core/tests/test_pack_grounding.py
Shay 51aad0c2cd feat(adr-0065): OOV cliff → five-tier honesty gradient (Phase 2.1 + 2.2)
Replaces the flat "I don't know — insufficient grounding" disclosure
with a deterministic gradient that names specific vocabulary gaps
and gives operators concrete next steps.

P2.1 — OOV "teach me" surface (chat/oov_surface.py).

  When the intent classifier extracts a clean subject lemma but that
  lemma is not resident in any mounted lexicon pack, the runtime now
  emits a deterministic learning-invitation surface tagged
  ``grounding_source="oov"`` instead of the universal disclosure.

  Surface format (fixed template):

    "I haven't learned '{token}' yet (intent: {intent}).
     Mounted lexicon packs: {pack_list}.
     Teach me via a reviewed PackMutationProposal."

  The OOV token passes through ``core._safe_display.safe_display``
  before persistence — user-input sanitization at the trust boundary.
  No vocabulary is invented; no domain is inferred.  Honours the
  ADR-0027 proposal-only invariant: the surface invites a reviewed
  pack mutation, never silently mutates any pack.

  Refactored ``_maybe_pack_grounded_surface`` so every existing
  intent branch (COMPARISON / CAUSE / VERIFICATION / CORRECTION /
  PROCEDURE / DEFINITION+RECALL) falls through on a None composer
  result instead of early-returning.  The OOV invitation is the
  deterministic fall-through for any clean-subject prompt whose
  subject doesn't resolve.

P2.2 — Partial-grounding tier (chat/partial_surface.py).

  When exactly one of two COMPARISON lemmas resolves, the runtime
  emits a hedged surface that grounds the known side verbatim and
  disclaims the OOV side explicitly:

    "Whatever '{oov}' is, I can ground '{known}' — pack-grounded
     ({pack_id}): {d1}; {d2}.  I cannot ground the comparison
     without learning '{oov}' — teach me via a reviewed
     PackMutationProposal."

  Tagged ``grounding_source="partial"``.  Falls through to OOV
  invitation when both lemmas are OOV, and to full pack-grounded
  COMPARISON when both resolve — partial is the middle tier in the
  five-tier gradient.

  Also normalises trailing sentence punctuation on
  intent.secondary_subject at the COMPARISON boundary so prompts
  like "Compare A and B." (with the period) still resolve B
  correctly.

Five-tier gradient (vault → teaching → pack → partial → oov → none).

Test debt retired: four pre-existing tests asserted "OOV → universal
disclosure", which is exactly the contract P2.1/P2.2 inverted.
Rewritten to the new contract.  Plus test_procedure_surface.py
gained a test for the OOV gradient on procedure intents.

Verification:
  tests/test_oov_surface.py                       22 passed
  tests/test_partial_surface.py                   16 passed
  Cognition eval byte-identical:
    public  100% / 100% / 91.7% / 100%
    holdout 100% / 100% / 83.3% / 100%
  Curated lanes all green.
2026-05-18 16:41:45 -07:00

151 lines
5.8 KiB
Python

"""ADR-0048 — pack-grounded fallback surface tests.
The contract these tests pin:
- Pack-grounded surfaces engage ONLY when the ``UnknownDomainGate``
fires with ``source="empty_vault"`` AND the intent is DEFINITION
or RECALL AND the subject lemma is in ``en_core_cognition_v1``.
- The surface is composed verbatim from the pack lexicon's
``semantic_domains`` and the lemma — no synthesis.
- The audit contract is preserved: ChatResponse and TurnEvent both
carry a ``grounding_source`` provenance tag set to ``"pack"`` on
the pack-grounded path, ``"none"`` on the universal disclosure,
and ``"vault"`` on the main walk path.
- Safety / ethics refusal still takes priority — pack-grounded
surfaces never bypass a SafetyVerdict violation.
"""
from __future__ import annotations
import pytest
from chat.pack_grounding import (
PACK_ID,
is_pack_lemma,
pack_grounded_surface,
)
from chat.runtime import ChatRuntime, _UNKNOWN_DOMAIN_SURFACE
# ---------------------------------------------------------------------------
# pack_grounding module — pure-function contracts
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("lemma", ["light", "knowledge", "meaning", "memory", "truth"])
def test_known_pack_lemmas_produce_grounded_surface(lemma: str) -> None:
surface = pack_grounded_surface(lemma)
assert surface is not None
assert lemma in surface
assert PACK_ID in surface
assert "No session evidence yet." in surface
def test_unknown_lemma_returns_none() -> None:
assert pack_grounded_surface("nonexistentwordxyz") is None
@pytest.mark.parametrize("bad", ["", " ", None])
def test_empty_or_invalid_lemma_returns_none(bad) -> None:
assert pack_grounded_surface(bad) is None # type: ignore[arg-type]
def test_is_pack_lemma_round_trips() -> None:
assert is_pack_lemma("light") is True
assert is_pack_lemma("nonexistentwordxyz") is False
assert is_pack_lemma("") is False
def test_surface_is_deterministic() -> None:
"""Same lemma must produce byte-identical surfaces on repeat calls
— pack is immutable, no randomness, no synthesis."""
a = pack_grounded_surface("light")
b = pack_grounded_surface("light")
assert a == b
assert a is not None
# ---------------------------------------------------------------------------
# ChatRuntime integration — cold-start path
# ---------------------------------------------------------------------------
def test_cold_start_definition_returns_pack_grounded_surface() -> None:
"""Cold-start DEFINITION on a pack-known lemma routes through the
pack-grounded surface, not the universal disclosure."""
rt = ChatRuntime()
resp = rt.chat("What is light?")
assert "pack-grounded" in resp.surface
assert "light" in resp.surface
assert resp.grounding_source == "pack"
def test_cold_start_recall_returns_pack_grounded_surface() -> None:
"""RECALL intent on a pack-known lemma also engages the pack path."""
rt = ChatRuntime()
resp = rt.chat("Remember light")
assert resp.grounding_source == "pack"
assert "light" in resp.surface
def test_cold_start_unknown_lemma_routes_to_oov_invitation() -> None:
"""ADR-0065 / P2.1 — when the classifier extracts a clean subject
that is OOV, the runtime emits the OOV "teach me" invitation
surface instead of the universal disclosure.
``How can I quoxulate the wxyzabc?`` is PROCEDURE intent;
``quoxulate`` is OOV. Pre-P2.1 this produced the universal
disclosure; post-P2.1 it produces an OOV invitation naming the
unknown token + the mounted-pack list."""
rt = ChatRuntime()
resp = rt.chat("How can I quoxulate the wxyzabc?")
assert resp.grounding_source == "oov"
assert "quoxulate" in resp.surface or "wxyzabc" in resp.surface
assert "PackMutationProposal" in resp.surface
def test_cold_start_cause_on_oov_routes_to_oov_invitation() -> None:
"""ADR-0065 / P2.1 — CAUSE on an OOV subject also routes to the
OOV invitation, not the universal disclosure.
Pre-P2.1 these prompts went silent (CAUSE branch early-returned
None when no teaching chain existed). Post-P2.1 the runtime
explicitly names the gap."""
rt = ChatRuntime()
resp = rt.chat("Why does wxyzabc exist?")
assert resp.grounding_source == "oov"
assert "wxyzabc" in resp.surface
def test_turn_event_carries_grounding_source() -> None:
"""ADR-0048 provenance propagates to TurnEvent for downstream audit."""
rt = ChatRuntime()
rt.chat("What is light?")
last_event = rt.turn_log[-1]
assert getattr(last_event, "grounding_source", None) == "pack"
def test_chat_response_grounding_source_default_for_main_path() -> None:
"""When the walk path runs, the ChatResponse carries
``grounding_source="vault"``. We force the walk by priming the
vault with one turn so the second turn's gate clears."""
rt = ChatRuntime()
rt.chat("light truth") # seed vault with one known-token turn
resp = rt.chat("light truth")
# The second turn may or may not have vault hits depending on
# gate threshold; what we assert is that grounding_source is set
# to one of the documented values.
assert resp.grounding_source in {"vault", "pack", "none"}
def test_pack_grounded_surface_passes_safety_check() -> None:
"""Pack-grounded surfaces preserve the audit contract — safety
and ethics verdicts still surface and refusal still takes
priority above pack grounding when triggered."""
rt = ChatRuntime()
resp = rt.chat("What is light?")
# Safety verdict must be present on every stub-path response
# (ADR-0035). Specific verdict outcome depends on the safety
# pack — we only assert the audit contract holds.
assert resp.safety_verdict is not None
assert resp.ethics_verdict is not None