Replaces the flat "I don't know — insufficient grounding" disclosure
with a deterministic gradient that names specific vocabulary gaps
and gives operators concrete next steps.
P2.1 — OOV "teach me" surface (chat/oov_surface.py).
When the intent classifier extracts a clean subject lemma but that
lemma is not resident in any mounted lexicon pack, the runtime now
emits a deterministic learning-invitation surface tagged
``grounding_source="oov"`` instead of the universal disclosure.
Surface format (fixed template):
"I haven't learned '{token}' yet (intent: {intent}).
Mounted lexicon packs: {pack_list}.
Teach me via a reviewed PackMutationProposal."
The OOV token passes through ``core._safe_display.safe_display``
before persistence — user-input sanitization at the trust boundary.
No vocabulary is invented; no domain is inferred. Honours the
ADR-0027 proposal-only invariant: the surface invites a reviewed
pack mutation, never silently mutates any pack.
Refactored ``_maybe_pack_grounded_surface`` so every existing
intent branch (COMPARISON / CAUSE / VERIFICATION / CORRECTION /
PROCEDURE / DEFINITION+RECALL) falls through on a None composer
result instead of early-returning. The OOV invitation is the
deterministic fall-through for any clean-subject prompt whose
subject doesn't resolve.
P2.2 — Partial-grounding tier (chat/partial_surface.py).
When exactly one of two COMPARISON lemmas resolves, the runtime
emits a hedged surface that grounds the known side verbatim and
disclaims the OOV side explicitly:
"Whatever '{oov}' is, I can ground '{known}' — pack-grounded
({pack_id}): {d1}; {d2}. I cannot ground the comparison
without learning '{oov}' — teach me via a reviewed
PackMutationProposal."
Tagged ``grounding_source="partial"``. Falls through to OOV
invitation when both lemmas are OOV, and to full pack-grounded
COMPARISON when both resolve — partial is the middle tier in the
five-tier gradient.
Also normalises trailing sentence punctuation on
intent.secondary_subject at the COMPARISON boundary so prompts
like "Compare A and B." (with the period) still resolve B
correctly.
Five-tier gradient (vault → teaching → pack → partial → oov → none).
Test debt retired: four pre-existing tests asserted "OOV → universal
disclosure", which is exactly the contract P2.1/P2.2 inverted.
Rewritten to the new contract. Plus test_procedure_surface.py
gained a test for the OOV gradient on procedure intents.
Verification:
tests/test_oov_surface.py 22 passed
tests/test_partial_surface.py 16 passed
Cognition eval byte-identical:
public 100% / 100% / 91.7% / 100%
holdout 100% / 100% / 83.3% / 100%
Curated lanes all green.
151 lines
5.8 KiB
Python
151 lines
5.8 KiB
Python
"""ADR-0048 — pack-grounded fallback surface tests.
|
|
|
|
The contract these tests pin:
|
|
|
|
- Pack-grounded surfaces engage ONLY when the ``UnknownDomainGate``
|
|
fires with ``source="empty_vault"`` AND the intent is DEFINITION
|
|
or RECALL AND the subject lemma is in ``en_core_cognition_v1``.
|
|
- The surface is composed verbatim from the pack lexicon's
|
|
``semantic_domains`` and the lemma — no synthesis.
|
|
- The audit contract is preserved: ChatResponse and TurnEvent both
|
|
carry a ``grounding_source`` provenance tag set to ``"pack"`` on
|
|
the pack-grounded path, ``"none"`` on the universal disclosure,
|
|
and ``"vault"`` on the main walk path.
|
|
- Safety / ethics refusal still takes priority — pack-grounded
|
|
surfaces never bypass a SafetyVerdict violation.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from chat.pack_grounding import (
|
|
PACK_ID,
|
|
is_pack_lemma,
|
|
pack_grounded_surface,
|
|
)
|
|
from chat.runtime import ChatRuntime, _UNKNOWN_DOMAIN_SURFACE
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# pack_grounding module — pure-function contracts
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize("lemma", ["light", "knowledge", "meaning", "memory", "truth"])
|
|
def test_known_pack_lemmas_produce_grounded_surface(lemma: str) -> None:
|
|
surface = pack_grounded_surface(lemma)
|
|
assert surface is not None
|
|
assert lemma in surface
|
|
assert PACK_ID in surface
|
|
assert "No session evidence yet." in surface
|
|
|
|
|
|
def test_unknown_lemma_returns_none() -> None:
|
|
assert pack_grounded_surface("nonexistentwordxyz") is None
|
|
|
|
|
|
@pytest.mark.parametrize("bad", ["", " ", None])
|
|
def test_empty_or_invalid_lemma_returns_none(bad) -> None:
|
|
assert pack_grounded_surface(bad) is None # type: ignore[arg-type]
|
|
|
|
|
|
def test_is_pack_lemma_round_trips() -> None:
|
|
assert is_pack_lemma("light") is True
|
|
assert is_pack_lemma("nonexistentwordxyz") is False
|
|
assert is_pack_lemma("") is False
|
|
|
|
|
|
def test_surface_is_deterministic() -> None:
|
|
"""Same lemma must produce byte-identical surfaces on repeat calls
|
|
— pack is immutable, no randomness, no synthesis."""
|
|
a = pack_grounded_surface("light")
|
|
b = pack_grounded_surface("light")
|
|
assert a == b
|
|
assert a is not None
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ChatRuntime integration — cold-start path
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_cold_start_definition_returns_pack_grounded_surface() -> None:
|
|
"""Cold-start DEFINITION on a pack-known lemma routes through the
|
|
pack-grounded surface, not the universal disclosure."""
|
|
rt = ChatRuntime()
|
|
resp = rt.chat("What is light?")
|
|
assert "pack-grounded" in resp.surface
|
|
assert "light" in resp.surface
|
|
assert resp.grounding_source == "pack"
|
|
|
|
|
|
def test_cold_start_recall_returns_pack_grounded_surface() -> None:
|
|
"""RECALL intent on a pack-known lemma also engages the pack path."""
|
|
rt = ChatRuntime()
|
|
resp = rt.chat("Remember light")
|
|
assert resp.grounding_source == "pack"
|
|
assert "light" in resp.surface
|
|
|
|
|
|
def test_cold_start_unknown_lemma_routes_to_oov_invitation() -> None:
|
|
"""ADR-0065 / P2.1 — when the classifier extracts a clean subject
|
|
that is OOV, the runtime emits the OOV "teach me" invitation
|
|
surface instead of the universal disclosure.
|
|
|
|
``How can I quoxulate the wxyzabc?`` is PROCEDURE intent;
|
|
``quoxulate`` is OOV. Pre-P2.1 this produced the universal
|
|
disclosure; post-P2.1 it produces an OOV invitation naming the
|
|
unknown token + the mounted-pack list."""
|
|
rt = ChatRuntime()
|
|
resp = rt.chat("How can I quoxulate the wxyzabc?")
|
|
assert resp.grounding_source == "oov"
|
|
assert "quoxulate" in resp.surface or "wxyzabc" in resp.surface
|
|
assert "PackMutationProposal" in resp.surface
|
|
|
|
|
|
def test_cold_start_cause_on_oov_routes_to_oov_invitation() -> None:
|
|
"""ADR-0065 / P2.1 — CAUSE on an OOV subject also routes to the
|
|
OOV invitation, not the universal disclosure.
|
|
|
|
Pre-P2.1 these prompts went silent (CAUSE branch early-returned
|
|
None when no teaching chain existed). Post-P2.1 the runtime
|
|
explicitly names the gap."""
|
|
rt = ChatRuntime()
|
|
resp = rt.chat("Why does wxyzabc exist?")
|
|
assert resp.grounding_source == "oov"
|
|
assert "wxyzabc" in resp.surface
|
|
|
|
|
|
def test_turn_event_carries_grounding_source() -> None:
|
|
"""ADR-0048 provenance propagates to TurnEvent for downstream audit."""
|
|
rt = ChatRuntime()
|
|
rt.chat("What is light?")
|
|
last_event = rt.turn_log[-1]
|
|
assert getattr(last_event, "grounding_source", None) == "pack"
|
|
|
|
|
|
def test_chat_response_grounding_source_default_for_main_path() -> None:
|
|
"""When the walk path runs, the ChatResponse carries
|
|
``grounding_source="vault"``. We force the walk by priming the
|
|
vault with one turn so the second turn's gate clears."""
|
|
rt = ChatRuntime()
|
|
rt.chat("light truth") # seed vault with one known-token turn
|
|
resp = rt.chat("light truth")
|
|
# The second turn may or may not have vault hits depending on
|
|
# gate threshold; what we assert is that grounding_source is set
|
|
# to one of the documented values.
|
|
assert resp.grounding_source in {"vault", "pack", "none"}
|
|
|
|
|
|
def test_pack_grounded_surface_passes_safety_check() -> None:
|
|
"""Pack-grounded surfaces preserve the audit contract — safety
|
|
and ethics verdicts still surface and refusal still takes
|
|
priority above pack grounding when triggered."""
|
|
rt = ChatRuntime()
|
|
resp = rt.chat("What is light?")
|
|
# Safety verdict must be present on every stub-path response
|
|
# (ADR-0035). Specific verdict outcome depends on the safety
|
|
# pack — we only assert the audit contract holds.
|
|
assert resp.safety_verdict is not None
|
|
assert resp.ethics_verdict is not None
|