core/tests/test_pack_grounding.py
Shay 07da601641 feat(packs): seed 323 reviewed glosses across 9 English content packs
Phase C of the gloss feature.  Lands the natural-language gloss
content that the resolver (Phase B2) and the runtime composer
(Phase B3) were prepared for.  This is the user-visible payoff:
cold-start DEFINITION / RECALL prompts on pack-resident lemmas now
emit fluent grounded sentences instead of dotted-domain disclosure.

Authoring: five parallel subagents in ONE message block (a single
parallel dispatch, ~20s wall-clock vs ~95s sequential).  Each
subagent received its pack's complete lemma + POS list and a strict
JSON-shape exemplar.  Total returned: 326 raw gloss entries.

Assembly (this commit): the raw entries were partitioned by
lexicon-residency lookup (the resolve_gloss invariant enforced at
storage time), deduplicated within pack, sorted by lemma, written
to ``language_packs/data/<pack>/glosses.jsonl``, and each pack's
manifest received a new ``glosses_checksum`` field.  323 glosses
landed clean; 0 rejected.

Per-pack distribution:
  en_core_cognition_v1     78 glosses
  en_core_meta_v1          72 glosses
  en_core_attitude_v1      40 glosses
  en_core_temporal_v1      28 glosses
  en_core_action_v1        26 glosses
  en_core_quantitative_v1  24 glosses
  en_core_spatial_v1       24 glosses
  en_core_polarity_v1      16 glosses
  en_core_causation_v1     15 glosses

Live-probe lift (fresh ChatRuntime per prompt):

  BEFORE:
    truth — pack-grounded (en_core_cognition_v1):
      cognition.truth; logos.core; epistemic.ground.
      No session evidence yet.

  AFTER:
    Truth is a claim or state grounded by evidence and coherent
    judgment.  pack-grounded (en_core_cognition_v1).

Same provenance.  Same audit-trail content (the dotted domains are
still in lexicon.jsonl, the resolver can still read them, the
candidate object carries them verbatim).  But the user-facing
surface is a sentence the user can actually read.

Eval-lane lift:

  deterministic_fluency       BEFORE      AFTER
    no_dotted_inventory_rate  0.3333  →   1.0000
    no_provenance_only_rate   1.0000  →   1.0000  (held)
    no_placeholder_rate       1.0000  →   1.0000  (held)
    complete_punctuation_rate 1.0000  →   1.0000  (held)
    finite_predicate_shape    1.0000  →   1.0000  (held)
    surface_provenance_match  1.0000  →   1.0000  (held)
  cold_start_grounding         all metrics held at 1.0
  warmed_session_consistency   no_placeholder + telemetry_match held at 1.0
                              (warm_grounding_stability still 0 — separate fix)
  cognition eval public        100 / 100 / 91.7 / 100   (BYTE-IDENTICAL)
  cognition eval holdout       100 / 100 / 83.3 / 100   (BYTE-IDENTICAL)

  The cognition eval bytes-identity holds because the eval checks
  substring containment (case-insensitive after the format change).
  Every lemma still appears in its fluent surface.

Hardening this commit enforces:

  Lexicon-residency at storage time
    tests/test_pack_glosses_content.py::test_every_gloss_lemma_is_lexicon_resident
    walks every glosses.jsonl and asserts every lemma is present in
    the same pack's lexicon.jsonl.  Drift in glosses (an unratified
    lemma sneaking in) fails the lane immediately.

  Dual-checksum discipline
    tests/test_pack_glosses_content.py::test_every_glossed_pack_has_matching_checksum
    re-hashes glosses.jsonl bytes-on-disk and compares against the
    manifest's glosses_checksum.  Any tampering fails.

  Immutable-lexicon invariant
    tests/test_pack_glosses_content.py::test_lexicon_checksum_unchanged_by_gloss_landing
    re-hashes lexicon.jsonl and compares against the manifest's
    (original) checksum.  Proves that adding glosses did NOT perturb
    the lexicon seal.

  High-freq lemma resolution
    32 of the most-common conversational lemmas (truth, doubt,
    fact, idea, self, true, important, now, place, make, effect,
    always, ...) all resolve to a fluent surface end-to-end.

Test-suite drift this commit absorbed:

  - tests/test_pack_grounding.py — three substring assertions
    updated to be case-insensitive (gloss-backed surfaces capitalize
    lemmas at sentence start, dotted-disclosure surfaces don't).
    "No session evidence yet" assertion replaced with the
    common-substring "pack-grounded" marker that BOTH forms emit.
  - tests/test_pack_resolver_glosses.py — the back-compat test
    pivots from en_core_cognition_v1 (now glossed) to en_minimal_v1
    (deliberately unglossed).  A new test pins the glossed case.

Files added:
  language_packs/data/<pack>/glosses.jsonl  (9 files, 323 entries)
  tests/test_pack_glosses_content.py        (9 contract tests)

Files modified:
  language_packs/data/<pack>/manifest.json  (9 files, glosses_checksum field)
  chat/pack_grounding.py                    (lowercase "pack-grounded" tag)
  tests/test_pack_grounding.py              (3 substring assertions relaxed)
  tests/test_pack_resolver_glosses.py       (back-compat test pivoted)

Verification:
  127/127 affected tests green.
  9/9 new gloss-content tests green.
  All three eval lanes report the lift documented above.
  Cognition eval byte-identical.
2026-05-19 07:34:33 -07:00

158 lines
6.3 KiB
Python

"""ADR-0048 — pack-grounded fallback surface tests.
The contract these tests pin:
- Pack-grounded surfaces engage ONLY when the ``UnknownDomainGate``
fires with ``source="empty_vault"`` AND the intent is DEFINITION
or RECALL AND the subject lemma is in ``en_core_cognition_v1``.
- The surface is composed verbatim from the pack lexicon's
``semantic_domains`` and the lemma — no synthesis.
- The audit contract is preserved: ChatResponse and TurnEvent both
carry a ``grounding_source`` provenance tag set to ``"pack"`` on
the pack-grounded path, ``"none"`` on the universal disclosure,
and ``"vault"`` on the main walk path.
- Safety / ethics refusal still takes priority — pack-grounded
surfaces never bypass a SafetyVerdict violation.
"""
from __future__ import annotations
import pytest
from chat.pack_grounding import (
PACK_ID,
is_pack_lemma,
pack_grounded_surface,
)
from chat.runtime import ChatRuntime, _UNKNOWN_DOMAIN_SURFACE
# ---------------------------------------------------------------------------
# pack_grounding module — pure-function contracts
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("lemma", ["light", "knowledge", "meaning", "memory", "truth"])
def test_known_pack_lemmas_produce_grounded_surface(lemma: str) -> None:
surface = pack_grounded_surface(lemma)
assert surface is not None
# Case-insensitive: gloss-backed surfaces capitalize the lemma at
# the start of the sentence ("Truth is a claim..."), while the
# dotted-disclosure fallback keeps it lowercase. Both forms must
# contain the lemma.
assert lemma.lower() in surface.lower()
assert PACK_ID in surface
# Surface carries either the gloss-backed "pack-grounded (pack_id)."
# tag or the dotted-disclosure "No session evidence yet." trailer.
assert "pack-grounded" in surface
def test_unknown_lemma_returns_none() -> None:
assert pack_grounded_surface("nonexistentwordxyz") is None
@pytest.mark.parametrize("bad", ["", " ", None])
def test_empty_or_invalid_lemma_returns_none(bad) -> None:
assert pack_grounded_surface(bad) is None # type: ignore[arg-type]
def test_is_pack_lemma_round_trips() -> None:
assert is_pack_lemma("light") is True
assert is_pack_lemma("nonexistentwordxyz") is False
assert is_pack_lemma("") is False
def test_surface_is_deterministic() -> None:
"""Same lemma must produce byte-identical surfaces on repeat calls
— pack is immutable, no randomness, no synthesis."""
a = pack_grounded_surface("light")
b = pack_grounded_surface("light")
assert a == b
assert a is not None
# ---------------------------------------------------------------------------
# ChatRuntime integration — cold-start path
# ---------------------------------------------------------------------------
def test_cold_start_definition_returns_pack_grounded_surface() -> None:
"""Cold-start DEFINITION on a pack-known lemma routes through the
pack-grounded surface, not the universal disclosure."""
rt = ChatRuntime()
resp = rt.chat("What is light?")
assert "pack-grounded" in resp.surface
# Case-insensitive: fluent gloss surfaces capitalize at sentence start.
assert "light" in resp.surface.lower()
assert resp.grounding_source == "pack"
def test_cold_start_recall_returns_pack_grounded_surface() -> None:
"""RECALL intent on a pack-known lemma also engages the pack path."""
rt = ChatRuntime()
resp = rt.chat("Remember light")
assert resp.grounding_source == "pack"
assert "light" in resp.surface.lower()
def test_cold_start_unknown_lemma_routes_to_oov_invitation() -> None:
"""ADR-0065 / P2.1 — when the classifier extracts a clean subject
that is OOV, the runtime emits the OOV "teach me" invitation
surface instead of the universal disclosure.
``How can I quoxulate the wxyzabc?`` is PROCEDURE intent;
``quoxulate`` is OOV. Pre-P2.1 this produced the universal
disclosure; post-P2.1 it produces an OOV invitation naming the
unknown token + the mounted-pack list."""
rt = ChatRuntime()
resp = rt.chat("How can I quoxulate the wxyzabc?")
assert resp.grounding_source == "oov"
assert "quoxulate" in resp.surface or "wxyzabc" in resp.surface
assert "PackMutationProposal" in resp.surface
def test_cold_start_cause_on_oov_routes_to_oov_invitation() -> None:
"""ADR-0065 / P2.1 — CAUSE on an OOV subject also routes to the
OOV invitation, not the universal disclosure.
Pre-P2.1 these prompts went silent (CAUSE branch early-returned
None when no teaching chain existed). Post-P2.1 the runtime
explicitly names the gap."""
rt = ChatRuntime()
resp = rt.chat("Why does wxyzabc exist?")
assert resp.grounding_source == "oov"
assert "wxyzabc" in resp.surface
def test_turn_event_carries_grounding_source() -> None:
"""ADR-0048 provenance propagates to TurnEvent for downstream audit."""
rt = ChatRuntime()
rt.chat("What is light?")
last_event = rt.turn_log[-1]
assert getattr(last_event, "grounding_source", None) == "pack"
def test_chat_response_grounding_source_default_for_main_path() -> None:
"""When the walk path runs, the ChatResponse carries
``grounding_source="vault"``. We force the walk by priming the
vault with one turn so the second turn's gate clears."""
rt = ChatRuntime()
rt.chat("light truth") # seed vault with one known-token turn
resp = rt.chat("light truth")
# The second turn may or may not have vault hits depending on
# gate threshold; what we assert is that grounding_source is set
# to one of the documented values.
assert resp.grounding_source in {"vault", "pack", "none"}
def test_pack_grounded_surface_passes_safety_check() -> None:
"""Pack-grounded surfaces preserve the audit contract — safety
and ethics verdicts still surface and refusal still takes
priority above pack grounding when triggered."""
rt = ChatRuntime()
resp = rt.chat("What is light?")
# Safety verdict must be present on every stub-path response
# (ADR-0035). Specific verdict outcome depends on the safety
# pack — we only assert the audit contract holds.
assert resp.safety_verdict is not None
assert resp.ethics_verdict is not None