fix(evals): prompt_diversity gloss-quote heuristic — 4-token window → substring (#69)

The v1 gloss-quote detector used a 4-token contiguous window of
≥4-char tokens.  That heuristic was too strict for the actual ADR-0084
brief gloss style, which is deliberately short and primitive-only:

  light    "visible medium that reveal truth"   5 tokens ≥4 chars
  parent   "person with a child"                3 tokens ≥4 chars   ← can't window
  recall   "get memory from before"             3 tokens ≥4 chars   ← can't window
  wisdom   "good use of knowledge"              2 tokens ≥4 chars   ← can't window

Result: post-PR #65 baseline showed gloss_quote_rate=0.0% even though
the pack-grounded composer was visibly emitting glosses verbatim:

  surface: "Parent is person with a child. pack-grounded (en_core_relations_v1)."
  gloss:   "person with a child"
  window:  could not even form

Replace with substring match against the gloss text.  The composer
emits the gloss verbatim (no paraphrasing — that's the no-LLM
discipline), so substring is exact, high-confidence, and trivially
correct:

  gloss_quoted ⟺ gloss.lower().strip() in surface.lower()

Re-baselined v1/public (26 cases):
  gloss_quote_rate: 7.7% (false-positive 4-token window noise)
                  → 0.0% (post-#65, broken metric)
                  → 11.5% (this PR, real signal)

The other four metrics unchanged.  3/26 cases (DEFINITION on
``evidence``/``recall``/``parent``) are detected as gloss-quoted now,
which matches reality — the pack-grounded composer at
chat/pack_grounding.py:398 has been gloss-aware all along; it just
had no glosses to quote pre-#65.

Why this is just a heuristic refinement, not a contract change:

The contract.md still says v1 has NO pass thresholds beyond
versor_closure_rate==1.00.  The lane's job is to establish baseline
distribution.  The heuristic was *measuring the wrong thing* — fixing
the measurement is a contract clarification, not a contract change.

Tests added (TestGlossQuote, 4 cases):
  - short brief-style gloss detected via substring
  - chain-walk surface for same lemma NOT counted as gloss-quoted
  - unknown term returns False
  - empty terms returns False

Updated the function docstring with the post-#65 context so future
readers understand why v1's contract predicted 0% but reality is ~12%.
This commit is contained in:
Shay 2026-05-20 15:43:01 -07:00 committed by GitHub
parent 1938aaa674
commit 6b0d723987
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 66 additions and 13 deletions

View file

@ -189,12 +189,21 @@ def _surface_has_audit_leak(surface: str) -> bool:
def _surface_quotes_gloss(surface: str, expected_terms: tuple[str, ...]) -> bool:
"""Return True iff the surface visibly draws from a pack gloss.
Uses :func:`chat.pack_resolver.resolve_gloss` to fetch any gloss
bound to each expected term, then checks for a substantive
substring (4-token window) of the gloss in the surface.
Resolves each expected term via
:func:`chat.pack_resolver.resolve_gloss`, then asks: does the
surface contain the gloss text verbatim? The pack-grounded
composer emits the gloss without paraphrasing
(``"{Lemma} is {gloss}."``), so substring match is an exact and
high-confidence "gloss actually quoted" signal no fuzzy windows,
no false-positives from one shared content word.
v1 0% by design the composer does not consume glosses yet.
The metric exists so ADR-0085's lift is quantifiable.
Note on the v1 prediction: the contract predicted `` 0%`` here,
on the assumption that the composer would not consume glosses
until ADR-0085 landed. In fact the pack-grounded composer at
``chat/pack_grounding.py:398-434`` was *already* gloss-aware
pre-ADR-0084 but had no glosses to consume. Once PR #65's content
landed, the composer immediately started emitting glosses on
DEFINITION/RECALL. This metric now reflects that reality.
"""
if not expected_terms:
return False
@ -207,15 +216,10 @@ def _surface_quotes_gloss(surface: str, expected_terms: tuple[str, ...]) -> bool
if resolved is None:
continue
_pack_id, _pos, gloss = resolved
gloss_tokens = [t for t in re.findall(r"[a-z]+", gloss.lower()) if len(t) >= 4]
if not gloss_tokens:
if not gloss.strip():
continue
# 4-token contiguous window match — high-confidence "gloss
# actually quoted", not "shared one common word".
for i in range(len(gloss_tokens) - 3):
window = " ".join(gloss_tokens[i : i + 4])
if window in surface_lower:
return True
if gloss.lower().strip() in surface_lower:
return True
return False

View file

@ -82,6 +82,55 @@ class TestAuditLeak:
assert _surface_has_audit_leak(surface) is is_leak
# --------------------------------------------------------------------------- #
# Gloss-quote detector
# --------------------------------------------------------------------------- #
class TestGlossQuote:
"""The detector is exact-substring against the pack's gloss text,
not a fuzzy window. The pack-grounded composer emits gloss text
verbatim, so substring match is the right signal: zero false
positives, zero false negatives on brief-style short glosses where
a 4-token window would be impossible (e.g. ``person`` ``"person
with a child"`` has only 3 tokens ≥4 chars).
"""
def _make(self, surface: str, terms: tuple[str, ...]) -> bool:
from evals.prompt_diversity.runner import _surface_quotes_gloss
return _surface_quotes_gloss(surface, terms)
def test_quoted_short_gloss_detected(self) -> None:
# ``light`` gloss is ``"visible medium that reveal truth"`` —
# 5 tokens, but only 5 are ≥4 chars; the old 4-token window
# would barely fit. ``parent`` gloss is ``"person with a child"``
# — 4 tokens, 3 are ≥4 chars; the old window could never match.
# Substring match handles both natively.
assert self._make(
"Parent is person with a child. pack-grounded (en_core_relations_v1).",
("parent",),
) is True
assert self._make(
"Light is visible medium that reveal truth. pack-grounded (en_core_cognition_v1).",
("light",),
) is True
def test_unquoted_surface_returns_false(self) -> None:
# Chain-walk surface for the same lemma must NOT count as
# gloss-quoted — it shares vocabulary but doesn't quote the
# gloss itself.
assert self._make(
"light — teaching-grounded (cognition_chains_v1): cognition.illumination; logos.core.",
("light",),
) is False
def test_unknown_term_returns_false(self) -> None:
assert self._make("anything", ("nonsense_lemma_42",)) is False
def test_empty_terms_returns_false(self) -> None:
assert self._make("anything", ()) is False
# --------------------------------------------------------------------------- #
# End-to-end run on the v1 public split
# --------------------------------------------------------------------------- #