core/evals/prompt_diversity
Shay 6b0d723987
fix(evals): prompt_diversity gloss-quote heuristic — 4-token window → substring (#69)
The v1 gloss-quote detector used a 4-token contiguous window of
≥4-char tokens.  That heuristic was too strict for the actual ADR-0084
brief gloss style, which is deliberately short and primitive-only:

  light    "visible medium that reveal truth"   5 tokens ≥4 chars
  parent   "person with a child"                3 tokens ≥4 chars   ← can't window
  recall   "get memory from before"             3 tokens ≥4 chars   ← can't window
  wisdom   "good use of knowledge"              2 tokens ≥4 chars   ← can't window

Result: post-PR #65 baseline showed gloss_quote_rate=0.0% even though
the pack-grounded composer was visibly emitting glosses verbatim:

  surface: "Parent is person with a child. pack-grounded (en_core_relations_v1)."
  gloss:   "person with a child"
  window:  could not even form

Replace with substring match against the gloss text.  The composer
emits the gloss verbatim (no paraphrasing — that's the no-LLM
discipline), so substring is exact, high-confidence, and trivially
correct:

  gloss_quoted ⟺ gloss.lower().strip() in surface.lower()

Re-baselined v1/public (26 cases):
  gloss_quote_rate: 7.7% (false-positive 4-token window noise)
                  → 0.0% (post-#65, broken metric)
                  → 11.5% (this PR, real signal)

The other four metrics unchanged.  3/26 cases (DEFINITION on
``evidence``/``recall``/``parent``) are detected as gloss-quoted now,
which matches reality — the pack-grounded composer at
chat/pack_grounding.py:398 has been gloss-aware all along; it just
had no glosses to quote pre-#65.

Why this is just a heuristic refinement, not a contract change:

The contract.md still says v1 has NO pass thresholds beyond
versor_closure_rate==1.00.  The lane's job is to establish baseline
distribution.  The heuristic was *measuring the wrong thing* — fixing
the measurement is a contract clarification, not a contract change.

Tests added (TestGlossQuote, 4 cases):
  - short brief-style gloss detected via substring
  - chain-walk surface for same lemma NOT counted as gloss-quoted
  - unknown term returns False
  - empty terms returns False

Updated the function docstring with the post-#65 context so future
readers understand why v1's contract predicted 0% but reality is ~12%.
2026-05-20 15:43:01 -07:00
..
dev feat(adr-0084): definitional layer — proposal + substrate (schema/loader/closure) (#64) 2026-05-20 15:25:25 -07:00
public/v1 feat(adr-0084): definitional layer — proposal + substrate (schema/loader/closure) (#64) 2026-05-20 15:25:25 -07:00
__init__.py feat(adr-0084): definitional layer — proposal + substrate (schema/loader/closure) (#64) 2026-05-20 15:25:25 -07:00
contract.md feat(adr-0084): definitional layer — proposal + substrate (schema/loader/closure) (#64) 2026-05-20 15:25:25 -07:00
runner.py fix(evals): prompt_diversity gloss-quote heuristic — 4-token window → substring (#69) 2026-05-20 15:43:01 -07:00