Shay a084f1db21 feat(evals): cold_start_grounding lane — 44-prompt routing probe

Commits the 2026-05-19 probe as a durable, replayable eval lane.
This is *step 1* of the gloss-feature rollout sequence agreed
upstream: establish a stable measurement substrate before any
further intent/grounding changes, so the 52%→0% lift (and any
future regression) is reproducible and CI-pinned.

The lane is deliberately named ``cold_start_grounding`` rather than
``fluency``:
  - It measures **routing** (intent → grounding source), not
    sentence quality, morphology, or surface diversity.
  - The cold-start qualifier reflects the fresh-``ChatRuntime()``-
    per-case design.  Re-using a runtime across cases would
    contaminate the vault from earlier turns and was the exact bug
    observed during the probe before the per-case-runtime fix.

Files:

  evals/cold_start_grounding/contract.md
    Lane contract: what is measured, scoring rubric, pass thresholds
    (intent ≥ 0.95 / grounding ≥ 0.95 / subject ≥ 0.90), and the
    rationale for the deliberate non-fallback on CAUSE/VERIFICATION
    without teaching chains.
  evals/cold_start_grounding/public/v1/cases.jsonl
    44 cases across 16 categories.  Each case carries id, prompt,
    category, expected_intent, expected_grounding_source, and an
    optional expected_subject.  Categories cover every intent
    pattern fixed in b52e04a (Define, What-does-X-mean, infinitive,
    How-does-X-work, What-causes-X) plus OOV controls and CAUSE
    cases with/without teaching chains.
  evals/cold_start_grounding/dev/cases.jsonl
    5 representative cases for fast local iteration.
  evals/cold_start_grounding/runner.py
    Framework-compliant ``run_lane(cases, config=None) -> LaneReport``.
    Constructs a fresh ChatRuntime() inside ``_run_case`` (cold-start
    invariant).  Emits intent_accuracy, grounding_accuracy,
    subject_accuracy, full grounding distributions, and a per-
    category breakdown for regression attribution.
  tests/test_cold_start_grounding_lane.py
    16 contract tests covering: case-set integrity, valid enum
    values, unique ids, lane discovery, pass thresholds, expected-
    vs-actual distribution match (drift detection), the architectural
    invariants on oov_control and cause_no_teaching_chain cases, the
    cold-start invariant (static check that the runner constructs
    ChatRuntime() inside the per-case helper, not at module scope),
    and result JSON-serialization round-trip.

Baseline metrics (this commit, all v1 public cases):
  intent_accuracy:    1.0000  (44/44)
  grounding_accuracy: 1.0000  (44/44)
  subject_accuracy:   1.0000  (44/44)

  grounding distribution (actual == expected exactly):
    pack:      37
    oov:        4
    teaching:   1
    none:       2  (deliberate — CAUSE without teaching chain)

Why "none" cases are *expected* to ground as none:
  CAUSE / VERIFICATION on a pack-resident lemma WITHOUT an active
  teaching chain stays grounding_source='none' on purpose.  Falling
  through to pack_grounded_surface here would mask the discovery-
  candidate signal the teaching pipeline uses to identify chains
  worth authoring.  The contract test in
  TestArchitecturalInvariants::test_cause_no_chain_cases_route_to_none
  pins this doctrine.

Verification: 16/16 lane tests green; full lane run via
``core eval cold_start_grounding`` reports 100% on every metric.

Subsequent steps in the agreed sequence (NOT in this commit):
  2. Hygiene: runtime API wrappers (achat/arespond/respond) + the
     stale CAUSE/VERIFICATION docstring in
     tests/test_intent_classification_extensions.py.
  3. Harden gloss resolver in feat/pack-glosses-wip
     (lexicon-residency check, dual checksum, cache clearing,
     malformed-JSONL skip tests).
  4. Wire gloss-backed pack_grounded_surface().
  5. Author starter glosses with checksum discipline.

2026-05-19 06:33:42 -07:00

3.4 KiB

Raw Blame History

Cold-Start Grounding Eval Lane — Contract

Lane: cold_start_grounding Version: v1 Created: 2026-05-19

What this lane measures

Cold-start routing of conversational prompts to the correct grounding source. Each case is fed through a fresh ChatRuntime() (no vault accumulation, no prior turn) and the runtime's ChatResponse.grounding_source is compared against the case's expected_grounding_source.

This is a routing probe, not a fluency probe. It does not score sentence quality, morphology, or surface diversity. It scores:

"For a realistic conversational prompt about a pack-resident lemma, does the runtime correctly route to a pack/teaching surface — and for a genuinely OOV lemma or an honest knowledge gap, does it route to OOV/none?"

Two architectural invariants are pinned by this lane:

Pack-resident DEFINITION subjects always route to pack.
CAUSE / VERIFICATION subjects without an active teaching chain stay none (deliberate non-fallback — preserves the discovery-candidate signal the teaching pipeline uses).

Scoring rubric

Each case produces three binary signals:

Signal	Definition
`intent_match`	`actual_intent.tag.value == expected_intent`
`grounding_match`	`actual_grounding_source == expected_grounding_source`
`subject_match`	`actual_intent.subject == expected_subject` (optional; only checked when case carries `expected_subject`)

Lane-level metrics:

Metric	Definition	v1 pass threshold
`grounding_accuracy`	Fraction of cases with correct grounding source	>= 0.95
`intent_accuracy`	Fraction of cases with correct intent tag	>= 0.95
`subject_accuracy`	Fraction of cases with correct extracted subject (subset that asserts subject)	>= 0.90

Pass criteria

All three thresholds satisfied on the public v1 split.

Cold-start invariant

The runner constructs a new ChatRuntime() for every case. This is deliberate: the lane measures cold-start routing, not multi-turn accumulation behaviour. Re-using a runtime across cases would contaminate vault content from earlier prompts (this is exactly the bug observed during the 2026-05-19 probe — when the same runtime processed multiple prompts the vault grounding source overrode the pack source on later turns, producing garbled surfaces).

Why this lane exists

The 2026-05-19 cumulative live probe surfaced that ~52% of realistic conversational DEFINITION prompts on pack-resident lemmas were returning grounding_source="none". The bottleneck was intent classification + subject extraction, not lexicon coverage. Five specific patterns (Define X, What does X mean?, What is to V?, How does X work?, What causes X?) had no rule or routed to an intent the runtime dispatcher couldn't handle.

This lane commits that probe set as a durable, replayable artifact so the lift is reproducible and any future regression in intent routing fails the lane immediately.

Case schema

{
  "id": "definition_doubt_001",
  "prompt": "What is doubt?",
  "category": "definition_meta_pack",
  "expected_intent": "definition",
  "expected_grounding_source": "pack",
  "expected_subject": "doubt"
}

expected_grounding_source is one of: pack, teaching, oov, none, vault, partial.

expected_subject is optional; when present it pins the extracted-subject invariant.

category is freeform and used to group cases in reports.

3.4 KiB Raw Blame History