Shay e06fda5b8b feat(runtime+evals): warm-path pack grounding + three long-span lanes

Step 1 — warm_grounding_stability targeted patch
- chat/runtime.py:_maybe_pack_grounded_surface accepts allow_warm=True;
  warm path invokes it after articulation and overrides
  response_surface / articulation / grounding_source when pack-grounded
  or teaching-grounded.
- CAUSE / VERIFICATION without a teaching chain on warm path emits the
  unknown-domain disclosure (matches cold-path discovery-signal doctrine
  — no fabricated vault content).
- warmed_session_consistency public lane: warm_grounding_stability
  0.0 → 1.0, grounding_match_rate 1.0, telemetry_consistency 1.0.
- Cognition lane byte-identical (public 100/100/91.7/100, holdout
  100/100/83.3/100).  Full suite 2294 passed.

Step 2 — three new red eval lanes (measurement substrate)
- conversational_thread_coherence: 6 cases / 45 turns; per-turn
  no_placeholder / not_walk_fragment / length / is_grounded predicates
  + per-case topic_anchor and no_topic_drift.  Baseline: grounded
  0.93, topic_anchor 0.50, no_topic_drift 0.83.
- multi_sentence_response: 15 cases over Explain/Tell/Describe/Walk/
  Example/Essay shapes; predicates sentence_count >= 2, non-fragment,
  connective_present, subject_named.  Baseline: multi_sentence 0.53,
  connective 0.10 — biggest architectural gap.
- self_consistency_over_time: 7 cases; same probe at multiple turn
  indices with unrelated fillers interleaved.  Baseline: byte_identical
  0.86 (one CAUSE-no-chain disclosure drifts under accumulation).

All three lanes deterministic, lexical-predicate-only — no LLM judge,
no embedding similarity.  Red-on-creation by design.  See
notes/long_span_fluency_baseline_2026-05-19.md.

2026-05-19 08:26:38 -07:00

2 KiB

Raw Permalink Blame History

Conversational Thread Coherence Eval Lane — Contract

Lane: conversational_thread_coherence Version: v1 Created: 2026-05-19 Status: Red on creation — measurement substrate for long-span fluency.

What this lane measures

Whether ChatRuntime maintains coherent grounding and topic continuity across an 8–12 turn thread that includes topic shifts, follow-up questions, and re-introductions of earlier subjects.

The asymmetric pair to warmed_session_consistency: that lane pins replay stability of identical prompts; this lane pins evolving conversation across structurally-different turns.

Per-turn predicates

Predicate	Definition
`no_placeholder`	surface contains none of: `...`, `<pending>`, `<prior>`, `<empty>`
`is_grounded`	`grounding_source` ∈ {pack, teaching, vault, oov, partial} (not `none`) on turns whose prompt is expected to ground
`not_walk_fragment`	surface has ≥ 4 alphabetic tokens AND ≥ 1 finite verb form
`length_adequate`	`len(surface.strip()) ≥ 20`

Per-case predicates

Predicate	Definition
`topic_anchor_present`	When a follow-up prompt does not name the prior topic explicitly (e.g. "What about that?"), the response surface mentions the prior topic's subject lemma OR explicitly discloses
`no_topic_drift_to_none`	After any `pack`/`teaching` turn, no subsequent turn on the SAME prompt-subject drops to `none` (would indicate state corruption)

Scoring rubric

per_turn_grounded_rate     = grounded_turns / total_turns
per_turn_not_fragment_rate = non_fragment_turns / total_turns
case_topic_anchor_rate     = cases_passing_anchor / cases_with_anaphora
case_no_drift_rate         = cases_passing_no_drift / replay_cases

Doctrine constraints

No LLM judge. All predicates are deterministic, lexical / structural.
No paraphrase equivalence by embedding. Lexical overlap only.
Red-on-creation is acceptable and expected — this lane is a target, not a regression gate (yet).

2 KiB Raw Permalink Blame History Unescape Escape