Shay e06fda5b8b feat(runtime+evals): warm-path pack grounding + three long-span lanes

Step 1 — warm_grounding_stability targeted patch
- chat/runtime.py:_maybe_pack_grounded_surface accepts allow_warm=True;
  warm path invokes it after articulation and overrides
  response_surface / articulation / grounding_source when pack-grounded
  or teaching-grounded.
- CAUSE / VERIFICATION without a teaching chain on warm path emits the
  unknown-domain disclosure (matches cold-path discovery-signal doctrine
  — no fabricated vault content).
- warmed_session_consistency public lane: warm_grounding_stability
  0.0 → 1.0, grounding_match_rate 1.0, telemetry_consistency 1.0.
- Cognition lane byte-identical (public 100/100/91.7/100, holdout
  100/100/83.3/100).  Full suite 2294 passed.

Step 2 — three new red eval lanes (measurement substrate)
- conversational_thread_coherence: 6 cases / 45 turns; per-turn
  no_placeholder / not_walk_fragment / length / is_grounded predicates
  + per-case topic_anchor and no_topic_drift.  Baseline: grounded
  0.93, topic_anchor 0.50, no_topic_drift 0.83.
- multi_sentence_response: 15 cases over Explain/Tell/Describe/Walk/
  Example/Essay shapes; predicates sentence_count >= 2, non-fragment,
  connective_present, subject_named.  Baseline: multi_sentence 0.53,
  connective 0.10 — biggest architectural gap.
- self_consistency_over_time: 7 cases; same probe at multiple turn
  indices with unrelated fillers interleaved.  Baseline: byte_identical
  0.86 (one CAUSE-no-chain disclosure drifts under accumulation).

All three lanes deterministic, lexical-predicate-only — no LLM judge,
no embedding similarity.  Red-on-creation by design.  See
notes/long_span_fluency_baseline_2026-05-19.md.

2026-05-19 08:26:38 -07:00

2 KiB

Raw Blame History

Multi-Sentence Response Eval Lane — Contract

Lane: multi_sentence_response Version: v1 Created: 2026-05-19 Status: Red on creation — measurement substrate for compositional surface.

What this lane measures

Whether ChatRuntime can emit a response that is more than a single sentence when the prompt structurally calls for elaboration ("Explain X", "Tell me about X", "Describe X", "Walk me through X").

Currently every pack-grounded surface is a single sentence emitted by _frame_gloss. NARRATIVE and EXAMPLE intents already compose multi-clause output via teaching chains, so they are tested here too as the only multi-sentence-capable code path.

Per-case predicates

Predicate	Definition
`sentence_count_>=_2`	the surface contains at least 2 terminated sentences (`.`, `?`, `!`)
`each_sentence_>=_4_tokens`	every sentence has ≥ 4 alphabetic tokens (no fragments)
`connective_present`	the surface contains at least one connective (`and`, `because`, `therefore`, `which`, `since`, `also`, `furthermore`, `however`, `consequently`) — only enforced when `expects_connective=true`
`not_just_provenance_tag`	sentence_count counts BEFORE the trailing provenance tag (`pack-grounded (…).`) is treated as its own sentence
`grounded`	`grounding_source` ∈ {pack, teaching}
`subject_named`	the prompt's subject lemma appears in the surface

Scoring rubric

multi_sentence_rate     = cases_with_>=2_sentences / total_cases
non_fragment_rate       = cases_where_every_sentence_>=4_tokens / total_cases
connective_present_rate = cases_with_connective / cases_expecting_connective

Doctrine constraints

The "trailing provenance tag" is structural, not a real sentence — predicate logic strips it before counting.
No LLM judge. Pure structural counting.
Red-on-creation expected: only NARRATIVE / EXAMPLE / cross-pack / composed_surface code paths can possibly satisfy sentence_count_>=_2 today.

2 KiB Raw Blame History