Asymmetric counterpart to cold_start_grounding. Builds the
measurement substrate for the Phase B1 pipeline-override usefulness
gate. Lane is committed now (red baseline measured) so the fix is
landed against a fixed regression target.
The 2026-05-19 design review surfaced the bug this lane catches:
> pipeline overrode a runtime surface with a placeholder realizer
> surface because realized_plan.surface was non-empty, even though
> it contained '...'. The runtime audit log still held a different
> surface. This is the central fluency/design fault: the system
> can be "green" while user-facing selection, pipeline selection,
> and telemetry selection disagree.
The lane reproduces this exactly on the current main:
Surface "Soon is defined as ..." emitted on turn 2 of "What does
soon mean?" (where turn 1 grounded as pack correctly). Telemetry
recorded a different surface than the pipeline returned.
Initial red baseline (THIS commit):
no_placeholder_rate = 0.4444 (target after Phase B1: 1.00)
telemetry_consistency_rate = 0.4444 (target after Phase B1: 1.00)
warm_grounding_stability = 0.0000 (target after Phase B1: >=0.95)
Cold-start-grounding stays at 1.00 on its own metrics. The cold lane
measures routing, the warmed lane measures override discipline; they
are deliberately not the same.
Files:
evals/warmed_session_consistency/contract.md
What is measured, why, and the asymmetry with cold_start_grounding.
Documents the four binary per-turn signals (no_placeholder,
pipeline_match_telemetry, pipeline_match_walk, grounded_holds_on_warm)
and the per-case warm_grounding_stable invariant.
evals/warmed_session_consistency/public/v1/cases.jsonl
8 cases / 18 turns. Mix of:
- replay-the-same-prompt (catches override drift)
- mixed-intent sequences (catches OOV / pack interaction)
- cause-no-chain (must stay none across replays)
- what-does-x-mean (the warmed variant of the cold-start test)
evals/warmed_session_consistency/dev/cases.jsonl
2 representative cases for fast iteration.
evals/warmed_session_consistency/runner.py
Framework-compliant run_lane(cases, config=None) -> LaneReport.
Constructs ONE ChatRuntime + CognitiveTurnPipeline per case,
plays the turn sequence through them. Per-turn signals:
no_placeholder — surface free of ..., <pending>, <prior>
telemetry_match — pipeline result.surface == turn_log[-1].surface
grounding_match — actual_grounding == expected_grounding
Per-case signal:
warm_grounding_stable — every replayed prompt produces the same
grounding across turns
tests/test_warmed_session_lane.py
8 contract tests covering: case-set integrity, replay-pattern
presence, lane discovery, runner emits every required metric,
per-turn details carry all signals, and the warmed-runtime
invariant (static check that ChatRuntime is constructed
per-case, not per-turn and not module-scope).
NOT pinned in this commit (deliberate):
Threshold assertions are NOT in the test file. They will land in
Phase B1 alongside the pipeline-override usefulness gate. This
lane's role at present is to PROVIDE the regression target, not
to enforce it before the fix.
Verification: 8/8 lane tests green; the lane itself runs and emits
the red metrics documented above.
Shay