Shay 0cf1a8fdc4 feat(evals): warmed_session_consistency lane — pipeline override regression substrate

Asymmetric counterpart to cold_start_grounding.  Builds the
measurement substrate for the Phase B1 pipeline-override usefulness
gate.  Lane is committed now (red baseline measured) so the fix is
landed against a fixed regression target.

The 2026-05-19 design review surfaced the bug this lane catches:

  > pipeline overrode a runtime surface with a placeholder realizer
  > surface because realized_plan.surface was non-empty, even though
  > it contained '...'.  The runtime audit log still held a different
  > surface.  This is the central fluency/design fault: the system
  > can be "green" while user-facing selection, pipeline selection,
  > and telemetry selection disagree.

The lane reproduces this exactly on the current main:

  Surface "Soon is defined as ..." emitted on turn 2 of "What does
  soon mean?" (where turn 1 grounded as pack correctly).  Telemetry
  recorded a different surface than the pipeline returned.

Initial red baseline (THIS commit):
  no_placeholder_rate        = 0.4444  (target after Phase B1: 1.00)
  telemetry_consistency_rate = 0.4444  (target after Phase B1: 1.00)
  warm_grounding_stability   = 0.0000  (target after Phase B1: >=0.95)

Cold-start-grounding stays at 1.00 on its own metrics.  The cold lane
measures routing, the warmed lane measures override discipline; they
are deliberately not the same.

Files:
  evals/warmed_session_consistency/contract.md
    What is measured, why, and the asymmetry with cold_start_grounding.
    Documents the four binary per-turn signals (no_placeholder,
    pipeline_match_telemetry, pipeline_match_walk, grounded_holds_on_warm)
    and the per-case warm_grounding_stable invariant.
  evals/warmed_session_consistency/public/v1/cases.jsonl
    8 cases / 18 turns.  Mix of:
      - replay-the-same-prompt (catches override drift)
      - mixed-intent sequences (catches OOV / pack interaction)
      - cause-no-chain (must stay none across replays)
      - what-does-x-mean (the warmed variant of the cold-start test)
  evals/warmed_session_consistency/dev/cases.jsonl
    2 representative cases for fast iteration.
  evals/warmed_session_consistency/runner.py
    Framework-compliant run_lane(cases, config=None) -> LaneReport.
    Constructs ONE ChatRuntime + CognitiveTurnPipeline per case,
    plays the turn sequence through them.  Per-turn signals:
      no_placeholder       — surface free of ..., <pending>, <prior>
      telemetry_match      — pipeline result.surface == turn_log[-1].surface
      grounding_match      — actual_grounding == expected_grounding
    Per-case signal:
      warm_grounding_stable — every replayed prompt produces the same
                              grounding across turns
  tests/test_warmed_session_lane.py
    8 contract tests covering: case-set integrity, replay-pattern
    presence, lane discovery, runner emits every required metric,
    per-turn details carry all signals, and the warmed-runtime
    invariant (static check that ChatRuntime is constructed
    per-case, not per-turn and not module-scope).

NOT pinned in this commit (deliberate):
  Threshold assertions are NOT in the test file.  They will land in
  Phase B1 alongside the pipeline-override usefulness gate.  This
  lane's role at present is to PROVIDE the regression target, not
  to enforce it before the fix.

Verification: 8/8 lane tests green; the lane itself runs and emits
the red metrics documented above.

2026-05-19 07:13:41 -07:00

4 KiB

Raw Blame History

Warmed-Session Consistency Eval Lane — Contract

Lane: warmed_session_consistency Version: v1 Created: 2026-05-19

What this lane measures

Pipeline / runtime / telemetry surface consistency across a warmed session — i.e. a single CognitiveTurnPipeline running multiple turns through the same ChatRuntime, where vault state, prior-turn context, and session-level memory accumulate.

This is the asymmetric counterpart to cold_start_grounding. Cold start measures routing on fresh runtimes. Warmed session measures whether accumulated state, pipeline overrides, or telemetry-vs-final- result drift can corrupt an answer that was correctly grounded at the runtime level.

The 2026-05-19 design review surfaced the bug this lane pins:

first pipeline run: truth - pack-grounded (...). second pipeline run: Truth is defined as ... second walk: Truth infer. runtime turn log: Truth infer.

The pipeline's realized_plan.surface overrode a perfectly good runtime-level pack-grounded surface with a <pending>-rendered placeholder. Telemetry recorded yet a third surface.

Scoring rubric

Each case runs N turns through one CognitiveTurnPipeline / ChatRuntime pair. After every turn the lane checks:

Signal	Definition
`no_placeholder`	surface contains none of: `...`, `<pending>`, `<prior>`, `placeholder`
`pipeline_match_telemetry`	`result.surface` equals the surface in the most recent `runtime.turn_log` entry
`pipeline_match_walk`	`result.surface` is either equal to `result.walk_surface` (when the pipeline did not override) OR is the realized-plan output (when it did and the override was a useful surface)
`grounded_holds_on_warm`	a turn that grounded as `pack`/`teaching` on turn 1 must not regress to `none`/`vault`/walk-fragment on subsequent turns for the same prompt

Lane-level metrics (rates over all (case × turn) pairs):

Metric	Definition	v1 pass threshold
`no_placeholder_rate`	fraction of turns whose surface contains no placeholder	1.00
`telemetry_consistency_rate`	fraction of turns where pipeline-final == turn_log-emitted	1.00
`warm_grounding_stability`	fraction of replayed prompts whose grounding_source is byte-identical across all replays	>= 0.95

no_placeholder_rate and telemetry_consistency_rate are hard 1.00 because either failure is a doctrine violation, not a tunable metric.

Why cold-start alone is not enough

cold_start_grounding constructs a fresh ChatRuntime per case to isolate routing logic. That is correct for what it measures. But it hides every bug class that requires accumulated state:

pipeline overriding a good runtime surface with a worse realizer one
telemetry emitting a surface that doesn't match the final pipeline return value
vault accumulation winning over pack grounding on later turns
correction-pass injecting a backward perturbation that mis-attributes to the wrong turn

The warmed-session lane runs these exact paths and asserts they hold.

Cold-start invariant — INVERTED here

Unlike cold_start_grounding, this lane DOES NOT construct a fresh runtime per case. Each case explicitly carries a turn sequence; the runner constructs one runtime and one pipeline, then plays the sequence through them. Each turn's expected behavior may reference prior-turn state (e.g. "after a CORRECTION on turn 2, turn 3's surface must still ground correctly on a fresh DEFINITION prompt").

Case schema

{
  "id": "warm_replay_truth_001",
  "category": "pipeline_override_no_placeholder",
  "turns": [
    {"prompt": "What is truth?", "expected_grounding_source": "pack"},
    {"prompt": "What is truth?", "expected_grounding_source": "pack"}
  ],
  "warm_invariants": ["no_placeholder", "pipeline_match_telemetry"]
}

Each case carries a turns list (ordered). Optional warm_invariants names the subset of contract checks to enforce on this case (default: all four).

4 KiB Raw Blame History Unescape Escape