*** VERIFICATION PENDING — see the bottom of this message. ***
99 of the 107 full-suite failures are one test, red on clean main since
2026-07-23, and it is guarding the wrong field.
WHAT IS NOT WRONG. test_register_matrix_trace_hash_invariant — the load-bearing
ADR-0072 truth-path-isolation claim — passes on all 99 registers on pristine
main. trace_hash does not vary across the register axis. The invariant holds;
the seals and wrong=0 are untouched.
WHAT IS. test_register_matrix_canonical_surface_byte_identical asserts on
CognitiveTurnResult.surface, on its docstring's claim that this is "the
truth-path field compute_trace_hash consumes". That stopped being true when
Phase 0 flipped resolve_surface to response-first precedence and introduced
SurfaceResolution.hash_surface as the register-invariant capture. Since then
pipeline.py folds hash_surface into compute_trace_hash, while result.py
documents surface as "final voiced surface (what the user sees)".
So the test demanded that registers NOT differ on the served surface — the
opposite of what the register axis exists for, and a direct contradiction of
test_register_substantive_consumption.py, which pins that they DO differ and is
green in smoke. Two tests on main asserting opposite things about the same
bytes; the red one is in neither gate, so it stayed red.
ROOT CAUSE, and why this is a code fix rather than a test edit: hash_surface was
pipeline-local. The invariant was unobservable from a turn result, which is why
the test reached for the nearest visible field and drifted. Exposing it makes
the contract testable at the seam that owns it:
- CognitiveTurnResult.hash_surface — new field, populated from the value the
pipeline already computed
- evals/cognition CaseResult carries it through
- the test asserts byte-identity on hash_surface, with the history recorded so
it cannot drift back
No stored baseline to regenerate — the fixture computes it live.
[Verification]: NOT YET RUN. The register matrix is ~16 min and was still
executing when the working tree had to be committed; its output buffers through
tail, so no partial signal was available. Committed to preserve the work with an
honest label rather than to claim it passes. DO NOT MERGE until
`pytest tests/test_cognition_eval_register_matrix.py` is green — expected 0
failed where main has 99, with test_register_matrix_trace_hash_invariant still
passing. Result to follow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FduW6Krm3PPQv3P5iwBYtx
Implement the eval infrastructure defined in ADR-0016 before building new
eval lanes. This establishes the discipline that governs the entire
capability roadmap.
- Generic eval framework (evals/framework.py): lane discovery, versioned
scoring, result persistence
- Cognition lane retrofitted into new convention: 45 cases split into
stratified dev (13) / public v1 (13) / holdout (19) sets with contract,
runner, and recorded results
- Generalized `core eval <lane>` CLI: dynamic lane discovery, --list,
--version, --split, --save, --json flags
- Holdout runner scaffold: plaintext fallback, encryption interface ready
- Baseline runner scaffold: pluggable frontier model interface
- Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the
unknown-domain gate returned a stub without appending to turn_log
- ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log
Phase 0 exit audit found two methodology issues:
1. Pipeline turn_log crash (fixed here)
2. Versor drift in multi-turn sessions (pre-existing, under investigation)
