Three lane SHA pins drifted because intentional surface/serialization
changes shipped without re-running scripts/verify_lane_shas.py --update.
Bisect attributing the drift:
- demo_composition + public_demo broke at 5cad0a4 (#118 ADR-0110
mathematics_logic → expert_demo) — the demos enumerate the expert set.
- demo_composition drifted a second time at ab4c7cb (#220 Phase 3
state tagging spine) — additional epistemic fields shifted the surface.
- domain_contract_validation broke at a45eab1 (#219 Phase 2 epistemic
bug repairs) — normative/epistemic field shape changed.
The in-tree canonical report for fabrication_control_summary was also
stale vs. its (correct) pin; refreshed here for byte-alignment.
After this commit: 7/7 lanes match pinned SHAs; verify_lane_shas.py
runs green locally and in CI.
Followup (separate PR): hook/template guard so future PRs that touch
core/cognition/result.py, chat/runtime.py, or capability registries
re-run --update before merge.
First negative-control measure. Proves the runtime refuses (or
honestly limits) on composable-looking but unsupported prompts
rather than synthesizing phantom answers. Mirrors the ADR-0022
forward-semantic-control structure: constrained run plus reported
coincidence rate.
- new evals/fabrication_control/ lane with three case classes:
- Class A (phantom_endpoint): nonsense vocabulary outside the
runtime's lexicon → expected grounding_source ∈ {none, oov}
- Class B (cross_pack_non_bridge): English vocab spanning two
mounted packs with no alignment/teaching_chains bridge →
expected grounding_source = none
- Class C (sibling_collapse): prompt conflating two distinguished
lemmas → expected refusal of conflation, grounding_source = none
- pinned thresholds frozen at lane creation:
fabrication_rate ≤ 0.01, refusal_recall ≥ 0.95,
trace_evidence_present == 1.00,
grounding_source_matches_expected == 1.00
- three-set discipline per docs/capability_roadmap.md Rule 1:
cases/dev.jsonl (12 cases, 4/class), cases/public.jsonl (9 cases),
cases/holdout.jsonl (empty — reserved for first version cut)
- runner.py drives each case through ChatRuntime.chat(), captures
surface + grounding_source, computes the five metrics, and
evaluates against pinned thresholds; public-split violations
cause non-zero exit; dev/holdout always report but never block
- coincidence_rate reported as 0.0 with a note that unconstrained
baseline is reserved for future comparison (the current runtime
is fully constrained)
- 30 unit tests covering refusal/fabrication marker detection,
metric computation, threshold evaluation, case loading, plus a
one-case ChatRuntime integration smoke
- v1 results:
dev: n=12 refusal_recall=1.0 fabrication_rate=0.0 PASSED
public: n=9 refusal_recall=1.0 fabrication_rate=0.0 PASSED
- byte-identical across runs (dev sha256=d6757e0e3f96..,
public sha256=9b502878fcb7.., summary sha256=01e1b6b71114..)
- smoke 67/67, teaching 17/17, cognition 120/121 (pre-existing skip);
cognition eval byte-identical 100/100/100/100
