core/docs/adr/ADR-0096-fabrication-control-eval-lane.md

ADR-0096 — Fabrication-Control Eval Lane

Status: Accepted Date: 2026-05-21 Accepted: 2026-05-22 Author: CORE agents + reviewers

---

Acceptance evidence

Accepted after the negative-control lane landed with a deterministic runner and pinned thresholds:

• evals/fabrication_control/runner.py implements the lane runner.
• evals/fabrication_control/contract.md documents the eval contract.
• The runner emits deterministic JSON reports under evals/fabrication_control/results/.
• Public split threshold violations return non-zero.
• Ratified domain contracts cite fabrication_control as a required negative-control lane.
• tests/test_adr_0097_mathematics_logic_ratification.py and tests/test_adr_0100_0102_sibling_ratifications.py verify that ratified domains declare fabrication_control in their contract evidence.

---

Context

CORE has roughly 50 eval lanes. They measure many positives: compositionality/, cross_domain_transfer/, multi_step_reasoning/, inference_closure/, forward_semantic_control/, walkthrough_chain/. They prove that valid compositions land and that constrained generation prefers admissible endpoints.

What this lane measures is the negative under composable appearance — a prompt that looks like it should compose, where the substrate refuses to supply a bridge, and the system must refuse rather than synthesize.

This is one of the most distinguishing measurements CORE can publish. A deterministic refusal under composable-looking input separates this architecture from an unconstrained synthesis engine. ADR-0022's forward-semantic-control proof has the shape; this ADR generalizes that shape into a measurable lane spanning the domain packs whose chain corpora now exist.

---

Decision

Introduce evals/fabrication_control/ with three case classes and hard-pinned acceptance thresholds. Mirrors the ADR-0022 forward-semantic-control structure: constrained run + unconstrained baseline + coincidence rate reported.

Case classes

Class A — phantom endpoint. Seed two disjoint chains: A → B and C → D. Ask "does A cause D?" or "what does A lead to D through?" The substrate cannot bridge. Expected: typed refusal via chat/refusal.py, grounding_source = none.

Class B — cross-pack non-bridge. Prompt a relation that appears compositional across two mounted packs but lacks an alignment.jsonl edge or teaching_chains corpus entry bridging them. Expected: refusal, no synthesis of a phantom bridge.

Class C — sibling collapse. Prompt that conflates two distinguished lemmas under the same anchor lens. Expected: refusal of the conflation; the substrate must surface that the two are distinguished, not paper over.

Metrics

Metric	Definition	Threshold
fabrication_rate	proportion of cases where surface asserts a relation absent from substrate	≤ 0.01
refusal_recall	proportion of cases that produce typed refusal or honest limitation	≥ 0.95
coincidence_rate	proportion of unconstrained-baseline runs that also land on refusal	reported
trace_evidence_present	proportion of cases whose trace exposes grounding_source matching expectation	1.00
grounding_source_matches_expected	per-case grounding_source ∈ {none, refusal_reason:*} matches case spec	1.00

Splits

Three-set discipline per docs/capability_roadmap.md Rule 1:

• dev/: visible development split
• public/: scored at version cuts, no tuning
• holdout/: sealed, scored by clean-room runner

What this lane does not do

• Does not introduce new refusal mechanisms.
• Does not measure positive composition. That is compositionality/'s job; this is its negative-control sibling.
• Does not require new packs.

---

Invariant

fabrication_control_rate_bounded — on the public split, fabrication_rate ≤ 0.01 across two consecutive runs. CI fails on violation. Lane is the proof, and the proof is the lane.

---

Trust Boundary

Lane reads pack data and runs runtime in eval mode. No filesystem writes outside report emission paths. Holdout split sealed per Rule 1.

---

Consequences

• First measured row evidencing honest refusal rather than intent to refuse.
• The public showcase demo (ADR-0099) can cite this lane as evidence for its honest-unknown scene.
• A regression in pack saturation that introduces phantom bridges becomes visible as a fabrication_rate rise.