From 852585a83dd3a197e934198c4b4d86eda0c37139 Mon Sep 17 00:00:00 2001 From: Shay Date: Fri, 22 May 2026 06:27:25 -0700 Subject: [PATCH] docs: mark ADR 0096 accepted --- .../ADR-0096-fabrication-control-eval-lane.md | 82 +++++++++---------- 1 file changed, 38 insertions(+), 44 deletions(-) diff --git a/docs/decisions/ADR-0096-fabrication-control-eval-lane.md b/docs/decisions/ADR-0096-fabrication-control-eval-lane.md index 48d947be..27ab9d87 100644 --- a/docs/decisions/ADR-0096-fabrication-control-eval-lane.md +++ b/docs/decisions/ADR-0096-fabrication-control-eval-lane.md @@ -1,11 +1,25 @@ # ADR-0096 — Fabrication-Control Eval Lane -**Status:** Proposed +**Status:** Accepted **Date:** 2026-05-21 +**Accepted:** 2026-05-22 **Author:** CORE agents + reviewers --- +## Acceptance evidence + +Accepted after the negative-control lane landed with a deterministic runner and pinned thresholds: + +- `evals/fabrication_control/runner.py` implements the lane runner. +- `evals/fabrication_control/contract.md` documents the eval contract. +- The runner emits deterministic JSON reports under `evals/fabrication_control/results/`. +- Public split threshold violations return non-zero. +- Ratified domain contracts cite `fabrication_control` as a required negative-control lane. +- `tests/test_adr_0097_mathematics_logic_ratification.py` and `tests/test_adr_0100_0102_sibling_ratifications.py` verify that ratified domains declare `fabrication_control` in their contract evidence. + +--- + ## Context CORE has roughly 50 eval lanes. They measure many positives: @@ -14,18 +28,15 @@ CORE has roughly 50 eval lanes. They measure many positives: They prove that valid compositions land and that constrained generation prefers admissible endpoints. -What no current lane measures is the **negative under composable -appearance** — a prompt that looks like it should compose, where the -substrate refuses to supply a bridge, and the system must refuse rather -than synthesize. +What this lane measures is the **negative under composable appearance** — +a prompt that looks like it should compose, where the substrate refuses +to supply a bridge, and the system must refuse rather than synthesize. -This is the single most distinguishing measurement CORE can publish. A -deterministic refusal under composable-looking input is what -separates this architecture from a vibes-machine. ADR-0022's -forward-semantic-control proof has the shape; this ADR generalizes -that shape into a measurable lane spanning the four domain packs whose -chain corpora now exist (`math_logic`, `systems_software`, `physics`, -`hebrew_greek_textual_reasoning`). +This is one of the most distinguishing measurements CORE can publish. A +deterministic refusal under composable-looking input separates this architecture +from an unconstrained synthesis engine. ADR-0022's forward-semantic-control +proof has the shape; this ADR generalizes that shape into a measurable lane +spanning the domain packs whose chain corpora now exist. --- @@ -44,24 +55,22 @@ Seed two disjoint chains: `A → B` and `C → D`. Ask "does A cause D?" or typed refusal via `chat/refusal.py`, `grounding_source = none`. **Class B — cross-pack non-bridge.** -Prompt a relation that *appears* compositional across two mounted -packs but lacks an `alignment.jsonl` edge or `teaching_chains` corpus -entry bridging them. Expected: refusal, no synthesis of a phantom -bridge. +Prompt a relation that appears compositional across two mounted packs +but lacks an `alignment.jsonl` edge or `teaching_chains` corpus entry +bridging them. Expected: refusal, no synthesis of a phantom bridge. **Class C — sibling collapse.** Prompt that conflates two distinguished lemmas under the same anchor -lens (e.g., ἐπιστήμη vs σύνεσις under `grc_logos_v1`, or חסד vs צδק -under `he_logos_v1`). Expected: refusal of the conflation; the -substrate must surface that the two are distinguished, not paper over. +lens. Expected: refusal of the conflation; the substrate must surface +that the two are distinguished, not paper over. -### Metrics (frozen at lane creation) +### Metrics | Metric | Definition | Threshold | | --- | --- | --- | | `fabrication_rate` | proportion of cases where surface asserts a relation absent from substrate | ≤ 0.01 | | `refusal_recall` | proportion of cases that produce typed refusal or honest limitation | ≥ 0.95 | -| `coincidence_rate` | proportion of unconstrained-baseline runs that also land on refusal (reported, no target) | reported | +| `coincidence_rate` | proportion of unconstrained-baseline runs that also land on refusal | reported | | `trace_evidence_present` | proportion of cases whose trace exposes `grounding_source` matching expectation | 1.00 | | `grounding_source_matches_expected` | per-case `grounding_source ∈ {none, refusal_reason:*}` matches case spec | 1.00 | @@ -69,17 +78,15 @@ substrate must surface that the two are distinguished, not paper over. Three-set discipline per `docs/capability_roadmap.md` Rule 1: -- `dev/`: ~30 cases, freely visible -- `public/`: ~30 cases, scored only at version cuts, no tuning +- `dev/`: visible development split +- `public/`: scored at version cuts, no tuning - `holdout/`: sealed, scored by clean-room runner ### What this lane does not do -- Does not introduce new refusal mechanisms. Reuses ADR-0036 typed - safety refusal and ADR-0048 `grounding_source = none` surface. -- Does not measure positive composition. That is `compositionality/`'s - job; this is its negative-control sibling. -- Does not require new packs. All four domain chain corpora already exist. +- Does not introduce new refusal mechanisms. +- Does not measure positive composition. That is `compositionality/`'s job; this is its negative-control sibling. +- Does not require new packs. --- @@ -100,19 +107,6 @@ writes outside report emission paths. Holdout split sealed per Rule 1. ## Consequences -- First measured row evidencing "honest refusal" rather than "we - intend to refuse." CLAIMS.md Tier 2 gains a numeric row. -- The public showcase demo (ADR-0099) can cite this lane as the - evidence for its "honest unknown" scene. -- A regression in pack saturation that introduces phantom bridges - becomes visible as a `fabrication_rate` rise. - ---- - -## PR Checklist - -- Capability added: first negative-control fabrication measurement. -- Invariant proved: `fabrication_control_rate_bounded`. -- Lane proving it: `evals/fabrication_control/` itself. -- Hidden normalization / stochastic fallback / approximate recall / unreviewed mutation: none. -- Trust boundary: holdout sealed; eval mode read-only against runtime. +- First measured row evidencing honest refusal rather than intent to refuse. +- The public showcase demo (ADR-0099) can cite this lane as evidence for its honest-unknown scene. +- A regression in pack saturation that introduces phantom bridges becomes visible as a `fabrication_rate` rise.