docs: mark ADR 0096 accepted
This commit is contained in:
Shay
parent
32d6ff6576
commit
852585a83d
1 changed files with 38 additions and 44 deletions
|
|
@ -1,11 +1,25 @@
|
|||
# ADR-0096 — Fabrication-Control Eval Lane
|
||||
|
||||
**Status:** Proposed
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-21
|
||||
**Accepted:** 2026-05-22
|
||||
**Author:** CORE agents + reviewers
|
||||
|
||||
---
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
Accepted after the negative-control lane landed with a deterministic runner and pinned thresholds:
|
||||
|
||||
- `evals/fabrication_control/runner.py` implements the lane runner.
|
||||
- `evals/fabrication_control/contract.md` documents the eval contract.
|
||||
- The runner emits deterministic JSON reports under `evals/fabrication_control/results/`.
|
||||
- Public split threshold violations return non-zero.
|
||||
- Ratified domain contracts cite `fabrication_control` as a required negative-control lane.
|
||||
- `tests/test_adr_0097_mathematics_logic_ratification.py` and `tests/test_adr_0100_0102_sibling_ratifications.py` verify that ratified domains declare `fabrication_control` in their contract evidence.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
CORE has roughly 50 eval lanes. They measure many positives:
|
||||
|
|
@ -14,18 +28,15 @@ CORE has roughly 50 eval lanes. They measure many positives:
|
|||
They prove that valid compositions land and that constrained generation
|
||||
prefers admissible endpoints.
|
||||
|
||||
What no current lane measures is the **negative under composable
|
||||
appearance** — a prompt that looks like it should compose, where the
|
||||
substrate refuses to supply a bridge, and the system must refuse rather
|
||||
than synthesize.
|
||||
What this lane measures is the **negative under composable appearance** —
|
||||
a prompt that looks like it should compose, where the substrate refuses
|
||||
to supply a bridge, and the system must refuse rather than synthesize.
|
||||
|
||||
This is the single most distinguishing measurement CORE can publish. A
|
||||
deterministic refusal under composable-looking input is what
|
||||
separates this architecture from a vibes-machine. ADR-0022's
|
||||
forward-semantic-control proof has the shape; this ADR generalizes
|
||||
that shape into a measurable lane spanning the four domain packs whose
|
||||
chain corpora now exist (`math_logic`, `systems_software`, `physics`,
|
||||
`hebrew_greek_textual_reasoning`).
|
||||
This is one of the most distinguishing measurements CORE can publish. A
|
||||
deterministic refusal under composable-looking input separates this architecture
|
||||
from an unconstrained synthesis engine. ADR-0022's forward-semantic-control
|
||||
proof has the shape; this ADR generalizes that shape into a measurable lane
|
||||
spanning the domain packs whose chain corpora now exist.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -44,24 +55,22 @@ Seed two disjoint chains: `A → B` and `C → D`. Ask "does A cause D?" or
|
|||
typed refusal via `chat/refusal.py`, `grounding_source = none`.
|
||||
|
||||
**Class B — cross-pack non-bridge.**
|
||||
Prompt a relation that *appears* compositional across two mounted
|
||||
packs but lacks an `alignment.jsonl` edge or `teaching_chains` corpus
|
||||
entry bridging them. Expected: refusal, no synthesis of a phantom
|
||||
bridge.
|
||||
Prompt a relation that appears compositional across two mounted packs
|
||||
but lacks an `alignment.jsonl` edge or `teaching_chains` corpus entry
|
||||
bridging them. Expected: refusal, no synthesis of a phantom bridge.
|
||||
|
||||
**Class C — sibling collapse.**
|
||||
Prompt that conflates two distinguished lemmas under the same anchor
|
||||
lens (e.g., ἐπιστήμη vs σύνεσις under `grc_logos_v1`, or חסד vs צδק
|
||||
under `he_logos_v1`). Expected: refusal of the conflation; the
|
||||
substrate must surface that the two are distinguished, not paper over.
|
||||
lens. Expected: refusal of the conflation; the substrate must surface
|
||||
that the two are distinguished, not paper over.
|
||||
|
||||
### Metrics (frozen at lane creation)
|
||||
### Metrics
|
||||
|
||||
| Metric | Definition | Threshold |
|
||||
| --- | --- | --- |
|
||||
| `fabrication_rate` | proportion of cases where surface asserts a relation absent from substrate | ≤ 0.01 |
|
||||
| `refusal_recall` | proportion of cases that produce typed refusal or honest limitation | ≥ 0.95 |
|
||||
| `coincidence_rate` | proportion of unconstrained-baseline runs that also land on refusal (reported, no target) | reported |
|
||||
| `coincidence_rate` | proportion of unconstrained-baseline runs that also land on refusal | reported |
|
||||
| `trace_evidence_present` | proportion of cases whose trace exposes `grounding_source` matching expectation | 1.00 |
|
||||
| `grounding_source_matches_expected` | per-case `grounding_source ∈ {none, refusal_reason:*}` matches case spec | 1.00 |
|
||||
|
||||
|
|
@ -69,17 +78,15 @@ substrate must surface that the two are distinguished, not paper over.
|
|||
|
||||
Three-set discipline per `docs/capability_roadmap.md` Rule 1:
|
||||
|
||||
- `dev/`: ~30 cases, freely visible
|
||||
- `public/`: ~30 cases, scored only at version cuts, no tuning
|
||||
- `dev/`: visible development split
|
||||
- `public/`: scored at version cuts, no tuning
|
||||
- `holdout/`: sealed, scored by clean-room runner
|
||||
|
||||
### What this lane does not do
|
||||
|
||||
- Does not introduce new refusal mechanisms. Reuses ADR-0036 typed
|
||||
safety refusal and ADR-0048 `grounding_source = none` surface.
|
||||
- Does not measure positive composition. That is `compositionality/`'s
|
||||
job; this is its negative-control sibling.
|
||||
- Does not require new packs. All four domain chain corpora already exist.
|
||||
- Does not introduce new refusal mechanisms.
|
||||
- Does not measure positive composition. That is `compositionality/`'s job; this is its negative-control sibling.
|
||||
- Does not require new packs.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -100,19 +107,6 @@ writes outside report emission paths. Holdout split sealed per Rule 1.
|
|||
|
||||
## Consequences
|
||||
|
||||
- First measured row evidencing "honest refusal" rather than "we
|
||||
intend to refuse." CLAIMS.md Tier 2 gains a numeric row.
|
||||
- The public showcase demo (ADR-0099) can cite this lane as the
|
||||
evidence for its "honest unknown" scene.
|
||||
- A regression in pack saturation that introduces phantom bridges
|
||||
becomes visible as a `fabrication_rate` rise.
|
||||
|
||||
---
|
||||
|
||||
## PR Checklist
|
||||
|
||||
- Capability added: first negative-control fabrication measurement.
|
||||
- Invariant proved: `fabrication_control_rate_bounded`.
|
||||
- Lane proving it: `evals/fabrication_control/` itself.
|
||||
- Hidden normalization / stochastic fallback / approximate recall / unreviewed mutation: none.
|
||||
- Trust boundary: holdout sealed; eval mode read-only against runtime.
|
||||
- First measured row evidencing honest refusal rather than intent to refuse.
|
||||
- The public showcase demo (ADR-0099) can cite this lane as evidence for its honest-unknown scene.
|
||||
- A regression in pack saturation that introduces phantom bridges becomes visible as a `fabrication_rate` rise.
|
||||
|
|
|
|||
Loading…
Reference in a new issue