bc90622e53
Closes the documentation audit. All four lower-priority surfaces from the earlier scan now reflect the post-ADR-0110 reality. New: - docs/sessions/SESSION-2026-05-22-contract-layer-arc.md: full narrative of the ADR-0103 -> ADR-0110 arc, including the refused/amended/succeeded narrative, infrastructure bridges, and ledger state at session close. Pairs with the 2026-05-21 articulation-arc session log. Extended (additive, no rewrites): - docs/runtime_contracts.md: new 'Expert-Demo Promotion Contract (ADR-0106 + ADR-0109)' section. Surfaces ledger_report() shape, reviewer-yaml schema, lane-shape registry, replay invariant, fail-closed registry behavior, and trust boundary. - docs/eval_methodology.md: new 'Lane-shape registry (ADR-0109)' section. Documents what to do when adding a new eval lane: pick a shape or amend ADR-0109. Notes the holdout-runner gating (cases_plaintext.jsonl vs sealed). Header bumped to 2026-05-22. - docs/Whitepaper.md: new section XIII 'Evidence-Governed Domain Layer' between XII (Forward Semantic Control) and the Extensions/closing. Narrates the contract chain at the philosophical / external-reader level. The original §XIII becomes §XIV. - docs/Yellowpaper.md: new section XII 'Ratification Contract (ADR-0091 + ADR-0106 + ADR-0109)'. Formal specification of the nine predicates, the promotion predicate (with explicit logical form), the digest function, and the lane-shape registry table. The original §XII becomes §XIII. No code, no tests, no contract changes.
162 lines
6.1 KiB
Markdown
162 lines
6.1 KiB
Markdown
# Eval Methodology — Benchmark Discipline Contract
|
|
|
|
**Status:** Accepted (ADR-0016, extended by ADR-0109)
|
|
**Last updated:** 2026-05-22 (lane-shape registry section added)
|
|
|
|
This document defines the five rules that govern every eval lane in the CORE
|
|
capability roadmap. No exceptions per phase. A lane that does not satisfy these
|
|
rules is exploration, not a gate.
|
|
|
|
---
|
|
|
|
## Rule 1 — Three-set split per lane
|
|
|
|
Every lane maintains three disjoint corpora:
|
|
|
|
- **Dev set.** Freely visible during development. Used to iterate.
|
|
- **Public test set.** Visible, but tuning against it is forbidden. Scored at
|
|
version-cut time only. Drift in dev-vs-public scores is a red flag for
|
|
overfitting.
|
|
- **Private holdout.** Sealed. Never read by Claude, never committed in
|
|
plaintext, only scored by a clean-room runner at release events. Stored
|
|
encrypted in `evals/holdouts/` with key held by the human reviewer.
|
|
|
|
If a lane has only a dev set, it does not count as a gate. It is exploration.
|
|
|
|
## Rule 2 — Versioned difficulty escalation
|
|
|
|
Each lane has versions: `v1`, `v2`, `v3`, ... with monotonically harder
|
|
distributions. Passing a version is not a terminal state; it is a checkpoint
|
|
that unlocks generating the next version.
|
|
|
|
- **v1** — baseline competence demonstration. The construction is shown clearly.
|
|
- **v2** — distributional shift: longer chains, deeper nesting, rarer
|
|
vocabulary, paraphrased surface forms.
|
|
- **v3** — adversarial: items generated specifically by inspecting model
|
|
failures on v2.
|
|
- **v4+** — out-of-distribution: items drawn from domains, registers, or
|
|
constructions not present at training time.
|
|
|
|
Score is always reported as a tuple `(v1_score, v2_score, v3_score, ...)`,
|
|
never collapsed to a single number.
|
|
|
|
## Rule 3 — Adversarial regeneration on pass
|
|
|
|
When a model passes a version (>=95% on the public test set with >=90% on
|
|
private holdout), the next version is generated by adversarial process:
|
|
|
|
- Human review finds construction families the model handled accidentally
|
|
rather than structurally.
|
|
- A separate generator produces items targeting the weakest decile of the
|
|
previous version.
|
|
- The new version is reviewed for legitimacy — no impossible items, no
|
|
ambiguous items, no items that depend on world knowledge the system was
|
|
never given.
|
|
|
|
## Rule 4 — Frontier baseline tracking
|
|
|
|
For each lane, a baseline score is computed for at least one frontier
|
|
transformer-based model on the same public test set. Baselines are:
|
|
|
|
- Re-scored every time a version is cut.
|
|
- Published alongside CORE's score.
|
|
- Never tuned, never prompt-engineered to maximize — the prompt is the eval
|
|
task as written.
|
|
|
|
## Rule 5 — Honest reporting
|
|
|
|
- Failures are reported with the same prominence as passes.
|
|
- Confidence intervals on every score (bootstrapped over the test set).
|
|
- Per-construction breakdowns published — never a single aggregate hiding
|
|
structural failures.
|
|
- Regressions across versions are surfaced, never silently dropped.
|
|
- "Did not test" is a valid result; "tested and failed" is preferred over
|
|
"did not test."
|
|
|
|
If a number cannot be reported honestly under these rules, the lane is not
|
|
ready. Do not ship the lane.
|
|
|
|
---
|
|
|
|
## Eval lane directory contract
|
|
|
|
Every eval lane lives in `evals/<lane_name>/` with this layout:
|
|
|
|
```
|
|
evals/<lane_name>/
|
|
contract.md # what the lane measures, scoring rubric, pass thresholds
|
|
dev/ # dev set, freely visible
|
|
public/v1/ # public test set, version 1
|
|
public/v2/ # ...
|
|
holdouts/ # encrypted, sealed
|
|
runner.py # deterministic scorer
|
|
baselines/ # frontier model scores per version
|
|
results/ # CORE scores per version per release
|
|
```
|
|
|
|
A lane without a `contract.md` does not run.
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- ADR-0016: Capability Roadmap
|
|
- `docs/capability_roadmap.md`: Full phased plan
|
|
|
|
---
|
|
|
|
## Lane-shape registry (ADR-0109)
|
|
|
|
ADR-0091's Domain Pack Contract v1 introduced a `dev/public/holdout`
|
|
discipline that every ratified pack must declare. ADR-0106 added a
|
|
reviewer-signed expert-demo promotion gate that consults those same
|
|
lane outputs at the ledger level. ADR-0109 then formalized the rule
|
|
that **threshold dispatch is lane-shape-aware**, not lane-uniform.
|
|
|
|
### What this means for new lanes
|
|
|
|
Adding a new eval lane requires deciding which shape it reports:
|
|
|
|
| Shape | Required metrics | Threshold |
|
|
|---|---|---|
|
|
| `cognition_shape` | `surface_groundedness`, `term_capture_rate`, `intent_accuracy`, `versor_closure_rate` | 0.95 / 0.85 / 0.95 / 1.0 |
|
|
| `accuracy_shape` | `accuracy` (or `passed`/`total` fallback) | ≥ 0.95 |
|
|
| `inference_shape` | `all_pass_rate`, `replay_determinism`, `overall_pass` | 0.95 / 1.0 / true |
|
|
| `refusal_shape` | `by_class[*].n`, `by_class[*].refused`, `by_class[*].fabricated` | refused == n, fabricated == 0 |
|
|
| `symbolic_logic_shape` | `accuracy` | ≥ 0.95 |
|
|
|
|
A lane that does not fit any existing shape **must not be silently
|
|
broadened**. The path is:
|
|
|
|
1. Open an ADR amending ADR-0109 to add the new shape.
|
|
2. Add the shape checker to `SHAPE_CHECKERS` in
|
|
`core/capability/expert_demo.py`.
|
|
3. Add the lane → shape mapping to `LANE_SHAPE_REGISTRY`.
|
|
|
|
A lane id absent from the registry is **fail-closed** at the
|
|
expert-demo gate (reason:
|
|
`lane <id> has no registered shape — introduce via ADR amendment`).
|
|
Unregistered lanes can still run as exploration; they just cannot
|
|
contribute evidence to a `reviewer-signed expert_demo` promotion.
|
|
|
|
### Holdout-runner gating (ADR-0105)
|
|
|
|
`split='holdout'` runs go through `evals.holdout_runner._decrypt_holdout`,
|
|
which expects either:
|
|
|
|
- `holdouts/v1/cases.jsonl.age` (sealed, requires `CORE_HOLDOUT_KEY`), **or**
|
|
- `holdouts/v1/cases_plaintext.jsonl` (dev-mode fallback, no key required).
|
|
|
|
A bare `holdouts/v1/cases.jsonl` is invisible to the runner. Lanes
|
|
authored before ADR-0105 must either rename their plaintext file or
|
|
seal it against an age recipient to be runnable on the `holdout`
|
|
split.
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- ADR-0091: Domain Pack Contract v1 (`evals/<lane>/holdouts/` discipline)
|
|
- ADR-0105: Sealed-holdout encryption (dev-mode fallback preserved)
|
|
- ADR-0106: Expert-demo promotion contract (consumes lane results)
|
|
- ADR-0109: Lane-shape-aware thresholds (this section's authority)
|