# Eval Methodology — Benchmark Discipline Contract **Status:** Accepted (ADR-0016, extended by ADR-0109) **Last updated:** 2026-05-22 (lane-shape registry section added) This document defines the five rules that govern every eval lane in the CORE capability roadmap. No exceptions per phase. A lane that does not satisfy these rules is exploration, not a gate. --- ## Rule 1 — Three-set split per lane Every lane maintains three disjoint corpora: - **Dev set.** Freely visible during development. Used to iterate. - **Public test set.** Visible, but tuning against it is forbidden. Scored at version-cut time only. Drift in dev-vs-public scores is a red flag for overfitting. - **Private holdout.** Sealed. Never read by Claude, never committed in plaintext, only scored by a clean-room runner at release events. Stored encrypted in `evals/holdouts/` with key held by the human reviewer. If a lane has only a dev set, it does not count as a gate. It is exploration. ## Rule 2 — Versioned difficulty escalation Each lane has versions: `v1`, `v2`, `v3`, ... with monotonically harder distributions. Passing a version is not a terminal state; it is a checkpoint that unlocks generating the next version. - **v1** — baseline competence demonstration. The construction is shown clearly. - **v2** — distributional shift: longer chains, deeper nesting, rarer vocabulary, paraphrased surface forms. - **v3** — adversarial: items generated specifically by inspecting model failures on v2. - **v4+** — out-of-distribution: items drawn from domains, registers, or constructions not present at training time. Score is always reported as a tuple `(v1_score, v2_score, v3_score, ...)`, never collapsed to a single number. ## Rule 3 — Adversarial regeneration on pass When a model passes a version (>=95% on the public test set with >=90% on private holdout), the next version is generated by adversarial process: - Human review finds construction families the model handled accidentally rather than structurally. - A separate generator produces items targeting the weakest decile of the previous version. - The new version is reviewed for legitimacy — no impossible items, no ambiguous items, no items that depend on world knowledge the system was never given. ## Rule 4 — Frontier baseline tracking For each lane, a baseline score is computed for at least one frontier transformer-based model on the same public test set. Baselines are: - Re-scored every time a version is cut. - Published alongside CORE's score. - Never tuned, never prompt-engineered to maximize — the prompt is the eval task as written. ## Rule 5 — Honest reporting - Failures are reported with the same prominence as passes. - Confidence intervals on every score (bootstrapped over the test set). - Per-construction breakdowns published — never a single aggregate hiding structural failures. - Regressions across versions are surfaced, never silently dropped. - "Did not test" is a valid result; "tested and failed" is preferred over "did not test." If a number cannot be reported honestly under these rules, the lane is not ready. Do not ship the lane. --- ## Eval lane directory contract Every eval lane lives in `evals//` with this layout: ``` evals// contract.md # what the lane measures, scoring rubric, pass thresholds dev/ # dev set, freely visible public/v1/ # public test set, version 1 public/v2/ # ... holdouts/ # encrypted, sealed runner.py # deterministic scorer baselines/ # frontier model scores per version results/ # CORE scores per version per release ``` A lane without a `contract.md` does not run. --- ## References - ADR-0016: Capability Roadmap - `docs/capability_roadmap.md`: Full phased plan --- ## Lane-shape registry (ADR-0109) ADR-0091's Domain Pack Contract v1 introduced a `dev/public/holdout` discipline that every ratified pack must declare. ADR-0106 added a reviewer-signed expert-demo promotion gate that consults those same lane outputs at the ledger level. ADR-0109 then formalized the rule that **threshold dispatch is lane-shape-aware**, not lane-uniform. ### What this means for new lanes Adding a new eval lane requires deciding which shape it reports: | Shape | Required metrics | Threshold | |---|---|---| | `cognition_shape` | `surface_groundedness`, `term_capture_rate`, `intent_accuracy`, `versor_closure_rate` | 0.95 / 0.85 / 0.95 / 1.0 | | `accuracy_shape` | `accuracy` (or `passed`/`total` fallback) | ≥ 0.95 | | `inference_shape` | `all_pass_rate`, `replay_determinism`, `overall_pass` | 0.95 / 1.0 / true | | `refusal_shape` | `by_class[*].n`, `by_class[*].refused`, `by_class[*].fabricated` | refused == n, fabricated == 0 | | `symbolic_logic_shape` | `accuracy` | ≥ 0.95 | A lane that does not fit any existing shape **must not be silently broadened**. The path is: 1. Open an ADR amending ADR-0109 to add the new shape. 2. Add the shape checker to `SHAPE_CHECKERS` in `core/capability/expert_demo.py`. 3. Add the lane → shape mapping to `LANE_SHAPE_REGISTRY`. A lane id absent from the registry is **fail-closed** at the expert-demo gate (reason: `lane has no registered shape — introduce via ADR amendment`). Unregistered lanes can still run as exploration; they just cannot contribute evidence to a `reviewer-signed expert_demo` promotion. ### Holdout-runner gating (ADR-0105) `split='holdout'` runs go through `evals.holdout_runner._decrypt_holdout`, which expects either: - `holdouts/v1/cases.jsonl.age` (sealed, requires `CORE_HOLDOUT_KEY`), **or** - `holdouts/v1/cases_plaintext.jsonl` (dev-mode fallback, no key required). A bare `holdouts/v1/cases.jsonl` is invisible to the runner. Lanes authored before ADR-0105 must either rename their plaintext file or seal it against an age recipient to be runnable on the `holdout` split. --- ## References - ADR-0091: Domain Pack Contract v1 (`evals//holdouts/` discipline) - ADR-0105: Sealed-holdout encryption (dev-mode fallback preserved) - ADR-0106: Expert-demo promotion contract (consumes lane results) - ADR-0109: Lane-shape-aware thresholds (this section's authority)