# Eval Methodology — Benchmark Discipline Contract

**Status:** Accepted (ADR-0016)
**Last updated:** 2026-05-15

This document defines the five rules that govern every eval lane in the CORE
capability roadmap. No exceptions per phase. A lane that does not satisfy these
rules is exploration, not a gate.

---

## Rule 1 — Three-set split per lane

Every lane maintains three disjoint corpora:

- **Dev set.** Freely visible during development. Used to iterate.
- **Public test set.** Visible, but tuning against it is forbidden. Scored at
  version-cut time only. Drift in dev-vs-public scores is a red flag for
  overfitting.
- **Private holdout.** Sealed. Never read by Claude, never committed in
  plaintext, only scored by a clean-room runner at release events. Stored
  encrypted in `evals/holdouts/` with key held by the human reviewer.

If a lane has only a dev set, it does not count as a gate. It is exploration.

## Rule 2 — Versioned difficulty escalation

Each lane has versions: `v1`, `v2`, `v3`, ... with monotonically harder
distributions. Passing a version is not a terminal state; it is a checkpoint
that unlocks generating the next version.

- **v1** — baseline competence demonstration. The construction is shown clearly.
- **v2** — distributional shift: longer chains, deeper nesting, rarer
  vocabulary, paraphrased surface forms.
- **v3** — adversarial: items generated specifically by inspecting model
  failures on v2.
- **v4+** — out-of-distribution: items drawn from domains, registers, or
  constructions not present at training time.

Score is always reported as a tuple `(v1_score, v2_score, v3_score, ...)`,
never collapsed to a single number.

## Rule 3 — Adversarial regeneration on pass

When a model passes a version (>=95% on the public test set with >=90% on
private holdout), the next version is generated by adversarial process:

- Human review finds construction families the model handled accidentally
  rather than structurally.
- A separate generator produces items targeting the weakest decile of the
  previous version.
- The new version is reviewed for legitimacy — no impossible items, no
  ambiguous items, no items that depend on world knowledge the system was
  never given.

## Rule 4 — Frontier baseline tracking

For each lane, a baseline score is computed for at least one frontier
transformer-based model on the same public test set. Baselines are:

- Re-scored every time a version is cut.
- Published alongside CORE's score.
- Never tuned, never prompt-engineered to maximize — the prompt is the eval
  task as written.

## Rule 5 — Honest reporting

- Failures are reported with the same prominence as passes.
- Confidence intervals on every score (bootstrapped over the test set).
- Per-construction breakdowns published — never a single aggregate hiding
  structural failures.
- Regressions across versions are surfaced, never silently dropped.
- "Did not test" is a valid result; "tested and failed" is preferred over
  "did not test."

If a number cannot be reported honestly under these rules, the lane is not
ready. Do not ship the lane.

---

## Eval lane directory contract

Every eval lane lives in `evals/<lane_name>/` with this layout:

```
evals/<lane_name>/
  contract.md           # what the lane measures, scoring rubric, pass thresholds
  dev/                  # dev set, freely visible
  public/v1/            # public test set, version 1
  public/v2/            # ...
  holdouts/             # encrypted, sealed
  runner.py             # deterministic scorer
  baselines/            # frontier model scores per version
  results/              # CORE scores per version per release
```

A lane without a `contract.md` does not run.

---

## References

- ADR-0016: Capability Roadmap
- `docs/capability_roadmap.md`: Full phased plan