core/docs/eval_methodology.md

Eval Methodology — Benchmark Discipline Contract

Status: Accepted (ADR-0016) Last updated: 2026-05-15

This document defines the five rules that govern every eval lane in the CORE capability roadmap. No exceptions per phase. A lane that does not satisfy these rules is exploration, not a gate.

---

Rule 1 — Three-set split per lane

Every lane maintains three disjoint corpora:

• Dev set. Freely visible during development. Used to iterate.
• Public test set. Visible, but tuning against it is forbidden. Scored at version-cut time only. Drift in dev-vs-public scores is a red flag for overfitting.
• Private holdout. Sealed. Never read by Claude, never committed in plaintext, only scored by a clean-room runner at release events. Stored encrypted in evals/holdouts/ with key held by the human reviewer.

If a lane has only a dev set, it does not count as a gate. It is exploration.

Rule 2 — Versioned difficulty escalation

Each lane has versions: v1, v2, v3, ... with monotonically harder distributions. Passing a version is not a terminal state; it is a checkpoint that unlocks generating the next version.

• v1 — baseline competence demonstration. The construction is shown clearly.
• v2 — distributional shift: longer chains, deeper nesting, rarer vocabulary, paraphrased surface forms.
• v3 — adversarial: items generated specifically by inspecting model failures on v2.
• v4+ — out-of-distribution: items drawn from domains, registers, or constructions not present at training time.

Score is always reported as a tuple (v1_score, v2_score, v3_score, ...), never collapsed to a single number.

Rule 3 — Adversarial regeneration on pass

When a model passes a version (>=95% on the public test set with >=90% on private holdout), the next version is generated by adversarial process:

• Human review finds construction families the model handled accidentally rather than structurally.
• A separate generator produces items targeting the weakest decile of the previous version.
• The new version is reviewed for legitimacy — no impossible items, no ambiguous items, no items that depend on world knowledge the system was never given.

Rule 4 — Frontier baseline tracking

For each lane, a baseline score is computed for at least one frontier transformer-based model on the same public test set. Baselines are:

• Re-scored every time a version is cut.
• Published alongside CORE's score.
• Never tuned, never prompt-engineered to maximize — the prompt is the eval task as written.

Rule 5 — Honest reporting

• Failures are reported with the same prominence as passes.
• Confidence intervals on every score (bootstrapped over the test set).
• Per-construction breakdowns published — never a single aggregate hiding structural failures.
• Regressions across versions are surfaced, never silently dropped.
• "Did not test" is a valid result; "tested and failed" is preferred over "did not test."

If a number cannot be reported honestly under these rules, the lane is not ready. Do not ship the lane.

---

Eval lane directory contract

Every eval lane lives in evals/<lane_name>/ with this layout:

evals/<lane_name>/
  contract.md           # what the lane measures, scoring rubric, pass thresholds
  dev/                  # dev set, freely visible
  public/v1/            # public test set, version 1
  public/v2/            # ...
  holdouts/             # encrypted, sealed
  runner.py             # deterministic scorer
  baselines/            # frontier model scores per version
  results/              # CORE scores per version per release

A lane without a contract.md does not run.

---

References

• ADR-0016: Capability Roadmap
• docs/capability_roadmap.md: Full phased plan