core/docs/eval_methodology.md

Eval Methodology — Benchmark Discipline Contract

Status: Accepted (ADR-0016, extended by ADR-0109) Last updated: 2026-05-22 (lane-shape registry section added)

This document defines the five rules that govern every eval lane in the CORE capability roadmap. No exceptions per phase. A lane that does not satisfy these rules is exploration, not a gate.

---

Rule 1 — Three-set split per lane

Every lane maintains three disjoint corpora:

• Dev set. Freely visible during development. Used to iterate.
• Public test set. Visible, but tuning against it is forbidden. Scored at version-cut time only. Drift in dev-vs-public scores is a red flag for overfitting.
• Private holdout. Sealed. Never read by Claude, never committed in plaintext, only scored by a clean-room runner at release events. Stored encrypted in evals/holdouts/ with key held by the human reviewer.

If a lane has only a dev set, it does not count as a gate. It is exploration.

Rule 2 — Versioned difficulty escalation

Each lane has versions: v1, v2, v3, ... with monotonically harder distributions. Passing a version is not a terminal state; it is a checkpoint that unlocks generating the next version.

• v1 — baseline competence demonstration. The construction is shown clearly.
• v2 — distributional shift: longer chains, deeper nesting, rarer vocabulary, paraphrased surface forms.
• v3 — adversarial: items generated specifically by inspecting model failures on v2.
• v4+ — out-of-distribution: items drawn from domains, registers, or constructions not present at training time.

Score is always reported as a tuple (v1_score, v2_score, v3_score, ...), never collapsed to a single number.

Rule 3 — Adversarial regeneration on pass

When a model passes a version (>=95% on the public test set with >=90% on private holdout), the next version is generated by adversarial process:

• Human review finds construction families the model handled accidentally rather than structurally.
• A separate generator produces items targeting the weakest decile of the previous version.
• The new version is reviewed for legitimacy — no impossible items, no ambiguous items, no items that depend on world knowledge the system was never given.

Rule 4 — Frontier baseline tracking

For each lane, a baseline score is computed for at least one frontier transformer-based model on the same public test set. Baselines are:

• Re-scored every time a version is cut.
• Published alongside CORE's score.
• Never tuned, never prompt-engineered to maximize — the prompt is the eval task as written.

Rule 5 — Honest reporting

• Failures are reported with the same prominence as passes.
• Confidence intervals on every score (bootstrapped over the test set).
• Per-construction breakdowns published — never a single aggregate hiding structural failures.
• Regressions across versions are surfaced, never silently dropped.
• "Did not test" is a valid result; "tested and failed" is preferred over "did not test."

If a number cannot be reported honestly under these rules, the lane is not ready. Do not ship the lane.

---

Eval lane directory contract

Every eval lane lives in evals/<lane_name>/ with this layout:

evals/<lane_name>/
  contract.md           # what the lane measures, scoring rubric, pass thresholds
  dev/                  # dev set, freely visible
  public/v1/            # public test set, version 1
  public/v2/            # ...
  holdouts/             # encrypted, sealed
  runner.py             # deterministic scorer
  baselines/            # frontier model scores per version
  results/              # CORE scores per version per release

A lane without a contract.md does not run.

---

References

• ADR-0016: Capability Roadmap
• docs/capability_roadmap.md: Full phased plan

---

Lane-shape registry (ADR-0109)

ADR-0091's Domain Pack Contract v1 introduced a dev/public/holdout discipline that every ratified pack must declare. ADR-0106 added a reviewer-signed expert-demo promotion gate that consults those same lane outputs at the ledger level. ADR-0109 then formalized the rule that threshold dispatch is lane-shape-aware, not lane-uniform.

What this means for new lanes

Adding a new eval lane requires deciding which shape it reports:

Shape	Required metrics	Threshold
cognition_shape	surface_groundedness, term_capture_rate, intent_accuracy, versor_closure_rate	0.95 / 0.85 / 0.95 / 1.0
accuracy_shape	accuracy (or passed/total fallback)	≥ 0.95
inference_shape	all_pass_rate, replay_determinism, overall_pass	0.95 / 1.0 / true
refusal_shape	by_class[*].n, by_class[*].refused, by_class[*].fabricated	refused == n, fabricated == 0
symbolic_logic_shape	accuracy	≥ 0.95

A lane that does not fit any existing shape must not be silently broadened. The path is:

1. Open an ADR amending ADR-0109 to add the new shape.
2. Add the shape checker to SHAPE_CHECKERS in core/capability/expert_demo.py.
3. Add the lane → shape mapping to LANE_SHAPE_REGISTRY.

A lane id absent from the registry is fail-closed at the expert-demo gate (reason: lane <id> has no registered shape — introduce via ADR amendment). Unregistered lanes can still run as exploration; they just cannot contribute evidence to a reviewer-signed expert_demo promotion.

Holdout-runner gating (ADR-0105)

split='holdout' runs go through evals.holdout_runner._decrypt_holdout, which expects either:

• holdouts/v1/cases.jsonl.age (sealed, requires CORE_HOLDOUT_KEY), or
• holdouts/v1/cases_plaintext.jsonl (dev-mode fallback, no key required).

A bare holdouts/v1/cases.jsonl is invisible to the runner. Lanes authored before ADR-0105 must either rename their plaintext file or seal it against an age recipient to be runnable on the holdout split.

---

References

• ADR-0091: Domain Pack Contract v1 (evals/<lane>/holdouts/ discipline)
• ADR-0105: Sealed-holdout encryption (dev-mode fallback preserved)
• ADR-0106: Expert-demo promotion contract (consumes lane results)
• ADR-0109: Lane-shape-aware thresholds (this section's authority)