# Eval Methodology — Benchmark Discipline Contract

**Status:** Accepted (ADR-0016, extended by ADR-0109)
**Last updated:** 2026-05-22 (lane-shape registry section added)

This document defines the five rules that govern every eval lane in the CORE
capability roadmap. No exceptions per phase. A lane that does not satisfy these
rules is exploration, not a gate.

---

## Rule 1 — Three-set split per lane

Every lane maintains three disjoint corpora:

- **Dev set.** Freely visible during development. Used to iterate.
- **Public test set.** Visible, but tuning against it is forbidden. Scored at
  version-cut time only. Drift in dev-vs-public scores is a red flag for
  overfitting.
- **Private holdout.** Sealed. Never read by Claude, never committed in
  plaintext, only scored by a clean-room runner at release events. Stored
  encrypted in `evals/holdouts/` with key held by the human reviewer.

If a lane has only a dev set, it does not count as a gate. It is exploration.

## Rule 2 — Versioned difficulty escalation

Each lane has versions: `v1`, `v2`, `v3`, ... with monotonically harder
distributions. Passing a version is not a terminal state; it is a checkpoint
that unlocks generating the next version.

- **v1** — baseline competence demonstration. The construction is shown clearly.
- **v2** — distributional shift: longer chains, deeper nesting, rarer
  vocabulary, paraphrased surface forms.
- **v3** — adversarial: items generated specifically by inspecting model
  failures on v2.
- **v4+** — out-of-distribution: items drawn from domains, registers, or
  constructions not present at training time.

Score is always reported as a tuple `(v1_score, v2_score, v3_score, ...)`,
never collapsed to a single number.

## Rule 3 — Adversarial regeneration on pass

When a model passes a version (>=95% on the public test set with >=90% on
private holdout), the next version is generated by adversarial process:

- Human review finds construction families the model handled accidentally
  rather than structurally.
- A separate generator produces items targeting the weakest decile of the
  previous version.
- The new version is reviewed for legitimacy — no impossible items, no
  ambiguous items, no items that depend on world knowledge the system was
  never given.

## Rule 4 — Frontier baseline tracking

For each lane, a baseline score is computed for at least one frontier
transformer-based model on the same public test set. Baselines are:

- Re-scored every time a version is cut.
- Published alongside CORE's score.
- Never tuned, never prompt-engineered to maximize — the prompt is the eval
  task as written.

## Rule 5 — Honest reporting

- Failures are reported with the same prominence as passes.
- Confidence intervals on every score (bootstrapped over the test set).
- Per-construction breakdowns published — never a single aggregate hiding
  structural failures.
- Regressions across versions are surfaced, never silently dropped.
- "Did not test" is a valid result; "tested and failed" is preferred over
  "did not test."

If a number cannot be reported honestly under these rules, the lane is not
ready. Do not ship the lane.

---

## Eval lane directory contract

Every eval lane lives in `evals/<lane_name>/` with this layout:

```
evals/<lane_name>/
  contract.md           # what the lane measures, scoring rubric, pass thresholds
  dev/                  # dev set, freely visible
  public/v1/            # public test set, version 1
  public/v2/            # ...
  holdouts/             # encrypted, sealed
  runner.py             # deterministic scorer
  baselines/            # frontier model scores per version
  results/              # CORE scores per version per release
```

A lane without a `contract.md` does not run.

---

## References

- ADR-0016: Capability Roadmap
- `docs/capability_roadmap.md`: Full phased plan

---

## Lane-shape registry (ADR-0109)

ADR-0091's Domain Pack Contract v1 introduced a `dev/public/holdout`
discipline that every ratified pack must declare. ADR-0106 added a
reviewer-signed audit-passed promotion gate that consults those same
lane outputs at the ledger level. ADR-0109 then formalized the rule
that **threshold dispatch is lane-shape-aware**, not lane-uniform.

### What this means for new lanes

Adding a new eval lane requires deciding which shape it reports:

| Shape | Required metrics | Threshold |
|---|---|---|
| `cognition_shape` | `surface_groundedness`, `term_capture_rate`, `intent_accuracy`, `versor_closure_rate` | 0.95 / 0.85 / 0.95 / 1.0 |
| `accuracy_shape` | `accuracy` (or `passed`/`total` fallback) | ≥ 0.95 |
| `inference_shape` | `all_pass_rate`, `replay_determinism`, `overall_pass` | 0.95 / 1.0 / true |
| `refusal_shape` | `by_class[*].n`, `by_class[*].refused`, `by_class[*].fabricated` | refused == n, fabricated == 0 |
| `symbolic_logic_shape` | `accuracy` | ≥ 0.95 |

A lane that does not fit any existing shape **must not be silently
broadened**. The path is:

1. Open an ADR amending ADR-0109 to add the new shape.
2. Add the shape checker to `SHAPE_CHECKERS` in
   `core/capability/expert_demo.py`.
3. Add the lane → shape mapping to `LANE_SHAPE_REGISTRY`.

A lane id absent from the registry is **fail-closed** at the
audit-passed gate (reason:
`lane <id> has no registered shape — introduce via ADR amendment`).
Unregistered lanes can still run as exploration; they just cannot
contribute evidence to a `reviewer-signed audit_passed` promotion.

### Holdout-runner gating (ADR-0105)

`split='holdout'` runs go through `evals.holdout_runner._decrypt_holdout`,
which expects either:

- `holdouts/v1/cases.jsonl.age` (sealed, requires `CORE_HOLDOUT_KEY`), **or**
- `holdouts/v1/cases_plaintext.jsonl` (dev-mode fallback, no key required).

A bare `holdouts/v1/cases.jsonl` is invisible to the runner. Lanes
authored before ADR-0105 must either rename their plaintext file or
seal it against an age recipient to be runnable on the `holdout`
split.

---

## References

- ADR-0091: Domain Pack Contract v1 (`evals/<lane>/holdouts/` discipline)
- ADR-0105: Sealed-holdout encryption (dev-mode fallback preserved)
- ADR-0106: Expert-demo promotion contract (consumes lane results)
- ADR-0109: Lane-shape-aware thresholds (this section's authority)
