1e01f7794e
Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)
104 lines
3.7 KiB
Markdown
104 lines
3.7 KiB
Markdown
# Eval Methodology — Benchmark Discipline Contract
|
|
|
|
**Status:** Accepted (ADR-0016)
|
|
**Last updated:** 2026-05-15
|
|
|
|
This document defines the five rules that govern every eval lane in the CORE
|
|
capability roadmap. No exceptions per phase. A lane that does not satisfy these
|
|
rules is exploration, not a gate.
|
|
|
|
---
|
|
|
|
## Rule 1 — Three-set split per lane
|
|
|
|
Every lane maintains three disjoint corpora:
|
|
|
|
- **Dev set.** Freely visible during development. Used to iterate.
|
|
- **Public test set.** Visible, but tuning against it is forbidden. Scored at
|
|
version-cut time only. Drift in dev-vs-public scores is a red flag for
|
|
overfitting.
|
|
- **Private holdout.** Sealed. Never read by Claude, never committed in
|
|
plaintext, only scored by a clean-room runner at release events. Stored
|
|
encrypted in `evals/holdouts/` with key held by the human reviewer.
|
|
|
|
If a lane has only a dev set, it does not count as a gate. It is exploration.
|
|
|
|
## Rule 2 — Versioned difficulty escalation
|
|
|
|
Each lane has versions: `v1`, `v2`, `v3`, ... with monotonically harder
|
|
distributions. Passing a version is not a terminal state; it is a checkpoint
|
|
that unlocks generating the next version.
|
|
|
|
- **v1** — baseline competence demonstration. The construction is shown clearly.
|
|
- **v2** — distributional shift: longer chains, deeper nesting, rarer
|
|
vocabulary, paraphrased surface forms.
|
|
- **v3** — adversarial: items generated specifically by inspecting model
|
|
failures on v2.
|
|
- **v4+** — out-of-distribution: items drawn from domains, registers, or
|
|
constructions not present at training time.
|
|
|
|
Score is always reported as a tuple `(v1_score, v2_score, v3_score, ...)`,
|
|
never collapsed to a single number.
|
|
|
|
## Rule 3 — Adversarial regeneration on pass
|
|
|
|
When a model passes a version (>=95% on the public test set with >=90% on
|
|
private holdout), the next version is generated by adversarial process:
|
|
|
|
- Human review finds construction families the model handled accidentally
|
|
rather than structurally.
|
|
- A separate generator produces items targeting the weakest decile of the
|
|
previous version.
|
|
- The new version is reviewed for legitimacy — no impossible items, no
|
|
ambiguous items, no items that depend on world knowledge the system was
|
|
never given.
|
|
|
|
## Rule 4 — Frontier baseline tracking
|
|
|
|
For each lane, a baseline score is computed for at least one frontier
|
|
transformer-based model on the same public test set. Baselines are:
|
|
|
|
- Re-scored every time a version is cut.
|
|
- Published alongside CORE's score.
|
|
- Never tuned, never prompt-engineered to maximize — the prompt is the eval
|
|
task as written.
|
|
|
|
## Rule 5 — Honest reporting
|
|
|
|
- Failures are reported with the same prominence as passes.
|
|
- Confidence intervals on every score (bootstrapped over the test set).
|
|
- Per-construction breakdowns published — never a single aggregate hiding
|
|
structural failures.
|
|
- Regressions across versions are surfaced, never silently dropped.
|
|
- "Did not test" is a valid result; "tested and failed" is preferred over
|
|
"did not test."
|
|
|
|
If a number cannot be reported honestly under these rules, the lane is not
|
|
ready. Do not ship the lane.
|
|
|
|
---
|
|
|
|
## Eval lane directory contract
|
|
|
|
Every eval lane lives in `evals/<lane_name>/` with this layout:
|
|
|
|
```
|
|
evals/<lane_name>/
|
|
contract.md # what the lane measures, scoring rubric, pass thresholds
|
|
dev/ # dev set, freely visible
|
|
public/v1/ # public test set, version 1
|
|
public/v2/ # ...
|
|
holdouts/ # encrypted, sealed
|
|
runner.py # deterministic scorer
|
|
baselines/ # frontier model scores per version
|
|
results/ # CORE scores per version per release
|
|
```
|
|
|
|
A lane without a `contract.md` does not run.
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- ADR-0016: Capability Roadmap
|
|
- `docs/capability_roadmap.md`: Full phased plan
|