696f62abdd
The word "expert" in the previous status name implied raw-capability parity
with frontier LLMs on the same benchmark — which the gate does NOT verify.
What the gate actually verifies is CORE *claim-shape compliance*:
* signed digest (replay-reproducible from on-disk lane results)
* replay determinism (same inputs → byte-equal trace_hash)
* typed refusal (fabrication refused, not paraphrased)
* exact recall (no ANN, no cosine, no attention bottleneck)
* grounding-source provenance
These are claim shapes a transformer LLM cannot structurally produce
regardless of raw accuracy. A frontier LLM might score higher on the
same benchmark but cannot pass this contract.
Rename scope (semantics only, per ADR-0113):
status string "expert-demo" → "audit-passed"
predicate key predicates.expert_demo → predicates.audit_passed
reason key expert_demo_reason → audit_passed_reason
YAML key expert_demo_claims → audit_passed_claims
CLI command core demo expert → core demo audit-passed
output dir evals/expert_demos/ → evals/audit_passed/
artifact filenames expert_demo.{json,html} → audit_passed.{json,html}
HTML title CORE Expert-Demo: X → CORE Audit-Passed: X
Internal Python identifiers (module/file/function/class names like
`expert_demo.py`, `evaluate_expert_demo`, `ExpertDemoClaim`,
`expert_demo_claim_for`) are deliberately kept to minimize churn. ADR
file titles (ADR-0106..0112) preserved as historical record.
`expert` namespace reserved for ADR-0114+: an actual capability tier
above `audit-passed` backed by a public benchmark with a stated
threshold. ADR-0114 proposes the first such target — GSM8K-math —
laying out a falsifiable 7-phase arc (parser → solver → verifier →
stepped-realizer → eval lane → first `expert` ledger tier promotion).
Tests: 184 directly-affected tests green (140 capability/expert-demo
suite + 34 demo/audit-tour + 10 correction-cue). Smoke suite 67/67.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
162 lines
6.1 KiB
Markdown
162 lines
6.1 KiB
Markdown
# Eval Methodology — Benchmark Discipline Contract
|
|
|
|
**Status:** Accepted (ADR-0016, extended by ADR-0109)
|
|
**Last updated:** 2026-05-22 (lane-shape registry section added)
|
|
|
|
This document defines the five rules that govern every eval lane in the CORE
|
|
capability roadmap. No exceptions per phase. A lane that does not satisfy these
|
|
rules is exploration, not a gate.
|
|
|
|
---
|
|
|
|
## Rule 1 — Three-set split per lane
|
|
|
|
Every lane maintains three disjoint corpora:
|
|
|
|
- **Dev set.** Freely visible during development. Used to iterate.
|
|
- **Public test set.** Visible, but tuning against it is forbidden. Scored at
|
|
version-cut time only. Drift in dev-vs-public scores is a red flag for
|
|
overfitting.
|
|
- **Private holdout.** Sealed. Never read by Claude, never committed in
|
|
plaintext, only scored by a clean-room runner at release events. Stored
|
|
encrypted in `evals/holdouts/` with key held by the human reviewer.
|
|
|
|
If a lane has only a dev set, it does not count as a gate. It is exploration.
|
|
|
|
## Rule 2 — Versioned difficulty escalation
|
|
|
|
Each lane has versions: `v1`, `v2`, `v3`, ... with monotonically harder
|
|
distributions. Passing a version is not a terminal state; it is a checkpoint
|
|
that unlocks generating the next version.
|
|
|
|
- **v1** — baseline competence demonstration. The construction is shown clearly.
|
|
- **v2** — distributional shift: longer chains, deeper nesting, rarer
|
|
vocabulary, paraphrased surface forms.
|
|
- **v3** — adversarial: items generated specifically by inspecting model
|
|
failures on v2.
|
|
- **v4+** — out-of-distribution: items drawn from domains, registers, or
|
|
constructions not present at training time.
|
|
|
|
Score is always reported as a tuple `(v1_score, v2_score, v3_score, ...)`,
|
|
never collapsed to a single number.
|
|
|
|
## Rule 3 — Adversarial regeneration on pass
|
|
|
|
When a model passes a version (>=95% on the public test set with >=90% on
|
|
private holdout), the next version is generated by adversarial process:
|
|
|
|
- Human review finds construction families the model handled accidentally
|
|
rather than structurally.
|
|
- A separate generator produces items targeting the weakest decile of the
|
|
previous version.
|
|
- The new version is reviewed for legitimacy — no impossible items, no
|
|
ambiguous items, no items that depend on world knowledge the system was
|
|
never given.
|
|
|
|
## Rule 4 — Frontier baseline tracking
|
|
|
|
For each lane, a baseline score is computed for at least one frontier
|
|
transformer-based model on the same public test set. Baselines are:
|
|
|
|
- Re-scored every time a version is cut.
|
|
- Published alongside CORE's score.
|
|
- Never tuned, never prompt-engineered to maximize — the prompt is the eval
|
|
task as written.
|
|
|
|
## Rule 5 — Honest reporting
|
|
|
|
- Failures are reported with the same prominence as passes.
|
|
- Confidence intervals on every score (bootstrapped over the test set).
|
|
- Per-construction breakdowns published — never a single aggregate hiding
|
|
structural failures.
|
|
- Regressions across versions are surfaced, never silently dropped.
|
|
- "Did not test" is a valid result; "tested and failed" is preferred over
|
|
"did not test."
|
|
|
|
If a number cannot be reported honestly under these rules, the lane is not
|
|
ready. Do not ship the lane.
|
|
|
|
---
|
|
|
|
## Eval lane directory contract
|
|
|
|
Every eval lane lives in `evals/<lane_name>/` with this layout:
|
|
|
|
```
|
|
evals/<lane_name>/
|
|
contract.md # what the lane measures, scoring rubric, pass thresholds
|
|
dev/ # dev set, freely visible
|
|
public/v1/ # public test set, version 1
|
|
public/v2/ # ...
|
|
holdouts/ # encrypted, sealed
|
|
runner.py # deterministic scorer
|
|
baselines/ # frontier model scores per version
|
|
results/ # CORE scores per version per release
|
|
```
|
|
|
|
A lane without a `contract.md` does not run.
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- ADR-0016: Capability Roadmap
|
|
- `docs/capability_roadmap.md`: Full phased plan
|
|
|
|
---
|
|
|
|
## Lane-shape registry (ADR-0109)
|
|
|
|
ADR-0091's Domain Pack Contract v1 introduced a `dev/public/holdout`
|
|
discipline that every ratified pack must declare. ADR-0106 added a
|
|
reviewer-signed audit-passed promotion gate that consults those same
|
|
lane outputs at the ledger level. ADR-0109 then formalized the rule
|
|
that **threshold dispatch is lane-shape-aware**, not lane-uniform.
|
|
|
|
### What this means for new lanes
|
|
|
|
Adding a new eval lane requires deciding which shape it reports:
|
|
|
|
| Shape | Required metrics | Threshold |
|
|
|---|---|---|
|
|
| `cognition_shape` | `surface_groundedness`, `term_capture_rate`, `intent_accuracy`, `versor_closure_rate` | 0.95 / 0.85 / 0.95 / 1.0 |
|
|
| `accuracy_shape` | `accuracy` (or `passed`/`total` fallback) | ≥ 0.95 |
|
|
| `inference_shape` | `all_pass_rate`, `replay_determinism`, `overall_pass` | 0.95 / 1.0 / true |
|
|
| `refusal_shape` | `by_class[*].n`, `by_class[*].refused`, `by_class[*].fabricated` | refused == n, fabricated == 0 |
|
|
| `symbolic_logic_shape` | `accuracy` | ≥ 0.95 |
|
|
|
|
A lane that does not fit any existing shape **must not be silently
|
|
broadened**. The path is:
|
|
|
|
1. Open an ADR amending ADR-0109 to add the new shape.
|
|
2. Add the shape checker to `SHAPE_CHECKERS` in
|
|
`core/capability/expert_demo.py`.
|
|
3. Add the lane → shape mapping to `LANE_SHAPE_REGISTRY`.
|
|
|
|
A lane id absent from the registry is **fail-closed** at the
|
|
audit-passed gate (reason:
|
|
`lane <id> has no registered shape — introduce via ADR amendment`).
|
|
Unregistered lanes can still run as exploration; they just cannot
|
|
contribute evidence to a `reviewer-signed audit_passed` promotion.
|
|
|
|
### Holdout-runner gating (ADR-0105)
|
|
|
|
`split='holdout'` runs go through `evals.holdout_runner._decrypt_holdout`,
|
|
which expects either:
|
|
|
|
- `holdouts/v1/cases.jsonl.age` (sealed, requires `CORE_HOLDOUT_KEY`), **or**
|
|
- `holdouts/v1/cases_plaintext.jsonl` (dev-mode fallback, no key required).
|
|
|
|
A bare `holdouts/v1/cases.jsonl` is invisible to the runner. Lanes
|
|
authored before ADR-0105 must either rename their plaintext file or
|
|
seal it against an age recipient to be runnable on the `holdout`
|
|
split.
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- ADR-0091: Domain Pack Contract v1 (`evals/<lane>/holdouts/` discipline)
|
|
- ADR-0105: Sealed-holdout encryption (dev-mode fallback preserved)
|
|
- ADR-0106: Expert-demo promotion contract (consumes lane results)
|
|
- ADR-0109: Lane-shape-aware thresholds (this section's authority)
|