1e01f7794e
Implement the eval infrastructure defined in ADR-0016 before building new eval lanes. This establishes the discipline that governs the entire capability roadmap. - Generic eval framework (evals/framework.py): lane discovery, versioned scoring, result persistence - Cognition lane retrofitted into new convention: 45 cases split into stratified dev (13) / public v1 (13) / holdout (19) sets with contract, runner, and recorded results - Generalized `core eval <lane>` CLI: dynamic lane discovery, --list, --version, --split, --save, --json flags - Holdout runner scaffold: plaintext fallback, encryption interface ready - Baseline runner scaffold: pluggable frontier model interface - Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the unknown-domain gate returned a stub without appending to turn_log - ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log Phase 0 exit audit found two methodology issues: 1. Pipeline turn_log crash (fixed here) 2. Versor drift in multi-turn sessions (pre-existing, under investigation)
48 lines
1.7 KiB
Markdown
48 lines
1.7 KiB
Markdown
# Cognition Eval Lane — Contract
|
|
|
|
**Lane:** `cognition`
|
|
**Version:** v1
|
|
**Created:** 2026-05-15
|
|
|
|
## What this lane measures
|
|
|
|
End-to-end cognitive pipeline correctness: given a natural-language prompt, does
|
|
the `CognitiveTurnPipeline` produce a response that:
|
|
|
|
1. Classifies intent correctly.
|
|
2. Captures expected domain terms in the realized surface.
|
|
3. Contains expected surface fragments (grounding check).
|
|
4. Maintains versor closure (`versor_condition < 1e-6`).
|
|
5. Produces a deterministic trace hash across runs.
|
|
|
|
## Scoring rubric
|
|
|
|
Each case produces five binary signals. Lane-level metrics are rates over cases:
|
|
|
|
| Metric | Definition | v1 pass threshold |
|
|
|--------|-----------|-------------------|
|
|
| `intent_accuracy` | Fraction of cases with correct intent classification | >= 0.90 |
|
|
| `term_capture_rate` | Fraction of expected terms found in surface | >= 0.80 |
|
|
| `surface_groundedness` | Fraction of cases where all expected surface fragments present | >= 0.80 |
|
|
| `versor_closure_rate` | Fraction of cases with `versor_condition < 1e-6` | 1.00 |
|
|
| `determinism` | All trace hashes identical across 2 runs | true |
|
|
|
|
## Pass criteria
|
|
|
|
- **Public v1:** All five metrics meet or exceed thresholds above.
|
|
- **Holdout:** intent_accuracy >= 0.85, versor_closure_rate == 1.00.
|
|
|
|
## Version escalation plan
|
|
|
|
- **v2:** Longer prompts, paraphrased surface forms, rarer vocabulary (e.g.
|
|
"elucidate" instead of "what is"), multi-clause prompts.
|
|
- **v3:** Adversarial items targeting weakest category from v2 results.
|
|
|
|
## Categories tested
|
|
|
|
definition, comparison, cause, procedure, recall, correction, verification,
|
|
unknown
|
|
|
|
## Runner
|
|
|
|
`runner.py` in this directory. Invoked via `core eval cognition`.
|