Shay 1e01f7794e feat(evals): Phase 0 — benchmark methodology lock-in and eval framework

Implement the eval infrastructure defined in ADR-0016 before building new
eval lanes. This establishes the discipline that governs the entire
capability roadmap.

- Generic eval framework (evals/framework.py): lane discovery, versioned
  scoring, result persistence
- Cognition lane retrofitted into new convention: 45 cases split into
  stratified dev (13) / public v1 (13) / holdout (19) sets with contract,
  runner, and recorded results
- Generalized `core eval <lane>` CLI: dynamic lane discovery, --list,
  --version, --split, --save, --json flags
- Holdout runner scaffold: plaintext fallback, encryption interface ready
- Baseline runner scaffold: pluggable frontier model interface
- Fix: CognitiveTurnPipeline.run() crashed on turn_log[-1] when the
  unknown-domain gate returned a stub without appending to turn_log
- ADR-0016, eval_methodology.md, PROGRESS.md, capability gates session log

Phase 0 exit audit found two methodology issues:
1. Pipeline turn_log crash (fixed here)
2. Versor drift in multi-turn sessions (pre-existing, under investigation)

2026-05-15 22:36:53 -07:00

1.7 KiB

Raw Blame History

Cognition Eval Lane — Contract

Lane: cognition Version: v1 Created: 2026-05-15

What this lane measures

End-to-end cognitive pipeline correctness: given a natural-language prompt, does the CognitiveTurnPipeline produce a response that:

Classifies intent correctly.
Captures expected domain terms in the realized surface.
Contains expected surface fragments (grounding check).
Maintains versor closure (versor_condition < 1e-6).
Produces a deterministic trace hash across runs.

Scoring rubric

Each case produces five binary signals. Lane-level metrics are rates over cases:

Metric	Definition	v1 pass threshold
`intent_accuracy`	Fraction of cases with correct intent classification	>= 0.90
`term_capture_rate`	Fraction of expected terms found in surface	>= 0.80
`surface_groundedness`	Fraction of cases where all expected surface fragments present	>= 0.80
`versor_closure_rate`	Fraction of cases with `versor_condition < 1e-6`	1.00
`determinism`	All trace hashes identical across 2 runs	true

Pass criteria

Public v1: All five metrics meet or exceed thresholds above.
Holdout: intent_accuracy >= 0.85, versor_closure_rate == 1.00.

Version escalation plan

v2: Longer prompts, paraphrased surface forms, rarer vocabulary (e.g. "elucidate" instead of "what is"), multi-clause prompts.
v3: Adversarial items targeting weakest category from v2 results.

Categories tested

definition, comparison, cause, procedure, recall, correction, verification, unknown

Runner

runner.py in this directory. Invoked via core eval cognition.

1.7 KiB Raw Blame History