core/evals
Shay 64268436fb feat(evals): calibration lane v1 — typed cognitive signals
Adds the third Phase 2 lane: calibration measures whether CORE's runtime
emits distinguishable, typed evidence for three cognitive states:

  no_grounding         vault_hits == 0 (gate fired, no recall)
  coherent             vault_hits > 0  (vault recall fired)
  correction_proposed  pack_mutation_proposal is not None

Each case runs on its own fresh CognitiveTurnPipeline to avoid
cross-case field-state drift (the gate's geometric recall score is
sensitive to vault content drift across turns).

v1 results: dev 12/12, public/v1 24/24, holdouts/v1 18/18 — all classes
score 1.0 across all splits.

Architectural findings logged in evals/calibration/gaps.md:

  1. The ingest gate fires on a *geometric* CGA-recall score, not on
     semantic OOD. 6/42 hand-chosen OOD prompts fire the gate with a
     warmed vault; the other 36 land geometrically near in-pack
     versors after morphological grounding. v1 measures the reliable
     recall/correction signals, not semantic OOD detection.

  2. CognitiveTurnPipeline.run() unconditionally overrides the
     runtime's gate-safety surface with the realizer surface. The OOD
     marker survives in walk_surface but not in surface. v1 classifies
     on vault_hits (preserved) rather than surface (overridden).

Both findings are filed as suggested follow-up work, not v1 blockers.
2026-05-16 12:22:16 -07:00
..
calibration feat(evals): calibration lane v1 — typed cognitive signals 2026-05-16 12:22:16 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
grammatical_coverage feat(evals): grammatical-coverage v2 cases - 36 cases with deeper nesting and rarer vocabulary (100% pass) 2026-05-16 06:40:55 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
monotonic_learning feat(evals): monotonic-learning lane v1 — no regression across cycles 2026-05-16 11:56:34 -07:00
provenance feat(evals): provenance lane v1 — replay determinism + source back-pointers 2026-05-16 11:45:00 -07:00
reports Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00