core/evals
Shay 119d97f9c0 feat(evals): v2 lanes for calibration and symbolic-logic
Closes Phase 2 v2 coverage — all five lanes now pass v2 public + holdouts.

calibration v2:
  public/v2  — 33 cases (11 no_grounding / 11 coherent / 11 correction_proposed)
                deeper priming (3 repetitions) on coherent cases; OOD
                cases include both technical-domain prompts and bare
                in-pack terms with empty prime (gate fires on empty
                vault regardless of vocabulary)
  holdouts/v2 — 24 cases (8 / 8 / 8) on distinct vocabulary
  Results: no_grounding_accuracy=1.0, coherent_accuracy=1.0,
            correction_proposed_accuracy=1.0 on both splits.

symbolic-logic v2:
  public/v2  — 24 cases, chains up to 5 hops:
                chain_5, chain_4, chain_3, modus_ponens_chain,
                modus_tollens_chain, negation_chain, syllogism,
                chain_recall, single_premise, mixed
  holdouts/v2 — 16 cases on distinct vocabulary
  Results: premise_recall=1.0, replay_determinism=1.0,
            proposal_storage=1.0 on both splits.

  Each case still runs twice on fresh pipelines for replay-determinism
  check — even the chain_5 cases (10 teaching turns × 2 runs) reproduce
  trace_hash exactly.

Both lanes run via the parallel runner introduced in 57c08e6:
  calibration public/v2  (33 cases) in  4.9s
  calibration holdouts/v2 (24 cases) in  3.7s
  symbolic_logic public/v2 (24 cases) in 21.9s
  symbolic_logic holdouts/v2 (16 cases) in 16.5s
  (symbolic_logic is slower per-case because of the 2x replay run +
   long chains)

Test suite: 596 passing (no regression).

Phase 2 v2 milestone: all five lanes complete + v1 baselines documented.
Next: v3 lanes for at least two of the five (roadmap exit gate).
2026-05-16 13:17:41 -07:00
..
adversarial_identity feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
calibration feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
grammatical_coverage feat(evals): grammatical-coverage v2 cases - 36 cases with deeper nesting and rarer vocabulary (100% pass) 2026-05-16 06:40:55 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
monotonic_learning feat(evals): v2 lanes — monotonic-learning + provenance 2026-05-16 13:03:28 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
reports Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00