core/evals
Shay 0053648efd feat(evals): symbolic-logic lane v1 — premise-chain foundations
Adds the fourth Phase 2 lane. v1 measures the structural foundations
on which a future inference engine would be built:

  M1. premise_recall    — probe vault_hits >= min after chain teaching
  M2. replay_determinism — same chain + probe → same trace_hash
  M3. proposal_storage  — correction premises store as PackMutationProposals

Patterns covered: modus_ponens_chain, modus_tollens_chain, syllogism,
negation, chain_recall (up to 4-hop chains).

v1 results across 38 cases (8 dev + 18 public + 12 holdouts):
  premise_recall=1.0, replay_determinism=1.0, proposal_storage=1.0.

Each case runs twice on fresh CognitiveTurnPipelines to verify the
trace_hash matches — confirming deterministic replay over premise chains.

Architectural finding logged in evals/symbolic_logic/gaps.md:

  CORE has no first-class inference operator. Chain "inference" today is
  emergent from teaching-store commits + cumulative vault recall, not a
  named-rule symbolic engine. v1 honestly tests what CORE deterministically
  *does* (store, replay, recall chains) without overclaiming that CORE
  reasons symbolically. v2 would assert specific transitive recall
  contents in the probe surface, which requires either a
  PropositionGraph traversal operator or pack-axiom rules — both filed
  as suggested follow-up work.
2026-05-16 12:34:55 -07:00
..
calibration feat(evals): calibration lane v1 — typed cognitive signals 2026-05-16 12:22:16 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
grammatical_coverage feat(evals): grammatical-coverage v2 cases - 36 cases with deeper nesting and rarer vocabulary (100% pass) 2026-05-16 06:40:55 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
monotonic_learning feat(evals): monotonic-learning lane v1 — no regression across cycles 2026-05-16 11:56:34 -07:00
provenance feat(evals): provenance lane v1 — replay determinism + source back-pointers 2026-05-16 11:45:00 -07:00
reports Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
symbolic_logic feat(evals): symbolic-logic lane v1 — premise-chain foundations 2026-05-16 12:34:55 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00