core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/evals
History
Shay 6b25069da8 feat(adr-0054): vault recall indexing/batching + holdout split wired 
Two doctrine-aligned CLAUDE.md items closed together.

Part 1 — vault indexing + batching (item #4):
- VaultStore lazy _matrix_cache (invalidated on store / reproject /
  eviction); vault_recall(prebuilt_matrix=...) skips deque→ndarray
  rebuild on hot path
- New vault_recall_batch + VaultStore.recall_batch — B queries
  scored in one component-serial sweep, bit-identical to per-query
  vault_recall (3 seeds × 7 queries × N=137 parity test)
- No approximation, no hot-path repair, scoring arithmetic
  unchanged

Part 2 — holdout split wired:
- LaneInfo.holdout_cases_path resolves plaintext holdouts in fixed
  priority; sealed (.age) holdouts stay in holdout_runner
- framework.run_lane(split="holdout") + argparse --split choices
- First official cognition holdout numbers: 19 cases, intent 100%,
  surface 94.7%, term_capture 70.8%, versor 100% — single miss is
  predicted correction_truth_040 (ADR-0053 scope-limit)

Tests: 21 new vault tests + 10 new framework tests. Lanes: smoke
67, cognition 121, runtime 19, teaching 17, packs 6, algebra 132 —
all green. versor_condition < 1e-6 invariant preserved.
2026-05-18 07:58:57 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
articulation_of_status feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
audit_tour feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition feat(adr-0053): cognition lane closure — corpus expansion + CORRECTION acknowledgement 2026-05-18 07:43:39 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
elementary_mathematics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
forward_semantic_control docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
hebrew_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
identity_divergence feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
industry_demos fix(adr-0046): make forward-graph-constraint branch mergeable 2026-05-18 05:57:46 -07:00
inference_closure feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
introspection feat(phase3): core/cognition/explain.py — close Gap 3 introspection 2026-05-16 15:09:48 -07:00
koine_greek_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
long_context_cost feat(adr-0044, adr-0045): domain ethics pack + long-context comparison 2026-05-17 22:31:47 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
refusal_calibration feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
results feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
sample_efficiency feat(phase4): sample-efficiency v1 — first quantitative-curve lane 2026-05-16 15:39:28 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
zero_code_domain_acquisition
__init__.py
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
CLAIMS.md feat(bench): bench cost — $/1000 turns + latency, with disclosed assumptions 2026-05-17 10:53:08 -07:00
cognition_cases.jsonl
framework.py feat(adr-0054): vault recall indexing/batching + holdout split wired 2026-05-18 07:58:57 -07:00
holdout_runner.py
metrics.py
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py