core/evals
Shay 64c5bc4619 feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants
Audit of the one-mutation-path invariant (ADR-0021 §3) found three leaks
where pack authority or session-state writes could substitute for coherence
judgment. All three landed fixes or partial closures in this push.

Leaks closed:
- Leak A: pack vocab defaulted to COHERENT — flipped to SPECULATIVE in
  language_packs/{compiler,schema}.py; docstring corrected to align with
  ADR-0021 (it was rationalizing the leak).
- Leak B: vault.recall was epistemic-blind — VaultStore.store() now stamps
  every entry with EpistemicStatus (default SPECULATIVE); recall(min_status=)
  filters to admissible-as-evidence tier. All 4 vault-write sites updated.
- Leak C (write-side): generate/proposition.py:198 stored articulated
  propositions unmarked — now stamps SPECULATIVE, breaking the
  fabrication-feedback loop in principle. Read-side audit of 5 call sites
  is the residual.

New architectural invariants (tests/test_architectural_invariants.py):
- INV-21: one-mutation-path allowlist (caught Leak C on first run)
- INV-22: pack lexicon default is SPECULATIVE (Leak A guard)
- INV-23: vault recall epistemic-aware (Leak B guard)

New eval lanes:
- teaching_injection_resistance — ships GREEN at 1.00/1.00/0 (the
  structural anti-injection claim is real and measurable)
- refusal_calibration — honest gap: 0% refusal, 0% fabrication
- contradiction_detection — honest gap: 50% flag via versor-delta heuristic,
  100% false-positive; motivates the proper coherence-checker
- articulation_of_status — honest gap: 0% speculative articulation, 60%
  false certainty; output-side leak surface

New benchmarks:
- benchmarks/footprint.py — total deployed runtime is 7.06 MiB
  (109,358x smaller than Llama 3.1 405B, runs offline, no GPU)
- benchmarks/learning_curve.py — monotonic + replay-deterministic curve
  per lane

Documentation:
- docs/truth_seeking_schema.md — foundational architectural commitment,
  five rules, mapped to human failure modes, leaks published openly
- evals/CLAIMS.md — five-tier public claims doc; Tier 4.5 publishes
  known gaps with named fixes; verification contract at top
- README.md — new pillar between algebraic substrate and language pillar

Includes in-flight formation pipeline scaffolding (formation/, tests/formation/,
docs/formation_pipeline_plan.md) and minor CLI/contracts/gitignore edits
that were already in the working tree at session start.

Verification: 798 passed, 2 skipped, 1 deselected (pre-existing pack-count
test drift unrelated to schema changes).
2026-05-17 07:27:41 -07:00
..
adversarial_identity docs(identity): empirical finding — fix #3 needs upstream ingest-gate work 2026-05-16 14:23:20 -07:00
articulation_of_status feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
contradiction_detection feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
elementary_mathematics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
hebrew_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
identity_divergence feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
inference_closure feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
introspection feat(phase3): core/cognition/explain.py — close Gap 3 introspection 2026-05-16 15:09:48 -07:00
koine_greek_fluency feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
long_context_cost feat(phase4): long-context-cost lane + ADR-0019 Stage 1 vault recall vectorisation 2026-05-16 16:39:30 -07:00
monotonic_learning feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
provenance feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
refusal_calibration feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
reports feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
sample_efficiency feat(phase4): sample-efficiency v1 — first quantitative-curve lane 2026-05-16 15:39:28 -07:00
symbolic_logic feat(evals): v2 lanes for calibration and symbolic-logic 2026-05-16 13:17:41 -07:00
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
zero_code_domain_acquisition feat began creation of zero code domain acquisition. did not complete yet. 2026-05-16 06:31:01 -07:00
__init__.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
baseline_runner.py feat(evals): frontier structural-zero baselines for Phase 2 v1 lanes 2026-05-16 12:45:28 -07:00
CLAIMS.md feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
cognition_cases.jsonl feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
framework.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
holdout_runner.py feat(evals): Phase 0 — benchmark methodology lock-in and eval framework 2026-05-15 22:36:53 -07:00
metrics.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00
parallel.py feat(evals): parallel runner + adversarial-identity v2 2026-05-16 13:10:26 -07:00
run_cognition_eval.py Add cognitive eval harness and calibration replay (#30) 2026-05-15 07:41:36 -07:00