core/benchmarks
Shay 64c5bc4619 feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants
Audit of the one-mutation-path invariant (ADR-0021 §3) found three leaks
where pack authority or session-state writes could substitute for coherence
judgment. All three landed fixes or partial closures in this push.

Leaks closed:
- Leak A: pack vocab defaulted to COHERENT — flipped to SPECULATIVE in
  language_packs/{compiler,schema}.py; docstring corrected to align with
  ADR-0021 (it was rationalizing the leak).
- Leak B: vault.recall was epistemic-blind — VaultStore.store() now stamps
  every entry with EpistemicStatus (default SPECULATIVE); recall(min_status=)
  filters to admissible-as-evidence tier. All 4 vault-write sites updated.
- Leak C (write-side): generate/proposition.py:198 stored articulated
  propositions unmarked — now stamps SPECULATIVE, breaking the
  fabrication-feedback loop in principle. Read-side audit of 5 call sites
  is the residual.

New architectural invariants (tests/test_architectural_invariants.py):
- INV-21: one-mutation-path allowlist (caught Leak C on first run)
- INV-22: pack lexicon default is SPECULATIVE (Leak A guard)
- INV-23: vault recall epistemic-aware (Leak B guard)

New eval lanes:
- teaching_injection_resistance — ships GREEN at 1.00/1.00/0 (the
  structural anti-injection claim is real and measurable)
- refusal_calibration — honest gap: 0% refusal, 0% fabrication
- contradiction_detection — honest gap: 50% flag via versor-delta heuristic,
  100% false-positive; motivates the proper coherence-checker
- articulation_of_status — honest gap: 0% speculative articulation, 60%
  false certainty; output-side leak surface

New benchmarks:
- benchmarks/footprint.py — total deployed runtime is 7.06 MiB
  (109,358x smaller than Llama 3.1 405B, runs offline, no GPU)
- benchmarks/learning_curve.py — monotonic + replay-deterministic curve
  per lane

Documentation:
- docs/truth_seeking_schema.md — foundational architectural commitment,
  five rules, mapped to human failure modes, leaks published openly
- evals/CLAIMS.md — five-tier public claims doc; Tier 4.5 publishes
  known gaps with named fixes; verification contract at top
- README.md — new pillar between algebraic substrate and language pillar

Includes in-flight formation pipeline scaffolding (formation/, tests/formation/,
docs/formation_pipeline_plan.md) and minor CLI/contracts/gitignore edits
that were already in the working tree at session start.

Verification: 798 passed, 2 skipped, 1 deselected (pre-existing pack-count
test drift unrelated to schema changes).
2026-05-17 07:27:41 -07:00
..
__init__.py feat: Full Proof — surface realizer join, Rust diffusion parity, benchmark harness 2026-05-15 17:39:14 -07:00
footprint.py feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
learning_curve.py feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
pipeline_profiler.py feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
replay_vs_llm.py feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
run_benchmarks.py feat: Full Proof — surface realizer join, Rust diffusion parity, benchmark harness 2026-05-15 17:39:14 -07:00
word_selection_tracer.py feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00