benchmarks/cost.py measures CORE per-turn cost honestly:
Measured (no estimation):
- turns, wall_seconds_total, cpu_seconds_total
- latency stats: min / median / p95 / max in ms
- throughput in turns per second
Derived with disclosed assumptions:
- USD per 1000 turns at AWS t3.medium on-demand
($0.0416/hr, source cited in CloudReference.source_note)
- Frontier pricing comparison: Anthropic Claude Sonnet 4.5 /
Haiku 4.5 and OpenAI GPT-4o, public per-token rates with
source notes, derived using a conservative 20-in / 40-out
tokens-per-turn assumption.
Explicitly NOT reported:
- Joules per turn. Honest energy measurement requires RAPL
(Linux) or IOKit/powermetrics (macOS) with privileged access
that a plain Python process cannot get. Reporting a fabricated
figure from a hand-waved TDP would violate "speculation is not
evidence." cpu_seconds_total is the available proxy.
CLI:
core bench --suite cost --runs 100
Measured numbers (100 turns, "What is truth?", warmup 5):
median latency: 444.88 ms
p95 latency: 447.10 ms
throughput: 2.61 turns/s
$/1000 turns: $0.0044
vs frontier: 48–149× cheaper depending on provider
CLAIMS.md Tier 4 cost/latency rows updated with real numbers
replacing TBDs. evals/reports/cost_latest.json committed as the
captured baseline.
Verified: smoke (67), bench --suite cost CLI works.
Audit of the one-mutation-path invariant (ADR-0021 §3) found three leaks
where pack authority or session-state writes could substitute for coherence
judgment. All three landed fixes or partial closures in this push.
Leaks closed:
- Leak A: pack vocab defaulted to COHERENT — flipped to SPECULATIVE in
language_packs/{compiler,schema}.py; docstring corrected to align with
ADR-0021 (it was rationalizing the leak).
- Leak B: vault.recall was epistemic-blind — VaultStore.store() now stamps
every entry with EpistemicStatus (default SPECULATIVE); recall(min_status=)
filters to admissible-as-evidence tier. All 4 vault-write sites updated.
- Leak C (write-side): generate/proposition.py:198 stored articulated
propositions unmarked — now stamps SPECULATIVE, breaking the
fabrication-feedback loop in principle. Read-side audit of 5 call sites
is the residual.
New architectural invariants (tests/test_architectural_invariants.py):
- INV-21: one-mutation-path allowlist (caught Leak C on first run)
- INV-22: pack lexicon default is SPECULATIVE (Leak A guard)
- INV-23: vault recall epistemic-aware (Leak B guard)
New eval lanes:
- teaching_injection_resistance — ships GREEN at 1.00/1.00/0 (the
structural anti-injection claim is real and measurable)
- refusal_calibration — honest gap: 0% refusal, 0% fabrication
- contradiction_detection — honest gap: 50% flag via versor-delta heuristic,
100% false-positive; motivates the proper coherence-checker
- articulation_of_status — honest gap: 0% speculative articulation, 60%
false certainty; output-side leak surface
New benchmarks:
- benchmarks/footprint.py — total deployed runtime is 7.06 MiB
(109,358x smaller than Llama 3.1 405B, runs offline, no GPU)
- benchmarks/learning_curve.py — monotonic + replay-deterministic curve
per lane
Documentation:
- docs/truth_seeking_schema.md — foundational architectural commitment,
five rules, mapped to human failure modes, leaks published openly
- evals/CLAIMS.md — five-tier public claims doc; Tier 4.5 publishes
known gaps with named fixes; verification contract at top
- README.md — new pillar between algebraic substrate and language pillar
Includes in-flight formation pipeline scaffolding (formation/, tests/formation/,
docs/formation_pipeline_plan.md) and minor CLI/contracts/gitignore edits
that were already in the working tree at session start.
Verification: 798 passed, 2 skipped, 1 deselected (pre-existing pack-count
test drift unrelated to schema changes).
