benchmarks/cost.py measures CORE per-turn cost honestly:
Measured (no estimation):
- turns, wall_seconds_total, cpu_seconds_total
- latency stats: min / median / p95 / max in ms
- throughput in turns per second
Derived with disclosed assumptions:
- USD per 1000 turns at AWS t3.medium on-demand
($0.0416/hr, source cited in CloudReference.source_note)
- Frontier pricing comparison: Anthropic Claude Sonnet 4.5 /
Haiku 4.5 and OpenAI GPT-4o, public per-token rates with
source notes, derived using a conservative 20-in / 40-out
tokens-per-turn assumption.
Explicitly NOT reported:
- Joules per turn. Honest energy measurement requires RAPL
(Linux) or IOKit/powermetrics (macOS) with privileged access
that a plain Python process cannot get. Reporting a fabricated
figure from a hand-waved TDP would violate "speculation is not
evidence." cpu_seconds_total is the available proxy.
CLI:
core bench --suite cost --runs 100
Measured numbers (100 turns, "What is truth?", warmup 5):
median latency: 444.88 ms
p95 latency: 447.10 ms
throughput: 2.61 turns/s
$/1000 turns: $0.0044
vs frontier: 48–149× cheaper depending on provider
CLAIMS.md Tier 4 cost/latency rows updated with real numbers
replacing TBDs. evals/reports/cost_latest.json committed as the
captured baseline.
Verified: smoke (67), bench --suite cost CLI works.
79a4125d24