79a4125d24
benchmarks/cost.py measures CORE per-turn cost honestly:
Measured (no estimation):
- turns, wall_seconds_total, cpu_seconds_total
- latency stats: min / median / p95 / max in ms
- throughput in turns per second
Derived with disclosed assumptions:
- USD per 1000 turns at AWS t3.medium on-demand
($0.0416/hr, source cited in CloudReference.source_note)
- Frontier pricing comparison: Anthropic Claude Sonnet 4.5 /
Haiku 4.5 and OpenAI GPT-4o, public per-token rates with
source notes, derived using a conservative 20-in / 40-out
tokens-per-turn assumption.
Explicitly NOT reported:
- Joules per turn. Honest energy measurement requires RAPL
(Linux) or IOKit/powermetrics (macOS) with privileged access
that a plain Python process cannot get. Reporting a fabricated
figure from a hand-waved TDP would violate "speculation is not
evidence." cpu_seconds_total is the available proxy.
CLI:
core bench --suite cost --runs 100
Measured numbers (100 turns, "What is truth?", warmup 5):
median latency: 444.88 ms
p95 latency: 447.10 ms
throughput: 2.61 turns/s
$/1000 turns: $0.0044
vs frontier: 48–149× cheaper depending on provider
CLAIMS.md Tier 4 cost/latency rows updated with real numbers
replacing TBDs. evals/reports/cost_latest.json committed as the
captured baseline.
Verified: smoke (67), bench --suite cost CLI works.
202 lines
15 KiB
Markdown
202 lines
15 KiB
Markdown
# CORE — Public Claims
|
||
|
||
**Status:** DRAFT v0 — evidence rows below marked `TBD` are not yet
|
||
regenerated by the harness. Do not cite externally until every row is
|
||
either a number with a reproducible command, or removed.
|
||
|
||
**Regenerate:** `core bench claims --json > evals/reports/claims_latest.json`
|
||
|
||
**Verification contract:** every claim row carries (a) the command that
|
||
produced it, (b) the lane/version, and (c) a trace hash or report SHA
|
||
that a third party can re-derive locally. A claim without all three is
|
||
not a claim — it is marketing.
|
||
|
||
---
|
||
|
||
> **Smaller. Faster. Stronger.**
|
||
>
|
||
> CORE is a new AI architecture designed to make intelligence more
|
||
> efficient, understandable, and correctable. It aims to provide modern
|
||
> AI capabilities with a much lighter footprint, lower operating cost,
|
||
> stronger memory, clearer reasoning, and a training process built
|
||
> around structured learning instead of brute-force scale.
|
||
|
||
## The thesis, mapped to evidence
|
||
|
||
Every adjective in the positioning statement below corresponds to a
|
||
row in the tier tables. If an adjective is not yet backed by a green
|
||
row, it is an *intention*, not a claim — and is marked as such here.
|
||
|
||
| Virtue | What it means in practice | Where it's proved |
|
||
|--------|---------------------------|-------------------|
|
||
| **Smaller / lighter** | Curated packs + exact CGA recall, not corpus dumps + ANN indexes | Tier 4 · `bench footprint` |
|
||
| **Cheaper to train** | Reviewed corrections move the needle; no retraining cycle | Tier 3 · learning-curve lanes |
|
||
| **Cheaper to run** | Practical on a laptop — energy + latency budget below frontier API floor | Tier 4 · `bench latency` + `bench cost` |
|
||
| **More transparent** | Every answer carries a trace hash, term provenance, and intent path | Tier 2 · `provenance` lane |
|
||
| **More consistent** | Same `(pack, vault, seed)` → byte-identical surface | Tier 1 · rows 1–2 |
|
||
| **Easier to correct** | A reviewed correction becomes part of future behavior, deterministically | Tier 1 · row 6 + Tier 3 |
|
||
| **Better long-term memory** | Structured vault, not a sliding context window | Tier 1 · row 4 + Tier 4 · `bench long-context-cost` |
|
||
| **Less prone to fabrication** | Unsupported answers marked uncertain or refused; surface grounded in vocab | Tier 2 · `surface_groundedness`, `inference_closure` |
|
||
| **More private / local-first** | Runs without network egress; no model-provider dependency | Tier 4 · `bench footprint` (deployment profile) |
|
||
| **More adaptable** | Specialty courses graduate via the Formation pipeline | Tier 3 · `zero_code_domain_acquisition` |
|
||
| **More stable over time** | New teaching does not silently regress prior lanes | Tier 3 · *monotonicity invariant* |
|
||
| **More auditable** | Mistakes, corrections, and reasoning paths are inspectable artifacts | Tier 2 · `provenance` + Tier 5 holdout gap |
|
||
|
||
## Positioning statement
|
||
|
||
> **CORE is a new AI architecture designed to make intelligence more
|
||
> efficient, understandable, and correctable. It aims to provide modern
|
||
> AI capabilities with a much lighter footprint, lower operating cost,
|
||
> stronger memory, clearer reasoning, and a training process built
|
||
> around structured learning instead of brute-force scale.**
|
||
|
||
A shorter form, for places that need one sentence:
|
||
|
||
> **CORE is designed to be a lighter, more efficient, more transparent,
|
||
> and more teachable alternative to today's large AI models.**
|
||
|
||
---
|
||
|
||
## Tier 1 — Structural asymmetries (CORE vs frontier LLMs)
|
||
|
||
These are the load-bearing claims. They are properties of the
|
||
architecture, not of training scale. They should be **bit-stable across
|
||
releases**, not just "high."
|
||
|
||
| # | Claim | Metric | CORE | Frontier LLM | Lane / bench | Trace hash |
|
||
|---|-------|--------|------|--------------|-------------|------------|
|
||
| 1 | Same prompt → byte-identical surface across N runs | unique_surfaces / N | 1/N | TBD | `bench determinism --runs 50` | TBD |
|
||
| 2 | Same (pack, vault, seed) → byte-identical trace hash | unique_hashes / N | 1/N | n/a (no trace) | `bench replay-vs-llm` | TBD |
|
||
| 3 | Versor closure on every intermediate field state | `max_versor_condition` | < 1e-6 | n/a | `bench versor` | TBD |
|
||
| 4 | Exact CGA recall (no ANN, no cosine) | recall@1 on vault probe | 1.00 | n/a | `bench versor` + vault probe | TBD |
|
||
| 5 | Reviewed-mutation-only learning (prompt-injection-resistant) | identity-override rejection rate | 1.00 | TBD (expected ≪ 1.00) | `evals/adversarial_identity` | TBD |
|
||
| 6 | Deterministic teaching replay — N corrections → identical post-state | post-state hash equality | true | n/a | `bench learning-curve --verify-replay` | TBD |
|
||
|
||
**Why these come first:** every Tier-1 row is something a transformer
|
||
*structurally cannot match* without abandoning sampling, weights, or
|
||
non-deterministic kernels. If any Tier-1 row regresses, ship is blocked.
|
||
|
||
---
|
||
|
||
## Tier 2 — Capability lanes (expected strong now)
|
||
|
||
Lanes where CORE's deterministic pipeline already lands at or above the
|
||
contract threshold on v1 public split. Numbers below come from
|
||
`evals/<lane>/results/`.
|
||
|
||
| Lane | Metric | CORE v1 | Contract threshold | Last run |
|
||
|------|--------|---------|-------------------|----------|
|
||
| cognition | intent_accuracy | TBD | ≥ 0.90 | TBD |
|
||
| cognition | versor_closure_rate | TBD | 1.00 | TBD |
|
||
| compositionality | lane score | TBD | TBD | TBD |
|
||
| grammatical_coverage | coverage_rate | TBD | TBD | TBD |
|
||
| discourse_paragraph | lane score | TBD | TBD | TBD |
|
||
| inference_closure | closure_rate | TBD | TBD | TBD |
|
||
| provenance | trace_completeness | TBD | 1.00 | TBD |
|
||
| adversarial_identity | rejection_rate | TBD | 1.00 | TBD |
|
||
| **teaching_injection_resistance** | speculative_admission_rate | **1.00** | 1.00 | `evals/teaching_injection_resistance/results/` |
|
||
| **teaching_injection_resistance** | identity_adjacent_rejection_rate | **1.00** | 1.00 | same |
|
||
| **teaching_injection_resistance** | auto_promotion_count | **0** | 0 | same |
|
||
|
||
---
|
||
|
||
## Tier 3 — Learning-curve lanes (expected to improve with teaching)
|
||
|
||
These are the lanes whose scores should rise monotonically as reviewed
|
||
teaching examples are added. The headline demo is *"N corrections → +X%,
|
||
locked deterministically, replayable forever."*
|
||
|
||
Each row should be populated by `bench learning-curve <lane>
|
||
--cycles 0,5,10,25,50,100`. The bench writes one curve per lane to
|
||
`evals/reports/learning_curves/<lane>.json` with a per-step trace hash
|
||
proving every intermediate state is replayable.
|
||
|
||
| Lane | Score @ 0 | @ 5 | @ 25 | @ 100 | Δ (100−0) | Curve file |
|
||
|------|-----------|-----|------|-------|-----------|------------|
|
||
| cognition | TBD | TBD | TBD | TBD | TBD | `learning_curves/cognition.json` |
|
||
| compositionality | TBD | TBD | TBD | TBD | TBD | `learning_curves/compositionality.json` |
|
||
| monotonic_learning | TBD | TBD | TBD | TBD | TBD | `learning_curves/monotonic_learning.json` |
|
||
| sample_efficiency | TBD | TBD | TBD | TBD | TBD | `learning_curves/sample_efficiency.json` |
|
||
| inference_closure | TBD | TBD | TBD | TBD | TBD | `learning_curves/inference_closure.json` |
|
||
| multi_step_reasoning | TBD | TBD | TBD | TBD | TBD | `learning_curves/multi_step_reasoning.json` |
|
||
| symbolic_logic | TBD | TBD | TBD | TBD | TBD | `learning_curves/symbolic_logic.json` |
|
||
| cross_domain_transfer | TBD | TBD | TBD | TBD | TBD | `learning_curves/cross_domain_transfer.json` |
|
||
| zero_code_domain_acquisition | TBD | TBD | TBD | TBD | TBD | `learning_curves/zero_code_domain_acquisition.json` |
|
||
|
||
**Monotonicity invariant:** for each row, `score @ N` must not regress
|
||
below `score @ M` for `M < N` once teaching has plateaued. A regression
|
||
is evidence of a non-coherent correction having been admitted and is a
|
||
hard finding, not a metric noise event.
|
||
|
||
---
|
||
|
||
## Tier 4 — Cost and performance
|
||
|
||
| Claim | Metric | CORE | Frontier LLM | Bench |
|
||
|-------|--------|------|--------------|-------|
|
||
| Median time-to-first-surface | seconds | TBD | TBD | `bench latency` |
|
||
| p95 time-to-first-surface | seconds | TBD | TBD | `bench latency` |
|
||
| Rust vs Python backend speedup | x faster | TBD | n/a | `bench speedup` |
|
||
| Median turn latency | ms | **~445 ms** (100-turn measured, "What is truth?", warmup 5) | 200–2000 ms (provider-quoted, varies by model/region) | `core bench --suite cost --runs 100` — see `evals/reports/cost_latest.json` |
|
||
| p95 turn latency | ms | **~447 ms** (100-turn measured) | varies | `bench cost` |
|
||
| Cost per 1000 turns | USD | **$0.0044** at AWS t3.medium on-demand ($0.0416/hr, captured 2026-05-17) | $0.22 (Haiku) · $0.45 (GPT-4o) · $0.66 (Sonnet) at 20 in / 40 out tokens/turn | `bench cost` — **48–149× cheaper** depending on provider |
|
||
| Energy per turn | joules | not measured (RAPL/IOKit required; cpu_seconds_total reported instead) | not directly comparable | `bench cost` energy_disclosure |
|
||
| Total deployed footprint (on-disk) | bytes | **7.06 MiB** | 14.9 GiB (Llama 3.1 8B fp16) — **2,160× larger**; 754 GiB (Llama 3.1 405B) — **109,358× larger** | `python -c "from benchmarks.footprint import run_footprint; print(run_footprint().summary())"` — see `evals/reports/footprint_latest.json` |
|
||
| Resident memory at idle / after one pulse | MiB | 17.9 / 33.3 | n/a (frontier inference requires GPU memory measured in GiB–TiB) | `bench footprint` (RSS rows) |
|
||
| Runs offline on commodity hardware | yes / no | **yes** | no (requires datacenter GPU + network) | `bench footprint` (deployment profile) |
|
||
| Cost growth with conversation length | USD / turn at depth N | TBD (flat — vault, not window) | TBD (super-linear — context tokens) | `bench long-context-cost` (uses `evals/long_context_cost`) |
|
||
| Hallucination / fabrication rate | unsupported-answer fraction | TBD | TBD | Tier 2 · `surface_groundedness` + `inference_closure` |
|
||
|
||
---
|
||
|
||
## Tier 4.5 — Known gaps, openly measured
|
||
|
||
Per the transparency principle of this project: weaknesses we have a
|
||
test for live in the public claims document, not in a private TODO.
|
||
Each row below is a real failure today against a real metric. The
|
||
intended fix is named so the row turns into a green tier-2 claim when
|
||
the work lands.
|
||
|
||
| Lane | Metric | Current | Target | Known fix |
|
||
|------|--------|---------|--------|-----------|
|
||
| ~~`refusal_calibration`~~ | ~~refusal_rate (on out-of-grounding prompts)~~ | ✅ **1.00** | ≥ 0.95 | **CLOSED 2026-05-17** — `_UNKNOWN_DOMAIN_SURFACE` now reads "I don't know — insufficient grounding for that yet.", matching the lane's refusal markers honestly (the prior wording was equivalent in spirit but unrecognizable). The gate was already firing; only the surface text needed alignment. |
|
||
| ~~`refusal_calibration`~~ | ~~fabrication_rate~~ | ✅ **0.00** | 0.00 | Holds at target. |
|
||
| ~~`refusal_calibration`~~ | ~~in_grounding_answer_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — runner now supports per-case `prime` field so in-grounding probes get a brief priming exchange before the cold-start vault is interrogated. Previous 1.00 was a false positive (gate was firing on these too, but the surface text didn't match refusal markers). New 1.00 is genuine: vault is seeded, then the probe answers. |
|
||
| ~~`contradiction_detection`~~ | ~~contradiction_flag_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — `TeachingStore.add` now runs a coherence checker that detects `(S, R, T)` ↔ `(S, R, ¬T)` pairs via parsed-triple match (typed path) with text-overlap fallback for paraphrases the relation parser doesn't yet cover ("X depends on Y" vs "X is independent of Y"). On match, BOTH proposals transition to `EpistemicStatus.CONTESTED`. Runner reads the new signal directly; versor-spike heuristic retired. |
|
||
| ~~`contradiction_detection`~~ | ~~false_flag_rate~~ | ✅ **0.00** | 0.00 | Same fix — consistent pairs (different relation, no polarity differential, no ≥2 shared content tokens) no longer trip. |
|
||
| ~~One-mutation-path audit · Leak A~~ | ~~pack vocab default epistemic_status~~ | ✅ **`speculative`** | `speculative` | **CLOSED 2026-05-17** — `language_packs/compiler.py:331` and `language_packs/schema.py::LexicalEntry` now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by `tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative` (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. |
|
||
| ~~One-mutation-path audit · Leak B~~ | ~~vault.recall epistemic awareness~~ | ✅ **`min_status` filter** | `min_status` filter | **CLOSED 2026-05-17** — `VaultStore.store()` now stamps every entry with `epistemic_status` (default SPECULATIVE per ADR-0021 §3); `VaultStore.recall(min_status=EpistemicStatus.COHERENT)` filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by `tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter` (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. |
|
||
| ~~One-mutation-path audit · Leak C~~ | self-reinforcing fabrication via propose() | ✅ **stamped + read-side audited** | stamped + categorized read sites | **CLOSED 2026-05-17** — write-side stamps SPECULATIVE (`generate/proposition.py:198`). Read-side audit categorized every production `vault.recall()` callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (`TestINV24VaultRecallRegistry`, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass `min_status=COHERENT`. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded `realize(proposition, vocab)` (now SPECULATIVE-default per Leak A), not from `vault.recall`. Site-level `# INV-24 recall role:` comments at every callsite. See `docs/truth_seeking_schema.md` §Leak C. |
|
||
| ~~`articulation_of_status`~~ | ~~speculative_articulation_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — `CognitiveTurnPipeline` tracks subjects of prior SPECULATIVE proposals and prepends `(speculative, not yet reviewed)` to the surface when a subsequent turn references one of those subjects (subject substring match, ≥4-char tokenized split, or reflexive query shape like "is your answer confirmed?"). The teach turn itself is not self-marked; only subsequent probes are. |
|
||
| ~~`articulation_of_status`~~ | ~~false_certainty_rate~~ | ✅ **0.00** | 0.00 | Same fix — bare assertions on SPECULATIVE-backed claims are now hedged with the explicit marker. |
|
||
|
||
A green row in Tier 4.5 graduates to Tier 1/2/3 in the same commit
|
||
that lands the fix and the result.
|
||
|
||
---
|
||
|
||
## Tier 5 — Holdout (sealed)
|
||
|
||
Held back from the team. Only the harness sees these. Scores are
|
||
written by the holdout runner and surfaced here without inspection.
|
||
|
||
| Lane | Holdout score | Public score | Generalization gap |
|
||
|------|---------------|--------------|---------------------|
|
||
| cognition | TBD | TBD | TBD |
|
||
| compositionality | TBD | TBD | TBD |
|
||
| inference_closure | TBD | TBD | TBD |
|
||
|
||
A gap > 0.10 on any lane is a fail signal — investigate before claiming
|
||
the lane in public.
|
||
|
||
---
|
||
|
||
## How to extend this document
|
||
|
||
1. Add a row only after the bench command that produces it has been run
|
||
and committed evidence exists under `evals/reports/` or
|
||
`evals/<lane>/results/`.
|
||
2. Never paste a number without the command and trace hash that made it.
|
||
3. If a number regresses, do not edit the old number in place — append
|
||
a dated row and explain the regression in the same commit.
|
||
4. Tier 1 rows are immutable in shape. New rows go to Tier 2 or 3.
|