core/evals/CLAIMS.md

CORE — Public Claims

Status: DRAFT v0 — evidence rows below marked TBD are not yet regenerated by the harness. Do not cite externally until every row is either a number with a reproducible command, or removed.

Regenerate: core bench claims --json > evals/reports/claims_latest.json

Verification contract: every claim row carries (a) the command that produced it, (b) the lane/version, and (c) a trace hash or report SHA that a third party can re-derive locally. A claim without all three is not a claim — it is marketing.

---

Smaller. Faster. Stronger.

CORE is a new AI architecture designed to make intelligence more efficient, understandable, and correctable. It aims to provide modern AI capabilities with a much lighter footprint, lower operating cost, stronger memory, clearer reasoning, and a training process built around structured learning instead of brute-force scale.

The thesis, mapped to evidence

Every adjective in the positioning statement below corresponds to a row in the tier tables. If an adjective is not yet backed by a green row, it is an intention, not a claim — and is marked as such here.

Virtue	What it means in practice	Where it's proved
Smaller / lighter	Curated packs + exact CGA recall, not corpus dumps + ANN indexes	Tier 4 · bench footprint
Cheaper to train	Reviewed corrections move the needle; no retraining cycle	Tier 3 · learning-curve lanes
Cheaper to run	Practical on a laptop — energy + latency budget below frontier API floor	Tier 4 · bench latency + bench cost
More transparent	Every answer carries a trace hash, term provenance, and intent path	Tier 2 · provenance lane
More consistent	Same (pack, vault, seed) → byte-identical surface	Tier 1 · rows 1–2
Easier to correct	A reviewed correction becomes part of future behavior, deterministically	Tier 1 · row 6 + Tier 3
Better long-term memory	Structured vault, not a sliding context window	Tier 1 · row 4 + Tier 4 · bench long-context-cost
Less prone to fabrication	Unsupported answers marked uncertain or refused; surface grounded in vocab	Tier 2 · surface_groundedness, inference_closure
More private / local-first	Runs without network egress; no model-provider dependency	Tier 4 · bench footprint (deployment profile)
More adaptable	Specialty courses graduate via the Formation pipeline	Tier 3 · zero_code_domain_acquisition
More stable over time	New teaching does not silently regress prior lanes	Tier 3 · monotonicity invariant
More auditable	Mistakes, corrections, and reasoning paths are inspectable artifacts	Tier 2 · provenance + Tier 5 holdout gap

Positioning statement

CORE is a new AI architecture designed to make intelligence more efficient, understandable, and correctable. It aims to provide modern AI capabilities with a much lighter footprint, lower operating cost, stronger memory, clearer reasoning, and a training process built around structured learning instead of brute-force scale.

A shorter form, for places that need one sentence:

CORE is designed to be a lighter, more efficient, more transparent, and more teachable alternative to today's large AI models.

---

Tier 1 — Structural asymmetries (CORE vs frontier LLMs)

These are the load-bearing claims. They are properties of the architecture, not of training scale. They should be bit-stable across releases, not just "high."

#	Claim	Metric	CORE	Frontier LLM	Lane / bench	Trace hash
1	Same prompt → byte-identical surface across N runs	unique_surfaces / N	1/N	TBD	bench determinism --runs 50	TBD
2	Same (pack, vault, seed) → byte-identical trace hash	unique_hashes / N	1/N	n/a (no trace)	bench replay-vs-llm	TBD
3	Versor closure on every intermediate field state	max_versor_condition	< 1e-6	n/a	bench versor	TBD
4	Exact CGA recall (no ANN, no cosine)	recall@1 on vault probe	1.00	n/a	bench versor + vault probe	TBD
5	Reviewed-mutation-only learning (prompt-injection-resistant)	identity-override rejection rate	1.00	TBD (expected ≪ 1.00)	evals/adversarial_identity	TBD
6	Deterministic teaching replay — N corrections → identical post-state	post-state hash equality	true	n/a	bench learning-curve --verify-replay	TBD

Why these come first: every Tier-1 row is something a transformer structurally cannot match without abandoning sampling, weights, or non-deterministic kernels. If any Tier-1 row regresses, ship is blocked.

---

Tier 2 — Capability lanes (expected strong now)

Lanes where CORE's deterministic pipeline already lands at or above the contract threshold on v1 public split. Numbers below come from evals/<lane>/results/.

Lane	Metric	CORE v1	Contract threshold	Last run
cognition	intent_accuracy	TBD	≥ 0.90	TBD
cognition	versor_closure_rate	TBD	1.00	TBD
compositionality	lane score	TBD	TBD	TBD
grammatical_coverage	coverage_rate	TBD	TBD	TBD
discourse_paragraph	lane score	TBD	TBD	TBD
inference_closure	closure_rate	TBD	TBD	TBD
provenance	trace_completeness	TBD	1.00	TBD
adversarial_identity	rejection_rate	TBD	1.00	TBD
teaching_injection_resistance	speculative_admission_rate	1.00	1.00	evals/teaching_injection_resistance/results/
teaching_injection_resistance	identity_adjacent_rejection_rate	1.00	1.00	same
teaching_injection_resistance	auto_promotion_count	0	0	same

---

Tier 3 — Learning-curve lanes (expected to improve with teaching)

These are the lanes whose scores should rise monotonically as reviewed teaching examples are added. The headline demo is "N corrections → +X%, locked deterministically, replayable forever."

Each row should be populated by bench learning-curve <lane> --cycles 0,5,10,25,50,100. The bench writes one curve per lane to evals/reports/learning_curves/<lane>.json with a per-step trace hash proving every intermediate state is replayable.

Lane	Score @ 0	@ 5	@ 25	@ 100	Δ (100−0)	Curve file
cognition	TBD	TBD	TBD	TBD	TBD	learning_curves/cognition.json
compositionality	TBD	TBD	TBD	TBD	TBD	learning_curves/compositionality.json
monotonic_learning	TBD	TBD	TBD	TBD	TBD	learning_curves/monotonic_learning.json
sample_efficiency	TBD	TBD	TBD	TBD	TBD	learning_curves/sample_efficiency.json
inference_closure	TBD	TBD	TBD	TBD	TBD	learning_curves/inference_closure.json
multi_step_reasoning	TBD	TBD	TBD	TBD	TBD	learning_curves/multi_step_reasoning.json
symbolic_logic	TBD	TBD	TBD	TBD	TBD	learning_curves/symbolic_logic.json
cross_domain_transfer	TBD	TBD	TBD	TBD	TBD	learning_curves/cross_domain_transfer.json
zero_code_domain_acquisition	TBD	TBD	TBD	TBD	TBD	learning_curves/zero_code_domain_acquisition.json

Monotonicity invariant: for each row, score @ N must not regress below score @ M for M < N once teaching has plateaued. A regression is evidence of a non-coherent correction having been admitted and is a hard finding, not a metric noise event.

---

Tier 4 — Cost and performance

Claim	Metric	CORE	Frontier LLM	Bench
Median time-to-first-surface	seconds	TBD	TBD	bench latency
p95 time-to-first-surface	seconds	TBD	TBD	bench latency
Rust vs Python backend speedup	x faster	TBD	n/a	bench speedup
Energy per turn (estimated)	joules	TBD	TBD	bench cost (not yet implemented)
Cost per 1000 turns	USD	TBD (≈ electricity only)	TBD (API list)	bench cost (not yet implemented)
Total deployed footprint (on-disk)	bytes	7.06 MiB	14.9 GiB (Llama 3.1 8B fp16) — 2,160× larger; 754 GiB (Llama 3.1 405B) — 109,358× larger	python -c "from benchmarks.footprint import run_footprint; print(run_footprint().summary())" — see evals/reports/footprint_latest.json
Resident memory at idle / after one pulse	MiB	17.9 / 33.3	n/a (frontier inference requires GPU memory measured in GiB–TiB)	bench footprint (RSS rows)
Runs offline on commodity hardware	yes / no	yes	no (requires datacenter GPU + network)	bench footprint (deployment profile)
Cost growth with conversation length	USD / turn at depth N	TBD (flat — vault, not window)	TBD (super-linear — context tokens)	bench long-context-cost (uses evals/long_context_cost)
Hallucination / fabrication rate	unsupported-answer fraction	TBD	TBD	Tier 2 · surface_groundedness + inference_closure

---

Tier 4.5 — Known gaps, openly measured

Per the transparency principle of this project: weaknesses we have a test for live in the public claims document, not in a private TODO. Each row below is a real failure today against a real metric. The intended fix is named so the row turns into a green tier-2 claim when the work lands.

Lane	Metric	Current	Target	Known fix
refusal_calibration	refusal_rate (on out-of-grounding prompts)	0.00	≥ 0.95	Realizer emits an explicit unknown surface when no vault hit lands and no grounded vocabulary covers the subject.
refusal_calibration	fabrication_rate	0.00	0.00	Already at target — system is not asserting ungrounded definitions, but is not yet refusing either.
refusal_calibration	in_grounding_answer_rate	1.00	≥ 0.90	Already at target.
contradiction_detection	contradiction_flag_rate	0.50	≥ 0.90	Coherence checker at TeachingStore.add that detects (S, R, O) ↔ (S, ¬R, O) pairs and transitions both to EpistemicStatus.CONTESTED. Versor-delta alone is not a clean signal.
contradiction_detection	false_flag_rate	1.00	0.00	Same fix — replace the versor-delta heuristic with the coherence checker.
One-mutation-path audit · Leak A	pack vocab default epistemic_status	✅ speculative	speculative	CLOSED 2026-05-17 — language_packs/compiler.py:331 and language_packs/schema.py::LexicalEntry now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp.
One-mutation-path audit · Leak B	vault.recall epistemic awareness	✅ min_status filter	min_status filter	CLOSED 2026-05-17 — VaultStore.store() now stamps every entry with epistemic_status (default SPECULATIVE per ADR-0021 §3); VaultStore.recall(min_status=EpistemicStatus.COHERENT) filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default.
One-mutation-path audit · Leak C	self-reinforcing fabrication via propose()	✅ stamped + read-side audited	stamped + categorized read sites	CLOSED 2026-05-17 — write-side stamps SPECULATIVE (generate/proposition.py:198). Read-side audit categorized every production vault.recall() callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (TestINV24VaultRecallRegistry, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass min_status=COHERENT. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded realize(proposition, vocab) (now SPECULATIVE-default per Leak A), not from vault.recall. Site-level # INV-24 recall role: comments at every callsite. See docs/truth_seeking_schema.md §Leak C.
articulation_of_status	speculative_articulation_rate	0.00	≥ 0.90	Realizer consults pack_mutation_proposal.epistemic_status (or the read-path filter from Leak B) and emits a status marker when the answer draws on non-COHERENT material.
articulation_of_status	false_certainty_rate	0.60	0.00	Same fix — bare assertions on SPECULATIVE-backed claims become hedged surfaces.

A green row in Tier 4.5 graduates to Tier 1/2/3 in the same commit that lands the fix and the result.

---

Tier 5 — Holdout (sealed)

Held back from the team. Only the harness sees these. Scores are written by the holdout runner and surfaced here without inspection.

Lane	Holdout score	Public score	Generalization gap
cognition	TBD	TBD	TBD
compositionality	TBD	TBD	TBD
inference_closure	TBD	TBD	TBD

A gap > 0.10 on any lane is a fail signal — investigate before claiming the lane in public.

---

How to extend this document

1. Add a row only after the bench command that produces it has been run and committed evidence exists under evals/reports/ or evals/<lane>/results/.
2. Never paste a number without the command and trace hash that made it.
3. If a number regresses, do not edit the old number in place — append a dated row and explain the regression in the same commit.
4. Tier 1 rows are immutable in shape. New rows go to Tier 2 or 3.