core/evals/CLAIMS.md

# CORE — Public Claims

**Status:** DRAFT v0 — evidence rows below marked `TBD` are not yet
regenerated by the harness. Do not cite externally until every row is
either a number with a reproducible command, or removed.

**Regenerate:** `core bench claims --json > evals/reports/claims_latest.json`

**Verification contract:** every claim row carries (a) the command that
produced it, (b) the lane/version, and (c) a trace hash or report SHA
that a third party can re-derive locally. A claim without all three is
not a claim — it is marketing.

---

> **Smaller. Faster. Stronger.**
>
> CORE is a new AI architecture designed to make intelligence more
> efficient, understandable, and correctable. It aims to provide modern
> AI capabilities with a much lighter footprint, lower operating cost,
> stronger memory, clearer reasoning, and a training process built
> around structured learning instead of brute-force scale.

## The thesis, mapped to evidence

Every adjective in the positioning statement below corresponds to a
row in the tier tables. If an adjective is not yet backed by a green
row, it is an *intention*, not a claim — and is marked as such here.

| Virtue | What it means in practice | Where it's proved |
|--------|---------------------------|-------------------|
| **Smaller / lighter** | Curated packs + exact CGA recall, not corpus dumps + ANN indexes | Tier 4 · `bench footprint` |
| **Cheaper to train** | Reviewed corrections move the needle; no retraining cycle | Tier 3 · learning-curve lanes |
| **Cheaper to run** | Practical on a laptop — energy + latency budget below frontier API floor | Tier 4 · `bench latency` + `bench cost` |
| **More transparent** | Every answer carries a trace hash, term provenance, and intent path | Tier 2 · `provenance` lane |
| **More consistent** | Same `(pack, vault, seed)` → byte-identical surface | Tier 1 · rows 1–2 |
| **Easier to correct** | A reviewed correction becomes part of future behavior, deterministically | Tier 1 · row 6 + Tier 3 |
| **Better long-term memory** | Structured vault, not a sliding context window | Tier 1 · row 4 + Tier 4 · `bench long-context-cost` |
| **Less prone to fabrication** | Unsupported answers marked uncertain or refused; surface grounded in vocab | Tier 2 · `surface_groundedness`, `inference_closure` |
| **More private / local-first** | Runs without network egress; no model-provider dependency | Tier 4 · `bench footprint` (deployment profile) |
| **More adaptable** | Specialty courses graduate via the Formation pipeline | Tier 3 · `zero_code_domain_acquisition` |
| **More stable over time** | New teaching does not silently regress prior lanes | Tier 3 · *monotonicity invariant* |
| **More auditable** | Mistakes, corrections, and reasoning paths are inspectable artifacts | Tier 2 · `provenance` + Tier 5 holdout gap |

## Positioning statement

> **CORE is a new AI architecture designed to make intelligence more
> efficient, understandable, and correctable. It aims to provide modern
> AI capabilities with a much lighter footprint, lower operating cost,
> stronger memory, clearer reasoning, and a training process built
> around structured learning instead of brute-force scale.**

A shorter form, for places that need one sentence:

> **CORE is designed to be a lighter, more efficient, more transparent,
> and more teachable alternative to today's large AI models.**

---

## Tier 1 — Structural asymmetries (CORE vs frontier LLMs)

These are the load-bearing claims. They are properties of the
architecture, not of training scale. They should be **bit-stable across
releases**, not just "high."

| # | Claim | Metric | CORE | Frontier LLM | Lane / bench | Trace hash |
|---|-------|--------|------|--------------|-------------|------------|
| 1 | Same prompt → byte-identical surface across N runs | unique_surfaces / N | 1/N | TBD | `bench determinism --runs 50` | TBD |
| 2 | Same (pack, vault, seed) → byte-identical trace hash | unique_hashes / N | 1/N | n/a (no trace) | `bench replay-vs-llm` | TBD |
| 3 | Versor closure on every intermediate field state | `max_versor_condition` | < 1e-6 | n/a | `bench versor` | TBD |
| 4 | Exact CGA recall (no ANN, no cosine) | recall@1 on vault probe | 1.00 | n/a | `bench versor` + vault probe | TBD |
| 5 | Reviewed-mutation-only learning (prompt-injection-resistant) | identity-override rejection rate | 1.00 | TBD (expected ≪ 1.00) | `evals/adversarial_identity` | TBD |
| 6 | Deterministic teaching replay — N corrections → identical post-state | post-state hash equality | true | n/a | `bench learning-curve --verify-replay` | TBD |

**Why these come first:** every Tier-1 row is something a transformer
*structurally cannot match* without abandoning sampling, weights, or
non-deterministic kernels. If any Tier-1 row regresses, ship is blocked.

---

## Tier 2 — Capability lanes (expected strong now)

Lanes where CORE's deterministic pipeline already lands at or above the
contract threshold on v1 public split. Numbers below come from
`evals/<lane>/results/`.

| Lane | Metric | CORE v1 | Contract threshold | Last run |
|------|--------|---------|-------------------|----------|
| cognition | intent_accuracy | TBD | ≥ 0.90 | TBD |
| cognition | versor_closure_rate | TBD | 1.00 | TBD |
| compositionality | lane score | TBD | TBD | TBD |
| grammatical_coverage | coverage_rate | TBD | TBD | TBD |
| discourse_paragraph | lane score | TBD | TBD | TBD |
| inference_closure | closure_rate | TBD | TBD | TBD |
| provenance | trace_completeness | TBD | 1.00 | TBD |
| adversarial_identity | rejection_rate | TBD | 1.00 | TBD |
| **teaching_injection_resistance** | speculative_admission_rate | **1.00** | 1.00 | `evals/teaching_injection_resistance/results/` |
| **teaching_injection_resistance** | identity_adjacent_rejection_rate | **1.00** | 1.00 | same |
| **teaching_injection_resistance** | auto_promotion_count | **0** | 0 | same |

---

## Tier 3 — Learning-curve lanes (expected to improve with teaching)

These are the lanes whose scores should rise monotonically as reviewed
teaching examples are added. The headline demo is *"N corrections → +X%,
locked deterministically, replayable forever."*

Each row should be populated by `bench learning-curve <lane>
--cycles 0,5,10,25,50,100`. The bench writes one curve per lane to
`evals/reports/learning_curves/<lane>.json` with a per-step trace hash
proving every intermediate state is replayable.

| Lane | Score @ 0 | @ 5 | @ 25 | @ 100 | Δ (100−0) | Curve file |
|------|-----------|-----|------|-------|-----------|------------|
| cognition | TBD | TBD | TBD | TBD | TBD | `learning_curves/cognition.json` |
| compositionality | TBD | TBD | TBD | TBD | TBD | `learning_curves/compositionality.json` |
| monotonic_learning | TBD | TBD | TBD | TBD | TBD | `learning_curves/monotonic_learning.json` |
| sample_efficiency | TBD | TBD | TBD | TBD | TBD | `learning_curves/sample_efficiency.json` |
| inference_closure | TBD | TBD | TBD | TBD | TBD | `learning_curves/inference_closure.json` |
| multi_step_reasoning | TBD | TBD | TBD | TBD | TBD | `learning_curves/multi_step_reasoning.json` |
| symbolic_logic | TBD | TBD | TBD | TBD | TBD | `learning_curves/symbolic_logic.json` |
| cross_domain_transfer | TBD | TBD | TBD | TBD | TBD | `learning_curves/cross_domain_transfer.json` |
| zero_code_domain_acquisition | TBD | TBD | TBD | TBD | TBD | `learning_curves/zero_code_domain_acquisition.json` |

**Monotonicity invariant:** for each row, `score @ N` must not regress
below `score @ M` for `M < N` once teaching has plateaued. A regression
is evidence of a non-coherent correction having been admitted and is a
hard finding, not a metric noise event.

---

## Tier 4 — Cost and performance

| Claim | Metric | CORE | Frontier LLM | Bench |
|-------|--------|------|--------------|-------|
| Median time-to-first-surface | seconds | TBD | TBD | `bench latency` |
| p95 time-to-first-surface | seconds | TBD | TBD | `bench latency` |
| Rust vs Python backend speedup | x faster | TBD | n/a | `bench speedup` |
| Energy per turn (estimated) | joules | TBD | TBD | `bench cost` (not yet implemented) |
| Cost per 1000 turns | USD | TBD (≈ electricity only) | TBD (API list) | `bench cost` (not yet implemented) |
| Total deployed footprint (on-disk) | bytes | **7.06 MiB** | 14.9 GiB (Llama 3.1 8B fp16) — **2,160× larger**; 754 GiB (Llama 3.1 405B) — **109,358× larger** | `python -c "from benchmarks.footprint import run_footprint; print(run_footprint().summary())"` — see `evals/reports/footprint_latest.json` |
| Resident memory at idle / after one pulse | MiB | 17.9 / 33.3 | n/a (frontier inference requires GPU memory measured in GiB–TiB) | `bench footprint` (RSS rows) |
| Runs offline on commodity hardware | yes / no | **yes** | no (requires datacenter GPU + network) | `bench footprint` (deployment profile) |
| Cost growth with conversation length | USD / turn at depth N | TBD (flat — vault, not window) | TBD (super-linear — context tokens) | `bench long-context-cost` (uses `evals/long_context_cost`) |
| Hallucination / fabrication rate | unsupported-answer fraction | TBD | TBD | Tier 2 · `surface_groundedness` + `inference_closure` |

---

## Tier 4.5 — Known gaps, openly measured

Per the transparency principle of this project: weaknesses we have a
test for live in the public claims document, not in a private TODO.
Each row below is a real failure today against a real metric. The
intended fix is named so the row turns into a green tier-2 claim when
the work lands.

| Lane | Metric | Current | Target | Known fix |
|------|--------|---------|--------|-----------|
| `refusal_calibration` | refusal_rate (on out-of-grounding prompts) | 0.00 | ≥ 0.95 | Realizer emits an explicit unknown surface when no vault hit lands and no grounded vocabulary covers the subject. |
| `refusal_calibration` | fabrication_rate | 0.00 | 0.00 | Already at target — system is not asserting ungrounded definitions, but is not yet refusing either. |
| `refusal_calibration` | in_grounding_answer_rate | 1.00 | ≥ 0.90 | Already at target. |
| `contradiction_detection` | contradiction_flag_rate | 0.50 | ≥ 0.90 | Coherence checker at `TeachingStore.add` that detects `(S, R, O)` ↔ `(S, ¬R, O)` pairs and transitions both to `EpistemicStatus.CONTESTED`. Versor-delta alone is not a clean signal. |
| `contradiction_detection` | false_flag_rate | 1.00 | 0.00 | Same fix — replace the versor-delta heuristic with the coherence checker. |
| ~~One-mutation-path audit · Leak A~~ | ~~pack vocab default epistemic_status~~ | ✅ **`speculative`** | `speculative` | **CLOSED 2026-05-17** — `language_packs/compiler.py:331` and `language_packs/schema.py::LexicalEntry` now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by `tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative` (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. |
| ~~One-mutation-path audit · Leak B~~ | ~~vault.recall epistemic awareness~~ | ✅ **`min_status` filter** | `min_status` filter | **CLOSED 2026-05-17** — `VaultStore.store()` now stamps every entry with `epistemic_status` (default SPECULATIVE per ADR-0021 §3); `VaultStore.recall(min_status=EpistemicStatus.COHERENT)` filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by `tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter` (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. |
| ~~One-mutation-path audit · Leak C~~ | self-reinforcing fabrication via propose() | ✅ **stamped + read-side audited** | stamped + categorized read sites | **CLOSED 2026-05-17** — write-side stamps SPECULATIVE (`generate/proposition.py:198`). Read-side audit categorized every production `vault.recall()` callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (`TestINV24VaultRecallRegistry`, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass `min_status=COHERENT`. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded `realize(proposition, vocab)` (now SPECULATIVE-default per Leak A), not from `vault.recall`. Site-level `# INV-24 recall role:` comments at every callsite. See `docs/truth_seeking_schema.md` §Leak C. |
| `articulation_of_status` | speculative_articulation_rate | 0.00 | ≥ 0.90 | Realizer consults `pack_mutation_proposal.epistemic_status` (or the read-path filter from Leak B) and emits a status marker when the answer draws on non-COHERENT material. |
| `articulation_of_status` | false_certainty_rate | 0.60 | 0.00 | Same fix — bare assertions on SPECULATIVE-backed claims become hedged surfaces. |

A green row in Tier 4.5 graduates to Tier 1/2/3 in the same commit
that lands the fix and the result.

---

## Tier 5 — Holdout (sealed)

Held back from the team. Only the harness sees these. Scores are
written by the holdout runner and surfaced here without inspection.

| Lane | Holdout score | Public score | Generalization gap |
|------|---------------|--------------|---------------------|
| cognition | TBD | TBD | TBD |
| compositionality | TBD | TBD | TBD |
| inference_closure | TBD | TBD | TBD |

A gap > 0.10 on any lane is a fail signal — investigate before claiming
the lane in public.

---

## How to extend this document

1. Add a row only after the bench command that produces it has been run
   and committed evidence exists under `evals/reports/` or
   `evals/<lane>/results/`.
2. Never paste a number without the command and trace hash that made it.
3. If a number regresses, do not edit the old number in place — append
   a dated row and explain the regression in the same commit.
4. Tier 1 rows are immutable in shape. New rows go to Tier 2 or 3.