17 KiB
CORE — Public Claims
Status: DRAFT v0 — evidence rows below marked TBD are not yet
regenerated by the harness. Do not cite externally until every row is
either a number with a reproducible command, or removed.
Regenerate: core bench claims --json > evals/reports/claims_latest.json
Verification contract: every claim row carries (a) the command that produced it, (b) the lane/version, and (c) a trace hash or report SHA that a third party can re-derive locally. A claim without all three is not a claim — it is marketing.
Smaller. Faster. Stronger.
CORE is a new AI architecture designed to make intelligence more efficient, understandable, and correctable. It aims to provide modern AI capabilities with a much lighter footprint, lower operating cost, stronger memory, clearer reasoning, and a training process built around structured learning instead of brute-force scale.
The thesis, mapped to evidence
Every adjective in the positioning statement below corresponds to a row in the tier tables. If an adjective is not yet backed by a green row, it is an intention, not a claim — and is marked as such here.
| Virtue | What it means in practice | Where it's proved |
|---|---|---|
| Smaller / lighter | Curated packs + exact CGA recall, not corpus dumps + ANN indexes | Tier 4 · bench footprint |
| Cheaper to train | Reviewed corrections move the needle; no retraining cycle | Tier 3 · learning-curve lanes |
| Cheaper to run | Practical on a laptop — energy + latency budget below frontier API floor | Tier 4 · bench latency + bench cost |
| More transparent | Every answer carries a trace hash, term provenance, and intent path | Tier 2 · provenance lane |
| More consistent | Same (pack, vault, seed) → byte-identical surface |
Tier 1 · rows 1–2 |
| Easier to correct | A reviewed correction becomes part of future behavior, deterministically | Tier 1 · row 6 + Tier 3 |
| Better long-term memory | Structured vault, not a sliding context window | Tier 1 · row 4 + Tier 4 · bench long-context-cost |
| Less prone to fabrication | Unsupported answers marked uncertain or refused; surface grounded in vocab | Tier 2 · surface_groundedness, inference_closure |
| More private / local-first | Runs without network egress; no model-provider dependency | Tier 4 · bench footprint (deployment profile) |
| More adaptable | Specialty courses graduate via the Formation pipeline | Tier 3 · zero_code_domain_acquisition |
| More stable over time | New teaching does not silently regress prior lanes | Tier 3 · monotonicity invariant |
| More auditable | Mistakes, corrections, and reasoning paths are inspectable artifacts | Tier 2 · provenance + Tier 5 holdout gap |
Positioning statement
CORE is a new AI architecture designed to make intelligence more efficient, understandable, and correctable. It aims to provide modern AI capabilities with a much lighter footprint, lower operating cost, stronger memory, clearer reasoning, and a training process built around structured learning instead of brute-force scale.
A shorter form, for places that need one sentence:
CORE is designed to be a lighter, more efficient, more transparent, and more teachable alternative to today's large AI models.
Tier 1 — Structural asymmetries (CORE vs frontier LLMs)
These are the load-bearing claims. They are properties of the architecture, not of training scale. They should be bit-stable across releases, not just "high."
| # | Claim | Metric | CORE | Frontier LLM | Lane / bench | Trace hash |
|---|---|---|---|---|---|---|
| 1 | Same prompt → byte-identical surface across N runs | unique_surfaces / N | 1/N | TBD | bench determinism --runs 50 |
TBD |
| 2 | Same (pack, vault, seed) → byte-identical trace hash | unique_hashes / N | 1/N | n/a (no trace) | bench replay-vs-llm |
TBD |
| 3 | Versor closure on every intermediate field state | max_versor_condition |
< 1e-6 | n/a | bench versor |
TBD |
| 4 | Exact CGA recall (no ANN, no cosine) | recall@1 on vault probe | 1.00 | n/a | bench versor + vault probe |
TBD |
| 5 | Reviewed-mutation-only learning (prompt-injection-resistant) | identity-override rejection rate | 1.00 | TBD (expected ≪ 1.00) | evals/adversarial_identity |
TBD |
| 6 | Deterministic teaching replay — N corrections → identical post-state | post-state hash equality | true | n/a | bench learning-curve --verify-replay |
TBD |
Why these come first: every Tier-1 row is something a transformer structurally cannot match without abandoning sampling, weights, or non-deterministic kernels. If any Tier-1 row regresses, ship is blocked.
Tier 2 — Capability lanes (expected strong now)
Lanes where CORE's deterministic pipeline already lands at or above the
contract threshold on v1 public split. Numbers below come from
evals/<lane>/results/.
| Lane | Metric | CORE v1 | Contract threshold | Last run |
|---|---|---|---|---|
| cognition | intent_accuracy | TBD | ≥ 0.90 | TBD |
| cognition | versor_closure_rate | TBD | 1.00 | TBD |
| compositionality | lane score | TBD | TBD | TBD |
| grammatical_coverage | coverage_rate | TBD | TBD | TBD |
| discourse_paragraph | lane score | TBD | TBD | TBD |
| inference_closure | closure_rate | TBD | TBD | TBD |
| provenance | trace_completeness | TBD | 1.00 | TBD |
| adversarial_identity | rejection_rate | TBD | 1.00 | TBD |
| teaching_injection_resistance | speculative_admission_rate | 1.00 | 1.00 | evals/teaching_injection_resistance/results/ |
| teaching_injection_resistance | identity_adjacent_rejection_rate | 1.00 | 1.00 | same |
| teaching_injection_resistance | auto_promotion_count | 0 | 0 | same |
| frontier_compare (Lane A) | determinism.primary_score |
1.00 | 1.00 | python -m evals.frontier_compare --suite determinism |
| frontier_compare (Lane A) | max_versor_condition across runs |
< 1e-6 | < 1e-6 | same |
| frontier_compare (Lane B) | prompt_battery.primary_score (CORE adapter) |
1.00 | 1.00 | python -m evals.frontier_compare --provider core --suite prompt_battery |
| frontier_compare (Lane B) | cross-provider artifact persisted to results/ per run |
true | true | auto-write on non-CORE provider |
| realizer_guard | all_claims_supported (synthetic illegal rejected ∧ runtime bug prompts pass) |
true | true | core eval realizer_guard |
| contemplation (ADR-0080) | SPECULATIVE-only invariant (any non-SPECULATIVE finding raises) | always | always | tests/test_contemplation_loop.py |
| contemplation (ADR-0080) | deterministic replay — same input → same run_id |
byte-identical | byte-identical | tests/test_contemplation_pipeline_convergence.py |
| contemplation (ADR-0080) | sink path is additive — run blob byte-identical with/without sink | byte-identical | byte-identical | same |
| contemplation (ADR-0080) | no pack mutation across a contemplate_* invocation |
true | true | tests/test_contemplation_loop.py::test_contemplation_runner_does_not_mutate_pack_tree |
Tier 3 — Learning-curve lanes (expected to improve with teaching)
These are the lanes whose scores should rise monotonically as reviewed teaching examples are added. The headline demo is "N corrections → +X%, locked deterministically, replayable forever."
Each row should be populated by bench learning-curve <lane> --cycles 0,5,10,25,50,100. The bench writes one curve per lane to
evals/reports/learning_curves/<lane>.json with a per-step trace hash
proving every intermediate state is replayable.
| Lane | Score @ 0 | @ 5 | @ 25 | @ 100 | Δ (100−0) | Curve file |
|---|---|---|---|---|---|---|
| cognition | TBD | TBD | TBD | TBD | TBD | learning_curves/cognition.json |
| compositionality | TBD | TBD | TBD | TBD | TBD | learning_curves/compositionality.json |
| monotonic_learning | TBD | TBD | TBD | TBD | TBD | learning_curves/monotonic_learning.json |
| sample_efficiency | TBD | TBD | TBD | TBD | TBD | learning_curves/sample_efficiency.json |
| inference_closure | TBD | TBD | TBD | TBD | TBD | learning_curves/inference_closure.json |
| multi_step_reasoning | TBD | TBD | TBD | TBD | TBD | learning_curves/multi_step_reasoning.json |
| symbolic_logic | TBD | TBD | TBD | TBD | TBD | learning_curves/symbolic_logic.json |
| cross_domain_transfer | TBD | TBD | TBD | TBD | TBD | learning_curves/cross_domain_transfer.json |
| zero_code_domain_acquisition | TBD | TBD | TBD | TBD | TBD | learning_curves/zero_code_domain_acquisition.json |
Monotonicity invariant: for each row, score @ N must not regress
below score @ M for M < N once teaching has plateaued. A regression
is evidence of a non-coherent correction having been admitted and is a
hard finding, not a metric noise event.
Tier 4 — Cost and performance
| Claim | Metric | CORE | Frontier LLM | Bench |
|---|---|---|---|---|
| Median time-to-first-surface | seconds | TBD | TBD | bench latency |
| p95 time-to-first-surface | seconds | TBD | TBD | bench latency |
| Rust vs Python backend speedup | x faster | TBD | n/a | bench speedup |
| Median turn latency | ms | ~445 ms (100-turn measured, "What is truth?", warmup 5) | 200–2000 ms (provider-quoted, varies by model/region) | core bench --suite cost --runs 100 — see evals/reports/cost_latest.json |
| p95 turn latency | ms | ~447 ms (100-turn measured) | varies | bench cost |
| Cost per 1000 turns | USD | $0.0044 at AWS t3.medium on-demand ($0.0416/hr, captured 2026-05-17) | $0.22 (Haiku) · $0.45 (GPT-4o) · $0.66 (Sonnet) at 20 in / 40 out tokens/turn | bench cost — 48–149× cheaper depending on provider |
| Energy per turn | joules | not measured (RAPL/IOKit required; cpu_seconds_total reported instead) | not directly comparable | bench cost energy_disclosure |
| Total deployed footprint (on-disk) | bytes | 7.06 MiB | 14.9 GiB (Llama 3.1 8B fp16) — 2,160× larger; 754 GiB (Llama 3.1 405B) — 109,358× larger | python -c "from benchmarks.footprint import run_footprint; print(run_footprint().summary())" — see evals/reports/footprint_latest.json |
| Resident memory at idle / after one pulse | MiB | 17.9 / 33.3 | n/a (frontier inference requires GPU memory measured in GiB–TiB) | bench footprint (RSS rows) |
| Runs offline on commodity hardware | yes / no | yes | no (requires datacenter GPU + network) | bench footprint (deployment profile) |
| Cost growth with conversation length | USD / turn at depth N | TBD (flat — vault, not window) | TBD (super-linear — context tokens) | bench long-context-cost (uses evals/long_context_cost) |
| Hallucination / fabrication rate | unsupported-answer fraction | TBD | TBD | Tier 2 · surface_groundedness + inference_closure |
Tier 4.5 — Known gaps, openly measured
Per the transparency principle of this project: weaknesses we have a test for live in the public claims document, not in a private TODO. Each row below is a real failure today against a real metric. The intended fix is named so the row turns into a green tier-2 claim when the work lands.
| Lane | Metric | Current | Target | Known fix |
|---|---|---|---|---|
refusal_calibration |
✅ 1.00 | ≥ 0.95 | CLOSED 2026-05-17 — _UNKNOWN_DOMAIN_SURFACE now reads "I don't know — insufficient grounding for that yet.", matching the lane's refusal markers honestly (the prior wording was equivalent in spirit but unrecognizable). The gate was already firing; only the surface text needed alignment. |
|
refusal_calibration |
✅ 0.00 | 0.00 | Holds at target. | |
refusal_calibration |
✅ 1.00 | ≥ 0.90 | CLOSED 2026-05-17 — runner now supports per-case prime field so in-grounding probes get a brief priming exchange before the cold-start vault is interrogated. Previous 1.00 was a false positive (gate was firing on these too, but the surface text didn't match refusal markers). New 1.00 is genuine: vault is seeded, then the probe answers. |
|
contradiction_detection |
✅ 1.00 | ≥ 0.90 | CLOSED 2026-05-17 — TeachingStore.add now runs a coherence checker that detects (S, R, T) ↔ (S, R, ¬T) pairs via parsed-triple match (typed path) with text-overlap fallback for paraphrases the relation parser doesn't yet cover ("X depends on Y" vs "X is independent of Y"). On match, BOTH proposals transition to EpistemicStatus.CONTESTED. Runner reads the new signal directly; versor-spike heuristic retired. |
|
contradiction_detection |
✅ 0.00 | 0.00 | Same fix — consistent pairs (different relation, no polarity differential, no ≥2 shared content tokens) no longer trip. | |
✅ speculative |
speculative |
CLOSED 2026-05-17 — packs/compiler.py:331 and packs/schema.py::LexicalEntry now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. |
||
✅ min_status filter |
min_status filter |
CLOSED 2026-05-17 — VaultStore.store() now stamps every entry with epistemic_status (default SPECULATIVE per ADR-0021 §3); VaultStore.recall(min_status=EpistemicStatus.COHERENT) filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. |
||
| self-reinforcing fabrication via propose() | ✅ stamped + read-side audited | stamped + categorized read sites | CLOSED 2026-05-17 — write-side stamps SPECULATIVE (generate/proposition.py:198). Read-side audit categorized every production vault.recall() callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (TestINV24VaultRecallRegistry, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass min_status=COHERENT. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded realize(proposition, vocab) (now SPECULATIVE-default per Leak A), not from vault.recall. Site-level # INV-24 recall role: comments at every callsite. See docs/truth_seeking_schema.md §Leak C. |
|
articulation_of_status |
✅ 1.00 | ≥ 0.90 | CLOSED 2026-05-17 — CognitiveTurnPipeline tracks subjects of prior SPECULATIVE proposals and prepends (speculative, not yet reviewed) to the surface when a subsequent turn references one of those subjects (subject substring match, ≥4-char tokenized split, or reflexive query shape like "is your answer confirmed?"). The teach turn itself is not self-marked; only subsequent probes are. |
|
articulation_of_status |
✅ 0.00 | 0.00 | Same fix — bare assertions on SPECULATIVE-backed claims are now hedged with the explicit marker. |
A green row in Tier 4.5 graduates to Tier 1/2/3 in the same commit that lands the fix and the result.
Tier 5 — Holdout (sealed)
Held back from the team. Only the harness sees these. Scores are written by the holdout runner and surfaced here without inspection.
| Lane | Holdout score | Public score | Generalization gap |
|---|---|---|---|
| cognition | TBD | TBD | TBD |
| compositionality | TBD | TBD | TBD |
| inference_closure | TBD | TBD | TBD |
A gap > 0.10 on any lane is a fail signal — investigate before claiming the lane in public.
How to extend this document
- Add a row only after the bench command that produces it has been run
and committed evidence exists under
evals/reports/orevals/<lane>/results/. - Never paste a number without the command and trace hash that made it.
- If a number regresses, do not edit the old number in place — append a dated row and explain the regression in the same commit.
- Tier 1 rows are immutable in shape. New rows go to Tier 2 or 3.