core/benchmarks

Benchmarks

Operator-runnable measurement harnesses for CORE. Each suite anchors a specific load-bearing claim in numbers an outsider can reproduce.

core bench --suite articulation
core bench --suite determinism --runs 50
core bench --suite teaching-loop --runs 100
core bench --suite cost --runs 100
core bench --suite latency
core bench --suite speedup
core bench --suite versor
core bench --suite convergence
core bench --suite realizer

This document covers all benchmark suites in the harness, with emphasis on the post-Phase-4 articulation suite (the newest and most comparative one).

---

Why these benchmarks exist

CORE makes three structurally unusual claims:

1. Reconstruction over storage. The user-facing surface is re-derived from immutable corpora + ratified packs every turn, not sampled from a stochastic process. → Same input must produce byte-identical output across runs.
2. Bounded footprint. No transformer weights, no rolling token history, no embedding store. → Memory stays flat across thousands of turns.
3. Traceable grounding. Every surface carries a grounding tag pointing at the resolving pack id or chain id. → We can audit why a sentence was produced.

These claims need evidence, not assertions. The benchmarks here are the evidence.

---

Suite catalogue

Suite	Anchors	What it does	How to read it
articulation	Phase 4 capability claim	Fires every supported intent shape; reruns to prove determinism; samples RSS across many turns; walks cross-topic prompts; side-by-side with Ollama	All-identical on determinism; flat RSS; OOV fall-through visible; ollama shows ≥2 unique surfaces per prompt
determinism	"Same input → same output"	N reruns of the pulse loop, compares trace hashes	unique_hashes should be 1
teaching-loop	"Replayable learning"	Builds a proposal, runs replay, accepts, asserts active corpus is byte-identical to a deterministic baseline across N runs	unique(proposal_id) and unique(chain_id) should both be 1
cost	Tier-4 CLAIMS.md cost claim	Wall + CPU seconds per turn, $/1000-turn at a disclosed cloud rate, frontier price context	Higher throughput / lower $ = better; frontier pricing context for apples-to-apples
latency	Time-to-first-surface	Single pulse call timed	Lower ms = better
speedup	Rust backend lift	Python vs Rust on identical workload	Speedup factor > 1× when Rust available
versor	CGA field invariant	Walks the pulse loop and checks versor_condition(F) < 1e-6 on every transition	Must always pass; failures point at the operator that broke closure
convergence	Pulse loop terminates	Field returns to a stable state within bounded iterations	Bounded — no runaway
realizer	Pack template coverage	Counts realizer hits vs misses on a fixed prompt set	Higher coverage = better

---

The articulation suite (Phase 4)

The newest and most operator-facing bench. Anchors the post-ADR-0067 claim that CORE can:

• Reach every supported intent shape (DEFINITION / RECALL / CAUSE / VERIFICATION / COMPARISON / CORRECTION / PROCEDURE / NARRATIVE / EXAMPLE) plus the OOV fall-through plus cross-pack chains.
• Emit byte-identical surfaces across reruns.
• Hold memory roughly flat across hundreds of turns.
• Maintain thread context across topic shifts.
• Outperform stochastic models on the determinism axis the system is actually designed for.

Running it

# Quick smoke (no Ollama):
core bench --suite articulation --runs 5 --turns 50

# Full run with Ollama side-by-side (needs `ollama` on PATH + a model):
core bench --suite articulation \
    --runs 10 --turns 200 \
    --ollama-model llama3:8b --ollama-reruns 3 \
    --report bench_reports/articulation.json

# Machine-readable JSON:
core bench --suite articulation --json

Flags:

• --runs N — Determinism rerun count per prompt. Higher is stricter.
• --turns N — Footprint sample run length. The bench drives one ChatRuntime through N cold-start prompts and samples RSS.
• --ollama-model MODEL — Ollama model id (e.g. llama3:8b, granite3.3:8b). Omit to skip the side-by-side.
• --ollama-reruns N — Per-prompt rerun count for Ollama. The bench measures unique_surfaces / reruns for each side.
• --report PATH — Write the full JSON report to disk in addition to printing.

Sub-benches

1. Intent breadth

Sends one prompt per intent shape (12 prompts total covering 9 intents + OOV + 2 cross-pack variants) through fresh ChatRuntime instances. Reports the classified intent, the grounding tag, and a surface snippet.

Read it like this:

• Good: Every intent fires; grounding tier matches the prompt (e.g. CAUSE on knowledge should route to teaching).
• Neutral: A prompt routes to vault instead of pack/teaching — that's CORE's normal recall path on warm vaults, but the breadth bench uses fresh runtimes so vault hits would indicate a stub injection issue worth investigating.
• Bad: Any prompt routes to none when the breadth set says it shouldn't. Means a pack/teaching path regressed.

2. Determinism

Five prompts × N reruns × fresh ChatRuntime each time. Counts unique surfaces per prompt.

Read it like this:

• Good: unique_surfaces == 1 for every prompt. This is the primary claim — any failure here is load-bearing.
• Neutral: unique_surfaces > 1 only for prompts that route through vault (vault recall is content-similar but not byte-stable across compile permutations); reroute the prompt or fix the cold-start path.
• Bad: unique_surfaces > 1 on a pack/teaching/cross-pack prompt. That means a supposedly deterministic composer is reading non-deterministic state. Stop and bisect.

3. Memory footprint

Single ChatRuntime, turns prompts (cycling through the breadth set), RSS sampled every sample_every turns via psutil.

Read it like this:

• Good: Per-turn ΔRSS in the low tens of KiB. Vault grows bounded with stored states; pack caches are immutable.
• Neutral: Linear growth on the first ~100 turns as caches warm up, then flat. Inspect the samples list — if it plateaus, healthy.
• Bad: Per-turn ΔRSS in MiB or growth that does not plateau. Points at unbounded list/dict accumulation; check vault eviction and any lru_cache(maxsize=None) introductions.

4. Cross-topic context

One runtime with thread_anaphora=True. Walks 8 prompts that switch between cognition, relations, and cross-pack subjects. Reports per- turn anaphora-fire status.

Read it like this:

• Good: The state survives across the walk — thread_context retains every turn's TurnSummary. The bench prints which turns fired anaphora.
• Neutral / expected: anaphora_fire_count == 0 once the vault has content. Per ADR-0066 §Future ADRs, anaphora today fires only when BOTH the prior and current turn are pack/teaching tier. After the first turn populates the vault, recall hits the vault and the anaphora prefix is suppressed. This is the architectural ceiling, not a defect.
• Bad: Exceptions or grounding_source == "none" on prompts that should ground. Means an intent route or pack mount broke.

5. Ollama side-by-side

Opt-in. Runs three prompts through both CORE and an Ollama model, each N times. Reports unique-surface count per side.

Read it like this:

• Good (the whole point): CORE shows unique_surfaces == 1 for every prompt regardless of rerun count. Ollama shows unique_surfaces >= 2 on most prompts even with low rerun count because LLMs are stochastic.
• Why this matters: A user asking the same question twice should get the same answer. CORE guarantees this structurally. LLMs at temperature=0 come close but still vary because of GPU non-determinism, MoE routing, and sampling on tie-break logits.
• Comparing surfaces: Don't compare content quality between CORE and Ollama. They optimise different objectives:
  • CORE: traceable, deterministic, every token sourced.
  • Ollama: fluent, broad, stochastic, no provenance.
• What "fail" looks like: Ollama is not on PATH → skipped (not failed). Ollama returns <ollama error: ...> → that prompt is excluded from the unique-surface count.

Comparison caveat

CORE and Ollama are not running the same task in a fair sense:

• Different vocabulary surface area. Ollama has read most of the internet. CORE has 3 ratified packs (~150 lemmas total). Asking "What is photosynthesis?" of CORE produces the OOV invitation by design; asking it of Ollama produces a paragraph. Neither is wrong — the systems make different promises.
• Different success criteria. CORE wins on determinism, provenance, footprint, replayability. Ollama wins on coverage and fluency. The bench measures the axes CORE was designed for.
• Different latency profile. CORE: single-digit ms. Ollama: hundreds of ms to seconds depending on model. The articulation bench does not time Ollama; the cost bench is the place for per-turn timing.

If a prompt is in CORE's pack vocabulary, the side-by-side is direct: same prompt, identical surface from CORE every run, varying surface from Ollama every run. If a prompt is OOV for CORE, the side-by-side is informational: shows what the gradient does (CORE admits it doesn't know; Ollama hallucinates plausibly).

---

Other suites — quick reference

determinism

core bench --suite determinism --runs 50

The original determinism bench: drives the pulse loop (not the chat runtime) N times, hashes the full trace, asserts a single unique hash. The articulation suite's determinism sub-bench tests the chat runtime; this one tests the deeper pulse loop.

teaching-loop

core bench --suite teaching-loop --runs 100

Anchors ADR-0055..0057. Drives the discovery → proposal → replay → accept loop end-to-end N times. Asserts the resulting active corpus is byte-identical to a deterministic baseline and that the proposal id + chain id are stable across runs.

cost

core bench --suite cost --runs 100

Measures wall-seconds, CPU-seconds, throughput, and derives $/1000-turn at a disclosed cloud-instance rate. Reports frontier LLM per-token pricing context. Energy/joules is not reported because honest measurement requires privileged RAPL/IOKit access.

latency / speedup / versor / convergence / realizer

These are the original benches in benchmarks/run_benchmarks.py — each runs a single measurement and emits a BenchResult(passed, metric, unit, detail). Run them when investigating regressions in the specific axis named.

---

Operator workflow

1. After any non-trivial change, run:

   core test --suite cognition -q
   core eval cognition
   core bench --suite articulation --runs 5 --turns 50
   

2. Before merging a PR that touches surface composers, runtime, packs, or teaching corpora, also run:

   core bench --suite teaching-loop --runs 50
   core bench --suite articulation --runs 20 --turns 200 \
       --ollama-model llama3:8b --report bench_reports/<branch>.json
   

3. When investigating a regression, run the targeted suite:
   • Determinism breakage → --suite determinism --runs 50
   • Memory growth → --suite articulation --turns 1000
   • Versor closure error → --suite versor
   • Pack template miss → --suite realizer

The bench reports go under bench_reports/ (gitignored by default). Include the JSON report path in the PR description when the change is bench-relevant.

---

What the benchmarks intentionally do NOT do

• Score linguistic quality. Fluency, helpfulness, "naturalness" — none of these are CORE's optimisation target. The system optimises for determinism, provenance, and bounded footprint.
• Report fabricated joules. Energy measurement requires privileged RAPL/IOKit access we don't have in a plain Python process. cost.cpu_seconds is the honest proxy.
• Compare against LLMs on tasks LLMs are designed for. A bench that asks "who writes the better essay?" is the wrong question for CORE. The Ollama side-by-side measures the one axis where the comparison is meaningful: same input → same output.
• Bench-game CORE. Every prompt set is in-tree; modifications show up in PRs; no "private eval set" trick.

---

Adding a new bench

1. Drop a module in benchmarks/<name>.py with a run_<name>() entrypoint returning a dataclass Report with as_dict() and summary() methods.
2. Add a dispatch branch in core/cli.py:cmd_bench.
3. Add the suite name to bench.add_argument("--suite", choices=...).
4. Add a row to the catalogue table at the top of this README.
5. Add tests under tests/test_<name>_bench.py that pin the report shape (not the runtime behaviour — that's covered by lanes).