8f1903e8e7
* chore(evals, cli): contract standardization + bench --json stdout cleanliness
End-of-session shippability pass. Three concrete fixes:
1. core/cli.py — bench --json no longer pollutes stdout
Several bench paths call scripts.run_pulse.run_pulse which prints
verbose [pulse] traces unconditionally to stdout, breaking jq /
programmatic consumers of --json output.
New _bench_stdout_guard() redirects stdout → stderr for the
duration of the bench run when --json is set. Operator still sees
the pulse trace (on stderr), but --json consumers get a clean JSON
document on stdout. Applied to all four bench paths: cost,
articulation, default suite, and --suite all.
Verified: core bench --suite determinism --json now produces
parseable JSON; human path still shows 1140 [pulse] lines.
2. evals/{frontier_compare,realizer_guard}/contract.md (new)
core/contemplation/contract.md (new)
Each new contract follows the established pattern (37 contracts
already exist under evals/<lane>/contract.md):
- What it measures
- Why it matters (structural win)
- How to run
- How to read the output
- Pass criteria table
- When it has failed and why
- Runner / module layout
Coverage:
- frontier_compare: both Lane A (CORE-only suites) and Lane B
(cross-provider prompt_battery) with explicit guardrails
against mixing — operator asks for the wrong lane combination,
runner exits 2 with helpful error.
- realizer_guard: C1/C2 articulation safety boundary — synthetic
illegal candidates rejected directly by check_surface AND
former-bug runtime prompts now produce legal articulations.
- contemplation (ADR-0080): not under evals/ since it's runtime
infrastructure that consumes eval reports — contract lives at
core/contemplation/contract.md. Documents the read-only +
SPECULATIVE-only + deterministic-replay invariants and the
shared DiscoveryCandidateSink plumbing convergence (ADR-0080).
3. evals/CLAIMS.md — Tier 2 rows added
- frontier_compare Lane A: determinism.primary_score, max_versor_condition
- frontier_compare Lane B: prompt_battery.primary_score (CORE adapter),
cross-provider artifact persistence
- realizer_guard: all_claims_supported
- contemplation: SPECULATIVE-only invariant, deterministic replay,
additive sink path, no pack mutation (all CI-pinned by tests)
Verification
------------
$ core test --suite smoke -q
67 passed in 27.22s (no regression)
$ uv run pytest -q tests/test_contemplation_loop.py \
tests/test_contemplation_pipeline_convergence.py \
tests/test_frontier_compare_cross_provider.py
27 passed in 4.87s
$ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed
true (was: JSONDecodeError on prior [pulse] pollution)
* feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart
Stop-hook caught that #62 only covered contracts — the 929-line
report_viewer.html was never audited against the new cross-provider
report shape from #61. Two real gaps:
1. Lane-aware observation drawer
The drawer hardcoded Lane A (CORE-native) fields: surface,
grounding_source, anchor_lens_mode_label, versor_condition.
Lane B (cross-provider) observations carry different fields:
provider, model, elapsed_ms, error_type, error_message.
Loading a cross-provider report rendered only the surface row
with empty `grounding` — the provider + model + timing data
was unreachable without expanding "Show raw JSON".
Fix: detect Lane B (presence of `obs.provider`) and render the
appropriate field set. Lane A still renders identically (now
also surfaces trace_hash + register_id when present, which were
silently buried in the raw JSON before).
2. Pass-rate chart per suite
The summary strip showed one aggregate Primary % across all
suites, with no way to see WHICH suite is dragging the score.
Multi-suite runs (e.g. --suite all) had to expand each panel
individually to find the failing one.
Fix: new .passrate-chart element below the summary strip,
one horizontal bar per suite showing passed/total. All-pass =
solid green, all-fail = solid red, partial = green/red split
at the pass fraction. CSS only — no new dependencies.
3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar
shows the "side-by-side surface evidence across providers"
description when loading a Lane B report.
Verified
--------
- Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54)
- One <script> tag pair preserved
- Generated a real Lane B report via
`python -m evals.frontier_compare --provider core --suite prompt_battery`
for visual confirmation
Out of scope (noted for future PR)
----------------------------------
Sampled 3 `core demo` targets:
- register-tour: clean schema (all_claims_supported, claims, grid)
- audit-tour: both scene_1_* keys AND an empty scenes:[] array — inconsistent
- anti-regression: no all_claims_supported key, uses all_gates_held instead
Demo schema standardization deserves its own PR — operator tooling
would benefit from a uniform top-level success field across demos.
* docs(evals) + chore(demos): systematic audit + uniform success field
Stop-hook caught two real gaps after the contract+UI PR:
- demos had divergent success-field names (all_gates_held vs
learning_loop_closed vs claim_supported vs nested claims_supported)
- no systematic look at the 48 eval directories had been done
Both addressed concretely; remaining work captured in audit doc
rather than vaguely deferred.
1. Demo schema standardization — uniform all_claims_supported field
----------------------------------------------------------------------
All 9 ``core demo`` targets now emit a top-level
``all_claims_supported: bool`` field. Existing per-demo fields
(``all_gates_held``, ``learning_loop_closed``, ``claim_supported``,
nested ``claims_supported``) are preserved for backwards compat —
the new field is an alias derived from the demo's existing success
signal, not a replacement.
Operator tooling and the CI gate can now target
``all_claims_supported`` without knowing each demo's idiomatic
field name.
Files touched:
- evals/anti_regression/run_demo.py — adds AND of all_gates_held +
active_corpus_byte_identical
- evals/learning_loop/run_demo.py — adds AND of learning_loop_closed +
active_corpus_byte_identical
- scripts/publish_pack_measurements.py — adds AND of the three
entries in the nested claims_supported dict
- evals/long_context_cost/comparison_runner.py — adds alias for
claim_supported (singular)
The 5 demos already using ``all_claims_supported`` (audit-tour,
register-tour, anchor-lens-tour, orthogonality-tour, articulation)
are unchanged.
Verified across all 9 demos:
audit-tour : True
register-tour : True
anchor-lens-tour : True
orthogonality-tour : True
pack-measurements : True ← new alias
anti-regression : True ← new alias
learning-loop : True ← new alias
articulation : True
long-context-comparison : True ← new alias
2. docs/EVAL_AUDIT_2026-05-20.md — systematic 48-lane audit
------------------------------------------------------------
Replaces the "future PR" deferral with a concrete document.
Contains:
- Method (what was inspected for each lane).
- Summary (40/48 have contract.md; 18/48 have saved results;
empty results/ ≠ broken — most lanes regenerate on demand).
- Cross-provider relevance triage:
* 9 lanes are cross-provider-relevant and could benefit
from the prompt_battery-style adapter pattern (cognition,
english_fluency_ood, hebrew_fluency, koine_greek_fluency,
grammatical_coverage, inference_closure, multi_step_reasoning,
discourse_paragraph, foundational_*_ood, etc.).
* 29 lanes are CORE-only by design (versor closure, anchor
lens, identity divergence, provenance, etc.) — wiring
providers would be category-erroneous.
- Demo schema standardization status (this PR closes that).
- UI/UX coverage matrix.
- 5 concrete follow-up items, each focused enough for a single
PR, none requiring architectural change.
Regenerated reports
-------------------
evals/long_context_cost/results/comparison_v1.json and
evals/results/phase2_pack_measurements.json now contain the new
all_claims_supported field (auto-regenerated when validating the
schema change).
evals/frontier_compare/results/sample_core_promptbattery.json
added as a reference Lane B report so the new viewer always has
something to load on first open.
17 KiB
17 KiB
CORE — Public Claims
Status: DRAFT v0 — evidence rows below marked TBD are not yet
regenerated by the harness. Do not cite externally until every row is
either a number with a reproducible command, or removed.
Regenerate: core bench claims --json > evals/reports/claims_latest.json
Verification contract: every claim row carries (a) the command that produced it, (b) the lane/version, and (c) a trace hash or report SHA that a third party can re-derive locally. A claim without all three is not a claim — it is marketing.
Smaller. Faster. Stronger.
CORE is a new AI architecture designed to make intelligence more efficient, understandable, and correctable. It aims to provide modern AI capabilities with a much lighter footprint, lower operating cost, stronger memory, clearer reasoning, and a training process built around structured learning instead of brute-force scale.
The thesis, mapped to evidence
Every adjective in the positioning statement below corresponds to a row in the tier tables. If an adjective is not yet backed by a green row, it is an intention, not a claim — and is marked as such here.
| Virtue | What it means in practice | Where it's proved |
|---|---|---|
| Smaller / lighter | Curated packs + exact CGA recall, not corpus dumps + ANN indexes | Tier 4 · bench footprint |
| Cheaper to train | Reviewed corrections move the needle; no retraining cycle | Tier 3 · learning-curve lanes |
| Cheaper to run | Practical on a laptop — energy + latency budget below frontier API floor | Tier 4 · bench latency + bench cost |
| More transparent | Every answer carries a trace hash, term provenance, and intent path | Tier 2 · provenance lane |
| More consistent | Same (pack, vault, seed) → byte-identical surface |
Tier 1 · rows 1–2 |
| Easier to correct | A reviewed correction becomes part of future behavior, deterministically | Tier 1 · row 6 + Tier 3 |
| Better long-term memory | Structured vault, not a sliding context window | Tier 1 · row 4 + Tier 4 · bench long-context-cost |
| Less prone to fabrication | Unsupported answers marked uncertain or refused; surface grounded in vocab | Tier 2 · surface_groundedness, inference_closure |
| More private / local-first | Runs without network egress; no model-provider dependency | Tier 4 · bench footprint (deployment profile) |
| More adaptable | Specialty courses graduate via the Formation pipeline | Tier 3 · zero_code_domain_acquisition |
| More stable over time | New teaching does not silently regress prior lanes | Tier 3 · monotonicity invariant |
| More auditable | Mistakes, corrections, and reasoning paths are inspectable artifacts | Tier 2 · provenance + Tier 5 holdout gap |
Positioning statement
CORE is a new AI architecture designed to make intelligence more efficient, understandable, and correctable. It aims to provide modern AI capabilities with a much lighter footprint, lower operating cost, stronger memory, clearer reasoning, and a training process built around structured learning instead of brute-force scale.
A shorter form, for places that need one sentence:
CORE is designed to be a lighter, more efficient, more transparent, and more teachable alternative to today's large AI models.
Tier 1 — Structural asymmetries (CORE vs frontier LLMs)
These are the load-bearing claims. They are properties of the architecture, not of training scale. They should be bit-stable across releases, not just "high."
| # | Claim | Metric | CORE | Frontier LLM | Lane / bench | Trace hash |
|---|---|---|---|---|---|---|
| 1 | Same prompt → byte-identical surface across N runs | unique_surfaces / N | 1/N | TBD | bench determinism --runs 50 |
TBD |
| 2 | Same (pack, vault, seed) → byte-identical trace hash | unique_hashes / N | 1/N | n/a (no trace) | bench replay-vs-llm |
TBD |
| 3 | Versor closure on every intermediate field state | max_versor_condition |
< 1e-6 | n/a | bench versor |
TBD |
| 4 | Exact CGA recall (no ANN, no cosine) | recall@1 on vault probe | 1.00 | n/a | bench versor + vault probe |
TBD |
| 5 | Reviewed-mutation-only learning (prompt-injection-resistant) | identity-override rejection rate | 1.00 | TBD (expected ≪ 1.00) | evals/adversarial_identity |
TBD |
| 6 | Deterministic teaching replay — N corrections → identical post-state | post-state hash equality | true | n/a | bench learning-curve --verify-replay |
TBD |
Why these come first: every Tier-1 row is something a transformer structurally cannot match without abandoning sampling, weights, or non-deterministic kernels. If any Tier-1 row regresses, ship is blocked.
Tier 2 — Capability lanes (expected strong now)
Lanes where CORE's deterministic pipeline already lands at or above the
contract threshold on v1 public split. Numbers below come from
evals/<lane>/results/.
| Lane | Metric | CORE v1 | Contract threshold | Last run |
|---|---|---|---|---|
| cognition | intent_accuracy | TBD | ≥ 0.90 | TBD |
| cognition | versor_closure_rate | TBD | 1.00 | TBD |
| compositionality | lane score | TBD | TBD | TBD |
| grammatical_coverage | coverage_rate | TBD | TBD | TBD |
| discourse_paragraph | lane score | TBD | TBD | TBD |
| inference_closure | closure_rate | TBD | TBD | TBD |
| provenance | trace_completeness | TBD | 1.00 | TBD |
| adversarial_identity | rejection_rate | TBD | 1.00 | TBD |
| teaching_injection_resistance | speculative_admission_rate | 1.00 | 1.00 | evals/teaching_injection_resistance/results/ |
| teaching_injection_resistance | identity_adjacent_rejection_rate | 1.00 | 1.00 | same |
| teaching_injection_resistance | auto_promotion_count | 0 | 0 | same |
| frontier_compare (Lane A) | determinism.primary_score |
1.00 | 1.00 | python -m evals.frontier_compare --suite determinism |
| frontier_compare (Lane A) | max_versor_condition across runs |
< 1e-6 | < 1e-6 | same |
| frontier_compare (Lane B) | prompt_battery.primary_score (CORE adapter) |
1.00 | 1.00 | python -m evals.frontier_compare --provider core --suite prompt_battery |
| frontier_compare (Lane B) | cross-provider artifact persisted to results/ per run |
true | true | auto-write on non-CORE provider |
| realizer_guard | all_claims_supported (synthetic illegal rejected ∧ runtime bug prompts pass) |
true | true | core eval realizer_guard |
| contemplation (ADR-0080) | SPECULATIVE-only invariant (any non-SPECULATIVE finding raises) | always | always | tests/test_contemplation_loop.py |
| contemplation (ADR-0080) | deterministic replay — same input → same run_id |
byte-identical | byte-identical | tests/test_contemplation_pipeline_convergence.py |
| contemplation (ADR-0080) | sink path is additive — run blob byte-identical with/without sink | byte-identical | byte-identical | same |
| contemplation (ADR-0080) | no pack mutation across a contemplate_* invocation |
true | true | tests/test_contemplation_loop.py::test_contemplation_runner_does_not_mutate_pack_tree |
Tier 3 — Learning-curve lanes (expected to improve with teaching)
These are the lanes whose scores should rise monotonically as reviewed teaching examples are added. The headline demo is "N corrections → +X%, locked deterministically, replayable forever."
Each row should be populated by bench learning-curve <lane> --cycles 0,5,10,25,50,100. The bench writes one curve per lane to
evals/reports/learning_curves/<lane>.json with a per-step trace hash
proving every intermediate state is replayable.
| Lane | Score @ 0 | @ 5 | @ 25 | @ 100 | Δ (100−0) | Curve file |
|---|---|---|---|---|---|---|
| cognition | TBD | TBD | TBD | TBD | TBD | learning_curves/cognition.json |
| compositionality | TBD | TBD | TBD | TBD | TBD | learning_curves/compositionality.json |
| monotonic_learning | TBD | TBD | TBD | TBD | TBD | learning_curves/monotonic_learning.json |
| sample_efficiency | TBD | TBD | TBD | TBD | TBD | learning_curves/sample_efficiency.json |
| inference_closure | TBD | TBD | TBD | TBD | TBD | learning_curves/inference_closure.json |
| multi_step_reasoning | TBD | TBD | TBD | TBD | TBD | learning_curves/multi_step_reasoning.json |
| symbolic_logic | TBD | TBD | TBD | TBD | TBD | learning_curves/symbolic_logic.json |
| cross_domain_transfer | TBD | TBD | TBD | TBD | TBD | learning_curves/cross_domain_transfer.json |
| zero_code_domain_acquisition | TBD | TBD | TBD | TBD | TBD | learning_curves/zero_code_domain_acquisition.json |
Monotonicity invariant: for each row, score @ N must not regress
below score @ M for M < N once teaching has plateaued. A regression
is evidence of a non-coherent correction having been admitted and is a
hard finding, not a metric noise event.
Tier 4 — Cost and performance
| Claim | Metric | CORE | Frontier LLM | Bench |
|---|---|---|---|---|
| Median time-to-first-surface | seconds | TBD | TBD | bench latency |
| p95 time-to-first-surface | seconds | TBD | TBD | bench latency |
| Rust vs Python backend speedup | x faster | TBD | n/a | bench speedup |
| Median turn latency | ms | ~445 ms (100-turn measured, "What is truth?", warmup 5) | 200–2000 ms (provider-quoted, varies by model/region) | core bench --suite cost --runs 100 — see evals/reports/cost_latest.json |
| p95 turn latency | ms | ~447 ms (100-turn measured) | varies | bench cost |
| Cost per 1000 turns | USD | $0.0044 at AWS t3.medium on-demand ($0.0416/hr, captured 2026-05-17) | $0.22 (Haiku) · $0.45 (GPT-4o) · $0.66 (Sonnet) at 20 in / 40 out tokens/turn | bench cost — 48–149× cheaper depending on provider |
| Energy per turn | joules | not measured (RAPL/IOKit required; cpu_seconds_total reported instead) | not directly comparable | bench cost energy_disclosure |
| Total deployed footprint (on-disk) | bytes | 7.06 MiB | 14.9 GiB (Llama 3.1 8B fp16) — 2,160× larger; 754 GiB (Llama 3.1 405B) — 109,358× larger | python -c "from benchmarks.footprint import run_footprint; print(run_footprint().summary())" — see evals/reports/footprint_latest.json |
| Resident memory at idle / after one pulse | MiB | 17.9 / 33.3 | n/a (frontier inference requires GPU memory measured in GiB–TiB) | bench footprint (RSS rows) |
| Runs offline on commodity hardware | yes / no | yes | no (requires datacenter GPU + network) | bench footprint (deployment profile) |
| Cost growth with conversation length | USD / turn at depth N | TBD (flat — vault, not window) | TBD (super-linear — context tokens) | bench long-context-cost (uses evals/long_context_cost) |
| Hallucination / fabrication rate | unsupported-answer fraction | TBD | TBD | Tier 2 · surface_groundedness + inference_closure |
Tier 4.5 — Known gaps, openly measured
Per the transparency principle of this project: weaknesses we have a test for live in the public claims document, not in a private TODO. Each row below is a real failure today against a real metric. The intended fix is named so the row turns into a green tier-2 claim when the work lands.
| Lane | Metric | Current | Target | Known fix |
|---|---|---|---|---|
refusal_calibration |
✅ 1.00 | ≥ 0.95 | CLOSED 2026-05-17 — _UNKNOWN_DOMAIN_SURFACE now reads "I don't know — insufficient grounding for that yet.", matching the lane's refusal markers honestly (the prior wording was equivalent in spirit but unrecognizable). The gate was already firing; only the surface text needed alignment. |
|
refusal_calibration |
✅ 0.00 | 0.00 | Holds at target. | |
refusal_calibration |
✅ 1.00 | ≥ 0.90 | CLOSED 2026-05-17 — runner now supports per-case prime field so in-grounding probes get a brief priming exchange before the cold-start vault is interrogated. Previous 1.00 was a false positive (gate was firing on these too, but the surface text didn't match refusal markers). New 1.00 is genuine: vault is seeded, then the probe answers. |
|
contradiction_detection |
✅ 1.00 | ≥ 0.90 | CLOSED 2026-05-17 — TeachingStore.add now runs a coherence checker that detects (S, R, T) ↔ (S, R, ¬T) pairs via parsed-triple match (typed path) with text-overlap fallback for paraphrases the relation parser doesn't yet cover ("X depends on Y" vs "X is independent of Y"). On match, BOTH proposals transition to EpistemicStatus.CONTESTED. Runner reads the new signal directly; versor-spike heuristic retired. |
|
contradiction_detection |
✅ 0.00 | 0.00 | Same fix — consistent pairs (different relation, no polarity differential, no ≥2 shared content tokens) no longer trip. | |
✅ speculative |
speculative |
CLOSED 2026-05-17 — language_packs/compiler.py:331 and language_packs/schema.py::LexicalEntry now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. |
||
✅ min_status filter |
min_status filter |
CLOSED 2026-05-17 — VaultStore.store() now stamps every entry with epistemic_status (default SPECULATIVE per ADR-0021 §3); VaultStore.recall(min_status=EpistemicStatus.COHERENT) filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. |
||
| self-reinforcing fabrication via propose() | ✅ stamped + read-side audited | stamped + categorized read sites | CLOSED 2026-05-17 — write-side stamps SPECULATIVE (generate/proposition.py:198). Read-side audit categorized every production vault.recall() callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (TestINV24VaultRecallRegistry, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass min_status=COHERENT. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded realize(proposition, vocab) (now SPECULATIVE-default per Leak A), not from vault.recall. Site-level # INV-24 recall role: comments at every callsite. See docs/truth_seeking_schema.md §Leak C. |
|
articulation_of_status |
✅ 1.00 | ≥ 0.90 | CLOSED 2026-05-17 — CognitiveTurnPipeline tracks subjects of prior SPECULATIVE proposals and prepends (speculative, not yet reviewed) to the surface when a subsequent turn references one of those subjects (subject substring match, ≥4-char tokenized split, or reflexive query shape like "is your answer confirmed?"). The teach turn itself is not self-marked; only subsequent probes are. |
|
articulation_of_status |
✅ 0.00 | 0.00 | Same fix — bare assertions on SPECULATIVE-backed claims are now hedged with the explicit marker. |
A green row in Tier 4.5 graduates to Tier 1/2/3 in the same commit that lands the fix and the result.
Tier 5 — Holdout (sealed)
Held back from the team. Only the harness sees these. Scores are written by the holdout runner and surfaced here without inspection.
| Lane | Holdout score | Public score | Generalization gap |
|---|---|---|---|
| cognition | TBD | TBD | TBD |
| compositionality | TBD | TBD | TBD |
| inference_closure | TBD | TBD | TBD |
A gap > 0.10 on any lane is a fail signal — investigate before claiming the lane in public.
How to extend this document
- Add a row only after the bench command that produces it has been run
and committed evidence exists under
evals/reports/orevals/<lane>/results/. - Never paste a number without the command and trace hash that made it.
- If a number regresses, do not edit the old number in place — append a dated row and explain the regression in the same commit.
- Tier 1 rows are immutable in shape. New rows go to Tier 2 or 3.