8f1903e8e7
* chore(evals, cli): contract standardization + bench --json stdout cleanliness
End-of-session shippability pass. Three concrete fixes:
1. core/cli.py — bench --json no longer pollutes stdout
Several bench paths call scripts.run_pulse.run_pulse which prints
verbose [pulse] traces unconditionally to stdout, breaking jq /
programmatic consumers of --json output.
New _bench_stdout_guard() redirects stdout → stderr for the
duration of the bench run when --json is set. Operator still sees
the pulse trace (on stderr), but --json consumers get a clean JSON
document on stdout. Applied to all four bench paths: cost,
articulation, default suite, and --suite all.
Verified: core bench --suite determinism --json now produces
parseable JSON; human path still shows 1140 [pulse] lines.
2. evals/{frontier_compare,realizer_guard}/contract.md (new)
core/contemplation/contract.md (new)
Each new contract follows the established pattern (37 contracts
already exist under evals/<lane>/contract.md):
- What it measures
- Why it matters (structural win)
- How to run
- How to read the output
- Pass criteria table
- When it has failed and why
- Runner / module layout
Coverage:
- frontier_compare: both Lane A (CORE-only suites) and Lane B
(cross-provider prompt_battery) with explicit guardrails
against mixing — operator asks for the wrong lane combination,
runner exits 2 with helpful error.
- realizer_guard: C1/C2 articulation safety boundary — synthetic
illegal candidates rejected directly by check_surface AND
former-bug runtime prompts now produce legal articulations.
- contemplation (ADR-0080): not under evals/ since it's runtime
infrastructure that consumes eval reports — contract lives at
core/contemplation/contract.md. Documents the read-only +
SPECULATIVE-only + deterministic-replay invariants and the
shared DiscoveryCandidateSink plumbing convergence (ADR-0080).
3. evals/CLAIMS.md — Tier 2 rows added
- frontier_compare Lane A: determinism.primary_score, max_versor_condition
- frontier_compare Lane B: prompt_battery.primary_score (CORE adapter),
cross-provider artifact persistence
- realizer_guard: all_claims_supported
- contemplation: SPECULATIVE-only invariant, deterministic replay,
additive sink path, no pack mutation (all CI-pinned by tests)
Verification
------------
$ core test --suite smoke -q
67 passed in 27.22s (no regression)
$ uv run pytest -q tests/test_contemplation_loop.py \
tests/test_contemplation_pipeline_convergence.py \
tests/test_frontier_compare_cross_provider.py
27 passed in 4.87s
$ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed
true (was: JSONDecodeError on prior [pulse] pollution)
* feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart
Stop-hook caught that #62 only covered contracts — the 929-line
report_viewer.html was never audited against the new cross-provider
report shape from #61. Two real gaps:
1. Lane-aware observation drawer
The drawer hardcoded Lane A (CORE-native) fields: surface,
grounding_source, anchor_lens_mode_label, versor_condition.
Lane B (cross-provider) observations carry different fields:
provider, model, elapsed_ms, error_type, error_message.
Loading a cross-provider report rendered only the surface row
with empty `grounding` — the provider + model + timing data
was unreachable without expanding "Show raw JSON".
Fix: detect Lane B (presence of `obs.provider`) and render the
appropriate field set. Lane A still renders identically (now
also surfaces trace_hash + register_id when present, which were
silently buried in the raw JSON before).
2. Pass-rate chart per suite
The summary strip showed one aggregate Primary % across all
suites, with no way to see WHICH suite is dragging the score.
Multi-suite runs (e.g. --suite all) had to expand each panel
individually to find the failing one.
Fix: new .passrate-chart element below the summary strip,
one horizontal bar per suite showing passed/total. All-pass =
solid green, all-fail = solid red, partial = green/red split
at the pass fraction. CSS only — no new dependencies.
3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar
shows the "side-by-side surface evidence across providers"
description when loading a Lane B report.
Verified
--------
- Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54)
- One <script> tag pair preserved
- Generated a real Lane B report via
`python -m evals.frontier_compare --provider core --suite prompt_battery`
for visual confirmation
Out of scope (noted for future PR)
----------------------------------
Sampled 3 `core demo` targets:
- register-tour: clean schema (all_claims_supported, claims, grid)
- audit-tour: both scene_1_* keys AND an empty scenes:[] array — inconsistent
- anti-regression: no all_claims_supported key, uses all_gates_held instead
Demo schema standardization deserves its own PR — operator tooling
would benefit from a uniform top-level success field across demos.
* docs(evals) + chore(demos): systematic audit + uniform success field
Stop-hook caught two real gaps after the contract+UI PR:
- demos had divergent success-field names (all_gates_held vs
learning_loop_closed vs claim_supported vs nested claims_supported)
- no systematic look at the 48 eval directories had been done
Both addressed concretely; remaining work captured in audit doc
rather than vaguely deferred.
1. Demo schema standardization — uniform all_claims_supported field
----------------------------------------------------------------------
All 9 ``core demo`` targets now emit a top-level
``all_claims_supported: bool`` field. Existing per-demo fields
(``all_gates_held``, ``learning_loop_closed``, ``claim_supported``,
nested ``claims_supported``) are preserved for backwards compat —
the new field is an alias derived from the demo's existing success
signal, not a replacement.
Operator tooling and the CI gate can now target
``all_claims_supported`` without knowing each demo's idiomatic
field name.
Files touched:
- evals/anti_regression/run_demo.py — adds AND of all_gates_held +
active_corpus_byte_identical
- evals/learning_loop/run_demo.py — adds AND of learning_loop_closed +
active_corpus_byte_identical
- scripts/publish_pack_measurements.py — adds AND of the three
entries in the nested claims_supported dict
- evals/long_context_cost/comparison_runner.py — adds alias for
claim_supported (singular)
The 5 demos already using ``all_claims_supported`` (audit-tour,
register-tour, anchor-lens-tour, orthogonality-tour, articulation)
are unchanged.
Verified across all 9 demos:
audit-tour : True
register-tour : True
anchor-lens-tour : True
orthogonality-tour : True
pack-measurements : True ← new alias
anti-regression : True ← new alias
learning-loop : True ← new alias
articulation : True
long-context-comparison : True ← new alias
2. docs/EVAL_AUDIT_2026-05-20.md — systematic 48-lane audit
------------------------------------------------------------
Replaces the "future PR" deferral with a concrete document.
Contains:
- Method (what was inspected for each lane).
- Summary (40/48 have contract.md; 18/48 have saved results;
empty results/ ≠ broken — most lanes regenerate on demand).
- Cross-provider relevance triage:
* 9 lanes are cross-provider-relevant and could benefit
from the prompt_battery-style adapter pattern (cognition,
english_fluency_ood, hebrew_fluency, koine_greek_fluency,
grammatical_coverage, inference_closure, multi_step_reasoning,
discourse_paragraph, foundational_*_ood, etc.).
* 29 lanes are CORE-only by design (versor closure, anchor
lens, identity divergence, provenance, etc.) — wiring
providers would be category-erroneous.
- Demo schema standardization status (this PR closes that).
- UI/UX coverage matrix.
- 5 concrete follow-up items, each focused enough for a single
PR, none requiring architectural change.
Regenerated reports
-------------------
evals/long_context_cost/results/comparison_v1.json and
evals/results/phase2_pack_measurements.json now contain the new
all_claims_supported field (auto-regenerated when validating the
schema change).
evals/frontier_compare/results/sample_core_promptbattery.json
added as a reference Lane B report so the new viewer always has
something to load on first open.
211 lines
17 KiB
Markdown
211 lines
17 KiB
Markdown
# CORE — Public Claims
|
||
|
||
**Status:** DRAFT v0 — evidence rows below marked `TBD` are not yet
|
||
regenerated by the harness. Do not cite externally until every row is
|
||
either a number with a reproducible command, or removed.
|
||
|
||
**Regenerate:** `core bench claims --json > evals/reports/claims_latest.json`
|
||
|
||
**Verification contract:** every claim row carries (a) the command that
|
||
produced it, (b) the lane/version, and (c) a trace hash or report SHA
|
||
that a third party can re-derive locally. A claim without all three is
|
||
not a claim — it is marketing.
|
||
|
||
---
|
||
|
||
> **Smaller. Faster. Stronger.**
|
||
>
|
||
> CORE is a new AI architecture designed to make intelligence more
|
||
> efficient, understandable, and correctable. It aims to provide modern
|
||
> AI capabilities with a much lighter footprint, lower operating cost,
|
||
> stronger memory, clearer reasoning, and a training process built
|
||
> around structured learning instead of brute-force scale.
|
||
|
||
## The thesis, mapped to evidence
|
||
|
||
Every adjective in the positioning statement below corresponds to a
|
||
row in the tier tables. If an adjective is not yet backed by a green
|
||
row, it is an *intention*, not a claim — and is marked as such here.
|
||
|
||
| Virtue | What it means in practice | Where it's proved |
|
||
|--------|---------------------------|-------------------|
|
||
| **Smaller / lighter** | Curated packs + exact CGA recall, not corpus dumps + ANN indexes | Tier 4 · `bench footprint` |
|
||
| **Cheaper to train** | Reviewed corrections move the needle; no retraining cycle | Tier 3 · learning-curve lanes |
|
||
| **Cheaper to run** | Practical on a laptop — energy + latency budget below frontier API floor | Tier 4 · `bench latency` + `bench cost` |
|
||
| **More transparent** | Every answer carries a trace hash, term provenance, and intent path | Tier 2 · `provenance` lane |
|
||
| **More consistent** | Same `(pack, vault, seed)` → byte-identical surface | Tier 1 · rows 1–2 |
|
||
| **Easier to correct** | A reviewed correction becomes part of future behavior, deterministically | Tier 1 · row 6 + Tier 3 |
|
||
| **Better long-term memory** | Structured vault, not a sliding context window | Tier 1 · row 4 + Tier 4 · `bench long-context-cost` |
|
||
| **Less prone to fabrication** | Unsupported answers marked uncertain or refused; surface grounded in vocab | Tier 2 · `surface_groundedness`, `inference_closure` |
|
||
| **More private / local-first** | Runs without network egress; no model-provider dependency | Tier 4 · `bench footprint` (deployment profile) |
|
||
| **More adaptable** | Specialty courses graduate via the Formation pipeline | Tier 3 · `zero_code_domain_acquisition` |
|
||
| **More stable over time** | New teaching does not silently regress prior lanes | Tier 3 · *monotonicity invariant* |
|
||
| **More auditable** | Mistakes, corrections, and reasoning paths are inspectable artifacts | Tier 2 · `provenance` + Tier 5 holdout gap |
|
||
|
||
## Positioning statement
|
||
|
||
> **CORE is a new AI architecture designed to make intelligence more
|
||
> efficient, understandable, and correctable. It aims to provide modern
|
||
> AI capabilities with a much lighter footprint, lower operating cost,
|
||
> stronger memory, clearer reasoning, and a training process built
|
||
> around structured learning instead of brute-force scale.**
|
||
|
||
A shorter form, for places that need one sentence:
|
||
|
||
> **CORE is designed to be a lighter, more efficient, more transparent,
|
||
> and more teachable alternative to today's large AI models.**
|
||
|
||
---
|
||
|
||
## Tier 1 — Structural asymmetries (CORE vs frontier LLMs)
|
||
|
||
These are the load-bearing claims. They are properties of the
|
||
architecture, not of training scale. They should be **bit-stable across
|
||
releases**, not just "high."
|
||
|
||
| # | Claim | Metric | CORE | Frontier LLM | Lane / bench | Trace hash |
|
||
|---|-------|--------|------|--------------|-------------|------------|
|
||
| 1 | Same prompt → byte-identical surface across N runs | unique_surfaces / N | 1/N | TBD | `bench determinism --runs 50` | TBD |
|
||
| 2 | Same (pack, vault, seed) → byte-identical trace hash | unique_hashes / N | 1/N | n/a (no trace) | `bench replay-vs-llm` | TBD |
|
||
| 3 | Versor closure on every intermediate field state | `max_versor_condition` | < 1e-6 | n/a | `bench versor` | TBD |
|
||
| 4 | Exact CGA recall (no ANN, no cosine) | recall@1 on vault probe | 1.00 | n/a | `bench versor` + vault probe | TBD |
|
||
| 5 | Reviewed-mutation-only learning (prompt-injection-resistant) | identity-override rejection rate | 1.00 | TBD (expected ≪ 1.00) | `evals/adversarial_identity` | TBD |
|
||
| 6 | Deterministic teaching replay — N corrections → identical post-state | post-state hash equality | true | n/a | `bench learning-curve --verify-replay` | TBD |
|
||
|
||
**Why these come first:** every Tier-1 row is something a transformer
|
||
*structurally cannot match* without abandoning sampling, weights, or
|
||
non-deterministic kernels. If any Tier-1 row regresses, ship is blocked.
|
||
|
||
---
|
||
|
||
## Tier 2 — Capability lanes (expected strong now)
|
||
|
||
Lanes where CORE's deterministic pipeline already lands at or above the
|
||
contract threshold on v1 public split. Numbers below come from
|
||
`evals/<lane>/results/`.
|
||
|
||
| Lane | Metric | CORE v1 | Contract threshold | Last run |
|
||
|------|--------|---------|-------------------|----------|
|
||
| cognition | intent_accuracy | TBD | ≥ 0.90 | TBD |
|
||
| cognition | versor_closure_rate | TBD | 1.00 | TBD |
|
||
| compositionality | lane score | TBD | TBD | TBD |
|
||
| grammatical_coverage | coverage_rate | TBD | TBD | TBD |
|
||
| discourse_paragraph | lane score | TBD | TBD | TBD |
|
||
| inference_closure | closure_rate | TBD | TBD | TBD |
|
||
| provenance | trace_completeness | TBD | 1.00 | TBD |
|
||
| adversarial_identity | rejection_rate | TBD | 1.00 | TBD |
|
||
| **teaching_injection_resistance** | speculative_admission_rate | **1.00** | 1.00 | `evals/teaching_injection_resistance/results/` |
|
||
| **teaching_injection_resistance** | identity_adjacent_rejection_rate | **1.00** | 1.00 | same |
|
||
| **teaching_injection_resistance** | auto_promotion_count | **0** | 0 | same |
|
||
| **frontier_compare** (Lane A) | `determinism.primary_score` | **1.00** | 1.00 | `python -m evals.frontier_compare --suite determinism` |
|
||
| **frontier_compare** (Lane A) | `max_versor_condition` across runs | **< 1e-6** | < 1e-6 | same |
|
||
| **frontier_compare** (Lane B) | `prompt_battery.primary_score` (CORE adapter) | **1.00** | 1.00 | `python -m evals.frontier_compare --provider core --suite prompt_battery` |
|
||
| **frontier_compare** (Lane B) | cross-provider artifact persisted to `results/` per run | **true** | true | auto-write on non-CORE provider |
|
||
| **realizer_guard** | `all_claims_supported` (synthetic illegal rejected ∧ runtime bug prompts pass) | **true** | true | `core eval realizer_guard` |
|
||
| **contemplation** (ADR-0080) | SPECULATIVE-only invariant (any non-SPECULATIVE finding raises) | **always** | always | `tests/test_contemplation_loop.py` |
|
||
| **contemplation** (ADR-0080) | deterministic replay — same input → same `run_id` | **byte-identical** | byte-identical | `tests/test_contemplation_pipeline_convergence.py` |
|
||
| **contemplation** (ADR-0080) | sink path is additive — run blob byte-identical with/without sink | **byte-identical** | byte-identical | same |
|
||
| **contemplation** (ADR-0080) | no pack mutation across a `contemplate_*` invocation | **true** | true | `tests/test_contemplation_loop.py::test_contemplation_runner_does_not_mutate_pack_tree` |
|
||
|
||
---
|
||
|
||
## Tier 3 — Learning-curve lanes (expected to improve with teaching)
|
||
|
||
These are the lanes whose scores should rise monotonically as reviewed
|
||
teaching examples are added. The headline demo is *"N corrections → +X%,
|
||
locked deterministically, replayable forever."*
|
||
|
||
Each row should be populated by `bench learning-curve <lane>
|
||
--cycles 0,5,10,25,50,100`. The bench writes one curve per lane to
|
||
`evals/reports/learning_curves/<lane>.json` with a per-step trace hash
|
||
proving every intermediate state is replayable.
|
||
|
||
| Lane | Score @ 0 | @ 5 | @ 25 | @ 100 | Δ (100−0) | Curve file |
|
||
|------|-----------|-----|------|-------|-----------|------------|
|
||
| cognition | TBD | TBD | TBD | TBD | TBD | `learning_curves/cognition.json` |
|
||
| compositionality | TBD | TBD | TBD | TBD | TBD | `learning_curves/compositionality.json` |
|
||
| monotonic_learning | TBD | TBD | TBD | TBD | TBD | `learning_curves/monotonic_learning.json` |
|
||
| sample_efficiency | TBD | TBD | TBD | TBD | TBD | `learning_curves/sample_efficiency.json` |
|
||
| inference_closure | TBD | TBD | TBD | TBD | TBD | `learning_curves/inference_closure.json` |
|
||
| multi_step_reasoning | TBD | TBD | TBD | TBD | TBD | `learning_curves/multi_step_reasoning.json` |
|
||
| symbolic_logic | TBD | TBD | TBD | TBD | TBD | `learning_curves/symbolic_logic.json` |
|
||
| cross_domain_transfer | TBD | TBD | TBD | TBD | TBD | `learning_curves/cross_domain_transfer.json` |
|
||
| zero_code_domain_acquisition | TBD | TBD | TBD | TBD | TBD | `learning_curves/zero_code_domain_acquisition.json` |
|
||
|
||
**Monotonicity invariant:** for each row, `score @ N` must not regress
|
||
below `score @ M` for `M < N` once teaching has plateaued. A regression
|
||
is evidence of a non-coherent correction having been admitted and is a
|
||
hard finding, not a metric noise event.
|
||
|
||
---
|
||
|
||
## Tier 4 — Cost and performance
|
||
|
||
| Claim | Metric | CORE | Frontier LLM | Bench |
|
||
|-------|--------|------|--------------|-------|
|
||
| Median time-to-first-surface | seconds | TBD | TBD | `bench latency` |
|
||
| p95 time-to-first-surface | seconds | TBD | TBD | `bench latency` |
|
||
| Rust vs Python backend speedup | x faster | TBD | n/a | `bench speedup` |
|
||
| Median turn latency | ms | **~445 ms** (100-turn measured, "What is truth?", warmup 5) | 200–2000 ms (provider-quoted, varies by model/region) | `core bench --suite cost --runs 100` — see `evals/reports/cost_latest.json` |
|
||
| p95 turn latency | ms | **~447 ms** (100-turn measured) | varies | `bench cost` |
|
||
| Cost per 1000 turns | USD | **$0.0044** at AWS t3.medium on-demand ($0.0416/hr, captured 2026-05-17) | $0.22 (Haiku) · $0.45 (GPT-4o) · $0.66 (Sonnet) at 20 in / 40 out tokens/turn | `bench cost` — **48–149× cheaper** depending on provider |
|
||
| Energy per turn | joules | not measured (RAPL/IOKit required; cpu_seconds_total reported instead) | not directly comparable | `bench cost` energy_disclosure |
|
||
| Total deployed footprint (on-disk) | bytes | **7.06 MiB** | 14.9 GiB (Llama 3.1 8B fp16) — **2,160× larger**; 754 GiB (Llama 3.1 405B) — **109,358× larger** | `python -c "from benchmarks.footprint import run_footprint; print(run_footprint().summary())"` — see `evals/reports/footprint_latest.json` |
|
||
| Resident memory at idle / after one pulse | MiB | 17.9 / 33.3 | n/a (frontier inference requires GPU memory measured in GiB–TiB) | `bench footprint` (RSS rows) |
|
||
| Runs offline on commodity hardware | yes / no | **yes** | no (requires datacenter GPU + network) | `bench footprint` (deployment profile) |
|
||
| Cost growth with conversation length | USD / turn at depth N | TBD (flat — vault, not window) | TBD (super-linear — context tokens) | `bench long-context-cost` (uses `evals/long_context_cost`) |
|
||
| Hallucination / fabrication rate | unsupported-answer fraction | TBD | TBD | Tier 2 · `surface_groundedness` + `inference_closure` |
|
||
|
||
---
|
||
|
||
## Tier 4.5 — Known gaps, openly measured
|
||
|
||
Per the transparency principle of this project: weaknesses we have a
|
||
test for live in the public claims document, not in a private TODO.
|
||
Each row below is a real failure today against a real metric. The
|
||
intended fix is named so the row turns into a green tier-2 claim when
|
||
the work lands.
|
||
|
||
| Lane | Metric | Current | Target | Known fix |
|
||
|------|--------|---------|--------|-----------|
|
||
| ~~`refusal_calibration`~~ | ~~refusal_rate (on out-of-grounding prompts)~~ | ✅ **1.00** | ≥ 0.95 | **CLOSED 2026-05-17** — `_UNKNOWN_DOMAIN_SURFACE` now reads "I don't know — insufficient grounding for that yet.", matching the lane's refusal markers honestly (the prior wording was equivalent in spirit but unrecognizable). The gate was already firing; only the surface text needed alignment. |
|
||
| ~~`refusal_calibration`~~ | ~~fabrication_rate~~ | ✅ **0.00** | 0.00 | Holds at target. |
|
||
| ~~`refusal_calibration`~~ | ~~in_grounding_answer_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — runner now supports per-case `prime` field so in-grounding probes get a brief priming exchange before the cold-start vault is interrogated. Previous 1.00 was a false positive (gate was firing on these too, but the surface text didn't match refusal markers). New 1.00 is genuine: vault is seeded, then the probe answers. |
|
||
| ~~`contradiction_detection`~~ | ~~contradiction_flag_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — `TeachingStore.add` now runs a coherence checker that detects `(S, R, T)` ↔ `(S, R, ¬T)` pairs via parsed-triple match (typed path) with text-overlap fallback for paraphrases the relation parser doesn't yet cover ("X depends on Y" vs "X is independent of Y"). On match, BOTH proposals transition to `EpistemicStatus.CONTESTED`. Runner reads the new signal directly; versor-spike heuristic retired. |
|
||
| ~~`contradiction_detection`~~ | ~~false_flag_rate~~ | ✅ **0.00** | 0.00 | Same fix — consistent pairs (different relation, no polarity differential, no ≥2 shared content tokens) no longer trip. |
|
||
| ~~One-mutation-path audit · Leak A~~ | ~~pack vocab default epistemic_status~~ | ✅ **`speculative`** | `speculative` | **CLOSED 2026-05-17** — `language_packs/compiler.py:331` and `language_packs/schema.py::LexicalEntry` now default to SPECULATIVE; docstring corrected to match ADR-0021 §Schema impact; regression guarded by `tests/test_architectural_invariants.py::TestINV22PackDefaultSpeculative` (3 tests, all passing). 365 existing unmarked pack rows now correctly report SPECULATIVE; explicit COHERENT remains the curator stamp. |
|
||
| ~~One-mutation-path audit · Leak B~~ | ~~vault.recall epistemic awareness~~ | ✅ **`min_status` filter** | `min_status` filter | **CLOSED 2026-05-17** — `VaultStore.store()` now stamps every entry with `epistemic_status` (default SPECULATIVE per ADR-0021 §3); `VaultStore.recall(min_status=EpistemicStatus.COHERENT)` filters out non-admissible entries. All 4 vault.store call sites updated with explicit status. Regression guarded by `tests/test_architectural_invariants.py::TestINV23VaultEpistemicFilter` (4 tests). Inference paths can now opt into evidence-only recall; session lookup retains tier-agnostic default. |
|
||
| ~~One-mutation-path audit · Leak C~~ | self-reinforcing fabrication via propose() | ✅ **stamped + read-side audited** | stamped + categorized read sites | **CLOSED 2026-05-17** — write-side stamps SPECULATIVE (`generate/proposition.py:198`). Read-side audit categorized every production `vault.recall()` callsite as RECOGNITION, EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. INV-24 (`TestINV24VaultRecallRegistry`, 3 tests) forces every new callsite to declare its role; EVIDENCE_USER_FACING sites must pass `min_status=COHERENT`. No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from pack-grounded `realize(proposition, vocab)` (now SPECULATIVE-default per Leak A), not from `vault.recall`. Site-level `# INV-24 recall role:` comments at every callsite. See `docs/truth_seeking_schema.md` §Leak C. |
|
||
| ~~`articulation_of_status`~~ | ~~speculative_articulation_rate~~ | ✅ **1.00** | ≥ 0.90 | **CLOSED 2026-05-17** — `CognitiveTurnPipeline` tracks subjects of prior SPECULATIVE proposals and prepends `(speculative, not yet reviewed)` to the surface when a subsequent turn references one of those subjects (subject substring match, ≥4-char tokenized split, or reflexive query shape like "is your answer confirmed?"). The teach turn itself is not self-marked; only subsequent probes are. |
|
||
| ~~`articulation_of_status`~~ | ~~false_certainty_rate~~ | ✅ **0.00** | 0.00 | Same fix — bare assertions on SPECULATIVE-backed claims are now hedged with the explicit marker. |
|
||
|
||
A green row in Tier 4.5 graduates to Tier 1/2/3 in the same commit
|
||
that lands the fix and the result.
|
||
|
||
---
|
||
|
||
## Tier 5 — Holdout (sealed)
|
||
|
||
Held back from the team. Only the harness sees these. Scores are
|
||
written by the holdout runner and surfaced here without inspection.
|
||
|
||
| Lane | Holdout score | Public score | Generalization gap |
|
||
|------|---------------|--------------|---------------------|
|
||
| cognition | TBD | TBD | TBD |
|
||
| compositionality | TBD | TBD | TBD |
|
||
| inference_closure | TBD | TBD | TBD |
|
||
|
||
A gap > 0.10 on any lane is a fail signal — investigate before claiming
|
||
the lane in public.
|
||
|
||
---
|
||
|
||
## How to extend this document
|
||
|
||
1. Add a row only after the bench command that produces it has been run
|
||
and committed evidence exists under `evals/reports/` or
|
||
`evals/<lane>/results/`.
|
||
2. Never paste a number without the command and trace hash that made it.
|
||
3. If a number regresses, do not edit the old number in place — append
|
||
a dated row and explain the regression in the same commit.
|
||
4. Tier 1 rows are immutable in shape. New rows go to Tier 2 or 3.
|