core/docs/plans/capability_roadmap.md

Capability Roadmap — Phased Plan to the Verifiable Competence Gates

Status: Draft, derived from docs/sessions/SESSION-2026-05-15-capability-gates.md Owner: Joshua Shay Last updated: 2026-05-22 (Phase 6 added; status update appended below)

Status update — 2026-05-17

Work landed since the 2026-05-15 draft (each item is a roadmap input, not yet a roadmap rewrite — that pass is queued):

• Forward Semantic Control chain — ADR-0022 through ADR-0026 accepted; inner-loop admissibility, rotor/frame admissibility, and ranked-with-margin gates implemented and CI-enforced. See README §"Forward Semantic Control".
• Epistemic schema closure — three leaks closed, four new lanes, three new architectural invariants (INV-21/22/23). Realizer-side refusal calibration and contradiction-coherence checker landed.
• Formation pipeline (back half) — Phases 1–7 implemented; 138/138 formation tests pass. See docs/formation_pipeline_plan.md.
• FSC v3 50-case threshold proof matrix — landed on feat/fsc-proof-suite-demo-rebased; PR pending.
• Cost benchmark — bench cost lane reports $/1000 turns + latency with disclosed assumptions; current measurement is 48–149× cheaper per turn than frontier LLMs.
• core pulse import fix — 2026-05-17 hex-literal bug in packs/en_seeder.py repaired; pulse path is operational again.

Closed gates (vs the Phase 1 "Foundational Triple"):

• identity-divergence — lane runner, axes, 93-event shared curriculum, dev / public / holdout splits all present.
• grammatical-coverage — partial; v1 work in flight, no formal contract yet.
• zero-code-domain-acquisition — not started.

A full rewrite of the phase exit-criteria against current state is queued for the next planning pass.

Status update — 2026-05-22

Phase 5 (Curriculum Era) work landed in volume during the 2026-05-17 → 2026-05-22 window. The roadmap gains a new Part-II phase (Phase 6 — Evidence-Governed Domain Layer, see below) that did not exist in the 2026-05-15 draft because the substrate it ratifies (multi-domain packs, signed reviewer registry, audit-passed gate) did not yet exist either. The phase is documented here as accepted and partially landed, not proposed.

Major chains accepted since 2026-05-17:

• Pack-layer chain — ADR-0027 through ADR-0045. Identity packs (0027/0028), safety packs (0029), ethics packs (0033/0036/0037), reviewed verdicts (0035), audit completeness (0039), telemetry sink (0040), CLI verdicts + fan-out (0041), audit-tour demo (0042), pack measurements (0043), medical-ethics worked example (0044), long-context comparison evidence (0045).
• Forward graph constraint → pack-grounded surface — ADR-0046 through ADR-0070. PropositionGraph becomes AdmissibilityRegion before generate (0046/0047); pack-grounded surface composers for DEFINITION/RECALL/COMPARISON/PROCEDURE/CORRECTION/NARRATIVE/EXAMPLE intents (0048–0066); cross-pack resolver and teaching-corpus chain (0063/0064); register terse v1 (0070); seeded variation (0071); register telemetry + tour (0072).
• Anchor lens substrate — ADR-0073 + sub-ADRs. Substantive variation axis sibling to register, with the opposite invariant (lens moves trace_hash DISTINCT; register holds it CONSTANT). Both orthogonal-axis demos CI-pinned simultaneously.
• Transitive chain surface and definitional groundwork — ADR-0078–0089. Composer/graph atom equivalence telemetry (0078); transitive chain surface (0083); compound-intent dispatch + discourse planner (0089).
• Contemplation Loop Phase 1 — ADR-0080 (accepted 2026-05-22). Read-only frontier-compare miner emits SPECULATIVE findings only; routes through teaching/review.py; no pack mutation. The first system-emitted gap-finding surface.
• Evidence-governed domain layer — ADR-0091 through ADR-0110. See Phase 6 below.

A full Phase 0–5 exit-criteria rewrite against this expanded state remains queued; that pass is independent of the Phase 6 documentation.

This document walks CORE from its present state through the gating framework defined in the 2026-05-15 session. It is organized into six phases (now seven with Phase 6 below). Each phase has entry criteria, work items, exit criteria, and a benchmark discipline contract.

The benchmark discipline is the spine of the plan. Without it, the phases become aspirational. With it, "are we there yet" becomes a CLI question.

---

Part I — Benchmark Discipline (read first)

The gates are only meaningful if the evals that prove them are honest. Five rules govern every eval lane in this roadmap. They apply uniformly; no exceptions per phase.

Rule 1 — Three-set split per lane

Every lane maintains three disjoint corpora:

• Dev set. Freely visible during development. Used to iterate.
• Public test set. Visible, but tuning against it is forbidden. Scored at version-cut time only. Drift in dev-vs-public scores is a red flag for overfitting.
• Private holdout. Sealed. Never read by Claude, never committed in plaintext, only scored by a clean-room runner at release events. Stored encrypted in evals/holdouts/ with key held by the human reviewer.

If a lane has only a dev set, it does not count as a gate. It is exploration.

Rule 2 — Versioned difficulty escalation

Each lane has versions: v1, v2, v3, … with monotonically harder distributions. Passing a version is not a terminal state; it is a checkpoint that unlocks generating the next version.

• v1 — baseline competence demonstration. The construction is shown clearly.
• v2 — distributional shift: longer chains, deeper nesting, rarer vocabulary, paraphrased surface forms.
• v3 — adversarial: items generated specifically by inspecting model failures on v2.
• v4+ — out-of-distribution: items drawn from domains, registers, or constructions not present at training time.

Score is always reported as a tuple (v1_score, v2_score, v3_score, …), never collapsed to a single number. A model that scores 99% on v1 and 12% on v3 is not a "99% model."

Rule 3 — Adversarial regeneration on pass

When a model passes a version (e.g., ≥95% on the public test set with ≥90% on private holdout), the next version is generated by adversarial process:

• Human review finds construction families the model handled accidentally rather than structurally.
• A separate generator (could be a different model, could be programmatic) produces items targeting the weakest decile of the previous version.
• The new version is reviewed for legitimacy — no impossible items, no ambiguous items, no items that depend on world knowledge the system was never given.

This is the protection against silent overfitting: every passed version triggers the construction of a harder one, so "progress" requires continuously rising scores against continuously harder tests.

Rule 4 — Frontier baseline tracking

For each lane, a baseline score is computed for at least one frontier transformer-based model (e.g., Claude Opus 4.7, GPT-5, Gemini 3 Ultra) on the same public test set. Baselines are:

• Re-scored every time a version is cut.
• Published alongside CORE's score.
• Never tuned, never prompted-engineered to maximize — the prompt is the eval task as written.

This serves two purposes: (a) it makes CORE's structural wins visible (frontier models score near zero on provenance, monotonic learning, etc.); (b) it prevents self-congratulation on lanes where CORE merely matches an LLM that was given no advantage.

Rule 5 — Honest reporting

• Failures are reported with the same prominence as passes.
• Confidence intervals on every score (bootstrapped over the test set).
• Per-construction breakdowns published — never a single aggregate hiding structural failures.
• Regressions across versions are surfaced, never silently dropped.
• "Did not test" is a valid result; "tested and failed" is preferred over "did not test."

If a number cannot be reported honestly under these rules, the lane is not ready. Do not ship the lane.

Eval contract template

Every eval lane lives in evals/<lane_name>/ with this layout:

evals/<lane_name>/
  contract.md           # what the lane measures, scoring rubric, pass thresholds
  dev/                  # dev set, freely visible
  public/v1/            # public test set, version 1
  public/v2/            # public test set, version 2
  ...
  holdouts/             # encrypted, sealed
  runner.py             # deterministic scorer
  baselines/            # frontier model scores per version
  results/              # CORE scores per version per release

A lane without a contract.md does not run.

---

Part II — The Phases

Phase 0 — Benchmark methodology lock-in

Entry criteria. Today.

Goal. Build the discipline infrastructure before building any new eval. Doing this first prevents the entire roadmap from drifting into vibes-based progress.

Work items.

1. Implement evals/ layout convention above.
2. Implement core eval <lane> CLI subcommand that loads contracts, runs the runner, writes results.
3. Implement the holdout-runner: a sandboxed process that decrypts the sealed test set, scores, writes only the aggregate score (never item-level results) back to the working tree.
4. Implement baseline-runner: a thin adapter that queries a frontier model on the public test set and records its score.
5. Write the methodology page in docs/eval_methodology.md (this Part I, extracted).
6. Pick one existing eval (the current core eval cognition) and retrofit it into the new convention as a forcing function. Versions become explicit; holdout is split out; results are reported per-version.

Exit criteria.

• core eval cognition runs under the new convention, with v1 public + private holdout + baseline.
• No new lane is allowed to be merged that does not follow the convention.
• The retrofit revealed at least one item-level methodology issue (silent ambiguity, leaked dev item, unstated assumption) — caught and documented. If the retrofit found nothing, the audit was not real.

Duration estimate. 1–2 weeks of focused work.

---

Phase 1 — Foundational Triple

Entry criteria. Phase 0 exit complete.

Goal. Implement and pass the three gates that determine whether CORE is ready to move from engineering into curriculum:

• grammatical-coverage (fluency)
• zero-code-domain-acquisition (engineering-vs-learning phase shift)
• identity-divergence (identity is load-bearing)

Work items.

1.1 grammatical-coverage

• Enumerate target grammatical constructions for English v1: simple declarative, negation, conjunction, disjunction, embedded clause, relative clause, quantification (universal/existential), basic tense (past/present/future), basic aspect (perfective/imperfective).
• For each construction, write contract test pairs: PropositionGraph → expected surface family. "Expected surface family" is a set of acceptable surfaces, not a single string, with a deterministic acceptance predicate.
• Implement v1 dev/public/holdout (target: ~50/50/50 items).
• Engineer realizer.py to pass v1.
• Once v1 ≥95% public and ≥90% holdout, generate v2 (deeper nesting, rare vocabulary substitution, longer sentences).
• Repeat for Hebrew and Koine Greek using their respective pack morphology.

1.2 zero-code-domain-acquisition

• Define three "surprise domains" never touched in development: pick from {kinship relations, basic arithmetic, simple spatial relations, color taxonomy, calendar relations}.
• Each domain has a pack-only authoring kit: vocabulary, relation predicates, axiom list, ~20 reviewed teaching examples, ~30 articulation prompts.
• Test: an author who knows the system but is forbidden from editing Python attempts to bring CORE to ≥80% articulation accuracy on the prompts using only pack + teaching loop.
• Each Python edit required is a logged "engineering gap" that goes onto the closing list.

1.3 identity-divergence

• Define two identity axis sets, deliberately oriented to produce different stances on the same proposition (e.g., axis-A weights novelty highly, axis-B weights tradition highly; or axis-A is precision-first, axis-B is generosity-first).
• Curate a shared curriculum: ~100 reviewed teaching events, identical for both agents.
• Curate a prompt set where identity should produce measurably different articulations.
• Scoring: an automated divergence metric (e.g., proposition-graph difference) plus a coherence metric (each output must be internally consistent with its own axes).
• Pass: divergence above floor, coherence above floor, both required.
• Also: identity-stripped baseline. The same curriculum with identity disabled should produce articulations whose divergence is at noise floor — proving identity is doing the work.

Exit criteria.

• All three lanes pass v1 on public + holdout.
• The engineering-gap list from 1.2 is either empty or has a documented closing plan.
• v2 generation has been attempted for at least one of the three.

Duration estimate. 4–8 weeks. The realizer work in 1.1 is the bottleneck and may expose deeper engineering gaps.

---

Phase 2 — Structural Wins Made Visible

Entry criteria. Phase 1 exit complete.

Goal. Build the lanes where CORE's architecture wins by design. These should pass relatively early for CORE and fail catastrophically for frontier baselines. The purpose is to publish the contrast.

Lanes:

• provenance — every articulated claim back-points to vault entries / teaching events / pack axioms; replay reproduces trace bit-for-bit.
• monotonic-learning — after N teaching cycles across unrelated domains, competence on domain 1 does not regress. Longitudinal: ≥10 teaching cycles, scored at each step.
• calibration — out-of-pack queries produce typed "no grounding" responses; in-pack queries do not. Distinguish "I don't know" / "incoherent" / "contradicts known."
• symbolic-logic — nested negation, modal operators (must/may/possible/necessary), counterfactual conditionals. Target ≥99% on v1.
• adversarial-identity — 1,000-turn red-team corpus; identity drift below noise floor.

Work items.

• Build each lane following the convention.
• Compute frontier baselines for each. Expected outcomes:
  • provenance — CORE: pass. Frontier: near-zero (no model can produce verifiable per-claim provenance).
  • monotonic-learning — CORE: pass by construction. Frontier: regression visible after fine-tuning rounds (this requires the frontier baseline to actually be fine-tuned, which complicates the comparison — may be reported as "not directly comparable, structural argument applies").
  • calibration — CORE: high if calibration is wired; frontier: confabulates on most OOD prompts.
  • symbolic-logic — CORE: target ≥99% v1; frontier: ~80% v1, collapses on v3.
  • adversarial-identity — CORE: target drift below noise; frontier: persona erodes within ~50–100 turns.
• Publish results page with per-lane comparison.

Anti-overfitting note. The structural wins are structural — the temptation to declare victory after v1 is large. Discipline: v2 and v3 must still be generated and scored. A "structural win" that fails on v3 is a structural claim that was actually a v1 coincidence.

Exit criteria.

• All five lanes have v1 + v2 results published with frontier baselines.
• At least two lanes have v3 results.
• The contrast page is honest about which results are "directly comparable" vs. "structural argument."

Duration estimate. 8–12 weeks. provenance and monotonic-learning may require new instrumentation in vault/ and teaching/.

---

Phase 3 — Reasoning Depth

Entry criteria. Phase 2 exit complete. This is the hardest phase. Expect engineering surprises.

Goal. Lanes that probe whether CORE actually thinks rather than retrieves and articulates.

Lanes:

• compositionality — novel combinations of taught primitives. SCAN/COGS-style splits adapted to proposition graphs.
• inference-closure — derive entailments never directly asserted (transitive, spatial, temporal, causal chains).
• introspection — explain(turn_id) produces a natural-language account that round-trips: a separate run conditioned on the explanation predicts the same articulation.
• multi-step-reasoning — pipeline produces and consumes intermediate proposition-graph states for problems whose solution requires ≥3 inferential hops.
• cross-domain-transfer — competence in domain B rises after teaching only in domain A, via shared structural elements.

Work items.

These will almost certainly expose engineering gaps. Expected gaps:

• generate/graph_planner.py may need an intermediate-state stack rather than a single planning pass (multi-step-reasoning).
• field/propagate.py may need to expose derivable-but-not-asserted recall paths (inference-closure).
• A new cognition/explain.py module may be needed for introspection.
• Cross-domain transfer may require examining how proposition graphs share structural sub-units, which may be a pack-design question more than a code question.

For each gap discovered, the work splits: (a) write the eval, (b) confirm it fails, (c) close the engineering gap, (d) re-run.

Anti-overfitting note. Compositionality is the lane most vulnerable to overfitting. The training-test split must be done by construction family, not by sampling. If the model has seen R(A,B) and R(C,D), the test set must use a novel relation R' applied to seen entities — not a fresh (A,B) pair under a seen R.

Exit criteria.

• All five lanes have v1 results with honest scores (which may be failing — that's acceptable for v1).
• Each failure has either a closed engineering gap or a documented architectural deferral.
• At least two lanes are passing v1 by phase exit.

Duration estimate. 12–24 weeks. This is the phase that decides whether CORE's design lives up to its philosophical claims.

---

Phase 4 — Scale and Efficiency

Entry criteria. Phase 3 exit complete. Phases 1–3 are pass/fail; this phase is quantitative curves.

Goal. Make CORE's quantitative behavior visible: how fast does it learn, how does cost scale, how does it compose at scale.

Lanes:

• sample-efficiency — corrections-to-competence curves across ten unrelated concepts. Plot, do not threshold.
• long-context-cost — vault size vs. per-turn latency curve at 10³, 10⁴, 10⁵, 10⁶ entries. Identify the asymptotic complexity. Decide indexing strategy if super-linear.
• multi-agent-composition — two CORE instances with different identities cooperate on a shared task; each maintains its own deterministic replay. Measure: task completion, replay determinism preserved per agent, no identity bleed.

Work items.

• Build infrastructure for longitudinal measurement (Phase 2's monotonic-learning runner is a starting point).
• Sample-efficiency requires running the teaching loop programmatically with controlled correction budgets.
• Long-context-cost may surface that the current vault/store.py is insufficient at scale — the response is exact indexing (B-tree, suffix array, signature-based bucketing), not approximate recall.
• Multi-agent composition surfaces orchestration questions that may justify a new module (society/ or similar) — defer unless the eval forces it.

Anti-overfitting note. Curves don't overfit the way thresholds do, but they can be selectively reported. Discipline: publish the full curve, not just the best operating point. Confidence intervals at each data point.

Exit criteria.

• Sample-efficiency curves published for ≥10 concepts.
• Vault cost curve published with asymptotic analysis. Indexing strategy decided.
• Multi-agent composition demonstrated for ≥2-agent cooperation with replay preserved.

Duration estimate. 8–16 weeks.

---

Phase 5 — Curriculum Era

Entry criteria. Phase 4 exit complete. From this point forward, engineering changes are exceptional, not routine. The work is curriculum design, reviewed teaching, and domain-specific evals.

Goal. Acquire human-comparable competence across school-level subjects, classical literature, foundational sciences, and the three foundational languages at fluency.

Structure.

The phase has no single exit criterion. Instead, each domain becomes its own sub-phase with its own evals:

• 5.1 English fluency — pack + curriculum sufficient that grammatical-coverage v5 (out-of-distribution registers: legal, poetic, technical, conversational) passes.
• 5.2 Hebrew fluency — analogous, with attention to root-and-pattern morphology.
• 5.3 Koine Greek fluency — analogous, with attention to case and aspect.
• 5.4 Elementary mathematics — number, arithmetic, basic algebra, geometry. Each topic becomes a pack + a domain-specific competence eval.
• 5.5 Foundational physics — kinematics, conservation, basic mechanics.
• 5.6 Foundational biology — taxonomy, cell, system.
• 5.7 Classical literature — reading comprehension at increasing complexity, eventually approaching the John 1:1–2 grounding case as a depth probe.
• (further sub-phases as curriculum expands)

Discipline during this phase.

• Every new domain ships with its own competence eval following the convention.
• The Phase 1–4 lanes are re-run on every release. A new domain that causes regression in a foundational gate is a curriculum bug, not a curriculum success.
• Frontier baselines are re-scored periodically; the contrast remains visible.

This phase has no estimated duration. It is the phase the project lives in after the engineering era ends. Frontier-LLM parity on breadth happens inside this phase if it happens at all — likely measured in years, not weeks, and at whatever sample efficiency Phase 4 demonstrated.

Phase 6 — Evidence-Governed Domain Layer

Entry criteria: Phase 5 corpus flywheel operational (curriculum + miner sourcing actively producing reviewed proposals); pack-layer chain (ADR-0027..0045) closed; forward-graph-constraint and surface-composer chains (ADR-0046..0089) shipping.

Goal. Distinguish contract-passing from demonstrated at the ledger surface. A pack that satisfies the nine ADR-0091 predicates earns a reasoning-capable ledger row; promotion to audit_passed=true requires a reviewer-signed evidence-bundle digest that reproduces byte-for-byte from on-disk lane results.

Substrate (all accepted as of 2026-05-22):

Layer	What it ratifies	ADR
Domain Pack Contract v1	Nine predicate checks on every ratified pack.	0091
Reviewer Registry v1	Schema-validated reviewer roster; primary vs. domain-scoped.	0092
Domain Contract v1 implementation	Validator + ledger enforcement at runtime.	0093
Proposal source provenance	Discriminated ProposalSource(kind=...) for every teaching proposal.	0094
Miner-sourced proposals	teaching/from_miner.py; SHA-pinned miner_loop_closure lane.	0095
Fabrication-control eval lane	Phantom / cross-pack / sibling-collapse refusals must clean.	0096
Demo composition contract	Demos compose from shipped modules; no parallel mechanism.	0098
Public showcase demo	Deterministic, byte-equal, <30s.	0099
Reasoning-capable ratifications	mathematics_logic (0097), physics (0100), systems_software (0101), hebrew_greek_textual_reasoning (0102).
Fluency lane attachment	Hebrew + Koine Greek lanes attached to ADR-0102 packs with dev/public/holdout.	0103
Curriculum-sourced proposals	teaching/from_curriculum.py; SHA-pinned curriculum_loop_closure lane.	0104
Sealed-holdout encryption	age-based; dev-mode plaintext fallback preserved.	0105
Expert-demo promotion contract	Domain-aware, reviewer-signed, replay-deterministic gate.	0106
Lane-shape registry	8 lane ids dispatch to 5 shapes; unknown lanes fail-closed.	0109 (amends 0106)

Worked promotion narrative. The contract has been demonstrated end-to-end:

1. ADR-0107 — first promotion attempt (mathematics_logic) honestly refused by the contract on two named blockers (metric-shape uniformity assumption; inference_closure substantively failing at 40% pass).
2. ADR-0109 — threshold rules amended with explicit lane-shape registry; cognition-shape thresholds preserved bit-identical; four new shapes added (accuracy_shape, inference_shape, refusal_shape, symbolic_logic_shape); unknown lanes fail-closed.
3. PR #117 fixed the intent-classifier regression that had broken inference_closure.
4. ADR-0110 — mathematics_logic promoted to audit_passed=true under the amended contract. Signed claim digest reproduces from on-disk lane results; first domain at audit-passed.
5. ADR-0111 — physics promoted to audit_passed=true without any contract change. Shares inference_closure + fabrication_control results with math; the digest distinguishes via domain_id + evidence_revision. Retires the "math-only" objection; confirms the bridges ADR-0110 landed were correctly scoped.

Exit criteria (cumulative; each can land independently of the others):

• ☑ Contract definition (ADR-0091..0093) and reviewer trust root (ADR-0092) accepted.
• ☑ Negative-control fabrication lane (ADR-0096) ratified and SHA-pinned.
• ☑ Four reasoning-capable domain ratifications (ADR-0097/0100/0101/0102).
• ☑ Expert-demo promotion contract accepted (ADR-0106 + ADR-0109 amendment).
• ☑ First worked promotion lands (ADR-0110 — mathematics_logic).
• ☑ Second worked promotion lands (ADR-0111 — physics; no contract change).
• ☐ Third worked promotion lands (systems_software or hebrew_greek_textual_reasoning, both eligible under the now-amended contract).
• ☐ Multi-reviewer signing (currently single-recipient; the open candidate frontier item from ADR-0105).

Why this is its own phase. Phases 1–5 measure capability internally. Phase 6 is the first layer that measures what the system has actually demonstrated to an external reader and forces the ledger to distinguish the two. Every prior phase makes claims about the substrate; Phase 6 makes the substrate's claims auditable.

This phase has no estimated duration. Each new domain promotion is a discrete unit of work; the contract is the durable artifact.

Audit-passed domains as of 2026-05-22: mathematics_logic, physics, systems_software.

---

Phase 7 — GSM8K-Math Expert Capability (ADR-0114 arc; substrate complete)

Status: Substrate complete as of 2026-05-23. ADR-0120 (first expert promotion contract) is the next gate. Roadmap ADRs: ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0118, ADR-0118a, ADR-0119 + sub-phases, ADR-0120 (proposed)

This phase was introduced by ADR-0114 as a 7-phase arc toward a first expert ledger tier claim. ADR-0114a defines 10 falsifiable anti-overfitting obligations any expert promotion must satisfy. ADR-0119 decomposes the GSM8K eval lane (Phases 1–4 outcome) into 8 sub-phases (5.1 through 5.8), all of which have now landed.

What "substrate complete" means. All machinery needed to score CORE against GSM8K is in place. The first honest measurement is on the books. ADR-0120 sets the numeric expert threshold and signs the first expert_claims entry.

gsm8k_math substrate — what has landed

Sub-phase	ADR	Deliverable
5.1	ADR-0119.1	fabrication_control holdout sealed; key-management pattern established
5.2	ADR-0119.2	200 CORE-original cases (50 dev + 150 public); verify.py 200/200
5.3	ADR-0119.3	evals/gsm8k_math/runner.py; correct/wrong/refused triple; wrong==0 gate
5.4	ADR-0119.4	Frontier-baseline comparison filed (Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%)
5.5	ADR-0119.5	Adversarial suite: 38 cases × 12 families; 0 wrong
5.6	ADR-0119.6	Depth-curve harness; flat at 1.0 across depths 1–8 on public
5.7	ADR-0119.7	Real GSM8K test set age-encrypted (1,319 cases); first honest measurement
5.8	ADR-0119.8	gsm8k_capability_shape registered in LANE_SHAPE_REGISTRY

First honest CORE-vs-real-GSM8K measurement

Split	Correct	Wrong	Refused
Real GSM8K test (sealed, 1,319 cases)	0	0	1,319
CORE-original public (150 cases)	150	0	0

0 wrong against an external benchmark is the load-bearing claim. CORE's grammar covers zero GSM8K test problems today — that is the honest gap. The load-bearing differentiator (zero confabulation) survives the move from CORE-original to real external benchmark.

ADR-0114a obligations: 10 of 10 discharged for gsm8k_math

ADR-0114a defined 10 falsifiable proof obligations any expert promotion must satisfy. All 10 are now discharged on main for the gsm8k_math lane.

#	Obligation	Discharged by
1	Sealed-holdout discipline	ADR-0119.1 (fab_control) + ADR-0119.7 (GSM8K test)
2	OOD surface variation ≥ 0.95 of dev	ADR-0118a (OOD surface generator)
3	Every correct answer ships replay-equal trace	ADR-0117 (solution trace verifier)
4	Typed refusal; wrong == 0	ADR-0116 + ADR-0119.3; holds against real GSM8K (ADR-0119.7)
5	Reasoning-isolation perturbation suite	ADR-0125
6	Compositional-depth curve published	ADR-0119.6 (harness); ε threshold to ADR-0120
7	Frontier-baseline comparison on identical items	ADR-0119.4
8	Adversarial generation; misparse rate zero	ADR-0119.5
9	Determinism across release boundaries	solver + verifier + realizer + runner (inherited)
10	Operation provenance via pack	ADR-0116 + en_arithmetic_v1 pack

What ADR-0120 will compose

ADR-0120 (first expert promotion contract) is the single remaining gate. It will:

1. Set the numeric expert threshold (minimum correct_rate; candidate: ≥ 0.40 / ≥ 0.60 / ≥ 0.85 per ADR-0114)
2. Set the depth-curve ε (per-step error tolerance for the flat-curve requirement)
3. Require all 10 ADR-0114a obligations as hard gates
4. Define the expert_claims schema (mirrors audit_passed_claims from ADR-0106)
5. Sign the first expert_claims entry — or refuse honestly if the correct_rate gate is not met

Per the gate-as-process pattern established by ADR-0107, the first promotion attempt (ADR-0121) will likely defer honestly on the correct_rate gate, since CORE's current grammar covers 0/1319 real GSM8K test problems. That is the expected and correct result; it makes the gap measurable and actionable.

---

Part III — Cross-Cutting Considerations

Versioning of the framework itself

This roadmap is v1. As phases complete, the framework may itself need amendment — new lanes added, methodology refined. Treat the roadmap with the same discipline as the evals: version it, never silently rewrite it. Each amendment is dated and explained.

Scope decisions deferred

Two scope decisions named in the 2026-05-15 session remain open and will be pinned before they cause drift:

• Agency — responsive vs. goal-directed. Defaulting to responsive for Phases 0–4. Phase 5 may revisit.
• Embodiment — symbolic-only vs. sensorimotor. ADR-0013 establishes the sensorium protocol; this roadmap does not assume sensorium-dependent gates in Phases 0–4. Phase 5 may add sensorium-dependent sub-phases.

Two further questions emerged during Phase planning that should be decided early:

• Tool use. Is the pipeline extensible to typed deterministic operators (calculator, search, code execution)? Decision needed before Phase 3, since multi-step-reasoning may benefit from operator delegation.
• Code generation. Is code a first-class proposition-graph articulation target? Decision needed before Phase 5 if computer-science is a curriculum domain.

What this roadmap is not

• Not a list of features. The features fall out of the gates.
• Not a competitive roadmap against frontier LLMs. The contrast is a side effect, not a target.
• Not a commitment to dates. The duration estimates are calibration aids, not deliverable schedules.
• Not a substitute for the work-sequencing list in CLAUDE.md. That list governs daily work; this document governs the arc.

Failure modes to watch for

• Vibes-based progress. "It feels smarter" is not a gate.
• Demo-driven development. Crafting a single impressive interaction is not progress; passing a sealed holdout is.
• Teaching-set leakage. If the same content appears in pack, teaching, and eval, scores are uninterpretable.
• Frontier envy. Trying to match frontier LLMs on lanes where they structurally win (e.g., long-tail stylistic breadth) compromises the lanes where CORE structurally wins.
• Lane proliferation. Adding lanes is cheap; maintaining honest holdouts is expensive. Resist new lanes unless they probe a distinct capability.

---

Part IV — Immediate Next Actions

1. Decide whether this roadmap is promoted to an ADR (likely ADR-0016).
2. Stub docs/eval_methodology.md as the extracted Part I (it's the contract every lane inherits).
3. Begin Phase 0: implement evals/ convention, retrofit core eval cognition into it.
4. Pin the agency scope decision in writing before Phase 3 begins.
5. Pin the tool use scope decision in writing before Phase 3 begins.

Phase 0 starts when the human reviewer signs off on this roadmap. The first measurable signal of progress is the core eval cognition retrofit landing under the new convention.