# Capability Roadmap — Phased Plan to the Verifiable Competence Gates **Status:** Draft, derived from `docs/sessions/SESSION-2026-05-15-capability-gates.md` **Owner:** Joshua Shay **Last updated:** 2026-05-22 (Phase 6 added; status update appended below) ## Status update — 2026-05-17 Work landed since the 2026-05-15 draft (each item is a roadmap input, not yet a roadmap rewrite — that pass is queued): * **Forward Semantic Control chain** — ADR-0022 through ADR-0026 accepted; inner-loop admissibility, rotor/frame admissibility, and ranked-with-margin gates implemented and CI-enforced. See README §"Forward Semantic Control". * **Epistemic schema closure** — three leaks closed, four new lanes, three new architectural invariants (INV-21/22/23). Realizer-side refusal calibration and contradiction-coherence checker landed. * **Formation pipeline (back half)** — Phases 1–7 implemented; 138/138 formation tests pass. See `docs/formation_pipeline_plan.md`. * **FSC v3 50-case threshold proof matrix** — landed on `feat/fsc-proof-suite-demo-rebased`; PR pending. * **Cost benchmark** — `bench cost` lane reports $/1000 turns + latency with disclosed assumptions; current measurement is 48–149× cheaper per turn than frontier LLMs. * **`core pulse` import fix** — 2026-05-17 hex-literal bug in `packs/en_seeder.py` repaired; pulse path is operational again. Closed gates (vs the Phase 1 "Foundational Triple"): * `identity-divergence` — lane runner, axes, 93-event shared curriculum, dev / public / holdout splits all present. * `grammatical-coverage` — partial; v1 work in flight, no formal contract yet. * `zero-code-domain-acquisition` — not started. A full rewrite of the phase exit-criteria against current state is queued for the next planning pass. ## Status update — 2026-05-22 Phase 5 (Curriculum Era) work landed in volume during the 2026-05-17 → 2026-05-22 window. The roadmap gains a new Part-II phase (**Phase 6 — Evidence-Governed Domain Layer**, see below) that did not exist in the 2026-05-15 draft because the substrate it ratifies (multi-domain packs, signed reviewer registry, audit-passed gate) did not yet exist either. The phase is documented here as accepted and partially landed, not proposed. Major chains accepted since 2026-05-17: * **Pack-layer chain — ADR-0027 through ADR-0045.** Identity packs (0027/0028), safety packs (0029), ethics packs (0033/0036/0037), reviewed verdicts (0035), audit completeness (0039), telemetry sink (0040), CLI verdicts + fan-out (0041), audit-tour demo (0042), pack measurements (0043), medical-ethics worked example (0044), long-context comparison evidence (0045). * **Forward graph constraint → pack-grounded surface — ADR-0046 through ADR-0070.** PropositionGraph becomes AdmissibilityRegion before generate (0046/0047); pack-grounded surface composers for DEFINITION/RECALL/COMPARISON/PROCEDURE/CORRECTION/NARRATIVE/EXAMPLE intents (0048–0066); cross-pack resolver and teaching-corpus chain (0063/0064); register terse v1 (0070); seeded variation (0071); register telemetry + tour (0072). * **Anchor lens substrate — ADR-0073 + sub-ADRs.** Substantive variation axis sibling to register, with the opposite invariant (lens moves trace_hash DISTINCT; register holds it CONSTANT). Both orthogonal-axis demos CI-pinned simultaneously. * **Transitive chain surface and definitional groundwork — ADR-0078–0089.** Composer/graph atom equivalence telemetry (0078); transitive chain surface (0083); compound-intent dispatch + discourse planner (0089). * **Contemplation Loop Phase 1 — ADR-0080 (accepted 2026-05-22).** Read-only frontier-compare miner emits `SPECULATIVE` findings only; routes through `teaching/review.py`; no pack mutation. The first system-emitted gap-finding surface. * **Evidence-governed domain layer — ADR-0091 through ADR-0110.** See Phase 6 below. A full Phase 0–5 exit-criteria rewrite against this expanded state remains queued; that pass is independent of the Phase 6 documentation. This document walks CORE from its present state through the gating framework defined in the 2026-05-15 session. It is organized into six phases (now seven with Phase 6 below). Each phase has entry criteria, work items, exit criteria, and a benchmark discipline contract. The benchmark discipline is the spine of the plan. Without it, the phases become aspirational. With it, "are we there yet" becomes a CLI question. --- ## Part I — Benchmark Discipline (read first) The gates are only meaningful if the evals that prove them are honest. Five rules govern every eval lane in this roadmap. They apply uniformly; no exceptions per phase. ### Rule 1 — Three-set split per lane Every lane maintains three disjoint corpora: - **Dev set.** Freely visible during development. Used to iterate. - **Public test set.** Visible, but tuning against it is forbidden. Scored at version-cut time only. Drift in dev-vs-public scores is a red flag for overfitting. - **Private holdout.** Sealed. Never read by Claude, never committed in plaintext, only scored by a clean-room runner at release events. Stored encrypted in `evals/holdouts/` with key held by the human reviewer. If a lane has only a dev set, it does not count as a gate. It is exploration. ### Rule 2 — Versioned difficulty escalation Each lane has versions: `v1`, `v2`, `v3`, … with monotonically harder distributions. Passing a version is not a terminal state; it is a checkpoint that unlocks generating the next version. - **v1** — baseline competence demonstration. The construction is shown clearly. - **v2** — distributional shift: longer chains, deeper nesting, rarer vocabulary, paraphrased surface forms. - **v3** — adversarial: items generated specifically by inspecting model failures on v2. - **v4+** — out-of-distribution: items drawn from domains, registers, or constructions not present at training time. Score is always reported as a tuple `(v1_score, v2_score, v3_score, …)`, never collapsed to a single number. A model that scores 99% on v1 and 12% on v3 is not a "99% model." ### Rule 3 — Adversarial regeneration on pass When a model passes a version (e.g., ≥95% on the public test set with ≥90% on private holdout), the next version is *generated by adversarial process*: - Human review finds construction families the model handled accidentally rather than structurally. - A separate generator (could be a different model, could be programmatic) produces items targeting the weakest decile of the previous version. - The new version is reviewed for legitimacy — no impossible items, no ambiguous items, no items that depend on world knowledge the system was never given. This is the protection against silent overfitting: every passed version triggers the construction of a harder one, so "progress" requires continuously rising scores against continuously harder tests. ### Rule 4 — Frontier baseline tracking For each lane, a baseline score is computed for at least one frontier transformer-based model (e.g., Claude Opus 4.7, GPT-5, Gemini 3 Ultra) on the *same* public test set. Baselines are: - Re-scored every time a version is cut. - Published alongside CORE's score. - Never tuned, never prompted-engineered to maximize — the prompt is the eval task as written. This serves two purposes: (a) it makes CORE's structural wins visible (frontier models score near zero on provenance, monotonic learning, etc.); (b) it prevents self-congratulation on lanes where CORE merely matches an LLM that was given no advantage. ### Rule 5 — Honest reporting - Failures are reported with the same prominence as passes. - Confidence intervals on every score (bootstrapped over the test set). - Per-construction breakdowns published — never a single aggregate hiding structural failures. - Regressions across versions are surfaced, never silently dropped. - "Did not test" is a valid result; "tested and failed" is preferred over "did not test." If a number cannot be reported honestly under these rules, the lane is not ready. Do not ship the lane. ### Eval contract template Every eval lane lives in `evals//` with this layout: ``` evals// contract.md # what the lane measures, scoring rubric, pass thresholds dev/ # dev set, freely visible public/v1/ # public test set, version 1 public/v2/ # public test set, version 2 ... holdouts/ # encrypted, sealed runner.py # deterministic scorer baselines/ # frontier model scores per version results/ # CORE scores per version per release ``` A lane without a `contract.md` does not run. --- ## Part II — The Phases ### Phase 0 — Benchmark methodology lock-in **Entry criteria.** Today. **Goal.** Build the discipline infrastructure before building any new eval. Doing this first prevents the entire roadmap from drifting into vibes-based progress. **Work items.** 1. Implement `evals/` layout convention above. 2. Implement `core eval ` CLI subcommand that loads contracts, runs the runner, writes results. 3. Implement the holdout-runner: a sandboxed process that decrypts the sealed test set, scores, writes only the aggregate score (never item-level results) back to the working tree. 4. Implement baseline-runner: a thin adapter that queries a frontier model on the public test set and records its score. 5. Write the methodology page in `docs/eval_methodology.md` (this Part I, extracted). 6. Pick one *existing* eval (the current `core eval cognition`) and retrofit it into the new convention as a forcing function. Versions become explicit; holdout is split out; results are reported per-version. **Exit criteria.** - `core eval cognition` runs under the new convention, with v1 public + private holdout + baseline. - No new lane is allowed to be merged that does not follow the convention. - The retrofit revealed at least one item-level methodology issue (silent ambiguity, leaked dev item, unstated assumption) — caught and documented. If the retrofit found nothing, the audit was not real. **Duration estimate.** 1–2 weeks of focused work. --- ### Phase 1 — Foundational Triple **Entry criteria.** Phase 0 exit complete. **Goal.** Implement and pass the three gates that determine whether CORE is ready to move from engineering into curriculum: - **grammatical-coverage** (fluency) - **zero-code-domain-acquisition** (engineering-vs-learning phase shift) - **identity-divergence** (identity is load-bearing) **Work items.** **1.1 grammatical-coverage** - Enumerate target grammatical constructions for English v1: simple declarative, negation, conjunction, disjunction, embedded clause, relative clause, quantification (universal/existential), basic tense (past/present/future), basic aspect (perfective/imperfective). - For each construction, write contract test pairs: `PropositionGraph → expected surface family`. "Expected surface family" is a set of acceptable surfaces, not a single string, with a deterministic acceptance predicate. - Implement v1 dev/public/holdout (target: ~50/50/50 items). - Engineer `realizer.py` to pass v1. - Once v1 ≥95% public and ≥90% holdout, generate v2 (deeper nesting, rare vocabulary substitution, longer sentences). - Repeat for Hebrew and Koine Greek using their respective pack morphology. **1.2 zero-code-domain-acquisition** - Define three "surprise domains" never touched in development: pick from {kinship relations, basic arithmetic, simple spatial relations, color taxonomy, calendar relations}. - Each domain has a pack-only authoring kit: vocabulary, relation predicates, axiom list, ~20 reviewed teaching examples, ~30 articulation prompts. - Test: an author who knows the system but is forbidden from editing Python attempts to bring CORE to ≥80% articulation accuracy on the prompts using only pack + teaching loop. - Each Python edit required is a logged "engineering gap" that goes onto the closing list. **1.3 identity-divergence** - Define two identity axis sets, deliberately oriented to produce different stances on the same proposition (e.g., axis-A weights novelty highly, axis-B weights tradition highly; or axis-A is precision-first, axis-B is generosity-first). - Curate a shared curriculum: ~100 reviewed teaching events, identical for both agents. - Curate a prompt set where identity should produce measurably different articulations. - Scoring: an automated divergence metric (e.g., proposition-graph difference) plus a coherence metric (each output must be internally consistent with its own axes). - Pass: divergence above floor, coherence above floor, *both required*. - Also: identity-stripped baseline. The same curriculum with identity disabled should produce articulations whose divergence is at noise floor — proving identity is doing the work. **Exit criteria.** - All three lanes pass v1 on public + holdout. - The engineering-gap list from 1.2 is either empty or has a documented closing plan. - v2 generation has been attempted for at least one of the three. **Duration estimate.** 4–8 weeks. The realizer work in 1.1 is the bottleneck and may expose deeper engineering gaps. --- ### Phase 2 — Structural Wins Made Visible **Entry criteria.** Phase 1 exit complete. **Goal.** Build the lanes where CORE's architecture wins by design. These should pass relatively early *for CORE* and fail catastrophically *for frontier baselines*. The purpose is to publish the contrast. **Lanes:** - **provenance** — every articulated claim back-points to vault entries / teaching events / pack axioms; replay reproduces trace bit-for-bit. - **monotonic-learning** — after N teaching cycles across unrelated domains, competence on domain 1 does not regress. Longitudinal: ≥10 teaching cycles, scored at each step. - **calibration** — out-of-pack queries produce typed "no grounding" responses; in-pack queries do not. Distinguish "I don't know" / "incoherent" / "contradicts known." - **symbolic-logic** — nested negation, modal operators (must/may/possible/necessary), counterfactual conditionals. Target ≥99% on v1. - **adversarial-identity** — 1,000-turn red-team corpus; identity drift below noise floor. **Work items.** - Build each lane following the convention. - Compute frontier baselines for each. Expected outcomes: - **provenance** — CORE: pass. Frontier: near-zero (no model can produce verifiable per-claim provenance). - **monotonic-learning** — CORE: pass by construction. Frontier: regression visible after fine-tuning rounds (this requires the frontier baseline to actually be fine-tuned, which complicates the comparison — may be reported as "not directly comparable, structural argument applies"). - **calibration** — CORE: high if calibration is wired; frontier: confabulates on most OOD prompts. - **symbolic-logic** — CORE: target ≥99% v1; frontier: ~80% v1, collapses on v3. - **adversarial-identity** — CORE: target drift below noise; frontier: persona erodes within ~50–100 turns. - Publish results page with per-lane comparison. **Anti-overfitting note.** The structural wins are *structural* — the temptation to declare victory after v1 is large. Discipline: v2 and v3 must still be generated and scored. A "structural win" that fails on v3 is a structural claim that was actually a v1 coincidence. **Exit criteria.** - All five lanes have v1 + v2 results published with frontier baselines. - At least two lanes have v3 results. - The contrast page is honest about which results are "directly comparable" vs. "structural argument." **Duration estimate.** 8–12 weeks. provenance and monotonic-learning may require new instrumentation in `vault/` and `teaching/`. --- ### Phase 3 — Reasoning Depth **Entry criteria.** Phase 2 exit complete. This is the hardest phase. Expect engineering surprises. **Goal.** Lanes that probe whether CORE actually *thinks* rather than retrieves and articulates. **Lanes:** - **compositionality** — novel combinations of taught primitives. SCAN/COGS-style splits adapted to proposition graphs. - **inference-closure** — derive entailments never directly asserted (transitive, spatial, temporal, causal chains). - **introspection** — `explain(turn_id)` produces a natural-language account that round-trips: a separate run conditioned on the explanation predicts the same articulation. - **multi-step-reasoning** — pipeline produces and consumes intermediate proposition-graph states for problems whose solution requires ≥3 inferential hops. - **cross-domain-transfer** — competence in domain B rises after teaching only in domain A, via shared structural elements. **Work items.** These will almost certainly expose engineering gaps. Expected gaps: - `generate/graph_planner.py` may need an intermediate-state stack rather than a single planning pass (multi-step-reasoning). - `field/propagate.py` may need to expose derivable-but-not-asserted recall paths (inference-closure). - A new `cognition/explain.py` module may be needed for introspection. - Cross-domain transfer may require examining how proposition graphs share structural sub-units, which may be a pack-design question more than a code question. For each gap discovered, the work splits: (a) write the eval, (b) confirm it fails, (c) close the engineering gap, (d) re-run. **Anti-overfitting note.** Compositionality is *the* lane most vulnerable to overfitting. The training-test split must be done by *construction family*, not by sampling. If the model has seen `R(A,B)` and `R(C,D)`, the test set must use a *novel relation R'* applied to seen entities — not a fresh `(A,B)` pair under a seen `R`. **Exit criteria.** - All five lanes have v1 results with honest scores (which may be failing — that's acceptable for v1). - Each failure has either a closed engineering gap or a documented architectural deferral. - At least two lanes are passing v1 by phase exit. **Duration estimate.** 12–24 weeks. This is the phase that decides whether CORE's design lives up to its philosophical claims. --- ### Phase 4 — Scale and Efficiency **Entry criteria.** Phase 3 exit complete. Phases 1–3 are pass/fail; this phase is *quantitative curves*. **Goal.** Make CORE's quantitative behavior visible: how fast does it learn, how does cost scale, how does it compose at scale. **Lanes:** - **sample-efficiency** — corrections-to-competence curves across ten unrelated concepts. Plot, do not threshold. - **long-context-cost** — vault size vs. per-turn latency curve at 10³, 10⁴, 10⁵, 10⁶ entries. Identify the asymptotic complexity. Decide indexing strategy if super-linear. - **multi-agent-composition** — two CORE instances with different identities cooperate on a shared task; each maintains its own deterministic replay. Measure: task completion, replay determinism preserved per agent, no identity bleed. **Work items.** - Build infrastructure for longitudinal measurement (Phase 2's monotonic-learning runner is a starting point). - Sample-efficiency requires running the teaching loop programmatically with controlled correction budgets. - Long-context-cost may surface that the current `vault/store.py` is insufficient at scale — the response is exact indexing (B-tree, suffix array, signature-based bucketing), not approximate recall. - Multi-agent composition surfaces orchestration questions that may justify a new module (`society/` or similar) — defer unless the eval forces it. **Anti-overfitting note.** Curves don't overfit the way thresholds do, but they can be selectively reported. Discipline: publish the full curve, not just the best operating point. Confidence intervals at each data point. **Exit criteria.** - Sample-efficiency curves published for ≥10 concepts. - Vault cost curve published with asymptotic analysis. Indexing strategy decided. - Multi-agent composition demonstrated for ≥2-agent cooperation with replay preserved. **Duration estimate.** 8–16 weeks. --- ### Phase 5 — Curriculum Era **Entry criteria.** Phase 4 exit complete. From this point forward, engineering changes are exceptional, not routine. The work is curriculum design, reviewed teaching, and domain-specific evals. **Goal.** Acquire human-comparable competence across school-level subjects, classical literature, foundational sciences, and the three foundational languages at fluency. **Structure.** The phase has no single exit criterion. Instead, each domain becomes its own sub-phase with its own evals: - **5.1 English fluency** — pack + curriculum sufficient that grammatical-coverage v5 (out-of-distribution registers: legal, poetic, technical, conversational) passes. - **5.2 Hebrew fluency** — analogous, with attention to root-and-pattern morphology. - **5.3 Koine Greek fluency** — analogous, with attention to case and aspect. - **5.4 Elementary mathematics** — number, arithmetic, basic algebra, geometry. Each topic becomes a pack + a domain-specific competence eval. - **5.5 Foundational physics** — kinematics, conservation, basic mechanics. - **5.6 Foundational biology** — taxonomy, cell, system. - **5.7 Classical literature** — reading comprehension at increasing complexity, eventually approaching the John 1:1–2 grounding case as a depth probe. - *(further sub-phases as curriculum expands)* **Discipline during this phase.** - Every new domain ships with its own competence eval following the convention. - The Phase 1–4 lanes are re-run on every release. A new domain that causes regression in a foundational gate is a curriculum bug, not a curriculum success. - Frontier baselines are re-scored periodically; the contrast remains visible. **This phase has no estimated duration.** It is the phase the project lives in after the engineering era ends. Frontier-LLM parity on breadth happens *inside* this phase if it happens at all — likely measured in years, not weeks, and at whatever sample efficiency Phase 4 demonstrated. ### Phase 6 — Evidence-Governed Domain Layer **Entry criteria:** Phase 5 corpus flywheel operational (curriculum + miner sourcing actively producing reviewed proposals); pack-layer chain (ADR-0027..0045) closed; forward-graph-constraint and surface-composer chains (ADR-0046..0089) shipping. **Goal.** Distinguish *contract-passing* from *demonstrated* at the ledger surface. A pack that satisfies the nine ADR-0091 predicates earns a `reasoning-capable` ledger row; promotion to `audit_passed=true` requires a reviewer-signed evidence-bundle digest that reproduces byte-for-byte from on-disk lane results. **Substrate (all accepted as of 2026-05-22):** | Layer | What it ratifies | ADR | |---|---|---| | Domain Pack Contract v1 | Nine predicate checks on every ratified pack. | 0091 | | Reviewer Registry v1 | Schema-validated reviewer roster; primary vs. domain-scoped. | 0092 | | Domain Contract v1 implementation | Validator + ledger enforcement at runtime. | 0093 | | Proposal source provenance | Discriminated `ProposalSource(kind=...)` for every teaching proposal. | 0094 | | Miner-sourced proposals | `teaching/from_miner.py`; SHA-pinned `miner_loop_closure` lane. | 0095 | | Fabrication-control eval lane | Phantom / cross-pack / sibling-collapse refusals must clean. | 0096 | | Demo composition contract | Demos compose from shipped modules; no parallel mechanism. | 0098 | | Public showcase demo | Deterministic, byte-equal, <30s. | 0099 | | Reasoning-capable ratifications | `mathematics_logic` (0097), `physics` (0100), `systems_software` (0101), `hebrew_greek_textual_reasoning` (0102). | | Fluency lane attachment | Hebrew + Koine Greek lanes attached to ADR-0102 packs with dev/public/holdout. | 0103 | | Curriculum-sourced proposals | `teaching/from_curriculum.py`; SHA-pinned `curriculum_loop_closure` lane. | 0104 | | Sealed-holdout encryption | age-based; dev-mode plaintext fallback preserved. | 0105 | | **Expert-demo promotion contract** | **Domain-aware, reviewer-signed, replay-deterministic gate.** | **0106** | | Lane-shape registry | 8 lane ids dispatch to 5 shapes; unknown lanes fail-closed. | 0109 (amends 0106) | **Worked promotion narrative.** The contract has been demonstrated end-to-end: 1. **ADR-0107** — first promotion attempt (`mathematics_logic`) honestly refused by the contract on two named blockers (metric-shape uniformity assumption; `inference_closure` substantively failing at 40% pass). 2. **ADR-0109** — threshold rules amended with explicit lane-shape registry; cognition-shape thresholds preserved bit-identical; four new shapes added (`accuracy_shape`, `inference_shape`, `refusal_shape`, `symbolic_logic_shape`); unknown lanes fail-closed. 3. PR #117 fixed the intent-classifier regression that had broken `inference_closure`. 4. **ADR-0110** — `mathematics_logic` promoted to `audit_passed=true` under the amended contract. Signed claim digest reproduces from on-disk lane results; first domain at audit-passed. 5. **ADR-0111** — `physics` promoted to `audit_passed=true` without any contract change. Shares `inference_closure` + `fabrication_control` results with math; the digest distinguishes via `domain_id` + `evidence_revision`. Retires the "math-only" objection; confirms the bridges ADR-0110 landed were correctly scoped. **Exit criteria (cumulative; each can land independently of the others):** - ☑ Contract definition (ADR-0091..0093) and reviewer trust root (ADR-0092) accepted. - ☑ Negative-control fabrication lane (ADR-0096) ratified and SHA-pinned. - ☑ Four reasoning-capable domain ratifications (ADR-0097/0100/0101/0102). - ☑ Expert-demo promotion contract accepted (ADR-0106 + ADR-0109 amendment). - ☑ First worked promotion lands (ADR-0110 — `mathematics_logic`). - ☑ Second worked promotion lands (ADR-0111 — `physics`; no contract change). - ☐ Third worked promotion lands (`systems_software` or `hebrew_greek_textual_reasoning`, both eligible under the now-amended contract). - ☐ Multi-reviewer signing (currently single-recipient; the open candidate frontier item from ADR-0105). **Why this is its own phase.** Phases 1–5 measure capability *internally*. Phase 6 is the first layer that measures *what the system has actually demonstrated to an external reader* and forces the ledger to distinguish the two. Every prior phase makes claims about the substrate; Phase 6 makes the substrate's claims auditable. **This phase has no estimated duration.** Each new domain promotion is a discrete unit of work; the contract is the durable artifact. Audit-passed domains as of 2026-05-22: `mathematics_logic`, `physics`, `systems_software`. --- ### Phase 7 — GSM8K-Math Expert Capability (ADR-0114 arc; substrate complete) **Status:** Substrate complete as of 2026-05-23. ADR-0120 (first `expert` promotion contract) is the next gate. **Roadmap ADRs:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0118, ADR-0118a, ADR-0119 + sub-phases, ADR-0120 (proposed) This phase was introduced by ADR-0114 as a 7-phase arc toward a first `expert` ledger tier claim. ADR-0114a defines 10 falsifiable anti-overfitting obligations any expert promotion must satisfy. ADR-0119 decomposes the GSM8K eval lane (Phases 1–4 outcome) into 8 sub-phases (5.1 through 5.8), all of which have now landed. **What "substrate complete" means.** All machinery needed to score CORE against GSM8K is in place. The first honest measurement is on the books. ADR-0120 sets the numeric expert threshold and signs the first expert_claims entry. #### gsm8k_math substrate — what has landed | Sub-phase | ADR | Deliverable | |---|---|---| | 5.1 | ADR-0119.1 | `fabrication_control` holdout sealed; key-management pattern established | | 5.2 | ADR-0119.2 | 200 CORE-original cases (50 dev + 150 public); verify.py 200/200 | | 5.3 | ADR-0119.3 | `evals/gsm8k_math/runner.py`; correct/wrong/refused triple; wrong==0 gate | | 5.4 | ADR-0119.4 | Frontier-baseline comparison filed (Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%) | | 5.5 | ADR-0119.5 | Adversarial suite: 38 cases × 12 families; 0 wrong | | 5.6 | ADR-0119.6 | Depth-curve harness; flat at 1.0 across depths 1–8 on public | | 5.7 | ADR-0119.7 | Real GSM8K test set age-encrypted (1,319 cases); first honest measurement | | 5.8 | ADR-0119.8 | `gsm8k_capability_shape` registered in `LANE_SHAPE_REGISTRY` | #### First honest CORE-vs-real-GSM8K measurement | Split | Correct | Wrong | Refused | |---|---|---|---| | Real GSM8K test (sealed, 1,319 cases) | 0 | **0** | 1,319 | | CORE-original public (150 cases) | 150 | **0** | 0 | **0 wrong against an external benchmark** is the load-bearing claim. CORE's grammar covers zero GSM8K test problems today — that is the honest gap. The load-bearing differentiator (zero confabulation) survives the move from CORE-original to real external benchmark. #### ADR-0114a obligations: 10 of 10 discharged for gsm8k_math ADR-0114a defined 10 falsifiable proof obligations any `expert` promotion must satisfy. All 10 are now discharged on main for the gsm8k_math lane. | # | Obligation | Discharged by | |---|---|---| | 1 | Sealed-holdout discipline | ADR-0119.1 (fab_control) + ADR-0119.7 (GSM8K test) | | 2 | OOD surface variation ≥ 0.95 of dev | ADR-0118a (OOD surface generator) | | 3 | Every correct answer ships replay-equal trace | ADR-0117 (solution trace verifier) | | 4 | Typed refusal; wrong == 0 | ADR-0116 + ADR-0119.3; holds against real GSM8K (ADR-0119.7) | | 5 | Reasoning-isolation perturbation suite | ADR-0125 | | 6 | Compositional-depth curve published | ADR-0119.6 (harness); ε threshold to ADR-0120 | | 7 | Frontier-baseline comparison on identical items | ADR-0119.4 | | 8 | Adversarial generation; misparse rate zero | ADR-0119.5 | | 9 | Determinism across release boundaries | solver + verifier + realizer + runner (inherited) | | 10 | Operation provenance via pack | ADR-0116 + en_arithmetic_v1 pack | #### What ADR-0120 will compose ADR-0120 (first `expert` promotion contract) is the single remaining gate. It will: 1. Set the numeric expert threshold (minimum correct_rate; candidate: ≥ 0.40 / ≥ 0.60 / ≥ 0.85 per ADR-0114) 2. Set the depth-curve ε (per-step error tolerance for the flat-curve requirement) 3. Require all 10 ADR-0114a obligations as hard gates 4. Define the `expert_claims` schema (mirrors `audit_passed_claims` from ADR-0106) 5. Sign the first expert_claims entry — or refuse honestly if the correct_rate gate is not met Per the gate-as-process pattern established by ADR-0107, the first promotion attempt (ADR-0121) will likely defer honestly on the correct_rate gate, since CORE's current grammar covers 0/1319 real GSM8K test problems. That is the expected and correct result; it makes the gap measurable and actionable. --- ## Part III — Cross-Cutting Considerations ### Versioning of the framework itself This roadmap is `v1`. As phases complete, the framework may itself need amendment — new lanes added, methodology refined. Treat the roadmap with the same discipline as the evals: version it, never silently rewrite it. Each amendment is dated and explained. ### Scope decisions deferred Two scope decisions named in the 2026-05-15 session remain open and will be pinned before they cause drift: - **Agency** — responsive vs. goal-directed. Defaulting to *responsive* for Phases 0–4. Phase 5 may revisit. - **Embodiment** — symbolic-only vs. sensorimotor. ADR-0013 establishes the sensorium protocol; this roadmap does not assume sensorium-dependent gates in Phases 0–4. Phase 5 may add sensorium-dependent sub-phases. Two further questions emerged during Phase planning that should be decided early: - **Tool use.** Is the pipeline extensible to typed deterministic operators (calculator, search, code execution)? Decision needed before Phase 3, since multi-step-reasoning may benefit from operator delegation. - **Code generation.** Is code a first-class proposition-graph articulation target? Decision needed before Phase 5 if computer-science is a curriculum domain. ### What this roadmap is not - Not a list of features. The features fall out of the gates. - Not a competitive roadmap against frontier LLMs. The contrast is a side effect, not a target. - Not a commitment to dates. The duration estimates are calibration aids, not deliverable schedules. - Not a substitute for the work-sequencing list in `CLAUDE.md`. That list governs daily work; this document governs the arc. ### Failure modes to watch for - **Vibes-based progress.** "It feels smarter" is not a gate. - **Demo-driven development.** Crafting a single impressive interaction is not progress; passing a sealed holdout is. - **Teaching-set leakage.** If the same content appears in pack, teaching, and eval, scores are uninterpretable. - **Frontier envy.** Trying to match frontier LLMs on lanes where they structurally win (e.g., long-tail stylistic breadth) compromises the lanes where CORE structurally wins. - **Lane proliferation.** Adding lanes is cheap; maintaining honest holdouts is expensive. Resist new lanes unless they probe a distinct capability. --- ## Part IV — Immediate Next Actions 1. Decide whether this roadmap is promoted to an ADR (likely `ADR-0016`). 2. Stub `docs/eval_methodology.md` as the extracted Part I (it's the contract every lane inherits). 3. Begin Phase 0: implement `evals/` convention, retrofit `core eval cognition` into it. 4. Pin the *agency* scope decision in writing before Phase 3 begins. 5. Pin the *tool use* scope decision in writing before Phase 3 begins. Phase 0 starts when the human reviewer signs off on this roadmap. The first measurable signal of progress is the `core eval cognition` retrofit landing under the new convention.