core/docs/capability_roadmap.md

# Capability Roadmap — Phased Plan to the Verifiable Competence Gates

**Status:** Draft, derived from `docs/sessions/SESSION-2026-05-15-capability-gates.md`
**Owner:** Joshua Shay
**Last updated:** 2026-05-15

This document walks CORE from its present state through the gating framework defined in the 2026-05-15 session. It is organized into six phases. Each phase has entry criteria, work items, exit criteria, and a benchmark discipline contract.

The benchmark discipline is the spine of the plan. Without it, the phases become aspirational. With it, "are we there yet" becomes a CLI question.

---

## Part I — Benchmark Discipline (read first)

The gates are only meaningful if the evals that prove them are honest. Five rules govern every eval lane in this roadmap. They apply uniformly; no exceptions per phase.

### Rule 1 — Three-set split per lane

Every lane maintains three disjoint corpora:

- **Dev set.** Freely visible during development. Used to iterate.
- **Public test set.** Visible, but tuning against it is forbidden. Scored at version-cut time only. Drift in dev-vs-public scores is a red flag for overfitting.
- **Private holdout.** Sealed. Never read by Claude, never committed in plaintext, only scored by a clean-room runner at release events. Stored encrypted in `evals/holdouts/` with key held by the human reviewer.

If a lane has only a dev set, it does not count as a gate. It is exploration.

### Rule 2 — Versioned difficulty escalation

Each lane has versions: `v1`, `v2`, `v3`, … with monotonically harder distributions. Passing a version is not a terminal state; it is a checkpoint that unlocks generating the next version.

- **v1** — baseline competence demonstration. The construction is shown clearly.
- **v2** — distributional shift: longer chains, deeper nesting, rarer vocabulary, paraphrased surface forms.
- **v3** — adversarial: items generated specifically by inspecting model failures on v2.
- **v4+** — out-of-distribution: items drawn from domains, registers, or constructions not present at training time.

Score is always reported as a tuple `(v1_score, v2_score, v3_score, …)`, never collapsed to a single number. A model that scores 99% on v1 and 12% on v3 is not a "99% model."

### Rule 3 — Adversarial regeneration on pass

When a model passes a version (e.g., ≥95% on the public test set with ≥90% on private holdout), the next version is *generated by adversarial process*:

- Human review finds construction families the model handled accidentally rather than structurally.
- A separate generator (could be a different model, could be programmatic) produces items targeting the weakest decile of the previous version.
- The new version is reviewed for legitimacy — no impossible items, no ambiguous items, no items that depend on world knowledge the system was never given.

This is the protection against silent overfitting: every passed version triggers the construction of a harder one, so "progress" requires continuously rising scores against continuously harder tests.

### Rule 4 — Frontier baseline tracking

For each lane, a baseline score is computed for at least one frontier transformer-based model (e.g., Claude Opus 4.7, GPT-5, Gemini 3 Ultra) on the *same* public test set. Baselines are:

- Re-scored every time a version is cut.
- Published alongside CORE's score.
- Never tuned, never prompted-engineered to maximize — the prompt is the eval task as written.

This serves two purposes: (a) it makes CORE's structural wins visible (frontier models score near zero on provenance, monotonic learning, etc.); (b) it prevents self-congratulation on lanes where CORE merely matches an LLM that was given no advantage.

### Rule 5 — Honest reporting

- Failures are reported with the same prominence as passes.
- Confidence intervals on every score (bootstrapped over the test set).
- Per-construction breakdowns published — never a single aggregate hiding structural failures.
- Regressions across versions are surfaced, never silently dropped.
- "Did not test" is a valid result; "tested and failed" is preferred over "did not test."

If a number cannot be reported honestly under these rules, the lane is not ready. Do not ship the lane.

### Eval contract template

Every eval lane lives in `evals/<lane_name>/` with this layout:

```
evals/<lane_name>/
  contract.md           # what the lane measures, scoring rubric, pass thresholds
  dev/                  # dev set, freely visible
  public/v1/            # public test set, version 1
  public/v2/            # public test set, version 2
  ...
  holdouts/             # encrypted, sealed
  runner.py             # deterministic scorer
  baselines/            # frontier model scores per version
  results/              # CORE scores per version per release
```

A lane without a `contract.md` does not run.

---

## Part II — The Phases

### Phase 0 — Benchmark methodology lock-in

**Entry criteria.** Today.

**Goal.** Build the discipline infrastructure before building any new eval. Doing this first prevents the entire roadmap from drifting into vibes-based progress.

**Work items.**

1. Implement `evals/` layout convention above.
2. Implement `core eval <lane>` CLI subcommand that loads contracts, runs the runner, writes results.
3. Implement the holdout-runner: a sandboxed process that decrypts the sealed test set, scores, writes only the aggregate score (never item-level results) back to the working tree.
4. Implement baseline-runner: a thin adapter that queries a frontier model on the public test set and records its score.
5. Write the methodology page in `docs/eval_methodology.md` (this Part I, extracted).
6. Pick one *existing* eval (the current `core eval cognition`) and retrofit it into the new convention as a forcing function. Versions become explicit; holdout is split out; results are reported per-version.

**Exit criteria.**

- `core eval cognition` runs under the new convention, with v1 public + private holdout + baseline.
- No new lane is allowed to be merged that does not follow the convention.
- The retrofit revealed at least one item-level methodology issue (silent ambiguity, leaked dev item, unstated assumption) — caught and documented. If the retrofit found nothing, the audit was not real.

**Duration estimate.** 1–2 weeks of focused work.

---

### Phase 1 — Foundational Triple

**Entry criteria.** Phase 0 exit complete.

**Goal.** Implement and pass the three gates that determine whether CORE is ready to move from engineering into curriculum:

- **grammatical-coverage** (fluency)
- **zero-code-domain-acquisition** (engineering-vs-learning phase shift)
- **identity-divergence** (identity is load-bearing)

**Work items.**

**1.1 grammatical-coverage**

- Enumerate target grammatical constructions for English v1: simple declarative, negation, conjunction, disjunction, embedded clause, relative clause, quantification (universal/existential), basic tense (past/present/future), basic aspect (perfective/imperfective).
- For each construction, write contract test pairs: `PropositionGraph → expected surface family`. "Expected surface family" is a set of acceptable surfaces, not a single string, with a deterministic acceptance predicate.
- Implement v1 dev/public/holdout (target: ~50/50/50 items).
- Engineer `realizer.py` to pass v1.
- Once v1 ≥95% public and ≥90% holdout, generate v2 (deeper nesting, rare vocabulary substitution, longer sentences).
- Repeat for Hebrew and Koine Greek using their respective pack morphology.

**1.2 zero-code-domain-acquisition**

- Define three "surprise domains" never touched in development: pick from {kinship relations, basic arithmetic, simple spatial relations, color taxonomy, calendar relations}.
- Each domain has a pack-only authoring kit: vocabulary, relation predicates, axiom list, ~20 reviewed teaching examples, ~30 articulation prompts.
- Test: an author who knows the system but is forbidden from editing Python attempts to bring CORE to ≥80% articulation accuracy on the prompts using only pack + teaching loop.
- Each Python edit required is a logged "engineering gap" that goes onto the closing list.

**1.3 identity-divergence**

- Define two identity axis sets, deliberately oriented to produce different stances on the same proposition (e.g., axis-A weights novelty highly, axis-B weights tradition highly; or axis-A is precision-first, axis-B is generosity-first).
- Curate a shared curriculum: ~100 reviewed teaching events, identical for both agents.
- Curate a prompt set where identity should produce measurably different articulations.
- Scoring: an automated divergence metric (e.g., proposition-graph difference) plus a coherence metric (each output must be internally consistent with its own axes).
- Pass: divergence above floor, coherence above floor, *both required*.
- Also: identity-stripped baseline. The same curriculum with identity disabled should produce articulations whose divergence is at noise floor — proving identity is doing the work.

**Exit criteria.**

- All three lanes pass v1 on public + holdout.
- The engineering-gap list from 1.2 is either empty or has a documented closing plan.
- v2 generation has been attempted for at least one of the three.

**Duration estimate.** 4–8 weeks. The realizer work in 1.1 is the bottleneck and may expose deeper engineering gaps.

---

### Phase 2 — Structural Wins Made Visible

**Entry criteria.** Phase 1 exit complete.

**Goal.** Build the lanes where CORE's architecture wins by design. These should pass relatively early *for CORE* and fail catastrophically *for frontier baselines*. The purpose is to publish the contrast.

**Lanes:**

- **provenance** — every articulated claim back-points to vault entries / teaching events / pack axioms; replay reproduces trace bit-for-bit.
- **monotonic-learning** — after N teaching cycles across unrelated domains, competence on domain 1 does not regress. Longitudinal: ≥10 teaching cycles, scored at each step.
- **calibration** — out-of-pack queries produce typed "no grounding" responses; in-pack queries do not. Distinguish "I don't know" / "incoherent" / "contradicts known."
- **symbolic-logic** — nested negation, modal operators (must/may/possible/necessary), counterfactual conditionals. Target ≥99% on v1.
- **adversarial-identity** — 1,000-turn red-team corpus; identity drift below noise floor.

**Work items.**

- Build each lane following the convention.
- Compute frontier baselines for each. Expected outcomes:
  - **provenance** — CORE: pass. Frontier: near-zero (no model can produce verifiable per-claim provenance).
  - **monotonic-learning** — CORE: pass by construction. Frontier: regression visible after fine-tuning rounds (this requires the frontier baseline to actually be fine-tuned, which complicates the comparison — may be reported as "not directly comparable, structural argument applies").
  - **calibration** — CORE: high if calibration is wired; frontier: confabulates on most OOD prompts.
  - **symbolic-logic** — CORE: target ≥99% v1; frontier: ~80% v1, collapses on v3.
  - **adversarial-identity** — CORE: target drift below noise; frontier: persona erodes within ~50–100 turns.
- Publish results page with per-lane comparison.

**Anti-overfitting note.** The structural wins are *structural* — the temptation to declare victory after v1 is large. Discipline: v2 and v3 must still be generated and scored. A "structural win" that fails on v3 is a structural claim that was actually a v1 coincidence.

**Exit criteria.**

- All five lanes have v1 + v2 results published with frontier baselines.
- At least two lanes have v3 results.
- The contrast page is honest about which results are "directly comparable" vs. "structural argument."

**Duration estimate.** 8–12 weeks. provenance and monotonic-learning may require new instrumentation in `vault/` and `teaching/`.

---

### Phase 3 — Reasoning Depth

**Entry criteria.** Phase 2 exit complete. This is the hardest phase. Expect engineering surprises.

**Goal.** Lanes that probe whether CORE actually *thinks* rather than retrieves and articulates.

**Lanes:**

- **compositionality** — novel combinations of taught primitives. SCAN/COGS-style splits adapted to proposition graphs.
- **inference-closure** — derive entailments never directly asserted (transitive, spatial, temporal, causal chains).
- **introspection** — `explain(turn_id)` produces a natural-language account that round-trips: a separate run conditioned on the explanation predicts the same articulation.
- **multi-step-reasoning** — pipeline produces and consumes intermediate proposition-graph states for problems whose solution requires ≥3 inferential hops.
- **cross-domain-transfer** — competence in domain B rises after teaching only in domain A, via shared structural elements.

**Work items.**

These will almost certainly expose engineering gaps. Expected gaps:

- `generate/graph_planner.py` may need an intermediate-state stack rather than a single planning pass (multi-step-reasoning).
- `field/propagate.py` may need to expose derivable-but-not-asserted recall paths (inference-closure).
- A new `cognition/explain.py` module may be needed for introspection.
- Cross-domain transfer may require examining how proposition graphs share structural sub-units, which may be a pack-design question more than a code question.

For each gap discovered, the work splits: (a) write the eval, (b) confirm it fails, (c) close the engineering gap, (d) re-run.

**Anti-overfitting note.** Compositionality is *the* lane most vulnerable to overfitting. The training-test split must be done by *construction family*, not by sampling. If the model has seen `R(A,B)` and `R(C,D)`, the test set must use a *novel relation R'* applied to seen entities — not a fresh `(A,B)` pair under a seen `R`.

**Exit criteria.**

- All five lanes have v1 results with honest scores (which may be failing — that's acceptable for v1).
- Each failure has either a closed engineering gap or a documented architectural deferral.
- At least two lanes are passing v1 by phase exit.

**Duration estimate.** 12–24 weeks. This is the phase that decides whether CORE's design lives up to its philosophical claims.

---

### Phase 4 — Scale and Efficiency

**Entry criteria.** Phase 3 exit complete. Phases 1–3 are pass/fail; this phase is *quantitative curves*.

**Goal.** Make CORE's quantitative behavior visible: how fast does it learn, how does cost scale, how does it compose at scale.

**Lanes:**

- **sample-efficiency** — corrections-to-competence curves across ten unrelated concepts. Plot, do not threshold.
- **long-context-cost** — vault size vs. per-turn latency curve at 10³, 10⁴, 10⁵, 10⁶ entries. Identify the asymptotic complexity. Decide indexing strategy if super-linear.
- **multi-agent-composition** — two CORE instances with different identities cooperate on a shared task; each maintains its own deterministic replay. Measure: task completion, replay determinism preserved per agent, no identity bleed.

**Work items.**

- Build infrastructure for longitudinal measurement (Phase 2's monotonic-learning runner is a starting point).
- Sample-efficiency requires running the teaching loop programmatically with controlled correction budgets.
- Long-context-cost may surface that the current `vault/store.py` is insufficient at scale — the response is exact indexing (B-tree, suffix array, signature-based bucketing), not approximate recall.
- Multi-agent composition surfaces orchestration questions that may justify a new module (`society/` or similar) — defer unless the eval forces it.

**Anti-overfitting note.** Curves don't overfit the way thresholds do, but they can be selectively reported. Discipline: publish the full curve, not just the best operating point. Confidence intervals at each data point.

**Exit criteria.**

- Sample-efficiency curves published for ≥10 concepts.
- Vault cost curve published with asymptotic analysis. Indexing strategy decided.
- Multi-agent composition demonstrated for ≥2-agent cooperation with replay preserved.

**Duration estimate.** 8–16 weeks.

---

### Phase 5 — Curriculum Era

**Entry criteria.** Phase 4 exit complete. From this point forward, engineering changes are exceptional, not routine. The work is curriculum design, reviewed teaching, and domain-specific evals.

**Goal.** Acquire human-comparable competence across school-level subjects, classical literature, foundational sciences, and the three foundational languages at fluency.

**Structure.**

The phase has no single exit criterion. Instead, each domain becomes its own sub-phase with its own evals:

- **5.1 English fluency** — pack + curriculum sufficient that grammatical-coverage v5 (out-of-distribution registers: legal, poetic, technical, conversational) passes.
- **5.2 Hebrew fluency** — analogous, with attention to root-and-pattern morphology.
- **5.3 Koine Greek fluency** — analogous, with attention to case and aspect.
- **5.4 Elementary mathematics** — number, arithmetic, basic algebra, geometry. Each topic becomes a pack + a domain-specific competence eval.
- **5.5 Foundational physics** — kinematics, conservation, basic mechanics.
- **5.6 Foundational biology** — taxonomy, cell, system.
- **5.7 Classical literature** — reading comprehension at increasing complexity, eventually approaching the John 1:1–2 grounding case as a depth probe.
- *(further sub-phases as curriculum expands)*

**Discipline during this phase.**

- Every new domain ships with its own competence eval following the convention.
- The Phase 1–4 lanes are re-run on every release. A new domain that causes regression in a foundational gate is a curriculum bug, not a curriculum success.
- Frontier baselines are re-scored periodically; the contrast remains visible.

**This phase has no estimated duration.** It is the phase the project lives in after the engineering era ends. Frontier-LLM parity on breadth happens *inside* this phase if it happens at all — likely measured in years, not weeks, and at whatever sample efficiency Phase 4 demonstrated.

---

## Part III — Cross-Cutting Considerations

### Versioning of the framework itself

This roadmap is `v1`. As phases complete, the framework may itself need amendment — new lanes added, methodology refined. Treat the roadmap with the same discipline as the evals: version it, never silently rewrite it. Each amendment is dated and explained.

### Scope decisions deferred

Two scope decisions named in the 2026-05-15 session remain open and will be pinned before they cause drift:

- **Agency** — responsive vs. goal-directed. Defaulting to *responsive* for Phases 0–4. Phase 5 may revisit.
- **Embodiment** — symbolic-only vs. sensorimotor. ADR-0013 establishes the sensorium protocol; this roadmap does not assume sensorium-dependent gates in Phases 0–4. Phase 5 may add sensorium-dependent sub-phases.

Two further questions emerged during Phase planning that should be decided early:

- **Tool use.** Is the pipeline extensible to typed deterministic operators (calculator, search, code execution)? Decision needed before Phase 3, since multi-step-reasoning may benefit from operator delegation.
- **Code generation.** Is code a first-class proposition-graph articulation target? Decision needed before Phase 5 if computer-science is a curriculum domain.

### What this roadmap is not

- Not a list of features. The features fall out of the gates.
- Not a competitive roadmap against frontier LLMs. The contrast is a side effect, not a target.
- Not a commitment to dates. The duration estimates are calibration aids, not deliverable schedules.
- Not a substitute for the work-sequencing list in `CLAUDE.md`. That list governs daily work; this document governs the arc.

### Failure modes to watch for

- **Vibes-based progress.** "It feels smarter" is not a gate.
- **Demo-driven development.** Crafting a single impressive interaction is not progress; passing a sealed holdout is.
- **Teaching-set leakage.** If the same content appears in pack, teaching, and eval, scores are uninterpretable.
- **Frontier envy.** Trying to match frontier LLMs on lanes where they structurally win (e.g., long-tail stylistic breadth) compromises the lanes where CORE structurally wins.
- **Lane proliferation.** Adding lanes is cheap; maintaining honest holdouts is expensive. Resist new lanes unless they probe a distinct capability.

---

## Part IV — Immediate Next Actions

1. Decide whether this roadmap is promoted to an ADR (likely `ADR-0016`).
2. Stub `docs/eval_methodology.md` as the extracted Part I (it's the contract every lane inherits).
3. Begin Phase 0: implement `evals/` convention, retrofit `core eval cognition` into it.
4. Pin the *agency* scope decision in writing before Phase 3 begins.
5. Pin the *tool use* scope decision in writing before Phase 3 begins.

Phase 0 starts when the human reviewer signs off on this roadmap. The first measurable signal of progress is the `core eval cognition` retrofit landing under the new convention.