core/docs/decisions/ADR-0114-expert-capability-roadmap-gsm8k-first.md

# ADR-0114 — Expert-Capability Roadmap: GSM8K-Math First

**Status:** Proposed
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Depends on:** ADR-0091, ADR-0106, ADR-0109, ADR-0110, ADR-0111, ADR-0112, ADR-0113
**Supersedes:** none (greenfield)

---

## Context

ADR-0113 reserved the `expert` namespace for a future ledger tier
above `audit-passed`. `audit-passed` verifies CORE *claim-shape
compliance* (signed digest, replay determinism, typed refusal, exact
recall) — a real, transformer-unreachable property. It does **not**
verify raw task performance against external benchmarks.

This ADR proposes the first concrete path toward an `expert` ledger
status. It is greenfield: no engine for it exists today.

### Honest framing of distance

The current architecture is impressive at the substrate level
(determinism, exact CGA recall, reviewed teaching, audit trail). It
is **far from expert-level on generative tasks**:

- The deterministic realizer composes pack lemmas via fixed templates
  and reviewed teaching chains. There is no symbolic solver, no proof
  search, no multi-step problem decomposition.
- `en_mathematics_logic_v1` pack has 16 lemmas. Solving an
  AMC-10 problem requires a working pipeline that does not exist.
- "Expert prose" is fuzzily defined and would need ~30k+ lemma
  lexicon plus discourse-level planning beyond current capability.

The honest current state is: **no domain is at expert-level capability
by any external measure**. The `audit-passed` rows for math and
physics describe CORE-specific claim shapes, not raw-task supremacy.

This ADR proposes building toward that — starting with the lowest-
hanging credible target: **grade-school math word problems on GSM8K**.

### Why GSM8K first

- **Public benchmark with established frontier baselines.** Frontier
  LLMs report 92-95% on GSM8K. Smaller open models report 30-70%. The
  comparison space is well-mapped.
- **Checkable answers.** Each problem has a single integer answer; no
  fuzzy grading required.
- **Smallest architectural delta.** The math pack already names the
  right operators (`adds`, `subtracts`, `multiplies`, `divides`); the
  realizer already composes deterministic surfaces. The missing piece
  is a real solver loop, not a fundamentally different substrate.
- **Honest first step.** If CORE scores at 6th-grader level on GSM8K,
  the result is *still load-bearing* — because the score would be
  replayable, traceable, and grounded in a way frontier LLM scores
  structurally cannot be. We can claim "Nth-percentile GSM8K, with
  trace-verifiable solutions" — a claim no LLM can make.
- **Compositional fit.** GSM8K's reasoning depth (typically 2-8 steps)
  fits CORE's proposition-graph + teaching-chain substrate without
  requiring an unbounded search.

---

## Decision

Establish a phased path to a first `expert` ledger tier claim, scoped
to **`mathematics_logic`** as the first domain and **GSM8K** as the
first benchmark. Each phase is its own ADR; this ADR proposes the
sequence and defines exit criteria.

### Phase 1 — Problem Parser (ADR-0115, future)

Build `generate/math_parser.py`: turns a natural-language word problem
into a typed proposition graph.

- Input: GSM8K problem string.
- Output: a `MathProblemGraph` with typed nodes: `entity`, `quantity`,
  `unit`, `operation`, `unknown`, `relation`.
- Constraint: deterministic. Same input → byte-identical graph.
- No solver yet — just structured parse.

Exit criterion: on a curated dev set of 50 GSM8K-style problems
(authored, not from GSM8K to avoid contamination), parse correctness
≥ 0.90 measured by human review against a published rubric.

### Phase 2 — Deterministic Solver (ADR-0116, future)

Build `generate/math_solver.py`: a tiny term-rewriting system over the
`MathProblemGraph` using the existing math-pack operator vocabulary.

- Input: `MathProblemGraph`.
- Output: a `SolutionTrace` — ordered list of operation applications
  ending at a numeric answer (or a typed refusal if the graph is
  under-determined).
- Constraint: pure function. No sampling. Trace is byte-deterministic
  from input.

Exit criterion: on the Phase 1 curated dev set, solver yields correct
final answer on ≥ 80% of graphs the parser produces correctly. The
solver does **not** need to solve every problem; it needs to be honest
about which it can.

### Phase 3 — Verifier (ADR-0117, future)

The verifier re-derives the answer from the `SolutionTrace` and emits
a typed verdict. This is the easy phase — CORE's substrate already
has replayability. The verifier just enforces it on this new artifact.

Exit criterion: replay determinism = 1.0 on all Phase 2 outputs.

### Phase 4 — Stepped-Realizer Extension (ADR-0118, future)

Extend the existing realizer to emit show-your-work prose from a
`SolutionTrace` — one sentence per operation, with pack-grounded
operator vocabulary.

Exit criterion: every Phase 2 success produces a stepped explanation
of length proportional to trace length, with each sentence
pack-grounded.

### Phase 5 — GSM8K Eval Lane (ADR-0119, future)

Author `evals/gsm8k/`:

- `dev/cases.jsonl` — curated subset of GSM8K train (300 problems).
- `public/v1/cases.jsonl` — curated subset of GSM8K train (1500
  problems; disjoint from dev).
- `holdouts/v1/cases_plaintext.jsonl` — curated subset of GSM8K
  *test* (300 problems; never read during development).
- `runner.py` — drives parser → solver → verifier → realizer and
  scores against the integer answer.
- `contract.md` — lane contract; lane shape `gsm8k_capability_shape`
  (new shape; introduced by ADR-0119 amendment to ADR-0109).

Exit criterion: lane runner produces deterministic results. Honest
first number reported (whatever it is).

### Phase 6 — First `expert` Promotion Contract (ADR-0120, future)

Define the `expert` ledger status:

- `expert=true` iff:
  1. `audit_passed=true` predicates pass (ADR-0106 + ADR-0109)
  2. At least one **capability lane** attached to the domain meets a
     **human-expert-calibrated threshold** declared in the lane's
     `contract.md`
  3. Reviewer-signed `expert_claims` entry whose evidence-bundle
     digest reproduces byte-for-byte (mirrors ADR-0106 §1.5 exactly)
  4. The capability lane's threshold is declared **publicly** in the
     ADR, not buried in a config — so external readers can debate the
     calibration.

For GSM8K specifically, candidate human-expert-calibrated thresholds:

| Threshold | Interpretation |
|---|---|
| ≥ 0.40 | "competent" — beats average 14-year-old human |
| ≥ 0.60 | "advanced" — competitive open-source LLM territory |
| ≥ 0.85 | "expert" — frontier-LLM territory; published-paper bar |

ADR-0120 must pick one and justify the choice. The chosen number is
*falsifiable*: if CORE scores below it, no `expert` row.

### Phase 7 — Second Capability Domain (ADR-0121+, future)

Pick a second capability domain only after Phase 6 lands. Likely
candidates: symbolic logic with quantifiers (closest cousin to math),
or DSL code generation (checkable, lexicon-bounded). Defer until
math expertise is real.

### Writing / open-prose capability — explicitly deferred

Open prose ("expert essay" / "expert article") is **not** on this
roadmap. Reasons:

- "Correct prose" is fuzzily defined; no GSM8K-equivalent benchmark
  exists with sharp scoring.
- Required lexicon (~30k+ working English lemmas) is two orders of
  magnitude larger than the current cognition pack.
- Discourse-level planning beyond current capability would need a
  separate architectural arc.

Writing capability becomes appropriate once at least two symbolic
domains have landed at `expert` and the substrate has been stress-tested
against checkable benchmarks.

---

## Non-Decisions

This ADR explicitly does **not**:

- Commit to a phase timeline. Each phase is its own ADR with its own
  scope. The ADR sequence is the durable artifact; the schedule is not.
- Promise CORE will reach 85% GSM8K. We commit to *honest scoring*,
  not to a target. If the architecture caps out at 35%, we report 35%.
- Pretend the audit-passed gate is a capability claim. The two tiers
  are distinct and remain distinct.
- Re-render existing audit-passed claims as expert claims. Math and
  physics audit-passed promotions stand independently of any future
  expert promotion on the same domain.

---

## Invariants

### `adr_0114_expert_namespace_undefined_until_adr_0120`

No code ships an `expert` ledger status before ADR-0120 lands. The
`_EXPERT_DOMAIN_STATUSES` tuple in `core.capability.reporting` stays
at 5 entries. Tested by the existing reporting tests.

### `adr_0114_gsm8k_is_first_capability_target`

The first capability lane authored under this roadmap is GSM8K
(ADR-0119). No other capability lane lands first.

### `adr_0114_expert_requires_explicit_threshold`

The first `expert` promotion contract (ADR-0120) must declare a
public threshold number for the underlying capability lane. No
hidden calibration.

---

## Acceptance evidence (for this proposed ADR)

ADR-0114 is accepted when:

- The ADR file exists in `docs/decisions/` and is linked from
  `docs/decisions/README.md`
- No code changes — this is a roadmap ADR only
- README updated to point at this sequence as the path-to-expert

No tests need to be added by this ADR. Tests are scoped to each
implementation ADR.

---

## Consequences

- The repo has a public, dated commitment to a first expert-capability
  target. The "what are you actually claiming is expert-level?"
  question now has a written answer: nothing yet; the path is ADR-0115
  through ADR-0120.
- The first `expert` claim, when it lands, will be *falsifiable*: tied
  to a public benchmark with a stated threshold. If CORE underperforms,
  the row stays at `audit-passed`.
- The `audit-passed` status remains the load-bearing CORE-vs-LLM claim
  in the interim. Nothing about the expert roadmap diminishes it; the
  two tiers measure orthogonal properties.

---

## Out of scope

- Implementation of any of Phases 1-7. Each is its own ADR.
- Writing / open-prose roadmap. Deferred until at least one symbolic
  domain lands at `expert`.
- Specific GSM8K threshold choice. ADR-0120's job.
- Alternative first benchmarks (MATH, MMLU-math, AIME). All are
  candidates after GSM8K; none replace it as Phase 1.