core/docs/decisions/ADR-0114-expert-capability-roadmap-gsm8k-first.md

ADR-0114 — Expert-Capability Roadmap: GSM8K-Math First

Status: Proposed Date: 2026-05-22 Author: CORE agents + reviewers Depends on: ADR-0091, ADR-0106, ADR-0109, ADR-0110, ADR-0111, ADR-0112, ADR-0113 Supersedes: none (greenfield)

---

Context

ADR-0113 reserved the expert namespace for a future ledger tier above audit-passed. audit-passed verifies CORE claim-shape compliance (signed digest, replay determinism, typed refusal, exact recall) — a real, transformer-unreachable property. It does not verify raw task performance against external benchmarks.

This ADR proposes the first concrete path toward an expert ledger status. It is greenfield: no engine for it exists today.

Honest framing of distance

The current architecture is impressive at the substrate level (determinism, exact CGA recall, reviewed teaching, audit trail). It is far from expert-level on generative tasks:

• The deterministic realizer composes pack lemmas via fixed templates and reviewed teaching chains. There is no symbolic solver, no proof search, no multi-step problem decomposition.
• en_mathematics_logic_v1 pack has 16 lemmas. Solving an AMC-10 problem requires a working pipeline that does not exist.
• "Expert prose" is fuzzily defined and would need ~30k+ lemma lexicon plus discourse-level planning beyond current capability.

The honest current state is: no domain is at expert-level capability by any external measure. The audit-passed rows for math and physics describe CORE-specific claim shapes, not raw-task supremacy.

This ADR proposes building toward that — starting with the lowest- hanging credible target: grade-school math word problems on GSM8K.

Why GSM8K first

• Public benchmark with established frontier baselines. Frontier LLMs report 92-95% on GSM8K. Smaller open models report 30-70%. The comparison space is well-mapped.
• Checkable answers. Each problem has a single integer answer; no fuzzy grading required.
• Smallest architectural delta. The math pack already names the right operators (adds, subtracts, multiplies, divides); the realizer already composes deterministic surfaces. The missing piece is a real solver loop, not a fundamentally different substrate.
• Honest first step. If CORE scores at 6th-grader level on GSM8K, the result is still load-bearing — because the score would be replayable, traceable, and grounded in a way frontier LLM scores structurally cannot be. We can claim "Nth-percentile GSM8K, with trace-verifiable solutions" — a claim no LLM can make.
• Compositional fit. GSM8K's reasoning depth (typically 2-8 steps) fits CORE's proposition-graph + teaching-chain substrate without requiring an unbounded search.

---

Decision

Establish a phased path to a first expert ledger tier claim, scoped to mathematics_logic as the first domain and GSM8K as the first benchmark. Each phase is its own ADR; this ADR proposes the sequence and defines exit criteria.

Phase 1 — Problem Parser (ADR-0115, future)

Build generate/math_parser.py: turns a natural-language word problem into a typed proposition graph.

• Input: GSM8K problem string.
• Output: a MathProblemGraph with typed nodes: entity, quantity, unit, operation, unknown, relation.
• Constraint: deterministic. Same input → byte-identical graph.
• No solver yet — just structured parse.

Exit criterion: on a curated dev set of 50 GSM8K-style problems (authored, not from GSM8K to avoid contamination), parse correctness ≥ 0.90 measured by human review against a published rubric.

Phase 2 — Deterministic Solver (ADR-0116, future)

Build generate/math_solver.py: a tiny term-rewriting system over the MathProblemGraph using the existing math-pack operator vocabulary.

• Input: MathProblemGraph.
• Output: a SolutionTrace — ordered list of operation applications ending at a numeric answer (or a typed refusal if the graph is under-determined).
• Constraint: pure function. No sampling. Trace is byte-deterministic from input.

Exit criterion: on the Phase 1 curated dev set, solver yields correct final answer on ≥ 80% of graphs the parser produces correctly. The solver does not need to solve every problem; it needs to be honest about which it can.

Phase 3 — Verifier (ADR-0117, future)

The verifier re-derives the answer from the SolutionTrace and emits a typed verdict. This is the easy phase — CORE's substrate already has replayability. The verifier just enforces it on this new artifact.

Exit criterion: replay determinism = 1.0 on all Phase 2 outputs.

Phase 4 — Stepped-Realizer Extension (ADR-0118, future)

Extend the existing realizer to emit show-your-work prose from a SolutionTrace — one sentence per operation, with pack-grounded operator vocabulary.

Exit criterion: every Phase 2 success produces a stepped explanation of length proportional to trace length, with each sentence pack-grounded.

Phase 5 — GSM8K Eval Lane (ADR-0119, future)

Author evals/gsm8k/:

• dev/cases.jsonl — curated subset of GSM8K train (300 problems).
• public/v1/cases.jsonl — curated subset of GSM8K train (1500 problems; disjoint from dev).
• holdouts/v1/cases_plaintext.jsonl — curated subset of GSM8K test (300 problems; never read during development).
• runner.py — drives parser → solver → verifier → realizer and scores against the integer answer.
• contract.md — lane contract; lane shape gsm8k_capability_shape (new shape; introduced by ADR-0119 amendment to ADR-0109).

Exit criterion: lane runner produces deterministic results. Honest first number reported (whatever it is).

Phase 6 — First expert Promotion Contract (ADR-0120, future)

Define the expert ledger status:

• expert=true iff:
  1. audit_passed=true predicates pass (ADR-0106 + ADR-0109)
  2. At least one capability lane attached to the domain meets a human-expert-calibrated threshold declared in the lane's contract.md
  3. Reviewer-signed expert_claims entry whose evidence-bundle digest reproduces byte-for-byte (mirrors ADR-0106 §1.5 exactly)
  4. The capability lane's threshold is declared publicly in the ADR, not buried in a config — so external readers can debate the calibration.

For GSM8K specifically, candidate human-expert-calibrated thresholds:

Threshold	Interpretation
≥ 0.40	"competent" — beats average 14-year-old human
≥ 0.60	"advanced" — competitive open-source LLM territory
≥ 0.85	"expert" — frontier-LLM territory; published-paper bar

ADR-0120 must pick one and justify the choice. The chosen number is falsifiable: if CORE scores below it, no expert row.

Phase 7 — Second Capability Domain (ADR-0121+, future)

Pick a second capability domain only after Phase 6 lands. Likely candidates: symbolic logic with quantifiers (closest cousin to math), or DSL code generation (checkable, lexicon-bounded). Defer until math expertise is real.

Writing / open-prose capability — explicitly deferred

Open prose ("expert essay" / "expert article") is not on this roadmap. Reasons:

• "Correct prose" is fuzzily defined; no GSM8K-equivalent benchmark exists with sharp scoring.
• Required lexicon (~30k+ working English lemmas) is two orders of magnitude larger than the current cognition pack.
• Discourse-level planning beyond current capability would need a separate architectural arc.

Writing capability becomes appropriate once at least two symbolic domains have landed at expert and the substrate has been stress-tested against checkable benchmarks.

---

Non-Decisions

This ADR explicitly does not:

• Commit to a phase timeline. Each phase is its own ADR with its own scope. The ADR sequence is the durable artifact; the schedule is not.
• Promise CORE will reach 85% GSM8K. We commit to honest scoring, not to a target. If the architecture caps out at 35%, we report 35%.
• Pretend the audit-passed gate is a capability claim. The two tiers are distinct and remain distinct.
• Re-render existing audit-passed claims as expert claims. Math and physics audit-passed promotions stand independently of any future expert promotion on the same domain.

---

Invariants

adr_0114_expert_namespace_undefined_until_adr_0120

No code ships an expert ledger status before ADR-0120 lands. The _EXPERT_DOMAIN_STATUSES tuple in core.capability.reporting stays at 5 entries. Tested by the existing reporting tests.

adr_0114_gsm8k_is_first_capability_target

The first capability lane authored under this roadmap is GSM8K (ADR-0119). No other capability lane lands first.

adr_0114_expert_requires_explicit_threshold

The first expert promotion contract (ADR-0120) must declare a public threshold number for the underlying capability lane. No hidden calibration.

---

Acceptance evidence (for this proposed ADR)

ADR-0114 is accepted when:

• The ADR file exists in docs/decisions/ and is linked from docs/decisions/README.md
• No code changes — this is a roadmap ADR only
• README updated to point at this sequence as the path-to-expert

No tests need to be added by this ADR. Tests are scoped to each implementation ADR.

---

Consequences

• The repo has a public, dated commitment to a first expert-capability target. The "what are you actually claiming is expert-level?" question now has a written answer: nothing yet; the path is ADR-0115 through ADR-0120.
• The first expert claim, when it lands, will be falsifiable: tied to a public benchmark with a stated threshold. If CORE underperforms, the row stays at audit-passed.
• The audit-passed status remains the load-bearing CORE-vs-LLM claim in the interim. Nothing about the expert roadmap diminishes it; the two tiers measure orthogonal properties.

---

Out of scope

• Implementation of any of Phases 1-7. Each is its own ADR.
• Writing / open-prose roadmap. Deferred until at least one symbolic domain lands at expert.
• Specific GSM8K threshold choice. ADR-0120's job.
• Alternative first benchmarks (MATH, MMLU-math, AIME). All are candidates after GSM8K; none replace it as Phase 1.