core/docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md

ADR-0131.G — GSM8K Coverage Probe: Honest Measurement Under the Safety Rail

Status: Proposed Date: 2026-05-23 Author: CORE agents + reviewers Parent: ADR-0131 Depends on: ADR-0115, ADR-0116, ADR-0117, ADR-0131.3, ADR-0132..ADR-0135

---

Context

ADR-0131 re-targeted the math-expert promotion away from GSM8K because GSM8K rewards paraphrase flexibility, which is the deterministic engine's structural weakness. The composite architecture-aligned gate (B1 symbolic equivalence, B2 teaching corpus, B3 bounded-grammar word problems) is the true gate.

GSM8K does not vanish from the project; it returns as a coverage probe of the bounded grammar + binding-graph layer. The framing this ADR pins:

GSM8K is not a promotion gate. It is a coverage measurement of what the bounded grammar admits. Of the admitted subset, the engine solves deterministically with wrong == 0.

What we never claim:

"We score X% on GSM8K"

What we may claim:

"We admit X% of GSM8K with the current bounded grammar. Within that admitted subset we solve 100% with zero confabulation; outside it we refuse cleanly with typed reasons."

Those two sentences look superficially similar. They are not. The first conflates parser scope with reasoning ability and would re-open the parser-shape treadmill ADR-0131 closed. The second preserves the architecture-aligned discipline: refusal-first, wrong == 0 invariant, no confabulation under any input.

---

Decision

Introduce a coverage probe: evals/gsm8k_math/train_sample/v1/run_coverage_probe.py. It adapts the GSM8K train_sample (ADR-0126 P5) into the existing evals.gsm8k_math.runner.run_lane shape and writes a deterministic train_sample_coverage_report.json. The report fields are pinned by this ADR:

Field	Meaning
admitted_solved	Parsed + solved + verifier-passed + correct answer/unit
admitted_wrong	Parsed + solved BUT verifier or answer mismatch — gate: must == 0
refused	Refused at parser or solver with a typed error
admission_rate	admitted_solved / cases_total
safety_rail_intact	admitted_wrong == 0
refused_reasons_top	Counter of the most common refusal reasons — the work queue for grammar expansion
per_case	Full audit trail (deterministic order)

The probe runs the existing run_lane — the runner is the contract, not the probe. The probe's job is to adapt input + emit the report.

Iteration discipline

Capability-first, not coverage-first. Each subsequent ADR-0131.G. (G.1, G.2, ...) lands a capability extension — not a list of GSM8K-specific templates. The distinction is the load-bearing one in this ADR:

• ❌ Wrong mode: "Find templates that admit more GSM8K cases." This leads to narrow patterns that score on the test but do not generalize. Coverage rises, capability does not.
• ✅ Right mode: "Extend the model's NL-to-typed-graph capability along a principled axis (verb class, comparative structure, numeric form, multi-clause grammar). GSM8K admission rises as a symptom of that growth — alongside any other word-problem corpus."

Concretely, every iteration must:

1. Name a single capability axis it extends (e.g. "rate verbs as initial-state-introducing predicates", "comparative- multiplicative as a binding-graph operation kind").
2. Add B3-style curated coverage cases that exercise the new capability independently of GSM8K — i.e. the new template class is exercised by hand-authored sentences in evals/math_bounded_grammar/v1/cases.jsonl (or a sibling), not only by GSM8K cases.
3. Re-run both the B3 lane and the GSM8K coverage probe. The B3 lane must still pass; the GSM8K probe admission_rate moves as a side effect.
4. Reject any expansion that only moves GSM8K admission and not B3 / curated coverage. That is the smell test for a template in disguise.

Gating at each iteration:

1. admitted_wrong == 0 on the GSM8K probe (the safety rail)
2. B3 lane still passes (no regression on the curated benchmark)
3. The new capability has its own curated test cases — landed in the same PR — that the iteration's PR diff exercises
4. admission_rate strictly increased OR refused_reasons_top shows a deliberate, documented reduction in a targeted clause family

We never grow admission by relaxing the parser into "best-effort" guessing. Every admitted case must round-trip through the deterministic solver + independent verifier (ADR-0117). Every admitted case is admitted because a general capability now recognizes it — not because a specific GSM8K phrasing got hard-coded.

Baseline (this ADR pins the starting line)

admission_rate: 0/50 = 0.0%
admitted_wrong: 0  (gate intact)
refused:        50/50 = 100.0%

Every refusal is at the parser layer with a typed error citing the specific clause that did not match a template. Zero crashes, zero confabulations. The safety rail is bulletproof at admission rate zero.

What "passing GSM8K" means under this ADR

The phrase is informal shorthand for: get the model to a capability level where GSM8K passes as a side effect. It is explicitly not "tune the model to score on GSM8K." The mindset distinction is the load-bearing one:

• Shoot for test-passing capability → game the benchmark, produce numbers that don't transfer
• Shoot for the level at which the test passes → grow real NL-to-typed-graph capability; GSM8K passes as a symptom

On the question of an architectural ceiling

There is no theoretical capability ceiling for unambiguous, well-specified word problems. Determinism excludes three things:

1. Stochastic sampling — CORE will not sample-and-pray.
2. Probabilistic best-guess on ambiguous input — CORE refuses when interpretation is underdetermined.
3. Not NL understanding. The binding graph + typed compiler is an NL-understanding layer. Determinism does not exclude it; it constrains how it commits.

What CORE has instead of a confabulation ceiling is a refusal floor on genuinely ambiguous or underspecified inputs. On benchmarks like GSM8K that floor is small (problems are mostly well-specified). The contrast with frontier:

• Frontier hits ~95% on GSM8K because it confabulates the last ~5% — cases that should have refused but produced confident wrong answers instead.
• CORE has the opposite shape on that ~5%: typed refusal with a cited reason, not a confidently wrong number.

The practical limit on admission_rate is engineering effort on the parser + binding-graph layer, not architecture. The honest claim, at every admission level, is the same two-sentence framing in the Context section. If a capability extension also makes B3 stronger, makes new word-problem corpora work out-of-the-box, and survives adversarial paraphrase tests, then GSM8K admission gain is a real signal. If the capability only moves GSM8K, the iteration failed its own discipline.

---

Invariants

• wrong_count_is_zero — every report iteration must show admitted_wrong == 0. This is the load-bearing safety rail inherited from B1 / B2 / B3.
• refusal_is_typed — every refused case has a non-empty reason citing the parser or solver layer. No crash-shaped failures, no silent failures.
• report_deterministic — two runs over the same case set produce byte-equal train_sample_coverage_report.json.
• adapter_pure — _adapt_case is a pure function from the train_sample dict shape to the runner shape; no I/O, no global state.

---

Acceptance evidence

Accepted when:

• evals/gsm8k_math/train_sample/v1/run_coverage_probe.py ships and exits 0 on the current 50-case sample.
• evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json is committed and matches the script's deterministic output.
• tests/test_adr_0131_G_gsm8k_coverage_probe.py passes — pins the safety rail (admitted_wrong == 0), the typed-refusal invariant (every refused case has a non-empty reason), and the report-byte-equality invariant.
• This ADR is included.

---

Consequences

• The train_sample corpus stops being eval-adjacent dead weight. It becomes a measured progress artifact every iteration can move.
• Each ADR-0131.G. iteration's PR diff includes a real number delta in the report's admission_rate and a real reduction in the refused_reasons_top work queue.
• The grammar-expansion work is auditable: each new template comes with B3-style coverage tests and a documented refusal-reason family it closes.
• The promotion-gate composite (B1 + B2 + B3) is unaffected. ADR-0131.4 (promotion wiring) consumes those three; it does NOT consume the GSM8K coverage probe. The probe is empirical context, not a gate.

---

Out of scope

• Pushing admission beyond the train sample's 50 cases — sealed holdout (ADR-0119.7) and public split (ADR-0119.2) remain separate measurement surfaces.
• Frontier head-to-head on GSM8K — deferred until admission is characterized enough that a CORE-vs-frontier story is architecture-aligned rather than treadmill-shaped.
• Promotion-gate amendment — ADR-0131.4 is the place to decide whether GSM8K admission ever enters the promotion contract. This ADR deliberately keeps that question open.