# ADR-0131.G — GSM8K Coverage Probe: Honest Measurement Under the Safety Rail **Status:** Proposed **Date:** 2026-05-23 **Author:** CORE agents + reviewers **Parent:** [ADR-0131](./ADR-0131-math-expert-rebench.md) **Depends on:** [ADR-0115](./ADR-0115-math-problem-parser-and-graph.md), [ADR-0116](./ADR-0116-deterministic-solver.md), [ADR-0117](./ADR-0117-solution-trace-verifier.md), [ADR-0131.3](./ADR-0131.3-bounded-grammar.md), [ADR-0132](./ADR-0132-binding-graph-data-model.md)..ADR-0135 --- ## Context ADR-0131 re-targeted the math-expert promotion away from GSM8K because GSM8K rewards paraphrase flexibility, which is the deterministic engine's structural weakness. The composite architecture-aligned gate (B1 symbolic equivalence, B2 teaching corpus, B3 bounded-grammar word problems) is the *true* gate. GSM8K does not vanish from the project; it returns as a **coverage probe** of the bounded grammar + binding-graph layer. The framing this ADR pins: > GSM8K is not a promotion gate. It is a coverage measurement of > what the bounded grammar admits. Of the admitted subset, the > engine solves deterministically with ``wrong == 0``. What we never claim: > "We score X% on GSM8K" What we may claim: > "We admit X% of GSM8K with the current bounded grammar. Within > that admitted subset we solve 100% with zero confabulation; > outside it we refuse cleanly with typed reasons." Those two sentences look superficially similar. They are not. The first conflates parser scope with reasoning ability and would re-open the parser-shape treadmill ADR-0131 closed. The second preserves the architecture-aligned discipline: refusal-first, ``wrong == 0`` invariant, no confabulation under any input. --- ## Decision Introduce a **coverage probe**: ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py``. It adapts the GSM8K train_sample (ADR-0126 P5) into the existing ``evals.gsm8k_math.runner.run_lane`` shape and writes a deterministic ``train_sample_coverage_report.json``. The report fields are pinned by this ADR: | Field | Meaning | |---|---| | ``admitted_solved`` | Parsed + solved + verifier-passed + correct answer/unit | | ``admitted_wrong`` | Parsed + solved BUT verifier or answer mismatch — **gate: must == 0** | | ``refused`` | Refused at parser or solver with a typed error | | ``admission_rate`` | ``admitted_solved / cases_total`` | | ``safety_rail_intact`` | ``admitted_wrong == 0`` | | ``refused_reasons_top`` | Counter of the most common refusal reasons — the *work queue* for grammar expansion | | ``per_case`` | Full audit trail (deterministic order) | The probe runs the existing ``run_lane`` — the *runner is the contract*, not the probe. The probe's job is to adapt input + emit the report. ### Iteration discipline **Capability-first, not coverage-first.** Each subsequent ADR-0131.G. (G.1, G.2, ...) lands a *capability extension* — not a list of GSM8K-specific templates. The distinction is the load-bearing one in this ADR: - ❌ Wrong mode: "Find templates that admit more GSM8K cases." This leads to narrow patterns that score on the test but do not generalize. Coverage rises, capability does not. - ✅ Right mode: "Extend the model's NL-to-typed-graph capability along a principled axis (verb class, comparative structure, numeric form, multi-clause grammar). GSM8K admission rises as a *symptom* of that growth — alongside any other word-problem corpus." Concretely, every iteration must: 1. Name a single capability axis it extends (e.g. "rate verbs as initial-state-introducing predicates", "comparative- multiplicative as a binding-graph operation kind"). 2. Add B3-style curated coverage cases that exercise the new capability **independently of GSM8K** — i.e. the new template class is exercised by hand-authored sentences in ``evals/math_bounded_grammar/v1/cases.jsonl`` (or a sibling), not only by GSM8K cases. 3. Re-run *both* the B3 lane and the GSM8K coverage probe. The B3 lane must still pass; the GSM8K probe ``admission_rate`` moves as a side effect. 4. **Reject any expansion that only moves GSM8K admission and not B3 / curated coverage.** That is the smell test for a template in disguise. Gating at each iteration: 1. ``admitted_wrong == 0`` on the GSM8K probe (the safety rail) 2. B3 lane still passes (no regression on the curated benchmark) 3. The new capability has its own curated test cases — landed in the same PR — that the iteration's PR diff exercises 4. ``admission_rate`` strictly increased OR ``refused_reasons_top`` shows a deliberate, documented reduction in a targeted clause family We never grow admission by relaxing the parser into "best-effort" guessing. Every admitted case must round-trip through the deterministic solver + independent verifier (ADR-0117). Every admitted case is admitted because a *general capability* now recognizes it — not because a specific GSM8K phrasing got hard-coded. ### Baseline (this ADR pins the starting line) ``` admission_rate: 0/50 = 0.0% admitted_wrong: 0 (gate intact) refused: 50/50 = 100.0% ``` Every refusal is at the parser layer with a typed error citing the specific clause that did not match a template. **Zero crashes, zero confabulations.** The safety rail is bulletproof at admission rate zero. ### What "passing GSM8K" means under this ADR The phrase is informal shorthand for: *get the model to a capability level where GSM8K passes as a side effect.* It is explicitly **not** "tune the model to score on GSM8K." The mindset distinction is the load-bearing one: - Shoot for **test-passing capability** → game the benchmark, produce numbers that don't transfer - Shoot for **the level at which the test passes** → grow real NL-to-typed-graph capability; GSM8K passes as a symptom ### On the question of an architectural ceiling There is **no theoretical capability ceiling** for unambiguous, well-specified word problems. Determinism excludes three things: 1. Stochastic sampling — CORE will not sample-and-pray. 2. Probabilistic best-guess on ambiguous input — CORE refuses when interpretation is underdetermined. 3. *Not* NL understanding. The binding graph + typed compiler *is* an NL-understanding layer. Determinism does not exclude it; it constrains how it commits. What CORE has instead of a confabulation ceiling is a **refusal floor** on genuinely ambiguous or underspecified inputs. On benchmarks like GSM8K that floor is small (problems are mostly well-specified). The contrast with frontier: - Frontier hits ~95% on GSM8K because it confabulates the last ~5% — cases that should have refused but produced confident wrong answers instead. - CORE has the opposite shape on that ~5%: typed refusal with a cited reason, not a confidently wrong number. The practical limit on admission_rate is **engineering effort on the parser + binding-graph layer**, not architecture. The honest claim, at every admission level, is the same two-sentence framing in the Context section. If a capability extension also makes B3 stronger, makes new word-problem corpora work out-of-the-box, and survives adversarial paraphrase tests, then GSM8K admission gain is a real signal. If the capability only moves GSM8K, the iteration failed its own discipline. --- ## Invariants - **``wrong_count_is_zero``** — every report iteration must show ``admitted_wrong == 0``. This is the load-bearing safety rail inherited from B1 / B2 / B3. - **``refusal_is_typed``** — every refused case has a non-empty reason citing the parser or solver layer. No crash-shaped failures, no silent failures. - **``report_deterministic``** — two runs over the same case set produce byte-equal ``train_sample_coverage_report.json``. - **``adapter_pure``** — ``_adapt_case`` is a pure function from the train_sample dict shape to the runner shape; no I/O, no global state. --- ## Acceptance evidence Accepted when: - ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`` ships and exits 0 on the current 50-case sample. - ``evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`` is committed and matches the script's deterministic output. - ``tests/test_adr_0131_G_gsm8k_coverage_probe.py`` passes — pins the safety rail (``admitted_wrong == 0``), the typed-refusal invariant (every refused case has a non-empty reason), and the report-byte-equality invariant. - This ADR is included. --- ## Consequences - The ``train_sample`` corpus stops being eval-adjacent dead weight. It becomes a measured progress artifact every iteration can move. - Each ADR-0131.G. iteration's PR diff includes a real number delta in the report's ``admission_rate`` and a real reduction in the ``refused_reasons_top`` work queue. - The grammar-expansion work is auditable: each new template comes with B3-style coverage tests and a documented refusal-reason family it closes. - The promotion-gate composite (B1 + B2 + B3) is unaffected. ``ADR-0131.4`` (promotion wiring) consumes those three; it does NOT consume the GSM8K coverage probe. The probe is empirical context, not a gate. --- ## Out of scope - Pushing admission beyond the train sample's 50 cases — sealed holdout (ADR-0119.7) and public split (ADR-0119.2) remain separate measurement surfaces. - Frontier head-to-head on GSM8K — deferred until admission is characterized enough that a CORE-vs-frontier story is architecture-aligned rather than treadmill-shaped. - Promotion-gate amendment — ADR-0131.4 is the place to decide whether GSM8K admission ever enters the promotion contract. This ADR deliberately keeps that question open.