core/docs/decisions/ADR-0131.G-gsm8k-coverage-probe.md

# ADR-0131.G — GSM8K Coverage Probe: Honest Measurement Under the Safety Rail

**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Parent:** [ADR-0131](./ADR-0131-math-expert-rebench.md)
**Depends on:**
[ADR-0115](./ADR-0115-math-problem-parser-and-graph.md),
[ADR-0116](./ADR-0116-deterministic-solver.md),
[ADR-0117](./ADR-0117-solution-trace-verifier.md),
[ADR-0131.3](./ADR-0131.3-bounded-grammar.md),
[ADR-0132](./ADR-0132-binding-graph-data-model.md)..ADR-0135

---

## Context

ADR-0131 re-targeted the math-expert promotion away from GSM8K
because GSM8K rewards paraphrase flexibility, which is the
deterministic engine's structural weakness. The composite
architecture-aligned gate (B1 symbolic equivalence, B2 teaching
corpus, B3 bounded-grammar word problems) is the *true* gate.

GSM8K does not vanish from the project; it returns as a **coverage
probe** of the bounded grammar + binding-graph layer. The framing
this ADR pins:

> GSM8K is not a promotion gate. It is a coverage measurement of
> what the bounded grammar admits. Of the admitted subset, the
> engine solves deterministically with ``wrong == 0``.

What we never claim:

> "We score X% on GSM8K"

What we may claim:

> "We admit X% of GSM8K with the current bounded grammar. Within
> that admitted subset we solve 100% with zero confabulation;
> outside it we refuse cleanly with typed reasons."

Those two sentences look superficially similar. They are not. The
first conflates parser scope with reasoning ability and would
re-open the parser-shape treadmill ADR-0131 closed. The second
preserves the architecture-aligned discipline: refusal-first,
``wrong == 0`` invariant, no confabulation under any input.

---

## Decision

Introduce a **coverage probe**:
``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py``. It
adapts the GSM8K train_sample (ADR-0126 P5) into the existing
``evals.gsm8k_math.runner.run_lane`` shape and writes a
deterministic ``train_sample_coverage_report.json``. The report
fields are pinned by this ADR:

| Field | Meaning |
|---|---|
| ``admitted_solved`` | Parsed + solved + verifier-passed + correct answer/unit |
| ``admitted_wrong`` | Parsed + solved BUT verifier or answer mismatch — **gate: must == 0** |
| ``refused`` | Refused at parser or solver with a typed error |
| ``admission_rate`` | ``admitted_solved / cases_total`` |
| ``safety_rail_intact`` | ``admitted_wrong == 0`` |
| ``refused_reasons_top`` | Counter of the most common refusal reasons — the *work queue* for grammar expansion |
| ``per_case`` | Full audit trail (deterministic order) |

The probe runs the existing ``run_lane`` — the *runner is the
contract*, not the probe. The probe's job is to adapt input + emit
the report.

### Iteration discipline

**Capability-first, not coverage-first.** Each subsequent
ADR-0131.G.<n> (G.1, G.2, ...) lands a *capability extension* —
not a list of GSM8K-specific templates. The distinction is the
load-bearing one in this ADR:

- ❌ Wrong mode: "Find templates that admit more GSM8K cases."
  This leads to narrow patterns that score on the test but do
  not generalize. Coverage rises, capability does not.
- ✅ Right mode: "Extend the model's NL-to-typed-graph capability
  along a principled axis (verb class, comparative structure,
  numeric form, multi-clause grammar). GSM8K admission rises as
  a *symptom* of that growth — alongside any other word-problem
  corpus."

Concretely, every iteration must:

1. Name a single capability axis it extends (e.g. "rate verbs
   as initial-state-introducing predicates", "comparative-
   multiplicative as a binding-graph operation kind").
2. Add B3-style curated coverage cases that exercise the new
   capability **independently of GSM8K** — i.e. the new template
   class is exercised by hand-authored sentences in
   ``evals/math_bounded_grammar/v1/cases.jsonl`` (or a sibling),
   not only by GSM8K cases.
3. Re-run *both* the B3 lane and the GSM8K coverage probe.
   The B3 lane must still pass; the GSM8K probe ``admission_rate``
   moves as a side effect.
4. **Reject any expansion that only moves GSM8K admission and
   not B3 / curated coverage.** That is the smell test for a
   template in disguise.

Gating at each iteration:

1. ``admitted_wrong == 0`` on the GSM8K probe (the safety rail)
2. B3 lane still passes (no regression on the curated benchmark)
3. The new capability has its own curated test cases — landed
   in the same PR — that the iteration's PR diff exercises
4. ``admission_rate`` strictly increased OR ``refused_reasons_top``
   shows a deliberate, documented reduction in a targeted clause
   family

We never grow admission by relaxing the parser into "best-effort"
guessing. Every admitted case must round-trip through the
deterministic solver + independent verifier (ADR-0117). Every
admitted case is admitted because a *general capability* now
recognizes it — not because a specific GSM8K phrasing got
hard-coded.

### Baseline (this ADR pins the starting line)

```
admission_rate: 0/50 = 0.0%
admitted_wrong: 0  (gate intact)
refused:        50/50 = 100.0%
```

Every refusal is at the parser layer with a typed error citing the
specific clause that did not match a template. **Zero crashes,
zero confabulations.** The safety rail is bulletproof at admission
rate zero.

### What "passing GSM8K" means under this ADR

The phrase is informal shorthand for: *get the model to a
capability level where GSM8K passes as a side effect.* It is
explicitly **not** "tune the model to score on GSM8K." The mindset
distinction is the load-bearing one:

- Shoot for **test-passing capability** → game the benchmark,
  produce numbers that don't transfer
- Shoot for **the level at which the test passes** → grow real
  NL-to-typed-graph capability; GSM8K passes as a symptom

### On the question of an architectural ceiling

There is **no theoretical capability ceiling** for unambiguous,
well-specified word problems. Determinism excludes three things:

1. Stochastic sampling — CORE will not sample-and-pray.
2. Probabilistic best-guess on ambiguous input — CORE refuses
   when interpretation is underdetermined.
3. *Not* NL understanding. The binding graph + typed compiler
   *is* an NL-understanding layer. Determinism does not exclude
   it; it constrains how it commits.

What CORE has instead of a confabulation ceiling is a **refusal
floor** on genuinely ambiguous or underspecified inputs. On
benchmarks like GSM8K that floor is small (problems are mostly
well-specified). The contrast with frontier:

- Frontier hits ~95% on GSM8K because it confabulates the last
  ~5% — cases that should have refused but produced confident
  wrong answers instead.
- CORE has the opposite shape on that ~5%: typed refusal with
  a cited reason, not a confidently wrong number.

The practical limit on admission_rate is **engineering effort on
the parser + binding-graph layer**, not architecture. The honest
claim, at every admission level, is the same two-sentence framing
in the Context section. If a capability extension also makes B3
stronger, makes new word-problem corpora work out-of-the-box, and
survives adversarial paraphrase tests, then GSM8K admission gain
is a real signal. If the capability only moves GSM8K, the
iteration failed its own discipline.

---

## Invariants

- **``wrong_count_is_zero``** — every report iteration must show
  ``admitted_wrong == 0``. This is the load-bearing safety rail
  inherited from B1 / B2 / B3.
- **``refusal_is_typed``** — every refused case has a non-empty
  reason citing the parser or solver layer. No crash-shaped
  failures, no silent failures.
- **``report_deterministic``** — two runs over the same case set
  produce byte-equal ``train_sample_coverage_report.json``.
- **``adapter_pure``** — ``_adapt_case`` is a pure function from
  the train_sample dict shape to the runner shape; no I/O, no
  global state.

---

## Acceptance evidence

Accepted when:

- ``evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`` ships
  and exits 0 on the current 50-case sample.
- ``evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json``
  is committed and matches the script's deterministic output.
- ``tests/test_adr_0131_G_gsm8k_coverage_probe.py`` passes — pins
  the safety rail (``admitted_wrong == 0``), the typed-refusal
  invariant (every refused case has a non-empty reason), and the
  report-byte-equality invariant.
- This ADR is included.

---

## Consequences

- The ``train_sample`` corpus stops being eval-adjacent dead
  weight. It becomes a measured progress artifact every iteration
  can move.
- Each ADR-0131.G.<n> iteration's PR diff includes a real number
  delta in the report's ``admission_rate`` and a real reduction in
  the ``refused_reasons_top`` work queue.
- The grammar-expansion work is auditable: each new template comes
  with B3-style coverage tests and a documented refusal-reason
  family it closes.
- The promotion-gate composite (B1 + B2 + B3) is unaffected.
  ``ADR-0131.4`` (promotion wiring) consumes those three; it does
  NOT consume the GSM8K coverage probe. The probe is empirical
  context, not a gate.

---

## Out of scope

- Pushing admission beyond the train sample's 50 cases — sealed
  holdout (ADR-0119.7) and public split (ADR-0119.2) remain
  separate measurement surfaces.
- Frontier head-to-head on GSM8K — deferred until admission is
  characterized enough that a CORE-vs-frontier story is
  architecture-aligned rather than treadmill-shaped.
- Promotion-gate amendment — ADR-0131.4 is the place to decide
  whether GSM8K admission ever enters the promotion contract.
  This ADR deliberately keeps that question open.