docs(ADR-0131.5): retire GSM8K probe as per-iteration gate after G.x completion (#199 )

All five G.x capability axes shipped (G.1–G.5); admission remained 0/50
throughout — expected, because the 50 sample cases fail at the statement
layer (rate verbs, implicit entities, multi-step ops), orthogonal to the
question-layer and initial-state-verb work the G.x axes targeted.

Decisions:
- Probe retired as a per-iteration gate (signal-to-noise ≈ 0 at 0/50)
- `admitted_wrong == 0` invariant preserved as a standing architectural
  invariant, CI-pinned by existing test + per-axis regression guards
- Re-activation criterion named: statement-layer expansion that
  plausibly admits ≥1 new case (Δadmission_rate ≥ 0.02)
- Probe runner + report stay in-repo for future statement-layer push

Docs-only. No test, parser, solver, or eval changes.

2026-05-23 19:49:45 -07:00

5 KiB

Raw Permalink Blame History

ADR-0131.5 — GSM8K Coverage Probe: Retirement After G.x Axis Completion

Status: Accepted
Date: 2026-05-23
Parent: ADR-0131.G — GSM8K Coverage Probe
Amends: ADR-0131.G iteration discipline section

Context

ADR-0131.G introduced the GSM8K coverage probe as a diff-able baseline (admitted_wrong == 0, admission_rate tracked across G.x iterations) and pinned capability-first iteration discipline for the G.x capability axes. Five axes shipped and merged to main:

Axis	ADR	PR	Axis cases
G.1 — verb classes for initial-state	ADR-0131.G.1	#187	curated lane
G.2 — comparative operations	ADR-0131.G.2	#196	31 cases
G.3 — numeric literals	ADR-0131.G.3	#183	curated lane
G.3.1 — numerics extensions	ADR-0131.G.3.1	#186	curated lane
G.4 — multi-clause composition	ADR-0131.G.4	#185	curated lane
G.5 — aggregate answer composition	ADR-0131.G.5	#197	20 cases

GSM8K admission after all five axes: 0/50 (unchanged from baseline).

This was expected and documented in each G.x PR. The bottleneck for all 50 sample cases is statement-layer parsing — rate verbs, compound sentences, implicit entities, multi-step operations. The G.x axes targeted the question layer (aggregate cues, comparative structure, numeric literals, multi-clause subjects) and the initial-state verb layer. None of the 50 GSM8K sample cases fail at those layers; they fail earlier.

The safety rail (admitted_wrong == 0) never wavered.

Decision

1. Retire the GSM8K probe from the per-iteration gate

Effective immediately, train_sample_coverage_report.json is a frozen evidence artifact, not a live gate for future iterations. The reasons:

Admission is still 0/50. A gate that never moves does not gate.
The G.x axes demonstrated that architecture-aligned capability growth does not necessarily lift GSM8K admission — and that is correct behavior, not a failure. The 50 sample cases are hard precisely because of statement-layer features (rate verbs, implicit entities) that are deliberately out of scope for the closed bounded-grammar model.
Keeping the probe as a per-iteration required re-run creates process overhead whose signal-to-noise ratio is now close to zero.

2. Preserve the `admitted_wrong == 0` invariant permanently

The safety rail is not retired. It becomes a standing architectural invariant rather than a per-iteration gate:

Any future change to the parser, solver, or verifier must preserve admitted_wrong == 0 on the GSM8K train sample.

The existing test tests/test_adr_0131_G_gsm8k_coverage_probe.py already CI-pins this; it is not removed. Each per-axis test file (G.2, G.5, etc.) also asserts admitted_wrong == 0 as a regression guard. This coverage is sufficient; no per-iteration re-report is needed.

3. Conditions for re-activating the probe as a live gate

If a future ADR targets statement-layer expansion (rate verbs, multi-step implicit operations, compound sentence parsing), re-run the probe and include the delta. The probe infrastructure remains in-repo for this purpose. The re-activation criterion is:

The new capability plausibly changes the admission count on ≥1 of the 50 sample cases.
The PR diff includes evidence that it does (pre/post admission_rate delta ≥ 0.02, i.e. ≥1 additional admission).

4. Archive note in ADR-0131.G

ADR-0131.G iteration discipline section 4 ("Reject any expansion that only moves GSM8K admission") remains valid. This ADR adds a symmetric note: expansions that do not move GSM8K admission are not defects — they are working as intended when the bottleneck is orthogonal to the capability axis being extended.

Evidence

evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json — frozen at admission_rate: 0.0, admitted_wrong: 0, refused: 50 after all G.x axes.
tests/test_adr_0131_G_gsm8k_coverage_probe.py — CI-pins admitted_wrong == 0 as a standing regression guard.
Per-axis tests (G.2, G.5) each include a test_gsm8k_*_admitted_wrong_zero assertion as belt-and-suspenders.

Consequences

Future capability-axis PRs do not need to re-run or delta the GSM8K probe unless their scope touches statement-layer parsing.
The probe file and runner remain in-repo as infrastructure for the eventual statement-layer push.
The ADR-0131.G "capability-first, not coverage-first" discipline is unchanged. GSM8K admission is still a symptom, never a target. This ADR merely removes the requirement to measure the symptom on every iteration when the symptom cannot move.

Out of scope

Statement-layer expansion (rate verbs, multi-step implicit operations) — that is a separate ADR when the time comes.
Promotion-gate amendment — ADR-0131.4 is the gate; it does not reference the GSM8K probe, and this ADR does not change it.
Removal of the probe runner or report file — both stay in-repo.

5 KiB Raw Permalink Blame History