From a32ffdc5255e06bbd998cb2fd19ab23996378f86 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 23 May 2026 19:49:45 -0700 Subject: [PATCH] docs(ADR-0131.5): retire GSM8K probe as per-iteration gate after G.x completion (#199) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five G.x capability axes shipped (G.1–G.5); admission remained 0/50 throughout — expected, because the 50 sample cases fail at the statement layer (rate verbs, implicit entities, multi-step ops), orthogonal to the question-layer and initial-state-verb work the G.x axes targeted. Decisions: - Probe retired as a per-iteration gate (signal-to-noise ≈ 0 at 0/50) - `admitted_wrong == 0` invariant preserved as a standing architectural invariant, CI-pinned by existing test + per-axis regression guards - Re-activation criterion named: statement-layer expansion that plausibly admits ≥1 new case (Δadmission_rate ≥ 0.02) - Probe runner + report stay in-repo for future statement-layer push Docs-only. No test, parser, solver, or eval changes. --- .../ADR-0131.5-gsm8k-probe-retirement.md | 124 ++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md diff --git a/docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md b/docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md new file mode 100644 index 00000000..759bd328 --- /dev/null +++ b/docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md @@ -0,0 +1,124 @@ +# ADR-0131.5 — GSM8K Coverage Probe: Retirement After G.x Axis Completion + +**Status:** Accepted +**Date:** 2026-05-23 +**Parent:** [ADR-0131.G — GSM8K Coverage Probe](ADR-0131.G-gsm8k-coverage-probe.md) +**Amends:** ADR-0131.G iteration discipline section + +--- + +## Context + +ADR-0131.G introduced the GSM8K coverage probe as a diff-able baseline +(`admitted_wrong == 0`, `admission_rate` tracked across G.x iterations) +and pinned capability-first iteration discipline for the G.x capability +axes. Five axes shipped and merged to `main`: + +| Axis | ADR | PR | Axis cases | Wrong | +|------|-----|----|-----------|-------| +| G.1 — verb classes for initial-state | ADR-0131.G.1 | #187 | curated lane | 0 | +| G.2 — comparative operations | ADR-0131.G.2 | #196 | 31 cases | 0 | +| G.3 — numeric literals | ADR-0131.G.3 | #183 | curated lane | 0 | +| G.3.1 — numerics extensions | ADR-0131.G.3.1 | #186 | curated lane | 0 | +| G.4 — multi-clause composition | ADR-0131.G.4 | #185 | curated lane | 0 | +| G.5 — aggregate answer composition | ADR-0131.G.5 | #197 | 20 cases | 0 | + +**GSM8K admission after all five axes: 0/50 (unchanged from baseline).** + +This was expected and documented in each G.x PR. The bottleneck for all +50 sample cases is statement-layer parsing — rate verbs, compound +sentences, implicit entities, multi-step operations. The G.x axes +targeted the *question layer* (aggregate cues, comparative structure, +numeric literals, multi-clause subjects) and the *initial-state verb +layer*. None of the 50 GSM8K sample cases fail at those layers; they +fail earlier. + +The safety rail (`admitted_wrong == 0`) never wavered. + +--- + +## Decision + +### 1. Retire the GSM8K probe from the per-iteration gate + +Effective immediately, `train_sample_coverage_report.json` is a **frozen +evidence artifact**, not a live gate for future iterations. The reasons: + +- Admission is still 0/50. A gate that never moves does not gate. +- The G.x axes demonstrated that architecture-aligned capability growth + does not necessarily lift GSM8K admission — and that is correct + behavior, not a failure. The 50 sample cases are hard precisely + because of statement-layer features (rate verbs, implicit entities) + that are deliberately out of scope for the closed bounded-grammar + model. +- Keeping the probe as a per-iteration required re-run creates process + overhead whose signal-to-noise ratio is now close to zero. + +### 2. Preserve the `admitted_wrong == 0` invariant permanently + +The safety rail is not retired. It becomes a **standing architectural +invariant** rather than a per-iteration gate: + +> Any future change to the parser, solver, or verifier must preserve +> `admitted_wrong == 0` on the GSM8K train sample. + +The existing test +`tests/test_adr_0131_G_gsm8k_coverage_probe.py` already CI-pins this; +it is not removed. Each per-axis test file (G.2, G.5, etc.) also +asserts `admitted_wrong == 0` as a regression guard. This coverage is +sufficient; no per-iteration re-report is needed. + +### 3. Conditions for re-activating the probe as a live gate + +If a future ADR targets **statement-layer** expansion (rate verbs, +multi-step implicit operations, compound sentence parsing), re-run the +probe and include the delta. The probe infrastructure remains in-repo +for this purpose. The re-activation criterion is: + +- The new capability plausibly changes the admission count on ≥1 + of the 50 sample cases. +- The PR diff includes evidence that it does (pre/post `admission_rate` + delta ≥ 0.02, i.e. ≥1 additional admission). + +### 4. Archive note in ADR-0131.G + +ADR-0131.G iteration discipline section 4 ("Reject any expansion that +only moves GSM8K admission") remains valid. This ADR adds a symmetric +note: expansions that do not move GSM8K admission are not defects — they +are working as intended when the bottleneck is orthogonal to the +capability axis being extended. + +--- + +## Evidence + +- `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json` + — frozen at `admission_rate: 0.0`, `admitted_wrong: 0`, `refused: 50` + after all G.x axes. +- `tests/test_adr_0131_G_gsm8k_coverage_probe.py` — CI-pins + `admitted_wrong == 0` as a standing regression guard. +- Per-axis tests (G.2, G.5) each include a `test_gsm8k_*_admitted_wrong_zero` + assertion as belt-and-suspenders. + +--- + +## Consequences + +- Future capability-axis PRs do not need to re-run or delta the GSM8K + probe unless their scope touches statement-layer parsing. +- The probe file and runner remain in-repo as infrastructure for the + eventual statement-layer push. +- The ADR-0131.G "capability-first, not coverage-first" discipline is + unchanged. GSM8K admission is still a *symptom*, never a target. + This ADR merely removes the requirement to measure the symptom on + every iteration when the symptom cannot move. + +--- + +## Out of scope + +- Statement-layer expansion (rate verbs, multi-step implicit operations) + — that is a separate ADR when the time comes. +- Promotion-gate amendment — ADR-0131.4 is the gate; it does not + reference the GSM8K probe, and this ADR does not change it. +- Removal of the probe runner or report file — both stay in-repo.