docs(ADR-0131.5): retire GSM8K probe as per-iteration gate after G.x completion (#199)
All five G.x capability axes shipped (G.1–G.5); admission remained 0/50 throughout — expected, because the 50 sample cases fail at the statement layer (rate verbs, implicit entities, multi-step ops), orthogonal to the question-layer and initial-state-verb work the G.x axes targeted. Decisions: - Probe retired as a per-iteration gate (signal-to-noise ≈ 0 at 0/50) - `admitted_wrong == 0` invariant preserved as a standing architectural invariant, CI-pinned by existing test + per-axis regression guards - Re-activation criterion named: statement-layer expansion that plausibly admits ≥1 new case (Δadmission_rate ≥ 0.02) - Probe runner + report stay in-repo for future statement-layer push Docs-only. No test, parser, solver, or eval changes.
This commit is contained in:
parent
7f67cea400
commit
a32ffdc525
1 changed files with 124 additions and 0 deletions
124
docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md
Normal file
124
docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md
Normal file
|
|
@ -0,0 +1,124 @@
|
||||||
|
# ADR-0131.5 — GSM8K Coverage Probe: Retirement After G.x Axis Completion
|
||||||
|
|
||||||
|
**Status:** Accepted
|
||||||
|
**Date:** 2026-05-23
|
||||||
|
**Parent:** [ADR-0131.G — GSM8K Coverage Probe](ADR-0131.G-gsm8k-coverage-probe.md)
|
||||||
|
**Amends:** ADR-0131.G iteration discipline section
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
ADR-0131.G introduced the GSM8K coverage probe as a diff-able baseline
|
||||||
|
(`admitted_wrong == 0`, `admission_rate` tracked across G.x iterations)
|
||||||
|
and pinned capability-first iteration discipline for the G.x capability
|
||||||
|
axes. Five axes shipped and merged to `main`:
|
||||||
|
|
||||||
|
| Axis | ADR | PR | Axis cases | Wrong |
|
||||||
|
|------|-----|----|-----------|-------|
|
||||||
|
| G.1 — verb classes for initial-state | ADR-0131.G.1 | #187 | curated lane | 0 |
|
||||||
|
| G.2 — comparative operations | ADR-0131.G.2 | #196 | 31 cases | 0 |
|
||||||
|
| G.3 — numeric literals | ADR-0131.G.3 | #183 | curated lane | 0 |
|
||||||
|
| G.3.1 — numerics extensions | ADR-0131.G.3.1 | #186 | curated lane | 0 |
|
||||||
|
| G.4 — multi-clause composition | ADR-0131.G.4 | #185 | curated lane | 0 |
|
||||||
|
| G.5 — aggregate answer composition | ADR-0131.G.5 | #197 | 20 cases | 0 |
|
||||||
|
|
||||||
|
**GSM8K admission after all five axes: 0/50 (unchanged from baseline).**
|
||||||
|
|
||||||
|
This was expected and documented in each G.x PR. The bottleneck for all
|
||||||
|
50 sample cases is statement-layer parsing — rate verbs, compound
|
||||||
|
sentences, implicit entities, multi-step operations. The G.x axes
|
||||||
|
targeted the *question layer* (aggregate cues, comparative structure,
|
||||||
|
numeric literals, multi-clause subjects) and the *initial-state verb
|
||||||
|
layer*. None of the 50 GSM8K sample cases fail at those layers; they
|
||||||
|
fail earlier.
|
||||||
|
|
||||||
|
The safety rail (`admitted_wrong == 0`) never wavered.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### 1. Retire the GSM8K probe from the per-iteration gate
|
||||||
|
|
||||||
|
Effective immediately, `train_sample_coverage_report.json` is a **frozen
|
||||||
|
evidence artifact**, not a live gate for future iterations. The reasons:
|
||||||
|
|
||||||
|
- Admission is still 0/50. A gate that never moves does not gate.
|
||||||
|
- The G.x axes demonstrated that architecture-aligned capability growth
|
||||||
|
does not necessarily lift GSM8K admission — and that is correct
|
||||||
|
behavior, not a failure. The 50 sample cases are hard precisely
|
||||||
|
because of statement-layer features (rate verbs, implicit entities)
|
||||||
|
that are deliberately out of scope for the closed bounded-grammar
|
||||||
|
model.
|
||||||
|
- Keeping the probe as a per-iteration required re-run creates process
|
||||||
|
overhead whose signal-to-noise ratio is now close to zero.
|
||||||
|
|
||||||
|
### 2. Preserve the `admitted_wrong == 0` invariant permanently
|
||||||
|
|
||||||
|
The safety rail is not retired. It becomes a **standing architectural
|
||||||
|
invariant** rather than a per-iteration gate:
|
||||||
|
|
||||||
|
> Any future change to the parser, solver, or verifier must preserve
|
||||||
|
> `admitted_wrong == 0` on the GSM8K train sample.
|
||||||
|
|
||||||
|
The existing test
|
||||||
|
`tests/test_adr_0131_G_gsm8k_coverage_probe.py` already CI-pins this;
|
||||||
|
it is not removed. Each per-axis test file (G.2, G.5, etc.) also
|
||||||
|
asserts `admitted_wrong == 0` as a regression guard. This coverage is
|
||||||
|
sufficient; no per-iteration re-report is needed.
|
||||||
|
|
||||||
|
### 3. Conditions for re-activating the probe as a live gate
|
||||||
|
|
||||||
|
If a future ADR targets **statement-layer** expansion (rate verbs,
|
||||||
|
multi-step implicit operations, compound sentence parsing), re-run the
|
||||||
|
probe and include the delta. The probe infrastructure remains in-repo
|
||||||
|
for this purpose. The re-activation criterion is:
|
||||||
|
|
||||||
|
- The new capability plausibly changes the admission count on ≥1
|
||||||
|
of the 50 sample cases.
|
||||||
|
- The PR diff includes evidence that it does (pre/post `admission_rate`
|
||||||
|
delta ≥ 0.02, i.e. ≥1 additional admission).
|
||||||
|
|
||||||
|
### 4. Archive note in ADR-0131.G
|
||||||
|
|
||||||
|
ADR-0131.G iteration discipline section 4 ("Reject any expansion that
|
||||||
|
only moves GSM8K admission") remains valid. This ADR adds a symmetric
|
||||||
|
note: expansions that do not move GSM8K admission are not defects — they
|
||||||
|
are working as intended when the bottleneck is orthogonal to the
|
||||||
|
capability axis being extended.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
|
||||||
|
— frozen at `admission_rate: 0.0`, `admitted_wrong: 0`, `refused: 50`
|
||||||
|
after all G.x axes.
|
||||||
|
- `tests/test_adr_0131_G_gsm8k_coverage_probe.py` — CI-pins
|
||||||
|
`admitted_wrong == 0` as a standing regression guard.
|
||||||
|
- Per-axis tests (G.2, G.5) each include a `test_gsm8k_*_admitted_wrong_zero`
|
||||||
|
assertion as belt-and-suspenders.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Future capability-axis PRs do not need to re-run or delta the GSM8K
|
||||||
|
probe unless their scope touches statement-layer parsing.
|
||||||
|
- The probe file and runner remain in-repo as infrastructure for the
|
||||||
|
eventual statement-layer push.
|
||||||
|
- The ADR-0131.G "capability-first, not coverage-first" discipline is
|
||||||
|
unchanged. GSM8K admission is still a *symptom*, never a target.
|
||||||
|
This ADR merely removes the requirement to measure the symptom on
|
||||||
|
every iteration when the symptom cannot move.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Statement-layer expansion (rate verbs, multi-step implicit operations)
|
||||||
|
— that is a separate ADR when the time comes.
|
||||||
|
- Promotion-gate amendment — ADR-0131.4 is the gate; it does not
|
||||||
|
reference the GSM8K probe, and this ADR does not change it.
|
||||||
|
- Removal of the probe runner or report file — both stay in-repo.
|
||||||
Loading…
Reference in a new issue