docs(ADR-0131.5): retire GSM8K probe as per-iteration gate after G.x completion (#199)
All five G.x capability axes shipped (G.1–G.5); admission remained 0/50 throughout — expected, because the 50 sample cases fail at the statement layer (rate verbs, implicit entities, multi-step ops), orthogonal to the question-layer and initial-state-verb work the G.x axes targeted. Decisions: - Probe retired as a per-iteration gate (signal-to-noise ≈ 0 at 0/50) - `admitted_wrong == 0` invariant preserved as a standing architectural invariant, CI-pinned by existing test + per-axis regression guards - Re-activation criterion named: statement-layer expansion that plausibly admits ≥1 new case (Δadmission_rate ≥ 0.02) - Probe runner + report stay in-repo for future statement-layer push Docs-only. No test, parser, solver, or eval changes.
This commit is contained in:
Shay
GitHub
parent
7f67cea400
commit
a32ffdc525
1 changed files with 124 additions and 0 deletions
124
docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md
Normal file
124
docs/decisions/ADR-0131.5-gsm8k-probe-retirement.md
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
# ADR-0131.5 — GSM8K Coverage Probe: Retirement After G.x Axis Completion
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-23
|
||||
**Parent:** [ADR-0131.G — GSM8K Coverage Probe](ADR-0131.G-gsm8k-coverage-probe.md)
|
||||
**Amends:** ADR-0131.G iteration discipline section
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0131.G introduced the GSM8K coverage probe as a diff-able baseline
|
||||
(`admitted_wrong == 0`, `admission_rate` tracked across G.x iterations)
|
||||
and pinned capability-first iteration discipline for the G.x capability
|
||||
axes. Five axes shipped and merged to `main`:
|
||||
|
||||
| Axis | ADR | PR | Axis cases | Wrong |
|
||||
|------|-----|----|-----------|-------|
|
||||
| G.1 — verb classes for initial-state | ADR-0131.G.1 | #187 | curated lane | 0 |
|
||||
| G.2 — comparative operations | ADR-0131.G.2 | #196 | 31 cases | 0 |
|
||||
| G.3 — numeric literals | ADR-0131.G.3 | #183 | curated lane | 0 |
|
||||
| G.3.1 — numerics extensions | ADR-0131.G.3.1 | #186 | curated lane | 0 |
|
||||
| G.4 — multi-clause composition | ADR-0131.G.4 | #185 | curated lane | 0 |
|
||||
| G.5 — aggregate answer composition | ADR-0131.G.5 | #197 | 20 cases | 0 |
|
||||
|
||||
**GSM8K admission after all five axes: 0/50 (unchanged from baseline).**
|
||||
|
||||
This was expected and documented in each G.x PR. The bottleneck for all
|
||||
50 sample cases is statement-layer parsing — rate verbs, compound
|
||||
sentences, implicit entities, multi-step operations. The G.x axes
|
||||
targeted the *question layer* (aggregate cues, comparative structure,
|
||||
numeric literals, multi-clause subjects) and the *initial-state verb
|
||||
layer*. None of the 50 GSM8K sample cases fail at those layers; they
|
||||
fail earlier.
|
||||
|
||||
The safety rail (`admitted_wrong == 0`) never wavered.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Retire the GSM8K probe from the per-iteration gate
|
||||
|
||||
Effective immediately, `train_sample_coverage_report.json` is a **frozen
|
||||
evidence artifact**, not a live gate for future iterations. The reasons:
|
||||
|
||||
- Admission is still 0/50. A gate that never moves does not gate.
|
||||
- The G.x axes demonstrated that architecture-aligned capability growth
|
||||
does not necessarily lift GSM8K admission — and that is correct
|
||||
behavior, not a failure. The 50 sample cases are hard precisely
|
||||
because of statement-layer features (rate verbs, implicit entities)
|
||||
that are deliberately out of scope for the closed bounded-grammar
|
||||
model.
|
||||
- Keeping the probe as a per-iteration required re-run creates process
|
||||
overhead whose signal-to-noise ratio is now close to zero.
|
||||
|
||||
### 2. Preserve the `admitted_wrong == 0` invariant permanently
|
||||
|
||||
The safety rail is not retired. It becomes a **standing architectural
|
||||
invariant** rather than a per-iteration gate:
|
||||
|
||||
> Any future change to the parser, solver, or verifier must preserve
|
||||
> `admitted_wrong == 0` on the GSM8K train sample.
|
||||
|
||||
The existing test
|
||||
`tests/test_adr_0131_G_gsm8k_coverage_probe.py` already CI-pins this;
|
||||
it is not removed. Each per-axis test file (G.2, G.5, etc.) also
|
||||
asserts `admitted_wrong == 0` as a regression guard. This coverage is
|
||||
sufficient; no per-iteration re-report is needed.
|
||||
|
||||
### 3. Conditions for re-activating the probe as a live gate
|
||||
|
||||
If a future ADR targets **statement-layer** expansion (rate verbs,
|
||||
multi-step implicit operations, compound sentence parsing), re-run the
|
||||
probe and include the delta. The probe infrastructure remains in-repo
|
||||
for this purpose. The re-activation criterion is:
|
||||
|
||||
- The new capability plausibly changes the admission count on ≥1
|
||||
of the 50 sample cases.
|
||||
- The PR diff includes evidence that it does (pre/post `admission_rate`
|
||||
delta ≥ 0.02, i.e. ≥1 additional admission).
|
||||
|
||||
### 4. Archive note in ADR-0131.G
|
||||
|
||||
ADR-0131.G iteration discipline section 4 ("Reject any expansion that
|
||||
only moves GSM8K admission") remains valid. This ADR adds a symmetric
|
||||
note: expansions that do not move GSM8K admission are not defects — they
|
||||
are working as intended when the bottleneck is orthogonal to the
|
||||
capability axis being extended.
|
||||
|
||||
---
|
||||
|
||||
## Evidence
|
||||
|
||||
- `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
|
||||
— frozen at `admission_rate: 0.0`, `admitted_wrong: 0`, `refused: 50`
|
||||
after all G.x axes.
|
||||
- `tests/test_adr_0131_G_gsm8k_coverage_probe.py` — CI-pins
|
||||
`admitted_wrong == 0` as a standing regression guard.
|
||||
- Per-axis tests (G.2, G.5) each include a `test_gsm8k_*_admitted_wrong_zero`
|
||||
assertion as belt-and-suspenders.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
- Future capability-axis PRs do not need to re-run or delta the GSM8K
|
||||
probe unless their scope touches statement-layer parsing.
|
||||
- The probe file and runner remain in-repo as infrastructure for the
|
||||
eventual statement-layer push.
|
||||
- The ADR-0131.G "capability-first, not coverage-first" discipline is
|
||||
unchanged. GSM8K admission is still a *symptom*, never a target.
|
||||
This ADR merely removes the requirement to measure the symptom on
|
||||
every iteration when the symptom cannot move.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Statement-layer expansion (rate verbs, multi-step implicit operations)
|
||||
— that is a separate ADR when the time comes.
|
||||
- Promotion-gate amendment — ADR-0131.4 is the gate; it does not
|
||||
reference the GSM8K probe, and this ADR does not change it.
|
||||
- Removal of the probe runner or report file — both stay in-repo.
|
||||
Loading…
Reference in a new issue