# ADR-0119.3 — gsm8k_math Lane Runner (Phase 5.3)

**Status:** Accepted
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0118, ADR-0119, ADR-0119.1, ADR-0119.2

---

## Context

Phase 5.3 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). Composes
the Phases 1–4 pipeline (parser → solver → verifier → realizer) into a
per-case scoring decision that the GSM8K eval lane consumes.

Per ADR-0114a Obligation #4, every case must land in exactly one of
three outcomes: `correct`, `wrong`, or `refused`. The lane gates on
**`wrong == 0`** — CORE must refuse rather than confabulate. A
nonzero `wrong` count invalidates the lane regardless of `correct`
rate.

---

## Decision

### `evals/gsm8k_math/runner.py`

Exposes `run_lane(cases, *, config=None) → LaneReport` per the
framework runner interface (`evals/framework.py`). Pure function;
same input → byte-equal `LaneReport.canonical_bytes()`.

Per-case pipeline:

```text
parse_problem(text) → graph     # ParseError → refused
solve(graph)        → trace     # SolveError → refused
verify(graph, trace)            # passed=False → wrong
realize(initial_state, trace)   # RealizerError → wrong
compare answer/unit to expected # mismatch → wrong; match → correct
```

`CaseOutcome` dataclass records: `case_id`, `outcome`, `reason`,
`expected_answer/unit`, `actual_answer/unit`, `trace_hash`,
`realized_prose`. The trace hash + prose are present on every
non-refused outcome, so a downstream depth-curve harness (ADR-0119.6)
or frontier-comparison report (ADR-0119.4) has a full per-case
audit trail to consume.

### Aggregate metrics

| Key | Type | Meaning |
|---|---|---|
| `cases_total` | int | input size |
| `correct` | int | answer matches AND verifier passes AND realizer succeeds |
| `wrong` | int | verifier failed OR realizer failed OR answer mismatched |
| `refused` | int | parser or solver raised typed refusal |
| `correct_rate` / `wrong_rate` / `refused_rate` | float | counts ÷ total |
| `wrong_count_is_zero` | bool | the load-bearing gate |
| `overall_pass` | bool | `wrong == 0 AND correct + refused == total` |

`overall_pass` is the single line that consumers (ADR-0120 eventual
expert-tier gate; ADR-0119.8 lane-gate composer) check.

### Initial measurement (Phase 5.2 corpus, dev + public)

Run against the on-main `evals/gsm8k_math/` corpus (200 CORE-original
cases authored under ADR-0119.2):

| Split | Total | Correct | Wrong | Refused | Overall pass |
|---|---|---|---|---|---|
| dev | 50 | 50 | 0 | 0 | True |
| public | 150 | 150 | 0 | 0 | True |

**200/200 correct; zero wrong; zero refused.** Establishes the lane
baseline. ADR-0119.4 (frontier comparison) and ADR-0119.6 (depth
curve) will pair this measurement with the relevant context. ADR-0119.7
(sealed GSM8K test holdout) will eventually exercise the same runner
against the real GSM8K test set.

---

## ADR-0114a obligation discharge update

ADR-0119.3 hardens **#4 (typed refusal + wrong==0)** at the lane
layer. Previously the discipline lived in the solver (ADR-0116); now
the lane runner enforces and reports it.

| # | Obligation | Status |
|---|---|---|
| 1 | Sealed-holdout discipline | ADR-0119.1 (fab_control); 5.7 for GSM8K test |
| 2 | OOD surface variation | ADR-0118a |
| 3 | Replay-equal trace | ADR-0117 |
| 4 | Typed refusal + wrong==0 | **hardened at lane layer (this ADR)** |
| 5 | Reasoning-isolation perturbation suite | ADR-0125 |
| 6 | Compositional-depth curve | ADR-0119.6 (in flight) |
| 7 | Frontier-baseline comparison | ADR-0119.4 (in flight) |
| 8 | Adversarial generation | ADR-0119.5 (pending) |
| 9 | Determinism | discharged at solver + verifier + realizer + lane layers |
| 10 | Operation provenance via pack | ADR-0116 |

---

## Invariants

### `adr_0119_3_determinism`

Two `run_lane(cases)` calls produce byte-equal `LaneReport.canonical_bytes()`.

### `adr_0119_3_outcome_exhaustive`

Every case lands in exactly one of `correct` / `wrong` / `refused`.

### `adr_0119_3_zero_wrong_on_phase_5_2_corpus`

On the gsm8k_math/dev + public splits (200 cases authored under
ADR-0119.2), `wrong == 0`. Today: dev 50/50, public 150/150.

### `adr_0119_3_refusal_is_typed`

A graph that triggers ParseError or SolveError produces
`outcome == "refused"` with a reason that names the failing stage
(`parser: ...` or `solver: ...`).

### `adr_0119_3_correct_outcomes_carry_audit_trail`

Every `correct` case has a `trace_hash` (64-char hex) and a
`realized_prose` field set. The audit trail enables ADR-0119.4
(frontier-comparison report) and ADR-0119.6 (depth curve) to
inspect per-case reasoning without re-running the pipeline.

---

## Acceptance evidence

- `evals/gsm8k_math/runner.py` exports `run_lane`, `LaneReport`,
  `CaseOutcome`
- `tests/test_gsm8k_math_runner.py` (13 cases) green
- 200/200 correct, 0 wrong, 0 refused on the Phase 5.2 corpus
- Smoke suite green
- ADR linked from `docs/decisions/README.md` index + frontier

---

## Consequences

- Phase 5.3 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md) lands.
  Phases 5.4, 5.5, 5.6 can now run against this runner's outputs.
- The "every correct answer ships with replay-equal trace + readable
  prose" promise (ADR-0114a #3 + #10) is now mechanically inspectable
  per-case via `report.case_details[i]["trace_hash"]` and
  `["realized_prose"]`.
- The lane runner is the substrate the eventual ADR-0120 expert-tier
  gate consumes. ADR-0120 will require `overall_pass == True` on both
  the public split AND the sealed holdout.

---

## Out of scope

- Adversarial case authoring + scoring (ADR-0119.5)
- Depth-curve bucketing harness (ADR-0119.6)
- Frontier-baseline comparison report (ADR-0119.4)
- Lane registry shape + ADR-0120 numeric thresholds (ADR-0119.8 +
  ADR-0120)
- Parallel execution / case batching (current runner is single-threaded;
  small dev/public corpora don't need it. Future ADR may add a
  parallel mode that produces the same byte-equal report.)