9288688640
Composes the Phases 1-4 pipeline (parser → solver → verifier → realizer) into a per-case scoring decision: correct / wrong / refused. Outcome categorization (ADR-0114a Obligation #4): parser ParseError → refused solver SolveError → refused verifier verdict failed → wrong realizer error → wrong answer/unit mismatch → wrong all match → correct `wrong == 0` is the load-bearing gate. The lane's overall_pass holds only if wrong == 0 AND correct + refused == total. Initial measurement on the Phase 5.2 corpus: dev (50) : 50 correct, 0 wrong, 0 refused, overall_pass=True public (150) : 150 correct, 0 wrong, 0 refused, overall_pass=True Every correct case carries a trace_hash (64-char SHA-256) and realized prose — full audit trail per case, consumable by ADR-0119.4 (frontier comparison), ADR-0119.6 (depth curve), and ADR-0120 (eventual expert-tier gate). Tests: 13/13 green; 443 total green across runner + realizer + solver + verifier; 67/67 smoke green. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
167 lines
5.9 KiB
Markdown
167 lines
5.9 KiB
Markdown
# ADR-0119.3 — gsm8k_math Lane Runner (Phase 5.3)
|
||
|
||
**Status:** Accepted
|
||
**Date:** 2026-05-22
|
||
**Author:** CORE agents + reviewers
|
||
**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0118, ADR-0119, ADR-0119.1, ADR-0119.2
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
Phase 5.3 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). Composes
|
||
the Phases 1–4 pipeline (parser → solver → verifier → realizer) into a
|
||
per-case scoring decision that the GSM8K eval lane consumes.
|
||
|
||
Per ADR-0114a Obligation #4, every case must land in exactly one of
|
||
three outcomes: `correct`, `wrong`, or `refused`. The lane gates on
|
||
**`wrong == 0`** — CORE must refuse rather than confabulate. A
|
||
nonzero `wrong` count invalidates the lane regardless of `correct`
|
||
rate.
|
||
|
||
---
|
||
|
||
## Decision
|
||
|
||
### `evals/gsm8k_math/runner.py`
|
||
|
||
Exposes `run_lane(cases, *, config=None) → LaneReport` per the
|
||
framework runner interface (`evals/framework.py`). Pure function;
|
||
same input → byte-equal `LaneReport.canonical_bytes()`.
|
||
|
||
Per-case pipeline:
|
||
|
||
```text
|
||
parse_problem(text) → graph # ParseError → refused
|
||
solve(graph) → trace # SolveError → refused
|
||
verify(graph, trace) # passed=False → wrong
|
||
realize(initial_state, trace) # RealizerError → wrong
|
||
compare answer/unit to expected # mismatch → wrong; match → correct
|
||
```
|
||
|
||
`CaseOutcome` dataclass records: `case_id`, `outcome`, `reason`,
|
||
`expected_answer/unit`, `actual_answer/unit`, `trace_hash`,
|
||
`realized_prose`. The trace hash + prose are present on every
|
||
non-refused outcome, so a downstream depth-curve harness (ADR-0119.6)
|
||
or frontier-comparison report (ADR-0119.4) has a full per-case
|
||
audit trail to consume.
|
||
|
||
### Aggregate metrics
|
||
|
||
| Key | Type | Meaning |
|
||
|---|---|---|
|
||
| `cases_total` | int | input size |
|
||
| `correct` | int | answer matches AND verifier passes AND realizer succeeds |
|
||
| `wrong` | int | verifier failed OR realizer failed OR answer mismatched |
|
||
| `refused` | int | parser or solver raised typed refusal |
|
||
| `correct_rate` / `wrong_rate` / `refused_rate` | float | counts ÷ total |
|
||
| `wrong_count_is_zero` | bool | the load-bearing gate |
|
||
| `overall_pass` | bool | `wrong == 0 AND correct + refused == total` |
|
||
|
||
`overall_pass` is the single line that consumers (ADR-0120 eventual
|
||
expert-tier gate; ADR-0119.8 lane-gate composer) check.
|
||
|
||
### Initial measurement (Phase 5.2 corpus, dev + public)
|
||
|
||
Run against the on-main `evals/gsm8k_math/` corpus (200 CORE-original
|
||
cases authored under ADR-0119.2):
|
||
|
||
| Split | Total | Correct | Wrong | Refused | Overall pass |
|
||
|---|---|---|---|---|---|
|
||
| dev | 50 | 50 | 0 | 0 | True |
|
||
| public | 150 | 150 | 0 | 0 | True |
|
||
|
||
**200/200 correct; zero wrong; zero refused.** Establishes the lane
|
||
baseline. ADR-0119.4 (frontier comparison) and ADR-0119.6 (depth
|
||
curve) will pair this measurement with the relevant context. ADR-0119.7
|
||
(sealed GSM8K test holdout) will eventually exercise the same runner
|
||
against the real GSM8K test set.
|
||
|
||
---
|
||
|
||
## ADR-0114a obligation discharge update
|
||
|
||
ADR-0119.3 hardens **#4 (typed refusal + wrong==0)** at the lane
|
||
layer. Previously the discipline lived in the solver (ADR-0116); now
|
||
the lane runner enforces and reports it.
|
||
|
||
| # | Obligation | Status |
|
||
|---|---|---|
|
||
| 1 | Sealed-holdout discipline | ADR-0119.1 (fab_control); 5.7 for GSM8K test |
|
||
| 2 | OOD surface variation | ADR-0118a |
|
||
| 3 | Replay-equal trace | ADR-0117 |
|
||
| 4 | Typed refusal + wrong==0 | **hardened at lane layer (this ADR)** |
|
||
| 5 | Reasoning-isolation perturbation suite | ADR-0125 |
|
||
| 6 | Compositional-depth curve | ADR-0119.6 (in flight) |
|
||
| 7 | Frontier-baseline comparison | ADR-0119.4 (in flight) |
|
||
| 8 | Adversarial generation | ADR-0119.5 (pending) |
|
||
| 9 | Determinism | discharged at solver + verifier + realizer + lane layers |
|
||
| 10 | Operation provenance via pack | ADR-0116 |
|
||
|
||
---
|
||
|
||
## Invariants
|
||
|
||
### `adr_0119_3_determinism`
|
||
|
||
Two `run_lane(cases)` calls produce byte-equal `LaneReport.canonical_bytes()`.
|
||
|
||
### `adr_0119_3_outcome_exhaustive`
|
||
|
||
Every case lands in exactly one of `correct` / `wrong` / `refused`.
|
||
|
||
### `adr_0119_3_zero_wrong_on_phase_5_2_corpus`
|
||
|
||
On the gsm8k_math/dev + public splits (200 cases authored under
|
||
ADR-0119.2), `wrong == 0`. Today: dev 50/50, public 150/150.
|
||
|
||
### `adr_0119_3_refusal_is_typed`
|
||
|
||
A graph that triggers ParseError or SolveError produces
|
||
`outcome == "refused"` with a reason that names the failing stage
|
||
(`parser: ...` or `solver: ...`).
|
||
|
||
### `adr_0119_3_correct_outcomes_carry_audit_trail`
|
||
|
||
Every `correct` case has a `trace_hash` (64-char hex) and a
|
||
`realized_prose` field set. The audit trail enables ADR-0119.4
|
||
(frontier-comparison report) and ADR-0119.6 (depth curve) to
|
||
inspect per-case reasoning without re-running the pipeline.
|
||
|
||
---
|
||
|
||
## Acceptance evidence
|
||
|
||
- `evals/gsm8k_math/runner.py` exports `run_lane`, `LaneReport`,
|
||
`CaseOutcome`
|
||
- `tests/test_gsm8k_math_runner.py` (13 cases) green
|
||
- 200/200 correct, 0 wrong, 0 refused on the Phase 5.2 corpus
|
||
- Smoke suite green
|
||
- ADR linked from `docs/decisions/README.md` index + frontier
|
||
|
||
---
|
||
|
||
## Consequences
|
||
|
||
- Phase 5.3 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md) lands.
|
||
Phases 5.4, 5.5, 5.6 can now run against this runner's outputs.
|
||
- The "every correct answer ships with replay-equal trace + readable
|
||
prose" promise (ADR-0114a #3 + #10) is now mechanically inspectable
|
||
per-case via `report.case_details[i]["trace_hash"]` and
|
||
`["realized_prose"]`.
|
||
- The lane runner is the substrate the eventual ADR-0120 expert-tier
|
||
gate consumes. ADR-0120 will require `overall_pass == True` on both
|
||
the public split AND the sealed holdout.
|
||
|
||
---
|
||
|
||
## Out of scope
|
||
|
||
- Adversarial case authoring + scoring (ADR-0119.5)
|
||
- Depth-curve bucketing harness (ADR-0119.6)
|
||
- Frontier-baseline comparison report (ADR-0119.4)
|
||
- Lane registry shape + ADR-0120 numeric thresholds (ADR-0119.8 +
|
||
ADR-0120)
|
||
- Parallel execution / case batching (current runner is single-threaded;
|
||
small dev/public corpora don't need it. Future ADR may add a
|
||
parallel mode that produces the same byte-equal report.)
|