core/docs/decisions/ADR-0119.3-lane-runner.md

ADR-0119.3 — gsm8k_math Lane Runner (Phase 5.3)

Status: Accepted Date: 2026-05-22 Author: CORE agents + reviewers Depends on: ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0118, ADR-0119, ADR-0119.1, ADR-0119.2

---

Context

Phase 5.3 of ADR-0119. Composes the Phases 1–4 pipeline (parser → solver → verifier → realizer) into a per-case scoring decision that the GSM8K eval lane consumes.

Per ADR-0114a Obligation #4, every case must land in exactly one of three outcomes: correct, wrong, or refused. The lane gates on wrong == 0 — CORE must refuse rather than confabulate. A nonzero wrong count invalidates the lane regardless of correct rate.

---

Decision

evals/gsm8k_math/runner.py

Exposes run_lane(cases, *, config=None) → LaneReport per the framework runner interface (evals/framework.py). Pure function; same input → byte-equal LaneReport.canonical_bytes().

Per-case pipeline:

parse_problem(text) → graph     # ParseError → refused
solve(graph)        → trace     # SolveError → refused
verify(graph, trace)            # passed=False → wrong
realize(initial_state, trace)   # RealizerError → wrong
compare answer/unit to expected # mismatch → wrong; match → correct

CaseOutcome dataclass records: case_id, outcome, reason, expected_answer/unit, actual_answer/unit, trace_hash, realized_prose. The trace hash + prose are present on every non-refused outcome, so a downstream depth-curve harness (ADR-0119.6) or frontier-comparison report (ADR-0119.4) has a full per-case audit trail to consume.

Aggregate metrics

Key	Type	Meaning
cases_total	int	input size
correct	int	answer matches AND verifier passes AND realizer succeeds
wrong	int	verifier failed OR realizer failed OR answer mismatched
refused	int	parser or solver raised typed refusal
correct_rate / wrong_rate / refused_rate	float	counts ÷ total
wrong_count_is_zero	bool	the load-bearing gate
overall_pass	bool	wrong == 0 AND correct + refused == total

overall_pass is the single line that consumers (ADR-0120 eventual expert-tier gate; ADR-0119.8 lane-gate composer) check.

Initial measurement (Phase 5.2 corpus, dev + public)

Run against the on-main evals/gsm8k_math/ corpus (200 CORE-original cases authored under ADR-0119.2):

Split	Total	Correct	Wrong	Refused	Overall pass
dev	50	50	0	0	True
public	150	150	0	0	True

200/200 correct; zero wrong; zero refused. Establishes the lane baseline. ADR-0119.4 (frontier comparison) and ADR-0119.6 (depth curve) will pair this measurement with the relevant context. ADR-0119.7 (sealed GSM8K test holdout) will eventually exercise the same runner against the real GSM8K test set.

---

ADR-0114a obligation discharge update

ADR-0119.3 hardens #4 (typed refusal + wrong==0) at the lane layer. Previously the discipline lived in the solver (ADR-0116); now the lane runner enforces and reports it.

#	Obligation	Status
1	Sealed-holdout discipline	ADR-0119.1 (fab_control); 5.7 for GSM8K test
2	OOD surface variation	ADR-0118a
3	Replay-equal trace	ADR-0117
4	Typed refusal + wrong==0	hardened at lane layer (this ADR)
5	Reasoning-isolation perturbation suite	ADR-0125
6	Compositional-depth curve	ADR-0119.6 (in flight)
7	Frontier-baseline comparison	ADR-0119.4 (in flight)
8	Adversarial generation	ADR-0119.5 (pending)
9	Determinism	discharged at solver + verifier + realizer + lane layers
10	Operation provenance via pack	ADR-0116

---

Invariants

adr_0119_3_determinism

Two run_lane(cases) calls produce byte-equal LaneReport.canonical_bytes().

adr_0119_3_outcome_exhaustive

Every case lands in exactly one of correct / wrong / refused.

adr_0119_3_zero_wrong_on_phase_5_2_corpus

On the gsm8k_math/dev + public splits (200 cases authored under ADR-0119.2), wrong == 0. Today: dev 50/50, public 150/150.

adr_0119_3_refusal_is_typed

A graph that triggers ParseError or SolveError produces outcome == "refused" with a reason that names the failing stage (parser: ... or solver: ...).

adr_0119_3_correct_outcomes_carry_audit_trail

Every correct case has a trace_hash (64-char hex) and a realized_prose field set. The audit trail enables ADR-0119.4 (frontier-comparison report) and ADR-0119.6 (depth curve) to inspect per-case reasoning without re-running the pipeline.

---

Acceptance evidence

• evals/gsm8k_math/runner.py exports run_lane, LaneReport, CaseOutcome
• tests/test_gsm8k_math_runner.py (13 cases) green
• 200/200 correct, 0 wrong, 0 refused on the Phase 5.2 corpus
• Smoke suite green
• ADR linked from docs/decisions/README.md index + frontier

---

Consequences

• Phase 5.3 of ADR-0119 lands. Phases 5.4, 5.5, 5.6 can now run against this runner's outputs.
• The "every correct answer ships with replay-equal trace + readable prose" promise (ADR-0114a #3 + #10) is now mechanically inspectable per-case via report.case_details[i]["trace_hash"] and ["realized_prose"].
• The lane runner is the substrate the eventual ADR-0120 expert-tier gate consumes. ADR-0120 will require overall_pass == True on both the public split AND the sealed holdout.

---

Out of scope

• Adversarial case authoring + scoring (ADR-0119.5)
• Depth-curve bucketing harness (ADR-0119.6)
• Frontier-baseline comparison report (ADR-0119.4)
• Lane registry shape + ADR-0120 numeric thresholds (ADR-0119.8 + ADR-0120)
• Parallel execution / case batching (current runner is single-threaded; small dev/public corpora don't need it. Future ADR may add a parallel mode that produces the same byte-equal report.)