# ADR-0119.8 — gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) **Status:** Accepted **Date:** 2026-05-23 **Author:** CORE agents + reviewers **Depends on:** ADR-0109 (lane-shape registry), ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.2, ADR-0119.3, ADR-0119.4, ADR-0119.5, ADR-0119.6 --- ## Context Phase 5.8 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). Composes the per-sub-phase work (5.1..5.6) into a single per-split lane verdict the eventual ADR-0120 `expert` promotion contract can consume. Per the ADR-0119 §5.8 brief: > "A new lane shape `gsm8k_capability_shape` is registered in > `LANE_SHAPE_REGISTRY` with the above thresholds. ADR-0119.8 ships > the shape; ADR-0120 invokes it." The shape's load-bearing constraints (per ADR-0114a Obligation #4): - **`wrong == 0`** — the misparse-discipline gate - **`correct + refused == cases_total`** — accounting completeness - **`cases_total > 0`** — non-empty input - **`overall_pass == True`** (when present) — runner self-consistency Substrate prerequisites (5.1..5.6 artifacts) are checked by the companion test module, not by the shape checker itself — that keeps shape checkers narrowly metrics-focused (consistent with ADR-0109's existing checker pattern). --- ## Decision ### `LANE_SHAPE_REGISTRY` mapping ```python LANE_SHAPE_REGISTRY["gsm8k_math"] = "gsm8k_capability_shape" ``` Distinct from the existing four shapes because the metrics keys differ (`cases_total` / `correct` / `wrong` / `refused` / `overall_pass`) and the composition rule (`correct + refused == total AND wrong == 0`) is unique to this lane's runner contract. ### `_check_gsm8k_capability_shape` checker Refuses on any of: | Condition | Reason | |---|---| | Any of `cases_total` / `correct` / `wrong` / `refused` missing | "missing required metric X" | | `cases_total <= 0` | "cases_total=N (must be > 0)" | | `wrong != 0` | "wrong=N (must be 0 — ADR-0114a Obligation #4)" | | `correct + refused != cases_total` | "outcome accounting incomplete" | | `overall_pass` present and false | "overall_pass is False despite wrong=0 and accounting balanced" | Accepts on everything else. Edge case: **all-refused (0 correct, 0 wrong, N refused) PASSES the shape gate.** Whether that's acceptable for an `expert` promotion is ADR-0120's job (it sets the correct-rate-minimum); this layer just verifies the runner's contract is internally consistent. ### Current measurement on main Running the gate against the on-main `evals/gsm8k_math/` corpus: | Split | cases_total | correct | wrong | refused | gate | |---|---|---|---|---|---| | dev | 50 | 50 | 0 | 0 | ✓ | | public | 150 | 150 | 0 | 0 | ✓ | --- ## ADR-0114a obligation roll-up after Phase 5.8 All 10 obligations are now mechanically gated on `main`: | # | Obligation | Discharged by | |---|---|---| | 1 | Sealed-holdout discipline | ADR-0119.1 (fab_control); ADR-0119.7 for the GSM8K test set | | 2 | OOD surface variation | ADR-0118a | | 3 | Replay-equal trace | ADR-0117 | | 4 | Typed refusal + wrong==0 | ADR-0116 + ADR-0119.3 + **gate enforces here** | | 5 | Reasoning-isolation perturbation suite | ADR-0125 | | 6 | Compositional-depth curve | ADR-0119.6 (ε threshold to ADR-0120) | | 7 | Frontier-baseline comparison | ADR-0119.4 | | 8 | Adversarial generation; wrong==0 | ADR-0119.5 | | 9 | Determinism | solver + verifier + realizer + runner + shape checker | | 10 | Operation provenance via pack | ADR-0116 | The only piece left before ADR-0120 (first `expert` promotion contract) is **Phase 5.7** — sealing the real GSM8K test set into `evals/gsm8k_math/holdouts/v1/cases.jsonl.age`. That requires a human to (a) acquire the GSM8K test set and (b) encrypt it against the existing fab_control recipient key (or a fresh GSM8K-specific recipient). All other machinery is in place. --- ## Invariants ### `adr_0119_8_registry_mapping` `LANE_SHAPE_REGISTRY["gsm8k_math"] == "gsm8k_capability_shape"`; `SHAPE_CHECKERS["gsm8k_capability_shape"]` is callable. ### `adr_0119_8_live_runner_passes_gate` Running the on-main lane runner against the dev or public split produces metrics the shape checker accepts. Today: dev 50/50, public 150/150, both green. ### `adr_0119_8_nonzero_wrong_refuses` A metrics block with `wrong > 0` refuses the gate with a reason naming ADR-0114a Obligation #4. ### `adr_0119_8_incomplete_accounting_refuses` A metrics block where `correct + refused != cases_total` refuses with a typed "outcome accounting incomplete" reason. ### `adr_0119_8_missing_field_refuses` Removing any of `cases_total` / `correct` / `wrong` / `refused` from the metrics refuses with a "missing required metric" reason naming the field. ### `adr_0119_8_substrate_artifacts_present` Phase 5.1..5.6 artifacts exist on disk (sealed holdout for fab_control; corpus + verify.py; runner; frontier + comparison; adversarial generator + score; depth-curve harness). Phase 5.7 placeholder is also present (empty cases.jsonl files reserved for the eventual real-GSM8K-test seal). --- ## Acceptance evidence - `core/capability/expert_demo.py` registers `gsm8k_math → gsm8k_capability_shape`; `SHAPE_CHECKERS` exports the checker - `tests/test_adr_0119_8_lane_gate.py` (21 cases) green; pins all six invariants - Live runner on dev + public both pass the gate - 101 tests green across Phase 5 + lane-shape-thresholds + expert-demo-contract + capability-reports - Smoke suite green (67/67) - ADR linked from `docs/decisions/README.md` index + frontier --- ## Consequences - The gsm8k_math lane is now first-class in the lane-shape registry. Adding it to a domain pack's `eval_lanes` manifest and signing an `audit_passed_claim` would route through the new gate cleanly. - ADR-0120 (first `expert` promotion contract) can now reference `gsm8k_capability_shape` as the lane shape it requires. The only remaining gap is the real-GSM8K-test seal (Phase 5.7) and the numeric ε threshold for the depth curve (ADR-0120's call). - The "all-refused" edge case (where the lane refuses every input) passes the shape gate intentionally. That makes the shape narrowly about *runner self-consistency*, not capability. ADR-0120 sets the capability bar. --- ## Out of scope - Sealing the real GSM8K test set (ADR-0119.7). - The numeric ε threshold for the depth curve (ADR-0120). - A minimum `correct_rate` for the lane to qualify as `expert` evidence — that's a capability claim, lives in ADR-0120. - Cross-split gates (e.g. holdout must score within Δ of public). Those are anti-overfitting properties already handled by Obligation #2 (OOD) and Obligation #6 (depth curve); the shape checker doesn't duplicate them.