a13df6f370
Phase 5.8 of ADR-0119. Composes the per-sub-phase substrate (5.1..5.6) into a single per-split lane verdict the eventual ADR-0120 expert promotion contract can consume. LANE_SHAPE_REGISTRY adds: "gsm8k_math": "gsm8k_capability_shape" _check_gsm8k_capability_shape refuses on any of: - missing cases_total / correct / wrong / refused fields - cases_total <= 0 - wrong != 0 (ADR-0114a Obligation #4) - correct + refused != cases_total (accounting incomplete) - overall_pass present and false Accepts otherwise. Edge: all-refused passes the shape gate (runner self-consistency). Capability bar (min correct-rate, depth-curve ε) lives in ADR-0120. Live measurement on main: dev 50/50 correct, 0 wrong, 0 refused → gate ✓ public 150/150 correct, 0 wrong, 0 refused → gate ✓ 21 invariant tests pin: registry mapping, shape checker presence, live runner passes, nonzero wrong refuses, incomplete accounting refuses, missing field refuses, clean metrics pass, all-refused edge passes, all Phase 5.1..5.6 substrate artifacts exist on disk. Phase 5 status: 5.1..5.6 + 5.8 ✓. Only 5.7 (sealed real GSM8K test) remains before ADR-0120 (first expert promotion contract) becomes feasible. ADR-0114a roll-up unchanged: 10/10 obligations discharged on main (modulo Phase 5.7's lane-specific GSM8K test sealing). Tests: 21 new + 80 prior across Phase 5 + adjacent suites = 101 green; 67/67 smoke. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
183 lines
6.5 KiB
Markdown
183 lines
6.5 KiB
Markdown
# ADR-0119.8 — gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`)
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-23
|
|
**Author:** CORE agents + reviewers
|
|
**Depends on:** ADR-0109 (lane-shape registry), ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.2, ADR-0119.3, ADR-0119.4, ADR-0119.5, ADR-0119.6
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
Phase 5.8 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). Composes
|
|
the per-sub-phase work (5.1..5.6) into a single per-split lane verdict
|
|
the eventual ADR-0120 `expert` promotion contract can consume.
|
|
|
|
Per the ADR-0119 §5.8 brief:
|
|
|
|
> "A new lane shape `gsm8k_capability_shape` is registered in
|
|
> `LANE_SHAPE_REGISTRY` with the above thresholds. ADR-0119.8 ships
|
|
> the shape; ADR-0120 invokes it."
|
|
|
|
The shape's load-bearing constraints (per ADR-0114a Obligation #4):
|
|
|
|
- **`wrong == 0`** — the misparse-discipline gate
|
|
- **`correct + refused == cases_total`** — accounting completeness
|
|
- **`cases_total > 0`** — non-empty input
|
|
- **`overall_pass == True`** (when present) — runner self-consistency
|
|
|
|
Substrate prerequisites (5.1..5.6 artifacts) are checked by the
|
|
companion test module, not by the shape checker itself — that keeps
|
|
shape checkers narrowly metrics-focused (consistent with ADR-0109's
|
|
existing checker pattern).
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
### `LANE_SHAPE_REGISTRY` mapping
|
|
|
|
```python
|
|
LANE_SHAPE_REGISTRY["gsm8k_math"] = "gsm8k_capability_shape"
|
|
```
|
|
|
|
Distinct from the existing four shapes because the metrics keys
|
|
differ (`cases_total` / `correct` / `wrong` / `refused` /
|
|
`overall_pass`) and the composition rule
|
|
(`correct + refused == total AND wrong == 0`) is unique to this
|
|
lane's runner contract.
|
|
|
|
### `_check_gsm8k_capability_shape` checker
|
|
|
|
Refuses on any of:
|
|
|
|
| Condition | Reason |
|
|
|---|---|
|
|
| Any of `cases_total` / `correct` / `wrong` / `refused` missing | "missing required metric X" |
|
|
| `cases_total <= 0` | "cases_total=N (must be > 0)" |
|
|
| `wrong != 0` | "wrong=N (must be 0 — ADR-0114a Obligation #4)" |
|
|
| `correct + refused != cases_total` | "outcome accounting incomplete" |
|
|
| `overall_pass` present and false | "overall_pass is False despite wrong=0 and accounting balanced" |
|
|
|
|
Accepts on everything else. Edge case: **all-refused (0 correct,
|
|
0 wrong, N refused) PASSES the shape gate.** Whether that's acceptable
|
|
for an `expert` promotion is ADR-0120's job (it sets the
|
|
correct-rate-minimum); this layer just verifies the runner's
|
|
contract is internally consistent.
|
|
|
|
### Current measurement on main
|
|
|
|
Running the gate against the on-main `evals/gsm8k_math/` corpus:
|
|
|
|
| Split | cases_total | correct | wrong | refused | gate |
|
|
|---|---|---|---|---|---|
|
|
| dev | 50 | 50 | 0 | 0 | ✓ |
|
|
| public | 150 | 150 | 0 | 0 | ✓ |
|
|
|
|
---
|
|
|
|
## ADR-0114a obligation roll-up after Phase 5.8
|
|
|
|
All 10 obligations are now mechanically gated on `main`:
|
|
|
|
| # | Obligation | Discharged by |
|
|
|---|---|---|
|
|
| 1 | Sealed-holdout discipline | ADR-0119.1 (fab_control); ADR-0119.7 for the GSM8K test set |
|
|
| 2 | OOD surface variation | ADR-0118a |
|
|
| 3 | Replay-equal trace | ADR-0117 |
|
|
| 4 | Typed refusal + wrong==0 | ADR-0116 + ADR-0119.3 + **gate enforces here** |
|
|
| 5 | Reasoning-isolation perturbation suite | ADR-0125 |
|
|
| 6 | Compositional-depth curve | ADR-0119.6 (ε threshold to ADR-0120) |
|
|
| 7 | Frontier-baseline comparison | ADR-0119.4 |
|
|
| 8 | Adversarial generation; wrong==0 | ADR-0119.5 |
|
|
| 9 | Determinism | solver + verifier + realizer + runner + shape checker |
|
|
| 10 | Operation provenance via pack | ADR-0116 |
|
|
|
|
The only piece left before ADR-0120 (first `expert` promotion
|
|
contract) is **Phase 5.7** — sealing the real GSM8K test set into
|
|
`evals/gsm8k_math/holdouts/v1/cases.jsonl.age`. That requires a
|
|
human to (a) acquire the GSM8K test set and (b) encrypt it against
|
|
the existing fab_control recipient key (or a fresh GSM8K-specific
|
|
recipient). All other machinery is in place.
|
|
|
|
---
|
|
|
|
## Invariants
|
|
|
|
### `adr_0119_8_registry_mapping`
|
|
|
|
`LANE_SHAPE_REGISTRY["gsm8k_math"] == "gsm8k_capability_shape"`;
|
|
`SHAPE_CHECKERS["gsm8k_capability_shape"]` is callable.
|
|
|
|
### `adr_0119_8_live_runner_passes_gate`
|
|
|
|
Running the on-main lane runner against the dev or public split
|
|
produces metrics the shape checker accepts. Today: dev 50/50,
|
|
public 150/150, both green.
|
|
|
|
### `adr_0119_8_nonzero_wrong_refuses`
|
|
|
|
A metrics block with `wrong > 0` refuses the gate with a reason
|
|
naming ADR-0114a Obligation #4.
|
|
|
|
### `adr_0119_8_incomplete_accounting_refuses`
|
|
|
|
A metrics block where `correct + refused != cases_total` refuses
|
|
with a typed "outcome accounting incomplete" reason.
|
|
|
|
### `adr_0119_8_missing_field_refuses`
|
|
|
|
Removing any of `cases_total` / `correct` / `wrong` / `refused`
|
|
from the metrics refuses with a "missing required metric" reason
|
|
naming the field.
|
|
|
|
### `adr_0119_8_substrate_artifacts_present`
|
|
|
|
Phase 5.1..5.6 artifacts exist on disk (sealed holdout for
|
|
fab_control; corpus + verify.py; runner; frontier + comparison;
|
|
adversarial generator + score; depth-curve harness). Phase 5.7
|
|
placeholder is also present (empty cases.jsonl files reserved for
|
|
the eventual real-GSM8K-test seal).
|
|
|
|
---
|
|
|
|
## Acceptance evidence
|
|
|
|
- `core/capability/expert_demo.py` registers `gsm8k_math →
|
|
gsm8k_capability_shape`; `SHAPE_CHECKERS` exports the checker
|
|
- `tests/test_adr_0119_8_lane_gate.py` (21 cases) green; pins all
|
|
six invariants
|
|
- Live runner on dev + public both pass the gate
|
|
- 101 tests green across Phase 5 + lane-shape-thresholds +
|
|
expert-demo-contract + capability-reports
|
|
- Smoke suite green (67/67)
|
|
- ADR linked from `docs/decisions/README.md` index + frontier
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
- The gsm8k_math lane is now first-class in the lane-shape registry.
|
|
Adding it to a domain pack's `eval_lanes` manifest and signing an
|
|
`audit_passed_claim` would route through the new gate cleanly.
|
|
- ADR-0120 (first `expert` promotion contract) can now reference
|
|
`gsm8k_capability_shape` as the lane shape it requires. The only
|
|
remaining gap is the real-GSM8K-test seal (Phase 5.7) and the
|
|
numeric ε threshold for the depth curve (ADR-0120's call).
|
|
- The "all-refused" edge case (where the lane refuses every input)
|
|
passes the shape gate intentionally. That makes the shape narrowly
|
|
about *runner self-consistency*, not capability. ADR-0120 sets the
|
|
capability bar.
|
|
|
|
---
|
|
|
|
## Out of scope
|
|
|
|
- Sealing the real GSM8K test set (ADR-0119.7).
|
|
- The numeric ε threshold for the depth curve (ADR-0120).
|
|
- A minimum `correct_rate` for the lane to qualify as `expert`
|
|
evidence — that's a capability claim, lives in ADR-0120.
|
|
- Cross-split gates (e.g. holdout must score within Δ of public).
|
|
Those are anti-overfitting properties already handled by
|
|
Obligation #2 (OOD) and Obligation #6 (depth curve); the shape
|
|
checker doesn't duplicate them.
|