core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/decisions/ADR-0119.8-lane-gate.md
Shay a13df6f370 feat: ADR-0119.8 — gsm8k_math overall lane gate (gsm8k_capability_shape) 
Phase 5.8 of ADR-0119. Composes the per-sub-phase substrate
(5.1..5.6) into a single per-split lane verdict the eventual
ADR-0120 expert promotion contract can consume.

LANE_SHAPE_REGISTRY adds:
  "gsm8k_math": "gsm8k_capability_shape"

_check_gsm8k_capability_shape refuses on any of:
  - missing cases_total / correct / wrong / refused fields
  - cases_total <= 0
  - wrong != 0                          (ADR-0114a Obligation #4)
  - correct + refused != cases_total    (accounting incomplete)
  - overall_pass present and false

Accepts otherwise. Edge: all-refused passes the shape gate (runner
self-consistency). Capability bar (min correct-rate, depth-curve
ε) lives in ADR-0120.

Live measurement on main:
  dev    50/50 correct, 0 wrong, 0 refused  → gate ✓
  public 150/150 correct, 0 wrong, 0 refused → gate ✓

21 invariant tests pin: registry mapping, shape checker presence,
live runner passes, nonzero wrong refuses, incomplete accounting
refuses, missing field refuses, clean metrics pass, all-refused
edge passes, all Phase 5.1..5.6 substrate artifacts exist on disk.

Phase 5 status: 5.1..5.6 + 5.8 ✓. Only 5.7 (sealed real GSM8K
test) remains before ADR-0120 (first expert promotion contract)
becomes feasible.

ADR-0114a roll-up unchanged: 10/10 obligations discharged on main
(modulo Phase 5.7's lane-specific GSM8K test sealing).

Tests: 21 new + 80 prior across Phase 5 + adjacent suites = 101
green; 67/67 smoke.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 19:45:44 -07:00

6.5 KiB
Raw Blame History

ADR-0119.8 — gsm8k_math Overall Lane Gate (gsm8k_capability_shape)

Status: Accepted Date: 2026-05-23 Author: CORE agents + reviewers Depends on: ADR-0109 (lane-shape registry), ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.2, ADR-0119.3, ADR-0119.4, ADR-0119.5, ADR-0119.6

Context

Phase 5.8 of ADR-0119. Composes the per-sub-phase work (5.1..5.6) into a single per-split lane verdict the eventual ADR-0120 expert promotion contract can consume.

Per the ADR-0119 §5.8 brief:

"A new lane shape gsm8k_capability_shape is registered in LANE_SHAPE_REGISTRY with the above thresholds. ADR-0119.8 ships the shape; ADR-0120 invokes it."

The shape's load-bearing constraints (per ADR-0114a Obligation #4):

  • wrong == 0 — the misparse-discipline gate
  • correct + refused == cases_total — accounting completeness
  • cases_total > 0 — non-empty input
  • overall_pass == True (when present) — runner self-consistency

Substrate prerequisites (5.1..5.6 artifacts) are checked by the companion test module, not by the shape checker itself — that keeps shape checkers narrowly metrics-focused (consistent with ADR-0109's existing checker pattern).

Decision

LANE_SHAPE_REGISTRY mapping

LANE_SHAPE_REGISTRY["gsm8k_math"] = "gsm8k_capability_shape"

Distinct from the existing four shapes because the metrics keys differ (cases_total / correct / wrong / refused / overall_pass) and the composition rule (correct + refused == total AND wrong == 0) is unique to this lane's runner contract.

_check_gsm8k_capability_shape checker

Refuses on any of:

Condition Reason
Any of cases_total / correct / wrong / refused missing "missing required metric X"
cases_total <= 0 "cases_total=N (must be > 0)"
wrong != 0 "wrong=N (must be 0 — ADR-0114a Obligation #4)"
correct + refused != cases_total "outcome accounting incomplete"
overall_pass present and false "overall_pass is False despite wrong=0 and accounting balanced"

Accepts on everything else. Edge case: all-refused (0 correct, 0 wrong, N refused) PASSES the shape gate. Whether that's acceptable for an expert promotion is ADR-0120's job (it sets the correct-rate-minimum); this layer just verifies the runner's contract is internally consistent.

Current measurement on main

Running the gate against the on-main evals/gsm8k_math/ corpus:

Split cases_total correct wrong refused gate
dev 50 50 0 0 ✓
public 150 150 0 0 ✓

ADR-0114a obligation roll-up after Phase 5.8

All 10 obligations are now mechanically gated on main:

# Obligation Discharged by
1 Sealed-holdout discipline ADR-0119.1 (fab_control); ADR-0119.7 for the GSM8K test set
2 OOD surface variation ADR-0118a
3 Replay-equal trace ADR-0117
4 Typed refusal + wrong==0 ADR-0116 + ADR-0119.3 + gate enforces here
5 Reasoning-isolation perturbation suite ADR-0125
6 Compositional-depth curve ADR-0119.6 (ε threshold to ADR-0120)
7 Frontier-baseline comparison ADR-0119.4
8 Adversarial generation; wrong==0 ADR-0119.5
9 Determinism solver + verifier + realizer + runner + shape checker
10 Operation provenance via pack ADR-0116

The only piece left before ADR-0120 (first expert promotion contract) is Phase 5.7 — sealing the real GSM8K test set into evals/gsm8k_math/holdouts/v1/cases.jsonl.age. That requires a human to (a) acquire the GSM8K test set and (b) encrypt it against the existing fab_control recipient key (or a fresh GSM8K-specific recipient). All other machinery is in place.

Invariants

adr_0119_8_registry_mapping

LANE_SHAPE_REGISTRY["gsm8k_math"] == "gsm8k_capability_shape"; SHAPE_CHECKERS["gsm8k_capability_shape"] is callable.

adr_0119_8_live_runner_passes_gate

Running the on-main lane runner against the dev or public split produces metrics the shape checker accepts. Today: dev 50/50, public 150/150, both green.

adr_0119_8_nonzero_wrong_refuses

A metrics block with wrong > 0 refuses the gate with a reason naming ADR-0114a Obligation #4.

adr_0119_8_incomplete_accounting_refuses

A metrics block where correct + refused != cases_total refuses with a typed "outcome accounting incomplete" reason.

adr_0119_8_missing_field_refuses

Removing any of cases_total / correct / wrong / refused from the metrics refuses with a "missing required metric" reason naming the field.

adr_0119_8_substrate_artifacts_present

Phase 5.1..5.6 artifacts exist on disk (sealed holdout for fab_control; corpus + verify.py; runner; frontier + comparison; adversarial generator + score; depth-curve harness). Phase 5.7 placeholder is also present (empty cases.jsonl files reserved for the eventual real-GSM8K-test seal).

Acceptance evidence

  • core/capability/expert_demo.py registers gsm8k_math → gsm8k_capability_shape; SHAPE_CHECKERS exports the checker
  • tests/test_adr_0119_8_lane_gate.py (21 cases) green; pins all six invariants
  • Live runner on dev + public both pass the gate
  • 101 tests green across Phase 5 + lane-shape-thresholds + expert-demo-contract + capability-reports
  • Smoke suite green (67/67)
  • ADR linked from docs/decisions/README.md index + frontier

Consequences

  • The gsm8k_math lane is now first-class in the lane-shape registry. Adding it to a domain pack's eval_lanes manifest and signing an audit_passed_claim would route through the new gate cleanly.
  • ADR-0120 (first expert promotion contract) can now reference gsm8k_capability_shape as the lane shape it requires. The only remaining gap is the real-GSM8K-test seal (Phase 5.7) and the numeric ε threshold for the depth curve (ADR-0120's call).
  • The "all-refused" edge case (where the lane refuses every input) passes the shape gate intentionally. That makes the shape narrowly about runner self-consistency, not capability. ADR-0120 sets the capability bar.

Out of scope

  • Sealing the real GSM8K test set (ADR-0119.7).
  • The numeric ε threshold for the depth curve (ADR-0120).
  • A minimum correct_rate for the lane to qualify as expert evidence — that's a capability claim, lives in ADR-0120.
  • Cross-split gates (e.g. holdout must score within Δ of public). Those are anti-overfitting properties already handled by Obligation #2 (OOD) and Obligation #6 (depth curve); the shape checker doesn't duplicate them.