Shay 801287bba6 feat(ADR-0131.G.0): switch GSM8K coverage probe to candidate-graph pipeline

Zero behavior delta on the main baseline (both substrates produce
0/50 admission today) — but every subsequent ADR-0131.G.<n> iteration
now produces attributable admission deltas on the probe, instead of
silently extending a parser layer the probe wasn't measuring.

Background: ADR-0131.G's probe consulted run_lane → _score_one →
parse_problem (legacy first-match-wins parser, pre-ADR-0126). Every
G.<n> iteration extends the candidate-graph parser via
_score_one_candidate_graph → parse_and_solve. The mismatch was
discovered during G.3 development and explicitly reserved as this
follow-up.

Changes:
  - run_coverage_probe.py: switch import to _score_one_candidate_graph;
    new private _score_lane aggregator mirrors run_lane's output shape
    via per-case scoring; report root adds "substrate": "candidate_graph"
    for audit trail.
  - train_sample_coverage_report.json: regenerated. All metrics
    byte-identical to prior baseline (0/50 admission, wrong=0).
    refused_reasons_top text differs (candidate_graph: prefix instead
    of parser:) — expected and part of the substrate audit-trail shift.

Discipline: separate small PR per ADR-0131.G's "expansion that only
moves admission must be a standalone PR" principle. Substrate swap
attributable; future G.<n> deltas attributable.

Evidence:
  - python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe
    → admission 0/50, wrong=0, safety_rail_intact=True, exit 0
  - pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py
    → 8/8 pass in 0.18s (no test edits needed; tests pin invariants
    not numbers)
  - No changes to runner.py, no changes to any G.<n> work in flight.

Effect on in-flight iterations: each G.<n> PR (G.1 Gemini / G.2 #182 /
G.3 #183 / G.4 Opus#2) rebases after this lands and refreshes its
committed train_sample_coverage_report.json with the new substrate's
numbers. Rebase is mechanical.

2026-05-23 14:43:05 -07:00

7.3 KiB

Raw Blame History

ADR-0131.G.0 — Probe Substrate: Candidate-Graph Pipeline

Status: Proposed Date: 2026-05-23 Author: CORE main agent (Opus 4.7) Depends on: ADR-0126 (candidate-graph parser), ADR-0131.G (GSM8K coverage probe + iteration discipline) Parent: ADR-0131.G Sibling iterations (in flight): ADR-0131.G.1 (verbs), ADR-0131.G.2 (comparatives), ADR-0131.G.3 (numerics), ADR-0131.G.4 (multi-clause)

Context

ADR-0131.G (PR #181) pinned a GSM8K coverage probe as a diff-able admission number with capability-first iteration discipline. Each subsequent ADR-0131.G. extends a single capability axis of the parser layer and re-runs the probe, with the gate that admission_rate strictly increases (or a refused-reason family deliberately shrinks).

But the probe currently consults evals.gsm8k_math.runner.run_lane, which scores each case via _score_one → parse_problem — the legacy first-match-wins parser that pre-dates ADR-0126. Every ADR-0131.G. iteration extends the candidate-graph parser (ADR-0126 topology), via _score_one_candidate_graph → parse_and_solve. The probe and the iteration work measure two different parsers.

The mismatch was discovered during ADR-0131.G.3 development: the G.3 axis lane (which routes through the candidate-graph pipeline via _score_one_candidate_graph) showed full end-to-end correctness on 20 curated money + hyphenated-cardinal cases, while the GSM8K probe (legacy substrate) showed zero admission delta. The G.3 ADR documented the limit honestly and reserved this follow-up.

Decision

Switch evals/gsm8k_math/train_sample/v1/run_coverage_probe.py to call _score_one_candidate_graph instead of run_lane. The probe now measures the parser layer the iteration discipline extends.

Mechanically, the probe builds its own thin _score_lane aggregator that mirrors run_lane's shape (cases_total / correct / wrong / refused / rates / wrong_count_is_zero) but invokes _score_one_candidate_graph per case. No changes to runner.py itself.

Report schema gains one field — "substrate": "candidate_graph" — so the audit trail records which pipeline produced the numbers. Every other field (metrics, refused_reasons_top, per_case) keeps its shape; the 8 contract tests in tests/test_adr_0131_G_gsm8k_coverage_probe.py pass without modification.

Why this is a separate (small) PR

ADR-0131.G's discipline explicitly forbids bundling probe-infra changes into capability-axis PRs ("Reject any expansion that only moves GSM8K admission and not B3 / curated coverage — that's the smell test for a template in disguise"). The mirror image applies here: a substrate swap that moves admission numbers must be a standalone PR so the delta is attributable to the substrate change, not silently absorbed into a capability iteration.

Today the substrate swap is zero-delta — both _score_one (legacy) and _score_one_candidate_graph produce 0/50 admission on the current main baseline. That makes this the cleanest-possible substrate PR: behavior unchanged, audit trail shifted, future iterations attributable.

What changed in code

`evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`

Import switched: run_lane → _score_one_candidate_graph (and the existing LaneReport no longer needed at the probe layer).
New private _score_lane aggregator mirrors run_lane's output shape via _score_one_candidate_graph per case.
build_report adds "substrate": "candidate_graph" to the report root.
Docstring updated to document the substrate choice.

`evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`

Regenerated. Top-level "substrate" field added. All metrics byte-identical to the prior baseline (both substrates produce the same 0/50 numbers today). refused_reasons_top text changes because the candidate-graph pipeline emits slightly different refusal strings (prefix candidate_graph: instead of parser:); this is expected and is part of the substrate audit-trail shift.

No changes to

evals/gsm8k_math/runner.py — run_lane, _score_one, _score_one_candidate_graph all unchanged.
tests/test_adr_0131_G_gsm8k_coverage_probe.py — the 8 contract tests pin invariants, not specific numbers, and all pass after the substrate swap.
Any G. iteration's lane code or axis cases.

Evidence

Probe: python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe → admission_rate: 0/50 = 0.0%, wrong: 0, safety_rail_intact: True, exit code 0.
Contract tests: 8/8 pass in 0.18s.
Substrate equivalence on main baseline: both pipelines produce 0/50 admission. The substrate swap is zero behavior delta today.

Effect on in-flight iterations

Iteration	Effect of this PR
ADR-0131.G.1 (verbs, Gemini)	After landing, probe admission will rise on cases the new verbs unlock (e.g., `Nicole collected 400 Pokemon cards`).
ADR-0131.G.2 (comparatives, PR #182)	Refused-reason family for comparatives moves at the probe layer for the first time.
ADR-0131.G.3 (numerics, PR #183)	Money + hyphenated cases the candidate-graph pipeline already admits start showing as probe admission (combined with G.1 unlocks `Tina makes $18.00 an hour` etc.).
ADR-0131.G.4 (multi-clause, Opus#2)	Conjoined-subject cases like `Aaron and his brother Carson each saved up $40` become probe-admittable (combined with G.3 for the money literal).

Reconciliation order after merge: each G. PR rebases onto main after this PR lands, runs the probe, refreshes its committed train_sample_coverage_report.json. The refreshed report becomes the iteration's load-bearing evidence on the probe, complementing the axis lane.

What this does NOT do

Does NOT change the legacy parser path. parse_problem and _score_one remain available for other lanes that depend on them.
Does NOT change the GSM8K case set, the sample, or the answer vocabulary.
Does NOT change the GSM8K probe's role: it remains a probe (honest disclosure of what the architecture can/can't admit), not a gate for the math-expert promotion (that's still B1+B2+B3 per ADR-0131).
Does NOT modify any G. capability work in flight. Each rebase is mechanical (one merged report file refresh).
Does NOT introduce stochastic behavior, opaque fallbacks, or approximate recall anywhere in the probe.

CLAUDE.md PR-checklist answers

Capability / performance / security added or protected: Aligns the probe's measurement substrate with the parser layer the iteration discipline extends. Audit-trail correctness for every future capability iteration.
Invariant proving the field remains valid: admitted_wrong == 0 preserved on the probe; safety_rail_intact: True; 8/8 contract tests pass.
CLI/eval lane proving the lane: python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe and pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py.
Avoided hidden normalization / stochastic / approximate / unreviewed mutation: Yes. Pure deterministic substrate swap, same case set, same scoring rules, different parser entry point.
Trust boundary: No new user-input surface, no new filesystem paths, no new dynamic imports. The probe still reads only cases.jsonl and writes only train_sample_coverage_report.json under evals/gsm8k_math/train_sample/v1/.

7.3 KiB Raw Blame History