core/docs/adr/ADR-0131.G.0-probe-substrate.md

ADR-0131.G.0 — Probe Substrate: Candidate-Graph Pipeline

Status: Proposed Date: 2026-05-23 Author: CORE main agent (Opus 4.7) Depends on: ADR-0126 (candidate-graph parser), ADR-0131.G (GSM8K coverage probe + iteration discipline) Parent: ADR-0131.G Sibling iterations (in flight): ADR-0131.G.1 (verbs), ADR-0131.G.2 (comparatives), ADR-0131.G.3 (numerics), ADR-0131.G.4 (multi-clause)

---

Context

ADR-0131.G (PR #181) pinned a GSM8K coverage probe as a diff-able admission number with capability-first iteration discipline. Each subsequent ADR-0131.G. extends a single capability axis of the parser layer and re-runs the probe, with the gate that admission_rate strictly increases (or a refused-reason family deliberately shrinks).

But the probe currently consults evals.gsm8k_math.runner.run_lane, which scores each case via _score_one → parse_problem — the legacy first-match-wins parser that pre-dates ADR-0126. Every ADR-0131.G. iteration extends the candidate-graph parser (ADR-0126 topology), via _score_one_candidate_graph → parse_and_solve. The probe and the iteration work measure two different parsers.

The mismatch was discovered during ADR-0131.G.3 development: the G.3 axis lane (which routes through the candidate-graph pipeline via _score_one_candidate_graph) showed full end-to-end correctness on 20 curated money + hyphenated-cardinal cases, while the GSM8K probe (legacy substrate) showed zero admission delta. The G.3 ADR documented the limit honestly and reserved this follow-up.

Decision

Switch evals/gsm8k_math/train_sample/v1/run_coverage_probe.py to call _score_one_candidate_graph instead of run_lane. The probe now measures the parser layer the iteration discipline extends.

Mechanically, the probe builds its own thin _score_lane aggregator that mirrors run_lane's shape (cases_total / correct / wrong / refused / rates / wrong_count_is_zero) but invokes _score_one_candidate_graph per case. No changes to runner.py itself.

Report schema gains one field — "substrate": "candidate_graph" — so the audit trail records which pipeline produced the numbers. Every other field (metrics, refused_reasons_top, per_case) keeps its shape; the 8 contract tests in tests/test_adr_0131_G_gsm8k_coverage_probe.py pass without modification.

Why this is a separate (small) PR

ADR-0131.G's discipline explicitly forbids bundling probe-infra changes into capability-axis PRs ("Reject any expansion that only moves GSM8K admission and not B3 / curated coverage — that's the smell test for a template in disguise"). The mirror image applies here: a substrate swap that moves admission numbers must be a standalone PR so the delta is attributable to the substrate change, not silently absorbed into a capability iteration.

Today the substrate swap is zero-delta — both _score_one (legacy) and _score_one_candidate_graph produce 0/50 admission on the current main baseline. That makes this the cleanest-possible substrate PR: behavior unchanged, audit trail shifted, future iterations attributable.

What changed in code

evals/gsm8k_math/train_sample/v1/run_coverage_probe.py

• Import switched: run_lane → _score_one_candidate_graph (and the existing LaneReport no longer needed at the probe layer).
• New private _score_lane aggregator mirrors run_lane's output shape via _score_one_candidate_graph per case.
• build_report adds "substrate": "candidate_graph" to the report root.
• Docstring updated to document the substrate choice.

evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json

Regenerated. Top-level "substrate" field added. All metrics byte-identical to the prior baseline (both substrates produce the same 0/50 numbers today). refused_reasons_top text changes because the candidate-graph pipeline emits slightly different refusal strings (prefix candidate_graph: instead of parser:); this is expected and is part of the substrate audit-trail shift.

No changes to

• evals/gsm8k_math/runner.py — run_lane, _score_one, _score_one_candidate_graph all unchanged.
• tests/test_adr_0131_G_gsm8k_coverage_probe.py — the 8 contract tests pin invariants, not specific numbers, and all pass after the substrate swap.
• Any G. iteration's lane code or axis cases.

Evidence

• Probe: python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe → admission_rate: 0/50 = 0.0%, wrong: 0, safety_rail_intact: True, exit code 0.
• Contract tests: 8/8 pass in 0.18s.
• Substrate equivalence on main baseline: both pipelines produce 0/50 admission. The substrate swap is zero behavior delta today.

Effect on in-flight iterations

Iteration	Effect of this PR
ADR-0131.G.1 (verbs, Gemini)	After landing, probe admission will rise on cases the new verbs unlock (e.g., Nicole collected 400 Pokemon cards).
ADR-0131.G.2 (comparatives, PR #182)	Refused-reason family for comparatives moves at the probe layer for the first time.
ADR-0131.G.3 (numerics, PR #183)	Money + hyphenated cases the candidate-graph pipeline already admits start showing as probe admission (combined with G.1 unlocks Tina makes $18.00 an hour etc.).
ADR-0131.G.4 (multi-clause, Opus#2)	Conjoined-subject cases like Aaron and his brother Carson each saved up $40 become probe-admittable (combined with G.3 for the money literal).

Reconciliation order after merge: each G. PR rebases onto main after this PR lands, runs the probe, refreshes its committed train_sample_coverage_report.json. The refreshed report becomes the iteration's load-bearing evidence on the probe, complementing the axis lane.

What this does NOT do

• Does NOT change the legacy parser path. parse_problem and _score_one remain available for other lanes that depend on them.
• Does NOT change the GSM8K case set, the sample, or the answer vocabulary.
• Does NOT change the GSM8K probe's role: it remains a probe (honest disclosure of what the architecture can/can't admit), not a gate for the math-expert promotion (that's still B1+B2+B3 per ADR-0131).
• Does NOT modify any G. capability work in flight. Each rebase is mechanical (one merged report file refresh).
• Does NOT introduce stochastic behavior, opaque fallbacks, or approximate recall anywhere in the probe.

CLAUDE.md PR-checklist answers

• Capability / performance / security added or protected: Aligns the probe's measurement substrate with the parser layer the iteration discipline extends. Audit-trail correctness for every future capability iteration.
• Invariant proving the field remains valid: admitted_wrong == 0 preserved on the probe; safety_rail_intact: True; 8/8 contract tests pass.
• CLI/eval lane proving the lane: python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe and pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py.
• Avoided hidden normalization / stochastic / approximate / unreviewed mutation: Yes. Pure deterministic substrate swap, same case set, same scoring rules, different parser entry point.
• Trust boundary: No new user-input surface, no new filesystem paths, no new dynamic imports. The probe still reads only cases.jsonl and writes only train_sample_coverage_report.json under evals/gsm8k_math/train_sample/v1/.