core/docs/adr/ADR-0131.G.0-probe-substrate.md

# ADR-0131.G.0 — Probe Substrate: Candidate-Graph Pipeline

**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE main agent (Opus 4.7)
**Depends on:** ADR-0126 (candidate-graph parser),
ADR-0131.G (GSM8K coverage probe + iteration discipline)
**Parent:** ADR-0131.G
**Sibling iterations (in flight):** ADR-0131.G.1 (verbs),
ADR-0131.G.2 (comparatives), ADR-0131.G.3 (numerics),
ADR-0131.G.4 (multi-clause)

---

## Context

ADR-0131.G (PR #181) pinned a GSM8K coverage probe as a diff-able
admission number with capability-first iteration discipline. Each
subsequent ADR-0131.G.<n> extends a single capability axis of the
parser layer and re-runs the probe, with the gate that
`admission_rate` strictly increases (or a refused-reason family
deliberately shrinks).

But the probe currently consults
`evals.gsm8k_math.runner.run_lane`, which scores each case via
`_score_one` → `parse_problem` — the **legacy first-match-wins
parser** that pre-dates ADR-0126. Every ADR-0131.G.<n> iteration
extends the **candidate-graph parser** (ADR-0126 topology), via
`_score_one_candidate_graph` → `parse_and_solve`. **The probe and
the iteration work measure two different parsers.**

The mismatch was discovered during ADR-0131.G.3 development: the
G.3 axis lane (which routes through the candidate-graph pipeline
via `_score_one_candidate_graph`) showed full end-to-end
correctness on 20 curated money + hyphenated-cardinal cases, while
the GSM8K probe (legacy substrate) showed zero admission delta.
The G.3 ADR documented the limit honestly and reserved this
follow-up.

## Decision

Switch `evals/gsm8k_math/train_sample/v1/run_coverage_probe.py` to
call `_score_one_candidate_graph` instead of `run_lane`. The probe
now measures the parser layer the iteration discipline extends.

Mechanically, the probe builds its own thin `_score_lane` aggregator
that mirrors `run_lane`'s shape (cases_total / correct / wrong /
refused / rates / wrong_count_is_zero) but invokes
`_score_one_candidate_graph` per case. No changes to
`runner.py` itself.

Report schema gains one field — `"substrate": "candidate_graph"` —
so the audit trail records which pipeline produced the numbers.
Every other field (metrics, refused_reasons_top, per_case) keeps
its shape; the 8 contract tests in
`tests/test_adr_0131_G_gsm8k_coverage_probe.py` pass without
modification.

## Why this is a separate (small) PR

ADR-0131.G's discipline explicitly forbids bundling probe-infra
changes into capability-axis PRs ("Reject any expansion that only
moves GSM8K admission and not B3 / curated coverage — that's the
smell test for a template in disguise"). The mirror image applies
here: a substrate swap that moves admission numbers must be a
standalone PR so the delta is attributable to the substrate
change, not silently absorbed into a capability iteration.

**Today the substrate swap is zero-delta** — both
`_score_one` (legacy) and `_score_one_candidate_graph` produce
0/50 admission on the current main baseline. That makes this the
cleanest-possible substrate PR: behavior unchanged, audit trail
shifted, future iterations attributable.

## What changed in code

### `evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`

- Import switched: `run_lane` → `_score_one_candidate_graph` (and
  the existing `LaneReport` no longer needed at the probe layer).
- New private `_score_lane` aggregator mirrors `run_lane`'s
  output shape via `_score_one_candidate_graph` per case.
- `build_report` adds `"substrate": "candidate_graph"` to the
  report root.
- Docstring updated to document the substrate choice.

### `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`

Regenerated. Top-level `"substrate"` field added. All metrics
byte-identical to the prior baseline (both substrates produce the
same 0/50 numbers today). `refused_reasons_top` text changes
because the candidate-graph pipeline emits slightly different
refusal strings (prefix `candidate_graph:` instead of
`parser:`); this is expected and is part of the substrate
audit-trail shift.

### No changes to

- `evals/gsm8k_math/runner.py` — `run_lane`,
  `_score_one`, `_score_one_candidate_graph` all unchanged.
- `tests/test_adr_0131_G_gsm8k_coverage_probe.py` — the 8
  contract tests pin invariants, not specific numbers, and all
  pass after the substrate swap.
- Any G.<n> iteration's lane code or axis cases.

## Evidence

- **Probe**: `python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe`
  → `admission_rate: 0/50 = 0.0%`, `wrong: 0`,
  `safety_rail_intact: True`, exit code 0.
- **Contract tests**: 8/8 pass in 0.18s.
- **Substrate equivalence on main baseline**: both pipelines
  produce 0/50 admission. The substrate swap is zero behavior
  delta today.

## Effect on in-flight iterations

| Iteration | Effect of this PR |
|---|---|
| ADR-0131.G.1 (verbs, Gemini) | After landing, probe admission will rise on cases the new verbs unlock (e.g., `Nicole collected 400 Pokemon cards`). |
| ADR-0131.G.2 (comparatives, PR #182) | Refused-reason family for comparatives moves at the probe layer for the first time. |
| ADR-0131.G.3 (numerics, PR #183) | Money + hyphenated cases the candidate-graph pipeline already admits start showing as probe admission (combined with G.1 unlocks `Tina makes $18.00 an hour` etc.). |
| ADR-0131.G.4 (multi-clause, Opus#2) | Conjoined-subject cases like `Aaron and his brother Carson each saved up $40` become probe-admittable (combined with G.3 for the money literal). |

Reconciliation order after merge: each G.<n> PR rebases onto
`main` after this PR lands, runs the probe, refreshes its
committed `train_sample_coverage_report.json`. The refreshed
report becomes the iteration's load-bearing evidence on the
probe, complementing the axis lane.

## What this does NOT do

- Does NOT change the legacy parser path. `parse_problem` and
  `_score_one` remain available for other lanes that depend on
  them.
- Does NOT change the GSM8K case set, the sample, or the answer
  vocabulary.
- Does NOT change the GSM8K probe's role: it remains a *probe*
  (honest disclosure of what the architecture can/can't admit),
  not a *gate* for the math-expert promotion (that's still
  B1+B2+B3 per ADR-0131).
- Does NOT modify any G.<n> capability work in flight. Each
  rebase is mechanical (one merged report file refresh).
- Does NOT introduce stochastic behavior, opaque fallbacks, or
  approximate recall anywhere in the probe.

## CLAUDE.md PR-checklist answers

- **Capability / performance / security added or protected:**
  Aligns the probe's measurement substrate with the parser layer
  the iteration discipline extends. Audit-trail correctness for
  every future capability iteration.
- **Invariant proving the field remains valid:**
  `admitted_wrong == 0` preserved on the probe;
  `safety_rail_intact: True`; 8/8 contract tests pass.
- **CLI/eval lane proving the lane:**
  `python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe`
  and `pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py`.
- **Avoided hidden normalization / stochastic / approximate /
  unreviewed mutation:** Yes. Pure deterministic substrate swap,
  same case set, same scoring rules, different parser entry
  point.
- **Trust boundary:** No new user-input surface, no new
  filesystem paths, no new dynamic imports. The probe still
  reads only `cases.jsonl` and writes only
  `train_sample_coverage_report.json` under
  `evals/gsm8k_math/train_sample/v1/`.