801287bba6
Zero behavior delta on the main baseline (both substrates produce
0/50 admission today) — but every subsequent ADR-0131.G.<n> iteration
now produces attributable admission deltas on the probe, instead of
silently extending a parser layer the probe wasn't measuring.
Background: ADR-0131.G's probe consulted run_lane → _score_one →
parse_problem (legacy first-match-wins parser, pre-ADR-0126). Every
G.<n> iteration extends the candidate-graph parser via
_score_one_candidate_graph → parse_and_solve. The mismatch was
discovered during G.3 development and explicitly reserved as this
follow-up.
Changes:
- run_coverage_probe.py: switch import to _score_one_candidate_graph;
new private _score_lane aggregator mirrors run_lane's output shape
via per-case scoring; report root adds "substrate": "candidate_graph"
for audit trail.
- train_sample_coverage_report.json: regenerated. All metrics
byte-identical to prior baseline (0/50 admission, wrong=0).
refused_reasons_top text differs (candidate_graph: prefix instead
of parser:) — expected and part of the substrate audit-trail shift.
Discipline: separate small PR per ADR-0131.G's "expansion that only
moves admission must be a standalone PR" principle. Substrate swap
attributable; future G.<n> deltas attributable.
Evidence:
- python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe
→ admission 0/50, wrong=0, safety_rail_intact=True, exit 0
- pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py
→ 8/8 pass in 0.18s (no test edits needed; tests pin invariants
not numbers)
- No changes to runner.py, no changes to any G.<n> work in flight.
Effect on in-flight iterations: each G.<n> PR (G.1 Gemini / G.2 #182 /
G.3 #183 / G.4 Opus#2) rebases after this lands and refreshes its
committed train_sample_coverage_report.json with the new substrate's
numbers. Rebase is mechanical.
167 lines
7.3 KiB
Markdown
167 lines
7.3 KiB
Markdown
# ADR-0131.G.0 — Probe Substrate: Candidate-Graph Pipeline
|
|
|
|
**Status:** Proposed
|
|
**Date:** 2026-05-23
|
|
**Author:** CORE main agent (Opus 4.7)
|
|
**Depends on:** ADR-0126 (candidate-graph parser),
|
|
ADR-0131.G (GSM8K coverage probe + iteration discipline)
|
|
**Parent:** ADR-0131.G
|
|
**Sibling iterations (in flight):** ADR-0131.G.1 (verbs),
|
|
ADR-0131.G.2 (comparatives), ADR-0131.G.3 (numerics),
|
|
ADR-0131.G.4 (multi-clause)
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
ADR-0131.G (PR #181) pinned a GSM8K coverage probe as a diff-able
|
|
admission number with capability-first iteration discipline. Each
|
|
subsequent ADR-0131.G.<n> extends a single capability axis of the
|
|
parser layer and re-runs the probe, with the gate that
|
|
`admission_rate` strictly increases (or a refused-reason family
|
|
deliberately shrinks).
|
|
|
|
But the probe currently consults
|
|
`evals.gsm8k_math.runner.run_lane`, which scores each case via
|
|
`_score_one` → `parse_problem` — the **legacy first-match-wins
|
|
parser** that pre-dates ADR-0126. Every ADR-0131.G.<n> iteration
|
|
extends the **candidate-graph parser** (ADR-0126 topology), via
|
|
`_score_one_candidate_graph` → `parse_and_solve`. **The probe and
|
|
the iteration work measure two different parsers.**
|
|
|
|
The mismatch was discovered during ADR-0131.G.3 development: the
|
|
G.3 axis lane (which routes through the candidate-graph pipeline
|
|
via `_score_one_candidate_graph`) showed full end-to-end
|
|
correctness on 20 curated money + hyphenated-cardinal cases, while
|
|
the GSM8K probe (legacy substrate) showed zero admission delta.
|
|
The G.3 ADR documented the limit honestly and reserved this
|
|
follow-up.
|
|
|
|
## Decision
|
|
|
|
Switch `evals/gsm8k_math/train_sample/v1/run_coverage_probe.py` to
|
|
call `_score_one_candidate_graph` instead of `run_lane`. The probe
|
|
now measures the parser layer the iteration discipline extends.
|
|
|
|
Mechanically, the probe builds its own thin `_score_lane` aggregator
|
|
that mirrors `run_lane`'s shape (cases_total / correct / wrong /
|
|
refused / rates / wrong_count_is_zero) but invokes
|
|
`_score_one_candidate_graph` per case. No changes to
|
|
`runner.py` itself.
|
|
|
|
Report schema gains one field — `"substrate": "candidate_graph"` —
|
|
so the audit trail records which pipeline produced the numbers.
|
|
Every other field (metrics, refused_reasons_top, per_case) keeps
|
|
its shape; the 8 contract tests in
|
|
`tests/test_adr_0131_G_gsm8k_coverage_probe.py` pass without
|
|
modification.
|
|
|
|
## Why this is a separate (small) PR
|
|
|
|
ADR-0131.G's discipline explicitly forbids bundling probe-infra
|
|
changes into capability-axis PRs ("Reject any expansion that only
|
|
moves GSM8K admission and not B3 / curated coverage — that's the
|
|
smell test for a template in disguise"). The mirror image applies
|
|
here: a substrate swap that moves admission numbers must be a
|
|
standalone PR so the delta is attributable to the substrate
|
|
change, not silently absorbed into a capability iteration.
|
|
|
|
**Today the substrate swap is zero-delta** — both
|
|
`_score_one` (legacy) and `_score_one_candidate_graph` produce
|
|
0/50 admission on the current main baseline. That makes this the
|
|
cleanest-possible substrate PR: behavior unchanged, audit trail
|
|
shifted, future iterations attributable.
|
|
|
|
## What changed in code
|
|
|
|
### `evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`
|
|
|
|
- Import switched: `run_lane` → `_score_one_candidate_graph` (and
|
|
the existing `LaneReport` no longer needed at the probe layer).
|
|
- New private `_score_lane` aggregator mirrors `run_lane`'s
|
|
output shape via `_score_one_candidate_graph` per case.
|
|
- `build_report` adds `"substrate": "candidate_graph"` to the
|
|
report root.
|
|
- Docstring updated to document the substrate choice.
|
|
|
|
### `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
|
|
|
|
Regenerated. Top-level `"substrate"` field added. All metrics
|
|
byte-identical to the prior baseline (both substrates produce the
|
|
same 0/50 numbers today). `refused_reasons_top` text changes
|
|
because the candidate-graph pipeline emits slightly different
|
|
refusal strings (prefix `candidate_graph:` instead of
|
|
`parser:`); this is expected and is part of the substrate
|
|
audit-trail shift.
|
|
|
|
### No changes to
|
|
|
|
- `evals/gsm8k_math/runner.py` — `run_lane`,
|
|
`_score_one`, `_score_one_candidate_graph` all unchanged.
|
|
- `tests/test_adr_0131_G_gsm8k_coverage_probe.py` — the 8
|
|
contract tests pin invariants, not specific numbers, and all
|
|
pass after the substrate swap.
|
|
- Any G.<n> iteration's lane code or axis cases.
|
|
|
|
## Evidence
|
|
|
|
- **Probe**: `python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe`
|
|
→ `admission_rate: 0/50 = 0.0%`, `wrong: 0`,
|
|
`safety_rail_intact: True`, exit code 0.
|
|
- **Contract tests**: 8/8 pass in 0.18s.
|
|
- **Substrate equivalence on main baseline**: both pipelines
|
|
produce 0/50 admission. The substrate swap is zero behavior
|
|
delta today.
|
|
|
|
## Effect on in-flight iterations
|
|
|
|
| Iteration | Effect of this PR |
|
|
|---|---|
|
|
| ADR-0131.G.1 (verbs, Gemini) | After landing, probe admission will rise on cases the new verbs unlock (e.g., `Nicole collected 400 Pokemon cards`). |
|
|
| ADR-0131.G.2 (comparatives, PR #182) | Refused-reason family for comparatives moves at the probe layer for the first time. |
|
|
| ADR-0131.G.3 (numerics, PR #183) | Money + hyphenated cases the candidate-graph pipeline already admits start showing as probe admission (combined with G.1 unlocks `Tina makes $18.00 an hour` etc.). |
|
|
| ADR-0131.G.4 (multi-clause, Opus#2) | Conjoined-subject cases like `Aaron and his brother Carson each saved up $40` become probe-admittable (combined with G.3 for the money literal). |
|
|
|
|
Reconciliation order after merge: each G.<n> PR rebases onto
|
|
`main` after this PR lands, runs the probe, refreshes its
|
|
committed `train_sample_coverage_report.json`. The refreshed
|
|
report becomes the iteration's load-bearing evidence on the
|
|
probe, complementing the axis lane.
|
|
|
|
## What this does NOT do
|
|
|
|
- Does NOT change the legacy parser path. `parse_problem` and
|
|
`_score_one` remain available for other lanes that depend on
|
|
them.
|
|
- Does NOT change the GSM8K case set, the sample, or the answer
|
|
vocabulary.
|
|
- Does NOT change the GSM8K probe's role: it remains a *probe*
|
|
(honest disclosure of what the architecture can/can't admit),
|
|
not a *gate* for the math-expert promotion (that's still
|
|
B1+B2+B3 per ADR-0131).
|
|
- Does NOT modify any G.<n> capability work in flight. Each
|
|
rebase is mechanical (one merged report file refresh).
|
|
- Does NOT introduce stochastic behavior, opaque fallbacks, or
|
|
approximate recall anywhere in the probe.
|
|
|
|
## CLAUDE.md PR-checklist answers
|
|
|
|
- **Capability / performance / security added or protected:**
|
|
Aligns the probe's measurement substrate with the parser layer
|
|
the iteration discipline extends. Audit-trail correctness for
|
|
every future capability iteration.
|
|
- **Invariant proving the field remains valid:**
|
|
`admitted_wrong == 0` preserved on the probe;
|
|
`safety_rail_intact: True`; 8/8 contract tests pass.
|
|
- **CLI/eval lane proving the lane:**
|
|
`python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe`
|
|
and `pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py`.
|
|
- **Avoided hidden normalization / stochastic / approximate /
|
|
unreviewed mutation:** Yes. Pure deterministic substrate swap,
|
|
same case set, same scoring rules, different parser entry
|
|
point.
|
|
- **Trust boundary:** No new user-input surface, no new
|
|
filesystem paths, no new dynamic imports. The probe still
|
|
reads only `cases.jsonl` and writes only
|
|
`train_sample_coverage_report.json` under
|
|
`evals/gsm8k_math/train_sample/v1/`.
|