89defef30b
C1: delete generate/math_versor_arithmetic.py and its 3 tests (ADR-0139 add-only arithmetic spike; no runtime consumers, no pipeline wiring, follow-on lift paused per module docstring). C3: gitignore engine_state runtime artifacts (manifest.json, recognizers.jsonl, discovery_candidates.jsonl). Module code (engine_state/__init__.py) remains tracked; generated checkpoint files should not be. C5: document reader zero-delta root cause in train_sample/v1/README.md. Both Phase 2 (whole-problem) and Phase 1 (question-only) reader paths are called but inert because all 47 refusals are statement-level NO_INJECTOR gaps, not question-sentence gaps. Reader unblocks when injector coverage expands (C2 work). report.json use_reader flag corrected to reflect last run. C6: add deprecation header to generate/math_parser.py pointing at generate.math_candidate_graph.parse_and_solve as the live path. C2/C4 briefs: docs/handoff/CLEANUP-C2-run-lane-migration.md and docs/handoff/CLEANUP-C4-compositions-compile.md added as operator dispatch docs for the medium-scope wiring tasks.
126 lines
4 KiB
Markdown
126 lines
4 KiB
Markdown
# C2 — Wire `run_lane` to `_score_one_candidate_graph`
|
||
|
||
**Classification**: Unfinished migration (audit finding C2)
|
||
**Risk**: Medium — must verify baseline comparison files still reproduce
|
||
**Scope**: `evals/gsm8k_math/runner.py` only
|
||
|
||
---
|
||
|
||
## Problem
|
||
|
||
`run_lane` (line 449) calls `_score_one`, the legacy regex path
|
||
(`generate/math_parser.py` → `MathProblemGraph` → `solve`). Only the
|
||
per-lane override in `train_sample/v1/runner.py:33` uses
|
||
`_score_one_candidate_graph`. Any caller that goes through `run_lane` —
|
||
including the `public/`, `dev/`, and `holdout` lanes — is evaluated on the
|
||
legacy parser, not the candidate-graph architecture we've been measuring.
|
||
|
||
**Impact**: lift numbers on `train_sample/v1` are not comparable to what
|
||
`run_lane` would report on the other slices. The three-slice concern from the
|
||
audit is real.
|
||
|
||
---
|
||
|
||
## What `_score_one_candidate_graph` does differently
|
||
|
||
`_score_one_candidate_graph` (line 241) calls `parse_and_solve` from
|
||
`generate/math_candidate_graph`, which layers:
|
||
|
||
1. Whole-problem comprehension reader (ADR-0164 Phase 2, flag-gated)
|
||
2. Per-statement recognizer candidate graph (ME-1..ME-5 matchers)
|
||
3. Question-sentence hybrid reader (ADR-0164 Phase 1)
|
||
4. Injector dispatch (ADR-0170 + recognizer_anchor_inject.py)
|
||
5. Cartesian-product branch enumeration + admissibility gate
|
||
6. Existing solver + verifier (same as `_score_one`)
|
||
|
||
`_score_one` calls the regex parser directly, skipping steps 1–5.
|
||
|
||
---
|
||
|
||
## Migration plan
|
||
|
||
### Step 1 — Baseline snapshot (before changing `run_lane`)
|
||
|
||
```bash
|
||
# Capture current run_lane output on all slices that have unsealed cases
|
||
uv run python -m evals.gsm8k_math.runner --lane public > /tmp/pre_public.json
|
||
uv run python -m evals.gsm8k_math.runner --lane dev > /tmp/pre_dev.json
|
||
# Do NOT touch holdout; it is sealed.
|
||
```
|
||
|
||
### Step 2 — Wire `run_lane`
|
||
|
||
In `evals/gsm8k_math/runner.py` line 449, replace:
|
||
|
||
```python
|
||
outcomes = [_score_one(c) for c in cases]
|
||
```
|
||
|
||
with:
|
||
|
||
```python
|
||
outcomes = [_score_one_candidate_graph(c) for c in cases]
|
||
```
|
||
|
||
Add a `--legacy-parser` CLI flag that restores `_score_one` for the
|
||
comparison baseline file. The flag is only needed to reproduce
|
||
`baselines/frontier.json` and `baselines/comparison_v1.json`, both of which
|
||
were captured under the legacy path.
|
||
|
||
```python
|
||
# In run_lane signature:
|
||
def run_lane(
|
||
cases: list[dict[str, Any]],
|
||
*,
|
||
config: Any = None,
|
||
legacy_parser: bool = False,
|
||
) -> LaneReport:
|
||
score_fn = _score_one if legacy_parser else _score_one_candidate_graph
|
||
outcomes = [score_fn(c) for c in cases]
|
||
```
|
||
|
||
### Step 3 — Verify wrong == 0 on public/dev
|
||
|
||
```bash
|
||
uv run python -m evals.gsm8k_math.runner --lane public > /tmp/post_public.json
|
||
uv run python -m evals.gsm8k_math.runner --lane dev > /tmp/post_dev.json
|
||
# Assert wrong == 0 in both outputs
|
||
```
|
||
|
||
### Step 4 — Update baselines if counts differ
|
||
|
||
If `correct` counts differ (expected — the new path refuses differently),
|
||
update `baselines/frontier.json` and `baselines/comparison_v1.json` by
|
||
re-running with `--legacy-parser` so the files still reflect legacy baseline
|
||
numbers:
|
||
|
||
```bash
|
||
uv run python -m evals.gsm8k_math.runner --lane public --legacy-parser \
|
||
> evals/gsm8k_math/baselines/frontier.json
|
||
```
|
||
|
||
### Step 5 — Run smoke + packs suites
|
||
|
||
```bash
|
||
uv run core test --suite smoke -q
|
||
uv run core test --suite packs -q
|
||
```
|
||
|
||
---
|
||
|
||
## Invariant gates
|
||
|
||
- `wrong == 0` must hold on public + dev after the migration.
|
||
- `_score_one_candidate_graph` already enforces this by construction (same
|
||
verifier as `_score_one`, additive reader, refusal-preferring injector).
|
||
- The baseline snapshots captured in Step 1 give you a before/after diff to
|
||
explain any count change in a PR description.
|
||
|
||
---
|
||
|
||
## Relation to other findings
|
||
|
||
- **C4** (compositions compile): C4 should land first or simultaneously so
|
||
that `run_lane` immediately benefits from the registry being non-empty.
|
||
- **C5** (reader zero-delta): confirmed that the reader is inert until
|
||
statement-level injector gaps close. C2 does not depend on C5.
|