core/docs/decisions/ADR-0126-candidate-graph-parser.md

ADR-0126 — Candidate-Graph Parser with Round-Trip Verifier-Filter

Status: Proposed Date: 2026-05-23 Author: CORE agents + reviewers Depends on:

• ADR-0115 (parser substrate),
• ADR-0116 (deterministic solver),
• ADR-0117 (SolutionTrace verifier),
• ADR-0118 (stepped realizer),
• ADR-0119 (+ all 8 sub-phases — GSM8K-math lane substrate),
• ADR-0119.7 (sealed GSM8K test),
• ADR-0120 (expert promotion contract),
• ADR-0121 (math expert promotion DEFERRED with named blocker correct_rate = 0/1319 on sealed holdout). Supersedes: ADR-0123b (feat/adr-0123b-upstream-shape-gaps, commit 8c4070e, never opened as PR). Substrate of ADR-0122 / ADR-0123 / ADR-0123a remains in main and is consumed by this ADR.

---

Context

ADR-0121 deferred the mathematics_logic → expert promotion with the named blocker correct_rate = 0/1319 on sealed GSM8K holdout and proposed a parser-expansion arc of 4–8 construction classes. Three ADRs of that arc landed (ADR-0122 rate / per-unit, ADR-0123 comparison-phrasing, ADR-0123a comparison-shape-expansion) and a fourth was prepared (ADR-0123b upstream-shape-gaps). All four:

• preserved every invariant (wrong == 0, byte-equal trace, pack-binding, replay determinism),
• passed their author-drafted dev-set exit criteria,
• measured 0/1319 sealed-holdout lift (Gemini Tasks 6 and 8).

Two independent external assessments (Kimi K2.6, Grok 4.2) and an internal review converged on the same diagnosis:

1. The dev set is hermetically sealed from the real GSM8K distribution. It was authored by the people who authored the parser, against the parser's grammar. Dev-set success is uncorrelated (in expectation, anti-correlated) with sealed-holdout lift.

2. Per-axis vocabulary expansion is structurally optimistic. Each sealed case lives in the intersection of grammar gaps (verb × number-word × question-shape × ellipsis × multi-step), not the union. If each gap has independent coverage p, joint pass rate is p^k. Adding one verb raises p infinitesimally; joint pass rate barely moves.

3. The current parser fails hard at the first unmatched construction. A single brittle regex (e.g., an ambiguous verb match that steals the slot from a correct one) refuses an entire problem with ParseError. This makes vocabulary expansion adversarial against itself — adding new patterns can lose coverage on already-passing problems.

The architectural error is the topology of failure, not the size of vocabulary. Four zero-lift ADRs are the empirical evidence that more grammar in the same shape will not move the sealed-holdout score.

---

Decision

Replace the first-match-wins, single-graph, fail-hard parser topology with a candidate-graph + verifier-filter topology that preserves every existing invariant while converting compound-gap failure from multiplicative (p^k) to parallel (1 - (1-p)^k).

Pipeline change

OLD: surface → ONE graph → solve → verify(trace_replay)
NEW: surface → [graph_1, graph_2, … graph_n] → solve each
              → round-trip filter → admissibility filter → decision rule

Per-sentence change

Each sentence emits list[CandidateOperation] (possibly empty) instead of one Operation or ParseError. Multiple matching patterns no longer race for a single slot; they all emit candidates and the verifier filters wrong ones downstream.

Hard bound: MAX_CANDIDATES_PER_SENTENCE = 4. Exceeding raises ParseError (preserves determinism cost ceiling).

Graph assembly

Cartesian product of per-sentence candidate lists, enumerated depth-first in canonical pattern-priority order (most-specific-first). Hard bound: MAX_TOTAL_BRANCHES = 64. Exceeding refuses the entire problem (preserves runtime ceiling).

Round-trip filter (load-bearing new invariant)

A candidate operation is round-trip admissible iff reconstructing the source sentence from its parsed slots produces a string byte-equal to the original sentence under normalization N (lowercase + collapse whitespace + strip terminal punctuation).

Formally, for source sentence s and candidate operation op:

admissible(op, s) ⟺ N(reconstruct(op)) == N(s)

This is the wrong-answer firewall. A regex that misreads "gives" as subtract instead of transfer will reconstruct a sentence that doesn't byte-equal the source; the candidate is rejected before it can produce a wrong number.

Decision rule (final emit)

For the set A of admissible candidate graphs:

| |A| | answer distribution | decision | |-------|--------------------|----------| | 0 | — | refuse (preserves wrong == 0) | | 1 | — | emit the single answer | | ≥2 | all identical | emit the common answer | | ≥2 | non-identical | refuse (genuine ambiguity) |

Invariant preservation

Invariant	How preserved
wrong == 0	Round-trip filter rejects mis-parses; ambiguity refuses.
Trace determinism	Candidate enumeration canonical (lexical pattern key); branch order canonical (depth-first by sentence).
trace_hash byte-equality	Selected graph alone enters SolutionTrace; non-selected branches are not hashed.
Pack-binding	Candidate generators consume the same pack lemmas as today.
Replay equivalence	Replay re-runs candidate enumeration + filter; same input → same selected graph.
Adversarial wrong == 0	Adversarial suite gate runs unchanged; round-trip filter is stricter than today's gates.

New invariant added

Round-trip admissibility: any operation that emits in a final SolutionTrace must satisfy N(reconstruct(op)) == N(source_span) for the source span it claims to parse. This is a stricter contract than trace_hash byte-equality across runs (which only guards determinism) — it guards against semantic mis-parse.

---

Exit criterion

Inner-loop signal (new): draw 50 cases from GSM8K train split (deterministic seed, committed unsealed at evals/gsm8k_math/train_sample/v1/cases.jsonl). Run candidate-graph pipeline. The architecture is validated iff:

correct ≥ 10 / 50  (20% absolute)
wrong  == 0

This is the first non-synthetic gradient signal in the GSM8K lane. If the architecture is structurally sound, ≥20% lift on a random sample is achievable without any new vocabulary work — the gain comes purely from converting fail-hard to filter-and-pick.

Path-B trigger: if 50-case train sample shows correct < 10 after the candidate-graph pipeline is integrated, the parser-by-rule architecture (in any topology) is the wrong abstraction for GSM8K coverage. ADR-0126 itself is deferred and the work pivots to benchmark re-selection (see "Alternatives considered → Path B" below).

Sealed-holdout protocol unchanged: if the train-sample gate passes, the sealed holdout runs exactly once and the number is frozen in ADR-0126-results. The sealed .age ciphertext is never modified.

---

Alternatives considered

Path A.1 — More vocabulary (status-quo continuation)

Continue the ADR-0122/0123/0123a/0123b cadence: catalog refusal modes, add verbs/numbers/patterns, ship per-axis ADRs.

Rejected because four consecutive ADRs in this shape produced 0/1319 lift. The compound-gap math (p^k) explains why; no amount of p improvement at the rate we can produce ADRs moves joint pass rate meaningfully.

Path B — Benchmark re-selection

Demote GSM8K to a stress test we honestly refuse on; re-target the math-expert promotion to a benchmark where exact-recall, determinism, and provenance are the discriminators (formal-math symbolic equivalence, CORE-native teaching-corpus eval, MATH symbolic subset).

Held as fallback triggered by the Path-B condition above. Not chosen first because it forecloses the GSM8K target without having tried the architectural fix that the compound-gap diagnosis actually implies.

Path C — Learned (LLM-aided) parser

Replace the rule-based parser with an LLM-assisted candidate generator.

Rejected as a contract violation. ADR-0114a and the project-wide stance (CLAUDE.md "no opaque LLM fallbacks, no stochastic sampling, no hidden normalization") forbid this. The candidate-graph approach in this ADR is the deterministic analogue: parallel hypotheses with a verifier filter, no sampling, no learned scoring.

---

Implementation plan

Phase	Module	Description
P1	generate/math_roundtrip.py (new)	Standalone roundtrip_matches(op, source) -> bool. Unit tests over every existing op kind. This is the load-bearing primitive; nothing else matters if it doesn't work.
P2	generate/math_parser.py (refactor)	_process_statement / _process_question return list[CandidateOperation]. Verb tables widened permissively.
P3	generate/math_candidate_graph.py (new)	Branch enumerator + filter + decision rule.
P4	evals/gsm8k_math/runner.py (wire)	Replace single-graph call with candidate-graph call. Preserve all current passing cases.
P5	evals/gsm8k_math/train_sample/v1/cases.jsonl (new)	50-case deterministic train-split sample (unsealed).
P6	evals/gsm8k_math/train_sample_runner.py (new)	Run candidate-graph on train sample; emit correct_rate, wrong_count.

Regression gates (must remain green at every phase):

• core test --suite smoke
• core test --suite math (existing 507/507)
• core test --suite algebra (82/82)
• 150/150 public split
• 200/200 dev set
• adversarial suite (wrong == 0)

---

PR checklist (when proposing for merge)

What capability did this add?
  → Candidate-graph parser topology + round-trip verifier-filter.
What invariant proves the field remains valid?
  → Round-trip admissibility (new); wrong == 0 (preserved);
    trace_hash byte-equality (preserved).
Which CLI suite/eval proves the lane?
  → smoke + math + algebra + 200/200 dev + train_sample_runner.
Did this avoid hidden normalization, stochastic fallback,
approximate recall, unreviewed mutation?
  → Yes. Candidate enumeration is deterministic + bounded.
    Round-trip filter is deterministic byte comparison.
    No learned scoring, no sampling.
If it touches user input, what trust boundary was enforced?
  → No new user-input surfaces. Round-trip reconstruction does
    not echo unvalidated user content into logs.