docs(ADR-0126): candidate-graph parser + round-trip verifier-filter (proposed)
Architectural pivot from per-axis grammar expansion (ADR-0122/0123/0123a/ 0123b, all 0/1319 sealed lift) to candidate-graph topology with round-trip admissibility filter. Converts compound-gap failure from multiplicative p^k to parallel 1-(1-p)^k. Adds new invariant: round-trip admissibility (op slots must reconstruct to byte-equal source sentence under whitespace/case normalization). Preserves wrong==0, trace_hash byte-equality, pack-binding, replay determinism, no stochastic sampling. Exit criterion: 50-case GSM8K train-split sample (unsealed) must show correct >=10/50 with wrong==0 before any sealed-holdout run. If miss, escalate to Path B (benchmark re-selection). Supersedes ADR-0123b (never opened as PR).
This commit is contained in:
Shay
parent
5c6dfbe01d
commit
5a987ed577
1 changed files with 259 additions and 0 deletions
259
docs/decisions/ADR-0126-candidate-graph-parser.md
Normal file
259
docs/decisions/ADR-0126-candidate-graph-parser.md
Normal file
|
|
@ -0,0 +1,259 @@
|
||||||
|
# ADR-0126 — Candidate-Graph Parser with Round-Trip Verifier-Filter
|
||||||
|
|
||||||
|
**Status:** Proposed
|
||||||
|
**Date:** 2026-05-23
|
||||||
|
**Author:** CORE agents + reviewers
|
||||||
|
**Depends on:**
|
||||||
|
- ADR-0115 (parser substrate),
|
||||||
|
- ADR-0116 (deterministic solver),
|
||||||
|
- ADR-0117 (`SolutionTrace` verifier),
|
||||||
|
- ADR-0118 (stepped realizer),
|
||||||
|
- ADR-0119 (+ all 8 sub-phases — GSM8K-math lane substrate),
|
||||||
|
- ADR-0119.7 (sealed GSM8K test),
|
||||||
|
- ADR-0120 (expert promotion contract),
|
||||||
|
- ADR-0121 (math expert promotion DEFERRED with named blocker
|
||||||
|
`correct_rate = 0/1319` on sealed holdout).
|
||||||
|
**Supersedes:** ADR-0123b (`feat/adr-0123b-upstream-shape-gaps`,
|
||||||
|
commit `8c4070e`, never opened as PR). Substrate of ADR-0122 /
|
||||||
|
ADR-0123 / ADR-0123a remains in main and is consumed by this ADR.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
ADR-0121 deferred the `mathematics_logic` → `expert` promotion with
|
||||||
|
the named blocker `correct_rate = 0/1319 on sealed GSM8K holdout`
|
||||||
|
and proposed a parser-expansion arc of 4–8 construction classes.
|
||||||
|
Three ADRs of that arc landed (ADR-0122 rate / per-unit, ADR-0123
|
||||||
|
comparison-phrasing, ADR-0123a comparison-shape-expansion) and a
|
||||||
|
fourth was prepared (ADR-0123b upstream-shape-gaps). All four:
|
||||||
|
|
||||||
|
- preserved every invariant (`wrong == 0`, byte-equal trace,
|
||||||
|
pack-binding, replay determinism),
|
||||||
|
- passed their author-drafted dev-set exit criteria,
|
||||||
|
- **measured 0/1319 sealed-holdout lift** (Gemini Tasks 6 and 8).
|
||||||
|
|
||||||
|
Two independent external assessments (Kimi K2.6, Grok 4.2) and an
|
||||||
|
internal review converged on the same diagnosis:
|
||||||
|
|
||||||
|
1. **The dev set is hermetically sealed from the real GSM8K
|
||||||
|
distribution.** It was authored by the people who authored the
|
||||||
|
parser, against the parser's grammar. Dev-set success is
|
||||||
|
uncorrelated (in expectation, anti-correlated) with sealed-holdout
|
||||||
|
lift.
|
||||||
|
|
||||||
|
2. **Per-axis vocabulary expansion is structurally optimistic.**
|
||||||
|
Each sealed case lives in the *intersection* of grammar gaps
|
||||||
|
(verb × number-word × question-shape × ellipsis × multi-step),
|
||||||
|
not the union. If each gap has independent coverage `p`, joint
|
||||||
|
pass rate is `p^k`. Adding one verb raises `p` infinitesimally;
|
||||||
|
joint pass rate barely moves.
|
||||||
|
|
||||||
|
3. **The current parser fails hard at the first unmatched
|
||||||
|
construction.** A single brittle regex (e.g., an ambiguous verb
|
||||||
|
match that steals the slot from a correct one) refuses an entire
|
||||||
|
problem with `ParseError`. This makes vocabulary expansion
|
||||||
|
adversarial against itself — adding new patterns can *lose*
|
||||||
|
coverage on already-passing problems.
|
||||||
|
|
||||||
|
The architectural error is the *topology* of failure, not the
|
||||||
|
*size* of vocabulary. Four zero-lift ADRs are the empirical
|
||||||
|
evidence that more grammar in the same shape will not move the
|
||||||
|
sealed-holdout score.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Replace the **first-match-wins, single-graph, fail-hard** parser
|
||||||
|
topology with a **candidate-graph + verifier-filter** topology
|
||||||
|
that preserves every existing invariant while converting
|
||||||
|
compound-gap failure from multiplicative (`p^k`) to parallel
|
||||||
|
(`1 - (1-p)^k`).
|
||||||
|
|
||||||
|
### Pipeline change
|
||||||
|
|
||||||
|
```
|
||||||
|
OLD: surface → ONE graph → solve → verify(trace_replay)
|
||||||
|
NEW: surface → [graph_1, graph_2, … graph_n] → solve each
|
||||||
|
→ round-trip filter → admissibility filter → decision rule
|
||||||
|
```
|
||||||
|
|
||||||
|
### Per-sentence change
|
||||||
|
|
||||||
|
Each sentence emits `list[CandidateOperation]` (possibly empty)
|
||||||
|
instead of one `Operation` or `ParseError`. Multiple matching
|
||||||
|
patterns no longer race for a single slot; they all emit
|
||||||
|
candidates and the verifier filters wrong ones downstream.
|
||||||
|
|
||||||
|
Hard bound: `MAX_CANDIDATES_PER_SENTENCE = 4`. Exceeding raises
|
||||||
|
`ParseError` (preserves determinism cost ceiling).
|
||||||
|
|
||||||
|
### Graph assembly
|
||||||
|
|
||||||
|
Cartesian product of per-sentence candidate lists, enumerated
|
||||||
|
depth-first in canonical pattern-priority order
|
||||||
|
(most-specific-first). Hard bound: `MAX_TOTAL_BRANCHES = 64`.
|
||||||
|
Exceeding refuses the entire problem (preserves runtime ceiling).
|
||||||
|
|
||||||
|
### Round-trip filter (load-bearing new invariant)
|
||||||
|
|
||||||
|
A candidate operation is *round-trip admissible* iff
|
||||||
|
reconstructing the source sentence from its parsed slots produces
|
||||||
|
a string byte-equal to the original sentence under normalization
|
||||||
|
N (lowercase + collapse whitespace + strip terminal punctuation).
|
||||||
|
|
||||||
|
Formally, for source sentence `s` and candidate operation `op`:
|
||||||
|
|
||||||
|
```
|
||||||
|
admissible(op, s) ⟺ N(reconstruct(op)) == N(s)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the **wrong-answer firewall**. A regex that misreads
|
||||||
|
"gives" as `subtract` instead of `transfer` will reconstruct a
|
||||||
|
sentence that doesn't byte-equal the source; the candidate is
|
||||||
|
rejected before it can produce a wrong number.
|
||||||
|
|
||||||
|
### Decision rule (final emit)
|
||||||
|
|
||||||
|
For the set `A` of admissible candidate graphs:
|
||||||
|
|
||||||
|
| `|A|` | answer distribution | decision |
|
||||||
|
|-------|--------------------|----------|
|
||||||
|
| 0 | — | **refuse** (preserves `wrong == 0`) |
|
||||||
|
| 1 | — | **emit** the single answer |
|
||||||
|
| ≥2 | all identical | **emit** the common answer |
|
||||||
|
| ≥2 | non-identical | **refuse** (genuine ambiguity) |
|
||||||
|
|
||||||
|
### Invariant preservation
|
||||||
|
|
||||||
|
| Invariant | How preserved |
|
||||||
|
|-----------|---------------|
|
||||||
|
| `wrong == 0` | Round-trip filter rejects mis-parses; ambiguity refuses. |
|
||||||
|
| Trace determinism | Candidate enumeration canonical (lexical pattern key); branch order canonical (depth-first by sentence). |
|
||||||
|
| `trace_hash` byte-equality | Selected graph alone enters `SolutionTrace`; non-selected branches are not hashed. |
|
||||||
|
| Pack-binding | Candidate generators consume the same pack lemmas as today. |
|
||||||
|
| Replay equivalence | Replay re-runs candidate enumeration + filter; same input → same selected graph. |
|
||||||
|
| Adversarial `wrong == 0` | Adversarial suite gate runs unchanged; round-trip filter is *stricter* than today's gates. |
|
||||||
|
|
||||||
|
### New invariant added
|
||||||
|
|
||||||
|
**Round-trip admissibility:** any operation that emits in a final
|
||||||
|
`SolutionTrace` must satisfy `N(reconstruct(op)) == N(source_span)`
|
||||||
|
for the source span it claims to parse. This is a stricter
|
||||||
|
contract than `trace_hash` byte-equality across runs (which only
|
||||||
|
guards determinism) — it guards against semantic mis-parse.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exit criterion
|
||||||
|
|
||||||
|
**Inner-loop signal (new):** draw 50 cases from GSM8K *train*
|
||||||
|
split (deterministic seed, committed unsealed at
|
||||||
|
`evals/gsm8k_math/train_sample/v1/cases.jsonl`). Run candidate-graph
|
||||||
|
pipeline. The architecture is validated iff:
|
||||||
|
|
||||||
|
```
|
||||||
|
correct ≥ 10 / 50 (20% absolute)
|
||||||
|
wrong == 0
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the first non-synthetic gradient signal in the GSM8K
|
||||||
|
lane. If the architecture is structurally sound, ≥20% lift on a
|
||||||
|
random sample is achievable without any new vocabulary work — the
|
||||||
|
gain comes purely from converting fail-hard to filter-and-pick.
|
||||||
|
|
||||||
|
**Path-B trigger:** if 50-case train sample shows `correct < 10`
|
||||||
|
after the candidate-graph pipeline is integrated, the parser-by-rule
|
||||||
|
architecture (in any topology) is the wrong abstraction for GSM8K
|
||||||
|
coverage. ADR-0126 itself is deferred and the work pivots to
|
||||||
|
benchmark re-selection (see "Alternatives considered → Path B"
|
||||||
|
below).
|
||||||
|
|
||||||
|
**Sealed-holdout protocol unchanged:** if the train-sample gate
|
||||||
|
passes, the sealed holdout runs exactly once and the number is
|
||||||
|
frozen in ADR-0126-results. The sealed `.age` ciphertext is never
|
||||||
|
modified.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
### Path A.1 — More vocabulary (status-quo continuation)
|
||||||
|
|
||||||
|
Continue the ADR-0122/0123/0123a/0123b cadence: catalog refusal
|
||||||
|
modes, add verbs/numbers/patterns, ship per-axis ADRs.
|
||||||
|
|
||||||
|
**Rejected** because four consecutive ADRs in this shape produced
|
||||||
|
0/1319 lift. The compound-gap math (`p^k`) explains why; no
|
||||||
|
amount of `p` improvement at the rate we can produce ADRs moves
|
||||||
|
joint pass rate meaningfully.
|
||||||
|
|
||||||
|
### Path B — Benchmark re-selection
|
||||||
|
|
||||||
|
Demote GSM8K to a stress test we honestly refuse on; re-target
|
||||||
|
the math-expert promotion to a benchmark where exact-recall,
|
||||||
|
determinism, and provenance are the discriminators (formal-math
|
||||||
|
symbolic equivalence, CORE-native teaching-corpus eval, MATH
|
||||||
|
symbolic subset).
|
||||||
|
|
||||||
|
**Held as fallback** triggered by the Path-B condition above. Not
|
||||||
|
chosen first because it forecloses the GSM8K target without
|
||||||
|
having tried the architectural fix that the compound-gap diagnosis
|
||||||
|
actually implies.
|
||||||
|
|
||||||
|
### Path C — Learned (LLM-aided) parser
|
||||||
|
|
||||||
|
Replace the rule-based parser with an LLM-assisted candidate
|
||||||
|
generator.
|
||||||
|
|
||||||
|
**Rejected** as a contract violation. ADR-0114a and the
|
||||||
|
project-wide stance (CLAUDE.md "no opaque LLM fallbacks, no
|
||||||
|
stochastic sampling, no hidden normalization") forbid this. The
|
||||||
|
candidate-graph approach in this ADR is the deterministic
|
||||||
|
analogue: parallel hypotheses with a verifier filter, no
|
||||||
|
sampling, no learned scoring.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation plan
|
||||||
|
|
||||||
|
| Phase | Module | Description |
|
||||||
|
|-------|--------|-------------|
|
||||||
|
| P1 | `generate/math_roundtrip.py` (new) | Standalone `roundtrip_matches(op, source) -> bool`. Unit tests over every existing op kind. This is the load-bearing primitive; nothing else matters if it doesn't work. |
|
||||||
|
| P2 | `generate/math_parser.py` (refactor) | `_process_statement` / `_process_question` return `list[CandidateOperation]`. Verb tables widened permissively. |
|
||||||
|
| P3 | `generate/math_candidate_graph.py` (new) | Branch enumerator + filter + decision rule. |
|
||||||
|
| P4 | `evals/gsm8k_math/runner.py` (wire) | Replace single-graph call with candidate-graph call. Preserve all current passing cases. |
|
||||||
|
| P5 | `evals/gsm8k_math/train_sample/v1/cases.jsonl` (new) | 50-case deterministic train-split sample (unsealed). |
|
||||||
|
| P6 | `evals/gsm8k_math/train_sample_runner.py` (new) | Run candidate-graph on train sample; emit `correct_rate`, `wrong_count`. |
|
||||||
|
|
||||||
|
Regression gates (must remain green at every phase):
|
||||||
|
|
||||||
|
- `core test --suite smoke`
|
||||||
|
- `core test --suite math` (existing 507/507)
|
||||||
|
- `core test --suite algebra` (82/82)
|
||||||
|
- 150/150 public split
|
||||||
|
- 200/200 dev set
|
||||||
|
- adversarial suite (`wrong == 0`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## PR checklist (when proposing for merge)
|
||||||
|
|
||||||
|
```
|
||||||
|
What capability did this add?
|
||||||
|
→ Candidate-graph parser topology + round-trip verifier-filter.
|
||||||
|
What invariant proves the field remains valid?
|
||||||
|
→ Round-trip admissibility (new); wrong == 0 (preserved);
|
||||||
|
trace_hash byte-equality (preserved).
|
||||||
|
Which CLI suite/eval proves the lane?
|
||||||
|
→ smoke + math + algebra + 200/200 dev + train_sample_runner.
|
||||||
|
Did this avoid hidden normalization, stochastic fallback,
|
||||||
|
approximate recall, unreviewed mutation?
|
||||||
|
→ Yes. Candidate enumeration is deterministic + bounded.
|
||||||
|
Round-trip filter is deterministic byte comparison.
|
||||||
|
No learned scoring, no sampling.
|
||||||
|
If it touches user input, what trust boundary was enforced?
|
||||||
|
→ No new user-input surfaces. Round-trip reconstruction does
|
||||||
|
not echo unvalidated user content into logs.
|
||||||
|
```
|
||||||
Loading…
Reference in a new issue