From 8b3314f0604f0d4f0e6405d0d36896f26d087e5a Mon Sep 17 00:00:00 2001 From: Shay Date: Tue, 26 May 2026 10:56:12 -0700 Subject: [PATCH] =?UTF-8?q?docs(math):=20ADR-0163=20=E2=80=94=20path=20to?= =?UTF-8?q?=20GSM8K=20mastery=20via=20candidate-graph=20admissibility=20(p?= =?UTF-8?q?roposed)=20(#294)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit reframes the math roadmap entirely. State of main: every named math capability axis (G1..G5, S1) passes at 100% with wrong=0 on its controlled lane. binding_graph, math_versor_arithmetic, math_symbolic_equivalence, math_parser, math_candidate_parser, math_solver, math_verifier, math_realizer, math_problem_graph — all landed. The worktrees on disk are stale forks. State of GSM8K (50-case train sample): correct=0, refused=50, wrong=0. Every refusal reason is identical: "candidate_graph: no admissible candidate for statement: ". The reframe: the gap is NOT in operator algebra, NOT in binding graph internals, NOT in symbolic equivalence. The gap is in generate/math_candidate_graph.py — the admissibility surface that turns a natural-language statement into a candidate the downstream pipeline can consume. The capability axes pass at 100% because they test statement shapes the candidate-graph already admits. GSM8K refuses at 100% because its statements span shapes the candidate-graph has never been taught. Six-phase plan to lift GSM8K under the thesis "decodes, not generates": A. Refusal taxonomy (measure before building) B. Exemplar corpora per shape category (≤20 statements each, ≤3 per round) C. Contemplation runner ingests exemplars; emits DerivedRecognizer proposals D. Operator ratifies through ADR-0161 HITL queue (no new surface) E. Re-baseline GSM8K train sample. Round 1 exit: correct ≥ 10, wrong = 0. Round 2: ≥ 25. Round 3: ≥ 35. F. Scale to public/v1 (200 cases, target correct ≥ 100), then holdout (measurement-only — never tune against). Three non-negotiables: - wrong = 0 at every phase. Auto-rejected by replay gate, not by operator vigilance. - No hand-rolled recognizers in generate/. Every recognizer lands via contemplation → proposal → review corridor. - Active corpus mutation only via accept_proposal. Status: proposed. Implementation lands as three PRs starting with Phase A scaffolding. Scope discipline: docs-only. No code, no eval changes, no corpus mutation. --- .../ADR-0163-gsm8k-path-to-mastery.md | 426 ++++++++++++++++++ 1 file changed, 426 insertions(+) create mode 100644 docs/decisions/ADR-0163-gsm8k-path-to-mastery.md diff --git a/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md b/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md new file mode 100644 index 00000000..2e4582e4 --- /dev/null +++ b/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md @@ -0,0 +1,426 @@ +# ADR-0163 — Path to GSM8K mastery: candidate-graph admissibility via the contemplation/HITL corridor + +**Status:** Proposed +**Date:** 2026-05-26 +**Author:** Shay +**Anchor:** [[thesis-decoding-not-generating]] +**Parent:** [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md), [ADR-0119 — GSM8K eval lane](./) +**Companions:** [ADR-0149 — Recognizer pipeline](./), [ADR-0151 — Auto-proposal pipeline](./ADR-0151-auto-proposal-pipeline.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md), [ADR-0132/0133/0134/0135 — Binding graph](./) + +--- + +## Context — what the audit found + +A scoping pass across the unlanded math branches and the actually-shipped +state on `main` produced a result that reframes the math architecture +question entirely. + +### State of the math substrate on `main` + +The following components are **already landed** (worktrees on disk are +stale forks of work that landed via other PR paths): + +| Component | Status | +|---|---| +| `generate/binding_graph/` (all 7 modules: model, allocation, adapter, admissibility, units, question_target, `__init__`) | ✅ landed (ADR-0132/0133/0134/0135) | +| `generate/math_versor_arithmetic.py` (221 lines) | ✅ landed (ADR-0139/0140) | +| `generate/math_symbolic_equivalence.py` (97 lines) + `math_symbolic_normalizer.py` (371 lines) | ✅ landed (ADR-0131.1) | +| `generate/math_parser.py` (1,106 lines) | ✅ landed | +| `generate/math_candidate_parser.py` (2,232 lines) | ✅ landed | +| `generate/math_candidate_graph.py` (511 lines) | ✅ landed | +| `generate/math_problem_graph.py` (490 lines) | ✅ landed | +| `generate/math_solver.py` (506 lines), `math_verifier.py` (501 lines), `math_realizer.py` (422 lines), `math_roundtrip.py` (484 lines) | ✅ landed | +| Capability axis lanes G1..G5, S1 | ✅ landed with v1 corpora | + +### Capability axis lane results on `main` + +Every named capability axis passes its controlled lane at **100% with +`wrong = 0`**: + +| Lane | Cases | Solved correct | Refused as expected | Wrong | Verdict | +|---|---|---|---|---|---| +| G1 verb classes | 20 | 20 | 0 | 0 | ✅ exit_criterion passed | +| G2 comparatives | 29 | 29 | 0 | 0 | ✅ wrong_count_is_zero | +| G3 numerics v1 | 26 | 20 | 6 | 0 | ✅ overall_pass: true | +| G4 multi-clause | 32 | 32 | 0 | 0 | ✅ wrong_count_is_zero | +| G5 aggregate | 20 | 20 | 0 | 0 | ✅ wrong_count_is_zero | +| S1 rate events | 20 | 20 | 0 | 0 | ✅ wrong_count_is_zero | + +### GSM8K train-sample result on `main` (50 cases, ADR-0126) + +```text +correct: 0 +refused: 50 +wrong: 0 +exit_criterion: { correct_min: 10, wrong_max: 0, passed: false } +``` + +Every refusal reason is identical in shape: + +```text +candidate_graph: no admissible candidate for statement: "" +``` + +Sample refused statements: + +- `"Tina makes $18.00 an hour."` — rate with currency +- `"She splits it up into 25-foot sections."` — division-into-sections + unit +- `"The student council sells scented erasers in the morning before school starts to help raise money for school dances."` — descriptive setup, no extractable quantity +- `"There are some kids in camp."` — indefinite quantity ("some") +- `"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature."` — rate of change + fraction + +### The reframe + +The gap is **not** in operator algebra, **not** in the binding graph +internals, **not** in symbolic equivalence, **not** in the capability +axes themselves. The gap is in `generate/math_candidate_graph.py` — +the admissibility surface that turns a natural-language statement into +a candidate the downstream pipeline can consume. + +> **The capability axes pass at 100% because they test statement shapes +> the candidate-graph already admits. GSM8K refuses at 100% because its +> statements span shapes the candidate-graph has never been taught.** + +Every downstream component (binding graph, versor arithmetic, symbolic +equivalence, multi-clause decomposer, aggregator) is **mastered in +isolation**. The lift to GSM8K is *admissibility expansion*, not +operator development. + +This is the most consequential single finding in the math work to date. +It reframes the entire roadmap. + +--- + +## Decision — what to build, in what order, under what doctrine + +### Doctrine + +Three non-negotiables: + +1. **`wrong = 0` is invariant at every phase.** A `wrong` answer is an + architectural regression, not a tuning miss. A `refused` answer is + honest; a `wrong` answer is not. Every exit criterion in this ADR + reads `wrong_max: 0`. + +2. **No hand-rolled recognizers.** New statement shapes land via the + `DerivedRecognizer` pipeline that ADR-0149/0154 already wired. The + recognizer comes from corpus exemplars, not from operator-written + regex. This honors [[thesis-decoding-not-generating]]: we teach the + engine to *find* better, not stuff it with more found patterns. + +3. **Every new shape lands through the contemplation → proposal → + review corridor.** No parallel learning path. Recognizers are + proposed by contemplation (ADR-0150/0152), gated by replay-equivalence + (ADR-0057), reviewed by the operator via the HITL queue + (ADR-0161), and admitted to the active corpus only on ratification. + +These three rules, applied consistently, make admissibility expansion a +**capability** of the engine rather than an editing task on the +operator. + +### Phases + +#### Phase A — Refusal taxonomy (measure before building) + +Goal: categorize every refused statement in the GSM8K train sample by +*statement shape*, not by content. + +Deliverables: + +1. `evals/gsm8k_math/refusal_taxonomy/v1/taxonomy.jsonl` — one record + per refused statement, carrying `case_id`, `statement`, + `refusal_reason`, and a typed `shape_category` enum. +2. Initial shape categories (extend as the corpus grows): + - `rate_with_currency` — "Tina makes $18.00 an hour." + - `unit_partition` — "She splits it up into 25-foot sections." + - `descriptive_setup_no_quantity` — pure context with no extractable + measurement. + - `indefinite_quantity` — "some", "a few", "several". + - `fractional_rate_of_change` — "decreases to 3/4 of its temperature". + - `comparative_with_unit` — "20% more than", "twice as long as". + - `nested_question_target` — "How many more than X did Y have?" + - `temporal_aggregation` — "over five days, she earns…" + - `conditional_quantity` — "if she had 2 more, she would have…" +3. A new eval lane `evals/refusal_taxonomy/` that runs the categorizer + over an arbitrary refused-statement set and emits the histogram. +4. Acceptance: every refused statement in the 50-case sample has a + typed `shape_category`; "uncategorized" count is reported but + non-blocking. + +This phase produces no recognizers and no corpus changes. It is the +load-bearing measurement that prevents Phase B from chasing the wrong +gap. + +#### Phase B — Exemplar corpus per shape category + +Goal: for each top-N shape category from Phase A, hand-author a small +exemplar corpus (≤ 20 statements per category) with the expected +`MathProblemGraph` shape annotated. + +Deliverables: + +1. `teaching/admissibility_exemplars/_v1.jsonl` per + category, each line carrying `statement`, `expected_graph`, and + `provenance`. +2. The exemplar corpus is **reviewed-evidence floor** material under + ADR-0057 — pack-consistent, boundary-clean, polarity affirms. +3. Top-N is chosen by Phase A's histogram. Three categories per + round; ratchet rather than scope creep. + +This phase is the only place hand-authoring happens. Twenty +statements per category, three categories per round — sixty hand- +authored statements total per round. Each one is a *seed* the +contemplation loop generalizes. + +#### Phase C — Contemplation ingests exemplars and emits recognizer proposals + +Goal: the contemplation runner (ADR-0150/0152/0155) ingests each +exemplar corpus, decomposes the statements, and emits one or more +`DerivedRecognizer` proposals per shape category. + +Deliverables: + +1. Contemplation runner extended to ingest the exemplar corpus path as + a candidate source (alongside `discovery_candidates.jsonl`). +2. Each proposal carries: + - the shape category it generalizes, + - the recognizer's pattern in canonical form, + - replay-equivalence evidence against the active corpus + the + exemplar set, + - per-shape coverage metrics. +3. Proposals land in `teaching/proposals/proposals.jsonl` as usual + (ADR-0057), visible in the HITL queue (ADR-0161 §1). +4. **`wrong = 0` invariant**: each proposal's replay-equivalence gate + runs against the GSM8K train sample. If accepting the proposal + would lift `wrong` above 0 even on a single case, the proposal is + auto-rejected at the gate. + +#### Phase D — Operator ratifies through HITL queue + +Goal: the operator reviews each recognizer proposal through the +existing surfaces (CLI / workflow_dispatch / GitHub PR review) per +ADR-0161 §2. + +Deliverables: + +- No new operator surface. The proposals appear in the queue with + their shape category, exemplar coverage, replay evidence, and the + ratification CLI command. +- Operator accepts, rejects, or withdraws. + +#### Phase E — Re-baseline GSM8K train sample + +Goal: after each ratification round, re-run the train-sample eval and +update the counts. + +Deliverables: + +1. Automated re-baseline triggered by any merge that adds a recognizer. +2. Pass criteria for this ADR: + - **Round 1 exit**: `correct ≥ 10`, `wrong = 0` on the 50-case + sample (matches the existing exit criterion in the report's + `exit_criterion` block). + - **Round 2 exit**: `correct ≥ 25`, `wrong = 0`. + - **Round 3 exit**: `correct ≥ 35`, `wrong = 0`. +3. Each round runs Phases A → B → C → D → E in sequence. + +#### Phase F — Scale to public, holdout, full GSM8K + +Once the train sample clears Round 3, scope expands: + +| Split | Cases | Target | +|---|---|---| +| `public/v1` | 200 | `correct ≥ 0.5 × cases`, `wrong = 0` | +| `holdout/v1` | 200 | first run is measurement-only; do not tune against | +| Full GSM8K (8,500) | 8,500 | post-Phase F follow-up ADR; out of scope here | + +The holdout run is **never** used to drive recognizer additions. Per +ADR-0114a doctrine, holdout is the OOD ratio check; tuning against it +would invalidate the eval. + +--- + +## Constraints (non-negotiable) + +1. **`wrong = 0` at every phase, every round, every split.** Refusals + are honest; wrong answers are architectural regressions. Any + recognizer that would lift `wrong` above 0 is auto-rejected by the + replay gate, never by operator judgment alone. + +2. **No hand-rolled recognizers in `generate/`.** Every recognizer + added to the runtime comes from the contemplation → proposal → + review corridor. Phase B's exemplar corpus is **input** to that + corridor, not output of it. A PR that adds a regex-style + recognizer directly to `math_candidate_parser.py` violates this ADR + and must be rejected. + +3. **Replay-equivalence is a precondition, never permission.** Per + ADR-0057, replay-equivalence makes a proposal *eligible for + review*, not *automatically accepted*. This ADR does not weaken + that. + +4. **Active corpus mutation only via `accept_proposal`.** Per ADR-0152 + and ADR-0156/0158, the only path that mutates the active teaching + corpus is the reviewed accept path. Recognizer additions land via + that path or not at all. + +5. **No tuning against holdout.** Phase F's holdout split is + measurement-only. Tuning against it makes the eval lie. + +6. **Determinism preserved.** Each round's recognizer addition is a + reviewed, append-only mutation. GSM8K runs at any historical SHA + replay byte-identically given the corpus at that SHA. + +--- + +## Out of scope + +This ADR does not commit to: + +- a frontier-model comparison harness beyond what ADR-0119 already + scoped; +- a benchmark publication strategy; +- patent prep work; +- Rust backend parity for the math path (waiting on Python semantics + to lock, per CLAUDE.md work-sequencing); +- the full GSM8K split (8,500 problems) — that lives in a follow-up + ADR after Phase F clears `public`; +- non-GSM8K math benchmarks (MATH, AQuA, ASDiv) — scoped by separate + ADRs once the corridor proves itself on GSM8K; +- multimodal math (charts, geometry images); +- a math-specific workbench surface — the existing Workbench (ADR-0160 + / 0162) is sufficient; lane-level inspection of refusal histograms + becomes a `RefusalHistogramPanel` in the Eval Center (W-030) once + that lands. + +--- + +## Implementation plan — first three PRs + +### PR 1 — Phase A scaffolding (refusal taxonomy) + +- `evals/refusal_taxonomy/` lane: contract.md, runner.py, v1/cases.jsonl + (mirrors the 50 refused statements from `train_sample/v1/report.json`). +- `evals/refusal_taxonomy/v1/shape_categories.py` — the enum. +- `core teaching refusal-taxonomy --input ` CLI command for + re-running over an arbitrary refused set. +- Tests pin the enum coverage and the shape-categorizer's + deterministic output. +- Produces an initial histogram of the 50-case sample. + +### PR 2 — Phase B round 1 exemplar corpora + +For the top three shape categories from PR 1's histogram, hand-author +≤ 20 exemplar statements each with expected `MathProblemGraph` shape. +No runtime change. + +### PR 3 — Phase C contemplation extension + +Extend the contemplation runner to ingest exemplar paths as candidate +sources. Surface the per-shape coverage metric in the proposal log. +No new ratification path; existing HITL queue (ADR-0161) handles it. + +After PR 3 lands, the contemplation runner produces recognizer +proposals; the operator ratifies; Phase E re-baseline confirms `correct +≥ 10, wrong = 0`. Round 1 closes. + +--- + +## Acceptance criteria + +This ADR is ratifiable when: + +1. The audit findings above are independently verifiable by running + each capability axis lane on `main` and observing `wrong = 0`. +2. The GSM8K train-sample `correct: 0, refused: 50, wrong: 0` baseline + is reproducible at the current commit SHA. +3. The phase ordering (A → B → C → D → E → F) does not allow Phase B + to start before Phase A produces a histogram, nor Phase C before + Phase B writes exemplars. +4. The `wrong = 0` invariant is enforced as an auto-reject in the + replay-equivalence gate, not as a post-hoc operator check. + +This ADR is **delivered** when: + +5. GSM8K `public/v1` (200 cases) reaches `correct ≥ 100, wrong = 0`. +6. GSM8K `holdout/v1` measurement-only run is recorded once at the + end and never used to drive recognizer additions. + +--- + +## Consequences + +### Positive + +- The math roadmap is reduced from "build operators, build axes, build + decomposer, build aggregator" to **one** problem: expand the + candidate-graph's admissibility surface through the contemplation + corridor. Every other math component is mastered. +- Recognizer additions become a *capability* of the engine, not an + editing task on the operator. The thesis ("decodes, not generates") + manifests in the math lane directly. +- The exit criterion (`wrong = 0` at every round) is enforceable by the + replay gate, not by operator vigilance. +- The HITL queue (ADR-0161) absorbs the curriculum-expansion pressure + the master plan flagged as a future risk. Math is the first lane + that scales through it. + +### Negative + +- Phase A (refusal taxonomy) is upfront measurement work that ships no + capability. Two-three days of audit before any GSM8K case starts + passing. Worth it; the alternative is operators chasing whichever + problem shape caught their eye first. +- The exemplar corpus (Phase B) is hand-authored. Sixty statements + per round, hand-checked for shape correctness, is real work. The + alternative — auto-mining exemplars from GSM8K itself — would + violate the holdout discipline and tune against the benchmark we're + trying to honestly measure. +- The `wrong = 0` auto-reject gate may auto-reject proposals that are + *almost* right. This is intentional. An almost-right recognizer + that produces one wrong answer is worse than a refusal. + +### Risks + +- **The taxonomy could fragment.** Mitigation: cap initial shape + categories at ~9 and require every new category to cite ≥ 3 + refused statements. Phase A acceptance test enforces this. +- **The HITL queue could backlog.** ADR-0161 §4's pending cap (256) + applies here. If math proposals saturate the queue, the operator + raises the cap via repo variable or pauses contemplation runs. +- **Recognizer generalization could overfit to exemplars.** Mitigation: + every recognizer is replayed against the *entire* GSM8K train sample + and the public capability axes; regression on any axis auto-rejects. + +--- + +## Cross-references + +- [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md) — perturbation, OOD ratio, depth curve obligations the math lane must keep honoring +- [ADR-0119 — GSM8K eval lane](./) — eval lane definition +- [ADR-0131.G.* — capability axis lanes](./) — G1..G5, S1 mastered +- [ADR-0132/0133/0134/0135 — binding graph](./) — landed substrate +- [ADR-0139/0140 — versor arithmetic](./) — landed operator algebra +- [ADR-0149/0154 — recognizer pipeline](./) — substrate this ADR builds on +- [ADR-0150/0152 — autonomous contemplation + learning-arc corridor](./ADR-0152-learning-arc-demo.md) — proposal source +- [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md) — async producer +- [ADR-0057 — proposal review + replay-equivalence](./ADR-0057-teaching-chain-proposal-review.md) — gating discipline +- [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md) — review surface +- [CLAUDE.md](../../CLAUDE.md) — `wrong = 0` discipline, no hidden normalization, exact recall, proposal-only learning + +### Memory cross-references + +- [[thesis-decoding-not-generating]] — the load-bearing thesis this ADR + applies to math. Every recognizer comes from the engine learning a + shape, not from the operator stuffing a regex. +- [[feedback-address-critiques-dont-waive]] — the audit critique + ("the gap is admissibility, not operators") is acted on here, not + noted. +- [[feedback-adr-cross-reference-discipline]] — every substrate this + ADR builds on is cited; no parallel mechanism is introduced. +- [[feedback-cleanup-as-you-find]] — the stale `feat/adr-0131-*` and + `feat/binding-graph-phase*` branches on disk should be deleted as a + hygiene PR after this ADR ratifies; the work is already on main. +- [[feedback-scope-time-is-cheap]] — Phase A is the "pause and scope" + move applied to math. Two-three days of taxonomy before any + recognizer work prevents weeks of chasing the wrong gap.