core/docs/decisions/ADR-0163-gsm8k-path-to-mastery.md
Shay e705f27d2e
docs(ADR-0164,0165): incremental comprehension reader + regex scope rule (#317)
Replace the regex sentence-template front-end of the math admissibility
layer with an incremental compositional reader. Lock the architectural
boundary that regex is permitted only at the lexeme level, never as
sentence-structure templates.

ADR-0164 (Proposed) — Incremental Comprehension Reader. Word-by-word
state accumulation over a closed set of semantic categories, with the
operational lexicon living as a pack-shaped data artifact under
language_packs/data/en_core_math_v1/. Reader output type matches the
existing regex parser's output, so the binding-graph admissibility
(ADR-0132/0133/0134/0135), the solver (ADR-0116), and the verifier
(ADR-0117) stay unchanged. wrong=0 is preserved by construction —
the reader produces inputs to the existing admissibility gate, not a
bypass around it. Phased coexistence with the regex layer during
transition; regex sentence templates removed in Phase 3.

ADR-0165 (Proposed) — Regex Scope Rule. Structural invariant: regex
matches one piece of orthographic material with a closed rule
(currency literal, fraction literal, percentage, time-amount, closed
unit-noun sets), never a sentence shape. Lexeme-primitive registry is
closed and grown through the same contemplation -> proposal -> HITL
review corridor that grows vocabulary (ADR-0150 / 0152 / 0155 / 0161).
The engine acquires new recognition tools through reviewed teaching,
not through operator edits to parser code.

ADR-0163's diagnosis (front-end is the bottleneck) is reaffirmed.
Its Phase B-E prescription (regex DerivedRecognizers via
recognizer_match.py) is partially superseded by ADR-0164. ADR-0136
and its S-family (S.1 / S.2 / S.3 / S.4) have the same disposition:
regex sentence-template prescription superseded; empirical refusal
taxonomies and closed-set vocabulary preserved as lexicon seed.
The HITL corridor architecture is preserved; what flows through it
changes from regex recognizers to lexicon entries, categories, and
lexeme primitives.

Session log SESSION-2026-05-26-comprehension-reader.md captures the
narrative of how this decision emerged from the post-D.2 train-sample
baseline review (correct=3 refused=47 wrong=0, 34/47 refusals at the
question gate).

No runtime code changes. ADRs only.
2026-05-26 19:23:05 -07:00

610 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# ADR-0163 — Path to GSM8K mastery: candidate-graph admissibility via the contemplation/HITL corridor
**Status:** Proposed — *Phases BE prescription superseded by [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) (2026-05-26)*
**Date:** 2026-05-26
**Author:** Shay
**Anchor:** [[thesis-decoding-not-generating]]
**Parent:** [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md), [ADR-0119 — GSM8K eval lane](./)
**Companions:** [ADR-0149 — Recognizer pipeline](./), [ADR-0151 — Auto-proposal pipeline](./ADR-0151-auto-proposal-pipeline.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md), [ADR-0132/0133/0134/0135 — Binding graph](./)
**Superseded in part by:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
---
## Amendment 2026-05-26 — Prescription superseded by ADR-0164
After observing the post-D.2 train-sample baseline (`correct=3 refused=47
wrong=0`, with 34/47 refusals at the question gate), this ADR's *diagnosis*
is reaffirmed and its *prescription* is partially superseded.
**Preserved (load-bearing):**
- The diagnosis that the front-end (`math_candidate_parser.py`,
`math_candidate_graph.py`) is the bottleneck, not the binding graph or
the solver.
- The `wrong = 0` invariant doctrine.
- The contemplation → proposal → review HITL corridor as the
population mechanism for new recognition capability (now applied to
lexicon entries and lexeme primitives instead of regex recognizers).
- Phase A — the refusal taxonomy work. Its outputs
(`refusal_taxonomy_v*.json`) remain valid input evidence.
- Phase F — the scope expansion to public / holdout / full GSM8K.
**Superseded by ADR-0164:**
- Phases BE *prescription* — specifically, the production of regex-based
`DerivedRecognizer` records that land in
`generate/recognizer_match.py`. ADR-0164 replaces this with an
incremental compositional reader that consumes lexicon entries and
lexeme primitives. The corridor is unchanged; what flows through it
changes.
- Constraint #2 of this ADR ("No hand-rolled recognizers in `generate/`")
is *tightened* by ADR-0165: regex sentence-templates are forbidden
regardless of who writes them. Regex remains permitted at the
lexeme-primitive level only.
See [ADR-0164 §What's deprecated, what's preserved](./ADR-0164-incremental-comprehension-reader.md)
for the full transition plan and acceptance gates.
---
## Context — what the audit found
A scoping pass across the unlanded math branches and the actually-shipped
state on `main` produced a result that reframes the math architecture
question entirely.
### State of the math substrate on `main`
The following components are **already landed** (worktrees on disk are
stale forks of work that landed via other PR paths):
| Component | Status |
|---|---|
| `generate/binding_graph/` (all 7 modules: model, allocation, adapter, admissibility, units, question_target, `__init__`) | ✅ landed (ADR-0132/0133/0134/0135) |
| `generate/math_versor_arithmetic.py` (221 lines) | ✅ landed (ADR-0139/0140) |
| `generate/math_symbolic_equivalence.py` (97 lines) + `math_symbolic_normalizer.py` (371 lines) | ✅ landed (ADR-0131.1) |
| `generate/math_parser.py` (1,106 lines) | ✅ landed |
| `generate/math_candidate_parser.py` (2,232 lines) | ✅ landed |
| `generate/math_candidate_graph.py` (511 lines) | ✅ landed |
| `generate/math_problem_graph.py` (490 lines) | ✅ landed |
| `generate/math_solver.py` (506 lines), `math_verifier.py` (501 lines), `math_realizer.py` (422 lines), `math_roundtrip.py` (484 lines) | ✅ landed |
| Capability axis lanes G1..G5, S1 | ✅ landed with v1 corpora |
### Capability axis lane results on `main`
Every named capability axis passes its controlled lane at **100% with
`wrong = 0`**:
| Lane | Cases | Solved correct | Refused as expected | Wrong | Verdict |
|---|---|---|---|---|---|
| G1 verb classes | 20 | 20 | 0 | 0 | ✅ exit_criterion passed |
| G2 comparatives | 29 | 29 | 0 | 0 | ✅ wrong_count_is_zero |
| G3 numerics v1 | 26 | 20 | 6 | 0 | ✅ overall_pass: true |
| G4 multi-clause | 32 | 32 | 0 | 0 | ✅ wrong_count_is_zero |
| G5 aggregate | 20 | 20 | 0 | 0 | ✅ wrong_count_is_zero |
| S1 rate events | 20 | 20 | 0 | 0 | ✅ wrong_count_is_zero |
### GSM8K train-sample result on `main` (50 cases, ADR-0126)
```text
correct: 0
refused: 50
wrong: 0
exit_criterion: { correct_min: 10, wrong_max: 0, passed: false }
```
Every refusal reason is identical in shape:
```text
candidate_graph: no admissible candidate for statement: "<STATEMENT>"
```
Sample refused statements:
- `"Tina makes $18.00 an hour."` — rate with currency
- `"She splits it up into 25-foot sections."` — division-into-sections + unit
- `"The student council sells scented erasers in the morning before school starts to help raise money for school dances."` — descriptive setup, no extractable quantity
- `"There are some kids in camp."` — indefinite quantity ("some")
- `"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature."` — rate of change + fraction
### The reframe
The gap is **not** in operator algebra, **not** in the binding graph
internals, **not** in symbolic equivalence, **not** in the capability
axes themselves. The gap is in `generate/math_candidate_graph.py`
the admissibility surface that turns a natural-language statement into
a candidate the downstream pipeline can consume.
> **The capability axes pass at 100% because they test statement shapes
> the candidate-graph already admits. GSM8K refuses at 100% because its
> statements span shapes the candidate-graph has never been taught.**
Every downstream component (binding graph, versor arithmetic, symbolic
equivalence, multi-clause decomposer, aggregator) is **mastered in
isolation**. The lift to GSM8K is *admissibility expansion*, not
operator development.
This is the most consequential single finding in the math work to date.
It reframes the entire roadmap.
---
## Decision — what to build, in what order, under what doctrine
### Doctrine
Three non-negotiables:
1. **`wrong = 0` is invariant at every phase.** A `wrong` answer is an
architectural regression, not a tuning miss. A `refused` answer is
honest; a `wrong` answer is not. Every exit criterion in this ADR
reads `wrong_max: 0`.
2. **No hand-rolled recognizers.** New statement shapes land via the
`DerivedRecognizer` pipeline that ADR-0149/0154 already wired. The
recognizer comes from corpus exemplars, not from operator-written
regex. This honors [[thesis-decoding-not-generating]]: we teach the
engine to *find* better, not stuff it with more found patterns.
3. **Every new shape lands through the contemplation → proposal →
review corridor.** No parallel learning path. Recognizers are
proposed by contemplation (ADR-0150/0152), gated by replay-equivalence
(ADR-0057), reviewed by the operator via the HITL queue
(ADR-0161), and admitted to the active corpus only on ratification.
These three rules, applied consistently, make admissibility expansion a
**capability** of the engine rather than an editing task on the
operator.
### Phases
#### Phase A — Refusal taxonomy (measure before building)
Goal: categorize every refused statement in the GSM8K train sample by
*statement shape*, not by content.
Deliverables:
1. `evals/gsm8k_math/refusal_taxonomy/v1/taxonomy.jsonl` — one record
per refused statement, carrying `case_id`, `statement`,
`refusal_reason`, and a typed `shape_category` enum.
2. Initial shape categories (extend as the corpus grows):
- `rate_with_currency` — "Tina makes $18.00 an hour."
- `unit_partition` — "She splits it up into 25-foot sections."
- `descriptive_setup_no_quantity` — pure context with no extractable
measurement.
- `indefinite_quantity` — "some", "a few", "several".
- `fractional_rate_of_change` — "decreases to 3/4 of its temperature".
- `comparative_with_unit` — "20% more than", "twice as long as".
- `nested_question_target` — "How many more than X did Y have?"
- `temporal_aggregation` — "over five days, she earns…"
- `conditional_quantity` — "if she had 2 more, she would have…"
3. A new eval lane `evals/refusal_taxonomy/` that runs the categorizer
over an arbitrary refused-statement set and emits the histogram.
4. Acceptance: every refused statement in the 50-case sample has a
typed `shape_category`; "uncategorized" count is reported but
non-blocking.
This phase produces no recognizers and no corpus changes. It is the
load-bearing measurement that prevents Phase B from chasing the wrong
gap.
#### Phase B — Exemplar corpus per shape category
Goal: for each top-N shape category from Phase A, hand-author a small
exemplar corpus (≤ 20 statements per category) with the expected
`MathProblemGraph` shape annotated.
Deliverables:
1. `teaching/admissibility_exemplars/<shape_category>_v1.jsonl` per
category, each line carrying `statement`, `expected_graph`, and
`provenance`.
2. The exemplar corpus is **reviewed-evidence floor** material under
ADR-0057 — pack-consistent, boundary-clean, polarity affirms.
3. Top-N is chosen by Phase A's histogram. Three categories per
round; ratchet rather than scope creep.
This phase is the only place hand-authoring happens. Twenty
statements per category, three categories per round — sixty hand-
authored statements total per round. Each one is a *seed* the
contemplation loop generalizes.
#### Phase C — Contemplation ingests exemplars and emits recognizer proposals
Goal: the contemplation runner (ADR-0150/0152/0155) ingests each
exemplar corpus, decomposes the statements, and emits one or more
`DerivedRecognizer` proposals per shape category.
Deliverables:
1. Contemplation runner extended to ingest the exemplar corpus path as
a candidate source (alongside `discovery_candidates.jsonl`).
2. Each proposal carries:
- the shape category it generalizes,
- the recognizer's pattern in canonical form,
- replay-equivalence evidence against the active corpus + the
exemplar set,
- per-shape coverage metrics.
3. Proposals land in `teaching/proposals/proposals.jsonl` as usual
(ADR-0057), visible in the HITL queue (ADR-0161 §1).
4. **`wrong = 0` invariant**: each proposal's replay-equivalence gate
runs against the GSM8K train sample. If accepting the proposal
would lift `wrong` above 0 even on a single case, the proposal is
auto-rejected at the gate.
#### Phase D — Operator ratifies through HITL queue
Goal: the operator reviews each recognizer proposal through the
existing surfaces (CLI / workflow_dispatch / GitHub PR review) per
ADR-0161 §2.
Deliverables:
- No new operator surface. The proposals appear in the queue with
their shape category, exemplar coverage, replay evidence, and the
ratification CLI command.
- Operator accepts, rejects, or withdraws.
- Engineering wiring (landed alongside the operator surface, ADR-0163.D PR):
- `generate/recognizer_registry.py` — pure projection of
accepted `exemplar_corpus` proposals from the proposal log into
a sorted-tuple of :class:`RatifiedRecognizer` records.
In-process cache keyed on the log's (mtime, sha256).
- `generate/recognizer_match.py` — per-category rules-only
matchers (no LLM, no embedding) honoring the Phase C
synthesizer's narrowness rule: out-of-corpus surface forms
return None. ``parsed_anchors`` carry extracted tokens from
the statement.
- `generate/math_candidate_graph.py` — narrowest-edit guard at
the per-statement choice loop: before the existing "no
admissible candidate for statement" refusal, consult the
ratified registry. Recognized statements are skipped from
``per_sentence_choices`` (contribute zero math state),
preserving wrong=0 by construction. Empty registry is a
no-op.
- Downstream consumption of ``parsed_anchors`` (turning
recognized rate/temporal surfaces into solver state) is
Phase E follow-up.
#### Phase E — Re-baseline GSM8K train sample
Goal: after each ratification round, re-run the train-sample eval and
update the counts.
Deliverables:
1. Automated re-baseline triggered by any merge that adds a recognizer.
2. Pass criteria for this ADR:
- **Round 1 exit**: `correct ≥ 10`, `wrong = 0` on the 50-case
sample (matches the existing exit criterion in the report's
`exit_criterion` block).
- **Round 2 exit**: `correct ≥ 25`, `wrong = 0`.
- **Round 3 exit**: `correct ≥ 35`, `wrong = 0`.
3. Each round runs Phases A → B → C → D → E in sequence.
#### Phase F — Scale to public, holdout, full GSM8K
Once the train sample clears Round 3, scope expands:
| Split | Cases | Target |
|---|---|---|
| `public/v1` | 200 | `correct ≥ 0.5 × cases`, `wrong = 0` |
| `holdout/v1` | 200 | first run is measurement-only; do not tune against |
| Full GSM8K (8,500) | 8,500 | post-Phase F follow-up ADR; out of scope here |
The holdout run is **never** used to drive recognizer additions. Per
ADR-0114a doctrine, holdout is the OOD ratio check; tuning against it
would invalidate the eval.
---
## Constraints (non-negotiable)
1. **`wrong = 0` at every phase, every round, every split.** Refusals
are honest; wrong answers are architectural regressions. Any
recognizer that would lift `wrong` above 0 is auto-rejected by the
replay gate, never by operator judgment alone.
2. **No hand-rolled recognizers in `generate/`.** Every recognizer
added to the runtime comes from the contemplation → proposal →
review corridor. Phase B's exemplar corpus is **input** to that
corridor, not output of it. A PR that adds a regex-style
recognizer directly to `math_candidate_parser.py` violates this ADR
and must be rejected.
3. **Replay-equivalence is a precondition, never permission.** Per
ADR-0057, replay-equivalence makes a proposal *eligible for
review*, not *automatically accepted*. This ADR does not weaken
that.
4. **Active corpus mutation only via `accept_proposal`.** Per ADR-0152
and ADR-0156/0158, the only path that mutates the active teaching
corpus is the reviewed accept path. Recognizer additions land via
that path or not at all.
5. **No tuning against holdout.** Phase F's holdout split is
measurement-only. Tuning against it makes the eval lie.
6. **Determinism preserved.** Each round's recognizer addition is a
reviewed, append-only mutation. GSM8K runs at any historical SHA
replay byte-identically given the corpus at that SHA.
---
## Out of scope
This ADR does not commit to:
- a frontier-model comparison harness beyond what ADR-0119 already
scoped;
- a benchmark publication strategy;
- patent prep work;
- Rust backend parity for the math path (waiting on Python semantics
to lock, per CLAUDE.md work-sequencing);
- the full GSM8K split (8,500 problems) — that lives in a follow-up
ADR after Phase F clears `public`;
- non-GSM8K math benchmarks (MATH, AQuA, ASDiv) — scoped by separate
ADRs once the corridor proves itself on GSM8K;
- multimodal math (charts, geometry images);
- a math-specific workbench surface — the existing Workbench (ADR-0160
/ 0162) is sufficient; lane-level inspection of refusal histograms
becomes a `RefusalHistogramPanel` in the Eval Center (W-030) once
that lands.
---
## Implementation plan — first three PRs
### PR 1 — Phase A scaffolding (refusal taxonomy)
- `evals/refusal_taxonomy/` lane: contract.md, runner.py, v1/cases.jsonl
(mirrors the 50 refused statements from `train_sample/v1/report.json`).
- `evals/refusal_taxonomy/v1/shape_categories.py` — the enum.
- `core teaching refusal-taxonomy --input <path>` CLI command for
re-running over an arbitrary refused set.
- Tests pin the enum coverage and the shape-categorizer's
deterministic output.
- Produces an initial histogram of the 50-case sample.
### PR 2 — Phase B round 1 exemplar corpora
For the top three shape categories from PR 1's histogram, hand-author
≤ 20 exemplar statements each with expected `MathProblemGraph` shape.
No runtime change.
### PR 3 — Phase C contemplation extension
Extend the contemplation runner to ingest exemplar paths as candidate
sources. Surface the per-shape coverage metric in the proposal log.
No new ratification path; existing HITL queue (ADR-0161) handles it.
After PR 3 lands, the contemplation runner produces recognizer
proposals; the operator ratifies; Phase E re-baseline confirms `correct
≥ 10, wrong = 0`. Round 1 closes.
---
## Round 1 — what actually shipped (2026-05-26 amendment)
The corridor closed end-to-end in a single session, faster than the
implementation plan above projected. Five PRs landed in order:
| Phase | PR | Title |
|---|---|---|
| A | #297 | refusal taxonomy lane — 9 shape categories, 72% of 50-case sample categorized |
| B | #298 | exemplar corpora — descriptive_setup_no_quantity / temporal_aggregation / rate_with_currency (20 each) |
| C | #301 | recognizer synthesis + admissibility replay gate; three pending proposals in live log |
| D | #302 | candidate-graph wiring via single-edit skip-only `continue` |
| D-ratify | #304 | operator accepted all three Phase C recognizers |
### First measured lift on GSM8K train_sample
```
correct: 3 (up from 0 — first non-zero correct count ever)
refused: 47 (down from 50)
wrong: 0 (unchanged — the invariant holds)
exit_criterion: { correct_min: 10, wrong_max: 0, passed: false }
```
Lifted cases:
- `gsm8k-train-sample-v1-0014` — "Bob can shuck 10 oysters in 5 minutes.
How many oysters can he shuck in 2 hours?" → 240
- `gsm8k-train-sample-v1-0018` — "Xavier plays football. During 15
minutes Xavier can score 2 goals on average..." → 16
- `gsm8k-train-sample-v1-0042` — "Ella has 4 bags with 20 apples in each
bag and six bags with 25 apples in each bag. If Ella sells 200
apples, how many apples does she have left?" → 30
**Capability-axis floor preserved** — G1..G5 + S1 all report `wrong = 0`
post-ratification, byte-identical to the pre-Phase-D baseline.
### Unexpected positive observation
None of the three lifts are pure `descriptive_setup_no_quantity` cases —
they all involve temporal or aggregation framings. Phase D's skip-only
wiring is doing more useful work than the projection suggested: when a
previously-refusing statement is skipped, the *question* + *remaining
statements* together carry enough math for the existing solver to
produce the right answer. **A Phase B round 2 (more shape categories
from the uncategorized 14) may be a more direct path to clearing Round
1 exit (`correct ≥ 10`) than the originally-planned Phase D.2
(parsed_anchors solver plumbing).** Worth measuring before scoping
Phase D.2.
### Phase E status
The Phase E re-baseline harness (versioned baselines under
`evals/gsm8k_math/train_sample/v1/baselines/`, workflow_dispatch +
nightly schedule, `LiftReport` schema, append-only history) was briefed
but not dispatched in this session. The re-baseline above was produced
by running `evals.gsm8k_math.train_sample.v1.runner` against
`origin/main` post-#304. Phase E will automate this.
See [SESSION-2026-05-26-corridor-closure.md](../sessions/SESSION-2026-05-26-corridor-closure.md)
for the full session ledger.
## Phase D.2 amendment — discrete_count_statement injection v1
Phase D.2 v1 plumbs `parsed_anchors` from one round-2 recognizer
(`discrete_count_statement`) into the candidate-graph as
`CandidateInitial`. The wiring is the first PR where a recognizer's
matcher output becomes solver input; wrong=0 moves from "skip-only by
construction" to **five layered safety nets** that all must hold:
1. **Matcher narrowness**`_try_extract_discrete_count_anchor` refuses
on ambiguity: requires a single proper-noun subject, a closed
possession-verb whitelist (`has`/`have`/`had`), exactly one numeric
token, `count_kind ∈ observed_count_kinds`, `counted_noun ∈
observed_counted_nouns`, no clause-split connectives.
2. **Extraction correctness** — the recognizer's match returns
`parsed_anchors=()` (detection-only fallback) when the narrowness
rules fail; the per-category injector returns `()` on any
construction failure.
3. **Injection correctness** — the built `CandidateInitial` is gated by
`_initial_admissible` upstream of the Cartesian product; failures
under-admit (return `()`) rather than over-admit.
4. **Replay gate** — propose-time `run_admissibility_replay_gate`
auto-rejects extraction changes that lift GSM8K wrong count.
5. **Multi-branch decision rule** — when an injected candidate
disagrees with another branch's answer, the candidate-graph
refuses.
**Re-baseline (GSM8K train_sample v1, post-D.2 v1):**
`correct=3, refused=47, wrong=0`**identical to the pre-D.2 baseline**.
The framework lands and is operational, but no GSM8K train_sample case
has a discrete_count statement that simultaneously (a) the existing
parser misses, (b) carries a counted_noun in the spec's observed lemma
set, (c) carries exactly one numeric token, and (d) carries no
clause-split connectives. Empirical lift in v1 = 0 cases; the bottleneck
is **other recognizer categories** (rate_with_currency, temporal_aggregation,
multiplicative_aggregation, currency_amount) whose injectors return `()`
(skip-only fallback) until follow-up PRs D.2.2..D.2.5 plumb them.
**Operator caveat — matcher behavior, not canonical_pattern.**
Round-1's ratified `discrete_count_statement` spec is unchanged. The
matcher's behavior on the spec's `canonical_pattern` has been extended
from detection-only to populated `parsed_anchors`. Re-ratification is
not required for this extension; if policy requires re-ratification
when matcher behavior changes, the registry digest provides byte-stable
provenance.
**G1..G5 + S1 wrong=0 invariant:** 222 passed / 2 pre-existing
report-comparison failures / 3 skipped — byte-identical to pre-D.2.
**Solver code: unchanged.** The injector returns the same
`CandidateInitial` type the existing parser produces; the solver runs
unchanged.
**Follow-up PRs (D.2.x):**
- D.2.2 — `rate_with_currency` parsed_anchors → solver state
- D.2.3 — `temporal_aggregation` parsed_anchors → solver state
- D.2.4 — `multiplicative_aggregation` parsed_anchors → solver state
- D.2.5 — `currency_amount` parsed_anchors → solver state
Each ships in its own PR after the operator reviews D.2 v1's framework
and empirical lift; the dispatch table in
`generate/recognizer_anchor_inject.py` is the single registration site.
---
## Acceptance criteria
This ADR is ratifiable when:
1. The audit findings above are independently verifiable by running
each capability axis lane on `main` and observing `wrong = 0`.
2. The GSM8K train-sample `correct: 0, refused: 50, wrong: 0` baseline
is reproducible at the current commit SHA.
3. The phase ordering (A → B → C → D → E → F) does not allow Phase B
to start before Phase A produces a histogram, nor Phase C before
Phase B writes exemplars.
4. The `wrong = 0` invariant is enforced as an auto-reject in the
replay-equivalence gate, not as a post-hoc operator check.
This ADR is **delivered** when:
5. GSM8K `public/v1` (200 cases) reaches `correct ≥ 100, wrong = 0`.
6. GSM8K `holdout/v1` measurement-only run is recorded once at the
end and never used to drive recognizer additions.
---
## Consequences
### Positive
- The math roadmap is reduced from "build operators, build axes, build
decomposer, build aggregator" to **one** problem: expand the
candidate-graph's admissibility surface through the contemplation
corridor. Every other math component is mastered.
- Recognizer additions become a *capability* of the engine, not an
editing task on the operator. The thesis ("decodes, not generates")
manifests in the math lane directly.
- The exit criterion (`wrong = 0` at every round) is enforceable by the
replay gate, not by operator vigilance.
- The HITL queue (ADR-0161) absorbs the curriculum-expansion pressure
the master plan flagged as a future risk. Math is the first lane
that scales through it.
### Negative
- Phase A (refusal taxonomy) is upfront measurement work that ships no
capability. Two-three days of audit before any GSM8K case starts
passing. Worth it; the alternative is operators chasing whichever
problem shape caught their eye first.
- The exemplar corpus (Phase B) is hand-authored. Sixty statements
per round, hand-checked for shape correctness, is real work. The
alternative — auto-mining exemplars from GSM8K itself — would
violate the holdout discipline and tune against the benchmark we're
trying to honestly measure.
- The `wrong = 0` auto-reject gate may auto-reject proposals that are
*almost* right. This is intentional. An almost-right recognizer
that produces one wrong answer is worse than a refusal.
### Risks
- **The taxonomy could fragment.** Mitigation: cap initial shape
categories at ~9 and require every new category to cite ≥ 3
refused statements. Phase A acceptance test enforces this.
- **The HITL queue could backlog.** ADR-0161 §4's pending cap (256)
applies here. If math proposals saturate the queue, the operator
raises the cap via repo variable or pauses contemplation runs.
- **Recognizer generalization could overfit to exemplars.** Mitigation:
every recognizer is replayed against the *entire* GSM8K train sample
and the public capability axes; regression on any axis auto-rejects.
---
## Cross-references
- [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md) — perturbation, OOD ratio, depth curve obligations the math lane must keep honoring
- [ADR-0119 — GSM8K eval lane](./) — eval lane definition
- [ADR-0131.G.* — capability axis lanes](./) — G1..G5, S1 mastered
- [ADR-0132/0133/0134/0135 — binding graph](./) — landed substrate
- [ADR-0139/0140 — versor arithmetic](./) — landed operator algebra
- [ADR-0149/0154 — recognizer pipeline](./) — substrate this ADR builds on
- [ADR-0150/0152 — autonomous contemplation + learning-arc corridor](./ADR-0152-learning-arc-demo.md) — proposal source
- [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md) — async producer
- [ADR-0057 — proposal review + replay-equivalence](./ADR-0057-teaching-chain-proposal-review.md) — gating discipline
- [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md) — review surface
- [CLAUDE.md](../../CLAUDE.md) — `wrong = 0` discipline, no hidden normalization, exact recall, proposal-only learning
### Memory cross-references
- [[thesis-decoding-not-generating]] — the load-bearing thesis this ADR
applies to math. Every recognizer comes from the engine learning a
shape, not from the operator stuffing a regex.
- [[feedback-address-critiques-dont-waive]] — the audit critique
("the gap is admissibility, not operators") is acted on here, not
noted.
- [[feedback-adr-cross-reference-discipline]] — every substrate this
ADR builds on is cited; no parallel mechanism is introduced.
- [[feedback-cleanup-as-you-find]] — the stale `feat/adr-0131-*` and
`feat/binding-graph-phase*` branches on disk should be deleted as a
hygiene PR after this ADR ratifies; the work is already on main.
- [[feedback-scope-time-is-cheap]] — Phase A is the "pause and scope"
move applied to math. Two-three days of taxonomy before any
recognizer work prevents weeks of chasing the wrong gap.