core/docs/decisions/ADR-0164-incremental-comprehension-reader.md

ADR-0164 — Incremental Comprehension Reader (replaces regex sentence-template parsing)

Status: Accepted (ratified by ADR-0207, 2026-06-03) — Phase 1+2 shipped; remainder per ADR-0207 §5 Date: 2026-05-26 Author: Shay Anchor: thesis-decoding-not-generating Parent: ADR-0163 — Path to GSM8K mastery Companions: ADR-0165 — Regex Scope Rule, ADR-0132/0133/0134/0135 — Binding graph, ADR-0150/0152/0155/0161 — Contemplation / HITL corridor, ADR-0114a — Anti-overfitting proof obligations Supersedes in part:

• ADR-0163 §Phase B–E prescription (the regex-based DerivedRecognizer production path). Its diagnosis and its HITL corridor are preserved.
• ADR-0136 — Statement Layer Corridor and the ADR-0136.S.1–S.4 sub-family (regex sentence-template additions). Their empirical refusal taxonomies are preserved as input evidence; the regex prescription is replaced.

---

Context — why the front-end was the bottleneck, and why the prescribed fix doesn't fix it

ADR-0163 correctly identified that the GSM8K capability gap sits before the binding graph and solver, in generate/math_candidate_parser.py and generate/math_candidate_graph.py. The downstream substrate (MathProblemGraph, the binding-graph admissibility check, the solver, the verifier, the realizer) is mastered in isolation and passes every controlled capability axis at 100% with wrong = 0. GSM8K refuses at near-100% because its statements span surface shapes the front-end has never been taught.

That diagnosis is preserved verbatim by this ADR.

The prescription of ADR-0163 — broaden the recognizer set via the contemplation → proposal → review corridor, where each accepted recognizer is a typed regex matcher in generate/recognizer_match.py — does not fix the underlying problem. It institutionalizes it.

A regex template is, by construction, an enumeration of one surface shape. Each accepted recognizer covers exactly the cases its pattern matches and refuses on every novel phrasing of the same underlying mathematical structure. The post-D.2 baseline measured this directly:

GSM8K train_sample/v1:  correct=3  refused=47  wrong=0
  exit_criterion: { correct_min: 10, wrong_max: 0, passed: false }

The refusal split is diagnostic:

• 34/47 are no admissible candidate for question: — the statements parsed, but the question surface form did not match any of the ~6 question regexes in math_candidate_parser.py (Pattern A/B/C, capacity, earnings, conditional-op).
• 9/47 are no admissible candidate for statement: — a statement hit a recognizer gap (fractions, rate-with-currency, periodic temporal).
• 4/47 are no branch produced a solvable graph — statements + question admitted but the solver couldn't close.

The question grammar is the dominant bottleneck. The current question patterns try to enumerate ~6 frames of "what an English math-problem question looks like." English doesn't have a closed grammar for math-problem questions, so the enumeration is unbounded and the refusal rate climbs with linguistic diversity. Adding a seventh, eighth, twentieth pattern is not a limit-decreasing operation; the refusal-rate ceiling is set by the regex template approach itself.

---

Diagnosis — regex sentence-templates overfit by design

A regex template at the sentence-structure level claims that a class of meanings (e.g. "ask for a residual quantity") has a closed orthographic form (e.g. How\s+much\s+(money|...)\s+(will|did)\s+...\s+(make|earn|...)). This claim is false for natural language. Three consequences follow:

1. Refusal is brittle. "How much will it cost him?" and "how much did he pay in total?" and "how much money will she be left with after the purchase?" all ask the solver for the same kind of output — the value of one terminal-state quantity — but no template covers all three, and each missing template is a refusal.

2. The fix path is unbounded. Each refused phrasing produces a new recognizer. Each new recognizer adds vocabulary and structural assumptions. The set has no closure: there is no point at which "all GSM8K question shapes have been covered" because the set of question shapes is not finite.

3. The model loses comprehension. A template either matches or it doesn't. It has no partial understanding. There is no state in which the engine has "read three words and narrowed the interpretation" — the pattern matches the whole sentence or refuses. That is the opposite of how comprehension works.

ADR-0163's pathway (recognizer-via-contemplation) addresses who writes the regex (the contemplation loop, not the operator). It does not address whether regex sentence-templates are the right representation at all. They are not.

---

Decision — incremental comprehension reader

Replace the regex sentence-template front-end with an incremental compositional reader that processes one token at a time, maintains an immutable partial-comprehension state, and produces the same downstream types (CandidateInitial, Operation, MathProblemGraph, BoundUnknown-input fields) the regex parser produces today.

The downstream substrate is unchanged. The binding graph, admissibility check, solver, verifier, realizer, and round-trip filter all stay in place and continue to enforce the wrong = 0 invariant.

Three components

1. Operational lexicon (data, not code).

Each word in the comprehension vocabulary maps to a semantic category and an update rule. The category carries the generalization; adding a word is adding a lookup, never a rule.

Example category set (closed, ADR-tracked, extended only by ratification):

Category	Examples	Role in reader state
question_open	how, what	open question frame
question_continuous_qty	much, long, far, old	continuous-quantity question
question_discrete_qty	many	discrete-count question
question_comparative	more, less, longer, fewer	mark question as difference
residual_modifier	left, remaining, after	terminal-state residual
aggregate_modifier	total, in all, altogether, combined	sum across entities
accumulation_verb	earn, make, gain, accumulate, save	additive op
depletion_verb	spend, pay, lose, give	subtractive op
transfer_verb	give, send, pass	transfer op
distributive_modifier	each, per	bind rate or multiply
currency_unit_noun	money, dollars, profit, income, savings, cost	unit class: currency
count_unit_noun	apples, books, kids, chickens, …	unit class: countable
time_unit_noun	hour, day, week, minute, year	unit class: time
entity_pronoun	she, he, they, it	binds resolved entity
proper_noun_entity	Tina, Marion, Jen, …	binds entity directly

The lexicon lives under language_packs/data/en_core_math_v1/ parallel to en_core_cognition_v1 and en_core_relations_v1, with the same loader discipline, the same manifest-checksum rule (CLAUDE.md §Semantic Pack Discipline), and the same review pathway (ADR-0150/0152/0155/0161). New lexicon entries enter through reviewed teaching, never via operator edits.

The vocabulary already collected in math_candidate_parser.py — _MASS_NOUNS, _PATTERN_A_VERBS, _PATTERN_B_VERBS, _PATTERN_C_VERBS, _CAPACITY_VERB_PATTERN, _EARNINGS_VERB_PATTERN, ADD_VERBS, SUBTRACT_VERBS, TRANSFER_VERBS, _FEMALE_NAMES, _MALE_NAMES — is ported wholesale as the seed corpus of the new lexicon. That ratified vocabulary is good work; only its container (regex character classes inside sentence templates) is wrong.

2. Partial-comprehension state (immutable).

ComprehensionState:
  entities:        tuple[EntityRef, ...]      # who's been mentioned
  quantities:      tuple[QuantityRef, ...]    # numbers with units, attached or floating
  operations:      tuple[PartialOp, ...]      # verb-induced operations, possibly incomplete
  question_target: QuestionTargetSlot | None  # what's being asked, possibly partial
  expectation:     ExpectationFrame | None    # what category would close the current frame

expectation is the load-bearing field for recontextualization. After reading "How much money will she", the state's expectation is "an accumulation verb, a depletion verb, a residual modifier, or a state-continuation verb." Each closes the question frame differently. The expectation is what tells the reader how to interpret an ambiguous next word.

State is frozen-dataclass immutable. Canonical-bytes serialization (sorted-key, fixed-precision) keeps trace_hash deterministic per CLAUDE.md §Runtime Surface Contract.

3. Deterministic reader (state machine over categories).

apply_word(state, word) -> state | Refusal

For each token:

1. Lexical primitive scan (ADR-0165): try to match orthographic primitives — currency literal, fraction literal, numeric literal, percentage literal, time-unit noun — in priority order. If one fires, the token becomes a typed lexeme with extracted value(s) and category.
2. Lexicon lookup: if no primitive fired, look up the surface form in the operational lexicon. If absent, refuse with unknown_word: <token> (position N).
3. Expectation check: if the token's category satisfies state.expectation, apply the update rule. If not — and the category is a legal frame opener at this position — close the current frame and open a new one. If neither — refuse with unexpected_category: got <cat>, expected <frame>.
4. Emit new state.

End-of-sentence: the state must satisfy a finalization predicate (question_target is bound, operations have their operands, dangling quantities have unit attachments). Otherwise refuse with unfinished_frame.

The reader is a deterministic shift-reduce parser over semantic categories, not over tokens. The category set is ~20 items; the composition rules total 30–50. Adding a verb does not change a rule. Adding a category requires an ADR.

Output

The reader emits one of:

• A MathProblemGraph (and the underlying CandidateInitial / Operation tuple) ready for the existing candidate-graph admissibility layer, or
• A typed ReaderRefusal carrying the token position, the failed expectation, and the closest legal next category. Refusals are the evidence the teaching loop chews on (Phase E below).

Downstream consumption is unchanged. The binding-graph adapter (ADR-0133), the BoundUnknown resolver (ADR-0135), the admissibility check (ADR-0134), the solver (ADR-0116), and the verifier (ADR-0117) all act on the reader's output exactly as they act on the regex parser's output today. The wrong = 0 invariant is preserved by construction because the reader does not bypass admissibility — it produces inputs to it.

---

Constraints (non-negotiable)

1. wrong = 0 at every phase, every round, every split. The reader can be more permissive about which sentences it comprehends without weakening what comprehension produces. The existing admissibility, unit-proof, and multi-branch-disagreement refusal stay in force.

2. No hidden normalization, stochastic fallback, or "best guess." The reader refuses cleanly on novel structure. No softmax over candidate parses, no nearest-template selection, no default category.

3. No regex sentence-templates. Per ADR-0165, regex is allowed only at the lexeme level (currency literal, fraction literal, etc.). Any regex that matches across word combination is a grammar template and forbidden.

4. Lexicon and category set are closed and ADR-tracked. New lexicon entries land through reviewed teaching (the existing ADR-0150/0152/0155/ 0161 corridor — preserved from ADR-0163). New categories or new composition rules require an ADR.

5. Deterministic replay. Identical input → byte-equal reader output. The ComprehensionState has canonical-bytes serialization. The reader emits a deterministic trace that feeds trace_hash.

---

What's deprecated, what's preserved

Deprecated by this ADR

• ADR-0163 §Phase B–E prescription: the production of regex-based DerivedRecognizer records that land in generate/recognizer_match.py. New recognizers in this form are blocked starting with the reader's first acceptance round. Existing recognizers remain dormant during the transition (see Coexistence below) and are removed once their categories are covered by the reader.
• ADR-0136 — Statement Layer Corridor and the sub-family ADR-0136.S.1–S.4: regex sentence-template additions to math_candidate_parser.py. The empirical refusal taxonomies they produced are preserved as input evidence for lexicon and category work. The patterns themselves are scheduled for removal once the reader covers their cases.
• The Pattern A / Pattern B / Pattern C regex blocks introduced by ADR-0163.D.4 in generate/math_candidate_parser.py (_Q_MASS_NOUN_RE, _Q_COMPARATIVE_RE, _Q_PRONOUN_VERB_RE) — replaced by the reader's question-frame composition rules.

Preserved in full

• The binding graph (ADR-0132/0133/0134/0135). The reader produces the same input types it consumes today.
• The HITL corridor (ADR-0150/0152/0155/0161). New lexicon entries and new categories ride the same contemplation → proposal → review pathway. ADR-0163's corridor architecture is correct; only what flows through it changes (lexicon entries instead of regex recognizers).
• The capability-axis lanes (G1–G5, S1). They continue to validate the downstream substrate and act as the regression net for any reader change.
• wrong = 0 doctrine and the replay-equivalence gate.
• All closed-set vocabulary previously collected by the regex parser. It is the seed of the operational lexicon.

Untouched but adjacent

• The recognizer_registry / recognizer_match modules become the lexicon loader and lexical-primitive registry rather than the regex pattern store. The interface signature changes but the corridor-driven population of these registries is preserved.

---

Phasing

Phase 1 — Question reader (where 34/47 refusals live)

Build the reader for question sentences only. The output type is narrow: just the fields BoundUnknown consumes (entity, unit, question_form). Coexist with the existing regex question patterns: reader runs first; on refusal, falls through to existing regex; on reader acceptance, regex is not invoked. Measure pickup rate against train_sample/v1 per round.

Acceptance for Phase 1:

• Reader covers ≥20/34 currently-refused question cases.
• Combined (reader + legacy) correct ≥ 10 on the 50-case sample with wrong = 0. This satisfies the Round-1 exit criterion of ADR-0163.
• Reader has zero disagreement with regex on the 6 cases where both fire (3 correct + 3 secondary), per byte-equal BoundUnknown output.

Phase 2 — Statement reader

Extend the reader to statement sentences. Coexist with existing regex statement patterns the same way. Phase out the regex statement patterns incrementally as reader coverage grows.

Acceptance for Phase 2:

• Reader covers ≥30/50 train_sample cases end-to-end (statements + question both via reader).
• correct ≥ 25 (ADR-0163 Round-2 exit) with wrong = 0.

Phase 3 — Regex layer removal

Once reader coverage ≥ regex coverage on a case-by-case basis, the regex sentence-template layer is deleted. The lexical-primitive layer (regex applied to single orthographic shapes per ADR-0165) survives — that is the correct use of regex and is not what this ADR deprecates.

Acceptance for Phase 3:

• correct ≥ 35 on train_sample, wrong = 0 (ADR-0163 Round-3 exit).
• math_candidate_parser.py no longer contains sentence-level regex patterns. Closed-set vocabulary tables remain (now consumed by the lexicon loader rather than woven into regexes).

Phase 4 — Scale

Per ADR-0163 §Phase F: public, holdout, full GSM8K. No changes to that scope from this ADR; the reader simply replaces the front-end.

---

Acceptance criteria for this ADR (Proposed → Accepted)

This ADR moves to Accepted when:

1. A ComprehensionState prototype exists in generate/comprehension/ with frozen-dataclass shape, canonical-bytes serialization, and unit tests pinning determinism.
2. The seed lexicon pack en_core_math_v1 is materialized from the existing closed-set vocabulary in math_candidate_parser.py, with the standard pack-test discipline.
3. Phase 1 acceptance is met on train_sample/v1.
4. Capability-axis lanes G1–G5, S1 remain at 100% wrong = 0 (regression net unbroken).
5. verify pinned lane SHAs continues to pass.

---

Open questions (resolve before Phase 1 PR)

1. Lexical primitive set scope. Inventory of which orthographic shapes get primitives vs. lexicon entries (currency literal, fraction literal, percentage literal, decimal literal, time-unit noun, dollar-amount, ordinal). Likely a sub-ADR (ADR-0164.1).
2. Ambiguity resolution precedence. When a token could open two frames, the precedence order. Likely a sub-ADR after Phase 1 measurement reveals which collisions are real.
3. Pronoun-entity resolution. The reader needs entity resolution anyway; the regex parser's _resolve_question_entity heuristic is a reasonable starting point but should be reviewed against the compositional model.
4. Cross-sentence state. The current regex parser is per-sentence; GSM8K problems have cross-sentence references ("she" referring to "Tina" three sentences earlier). The reader will need a ProblemReadingState that persists across sentences. Scope this in Phase 1 design.

---

Cross-references

• Bottleneck evidence: evals/gsm8k_math/train_sample/v1/report.json, refusal_taxonomy_v4.json.
• Substrate that survives: generate/binding_graph/, generate/math_solver.py, generate/math_verifier.py, generate/math_realizer.py, capability-axis lanes.
• The corridor: ADR-0150 (contemplation), ADR-0152 (learning-arc), ADR-0155 (CI contemplation runner), ADR-0161 (HITL queue).
• The boundary rule: ADR-0165.
• The anti-overfitting doctrine: ADR-0114a.
• The thesis: [[thesis-decoding-not-generating]] — the reader is a decoder. Each word narrows the space; the meaning is the accumulation, not the match.

---

Current status (2026-05-27)

Phase 1 and Phase 2 are implemented. Measurement as of post-ME-5 (PR #404):

Phase	Implemented	Tests	Eval delta
Phase 1 (question hybrid)	✅	33 tests (coexistence + question_frame)	0 net new cases (case 0027 already correct via regex)
Phase 2 (whole-problem)	✅	19 tests (reader_phase2)	0 new cases
Phase 3 (retire regex question parser)	Not started	—	—

wrong = 0 is preserved under flag ON across all 50 train-sample cases.

Why zero eval delta today. The 47 refused cases fail the reader at or before the first non-trivial token:

• Fraction/percentage literals (0004, 0005, 0010, 0041, others) — explicit Phase 2.1 deferral in lifecycle.py:344.
• Unknown words (verbs, nouns absent from the math lexicon) — most of the 47.
• Multi-quantity composition structures — out of Phase 2 scope.

Next lift path. Lexicon expansion via the ratification corridor (ADR-0150/0152/0155/0161) is the highest-leverage first step — no code change required, and a batch of 10–15 common unknown verbs is estimated to unlock ≥ 1 new Phase 2 admission. If lexicon expansion yields 0 new admissions, the bottleneck is structural (frame rules) and Phase 2.1 fraction scope becomes the next ADR target.

See docs/handoff/COMPREHENSION-READER-AUDIT.md for the full investigation.