docs(ADR-0164,0165): incremental comprehension reader + regex scope rule (#317)
Replace the regex sentence-template front-end of the math admissibility layer with an incremental compositional reader. Lock the architectural boundary that regex is permitted only at the lexeme level, never as sentence-structure templates. ADR-0164 (Proposed) — Incremental Comprehension Reader. Word-by-word state accumulation over a closed set of semantic categories, with the operational lexicon living as a pack-shaped data artifact under language_packs/data/en_core_math_v1/. Reader output type matches the existing regex parser's output, so the binding-graph admissibility (ADR-0132/0133/0134/0135), the solver (ADR-0116), and the verifier (ADR-0117) stay unchanged. wrong=0 is preserved by construction — the reader produces inputs to the existing admissibility gate, not a bypass around it. Phased coexistence with the regex layer during transition; regex sentence templates removed in Phase 3. ADR-0165 (Proposed) — Regex Scope Rule. Structural invariant: regex matches one piece of orthographic material with a closed rule (currency literal, fraction literal, percentage, time-amount, closed unit-noun sets), never a sentence shape. Lexeme-primitive registry is closed and grown through the same contemplation -> proposal -> HITL review corridor that grows vocabulary (ADR-0150 / 0152 / 0155 / 0161). The engine acquires new recognition tools through reviewed teaching, not through operator edits to parser code. ADR-0163's diagnosis (front-end is the bottleneck) is reaffirmed. Its Phase B-E prescription (regex DerivedRecognizers via recognizer_match.py) is partially superseded by ADR-0164. ADR-0136 and its S-family (S.1 / S.2 / S.3 / S.4) have the same disposition: regex sentence-template prescription superseded; empirical refusal taxonomies and closed-set vocabulary preserved as lexicon seed. The HITL corridor architecture is preserved; what flows through it changes from regex recognizers to lexicon entries, categories, and lexeme primitives. Session log SESSION-2026-05-26-comprehension-reader.md captures the narrative of how this decision emerged from the post-D.2 train-sample baseline review (correct=3 refused=47 wrong=0, 34/47 refusals at the question gate). No runtime code changes. ADRs only.
This commit is contained in:
parent
0b4a87beae
commit
e705f27d2e
10 changed files with 1001 additions and 10 deletions
|
|
@ -1,9 +1,30 @@
|
||||||
# ADR-0136 — Statement-Layer Corridor: Graduated GSM8K Admission via Parser Extension
|
# ADR-0136 — Statement-Layer Corridor: Graduated GSM8K Admission via Parser Extension
|
||||||
|
|
||||||
**Status:** Active
|
**Status:** Active — *Regex sentence-template prescription superseded by [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) (2026-05-26). Empirical taxonomies preserved.*
|
||||||
**Date:** 2026-05-23
|
**Date:** 2026-05-23
|
||||||
**Author:** CORE agents + reviewers
|
**Author:** CORE agents + reviewers
|
||||||
**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md)
|
**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md)
|
||||||
|
**Superseded in part by:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Amendment 2026-05-26 — Regex prescription superseded
|
||||||
|
|
||||||
|
This corridor's *production mechanism* — adding regex sentence-template
|
||||||
|
patterns to `generate/math_candidate_parser.py` per S-stage sub-ADR — is
|
||||||
|
superseded by ADR-0164's incremental comprehension reader. The closed-set
|
||||||
|
vocabulary collected by each S-stage (verb lists, mass-noun lists, name
|
||||||
|
lists, unit-noun lists) is **preserved** as seed input to the new
|
||||||
|
operational lexicon (ADR-0164 §Decision §1). The empirical refusal
|
||||||
|
taxonomies (`refusal_taxonomy_v*.json`) are **preserved** as input
|
||||||
|
evidence for category and primitive development. The regex patterns
|
||||||
|
themselves are scheduled for removal during ADR-0164 Phase 3.
|
||||||
|
|
||||||
|
Per [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), regex
|
||||||
|
remains permitted at the lexeme-primitive level (currency literal,
|
||||||
|
fraction literal, etc.) and forbidden at the sentence-structure level.
|
||||||
|
|
||||||
|
---
|
||||||
**Depends on:**
|
**Depends on:**
|
||||||
[ADR-0115](./ADR-0115-math-problem-parser-and-graph.md),
|
[ADR-0115](./ADR-0115-math-problem-parser-and-graph.md),
|
||||||
[ADR-0116](./ADR-0116-deterministic-solver.md),
|
[ADR-0116](./ADR-0116-deterministic-solver.md),
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
# ADR-0136.S.1 — Rate/Event Statement Parsing
|
# ADR-0136.S.1 — Rate/Event Statement Parsing
|
||||||
|
|
||||||
**Status:** Accepted
|
**Status:** Accepted — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
**Parent:** ADR-0136 (Statement Layer Corridor) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||||
**Date:** 2026-05-23
|
**Date:** 2026-05-23
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
# ADR-0136.S.2 — Conditional-Op Question (Statement-Layer Corridor)
|
# ADR-0136.S.2 — Conditional-Op Question (Statement-Layer Corridor)
|
||||||
|
|
||||||
**Status:** Active
|
**Status:** Active — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||||
**Date:** 2026-05-23
|
**Date:** 2026-05-23
|
||||||
**Parent:** [ADR-0136](./ADR-0136-statement-layer-corridor.md)
|
**Parent:** [ADR-0136](./ADR-0136-statement-layer-corridor.md) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
# ADR-0136.S.4 — Novel Initial-Form Subject-Slot Widenings
|
# ADR-0136.S.4 — Novel Initial-Form Subject-Slot Widenings
|
||||||
|
|
||||||
**Status:** Accepted
|
**Status:** Accepted — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
**Parent:** ADR-0136 (Statement Layer Corridor) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||||
**Date:** 2026-05-23
|
**Date:** 2026-05-23
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
# ADR-0136.S.3 — Compound Initial-Mutation Extractor
|
# ADR-0136.S.3 — Compound Initial-Mutation Extractor
|
||||||
|
|
||||||
**Status:** Accepted
|
**Status:** Accepted — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
**Parent:** ADR-0136 (Statement Layer Corridor) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||||
**Date:** 2026-05-23
|
**Date:** 2026-05-23
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,49 @@
|
||||||
# ADR-0163 — Path to GSM8K mastery: candidate-graph admissibility via the contemplation/HITL corridor
|
# ADR-0163 — Path to GSM8K mastery: candidate-graph admissibility via the contemplation/HITL corridor
|
||||||
|
|
||||||
**Status:** Proposed
|
**Status:** Proposed — *Phases B–E prescription superseded by [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) (2026-05-26)*
|
||||||
**Date:** 2026-05-26
|
**Date:** 2026-05-26
|
||||||
**Author:** Shay
|
**Author:** Shay
|
||||||
**Anchor:** [[thesis-decoding-not-generating]]
|
**Anchor:** [[thesis-decoding-not-generating]]
|
||||||
**Parent:** [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md), [ADR-0119 — GSM8K eval lane](./)
|
**Parent:** [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md), [ADR-0119 — GSM8K eval lane](./)
|
||||||
**Companions:** [ADR-0149 — Recognizer pipeline](./), [ADR-0151 — Auto-proposal pipeline](./ADR-0151-auto-proposal-pipeline.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md), [ADR-0132/0133/0134/0135 — Binding graph](./)
|
**Companions:** [ADR-0149 — Recognizer pipeline](./), [ADR-0151 — Auto-proposal pipeline](./ADR-0151-auto-proposal-pipeline.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md), [ADR-0132/0133/0134/0135 — Binding graph](./)
|
||||||
|
**Superseded in part by:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Amendment 2026-05-26 — Prescription superseded by ADR-0164
|
||||||
|
|
||||||
|
After observing the post-D.2 train-sample baseline (`correct=3 refused=47
|
||||||
|
wrong=0`, with 34/47 refusals at the question gate), this ADR's *diagnosis*
|
||||||
|
is reaffirmed and its *prescription* is partially superseded.
|
||||||
|
|
||||||
|
**Preserved (load-bearing):**
|
||||||
|
|
||||||
|
- The diagnosis that the front-end (`math_candidate_parser.py`,
|
||||||
|
`math_candidate_graph.py`) is the bottleneck, not the binding graph or
|
||||||
|
the solver.
|
||||||
|
- The `wrong = 0` invariant doctrine.
|
||||||
|
- The contemplation → proposal → review HITL corridor as the
|
||||||
|
population mechanism for new recognition capability (now applied to
|
||||||
|
lexicon entries and lexeme primitives instead of regex recognizers).
|
||||||
|
- Phase A — the refusal taxonomy work. Its outputs
|
||||||
|
(`refusal_taxonomy_v*.json`) remain valid input evidence.
|
||||||
|
- Phase F — the scope expansion to public / holdout / full GSM8K.
|
||||||
|
|
||||||
|
**Superseded by ADR-0164:**
|
||||||
|
|
||||||
|
- Phases B–E *prescription* — specifically, the production of regex-based
|
||||||
|
`DerivedRecognizer` records that land in
|
||||||
|
`generate/recognizer_match.py`. ADR-0164 replaces this with an
|
||||||
|
incremental compositional reader that consumes lexicon entries and
|
||||||
|
lexeme primitives. The corridor is unchanged; what flows through it
|
||||||
|
changes.
|
||||||
|
- Constraint #2 of this ADR ("No hand-rolled recognizers in `generate/`")
|
||||||
|
is *tightened* by ADR-0165: regex sentence-templates are forbidden
|
||||||
|
regardless of who writes them. Regex remains permitted at the
|
||||||
|
lexeme-primitive level only.
|
||||||
|
|
||||||
|
See [ADR-0164 §What's deprecated, what's preserved](./ADR-0164-incremental-comprehension-reader.md)
|
||||||
|
for the full transition plan and acceptance gates.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
390
docs/decisions/ADR-0164-incremental-comprehension-reader.md
Normal file
390
docs/decisions/ADR-0164-incremental-comprehension-reader.md
Normal file
|
|
@ -0,0 +1,390 @@
|
||||||
|
# ADR-0164 — Incremental Comprehension Reader (replaces regex sentence-template parsing)
|
||||||
|
|
||||||
|
**Status:** Proposed
|
||||||
|
**Date:** 2026-05-26
|
||||||
|
**Author:** Shay
|
||||||
|
**Anchor:** [[thesis-decoding-not-generating]]
|
||||||
|
**Parent:** [ADR-0163 — Path to GSM8K mastery](./ADR-0163-gsm8k-path-to-mastery.md)
|
||||||
|
**Companions:** [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), [ADR-0132/0133/0134/0135 — Binding graph](./), [ADR-0150/0152/0155/0161 — Contemplation / HITL corridor](./), [ADR-0114a — Anti-overfitting proof obligations](./ADR-0114a-anti-overfitting-proof-obligations.md)
|
||||||
|
**Supersedes in part:**
|
||||||
|
- [ADR-0163](./ADR-0163-gsm8k-path-to-mastery.md) §Phase B–E *prescription* (the regex-based `DerivedRecognizer` production path). Its diagnosis and its HITL corridor are preserved.
|
||||||
|
- [ADR-0136 — Statement Layer Corridor](./ADR-0136-statement-layer-corridor.md) and the [ADR-0136.S.1–S.4](./) sub-family (regex sentence-template additions). Their empirical refusal taxonomies are preserved as input evidence; the regex prescription is replaced.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context — why the front-end was the bottleneck, and why the prescribed fix doesn't fix it
|
||||||
|
|
||||||
|
ADR-0163 correctly identified that the GSM8K capability gap sits *before* the
|
||||||
|
binding graph and solver, in `generate/math_candidate_parser.py` and
|
||||||
|
`generate/math_candidate_graph.py`. The downstream substrate
|
||||||
|
(`MathProblemGraph`, the binding-graph admissibility check, the solver, the
|
||||||
|
verifier, the realizer) is mastered in isolation and passes every controlled
|
||||||
|
capability axis at 100% with `wrong = 0`. GSM8K refuses at near-100% because
|
||||||
|
its statements span surface shapes the front-end has never been taught.
|
||||||
|
|
||||||
|
That diagnosis is preserved verbatim by this ADR.
|
||||||
|
|
||||||
|
The *prescription* of ADR-0163 — broaden the recognizer set via the
|
||||||
|
contemplation → proposal → review corridor, where each accepted recognizer is
|
||||||
|
a typed regex matcher in `generate/recognizer_match.py` — does not fix the
|
||||||
|
underlying problem. It institutionalizes it.
|
||||||
|
|
||||||
|
A regex template is, by construction, an enumeration of one surface shape.
|
||||||
|
Each accepted recognizer covers exactly the cases its pattern matches and
|
||||||
|
refuses on every novel phrasing of the same underlying mathematical
|
||||||
|
structure. The post-D.2 baseline measured this directly:
|
||||||
|
|
||||||
|
```text
|
||||||
|
GSM8K train_sample/v1: correct=3 refused=47 wrong=0
|
||||||
|
exit_criterion: { correct_min: 10, wrong_max: 0, passed: false }
|
||||||
|
```
|
||||||
|
|
||||||
|
The refusal split is diagnostic:
|
||||||
|
|
||||||
|
- **34/47** are `no admissible candidate for question:` — the statements
|
||||||
|
parsed, but the question surface form did not match any of the ~6 question
|
||||||
|
regexes in `math_candidate_parser.py` (Pattern A/B/C, capacity, earnings,
|
||||||
|
conditional-op).
|
||||||
|
- **9/47** are `no admissible candidate for statement:` — a statement hit a
|
||||||
|
recognizer gap (fractions, rate-with-currency, periodic temporal).
|
||||||
|
- **4/47** are `no branch produced a solvable graph` — statements + question
|
||||||
|
admitted but the solver couldn't close.
|
||||||
|
|
||||||
|
The question grammar is the dominant bottleneck. The current question
|
||||||
|
patterns try to enumerate ~6 frames of "what an English math-problem question
|
||||||
|
looks like." English doesn't have a closed grammar for math-problem
|
||||||
|
questions, so the enumeration is unbounded and the refusal rate climbs with
|
||||||
|
linguistic diversity. Adding a seventh, eighth, twentieth pattern is not a
|
||||||
|
limit-decreasing operation; the refusal-rate ceiling is set by the regex
|
||||||
|
template approach itself.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Diagnosis — regex sentence-templates overfit by design
|
||||||
|
|
||||||
|
A regex template at the **sentence-structure** level claims that a class of
|
||||||
|
meanings (e.g. "ask for a residual quantity") has a closed orthographic form
|
||||||
|
(e.g. `How\s+much\s+(money|...)\s+(will|did)\s+...\s+(make|earn|...)`). This
|
||||||
|
claim is false for natural language. Three consequences follow:
|
||||||
|
|
||||||
|
1. **Refusal is brittle.** "How much will it cost him?" and "how much did he
|
||||||
|
pay in total?" and "how much money will she be left with after the
|
||||||
|
purchase?" all ask the solver for the same kind of output — the value of
|
||||||
|
one terminal-state quantity — but no template covers all three, and each
|
||||||
|
missing template is a refusal.
|
||||||
|
|
||||||
|
2. **The fix path is unbounded.** Each refused phrasing produces a new
|
||||||
|
recognizer. Each new recognizer adds vocabulary and structural assumptions.
|
||||||
|
The set has no closure: there is no point at which "all GSM8K question
|
||||||
|
shapes have been covered" because the set of question shapes is not finite.
|
||||||
|
|
||||||
|
3. **The model loses comprehension.** A template either matches or it
|
||||||
|
doesn't. It has no partial understanding. There is no state in which the
|
||||||
|
engine has "read three words and narrowed the interpretation" — the
|
||||||
|
pattern matches the whole sentence or refuses. That is the opposite of
|
||||||
|
how comprehension works.
|
||||||
|
|
||||||
|
ADR-0163's pathway (recognizer-via-contemplation) addresses *who writes the
|
||||||
|
regex* (the contemplation loop, not the operator). It does not address
|
||||||
|
*whether regex sentence-templates are the right representation at all.* They
|
||||||
|
are not.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision — incremental comprehension reader
|
||||||
|
|
||||||
|
Replace the regex sentence-template front-end with an **incremental
|
||||||
|
compositional reader** that processes one token at a time, maintains an
|
||||||
|
immutable partial-comprehension state, and produces the same downstream
|
||||||
|
types (`CandidateInitial`, `Operation`, `MathProblemGraph`,
|
||||||
|
`BoundUnknown`-input fields) the regex parser produces today.
|
||||||
|
|
||||||
|
The downstream substrate is unchanged. The binding graph, admissibility
|
||||||
|
check, solver, verifier, realizer, and round-trip filter all stay in place
|
||||||
|
and continue to enforce the `wrong = 0` invariant.
|
||||||
|
|
||||||
|
### Three components
|
||||||
|
|
||||||
|
**1. Operational lexicon (data, not code).**
|
||||||
|
|
||||||
|
Each word in the comprehension vocabulary maps to a *semantic category* and
|
||||||
|
an *update rule*. The category carries the generalization; adding a word is
|
||||||
|
adding a lookup, never a rule.
|
||||||
|
|
||||||
|
Example category set (closed, ADR-tracked, extended only by ratification):
|
||||||
|
|
||||||
|
| Category | Examples | Role in reader state |
|
||||||
|
|---|---|---|
|
||||||
|
| `question_open` | how, what | open question frame |
|
||||||
|
| `question_continuous_qty` | much, long, far, old | continuous-quantity question |
|
||||||
|
| `question_discrete_qty` | many | discrete-count question |
|
||||||
|
| `question_comparative` | more, less, longer, fewer | mark question as `difference` |
|
||||||
|
| `residual_modifier` | left, remaining, after | terminal-state residual |
|
||||||
|
| `aggregate_modifier` | total, in all, altogether, combined | sum across entities |
|
||||||
|
| `accumulation_verb` | earn, make, gain, accumulate, save | additive op |
|
||||||
|
| `depletion_verb` | spend, pay, lose, give | subtractive op |
|
||||||
|
| `transfer_verb` | give, send, pass | transfer op |
|
||||||
|
| `distributive_modifier` | each, per | bind rate or multiply |
|
||||||
|
| `currency_unit_noun` | money, dollars, profit, income, savings, cost | unit class: currency |
|
||||||
|
| `count_unit_noun` | apples, books, kids, chickens, … | unit class: countable |
|
||||||
|
| `time_unit_noun` | hour, day, week, minute, year | unit class: time |
|
||||||
|
| `entity_pronoun` | she, he, they, it | binds resolved entity |
|
||||||
|
| `proper_noun_entity` | Tina, Marion, Jen, … | binds entity directly |
|
||||||
|
|
||||||
|
The lexicon lives under `language_packs/data/en_core_math_v1/` parallel to
|
||||||
|
`en_core_cognition_v1` and `en_core_relations_v1`, with the same loader
|
||||||
|
discipline, the same manifest-checksum rule (CLAUDE.md §Semantic Pack
|
||||||
|
Discipline), and the same review pathway (ADR-0150/0152/0155/0161). New
|
||||||
|
lexicon entries enter through reviewed teaching, never via operator edits.
|
||||||
|
|
||||||
|
The vocabulary already collected in `math_candidate_parser.py` —
|
||||||
|
`_MASS_NOUNS`, `_PATTERN_A_VERBS`, `_PATTERN_B_VERBS`, `_PATTERN_C_VERBS`,
|
||||||
|
`_CAPACITY_VERB_PATTERN`, `_EARNINGS_VERB_PATTERN`, `ADD_VERBS`,
|
||||||
|
`SUBTRACT_VERBS`, `TRANSFER_VERBS`, `_FEMALE_NAMES`, `_MALE_NAMES` — is
|
||||||
|
**ported wholesale** as the seed corpus of the new lexicon. That ratified
|
||||||
|
vocabulary is good work; only its container (regex character classes inside
|
||||||
|
sentence templates) is wrong.
|
||||||
|
|
||||||
|
**2. Partial-comprehension state (immutable).**
|
||||||
|
|
||||||
|
```text
|
||||||
|
ComprehensionState:
|
||||||
|
entities: tuple[EntityRef, ...] # who's been mentioned
|
||||||
|
quantities: tuple[QuantityRef, ...] # numbers with units, attached or floating
|
||||||
|
operations: tuple[PartialOp, ...] # verb-induced operations, possibly incomplete
|
||||||
|
question_target: QuestionTargetSlot | None # what's being asked, possibly partial
|
||||||
|
expectation: ExpectationFrame | None # what category would close the current frame
|
||||||
|
```
|
||||||
|
|
||||||
|
`expectation` is the load-bearing field for recontextualization. After
|
||||||
|
reading "How much money will she", the state's expectation is "an
|
||||||
|
accumulation verb, a depletion verb, a residual modifier, or a
|
||||||
|
state-continuation verb." Each closes the question frame differently. The
|
||||||
|
expectation is what tells the reader how to interpret an ambiguous next
|
||||||
|
word.
|
||||||
|
|
||||||
|
State is frozen-dataclass immutable. Canonical-bytes serialization
|
||||||
|
(sorted-key, fixed-precision) keeps `trace_hash` deterministic per CLAUDE.md
|
||||||
|
§Runtime Surface Contract.
|
||||||
|
|
||||||
|
**3. Deterministic reader (state machine over categories).**
|
||||||
|
|
||||||
|
```text
|
||||||
|
apply_word(state, word) -> state | Refusal
|
||||||
|
```
|
||||||
|
|
||||||
|
For each token:
|
||||||
|
1. **Lexical primitive scan** (ADR-0165): try to match orthographic
|
||||||
|
primitives — currency literal, fraction literal, numeric literal,
|
||||||
|
percentage literal, time-unit noun — in priority order. If one fires,
|
||||||
|
the token becomes a typed lexeme with extracted value(s) and category.
|
||||||
|
2. **Lexicon lookup**: if no primitive fired, look up the surface form in
|
||||||
|
the operational lexicon. If absent, refuse with
|
||||||
|
`unknown_word: <token> (position N)`.
|
||||||
|
3. **Expectation check**: if the token's category satisfies
|
||||||
|
`state.expectation`, apply the update rule. If not — and the category
|
||||||
|
is a legal frame opener at this position — close the current frame and
|
||||||
|
open a new one. If neither — refuse with
|
||||||
|
`unexpected_category: got <cat>, expected <frame>`.
|
||||||
|
4. Emit new state.
|
||||||
|
|
||||||
|
End-of-sentence: the state must satisfy a finalization predicate
|
||||||
|
(question_target is bound, operations have their operands, dangling
|
||||||
|
quantities have unit attachments). Otherwise refuse with `unfinished_frame`.
|
||||||
|
|
||||||
|
The reader is a deterministic shift-reduce parser **over semantic
|
||||||
|
categories**, not over tokens. The category set is ~20 items; the
|
||||||
|
composition rules total 30–50. Adding a verb does not change a rule. Adding
|
||||||
|
a category requires an ADR.
|
||||||
|
|
||||||
|
### Output
|
||||||
|
|
||||||
|
The reader emits one of:
|
||||||
|
- A `MathProblemGraph` (and the underlying `CandidateInitial` /
|
||||||
|
`Operation` tuple) ready for the existing candidate-graph admissibility
|
||||||
|
layer, or
|
||||||
|
- A typed `ReaderRefusal` carrying the token position, the failed
|
||||||
|
expectation, and the closest legal next category. Refusals are the
|
||||||
|
evidence the teaching loop chews on (Phase E below).
|
||||||
|
|
||||||
|
Downstream consumption is unchanged. The binding-graph adapter (ADR-0133),
|
||||||
|
the `BoundUnknown` resolver (ADR-0135), the admissibility check (ADR-0134),
|
||||||
|
the solver (ADR-0116), and the verifier (ADR-0117) all act on the reader's
|
||||||
|
output exactly as they act on the regex parser's output today. The
|
||||||
|
`wrong = 0` invariant is preserved by construction because the reader does
|
||||||
|
not *bypass* admissibility — it produces inputs to it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Constraints (non-negotiable)
|
||||||
|
|
||||||
|
1. **`wrong = 0` at every phase, every round, every split.** The reader can
|
||||||
|
be more permissive about *which sentences it comprehends* without
|
||||||
|
weakening *what comprehension produces*. The existing admissibility,
|
||||||
|
unit-proof, and multi-branch-disagreement refusal stay in force.
|
||||||
|
|
||||||
|
2. **No hidden normalization, stochastic fallback, or "best guess."** The
|
||||||
|
reader refuses cleanly on novel structure. No softmax over candidate
|
||||||
|
parses, no nearest-template selection, no default category.
|
||||||
|
|
||||||
|
3. **No regex sentence-templates.** Per [ADR-0165](./ADR-0165-regex-scope-rule.md),
|
||||||
|
regex is allowed only at the lexeme level (currency literal, fraction
|
||||||
|
literal, etc.). Any regex that matches across word combination is a
|
||||||
|
grammar template and forbidden.
|
||||||
|
|
||||||
|
4. **Lexicon and category set are closed and ADR-tracked.** New lexicon
|
||||||
|
entries land through reviewed teaching (the existing ADR-0150/0152/0155/
|
||||||
|
0161 corridor — preserved from ADR-0163). New categories or new
|
||||||
|
composition rules require an ADR.
|
||||||
|
|
||||||
|
5. **Deterministic replay.** Identical input → byte-equal reader output. The
|
||||||
|
`ComprehensionState` has canonical-bytes serialization. The reader emits
|
||||||
|
a deterministic trace that feeds `trace_hash`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's deprecated, what's preserved
|
||||||
|
|
||||||
|
### Deprecated by this ADR
|
||||||
|
|
||||||
|
- **ADR-0163 §Phase B–E prescription**: the production of regex-based
|
||||||
|
`DerivedRecognizer` records that land in
|
||||||
|
`generate/recognizer_match.py`. New recognizers in this form are blocked
|
||||||
|
starting with the reader's first acceptance round. Existing recognizers
|
||||||
|
remain dormant during the transition (see Coexistence below) and are
|
||||||
|
removed once their categories are covered by the reader.
|
||||||
|
- **ADR-0136 — Statement Layer Corridor** and the sub-family
|
||||||
|
[ADR-0136.S.1–S.4](./): regex sentence-template additions to
|
||||||
|
`math_candidate_parser.py`. The empirical refusal taxonomies they
|
||||||
|
produced are preserved as input evidence for lexicon and category work.
|
||||||
|
The patterns themselves are scheduled for removal once the reader
|
||||||
|
covers their cases.
|
||||||
|
- The `Pattern A` / `Pattern B` / `Pattern C` regex blocks introduced by
|
||||||
|
ADR-0163.D.4 in `generate/math_candidate_parser.py` (`_Q_MASS_NOUN_RE`,
|
||||||
|
`_Q_COMPARATIVE_RE`, `_Q_PRONOUN_VERB_RE`) — replaced by the reader's
|
||||||
|
question-frame composition rules.
|
||||||
|
|
||||||
|
### Preserved in full
|
||||||
|
|
||||||
|
- **The binding graph** (ADR-0132/0133/0134/0135). The reader produces the
|
||||||
|
same input types it consumes today.
|
||||||
|
- **The HITL corridor** (ADR-0150/0152/0155/0161). New lexicon entries and
|
||||||
|
new categories ride the same contemplation → proposal → review pathway.
|
||||||
|
ADR-0163's corridor architecture is correct; only what flows through it
|
||||||
|
changes (lexicon entries instead of regex recognizers).
|
||||||
|
- **The capability-axis lanes** (G1–G5, S1). They continue to validate the
|
||||||
|
downstream substrate and act as the regression net for any reader
|
||||||
|
change.
|
||||||
|
- **`wrong = 0` doctrine** and the replay-equivalence gate.
|
||||||
|
- **All closed-set vocabulary** previously collected by the regex parser.
|
||||||
|
It is the seed of the operational lexicon.
|
||||||
|
|
||||||
|
### Untouched but adjacent
|
||||||
|
|
||||||
|
- The `recognizer_registry` / `recognizer_match` modules become the lexicon
|
||||||
|
loader and lexical-primitive registry rather than the regex pattern
|
||||||
|
store. The interface signature changes but the corridor-driven
|
||||||
|
*population* of these registries is preserved.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phasing
|
||||||
|
|
||||||
|
### Phase 1 — Question reader (where 34/47 refusals live)
|
||||||
|
|
||||||
|
Build the reader for question sentences only. The output type is narrow:
|
||||||
|
just the fields `BoundUnknown` consumes (`entity`, `unit`, question_form).
|
||||||
|
Coexist with the existing regex question patterns: reader runs first; on
|
||||||
|
refusal, falls through to existing regex; on reader acceptance, regex is
|
||||||
|
not invoked. Measure pickup rate against `train_sample/v1` per round.
|
||||||
|
|
||||||
|
Acceptance for Phase 1:
|
||||||
|
- Reader covers ≥20/34 currently-refused question cases.
|
||||||
|
- Combined (reader + legacy) `correct ≥ 10` on the 50-case sample with
|
||||||
|
`wrong = 0`. This satisfies the Round-1 exit criterion of ADR-0163.
|
||||||
|
- Reader has zero disagreement with regex on the 6 cases where both fire
|
||||||
|
(3 correct + 3 secondary), per byte-equal `BoundUnknown` output.
|
||||||
|
|
||||||
|
### Phase 2 — Statement reader
|
||||||
|
|
||||||
|
Extend the reader to statement sentences. Coexist with existing regex
|
||||||
|
statement patterns the same way. Phase out the regex statement patterns
|
||||||
|
incrementally as reader coverage grows.
|
||||||
|
|
||||||
|
Acceptance for Phase 2:
|
||||||
|
- Reader covers ≥30/50 train_sample cases end-to-end (statements +
|
||||||
|
question both via reader).
|
||||||
|
- `correct ≥ 25` (ADR-0163 Round-2 exit) with `wrong = 0`.
|
||||||
|
|
||||||
|
### Phase 3 — Regex layer removal
|
||||||
|
|
||||||
|
Once reader coverage ≥ regex coverage on a case-by-case basis, the regex
|
||||||
|
sentence-template layer is deleted. The lexical-primitive layer (regex
|
||||||
|
applied to single orthographic shapes per ADR-0165) survives — that is the
|
||||||
|
correct use of regex and is not what this ADR deprecates.
|
||||||
|
|
||||||
|
Acceptance for Phase 3:
|
||||||
|
- `correct ≥ 35` on train_sample, `wrong = 0` (ADR-0163 Round-3 exit).
|
||||||
|
- `math_candidate_parser.py` no longer contains sentence-level regex
|
||||||
|
patterns. Closed-set vocabulary tables remain (now consumed by the
|
||||||
|
lexicon loader rather than woven into regexes).
|
||||||
|
|
||||||
|
### Phase 4 — Scale
|
||||||
|
|
||||||
|
Per ADR-0163 §Phase F: public, holdout, full GSM8K. No changes to that
|
||||||
|
scope from this ADR; the reader simply replaces the front-end.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acceptance criteria for this ADR (Proposed → Accepted)
|
||||||
|
|
||||||
|
This ADR moves to **Accepted** when:
|
||||||
|
|
||||||
|
1. A `ComprehensionState` prototype exists in `generate/comprehension/`
|
||||||
|
with frozen-dataclass shape, canonical-bytes serialization, and unit
|
||||||
|
tests pinning determinism.
|
||||||
|
2. The seed lexicon pack `en_core_math_v1` is materialized from the
|
||||||
|
existing closed-set vocabulary in `math_candidate_parser.py`, with the
|
||||||
|
standard pack-test discipline.
|
||||||
|
3. Phase 1 acceptance is met on `train_sample/v1`.
|
||||||
|
4. Capability-axis lanes G1–G5, S1 remain at 100% `wrong = 0` (regression
|
||||||
|
net unbroken).
|
||||||
|
5. `verify pinned lane SHAs` continues to pass.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions (resolve before Phase 1 PR)
|
||||||
|
|
||||||
|
1. **Lexical primitive set scope.** Inventory of which orthographic shapes
|
||||||
|
get primitives vs. lexicon entries (currency literal, fraction literal,
|
||||||
|
percentage literal, decimal literal, time-unit noun, dollar-amount,
|
||||||
|
ordinal). Likely a sub-ADR (ADR-0164.1).
|
||||||
|
2. **Ambiguity resolution precedence.** When a token could open two
|
||||||
|
frames, the precedence order. Likely a sub-ADR after Phase 1
|
||||||
|
measurement reveals which collisions are real.
|
||||||
|
3. **Pronoun-entity resolution.** The reader needs entity resolution
|
||||||
|
anyway; the regex parser's `_resolve_question_entity` heuristic is a
|
||||||
|
reasonable starting point but should be reviewed against the
|
||||||
|
compositional model.
|
||||||
|
4. **Cross-sentence state.** The current regex parser is per-sentence;
|
||||||
|
GSM8K problems have cross-sentence references ("she" referring to
|
||||||
|
"Tina" three sentences earlier). The reader will need a
|
||||||
|
`ProblemReadingState` that persists across sentences. Scope this in
|
||||||
|
Phase 1 design.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cross-references
|
||||||
|
|
||||||
|
- **Bottleneck evidence**: `evals/gsm8k_math/train_sample/v1/report.json`,
|
||||||
|
`refusal_taxonomy_v4.json`.
|
||||||
|
- **Substrate that survives**: `generate/binding_graph/`,
|
||||||
|
`generate/math_solver.py`, `generate/math_verifier.py`,
|
||||||
|
`generate/math_realizer.py`, capability-axis lanes.
|
||||||
|
- **The corridor**: ADR-0150 (contemplation), ADR-0152 (learning-arc),
|
||||||
|
ADR-0155 (CI contemplation runner), ADR-0161 (HITL queue).
|
||||||
|
- **The boundary rule**: ADR-0165.
|
||||||
|
- **The anti-overfitting doctrine**: ADR-0114a.
|
||||||
|
- **The thesis**: `[[thesis-decoding-not-generating]]` — the reader is a
|
||||||
|
decoder. Each word narrows the space; the meaning is the accumulation,
|
||||||
|
not the match.
|
||||||
270
docs/decisions/ADR-0165-regex-scope-rule.md
Normal file
270
docs/decisions/ADR-0165-regex-scope-rule.md
Normal file
|
|
@ -0,0 +1,270 @@
|
||||||
|
# ADR-0165 — Regex Scope Rule: Lexemes Only, Never Grammar
|
||||||
|
|
||||||
|
**Status:** Proposed
|
||||||
|
**Date:** 2026-05-26
|
||||||
|
**Author:** Shay
|
||||||
|
**Anchor:** [[thesis-decoding-not-generating]]
|
||||||
|
**Companions:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0114a — Anti-overfitting proof obligations](./ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0150 — Autonomous inter-session contemplation](./ADR-0150-autonomous-inter-session-contemplation.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context — where regex creeps in, and why the line matters
|
||||||
|
|
||||||
|
CORE's existing front-end parsers (`generate/math_candidate_parser.py`,
|
||||||
|
`generate/recognizer_match.py`) use regex for two structurally different
|
||||||
|
jobs that have been collapsed into one:
|
||||||
|
|
||||||
|
1. **Recognizing lexemes** — "this token chunk is a currency literal," "this
|
||||||
|
is a fraction `\d+/\d+`," "this is a numeric expression." These have
|
||||||
|
genuinely closed orthographic rules. Regex is the right primitive.
|
||||||
|
|
||||||
|
2. **Matching sentence structure** — "this whole sentence is the shape
|
||||||
|
`How much MASS_NOUN does ENTITY VERB ...`." These do not have a closed
|
||||||
|
rule, because natural language doesn't have one. Regex here is
|
||||||
|
enumeration of memorized shapes, dressed up as grammar.
|
||||||
|
|
||||||
|
Mixing the two has been the engine's most consistent source of overfitting.
|
||||||
|
[ADR-0164](./ADR-0164-incremental-comprehension-reader.md) replaces the
|
||||||
|
sentence-structure use with an incremental compositional reader. This ADR
|
||||||
|
locks the *boundary*: a structural invariant that any future code must
|
||||||
|
respect, independent of which front-end implementation is current.
|
||||||
|
|
||||||
|
The rule is in the same spirit as the existing structural invariants —
|
||||||
|
`versor_condition(F) < 1e-6`, "no normalization outside the gate", "no
|
||||||
|
approximate vault recall", "no hidden stochastic fallback." It is a typed
|
||||||
|
boundary, not a guideline.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rule
|
||||||
|
|
||||||
|
> **Regex is permitted only at the lexeme level.** Regex must operate on the
|
||||||
|
> orthographic shape of one structured token or contiguous token-class run
|
||||||
|
> whose meaning has a genuinely closed rule. Regex must not match across
|
||||||
|
> word combination, syntactic role, or sentence structure.
|
||||||
|
|
||||||
|
**Test for any regex literal in the runtime path:**
|
||||||
|
|
||||||
|
If the regex describes "what this piece of orthographic material looks
|
||||||
|
like," it is a *lexeme primitive* and is permitted.
|
||||||
|
|
||||||
|
If the regex describes "how these words combine to mean X," it is a
|
||||||
|
*grammar template* and is forbidden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Legitimate uses (lexeme primitives)
|
||||||
|
|
||||||
|
These are the canonical examples. The list is extensible through the
|
||||||
|
teaching corridor (§Population, below).
|
||||||
|
|
||||||
|
| Primitive | Example regex | Extracts | Category emitted |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `currency_literal` | `\$(\d+(?:\.\d+)?)` | numeric value | QUANTITY, unit_class=currency |
|
||||||
|
| `numeric_literal` | `\d+(?:\.\d+)?` | numeric value | QUANTITY (unit pending) |
|
||||||
|
| `fraction_literal` | `(\d+)\s*/\s*(\d+)` | numerator, denominator | QUANTITY, kind=fraction |
|
||||||
|
| `percentage_literal` | `(\d+(?:\.\d+)?)\s*%` | numeric value | QUANTITY, unit_class=ratio |
|
||||||
|
| `time_amount_literal` | `(\d+)[- ]?(hour\|minute\|day\|week\|month\|year)s?` | value, unit | QUANTITY, unit_class=time |
|
||||||
|
| `mass_noun_token` | `(?:money\|profit\|interest\|...)` | the lexeme | UNIT_CATEGORY_TOKEN |
|
||||||
|
| `decimal_currency` | `\$\d+\.\d{2}` | value | QUANTITY, unit_class=currency |
|
||||||
|
| `ordinal_token` | `(?:first\|second\|third\|...)` | rank | ORDINAL |
|
||||||
|
|
||||||
|
Each primitive has:
|
||||||
|
|
||||||
|
- a **name** (closed registry key),
|
||||||
|
- a **pattern** (a single, focused regex over orthographic shape),
|
||||||
|
- an **emission** (the typed category + extracted values it produces),
|
||||||
|
- a **provenance** (which ADR or teaching ratification introduced it).
|
||||||
|
|
||||||
|
Primitives never reach across roles. `currency_literal` recognizes
|
||||||
|
`$18.00`; it does not recognize `$18.00 an hour` (that composition is the
|
||||||
|
reader's job, not the primitive's).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Forbidden uses (grammar templates)
|
||||||
|
|
||||||
|
These are the patterns ADR-0164 deprecates and this rule forbids
|
||||||
|
recurring. Each example below is a *grammar template* and would be rejected
|
||||||
|
in code review under this ADR.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# FORBIDDEN — regex matching question shape:
|
||||||
|
_Q_MASS_NOUN_RE = re.compile(
|
||||||
|
r"^How\s+much\s+"
|
||||||
|
rf"(?P<unit>{_MASS_NOUN_PATTERN})"
|
||||||
|
r"\s+(?:will|did|does|do|would)\s+"
|
||||||
|
rf"(?P<entity>{_Q_ENTITY_OR_PRONOUN})\s+"
|
||||||
|
rf"(?:have\s+earned\s+|be\s+able\s+to\s+)?{_PATTERN_A_VERBS}"
|
||||||
|
r"(?:\s+.*?)?\??\s*$"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a sentence-structure template. It matches across question stem
|
||||||
|
("How much"), unit ("money"), auxiliary ("will"), entity ("Tina"), and
|
||||||
|
verb ("earn"). It claims the conjunction of these elements forms a closed
|
||||||
|
shape. It doesn't.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# FORBIDDEN — regex matching statement structure:
|
||||||
|
_INITIAL_HAS_RE = re.compile(
|
||||||
|
rf"^(?P<entity>{_ENTITY})\s+has\s+(?P<value>\d+)\s+(?P<unit>\w+)\.?\s*$"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Same problem: matches across subject, verb, value, unit, and clause
|
||||||
|
shape. Replace with reader composition rules over the categories
|
||||||
|
`proper_noun_entity`, `possession_verb`, `numeric_literal`, `count_unit_noun`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# PERMITTED — regex matching one orthographic shape:
|
||||||
|
_CURRENCY_LITERAL_RE = re.compile(r"\$(\d+(?:\.\d+)?)")
|
||||||
|
```
|
||||||
|
|
||||||
|
This recognizes the surface form of one token (with the `$` prefix). It
|
||||||
|
has a closed rule (currency notation). It is the correct use of regex.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Code-review test (apply to every new regex)
|
||||||
|
|
||||||
|
When reviewing or writing a regex, answer three questions:
|
||||||
|
|
||||||
|
1. **What does the regex match?** If the answer names a *piece of
|
||||||
|
orthographic material* (a number, a currency amount, a unit-noun set,
|
||||||
|
a date), it's a lexeme primitive. If it names *a way words combine*
|
||||||
|
(a question shape, an assertion shape, a clause pattern), it's a
|
||||||
|
grammar template.
|
||||||
|
|
||||||
|
2. **Could a competent linguist describe the matched class as a closed
|
||||||
|
set of orthographic rules?** If yes, the regex is appropriate. If the
|
||||||
|
class is "things people sometimes say to mean X," the regex is
|
||||||
|
enumerating memorized shapes and is forbidden.
|
||||||
|
|
||||||
|
3. **What happens when a novel phrasing of the same underlying meaning
|
||||||
|
appears?** A lexeme primitive's domain doesn't depend on the rest of
|
||||||
|
the sentence, so novel phrasings around it don't break it. A grammar
|
||||||
|
template refuses on every novel phrasing. If a refusal on novel
|
||||||
|
phrasing is the expected behavior of the regex, it's a grammar
|
||||||
|
template.
|
||||||
|
|
||||||
|
A regex that fails any of these three is a grammar template and must be
|
||||||
|
restructured (typically: extract the closed-set vocabulary as a primitive,
|
||||||
|
move the structural part into reader composition rules).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Population — how the primitive set grows
|
||||||
|
|
||||||
|
The closed registry of lexeme primitives is not static. It grows through
|
||||||
|
the same contemplation → proposal → review corridor that already grows the
|
||||||
|
language packs and (under ADR-0164) the operational lexicon. This means:
|
||||||
|
|
||||||
|
1. **The reader refuses on an unrecognized token shape** — for example,
|
||||||
|
it encounters `"$1.5M"` and no current primitive matches it.
|
||||||
|
2. The refusal is logged with its token shape and position
|
||||||
|
(`evals/discovery/discovery_candidates.jsonl` analog).
|
||||||
|
3. The contemplation runner (ADR-0150/0155) identifies the shape as a
|
||||||
|
candidate new primitive and emits a proposal carrying:
|
||||||
|
- the proposed primitive's pattern,
|
||||||
|
- its emission category and extracted fields,
|
||||||
|
- replay-equivalence evidence (acceptance does not lift `wrong` above
|
||||||
|
0 on the active corpus),
|
||||||
|
- the originating refused tokens.
|
||||||
|
4. The proposal lands in the HITL queue (ADR-0161). The operator reviews
|
||||||
|
the pattern and emission rule. On acceptance, the primitive enters the
|
||||||
|
closed registry with its provenance.
|
||||||
|
5. The reader picks up the new primitive on next run. No code edit. No
|
||||||
|
parser rewrite. The engine has been *taught* a new lexical recognizer.
|
||||||
|
|
||||||
|
This matches the user's original framing: regex is a **mental tool** the
|
||||||
|
engine wields. The toolset is bounded and reviewed, but not hard-wired.
|
||||||
|
Adding a tool follows the same ratification discipline as adding a lemma
|
||||||
|
or a category.
|
||||||
|
|
||||||
|
Two consequences of this design:
|
||||||
|
|
||||||
|
- **Operators do not write production regex.** They review proposed
|
||||||
|
regex against typed evidence. This eliminates the failure mode where
|
||||||
|
a regex sneaks in to "just unblock GSM8K case 0027."
|
||||||
|
- **The regex set has a measurable closure curve.** Each ratification
|
||||||
|
round either does or doesn't reduce the refused-token count.
|
||||||
|
Diminishing returns become visible — the regex set is on the same
|
||||||
|
measurement substrate as the lexicon and the recognizer corpus.
|
||||||
|
|
||||||
|
The bootstrap primitive set lands as part of the ADR-0164 Phase 1 PR. It
|
||||||
|
covers the closed orthographic forms already known to be needed from the
|
||||||
|
existing parser (currency, numeric, fraction, percentage, time-amount,
|
||||||
|
ordinal). Everything beyond bootstrap enters via the corridor.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Positive
|
||||||
|
|
||||||
|
1. **Overfit-by-design becomes structurally impossible at the regex
|
||||||
|
layer.** A grammar-template regex cannot land in code review without
|
||||||
|
violating this ADR explicitly.
|
||||||
|
2. **Closed-set vocabulary already collected in the existing parser is
|
||||||
|
preserved.** Mass-noun lists, possession-verb lists, name lists — all
|
||||||
|
become lexeme primitives or lexicon entries.
|
||||||
|
3. **One regex toolkit, one lexicon, one composition layer.** Three
|
||||||
|
layered concerns with separate population pathways and separate
|
||||||
|
review criteria. Maintenance scales linearly, not multiplicatively.
|
||||||
|
4. **The teaching corridor's purpose generalizes.** Today the corridor
|
||||||
|
teaches words, domains, and (per the deprecated path) regex
|
||||||
|
recognizers. Under this rule it teaches words, categories, and
|
||||||
|
lexeme primitives — three orthogonal kinds of evidence, each with a
|
||||||
|
clean review predicate.
|
||||||
|
|
||||||
|
### Negative / tradeoffs
|
||||||
|
|
||||||
|
1. **Refactoring cost is real.** Removing existing sentence-template
|
||||||
|
regexes from `math_candidate_parser.py` and `recognizer_match.py` is
|
||||||
|
a substantial edit, even with vocabulary preserved. The transition
|
||||||
|
plan in ADR-0164 (coexistence → incremental removal) absorbs this.
|
||||||
|
2. **Lexeme primitives are an attractive surface for new overfitting.**
|
||||||
|
A primitive author could try to smuggle structure ("a number followed
|
||||||
|
by a currency word followed by 'an hour'") into a single regex. The
|
||||||
|
review criteria above are explicit about this; the corridor enforces
|
||||||
|
it.
|
||||||
|
3. **The reader has to do more work.** Composition that used to live
|
||||||
|
inside a regex now lives in update rules. This is the point —
|
||||||
|
composition is the engine's job, not the regex's — but it shifts
|
||||||
|
complexity from one place to another.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Boundaries — what this ADR does **not** say
|
||||||
|
|
||||||
|
1. **It does not forbid regex.** Regex remains a primary tool for lexeme
|
||||||
|
recognition. The current `evals/`, `scripts/`, and CLI parsers
|
||||||
|
already use regex appropriately for log parsing, file-path matching,
|
||||||
|
etc. None of that is affected.
|
||||||
|
2. **It does not specify the reader.** The reader's design is
|
||||||
|
ADR-0164's scope. This ADR only constrains where regex may live
|
||||||
|
in whatever front-end is current.
|
||||||
|
3. **It does not retroactively reject older code wholesale.** ADRs
|
||||||
|
ADR-0136.S.1–S.4 and ADR-0163.D.2–D.4 introduced grammar templates
|
||||||
|
under previous policy. They are deprecated by ADR-0164 with an
|
||||||
|
explicit transition plan. New work follows this rule from acceptance
|
||||||
|
of this ADR forward.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cross-references
|
||||||
|
|
||||||
|
- **Sibling ADR**: ADR-0164 — the comprehension reader that occupies the
|
||||||
|
space this rule clears.
|
||||||
|
- **Existing structural invariants** (same spirit, different domain):
|
||||||
|
- `versor_condition(F) < 1e-6` (CLAUDE.md §Field Invariant)
|
||||||
|
- "Allowed normalization sites" (CLAUDE.md §Normalization Rules)
|
||||||
|
- "Exact CGA recall" (CLAUDE.md §Core Primitives)
|
||||||
|
- ADR-0114a Anti-overfitting proof obligations
|
||||||
|
- **Population corridor**: ADR-0150 (contemplation), ADR-0152
|
||||||
|
(learning-arc proof), ADR-0155 (CI contemplation runner), ADR-0161
|
||||||
|
(HITL async queue).
|
||||||
|
- **Anchor**: `[[thesis-decoding-not-generating]]` — regex is a
|
||||||
|
decoder's tool for recognizing fixed orthographic shapes. It is not a
|
||||||
|
generator's tool for hallucinating sentence grammars.
|
||||||
|
|
@ -205,6 +205,45 @@ The contract has been demonstrated end-to-end: refused once honestly (ADR-0107),
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### Comprehension-reader pivot — ADR-0164 and ADR-0165 (2026-05-26)
|
||||||
|
|
||||||
|
The GSM8K admissibility front-end is replaced. The regex sentence-template
|
||||||
|
parsing layer in `generate/math_candidate_parser.py` and
|
||||||
|
`generate/recognizer_match.py` is recognized as overfitting by
|
||||||
|
construction — it enumerates memorized surface shapes while pretending to
|
||||||
|
encode a closed grammar that English does not have.
|
||||||
|
|
||||||
|
- **[ADR-0164](ADR-0164-incremental-comprehension-reader.md)** —
|
||||||
|
Incremental Comprehension Reader. Word-by-word state accumulation over a
|
||||||
|
closed set of semantic categories. The reader is a deterministic
|
||||||
|
shift-reduce parser over *categories*, not over tokens. Output type is
|
||||||
|
identical to the regex parser's output, so the binding-graph
|
||||||
|
admissibility (ADR-0132/0133/0134/0135) downstream is unchanged.
|
||||||
|
Operational lexicon lives in `language_packs/data/en_core_math_v1/`
|
||||||
|
alongside the existing packs.
|
||||||
|
|
||||||
|
- **[ADR-0165](ADR-0165-regex-scope-rule.md)** — Regex Scope Rule.
|
||||||
|
Structural invariant: regex is permitted only at the lexeme level
|
||||||
|
(currency literal, fraction literal, percentage literal, time-amount,
|
||||||
|
closed-set unit-noun), never at the sentence-structure level. The
|
||||||
|
primitive set is a closed registry grown through the same contemplation
|
||||||
|
→ proposal → HITL review corridor that grows vocabulary.
|
||||||
|
|
||||||
|
These ADRs preserve every load-bearing piece of the prior work: the
|
||||||
|
binding graph (ADR-0132–0135), the solver / verifier / realizer
|
||||||
|
substrate (ADR-0116–0118), the capability-axis lanes G1–G5 and S1, the
|
||||||
|
HITL corridor (ADR-0150 / 0152 / 0155 / 0161), the `wrong = 0` doctrine
|
||||||
|
and the replay-equivalence gate (ADR-0057, ADR-0114a). The
|
||||||
|
contemplation → proposal → review corridor architecture is reaffirmed
|
||||||
|
and its scope is generalized: it now ratifies lexicon entries,
|
||||||
|
categories, and lexeme primitives — not regex recognizers.
|
||||||
|
|
||||||
|
ADR-0163's *diagnosis* (the front-end is the bottleneck) is reaffirmed;
|
||||||
|
its *prescription* (Phases B–E recognizer production) is partially
|
||||||
|
superseded. ADR-0136 and its S-family have the same disposition.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Session Logs
|
## Session Logs
|
||||||
|
|
||||||
Session logs record the decisions and rationale from individual working sessions. They are not ADRs — they are the narrative record that informed the ADRs.
|
Session logs record the decisions and rationale from individual working sessions. They are not ADRs — they are the narrative record that informed the ADRs.
|
||||||
|
|
@ -215,3 +254,4 @@ Session logs record the decisions and rationale from individual working sessions
|
||||||
| 2026-05-12 (addendum) | [SESSION-2026-05-12-b.md](SESSION-2026-05-12-b.md) |
|
| 2026-05-12 (addendum) | [SESSION-2026-05-12-b.md](SESSION-2026-05-12-b.md) |
|
||||||
| 2026-05-12 (language packs) | [SESSION-2026-05-12-language-packs-addendum.md](SESSION-2026-05-12-language-packs-addendum.md) |
|
| 2026-05-12 (language packs) | [SESSION-2026-05-12-language-packs-addendum.md](SESSION-2026-05-12-language-packs-addendum.md) |
|
||||||
| 2026-05-13 | [SESSION-2026-05-13.md](SESSION-2026-05-13.md) |
|
| 2026-05-13 | [SESSION-2026-05-13.md](SESSION-2026-05-13.md) |
|
||||||
|
| 2026-05-26 (comprehension reader) | [SESSION-2026-05-26-comprehension-reader.md](SESSION-2026-05-26-comprehension-reader.md) |
|
||||||
|
|
|
||||||
232
docs/decisions/SESSION-2026-05-26-comprehension-reader.md
Normal file
232
docs/decisions/SESSION-2026-05-26-comprehension-reader.md
Normal file
|
|
@ -0,0 +1,232 @@
|
||||||
|
# SESSION 2026-05-26 — Comprehension Reader Decision
|
||||||
|
|
||||||
|
**Participants:** Shay, Claude (Sonnet 4.6 → Opus 4.7 for ADR drafting)
|
||||||
|
**Outputs:**
|
||||||
|
[ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md),
|
||||||
|
[ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md).
|
||||||
|
**Affected:** [ADR-0163](./ADR-0163-gsm8k-path-to-mastery.md) (prescription
|
||||||
|
partially superseded), [ADR-0136 + sub-family](./ADR-0136-statement-layer-corridor.md)
|
||||||
|
(regex prescription superseded; vocabulary preserved as lexicon seed).
|
||||||
|
**Anchor:** [[thesis-decoding-not-generating]]
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What triggered the session
|
||||||
|
|
||||||
|
PR cleanup turn that started as "merge the open PRs" became an architectural
|
||||||
|
session when the operator asked why the post-D.2 GSM8K train_sample baseline
|
||||||
|
remained at `correct=3 refused=47 wrong=0`.
|
||||||
|
|
||||||
|
Three open PRs were on the board at session start:
|
||||||
|
|
||||||
|
- **#316** — `fix(INV-02): allowlist test_issue_300_versor_margin.py` — all
|
||||||
|
checks green, mergeable. Merged first.
|
||||||
|
- **#315** — `feat(ADR-0163.D.2): parsed_anchors → MathProblemGraph state
|
||||||
|
— discrete_count_statement injection v1` — smoke failing because the
|
||||||
|
INV-02 allowlist fix wasn't in its base. Rebased onto new main after
|
||||||
|
#316, smoke turned green, merged.
|
||||||
|
- **#314** — `docs(plan): CORE general advancement path` — rebased onto
|
||||||
|
new main, all checks green, merged.
|
||||||
|
|
||||||
|
Board cleared. Then the substantive question.
|
||||||
|
|
||||||
|
## The diagnostic dive
|
||||||
|
|
||||||
|
The operator asked: "Why aren't we getting more of these answers right?"
|
||||||
|
|
||||||
|
Running the train_sample runner directly produced
|
||||||
|
`correct=3 refused=47 wrong=0` with `exit_criterion: correct_min=10, passed=false`.
|
||||||
|
Refusal-reason aggregation showed the bottleneck precisely:
|
||||||
|
|
||||||
|
- **34 / 47** refusals were `no admissible candidate for question: '<text>'`
|
||||||
|
— statements parsed successfully, but the question surface form did not
|
||||||
|
match any of the ~6 question regexes in
|
||||||
|
`generate/math_candidate_parser.py` (Pattern A / B / C, capacity,
|
||||||
|
earnings, conditional-op).
|
||||||
|
- **9 / 47** refusals were `no admissible candidate for statement: '<text>'`
|
||||||
|
— statement-side regex gaps (5 of them fraction operands).
|
||||||
|
- **4 / 47** refusals were `no branch produced a solvable graph` —
|
||||||
|
statements + question admitted but the solver couldn't close.
|
||||||
|
|
||||||
|
The 3 admitted cases shared a tight structural signature: rate × time
|
||||||
|
patterns plus one distributive multiply + subtract. The exact patterns the
|
||||||
|
regex front-end was originally written to handle.
|
||||||
|
|
||||||
|
The v4 refusal taxonomy
|
||||||
|
(`evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v4.json`) reinforced
|
||||||
|
the picture: 23 distinct primary-barrier categories across 47 cases, with
|
||||||
|
no single category larger than 5 cases. The long tail of distinct shapes
|
||||||
|
is the long tail of English question surface forms.
|
||||||
|
|
||||||
|
## The operator's diagnosis (load-bearing)
|
||||||
|
|
||||||
|
The operator said, plainly:
|
||||||
|
|
||||||
|
> "Obviously the whole regex stuff is overfitting by design… lol. I was
|
||||||
|
> literally wondering about that when it was being built… just thought
|
||||||
|
> you knew what you were doing."
|
||||||
|
|
||||||
|
And:
|
||||||
|
|
||||||
|
> "Regex wasn't meant to be there. And I said, if we are going to allow
|
||||||
|
> regex in, then we teach the model how to use regex itself as a 'mental
|
||||||
|
> tool' of sorts. but not use it to overfit templates to what we want.
|
||||||
|
> That's only ever going to end up being a bottleneck risk. Makes no
|
||||||
|
> sense. If there truly were a said, rule-based system for sentence
|
||||||
|
> structure then that would be different, and we could use all the
|
||||||
|
> 'known' templates."
|
||||||
|
|
||||||
|
This is the architectural pivot. Three points compress into it:
|
||||||
|
|
||||||
|
1. **Sentence-template regex is overfitting by definition.** A regex
|
||||||
|
sentence-template is an enumeration of memorized surface shapes
|
||||||
|
pretending to be a grammar rule. English does not have a closed
|
||||||
|
grammar for math-problem questions. Adding more templates does not
|
||||||
|
approach a limit; the refusal-rate ceiling is set by the *method*,
|
||||||
|
not by template count.
|
||||||
|
|
||||||
|
2. **Regex has a legitimate role at the lexeme level.** Currency
|
||||||
|
literals, fractions, percentages, numeric expressions, closed-set
|
||||||
|
unit-noun lists — these have genuinely closed orthographic rules.
|
||||||
|
Regex is the honest tool for recognizing them. The boundary is:
|
||||||
|
regex describes "what one piece of orthographic material looks like,"
|
||||||
|
never "how words combine to mean X."
|
||||||
|
|
||||||
|
3. **The model should be able to acquire regex tools through review,
|
||||||
|
not have them hard-coded.** The operator had already designed the
|
||||||
|
teaching/contemplation/HITL corridor (ADR-0150 / 0152 / 0155 / 0161)
|
||||||
|
for exactly this purpose. The corridor is general: it can ratify new
|
||||||
|
vocabulary, new categories, and new lexeme primitives through the
|
||||||
|
same review pathway. Regex tools become *data* the engine
|
||||||
|
accumulates through reviewed teaching, not code the operator writes.
|
||||||
|
|
||||||
|
The operator's framing of point 3 was the moment the corridor's purpose
|
||||||
|
generalized in scope: it teaches *recognition capability*, not just
|
||||||
|
*recognized content*. That is the structural difference between a fixed
|
||||||
|
toolkit and an intelligence that can grow its own tools.
|
||||||
|
|
||||||
|
## The architectural shape of the answer
|
||||||
|
|
||||||
|
The downstream substrate is correct and stays:
|
||||||
|
|
||||||
|
```
|
||||||
|
... → MathProblemGraph → BoundUnknown (ADR-0135) → Admissibility
|
||||||
|
(ADR-0132/0133) via question_target.py (ADR-0134)
|
||||||
|
→ Solver (ADR-0116)
|
||||||
|
→ Verifier (ADR-0117)
|
||||||
|
→ Realizer (ADR-0118)
|
||||||
|
```
|
||||||
|
|
||||||
|
The binding-graph layer already operates on typed structure rather than
|
||||||
|
surface words. It infers `question_form` (count / total / rate /
|
||||||
|
difference / ratio / identity) from the operations touching the unknown.
|
||||||
|
That's the correct level. It just doesn't get fed enough graphs because
|
||||||
|
the front-end refuses too often.
|
||||||
|
|
||||||
|
The front-end is replaced. The new shape:
|
||||||
|
|
||||||
|
```
|
||||||
|
Text
|
||||||
|
→ Lexical Primitives (regex, lexeme-scope only — ADR-0165)
|
||||||
|
→ Lexicon Lookup (word → semantic category, ADR-0164)
|
||||||
|
→ Incremental Reader (word-by-word state accumulation)
|
||||||
|
→ MathProblemGraph (same downstream type as before)
|
||||||
|
→ [unchanged downstream]
|
||||||
|
```
|
||||||
|
|
||||||
|
The reader is a deterministic shift-reduce parser **over semantic
|
||||||
|
categories**, not over surface tokens. The category set is closed
|
||||||
|
(~20 items), the composition rules are bounded (30–50). Adding a verb
|
||||||
|
adds a lexicon lookup, not a new code path. The vocabulary already
|
||||||
|
collected in `math_candidate_parser.py` (`_MASS_NOUNS`,
|
||||||
|
`_PATTERN_A_VERBS`, `_PATTERN_B_VERBS`, `_PATTERN_C_VERBS`, name lists,
|
||||||
|
add/subtract/transfer verb sets) ports wholesale as the lexicon seed —
|
||||||
|
the vocabulary work is salvaged; only the regex template wrappers are
|
||||||
|
removed.
|
||||||
|
|
||||||
|
## Why this preserves wrong = 0
|
||||||
|
|
||||||
|
The reader can be more permissive about *which sentences it
|
||||||
|
comprehends* without being more permissive about *what comprehension
|
||||||
|
produces*. The output type is identical to what the regex parser
|
||||||
|
produces today, so the existing admissibility gate (unit proofs,
|
||||||
|
multi-branch disagreement refusal, replay-equivalence) stays in force.
|
||||||
|
A malformed comprehension produces a graph that admissibility rejects.
|
||||||
|
wrong = 0 is preserved by construction.
|
||||||
|
|
||||||
|
## The corridor generalizes
|
||||||
|
|
||||||
|
The teaching → contemplation → review corridor (ADR-0150 / 0152 / 0155 /
|
||||||
|
0161) already exists for vocabulary. Under ADR-0164 it expands in scope
|
||||||
|
to also ratify:
|
||||||
|
|
||||||
|
- **Lexicon entries** (word → category mappings)
|
||||||
|
- **Composition rules** (rare — bounded set, ADR-tracked)
|
||||||
|
- **Lexeme primitives** (new regex tools the engine can wield)
|
||||||
|
|
||||||
|
Three orthogonal kinds of evidence, three orthogonal review predicates,
|
||||||
|
one shared corridor. The engine becomes able to acquire new recognition
|
||||||
|
capability through reviewed experience instead of through operator
|
||||||
|
edits to parser code.
|
||||||
|
|
||||||
|
The operator's reaction at the moment this clicked into place:
|
||||||
|
|
||||||
|
> "That's the absolute fundamental key to intelligence. Truly. That's
|
||||||
|
> what I had been hoping we could figure out."
|
||||||
|
|
||||||
|
## Deprecation discipline
|
||||||
|
|
||||||
|
ADR-0163's *diagnosis* (the front-end is the bottleneck) is reaffirmed.
|
||||||
|
ADR-0163's *prescription* (Phases B–E producing regex-based
|
||||||
|
`DerivedRecognizer` records in `generate/recognizer_match.py`) is
|
||||||
|
superseded — what flows through the corridor changes, the corridor
|
||||||
|
itself does not.
|
||||||
|
|
||||||
|
ADR-0136 and its S-family (S.1 / S.2 / S.3 / S.4 / post-rescan
|
||||||
|
variants): regex sentence-template prescription superseded. Empirical
|
||||||
|
refusal taxonomies preserved. Closed-set vocabulary tables preserved as
|
||||||
|
lexicon seed.
|
||||||
|
|
||||||
|
All ratified work survives in some form. The regex *wrappers* go;
|
||||||
|
everything else carries forward.
|
||||||
|
|
||||||
|
## Phasing committed in ADR-0164
|
||||||
|
|
||||||
|
1. **Phase 1 — Question reader.** Build the reader for question
|
||||||
|
sentences only. Coexist with existing regex; reader runs first, regex
|
||||||
|
is fallback. Target: `correct ≥ 10` on train_sample/v1, satisfying
|
||||||
|
ADR-0163 Round-1 exit. Reader covers ≥20/34 currently-refused
|
||||||
|
question cases.
|
||||||
|
2. **Phase 2 — Statement reader.** Extend to statements. Target:
|
||||||
|
`correct ≥ 25`.
|
||||||
|
3. **Phase 3 — Regex layer removal.** `math_candidate_parser.py` no
|
||||||
|
longer contains sentence-level regex patterns. Target: `correct ≥ 35`.
|
||||||
|
4. **Phase 4 — Scale.** Per ADR-0163 §Phase F: public, holdout, full
|
||||||
|
GSM8K.
|
||||||
|
|
||||||
|
## What the session did not decide
|
||||||
|
|
||||||
|
- Specific category set and composition-rule closure beyond a sketch.
|
||||||
|
These will be sub-ADRs once Phase 1 measurement reveals real
|
||||||
|
collisions.
|
||||||
|
- Cross-sentence reading state (pronoun resolution across the problem
|
||||||
|
body). Scoped in Phase 1 design.
|
||||||
|
- The lexicon pack's exact frontmatter and merging policy with
|
||||||
|
existing packs (`en_core_cognition_v1`, `en_core_relations_v1`).
|
||||||
|
- Whether the existing `recognizer_registry` / `recognizer_match`
|
||||||
|
modules become the new primitive registry or are replaced with
|
||||||
|
fresh modules under `generate/comprehension/`.
|
||||||
|
|
||||||
|
## Closing observation
|
||||||
|
|
||||||
|
The hard part of the session was not the new architecture — it was
|
||||||
|
recognizing that ADR-0163's prescription, which had landed only days
|
||||||
|
earlier and was actively being extended (PRs #305, #306, #307, #308,
|
||||||
|
#309, #310, #315), was wrong in its mechanism even though right in its
|
||||||
|
diagnosis. The mechanism was *institutionalizing* the regex template
|
||||||
|
approach by routing it through the corridor.
|
||||||
|
|
||||||
|
The operator had been holding the right intuition the whole time:
|
||||||
|
"sentences come in all shapes and forms." That intuition is now an
|
||||||
|
ADR, an invariant boundary, and an architectural transition plan with
|
||||||
|
acceptance gates.
|
||||||
Loading…
Reference in a new issue