docs(ADR-0164,0165): incremental comprehension reader + regex scope rule (#317)
Replace the regex sentence-template front-end of the math admissibility layer with an incremental compositional reader. Lock the architectural boundary that regex is permitted only at the lexeme level, never as sentence-structure templates. ADR-0164 (Proposed) — Incremental Comprehension Reader. Word-by-word state accumulation over a closed set of semantic categories, with the operational lexicon living as a pack-shaped data artifact under language_packs/data/en_core_math_v1/. Reader output type matches the existing regex parser's output, so the binding-graph admissibility (ADR-0132/0133/0134/0135), the solver (ADR-0116), and the verifier (ADR-0117) stay unchanged. wrong=0 is preserved by construction — the reader produces inputs to the existing admissibility gate, not a bypass around it. Phased coexistence with the regex layer during transition; regex sentence templates removed in Phase 3. ADR-0165 (Proposed) — Regex Scope Rule. Structural invariant: regex matches one piece of orthographic material with a closed rule (currency literal, fraction literal, percentage, time-amount, closed unit-noun sets), never a sentence shape. Lexeme-primitive registry is closed and grown through the same contemplation -> proposal -> HITL review corridor that grows vocabulary (ADR-0150 / 0152 / 0155 / 0161). The engine acquires new recognition tools through reviewed teaching, not through operator edits to parser code. ADR-0163's diagnosis (front-end is the bottleneck) is reaffirmed. Its Phase B-E prescription (regex DerivedRecognizers via recognizer_match.py) is partially superseded by ADR-0164. ADR-0136 and its S-family (S.1 / S.2 / S.3 / S.4) have the same disposition: regex sentence-template prescription superseded; empirical refusal taxonomies and closed-set vocabulary preserved as lexicon seed. The HITL corridor architecture is preserved; what flows through it changes from regex recognizers to lexicon entries, categories, and lexeme primitives. Session log SESSION-2026-05-26-comprehension-reader.md captures the narrative of how this decision emerged from the post-D.2 train-sample baseline review (correct=3 refused=47 wrong=0, 34/47 refusals at the question gate). No runtime code changes. ADRs only.
This commit is contained in:
Shay
GitHub
parent
0b4a87beae
commit
e705f27d2e
10 changed files with 1001 additions and 10 deletions
|
|
@ -1,9 +1,30 @@
|
|||
# ADR-0136 — Statement-Layer Corridor: Graduated GSM8K Admission via Parser Extension
|
||||
|
||||
**Status:** Active
|
||||
**Status:** Active — *Regex sentence-template prescription superseded by [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) (2026-05-26). Empirical taxonomies preserved.*
|
||||
**Date:** 2026-05-23
|
||||
**Author:** CORE agents + reviewers
|
||||
**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md)
|
||||
**Superseded in part by:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
|
||||
|
||||
---
|
||||
|
||||
## Amendment 2026-05-26 — Regex prescription superseded
|
||||
|
||||
This corridor's *production mechanism* — adding regex sentence-template
|
||||
patterns to `generate/math_candidate_parser.py` per S-stage sub-ADR — is
|
||||
superseded by ADR-0164's incremental comprehension reader. The closed-set
|
||||
vocabulary collected by each S-stage (verb lists, mass-noun lists, name
|
||||
lists, unit-noun lists) is **preserved** as seed input to the new
|
||||
operational lexicon (ADR-0164 §Decision §1). The empirical refusal
|
||||
taxonomies (`refusal_taxonomy_v*.json`) are **preserved** as input
|
||||
evidence for category and primitive development. The regex patterns
|
||||
themselves are scheduled for removal during ADR-0164 Phase 3.
|
||||
|
||||
Per [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), regex
|
||||
remains permitted at the lexeme-primitive level (currency literal,
|
||||
fraction literal, etc.) and forbidden at the sentence-structure level.
|
||||
|
||||
---
|
||||
**Depends on:**
|
||||
[ADR-0115](./ADR-0115-math-problem-parser-and-graph.md),
|
||||
[ADR-0116](./ADR-0116-deterministic-solver.md),
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# ADR-0136.S.1 — Rate/Event Statement Parsing
|
||||
|
||||
**Status:** Accepted
|
||||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
||||
**Status:** Accepted — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||
**Parent:** ADR-0136 (Statement Layer Corridor) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||
**Date:** 2026-05-23
|
||||
|
||||
## Context
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# ADR-0136.S.2 — Conditional-Op Question (Statement-Layer Corridor)
|
||||
|
||||
**Status:** Active
|
||||
**Status:** Active — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||
**Date:** 2026-05-23
|
||||
**Parent:** [ADR-0136](./ADR-0136-statement-layer-corridor.md)
|
||||
**Parent:** [ADR-0136](./ADR-0136-statement-layer-corridor.md) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# ADR-0136.S.4 — Novel Initial-Form Subject-Slot Widenings
|
||||
|
||||
**Status:** Accepted
|
||||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
||||
**Status:** Accepted — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||
**Parent:** ADR-0136 (Statement Layer Corridor) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||
**Date:** 2026-05-23
|
||||
|
||||
## Context
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# ADR-0136.S.3 — Compound Initial-Mutation Extractor
|
||||
|
||||
**Status:** Accepted
|
||||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
||||
**Status:** Accepted — *regex patterns scheduled for removal under [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) Phase 3; closed-set vocabulary preserved as lexicon seed*
|
||||
**Parent:** ADR-0136 (Statement Layer Corridor) — see [ADR-0136 §Amendment 2026-05-26](./ADR-0136-statement-layer-corridor.md)
|
||||
**Date:** 2026-05-23
|
||||
|
||||
## Context
|
||||
|
|
|
|||
|
|
@ -1,11 +1,49 @@
|
|||
# ADR-0163 — Path to GSM8K mastery: candidate-graph admissibility via the contemplation/HITL corridor
|
||||
|
||||
**Status:** Proposed
|
||||
**Status:** Proposed — *Phases B–E prescription superseded by [ADR-0164](./ADR-0164-incremental-comprehension-reader.md) (2026-05-26)*
|
||||
**Date:** 2026-05-26
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Parent:** [ADR-0114a — Capability Obligations](./ADR-0114a-capability-obligations.md), [ADR-0119 — GSM8K eval lane](./)
|
||||
**Companions:** [ADR-0149 — Recognizer pipeline](./), [ADR-0151 — Auto-proposal pipeline](./ADR-0151-auto-proposal-pipeline.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0155 — CI contemplation runner](./ADR-0155-ci-contemplation-runner.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md), [ADR-0132/0133/0134/0135 — Binding graph](./)
|
||||
**Superseded in part by:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
|
||||
|
||||
---
|
||||
|
||||
## Amendment 2026-05-26 — Prescription superseded by ADR-0164
|
||||
|
||||
After observing the post-D.2 train-sample baseline (`correct=3 refused=47
|
||||
wrong=0`, with 34/47 refusals at the question gate), this ADR's *diagnosis*
|
||||
is reaffirmed and its *prescription* is partially superseded.
|
||||
|
||||
**Preserved (load-bearing):**
|
||||
|
||||
- The diagnosis that the front-end (`math_candidate_parser.py`,
|
||||
`math_candidate_graph.py`) is the bottleneck, not the binding graph or
|
||||
the solver.
|
||||
- The `wrong = 0` invariant doctrine.
|
||||
- The contemplation → proposal → review HITL corridor as the
|
||||
population mechanism for new recognition capability (now applied to
|
||||
lexicon entries and lexeme primitives instead of regex recognizers).
|
||||
- Phase A — the refusal taxonomy work. Its outputs
|
||||
(`refusal_taxonomy_v*.json`) remain valid input evidence.
|
||||
- Phase F — the scope expansion to public / holdout / full GSM8K.
|
||||
|
||||
**Superseded by ADR-0164:**
|
||||
|
||||
- Phases B–E *prescription* — specifically, the production of regex-based
|
||||
`DerivedRecognizer` records that land in
|
||||
`generate/recognizer_match.py`. ADR-0164 replaces this with an
|
||||
incremental compositional reader that consumes lexicon entries and
|
||||
lexeme primitives. The corridor is unchanged; what flows through it
|
||||
changes.
|
||||
- Constraint #2 of this ADR ("No hand-rolled recognizers in `generate/`")
|
||||
is *tightened* by ADR-0165: regex sentence-templates are forbidden
|
||||
regardless of who writes them. Regex remains permitted at the
|
||||
lexeme-primitive level only.
|
||||
|
||||
See [ADR-0164 §What's deprecated, what's preserved](./ADR-0164-incremental-comprehension-reader.md)
|
||||
for the full transition plan and acceptance gates.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
390
docs/decisions/ADR-0164-incremental-comprehension-reader.md
Normal file
390
docs/decisions/ADR-0164-incremental-comprehension-reader.md
Normal file
|
|
@ -0,0 +1,390 @@
|
|||
# ADR-0164 — Incremental Comprehension Reader (replaces regex sentence-template parsing)
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-26
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Parent:** [ADR-0163 — Path to GSM8K mastery](./ADR-0163-gsm8k-path-to-mastery.md)
|
||||
**Companions:** [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md), [ADR-0132/0133/0134/0135 — Binding graph](./), [ADR-0150/0152/0155/0161 — Contemplation / HITL corridor](./), [ADR-0114a — Anti-overfitting proof obligations](./ADR-0114a-anti-overfitting-proof-obligations.md)
|
||||
**Supersedes in part:**
|
||||
- [ADR-0163](./ADR-0163-gsm8k-path-to-mastery.md) §Phase B–E *prescription* (the regex-based `DerivedRecognizer` production path). Its diagnosis and its HITL corridor are preserved.
|
||||
- [ADR-0136 — Statement Layer Corridor](./ADR-0136-statement-layer-corridor.md) and the [ADR-0136.S.1–S.4](./) sub-family (regex sentence-template additions). Their empirical refusal taxonomies are preserved as input evidence; the regex prescription is replaced.
|
||||
|
||||
---
|
||||
|
||||
## Context — why the front-end was the bottleneck, and why the prescribed fix doesn't fix it
|
||||
|
||||
ADR-0163 correctly identified that the GSM8K capability gap sits *before* the
|
||||
binding graph and solver, in `generate/math_candidate_parser.py` and
|
||||
`generate/math_candidate_graph.py`. The downstream substrate
|
||||
(`MathProblemGraph`, the binding-graph admissibility check, the solver, the
|
||||
verifier, the realizer) is mastered in isolation and passes every controlled
|
||||
capability axis at 100% with `wrong = 0`. GSM8K refuses at near-100% because
|
||||
its statements span surface shapes the front-end has never been taught.
|
||||
|
||||
That diagnosis is preserved verbatim by this ADR.
|
||||
|
||||
The *prescription* of ADR-0163 — broaden the recognizer set via the
|
||||
contemplation → proposal → review corridor, where each accepted recognizer is
|
||||
a typed regex matcher in `generate/recognizer_match.py` — does not fix the
|
||||
underlying problem. It institutionalizes it.
|
||||
|
||||
A regex template is, by construction, an enumeration of one surface shape.
|
||||
Each accepted recognizer covers exactly the cases its pattern matches and
|
||||
refuses on every novel phrasing of the same underlying mathematical
|
||||
structure. The post-D.2 baseline measured this directly:
|
||||
|
||||
```text
|
||||
GSM8K train_sample/v1: correct=3 refused=47 wrong=0
|
||||
exit_criterion: { correct_min: 10, wrong_max: 0, passed: false }
|
||||
```
|
||||
|
||||
The refusal split is diagnostic:
|
||||
|
||||
- **34/47** are `no admissible candidate for question:` — the statements
|
||||
parsed, but the question surface form did not match any of the ~6 question
|
||||
regexes in `math_candidate_parser.py` (Pattern A/B/C, capacity, earnings,
|
||||
conditional-op).
|
||||
- **9/47** are `no admissible candidate for statement:` — a statement hit a
|
||||
recognizer gap (fractions, rate-with-currency, periodic temporal).
|
||||
- **4/47** are `no branch produced a solvable graph` — statements + question
|
||||
admitted but the solver couldn't close.
|
||||
|
||||
The question grammar is the dominant bottleneck. The current question
|
||||
patterns try to enumerate ~6 frames of "what an English math-problem question
|
||||
looks like." English doesn't have a closed grammar for math-problem
|
||||
questions, so the enumeration is unbounded and the refusal rate climbs with
|
||||
linguistic diversity. Adding a seventh, eighth, twentieth pattern is not a
|
||||
limit-decreasing operation; the refusal-rate ceiling is set by the regex
|
||||
template approach itself.
|
||||
|
||||
---
|
||||
|
||||
## Diagnosis — regex sentence-templates overfit by design
|
||||
|
||||
A regex template at the **sentence-structure** level claims that a class of
|
||||
meanings (e.g. "ask for a residual quantity") has a closed orthographic form
|
||||
(e.g. `How\s+much\s+(money|...)\s+(will|did)\s+...\s+(make|earn|...)`). This
|
||||
claim is false for natural language. Three consequences follow:
|
||||
|
||||
1. **Refusal is brittle.** "How much will it cost him?" and "how much did he
|
||||
pay in total?" and "how much money will she be left with after the
|
||||
purchase?" all ask the solver for the same kind of output — the value of
|
||||
one terminal-state quantity — but no template covers all three, and each
|
||||
missing template is a refusal.
|
||||
|
||||
2. **The fix path is unbounded.** Each refused phrasing produces a new
|
||||
recognizer. Each new recognizer adds vocabulary and structural assumptions.
|
||||
The set has no closure: there is no point at which "all GSM8K question
|
||||
shapes have been covered" because the set of question shapes is not finite.
|
||||
|
||||
3. **The model loses comprehension.** A template either matches or it
|
||||
doesn't. It has no partial understanding. There is no state in which the
|
||||
engine has "read three words and narrowed the interpretation" — the
|
||||
pattern matches the whole sentence or refuses. That is the opposite of
|
||||
how comprehension works.
|
||||
|
||||
ADR-0163's pathway (recognizer-via-contemplation) addresses *who writes the
|
||||
regex* (the contemplation loop, not the operator). It does not address
|
||||
*whether regex sentence-templates are the right representation at all.* They
|
||||
are not.
|
||||
|
||||
---
|
||||
|
||||
## Decision — incremental comprehension reader
|
||||
|
||||
Replace the regex sentence-template front-end with an **incremental
|
||||
compositional reader** that processes one token at a time, maintains an
|
||||
immutable partial-comprehension state, and produces the same downstream
|
||||
types (`CandidateInitial`, `Operation`, `MathProblemGraph`,
|
||||
`BoundUnknown`-input fields) the regex parser produces today.
|
||||
|
||||
The downstream substrate is unchanged. The binding graph, admissibility
|
||||
check, solver, verifier, realizer, and round-trip filter all stay in place
|
||||
and continue to enforce the `wrong = 0` invariant.
|
||||
|
||||
### Three components
|
||||
|
||||
**1. Operational lexicon (data, not code).**
|
||||
|
||||
Each word in the comprehension vocabulary maps to a *semantic category* and
|
||||
an *update rule*. The category carries the generalization; adding a word is
|
||||
adding a lookup, never a rule.
|
||||
|
||||
Example category set (closed, ADR-tracked, extended only by ratification):
|
||||
|
||||
| Category | Examples | Role in reader state |
|
||||
|---|---|---|
|
||||
| `question_open` | how, what | open question frame |
|
||||
| `question_continuous_qty` | much, long, far, old | continuous-quantity question |
|
||||
| `question_discrete_qty` | many | discrete-count question |
|
||||
| `question_comparative` | more, less, longer, fewer | mark question as `difference` |
|
||||
| `residual_modifier` | left, remaining, after | terminal-state residual |
|
||||
| `aggregate_modifier` | total, in all, altogether, combined | sum across entities |
|
||||
| `accumulation_verb` | earn, make, gain, accumulate, save | additive op |
|
||||
| `depletion_verb` | spend, pay, lose, give | subtractive op |
|
||||
| `transfer_verb` | give, send, pass | transfer op |
|
||||
| `distributive_modifier` | each, per | bind rate or multiply |
|
||||
| `currency_unit_noun` | money, dollars, profit, income, savings, cost | unit class: currency |
|
||||
| `count_unit_noun` | apples, books, kids, chickens, … | unit class: countable |
|
||||
| `time_unit_noun` | hour, day, week, minute, year | unit class: time |
|
||||
| `entity_pronoun` | she, he, they, it | binds resolved entity |
|
||||
| `proper_noun_entity` | Tina, Marion, Jen, … | binds entity directly |
|
||||
|
||||
The lexicon lives under `language_packs/data/en_core_math_v1/` parallel to
|
||||
`en_core_cognition_v1` and `en_core_relations_v1`, with the same loader
|
||||
discipline, the same manifest-checksum rule (CLAUDE.md §Semantic Pack
|
||||
Discipline), and the same review pathway (ADR-0150/0152/0155/0161). New
|
||||
lexicon entries enter through reviewed teaching, never via operator edits.
|
||||
|
||||
The vocabulary already collected in `math_candidate_parser.py` —
|
||||
`_MASS_NOUNS`, `_PATTERN_A_VERBS`, `_PATTERN_B_VERBS`, `_PATTERN_C_VERBS`,
|
||||
`_CAPACITY_VERB_PATTERN`, `_EARNINGS_VERB_PATTERN`, `ADD_VERBS`,
|
||||
`SUBTRACT_VERBS`, `TRANSFER_VERBS`, `_FEMALE_NAMES`, `_MALE_NAMES` — is
|
||||
**ported wholesale** as the seed corpus of the new lexicon. That ratified
|
||||
vocabulary is good work; only its container (regex character classes inside
|
||||
sentence templates) is wrong.
|
||||
|
||||
**2. Partial-comprehension state (immutable).**
|
||||
|
||||
```text
|
||||
ComprehensionState:
|
||||
entities: tuple[EntityRef, ...] # who's been mentioned
|
||||
quantities: tuple[QuantityRef, ...] # numbers with units, attached or floating
|
||||
operations: tuple[PartialOp, ...] # verb-induced operations, possibly incomplete
|
||||
question_target: QuestionTargetSlot | None # what's being asked, possibly partial
|
||||
expectation: ExpectationFrame | None # what category would close the current frame
|
||||
```
|
||||
|
||||
`expectation` is the load-bearing field for recontextualization. After
|
||||
reading "How much money will she", the state's expectation is "an
|
||||
accumulation verb, a depletion verb, a residual modifier, or a
|
||||
state-continuation verb." Each closes the question frame differently. The
|
||||
expectation is what tells the reader how to interpret an ambiguous next
|
||||
word.
|
||||
|
||||
State is frozen-dataclass immutable. Canonical-bytes serialization
|
||||
(sorted-key, fixed-precision) keeps `trace_hash` deterministic per CLAUDE.md
|
||||
§Runtime Surface Contract.
|
||||
|
||||
**3. Deterministic reader (state machine over categories).**
|
||||
|
||||
```text
|
||||
apply_word(state, word) -> state | Refusal
|
||||
```
|
||||
|
||||
For each token:
|
||||
1. **Lexical primitive scan** (ADR-0165): try to match orthographic
|
||||
primitives — currency literal, fraction literal, numeric literal,
|
||||
percentage literal, time-unit noun — in priority order. If one fires,
|
||||
the token becomes a typed lexeme with extracted value(s) and category.
|
||||
2. **Lexicon lookup**: if no primitive fired, look up the surface form in
|
||||
the operational lexicon. If absent, refuse with
|
||||
`unknown_word: <token> (position N)`.
|
||||
3. **Expectation check**: if the token's category satisfies
|
||||
`state.expectation`, apply the update rule. If not — and the category
|
||||
is a legal frame opener at this position — close the current frame and
|
||||
open a new one. If neither — refuse with
|
||||
`unexpected_category: got <cat>, expected <frame>`.
|
||||
4. Emit new state.
|
||||
|
||||
End-of-sentence: the state must satisfy a finalization predicate
|
||||
(question_target is bound, operations have their operands, dangling
|
||||
quantities have unit attachments). Otherwise refuse with `unfinished_frame`.
|
||||
|
||||
The reader is a deterministic shift-reduce parser **over semantic
|
||||
categories**, not over tokens. The category set is ~20 items; the
|
||||
composition rules total 30–50. Adding a verb does not change a rule. Adding
|
||||
a category requires an ADR.
|
||||
|
||||
### Output
|
||||
|
||||
The reader emits one of:
|
||||
- A `MathProblemGraph` (and the underlying `CandidateInitial` /
|
||||
`Operation` tuple) ready for the existing candidate-graph admissibility
|
||||
layer, or
|
||||
- A typed `ReaderRefusal` carrying the token position, the failed
|
||||
expectation, and the closest legal next category. Refusals are the
|
||||
evidence the teaching loop chews on (Phase E below).
|
||||
|
||||
Downstream consumption is unchanged. The binding-graph adapter (ADR-0133),
|
||||
the `BoundUnknown` resolver (ADR-0135), the admissibility check (ADR-0134),
|
||||
the solver (ADR-0116), and the verifier (ADR-0117) all act on the reader's
|
||||
output exactly as they act on the regex parser's output today. The
|
||||
`wrong = 0` invariant is preserved by construction because the reader does
|
||||
not *bypass* admissibility — it produces inputs to it.
|
||||
|
||||
---
|
||||
|
||||
## Constraints (non-negotiable)
|
||||
|
||||
1. **`wrong = 0` at every phase, every round, every split.** The reader can
|
||||
be more permissive about *which sentences it comprehends* without
|
||||
weakening *what comprehension produces*. The existing admissibility,
|
||||
unit-proof, and multi-branch-disagreement refusal stay in force.
|
||||
|
||||
2. **No hidden normalization, stochastic fallback, or "best guess."** The
|
||||
reader refuses cleanly on novel structure. No softmax over candidate
|
||||
parses, no nearest-template selection, no default category.
|
||||
|
||||
3. **No regex sentence-templates.** Per [ADR-0165](./ADR-0165-regex-scope-rule.md),
|
||||
regex is allowed only at the lexeme level (currency literal, fraction
|
||||
literal, etc.). Any regex that matches across word combination is a
|
||||
grammar template and forbidden.
|
||||
|
||||
4. **Lexicon and category set are closed and ADR-tracked.** New lexicon
|
||||
entries land through reviewed teaching (the existing ADR-0150/0152/0155/
|
||||
0161 corridor — preserved from ADR-0163). New categories or new
|
||||
composition rules require an ADR.
|
||||
|
||||
5. **Deterministic replay.** Identical input → byte-equal reader output. The
|
||||
`ComprehensionState` has canonical-bytes serialization. The reader emits
|
||||
a deterministic trace that feeds `trace_hash`.
|
||||
|
||||
---
|
||||
|
||||
## What's deprecated, what's preserved
|
||||
|
||||
### Deprecated by this ADR
|
||||
|
||||
- **ADR-0163 §Phase B–E prescription**: the production of regex-based
|
||||
`DerivedRecognizer` records that land in
|
||||
`generate/recognizer_match.py`. New recognizers in this form are blocked
|
||||
starting with the reader's first acceptance round. Existing recognizers
|
||||
remain dormant during the transition (see Coexistence below) and are
|
||||
removed once their categories are covered by the reader.
|
||||
- **ADR-0136 — Statement Layer Corridor** and the sub-family
|
||||
[ADR-0136.S.1–S.4](./): regex sentence-template additions to
|
||||
`math_candidate_parser.py`. The empirical refusal taxonomies they
|
||||
produced are preserved as input evidence for lexicon and category work.
|
||||
The patterns themselves are scheduled for removal once the reader
|
||||
covers their cases.
|
||||
- The `Pattern A` / `Pattern B` / `Pattern C` regex blocks introduced by
|
||||
ADR-0163.D.4 in `generate/math_candidate_parser.py` (`_Q_MASS_NOUN_RE`,
|
||||
`_Q_COMPARATIVE_RE`, `_Q_PRONOUN_VERB_RE`) — replaced by the reader's
|
||||
question-frame composition rules.
|
||||
|
||||
### Preserved in full
|
||||
|
||||
- **The binding graph** (ADR-0132/0133/0134/0135). The reader produces the
|
||||
same input types it consumes today.
|
||||
- **The HITL corridor** (ADR-0150/0152/0155/0161). New lexicon entries and
|
||||
new categories ride the same contemplation → proposal → review pathway.
|
||||
ADR-0163's corridor architecture is correct; only what flows through it
|
||||
changes (lexicon entries instead of regex recognizers).
|
||||
- **The capability-axis lanes** (G1–G5, S1). They continue to validate the
|
||||
downstream substrate and act as the regression net for any reader
|
||||
change.
|
||||
- **`wrong = 0` doctrine** and the replay-equivalence gate.
|
||||
- **All closed-set vocabulary** previously collected by the regex parser.
|
||||
It is the seed of the operational lexicon.
|
||||
|
||||
### Untouched but adjacent
|
||||
|
||||
- The `recognizer_registry` / `recognizer_match` modules become the lexicon
|
||||
loader and lexical-primitive registry rather than the regex pattern
|
||||
store. The interface signature changes but the corridor-driven
|
||||
*population* of these registries is preserved.
|
||||
|
||||
---
|
||||
|
||||
## Phasing
|
||||
|
||||
### Phase 1 — Question reader (where 34/47 refusals live)
|
||||
|
||||
Build the reader for question sentences only. The output type is narrow:
|
||||
just the fields `BoundUnknown` consumes (`entity`, `unit`, question_form).
|
||||
Coexist with the existing regex question patterns: reader runs first; on
|
||||
refusal, falls through to existing regex; on reader acceptance, regex is
|
||||
not invoked. Measure pickup rate against `train_sample/v1` per round.
|
||||
|
||||
Acceptance for Phase 1:
|
||||
- Reader covers ≥20/34 currently-refused question cases.
|
||||
- Combined (reader + legacy) `correct ≥ 10` on the 50-case sample with
|
||||
`wrong = 0`. This satisfies the Round-1 exit criterion of ADR-0163.
|
||||
- Reader has zero disagreement with regex on the 6 cases where both fire
|
||||
(3 correct + 3 secondary), per byte-equal `BoundUnknown` output.
|
||||
|
||||
### Phase 2 — Statement reader
|
||||
|
||||
Extend the reader to statement sentences. Coexist with existing regex
|
||||
statement patterns the same way. Phase out the regex statement patterns
|
||||
incrementally as reader coverage grows.
|
||||
|
||||
Acceptance for Phase 2:
|
||||
- Reader covers ≥30/50 train_sample cases end-to-end (statements +
|
||||
question both via reader).
|
||||
- `correct ≥ 25` (ADR-0163 Round-2 exit) with `wrong = 0`.
|
||||
|
||||
### Phase 3 — Regex layer removal
|
||||
|
||||
Once reader coverage ≥ regex coverage on a case-by-case basis, the regex
|
||||
sentence-template layer is deleted. The lexical-primitive layer (regex
|
||||
applied to single orthographic shapes per ADR-0165) survives — that is the
|
||||
correct use of regex and is not what this ADR deprecates.
|
||||
|
||||
Acceptance for Phase 3:
|
||||
- `correct ≥ 35` on train_sample, `wrong = 0` (ADR-0163 Round-3 exit).
|
||||
- `math_candidate_parser.py` no longer contains sentence-level regex
|
||||
patterns. Closed-set vocabulary tables remain (now consumed by the
|
||||
lexicon loader rather than woven into regexes).
|
||||
|
||||
### Phase 4 — Scale
|
||||
|
||||
Per ADR-0163 §Phase F: public, holdout, full GSM8K. No changes to that
|
||||
scope from this ADR; the reader simply replaces the front-end.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria for this ADR (Proposed → Accepted)
|
||||
|
||||
This ADR moves to **Accepted** when:
|
||||
|
||||
1. A `ComprehensionState` prototype exists in `generate/comprehension/`
|
||||
with frozen-dataclass shape, canonical-bytes serialization, and unit
|
||||
tests pinning determinism.
|
||||
2. The seed lexicon pack `en_core_math_v1` is materialized from the
|
||||
existing closed-set vocabulary in `math_candidate_parser.py`, with the
|
||||
standard pack-test discipline.
|
||||
3. Phase 1 acceptance is met on `train_sample/v1`.
|
||||
4. Capability-axis lanes G1–G5, S1 remain at 100% `wrong = 0` (regression
|
||||
net unbroken).
|
||||
5. `verify pinned lane SHAs` continues to pass.
|
||||
|
||||
---
|
||||
|
||||
## Open questions (resolve before Phase 1 PR)
|
||||
|
||||
1. **Lexical primitive set scope.** Inventory of which orthographic shapes
|
||||
get primitives vs. lexicon entries (currency literal, fraction literal,
|
||||
percentage literal, decimal literal, time-unit noun, dollar-amount,
|
||||
ordinal). Likely a sub-ADR (ADR-0164.1).
|
||||
2. **Ambiguity resolution precedence.** When a token could open two
|
||||
frames, the precedence order. Likely a sub-ADR after Phase 1
|
||||
measurement reveals which collisions are real.
|
||||
3. **Pronoun-entity resolution.** The reader needs entity resolution
|
||||
anyway; the regex parser's `_resolve_question_entity` heuristic is a
|
||||
reasonable starting point but should be reviewed against the
|
||||
compositional model.
|
||||
4. **Cross-sentence state.** The current regex parser is per-sentence;
|
||||
GSM8K problems have cross-sentence references ("she" referring to
|
||||
"Tina" three sentences earlier). The reader will need a
|
||||
`ProblemReadingState` that persists across sentences. Scope this in
|
||||
Phase 1 design.
|
||||
|
||||
---
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Bottleneck evidence**: `evals/gsm8k_math/train_sample/v1/report.json`,
|
||||
`refusal_taxonomy_v4.json`.
|
||||
- **Substrate that survives**: `generate/binding_graph/`,
|
||||
`generate/math_solver.py`, `generate/math_verifier.py`,
|
||||
`generate/math_realizer.py`, capability-axis lanes.
|
||||
- **The corridor**: ADR-0150 (contemplation), ADR-0152 (learning-arc),
|
||||
ADR-0155 (CI contemplation runner), ADR-0161 (HITL queue).
|
||||
- **The boundary rule**: ADR-0165.
|
||||
- **The anti-overfitting doctrine**: ADR-0114a.
|
||||
- **The thesis**: `[[thesis-decoding-not-generating]]` — the reader is a
|
||||
decoder. Each word narrows the space; the meaning is the accumulation,
|
||||
not the match.
|
||||
270
docs/decisions/ADR-0165-regex-scope-rule.md
Normal file
270
docs/decisions/ADR-0165-regex-scope-rule.md
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
# ADR-0165 — Regex Scope Rule: Lexemes Only, Never Grammar
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-26
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Companions:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md), [ADR-0114a — Anti-overfitting proof obligations](./ADR-0114a-anti-overfitting-proof-obligations.md), [ADR-0150 — Autonomous inter-session contemplation](./ADR-0150-autonomous-inter-session-contemplation.md), [ADR-0152 — Learning-arc proof corridor](./ADR-0152-learning-arc-demo.md), [ADR-0161 — HITL async queue](./ADR-0161-hitl-async-queue.md)
|
||||
|
||||
---
|
||||
|
||||
## Context — where regex creeps in, and why the line matters
|
||||
|
||||
CORE's existing front-end parsers (`generate/math_candidate_parser.py`,
|
||||
`generate/recognizer_match.py`) use regex for two structurally different
|
||||
jobs that have been collapsed into one:
|
||||
|
||||
1. **Recognizing lexemes** — "this token chunk is a currency literal," "this
|
||||
is a fraction `\d+/\d+`," "this is a numeric expression." These have
|
||||
genuinely closed orthographic rules. Regex is the right primitive.
|
||||
|
||||
2. **Matching sentence structure** — "this whole sentence is the shape
|
||||
`How much MASS_NOUN does ENTITY VERB ...`." These do not have a closed
|
||||
rule, because natural language doesn't have one. Regex here is
|
||||
enumeration of memorized shapes, dressed up as grammar.
|
||||
|
||||
Mixing the two has been the engine's most consistent source of overfitting.
|
||||
[ADR-0164](./ADR-0164-incremental-comprehension-reader.md) replaces the
|
||||
sentence-structure use with an incremental compositional reader. This ADR
|
||||
locks the *boundary*: a structural invariant that any future code must
|
||||
respect, independent of which front-end implementation is current.
|
||||
|
||||
The rule is in the same spirit as the existing structural invariants —
|
||||
`versor_condition(F) < 1e-6`, "no normalization outside the gate", "no
|
||||
approximate vault recall", "no hidden stochastic fallback." It is a typed
|
||||
boundary, not a guideline.
|
||||
|
||||
---
|
||||
|
||||
## Rule
|
||||
|
||||
> **Regex is permitted only at the lexeme level.** Regex must operate on the
|
||||
> orthographic shape of one structured token or contiguous token-class run
|
||||
> whose meaning has a genuinely closed rule. Regex must not match across
|
||||
> word combination, syntactic role, or sentence structure.
|
||||
|
||||
**Test for any regex literal in the runtime path:**
|
||||
|
||||
If the regex describes "what this piece of orthographic material looks
|
||||
like," it is a *lexeme primitive* and is permitted.
|
||||
|
||||
If the regex describes "how these words combine to mean X," it is a
|
||||
*grammar template* and is forbidden.
|
||||
|
||||
---
|
||||
|
||||
## Legitimate uses (lexeme primitives)
|
||||
|
||||
These are the canonical examples. The list is extensible through the
|
||||
teaching corridor (§Population, below).
|
||||
|
||||
| Primitive | Example regex | Extracts | Category emitted |
|
||||
|---|---|---|---|
|
||||
| `currency_literal` | `\$(\d+(?:\.\d+)?)` | numeric value | QUANTITY, unit_class=currency |
|
||||
| `numeric_literal` | `\d+(?:\.\d+)?` | numeric value | QUANTITY (unit pending) |
|
||||
| `fraction_literal` | `(\d+)\s*/\s*(\d+)` | numerator, denominator | QUANTITY, kind=fraction |
|
||||
| `percentage_literal` | `(\d+(?:\.\d+)?)\s*%` | numeric value | QUANTITY, unit_class=ratio |
|
||||
| `time_amount_literal` | `(\d+)[- ]?(hour\|minute\|day\|week\|month\|year)s?` | value, unit | QUANTITY, unit_class=time |
|
||||
| `mass_noun_token` | `(?:money\|profit\|interest\|...)` | the lexeme | UNIT_CATEGORY_TOKEN |
|
||||
| `decimal_currency` | `\$\d+\.\d{2}` | value | QUANTITY, unit_class=currency |
|
||||
| `ordinal_token` | `(?:first\|second\|third\|...)` | rank | ORDINAL |
|
||||
|
||||
Each primitive has:
|
||||
|
||||
- a **name** (closed registry key),
|
||||
- a **pattern** (a single, focused regex over orthographic shape),
|
||||
- an **emission** (the typed category + extracted values it produces),
|
||||
- a **provenance** (which ADR or teaching ratification introduced it).
|
||||
|
||||
Primitives never reach across roles. `currency_literal` recognizes
|
||||
`$18.00`; it does not recognize `$18.00 an hour` (that composition is the
|
||||
reader's job, not the primitive's).
|
||||
|
||||
---
|
||||
|
||||
## Forbidden uses (grammar templates)
|
||||
|
||||
These are the patterns ADR-0164 deprecates and this rule forbids
|
||||
recurring. Each example below is a *grammar template* and would be rejected
|
||||
in code review under this ADR.
|
||||
|
||||
```python
|
||||
# FORBIDDEN — regex matching question shape:
|
||||
_Q_MASS_NOUN_RE = re.compile(
|
||||
r"^How\s+much\s+"
|
||||
rf"(?P<unit>{_MASS_NOUN_PATTERN})"
|
||||
r"\s+(?:will|did|does|do|would)\s+"
|
||||
rf"(?P<entity>{_Q_ENTITY_OR_PRONOUN})\s+"
|
||||
rf"(?:have\s+earned\s+|be\s+able\s+to\s+)?{_PATTERN_A_VERBS}"
|
||||
r"(?:\s+.*?)?\??\s*$"
|
||||
)
|
||||
```
|
||||
|
||||
This is a sentence-structure template. It matches across question stem
|
||||
("How much"), unit ("money"), auxiliary ("will"), entity ("Tina"), and
|
||||
verb ("earn"). It claims the conjunction of these elements forms a closed
|
||||
shape. It doesn't.
|
||||
|
||||
```python
|
||||
# FORBIDDEN — regex matching statement structure:
|
||||
_INITIAL_HAS_RE = re.compile(
|
||||
rf"^(?P<entity>{_ENTITY})\s+has\s+(?P<value>\d+)\s+(?P<unit>\w+)\.?\s*$"
|
||||
)
|
||||
```
|
||||
|
||||
Same problem: matches across subject, verb, value, unit, and clause
|
||||
shape. Replace with reader composition rules over the categories
|
||||
`proper_noun_entity`, `possession_verb`, `numeric_literal`, `count_unit_noun`.
|
||||
|
||||
```python
|
||||
# PERMITTED — regex matching one orthographic shape:
|
||||
_CURRENCY_LITERAL_RE = re.compile(r"\$(\d+(?:\.\d+)?)")
|
||||
```
|
||||
|
||||
This recognizes the surface form of one token (with the `$` prefix). It
|
||||
has a closed rule (currency notation). It is the correct use of regex.
|
||||
|
||||
---
|
||||
|
||||
## Code-review test (apply to every new regex)
|
||||
|
||||
When reviewing or writing a regex, answer three questions:
|
||||
|
||||
1. **What does the regex match?** If the answer names a *piece of
|
||||
orthographic material* (a number, a currency amount, a unit-noun set,
|
||||
a date), it's a lexeme primitive. If it names *a way words combine*
|
||||
(a question shape, an assertion shape, a clause pattern), it's a
|
||||
grammar template.
|
||||
|
||||
2. **Could a competent linguist describe the matched class as a closed
|
||||
set of orthographic rules?** If yes, the regex is appropriate. If the
|
||||
class is "things people sometimes say to mean X," the regex is
|
||||
enumerating memorized shapes and is forbidden.
|
||||
|
||||
3. **What happens when a novel phrasing of the same underlying meaning
|
||||
appears?** A lexeme primitive's domain doesn't depend on the rest of
|
||||
the sentence, so novel phrasings around it don't break it. A grammar
|
||||
template refuses on every novel phrasing. If a refusal on novel
|
||||
phrasing is the expected behavior of the regex, it's a grammar
|
||||
template.
|
||||
|
||||
A regex that fails any of these three is a grammar template and must be
|
||||
restructured (typically: extract the closed-set vocabulary as a primitive,
|
||||
move the structural part into reader composition rules).
|
||||
|
||||
---
|
||||
|
||||
## Population — how the primitive set grows
|
||||
|
||||
The closed registry of lexeme primitives is not static. It grows through
|
||||
the same contemplation → proposal → review corridor that already grows the
|
||||
language packs and (under ADR-0164) the operational lexicon. This means:
|
||||
|
||||
1. **The reader refuses on an unrecognized token shape** — for example,
|
||||
it encounters `"$1.5M"` and no current primitive matches it.
|
||||
2. The refusal is logged with its token shape and position
|
||||
(`evals/discovery/discovery_candidates.jsonl` analog).
|
||||
3. The contemplation runner (ADR-0150/0155) identifies the shape as a
|
||||
candidate new primitive and emits a proposal carrying:
|
||||
- the proposed primitive's pattern,
|
||||
- its emission category and extracted fields,
|
||||
- replay-equivalence evidence (acceptance does not lift `wrong` above
|
||||
0 on the active corpus),
|
||||
- the originating refused tokens.
|
||||
4. The proposal lands in the HITL queue (ADR-0161). The operator reviews
|
||||
the pattern and emission rule. On acceptance, the primitive enters the
|
||||
closed registry with its provenance.
|
||||
5. The reader picks up the new primitive on next run. No code edit. No
|
||||
parser rewrite. The engine has been *taught* a new lexical recognizer.
|
||||
|
||||
This matches the user's original framing: regex is a **mental tool** the
|
||||
engine wields. The toolset is bounded and reviewed, but not hard-wired.
|
||||
Adding a tool follows the same ratification discipline as adding a lemma
|
||||
or a category.
|
||||
|
||||
Two consequences of this design:
|
||||
|
||||
- **Operators do not write production regex.** They review proposed
|
||||
regex against typed evidence. This eliminates the failure mode where
|
||||
a regex sneaks in to "just unblock GSM8K case 0027."
|
||||
- **The regex set has a measurable closure curve.** Each ratification
|
||||
round either does or doesn't reduce the refused-token count.
|
||||
Diminishing returns become visible — the regex set is on the same
|
||||
measurement substrate as the lexicon and the recognizer corpus.
|
||||
|
||||
The bootstrap primitive set lands as part of the ADR-0164 Phase 1 PR. It
|
||||
covers the closed orthographic forms already known to be needed from the
|
||||
existing parser (currency, numeric, fraction, percentage, time-amount,
|
||||
ordinal). Everything beyond bootstrap enters via the corridor.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
1. **Overfit-by-design becomes structurally impossible at the regex
|
||||
layer.** A grammar-template regex cannot land in code review without
|
||||
violating this ADR explicitly.
|
||||
2. **Closed-set vocabulary already collected in the existing parser is
|
||||
preserved.** Mass-noun lists, possession-verb lists, name lists — all
|
||||
become lexeme primitives or lexicon entries.
|
||||
3. **One regex toolkit, one lexicon, one composition layer.** Three
|
||||
layered concerns with separate population pathways and separate
|
||||
review criteria. Maintenance scales linearly, not multiplicatively.
|
||||
4. **The teaching corridor's purpose generalizes.** Today the corridor
|
||||
teaches words, domains, and (per the deprecated path) regex
|
||||
recognizers. Under this rule it teaches words, categories, and
|
||||
lexeme primitives — three orthogonal kinds of evidence, each with a
|
||||
clean review predicate.
|
||||
|
||||
### Negative / tradeoffs
|
||||
|
||||
1. **Refactoring cost is real.** Removing existing sentence-template
|
||||
regexes from `math_candidate_parser.py` and `recognizer_match.py` is
|
||||
a substantial edit, even with vocabulary preserved. The transition
|
||||
plan in ADR-0164 (coexistence → incremental removal) absorbs this.
|
||||
2. **Lexeme primitives are an attractive surface for new overfitting.**
|
||||
A primitive author could try to smuggle structure ("a number followed
|
||||
by a currency word followed by 'an hour'") into a single regex. The
|
||||
review criteria above are explicit about this; the corridor enforces
|
||||
it.
|
||||
3. **The reader has to do more work.** Composition that used to live
|
||||
inside a regex now lives in update rules. This is the point —
|
||||
composition is the engine's job, not the regex's — but it shifts
|
||||
complexity from one place to another.
|
||||
|
||||
---
|
||||
|
||||
## Boundaries — what this ADR does **not** say
|
||||
|
||||
1. **It does not forbid regex.** Regex remains a primary tool for lexeme
|
||||
recognition. The current `evals/`, `scripts/`, and CLI parsers
|
||||
already use regex appropriately for log parsing, file-path matching,
|
||||
etc. None of that is affected.
|
||||
2. **It does not specify the reader.** The reader's design is
|
||||
ADR-0164's scope. This ADR only constrains where regex may live
|
||||
in whatever front-end is current.
|
||||
3. **It does not retroactively reject older code wholesale.** ADRs
|
||||
ADR-0136.S.1–S.4 and ADR-0163.D.2–D.4 introduced grammar templates
|
||||
under previous policy. They are deprecated by ADR-0164 with an
|
||||
explicit transition plan. New work follows this rule from acceptance
|
||||
of this ADR forward.
|
||||
|
||||
---
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Sibling ADR**: ADR-0164 — the comprehension reader that occupies the
|
||||
space this rule clears.
|
||||
- **Existing structural invariants** (same spirit, different domain):
|
||||
- `versor_condition(F) < 1e-6` (CLAUDE.md §Field Invariant)
|
||||
- "Allowed normalization sites" (CLAUDE.md §Normalization Rules)
|
||||
- "Exact CGA recall" (CLAUDE.md §Core Primitives)
|
||||
- ADR-0114a Anti-overfitting proof obligations
|
||||
- **Population corridor**: ADR-0150 (contemplation), ADR-0152
|
||||
(learning-arc proof), ADR-0155 (CI contemplation runner), ADR-0161
|
||||
(HITL async queue).
|
||||
- **Anchor**: `[[thesis-decoding-not-generating]]` — regex is a
|
||||
decoder's tool for recognizing fixed orthographic shapes. It is not a
|
||||
generator's tool for hallucinating sentence grammars.
|
||||
|
|
@ -205,6 +205,45 @@ The contract has been demonstrated end-to-end: refused once honestly (ADR-0107),
|
|||
|
||||
---
|
||||
|
||||
### Comprehension-reader pivot — ADR-0164 and ADR-0165 (2026-05-26)
|
||||
|
||||
The GSM8K admissibility front-end is replaced. The regex sentence-template
|
||||
parsing layer in `generate/math_candidate_parser.py` and
|
||||
`generate/recognizer_match.py` is recognized as overfitting by
|
||||
construction — it enumerates memorized surface shapes while pretending to
|
||||
encode a closed grammar that English does not have.
|
||||
|
||||
- **[ADR-0164](ADR-0164-incremental-comprehension-reader.md)** —
|
||||
Incremental Comprehension Reader. Word-by-word state accumulation over a
|
||||
closed set of semantic categories. The reader is a deterministic
|
||||
shift-reduce parser over *categories*, not over tokens. Output type is
|
||||
identical to the regex parser's output, so the binding-graph
|
||||
admissibility (ADR-0132/0133/0134/0135) downstream is unchanged.
|
||||
Operational lexicon lives in `language_packs/data/en_core_math_v1/`
|
||||
alongside the existing packs.
|
||||
|
||||
- **[ADR-0165](ADR-0165-regex-scope-rule.md)** — Regex Scope Rule.
|
||||
Structural invariant: regex is permitted only at the lexeme level
|
||||
(currency literal, fraction literal, percentage literal, time-amount,
|
||||
closed-set unit-noun), never at the sentence-structure level. The
|
||||
primitive set is a closed registry grown through the same contemplation
|
||||
→ proposal → HITL review corridor that grows vocabulary.
|
||||
|
||||
These ADRs preserve every load-bearing piece of the prior work: the
|
||||
binding graph (ADR-0132–0135), the solver / verifier / realizer
|
||||
substrate (ADR-0116–0118), the capability-axis lanes G1–G5 and S1, the
|
||||
HITL corridor (ADR-0150 / 0152 / 0155 / 0161), the `wrong = 0` doctrine
|
||||
and the replay-equivalence gate (ADR-0057, ADR-0114a). The
|
||||
contemplation → proposal → review corridor architecture is reaffirmed
|
||||
and its scope is generalized: it now ratifies lexicon entries,
|
||||
categories, and lexeme primitives — not regex recognizers.
|
||||
|
||||
ADR-0163's *diagnosis* (the front-end is the bottleneck) is reaffirmed;
|
||||
its *prescription* (Phases B–E recognizer production) is partially
|
||||
superseded. ADR-0136 and its S-family have the same disposition.
|
||||
|
||||
---
|
||||
|
||||
## Session Logs
|
||||
|
||||
Session logs record the decisions and rationale from individual working sessions. They are not ADRs — they are the narrative record that informed the ADRs.
|
||||
|
|
@ -215,3 +254,4 @@ Session logs record the decisions and rationale from individual working sessions
|
|||
| 2026-05-12 (addendum) | [SESSION-2026-05-12-b.md](SESSION-2026-05-12-b.md) |
|
||||
| 2026-05-12 (language packs) | [SESSION-2026-05-12-language-packs-addendum.md](SESSION-2026-05-12-language-packs-addendum.md) |
|
||||
| 2026-05-13 | [SESSION-2026-05-13.md](SESSION-2026-05-13.md) |
|
||||
| 2026-05-26 (comprehension reader) | [SESSION-2026-05-26-comprehension-reader.md](SESSION-2026-05-26-comprehension-reader.md) |
|
||||
|
|
|
|||
232
docs/decisions/SESSION-2026-05-26-comprehension-reader.md
Normal file
232
docs/decisions/SESSION-2026-05-26-comprehension-reader.md
Normal file
|
|
@ -0,0 +1,232 @@
|
|||
# SESSION 2026-05-26 — Comprehension Reader Decision
|
||||
|
||||
**Participants:** Shay, Claude (Sonnet 4.6 → Opus 4.7 for ADR drafting)
|
||||
**Outputs:**
|
||||
[ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md),
|
||||
[ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md).
|
||||
**Affected:** [ADR-0163](./ADR-0163-gsm8k-path-to-mastery.md) (prescription
|
||||
partially superseded), [ADR-0136 + sub-family](./ADR-0136-statement-layer-corridor.md)
|
||||
(regex prescription superseded; vocabulary preserved as lexicon seed).
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
|
||||
---
|
||||
|
||||
## What triggered the session
|
||||
|
||||
PR cleanup turn that started as "merge the open PRs" became an architectural
|
||||
session when the operator asked why the post-D.2 GSM8K train_sample baseline
|
||||
remained at `correct=3 refused=47 wrong=0`.
|
||||
|
||||
Three open PRs were on the board at session start:
|
||||
|
||||
- **#316** — `fix(INV-02): allowlist test_issue_300_versor_margin.py` — all
|
||||
checks green, mergeable. Merged first.
|
||||
- **#315** — `feat(ADR-0163.D.2): parsed_anchors → MathProblemGraph state
|
||||
— discrete_count_statement injection v1` — smoke failing because the
|
||||
INV-02 allowlist fix wasn't in its base. Rebased onto new main after
|
||||
#316, smoke turned green, merged.
|
||||
- **#314** — `docs(plan): CORE general advancement path` — rebased onto
|
||||
new main, all checks green, merged.
|
||||
|
||||
Board cleared. Then the substantive question.
|
||||
|
||||
## The diagnostic dive
|
||||
|
||||
The operator asked: "Why aren't we getting more of these answers right?"
|
||||
|
||||
Running the train_sample runner directly produced
|
||||
`correct=3 refused=47 wrong=0` with `exit_criterion: correct_min=10, passed=false`.
|
||||
Refusal-reason aggregation showed the bottleneck precisely:
|
||||
|
||||
- **34 / 47** refusals were `no admissible candidate for question: '<text>'`
|
||||
— statements parsed successfully, but the question surface form did not
|
||||
match any of the ~6 question regexes in
|
||||
`generate/math_candidate_parser.py` (Pattern A / B / C, capacity,
|
||||
earnings, conditional-op).
|
||||
- **9 / 47** refusals were `no admissible candidate for statement: '<text>'`
|
||||
— statement-side regex gaps (5 of them fraction operands).
|
||||
- **4 / 47** refusals were `no branch produced a solvable graph` —
|
||||
statements + question admitted but the solver couldn't close.
|
||||
|
||||
The 3 admitted cases shared a tight structural signature: rate × time
|
||||
patterns plus one distributive multiply + subtract. The exact patterns the
|
||||
regex front-end was originally written to handle.
|
||||
|
||||
The v4 refusal taxonomy
|
||||
(`evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v4.json`) reinforced
|
||||
the picture: 23 distinct primary-barrier categories across 47 cases, with
|
||||
no single category larger than 5 cases. The long tail of distinct shapes
|
||||
is the long tail of English question surface forms.
|
||||
|
||||
## The operator's diagnosis (load-bearing)
|
||||
|
||||
The operator said, plainly:
|
||||
|
||||
> "Obviously the whole regex stuff is overfitting by design… lol. I was
|
||||
> literally wondering about that when it was being built… just thought
|
||||
> you knew what you were doing."
|
||||
|
||||
And:
|
||||
|
||||
> "Regex wasn't meant to be there. And I said, if we are going to allow
|
||||
> regex in, then we teach the model how to use regex itself as a 'mental
|
||||
> tool' of sorts. but not use it to overfit templates to what we want.
|
||||
> That's only ever going to end up being a bottleneck risk. Makes no
|
||||
> sense. If there truly were a said, rule-based system for sentence
|
||||
> structure then that would be different, and we could use all the
|
||||
> 'known' templates."
|
||||
|
||||
This is the architectural pivot. Three points compress into it:
|
||||
|
||||
1. **Sentence-template regex is overfitting by definition.** A regex
|
||||
sentence-template is an enumeration of memorized surface shapes
|
||||
pretending to be a grammar rule. English does not have a closed
|
||||
grammar for math-problem questions. Adding more templates does not
|
||||
approach a limit; the refusal-rate ceiling is set by the *method*,
|
||||
not by template count.
|
||||
|
||||
2. **Regex has a legitimate role at the lexeme level.** Currency
|
||||
literals, fractions, percentages, numeric expressions, closed-set
|
||||
unit-noun lists — these have genuinely closed orthographic rules.
|
||||
Regex is the honest tool for recognizing them. The boundary is:
|
||||
regex describes "what one piece of orthographic material looks like,"
|
||||
never "how words combine to mean X."
|
||||
|
||||
3. **The model should be able to acquire regex tools through review,
|
||||
not have them hard-coded.** The operator had already designed the
|
||||
teaching/contemplation/HITL corridor (ADR-0150 / 0152 / 0155 / 0161)
|
||||
for exactly this purpose. The corridor is general: it can ratify new
|
||||
vocabulary, new categories, and new lexeme primitives through the
|
||||
same review pathway. Regex tools become *data* the engine
|
||||
accumulates through reviewed teaching, not code the operator writes.
|
||||
|
||||
The operator's framing of point 3 was the moment the corridor's purpose
|
||||
generalized in scope: it teaches *recognition capability*, not just
|
||||
*recognized content*. That is the structural difference between a fixed
|
||||
toolkit and an intelligence that can grow its own tools.
|
||||
|
||||
## The architectural shape of the answer
|
||||
|
||||
The downstream substrate is correct and stays:
|
||||
|
||||
```
|
||||
... → MathProblemGraph → BoundUnknown (ADR-0135) → Admissibility
|
||||
(ADR-0132/0133) via question_target.py (ADR-0134)
|
||||
→ Solver (ADR-0116)
|
||||
→ Verifier (ADR-0117)
|
||||
→ Realizer (ADR-0118)
|
||||
```
|
||||
|
||||
The binding-graph layer already operates on typed structure rather than
|
||||
surface words. It infers `question_form` (count / total / rate /
|
||||
difference / ratio / identity) from the operations touching the unknown.
|
||||
That's the correct level. It just doesn't get fed enough graphs because
|
||||
the front-end refuses too often.
|
||||
|
||||
The front-end is replaced. The new shape:
|
||||
|
||||
```
|
||||
Text
|
||||
→ Lexical Primitives (regex, lexeme-scope only — ADR-0165)
|
||||
→ Lexicon Lookup (word → semantic category, ADR-0164)
|
||||
→ Incremental Reader (word-by-word state accumulation)
|
||||
→ MathProblemGraph (same downstream type as before)
|
||||
→ [unchanged downstream]
|
||||
```
|
||||
|
||||
The reader is a deterministic shift-reduce parser **over semantic
|
||||
categories**, not over surface tokens. The category set is closed
|
||||
(~20 items), the composition rules are bounded (30–50). Adding a verb
|
||||
adds a lexicon lookup, not a new code path. The vocabulary already
|
||||
collected in `math_candidate_parser.py` (`_MASS_NOUNS`,
|
||||
`_PATTERN_A_VERBS`, `_PATTERN_B_VERBS`, `_PATTERN_C_VERBS`, name lists,
|
||||
add/subtract/transfer verb sets) ports wholesale as the lexicon seed —
|
||||
the vocabulary work is salvaged; only the regex template wrappers are
|
||||
removed.
|
||||
|
||||
## Why this preserves wrong = 0
|
||||
|
||||
The reader can be more permissive about *which sentences it
|
||||
comprehends* without being more permissive about *what comprehension
|
||||
produces*. The output type is identical to what the regex parser
|
||||
produces today, so the existing admissibility gate (unit proofs,
|
||||
multi-branch disagreement refusal, replay-equivalence) stays in force.
|
||||
A malformed comprehension produces a graph that admissibility rejects.
|
||||
wrong = 0 is preserved by construction.
|
||||
|
||||
## The corridor generalizes
|
||||
|
||||
The teaching → contemplation → review corridor (ADR-0150 / 0152 / 0155 /
|
||||
0161) already exists for vocabulary. Under ADR-0164 it expands in scope
|
||||
to also ratify:
|
||||
|
||||
- **Lexicon entries** (word → category mappings)
|
||||
- **Composition rules** (rare — bounded set, ADR-tracked)
|
||||
- **Lexeme primitives** (new regex tools the engine can wield)
|
||||
|
||||
Three orthogonal kinds of evidence, three orthogonal review predicates,
|
||||
one shared corridor. The engine becomes able to acquire new recognition
|
||||
capability through reviewed experience instead of through operator
|
||||
edits to parser code.
|
||||
|
||||
The operator's reaction at the moment this clicked into place:
|
||||
|
||||
> "That's the absolute fundamental key to intelligence. Truly. That's
|
||||
> what I had been hoping we could figure out."
|
||||
|
||||
## Deprecation discipline
|
||||
|
||||
ADR-0163's *diagnosis* (the front-end is the bottleneck) is reaffirmed.
|
||||
ADR-0163's *prescription* (Phases B–E producing regex-based
|
||||
`DerivedRecognizer` records in `generate/recognizer_match.py`) is
|
||||
superseded — what flows through the corridor changes, the corridor
|
||||
itself does not.
|
||||
|
||||
ADR-0136 and its S-family (S.1 / S.2 / S.3 / S.4 / post-rescan
|
||||
variants): regex sentence-template prescription superseded. Empirical
|
||||
refusal taxonomies preserved. Closed-set vocabulary tables preserved as
|
||||
lexicon seed.
|
||||
|
||||
All ratified work survives in some form. The regex *wrappers* go;
|
||||
everything else carries forward.
|
||||
|
||||
## Phasing committed in ADR-0164
|
||||
|
||||
1. **Phase 1 — Question reader.** Build the reader for question
|
||||
sentences only. Coexist with existing regex; reader runs first, regex
|
||||
is fallback. Target: `correct ≥ 10` on train_sample/v1, satisfying
|
||||
ADR-0163 Round-1 exit. Reader covers ≥20/34 currently-refused
|
||||
question cases.
|
||||
2. **Phase 2 — Statement reader.** Extend to statements. Target:
|
||||
`correct ≥ 25`.
|
||||
3. **Phase 3 — Regex layer removal.** `math_candidate_parser.py` no
|
||||
longer contains sentence-level regex patterns. Target: `correct ≥ 35`.
|
||||
4. **Phase 4 — Scale.** Per ADR-0163 §Phase F: public, holdout, full
|
||||
GSM8K.
|
||||
|
||||
## What the session did not decide
|
||||
|
||||
- Specific category set and composition-rule closure beyond a sketch.
|
||||
These will be sub-ADRs once Phase 1 measurement reveals real
|
||||
collisions.
|
||||
- Cross-sentence reading state (pronoun resolution across the problem
|
||||
body). Scoped in Phase 1 design.
|
||||
- The lexicon pack's exact frontmatter and merging policy with
|
||||
existing packs (`en_core_cognition_v1`, `en_core_relations_v1`).
|
||||
- Whether the existing `recognizer_registry` / `recognizer_match`
|
||||
modules become the new primitive registry or are replaced with
|
||||
fresh modules under `generate/comprehension/`.
|
||||
|
||||
## Closing observation
|
||||
|
||||
The hard part of the session was not the new architecture — it was
|
||||
recognizing that ADR-0163's prescription, which had landed only days
|
||||
earlier and was actively being extended (PRs #305, #306, #307, #308,
|
||||
#309, #310, #315), was wrong in its mechanism even though right in its
|
||||
diagnosis. The mechanism was *institutionalizing* the regex template
|
||||
approach by routing it through the corridor.
|
||||
|
||||
The operator had been holding the right intuition the whole time:
|
||||
"sentences come in all shapes and forms." That intuition is now an
|
||||
ADR, an invariant boundary, and an architectural transition plan with
|
||||
acceptance gates.
|
||||
Loading…
Reference in a new issue