From 3b8f441ae0773263f221b249615ce1602de63719 Mon Sep 17 00:00:00 2001 From: Shay Date: Tue, 26 May 2026 19:27:30 -0700 Subject: [PATCH] docs(ADR-0164.1): lexical primitive set scope (#318) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes ADR-0164 §Open question #1. Enumerates the 8-primitive seed registry for en_core_math_v1 (decimal-currency, currency, percentage, fraction, time-amount, numeric, ordinal, mass-noun-token), fixes the record schema (name/pattern/emits/extracted_fields/provenance/priority), documents pairwise overlap precedence with rationale, and records 4 rejected temptations (rate phrases, compound entities, question stems, compound numerics) so the ADR-0165 grammar/lexeme boundary doesn't get relitigated by future authors. --- .../ADR-0164.1-lexical-primitive-scope.md | 413 ++++++++++++++++++ 1 file changed, 413 insertions(+) create mode 100644 docs/decisions/ADR-0164.1-lexical-primitive-scope.md diff --git a/docs/decisions/ADR-0164.1-lexical-primitive-scope.md b/docs/decisions/ADR-0164.1-lexical-primitive-scope.md new file mode 100644 index 00000000..0d76d272 --- /dev/null +++ b/docs/decisions/ADR-0164.1-lexical-primitive-scope.md @@ -0,0 +1,413 @@ +# ADR-0164.1 — Lexical Primitive Set Scope (seed registry for `en_core_math_v1`) + +**Status:** Proposed +**Date:** 2026-05-26 +**Author:** Shay +**Anchor:** [[thesis-decoding-not-generating]] +**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md) +**Companion:** [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md) +**Resolves:** ADR-0164 §Open question #1 ("Lexical primitive set scope") + +--- + +## Context — why this sub-ADR exists + +ADR-0164 specifies the reader and leaves the *exact bootstrap primitive set* +open (§Open question #1). ADR-0165 specifies the *rule* that bounds what may +become a primitive (lexeme-level, closed orthographic shape) but does not +enumerate the set. + +This ADR closes the gap. It enumerates the seed primitive registry that +ships with the ADR-0164 Phase 1 PR, fixes the registry record schema, +documents overlap precedence (the only place primitives can interact at +recognition time), and records the temptations that are explicitly **not** +admitted, so future authors don't relitigate them. + +The reader's Phase 1 acceptance gate (see ADR-0164 §Phasing) depends on +this set being closed before scan-time. Adding a primitive after Phase 1 +follows the population corridor in ADR-0165 §Population (contemplation → +proposal → review). + +--- + +## Decision + +The seed lexeme-primitive registry for `en_core_math_v1` contains the +**eight** primitives below. Each is a closed orthographic shape per the +ADR-0165 three-question test (§Code-review test). Each carries the schema +fields in §Registry record schema. Overlap precedence between primitives +is fixed by §Overlap precedence and is the only behavior that may differ +between "primitive A fires" and "primitive A fires given primitive B +already matched the same span." + +Population beyond this seed set rides the ADR-0165 corridor and is +out of scope for this ADR. + +--- + +## Registry record schema + +Each primitive is a frozen record with the following fields. Field order +is canonical (used by the canonical-bytes serialization that feeds +`trace_hash` per CLAUDE.md §Runtime Surface Contract). + +| Field | Type | Meaning | +|---|---|---| +| `name` | kebab-case string, unique | Registry key. Stable across rounds. Forms the trace-evidence label when the primitive fires. | +| `pattern` | regex source string (Python `re` flavor) | The orthographic recognizer. Must be anchorable to a single token or contiguous token-class run (ADR-0165 §Rule). No `.*` across word combinations. No `\s+VERB\s+` constructions. | +| `emits` | enum: `QUANTITY`, `ORDINAL`, `UNIT_CATEGORY_TOKEN` | The reader category emitted on match. The category is what the reader's composition rules consume; the surface form is discarded after extraction. | +| `extracted_fields` | typed tuple `(name: type, ...)` | The structured payload produced. Types are `int`, `Decimal`, `str` from a closed enum, or `tuple[int, int]` for compound shapes (e.g. fractions). Reader composition is allowed to read these fields and only these fields. | +| `provenance` | string `"ADR-"` or `"teaching-ratified #"` | Audit trail. Seed primitives in this ADR are stamped `"ADR-0164.1"`. New primitives stamped by the HITL queue at acceptance time (ADR-0161). | +| `priority` | integer ≥ 0 (lower wins) | Tiebreaker when two primitives match the same span. The overlap-precedence table (§Overlap precedence) fixes seed priorities; new primitives declare priority at proposal time and the operator ratifies it. | + +The registry is a tuple of records, ordered by `priority` ascending then +`name` ascending (stable ordering for replay equivalence). + +--- + +## Seed primitive set (n = 8) + +Each entry below is a populated record. Patterns are shown as Python `re` +source; the runtime compiles them with `re.IGNORECASE` unless otherwise +noted, anchored to a single token / contiguous span (the reader is +responsible for token-boundary alignment; the regex itself does not +consume surrounding whitespace). + +### 1. `decimal-currency-literal` + +```yaml +name: decimal-currency-literal +pattern: \$(\d+)\.(\d{2})\b +emits: QUANTITY +extracted_fields: (whole: int, cents: int, unit_class: str = "currency") +provenance: ADR-0164.1 +priority: 10 +``` + +Rationale: `$18.00`, `$1.50`. The two-decimal-place currency form is its +own shape because (a) the cents field is structurally significant +(rounding semantics), (b) it must beat both `currency-literal` and +`numeric-literal` on the same span. Closed orthographic class: +"dollar-sign, integer, dot, exactly two digits." + +### 2. `currency-literal` + +```yaml +name: currency-literal +pattern: \$(\d+(?:\.\d+)?)\b +emits: QUANTITY +extracted_fields: (value: Decimal, unit_class: str = "currency") +provenance: ADR-0164.1 +priority: 20 +``` + +Rationale: `$18`, `$1.5`, `$1000`. Currency notation with arbitrary +(non-cents-shaped) decimal. Beats `numeric-literal` because the `$` +prefix carries the unit. Does *not* cover `$1.5M`, `$18/hour`, +`USD 18` — see §Rejected temptations. + +### 3. `percentage-literal` + +```yaml +name: percentage-literal +pattern: (\d+(?:\.\d+)?)\s?% +emits: QUANTITY +extracted_fields: (value: Decimal, unit_class: str = "ratio") +provenance: ADR-0164.1 +priority: 30 +``` + +Rationale: `25%`, `7.5 %`. The `%` glyph is a closed unit suffix. +Single optional space allowed because percent signs frequently appear +detached (`7 %`). The space inside the pattern is a single character, +not a `\s+` across structure — the run is still one orthographic shape. + +### 4. `fraction-literal` + +```yaml +name: fraction-literal +pattern: (\d+)\s?/\s?(\d+)\b +emits: QUANTITY +extracted_fields: (numerator: int, denominator: int, unit_class: str = "fraction") +provenance: ADR-0164.1 +priority: 40 +``` + +Rationale: `1/2`, `3 / 4`. The slash-separated integer pair is a +genuine closed shape. Optional single space on either side of the +slash. Reader composes "1/2 of " through composition rules, not +through extending this pattern. + +### 5. `time-amount-literal` + +```yaml +name: time-amount-literal +pattern: (\d+)[-\s]?(hour|minute|day|week|month|year|second)s?\b +emits: QUANTITY +extracted_fields: (value: int, unit: str, unit_class: str = "time") +provenance: ADR-0164.1 +priority: 50 +``` + +Rationale: `3 hours`, `30-minute`, `2 days`, `1 week`. The +"number-plus-time-unit-noun" form is the canonical closed time-amount +shape. Unit set is a closed enum; the reader treats `hour`/`hours`/ +`hour-` uniformly via the singular-form extraction. Does *not* cover +`an hour` (article + bare unit, that's grammar — see §Rejected +temptations), and does *not* cover `3 hours per week` (rate phrase, +grammar). + +### 6. `numeric-literal` + +```yaml +name: numeric-literal +pattern: \d+(?:\.\d+)?\b +emits: QUANTITY +extracted_fields: (value: Decimal, unit_class: str = "pending") +provenance: ADR-0164.1 +priority: 100 +``` + +Rationale: `18`, `1.5`, `12`. The bare number. `unit_class=pending` +signals to the reader that a unit attachment is expected from a +downstream token (a `count_unit_noun`, `currency_unit_noun`, +`time_unit_noun`, etc., from the operational lexicon — ADR-0164 +§Operational lexicon). Highest numeric `priority` ensures all other +numeric-bearing primitives win the overlap. + +### 7. `ordinal-literal` + +```yaml +name: ordinal-literal +pattern: (first|second|third|fourth|fifth|sixth|seventh|eighth|ninth|tenth)\b +emits: ORDINAL +extracted_fields: (rank: int) +provenance: ADR-0164.1 +priority: 60 +``` + +Rationale: `first`, `second`, `third`. Closed list of English ordinal +spellings 1–10. Extending the set to `eleventh`–`twentieth` or to +numeric ordinals (`1st`, `2nd`) is a teaching-corridor decision, not a +silent extension of this ADR. Extracted `rank` is the integer the +ordinal denotes. + +### 8. `mass-noun-token` + +```yaml +name: mass-noun-token +pattern: (money|profit|interest|income|savings|cost|amount|total)\b +emits: UNIT_CATEGORY_TOKEN +extracted_fields: (lemma: str, unit_class: str = "currency-mass") +provenance: ADR-0164.1 +priority: 70 +``` + +Rationale: ports `_MASS_NOUNS` from `generate/math_candidate_parser.py` +into a primitive form. The set is closed (8 lemmas) and orthographic; +each lemma is a single token recognized by spelling. The reader +composes "how much " via composition rules over +`question_open` + `question_continuous_qty` + `UNIT_CATEGORY_TOKEN`, +not via extending this pattern across the question stem. + +> Boundary note: this primitive is at the edge of the rule. A +> `mass-noun-token` is a closed set of single tokens, which satisfies +> ADR-0165's "contiguous token-class run." It is included as a +> primitive (rather than as an operational-lexicon entry under +> ADR-0164) because the reader's question-frame composition needs a +> structured category at scan time, not a lexicon hit during the +> composition pass. If Phase 1 measurement shows this is better +> modeled as a lexicon category, supersede this entry. + +--- + +## Overlap precedence + +A token span can be matched by more than one primitive. The reader runs +primitives in `priority` order (lower first), commits the first match +that consumes a non-empty span, and skips the consumed span for +subsequent primitives. The pairwise table below is the *normative* +record of every overlap relevant to the seed set. Each row is a +deliberate decision; review of a new primitive must extend this table. + +| Span shape | Candidate primitives | Winner | Rationale | +|---|---|---|---| +| `$18.00` | `decimal-currency-literal`, `currency-literal`, `numeric-literal` | `decimal-currency-literal` | Two-decimal cents form is structurally distinct (rounding). Currency-literal would lose the cents semantics; numeric-literal would lose the `$`. Lowest priority (10) wins. | +| `$18` | `currency-literal`, `numeric-literal` | `currency-literal` | The `$` prefix carries the unit class. Numeric-literal alone would emit `unit_class=pending` and force the reader to recover the currency unit from context, which is exactly the kind of grammar inference the comprehension layer is meant to do *with* typed evidence, not despite it. | +| `25%` | `percentage-literal`, `numeric-literal` | `percentage-literal` | The `%` suffix carries the unit class (ratio). Same rationale as currency. | +| `3 hours` | `time-amount-literal`, `numeric-literal` (matching `3`) | `time-amount-literal` | The time-unit noun is part of the closed shape; allowing `numeric-literal` to consume `3` first would orphan `hours` and force the reader to compose a time quantity through the lexicon — bypassing a typed primitive that already exists. The span-commit rule (longest valid match at lowest priority among ties) handles this: time-amount's pattern consumes the full `3 hours` span before numeric-literal's anchor matches. | +| `1/2` | `fraction-literal`, `numeric-literal` (matching `1`, then `2`) | `fraction-literal` | The slash makes the two integers a structured pair, not two independent quantities. | +| `first` | `ordinal-literal`, (nothing else) | `ordinal-literal` | Listed for completeness; no overlap, but the priority is set above `numeric-literal` so future `ordinal-numeric-literal` (`1st`, `2nd`) — if and when ratified — fits cleanly into the same slot. | +| `money` | `mass-noun-token`, (operational-lexicon `currency_unit_noun` per ADR-0164) | `mass-noun-token` | Lexical-primitive scan runs before lexicon lookup (ADR-0164 §Deterministic reader, step 1 → step 2). The primitive emits a typed `UNIT_CATEGORY_TOKEN` carrying `unit_class=currency-mass`, which the lexicon entry would also produce — but the primitive's emission is what the question-frame composition expects. | + +The precedence is **fixed**; a primitive proposal that would change an +existing row in this table requires explicit supersession of this ADR, +not a silent priority bump. + +--- + +## Rejected temptations + +The patterns below are *not* primitives and must not be added as +primitives. Each is a grammar template per ADR-0165 §Code-review test +and belongs in the reader's composition rules over the operational +lexicon. + +### Rejected #1 — Rate phrase `$18/hour`, `$18 per hour` + +A naive primitive might be `\$(\d+(?:\.\d+)?)\s?/\s?(hour|day|week)`. + +- **What does it match?** A composition of three role-distinct + elements: a currency amount, a connector (`/` or the word `per`), + and a time unit. Not "one piece of orthographic material" but "a + way three pieces of material combine to mean *rate*." +- **Closed-set test:** No. The connector class is open (`/`, `per`, + `an`, `each`, `every`, `for each`, …). The denominator class is + open (any unit noun, not just time). Recognizing rate via regex + enumerates surface forms. +- **Novel-phrasing test:** Refuses `$18 every hour`, `$18 for each + hour worked`, `eighteen dollars an hour`. The refusal is brittle + on the same underlying meaning — the diagnostic of a grammar + template. + +Correct home: reader composition rule `[QUANTITY{currency}] + +[distributive_modifier|"/"|"per"] + [time_unit_noun]` → `RATE` +operation. Currency and time-amount primitives feed the composition; +the rate semantics emerge there. + +### Rejected #2 — Compound entity `her three friends`, `Tina and Marion` + +A naive primitive might be +`(she|he|her|his|their)\s+(\d+|two|three|four|five|several|a few)\s+(friends|sisters|brothers|cousins)`. + +- **What does it match?** A possessive determiner, a count, and a + relational noun — three role-distinct elements with an open + combinatorial product. Pure grammar. +- **Closed-set test:** No. Possessive determiners are a small but + context-dependent set; counts include numeric literals *and* + spelled-out numerals *and* "a few" / "several" / "many" hedges; + relational nouns are open-ended (`friends`, `co-workers`, + `neighbors`, `kids`, `students`, …). +- **Novel-phrasing test:** Refuses `the three friends she invited`, + `Tina, Marion, and Jen`, `each of her friends`. Every novel + reference shape refuses on the same underlying meaning — group + entity binding. + +Correct home: cross-sentence `ProblemReadingState` (ADR-0164 §Open +question #4) plus reader composition rules over `entity_pronoun`, +`numeric-literal`, and a `relational_noun` lexicon category. Group +binding is the reader's job, not the primitive layer's. + +### Rejected #3 — Question stem `How much money will she earn` + +A naive primitive might be `How\s+much\s+(money|profit|...)\s+(will +|did|does)\s+(she|he|they|it)`. + +- **What does it match?** A five-role grammar template: question + opener + continuous-quantity word + mass-noun + auxiliary + + pronoun. This is the canonical example of what ADR-0164 deprecates + and ADR-0165 §Forbidden uses cites verbatim. +- **Closed-set test:** No on every role except `much` and `she/he/ + they/it`. The mass-noun set is reviewable but the rest of the + composition opens onto open lexical classes (auxiliaries, modal + verbs, perfect/progressive constructions, embedded clauses). +- **Novel-phrasing test:** Refuses `How much will she have earned + by Friday`, `How much money does Tina end up with`, `How much + did it cost him in total`. These are the exact 34/47 refusals + ADR-0164 §Context documents. + +Correct home: the reader's question-frame composition rules, +consuming primitive emissions (`UNIT_CATEGORY_TOKEN` from +`mass-noun-token`) plus operational-lexicon categories +(`question_open`, `question_continuous_qty`, `entity_pronoun`, +`accumulation_verb` from ADR-0164 §Operational lexicon). + +### Rejected #4 — Compound numeric `1,000`, `1.5M`, `1.5 million` + +A naive primitive might be `\d{1,3}(,\d{3})+(\.\d+)?` plus a +sibling `\d+(\.\d+)?[KMB]\b`. + +- **What does it match?** A composition between digit groups and + scaling tokens. Comma grouping is one shape; magnitude suffixes + are another; spelled-out scale words are a third. +- **Closed-set test:** Borderline. The comma-grouped form alone + (`1,000`, `12,345,678`) is a closed orthographic shape and *could* + be ratified as a primitive (`grouped-numeric-literal`) through the + ADR-0165 corridor when GSM8K evidence demands it. The `1.5M` and + `1.5 million` forms are compositions and must not be folded into + the same primitive. +- **Why rejected from the seed:** No GSM8K `train_sample/v1` case + observed in the ADR-0164 §Context evidence requires it. Adding it + speculatively violates ADR-0114a (no surface-form additions without + evidence) and the cleanup-as-you-find discipline (don't add what + you don't need). + +Correct home: defer to teaching-corridor ratification once a refusal +on a comma-grouped numeric is observed. The `M`/`million` scaling +case is composition (numeric + magnitude lexicon entry), not a +primitive. + +--- + +## Consequences + +### Positive + +1. **Phase 1 acceptance has a concrete scan-time vocabulary.** Eight + primitives, fully specified, ship with the Phase 1 PR. The reader's + step-1 (lexical primitive scan) is now implementable. +2. **Overlap behavior is recorded, not discovered.** Every overlap a + reviewer might worry about (`$18.00`, `3 hours`, `1/2`, `25%`) has + an explicit winner with rationale. +3. **The "where do I draw the line?" question is settled with + examples.** Future authors who want to add a primitive for a rate + phrase, compound entity, or question stem have §Rejected + temptations as the rejection precedent — no relitigation. + +### Negative / tradeoffs + +1. **Seed set is small.** Eight primitives won't cover every shape + GSM8K throws. That's the point — population is the corridor's job + (ADR-0165 §Population). Reader refusals on unknown token shapes + are evidence that flows back into the queue. +2. **`mass-noun-token` is at the rule boundary.** Listed honestly in + §Seed primitive set with an explicit supersede-if path. Phase 1 + measurement will settle whether it belongs here or in the + operational lexicon. +3. **Overlap precedence table will grow.** Every new primitive must + extend the table. This is the cost of fixed precedence, and it's + cheaper than the alternative (silent runtime tiebreaking). + +--- + +## Acceptance criteria for this sub-ADR + +This ADR moves to **Accepted** when: + +1. The seed registry above is materialized in + `language_packs/data/en_core_math_v1/lexical_primitives.json` (or + the equivalent loader format settled in the Phase 1 PR), one + record per entry, fields populated verbatim. +2. The manifest checksum hashes the bytes written to disk (CLAUDE.md + §Semantic Pack Discipline). +3. The overlap-precedence table has a pinned regression test — + given the seed registry, the eight overlap rows resolve to the + declared winner on synthetic minimum-pair inputs. +4. ADR-0164 §Open question #1 is checked off in that ADR's + open-questions list when ADR-0164 next ships an update. + +--- + +## Cross-references + +- **Parent:** ADR-0164 — Incremental Comprehension Reader +- **Companion (the rule):** ADR-0165 — Regex Scope Rule +- **Anti-overfitting doctrine:** ADR-0114a +- **Pack discipline:** CLAUDE.md §Semantic Pack Discipline +- **Population corridor:** ADR-0150 (contemplation), ADR-0152 + (learning-arc proof), ADR-0155 (CI contemplation runner), ADR-0161 + (HITL async queue) +- **Anchor:** `[[thesis-decoding-not-generating]]` — the primitive + set is a decoder's recognizer table. It enumerates the closed + orthographic shapes the reader can pick up *as such*; it does not + enumerate sentences.