docs(ADR-0164.1): lexical primitive set scope (#318)
Closes ADR-0164 §Open question #1. Enumerates the 8-primitive seed registry for en_core_math_v1 (decimal-currency, currency, percentage, fraction, time-amount, numeric, ordinal, mass-noun-token), fixes the record schema (name/pattern/emits/extracted_fields/provenance/priority), documents pairwise overlap precedence with rationale, and records 4 rejected temptations (rate phrases, compound entities, question stems, compound numerics) so the ADR-0165 grammar/lexeme boundary doesn't get relitigated by future authors.
This commit is contained in:
Shay
GitHub
parent
20f3a5d586
commit
3b8f441ae0
1 changed files with 413 additions and 0 deletions
413
docs/decisions/ADR-0164.1-lexical-primitive-scope.md
Normal file
413
docs/decisions/ADR-0164.1-lexical-primitive-scope.md
Normal file
|
|
@ -0,0 +1,413 @@
|
|||
# ADR-0164.1 — Lexical Primitive Set Scope (seed registry for `en_core_math_v1`)
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-26
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md)
|
||||
**Companion:** [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
|
||||
**Resolves:** ADR-0164 §Open question #1 ("Lexical primitive set scope")
|
||||
|
||||
---
|
||||
|
||||
## Context — why this sub-ADR exists
|
||||
|
||||
ADR-0164 specifies the reader and leaves the *exact bootstrap primitive set*
|
||||
open (§Open question #1). ADR-0165 specifies the *rule* that bounds what may
|
||||
become a primitive (lexeme-level, closed orthographic shape) but does not
|
||||
enumerate the set.
|
||||
|
||||
This ADR closes the gap. It enumerates the seed primitive registry that
|
||||
ships with the ADR-0164 Phase 1 PR, fixes the registry record schema,
|
||||
documents overlap precedence (the only place primitives can interact at
|
||||
recognition time), and records the temptations that are explicitly **not**
|
||||
admitted, so future authors don't relitigate them.
|
||||
|
||||
The reader's Phase 1 acceptance gate (see ADR-0164 §Phasing) depends on
|
||||
this set being closed before scan-time. Adding a primitive after Phase 1
|
||||
follows the population corridor in ADR-0165 §Population (contemplation →
|
||||
proposal → review).
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
The seed lexeme-primitive registry for `en_core_math_v1` contains the
|
||||
**eight** primitives below. Each is a closed orthographic shape per the
|
||||
ADR-0165 three-question test (§Code-review test). Each carries the schema
|
||||
fields in §Registry record schema. Overlap precedence between primitives
|
||||
is fixed by §Overlap precedence and is the only behavior that may differ
|
||||
between "primitive A fires" and "primitive A fires given primitive B
|
||||
already matched the same span."
|
||||
|
||||
Population beyond this seed set rides the ADR-0165 corridor and is
|
||||
out of scope for this ADR.
|
||||
|
||||
---
|
||||
|
||||
## Registry record schema
|
||||
|
||||
Each primitive is a frozen record with the following fields. Field order
|
||||
is canonical (used by the canonical-bytes serialization that feeds
|
||||
`trace_hash` per CLAUDE.md §Runtime Surface Contract).
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `name` | kebab-case string, unique | Registry key. Stable across rounds. Forms the trace-evidence label when the primitive fires. |
|
||||
| `pattern` | regex source string (Python `re` flavor) | The orthographic recognizer. Must be anchorable to a single token or contiguous token-class run (ADR-0165 §Rule). No `.*` across word combinations. No `\s+VERB\s+` constructions. |
|
||||
| `emits` | enum: `QUANTITY`, `ORDINAL`, `UNIT_CATEGORY_TOKEN` | The reader category emitted on match. The category is what the reader's composition rules consume; the surface form is discarded after extraction. |
|
||||
| `extracted_fields` | typed tuple `(name: type, ...)` | The structured payload produced. Types are `int`, `Decimal`, `str` from a closed enum, or `tuple[int, int]` for compound shapes (e.g. fractions). Reader composition is allowed to read these fields and only these fields. |
|
||||
| `provenance` | string `"ADR-<id>"` or `"teaching-ratified <ISO date> #<queue-id>"` | Audit trail. Seed primitives in this ADR are stamped `"ADR-0164.1"`. New primitives stamped by the HITL queue at acceptance time (ADR-0161). |
|
||||
| `priority` | integer ≥ 0 (lower wins) | Tiebreaker when two primitives match the same span. The overlap-precedence table (§Overlap precedence) fixes seed priorities; new primitives declare priority at proposal time and the operator ratifies it. |
|
||||
|
||||
The registry is a tuple of records, ordered by `priority` ascending then
|
||||
`name` ascending (stable ordering for replay equivalence).
|
||||
|
||||
---
|
||||
|
||||
## Seed primitive set (n = 8)
|
||||
|
||||
Each entry below is a populated record. Patterns are shown as Python `re`
|
||||
source; the runtime compiles them with `re.IGNORECASE` unless otherwise
|
||||
noted, anchored to a single token / contiguous span (the reader is
|
||||
responsible for token-boundary alignment; the regex itself does not
|
||||
consume surrounding whitespace).
|
||||
|
||||
### 1. `decimal-currency-literal`
|
||||
|
||||
```yaml
|
||||
name: decimal-currency-literal
|
||||
pattern: \$(\d+)\.(\d{2})\b
|
||||
emits: QUANTITY
|
||||
extracted_fields: (whole: int, cents: int, unit_class: str = "currency")
|
||||
provenance: ADR-0164.1
|
||||
priority: 10
|
||||
```
|
||||
|
||||
Rationale: `$18.00`, `$1.50`. The two-decimal-place currency form is its
|
||||
own shape because (a) the cents field is structurally significant
|
||||
(rounding semantics), (b) it must beat both `currency-literal` and
|
||||
`numeric-literal` on the same span. Closed orthographic class:
|
||||
"dollar-sign, integer, dot, exactly two digits."
|
||||
|
||||
### 2. `currency-literal`
|
||||
|
||||
```yaml
|
||||
name: currency-literal
|
||||
pattern: \$(\d+(?:\.\d+)?)\b
|
||||
emits: QUANTITY
|
||||
extracted_fields: (value: Decimal, unit_class: str = "currency")
|
||||
provenance: ADR-0164.1
|
||||
priority: 20
|
||||
```
|
||||
|
||||
Rationale: `$18`, `$1.5`, `$1000`. Currency notation with arbitrary
|
||||
(non-cents-shaped) decimal. Beats `numeric-literal` because the `$`
|
||||
prefix carries the unit. Does *not* cover `$1.5M`, `$18/hour`,
|
||||
`USD 18` — see §Rejected temptations.
|
||||
|
||||
### 3. `percentage-literal`
|
||||
|
||||
```yaml
|
||||
name: percentage-literal
|
||||
pattern: (\d+(?:\.\d+)?)\s?%
|
||||
emits: QUANTITY
|
||||
extracted_fields: (value: Decimal, unit_class: str = "ratio")
|
||||
provenance: ADR-0164.1
|
||||
priority: 30
|
||||
```
|
||||
|
||||
Rationale: `25%`, `7.5 %`. The `%` glyph is a closed unit suffix.
|
||||
Single optional space allowed because percent signs frequently appear
|
||||
detached (`7 %`). The space inside the pattern is a single character,
|
||||
not a `\s+` across structure — the run is still one orthographic shape.
|
||||
|
||||
### 4. `fraction-literal`
|
||||
|
||||
```yaml
|
||||
name: fraction-literal
|
||||
pattern: (\d+)\s?/\s?(\d+)\b
|
||||
emits: QUANTITY
|
||||
extracted_fields: (numerator: int, denominator: int, unit_class: str = "fraction")
|
||||
provenance: ADR-0164.1
|
||||
priority: 40
|
||||
```
|
||||
|
||||
Rationale: `1/2`, `3 / 4`. The slash-separated integer pair is a
|
||||
genuine closed shape. Optional single space on either side of the
|
||||
slash. Reader composes "1/2 of <X>" through composition rules, not
|
||||
through extending this pattern.
|
||||
|
||||
### 5. `time-amount-literal`
|
||||
|
||||
```yaml
|
||||
name: time-amount-literal
|
||||
pattern: (\d+)[-\s]?(hour|minute|day|week|month|year|second)s?\b
|
||||
emits: QUANTITY
|
||||
extracted_fields: (value: int, unit: str, unit_class: str = "time")
|
||||
provenance: ADR-0164.1
|
||||
priority: 50
|
||||
```
|
||||
|
||||
Rationale: `3 hours`, `30-minute`, `2 days`, `1 week`. The
|
||||
"number-plus-time-unit-noun" form is the canonical closed time-amount
|
||||
shape. Unit set is a closed enum; the reader treats `hour`/`hours`/
|
||||
`hour-` uniformly via the singular-form extraction. Does *not* cover
|
||||
`an hour` (article + bare unit, that's grammar — see §Rejected
|
||||
temptations), and does *not* cover `3 hours per week` (rate phrase,
|
||||
grammar).
|
||||
|
||||
### 6. `numeric-literal`
|
||||
|
||||
```yaml
|
||||
name: numeric-literal
|
||||
pattern: \d+(?:\.\d+)?\b
|
||||
emits: QUANTITY
|
||||
extracted_fields: (value: Decimal, unit_class: str = "pending")
|
||||
provenance: ADR-0164.1
|
||||
priority: 100
|
||||
```
|
||||
|
||||
Rationale: `18`, `1.5`, `12`. The bare number. `unit_class=pending`
|
||||
signals to the reader that a unit attachment is expected from a
|
||||
downstream token (a `count_unit_noun`, `currency_unit_noun`,
|
||||
`time_unit_noun`, etc., from the operational lexicon — ADR-0164
|
||||
§Operational lexicon). Highest numeric `priority` ensures all other
|
||||
numeric-bearing primitives win the overlap.
|
||||
|
||||
### 7. `ordinal-literal`
|
||||
|
||||
```yaml
|
||||
name: ordinal-literal
|
||||
pattern: (first|second|third|fourth|fifth|sixth|seventh|eighth|ninth|tenth)\b
|
||||
emits: ORDINAL
|
||||
extracted_fields: (rank: int)
|
||||
provenance: ADR-0164.1
|
||||
priority: 60
|
||||
```
|
||||
|
||||
Rationale: `first`, `second`, `third`. Closed list of English ordinal
|
||||
spellings 1–10. Extending the set to `eleventh`–`twentieth` or to
|
||||
numeric ordinals (`1st`, `2nd`) is a teaching-corridor decision, not a
|
||||
silent extension of this ADR. Extracted `rank` is the integer the
|
||||
ordinal denotes.
|
||||
|
||||
### 8. `mass-noun-token`
|
||||
|
||||
```yaml
|
||||
name: mass-noun-token
|
||||
pattern: (money|profit|interest|income|savings|cost|amount|total)\b
|
||||
emits: UNIT_CATEGORY_TOKEN
|
||||
extracted_fields: (lemma: str, unit_class: str = "currency-mass")
|
||||
provenance: ADR-0164.1
|
||||
priority: 70
|
||||
```
|
||||
|
||||
Rationale: ports `_MASS_NOUNS` from `generate/math_candidate_parser.py`
|
||||
into a primitive form. The set is closed (8 lemmas) and orthographic;
|
||||
each lemma is a single token recognized by spelling. The reader
|
||||
composes "how much <mass-noun>" via composition rules over
|
||||
`question_open` + `question_continuous_qty` + `UNIT_CATEGORY_TOKEN`,
|
||||
not via extending this pattern across the question stem.
|
||||
|
||||
> Boundary note: this primitive is at the edge of the rule. A
|
||||
> `mass-noun-token` is a closed set of single tokens, which satisfies
|
||||
> ADR-0165's "contiguous token-class run." It is included as a
|
||||
> primitive (rather than as an operational-lexicon entry under
|
||||
> ADR-0164) because the reader's question-frame composition needs a
|
||||
> structured category at scan time, not a lexicon hit during the
|
||||
> composition pass. If Phase 1 measurement shows this is better
|
||||
> modeled as a lexicon category, supersede this entry.
|
||||
|
||||
---
|
||||
|
||||
## Overlap precedence
|
||||
|
||||
A token span can be matched by more than one primitive. The reader runs
|
||||
primitives in `priority` order (lower first), commits the first match
|
||||
that consumes a non-empty span, and skips the consumed span for
|
||||
subsequent primitives. The pairwise table below is the *normative*
|
||||
record of every overlap relevant to the seed set. Each row is a
|
||||
deliberate decision; review of a new primitive must extend this table.
|
||||
|
||||
| Span shape | Candidate primitives | Winner | Rationale |
|
||||
|---|---|---|---|
|
||||
| `$18.00` | `decimal-currency-literal`, `currency-literal`, `numeric-literal` | `decimal-currency-literal` | Two-decimal cents form is structurally distinct (rounding). Currency-literal would lose the cents semantics; numeric-literal would lose the `$`. Lowest priority (10) wins. |
|
||||
| `$18` | `currency-literal`, `numeric-literal` | `currency-literal` | The `$` prefix carries the unit class. Numeric-literal alone would emit `unit_class=pending` and force the reader to recover the currency unit from context, which is exactly the kind of grammar inference the comprehension layer is meant to do *with* typed evidence, not despite it. |
|
||||
| `25%` | `percentage-literal`, `numeric-literal` | `percentage-literal` | The `%` suffix carries the unit class (ratio). Same rationale as currency. |
|
||||
| `3 hours` | `time-amount-literal`, `numeric-literal` (matching `3`) | `time-amount-literal` | The time-unit noun is part of the closed shape; allowing `numeric-literal` to consume `3` first would orphan `hours` and force the reader to compose a time quantity through the lexicon — bypassing a typed primitive that already exists. The span-commit rule (longest valid match at lowest priority among ties) handles this: time-amount's pattern consumes the full `3 hours` span before numeric-literal's anchor matches. |
|
||||
| `1/2` | `fraction-literal`, `numeric-literal` (matching `1`, then `2`) | `fraction-literal` | The slash makes the two integers a structured pair, not two independent quantities. |
|
||||
| `first` | `ordinal-literal`, (nothing else) | `ordinal-literal` | Listed for completeness; no overlap, but the priority is set above `numeric-literal` so future `ordinal-numeric-literal` (`1st`, `2nd`) — if and when ratified — fits cleanly into the same slot. |
|
||||
| `money` | `mass-noun-token`, (operational-lexicon `currency_unit_noun` per ADR-0164) | `mass-noun-token` | Lexical-primitive scan runs before lexicon lookup (ADR-0164 §Deterministic reader, step 1 → step 2). The primitive emits a typed `UNIT_CATEGORY_TOKEN` carrying `unit_class=currency-mass`, which the lexicon entry would also produce — but the primitive's emission is what the question-frame composition expects. |
|
||||
|
||||
The precedence is **fixed**; a primitive proposal that would change an
|
||||
existing row in this table requires explicit supersession of this ADR,
|
||||
not a silent priority bump.
|
||||
|
||||
---
|
||||
|
||||
## Rejected temptations
|
||||
|
||||
The patterns below are *not* primitives and must not be added as
|
||||
primitives. Each is a grammar template per ADR-0165 §Code-review test
|
||||
and belongs in the reader's composition rules over the operational
|
||||
lexicon.
|
||||
|
||||
### Rejected #1 — Rate phrase `$18/hour`, `$18 per hour`
|
||||
|
||||
A naive primitive might be `\$(\d+(?:\.\d+)?)\s?/\s?(hour|day|week)`.
|
||||
|
||||
- **What does it match?** A composition of three role-distinct
|
||||
elements: a currency amount, a connector (`/` or the word `per`),
|
||||
and a time unit. Not "one piece of orthographic material" but "a
|
||||
way three pieces of material combine to mean *rate*."
|
||||
- **Closed-set test:** No. The connector class is open (`/`, `per`,
|
||||
`an`, `each`, `every`, `for each`, …). The denominator class is
|
||||
open (any unit noun, not just time). Recognizing rate via regex
|
||||
enumerates surface forms.
|
||||
- **Novel-phrasing test:** Refuses `$18 every hour`, `$18 for each
|
||||
hour worked`, `eighteen dollars an hour`. The refusal is brittle
|
||||
on the same underlying meaning — the diagnostic of a grammar
|
||||
template.
|
||||
|
||||
Correct home: reader composition rule `[QUANTITY{currency}] +
|
||||
[distributive_modifier|"/"|"per"] + [time_unit_noun]` → `RATE`
|
||||
operation. Currency and time-amount primitives feed the composition;
|
||||
the rate semantics emerge there.
|
||||
|
||||
### Rejected #2 — Compound entity `her three friends`, `Tina and Marion`
|
||||
|
||||
A naive primitive might be
|
||||
`(she|he|her|his|their)\s+(\d+|two|three|four|five|several|a few)\s+(friends|sisters|brothers|cousins)`.
|
||||
|
||||
- **What does it match?** A possessive determiner, a count, and a
|
||||
relational noun — three role-distinct elements with an open
|
||||
combinatorial product. Pure grammar.
|
||||
- **Closed-set test:** No. Possessive determiners are a small but
|
||||
context-dependent set; counts include numeric literals *and*
|
||||
spelled-out numerals *and* "a few" / "several" / "many" hedges;
|
||||
relational nouns are open-ended (`friends`, `co-workers`,
|
||||
`neighbors`, `kids`, `students`, …).
|
||||
- **Novel-phrasing test:** Refuses `the three friends she invited`,
|
||||
`Tina, Marion, and Jen`, `each of her friends`. Every novel
|
||||
reference shape refuses on the same underlying meaning — group
|
||||
entity binding.
|
||||
|
||||
Correct home: cross-sentence `ProblemReadingState` (ADR-0164 §Open
|
||||
question #4) plus reader composition rules over `entity_pronoun`,
|
||||
`numeric-literal`, and a `relational_noun` lexicon category. Group
|
||||
binding is the reader's job, not the primitive layer's.
|
||||
|
||||
### Rejected #3 — Question stem `How much money will she earn`
|
||||
|
||||
A naive primitive might be `How\s+much\s+(money|profit|...)\s+(will
|
||||
|did|does)\s+(she|he|they|it)`.
|
||||
|
||||
- **What does it match?** A five-role grammar template: question
|
||||
opener + continuous-quantity word + mass-noun + auxiliary +
|
||||
pronoun. This is the canonical example of what ADR-0164 deprecates
|
||||
and ADR-0165 §Forbidden uses cites verbatim.
|
||||
- **Closed-set test:** No on every role except `much` and `she/he/
|
||||
they/it`. The mass-noun set is reviewable but the rest of the
|
||||
composition opens onto open lexical classes (auxiliaries, modal
|
||||
verbs, perfect/progressive constructions, embedded clauses).
|
||||
- **Novel-phrasing test:** Refuses `How much will she have earned
|
||||
by Friday`, `How much money does Tina end up with`, `How much
|
||||
did it cost him in total`. These are the exact 34/47 refusals
|
||||
ADR-0164 §Context documents.
|
||||
|
||||
Correct home: the reader's question-frame composition rules,
|
||||
consuming primitive emissions (`UNIT_CATEGORY_TOKEN` from
|
||||
`mass-noun-token`) plus operational-lexicon categories
|
||||
(`question_open`, `question_continuous_qty`, `entity_pronoun`,
|
||||
`accumulation_verb` from ADR-0164 §Operational lexicon).
|
||||
|
||||
### Rejected #4 — Compound numeric `1,000`, `1.5M`, `1.5 million`
|
||||
|
||||
A naive primitive might be `\d{1,3}(,\d{3})+(\.\d+)?` plus a
|
||||
sibling `\d+(\.\d+)?[KMB]\b`.
|
||||
|
||||
- **What does it match?** A composition between digit groups and
|
||||
scaling tokens. Comma grouping is one shape; magnitude suffixes
|
||||
are another; spelled-out scale words are a third.
|
||||
- **Closed-set test:** Borderline. The comma-grouped form alone
|
||||
(`1,000`, `12,345,678`) is a closed orthographic shape and *could*
|
||||
be ratified as a primitive (`grouped-numeric-literal`) through the
|
||||
ADR-0165 corridor when GSM8K evidence demands it. The `1.5M` and
|
||||
`1.5 million` forms are compositions and must not be folded into
|
||||
the same primitive.
|
||||
- **Why rejected from the seed:** No GSM8K `train_sample/v1` case
|
||||
observed in the ADR-0164 §Context evidence requires it. Adding it
|
||||
speculatively violates ADR-0114a (no surface-form additions without
|
||||
evidence) and the cleanup-as-you-find discipline (don't add what
|
||||
you don't need).
|
||||
|
||||
Correct home: defer to teaching-corridor ratification once a refusal
|
||||
on a comma-grouped numeric is observed. The `M`/`million` scaling
|
||||
case is composition (numeric + magnitude lexicon entry), not a
|
||||
primitive.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
1. **Phase 1 acceptance has a concrete scan-time vocabulary.** Eight
|
||||
primitives, fully specified, ship with the Phase 1 PR. The reader's
|
||||
step-1 (lexical primitive scan) is now implementable.
|
||||
2. **Overlap behavior is recorded, not discovered.** Every overlap a
|
||||
reviewer might worry about (`$18.00`, `3 hours`, `1/2`, `25%`) has
|
||||
an explicit winner with rationale.
|
||||
3. **The "where do I draw the line?" question is settled with
|
||||
examples.** Future authors who want to add a primitive for a rate
|
||||
phrase, compound entity, or question stem have §Rejected
|
||||
temptations as the rejection precedent — no relitigation.
|
||||
|
||||
### Negative / tradeoffs
|
||||
|
||||
1. **Seed set is small.** Eight primitives won't cover every shape
|
||||
GSM8K throws. That's the point — population is the corridor's job
|
||||
(ADR-0165 §Population). Reader refusals on unknown token shapes
|
||||
are evidence that flows back into the queue.
|
||||
2. **`mass-noun-token` is at the rule boundary.** Listed honestly in
|
||||
§Seed primitive set with an explicit supersede-if path. Phase 1
|
||||
measurement will settle whether it belongs here or in the
|
||||
operational lexicon.
|
||||
3. **Overlap precedence table will grow.** Every new primitive must
|
||||
extend the table. This is the cost of fixed precedence, and it's
|
||||
cheaper than the alternative (silent runtime tiebreaking).
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria for this sub-ADR
|
||||
|
||||
This ADR moves to **Accepted** when:
|
||||
|
||||
1. The seed registry above is materialized in
|
||||
`language_packs/data/en_core_math_v1/lexical_primitives.json` (or
|
||||
the equivalent loader format settled in the Phase 1 PR), one
|
||||
record per entry, fields populated verbatim.
|
||||
2. The manifest checksum hashes the bytes written to disk (CLAUDE.md
|
||||
§Semantic Pack Discipline).
|
||||
3. The overlap-precedence table has a pinned regression test —
|
||||
given the seed registry, the eight overlap rows resolve to the
|
||||
declared winner on synthetic minimum-pair inputs.
|
||||
4. ADR-0164 §Open question #1 is checked off in that ADR's
|
||||
open-questions list when ADR-0164 next ships an update.
|
||||
|
||||
---
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Parent:** ADR-0164 — Incremental Comprehension Reader
|
||||
- **Companion (the rule):** ADR-0165 — Regex Scope Rule
|
||||
- **Anti-overfitting doctrine:** ADR-0114a
|
||||
- **Pack discipline:** CLAUDE.md §Semantic Pack Discipline
|
||||
- **Population corridor:** ADR-0150 (contemplation), ADR-0152
|
||||
(learning-arc proof), ADR-0155 (CI contemplation runner), ADR-0161
|
||||
(HITL async queue)
|
||||
- **Anchor:** `[[thesis-decoding-not-generating]]` — the primitive
|
||||
set is a decoder's recognizer table. It enumerates the closed
|
||||
orthographic shapes the reader can pick up *as such*; it does not
|
||||
enumerate sentences.
|
||||
Loading…
Reference in a new issue