core/docs/decisions/ADR-0164.1-lexical-primitive-scope.md

# ADR-0164.1 — Lexical Primitive Set Scope (seed registry for `en_core_math_v1`)

**Status:** Proposed
**Date:** 2026-05-26
**Author:** Shay
**Anchor:** [[thesis-decoding-not-generating]]
**Parent:** [ADR-0164 — Incremental Comprehension Reader](./ADR-0164-incremental-comprehension-reader.md)
**Companion:** [ADR-0165 — Regex Scope Rule](./ADR-0165-regex-scope-rule.md)
**Resolves:** ADR-0164 §Open question #1 ("Lexical primitive set scope")

---

## Context — why this sub-ADR exists

ADR-0164 specifies the reader and leaves the *exact bootstrap primitive set*
open (§Open question #1). ADR-0165 specifies the *rule* that bounds what may
become a primitive (lexeme-level, closed orthographic shape) but does not
enumerate the set.

This ADR closes the gap. It enumerates the seed primitive registry that
ships with the ADR-0164 Phase 1 PR, fixes the registry record schema,
documents overlap precedence (the only place primitives can interact at
recognition time), and records the temptations that are explicitly **not**
admitted, so future authors don't relitigate them.

The reader's Phase 1 acceptance gate (see ADR-0164 §Phasing) depends on
this set being closed before scan-time. Adding a primitive after Phase 1
follows the population corridor in ADR-0165 §Population (contemplation →
proposal → review).

---

## Decision

The seed lexeme-primitive registry for `en_core_math_v1` contains the
**eight** primitives below. Each is a closed orthographic shape per the
ADR-0165 three-question test (§Code-review test). Each carries the schema
fields in §Registry record schema. Overlap precedence between primitives
is fixed by §Overlap precedence and is the only behavior that may differ
between "primitive A fires" and "primitive A fires given primitive B
already matched the same span."

Population beyond this seed set rides the ADR-0165 corridor and is
out of scope for this ADR.

---

## Registry record schema

Each primitive is a frozen record with the following fields. Field order
is canonical (used by the canonical-bytes serialization that feeds
`trace_hash` per CLAUDE.md §Runtime Surface Contract).

| Field | Type | Meaning |
|---|---|---|
| `name` | kebab-case string, unique | Registry key. Stable across rounds. Forms the trace-evidence label when the primitive fires. |
| `pattern` | regex source string (Python `re` flavor) | The orthographic recognizer. Must be anchorable to a single token or contiguous token-class run (ADR-0165 §Rule). No `.*` across word combinations. No `\s+VERB\s+` constructions. |
| `emits` | enum: `QUANTITY`, `ORDINAL`, `UNIT_CATEGORY_TOKEN` | The reader category emitted on match. The category is what the reader's composition rules consume; the surface form is discarded after extraction. |
| `extracted_fields` | typed tuple `(name: type, ...)` | The structured payload produced. Types are `int`, `Decimal`, `str` from a closed enum, or `tuple[int, int]` for compound shapes (e.g. fractions). Reader composition is allowed to read these fields and only these fields. |
| `provenance` | string `"ADR-<id>"` or `"teaching-ratified <ISO date> #<queue-id>"` | Audit trail. Seed primitives in this ADR are stamped `"ADR-0164.1"`. New primitives stamped by the HITL queue at acceptance time (ADR-0161). |
| `priority` | integer ≥ 0 (lower wins) | Tiebreaker when two primitives match the same span. The overlap-precedence table (§Overlap precedence) fixes seed priorities; new primitives declare priority at proposal time and the operator ratifies it. |

The registry is a tuple of records, ordered by `priority` ascending then
`name` ascending (stable ordering for replay equivalence).

---

## Seed primitive set (n = 8)

Each entry below is a populated record. Patterns are shown as Python `re`
source; the runtime compiles them with `re.IGNORECASE` unless otherwise
noted, anchored to a single token / contiguous span (the reader is
responsible for token-boundary alignment; the regex itself does not
consume surrounding whitespace).

### 1. `decimal-currency-literal`

```yaml
name: decimal-currency-literal
pattern: \$(\d+)\.(\d{2})\b
emits: QUANTITY
extracted_fields: (whole: int, cents: int, unit_class: str = "currency")
provenance: ADR-0164.1
priority: 10
```

Rationale: `$18.00`, `$1.50`. The two-decimal-place currency form is its
own shape because (a) the cents field is structurally significant
(rounding semantics), (b) it must beat both `currency-literal` and
`numeric-literal` on the same span. Closed orthographic class:
"dollar-sign, integer, dot, exactly two digits."

### 2. `currency-literal`

```yaml
name: currency-literal
pattern: \$(\d+(?:\.\d+)?)\b
emits: QUANTITY
extracted_fields: (value: Decimal, unit_class: str = "currency")
provenance: ADR-0164.1
priority: 20
```

Rationale: `$18`, `$1.5`, `$1000`. Currency notation with arbitrary
(non-cents-shaped) decimal. Beats `numeric-literal` because the `$`
prefix carries the unit. Does *not* cover `$1.5M`, `$18/hour`,
`USD 18` — see §Rejected temptations.

### 3. `percentage-literal`

```yaml
name: percentage-literal
pattern: (\d+(?:\.\d+)?)\s?%
emits: QUANTITY
extracted_fields: (value: Decimal, unit_class: str = "ratio")
provenance: ADR-0164.1
priority: 30
```

Rationale: `25%`, `7.5 %`. The `%` glyph is a closed unit suffix.
Single optional space allowed because percent signs frequently appear
detached (`7 %`). The space inside the pattern is a single character,
not a `\s+` across structure — the run is still one orthographic shape.

### 4. `fraction-literal`

```yaml
name: fraction-literal
pattern: (\d+)\s?/\s?(\d+)\b
emits: QUANTITY
extracted_fields: (numerator: int, denominator: int, unit_class: str = "fraction")
provenance: ADR-0164.1
priority: 40
```

Rationale: `1/2`, `3 / 4`. The slash-separated integer pair is a
genuine closed shape. Optional single space on either side of the
slash. Reader composes "1/2 of <X>" through composition rules, not
through extending this pattern.

### 5. `time-amount-literal`

```yaml
name: time-amount-literal
pattern: (\d+)[-\s]?(hour|minute|day|week|month|year|second)s?\b
emits: QUANTITY
extracted_fields: (value: int, unit: str, unit_class: str = "time")
provenance: ADR-0164.1
priority: 50
```

Rationale: `3 hours`, `30-minute`, `2 days`, `1 week`. The
"number-plus-time-unit-noun" form is the canonical closed time-amount
shape. Unit set is a closed enum; the reader treats `hour`/`hours`/
`hour-` uniformly via the singular-form extraction. Does *not* cover
`an hour` (article + bare unit, that's grammar — see §Rejected
temptations), and does *not* cover `3 hours per week` (rate phrase,
grammar).

### 6. `numeric-literal`

```yaml
name: numeric-literal
pattern: \d+(?:\.\d+)?\b
emits: QUANTITY
extracted_fields: (value: Decimal, unit_class: str = "pending")
provenance: ADR-0164.1
priority: 100
```

Rationale: `18`, `1.5`, `12`. The bare number. `unit_class=pending`
signals to the reader that a unit attachment is expected from a
downstream token (a `count_unit_noun`, `currency_unit_noun`,
`time_unit_noun`, etc., from the operational lexicon — ADR-0164
§Operational lexicon). Highest numeric `priority` ensures all other
numeric-bearing primitives win the overlap.

### 7. `ordinal-literal`

```yaml
name: ordinal-literal
pattern: (first|second|third|fourth|fifth|sixth|seventh|eighth|ninth|tenth)\b
emits: ORDINAL
extracted_fields: (rank: int)
provenance: ADR-0164.1
priority: 60
```

Rationale: `first`, `second`, `third`. Closed list of English ordinal
spellings 1–10. Extending the set to `eleventh`–`twentieth` or to
numeric ordinals (`1st`, `2nd`) is a teaching-corridor decision, not a
silent extension of this ADR. Extracted `rank` is the integer the
ordinal denotes.

### 8. `mass-noun-token`

```yaml
name: mass-noun-token
pattern: (money|profit|interest|income|savings|cost|amount|total)\b
emits: UNIT_CATEGORY_TOKEN
extracted_fields: (lemma: str, unit_class: str = "currency-mass")
provenance: ADR-0164.1
priority: 70
```

Rationale: ports `_MASS_NOUNS` from `generate/math_candidate_parser.py`
into a primitive form. The set is closed (8 lemmas) and orthographic;
each lemma is a single token recognized by spelling. The reader
composes "how much <mass-noun>" via composition rules over
`question_open` + `question_continuous_qty` + `UNIT_CATEGORY_TOKEN`,
not via extending this pattern across the question stem.

> Boundary note: this primitive is at the edge of the rule. A
> `mass-noun-token` is a closed set of single tokens, which satisfies
> ADR-0165's "contiguous token-class run." It is included as a
> primitive (rather than as an operational-lexicon entry under
> ADR-0164) because the reader's question-frame composition needs a
> structured category at scan time, not a lexicon hit during the
> composition pass. If Phase 1 measurement shows this is better
> modeled as a lexicon category, supersede this entry.

---

## Overlap precedence

A token span can be matched by more than one primitive. The reader runs
primitives in `priority` order (lower first), commits the first match
that consumes a non-empty span, and skips the consumed span for
subsequent primitives. The pairwise table below is the *normative*
record of every overlap relevant to the seed set. Each row is a
deliberate decision; review of a new primitive must extend this table.

| Span shape | Candidate primitives | Winner | Rationale |
|---|---|---|---|
| `$18.00` | `decimal-currency-literal`, `currency-literal`, `numeric-literal` | `decimal-currency-literal` | Two-decimal cents form is structurally distinct (rounding). Currency-literal would lose the cents semantics; numeric-literal would lose the `$`. Lowest priority (10) wins. |
| `$18` | `currency-literal`, `numeric-literal` | `currency-literal` | The `$` prefix carries the unit class. Numeric-literal alone would emit `unit_class=pending` and force the reader to recover the currency unit from context, which is exactly the kind of grammar inference the comprehension layer is meant to do *with* typed evidence, not despite it. |
| `25%` | `percentage-literal`, `numeric-literal` | `percentage-literal` | The `%` suffix carries the unit class (ratio). Same rationale as currency. |
| `3 hours` | `time-amount-literal`, `numeric-literal` (matching `3`) | `time-amount-literal` | The time-unit noun is part of the closed shape; allowing `numeric-literal` to consume `3` first would orphan `hours` and force the reader to compose a time quantity through the lexicon — bypassing a typed primitive that already exists. The span-commit rule (longest valid match at lowest priority among ties) handles this: time-amount's pattern consumes the full `3 hours` span before numeric-literal's anchor matches. |
| `1/2` | `fraction-literal`, `numeric-literal` (matching `1`, then `2`) | `fraction-literal` | The slash makes the two integers a structured pair, not two independent quantities. |
| `first` | `ordinal-literal`, (nothing else) | `ordinal-literal` | Listed for completeness; no overlap, but the priority is set above `numeric-literal` so future `ordinal-numeric-literal` (`1st`, `2nd`) — if and when ratified — fits cleanly into the same slot. |
| `money` | `mass-noun-token`, (operational-lexicon `currency_unit_noun` per ADR-0164) | `mass-noun-token` | Lexical-primitive scan runs before lexicon lookup (ADR-0164 §Deterministic reader, step 1 → step 2). The primitive emits a typed `UNIT_CATEGORY_TOKEN` carrying `unit_class=currency-mass`, which the lexicon entry would also produce — but the primitive's emission is what the question-frame composition expects. |

The precedence is **fixed**; a primitive proposal that would change an
existing row in this table requires explicit supersession of this ADR,
not a silent priority bump.

---

## Rejected temptations

The patterns below are *not* primitives and must not be added as
primitives. Each is a grammar template per ADR-0165 §Code-review test
and belongs in the reader's composition rules over the operational
lexicon.

### Rejected #1 — Rate phrase `$18/hour`, `$18 per hour`

A naive primitive might be `\$(\d+(?:\.\d+)?)\s?/\s?(hour|day|week)`.

- **What does it match?** A composition of three role-distinct
  elements: a currency amount, a connector (`/` or the word `per`),
  and a time unit. Not "one piece of orthographic material" but "a
  way three pieces of material combine to mean *rate*."
- **Closed-set test:** No. The connector class is open (`/`, `per`,
  `an`, `each`, `every`, `for each`, …). The denominator class is
  open (any unit noun, not just time). Recognizing rate via regex
  enumerates surface forms.
- **Novel-phrasing test:** Refuses `$18 every hour`, `$18 for each
  hour worked`, `eighteen dollars an hour`. The refusal is brittle
  on the same underlying meaning — the diagnostic of a grammar
  template.

Correct home: reader composition rule `[QUANTITY{currency}] +
[distributive_modifier|"/"|"per"] + [time_unit_noun]` → `RATE`
operation. Currency and time-amount primitives feed the composition;
the rate semantics emerge there.

### Rejected #2 — Compound entity `her three friends`, `Tina and Marion`

A naive primitive might be
`(she|he|her|his|their)\s+(\d+|two|three|four|five|several|a few)\s+(friends|sisters|brothers|cousins)`.

- **What does it match?** A possessive determiner, a count, and a
  relational noun — three role-distinct elements with an open
  combinatorial product. Pure grammar.
- **Closed-set test:** No. Possessive determiners are a small but
  context-dependent set; counts include numeric literals *and*
  spelled-out numerals *and* "a few" / "several" / "many" hedges;
  relational nouns are open-ended (`friends`, `co-workers`,
  `neighbors`, `kids`, `students`, …).
- **Novel-phrasing test:** Refuses `the three friends she invited`,
  `Tina, Marion, and Jen`, `each of her friends`. Every novel
  reference shape refuses on the same underlying meaning — group
  entity binding.

Correct home: cross-sentence `ProblemReadingState` (ADR-0164 §Open
question #4) plus reader composition rules over `entity_pronoun`,
`numeric-literal`, and a `relational_noun` lexicon category. Group
binding is the reader's job, not the primitive layer's.

### Rejected #3 — Question stem `How much money will she earn`

A naive primitive might be `How\s+much\s+(money|profit|...)\s+(will
|did|does)\s+(she|he|they|it)`.

- **What does it match?** A five-role grammar template: question
  opener + continuous-quantity word + mass-noun + auxiliary +
  pronoun. This is the canonical example of what ADR-0164 deprecates
  and ADR-0165 §Forbidden uses cites verbatim.
- **Closed-set test:** No on every role except `much` and `she/he/
  they/it`. The mass-noun set is reviewable but the rest of the
  composition opens onto open lexical classes (auxiliaries, modal
  verbs, perfect/progressive constructions, embedded clauses).
- **Novel-phrasing test:** Refuses `How much will she have earned
  by Friday`, `How much money does Tina end up with`, `How much
  did it cost him in total`. These are the exact 34/47 refusals
  ADR-0164 §Context documents.

Correct home: the reader's question-frame composition rules,
consuming primitive emissions (`UNIT_CATEGORY_TOKEN` from
`mass-noun-token`) plus operational-lexicon categories
(`question_open`, `question_continuous_qty`, `entity_pronoun`,
`accumulation_verb` from ADR-0164 §Operational lexicon).

### Rejected #4 — Compound numeric `1,000`, `1.5M`, `1.5 million`

A naive primitive might be `\d{1,3}(,\d{3})+(\.\d+)?` plus a
sibling `\d+(\.\d+)?[KMB]\b`.

- **What does it match?** A composition between digit groups and
  scaling tokens. Comma grouping is one shape; magnitude suffixes
  are another; spelled-out scale words are a third.
- **Closed-set test:** Borderline. The comma-grouped form alone
  (`1,000`, `12,345,678`) is a closed orthographic shape and *could*
  be ratified as a primitive (`grouped-numeric-literal`) through the
  ADR-0165 corridor when GSM8K evidence demands it. The `1.5M` and
  `1.5 million` forms are compositions and must not be folded into
  the same primitive.
- **Why rejected from the seed:** No GSM8K `train_sample/v1` case
  observed in the ADR-0164 §Context evidence requires it. Adding it
  speculatively violates ADR-0114a (no surface-form additions without
  evidence) and the cleanup-as-you-find discipline (don't add what
  you don't need).

Correct home: defer to teaching-corridor ratification once a refusal
on a comma-grouped numeric is observed. The `M`/`million` scaling
case is composition (numeric + magnitude lexicon entry), not a
primitive.

---

## Consequences

### Positive

1. **Phase 1 acceptance has a concrete scan-time vocabulary.** Eight
   primitives, fully specified, ship with the Phase 1 PR. The reader's
   step-1 (lexical primitive scan) is now implementable.
2. **Overlap behavior is recorded, not discovered.** Every overlap a
   reviewer might worry about (`$18.00`, `3 hours`, `1/2`, `25%`) has
   an explicit winner with rationale.
3. **The "where do I draw the line?" question is settled with
   examples.** Future authors who want to add a primitive for a rate
   phrase, compound entity, or question stem have §Rejected
   temptations as the rejection precedent — no relitigation.

### Negative / tradeoffs

1. **Seed set is small.** Eight primitives won't cover every shape
   GSM8K throws. That's the point — population is the corridor's job
   (ADR-0165 §Population). Reader refusals on unknown token shapes
   are evidence that flows back into the queue.
2. **`mass-noun-token` is at the rule boundary.** Listed honestly in
   §Seed primitive set with an explicit supersede-if path. Phase 1
   measurement will settle whether it belongs here or in the
   operational lexicon.
3. **Overlap precedence table will grow.** Every new primitive must
   extend the table. This is the cost of fixed precedence, and it's
   cheaper than the alternative (silent runtime tiebreaking).

---

## Acceptance criteria for this sub-ADR

This ADR moves to **Accepted** when:

1. The seed registry above is materialized in
   `language_packs/data/en_core_math_v1/lexical_primitives.json` (or
   the equivalent loader format settled in the Phase 1 PR), one
   record per entry, fields populated verbatim.
2. The manifest checksum hashes the bytes written to disk (CLAUDE.md
   §Semantic Pack Discipline).
3. The overlap-precedence table has a pinned regression test —
   given the seed registry, the eight overlap rows resolve to the
   declared winner on synthetic minimum-pair inputs.
4. ADR-0164 §Open question #1 is checked off in that ADR's
   open-questions list when ADR-0164 next ships an update.

---

## Cross-references

- **Parent:** ADR-0164 — Incremental Comprehension Reader
- **Companion (the rule):** ADR-0165 — Regex Scope Rule
- **Anti-overfitting doctrine:** ADR-0114a
- **Pack discipline:** CLAUDE.md §Semantic Pack Discipline
- **Population corridor:** ADR-0150 (contemplation), ADR-0152
  (learning-arc proof), ADR-0155 (CI contemplation runner), ADR-0161
  (HITL async queue)
- **Anchor:** `[[thesis-decoding-not-generating]]` — the primitive
  set is a decoder's recognizer table. It enumerates the closed
  orthographic shapes the reader can pick up *as such*; it does not
  enumerate sentences.