core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/adr/ADR-0164.1-lexical-primitive-scope.md

19 KiB
Raw Blame History

ADR-0164.1 — Lexical Primitive Set Scope (seed registry for en_core_math_v1)

Status: Proposed Date: 2026-05-26 Author: Shay Anchor: thesis-decoding-not-generating Parent: ADR-0164 — Incremental Comprehension Reader Companion: ADR-0165 — Regex Scope Rule Resolves: ADR-0164 §Open question #1 ("Lexical primitive set scope")

Context — why this sub-ADR exists

ADR-0164 specifies the reader and leaves the exact bootstrap primitive set open (§Open question #1). ADR-0165 specifies the rule that bounds what may become a primitive (lexeme-level, closed orthographic shape) but does not enumerate the set.

This ADR closes the gap. It enumerates the seed primitive registry that ships with the ADR-0164 Phase 1 PR, fixes the registry record schema, documents overlap precedence (the only place primitives can interact at recognition time), and records the temptations that are explicitly not admitted, so future authors don't relitigate them.

The reader's Phase 1 acceptance gate (see ADR-0164 §Phasing) depends on this set being closed before scan-time. Adding a primitive after Phase 1 follows the population corridor in ADR-0165 §Population (contemplation → proposal → review).

Decision

The seed lexeme-primitive registry for en_core_math_v1 contains the eight primitives below. Each is a closed orthographic shape per the ADR-0165 three-question test (§Code-review test). Each carries the schema fields in §Registry record schema. Overlap precedence between primitives is fixed by §Overlap precedence and is the only behavior that may differ between "primitive A fires" and "primitive A fires given primitive B already matched the same span."

Population beyond this seed set rides the ADR-0165 corridor and is out of scope for this ADR.

Registry record schema

Each primitive is a frozen record with the following fields. Field order is canonical (used by the canonical-bytes serialization that feeds trace_hash per CLAUDE.md §Runtime Surface Contract).

Field Type Meaning
name kebab-case string, unique Registry key. Stable across rounds. Forms the trace-evidence label when the primitive fires.
pattern regex source string (Python re flavor) The orthographic recognizer. Must be anchorable to a single token or contiguous token-class run (ADR-0165 §Rule). No .* across word combinations. No \s+VERB\s+ constructions.
emits enum: QUANTITY, ORDINAL, UNIT_CATEGORY_TOKEN The reader category emitted on match. The category is what the reader's composition rules consume; the surface form is discarded after extraction.
extracted_fields typed tuple (name: type, ...) The structured payload produced. Types are int, Decimal, str from a closed enum, or tuple[int, int] for compound shapes (e.g. fractions). Reader composition is allowed to read these fields and only these fields.
provenance string "ADR-<id>" or "teaching-ratified <ISO date> #<queue-id>" Audit trail. Seed primitives in this ADR are stamped "ADR-0164.1". New primitives stamped by the HITL queue at acceptance time (ADR-0161).
priority integer ≥ 0 (lower wins) Tiebreaker when two primitives match the same span. The overlap-precedence table (§Overlap precedence) fixes seed priorities; new primitives declare priority at proposal time and the operator ratifies it.

The registry is a tuple of records, ordered by priority ascending then name ascending (stable ordering for replay equivalence).

Seed primitive set (n = 8)

Each entry below is a populated record. Patterns are shown as Python re source; the runtime compiles them with re.IGNORECASE unless otherwise noted, anchored to a single token / contiguous span (the reader is responsible for token-boundary alignment; the regex itself does not consume surrounding whitespace).

1. decimal-currency-literal

name: decimal-currency-literal
pattern: \$(\d+)\.(\d{2})\b
emits: QUANTITY
extracted_fields: (whole: int, cents: int, unit_class: str = "currency")
provenance: ADR-0164.1
priority: 10

Rationale: $18.00, $1.50. The two-decimal-place currency form is its own shape because (a) the cents field is structurally significant (rounding semantics), (b) it must beat both currency-literal and numeric-literal on the same span. Closed orthographic class: "dollar-sign, integer, dot, exactly two digits."

2. currency-literal

name: currency-literal
pattern: \$(\d+(?:\.\d+)?)\b
emits: QUANTITY
extracted_fields: (value: Decimal, unit_class: str = "currency")
provenance: ADR-0164.1
priority: 20

Rationale: $18, $1.5, $1000. Currency notation with arbitrary (non-cents-shaped) decimal. Beats numeric-literal because the $ prefix carries the unit. Does not cover $1.5M, $18/hour, USD 18 — see §Rejected temptations.

3. percentage-literal

name: percentage-literal
pattern: (\d+(?:\.\d+)?)\s?%
emits: QUANTITY
extracted_fields: (value: Decimal, unit_class: str = "ratio")
provenance: ADR-0164.1
priority: 30

Rationale: 25%, 7.5 %. The % glyph is a closed unit suffix. Single optional space allowed because percent signs frequently appear detached (7 %). The space inside the pattern is a single character, not a \s+ across structure — the run is still one orthographic shape.

4. fraction-literal

name: fraction-literal
pattern: (\d+)\s?/\s?(\d+)\b
emits: QUANTITY
extracted_fields: (numerator: int, denominator: int, unit_class: str = "fraction")
provenance: ADR-0164.1
priority: 40

Rationale: 1/2, 3 / 4. The slash-separated integer pair is a genuine closed shape. Optional single space on either side of the slash. Reader composes "1/2 of " through composition rules, not through extending this pattern.

5. time-amount-literal

name: time-amount-literal
pattern: (\d+)[-\s]?(hour|minute|day|week|month|year|second)s?\b
emits: QUANTITY
extracted_fields: (value: int, unit: str, unit_class: str = "time")
provenance: ADR-0164.1
priority: 50

Rationale: 3 hours, 30-minute, 2 days, 1 week. The "number-plus-time-unit-noun" form is the canonical closed time-amount shape. Unit set is a closed enum; the reader treats hour/hours/ hour- uniformly via the singular-form extraction. Does not cover an hour (article + bare unit, that's grammar — see §Rejected temptations), and does not cover 3 hours per week (rate phrase, grammar).

6. numeric-literal

name: numeric-literal
pattern: \d+(?:\.\d+)?\b
emits: QUANTITY
extracted_fields: (value: Decimal, unit_class: str = "pending")
provenance: ADR-0164.1
priority: 100

Rationale: 18, 1.5, 12. The bare number. unit_class=pending signals to the reader that a unit attachment is expected from a downstream token (a count_unit_noun, currency_unit_noun, time_unit_noun, etc., from the operational lexicon — ADR-0164 §Operational lexicon). Highest numeric priority ensures all other numeric-bearing primitives win the overlap.

7. ordinal-literal

name: ordinal-literal
pattern: (first|second|third|fourth|fifth|sixth|seventh|eighth|ninth|tenth)\b
emits: ORDINAL
extracted_fields: (rank: int)
provenance: ADR-0164.1
priority: 60

Rationale: first, second, third. Closed list of English ordinal spellings 1–10. Extending the set to eleventh–twentieth or to numeric ordinals (1st, 2nd) is a teaching-corridor decision, not a silent extension of this ADR. Extracted rank is the integer the ordinal denotes.

8. mass-noun-token

name: mass-noun-token
pattern: (money|profit|interest|income|savings|cost|amount|total)\b
emits: UNIT_CATEGORY_TOKEN
extracted_fields: (lemma: str, unit_class: str = "currency-mass")
provenance: ADR-0164.1
priority: 70

Rationale: ports _MASS_NOUNS from generate/math_candidate_parser.py into a primitive form. The set is closed (8 lemmas) and orthographic; each lemma is a single token recognized by spelling. The reader composes "how much " via composition rules over question_open + question_continuous_qty + UNIT_CATEGORY_TOKEN, not via extending this pattern across the question stem.

Boundary note: this primitive is at the edge of the rule. A mass-noun-token is a closed set of single tokens, which satisfies ADR-0165's "contiguous token-class run." It is included as a primitive (rather than as an operational-lexicon entry under ADR-0164) because the reader's question-frame composition needs a structured category at scan time, not a lexicon hit during the composition pass. If Phase 1 measurement shows this is better modeled as a lexicon category, supersede this entry.

Overlap precedence

A token span can be matched by more than one primitive. The reader runs primitives in priority order (lower first), commits the first match that consumes a non-empty span, and skips the consumed span for subsequent primitives. The pairwise table below is the normative record of every overlap relevant to the seed set. Each row is a deliberate decision; review of a new primitive must extend this table.

Span shape Candidate primitives Winner Rationale
$18.00 decimal-currency-literal, currency-literal, numeric-literal decimal-currency-literal Two-decimal cents form is structurally distinct (rounding). Currency-literal would lose the cents semantics; numeric-literal would lose the $. Lowest priority (10) wins.
$18 currency-literal, numeric-literal currency-literal The $ prefix carries the unit class. Numeric-literal alone would emit unit_class=pending and force the reader to recover the currency unit from context, which is exactly the kind of grammar inference the comprehension layer is meant to do with typed evidence, not despite it.
25% percentage-literal, numeric-literal percentage-literal The % suffix carries the unit class (ratio). Same rationale as currency.
3 hours time-amount-literal, numeric-literal (matching 3) time-amount-literal The time-unit noun is part of the closed shape; allowing numeric-literal to consume 3 first would orphan hours and force the reader to compose a time quantity through the lexicon — bypassing a typed primitive that already exists. The span-commit rule (longest valid match at lowest priority among ties) handles this: time-amount's pattern consumes the full 3 hours span before numeric-literal's anchor matches.
1/2 fraction-literal, numeric-literal (matching 1, then 2) fraction-literal The slash makes the two integers a structured pair, not two independent quantities.
first ordinal-literal, (nothing else) ordinal-literal Listed for completeness; no overlap, but the priority is set above numeric-literal so future ordinal-numeric-literal (1st, 2nd) — if and when ratified — fits cleanly into the same slot.
money mass-noun-token, (operational-lexicon currency_unit_noun per ADR-0164) mass-noun-token Lexical-primitive scan runs before lexicon lookup (ADR-0164 §Deterministic reader, step 1 → step 2). The primitive emits a typed UNIT_CATEGORY_TOKEN carrying unit_class=currency-mass, which the lexicon entry would also produce — but the primitive's emission is what the question-frame composition expects.

The precedence is fixed; a primitive proposal that would change an existing row in this table requires explicit supersession of this ADR, not a silent priority bump.

Rejected temptations

The patterns below are not primitives and must not be added as primitives. Each is a grammar template per ADR-0165 §Code-review test and belongs in the reader's composition rules over the operational lexicon.

Rejected #1 — Rate phrase $18/hour, $18 per hour

A naive primitive might be \$(\d+(?:\.\d+)?)\s?/\s?(hour|day|week).

  • What does it match? A composition of three role-distinct elements: a currency amount, a connector (/ or the word per), and a time unit. Not "one piece of orthographic material" but "a way three pieces of material combine to mean rate."
  • Closed-set test: No. The connector class is open (/, per, an, each, every, for each, …). The denominator class is open (any unit noun, not just time). Recognizing rate via regex enumerates surface forms.
  • Novel-phrasing test: Refuses $18 every hour, $18 for each hour worked, eighteen dollars an hour. The refusal is brittle on the same underlying meaning — the diagnostic of a grammar template.

Correct home: reader composition rule [QUANTITY{currency}] + [distributive_modifier|"/"|"per"] + [time_unit_noun] → RATE operation. Currency and time-amount primitives feed the composition; the rate semantics emerge there.

Rejected #2 — Compound entity her three friends, Tina and Marion

A naive primitive might be (she|he|her|his|their)\s+(\d+|two|three|four|five|several|a few)\s+(friends|sisters|brothers|cousins).

  • What does it match? A possessive determiner, a count, and a relational noun — three role-distinct elements with an open combinatorial product. Pure grammar.
  • Closed-set test: No. Possessive determiners are a small but context-dependent set; counts include numeric literals and spelled-out numerals and "a few" / "several" / "many" hedges; relational nouns are open-ended (friends, co-workers, neighbors, kids, students, …).
  • Novel-phrasing test: Refuses the three friends she invited, Tina, Marion, and Jen, each of her friends. Every novel reference shape refuses on the same underlying meaning — group entity binding.

Correct home: cross-sentence ProblemReadingState (ADR-0164 §Open question #4) plus reader composition rules over entity_pronoun, numeric-literal, and a relational_noun lexicon category. Group binding is the reader's job, not the primitive layer's.

Rejected #3 — Question stem How much money will she earn

A naive primitive might be How\s+much\s+(money|profit|...)\s+(will |did|does)\s+(she|he|they|it).

  • What does it match? A five-role grammar template: question opener + continuous-quantity word + mass-noun + auxiliary + pronoun. This is the canonical example of what ADR-0164 deprecates and ADR-0165 §Forbidden uses cites verbatim.
  • Closed-set test: No on every role except much and she/he/ they/it. The mass-noun set is reviewable but the rest of the composition opens onto open lexical classes (auxiliaries, modal verbs, perfect/progressive constructions, embedded clauses).
  • Novel-phrasing test: Refuses How much will she have earned by Friday, How much money does Tina end up with, How much did it cost him in total. These are the exact 34/47 refusals ADR-0164 §Context documents.

Correct home: the reader's question-frame composition rules, consuming primitive emissions (UNIT_CATEGORY_TOKEN from mass-noun-token) plus operational-lexicon categories (question_open, question_continuous_qty, entity_pronoun, accumulation_verb from ADR-0164 §Operational lexicon).

Rejected #4 — Compound numeric 1,000, 1.5M, 1.5 million

A naive primitive might be \d{1,3}(,\d{3})+(\.\d+)? plus a sibling \d+(\.\d+)?[KMB]\b.

  • What does it match? A composition between digit groups and scaling tokens. Comma grouping is one shape; magnitude suffixes are another; spelled-out scale words are a third.
  • Closed-set test: Borderline. The comma-grouped form alone (1,000, 12,345,678) is a closed orthographic shape and could be ratified as a primitive (grouped-numeric-literal) through the ADR-0165 corridor when GSM8K evidence demands it. The 1.5M and 1.5 million forms are compositions and must not be folded into the same primitive.
  • Why rejected from the seed: No GSM8K train_sample/v1 case observed in the ADR-0164 §Context evidence requires it. Adding it speculatively violates ADR-0114a (no surface-form additions without evidence) and the cleanup-as-you-find discipline (don't add what you don't need).

Correct home: defer to teaching-corridor ratification once a refusal on a comma-grouped numeric is observed. The M/million scaling case is composition (numeric + magnitude lexicon entry), not a primitive.

Consequences

Positive

  1. Phase 1 acceptance has a concrete scan-time vocabulary. Eight primitives, fully specified, ship with the Phase 1 PR. The reader's step-1 (lexical primitive scan) is now implementable.
  2. Overlap behavior is recorded, not discovered. Every overlap a reviewer might worry about ($18.00, 3 hours, 1/2, 25%) has an explicit winner with rationale.
  3. The "where do I draw the line?" question is settled with examples. Future authors who want to add a primitive for a rate phrase, compound entity, or question stem have §Rejected temptations as the rejection precedent — no relitigation.

Negative / tradeoffs

  1. Seed set is small. Eight primitives won't cover every shape GSM8K throws. That's the point — population is the corridor's job (ADR-0165 §Population). Reader refusals on unknown token shapes are evidence that flows back into the queue.
  2. mass-noun-token is at the rule boundary. Listed honestly in §Seed primitive set with an explicit supersede-if path. Phase 1 measurement will settle whether it belongs here or in the operational lexicon.
  3. Overlap precedence table will grow. Every new primitive must extend the table. This is the cost of fixed precedence, and it's cheaper than the alternative (silent runtime tiebreaking).

Acceptance criteria for this sub-ADR

This ADR moves to Accepted when:

  1. The seed registry above is materialized in packs/data/en_core_math_v1/lexical_primitives.json (or the equivalent loader format settled in the Phase 1 PR), one record per entry, fields populated verbatim.
  2. The manifest checksum hashes the bytes written to disk (CLAUDE.md §Semantic Pack Discipline).
  3. The overlap-precedence table has a pinned regression test — given the seed registry, the eight overlap rows resolve to the declared winner on synthetic minimum-pair inputs.
  4. ADR-0164 §Open question #1 is checked off in that ADR's open-questions list when ADR-0164 next ships an update.

Cross-references

  • Parent: ADR-0164 — Incremental Comprehension Reader
  • Companion (the rule): ADR-0165 — Regex Scope Rule
  • Anti-overfitting doctrine: ADR-0114a
  • Pack discipline: CLAUDE.md §Semantic Pack Discipline
  • Population corridor: ADR-0150 (contemplation), ADR-0152 (learning-arc proof), ADR-0155 (CI contemplation runner), ADR-0161 (HITL async queue)
  • Anchor: [[thesis-decoding-not-generating]] — the primitive set is a decoder's recognizer table. It enumerates the closed orthographic shapes the reader can pick up as such; it does not enumerate sentences.