Commit graph

2 commits

Author SHA1 Message Date
Shay
b3dbde94b4
feat(comprehension/8.2): universal proper_noun_token primitive (#333)
ADR-0164.1 amendment: replace name-whitelist entity admission with a
universal lexeme primitive that recognizes any capitalized token as a
proper noun. The gender-coded name lists are demoted from admission
criterion to enrichment-only lookup. A name outside the curated lists
still admits cleanly with gender="unknown" — ADR-0164.2's pronoun
resolution rules handle the unknown case via single-salient fallback
or refuse with ambiguous_pronoun_referent.

Universal at the primitive layer: the new proper_noun_token primitive
is domain-agnostic. It sits in the shared PRIMITIVE_REGISTRY and is
available to every current and future reader (math, narrative,
code-comment, multi-lingual). The math reader is its first consumer.

Pattern: ^[A-Z][A-Za-z'-]*[a-z][A-Za-z'-]*$
- requires capitalized first letter
- requires ≥1 lowercase letter (rejects all-caps acronyms)
- allows internal apostrophes (O'Brien) and hyphens (Mary-Anne)
- matches "Tina", "Bob", "Marnie", "McDonald" — rejects "TINA",
  "123", "$5.00" (those go to their own primitives)

Sentence-initial lookup-first dispatch (lifecycle._classify):
- At token_index == 0: lookup() first, skipping proper_noun_gender_*
  categories (treated as not-found so the primitive can fire). If
  lookup misses, primitive scan picks up novel names. Inverts the
  question from "is this a name?" to "is this a known common word?"
- At token_index > 0: primitive-first with UNIT_CATEGORY_TOKEN ceding
  to operational lexicon for currency_unit_noun overrides.

Lexicon rename (per-category source files):
- proper_noun_entity_female.jsonl -> proper_noun_gender_female.jsonl
- proper_noun_entity_male.jsonl   -> proper_noun_gender_male.jsonl

Compiled lexicon.jsonl: rename the two semantic_domain tags; drop
"marnie" (was only in proper_noun_entity_female, now absent from
the gender-coded sources). Net: 208 -> 207 entries. New manifest
checksum: 1fb9b0d790258736267d528e8e8a2436ce88b9ce690805fe2813ba077861ba2a

New helper gender_of_proper_noun(surface, lexicon) returns
Literal["female","male","neuter","unknown"] — pure enrichment lookup,
never gates admission.

Measurement (reader_phase1_plus_proper_noun_delta.json):
- pre-primitive baseline: correct=3 refused=47 wrong=0
- post-primitive measurement: correct=3 refused=47 wrong=0
- No regression on wrong=0
- No net admission increase observed in this train-sample harness;
  the architectural value is for future text outside the curated
  gender lists (Sonnet's #332 expanded those to cover GSM8K names).

Tests:
- test_lexeme_primitives.py: registry count 8 -> 9, proper_noun_token
  fires + variants (Bob, Marnie, McDonald, O'Brien, Mary-Anne),
  numeric/all-caps refusals, numeric-literal still wins overlap on "123"
- test_reader_question_frame.py: 5 new tests for sentence-initial
  dispatch + unknown-gender pronoun resolution + novel-name admission
  via primitive (Zelda)
- test_en_core_math_v1_pack.py: category counts updated; mutual-exclusion
  between gender_female and gender_male preserved; total 208 -> 207
- test_lexicon.py: category list + lookup assertion updated to renamed
  proper_noun_gender_female
- test_proper_noun_primitive_universality.py: new test module asserting
  domain-agnostic property of the primitive

Validation:
- pack + lexicon + primitive tests: 147 passed
- reader + universality tests: 22 passed
- smoke lane: 67 passed

Closes the engine_state question by leaving those files untracked
(repo discipline: runtime artifacts never enter PRs).

Refs ADR-0164.1 amendment, ADR-0164.2 §EntityRegistry, ADR-0165
§Legitimate uses (the new primitive passes the three-question test).
2026-05-26 22:16:34 -07:00
Shay
4570c2c70e
feat(comprehension): operational lexicon loader for en_core_math_v1 (ADR-0164 §Decision §1) (#325)
Implements generate/comprehension/lexicon.py: loads per-category source
files from en_core_math_v1/lexicon/*.jsonl (full schema including aliases),
verifies manifest checksum against compiled lexicon.jsonl for pack integrity,
and provides O(1) case-folded surface lookups. Module-level cache keyed on
(path, mtime_ns, sha256) avoids redundant I/O.

Exports: LexiconEntry, Lexicon, LexiconLoadError, load_lexicon(), lookup().
MappingProxyType over internal dicts prevents callers from mutating cached state.
29 tests cover load, checksum, category completeness, alias resolution,
mutual-exclusion detection, determinism, and cache identity.
2026-05-26 20:08:27 -07:00