core/generate
Shay b3dbde94b4
feat(comprehension/8.2): universal proper_noun_token primitive (#333)
ADR-0164.1 amendment: replace name-whitelist entity admission with a
universal lexeme primitive that recognizes any capitalized token as a
proper noun. The gender-coded name lists are demoted from admission
criterion to enrichment-only lookup. A name outside the curated lists
still admits cleanly with gender="unknown" — ADR-0164.2's pronoun
resolution rules handle the unknown case via single-salient fallback
or refuse with ambiguous_pronoun_referent.

Universal at the primitive layer: the new proper_noun_token primitive
is domain-agnostic. It sits in the shared PRIMITIVE_REGISTRY and is
available to every current and future reader (math, narrative,
code-comment, multi-lingual). The math reader is its first consumer.

Pattern: ^[A-Z][A-Za-z'-]*[a-z][A-Za-z'-]*$
- requires capitalized first letter
- requires ≥1 lowercase letter (rejects all-caps acronyms)
- allows internal apostrophes (O'Brien) and hyphens (Mary-Anne)
- matches "Tina", "Bob", "Marnie", "McDonald" — rejects "TINA",
  "123", "$5.00" (those go to their own primitives)

Sentence-initial lookup-first dispatch (lifecycle._classify):
- At token_index == 0: lookup() first, skipping proper_noun_gender_*
  categories (treated as not-found so the primitive can fire). If
  lookup misses, primitive scan picks up novel names. Inverts the
  question from "is this a name?" to "is this a known common word?"
- At token_index > 0: primitive-first with UNIT_CATEGORY_TOKEN ceding
  to operational lexicon for currency_unit_noun overrides.

Lexicon rename (per-category source files):
- proper_noun_entity_female.jsonl -> proper_noun_gender_female.jsonl
- proper_noun_entity_male.jsonl   -> proper_noun_gender_male.jsonl

Compiled lexicon.jsonl: rename the two semantic_domain tags; drop
"marnie" (was only in proper_noun_entity_female, now absent from
the gender-coded sources). Net: 208 -> 207 entries. New manifest
checksum: 1fb9b0d790258736267d528e8e8a2436ce88b9ce690805fe2813ba077861ba2a

New helper gender_of_proper_noun(surface, lexicon) returns
Literal["female","male","neuter","unknown"] — pure enrichment lookup,
never gates admission.

Measurement (reader_phase1_plus_proper_noun_delta.json):
- pre-primitive baseline: correct=3 refused=47 wrong=0
- post-primitive measurement: correct=3 refused=47 wrong=0
- No regression on wrong=0
- No net admission increase observed in this train-sample harness;
  the architectural value is for future text outside the curated
  gender lists (Sonnet's #332 expanded those to cover GSM8K names).

Tests:
- test_lexeme_primitives.py: registry count 8 -> 9, proper_noun_token
  fires + variants (Bob, Marnie, McDonald, O'Brien, Mary-Anne),
  numeric/all-caps refusals, numeric-literal still wins overlap on "123"
- test_reader_question_frame.py: 5 new tests for sentence-initial
  dispatch + unknown-gender pronoun resolution + novel-name admission
  via primitive (Zelda)
- test_en_core_math_v1_pack.py: category counts updated; mutual-exclusion
  between gender_female and gender_male preserved; total 208 -> 207
- test_lexicon.py: category list + lookup assertion updated to renamed
  proper_noun_gender_female
- test_proper_noun_primitive_universality.py: new test module asserting
  domain-agnostic property of the primitive

Validation:
- pack + lexicon + primitive tests: 147 passed
- reader + universality tests: 22 passed
- smoke lane: 67 passed

Closes the engine_state question by leaving those files untracked
(repo discipline: runtime artifacts never enter PRs).

Refs ADR-0164.1 amendment, ADR-0164.2 §EntityRegistry, ADR-0165
§Legitimate uses (the new primitive passes the three-question test).
2026-05-26 22:16:34 -07:00
..
binding_graph feat(binding-graph): Phase 4 question-target binding (ADR-0135) (#179) 2026-05-23 11:24:49 -07:00
comprehension feat(comprehension/8.2): universal proper_noun_token primitive (#333) 2026-05-26 22:16:34 -07:00
__init__.py chore(generate): delete unreachable agenerate (#90) 2026-05-20 19:59:28 -07:00
admissibility.py feat(adr-0026): Phase 3 — ranked admissibility with margin 2026-05-17 15:03:03 -07:00
articulation.py Graceful fallback in realize() when slot versors are missing 2026-05-13 21:41:52 -07:00
articulation_legality.py feat(realizer): C1.5 — articulation legality at the realizer boundary 2026-05-20 11:11:28 -07:00
attention.py Add ADR-0008 salience attention 2026-05-13 22:40:36 -07:00
bridge_trace.py Phase 1 — bridge trace instrumentation (observation-only) 2026-05-18 18:04:57 -07:00
dialogue.py Fix test suite errors across core physics and generation 2026-05-14 13:02:32 -07:00
discourse_planner.py feat(discourse): Phase 2 — reflective rendering pronominalizes focus subject 2026-05-21 10:16:12 -07:00
exhaustion.py fix(W-011/W-012): propagate recognition refusal + catch InnerLoopExhaustion (#258) 2026-05-24 20:46:46 -07:00
graph_constraint.py fix(adr-0046): make forward-graph-constraint branch mergeable 2026-05-18 05:57:46 -07:00
graph_planner.py perf(graph): PropositionGraph.topo_order — Kahn's O(N+E) instead of O(N×E) (#92) 2026-05-20 20:37:21 -07:00
grounding_accessors.py feat(grounding): structured GroundedFact accessors for discourse planner 2026-05-19 11:19:59 -07:00
intent.py fix(intent): route 'Actually X R Y' premises to CORRECTION (inference_closure) (#117) 2026-05-22 12:33:56 -07:00
intent_bridge.py Phase 2 — proposition-slot grounding for articulate_with_intent 2026-05-18 18:18:31 -07:00
intent_ratifier.py feat(recognition): ADR-0144 — EpistemicGraph carrier + pipeline integration (#227) 2026-05-24 13:39:01 -07:00
math_candidate_graph.py feat(ADR-0164.P1): reader/regex hybrid coexistence + Phase 1 measurement gate (#331) 2026-05-26 21:14:11 -07:00
math_candidate_parser.py feat(ADR-0163.D.4): question grammar extension — mass nouns, comparatives, pronoun-entity resolution (#310) 2026-05-26 16:19:37 -07:00
math_parser.py fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
math_problem_graph.py feat(parser): ADR-0123 comparison-phrasing substrate (substrate-only; lift deferred) 2026-05-23 01:56:28 -07:00
math_realizer.py feat(realizer): ADR-0123 comparison-phrasing surface (closes substrate) 2026-05-23 02:03:49 -07:00
math_roundtrip.py feat(ADR-0131.G.3.1): numerics extensions — fractions + multi-currency + multi-token cardinals + word-num-adjective 2026-05-23 15:16:46 -07:00
math_solver.py feat(parser): ADR-0123 comparison-phrasing substrate (substrate-only; lift deferred) 2026-05-23 01:56:28 -07:00
math_symbolic_equivalence.py feat(ADR-0131.1.B): harden symbolic equivalence lane with generated corpus + exact algebra (#169) 2026-05-23 10:47:57 -07:00
math_symbolic_normalizer.py feat(ADR-0131.1.B): harden symbolic equivalence lane with generated corpus + exact algebra (#169) 2026-05-23 10:47:57 -07:00
math_verifier.py feat(parser): ADR-0123 comparison-phrasing substrate (substrate-only; lift deferred) 2026-05-23 01:56:28 -07:00
math_versor_arithmetic.py feat(ADR-0141): multiply as CGA dilator versor (positive non-zero) (#216) 2026-05-24 09:09:53 -07:00
morphology.py fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
ood_surface_generator.py fix(quarantine): drain all 60 quarantined tests — QUARANTINE=∅ (#267) 2026-05-25 11:22:12 -07:00
operators.py feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
perturbation_suite.py feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
proposition.py feat(adr-0022): Forward Semantic Control — Accepted 2026-05-17 12:10:20 -07:00
realizer.py [codex] Implement energy-modulated vault surface (#269) 2026-05-25 11:33:32 -07:00
realizer_guard.py feat(coherence): ADR-0075 — realizer slot-type guard (C1) 2026-05-19 22:35:09 -07:00
recognizer_anchor_inject.py feat(ADR-0163.D.2): parsed_anchors → MathProblemGraph state — discrete_count_statement injection v1 (#315) 2026-05-26 18:32:05 -07:00
recognizer_match.py feat(ADR-0163.D.2): parsed_anchors → MathProblemGraph state — discrete_count_statement injection v1 (#315) 2026-05-26 18:32:05 -07:00
recognizer_registry.py feat(ADR-0163.D): wire ratified RecognizerSpecs into math_candidate_graph admissibility surface (#302) 2026-05-26 13:11:47 -07:00
result.py feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
rotor_admissibility.py feat(adr-0025): Phase 4 — rotor / frame admissibility at the seam 2026-05-17 15:16:32 -07:00
salience.py Implement core physics and pack validation 2026-05-14 12:35:19 -07:00
semantic_templates.py feat: vault recall index, Rust versor parity, cognitive pack expansion 2026-05-15 15:34:39 -07:00
stream.py chore(generate): make stop-tokens caller-overridable via RuntimeConfig (#87) 2026-05-20 19:59:33 -07:00
surface.py feat(surface): ADR-0031 — score-decomposition surface (per-axis hedges) 2026-05-17 20:16:22 -07:00
templates.py feat(realizer): C1.5 — articulation legality at the realizer boundary 2026-05-20 11:11:28 -07:00