core/generate/lexicon.py

"""Single source of truth for CORE's closed English lexical tables.

Before this module the same linguistic facts were encoded up to five times
across the reading and writing paths, each copy written independently as a
successive deduction band landed. Measured on ``main`` @ ``0948f7cd``:
35 word-tables on the reading path vs 9 on the writing path, sharing 43 of
518 distinct words (Jaccard 0.083).

Every table here is **closed** — a finite enumeration, not a rule. Where two
call sites need different content, this module defines one literal and
*derives* the other, so a divergence is stated once and cannot drift.

Naming convention: a name is the linguistic fact, not the consumer. When a
consumer needs a narrower view, the narrow name says whose view it is
(``STRUCTURAL``, ``NEGATION_BEARING_WITH_NOT``).

**Deliberate non-goal:** this module does not decide anything. It holds
tables and derivations; the refusal/agreement logic stays with each band.
Consumers and their serving status are recorded in
``tests/test_lexicon_single_source.py``, which fails when a new consumer
appears. Phase 2A of ``docs/plans/grammar-unification-2026-07-26.md``.

**This module is a staging area, not the final home.** ADR-0258 §5 already
decided where number morphology belongs:

    the number-linking table is CORE's first morphology table, and it is
    exactly the artifact class the tri-language doctrine says must
    eventually enter through pack ratification (ADR-0253). [...] When a
    ``grc_*``/``he_*`` member band is built, the table moves from module
    constants to a ratified pack; no premature parameterization now.

That trigger has **not** fired — ``packs/{grc,he,el}/`` exist, but no
``grc``/``he`` member band does, so the promotion is correctly deferred and
this module does not pre-empt it. Consolidating five disagreeing copies into
one is a prerequisite for promotion either way: you cannot ratify a fact as
a pack row while the tree holds several answers to it.

Measured state of the destination, for whoever picks the promotion up:
``packs/en/morphology.jsonl`` holds **9 records** (7 forms of *be*, plus
``word``/``beginning`` — both regular plurals, present to support John 1:1),
so **no irregular English plural exists as pack data**. Neither end of the
pipe is connected: ``features.number`` has no consumer anywhere, and the only
code that opens ``morphology.jsonl``
(``chat/pack_resolver.py::_pack_morph_roots_for``) extracts a top-level
``root`` key that **0 of 31 records across all four packs** define, so it
returns ``{}`` every call.
"""

from __future__ import annotations

from typing import Final

# --------------------------------------------------------------------------
# Connectives
# --------------------------------------------------------------------------

#: Connective structure the member/verb/conditional bands do not compose.
#: Was three byte-identical copies: ``proof_chain/member.py::_CONNECTIVES``,
#: ``proof_chain/verb.py::_CONNECTIVES``,
#: ``proof_chain/cond_member.py::_CONNECTIVE_TOKENS``.
CONNECTIVES: Final[frozenset[str]] = frozenset({"if", "then", "or", "and", "either"})

#: The v2-EN band's structural-function-word set. Identical to
#: :data:`CONNECTIVES` plus ``therefore`` — that band consumes ``therefore``
#: as the conclusion marker, so a leftover one signals an unconsumed parse.
#: Derived, not duplicated: the two sets can no longer drift apart.
STRUCTURAL: Final[frozenset[str]] = CONNECTIVES | {"therefore"}

# --------------------------------------------------------------------------
# Copulas and negation
# --------------------------------------------------------------------------

#: The finite forms of *be* that CORE recognizes, in canonical order.
#:
#: **One inventory, four roles.** This was four separate literals — a copula
#: set (``english.py``), a copula tuple (``member.py``), an auxiliary set for
#: negation lookahead (``realizer_guard.py::_BE_AUX``), and an auxiliary set
#: for prompt-anchor filtering (``chat/runtime.py::_BE_FORMS``). Copula and
#: auxiliary are different syntactic *roles*, but the token inventory is one
#: fact, and all four copies held the same four words. The role views below
#: are derived, so a fifth role cannot introduce a fifth answer.
#:
#: Known-incomplete: ``am``, ``be``, ``been``, ``being`` are absent from all
#: four original copies and remain absent here — widening the inventory
#: changes what every consumer recognizes, so it is not a 2A change.
#: ``packs/en/morphology.jsonl`` already carries ``am`` as
#: ``en:be:present:1sg``, which is one reason the eventual home for this fact
#: is the pack rather than this module.
BE_FINITE_FORMS: Final[tuple[str, ...]] = ("is", "are", "was", "were")

#: Set view of :data:`BE_FINITE_FORMS` for membership tests.
BE_FINITE: Final[frozenset[str]] = frozenset(BE_FINITE_FORMS)

#: Copular role view, ordered. ``proof_chain/member.py`` relies on ordered
#: iteration for dispatch, so this stays a tuple.
COPULA_FORMS: Final[tuple[str, ...]] = BE_FINITE_FORMS

#: Copular role view, as a set.
COPULAS: Final[frozenset[str]] = BE_FINITE

#: The "it is not the case that <X>" sentential-negation prefix. Was two
#: byte-identical copies (``english.py``, ``member.py``).
SENTENTIAL_NOT: Final[tuple[str, ...]] = ("it", "is", "not", "the", "case", "that")

#: Negation-bearing tokens a band cannot normalize away. Leaving one inside
#: an opaque atom would hide a negation from the engine, so the bands refuse.
NEGATION_BEARING: Final[frozenset[str]] = frozenset(
    {"never", "cannot", "nor", "neither", "nothing", "none", "no"}
)

#: :data:`NEGATION_BEARING` plus bare ``not``.
#:
#: **Recorded divergence (Phase 2A decision).** ``english.py`` (v2-EN) uses
#: the 7-token set and ``member.py`` (v3-MEM) uses this 8-token one. The
#: difference is not an oversight: v2-EN normalizes ``<copula> not`` into
#: ``~atom`` and so must let ``not`` through to that rule, while v3-MEM
#: normalizes only the copular slot and the sentential prefix and refuses
#: every other ``not``.
#:
#: Both bands serve, so **unifying them changes what CORE refuses** and is
#: deferred to Phase 2B under the authorization gate. Deriving the larger
#: set from the smaller removes the duplication now without changing any
#: behaviour, which is the whole point of the 2A/2B split.
NEGATION_BEARING_WITH_NOT: Final[frozenset[str]] = NEGATION_BEARING | {"not"}

# --------------------------------------------------------------------------
# Quantifiers — three distinct facts, previously conflated
# --------------------------------------------------------------------------

#: Quantifiers that can *lead* a clause, making it categorical
#: ("all whales are mammals"). Used to detect internal structure a band
#: must not flatten.
QUANTIFIER_LEAD: Final[frozenset[str]] = frozenset(
    {"all", "every", "each", "no", "some", "none", "any", "most"}
)

#: What ``member.py`` recognizes beyond :data:`QUANTIFIER_LEAD`.
#:
#: This is a **provenance grouping, not a linguistic category** — it mixes
#: determiners (``few``, ``many``, ``both``) with pronouns (``everyone``,
#: ``nothing``). It is named for what it is (the delta) rather than given a
#: category name it would not earn. Splitting it into real categories needs
#: a linguistic decision, not a refactor.
QUANTIFIER_NON_LEAD: Final[frozenset[str]] = frozenset(
    {
        "few", "many", "several", "both", "either", "neither",
        "everyone", "everybody", "everything",
        "someone", "somebody", "something",
        "anyone", "anybody", "anything",
        "nobody", "nothing",
    }
)

#: Every quantifier token ``member.py`` refuses to lower. Measured to be a
#: strict superset of :data:`QUANTIFIER_LEAD`, so it is derived rather than
#: re-listed: the two sets agreed on all 8 shared tokens and can no longer
#: disagree.
QUANTIFIER_TOKENS: Final[frozenset[str]] = QUANTIFIER_LEAD | QUANTIFIER_NON_LEAD

#: Quantifiers that force **plural agreement** on the subject and verb.
#:
#: A genuinely different fact from :data:`QUANTIFIER_LEAD`, not a divergent
#: copy of it. ``every`` and ``each`` are quantifier-leads yet take a
#: *singular* noun by English rule ("each dog is"), so their absence here is
#: correct; ``few``/``many``/``several``/``various`` take plurals but cannot
#: lead a categorical reading. Overlap is 4 of 8 in each direction, which is
#: what made these look like a divergence in the §1.3 count.
PLURAL_QUANTIFIERS: Final[frozenset[str]] = frozenset(
    {"all", "some", "many", "few", "most", "several", "various", "no"}
)

# --------------------------------------------------------------------------
# Predicate display
# --------------------------------------------------------------------------

#: Machine predicate id -> human display phrase. Was two byte-identical
#: 26-entry copies: ``semantic_templates.py::_PREDICATE_HUMANIZE`` (the
#: serving writer) and ``templates.py::_PREDICATE_DISPLAY`` (the eval-only
#: realizer).
PREDICATE_DISPLAY: Final[dict[str, str]] = {
    "is_defined_as": "is defined as",
    "is_caused_by": "is caused by",
    "has_steps": "has the following steps",
    "contrasts_with": "contrasts with",
    "corrects": "corrects",
    "recalls": "recalls",
    "is_verified_as": "is verified as",
    "addresses": "addresses",
    "defines": "defines",
    "means": "means",
    "grounds": "grounds",
    "supports": "supports",
    "causes": "causes",
    "reveals": "reveals",
    "precedes": "precedes",
    "follows": "follows",
    "belongs_to": "belongs to",
    "answers": "answers",
    "is_grounded_in": "is grounded in",
    "is_distinguished_from": "is distinguished from",
    "implies": "implies",
    "entails": "entails",
    "requires": "requires",
    "verifies": "verifies",
    "evidences": "evidences",
    "orders": "orders",
}

#: The discourse planner's two-entry display table.
#:
#: **Recorded divergence (Phase 2A decision).** Two deliberate differences
#: from :data:`PREDICATE_DISPLAY`, both preserved:
#:
#: 1. ``is_defined_as`` renders as ``"is"``, not ``"is defined as"``. Inside
#:    a flowing paragraph the short copula reads as prose; the long form
#:    reads as a schema dump. This is a register choice.
#: 2. The table is **two entries, not 26**, on purpose. Every other
#:    predicate falls through to ``predicate.replace("_", " ")``. Widening it
#:    to the full table would silently change paragraph rendering for 24
#:    predicates, so scope is preserved exactly.
#:
#: ``belongs_to`` agreed with :data:`PREDICATE_DISPLAY` and is therefore
#: derived from it — the one entry that can drift no longer can.
DISCOURSE_PREDICATE_DISPLAY: Final[dict[str, str]] = {
    "is_defined_as": "is",
    "belongs_to": PREDICATE_DISPLAY["belongs_to"],
}

# --------------------------------------------------------------------------
# Number morphology — two inverse maps, not two divergent copies
# --------------------------------------------------------------------------
#
# §1.3 of the plan counted "irregular plurals: 3 copies, all three diverge."
# Measured against source, that count conflates two directions:
#
#   * ``templates.py``  singular -> plural  (a PLURALIZER, 25 entries)
#   * ``member.py``     plural -> singular  (a SINGULARIZER, 29 entries)
#   * ``reader.py``     plural -> singular  (a SINGULARIZER, 8 entries)
#
# and the two singularizers do **not** contradict each other: reader's 8
# entries are a *strict subset* of member's 29 (set difference in the
# reader's favour is empty). So there is no disagreement to arbitrate here —
# only a coverage asymmetry, plus one direction's knowledge being
# unavailable in the other direction.

#: Singular -> plural. The pluralizer, from ``templates.py``.
IRREGULAR_PLURALS: Final[dict[str, str]] = {
    "child": "children", "ox": "oxen", "foot": "feet", "tooth": "teeth",
    "man": "men", "woman": "women", "person": "people",
    "mouse": "mice", "louse": "lice", "goose": "geese",
    # invariant
    "sheep": "sheep", "fish": "fish", "deer": "deer", "moose": "moose",
    "series": "series", "species": "species",
    # latin/greek-origin domain vocabulary
    "datum": "data", "criterion": "criteria", "phenomenon": "phenomena",
    "analysis": "analyses", "axis": "axes", "basis": "bases",
    "thesis": "theses", "hypothesis": "hypotheses",
    "mitochondrion": "mitochondria",
    # Added in Phase 2B so the two directions are mutual inverses. The
    # singularizer knew cacti/fungi/dice; without these the pluralizer
    # produced "cactuses"/"funguses"/"dies" and CORE could read a word it
    # could not write back.
    "cactus": "cacti", "fungus": "fungi", "die": "dice",
}

#: Plural -> singular, including invariants that map to themselves. The
#: singularizer, from ``member.py``, which is the widest number table CORE
#: has. Consulted BEFORE any suffix rule: if either token of a candidate
#: pair appears here, this table is the only authority for that pair — which
#: is what keeps ``species``/``specie`` and ``news``/``new`` unlinked.
IRREGULAR_SINGULARS: Final[dict[str, str]] = {
    "men": "man", "women": "woman", "people": "person",
    "children": "child", "mice": "mouse", "geese": "goose",
    "feet": "foot", "teeth": "tooth", "oxen": "ox",
    "wolves": "wolf", "knives": "knife", "lives": "life",
    "leaves": "leaf", "halves": "half", "elves": "elf",
    "loaves": "loaf", "thieves": "thief", "cacti": "cactus",
    "fungi": "fungus", "dice": "die",
    # invariants (plural == singular)
    "sheep": "sheep", "fish": "fish", "deer": "deer",
    "species": "species", "series": "series", "means": "means",
    "offspring": "offspring", "aircraft": "aircraft", "news": "news",
}

#: Nouns whose plural equals their singular, **derived** from the invariant
#: rows of :data:`IRREGULAR_SINGULARS` (the rows where key == value).
#:
#: Derived rather than re-listed because a hand-written second list is exactly
#: how the two directions drifted apart in the first place: before Phase 2B the
#: pluralizer lacked ``aircraft``/``means``/``offspring`` and produced
#: "aircrafts", "meanses", "offsprings" while the singularizer knew all three.
#: ``news`` was saved only by coincidence — it is also a mass noun.
INVARIANT_NUMBER: Final[frozenset[str]] = frozenset(
    plural for plural, singular in IRREGULAR_SINGULARS.items() if plural == singular
)

#: Uncountable nouns — they take a quantifier but stay singular, so
#: ``all evidence`` is right and ``all evidences`` is wrong. The verb still
#: agrees singular ("all evidence supports truth"), which the categorical
#: renderer does **not** yet handle; see the Phase 2B note in
#: ``proof_chain/render.py``.
#:
#: Moved here in Phase 2B: it was a single copy in ``templates.py``, but the
#: pluralizer needs it and the pluralizer now serves, so the table has to be
#: reachable from both without a second literal. Covers the abstract/epistemic
#: vocabulary in ``en_core_cognition_v1`` plus common English mass nouns.
MASS_NOUNS: Final[frozenset[str]] = frozenset(
    {
        # epistemic / abstract (the seed-pack vocabulary)
        "evidence", "wisdom", "knowledge", "truth", "light", "darkness",
        "information", "data", "music", "art", "literature", "philosophy",
        "courage", "patience", "love", "hope", "fear", "grace",
        "meaning", "purpose", "beauty", "justice", "freedom",
        # physical mass
        "water", "air", "fire", "earth", "sand", "rain", "snow", "ice",
        "wood", "metal", "gold", "silver", "iron", "stone",
        "blood", "flesh", "bone",
        # collective / continuous
        "weather", "traffic", "furniture", "luggage", "advice",
        "equipment", "machinery", "scenery", "money", "news",
        "research", "progress", "feedback",
    }
)