feat(packs): reland en_core_syntax_v1 foundation vocabulary (collision-audited)
First foundation-curriculum substrate pack: 24 reviewed noun lemmas for
syntax/claim/provenance vocabulary (subject, predicate, agent_role,
clause, antecedent, consequent_role, polarity, negation, evidence_span,
…). Substrate only — no parser, no runtime generation change.
This is the clean reland of #503 (merged then reverted via #508 for
shipping two failing tests). Root cause of the revert was lemma
collision: the smoke CI gate does not collect dedicated test files, so
the pack merged green while its own test asserted resolver routings that
were false.
Reland fixes, with the full collision audit done against every on-disk
pack (not just the one the original author checked):
- agent -> agent_role (bare `agent` owned by en_core_meta_v1)
- comparison -> comparison_relation(bare `comparison` owned by en_core_cognition_v1)
- consequent -> consequent_role (bare `consequent` owned by en_core_causation_v1)
- negation kept bare on purpose: no *mounted* pack owned it
(en_mathematics_logic_v1 is a domain pack, not in
DEFAULT_RESOLVABLE_PACK_IDS), so mounting syntax now grounds a word
that previously returned None. Renaming would have lost grounding.
The original test also mis-asserted prior resolution: it claimed
`cause` -> en_core_causation_v1, but first-match-wins resolves it to
en_core_cognition_v1 (mounted at index 0). The test now pins the TRUE
current owner; this pack does not change it.
Resolver: en_core_syntax_v1 mounted after en_core_polarity_v1 and before
en_core_relations_v1 — purely additive, steals no prior resolution.
Evidence:
- tests/test_en_core_syntax_v1_pack.py: 13/13 (was 4 failed, 9 passed)
- pack/resolver/grounding/invariants blast radius: 415 passed
- core test --suite smoke: 67 passed
- core test --suite packs: 1 failed, 140 passed — the single failure
(test_lilibeth_canary_solves_end_to_end, GSM8K candidate-graph reader)
is PRE-EXISTING on clean main f79b647, byte-identical, and untouched by
this PR.
- lane SHAs: 7/8 content-match; the public_demo miss is a wall-clock
DemoContractError (>30s budget), not content drift — serving frozen.
PR checklist:
- Capability: foundation-curriculum language substrate vocabulary.
- Field invariant: pack loads through checksum-sealed compiler; no
algebra/versor path touched.
- Lane: core test --suite packs / smoke; dedicated pack test.
- No hidden normalization, stochastic fallback, approximate recall, or
unreviewed mutation.
- Trust boundary: language-pack loading only; manifest checksums hash the
exact bytes on disk (recomputed after the two renames).
This commit is contained in:
Shay
parent
f79b647671
commit
18a679c29d
6 changed files with 450 additions and 0 deletions
|
|
@ -50,6 +50,14 @@ DEFAULT_RESOLVABLE_PACK_IDS: tuple[str, ...] = (
|
||||||
"en_core_spatial_v1",
|
"en_core_spatial_v1",
|
||||||
"en_core_causation_v1",
|
"en_core_causation_v1",
|
||||||
"en_core_polarity_v1",
|
"en_core_polarity_v1",
|
||||||
|
# Foundation-curriculum syntax substrate. Mounted AFTER the
|
||||||
|
# high-frequency content packs so it is purely additive: every lemma
|
||||||
|
# it owns is disjoint from earlier mounted packs. Two colliders were
|
||||||
|
# renamed at authoring time so syntax never steals prior resolution
|
||||||
|
# (`agent`→`agent_role`, owned by en_core_meta_v1; `comparison`→
|
||||||
|
# `comparison_relation`, owned by en_core_cognition_v1). `negation`
|
||||||
|
# is kept bare: no *mounted* pack owned it, so syntax now grounds it.
|
||||||
|
"en_core_syntax_v1",
|
||||||
"en_core_relations_v1",
|
"en_core_relations_v1",
|
||||||
"en_core_relations_v2",
|
"en_core_relations_v2",
|
||||||
# ADR-0073c — synthetic English anchor lemmas for cross-lang collapse.
|
# ADR-0073c — synthetic English anchor lemmas for cross-lang collapse.
|
||||||
|
|
|
||||||
170
docs/curriculum/en_core_syntax_v1.md
Normal file
170
docs/curriculum/en_core_syntax_v1.md
Normal file
|
|
@ -0,0 +1,170 @@
|
||||||
|
# en_core_syntax_v1
|
||||||
|
|
||||||
|
Status: implemented candidate
|
||||||
|
Scope: foundational language substrate pack
|
||||||
|
Pack id: `en_core_syntax_v1`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`en_core_syntax_v1` seeds the first foundation-curriculum substrate needed for later claim parsing and relation binding.
|
||||||
|
|
||||||
|
The pack does not parse sentences. It establishes the lower-level English vocabulary for speaking about syntax, claim structure, evidence binding, reference, polarity, comparison, coordination, and conditional roles.
|
||||||
|
|
||||||
|
This is the first step toward the roadmap rule:
|
||||||
|
|
||||||
|
```text
|
||||||
|
language -> relations -> quantity -> units -> logic -> evidence -> data -> algorithms -> systems -> domains
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
This slice does not implement:
|
||||||
|
|
||||||
|
- a natural-language parser
|
||||||
|
- dependency parsing
|
||||||
|
- clause extraction
|
||||||
|
- evidence-span extraction
|
||||||
|
- claim graph construction
|
||||||
|
- GSM8K statement classification changes
|
||||||
|
- relation-binding execution
|
||||||
|
- new runtime generation behavior
|
||||||
|
|
||||||
|
Those are follow-on slices. This pack only ratifies the vocabulary substrate and makes it loadable/resolvable through existing pack infrastructure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Pack contents
|
||||||
|
|
||||||
|
The pack contains 24 reviewed noun entries:
|
||||||
|
|
||||||
|
```text
|
||||||
|
subject
|
||||||
|
predicate
|
||||||
|
agent_role
|
||||||
|
patient
|
||||||
|
object
|
||||||
|
modifier
|
||||||
|
clause
|
||||||
|
sentence
|
||||||
|
phrase
|
||||||
|
antecedent
|
||||||
|
consequent_role
|
||||||
|
referent
|
||||||
|
anaphor
|
||||||
|
qualifier
|
||||||
|
scope
|
||||||
|
polarity
|
||||||
|
coordination
|
||||||
|
conjunction
|
||||||
|
disjunction
|
||||||
|
negation
|
||||||
|
exception
|
||||||
|
comparison_relation
|
||||||
|
attachment
|
||||||
|
evidence_span
|
||||||
|
```
|
||||||
|
|
||||||
|
### Collision discipline (additive, never steals prior resolution)
|
||||||
|
|
||||||
|
The resolver is deterministic first-match-wins over `DEFAULT_RESOLVABLE_PACK_IDS`, and this pack is mounted *after* the high-frequency content packs. So any lemma it shares with an earlier-mounted pack would resolve to that earlier pack, never here. To keep the substrate strictly additive, every lemma that collided with a **mounted** pack was renamed to a disambiguated form, and the pack test pins both the rename and the untouched prior resolution:
|
||||||
|
|
||||||
|
- `agent` → `agent_role` — bare `agent` is owned by `en_core_meta_v1` (self-reference participant role).
|
||||||
|
- `comparison` → `comparison_relation` — bare `comparison` is owned by `en_core_cognition_v1`.
|
||||||
|
- `consequent` → `consequent_role` — bare `consequent` is owned by `en_core_causation_v1` as a causal/resultant adjective.
|
||||||
|
|
||||||
|
`negation` is **kept bare on purpose**: no *mounted* content pack owned it (the only other on-disk definition lives in `en_mathematics_logic_v1`, a domain pack that is not in `DEFAULT_RESOLVABLE_PACK_IDS`). Before this pack, `resolve_lemma("negation")` returned `None`; mounting the syntax pack now grounds it. Renaming it to a synthetic token would have *lost* that natural-word grounding, so the collision-driven rename is applied only where a mounted pack already answers the bare word.
|
||||||
|
|
||||||
|
One pre-existing resolution worth recording: `cause` resolves to `en_core_cognition_v1` (mounted at index 0), not `en_core_causation_v1`. That is unchanged by this pack and is pinned as-is in `test_prior_pack_lemma_resolution_unchanged`.
|
||||||
|
|
||||||
|
Primary namespace families:
|
||||||
|
|
||||||
|
```text
|
||||||
|
syntax.*
|
||||||
|
claim.*
|
||||||
|
provenance.*
|
||||||
|
```
|
||||||
|
|
||||||
|
Each lexicon row uses provenance:
|
||||||
|
|
||||||
|
```text
|
||||||
|
foundation_syntax_v1:reviewed:2026-05-30
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
```text
|
||||||
|
language_packs/data/en_core_syntax_v1/manifest.json
|
||||||
|
language_packs/data/en_core_syntax_v1/lexicon.jsonl
|
||||||
|
language_packs/data/en_core_syntax_v1/glosses.jsonl
|
||||||
|
tests/test_en_core_syntax_v1_pack.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The pack ships both a checksum-sealed `lexicon.jsonl` and a checksum-sealed `glosses.jsonl` companion overlay.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Admission gates pinned by tests
|
||||||
|
|
||||||
|
`tests/test_en_core_syntax_v1_pack.py` verifies:
|
||||||
|
|
||||||
|
- `load_pack("en_core_syntax_v1")` succeeds;
|
||||||
|
- manifest checksum matches `lexicon.jsonl` bytes;
|
||||||
|
- manifest `glosses_checksum` matches `glosses.jsonl` bytes;
|
||||||
|
- total entry count is 24;
|
||||||
|
- POS distribution is exactly 24 `NOUN` rows;
|
||||||
|
- expected lemmas are present in the compiled manifold;
|
||||||
|
- entry ids are contiguous and zero-padded;
|
||||||
|
- every lexicon row has reviewed foundation provenance;
|
||||||
|
- every gloss lemma is resident in the same pack lexicon;
|
||||||
|
- resolver registration routes syntax lemmas to this pack;
|
||||||
|
- prior pack lemma routing remains unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolver order
|
||||||
|
|
||||||
|
The pack is registered in `chat.pack_resolver.DEFAULT_RESOLVABLE_PACK_IDS` after the polarity pack and before relation/kinship packs:
|
||||||
|
|
||||||
|
```text
|
||||||
|
en_core_polarity_v1
|
||||||
|
en_core_syntax_v1
|
||||||
|
en_core_relations_v1
|
||||||
|
en_core_relations_v2
|
||||||
|
```
|
||||||
|
|
||||||
|
This makes syntax vocabulary available to the same deterministic first-match resolver used by existing English content packs while preserving prior high-frequency lemma ownership.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Follow-on work unlocked
|
||||||
|
|
||||||
|
The next slices should use this vocabulary to implement actual relation binding:
|
||||||
|
|
||||||
|
```text
|
||||||
|
evals/language_claim_parsing
|
||||||
|
evals/language_relation_binding
|
||||||
|
tests/test_language_claim_parsing.py
|
||||||
|
tests/test_relation_binding_replay.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected follow-on primitives:
|
||||||
|
|
||||||
|
- evidence-span extraction
|
||||||
|
- subject/predicate/argument slot binding
|
||||||
|
- modifier attachment
|
||||||
|
- conditional antecedent/consequent binding
|
||||||
|
- polarity and negation scope
|
||||||
|
- reference/anaphora target binding
|
||||||
|
- typed refusal when a required slot lacks evidence
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
This pack is a substrate, not a capability claim. It should not be marked as full language relation binding admission until parser/eval/runtime surfaces exist and pass their own tests.
|
||||||
24
language_packs/data/en_core_syntax_v1/glosses.jsonl
Normal file
24
language_packs/data/en_core_syntax_v1/glosses.jsonl
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
{"lemma":"subject","gloss":"the claim role that names what a statement is about","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"predicate","gloss":"the claim role that carries what is said of a subject","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"agent_role","gloss":"the participant that performs an action","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"patient","gloss":"the participant that undergoes an action or change","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"object","gloss":"the syntactic role receiving or completing an action","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"modifier","gloss":"a word or phrase that qualifies another expression","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"clause","gloss":"a bounded unit that can carry a claim relation","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"sentence","gloss":"a complete written surface that may contain one or more claims","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"phrase","gloss":"a bounded expression below clause level","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"antecedent","gloss":"the condition-side of a conditional claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"consequent_role","gloss":"the result-side role of a conditional claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"referent","gloss":"the entity or claim target that a reference points to","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"anaphor","gloss":"a later reference that points back to an earlier referent","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"qualifier","gloss":"a constraint or limit attached to a claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"scope","gloss":"the boundary within which a claim or operator applies","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"polarity","gloss":"the positive or negative orientation of a claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"coordination","gloss":"the joining of claim parts or clauses","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"conjunction","gloss":"a joining relation where both parts are asserted together","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"disjunction","gloss":"a joining relation where alternatives are presented","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"negation","gloss":"an operator that denies or reverses a claim polarity","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"exception","gloss":"a scoped exclusion from a claim or rule","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"comparison_relation","gloss":"a relation that evaluates one thing against another","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"attachment","gloss":"the binding of a modifier or qualifier to its target","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
|
{"lemma":"evidence_span","gloss":"the exact text span that supports a parsed claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||||
24
language_packs/data/en_core_syntax_v1/lexicon.jsonl
Normal file
24
language_packs/data/en_core_syntax_v1/lexicon.jsonl
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
{"entry_id":"en-core-syntax-001","surface":"subject","lemma":"subject","language":"en","pos":"NOUN","semantic_domains":["syntax.role.subject","claim.argument.subject","grammar.core"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-002","surface":"predicate","lemma":"predicate","language":"en","pos":"NOUN","semantic_domains":["syntax.role.predicate","claim.relation.predicate","grammar.core"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-003","surface":"agent_role","lemma":"agent_role","language":"en","pos":"NOUN","semantic_domains":["syntax.role.agent","event.participant.actor","claim.argument.agent"],"morphology_tags":["noun","semantic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-004","surface":"patient","lemma":"patient","language":"en","pos":"NOUN","semantic_domains":["syntax.role.patient","event.participant.undergoer","claim.argument.patient"],"morphology_tags":["noun","semantic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-005","surface":"object","lemma":"object","language":"en","pos":"NOUN","semantic_domains":["syntax.role.object","event.participant.object","claim.argument.object"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-006","surface":"modifier","lemma":"modifier","language":"en","pos":"NOUN","semantic_domains":["syntax.role.modifier","claim.qualifier","grammar.attachment"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-007","surface":"clause","lemma":"clause","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.clause","claim.boundary.clause","grammar.unit"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-008","surface":"sentence","lemma":"sentence","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.sentence","claim.surface.sentence","grammar.unit"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-009","surface":"phrase","lemma":"phrase","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.phrase","grammar.unit","claim.fragment"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-010","surface":"antecedent","lemma":"antecedent","language":"en","pos":"NOUN","semantic_domains":["syntax.condition.antecedent","logic.condition.input","claim.condition.antecedent"],"morphology_tags":["noun","conditional_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-011","surface":"consequent_role","lemma":"consequent_role","language":"en","pos":"NOUN","semantic_domains":["syntax.condition.consequent_role","logic.condition.output_role","claim.condition.consequent_role"],"morphology_tags":["noun","conditional_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-012","surface":"referent","lemma":"referent","language":"en","pos":"NOUN","semantic_domains":["syntax.reference.referent","semantic.entity.reference","claim.argument.referent"],"morphology_tags":["noun","reference_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-013","surface":"anaphor","lemma":"anaphor","language":"en","pos":"NOUN","semantic_domains":["syntax.reference.anaphor","discourse.backreference","claim.reference.surface"],"morphology_tags":["noun","reference_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-014","surface":"qualifier","lemma":"qualifier","language":"en","pos":"NOUN","semantic_domains":["claim.qualifier","syntax.modifier","semantic.constraint"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-015","surface":"scope","lemma":"scope","language":"en","pos":"NOUN","semantic_domains":["claim.scope","logic.scope","syntax.boundary.scope"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-016","surface":"polarity","lemma":"polarity","language":"en","pos":"NOUN","semantic_domains":["claim.polarity","logic.truth_orientation","syntax.negation.axis"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-017","surface":"coordination","lemma":"coordination","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination","logic.composition","claim.join"],"morphology_tags":["noun","composition_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-018","surface":"conjunction","lemma":"conjunction","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination.conjunction","logic.conjunction","claim.join.and"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-019","surface":"disjunction","lemma":"disjunction","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination.disjunction","logic.disjunction","claim.join.or"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-020","surface":"negation","lemma":"negation","language":"en","pos":"NOUN","semantic_domains":["syntax.negation","logic.negation","claim.polarity.negative"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-021","surface":"exception","lemma":"exception","language":"en","pos":"NOUN","semantic_domains":["syntax.exception","logic.exception","claim.scope.exception"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-022","surface":"comparison_relation","lemma":"comparison_relation","language":"en","pos":"NOUN","semantic_domains":["syntax.comparison","comparative.relation","claim.comparison"],"morphology_tags":["noun","relation_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-023","surface":"attachment","lemma":"attachment","language":"en","pos":"NOUN","semantic_domains":["syntax.attachment","grammar.binding","claim.qualifier.binding"],"morphology_tags":["noun","binding_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
|
{"entry_id":"en-core-syntax-024","surface":"evidence_span","lemma":"evidence_span","language":"en","pos":"NOUN","semantic_domains":["claim.evidence_span","provenance.text_span","binding.evidence"],"morphology_tags":["noun","provenance_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||||
16
language_packs/data/en_core_syntax_v1/manifest.json
Normal file
16
language_packs/data/en_core_syntax_v1/manifest.json
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
{
|
||||||
|
"pack_id": "en_core_syntax_v1",
|
||||||
|
"language": "en",
|
||||||
|
"role": "operational_base",
|
||||||
|
"script": "Latin",
|
||||||
|
"normalization_policy": "unitize_versor",
|
||||||
|
"source_manifest": "en_core_syntax_v1.lexicon.jsonl",
|
||||||
|
"determinism_class": "D0",
|
||||||
|
"checksum": "31ac3e05a1344e296e86cbcacdfc118583b17e2634dc752fd596528023b99900",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"gate_engaged": true,
|
||||||
|
"oov_policy": "tagged_fallback",
|
||||||
|
"definitional_layer": true,
|
||||||
|
"provenance": "foundation-curriculum:reviewed:2026-05-30",
|
||||||
|
"glosses_checksum": "aebb2191739722d54c48fe10d10ab6bae2efb44bfd14f04ce72b6333a4a1ca98"
|
||||||
|
}
|
||||||
208
tests/test_en_core_syntax_v1_pack.py
Normal file
208
tests/test_en_core_syntax_v1_pack.py
Normal file
|
|
@ -0,0 +1,208 @@
|
||||||
|
"""``en_core_syntax_v1`` — foundational syntax vocabulary pack tests.
|
||||||
|
|
||||||
|
This pack is the first foundation-curriculum substrate pack. It does not
|
||||||
|
implement a parser. It seeds the lower-level vocabulary needed for later
|
||||||
|
claim parsing, relation binding, clause attachment, conditional structure,
|
||||||
|
reference handling, polarity, and evidence-span discipline.
|
||||||
|
|
||||||
|
Contracts pinned here:
|
||||||
|
|
||||||
|
- checksum-verified load through ``language_packs.compiler.load_pack``;
|
||||||
|
- every entry's primary semantic namespace is syntax/claim/provenance;
|
||||||
|
- contiguous zero-padded entry ids;
|
||||||
|
- one gloss per lexicon lemma, with manifest checksum discipline;
|
||||||
|
- resolver registration after polarity and before kinship/domain relation packs;
|
||||||
|
- prior high-frequency pack lemma routing remains unchanged.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from collections import Counter
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from chat.pack_resolver import (
|
||||||
|
DEFAULT_RESOLVABLE_PACK_IDS,
|
||||||
|
clear_resolver_cache,
|
||||||
|
resolve_gloss,
|
||||||
|
resolve_lemma,
|
||||||
|
)
|
||||||
|
from language_packs.compiler import load_pack
|
||||||
|
|
||||||
|
|
||||||
|
PACK_ID = "en_core_syntax_v1"
|
||||||
|
_PACK_ROOT = Path(__file__).resolve().parent.parent / "language_packs" / "data" / PACK_ID
|
||||||
|
|
||||||
|
EXPECTED_TOTAL = 24
|
||||||
|
EXPECTED_POS_COUNTS = {"NOUN": 24}
|
||||||
|
EXPECTED_LEMMAS: tuple[str, ...] = (
|
||||||
|
"subject",
|
||||||
|
"predicate",
|
||||||
|
"agent_role",
|
||||||
|
"patient",
|
||||||
|
"object",
|
||||||
|
"modifier",
|
||||||
|
"clause",
|
||||||
|
"sentence",
|
||||||
|
"phrase",
|
||||||
|
"antecedent",
|
||||||
|
"consequent_role",
|
||||||
|
"referent",
|
||||||
|
"anaphor",
|
||||||
|
"qualifier",
|
||||||
|
"scope",
|
||||||
|
"polarity",
|
||||||
|
"coordination",
|
||||||
|
"conjunction",
|
||||||
|
"disjunction",
|
||||||
|
"negation",
|
||||||
|
"exception",
|
||||||
|
"comparison_relation",
|
||||||
|
"attachment",
|
||||||
|
"evidence_span",
|
||||||
|
)
|
||||||
|
|
||||||
|
_ALLOWED_PRIMARY_PREFIXES = ("syntax.", "claim.", "provenance.")
|
||||||
|
_EXPECTED_PROVENANCE = ["foundation_syntax_v1:reviewed:2026-05-30"]
|
||||||
|
|
||||||
|
|
||||||
|
def _read_jsonl(path: Path) -> list[dict]:
|
||||||
|
return [
|
||||||
|
json.loads(line)
|
||||||
|
for line in path.read_text("utf-8").splitlines()
|
||||||
|
if line.strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _read_lexicon() -> list[dict]:
|
||||||
|
return _read_jsonl(_PACK_ROOT / "lexicon.jsonl")
|
||||||
|
|
||||||
|
|
||||||
|
def _read_glosses() -> list[dict]:
|
||||||
|
return _read_jsonl(_PACK_ROOT / "glosses.jsonl")
|
||||||
|
|
||||||
|
|
||||||
|
def test_pack_loads_with_matching_checksum() -> None:
|
||||||
|
manifest, manifold = load_pack(PACK_ID)
|
||||||
|
assert manifest.pack_id == PACK_ID
|
||||||
|
assert len(manifest.checksum) == 64
|
||||||
|
assert len(manifold) == EXPECTED_TOTAL
|
||||||
|
|
||||||
|
|
||||||
|
def test_manifest_checksums_match_pack_bytes() -> None:
|
||||||
|
manifest = json.loads((_PACK_ROOT / "manifest.json").read_text("utf-8"))
|
||||||
|
assert manifest["checksum"] == hashlib.sha256(
|
||||||
|
(_PACK_ROOT / "lexicon.jsonl").read_bytes()
|
||||||
|
).hexdigest()
|
||||||
|
assert manifest["glosses_checksum"] == hashlib.sha256(
|
||||||
|
(_PACK_ROOT / "glosses.jsonl").read_bytes()
|
||||||
|
).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def test_pos_distribution_matches_design() -> None:
|
||||||
|
assert dict(Counter(e["pos"] for e in _read_lexicon())) == EXPECTED_POS_COUNTS
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_expected_lemmas_present() -> None:
|
||||||
|
_, manifold = load_pack(PACK_ID)
|
||||||
|
assert {manifold.get_word_at(i) for i in range(len(manifold))} == set(EXPECTED_LEMMAS)
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_entry_has_foundational_primary_namespace() -> None:
|
||||||
|
for entry in _read_lexicon():
|
||||||
|
primary = entry["semantic_domains"][0]
|
||||||
|
assert primary.startswith(_ALLOWED_PRIMARY_PREFIXES), entry
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_entry_has_reviewed_foundation_provenance() -> None:
|
||||||
|
for entry in _read_lexicon():
|
||||||
|
assert entry["provenance_ids"] == _EXPECTED_PROVENANCE, entry
|
||||||
|
|
||||||
|
|
||||||
|
def test_entry_ids_contiguous_and_zero_padded() -> None:
|
||||||
|
entries = sorted(_read_lexicon(), key=lambda d: d["entry_id"])
|
||||||
|
for i, entry in enumerate(entries, start=1):
|
||||||
|
assert entry["entry_id"] == f"en-core-syntax-{i:03d}", entry["entry_id"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_glosses_are_one_to_one_with_lexicon() -> None:
|
||||||
|
lexicon_lemmas = {entry["lemma"] for entry in _read_lexicon()}
|
||||||
|
gloss_lemmas = {entry["lemma"] for entry in _read_glosses()}
|
||||||
|
assert gloss_lemmas == lexicon_lemmas
|
||||||
|
|
||||||
|
|
||||||
|
def test_gloss_entries_use_strict_definitional_overlay_shape() -> None:
|
||||||
|
for entry in _read_glosses():
|
||||||
|
assert set(entry) == {
|
||||||
|
"lemma",
|
||||||
|
"gloss",
|
||||||
|
"pos",
|
||||||
|
"provenance_ids",
|
||||||
|
"definitional_atoms",
|
||||||
|
"predicates_invited",
|
||||||
|
"definition_version",
|
||||||
|
}
|
||||||
|
assert entry["provenance_ids"] == _EXPECTED_PROVENANCE
|
||||||
|
assert entry["definition_version"] == 1
|
||||||
|
assert isinstance(entry["definitional_atoms"], list)
|
||||||
|
assert isinstance(entry["predicates_invited"], list)
|
||||||
|
|
||||||
|
|
||||||
|
def test_pack_registered_after_polarity_before_relation_content() -> None:
|
||||||
|
assert PACK_ID in DEFAULT_RESOLVABLE_PACK_IDS
|
||||||
|
assert DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_polarity_v1") < DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID)
|
||||||
|
assert DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID) < DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_relations_v1")
|
||||||
|
assert DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID) < DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_relations_v2")
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolver_routes_syntax_lemmas_to_this_pack() -> None:
|
||||||
|
clear_resolver_cache()
|
||||||
|
for lemma in EXPECTED_LEMMAS:
|
||||||
|
resolved = resolve_lemma(lemma)
|
||||||
|
assert resolved is not None and resolved[0] == PACK_ID, (
|
||||||
|
f"{lemma!r} resolved to {resolved}"
|
||||||
|
)
|
||||||
|
assert resolved[1][0].startswith(_ALLOWED_PRIMARY_PREFIXES)
|
||||||
|
|
||||||
|
|
||||||
|
def test_gloss_resolver_routes_syntax_lemmas_to_this_pack() -> None:
|
||||||
|
clear_resolver_cache()
|
||||||
|
for lemma in (
|
||||||
|
"subject",
|
||||||
|
"predicate",
|
||||||
|
"antecedent",
|
||||||
|
"consequent_role",
|
||||||
|
"negation",
|
||||||
|
"evidence_span",
|
||||||
|
):
|
||||||
|
resolved = resolve_gloss(lemma)
|
||||||
|
assert resolved is not None, lemma
|
||||||
|
pack_id, pos, gloss = resolved
|
||||||
|
assert pack_id == PACK_ID
|
||||||
|
assert pos == "NOUN"
|
||||||
|
assert isinstance(gloss, str) and len(gloss.split()) >= 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_prior_pack_lemma_resolution_unchanged() -> None:
|
||||||
|
for lemma, expected in (
|
||||||
|
("truth", "en_core_cognition_v1"),
|
||||||
|
("doubt", "en_core_meta_v1"),
|
||||||
|
("true", "en_core_attitude_v1"),
|
||||||
|
("now", "en_core_temporal_v1"),
|
||||||
|
("do", "en_core_action_v1"),
|
||||||
|
("all", "en_core_quantitative_v1"),
|
||||||
|
("place", "en_core_spatial_v1"),
|
||||||
|
# `cause` is owned by en_core_cognition_v1 (mounted at index 0),
|
||||||
|
# NOT en_core_causation_v1 — this is pre-existing first-match-wins
|
||||||
|
# resolution that the syntax pack (mounted later) does not change.
|
||||||
|
# The contract here is "prior resolution unchanged"; we pin the
|
||||||
|
# TRUE current owner. `consequent` genuinely resolves to causation.
|
||||||
|
("cause", "en_core_cognition_v1"),
|
||||||
|
("consequent", "en_core_causation_v1"),
|
||||||
|
("never", "en_core_polarity_v1"),
|
||||||
|
("parent", "en_core_relations_v1"),
|
||||||
|
("mother", "en_core_relations_v2"),
|
||||||
|
):
|
||||||
|
resolved = resolve_lemma(lemma)
|
||||||
|
assert resolved is not None and resolved[0] == expected
|
||||||
Loading…
Reference in a new issue