Merge pull request #510 from AssetOverflow/feat/lang-foundation-reland
This commit is contained in:
Shay
GitHub
commit
497d1a9a22
6 changed files with 450 additions and 0 deletions
|
|
@ -50,6 +50,14 @@ DEFAULT_RESOLVABLE_PACK_IDS: tuple[str, ...] = (
|
|||
"en_core_spatial_v1",
|
||||
"en_core_causation_v1",
|
||||
"en_core_polarity_v1",
|
||||
# Foundation-curriculum syntax substrate. Mounted AFTER the
|
||||
# high-frequency content packs so it is purely additive: every lemma
|
||||
# it owns is disjoint from earlier mounted packs. Two colliders were
|
||||
# renamed at authoring time so syntax never steals prior resolution
|
||||
# (`agent`→`agent_role`, owned by en_core_meta_v1; `comparison`→
|
||||
# `comparison_relation`, owned by en_core_cognition_v1). `negation`
|
||||
# is kept bare: no *mounted* pack owned it, so syntax now grounds it.
|
||||
"en_core_syntax_v1",
|
||||
"en_core_relations_v1",
|
||||
"en_core_relations_v2",
|
||||
# ADR-0073c — synthetic English anchor lemmas for cross-lang collapse.
|
||||
|
|
|
|||
170
docs/curriculum/en_core_syntax_v1.md
Normal file
170
docs/curriculum/en_core_syntax_v1.md
Normal file
|
|
@ -0,0 +1,170 @@
|
|||
# en_core_syntax_v1
|
||||
|
||||
Status: implemented candidate
|
||||
Scope: foundational language substrate pack
|
||||
Pack id: `en_core_syntax_v1`
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
`en_core_syntax_v1` seeds the first foundation-curriculum substrate needed for later claim parsing and relation binding.
|
||||
|
||||
The pack does not parse sentences. It establishes the lower-level English vocabulary for speaking about syntax, claim structure, evidence binding, reference, polarity, comparison, coordination, and conditional roles.
|
||||
|
||||
This is the first step toward the roadmap rule:
|
||||
|
||||
```text
|
||||
language -> relations -> quantity -> units -> logic -> evidence -> data -> algorithms -> systems -> domains
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
This slice does not implement:
|
||||
|
||||
- a natural-language parser
|
||||
- dependency parsing
|
||||
- clause extraction
|
||||
- evidence-span extraction
|
||||
- claim graph construction
|
||||
- GSM8K statement classification changes
|
||||
- relation-binding execution
|
||||
- new runtime generation behavior
|
||||
|
||||
Those are follow-on slices. This pack only ratifies the vocabulary substrate and makes it loadable/resolvable through existing pack infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## Pack contents
|
||||
|
||||
The pack contains 24 reviewed noun entries:
|
||||
|
||||
```text
|
||||
subject
|
||||
predicate
|
||||
agent_role
|
||||
patient
|
||||
object
|
||||
modifier
|
||||
clause
|
||||
sentence
|
||||
phrase
|
||||
antecedent
|
||||
consequent_role
|
||||
referent
|
||||
anaphor
|
||||
qualifier
|
||||
scope
|
||||
polarity
|
||||
coordination
|
||||
conjunction
|
||||
disjunction
|
||||
negation
|
||||
exception
|
||||
comparison_relation
|
||||
attachment
|
||||
evidence_span
|
||||
```
|
||||
|
||||
### Collision discipline (additive, never steals prior resolution)
|
||||
|
||||
The resolver is deterministic first-match-wins over `DEFAULT_RESOLVABLE_PACK_IDS`, and this pack is mounted *after* the high-frequency content packs. So any lemma it shares with an earlier-mounted pack would resolve to that earlier pack, never here. To keep the substrate strictly additive, every lemma that collided with a **mounted** pack was renamed to a disambiguated form, and the pack test pins both the rename and the untouched prior resolution:
|
||||
|
||||
- `agent` → `agent_role` — bare `agent` is owned by `en_core_meta_v1` (self-reference participant role).
|
||||
- `comparison` → `comparison_relation` — bare `comparison` is owned by `en_core_cognition_v1`.
|
||||
- `consequent` → `consequent_role` — bare `consequent` is owned by `en_core_causation_v1` as a causal/resultant adjective.
|
||||
|
||||
`negation` is **kept bare on purpose**: no *mounted* content pack owned it (the only other on-disk definition lives in `en_mathematics_logic_v1`, a domain pack that is not in `DEFAULT_RESOLVABLE_PACK_IDS`). Before this pack, `resolve_lemma("negation")` returned `None`; mounting the syntax pack now grounds it. Renaming it to a synthetic token would have *lost* that natural-word grounding, so the collision-driven rename is applied only where a mounted pack already answers the bare word.
|
||||
|
||||
One pre-existing resolution worth recording: `cause` resolves to `en_core_cognition_v1` (mounted at index 0), not `en_core_causation_v1`. That is unchanged by this pack and is pinned as-is in `test_prior_pack_lemma_resolution_unchanged`.
|
||||
|
||||
Primary namespace families:
|
||||
|
||||
```text
|
||||
syntax.*
|
||||
claim.*
|
||||
provenance.*
|
||||
```
|
||||
|
||||
Each lexicon row uses provenance:
|
||||
|
||||
```text
|
||||
foundation_syntax_v1:reviewed:2026-05-30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files
|
||||
|
||||
```text
|
||||
language_packs/data/en_core_syntax_v1/manifest.json
|
||||
language_packs/data/en_core_syntax_v1/lexicon.jsonl
|
||||
language_packs/data/en_core_syntax_v1/glosses.jsonl
|
||||
tests/test_en_core_syntax_v1_pack.py
|
||||
```
|
||||
|
||||
The pack ships both a checksum-sealed `lexicon.jsonl` and a checksum-sealed `glosses.jsonl` companion overlay.
|
||||
|
||||
---
|
||||
|
||||
## Admission gates pinned by tests
|
||||
|
||||
`tests/test_en_core_syntax_v1_pack.py` verifies:
|
||||
|
||||
- `load_pack("en_core_syntax_v1")` succeeds;
|
||||
- manifest checksum matches `lexicon.jsonl` bytes;
|
||||
- manifest `glosses_checksum` matches `glosses.jsonl` bytes;
|
||||
- total entry count is 24;
|
||||
- POS distribution is exactly 24 `NOUN` rows;
|
||||
- expected lemmas are present in the compiled manifold;
|
||||
- entry ids are contiguous and zero-padded;
|
||||
- every lexicon row has reviewed foundation provenance;
|
||||
- every gloss lemma is resident in the same pack lexicon;
|
||||
- resolver registration routes syntax lemmas to this pack;
|
||||
- prior pack lemma routing remains unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Resolver order
|
||||
|
||||
The pack is registered in `chat.pack_resolver.DEFAULT_RESOLVABLE_PACK_IDS` after the polarity pack and before relation/kinship packs:
|
||||
|
||||
```text
|
||||
en_core_polarity_v1
|
||||
en_core_syntax_v1
|
||||
en_core_relations_v1
|
||||
en_core_relations_v2
|
||||
```
|
||||
|
||||
This makes syntax vocabulary available to the same deterministic first-match resolver used by existing English content packs while preserving prior high-frequency lemma ownership.
|
||||
|
||||
---
|
||||
|
||||
## Follow-on work unlocked
|
||||
|
||||
The next slices should use this vocabulary to implement actual relation binding:
|
||||
|
||||
```text
|
||||
evals/language_claim_parsing
|
||||
evals/language_relation_binding
|
||||
tests/test_language_claim_parsing.py
|
||||
tests/test_relation_binding_replay.py
|
||||
```
|
||||
|
||||
Expected follow-on primitives:
|
||||
|
||||
- evidence-span extraction
|
||||
- subject/predicate/argument slot binding
|
||||
- modifier attachment
|
||||
- conditional antecedent/consequent binding
|
||||
- polarity and negation scope
|
||||
- reference/anaphora target binding
|
||||
- typed refusal when a required slot lacks evidence
|
||||
|
||||
---
|
||||
|
||||
## Boundary
|
||||
|
||||
This pack is a substrate, not a capability claim. It should not be marked as full language relation binding admission until parser/eval/runtime surfaces exist and pass their own tests.
|
||||
24
language_packs/data/en_core_syntax_v1/glosses.jsonl
Normal file
24
language_packs/data/en_core_syntax_v1/glosses.jsonl
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{"lemma":"subject","gloss":"the claim role that names what a statement is about","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"predicate","gloss":"the claim role that carries what is said of a subject","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"agent_role","gloss":"the participant that performs an action","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"patient","gloss":"the participant that undergoes an action or change","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"object","gloss":"the syntactic role receiving or completing an action","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"modifier","gloss":"a word or phrase that qualifies another expression","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"clause","gloss":"a bounded unit that can carry a claim relation","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"sentence","gloss":"a complete written surface that may contain one or more claims","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"phrase","gloss":"a bounded expression below clause level","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"antecedent","gloss":"the condition-side of a conditional claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"consequent_role","gloss":"the result-side role of a conditional claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"referent","gloss":"the entity or claim target that a reference points to","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"anaphor","gloss":"a later reference that points back to an earlier referent","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"qualifier","gloss":"a constraint or limit attached to a claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"scope","gloss":"the boundary within which a claim or operator applies","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"polarity","gloss":"the positive or negative orientation of a claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"coordination","gloss":"the joining of claim parts or clauses","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"conjunction","gloss":"a joining relation where both parts are asserted together","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"disjunction","gloss":"a joining relation where alternatives are presented","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"negation","gloss":"an operator that denies or reverses a claim polarity","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"exception","gloss":"a scoped exclusion from a claim or rule","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"comparison_relation","gloss":"a relation that evaluates one thing against another","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"attachment","gloss":"the binding of a modifier or qualifier to its target","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
{"lemma":"evidence_span","gloss":"the exact text span that supports a parsed claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
||||
24
language_packs/data/en_core_syntax_v1/lexicon.jsonl
Normal file
24
language_packs/data/en_core_syntax_v1/lexicon.jsonl
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{"entry_id":"en-core-syntax-001","surface":"subject","lemma":"subject","language":"en","pos":"NOUN","semantic_domains":["syntax.role.subject","claim.argument.subject","grammar.core"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-002","surface":"predicate","lemma":"predicate","language":"en","pos":"NOUN","semantic_domains":["syntax.role.predicate","claim.relation.predicate","grammar.core"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-003","surface":"agent_role","lemma":"agent_role","language":"en","pos":"NOUN","semantic_domains":["syntax.role.agent","event.participant.actor","claim.argument.agent"],"morphology_tags":["noun","semantic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-004","surface":"patient","lemma":"patient","language":"en","pos":"NOUN","semantic_domains":["syntax.role.patient","event.participant.undergoer","claim.argument.patient"],"morphology_tags":["noun","semantic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-005","surface":"object","lemma":"object","language":"en","pos":"NOUN","semantic_domains":["syntax.role.object","event.participant.object","claim.argument.object"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-006","surface":"modifier","lemma":"modifier","language":"en","pos":"NOUN","semantic_domains":["syntax.role.modifier","claim.qualifier","grammar.attachment"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-007","surface":"clause","lemma":"clause","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.clause","claim.boundary.clause","grammar.unit"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-008","surface":"sentence","lemma":"sentence","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.sentence","claim.surface.sentence","grammar.unit"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-009","surface":"phrase","lemma":"phrase","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.phrase","grammar.unit","claim.fragment"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-010","surface":"antecedent","lemma":"antecedent","language":"en","pos":"NOUN","semantic_domains":["syntax.condition.antecedent","logic.condition.input","claim.condition.antecedent"],"morphology_tags":["noun","conditional_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-011","surface":"consequent_role","lemma":"consequent_role","language":"en","pos":"NOUN","semantic_domains":["syntax.condition.consequent_role","logic.condition.output_role","claim.condition.consequent_role"],"morphology_tags":["noun","conditional_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-012","surface":"referent","lemma":"referent","language":"en","pos":"NOUN","semantic_domains":["syntax.reference.referent","semantic.entity.reference","claim.argument.referent"],"morphology_tags":["noun","reference_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-013","surface":"anaphor","lemma":"anaphor","language":"en","pos":"NOUN","semantic_domains":["syntax.reference.anaphor","discourse.backreference","claim.reference.surface"],"morphology_tags":["noun","reference_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-014","surface":"qualifier","lemma":"qualifier","language":"en","pos":"NOUN","semantic_domains":["claim.qualifier","syntax.modifier","semantic.constraint"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-015","surface":"scope","lemma":"scope","language":"en","pos":"NOUN","semantic_domains":["claim.scope","logic.scope","syntax.boundary.scope"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-016","surface":"polarity","lemma":"polarity","language":"en","pos":"NOUN","semantic_domains":["claim.polarity","logic.truth_orientation","syntax.negation.axis"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-017","surface":"coordination","lemma":"coordination","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination","logic.composition","claim.join"],"morphology_tags":["noun","composition_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-018","surface":"conjunction","lemma":"conjunction","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination.conjunction","logic.conjunction","claim.join.and"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-019","surface":"disjunction","lemma":"disjunction","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination.disjunction","logic.disjunction","claim.join.or"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-020","surface":"negation","lemma":"negation","language":"en","pos":"NOUN","semantic_domains":["syntax.negation","logic.negation","claim.polarity.negative"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-021","surface":"exception","lemma":"exception","language":"en","pos":"NOUN","semantic_domains":["syntax.exception","logic.exception","claim.scope.exception"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-022","surface":"comparison_relation","lemma":"comparison_relation","language":"en","pos":"NOUN","semantic_domains":["syntax.comparison","comparative.relation","claim.comparison"],"morphology_tags":["noun","relation_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-023","surface":"attachment","lemma":"attachment","language":"en","pos":"NOUN","semantic_domains":["syntax.attachment","grammar.binding","claim.qualifier.binding"],"morphology_tags":["noun","binding_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
{"entry_id":"en-core-syntax-024","surface":"evidence_span","lemma":"evidence_span","language":"en","pos":"NOUN","semantic_domains":["claim.evidence_span","provenance.text_span","binding.evidence"],"morphology_tags":["noun","provenance_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
||||
16
language_packs/data/en_core_syntax_v1/manifest.json
Normal file
16
language_packs/data/en_core_syntax_v1/manifest.json
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"pack_id": "en_core_syntax_v1",
|
||||
"language": "en",
|
||||
"role": "operational_base",
|
||||
"script": "Latin",
|
||||
"normalization_policy": "unitize_versor",
|
||||
"source_manifest": "en_core_syntax_v1.lexicon.jsonl",
|
||||
"determinism_class": "D0",
|
||||
"checksum": "31ac3e05a1344e296e86cbcacdfc118583b17e2634dc752fd596528023b99900",
|
||||
"version": "1.0.0",
|
||||
"gate_engaged": true,
|
||||
"oov_policy": "tagged_fallback",
|
||||
"definitional_layer": true,
|
||||
"provenance": "foundation-curriculum:reviewed:2026-05-30",
|
||||
"glosses_checksum": "aebb2191739722d54c48fe10d10ab6bae2efb44bfd14f04ce72b6333a4a1ca98"
|
||||
}
|
||||
208
tests/test_en_core_syntax_v1_pack.py
Normal file
208
tests/test_en_core_syntax_v1_pack.py
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
"""``en_core_syntax_v1`` — foundational syntax vocabulary pack tests.
|
||||
|
||||
This pack is the first foundation-curriculum substrate pack. It does not
|
||||
implement a parser. It seeds the lower-level vocabulary needed for later
|
||||
claim parsing, relation binding, clause attachment, conditional structure,
|
||||
reference handling, polarity, and evidence-span discipline.
|
||||
|
||||
Contracts pinned here:
|
||||
|
||||
- checksum-verified load through ``language_packs.compiler.load_pack``;
|
||||
- every entry's primary semantic namespace is syntax/claim/provenance;
|
||||
- contiguous zero-padded entry ids;
|
||||
- one gloss per lexicon lemma, with manifest checksum discipline;
|
||||
- resolver registration after polarity and before kinship/domain relation packs;
|
||||
- prior high-frequency pack lemma routing remains unchanged.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
from chat.pack_resolver import (
|
||||
DEFAULT_RESOLVABLE_PACK_IDS,
|
||||
clear_resolver_cache,
|
||||
resolve_gloss,
|
||||
resolve_lemma,
|
||||
)
|
||||
from language_packs.compiler import load_pack
|
||||
|
||||
|
||||
PACK_ID = "en_core_syntax_v1"
|
||||
_PACK_ROOT = Path(__file__).resolve().parent.parent / "language_packs" / "data" / PACK_ID
|
||||
|
||||
EXPECTED_TOTAL = 24
|
||||
EXPECTED_POS_COUNTS = {"NOUN": 24}
|
||||
EXPECTED_LEMMAS: tuple[str, ...] = (
|
||||
"subject",
|
||||
"predicate",
|
||||
"agent_role",
|
||||
"patient",
|
||||
"object",
|
||||
"modifier",
|
||||
"clause",
|
||||
"sentence",
|
||||
"phrase",
|
||||
"antecedent",
|
||||
"consequent_role",
|
||||
"referent",
|
||||
"anaphor",
|
||||
"qualifier",
|
||||
"scope",
|
||||
"polarity",
|
||||
"coordination",
|
||||
"conjunction",
|
||||
"disjunction",
|
||||
"negation",
|
||||
"exception",
|
||||
"comparison_relation",
|
||||
"attachment",
|
||||
"evidence_span",
|
||||
)
|
||||
|
||||
_ALLOWED_PRIMARY_PREFIXES = ("syntax.", "claim.", "provenance.")
|
||||
_EXPECTED_PROVENANCE = ["foundation_syntax_v1:reviewed:2026-05-30"]
|
||||
|
||||
|
||||
def _read_jsonl(path: Path) -> list[dict]:
|
||||
return [
|
||||
json.loads(line)
|
||||
for line in path.read_text("utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def _read_lexicon() -> list[dict]:
|
||||
return _read_jsonl(_PACK_ROOT / "lexicon.jsonl")
|
||||
|
||||
|
||||
def _read_glosses() -> list[dict]:
|
||||
return _read_jsonl(_PACK_ROOT / "glosses.jsonl")
|
||||
|
||||
|
||||
def test_pack_loads_with_matching_checksum() -> None:
|
||||
manifest, manifold = load_pack(PACK_ID)
|
||||
assert manifest.pack_id == PACK_ID
|
||||
assert len(manifest.checksum) == 64
|
||||
assert len(manifold) == EXPECTED_TOTAL
|
||||
|
||||
|
||||
def test_manifest_checksums_match_pack_bytes() -> None:
|
||||
manifest = json.loads((_PACK_ROOT / "manifest.json").read_text("utf-8"))
|
||||
assert manifest["checksum"] == hashlib.sha256(
|
||||
(_PACK_ROOT / "lexicon.jsonl").read_bytes()
|
||||
).hexdigest()
|
||||
assert manifest["glosses_checksum"] == hashlib.sha256(
|
||||
(_PACK_ROOT / "glosses.jsonl").read_bytes()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def test_pos_distribution_matches_design() -> None:
|
||||
assert dict(Counter(e["pos"] for e in _read_lexicon())) == EXPECTED_POS_COUNTS
|
||||
|
||||
|
||||
def test_all_expected_lemmas_present() -> None:
|
||||
_, manifold = load_pack(PACK_ID)
|
||||
assert {manifold.get_word_at(i) for i in range(len(manifold))} == set(EXPECTED_LEMMAS)
|
||||
|
||||
|
||||
def test_every_entry_has_foundational_primary_namespace() -> None:
|
||||
for entry in _read_lexicon():
|
||||
primary = entry["semantic_domains"][0]
|
||||
assert primary.startswith(_ALLOWED_PRIMARY_PREFIXES), entry
|
||||
|
||||
|
||||
def test_every_entry_has_reviewed_foundation_provenance() -> None:
|
||||
for entry in _read_lexicon():
|
||||
assert entry["provenance_ids"] == _EXPECTED_PROVENANCE, entry
|
||||
|
||||
|
||||
def test_entry_ids_contiguous_and_zero_padded() -> None:
|
||||
entries = sorted(_read_lexicon(), key=lambda d: d["entry_id"])
|
||||
for i, entry in enumerate(entries, start=1):
|
||||
assert entry["entry_id"] == f"en-core-syntax-{i:03d}", entry["entry_id"]
|
||||
|
||||
|
||||
def test_glosses_are_one_to_one_with_lexicon() -> None:
|
||||
lexicon_lemmas = {entry["lemma"] for entry in _read_lexicon()}
|
||||
gloss_lemmas = {entry["lemma"] for entry in _read_glosses()}
|
||||
assert gloss_lemmas == lexicon_lemmas
|
||||
|
||||
|
||||
def test_gloss_entries_use_strict_definitional_overlay_shape() -> None:
|
||||
for entry in _read_glosses():
|
||||
assert set(entry) == {
|
||||
"lemma",
|
||||
"gloss",
|
||||
"pos",
|
||||
"provenance_ids",
|
||||
"definitional_atoms",
|
||||
"predicates_invited",
|
||||
"definition_version",
|
||||
}
|
||||
assert entry["provenance_ids"] == _EXPECTED_PROVENANCE
|
||||
assert entry["definition_version"] == 1
|
||||
assert isinstance(entry["definitional_atoms"], list)
|
||||
assert isinstance(entry["predicates_invited"], list)
|
||||
|
||||
|
||||
def test_pack_registered_after_polarity_before_relation_content() -> None:
|
||||
assert PACK_ID in DEFAULT_RESOLVABLE_PACK_IDS
|
||||
assert DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_polarity_v1") < DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID)
|
||||
assert DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID) < DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_relations_v1")
|
||||
assert DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID) < DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_relations_v2")
|
||||
|
||||
|
||||
def test_resolver_routes_syntax_lemmas_to_this_pack() -> None:
|
||||
clear_resolver_cache()
|
||||
for lemma in EXPECTED_LEMMAS:
|
||||
resolved = resolve_lemma(lemma)
|
||||
assert resolved is not None and resolved[0] == PACK_ID, (
|
||||
f"{lemma!r} resolved to {resolved}"
|
||||
)
|
||||
assert resolved[1][0].startswith(_ALLOWED_PRIMARY_PREFIXES)
|
||||
|
||||
|
||||
def test_gloss_resolver_routes_syntax_lemmas_to_this_pack() -> None:
|
||||
clear_resolver_cache()
|
||||
for lemma in (
|
||||
"subject",
|
||||
"predicate",
|
||||
"antecedent",
|
||||
"consequent_role",
|
||||
"negation",
|
||||
"evidence_span",
|
||||
):
|
||||
resolved = resolve_gloss(lemma)
|
||||
assert resolved is not None, lemma
|
||||
pack_id, pos, gloss = resolved
|
||||
assert pack_id == PACK_ID
|
||||
assert pos == "NOUN"
|
||||
assert isinstance(gloss, str) and len(gloss.split()) >= 4
|
||||
|
||||
|
||||
def test_prior_pack_lemma_resolution_unchanged() -> None:
|
||||
for lemma, expected in (
|
||||
("truth", "en_core_cognition_v1"),
|
||||
("doubt", "en_core_meta_v1"),
|
||||
("true", "en_core_attitude_v1"),
|
||||
("now", "en_core_temporal_v1"),
|
||||
("do", "en_core_action_v1"),
|
||||
("all", "en_core_quantitative_v1"),
|
||||
("place", "en_core_spatial_v1"),
|
||||
# `cause` is owned by en_core_cognition_v1 (mounted at index 0),
|
||||
# NOT en_core_causation_v1 — this is pre-existing first-match-wins
|
||||
# resolution that the syntax pack (mounted later) does not change.
|
||||
# The contract here is "prior resolution unchanged"; we pin the
|
||||
# TRUE current owner. `consequent` genuinely resolves to causation.
|
||||
("cause", "en_core_cognition_v1"),
|
||||
("consequent", "en_core_causation_v1"),
|
||||
("never", "en_core_polarity_v1"),
|
||||
("parent", "en_core_relations_v1"),
|
||||
("mother", "en_core_relations_v2"),
|
||||
):
|
||||
resolved = resolve_lemma(lemma)
|
||||
assert resolved is not None and resolved[0] == expected
|
||||
Loading…
Reference in a new issue