Merge pull request #508 from AssetOverflow/chore/revert-503-broken-syntax-pack
Revert #503: en_core_syntax_v1 pack shipped with failing tests
This commit is contained in:
Shay
GitHub
commit
f79b647671
6 changed files with 0 additions and 428 deletions
|
|
@ -50,7 +50,6 @@ DEFAULT_RESOLVABLE_PACK_IDS: tuple[str, ...] = (
|
||||||
"en_core_spatial_v1",
|
"en_core_spatial_v1",
|
||||||
"en_core_causation_v1",
|
"en_core_causation_v1",
|
||||||
"en_core_polarity_v1",
|
"en_core_polarity_v1",
|
||||||
"en_core_syntax_v1",
|
|
||||||
"en_core_relations_v1",
|
"en_core_relations_v1",
|
||||||
"en_core_relations_v2",
|
"en_core_relations_v2",
|
||||||
# ADR-0073c — synthetic English anchor lemmas for cross-lang collapse.
|
# ADR-0073c — synthetic English anchor lemmas for cross-lang collapse.
|
||||||
|
|
|
||||||
|
|
@ -1,160 +0,0 @@
|
||||||
# en_core_syntax_v1
|
|
||||||
|
|
||||||
Status: implemented candidate
|
|
||||||
Scope: foundational language substrate pack
|
|
||||||
Pack id: `en_core_syntax_v1`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
`en_core_syntax_v1` seeds the first foundation-curriculum substrate needed for later claim parsing and relation binding.
|
|
||||||
|
|
||||||
The pack does not parse sentences. It establishes the lower-level English vocabulary for speaking about syntax, claim structure, evidence binding, reference, polarity, comparison, coordination, and conditional roles.
|
|
||||||
|
|
||||||
This is the first step toward the roadmap rule:
|
|
||||||
|
|
||||||
```text
|
|
||||||
language -> relations -> quantity -> units -> logic -> evidence -> data -> algorithms -> systems -> domains
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Non-goals
|
|
||||||
|
|
||||||
This slice does not implement:
|
|
||||||
|
|
||||||
- a natural-language parser
|
|
||||||
- dependency parsing
|
|
||||||
- clause extraction
|
|
||||||
- evidence-span extraction
|
|
||||||
- claim graph construction
|
|
||||||
- GSM8K statement classification changes
|
|
||||||
- relation-binding execution
|
|
||||||
- new runtime generation behavior
|
|
||||||
|
|
||||||
Those are follow-on slices. This pack only ratifies the vocabulary substrate and makes it loadable/resolvable through existing pack infrastructure.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Pack contents
|
|
||||||
|
|
||||||
The pack contains 24 reviewed noun entries:
|
|
||||||
|
|
||||||
```text
|
|
||||||
subject
|
|
||||||
predicate
|
|
||||||
agent
|
|
||||||
patient
|
|
||||||
object
|
|
||||||
modifier
|
|
||||||
clause
|
|
||||||
sentence
|
|
||||||
phrase
|
|
||||||
antecedent
|
|
||||||
consequent_role
|
|
||||||
referent
|
|
||||||
anaphor
|
|
||||||
qualifier
|
|
||||||
scope
|
|
||||||
polarity
|
|
||||||
coordination
|
|
||||||
conjunction
|
|
||||||
disjunction
|
|
||||||
negation
|
|
||||||
exception
|
|
||||||
comparison
|
|
||||||
attachment
|
|
||||||
evidence_span
|
|
||||||
```
|
|
||||||
|
|
||||||
`consequent_role` deliberately avoids the bare lemma `consequent`, which is already owned by `en_core_causation_v1` as a causal/resultant adjective. The pack test pins that ownership so the syntax substrate remains additive and does not steal existing lemma resolution.
|
|
||||||
|
|
||||||
Primary namespace families:
|
|
||||||
|
|
||||||
```text
|
|
||||||
syntax.*
|
|
||||||
claim.*
|
|
||||||
provenance.*
|
|
||||||
```
|
|
||||||
|
|
||||||
Each lexicon row uses provenance:
|
|
||||||
|
|
||||||
```text
|
|
||||||
foundation_syntax_v1:reviewed:2026-05-30
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Files
|
|
||||||
|
|
||||||
```text
|
|
||||||
language_packs/data/en_core_syntax_v1/manifest.json
|
|
||||||
language_packs/data/en_core_syntax_v1/lexicon.jsonl
|
|
||||||
language_packs/data/en_core_syntax_v1/glosses.jsonl
|
|
||||||
tests/test_en_core_syntax_v1_pack.py
|
|
||||||
```
|
|
||||||
|
|
||||||
The pack ships both a checksum-sealed `lexicon.jsonl` and a checksum-sealed `glosses.jsonl` companion overlay.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Admission gates pinned by tests
|
|
||||||
|
|
||||||
`tests/test_en_core_syntax_v1_pack.py` verifies:
|
|
||||||
|
|
||||||
- `load_pack("en_core_syntax_v1")` succeeds;
|
|
||||||
- manifest checksum matches `lexicon.jsonl` bytes;
|
|
||||||
- manifest `glosses_checksum` matches `glosses.jsonl` bytes;
|
|
||||||
- total entry count is 24;
|
|
||||||
- POS distribution is exactly 24 `NOUN` rows;
|
|
||||||
- expected lemmas are present in the compiled manifold;
|
|
||||||
- entry ids are contiguous and zero-padded;
|
|
||||||
- every lexicon row has reviewed foundation provenance;
|
|
||||||
- every gloss lemma is resident in the same pack lexicon;
|
|
||||||
- resolver registration routes syntax lemmas to this pack;
|
|
||||||
- prior pack lemma routing remains unchanged.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Resolver order
|
|
||||||
|
|
||||||
The pack is registered in `chat.pack_resolver.DEFAULT_RESOLVABLE_PACK_IDS` after the polarity pack and before relation/kinship packs:
|
|
||||||
|
|
||||||
```text
|
|
||||||
en_core_polarity_v1
|
|
||||||
en_core_syntax_v1
|
|
||||||
en_core_relations_v1
|
|
||||||
en_core_relations_v2
|
|
||||||
```
|
|
||||||
|
|
||||||
This makes syntax vocabulary available to the same deterministic first-match resolver used by existing English content packs while preserving prior high-frequency lemma ownership.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Follow-on work unlocked
|
|
||||||
|
|
||||||
The next slices should use this vocabulary to implement actual relation binding:
|
|
||||||
|
|
||||||
```text
|
|
||||||
evals/language_claim_parsing
|
|
||||||
evals/language_relation_binding
|
|
||||||
tests/test_language_claim_parsing.py
|
|
||||||
tests/test_relation_binding_replay.py
|
|
||||||
```
|
|
||||||
|
|
||||||
Expected follow-on primitives:
|
|
||||||
|
|
||||||
- evidence-span extraction
|
|
||||||
- subject/predicate/argument slot binding
|
|
||||||
- modifier attachment
|
|
||||||
- conditional antecedent/consequent binding
|
|
||||||
- polarity and negation scope
|
|
||||||
- reference/anaphora target binding
|
|
||||||
- typed refusal when a required slot lacks evidence
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Boundary
|
|
||||||
|
|
||||||
This pack is a substrate, not a capability claim. It should not be marked as full language relation binding admission until parser/eval/runtime surfaces exist and pass their own tests.
|
|
||||||
|
|
@ -1,24 +0,0 @@
|
||||||
{"lemma":"subject","gloss":"the claim role that names what a statement is about","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"predicate","gloss":"the claim role that carries what is said of a subject","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"agent","gloss":"the participant that performs an action","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"patient","gloss":"the participant that undergoes an action or change","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"object","gloss":"the syntactic role receiving or completing an action","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"modifier","gloss":"a word or phrase that qualifies another expression","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"clause","gloss":"a bounded unit that can carry a claim relation","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"sentence","gloss":"a complete written surface that may contain one or more claims","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"phrase","gloss":"a bounded expression below clause level","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"antecedent","gloss":"the condition-side of a conditional claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"consequent_role","gloss":"the result-side role of a conditional claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"referent","gloss":"the entity or claim target that a reference points to","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"anaphor","gloss":"a later reference that points back to an earlier referent","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"qualifier","gloss":"a constraint or limit attached to a claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"scope","gloss":"the boundary within which a claim or operator applies","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"polarity","gloss":"the positive or negative orientation of a claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"coordination","gloss":"the joining of claim parts or clauses","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"conjunction","gloss":"a joining relation where both parts are asserted together","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"disjunction","gloss":"a joining relation where alternatives are presented","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"negation","gloss":"an operator that denies or reverses a claim polarity","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"exception","gloss":"a scoped exclusion from a claim or rule","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"comparison","gloss":"a relation that evaluates one thing against another","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"attachment","gloss":"the binding of a modifier or qualifier to its target","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
{"lemma":"evidence_span","gloss":"the exact text span that supports a parsed claim","pos":"NOUN","provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"],"definitional_atoms":[],"predicates_invited":[],"definition_version":1}
|
|
||||||
|
|
@ -1,24 +0,0 @@
|
||||||
{"entry_id":"en-core-syntax-001","surface":"subject","lemma":"subject","language":"en","pos":"NOUN","semantic_domains":["syntax.role.subject","claim.argument.subject","grammar.core"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-002","surface":"predicate","lemma":"predicate","language":"en","pos":"NOUN","semantic_domains":["syntax.role.predicate","claim.relation.predicate","grammar.core"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-003","surface":"agent","lemma":"agent","language":"en","pos":"NOUN","semantic_domains":["syntax.role.agent","event.participant.actor","claim.argument.agent"],"morphology_tags":["noun","semantic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-004","surface":"patient","lemma":"patient","language":"en","pos":"NOUN","semantic_domains":["syntax.role.patient","event.participant.undergoer","claim.argument.patient"],"morphology_tags":["noun","semantic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-005","surface":"object","lemma":"object","language":"en","pos":"NOUN","semantic_domains":["syntax.role.object","event.participant.object","claim.argument.object"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-006","surface":"modifier","lemma":"modifier","language":"en","pos":"NOUN","semantic_domains":["syntax.role.modifier","claim.qualifier","grammar.attachment"],"morphology_tags":["noun","syntactic_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-007","surface":"clause","lemma":"clause","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.clause","claim.boundary.clause","grammar.unit"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-008","surface":"sentence","lemma":"sentence","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.sentence","claim.surface.sentence","grammar.unit"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-009","surface":"phrase","lemma":"phrase","language":"en","pos":"NOUN","semantic_domains":["syntax.unit.phrase","grammar.unit","claim.fragment"],"morphology_tags":["noun","syntax_unit"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-010","surface":"antecedent","lemma":"antecedent","language":"en","pos":"NOUN","semantic_domains":["syntax.condition.antecedent","logic.condition.input","claim.condition.antecedent"],"morphology_tags":["noun","conditional_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-011","surface":"consequent_role","lemma":"consequent_role","language":"en","pos":"NOUN","semantic_domains":["syntax.condition.consequent_role","logic.condition.output_role","claim.condition.consequent_role"],"morphology_tags":["noun","conditional_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-012","surface":"referent","lemma":"referent","language":"en","pos":"NOUN","semantic_domains":["syntax.reference.referent","semantic.entity.reference","claim.argument.referent"],"morphology_tags":["noun","reference_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-013","surface":"anaphor","lemma":"anaphor","language":"en","pos":"NOUN","semantic_domains":["syntax.reference.anaphor","discourse.backreference","claim.reference.surface"],"morphology_tags":["noun","reference_role"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-014","surface":"qualifier","lemma":"qualifier","language":"en","pos":"NOUN","semantic_domains":["claim.qualifier","syntax.modifier","semantic.constraint"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-015","surface":"scope","lemma":"scope","language":"en","pos":"NOUN","semantic_domains":["claim.scope","logic.scope","syntax.boundary.scope"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-016","surface":"polarity","lemma":"polarity","language":"en","pos":"NOUN","semantic_domains":["claim.polarity","logic.truth_orientation","syntax.negation.axis"],"morphology_tags":["noun","claim_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-017","surface":"coordination","lemma":"coordination","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination","logic.composition","claim.join"],"morphology_tags":["noun","composition_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-018","surface":"conjunction","lemma":"conjunction","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination.conjunction","logic.conjunction","claim.join.and"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-019","surface":"disjunction","lemma":"disjunction","language":"en","pos":"NOUN","semantic_domains":["syntax.coordination.disjunction","logic.disjunction","claim.join.or"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-020","surface":"negation","lemma":"negation","language":"en","pos":"NOUN","semantic_domains":["syntax.negation","logic.negation","claim.polarity.negative"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-021","surface":"exception","lemma":"exception","language":"en","pos":"NOUN","semantic_domains":["syntax.exception","logic.exception","claim.scope.exception"],"morphology_tags":["noun","logical_operator"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-022","surface":"comparison","lemma":"comparison","language":"en","pos":"NOUN","semantic_domains":["syntax.comparison","comparative.relation","claim.comparison"],"morphology_tags":["noun","relation_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-023","surface":"attachment","lemma":"attachment","language":"en","pos":"NOUN","semantic_domains":["syntax.attachment","grammar.binding","claim.qualifier.binding"],"morphology_tags":["noun","binding_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
{"entry_id":"en-core-syntax-024","surface":"evidence_span","lemma":"evidence_span","language":"en","pos":"NOUN","semantic_domains":["claim.evidence_span","provenance.text_span","binding.evidence"],"morphology_tags":["noun","provenance_term"],"provenance_ids":["foundation_syntax_v1:reviewed:2026-05-30"]}
|
|
||||||
|
|
@ -1,16 +0,0 @@
|
||||||
{
|
|
||||||
"pack_id": "en_core_syntax_v1",
|
|
||||||
"language": "en",
|
|
||||||
"role": "operational_base",
|
|
||||||
"script": "Latin",
|
|
||||||
"normalization_policy": "unitize_versor",
|
|
||||||
"source_manifest": "en_core_syntax_v1.lexicon.jsonl",
|
|
||||||
"determinism_class": "D0",
|
|
||||||
"checksum": "7151bf1f98e4cb638f0fb8fd17a5d1912859b3ff8e94f7a2857169dacf9ee37e",
|
|
||||||
"version": "1.0.0",
|
|
||||||
"gate_engaged": true,
|
|
||||||
"oov_policy": "tagged_fallback",
|
|
||||||
"definitional_layer": true,
|
|
||||||
"provenance": "foundation-curriculum:reviewed:2026-05-30",
|
|
||||||
"glosses_checksum": "9eaed2d5756b325641892ce9b0fda3ec781298d181072ad5739533ee08710d5f"
|
|
||||||
}
|
|
||||||
|
|
@ -1,203 +0,0 @@
|
||||||
"""``en_core_syntax_v1`` — foundational syntax vocabulary pack tests.
|
|
||||||
|
|
||||||
This pack is the first foundation-curriculum substrate pack. It does not
|
|
||||||
implement a parser. It seeds the lower-level vocabulary needed for later
|
|
||||||
claim parsing, relation binding, clause attachment, conditional structure,
|
|
||||||
reference handling, polarity, and evidence-span discipline.
|
|
||||||
|
|
||||||
Contracts pinned here:
|
|
||||||
|
|
||||||
- checksum-verified load through ``language_packs.compiler.load_pack``;
|
|
||||||
- every entry's primary semantic namespace is syntax/claim/provenance;
|
|
||||||
- contiguous zero-padded entry ids;
|
|
||||||
- one gloss per lexicon lemma, with manifest checksum discipline;
|
|
||||||
- resolver registration after polarity and before kinship/domain relation packs;
|
|
||||||
- prior high-frequency pack lemma routing remains unchanged.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
from collections import Counter
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from chat.pack_resolver import (
|
|
||||||
DEFAULT_RESOLVABLE_PACK_IDS,
|
|
||||||
clear_resolver_cache,
|
|
||||||
resolve_gloss,
|
|
||||||
resolve_lemma,
|
|
||||||
)
|
|
||||||
from language_packs.compiler import load_pack
|
|
||||||
|
|
||||||
|
|
||||||
PACK_ID = "en_core_syntax_v1"
|
|
||||||
_PACK_ROOT = Path(__file__).resolve().parent.parent / "language_packs" / "data" / PACK_ID
|
|
||||||
|
|
||||||
EXPECTED_TOTAL = 24
|
|
||||||
EXPECTED_POS_COUNTS = {"NOUN": 24}
|
|
||||||
EXPECTED_LEMMAS: tuple[str, ...] = (
|
|
||||||
"subject",
|
|
||||||
"predicate",
|
|
||||||
"agent",
|
|
||||||
"patient",
|
|
||||||
"object",
|
|
||||||
"modifier",
|
|
||||||
"clause",
|
|
||||||
"sentence",
|
|
||||||
"phrase",
|
|
||||||
"antecedent",
|
|
||||||
"consequent_role",
|
|
||||||
"referent",
|
|
||||||
"anaphor",
|
|
||||||
"qualifier",
|
|
||||||
"scope",
|
|
||||||
"polarity",
|
|
||||||
"coordination",
|
|
||||||
"conjunction",
|
|
||||||
"disjunction",
|
|
||||||
"negation",
|
|
||||||
"exception",
|
|
||||||
"comparison",
|
|
||||||
"attachment",
|
|
||||||
"evidence_span",
|
|
||||||
)
|
|
||||||
|
|
||||||
_ALLOWED_PRIMARY_PREFIXES = ("syntax.", "claim.", "provenance.")
|
|
||||||
_EXPECTED_PROVENANCE = ["foundation_syntax_v1:reviewed:2026-05-30"]
|
|
||||||
|
|
||||||
|
|
||||||
def _read_jsonl(path: Path) -> list[dict]:
|
|
||||||
return [
|
|
||||||
json.loads(line)
|
|
||||||
for line in path.read_text("utf-8").splitlines()
|
|
||||||
if line.strip()
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def _read_lexicon() -> list[dict]:
|
|
||||||
return _read_jsonl(_PACK_ROOT / "lexicon.jsonl")
|
|
||||||
|
|
||||||
|
|
||||||
def _read_glosses() -> list[dict]:
|
|
||||||
return _read_jsonl(_PACK_ROOT / "glosses.jsonl")
|
|
||||||
|
|
||||||
|
|
||||||
def test_pack_loads_with_matching_checksum() -> None:
|
|
||||||
manifest, manifold = load_pack(PACK_ID)
|
|
||||||
assert manifest.pack_id == PACK_ID
|
|
||||||
assert len(manifest.checksum) == 64
|
|
||||||
assert len(manifold) == EXPECTED_TOTAL
|
|
||||||
|
|
||||||
|
|
||||||
def test_manifest_checksums_match_pack_bytes() -> None:
|
|
||||||
manifest = json.loads((_PACK_ROOT / "manifest.json").read_text("utf-8"))
|
|
||||||
assert manifest["checksum"] == hashlib.sha256(
|
|
||||||
(_PACK_ROOT / "lexicon.jsonl").read_bytes()
|
|
||||||
).hexdigest()
|
|
||||||
assert manifest["glosses_checksum"] == hashlib.sha256(
|
|
||||||
(_PACK_ROOT / "glosses.jsonl").read_bytes()
|
|
||||||
).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def test_pos_distribution_matches_design() -> None:
|
|
||||||
assert dict(Counter(e["pos"] for e in _read_lexicon())) == EXPECTED_POS_COUNTS
|
|
||||||
|
|
||||||
|
|
||||||
def test_all_expected_lemmas_present() -> None:
|
|
||||||
_, manifold = load_pack(PACK_ID)
|
|
||||||
assert {manifold.get_word_at(i) for i in range(len(manifold))} == set(EXPECTED_LEMMAS)
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_entry_has_foundational_primary_namespace() -> None:
|
|
||||||
for entry in _read_lexicon():
|
|
||||||
primary = entry["semantic_domains"][0]
|
|
||||||
assert primary.startswith(_ALLOWED_PRIMARY_PREFIXES), entry
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_entry_has_reviewed_foundation_provenance() -> None:
|
|
||||||
for entry in _read_lexicon():
|
|
||||||
assert entry["provenance_ids"] == _EXPECTED_PROVENANCE, entry
|
|
||||||
|
|
||||||
|
|
||||||
def test_entry_ids_contiguous_and_zero_padded() -> None:
|
|
||||||
entries = sorted(_read_lexicon(), key=lambda d: d["entry_id"])
|
|
||||||
for i, entry in enumerate(entries, start=1):
|
|
||||||
assert entry["entry_id"] == f"en-core-syntax-{i:03d}", entry["entry_id"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_glosses_are_one_to_one_with_lexicon() -> None:
|
|
||||||
lexicon_lemmas = {entry["lemma"] for entry in _read_lexicon()}
|
|
||||||
gloss_lemmas = {entry["lemma"] for entry in _read_glosses()}
|
|
||||||
assert gloss_lemmas == lexicon_lemmas
|
|
||||||
|
|
||||||
|
|
||||||
def test_gloss_entries_use_strict_definitional_overlay_shape() -> None:
|
|
||||||
for entry in _read_glosses():
|
|
||||||
assert set(entry) == {
|
|
||||||
"lemma",
|
|
||||||
"gloss",
|
|
||||||
"pos",
|
|
||||||
"provenance_ids",
|
|
||||||
"definitional_atoms",
|
|
||||||
"predicates_invited",
|
|
||||||
"definition_version",
|
|
||||||
}
|
|
||||||
assert entry["provenance_ids"] == _EXPECTED_PROVENANCE
|
|
||||||
assert entry["definition_version"] == 1
|
|
||||||
assert isinstance(entry["definitional_atoms"], list)
|
|
||||||
assert isinstance(entry["predicates_invited"], list)
|
|
||||||
|
|
||||||
|
|
||||||
def test_pack_registered_after_polarity_before_relation_content() -> None:
|
|
||||||
assert PACK_ID in DEFAULT_RESOLVABLE_PACK_IDS
|
|
||||||
assert DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_polarity_v1") < DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID)
|
|
||||||
assert DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID) < DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_relations_v1")
|
|
||||||
assert DEFAULT_RESOLVABLE_PACK_IDS.index(PACK_ID) < DEFAULT_RESOLVABLE_PACK_IDS.index("en_core_relations_v2")
|
|
||||||
|
|
||||||
|
|
||||||
def test_resolver_routes_syntax_lemmas_to_this_pack() -> None:
|
|
||||||
clear_resolver_cache()
|
|
||||||
for lemma in EXPECTED_LEMMAS:
|
|
||||||
resolved = resolve_lemma(lemma)
|
|
||||||
assert resolved is not None and resolved[0] == PACK_ID, (
|
|
||||||
f"{lemma!r} resolved to {resolved}"
|
|
||||||
)
|
|
||||||
assert resolved[1][0].startswith(_ALLOWED_PRIMARY_PREFIXES)
|
|
||||||
|
|
||||||
|
|
||||||
def test_gloss_resolver_routes_syntax_lemmas_to_this_pack() -> None:
|
|
||||||
clear_resolver_cache()
|
|
||||||
for lemma in (
|
|
||||||
"subject",
|
|
||||||
"predicate",
|
|
||||||
"antecedent",
|
|
||||||
"consequent_role",
|
|
||||||
"negation",
|
|
||||||
"evidence_span",
|
|
||||||
):
|
|
||||||
resolved = resolve_gloss(lemma)
|
|
||||||
assert resolved is not None, lemma
|
|
||||||
pack_id, pos, gloss = resolved
|
|
||||||
assert pack_id == PACK_ID
|
|
||||||
assert pos == "NOUN"
|
|
||||||
assert isinstance(gloss, str) and len(gloss.split()) >= 4
|
|
||||||
|
|
||||||
|
|
||||||
def test_prior_pack_lemma_resolution_unchanged() -> None:
|
|
||||||
for lemma, expected in (
|
|
||||||
("truth", "en_core_cognition_v1"),
|
|
||||||
("doubt", "en_core_meta_v1"),
|
|
||||||
("true", "en_core_attitude_v1"),
|
|
||||||
("now", "en_core_temporal_v1"),
|
|
||||||
("do", "en_core_action_v1"),
|
|
||||||
("all", "en_core_quantitative_v1"),
|
|
||||||
("place", "en_core_spatial_v1"),
|
|
||||||
("cause", "en_core_causation_v1"),
|
|
||||||
("consequent", "en_core_causation_v1"),
|
|
||||||
("never", "en_core_polarity_v1"),
|
|
||||||
("parent", "en_core_relations_v1"),
|
|
||||||
("mother", "en_core_relations_v2"),
|
|
||||||
):
|
|
||||||
resolved = resolve_lemma(lemma)
|
|
||||||
assert resolved is not None and resolved[0] == expected
|
|
||||||
Loading…
Reference in a new issue