test(adr-0084): integration test pins substrate gate against ratified content (#68)
After PR #64 (substrate) and PR #65 (content) both landed on main, this test is the promised follow-up that exercises the substrate-callable verify_definitional_closure against the real ratified content rather than fixture packs. It pins three contracts: 1. Substrate-vs-content handshake. The standalone scripts/verify_definitional_closure.py is the agent's dev-loop tool; this test is the gate-callable equivalent the ratification pipeline can invoke. Both must agree on what passes — divergence is a contract bug. 2. Content drift catcher. Any future content edit that adds an unresolved token / non-mounted dependency / silent staging leak fails this test before the edit lands on main. 3. Staging exclusion. en_minimal_v1 is staging per the ADR-0084 pack-content brief and must not be load-bearing for the closure rule. Test-pinned via a production-pool subtest. Substrate fix: allow empty definitional_atoms The substrate's strict parser previously rejected empty definitional_atoms. That stance was wrong: per the ADR-0084 pack- content brief, the per-entry atom list excludes articles, prepositions, and copulas. A gloss whose every content word is a function word (e.g. en_core_temporal_v1/prior → "before") has zero content atoms by construction. The closure rule passes vacuously when atoms is empty — there is nothing to close. The gloss-vs-atoms mismatch check in the standalone verifier is the second-layer gate that distinguishes by-construction emptiness (legitimate) from by-omission emptiness (laziness). Substrate parser shouldn't double-gate the same concern. The corresponding substrate test flipped from test_empty_definitional_atoms_rejected to test_empty_definitional_atoms_accepted, with comment explaining the reasoning. Primitives expansion: can + action Two content entries (en_core_cognition_v1/person → "who can know and do" and en_core_meta_v1/intend → "decide before an action") leaned on 'can' and 'action' as atom references. Today those lemmas resolve ONLY via en_minimal_v1/lexicon.jsonl — the staging pack. That's a production-vs-staging leak: production content should not be load- bearing on staging. Two clean alternatives: (a) rewrite the two glosses to avoid 'can' and 'action' (b) promote 'can' and 'action' to primitives Chose (b): both lemmas are genuinely terminal-feeling (can is a basic capability modal; action is an irreducible "what is done"); the content reads more naturally with them present than with paraphrased substitutes; and the floor was always going to need both eventually. The cost is two primitives.jsonl rows + checksum + count bump. Verification: scripts/verify_definitional_closure.py exit 0 tests/test_adr_0084_integration_closure.py 30/30 pass tests/test_adr_0084_definitional_substrate.py 39/39 pass core test --suite smoke -q 67/67 core test --suite packs -q 6/6 core eval cognition byte-identical (100/91.7/100/100) Two-layer gate now in place: - standalone verifier (dev loop, gloss/atom mismatch check) - substrate verifier (ratification gate, parametrized over every opted-in pack, staging-exclusion test, primitives floor coverage)
This commit is contained in:
Shay
GitHub
parent
48282eef8d
commit
1938aaa674
5 changed files with 250 additions and 6 deletions
|
|
@ -134,8 +134,19 @@ def parse_gloss_entry(payload: dict, *, strict: bool) -> GlossEntry:
|
|||
|
||||
lemma = _require_nonempty_str(payload, "lemma", lemma_hint=lemma_hint)
|
||||
gloss = _require_nonempty_str(payload, "gloss", lemma_hint=lemma)
|
||||
# Empty ``definitional_atoms`` is legal: per the ADR-0084
|
||||
# content brief, the per-entry atom list excludes
|
||||
# articles/prepositions/copulas. A gloss whose every content
|
||||
# word is a function word (e.g. ``prior`` → ``"before"``) has
|
||||
# zero content atoms by construction. The closure rule then
|
||||
# passes vacuously — which is fine, because there is nothing
|
||||
# *to* close. Note the substrate cannot tell whether atoms
|
||||
# are empty by-construction (legitimate) or by-omission
|
||||
# (laziness); the gloss-vs-atoms mismatch check in the
|
||||
# standalone verifier (``scripts/verify_definitional_closure.py``)
|
||||
# is the second-layer gate that distinguishes the two.
|
||||
definitional_atoms = _require_str_tuple(
|
||||
payload, "definitional_atoms", lemma_hint=lemma, allow_empty=False
|
||||
payload, "definitional_atoms", lemma_hint=lemma, allow_empty=True
|
||||
)
|
||||
predicates_invited = _require_str_tuple(
|
||||
payload, "predicates_invited", lemma_hint=lemma, allow_empty=True
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@
|
|||
"definitional_layer": true,
|
||||
"version": 1,
|
||||
"issued_at": "2026-05-20T00:00:00Z",
|
||||
"checksum": "93d01ff6d4812968a490021e0f5518ebe071c47325e506a6678a57a4154da284",
|
||||
"primitive_count": 52,
|
||||
"checksum": "e5a4782a2f0ec78ebe852ba66156534260ab1bda9f53148be098857ccd0e5e78",
|
||||
"primitive_count": 54,
|
||||
"never_auto_mutable": true,
|
||||
"provenance": "adr-0084:reviewed:2026-05-20"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -50,3 +50,5 @@
|
|||
{"lemma":"female","category":"identity","pos":"ADJ","primitive_version":1,"provenance_ids":["adr-0084:reviewed:2026-05-20"]}
|
||||
{"lemma":"other","category":"identity","pos":"ADJ","primitive_version":1,"provenance_ids":["adr-0084:reviewed:2026-05-20"]}
|
||||
{"lemma":"only","category":"quantity","pos":"ADV","primitive_version":1,"provenance_ids":["adr-0084:reviewed:2026-05-20"]}
|
||||
{"lemma":"can","category":"mode","pos":"AUX","primitive_version":1,"provenance_ids":["adr-0084:reviewed:2026-05-20","adr-0084:integration:2026-05-20"]}
|
||||
{"lemma":"action","category":"relation","pos":"NOUN","primitive_version":1,"provenance_ids":["adr-0084:reviewed:2026-05-20","adr-0084:integration:2026-05-20"]}
|
||||
|
|
|
|||
|
|
@ -89,11 +89,17 @@ class TestStrictParser:
|
|||
entry = parse_gloss_entry(payload, strict=True)
|
||||
assert entry.predicates_invited == ()
|
||||
|
||||
def test_empty_definitional_atoms_rejected(self) -> None:
|
||||
def test_empty_definitional_atoms_accepted(self) -> None:
|
||||
# Empty atoms list is legal — a gloss whose every content word
|
||||
# is a function word/preposition (per the ADR-0084 brief's skip
|
||||
# list) legitimately produces zero atoms. The closure rule
|
||||
# passes vacuously; the gloss-vs-atoms mismatch check in the
|
||||
# standalone verifier is the second-layer gate that catches
|
||||
# laziness vs by-construction emptiness.
|
||||
payload = _valid_strict_payload()
|
||||
payload["definitional_atoms"] = []
|
||||
with pytest.raises(DefinitionalSchemaError, match=r"non-empty 'definitional_atoms'"):
|
||||
parse_gloss_entry(payload, strict=True)
|
||||
entry = parse_gloss_entry(payload, strict=True)
|
||||
assert entry.definitional_atoms == ()
|
||||
|
||||
@pytest.mark.parametrize("bad_version", [0, -1, "1", 1.0, True])
|
||||
def test_invalid_definition_version_rejected(self, bad_version: object) -> None:
|
||||
|
|
|
|||
225
tests/test_adr_0084_integration_closure.py
Normal file
225
tests/test_adr_0084_integration_closure.py
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
"""ADR-0084 integration test — substrate gate against ratified content.
|
||||
|
||||
After PR #64 (substrate) and PR #65 (content) both landed on main, this
|
||||
test was promised as the follow-up that exercises the substrate-callable
|
||||
``verify_definitional_closure`` against the *real* ratified content,
|
||||
not fixture packs. Its job:
|
||||
|
||||
1. Pin the substrate-vs-content handshake. Today the standalone
|
||||
``scripts/verify_definitional_closure.py`` is the agent's dev-loop
|
||||
tool; this test is the *gate-callable* equivalent that the
|
||||
ratification pipeline can invoke. Both must stay in agreement on
|
||||
what passes — divergence is a contract bug.
|
||||
|
||||
2. Catch content drift. If a future content edit adds an unresolved
|
||||
token, an empty atom list that should not be empty, or a gloss
|
||||
that depends on a non-mounted pack, this test fails before the
|
||||
edit lands on main — independently of whether the agent's
|
||||
standalone script also catches it.
|
||||
|
||||
3. Document the resolution-pool contract. ``mounted_pack_lemmas`` is
|
||||
the union of lexicon-resident lemmas across *production* packs.
|
||||
``en_minimal_v1`` is staging per the ADR-0084 pack-content brief
|
||||
and must NOT be load-bearing for the closure rule (deliberately
|
||||
excluded here even though the standalone verifier currently
|
||||
pools it).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from language_packs.compiler import load_pack_entries
|
||||
from language_packs.definitions import (
|
||||
load_pack_glosses,
|
||||
verify_definitional_closure,
|
||||
)
|
||||
from packs.primitives import load_primitives_pack
|
||||
|
||||
|
||||
# Packs that flipped ``definitional_layer: true`` via PR #65.
|
||||
# Production-only — does NOT include ``en_minimal_v1`` (staging) or
|
||||
# the Greek/Hebrew packs (per-lens glosses deferred per ADR-0084
|
||||
# scope limits).
|
||||
OPTED_IN_PACKS: tuple[str, ...] = (
|
||||
"en_core_cognition_v1",
|
||||
"en_core_action_v1",
|
||||
"en_core_attitude_v1",
|
||||
"en_core_causation_v1",
|
||||
"en_core_meta_v1",
|
||||
"en_core_polarity_v1",
|
||||
"en_core_quantitative_v1",
|
||||
"en_core_spatial_v1",
|
||||
"en_core_temporal_v1",
|
||||
"en_core_relations_v1",
|
||||
"en_core_relations_v2",
|
||||
"en_core_relations_v3",
|
||||
"en_collapse_anchors_v1",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def primitives_lemmas() -> frozenset[str]:
|
||||
return load_primitives_pack().lemmas
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def mounted_lex_lemmas() -> frozenset[str]:
|
||||
"""Union of lexicon-resident lemmas across production opted-in packs.
|
||||
|
||||
Built from each pack's ``lexicon.jsonl`` — the same source the
|
||||
standalone verifier uses — because that's the operational meaning
|
||||
of "a lemma in another mounted pack" (gloss entries are an additive
|
||||
overlay on the immutable lexicon, not its replacement).
|
||||
"""
|
||||
lemmas: set[str] = set()
|
||||
for pack_id in OPTED_IN_PACKS:
|
||||
for entry in load_pack_entries(pack_id):
|
||||
lemmas.add(entry.lemma.lower())
|
||||
return frozenset(lemmas)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Strict-parse every opted-in pack
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class TestStrictParseOptedInPacks:
|
||||
"""Every opted-in pack must strict-parse under the substrate.
|
||||
|
||||
A strict-parse failure means the content carries a schema violation
|
||||
the substrate would refuse to accept — caught here before it ships.
|
||||
"""
|
||||
|
||||
@pytest.mark.parametrize("pack_id", OPTED_IN_PACKS)
|
||||
def test_strict_parse(self, pack_id: str) -> None:
|
||||
entries = load_pack_glosses(pack_id, strict=True)
|
||||
assert entries, f"{pack_id} has no parseable gloss entries"
|
||||
|
||||
def test_total_entry_count_matches_standalone_verifier(self) -> None:
|
||||
# Standalone ``scripts/verify_definitional_closure.py`` reports
|
||||
# 333 entries. Substrate strict-parse must see the same set
|
||||
# so the two verifiers agree on scope.
|
||||
total = sum(len(load_pack_glosses(p, strict=True)) for p in OPTED_IN_PACKS)
|
||||
assert total == 333, (
|
||||
f"Substrate parsed {total} entries; standalone verifier reports 333. "
|
||||
f"Divergence means one of the two verifiers is silently skipping rows."
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Closure rule against the production resolution pool
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class TestClosureAgainstProductionPool:
|
||||
"""Every opted-in pack must close against (same-pack + production-
|
||||
mounted lexicon + primitives) — staging packs deliberately excluded.
|
||||
"""
|
||||
|
||||
@pytest.mark.parametrize("pack_id", OPTED_IN_PACKS)
|
||||
def test_pack_closes(
|
||||
self,
|
||||
pack_id: str,
|
||||
mounted_lex_lemmas: frozenset[str],
|
||||
primitives_lemmas: frozenset[str],
|
||||
) -> None:
|
||||
violations = verify_definitional_closure(
|
||||
pack_id,
|
||||
mounted_pack_lemmas=mounted_lex_lemmas,
|
||||
primitive_lemmas=primitives_lemmas,
|
||||
strict=True,
|
||||
)
|
||||
assert violations == (), (
|
||||
f"{pack_id} has {len(violations)} unresolved tokens against the "
|
||||
f"production pool: "
|
||||
+ ", ".join(f"{v.lemma}→{v.unresolved_token!r}" for v in violations)
|
||||
)
|
||||
|
||||
def test_no_violations_total(
|
||||
self,
|
||||
mounted_lex_lemmas: frozenset[str],
|
||||
primitives_lemmas: frozenset[str],
|
||||
) -> None:
|
||||
# Aggregate gate — the integration contract for ADR-0084 as a
|
||||
# whole: every opted-in pack closes against the production
|
||||
# pool with zero unresolved tokens.
|
||||
total_violations = 0
|
||||
for pack_id in OPTED_IN_PACKS:
|
||||
total_violations += len(
|
||||
verify_definitional_closure(
|
||||
pack_id,
|
||||
mounted_pack_lemmas=mounted_lex_lemmas,
|
||||
primitive_lemmas=primitives_lemmas,
|
||||
strict=True,
|
||||
)
|
||||
)
|
||||
assert total_violations == 0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Staging exclusion contract
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class TestStagingExclusion:
|
||||
"""``en_minimal_v1`` is staging and must not be load-bearing for the
|
||||
closure rule. If a future content edit makes any opted-in pack
|
||||
depend on en_minimal_v1 to resolve, this test catches it — that
|
||||
dependency would be a production-vs-staging leak.
|
||||
"""
|
||||
|
||||
def test_no_production_pack_depends_on_en_minimal_v1(
|
||||
self,
|
||||
primitives_lemmas: frozenset[str],
|
||||
) -> None:
|
||||
# Build a pool WITHOUT en_minimal_v1 — the production pool.
|
||||
production_pool: set[str] = set()
|
||||
for pack_id in OPTED_IN_PACKS:
|
||||
for entry in load_pack_entries(pack_id):
|
||||
production_pool.add(entry.lemma.lower())
|
||||
|
||||
# Then check: every opted-in pack closes against that pool.
|
||||
# If a pack secretly leans on en_minimal_v1, this fails with a
|
||||
# concrete unresolved-token list.
|
||||
for pack_id in OPTED_IN_PACKS:
|
||||
violations = verify_definitional_closure(
|
||||
pack_id,
|
||||
mounted_pack_lemmas=production_pool,
|
||||
primitive_lemmas=primitives_lemmas,
|
||||
strict=True,
|
||||
)
|
||||
assert not violations, (
|
||||
f"{pack_id} leaks into en_minimal_v1 — unresolved without "
|
||||
f"staging pool: "
|
||||
+ ", ".join(f"{v.lemma}→{v.unresolved_token!r}" for v in violations)
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Primitives floor coverage
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
class TestPrimitivesFloor:
|
||||
"""Floor-level sanity: the primitives pack must carry the foundation
|
||||
words the brief named — and the ones the content ended up leaning
|
||||
on. Catches accidental removals from the primitives pack."""
|
||||
|
||||
REQUIRED_FLOOR = (
|
||||
# From the original ADR-0084 brief's category list
|
||||
"exist", "be", "not_be", "not",
|
||||
"same", "different",
|
||||
"cause", "because", "change",
|
||||
"say", "mean",
|
||||
"if", "then", "and", "or",
|
||||
# The ones added during integration because the content leaned
|
||||
# on them via en_minimal_v1 (which is staging and not load-
|
||||
# bearing) — promoted to primitives so production closure is
|
||||
# robust.
|
||||
"can", "action",
|
||||
)
|
||||
|
||||
def test_required_primitives_present(self, primitives_lemmas: frozenset[str]) -> None:
|
||||
missing = sorted(set(self.REQUIRED_FLOOR) - primitives_lemmas)
|
||||
assert not missing, f"primitives pack missing required floor lemmas: {missing}"
|
||||
Loading…
Reference in a new issue