After PR #64 (substrate) and PR #65 (content) both landed on main, this
test is the promised follow-up that exercises the substrate-callable
verify_definitional_closure against the real ratified content rather
than fixture packs. It pins three contracts:
1. Substrate-vs-content handshake. The standalone
scripts/verify_definitional_closure.py is the agent's dev-loop
tool; this test is the gate-callable equivalent the ratification
pipeline can invoke. Both must agree on what passes — divergence
is a contract bug.
2. Content drift catcher. Any future content edit that adds an
unresolved token / non-mounted dependency / silent staging leak
fails this test before the edit lands on main.
3. Staging exclusion. en_minimal_v1 is staging per the ADR-0084
pack-content brief and must not be load-bearing for the closure
rule. Test-pinned via a production-pool subtest.
Substrate fix: allow empty definitional_atoms
The substrate's strict parser previously rejected empty
definitional_atoms. That stance was wrong: per the ADR-0084 pack-
content brief, the per-entry atom list excludes articles, prepositions,
and copulas. A gloss whose every content word is a function word
(e.g. en_core_temporal_v1/prior → "before") has zero content atoms by
construction. The closure rule passes vacuously when atoms is empty
— there is nothing to close. The gloss-vs-atoms mismatch check in
the standalone verifier is the second-layer gate that distinguishes
by-construction emptiness (legitimate) from by-omission emptiness
(laziness). Substrate parser shouldn't double-gate the same concern.
The corresponding substrate test flipped from
test_empty_definitional_atoms_rejected to
test_empty_definitional_atoms_accepted, with comment explaining the
reasoning.
Primitives expansion: can + action
Two content entries (en_core_cognition_v1/person → "who can know and
do" and en_core_meta_v1/intend → "decide before an action") leaned on
'can' and 'action' as atom references. Today those lemmas resolve
ONLY via en_minimal_v1/lexicon.jsonl — the staging pack. That's a
production-vs-staging leak: production content should not be load-
bearing on staging.
Two clean alternatives:
(a) rewrite the two glosses to avoid 'can' and 'action'
(b) promote 'can' and 'action' to primitives
Chose (b): both lemmas are genuinely terminal-feeling (can is a basic
capability modal; action is an irreducible "what is done"); the
content reads more naturally with them present than with paraphrased
substitutes; and the floor was always going to need both eventually.
The cost is two primitives.jsonl rows + checksum + count bump.
Verification:
scripts/verify_definitional_closure.py exit 0
tests/test_adr_0084_integration_closure.py 30/30 pass
tests/test_adr_0084_definitional_substrate.py 39/39 pass
core test --suite smoke -q 67/67
core test --suite packs -q 6/6
core eval cognition byte-identical
(100/91.7/100/100)
Two-layer gate now in place:
- standalone verifier (dev loop, gloss/atom mismatch check)
- substrate verifier (ratification gate, parametrized over every
opted-in pack, staging-exclusion test, primitives floor coverage)
* docs(adr-0084): propose definitional layer + prompt-diversity suite
Three companion artifacts proposing the next substantive design step
after ADR-0083:
1. ADR-0084 (Proposed) — Definitional Layer for Lexicon Packs
Optional `definition` block on pack entries: gloss,
definitional_atoms, predicates_invited, definition_version,
provenance. Pack-level opt-in. Closure rule: every word in a
gloss must resolve to a same-pack lemma, another mounted pack's
lemma, or a primitive in a new `packs/primitives/` pack.
NO composer change in this ADR (sequenced for ADR-0085) —
ratify substrate before any consumer depends on it.
2. evals/prompt_diversity/ (Proposed) — companion eval lane
~50 cases across question-shape × sophistication × domain,
measuring three new metrics: response_shape_fit,
audit_in_surface_rate (quantifies the trust-boundary leak into
user surfaces), gloss_quote_rate (zero today; rises with future
gloss-aware composer). No v1 pass thresholds — the lane
establishes a baseline distribution so future work has
something to move. 26 seed cases authored covering all 21
categories.
3. docs/handoff/ADR-0084-pack-content-brief.md — paste-ready brief
for a cheaper/faster dev agent to produce the pack content in
parallel. Self-contained, 5 sequenced phases (primitives pack
→ extend 9 existing glosses → add to relations/anchors → write
closure verifier → run safety lanes), explicit don't-touch list
(no composer / runtime / algebra / Greek+Hebrew packs / schema
parser), no-LLM-glosses discipline, per-phase acceptance.
Discovery while drafting: 9 packs already carry glosses.jsonl
under language_packs/data/ with a flat schema (78 entries in
en_core_cognition_v1 alone). The brief reflects that — most
work is extending existing entries, not authoring from scratch.
Strategic context: ADR-0083 raised the *depth* ceiling on chain
composition; ADR-0084 raises the *fidelity* ceiling. The φ
separation probe (memory: phi-separation-falsified) established
that semantic capability lives in chain composition, not in φ
geometry, so deepening the composer's substrate is the natural
next step. ADR-0084 → 0085 (gloss-aware composer) → 0086
(predicate licensing at ratification) is the planned sequence.
* feat(adr-0084): substrate — schema parser, primitives loader, closure verifier
Substrate-only code-side for ADR-0084 (Definitional Layer for Lexicon Packs).
No composer touches the new fields yet; consumer integration is ADR-0085.
Schema (additive, default preserves byte-identity)
- LanguagePackManifest.definitional_layer: bool = False
- compiler loader propagates the flag from manifest.json
language_packs/definitions.py (new)
- GlossEntry dataclass: lemma, gloss, pos, definitional_atoms,
predicates_invited, definition_version, provenance_ids
- parse_gloss_entry(payload, *, strict) — strict mode enforces ADR-0084
§Schema validation row-by-row: required keys, typed lists, no
unknown keys, positive definition_version; lax mode preserves the
legacy two-field shape for back-compat
- load_pack_glosses(pack_id, *, strict) with cache + clear hook
- verify_definitional_closure(pack_id, *, mounted_pack_lemmas,
primitive_lemmas, strict) returning tuple[ClosureViolation, ...];
case-insensitive resolution; cycles permitted per ADR
packs/primitives/loader.py (new)
- Sister loader to packs/safety/ and packs/identity/
- PrimitivesPack frozen dataclass with .lemmas frozenset
- Gates: checksum match, kind=='primitives', definitional_layer:true,
never_auto_mutable:true, pack_id matches dir, primitive_count
cross-check, duplicate-lemma rejection, path-traversal rejection,
strict per-entry schema with allow-list
- DEFAULT_PRIMITIVES_PACK = 'en_semantic_primitives_v1'
tests/test_adr_0084_definitional_substrate.py
- 38 tests covering strict parser (each required key rejection, unknown
key rejection, empty predicates_invited allowed, empty
definitional_atoms rejected, invalid definition_version), lax
parser back-compat, load_pack_glosses (missing/strict raise/lax
skip/malformed JSON), closure verifier (same-pack/primitive/mounted/
unresolved/case-insensitive), primitives loader (every gate), and
a back-compat check that every shipped pack still ratifies with
definitional_layer=False
Lanes: smoke 67/0, cognition 120/0/1, teaching 17/0, runtime 19/0,
packs 6/0. Cognition eval byte-identical 100/91.7/100/100.
When the content PR lands (primitives.jsonl + extended glosses.jsonl
under ADR-0084-pack-content-brief.md), the gate catches any closure-rule
violation without further code change.
* feat(evals): prompt_diversity lane runner — measurement instrument for ADR-0084+
Implements the runner against the existing contract.md + 26-case v1
public split. Lane auto-discovered by evals.framework via the standard
contract + runner convention.
Runner (evals/prompt_diversity/runner.py)
- run_lane(cases, *, config, workers) -> LaneReport
- 5 metrics: intent_accuracy, versor_closure_rate (carried over from
cognition), plus the three new lane-specific metrics —
response_shape_fit, audit_in_surface_rate, gloss_quote_rate
- breakdown dict groups by (question_shape, sophistication, domain)
per contract §How to read the output
- mirrors evals.cognition.runner's parallel worker pattern
Per-shape classifier (deliberately substring/regex-simple at v1)
- predicate_identity, explanation, sequence, two_subject_contrast,
narrative, honest_disclosure
- Unknown shape => neutral pass (don't penalise new categories)
Audit-leak detector
- trust-boundary preamble markers (teaching-grounded (, pack-grounded
(, No session evidence yet.)
- dotted semantic-domain tag regex (cognition.illumination, etc.)
Gloss-quote detector
- resolves expected_terms via chat.pack_resolver.resolve_gloss
- 4-token contiguous-window match against surface (high-confidence
"gloss actually quoted", not "shared one common word")
Tests (tests/test_prompt_diversity_runner.py — 23)
- shape classifier parametrized over the six expected_shape values
- audit-leak detector parametrized over preamble + tag + clean cases
- end-to-end on v1 public:
* versor_closure_rate == 1.0 (only v1 pass threshold per contract)
* every metric in [0, 1]
* breakdown groups present with the four per-cell metrics
* diversity gate: >=5 question shapes, >=3 domains
(defends against future regressions that collapse the suite
back to a cognition-shaped fixture)
v1/public baseline (26 cases)
intent_accuracy : 65.4% (contract predicted 70-85%)
versor_closure_rate : 100.0% (only v1 pass threshold) PASS
response_shape_fit : 53.8% (contract predicted low)
audit_in_surface_rate: 42.3% (contract predicted ~100%)
gloss_quote_rate : 7.7% (contract predicted 0%)
Three baseline surprises worth noting in the report (NOT failures —
the v1 lane is explicitly there to establish the distribution):
- audit_in_surface_rate at 42% (not 100%) means the chain-walk leak
fires on ~11/26; the other 15 are honest-disclosure cases that
emit no audit envelope. Sharpens the future surface-vs-envelope
ADR's actual target: grounded surfaces specifically.
- response_shape_fit at 54% (not "low") — classifier likely has
false positives on the ", which " cause-marker. Worth tightening
once we have an ADR-0085 baseline to compare against.
- intent_accuracy at 65% (below predicted 70-85%) — classifier dips
harder on adversarial/cross-pack than expected. Real gap.
All five smoke/cognition/teaching/runtime/packs lanes still green;
core eval cognition byte-identical 100/91.7/100/100.
* feat(packs): ADR-0084 pack content (primitives + extend glosses + closure verifier) (#65)
* feat(packs): ADR-0084 pack content
* feat(packs): repair ADR-0084 definitional content
* test(adr-0084): adjust substrate manifest tests for post-#65 content reality
PR #65 flipped definitional_layer:true on 13 English packs (9 core +
4 relations + collapse-anchors). The substrate's previous test
test_existing_packs_unchanged asserted that en_core_cognition_v1 +
en_core_relations_v1 still had definitional_layer:False — which was
the right pre-content invariant but is wrong post-content.
Replace it with two complementary tests that hold against real content:
- test_non_opted_packs_default_false:
pins that packs that DIDN'T flip the flag (en_minimal_v1,
he_core_cognition_v1, grc_logos_cognition_v1) still surface
definitional_layer=False through the loader. Defends against
a future change accidentally flipping the flag on a non-opted
pack.
- test_opted_packs_carry_flag:
pins that packs that DID flip the flag (en_core_cognition_v1,
en_core_relations_v1) surface definitional_layer=True through
the loader. Proves the substrate's manifest-field propagation
works against real ratified content, not just fixture packs.
Net: +1 test, same intent (substrate ratifies the manifest field
correctly), now with real-content coverage on both sides of the gate.
All 62 ADR-0084 substrate + prompt-diversity tests pass.
Shay