Shay 7784c39f9f feat(capability): implement ADR-0093 — Domain Pack Contract v1 wired in

Promotes ADR-0091 from proposed-but-unenforced to enforced. The CLI
command core capability domain-contract now runs the nine ADR-0091
predicates plus eval-lane artifact resolution; legacy structural-only
output remains available via --structural-only.

- new core/capability/domain_contract_predicates.py:
  evaluate_domain_contract(pack_id, *, data_root, chain_inventory,
  reviewer_registry) → DomainContractPredicateReport
- predicates wired:
  P1 manifest/checksum valid (via language_packs.compiler.load_pack)
  P2 gloss checksum (gloss-bearing packs only; otherwise vacuously pass)
  P3 domain_id ∈ DOMAIN_PACKS
  P4 teaching_chains entries ∈ TEACHING_CORPORA ∪ DOMAIN_CAPABILITY_CORPORA
  P5 ≥ 8 reviewed chains per claimed operator family from chain_report
  P6 ≥ 3 populated intent shapes per domain
  P7 every eval_lanes entry covers dev/public/holdout
  P8 reviewers resolve via ADR-0092 registry (consults can_review with
     scope='pack' and domain_id from contract)
  P9 known_gaps reference docs/gaps.md entries marked closed [x]
- _parse_gap_states reads docs/gaps.md format (- [x] / - [ ]) → {gap_id: closed?}
- _resolve_eval_lane_artifacts walks declared eval_lanes and surfaces
  per-split report path + SHA-256 (ADR-0093 item 4)
- CLI: cmd_capability_domain_contract now exits non-zero on any
  predicate failure; --structural-only preserves legacy behavior
- core.capability package re-exports new symbols (PredicateResult,
  DomainContractPredicateReport, evaluate_domain_contract)
- 24 unit tests covering contract presence/absence, each predicate
  positive + negative, gap parser, eval lane artifact surfacing,
  CLI default + structural-only paths, and determinism
- new evals/domain_contract_validation/ lane: 9 cases (positive +
  one negative per semantic predicate P3-P9 + determinism) passing
  9/9 byte-identical across runs (sha256 f9c06cde…)
- smoke 67/67, teaching 17/17, cognition 120/121 (pre-existing skip),
  ADR-0092..0095 tests 101/101; cognition eval byte-identical
  100/100/100/100

2026-05-21 18:33:23 -07:00

1.8 KiB

Raw Blame History

evals/domain_contract_validation — Lane Contract

ADR: ADR-0093 Invariant: domain_contract_v1_predicates_enforced

Purpose

Prove that ADR-0091's nine validation predicates fire under the implementation in :mod:core.capability.domain_contract_predicates.

The lane verifies, against synthetic manifests built in the runner's temp area:

positive: a contract that satisfies all nine predicates passes
one negative case per predicate (P3, P4, P5, P6, P7, P8, P9) where the relevant rule is minimally broken
the legacy structural-only output remains shaped as it was, so existing callers don't break

P1 and P2 (manifest checksum / gloss closure) require a fully compiled pack and are exercised against the in-tree en_mathematics_logic_v1 pack rather than against synthetic fixtures.

Cases

positive_all_predicates_pass — synthetic pack with valid contract.
p3_unknown_domain — domain_id outside the ledger registry.
p4_unregistered_chain — teaching_chains references a non-existent corpus.
p5_chain_shortfall — one claimed operator family has < 8 chains.
p6_too_few_intents — domain has < 3 populated intent shapes.
p7_incomplete_splits — eval_lanes entry missing holdout.
p8_unknown_reviewer — reviewer id not in the registry.
p9_open_gap — known_gaps references an unknown gap id.
determinism — same inputs across two runs → byte-identical report.

Determinism

The runner emits results/v1_dev.json. Two consecutive runs must produce identical bytes (SHA-256 pinned). The chain inventory and reviewer registry are injected so the lane does not depend on git-state-sensitive sources.

Exit code

Non-zero on any case whose actual outcome diverges from the case spec.

1.8 KiB Raw Blame History