0ad97e5ef7
* perf(tests): extract math_teaching_corpus lane from pytest into CI lane SHAs
The two slowest tests in the pytest suite were:
388s test_adr_0131_2_teaching_corpus_lane::test_report_is_byte_equal_across_runs
161s test_adr_0131_2_teaching_corpus_lane::test_lane_passes_exit_criterion
Both invoked build_report() from evals.math_teaching_corpus.v1.runner —
the canonical math-teaching-corpus lane runner — once for the exit
criterion and again for byte-equality. Together: 549s = 9m 9s, 30% of
the full pytest suite, recomputed on every developer run.
This is the exact 'lane runner invoked from pytest' anti-pattern that
the existing scripts/verify_lane_shas.py CI job is designed to absorb.
The other 7 lanes (reviewer_registry, miner_loop_closure, etc.) all
run in CI via SHA pinning rather than in pytest.
Changes:
scripts/verify_lane_shas.py — add math_teaching_corpus_v1 spec +
PINNED_SHAS entry (eaf160d145da29f9..., computed locally from
a clean run of the lane in this commit's tree).
scripts/generate_claims.py — add _LANE_ADR entry (ADR-0131) +
claim text. Failing fast on missing lanes is by design.
CLAIMS.md — regenerated; one new row.
tests/test_adr_0131_2_teaching_corpus_lane.py — delete TestLaneGate
class (2 tests, 549s). Retain TestDatasetIntegrity (5 tests),
TestBoundedDomain (2), TestHonestEvidence (1) — these are
fast (0.26s total) and pin contracts the lane runner does not
cover (dataset shape, lemma boundedness, evidence reachability).
Replace deletion with an explanatory comment block.
The deleted contracts are still enforced — just in CI instead of
pytest:
exit criterion → runner exit code (returns 1 on failure)
byte-equality → PINNED_SHAS verification (SHA-256 of report.json)
Verified locally:
scripts/verify_lane_shas.py — 8/8 lanes match pinned SHAs
pytest tests/test_adr_0131_2_teaching_corpus_lane.py — 8/8 pass in 0.26s
Expected full-suite delta: -549s (from ~30m to ~21m). Further speedup
will come from the upcoming full-pytest CI gate with pytest-xdist -n4.
* ci: bump lane-shas timeout 12m → 20m for new math_teaching_corpus lane
The math_teaching_corpus_v1 lane added in this PR runs in ~5-6 min,
pushing the total lane-shas job over the previous 12-min timeout.
First CI run cancelled at 12m17s. Bumping to 20m gives ~8m headroom.
* fix(ci): bump lane subprocess timeout 300s→900s + add math_teaching_corpus to test_lane_sha_verifier EXPECTED_LANES
Two issues surfaced by CI run on the prior commit:
1. The math_teaching_corpus lane takes ~142s wall-clock locally (3.79
cores × ~538s CPU). On CI's single/dual-core runner that translates
to ~5-9 min, exceeding the 300s subprocess timeout in
scripts/verify_lane_shas.py. Bumping to 900s gives ~60% headroom.
2. tests/test_lane_sha_verifier.py::TestExpectedLaneCoverage::test_all_expected_lanes_covered
hardcodes the expected lane set. Adding math_teaching_corpus_v1 to
LANE_SPECS triggered the 'extra lanes' assertion. Adding it to
EXPECTED_LANES (the file's own contract: 'if intentional, add here').
3.3 KiB
3.3 KiB
CLAIMS
Every row below is mechanically derived from in-tree state and
verified by CI. Tier 1 rows come from core.capability.ledger_report;
Tier 2 rows come from scripts/verify_lane_shas.py's pinned SHAs.
Tier 1 — Ratified domains
Each row asserts: the domain's packs pass all nine ADR-0091 contract
predicates, declared operator chains meet the ≥8 / ≥3-intent floor,
the reviewer registry resolves the primary reviewer, and the ledger
status predicate evaluates to reasoning-capable with no open gaps.
| Domain | Status | ADR | Packs | Open gaps |
|---|---|---|---|---|
philosophy_theology |
reasoning-capable | ADR-0085 | 2 | 0 |
mathematics_logic |
expert | ADR-0097 | 1 | 0 |
physics |
audit-passed | ADR-0100 | 1 | 0 |
systems_software |
audit-passed | ADR-0101 | 1 | 0 |
hebrew_greek_textual_reasoning |
reasoning-capable | ADR-0102 | 4 | 0 |
Tier 2 — Pinned lane reports
Each row asserts: running the lane's runner produces a JSON
report whose SHA-256 matches the pinned value below. Mismatch
is a CI failure (.github/workflows/lane-shas.yml).
| ADR | Lane | Purpose | Report path | Pinned SHA-256 |
|---|---|---|---|---|
| ADR-0092 | reviewer_registry |
Reviewer registry schema validates + bootstrap entry self-seals | evals/reviewer_registry/results/v1_dev.json |
681a2aab5aa4ffd58cd837ce5673c8b2a9545b570117aec3c02726a12f6876e6 |
| ADR-0093 | domain_contract_validation |
All ratified packs satisfy the 9 ADR-0091 contract predicates | evals/domain_contract_validation/results/v1_dev.json |
98ace04e3f02bbc5a8ad655bb6593c3f1ee64cb67014f1122fe6c3c85f48d22f |
| ADR-0095 | miner_loop_closure |
Miner-sourced proposals route through single reviewed teaching path | evals/miner_loop_closure/results/v1_dev.json |
9f071733abe7dcacf759f928548ce738fb639af3fd6e4c621a651b306d7e77ce |
| ADR-0096 | fabrication_control_summary |
Phantom endpoints / cross-pack non-bridges / sibling collapses refuse | evals/fabrication_control/results/v1_summary.json |
01e1b6b711141f2b4a14551d7df3ea482d8d6dd7b364a25c509f4f8d08cda8a8 |
| ADR-0098 | demo_composition |
Demos compose from shipped modules; no parallel mechanism | evals/demo_composition/results/v1_dev.json |
3a3d09f3a87462737e615c2dd3481b9e13e5ff8fadee0043c37873494ded556d |
| ADR-0099 | public_demo |
Public showcase runs deterministically under 30s; all claims supported | evals/public_demo/results/v1_dev.json |
888ddd0d12635d709f91898cf06601062168694d870f776d2b0b7710e5d68cbd |
| ADR-0104 | curriculum_loop_closure |
Curriculum-sourced proposals route through single reviewed teaching path | evals/curriculum_loop_closure/results/v1_dev.json |
b46d56b2d209172cc3ffaf3776dc8dcfe55093f13587c5cb67372be6dfa23e8d |
| ADR-0131 | math_teaching_corpus_v1 |
Math teaching corpus replays deterministically; all chains pass exit criterion (correct_rate=1.0, wrong=0) | evals/math_teaching_corpus/v1/report.json |
eaf160d145da29f9050ede8d58bf111b0f651dd40aeae9201857d0b97e014dd4 |
Verification
python3 scripts/verify_lane_shas.py # verify all Tier 2 SHAs
core test --suite full -q # exercises Tier 1 invariants