0ad97e5ef7
* perf(tests): extract math_teaching_corpus lane from pytest into CI lane SHAs
The two slowest tests in the pytest suite were:
388s test_adr_0131_2_teaching_corpus_lane::test_report_is_byte_equal_across_runs
161s test_adr_0131_2_teaching_corpus_lane::test_lane_passes_exit_criterion
Both invoked build_report() from evals.math_teaching_corpus.v1.runner —
the canonical math-teaching-corpus lane runner — once for the exit
criterion and again for byte-equality. Together: 549s = 9m 9s, 30% of
the full pytest suite, recomputed on every developer run.
This is the exact 'lane runner invoked from pytest' anti-pattern that
the existing scripts/verify_lane_shas.py CI job is designed to absorb.
The other 7 lanes (reviewer_registry, miner_loop_closure, etc.) all
run in CI via SHA pinning rather than in pytest.
Changes:
scripts/verify_lane_shas.py — add math_teaching_corpus_v1 spec +
PINNED_SHAS entry (eaf160d145da29f9..., computed locally from
a clean run of the lane in this commit's tree).
scripts/generate_claims.py — add _LANE_ADR entry (ADR-0131) +
claim text. Failing fast on missing lanes is by design.
CLAIMS.md — regenerated; one new row.
tests/test_adr_0131_2_teaching_corpus_lane.py — delete TestLaneGate
class (2 tests, 549s). Retain TestDatasetIntegrity (5 tests),
TestBoundedDomain (2), TestHonestEvidence (1) — these are
fast (0.26s total) and pin contracts the lane runner does not
cover (dataset shape, lemma boundedness, evidence reachability).
Replace deletion with an explanatory comment block.
The deleted contracts are still enforced — just in CI instead of
pytest:
exit criterion → runner exit code (returns 1 on failure)
byte-equality → PINNED_SHAS verification (SHA-256 of report.json)
Verified locally:
scripts/verify_lane_shas.py — 8/8 lanes match pinned SHAs
pytest tests/test_adr_0131_2_teaching_corpus_lane.py — 8/8 pass in 0.26s
Expected full-suite delta: -549s (from ~30m to ~21m). Further speedup
will come from the upcoming full-pytest CI gate with pytest-xdist -n4.
* ci: bump lane-shas timeout 12m → 20m for new math_teaching_corpus lane
The math_teaching_corpus_v1 lane added in this PR runs in ~5-6 min,
pushing the total lane-shas job over the previous 12-min timeout.
First CI run cancelled at 12m17s. Bumping to 20m gives ~8m headroom.
* fix(ci): bump lane subprocess timeout 300s→900s + add math_teaching_corpus to test_lane_sha_verifier EXPECTED_LANES
Two issues surfaced by CI run on the prior commit:
1. The math_teaching_corpus lane takes ~142s wall-clock locally (3.79
cores × ~538s CPU). On CI's single/dual-core runner that translates
to ~5-9 min, exceeding the 300s subprocess timeout in
scripts/verify_lane_shas.py. Bumping to 900s gives ~60% headroom.
2. tests/test_lane_sha_verifier.py::TestExpectedLaneCoverage::test_all_expected_lanes_covered
hardcodes the expected lane set. Adding math_teaching_corpus_v1 to
LANE_SPECS triggered the 'extra lanes' assertion. Adding it to
EXPECTED_LANES (the file's own contract: 'if intentional, add here').
105 lines
3.7 KiB
Python
105 lines
3.7 KiB
Python
"""Pin scripts/verify_lane_shas.py's pin-block shape.
|
|
|
|
The full verify-all suite (which re-runs every lane) is expensive
|
|
(~30s). It is exercised in CI by the lane-shas workflow. Locally
|
|
``pytest`` covers two cheaper guarantees:
|
|
|
|
1. Every shipped ADR lane has a pin in ``PINNED_SHAS``.
|
|
2. Every ``LaneSpec.runner_module`` actually exists on disk and is
|
|
marked as a Python file.
|
|
|
|
These two together mean no lane silently drops out of CI coverage.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
|
|
|
|
|
def _load_verifier_module():
|
|
"""Load scripts/verify_lane_shas.py as a module.
|
|
|
|
The scripts directory is not on the default import path; load
|
|
directly so the pins/specs remain authoritative without forcing a
|
|
package reshuffle.
|
|
"""
|
|
path = REPO_ROOT / "scripts" / "verify_lane_shas.py"
|
|
spec = importlib.util.spec_from_file_location("_verify_lane_shas", path)
|
|
assert spec is not None and spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
sys.modules["_verify_lane_shas"] = module
|
|
spec.loader.exec_module(module)
|
|
return module
|
|
|
|
|
|
verifier = _load_verifier_module()
|
|
|
|
|
|
class TestPinBlockShape:
|
|
def test_every_lane_spec_has_a_pin(self) -> None:
|
|
lane_ids = {spec.lane_id for spec in verifier.LANE_SPECS}
|
|
pinned = set(verifier.PINNED_SHAS.keys())
|
|
missing = lane_ids - pinned
|
|
assert not missing, f"lanes without pinned SHA: {missing}"
|
|
|
|
def test_no_orphan_pins(self) -> None:
|
|
"""A pin without a matching LaneSpec is dead code."""
|
|
lane_ids = {spec.lane_id for spec in verifier.LANE_SPECS}
|
|
pinned = set(verifier.PINNED_SHAS.keys())
|
|
orphans = pinned - lane_ids
|
|
assert not orphans, f"orphan pins (no matching LaneSpec): {orphans}"
|
|
|
|
def test_every_pin_is_64_hex_chars(self) -> None:
|
|
for lane_id, sha in verifier.PINNED_SHAS.items():
|
|
assert re.fullmatch(r"[0-9a-f]{64}", sha), (
|
|
f"pin for {lane_id!r} is not a 64-char hex SHA-256: {sha!r}"
|
|
)
|
|
|
|
def test_every_lane_spec_runner_exists(self) -> None:
|
|
for spec in verifier.LANE_SPECS:
|
|
assert spec.runner_path.exists(), (
|
|
f"lane {spec.lane_id!r} runner not found at {spec.runner_path}"
|
|
)
|
|
assert spec.runner_path.suffix == ".py"
|
|
|
|
def test_every_lane_spec_canonical_report_path_under_repo(self) -> None:
|
|
for spec in verifier.LANE_SPECS:
|
|
assert spec.canonical_report.is_relative_to(REPO_ROOT)
|
|
|
|
|
|
class TestExpectedLaneCoverage:
|
|
"""The verifier MUST cover all six ADR-0092..0099 lanes.
|
|
|
|
Hard-code the canonical lane ids so silently dropping any one
|
|
fails this test. ADR-0094 and ADR-0097 are schema/ratification
|
|
only — no eval lane — and intentionally absent.
|
|
"""
|
|
|
|
EXPECTED_LANES = frozenset(
|
|
{
|
|
"reviewer_registry", # ADR-0092
|
|
"miner_loop_closure", # ADR-0095
|
|
"domain_contract_validation", # ADR-0093
|
|
"fabrication_control_summary", # ADR-0096
|
|
"demo_composition", # ADR-0098
|
|
"public_demo", # ADR-0099
|
|
"curriculum_loop_closure",
|
|
"math_teaching_corpus_v1", # ADR-0131
|
|
}
|
|
)
|
|
|
|
def test_all_expected_lanes_covered(self) -> None:
|
|
actual = {spec.lane_id for spec in verifier.LANE_SPECS}
|
|
missing = self.EXPECTED_LANES - actual
|
|
extra = actual - self.EXPECTED_LANES
|
|
assert not missing, f"missing expected lanes: {missing}"
|
|
assert not extra, (
|
|
f"unexpected extra lanes: {extra} (add to EXPECTED_LANES in test "
|
|
"if intentional)"
|
|
)
|