0ad97e5ef7
* perf(tests): extract math_teaching_corpus lane from pytest into CI lane SHAs
The two slowest tests in the pytest suite were:
388s test_adr_0131_2_teaching_corpus_lane::test_report_is_byte_equal_across_runs
161s test_adr_0131_2_teaching_corpus_lane::test_lane_passes_exit_criterion
Both invoked build_report() from evals.math_teaching_corpus.v1.runner —
the canonical math-teaching-corpus lane runner — once for the exit
criterion and again for byte-equality. Together: 549s = 9m 9s, 30% of
the full pytest suite, recomputed on every developer run.
This is the exact 'lane runner invoked from pytest' anti-pattern that
the existing scripts/verify_lane_shas.py CI job is designed to absorb.
The other 7 lanes (reviewer_registry, miner_loop_closure, etc.) all
run in CI via SHA pinning rather than in pytest.
Changes:
scripts/verify_lane_shas.py — add math_teaching_corpus_v1 spec +
PINNED_SHAS entry (eaf160d145da29f9..., computed locally from
a clean run of the lane in this commit's tree).
scripts/generate_claims.py — add _LANE_ADR entry (ADR-0131) +
claim text. Failing fast on missing lanes is by design.
CLAIMS.md — regenerated; one new row.
tests/test_adr_0131_2_teaching_corpus_lane.py — delete TestLaneGate
class (2 tests, 549s). Retain TestDatasetIntegrity (5 tests),
TestBoundedDomain (2), TestHonestEvidence (1) — these are
fast (0.26s total) and pin contracts the lane runner does not
cover (dataset shape, lemma boundedness, evidence reachability).
Replace deletion with an explanatory comment block.
The deleted contracts are still enforced — just in CI instead of
pytest:
exit criterion → runner exit code (returns 1 on failure)
byte-equality → PINNED_SHAS verification (SHA-256 of report.json)
Verified locally:
scripts/verify_lane_shas.py — 8/8 lanes match pinned SHAs
pytest tests/test_adr_0131_2_teaching_corpus_lane.py — 8/8 pass in 0.26s
Expected full-suite delta: -549s (from ~30m to ~21m). Further speedup
will come from the upcoming full-pytest CI gate with pytest-xdist -n4.
* ci: bump lane-shas timeout 12m → 20m for new math_teaching_corpus lane
The math_teaching_corpus_v1 lane added in this PR runs in ~5-6 min,
pushing the total lane-shas job over the previous 12-min timeout.
First CI run cancelled at 12m17s. Bumping to 20m gives ~8m headroom.
* fix(ci): bump lane subprocess timeout 300s→900s + add math_teaching_corpus to test_lane_sha_verifier EXPECTED_LANES
Two issues surfaced by CI run on the prior commit:
1. The math_teaching_corpus lane takes ~142s wall-clock locally (3.79
cores × ~538s CPU). On CI's single/dual-core runner that translates
to ~5-9 min, exceeding the 300s subprocess timeout in
scripts/verify_lane_shas.py. Bumping to 900s gives ~60% headroom.
2. tests/test_lane_sha_verifier.py::TestExpectedLaneCoverage::test_all_expected_lanes_covered
hardcodes the expected lane set. Adding math_teaching_corpus_v1 to
LANE_SPECS triggered the 'extra lanes' assertion. Adding it to
EXPECTED_LANES (the file's own contract: 'if intentional, add here').
64 lines
1.5 KiB
YAML
64 lines
1.5 KiB
YAML
name: lane-shas
|
|
|
|
# Verify that every ADR-0092..0099 lane produces its pinned SHA-256
|
|
# report. A failing job means a lane's deterministic output changed
|
|
# without an explicit ADR-tracked pin update via:
|
|
#
|
|
# python scripts/verify_lane_shas.py --update
|
|
#
|
|
# Single source of truth for the pinned values is scripts/verify_lane_shas.py.
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
branches: [main]
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: lane-shas-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
verify:
|
|
name: verify pinned lane SHAs
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
|
|
steps:
|
|
- name: checkout
|
|
uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 1
|
|
|
|
- name: set up python
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.11'
|
|
cache: 'pip'
|
|
|
|
- name: install dependencies
|
|
run: |
|
|
python -m pip install --upgrade pip
|
|
pip install -e . pyyaml pytest
|
|
|
|
- name: verify lane SHAs
|
|
env:
|
|
PYTHONPATH: ${{ github.workspace }}
|
|
run: |
|
|
python scripts/verify_lane_shas.py
|
|
|
|
- name: verify CLAIMS.md is current
|
|
env:
|
|
PYTHONPATH: ${{ github.workspace }}
|
|
run: |
|
|
python scripts/generate_claims.py --check
|
|
|
|
- name: emit machine-readable report (on failure)
|
|
if: failure()
|
|
env:
|
|
PYTHONPATH: ${{ github.workspace }}
|
|
run: |
|
|
python scripts/verify_lane_shas.py --json || true
|