fix(ci): stream lane-shas progress + distinguish timeout from content drift #50

Merged

core-labs merged 2 commits from fix/ci-lane-shas-timeout-diagnostics into main

2026-07-16 20:17:04 +00:00

core-labs commented

2026-07-16 13:40:32 +00:00

Owner

Summary

lane-shas.yml has intermittently failed under Act-runner contention with
ambiguous logs — see the #38 postmortem in
docs/plans/adr-0241-0242-session-completion-summary-2026-07-15.md: run
#167 failed at ~22m39s, but a local scripts/verify_lane_shas.py run showed
9/9 pins matching. The failure was never conclusively attributed to a
timeout vs. an incomplete Forgejo log, because the script buffers all
per-lane output until every lane finishes — so if the job gets killed
mid-run, the log shows nothing about which lane was slow or why.

This PR does not touch any lane content or logic. Two small, additive
changes to scripts/verify_lane_shas.py:

Stream progress per lane (-> lane_id ... done (Ns), flushed
immediately) instead of buffering output until the end. A hang is now
visible in the CI log the moment it happens.
Configurable, distinguishable timeout. The per-lane subprocess
timeout (previously a hardcoded 900) is now overridable via
CORE_LANE_VERIFY_TIMEOUT_S (default unchanged), matching the existing
CORE_SHOWCASE_HARD_BUDGET-style knob convention already in this repo.
A TimeoutExpired is now reported distinctly from a genuine SHA
mismatch, and the remediation text no longer suggests --update
(re-pinning) when the real cause is runner contention.

Test plan

uv run ruff check scripts/verify_lane_shas.py — clean
uv run python -m py_compile scripts/verify_lane_shas.py — OK
uv run pytest tests/test_lane_sha_verifier.py -q — 6 passed (pin/shape
coverage unaffected — no LANE_SPECS/PINNED_SHAS changes)
uv run python scripts/verify_lane_shas.py run end-to-end locally —
9/9 lanes matched pinned SHAs, streaming output confirmed correct
per-lane timing (math_teaching_corpus_v1 alone: 300s — notably,
nearly a third of the 900s per-lane budget on an idle 10-core Mac;
under 2-vCPU Act-runner contention this margin is exactly what could
disappear)

[Verification]: Smoke suite not required for this change (CI-tooling-only,
no runtime/algebra/cognition path touched); ruff + targeted pytest +
full local lane-shas run passed as above.

## Summary `lane-shas.yml` has intermittently failed under Act-runner contention with ambiguous logs — see the #38 postmortem in `docs/plans/adr-0241-0242-session-completion-summary-2026-07-15.md`: run #167 failed at ~22m39s, but a local `scripts/verify_lane_shas.py` run showed 9/9 pins matching. The failure was never conclusively attributed to a timeout vs. an incomplete Forgejo log, because the script buffers all per-lane output until every lane finishes — so if the job gets killed mid-run, the log shows nothing about which lane was slow or why. This PR does not touch any lane content or logic. Two small, additive changes to `scripts/verify_lane_shas.py`: 1. **Stream progress per lane** (`-> lane_id ... done (Ns)`, flushed immediately) instead of buffering output until the end. A hang is now visible in the CI log the moment it happens. 2. **Configurable, distinguishable timeout.** The per-lane subprocess timeout (previously a hardcoded `900`) is now overridable via `CORE_LANE_VERIFY_TIMEOUT_S` (default unchanged), matching the existing `CORE_SHOWCASE_HARD_BUDGET`-style knob convention already in this repo. A `TimeoutExpired` is now reported distinctly from a genuine SHA mismatch, and the remediation text no longer suggests `--update` (re-pinning) when the real cause is runner contention. ## Test plan - [x] `uv run ruff check scripts/verify_lane_shas.py` — clean - [x] `uv run python -m py_compile scripts/verify_lane_shas.py` — OK - [x] `uv run pytest tests/test_lane_sha_verifier.py -q` — 6 passed (pin/shape coverage unaffected — no LANE_SPECS/PINNED_SHAS changes) - [x] `uv run python scripts/verify_lane_shas.py` run end-to-end locally — 9/9 lanes matched pinned SHAs, streaming output confirmed correct per-lane timing (`math_teaching_corpus_v1` alone: 300s — notably, nearly a third of the 900s per-lane budget on an idle 10-core Mac; under 2-vCPU Act-runner contention this margin is exactly what could disappear) [Verification]: Smoke suite not required for this change (CI-tooling-only, no runtime/algebra/cognition path touched); ruff + targeted pytest + full local lane-shas run passed as above.

core-labs added 1 commit 2026-07-16 13:40:42 +00:00

fix(ci): stream lane-shas progress + distinguish timeout from content drift

smoke / smoke (-m "not quarantine") (pull_request) Failing after 53s

Details

lane-shas / verify pinned lane SHAs (pull_request) Failing after 14m15s

Details

05e9d6d28e

lane-shas.yml has intermittently failed under Act-runner contention with
truncated/ambiguous logs (see docs/plans/adr-0241-0242-session-completion-
summary-2026-07-15.md #38 postmortem: run #167 failed at ~22m39s, but local
verify_lane_shas.py showed 9/9 pins matching — the failure was never
conclusively attributed to a timeout vs an incomplete log, because the
script buffers all per-lane output until every lane finishes).

Two changes, no lane logic touched:

1. Stream progress per lane (`-> lane_id ... done (Ns)`, flush=True) instead
   of buffering until the end, so a hang shows exactly which lane and for
   how long even if the job is killed before the final summary prints.
2. Make the per-lane subprocess timeout configurable via
   CORE_LANE_VERIFY_TIMEOUT_S (default unchanged at 900s — same knob shape
   as CORE_SHOWCASE_HARD_BUDGET), and report a TimeoutExpired distinctly
   from a genuine SHA mismatch so the remediation text stops suggesting
   `--update` (re-pinning) for what is actually runner contention.

[Verification]: tests/test_lane_sha_verifier.py 6 passed; ruff clean;
scripts/verify_lane_shas.py run locally with the new streaming output
confirmed correct per-lane timing (e.g. demo_composition 154s) end to end.

core-labs added 1 commit 2026-07-16 20:00:41 +00:00

Merge remote-tracking branch 'forgejo/main' into fix/ci-lane-shas-timeout-diagnostics

smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m2s

Details

lane-shas / verify pinned lane SHAs (pull_request) Successful in 10m22s

Details

0e9051b643