fix(ci): stream lane-shas progress + distinguish timeout from content drift #50

Merged
core-labs merged 2 commits from fix/ci-lane-shas-timeout-diagnostics into main 2026-07-16 20:17:04 +00:00

2 commits

Author SHA1 Message Date
Shay
0e9051b643 Merge remote-tracking branch 'forgejo/main' into fix/ci-lane-shas-timeout-diagnostics
All checks were successful
smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m2s
lane-shas / verify pinned lane SHAs (pull_request) Successful in 10m22s
2026-07-16 12:58:03 -07:00
Shay
05e9d6d28e fix(ci): stream lane-shas progress + distinguish timeout from content drift
Some checks failed
smoke / smoke (-m "not quarantine") (pull_request) Failing after 53s
lane-shas / verify pinned lane SHAs (pull_request) Failing after 14m15s
lane-shas.yml has intermittently failed under Act-runner contention with
truncated/ambiguous logs (see docs/plans/adr-0241-0242-session-completion-
summary-2026-07-15.md #38 postmortem: run #167 failed at ~22m39s, but local
verify_lane_shas.py showed 9/9 pins matching — the failure was never
conclusively attributed to a timeout vs an incomplete log, because the
script buffers all per-lane output until every lane finishes).

Two changes, no lane logic touched:

1. Stream progress per lane (`-> lane_id ... done (Ns)`, flush=True) instead
   of buffering until the end, so a hang shows exactly which lane and for
   how long even if the job is killed before the final summary prints.
2. Make the per-lane subprocess timeout configurable via
   CORE_LANE_VERIFY_TIMEOUT_S (default unchanged at 900s — same knob shape
   as CORE_SHOWCASE_HARD_BUDGET), and report a TimeoutExpired distinctly
   from a genuine SHA mismatch so the remediation text stops suggesting
   `--update` (re-pinning) for what is actually runner contention.

[Verification]: tests/test_lane_sha_verifier.py 6 passed; ruff clean;
scripts/verify_lane_shas.py run locally with the new streaming output
confirmed correct per-lane timing (e.g. demo_composition 154s) end to end.
2026-07-16 06:33:25 -07:00