f93f756e44
Root cause of PR checks stuck "Waiting to run": full-pytest on main ran ~12k tests including the slow registry on a 1.5-CPU / 1.2 GiB job cgroup for hours, monopolizing the only runner. Workflows: - full-pytest.yml: fast lane only (`not quarantine and not slow`) - nightly-full-pytest.yml: full suite including slow (schedule + manual) - lane-shas.yml: PR path filters so pin verification is skipped when the change cannot affect lane report bytes Docs: docs/ci-optimization.md + testing-lanes CI mapping. Infra (VM, same change window): job cgroup 2 CPU / 1800 MiB; hung full-pytest task stopped so the queue can drain PR smoke/lane-shas.
3 KiB
CI optimization (single self-hosted Act runner)
Problem (measured 2026-07-15)
| Fact | Value |
|---|---|
| Host | 2 vCPU, ~3.8 GiB RAM, shared Forgejo + Postgres + Traefik |
| Job cgroup (was) | --cpus=1.5 --memory=1200m |
| Concurrent capacity | 1 job |
| Suite size | ~12.7k tests (not quarantine) |
| Post-merge job | full-pytest with slow included + -n 2 |
| Observed | One full-pytest ran 2+ hours at ~25% progress; PR smoke / lane-shas stayed Waiting to run |
That is not “slow tests” alone — it is queue starvation: one oversized job owns the only runner.
Order-of-magnitude strategy
1. Right-size the default CI lane (workflow) — largest lever
| Lane | When | Marker expression | Intent |
|---|---|---|---|
| smoke | every PR | fixed file list | minutes; merge gate |
| lane-shas | PR (path-filtered) + main | pin script | only when pin surface can change |
| full-pytest (main) | every push to main | not quarantine and not slow |
fast post-merge ratchet |
| nightly-full | schedule + manual | not quarantine (includes slow) |
soak / proof / eval-matrix |
Local doc already measured ~7.7× from “full serial” → “fast + parallel” on a laptop (docs/testing-lanes.md). On a 2‑vCPU VM the absolute times are worse, but excluding slow removes the 16‑minute parallel floor and hundreds of soak tests — typically multi‑× wall clock and frees the runner for PR gates.
2. Path-filter expensive PR jobs
lane-shas is multi‑minute (demos + showcase). Skip on PR when the diff cannot change pin bytes (docs-only, pure tooling, etc.). Main still always verifies after merge.
3. Runner cgroup (infra)
infra/runner/config.yaml job options should match host reality:
- Prefer 2 CPUs for the job when the host is 2‑vCPU (avoid 1.5 + xdist thrash).
- Memory ≤ available after Forgejo (~800 MiB limit on host stack).
- Keep capacity 1 until a second runner exists — concurrency >1 on 2 cores makes two jobs slower than one.
4. What will not give 100× on this box
- Throwing more xdist workers at a 2‑vCPU host.
- Running full+slow on every main push.
- Expecting MLX/Rust to speed pytest collection/runtime of the whole suite without suite redesign.
True second order of magnitude needs either more hardware (second runner / larger VM) or further suite partitioning (shard by package, change-based test selection).
Local policy (solo branch work)
- Targeted tests for files you touched.
- Dual-run only if pin surface changed.
- Full smoke before merge (or trust green smoke if unchanged smoke paths).
- Never rely on remote
full-pytestfor PR feedback — it is post-merge / nightly.
Related
docs/testing-lanes.md— fast / slow / full classificationconftest.py—SLOW_FILES/SLOW_TESTS/QUARANTINE.github/workflows/full-pytest.yml.github/workflows/nightly-full-pytest.yml.github/workflows/smoke.yml.github/workflows/lane-shas.yml