f93f756e44
Root cause of PR checks stuck "Waiting to run": full-pytest on main ran ~12k tests including the slow registry on a 1.5-CPU / 1.2 GiB job cgroup for hours, monopolizing the only runner. Workflows: - full-pytest.yml: fast lane only (`not quarantine and not slow`) - nightly-full-pytest.yml: full suite including slow (schedule + manual) - lane-shas.yml: PR path filters so pin verification is skipped when the change cannot affect lane report bytes Docs: docs/ci-optimization.md + testing-lanes CI mapping. Infra (VM, same change window): job cgroup 2 CPU / 1800 MiB; hung full-pytest task stopped so the queue can drain PR smoke/lane-shas.
63 lines
3 KiB
Markdown
63 lines
3 KiB
Markdown
# CI optimization (single self-hosted Act runner)
|
||
|
||
## Problem (measured 2026-07-15)
|
||
|
||
| Fact | Value |
|
||
|------|--------|
|
||
| Host | 2 vCPU, ~3.8 GiB RAM, shared Forgejo + Postgres + Traefik |
|
||
| Job cgroup (was) | `--cpus=1.5 --memory=1200m` |
|
||
| Concurrent capacity | 1 job |
|
||
| Suite size | ~12.7k tests (`not quarantine`) |
|
||
| Post-merge job | `full-pytest` with **slow included** + `-n 2` |
|
||
| Observed | One `full-pytest` ran **2+ hours** at ~25% progress; PR **smoke / lane-shas** stayed **Waiting to run** |
|
||
|
||
That is not “slow tests” alone — it is **queue starvation**: one oversized job owns the only runner.
|
||
|
||
## Order-of-magnitude strategy
|
||
|
||
### 1. Right-size the default CI lane (workflow) — largest lever
|
||
|
||
| Lane | When | Marker expression | Intent |
|
||
|------|------|-------------------|--------|
|
||
| **smoke** | every PR | fixed file list | minutes; merge gate |
|
||
| **lane-shas** | PR (path-filtered) + main | pin script | only when pin surface can change |
|
||
| **full-pytest (main)** | every push to main | `not quarantine and not slow` | fast post-merge ratchet |
|
||
| **nightly-full** | schedule + manual | `not quarantine` (includes slow) | soak / proof / eval-matrix |
|
||
|
||
Local doc already measured **~7.7×** from “full serial” → “fast + parallel” on a laptop (`docs/testing-lanes.md`). On a 2‑vCPU VM the absolute times are worse, but **excluding slow** removes the 16‑minute parallel floor and hundreds of soak tests — typically multi‑× wall clock and frees the runner for PR gates.
|
||
|
||
### 2. Path-filter expensive PR jobs
|
||
|
||
`lane-shas` is multi‑minute (demos + showcase). Skip on PR when the diff cannot change pin bytes (docs-only, pure tooling, etc.). Main still always verifies after merge.
|
||
|
||
### 3. Runner cgroup (infra)
|
||
|
||
`infra/runner/config.yaml` job options should match host reality:
|
||
|
||
- Prefer **2 CPUs** for the job when the host is 2‑vCPU (avoid 1.5 + xdist thrash).
|
||
- Memory **≤ available** after Forgejo (~800 MiB limit on host stack).
|
||
- Keep **capacity 1** until a second runner exists — concurrency >1 on 2 cores makes two jobs slower than one.
|
||
|
||
### 4. What will *not* give 100× on this box
|
||
|
||
- Throwing more xdist workers at a 2‑vCPU host.
|
||
- Running full+slow on every main push.
|
||
- Expecting MLX/Rust to speed pytest collection/runtime of the whole suite without suite redesign.
|
||
|
||
True second order of magnitude needs either **more hardware** (second runner / larger VM) or **further suite partitioning** (shard by package, change-based test selection).
|
||
|
||
## Local policy (solo branch work)
|
||
|
||
1. Targeted tests for files you touched.
|
||
2. Dual-run only if pin surface changed.
|
||
3. Full smoke before merge (or trust green smoke if unchanged smoke paths).
|
||
4. Never rely on remote `full-pytest` for PR feedback — it is post-merge / nightly.
|
||
|
||
## Related
|
||
|
||
- `docs/testing-lanes.md` — fast / slow / full classification
|
||
- `conftest.py` — `SLOW_FILES` / `SLOW_TESTS` / `QUARANTINE`
|
||
- `.github/workflows/full-pytest.yml`
|
||
- `.github/workflows/nightly-full-pytest.yml`
|
||
- `.github/workflows/smoke.yml`
|
||
- `.github/workflows/lane-shas.yml`
|