Compare commits
7 commits
44f7258b16
...
f93f756e44
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f93f756e44 | ||
|
|
2b3de74e80 | ||
|
|
6f5b2f0294 | ||
|
|
37fee86d7e | ||
|
|
e987b4dd34 | ||
|
|
2d8f0e910d | ||
|
|
340f5dd3a5 |
7 changed files with 130 additions and 209 deletions
45
.github/workflows/full-pytest.yml
vendored
45
.github/workflows/full-pytest.yml
vendored
|
|
@ -1,25 +1,20 @@
|
|||
name: full-pytest
|
||||
|
||||
# Post-merge FAST lane — runs on every push to main.
|
||||
# Post-merge gate — runs the **fast** lane on every push to main.
|
||||
#
|
||||
# Marker: -m "not quarantine and not slow"
|
||||
# ~9.5k unit/integration tests; excludes the slow registry in conftest.py
|
||||
# (soak / bench / proof / register-matrix; ~912 tests including the 16 min
|
||||
# phase2 fixture floor).
|
||||
# Why not full+slow here:
|
||||
# Host is a 2‑vCPU / ~4 GiB VM shared with Forgejo + Postgres + Traefik.
|
||||
# Job containers were capped at 1.5 CPU / 1.2 GiB. Running the entire
|
||||
# ~12k-test suite including the slow registry (one 16 min fixture floor)
|
||||
# monopolized the single Act runner for hours and starved PR smoke /
|
||||
# lane-shas ("Waiting to run").
|
||||
#
|
||||
# Why "full-pytest" still names this file:
|
||||
# Keep the workflow id stable for Forgejo required-check / history matching.
|
||||
# The job display name and this comment state the true contract: FAST on main.
|
||||
# The complete non-quarantine suite runs in nightly-full-pytest.yml.
|
||||
# Coverage contract:
|
||||
# - This workflow: ``not quarantine and not slow`` (make test-fast).
|
||||
# - Slow/soak/proof matrix: ``nightly-full-pytest.yml`` (schedule + manual).
|
||||
# - PR gate remains smoke.yml (minutes) + path-filtered lane-shas.yml.
|
||||
#
|
||||
# PR gate: smoke.yml (small critical subset).
|
||||
# Lane pins: lane-shas.yml.
|
||||
# Full soak: nightly-full-pytest.yml (schedule + workflow_dispatch).
|
||||
#
|
||||
# See:
|
||||
# conftest.py — QUARANTINE + SLOW_FILES / SLOW_TESTS registries
|
||||
# docs/testing-lanes.md — lane commands and CI policy (SSoT)
|
||||
# docs/ci-optimization.md — runner bottleneck + capacity notes
|
||||
# See docs/testing-lanes.md and docs/ci-optimization.md.
|
||||
|
||||
on:
|
||||
push:
|
||||
|
|
@ -34,9 +29,9 @@ concurrency:
|
|||
|
||||
jobs:
|
||||
pytest:
|
||||
name: fast pytest (-m "not quarantine and not slow" -n 2)
|
||||
name: full pytest (fast lane, not quarantine and not slow)
|
||||
runs-on: ubuntu-latest
|
||||
# Headroom over ~9.5 min parallel on a 10-core host; Act 2-vCPU is slower.
|
||||
# Fast lane should finish well under this on a healthy runner.
|
||||
timeout-minutes: 30
|
||||
|
||||
steps:
|
||||
|
|
@ -55,14 +50,17 @@ jobs:
|
|||
run: |
|
||||
uv pip install -e ".[dev]" pyyaml
|
||||
|
||||
- name: pytest (parallel, quarantine and slow excluded)
|
||||
- name: pytest fast lane (parallel)
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
CORE_SHOWCASE_SKIP_BUDGET: "1"
|
||||
# Avoid uv hardlink warning noise across filesystems on Act.
|
||||
UV_LINK_MODE: copy
|
||||
run: |
|
||||
# -n 2 matches prior CI; host is 2 vCPU — do not raise without more cores.
|
||||
uv run pytest -m "not quarantine and not slow" -n 2 --tb=short -q --maxfail=10
|
||||
|
||||
- name: report quarantine size (informational)
|
||||
- name: report quarantine + slow registry size (informational)
|
||||
if: always()
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
|
|
@ -70,6 +68,7 @@ jobs:
|
|||
uv run python -c "
|
||||
import sys
|
||||
sys.path.insert(0, '.')
|
||||
from conftest import QUARANTINE
|
||||
print(f'::notice title=Quarantine size::{len(QUARANTINE)} tests currently quarantined. Goal: shrink this number.')
|
||||
from conftest import QUARANTINE, SLOW_FILES, SLOW_TESTS
|
||||
print(f'::notice title=Quarantine size::{len(QUARANTINE)} tests quarantined.')
|
||||
print(f'::notice title=Slow registry::{len(SLOW_FILES)} files + {len(SLOW_TESTS)} exact tests classified slow (nightly only).')
|
||||
"
|
||||
|
|
|
|||
71
.github/workflows/lane-shas.yml
vendored
71
.github/workflows/lane-shas.yml
vendored
|
|
@ -1,6 +1,6 @@
|
|||
name: lane-shas
|
||||
|
||||
# Verify that every ADR-0092..0104 lane produces its pinned SHA-256
|
||||
# Verify that every ADR-0092..0099 lane produces its pinned SHA-256
|
||||
# report. A failing job means a lane's deterministic output changed
|
||||
# without an explicit ADR-tracked pin update via:
|
||||
#
|
||||
|
|
@ -8,19 +8,33 @@ name: lane-shas
|
|||
#
|
||||
# Single source of truth for the pinned values is scripts/verify_lane_shas.py.
|
||||
#
|
||||
# PR path policy (job-level, not workflow-level):
|
||||
# Pin bytes can move from Python, packs, eval fixtures/corpora, teaching
|
||||
# corpora, dependency pins, CLAIMS.md, or this workflow. When none of those
|
||||
# paths change on a PR, we skip the multi-minute runners and exit success so
|
||||
# a required check never sits "Waiting" forever (workflow-level `paths:`
|
||||
# would omit the job entirely and hang required status checks).
|
||||
# Main pushes always verify.
|
||||
# PR path filter: skip the multi-lane wall-clock cost when the change cannot
|
||||
# affect pin bytes (e.g. docs-only, pure physics off the demo path). Main
|
||||
# still always verifies after merge.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "scripts/verify_lane_shas.py"
|
||||
- "scripts/generate_claims.py"
|
||||
- "CLAIMS.md"
|
||||
- "evals/**"
|
||||
- "demos/**"
|
||||
- "core/demos/**"
|
||||
- "chat/**"
|
||||
- "generate/**"
|
||||
- "packs/**"
|
||||
- "teaching/**"
|
||||
- "vault/**"
|
||||
- "algebra/**"
|
||||
- "core/physics/**"
|
||||
- "core/cognition/**"
|
||||
- "core/capability/**"
|
||||
- "pyproject.toml"
|
||||
- ".github/workflows/lane-shas.yml"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
|
@ -39,54 +53,25 @@ jobs:
|
|||
- name: checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# Need base..head for PR path detection.
|
||||
fetch-depth: 0
|
||||
|
||||
- name: detect pin-relevant paths
|
||||
id: paths
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Always verify on main pushes.
|
||||
if [ "${{ github.event_name }}" = "push" ]; then
|
||||
echo "run=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Pin-relevant path check: always run on push"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
BASE="${{ github.event.pull_request.base.sha }}"
|
||||
HEAD="${{ github.event.pull_request.head.sha }}"
|
||||
# Paths that can change pin bytes or CLAIMS generation inputs.
|
||||
# Keep in sync with docs/testing-lanes.md § CI policy.
|
||||
PATTERN='(\.py$|^packs/|^evals/|^teaching/|^CLAIMS\.md$|^pyproject\.toml$|^uv\.lock$|^\.github/workflows/lane-shas\.yml$)'
|
||||
if git diff --name-only "$BASE" "$HEAD" | grep -E "$PATTERN" >/dev/null; then
|
||||
echo "run=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Pin-relevant paths changed; running lane SHA verification"
|
||||
git diff --name-only "$BASE" "$HEAD" | grep -E "$PATTERN" || true
|
||||
else
|
||||
echo "run=false" >> "$GITHUB_OUTPUT"
|
||||
echo "No pin-relevant paths changed; skipping lane SHA verification (job still green)"
|
||||
fi
|
||||
fetch-depth: 1
|
||||
|
||||
# setup-uv (not actions/setup-python) provisions Python on the aarch64
|
||||
# self-hosted runner; actions/setup-python has no arm64 build for the
|
||||
# pinned 3.12.13. Matches smoke.yml / full-pytest.yml.
|
||||
- name: set up uv
|
||||
if: steps.paths.outputs.run == 'true'
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
python-version: '3.12.13'
|
||||
enable-cache: true
|
||||
|
||||
- name: install dependencies
|
||||
if: steps.paths.outputs.run == 'true'
|
||||
run: |
|
||||
uv pip install -e . pyyaml pytest
|
||||
|
||||
- name: verify lane SHAs
|
||||
if: steps.paths.outputs.run == 'true'
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
UV_LINK_MODE: copy
|
||||
# public_demo wall-clock is soft by default (see evals/public_demo/runner.py).
|
||||
# Do not set CORE_SHOWCASE_HARD_BUDGET here — cold Act runners exceed 60s.
|
||||
# Content cases (claims, determinism, pure composition) remain hard gates.
|
||||
|
|
@ -94,20 +79,14 @@ jobs:
|
|||
uv run python scripts/verify_lane_shas.py
|
||||
|
||||
- name: verify CLAIMS.md is current
|
||||
if: steps.paths.outputs.run == 'true'
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
run: |
|
||||
uv run python scripts/generate_claims.py --check
|
||||
|
||||
- name: emit machine-readable report (on failure)
|
||||
if: failure() && steps.paths.outputs.run == 'true'
|
||||
if: failure()
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
run: |
|
||||
uv run python scripts/verify_lane_shas.py --json || true
|
||||
|
||||
- name: skip notice
|
||||
if: steps.paths.outputs.run != 'true'
|
||||
run: |
|
||||
echo "::notice title=lane-shas skipped::No pin-relevant paths in this PR; verification skipped (success)."
|
||||
|
|
|
|||
46
.github/workflows/nightly-full-pytest.yml
vendored
46
.github/workflows/nightly-full-pytest.yml
vendored
|
|
@ -1,22 +1,15 @@
|
|||
name: nightly-full-pytest
|
||||
|
||||
# Nightly FULL lane — complete non-quarantine suite including the slow registry.
|
||||
# Slow/soak lane — the heavyweight registry excluded from post-merge full-pytest.
|
||||
# Runs on a schedule so it cannot starve PR smoke / lane-shas on the single
|
||||
# self-hosted Act runner.
|
||||
#
|
||||
# Marker: -m "not quarantine"
|
||||
# Includes soak / bench / proof / register-matrix (SLOW_FILES + SLOW_TESTS).
|
||||
# Intentionally off the PR and post-merge critical path so a single 2-vCPU
|
||||
# Act runner is not held for 1–2h after every main push (see docs/ci-optimization.md).
|
||||
#
|
||||
# Risk owned here: a main merge can break slow tests until the next nightly
|
||||
# (or a manual workflow_dispatch). Treat red nightlies as release-blocking
|
||||
# debt; re-run via Actions → nightly-full-pytest → Run workflow after fixes.
|
||||
#
|
||||
# See docs/testing-lanes.md for the full CI policy.
|
||||
# Manual: Actions → nightly-full-pytest → Run workflow.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 02:00 UTC daily — off peak for human PR iteration.
|
||||
- cron: '0 2 * * *'
|
||||
# 06:00 UTC daily — adjust if runner is shared with other heavy jobs.
|
||||
- cron: "0 6 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
|
|
@ -24,15 +17,12 @@ permissions:
|
|||
|
||||
concurrency:
|
||||
group: nightly-full-pytest
|
||||
cancel-in-progress: false
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
pytest:
|
||||
name: full pytest (-m "not quarantine" -n 2)
|
||||
pytest-slow:
|
||||
name: nightly full (not quarantine, includes slow)
|
||||
runs-on: ubuntu-latest
|
||||
# Full suite parallel floor includes ~16 min phase2 fixture; thrashing on a
|
||||
# 2-vCPU Act host has been observed well past 60 min. Prefer a red timeout
|
||||
# only after a genuine hang, not under normal soak load.
|
||||
timeout-minutes: 120
|
||||
|
||||
steps:
|
||||
|
|
@ -52,21 +42,11 @@ jobs:
|
|||
run: |
|
||||
uv pip install -e ".[dev]" pyyaml
|
||||
|
||||
- name: pytest (parallel, full suite, quarantine excluded)
|
||||
- name: pytest full including slow registry
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
CORE_SHOWCASE_SKIP_BUDGET: "1"
|
||||
UV_LINK_MODE: copy
|
||||
run: |
|
||||
uv run pytest -m "not quarantine" -n 2 --tb=short -q --maxfail=10
|
||||
|
||||
- name: report quarantine size (informational)
|
||||
if: always()
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}
|
||||
run: |
|
||||
uv run python -c "
|
||||
import sys
|
||||
sys.path.insert(0, '.')
|
||||
from conftest import QUARANTINE
|
||||
print(f'::notice title=Quarantine size::{len(QUARANTINE)} tests currently quarantined. Goal: shrink this number.')
|
||||
"
|
||||
# Single worker on 2‑vCPU host avoids xdist thrash with 16 min fixtures.
|
||||
uv run pytest -m "not quarantine" -n 1 --tb=short -q --maxfail=20
|
||||
|
|
|
|||
7
.github/workflows/smoke.yml
vendored
7
.github/workflows/smoke.yml
vendored
|
|
@ -7,10 +7,9 @@ name: smoke
|
|||
# — ratified packs diverge directionally; pack-invariant refusal floor; no
|
||||
# fabrication). The falsifiability lane adds ~4 min but blocks-on-regression.
|
||||
#
|
||||
# Post-merge on main: full-pytest.yml runs the FAST lane
|
||||
# (-m "not quarantine and not slow"). Soak / proof / register-matrix coverage
|
||||
# is nightly-full-pytest.yml (not on the PR critical path).
|
||||
# See docs/testing-lanes.md.
|
||||
# Full pytest runs post-merge to main (see full-pytest.yml).
|
||||
# Regressions caught here block the PR; anything outside the smoke
|
||||
# suite is caught on main within minutes of merge.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
|
|
|
|||
22
conftest.py
22
conftest.py
|
|
@ -1,18 +1,19 @@
|
|||
"""Project-root conftest — test classification registries.
|
||||
|
||||
The QUARANTINE set is the only allowed registry for known-failing tests.
|
||||
It is currently empty. If it ever contains nodeids, CI excludes them via
|
||||
``-m "not quarantine"`` (smoke, full-pytest fast lane, nightly full).
|
||||
The suite is a ratchet: a quarantined test removed from this set must pass
|
||||
It is currently empty. If it ever contains nodeids, the CI gate at
|
||||
.github/workflows/full-pytest.yml runs ``pytest -m "not quarantine"``
|
||||
so those explicitly tracked failures do not block unrelated PRs. The
|
||||
suite is a ratchet: a quarantined test removed from this set must pass
|
||||
on its own merits.
|
||||
|
||||
See docs/test-debt-quarantine.md for current policy and historical cluster
|
||||
diagnoses. See docs/testing-lanes.md for CI lane policy (PR / main / nightly).
|
||||
diagnoses.
|
||||
|
||||
To remove a test from quarantine:
|
||||
1. Land a PR that makes the test pass.
|
||||
2. Delete its entry from QUARANTINE in the same PR.
|
||||
3. The main fast gate and nightly full gate will both require it to pass.
|
||||
3. The full-pytest CI gate will now require it to keep passing.
|
||||
|
||||
Adding a test to QUARANTINE is strongly discouraged. If a new
|
||||
failure surfaces, the right default is to fix it in the PR that
|
||||
|
|
@ -79,14 +80,13 @@ QUARANTINE: frozenset[str] = frozenset()
|
|||
# a developer run a fast lane locally. Classification adds the ``slow`` marker
|
||||
# ONLY — it never skips — so ``-m slow`` SELECTS these tests. Choose a lane:
|
||||
#
|
||||
# fast lane: pytest -m "not quarantine and not slow" (make test-fast;
|
||||
# also full-pytest.yml on main)
|
||||
# fast lane: pytest -m "not quarantine and not slow" (make test-fast)
|
||||
# slow lane: pytest -m "slow and not quarantine" (make test-slow)
|
||||
# full lane: pytest -m "not quarantine" (make test-full;
|
||||
# also nightly-full-pytest.yml)
|
||||
# full lane: pytest -m "not quarantine" (make test-full; CI)
|
||||
#
|
||||
# CI policy: PR = smoke subset; main = fast lane; nightly = full including
|
||||
# slow. See docs/testing-lanes.md.
|
||||
# CI: full-pytest.yml runs the fast lane (``not quarantine and not slow``).
|
||||
# Nightly workflow runs full including slow. See docs/testing-lanes.md and
|
||||
# docs/ci-optimization.md.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Whole-file: the cost is carried by a module/session-scoped fixture, so marking
|
||||
|
|
|
|||
|
|
@ -1,77 +1,63 @@
|
|||
# CI optimization — runner bottleneck and lane policy
|
||||
# CI optimization (single self-hosted Act runner)
|
||||
|
||||
Companion to [testing-lanes.md](./testing-lanes.md) (SSoT for markers and
|
||||
workflow → lane mapping). This note records **why** the multi-lane CI split
|
||||
exists and what capacity still limits PR queue time.
|
||||
## Problem (measured 2026-07-15)
|
||||
|
||||
## The bottleneck
|
||||
| Fact | Value |
|
||||
|------|--------|
|
||||
| Host | 2 vCPU, ~3.8 GiB RAM, shared Forgejo + Postgres + Traefik |
|
||||
| Job cgroup (was) | `--cpus=1.5 --memory=1200m` |
|
||||
| Concurrent capacity | 1 job |
|
||||
| Suite size | ~12.7k tests (`not quarantine`) |
|
||||
| Post-merge job | `full-pytest` with **slow included** + `-n 2` |
|
||||
| Observed | One `full-pytest` ran **2+ hours** at ~25% progress; PR **smoke / lane-shas** stayed **Waiting to run** |
|
||||
|
||||
A single self-hosted Act runner (≈2 vCPU / limited RAM) serializes all workflows
|
||||
that target `ubuntu-latest` for this repo. When post-merge CI ran the **full**
|
||||
non-quarantine suite (`-m "not quarantine"`) with `-n 2`, the slow registry
|
||||
dominated wall-clock:
|
||||
That is not “slow tests” alone — it is **queue starvation**: one oversized job owns the only runner.
|
||||
|
||||
- Module fixture floor: `test_inner_loop_phase2` ≈16 min setup alone
|
||||
- ~912 tests classified `slow` in `conftest.py` (including the register matrix)
|
||||
- Thrashing under oversubscription can stretch the job toward **1–2 hours**
|
||||
## Order-of-magnitude strategy
|
||||
|
||||
While that job holds the runner, PR gates (`smoke`, `lane-shas`) sit in
|
||||
**Waiting**. That is a queue problem first, a test-count problem second.
|
||||
### 1. Right-size the default CI lane (workflow) — largest lever
|
||||
|
||||
## What we changed (in-repo)
|
||||
| Lane | When | Marker expression | Intent |
|
||||
|------|------|-------------------|--------|
|
||||
| **smoke** | every PR | fixed file list | minutes; merge gate |
|
||||
| **lane-shas** | PR (path-filtered) + main | pin script | only when pin surface can change |
|
||||
| **full-pytest (main)** | every push to main | `not quarantine and not slow` | fast post-merge ratchet |
|
||||
| **nightly-full** | schedule + manual | `not quarantine` (includes slow) | soak / proof / eval-matrix |
|
||||
|
||||
| Change | Effect |
|
||||
|---|---|
|
||||
| `full-pytest.yml` → fast marker | Main push runs `not quarantine and not slow`; frees the runner much sooner after merge |
|
||||
| `nightly-full-pytest.yml` | Full suite (`not quarantine`) daily at 02:00 UTC + `workflow_dispatch`; timeout 120 min |
|
||||
| `lane-shas.yml` job-level path skip | PR pin verify runs only when pin-relevant paths change; job still green when skipped (no required-check hang) |
|
||||
| Contract comments | `smoke.yml`, `conftest.py`, `testing-lanes.md` match the real PR / main / nightly split |
|
||||
Local doc already measured **~7.7×** from “full serial” → “fast + parallel” on a laptop (`docs/testing-lanes.md`). On a 2‑vCPU VM the absolute times are worse, but **excluding slow** removes the 16‑minute parallel floor and hundreds of soak tests — typically multi‑× wall clock and frees the runner for PR gates.
|
||||
|
||||
Workflow **id** `full-pytest` is kept on purpose so Forgejo history / required-check
|
||||
names do not break; the job display name is `fast pytest (...)`.
|
||||
### 2. Path-filter expensive PR jobs
|
||||
|
||||
## What is out of band (ops, not git)
|
||||
`lane-shas` is multi‑minute (demos + showcase). Skip on PR when the diff cannot change pin bytes (docs-only, pure tooling, etc.). Main still always verifies after merge.
|
||||
|
||||
Host cgroup limits, hung-container cleanup, and adding a second runner are
|
||||
**VM/ops** actions. They are not encoded in this repository. Document them in
|
||||
ops runbooks when you change the host; do not treat this file as a substitute.
|
||||
### 3. Runner cgroup (infra)
|
||||
|
||||
## Expected feedback loop after this change
|
||||
`infra/runner/config.yaml` job options should match host reality:
|
||||
|
||||
| Stage | Typical hold |
|
||||
|---|---|
|
||||
| Local | Targeted tests; `make test-fast` before push |
|
||||
| PR | `smoke` (+ `lane-shas` when pin-relevant paths change) |
|
||||
| Main | Fast `full-pytest` + always-on `lane-shas` |
|
||||
| Nightly | Full soak including slow registry |
|
||||
- Prefer **2 CPUs** for the job when the host is 2‑vCPU (avoid 1.5 + xdist thrash).
|
||||
- Memory **≤ available** after Forgejo (~800 MiB limit on host stack).
|
||||
- Keep **capacity 1** until a second runner exists — concurrency >1 on 2 cores makes two jobs slower than one.
|
||||
|
||||
This is roughly **hours of runner hold after merge → tens of minutes** on the
|
||||
fast lane for the same host class — not infinite parallel capacity. Multiple
|
||||
open PRs still queue on one runner.
|
||||
### 4. What will *not* give 100× on this box
|
||||
|
||||
## Capacity still required
|
||||
- Throwing more xdist workers at a 2‑vCPU host.
|
||||
- Running full+slow on every main push.
|
||||
- Expecting MLX/Rust to speed pytest collection/runtime of the whole suite without suite redesign.
|
||||
|
||||
1. **Second Act runner or larger VM** — only real fix for concurrent PR + main
|
||||
execution.
|
||||
2. **xdist hermeticity** — see follow-ups in `testing-lanes.md` before raising
|
||||
`-n` further.
|
||||
3. **Warm-runtime fixture** — long tail of `ChatRuntime` construction in the
|
||||
fast lane.
|
||||
4. **Nightly failure signal** — wire Forgejo notification / issue on red
|
||||
`nightly-full-pytest` so ≤24h slow-break debt is not silent.
|
||||
5. **Optional path-triggered slow job** — if main must catch soak breaks same-day
|
||||
without nightly wait, add a non-blocking or path-filtered slow workflow later.
|
||||
True second order of magnitude needs either **more hardware** (second runner / larger VM) or **further suite partitioning** (shard by package, change-based test selection).
|
||||
|
||||
## Validation
|
||||
## Local policy (solo branch work)
|
||||
|
||||
```bash
|
||||
# Local parity with main CI
|
||||
make test-fast
|
||||
1. Targeted tests for files you touched.
|
||||
2. Dual-run only if pin surface changed.
|
||||
3. Full smoke before merge (or trust green smoke if unchanged smoke paths).
|
||||
4. Never rely on remote `full-pytest` for PR feedback — it is post-merge / nightly.
|
||||
|
||||
# Local parity with nightly
|
||||
make test-full
|
||||
## Related
|
||||
|
||||
# Lane pins (same as lane-shas job body)
|
||||
uv run python scripts/verify_lane_shas.py
|
||||
uv run python scripts/generate_claims.py --check
|
||||
```
|
||||
- `docs/testing-lanes.md` — fast / slow / full classification
|
||||
- `conftest.py` — `SLOW_FILES` / `SLOW_TESTS` / `QUARANTINE`
|
||||
- `.github/workflows/full-pytest.yml`
|
||||
- `.github/workflows/nightly-full-pytest.yml`
|
||||
- `.github/workflows/smoke.yml`
|
||||
- `.github/workflows/lane-shas.yml`
|
||||
|
|
|
|||
|
|
@ -2,56 +2,34 @@
|
|||
|
||||
The full pytest suite is ~10,600 tests and ~73 min serial. A small set of
|
||||
heavyweight tests dominates that wall-clock, so we classify them and offer a
|
||||
**fast lane** for local development and post-merge CI. Classification is
|
||||
empirical test-infrastructure metadata, so it lives in one auditable place
|
||||
**fast lane** for local development. Classification is empirical
|
||||
test-infrastructure metadata, so it lives in one auditable place
|
||||
(`conftest.py`), beside the `QUARANTINE` registry — not as `@pytest.mark.slow`
|
||||
decorators spread across ~24 files.
|
||||
|
||||
Runner capacity notes and the single-Act-runner queue story live in
|
||||
[ci-optimization.md](./ci-optimization.md). **This file is the SSoT for lane
|
||||
commands and which workflow runs which marker.**
|
||||
|
||||
## Lanes
|
||||
|
||||
| Lane | Command | What it runs |
|
||||
|---|---|---|
|
||||
| **fast** | `make test-fast` → `pytest -m "not quarantine and not slow"` | everything except the slow registry |
|
||||
| **slow** | `make test-slow` → `pytest -m "slow and not quarantine"` | only the heavyweight registry |
|
||||
| **full** | `make test-full` → `pytest -m "not quarantine"` | everything non-quarantine (local + nightly CI) |
|
||||
| **full** | `make test-full` → `pytest -m "not quarantine"` | everything (local + **nightly** CI) |
|
||||
|
||||
The marker is **classification only** — it never skips. `-m slow` *selects* the
|
||||
slow tests; you choose a lane with an explicit marker expression. Plain
|
||||
`pytest` (no `-m`) still runs the full suite.
|
||||
|
||||
## CI policy
|
||||
**CI mapping (2026-07-15 optimization):**
|
||||
|
||||
| Surface | Workflow | Marker / scope |
|
||||
|---|---|---|
|
||||
| **PR** | `smoke.yml` | Fixed critical subset (`not quarantine` within those files) |
|
||||
| **PR** | `lane-shas.yml` | Pinned ADR lane SHAs + `CLAIMS.md` check; **skipped green** when the PR does not touch pin-relevant paths |
|
||||
| **main push** | `full-pytest.yml` | **Fast lane** (`not quarantine and not slow`); workflow *id* kept as `full-pytest` for check stability |
|
||||
| **main push** | `lane-shas.yml` | Always runs (no path skip on push) |
|
||||
| **nightly / manual** | `nightly-full-pytest.yml` | **Full lane** (`not quarantine`), includes slow registry |
|
||||
| Workflow | Expression | When |
|
||||
|----------|------------|------|
|
||||
| `smoke.yml` | fixed smoke file list | every PR |
|
||||
| `full-pytest.yml` | `not quarantine and not slow` | every push to **main** |
|
||||
| `nightly-full-pytest.yml` | `not quarantine` (includes slow) | daily schedule + manual |
|
||||
| `lane-shas.yml` | pin script | main always; PR only if pin-affecting paths change |
|
||||
|
||||
### Pin-relevant paths (`lane-shas` PR skip)
|
||||
|
||||
On `pull_request`, verification runs only when the base…head diff touches any of:
|
||||
|
||||
- `*.py`
|
||||
- `packs/**`, `evals/**`, `teaching/**`
|
||||
- `CLAIMS.md`, `pyproject.toml`, `uv.lock`
|
||||
- `.github/workflows/lane-shas.yml`
|
||||
|
||||
Otherwise the job still starts and exits **success** (skip notice) so a required
|
||||
status check never sits on “Waiting” forever. Do **not** reintroduce workflow-level
|
||||
`on.pull_request.paths` for this gate without a always-green companion job.
|
||||
|
||||
### Owned risk
|
||||
|
||||
Slow/soak/proof tests (including the ~16 min phase2 fixture and the register
|
||||
matrix) are **not** on the PR or post-merge critical path. A merge can break them
|
||||
until the next nightly (02:00 UTC) or a manual `workflow_dispatch`. Treat a red
|
||||
nightly as release-blocking debt.
|
||||
See `docs/ci-optimization.md` for why the single Act runner cannot run full+slow
|
||||
on every merge without starving PR gates.
|
||||
|
||||
## Measured timings (10-core macOS, `CORE_BACKEND=numpy`)
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue