af4db1f79d
A handful of soak / bench / replay / proof / eval-matrix tests dominate the suite wall-clock (~50 tests ≈ the entire 73-min serial runtime; the other ~10k are near-instant). Classify them so developers can run a fast lane locally. Mechanism — central registry in conftest.py beside QUARANTINE (cost is empirical test-infra metadata, not test semantics, so it lives in one auditable place, not as decorators across ~24 files). Marker-only: it stamps `slow`, it NEVER skips, so `-m slow` SELECTS the slow tests. - SLOW_FILES (10): whole-file, where a module/session fixture carries the cost (marking one test would just shift the fixture cost to the next requester). - SLOW_TESTS (26 nodeids): mixed files, so the file's fast predicate/unit tests stay in the fast lane. - pytest_collection_modifyitems stamps `slow`; marker registered in pyproject. 912 of 10,596 tests classified (801 = test_cognition_eval_register_matrix eval-matrix; called out in docs — honest accounting). Lanes (Makefile + docs/testing-lanes.md): fast: pytest -m "not quarantine and not slow" slow: pytest -m "slow and not quarantine" full: pytest -m "not quarantine" (unchanged — what CI runs) CI is unaffected: smoke.yml and full-pytest.yml run `-m "not quarantine"`, which still includes the slow tests. No CI behavior change; no test logic touched (the change cannot alter pass/fail in the full lane — it only adds a marker). Timings (10-core, numpy): full 73min serial / 25min -n auto; fast ~26min serial / 9.5min -n auto (7.7x combined). The 975s test_inner_loop_phase2 outlier was probed (expected proof-scale work, not a bug; finding in docs). Deferred to follow-up PRs: -n auto by default (needs xdist hermeticity — fresh-env-dict subprocess + report.json/proposals writers race under parallel workers) and a warm-runtime fixture for the remaining 1-15s ChatRuntime tail.
87 lines
4.3 KiB
Markdown
87 lines
4.3 KiB
Markdown
# Testing lanes — fast / slow / full
|
||
|
||
The full pytest suite is ~10,600 tests and ~73 min serial. A small set of
|
||
heavyweight tests dominates that wall-clock, so we classify them and offer a
|
||
**fast lane** for local development. Classification is empirical
|
||
test-infrastructure metadata, so it lives in one auditable place
|
||
(`conftest.py`), beside the `QUARANTINE` registry — not as `@pytest.mark.slow`
|
||
decorators spread across ~24 files.
|
||
|
||
## Lanes
|
||
|
||
| Lane | Command | What it runs |
|
||
|---|---|---|
|
||
| **fast** | `make test-fast` → `pytest -m "not quarantine and not slow"` | everything except the slow registry |
|
||
| **slow** | `make test-slow` → `pytest -m "slow and not quarantine"` | only the heavyweight registry |
|
||
| **full** | `make test-full` → `pytest -m "not quarantine"` | everything (what CI runs) |
|
||
|
||
The marker is **classification only** — it never skips. `-m slow` *selects* the
|
||
slow tests; you choose a lane with an explicit marker expression. Plain
|
||
`pytest` (no `-m`) still runs the full suite.
|
||
|
||
CI is unchanged: `.github/workflows/smoke.yml` and `full-pytest.yml` both run
|
||
`-m "not quarantine"`, which includes the slow tests — so the split costs no CI
|
||
coverage.
|
||
|
||
## Measured timings (10-core macOS, `CORE_BACKEND=numpy`)
|
||
|
||
| Lane | Serial | Parallel (`-n auto`) |
|
||
|---|---|---|
|
||
| full | 73 min | 25 min |
|
||
| fast | ~26 min | **9.5 min** (9,590 passed) |
|
||
|
||
Combined (split + parallel) = **73 → 9.5 min (7.7×)**. The parallel fast lane
|
||
scales ~5.7× because it excludes the 975s parallel-floor monster (see below);
|
||
the full suite only reaches 2.9× because that one test pins a worker for 16 min.
|
||
|
||
`-n auto` is **not** wired into the `make` targets yet — see *Follow-up: xdist*.
|
||
|
||
## The slow registry (`conftest.py`)
|
||
|
||
Two registries, by cost shape:
|
||
|
||
- **`SLOW_FILES`** — whole-file: the cost is carried by a module/session-scoped
|
||
fixture, so marking one test is insufficient (skipping it just shifts the
|
||
fixture cost to the next test that requests it). 10 files.
|
||
- **`SLOW_TESTS`** — exact nodeids: mixed files where only specific tests are
|
||
soak/bench scale; the file's fast predicate/unit tests stay in the fast lane.
|
||
26 tests across 14 files.
|
||
|
||
**Honest accounting** — the registry marks **912** of 10,596 tests slow. 801 of
|
||
those are `test_cognition_eval_register_matrix.py` (a per-register × invariant
|
||
eval matrix: many cheap parametrized assertions gated behind expensive
|
||
per-register module-fixture setups). It is classified whole-file because the
|
||
cost is in the module fixture, but be aware the fast lane therefore omits the
|
||
register-matrix coverage; CI's full lane still runs it.
|
||
|
||
## Finding: the 975s `test_inner_loop_phase2` outlier
|
||
|
||
`test_inner_loop_phase2.py::TestCausalAttribution::test_null_control_matches_boundary_only`
|
||
showed a **975s (16 min) setup** — the single largest test, and the parallel
|
||
floor for the whole suite.
|
||
|
||
Probed: it is **expected proof-scale work, not a bug or runaway**. The cost is a
|
||
module-scoped `phase2_report` fixture that runs the FSC corpus (9 cases: 1
|
||
`public/v1` + 8 `dev`) through `run_lane`, which executes **4 conditions + 4
|
||
determinism reruns = 8 full real-runtime pipeline turns per case**, plus a fresh
|
||
`ChatRuntime()` per case (~5s each). 9 × 8 heavy pipeline turns ≈ 975s. The
|
||
fixture is shared across the file's 5 tests, so the cost is paid once.
|
||
|
||
A possible optimization exists — share the primed runtime across the 4 conditions
|
||
instead of reconstructing — but it touches the runner's determinism contract, so
|
||
it is deferred, not done here.
|
||
|
||
## Follow-ups (separate PRs)
|
||
|
||
1. **xdist by default.** The fast/full lanes are *not* xdist-hermetic yet:
|
||
fresh-env-dict subprocess tests (`tests/formation/*`, `test_identity_packs`)
|
||
write to the repo `engine_state/` dir, and other tests write
|
||
`evals/.../report.json` and `teaching/proposals/` — these **race** under
|
||
parallel workers (e.g. `test_workbench_replay::test_replay_leaves_no_trace`
|
||
fails under `-n auto`, passes serially). Isolate those writers, then wire
|
||
`-n auto` into `make test-fast` / `test-full`. This is the same hermeticity
|
||
theme as `docs/issues/default-engine-state-test-hygiene.md`.
|
||
2. **Warm-runtime fixture.** The fast lane's remaining ~9.5 min (parallel) is a
|
||
long tail of 1–15s `ChatRuntime` constructions, not outliers. A
|
||
shared/session-scoped warm-runtime fixture for read-only tests would cut this
|
||
further.
|