test: fast/slow/full lanes via central slow registry in conftest (#786 )

A handful of soak / bench / replay / proof / eval-matrix tests dominate the
suite wall-clock (~50 tests ≈ the entire 73-min serial runtime; the other ~10k
are near-instant). Classify them so developers can run a fast lane locally.

Mechanism — central registry in conftest.py beside QUARANTINE (cost is empirical
test-infra metadata, not test semantics, so it lives in one auditable place, not
as decorators across ~24 files). Marker-only: it stamps `slow`, it NEVER skips,
so `-m slow` SELECTS the slow tests.

- SLOW_FILES (10): whole-file, where a module/session fixture carries the cost
  (marking one test would just shift the fixture cost to the next requester).
- SLOW_TESTS (26 nodeids): mixed files, so the file's fast predicate/unit tests
  stay in the fast lane.
- pytest_collection_modifyitems stamps `slow`; marker registered in pyproject.
  912 of 10,596 tests classified (801 = test_cognition_eval_register_matrix
  eval-matrix; called out in docs — honest accounting).

Lanes (Makefile + docs/testing-lanes.md):
  fast: pytest -m "not quarantine and not slow"
  slow: pytest -m "slow and not quarantine"
  full: pytest -m "not quarantine"        (unchanged — what CI runs)

CI is unaffected: smoke.yml and full-pytest.yml run `-m "not quarantine"`, which
still includes the slow tests. No CI behavior change; no test logic touched (the
change cannot alter pass/fail in the full lane — it only adds a marker).

Timings (10-core, numpy): full 73min serial / 25min -n auto; fast ~26min serial
/ 9.5min -n auto (7.7x combined). The 975s test_inner_loop_phase2 outlier was
probed (expected proof-scale work, not a bug; finding in docs). Deferred to
follow-up PRs: -n auto by default (needs xdist hermeticity — fresh-env-dict
subprocess + report.json/proposals writers race under parallel workers) and a
warm-runtime fixture for the remaining 1-15s ChatRuntime tail.

2026-06-15 17:30:50 -07:00

4.3 KiB

Raw Blame History

Testing lanes — fast / slow / full

The full pytest suite is ~10,600 tests and ~73 min serial. A small set of heavyweight tests dominates that wall-clock, so we classify them and offer a fast lane for local development. Classification is empirical test-infrastructure metadata, so it lives in one auditable place (conftest.py), beside the QUARANTINE registry — not as @pytest.mark.slow decorators spread across ~24 files.

Lanes

Lane	Command	What it runs
fast	`make test-fast` → `pytest -m "not quarantine and not slow"`	everything except the slow registry
slow	`make test-slow` → `pytest -m "slow and not quarantine"`	only the heavyweight registry
full	`make test-full` → `pytest -m "not quarantine"`	everything (what CI runs)

The marker is classification only — it never skips. -m slow selects the slow tests; you choose a lane with an explicit marker expression. Plain pytest (no -m) still runs the full suite.

CI is unchanged: .github/workflows/smoke.yml and full-pytest.yml both run -m "not quarantine", which includes the slow tests — so the split costs no CI coverage.

Measured timings (10-core macOS, `CORE_BACKEND=numpy`)

Lane	Serial	Parallel (`-n auto`)
full	73 min	25 min
fast	~26 min	9.5 min (9,590 passed)

Combined (split + parallel) = 73 → 9.5 min (7.7×). The parallel fast lane scales ~5.7× because it excludes the 975s parallel-floor monster (see below); the full suite only reaches 2.9× because that one test pins a worker for 16 min.

-n auto is not wired into the make targets yet — see Follow-up: xdist.

The slow registry (`conftest.py`)

Two registries, by cost shape:

SLOW_FILES — whole-file: the cost is carried by a module/session-scoped fixture, so marking one test is insufficient (skipping it just shifts the fixture cost to the next test that requests it). 10 files.
SLOW_TESTS — exact nodeids: mixed files where only specific tests are soak/bench scale; the file's fast predicate/unit tests stay in the fast lane. 26 tests across 14 files.

Honest accounting — the registry marks 912 of 10,596 tests slow. 801 of those are test_cognition_eval_register_matrix.py (a per-register × invariant eval matrix: many cheap parametrized assertions gated behind expensive per-register module-fixture setups). It is classified whole-file because the cost is in the module fixture, but be aware the fast lane therefore omits the register-matrix coverage; CI's full lane still runs it.

Finding: the 975s `test_inner_loop_phase2` outlier

test_inner_loop_phase2.py::TestCausalAttribution::test_null_control_matches_boundary_only showed a 975s (16 min) setup — the single largest test, and the parallel floor for the whole suite.

Probed: it is expected proof-scale work, not a bug or runaway. The cost is a module-scoped phase2_report fixture that runs the FSC corpus (9 cases: 1 public/v1 + 8 dev) through run_lane, which executes 4 conditions + 4 determinism reruns = 8 full real-runtime pipeline turns per case, plus a fresh ChatRuntime() per case (~5s each). 9 × 8 heavy pipeline turns ≈ 975s. The fixture is shared across the file's 5 tests, so the cost is paid once.

A possible optimization exists — share the primed runtime across the 4 conditions instead of reconstructing — but it touches the runner's determinism contract, so it is deferred, not done here.

Follow-ups (separate PRs)

xdist by default. The fast/full lanes are not xdist-hermetic yet: fresh-env-dict subprocess tests (tests/formation/*, test_identity_packs) write to the repo engine_state/ dir, and other tests write evals/.../report.json and teaching/proposals/ — these race under parallel workers (e.g. test_workbench_replay::test_replay_leaves_no_trace fails under -n auto, passes serially). Isolate those writers, then wire -n auto into make test-fast / test-full. This is the same hermeticity theme as docs/issues/default-engine-state-test-hygiene.md.
Warm-runtime fixture. The fast lane's remaining ~9.5 min (parallel) is a long tail of 1–15s ChatRuntime constructions, not outliers. A shared/session-scoped warm-runtime fixture for read-only tests would cut this further.

4.3 KiB Raw Blame History Unescape Escape