af4db1f79d
A handful of soak / bench / replay / proof / eval-matrix tests dominate the suite wall-clock (~50 tests ≈ the entire 73-min serial runtime; the other ~10k are near-instant). Classify them so developers can run a fast lane locally. Mechanism — central registry in conftest.py beside QUARANTINE (cost is empirical test-infra metadata, not test semantics, so it lives in one auditable place, not as decorators across ~24 files). Marker-only: it stamps `slow`, it NEVER skips, so `-m slow` SELECTS the slow tests. - SLOW_FILES (10): whole-file, where a module/session fixture carries the cost (marking one test would just shift the fixture cost to the next requester). - SLOW_TESTS (26 nodeids): mixed files, so the file's fast predicate/unit tests stay in the fast lane. - pytest_collection_modifyitems stamps `slow`; marker registered in pyproject. 912 of 10,596 tests classified (801 = test_cognition_eval_register_matrix eval-matrix; called out in docs — honest accounting). Lanes (Makefile + docs/testing-lanes.md): fast: pytest -m "not quarantine and not slow" slow: pytest -m "slow and not quarantine" full: pytest -m "not quarantine" (unchanged — what CI runs) CI is unaffected: smoke.yml and full-pytest.yml run `-m "not quarantine"`, which still includes the slow tests. No CI behavior change; no test logic touched (the change cannot alter pass/fail in the full lane — it only adds a marker). Timings (10-core, numpy): full 73min serial / 25min -n auto; fast ~26min serial / 9.5min -n auto (7.7x combined). The 975s test_inner_loop_phase2 outlier was probed (expected proof-scale work, not a bug; finding in docs). Deferred to follow-up PRs: -n auto by default (needs xdist hermeticity — fresh-env-dict subprocess + report.json/proposals writers race under parallel workers) and a warm-runtime fixture for the remaining 1-15s ChatRuntime tail.
24 lines
956 B
Makefile
24 lines
956 B
Makefile
# Test lanes — see docs/testing-lanes.md.
|
|
#
|
|
# Slow tests are classified in conftest.py (SLOW_FILES / SLOW_TESTS), not via
|
|
# in-file decorators. Classification adds the `slow` marker only; it never
|
|
# skips, so the lanes below are selected explicitly by marker expression.
|
|
#
|
|
# These run serially. Parallel (`-n auto`) is intentionally NOT the default
|
|
# until the suite is xdist-hermetic (shared repo engine_state/ + report.json +
|
|
# teaching/proposals writers race under parallel workers). See
|
|
# docs/testing-lanes.md "Follow-up: xdist".
|
|
|
|
.PHONY: test-fast test-slow test-full
|
|
|
|
# Fast dev lane — excludes the slow soak/bench/proof/eval-matrix registry.
|
|
test-fast:
|
|
uv run pytest -m "not quarantine and not slow" -q
|
|
|
|
# Slow lane — only the heavyweight registry tests.
|
|
test-slow:
|
|
uv run pytest -m "slow and not quarantine" -q
|
|
|
|
# Full lane — everything except known-failing quarantine (what CI runs).
|
|
test-full:
|
|
uv run pytest -m "not quarantine" -q
|