From af4db1f79d4c3788e6aa8f7b9f34c49ab64fccd2 Mon Sep 17 00:00:00 2001 From: Shay Date: Mon, 15 Jun 2026 17:30:50 -0700 Subject: [PATCH] test: fast/slow/full lanes via central slow registry in conftest (#786) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A handful of soak / bench / replay / proof / eval-matrix tests dominate the suite wall-clock (~50 tests ≈ the entire 73-min serial runtime; the other ~10k are near-instant). Classify them so developers can run a fast lane locally. Mechanism — central registry in conftest.py beside QUARANTINE (cost is empirical test-infra metadata, not test semantics, so it lives in one auditable place, not as decorators across ~24 files). Marker-only: it stamps `slow`, it NEVER skips, so `-m slow` SELECTS the slow tests. - SLOW_FILES (10): whole-file, where a module/session fixture carries the cost (marking one test would just shift the fixture cost to the next requester). - SLOW_TESTS (26 nodeids): mixed files, so the file's fast predicate/unit tests stay in the fast lane. - pytest_collection_modifyitems stamps `slow`; marker registered in pyproject. 912 of 10,596 tests classified (801 = test_cognition_eval_register_matrix eval-matrix; called out in docs — honest accounting). Lanes (Makefile + docs/testing-lanes.md): fast: pytest -m "not quarantine and not slow" slow: pytest -m "slow and not quarantine" full: pytest -m "not quarantine" (unchanged — what CI runs) CI is unaffected: smoke.yml and full-pytest.yml run `-m "not quarantine"`, which still includes the slow tests. No CI behavior change; no test logic touched (the change cannot alter pass/fail in the full lane — it only adds a marker). Timings (10-core, numpy): full 73min serial / 25min -n auto; fast ~26min serial / 9.5min -n auto (7.7x combined). The 975s test_inner_loop_phase2 outlier was probed (expected proof-scale work, not a bug; finding in docs). Deferred to follow-up PRs: -n auto by default (needs xdist hermeticity — fresh-env-dict subprocess + report.json/proposals writers race under parallel workers) and a warm-runtime fixture for the remaining 1-15s ChatRuntime tail. --- Makefile | 24 +++++++++ conftest.py | 121 +++++++++++++++++++++++++++++++++++++++--- docs/testing-lanes.md | 87 ++++++++++++++++++++++++++++++ pyproject.toml | 1 + 4 files changed, 226 insertions(+), 7 deletions(-) create mode 100644 Makefile create mode 100644 docs/testing-lanes.md diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..56e52036 --- /dev/null +++ b/Makefile @@ -0,0 +1,24 @@ +# Test lanes — see docs/testing-lanes.md. +# +# Slow tests are classified in conftest.py (SLOW_FILES / SLOW_TESTS), not via +# in-file decorators. Classification adds the `slow` marker only; it never +# skips, so the lanes below are selected explicitly by marker expression. +# +# These run serially. Parallel (`-n auto`) is intentionally NOT the default +# until the suite is xdist-hermetic (shared repo engine_state/ + report.json + +# teaching/proposals writers race under parallel workers). See +# docs/testing-lanes.md "Follow-up: xdist". + +.PHONY: test-fast test-slow test-full + +# Fast dev lane — excludes the slow soak/bench/proof/eval-matrix registry. +test-fast: + uv run pytest -m "not quarantine and not slow" -q + +# Slow lane — only the heavyweight registry tests. +test-slow: + uv run pytest -m "slow and not quarantine" -q + +# Full lane — everything except known-failing quarantine (what CI runs). +test-full: + uv run pytest -m "not quarantine" -q diff --git a/conftest.py b/conftest.py index 2e04c864..8f3b7428 100644 --- a/conftest.py +++ b/conftest.py @@ -70,16 +70,123 @@ def _isolate_engine_state_default(request, tmp_path_factory, monkeypatch): QUARANTINE: frozenset[str] = frozenset() -def pytest_collection_modifyitems(config, items): - """Stamp the quarantine marker on any test whose nodeid is listed - in QUARANTINE. Tests not in the set are unaffected. +# --------------------------------------------------------------------------- +# Slow-test registry — empirical test-infrastructure metadata, NOT test +# semantics, so it lives here beside QUARANTINE in one auditable place rather +# than as @pytest.mark.slow decorators spread across ~24 files. +# +# A handful of soak / bench / replay / proof / eval-matrix tests dominate the +# suite wall-clock: ~50 tests account for essentially the entire serial runtime +# (~73 min), while the other ~10k tests are near-instant. Classifying them lets +# a developer run a fast lane locally. Classification adds the ``slow`` marker +# ONLY — it never skips — so ``-m slow`` SELECTS these tests. Choose a lane: +# +# fast lane: pytest -m "not quarantine and not slow" (make test-fast) +# slow lane: pytest -m "slow and not quarantine" (make test-slow) +# full lane: pytest -m "not quarantine" (make test-full; CI) +# +# CI is unchanged: smoke.yml and full-pytest.yml run ``-m "not quarantine"``, +# which still includes slow tests. See docs/testing-lanes.md. +# --------------------------------------------------------------------------- - The CI gate runs ``pytest -m "not quarantine"`` so quarantined - tests are silently skipped in CI. Local ``pytest`` runs include - them by default (still useful for debugging individual fixes). +# Whole-file: the cost is carried by a module/session-scoped fixture, so marking +# individual tests is insufficient — skipping one just shifts the fixture cost to +# the next test that requests it. The whole file is classified slow. +SLOW_FILES: frozenset[str] = frozenset({ + "tests/test_inner_loop_phase2.py", # module fixture run_lane(): 9 cases x 4 conditions x 5-rerun determinism (~975s) + "tests/test_cognition_eval_register_matrix.py", # module fixtures: full register x invariant eval matrix + "tests/test_articulation_bench.py", # module fixture: articulation bench corpus + "tests/test_register_invariant_grounding.py", # module fixtures: per-register runtime grounding + "tests/test_public_showcase.py", # module fixture: full showcase execution + "tests/test_conversation_demo.py", # module fixture: multi-turn demo run + "tests/test_edge_budget_gate.py", # module fixture: per-turn checkpoint cost soak + "tests/test_phase5_corpus.py", # module fixtures: replay-determinism corpus + "tests/test_realizer_guard_holdout.py", # module fixture: holdout-cluster run + "tests/test_teaching_loop_bench.py", # module fixture: teaching-loop determinism bench +}) + +# Exact nodeids: mixed files where only specific tests are soak/bench scale. +# Listed individually so the file's fast (predicate / unit) coverage stays in the +# fast lane. No scoped fixture carries the cost in these files, so per-test +# classification does not shift cost to a sibling. +SLOW_TESTS: frozenset[str] = frozenset({ + # test_l10_continuity.py — real-soak predicates among 28 mostly-fast tests + "tests/test_l10_continuity.py::test_p1_closure_holds_on_real_soak", + "tests/test_l10_continuity.py::test_p2a_determinism_holds_across_independent_runtimes", + "tests/test_l10_continuity.py::test_p2b_pre_reboot_invariant_holds_on_real_soak", + "tests/test_l10_continuity.py::test_p2b_reboot_is_transparent", + "tests/test_l10_continuity.py::test_p4_recovery_is_deterministic_across_orphan_crash", + "tests/test_l10_continuity.py::test_p5c_coherence_holds_over_multiple_corpus_cycles", + "tests/test_l10_continuity.py::test_report_panel_passes_and_records_not_covered", + # test_l10_arbitrary_interruption.py — partial/full generation interruption soak + "tests/test_l10_arbitrary_interruption.py::test_p4_arbitrary_interruption_full_gen_before_swap_holds", + "tests/test_l10_arbitrary_interruption.py::test_p4_arbitrary_interruption_partial_gen_holds", + # test_register_tour_demo.py — per-register tour runs (5 fast structural tests remain) + "tests/test_register_tour_demo.py::test_tour_grounding_sources_identical_across_registers", + "tests/test_register_tour_demo.py::test_tour_returns_structured_report", + "tests/test_register_tour_demo.py::test_tour_terse_substantively_differs_from_neutral", + "tests/test_register_tour_demo.py::test_tour_trace_hashes_identical_across_registers", + # test_register_firing_diagnostic.py — multi-register marker-engagement report + "tests/test_register_firing_diagnostic.py::test_build_report_records_marker_engagement_for_register_subset", + # test_operator_calibration_replay.py — replay/calibration proof tests + "tests/test_operator_calibration_replay.py::TestCalibrationRejectsInvariantRegression::test_versor_closure_must_not_regress", + "tests/test_operator_calibration_replay.py::TestCalibrationReportHasBeforeAfterMetrics::test_calibrate_returns_result", + # test_pack_measurements_phase2.py — falsifiability lane (ALSO run by CI smoke via + # -m "not quarantine", which does not exclude slow; classifying it is safe) + "tests/test_pack_measurements_phase2.py::TestRefusalCalibrationPackRunner::test_grounding_gate_is_pack_invariant", + "tests/test_pack_measurements_phase2.py::TestRefusalCalibrationPackRunner::test_no_fabrication_under_any_pack", + "tests/test_pack_measurements_phase2.py::TestRefusalCalibrationPackRunner::test_report_schema_is_stable", + # test_thread_context.py — eviction-over-capacity soak (19 fast tests remain) + "tests/test_thread_context.py::test_runtime_default_capacity_evicts_old_turns", + # test_correction_telemetry.py — cross-run determinism (6 fast tests remain) + "tests/test_correction_telemetry.py::test_correction_event_is_deterministic_across_runs", + # test_cold_start_grounding_lane.py — distribution lane (15 fast tests remain) + "tests/test_cold_start_grounding_lane.py::TestPassThresholds::test_distributions_match_expected", + # test_engine_state_session_persistence.py — reboot/restore soak (4 fast tests remain) + "tests/test_engine_state_session_persistence.py::test_chat_runtime_restores_lived_state_across_reboot", + "tests/test_engine_state_session_persistence.py::test_no_load_state_runtime_starts_fresh", + # test_cli_demo.py — combined-run demo subprocess (16 lighter demo tests remain) + "tests/test_cli_demo.py::TestDemoPreambles::test_all_preamble_explains_combined_run", + # test_identity_continuity_proof.py — byte-identical resumed-life proof + "tests/test_identity_continuity_proof.py::test_resumed_life_is_byte_identical_and_same_identity", + # test_warmed_session_lane.py — pipeline-override gate invariant + "tests/test_warmed_session_lane.py::TestPipelineOverrideGateInvariants::test_no_placeholder_rate_is_one", +}) + + +def _is_slow(nodeid: str) -> bool: + """True if ``nodeid`` is classified slow by the SLOW_FILES / SLOW_TESTS registry. + + ``nodeid`` is expected pre-normalized to forward slashes (the caller does this + once). Splitting on ``"::"`` yields exactly the file path, so the whole-file + check is an O(1) set lookup with no substring-prefix edge cases (e.g. + ``..._phase2.py`` can never match ``..._phase2_extra.py``). Exact-nodeid + match handles mixed files. + """ + if nodeid in SLOW_TESTS: + return True + file_path = nodeid.split("::", 1)[0] + return file_path in SLOW_FILES + + +def pytest_collection_modifyitems(config, items): + """Stamp the ``quarantine`` and ``slow`` markers from the conftest registries. + + - QUARANTINE nodeids get ``quarantine`` (CI runs ``-m "not quarantine"``). + - SLOW_FILES / SLOW_TESTS get ``slow`` — classification ONLY, never skipped + here, so ``-m slow`` SELECTS them. Lanes are chosen explicitly: + fast: pytest -m "not quarantine and not slow" + slow: pytest -m "slow and not quarantine" + full: pytest -m "not quarantine" """ _ = config # pluggy hook signature requires the name `config`; not used here quarantine_marker = pytest.mark.quarantine + slow_marker = pytest.mark.slow for item in items: - if item.nodeid in QUARANTINE: + # Normalize once: pytest nodeids are forward-slash, but normalize + # defensively so both registries match on Windows backslash nodeids too. + nodeid = item.nodeid.replace("\\", "/") + if nodeid in QUARANTINE: item.add_marker(quarantine_marker) + if _is_slow(nodeid): + item.add_marker(slow_marker) diff --git a/docs/testing-lanes.md b/docs/testing-lanes.md new file mode 100644 index 00000000..22e17fe1 --- /dev/null +++ b/docs/testing-lanes.md @@ -0,0 +1,87 @@ +# Testing lanes — fast / slow / full + +The full pytest suite is ~10,600 tests and ~73 min serial. A small set of +heavyweight tests dominates that wall-clock, so we classify them and offer a +**fast lane** for local development. Classification is empirical +test-infrastructure metadata, so it lives in one auditable place +(`conftest.py`), beside the `QUARANTINE` registry — not as `@pytest.mark.slow` +decorators spread across ~24 files. + +## Lanes + +| Lane | Command | What it runs | +|---|---|---| +| **fast** | `make test-fast` → `pytest -m "not quarantine and not slow"` | everything except the slow registry | +| **slow** | `make test-slow` → `pytest -m "slow and not quarantine"` | only the heavyweight registry | +| **full** | `make test-full` → `pytest -m "not quarantine"` | everything (what CI runs) | + +The marker is **classification only** — it never skips. `-m slow` *selects* the +slow tests; you choose a lane with an explicit marker expression. Plain +`pytest` (no `-m`) still runs the full suite. + +CI is unchanged: `.github/workflows/smoke.yml` and `full-pytest.yml` both run +`-m "not quarantine"`, which includes the slow tests — so the split costs no CI +coverage. + +## Measured timings (10-core macOS, `CORE_BACKEND=numpy`) + +| Lane | Serial | Parallel (`-n auto`) | +|---|---|---| +| full | 73 min | 25 min | +| fast | ~26 min | **9.5 min** (9,590 passed) | + +Combined (split + parallel) = **73 → 9.5 min (7.7×)**. The parallel fast lane +scales ~5.7× because it excludes the 975s parallel-floor monster (see below); +the full suite only reaches 2.9× because that one test pins a worker for 16 min. + +`-n auto` is **not** wired into the `make` targets yet — see *Follow-up: xdist*. + +## The slow registry (`conftest.py`) + +Two registries, by cost shape: + +- **`SLOW_FILES`** — whole-file: the cost is carried by a module/session-scoped + fixture, so marking one test is insufficient (skipping it just shifts the + fixture cost to the next test that requests it). 10 files. +- **`SLOW_TESTS`** — exact nodeids: mixed files where only specific tests are + soak/bench scale; the file's fast predicate/unit tests stay in the fast lane. + 26 tests across 14 files. + +**Honest accounting** — the registry marks **912** of 10,596 tests slow. 801 of +those are `test_cognition_eval_register_matrix.py` (a per-register × invariant +eval matrix: many cheap parametrized assertions gated behind expensive +per-register module-fixture setups). It is classified whole-file because the +cost is in the module fixture, but be aware the fast lane therefore omits the +register-matrix coverage; CI's full lane still runs it. + +## Finding: the 975s `test_inner_loop_phase2` outlier + +`test_inner_loop_phase2.py::TestCausalAttribution::test_null_control_matches_boundary_only` +showed a **975s (16 min) setup** — the single largest test, and the parallel +floor for the whole suite. + +Probed: it is **expected proof-scale work, not a bug or runaway**. The cost is a +module-scoped `phase2_report` fixture that runs the FSC corpus (9 cases: 1 +`public/v1` + 8 `dev`) through `run_lane`, which executes **4 conditions + 4 +determinism reruns = 8 full real-runtime pipeline turns per case**, plus a fresh +`ChatRuntime()` per case (~5s each). 9 × 8 heavy pipeline turns ≈ 975s. The +fixture is shared across the file's 5 tests, so the cost is paid once. + +A possible optimization exists — share the primed runtime across the 4 conditions +instead of reconstructing — but it touches the runner's determinism contract, so +it is deferred, not done here. + +## Follow-ups (separate PRs) + +1. **xdist by default.** The fast/full lanes are *not* xdist-hermetic yet: + fresh-env-dict subprocess tests (`tests/formation/*`, `test_identity_packs`) + write to the repo `engine_state/` dir, and other tests write + `evals/.../report.json` and `teaching/proposals/` — these **race** under + parallel workers (e.g. `test_workbench_replay::test_replay_leaves_no_trace` + fails under `-n auto`, passes serially). Isolate those writers, then wire + `-n auto` into `make test-fast` / `test-full`. This is the same hermeticity + theme as `docs/issues/default-engine-state-test-hygiene.md`. +2. **Warm-runtime fixture.** The fast lane's remaining ~9.5 min (parallel) is a + long tail of 1–15s `ChatRuntime` constructions, not outliers. A + shared/session-scoped warm-runtime fixture for read-only tests would cut this + further. diff --git a/pyproject.toml b/pyproject.toml index c3907707..dbb7f0d6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -62,4 +62,5 @@ testpaths = ["tests"] markers = [ "quarantine: pre-existing failure tracked in conftest.py QUARANTINE registry; excluded from full-pytest CI gate. See docs/test-debt-quarantine.md.", "uses_default_engine_state: test intentionally exercises the process default engine-state directory; opt out of per-test default isolation. See docs/issues/default-engine-state-test-hygiene.md.", + "slow: soak / bench / replay / proof / eval-matrix test that dominates suite wall-clock; classified in conftest.py SLOW_FILES/SLOW_TESTS (not in-file decorators). Fast lane: -m 'not quarantine and not slow'. See docs/testing-lanes.md.", ]