test: fast/slow/full lanes via central slow registry in conftest (#786)
A handful of soak / bench / replay / proof / eval-matrix tests dominate the suite wall-clock (~50 tests ≈ the entire 73-min serial runtime; the other ~10k are near-instant). Classify them so developers can run a fast lane locally. Mechanism — central registry in conftest.py beside QUARANTINE (cost is empirical test-infra metadata, not test semantics, so it lives in one auditable place, not as decorators across ~24 files). Marker-only: it stamps `slow`, it NEVER skips, so `-m slow` SELECTS the slow tests. - SLOW_FILES (10): whole-file, where a module/session fixture carries the cost (marking one test would just shift the fixture cost to the next requester). - SLOW_TESTS (26 nodeids): mixed files, so the file's fast predicate/unit tests stay in the fast lane. - pytest_collection_modifyitems stamps `slow`; marker registered in pyproject. 912 of 10,596 tests classified (801 = test_cognition_eval_register_matrix eval-matrix; called out in docs — honest accounting). Lanes (Makefile + docs/testing-lanes.md): fast: pytest -m "not quarantine and not slow" slow: pytest -m "slow and not quarantine" full: pytest -m "not quarantine" (unchanged — what CI runs) CI is unaffected: smoke.yml and full-pytest.yml run `-m "not quarantine"`, which still includes the slow tests. No CI behavior change; no test logic touched (the change cannot alter pass/fail in the full lane — it only adds a marker). Timings (10-core, numpy): full 73min serial / 25min -n auto; fast ~26min serial / 9.5min -n auto (7.7x combined). The 975s test_inner_loop_phase2 outlier was probed (expected proof-scale work, not a bug; finding in docs). Deferred to follow-up PRs: -n auto by default (needs xdist hermeticity — fresh-env-dict subprocess + report.json/proposals writers race under parallel workers) and a warm-runtime fixture for the remaining 1-15s ChatRuntime tail.
This commit is contained in:
Shay
GitHub
parent
90ed891ae0
commit
af4db1f79d
4 changed files with 226 additions and 7 deletions
24
Makefile
Normal file
24
Makefile
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Test lanes — see docs/testing-lanes.md.
|
||||
#
|
||||
# Slow tests are classified in conftest.py (SLOW_FILES / SLOW_TESTS), not via
|
||||
# in-file decorators. Classification adds the `slow` marker only; it never
|
||||
# skips, so the lanes below are selected explicitly by marker expression.
|
||||
#
|
||||
# These run serially. Parallel (`-n auto`) is intentionally NOT the default
|
||||
# until the suite is xdist-hermetic (shared repo engine_state/ + report.json +
|
||||
# teaching/proposals writers race under parallel workers). See
|
||||
# docs/testing-lanes.md "Follow-up: xdist".
|
||||
|
||||
.PHONY: test-fast test-slow test-full
|
||||
|
||||
# Fast dev lane — excludes the slow soak/bench/proof/eval-matrix registry.
|
||||
test-fast:
|
||||
uv run pytest -m "not quarantine and not slow" -q
|
||||
|
||||
# Slow lane — only the heavyweight registry tests.
|
||||
test-slow:
|
||||
uv run pytest -m "slow and not quarantine" -q
|
||||
|
||||
# Full lane — everything except known-failing quarantine (what CI runs).
|
||||
test-full:
|
||||
uv run pytest -m "not quarantine" -q
|
||||
121
conftest.py
121
conftest.py
|
|
@ -70,16 +70,123 @@ def _isolate_engine_state_default(request, tmp_path_factory, monkeypatch):
|
|||
QUARANTINE: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config, items):
|
||||
"""Stamp the quarantine marker on any test whose nodeid is listed
|
||||
in QUARANTINE. Tests not in the set are unaffected.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Slow-test registry — empirical test-infrastructure metadata, NOT test
|
||||
# semantics, so it lives here beside QUARANTINE in one auditable place rather
|
||||
# than as @pytest.mark.slow decorators spread across ~24 files.
|
||||
#
|
||||
# A handful of soak / bench / replay / proof / eval-matrix tests dominate the
|
||||
# suite wall-clock: ~50 tests account for essentially the entire serial runtime
|
||||
# (~73 min), while the other ~10k tests are near-instant. Classifying them lets
|
||||
# a developer run a fast lane locally. Classification adds the ``slow`` marker
|
||||
# ONLY — it never skips — so ``-m slow`` SELECTS these tests. Choose a lane:
|
||||
#
|
||||
# fast lane: pytest -m "not quarantine and not slow" (make test-fast)
|
||||
# slow lane: pytest -m "slow and not quarantine" (make test-slow)
|
||||
# full lane: pytest -m "not quarantine" (make test-full; CI)
|
||||
#
|
||||
# CI is unchanged: smoke.yml and full-pytest.yml run ``-m "not quarantine"``,
|
||||
# which still includes slow tests. See docs/testing-lanes.md.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
The CI gate runs ``pytest -m "not quarantine"`` so quarantined
|
||||
tests are silently skipped in CI. Local ``pytest`` runs include
|
||||
them by default (still useful for debugging individual fixes).
|
||||
# Whole-file: the cost is carried by a module/session-scoped fixture, so marking
|
||||
# individual tests is insufficient — skipping one just shifts the fixture cost to
|
||||
# the next test that requests it. The whole file is classified slow.
|
||||
SLOW_FILES: frozenset[str] = frozenset({
|
||||
"tests/test_inner_loop_phase2.py", # module fixture run_lane(): 9 cases x 4 conditions x 5-rerun determinism (~975s)
|
||||
"tests/test_cognition_eval_register_matrix.py", # module fixtures: full register x invariant eval matrix
|
||||
"tests/test_articulation_bench.py", # module fixture: articulation bench corpus
|
||||
"tests/test_register_invariant_grounding.py", # module fixtures: per-register runtime grounding
|
||||
"tests/test_public_showcase.py", # module fixture: full showcase execution
|
||||
"tests/test_conversation_demo.py", # module fixture: multi-turn demo run
|
||||
"tests/test_edge_budget_gate.py", # module fixture: per-turn checkpoint cost soak
|
||||
"tests/test_phase5_corpus.py", # module fixtures: replay-determinism corpus
|
||||
"tests/test_realizer_guard_holdout.py", # module fixture: holdout-cluster run
|
||||
"tests/test_teaching_loop_bench.py", # module fixture: teaching-loop determinism bench
|
||||
})
|
||||
|
||||
# Exact nodeids: mixed files where only specific tests are soak/bench scale.
|
||||
# Listed individually so the file's fast (predicate / unit) coverage stays in the
|
||||
# fast lane. No scoped fixture carries the cost in these files, so per-test
|
||||
# classification does not shift cost to a sibling.
|
||||
SLOW_TESTS: frozenset[str] = frozenset({
|
||||
# test_l10_continuity.py — real-soak predicates among 28 mostly-fast tests
|
||||
"tests/test_l10_continuity.py::test_p1_closure_holds_on_real_soak",
|
||||
"tests/test_l10_continuity.py::test_p2a_determinism_holds_across_independent_runtimes",
|
||||
"tests/test_l10_continuity.py::test_p2b_pre_reboot_invariant_holds_on_real_soak",
|
||||
"tests/test_l10_continuity.py::test_p2b_reboot_is_transparent",
|
||||
"tests/test_l10_continuity.py::test_p4_recovery_is_deterministic_across_orphan_crash",
|
||||
"tests/test_l10_continuity.py::test_p5c_coherence_holds_over_multiple_corpus_cycles",
|
||||
"tests/test_l10_continuity.py::test_report_panel_passes_and_records_not_covered",
|
||||
# test_l10_arbitrary_interruption.py — partial/full generation interruption soak
|
||||
"tests/test_l10_arbitrary_interruption.py::test_p4_arbitrary_interruption_full_gen_before_swap_holds",
|
||||
"tests/test_l10_arbitrary_interruption.py::test_p4_arbitrary_interruption_partial_gen_holds",
|
||||
# test_register_tour_demo.py — per-register tour runs (5 fast structural tests remain)
|
||||
"tests/test_register_tour_demo.py::test_tour_grounding_sources_identical_across_registers",
|
||||
"tests/test_register_tour_demo.py::test_tour_returns_structured_report",
|
||||
"tests/test_register_tour_demo.py::test_tour_terse_substantively_differs_from_neutral",
|
||||
"tests/test_register_tour_demo.py::test_tour_trace_hashes_identical_across_registers",
|
||||
# test_register_firing_diagnostic.py — multi-register marker-engagement report
|
||||
"tests/test_register_firing_diagnostic.py::test_build_report_records_marker_engagement_for_register_subset",
|
||||
# test_operator_calibration_replay.py — replay/calibration proof tests
|
||||
"tests/test_operator_calibration_replay.py::TestCalibrationRejectsInvariantRegression::test_versor_closure_must_not_regress",
|
||||
"tests/test_operator_calibration_replay.py::TestCalibrationReportHasBeforeAfterMetrics::test_calibrate_returns_result",
|
||||
# test_pack_measurements_phase2.py — falsifiability lane (ALSO run by CI smoke via
|
||||
# -m "not quarantine", which does not exclude slow; classifying it is safe)
|
||||
"tests/test_pack_measurements_phase2.py::TestRefusalCalibrationPackRunner::test_grounding_gate_is_pack_invariant",
|
||||
"tests/test_pack_measurements_phase2.py::TestRefusalCalibrationPackRunner::test_no_fabrication_under_any_pack",
|
||||
"tests/test_pack_measurements_phase2.py::TestRefusalCalibrationPackRunner::test_report_schema_is_stable",
|
||||
# test_thread_context.py — eviction-over-capacity soak (19 fast tests remain)
|
||||
"tests/test_thread_context.py::test_runtime_default_capacity_evicts_old_turns",
|
||||
# test_correction_telemetry.py — cross-run determinism (6 fast tests remain)
|
||||
"tests/test_correction_telemetry.py::test_correction_event_is_deterministic_across_runs",
|
||||
# test_cold_start_grounding_lane.py — distribution lane (15 fast tests remain)
|
||||
"tests/test_cold_start_grounding_lane.py::TestPassThresholds::test_distributions_match_expected",
|
||||
# test_engine_state_session_persistence.py — reboot/restore soak (4 fast tests remain)
|
||||
"tests/test_engine_state_session_persistence.py::test_chat_runtime_restores_lived_state_across_reboot",
|
||||
"tests/test_engine_state_session_persistence.py::test_no_load_state_runtime_starts_fresh",
|
||||
# test_cli_demo.py — combined-run demo subprocess (16 lighter demo tests remain)
|
||||
"tests/test_cli_demo.py::TestDemoPreambles::test_all_preamble_explains_combined_run",
|
||||
# test_identity_continuity_proof.py — byte-identical resumed-life proof
|
||||
"tests/test_identity_continuity_proof.py::test_resumed_life_is_byte_identical_and_same_identity",
|
||||
# test_warmed_session_lane.py — pipeline-override gate invariant
|
||||
"tests/test_warmed_session_lane.py::TestPipelineOverrideGateInvariants::test_no_placeholder_rate_is_one",
|
||||
})
|
||||
|
||||
|
||||
def _is_slow(nodeid: str) -> bool:
|
||||
"""True if ``nodeid`` is classified slow by the SLOW_FILES / SLOW_TESTS registry.
|
||||
|
||||
``nodeid`` is expected pre-normalized to forward slashes (the caller does this
|
||||
once). Splitting on ``"::"`` yields exactly the file path, so the whole-file
|
||||
check is an O(1) set lookup with no substring-prefix edge cases (e.g.
|
||||
``..._phase2.py`` can never match ``..._phase2_extra.py``). Exact-nodeid
|
||||
match handles mixed files.
|
||||
"""
|
||||
if nodeid in SLOW_TESTS:
|
||||
return True
|
||||
file_path = nodeid.split("::", 1)[0]
|
||||
return file_path in SLOW_FILES
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config, items):
|
||||
"""Stamp the ``quarantine`` and ``slow`` markers from the conftest registries.
|
||||
|
||||
- QUARANTINE nodeids get ``quarantine`` (CI runs ``-m "not quarantine"``).
|
||||
- SLOW_FILES / SLOW_TESTS get ``slow`` — classification ONLY, never skipped
|
||||
here, so ``-m slow`` SELECTS them. Lanes are chosen explicitly:
|
||||
fast: pytest -m "not quarantine and not slow"
|
||||
slow: pytest -m "slow and not quarantine"
|
||||
full: pytest -m "not quarantine"
|
||||
"""
|
||||
_ = config # pluggy hook signature requires the name `config`; not used here
|
||||
quarantine_marker = pytest.mark.quarantine
|
||||
slow_marker = pytest.mark.slow
|
||||
for item in items:
|
||||
if item.nodeid in QUARANTINE:
|
||||
# Normalize once: pytest nodeids are forward-slash, but normalize
|
||||
# defensively so both registries match on Windows backslash nodeids too.
|
||||
nodeid = item.nodeid.replace("\\", "/")
|
||||
if nodeid in QUARANTINE:
|
||||
item.add_marker(quarantine_marker)
|
||||
if _is_slow(nodeid):
|
||||
item.add_marker(slow_marker)
|
||||
|
|
|
|||
87
docs/testing-lanes.md
Normal file
87
docs/testing-lanes.md
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
# Testing lanes — fast / slow / full
|
||||
|
||||
The full pytest suite is ~10,600 tests and ~73 min serial. A small set of
|
||||
heavyweight tests dominates that wall-clock, so we classify them and offer a
|
||||
**fast lane** for local development. Classification is empirical
|
||||
test-infrastructure metadata, so it lives in one auditable place
|
||||
(`conftest.py`), beside the `QUARANTINE` registry — not as `@pytest.mark.slow`
|
||||
decorators spread across ~24 files.
|
||||
|
||||
## Lanes
|
||||
|
||||
| Lane | Command | What it runs |
|
||||
|---|---|---|
|
||||
| **fast** | `make test-fast` → `pytest -m "not quarantine and not slow"` | everything except the slow registry |
|
||||
| **slow** | `make test-slow` → `pytest -m "slow and not quarantine"` | only the heavyweight registry |
|
||||
| **full** | `make test-full` → `pytest -m "not quarantine"` | everything (what CI runs) |
|
||||
|
||||
The marker is **classification only** — it never skips. `-m slow` *selects* the
|
||||
slow tests; you choose a lane with an explicit marker expression. Plain
|
||||
`pytest` (no `-m`) still runs the full suite.
|
||||
|
||||
CI is unchanged: `.github/workflows/smoke.yml` and `full-pytest.yml` both run
|
||||
`-m "not quarantine"`, which includes the slow tests — so the split costs no CI
|
||||
coverage.
|
||||
|
||||
## Measured timings (10-core macOS, `CORE_BACKEND=numpy`)
|
||||
|
||||
| Lane | Serial | Parallel (`-n auto`) |
|
||||
|---|---|---|
|
||||
| full | 73 min | 25 min |
|
||||
| fast | ~26 min | **9.5 min** (9,590 passed) |
|
||||
|
||||
Combined (split + parallel) = **73 → 9.5 min (7.7×)**. The parallel fast lane
|
||||
scales ~5.7× because it excludes the 975s parallel-floor monster (see below);
|
||||
the full suite only reaches 2.9× because that one test pins a worker for 16 min.
|
||||
|
||||
`-n auto` is **not** wired into the `make` targets yet — see *Follow-up: xdist*.
|
||||
|
||||
## The slow registry (`conftest.py`)
|
||||
|
||||
Two registries, by cost shape:
|
||||
|
||||
- **`SLOW_FILES`** — whole-file: the cost is carried by a module/session-scoped
|
||||
fixture, so marking one test is insufficient (skipping it just shifts the
|
||||
fixture cost to the next test that requests it). 10 files.
|
||||
- **`SLOW_TESTS`** — exact nodeids: mixed files where only specific tests are
|
||||
soak/bench scale; the file's fast predicate/unit tests stay in the fast lane.
|
||||
26 tests across 14 files.
|
||||
|
||||
**Honest accounting** — the registry marks **912** of 10,596 tests slow. 801 of
|
||||
those are `test_cognition_eval_register_matrix.py` (a per-register × invariant
|
||||
eval matrix: many cheap parametrized assertions gated behind expensive
|
||||
per-register module-fixture setups). It is classified whole-file because the
|
||||
cost is in the module fixture, but be aware the fast lane therefore omits the
|
||||
register-matrix coverage; CI's full lane still runs it.
|
||||
|
||||
## Finding: the 975s `test_inner_loop_phase2` outlier
|
||||
|
||||
`test_inner_loop_phase2.py::TestCausalAttribution::test_null_control_matches_boundary_only`
|
||||
showed a **975s (16 min) setup** — the single largest test, and the parallel
|
||||
floor for the whole suite.
|
||||
|
||||
Probed: it is **expected proof-scale work, not a bug or runaway**. The cost is a
|
||||
module-scoped `phase2_report` fixture that runs the FSC corpus (9 cases: 1
|
||||
`public/v1` + 8 `dev`) through `run_lane`, which executes **4 conditions + 4
|
||||
determinism reruns = 8 full real-runtime pipeline turns per case**, plus a fresh
|
||||
`ChatRuntime()` per case (~5s each). 9 × 8 heavy pipeline turns ≈ 975s. The
|
||||
fixture is shared across the file's 5 tests, so the cost is paid once.
|
||||
|
||||
A possible optimization exists — share the primed runtime across the 4 conditions
|
||||
instead of reconstructing — but it touches the runner's determinism contract, so
|
||||
it is deferred, not done here.
|
||||
|
||||
## Follow-ups (separate PRs)
|
||||
|
||||
1. **xdist by default.** The fast/full lanes are *not* xdist-hermetic yet:
|
||||
fresh-env-dict subprocess tests (`tests/formation/*`, `test_identity_packs`)
|
||||
write to the repo `engine_state/` dir, and other tests write
|
||||
`evals/.../report.json` and `teaching/proposals/` — these **race** under
|
||||
parallel workers (e.g. `test_workbench_replay::test_replay_leaves_no_trace`
|
||||
fails under `-n auto`, passes serially). Isolate those writers, then wire
|
||||
`-n auto` into `make test-fast` / `test-full`. This is the same hermeticity
|
||||
theme as `docs/issues/default-engine-state-test-hygiene.md`.
|
||||
2. **Warm-runtime fixture.** The fast lane's remaining ~9.5 min (parallel) is a
|
||||
long tail of 1–15s `ChatRuntime` constructions, not outliers. A
|
||||
shared/session-scoped warm-runtime fixture for read-only tests would cut this
|
||||
further.
|
||||
|
|
@ -62,4 +62,5 @@ testpaths = ["tests"]
|
|||
markers = [
|
||||
"quarantine: pre-existing failure tracked in conftest.py QUARANTINE registry; excluded from full-pytest CI gate. See docs/test-debt-quarantine.md.",
|
||||
"uses_default_engine_state: test intentionally exercises the process default engine-state directory; opt out of per-test default isolation. See docs/issues/default-engine-state-test-hygiene.md.",
|
||||
"slow: soak / bench / replay / proof / eval-matrix test that dominates suite wall-clock; classified in conftest.py SLOW_FILES/SLOW_TESTS (not in-file decorators). Fast lane: -m 'not quarantine and not slow'. See docs/testing-lanes.md.",
|
||||
]
|
||||
|
|
|
|||
Loading…
Reference in a new issue