f03979f025
- docs/testing-lanes.md + Makefile: recommended determinism regression invocation (uv run python -m evals.close_derived_climb + contract pytest) as part of standard verification story / rerun flows. - evals/anti_regression/run_demo.py + tests/test_anti_regression_demo.py: hermetic embedding of the yardstick (lived IdleTickResult flag, semantic determine rule=direct, content_replay_checksum) into the core anti-regression / teaching demo flow (core demo anti-regression now carries + reports the Claim-B signals; active corpus remains untouched). - Supporting docs: contract.md (uv + refs), anti_regression_demo.md (complementary note), runtime_contracts.md (determination surface cross-ref). - New ratification artifact (pre-impl) + all success criteria met. See ratification for chosen approach + why only correct path. All existing tests/invariants green; hermeticity preserved; no core/engine changes.
119 lines
7.1 KiB
Markdown
119 lines
7.1 KiB
Markdown
# Testing lanes — fast / slow / full
|
||
|
||
The full pytest suite is ~10,600 tests and ~73 min serial. A small set of
|
||
heavyweight tests dominates that wall-clock, so we classify them and offer a
|
||
**fast lane** for local development. Classification is empirical
|
||
test-infrastructure metadata, so it lives in one auditable place
|
||
(`conftest.py`), beside the `QUARANTINE` registry — not as `@pytest.mark.slow`
|
||
decorators spread across ~24 files.
|
||
|
||
## Lanes
|
||
|
||
| Lane | Command | What it runs |
|
||
|---|---|---|
|
||
| **fast** | `make test-fast` → `pytest -m "not quarantine and not slow"` | everything except the slow registry |
|
||
| **slow** | `make test-slow` → `pytest -m "slow and not quarantine"` | only the heavyweight registry |
|
||
| **full** | `make test-full` → `pytest -m "not quarantine"` | everything (what CI runs) |
|
||
|
||
The marker is **classification only** — it never skips. `-m slow` *selects* the
|
||
slow tests; you choose a lane with an explicit marker expression. Plain
|
||
`pytest` (no `-m`) still runs the full suite.
|
||
|
||
CI is unchanged: `.github/workflows/smoke.yml` and `full-pytest.yml` both run
|
||
`-m "not quarantine"`, which includes the slow tests — so the split costs no CI
|
||
coverage.
|
||
|
||
## Measured timings (10-core macOS, `CORE_BACKEND=numpy`)
|
||
|
||
| Lane | Serial | Parallel (`-n auto`) |
|
||
|---|---|---|
|
||
| full | 73 min | 25 min |
|
||
| fast | ~26 min | **9.5 min** (9,590 passed) |
|
||
|
||
Combined (split + parallel) = **73 → 9.5 min (7.7×)**. The parallel fast lane
|
||
scales ~5.7× because it excludes the 975s parallel-floor monster (see below);
|
||
the full suite only reaches 2.9× because that one test pins a worker for 16 min.
|
||
|
||
`-n auto` is **not** wired into the `make` targets yet — see *Follow-up: xdist*.
|
||
|
||
## The slow registry (`conftest.py`)
|
||
|
||
Two registries, by cost shape:
|
||
|
||
- **`SLOW_FILES`** — whole-file: the cost is carried by a module/session-scoped
|
||
fixture, so marking one test is insufficient (skipping it just shifts the
|
||
fixture cost to the next test that requests it). 10 files.
|
||
- **`SLOW_TESTS`** — exact nodeids: mixed files where only specific tests are
|
||
soak/bench scale; the file's fast predicate/unit tests stay in the fast lane.
|
||
26 tests across 14 files.
|
||
|
||
**Honest accounting** — the registry marks **912** of 10,596 tests slow. 801 of
|
||
those are `test_cognition_eval_register_matrix.py` (a per-register × invariant
|
||
eval matrix: many cheap parametrized assertions gated behind expensive
|
||
per-register module-fixture setups). It is classified whole-file because the
|
||
cost is in the module fixture, but be aware the fast lane therefore omits the
|
||
register-matrix coverage; CI's full lane still runs it.
|
||
|
||
## Finding: the 975s `test_inner_loop_phase2` outlier
|
||
|
||
`test_inner_loop_phase2.py::TestCausalAttribution::test_null_control_matches_boundary_only`
|
||
showed a **975s (16 min) setup** — the single largest test, and the parallel
|
||
floor for the whole suite.
|
||
|
||
Probed: it is **expected proof-scale work, not a bug or runaway**. The cost is a
|
||
module-scoped `phase2_report` fixture that runs the FSC corpus (9 cases: 1
|
||
`public/v1` + 8 `dev`) through `run_lane`, which executes **4 conditions + 4
|
||
determinism reruns = 8 full real-runtime pipeline turns per case**, plus a fresh
|
||
`ChatRuntime()` per case (~5s each). 9 × 8 heavy pipeline turns ≈ 975s. The
|
||
fixture is shared across the file's 5 tests, so the cost is paid once.
|
||
|
||
A possible optimization exists — share the primed runtime across the 4 conditions
|
||
instead of reconstructing — but it touches the runner's determinism contract, so
|
||
it is deferred, not done here.
|
||
|
||
## Follow-ups (separate PRs)
|
||
|
||
1. **xdist by default.** The fast/full lanes are *not* xdist-hermetic yet:
|
||
fresh-env-dict subprocess tests (`tests/formation/*`, `test_identity_packs`)
|
||
write to the repo `engine_state/` dir, and other tests write
|
||
`evals/.../report.json` and `teaching/proposals/` — these **race** under
|
||
parallel workers (e.g. `test_workbench_replay::test_replay_leaves_no_trace`
|
||
fails under `-n auto`, passes serially). Isolate those writers, then wire
|
||
`-n auto` into `make test-fast` / `test-full`. This is the same hermeticity
|
||
theme as `docs/issues/default-engine-state-test-hygiene.md`.
|
||
2. **Warm-runtime fixture.** The fast lane's remaining ~9.5 min (parallel) is a
|
||
long tail of 1–15s `ChatRuntime` constructions, not outliers. A
|
||
shared/session-scoped warm-runtime fixture for read-only tests would cut this
|
||
further.
|
||
|
||
## Recommended determinism / teaching regression invocation (post-Claim-B hardening of CLOSE yardstick)
|
||
|
||
After any change touching CLOSE flywheel, idle_tick, realize_derived, consolidate_determinations, vault recall of realized facts, determine(), or the derived close proposal bridge, run the hardened yardstick as part of your determinism regression and anti-regression verification:
|
||
|
||
```bash
|
||
uv run python -m evals.close_derived_climb
|
||
uv run python -m pytest tests/test_derived_close_proposals.py tests/test_architectural_invariants.py -q
|
||
```
|
||
|
||
(Also available via `core demo anti-regression` which now embeds the yardstick — see below.)
|
||
|
||
This is the canonical "standard verification story" invocation for the CLOSE autonomous growth surface. It is the direct follow-up to the Claim-B hardening (#791) and makes the improved measurement recurring rather than isolated.
|
||
|
||
**What the hardened yardstick now exercises and measures (Claim B):**
|
||
- Real `ChatRuntime.idle_tick()` + `IdleTickResult.derived_close_proposals_emitted` (proposal flag gating via the lived runtime path, not a simulation).
|
||
- Explicit `determine()` calls on the post-fixed-point positive probes, asserting `Determined(True, rule='direct')` ("semantic_positives_determined_direct").
|
||
- `content_replay_checksum` covering canonical closure sets (with structure_key, Derivation, and premise_structure_keys) and full proposal bodies for exact-trajectory fidelity.
|
||
- Retained Claim A guarantees: strict/monotone growth (1/5/8), wrong_total == 0, negatives and excluded predicates refused, full determinism, hermetic (no serving, no ratification, SPECULATIVE/proposal_only only, all INV-30/31 etc. preserved).
|
||
|
||
See:
|
||
- `evals/close_derived_climb/contract.md` (metrics, scenarios, "no side effects")
|
||
- `docs/analysis/close-derived-climb-yardstick-claim-b-ratification-2026-06-16.md` (the hardening ratification)
|
||
- `docs/analysis/integrate-hardened-close-yardstick-determinism-teaching-regression-ratification-2026-06-16.md` (this integration ratification + "why only correct path")
|
||
- `docs/runtime_contracts.md` (determination surface contract exercised by the semantic asserts)
|
||
- `docs/evals/anti_regression_demo.md` (the anti-regression demo now runs the yardstick)
|
||
- `tests/test_anti_regression_demo.py` (contract test that pins the embedding)
|
||
- `Makefile` (comments under test lanes point here)
|
||
|
||
The yardstick itself remains hermetic per the rules in this document (fresh runtimes, internal temps only for proposal sink during flag test). It introduces no new writers to engine_state/, teaching/proposals/, or evals reports.
|
||
|
||
This integration (documentation promotion into the lanes + hermetic execution inside the anti-regression demo) is the highest-leverage way to ensure the project actually benefits from the hardened Claim-B measurement surface on every relevant regression run.
|