Commit graph

2 commits

Author SHA1 Message Date
Shay
f03979f025 feat: integrate hardened CLOSE yardstick into determinism & teaching regression surfaces
- docs/testing-lanes.md + Makefile: recommended determinism regression invocation (uv run python -m evals.close_derived_climb + contract pytest) as part of standard verification story / rerun flows.
- evals/anti_regression/run_demo.py + tests/test_anti_regression_demo.py: hermetic embedding of the yardstick (lived IdleTickResult flag, semantic determine rule=direct, content_replay_checksum) into the core anti-regression / teaching demo flow (core demo anti-regression now carries + reports the Claim-B signals; active corpus remains untouched).
- Supporting docs: contract.md (uv + refs), anti_regression_demo.md (complementary note), runtime_contracts.md (determination surface cross-ref).
- New ratification artifact (pre-impl) + all success criteria met.

See ratification for chosen approach + why only correct path.
All existing tests/invariants green; hermeticity preserved; no core/engine changes.
2026-06-16 17:25:05 -07:00
Shay
af4db1f79d
test: fast/slow/full lanes via central slow registry in conftest (#786)
A handful of soak / bench / replay / proof / eval-matrix tests dominate the
suite wall-clock (~50 tests ≈ the entire 73-min serial runtime; the other ~10k
are near-instant). Classify them so developers can run a fast lane locally.

Mechanism — central registry in conftest.py beside QUARANTINE (cost is empirical
test-infra metadata, not test semantics, so it lives in one auditable place, not
as decorators across ~24 files). Marker-only: it stamps `slow`, it NEVER skips,
so `-m slow` SELECTS the slow tests.

- SLOW_FILES (10): whole-file, where a module/session fixture carries the cost
  (marking one test would just shift the fixture cost to the next requester).
- SLOW_TESTS (26 nodeids): mixed files, so the file's fast predicate/unit tests
  stay in the fast lane.
- pytest_collection_modifyitems stamps `slow`; marker registered in pyproject.
  912 of 10,596 tests classified (801 = test_cognition_eval_register_matrix
  eval-matrix; called out in docs — honest accounting).

Lanes (Makefile + docs/testing-lanes.md):
  fast: pytest -m "not quarantine and not slow"
  slow: pytest -m "slow and not quarantine"
  full: pytest -m "not quarantine"        (unchanged — what CI runs)

CI is unaffected: smoke.yml and full-pytest.yml run `-m "not quarantine"`, which
still includes the slow tests. No CI behavior change; no test logic touched (the
change cannot alter pass/fail in the full lane — it only adds a marker).

Timings (10-core, numpy): full 73min serial / 25min -n auto; fast ~26min serial
/ 9.5min -n auto (7.7x combined). The 975s test_inner_loop_phase2 outlier was
probed (expected proof-scale work, not a bug; finding in docs). Deferred to
follow-up PRs: -n auto by default (needs xdist hermeticity — fresh-env-dict
subprocess + report.json/proposals writers race under parallel workers) and a
warm-runtime fixture for the remaining 1-15s ChatRuntime tail.
2026-06-15 17:30:50 -07:00