34295e55ce
Full lane wall-time: 6:35 → 2:25 (2.7× speedup). No behavioral changes; same 1933 passed, 2 skipped. Three wins, biggest first: 1. pytest-xdist as a project dependency. ``pyproject.toml`` gains ``pytest-xdist>=3.6``. ``cmd_test`` injects ``-n auto`` for ``--suite full`` when xdist is importable; curated suites stay single-process because worker-spawn overhead is net-negative on the smaller suites. Operator can override via passing ``-n <N>`` or ``--dist`` explicitly. Verified: ``core test --suite full -q`` prints ``bringing up nodes...`` and parallelises across the runner's CPUs. 2. Module-scoped fixture for run_demo() in test_learning_loop_demo.py. The 7 demo tests each previously called ``run_demo(emit_json=True)`` from scratch — and ``run_demo`` itself runs the cognition lane twice via the replay-equivalence gate. ~15s/file → ~3s/file. Module scope (not session) is intentional: pytest-xdist distributes by test, so a session-scoped fixture would still be re-evaluated per worker that picks up a test from this file. Module scope keeps the cost paid once per worker per file, which is the actual lower bound. 3. Module-scoped fixture for the teaching-loop bench. ``test_teaching_loop_bench.py``'s 5 tests previously each ran ``run_teaching_loop_determinism(runs=2 or 3)`` — 12 pipeline invocations across the file. One ``runs=3`` invocation shared across all 5 tests covers every assertion: ~25s → ~7s. For local iteration, ``core test --suite cognition -q`` etc. remain fast (no xdist overhead). The full-lane speedup is most visible under CI / pre-merge runs.
70 lines
2.6 KiB
Python
70 lines
2.6 KiB
Python
"""Teaching-loop determinism benchmark — falsifiable claim test.
|
||
|
||
The bench itself runs at any N ≥ 1; the test pins the headline claim
|
||
at a low N for fast CI. Headline claim:
|
||
|
||
"N identical inputs produce N byte-identical proposal artifacts,
|
||
the active corpus is byte-identical pre/post, and the wall-time
|
||
distribution is well-formed."
|
||
|
||
If determinism breaks anywhere in the pipeline (proposal_id hashing,
|
||
replay-equivalence gate, accept-side corpus_append, ProposalLog
|
||
replay), at least one of the ``unique_*`` counts in the bench report
|
||
will exceed 1 and this test fails.
|
||
|
||
Performance: a module-scoped fixture shares one ``runs=3`` invocation
|
||
across every test in this file. Each ``run_teaching_loop_determinism``
|
||
call runs the propose/replay/accept pipeline N times; cutting from 5
|
||
calls (runs=3,2,2,3,2 = 12 pipeline runs) to 1 call (3 runs) is a
|
||
~4× speedup with no contract loss.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
|
||
import pytest
|
||
|
||
from benchmarks.teaching_loop import run_teaching_loop_determinism
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def bench_report():
|
||
"""Single ``runs=3`` invocation shared across every test in this
|
||
file. ``runs=3`` is the highest N any individual test asks for,
|
||
so the determinism / latency / serialisation assertions all hold
|
||
against the same report."""
|
||
return run_teaching_loop_determinism(runs=3)
|
||
|
||
|
||
def test_teaching_loop_is_deterministic_across_three_runs(bench_report) -> None:
|
||
assert bench_report.deterministic is True
|
||
assert bench_report.active_corpus_byte_identical is True
|
||
assert bench_report.unique_proposal_ids == 1
|
||
assert bench_report.unique_replay_baselines == 1
|
||
assert bench_report.unique_replay_candidates == 1
|
||
assert bench_report.unique_regressed_metrics == 1
|
||
assert bench_report.unique_chain_ids == 1
|
||
|
||
|
||
def test_proposal_id_is_a_sha256_prefix(bench_report) -> None:
|
||
pid = bench_report.sample_proposal_id
|
||
assert len(pid) == 32
|
||
assert all(c in "0123456789abcdef" for c in pid)
|
||
|
||
|
||
def test_chain_id_matches_canonical_layout(bench_report) -> None:
|
||
assert bench_report.sample_chain_id == "cause_thought_reveals_meaning"
|
||
|
||
|
||
def test_latency_stats_are_well_formed(bench_report) -> None:
|
||
assert bench_report.elapsed_mean_s > 0.0
|
||
assert bench_report.elapsed_p50_s > 0.0
|
||
assert bench_report.elapsed_p95_s >= bench_report.elapsed_p50_s
|
||
assert bench_report.elapsed_total_s >= bench_report.elapsed_mean_s * bench_report.runs * 0.9
|
||
|
||
|
||
def test_report_serialises_to_json(bench_report) -> None:
|
||
blob = json.dumps(bench_report.as_dict(), sort_keys=True)
|
||
assert "deterministic" in blob
|
||
assert "active_corpus_byte_identical" in blob
|