Hunt for the -n auto fast-lane polluters flagged in docs/testing-lanes.md
("Follow-up: xdist by default"). Three root causes, all shared-repo-path
writers with no per-test isolation, following the #782 monkeypatch idiom:
1. tests/test_workbench_replay.py::test_replay_leaves_no_trace hardcoded
`Path("engine_state")` (the real shared dir) instead of reading
`engine_state._DEFAULT_DIR`, which the root-conftest autouse fixture
already redirects per-test. It was a victim, not a polluter: any
concurrent worker writing the real dir made this snapshot-diff flake.
Fixed by reading `engine_state._DEFAULT_DIR` dynamically.
2. evals/gsm8k_math/train_sample/v1/runner.py hardcoded its report.json
output to the committed repo path with no override. Two test files
(test_rat1_end_to_end_admission.py, test_wave_a_multiplicative_
aggregation_injector.py; 4 tests total) spawn it as a subprocess and
read the same file back — a write race under -n auto, and a confirmed
downstream victim (test_gsm8k_sealed_attempt_scout.py::
test_report_json_mtime_unchanged_by_scout_import asserts the file's
mtime is stable). Added an optional CORE_GSM8K_TRAIN_SAMPLE_REPORT_PATH
env override (default unchanged) and pointed the 4 call sites at
tmp_path.
3. core/cli.py's `_DEMO_RESULTS_DIR` (evals/forward_semantic_control/
results/) is written, glob-scanned, and index.json-rebuilt by ~11 tests
across tests/test_cli_demo.py's TestDemoSubcommand and TestDemoPreambles
classes. Added an autouse fixture monkeypatching `cli._DEMO_RESULTS_DIR`
to a per-test tmp dir. This also unmasked a latent order-dependent
coupling: test_demo_list_results_indexes_reports and
test_demo_list_results_json_well_formed never wrote their own report,
relying on a sibling test's leftover file in the shared dir (silently
correct only because pytest ran the file in declaration order). Made
both self-contained.
No assertion weakening, no test deletion, no global autouse fixture masking
real bugs. All three fixes preserve the real (non-test) default behavior
byte-for-byte when unpatched/env-unset.
Verification: 8x targeted -n 8 loop over the 7 affected files (before and
after) did not force-reproduce the underlying race live (narrow timing
window, small-scale run) — confirmation is source-level (hardcoded shared
paths bypassing the established isolation idiom) plus the prior documented
flake for test_replay_leaves_no_trace in docs/testing-lanes.md. Full
fast-lane -n auto run recorded in the PR description.
Two-pronged self-documentation pass so reviewers / investors / the
future team can revisit any artifact cold and immediately understand
what it tests, what to expect, and what to do if the numbers shift.
Inline preambles (`core demo`):
Before each demo's results table, print a structured preamble:
- WHAT THIS DEMO TESTS mechanism + corpus shape
- WHAT TO EXPECT IF WORKING concrete pass numbers
- WHAT TO LOOK FOR specific signals on regression
- WHEN TO TWEAK falsifiability + corpus authoring rules
Suppressed under --json so machine-readable output is uncluttered.
Wired into:
core demo phase5 (5-family stratified mechanism-isolation)
core demo phase6 (3-condition head-to-head vs baseline)
core demo all (combined; both preambles + a "what this means"
summary after the combined table)
Per-directory READMEs:
evals/forward_semantic_control/results/README.md
- Inventory of every JSON report with headline metrics
- Per-report interpretation guide ("when to look here")
- Per-case schema reference
- "When something looks wrong" troubleshooting tree
- Cross-links to ADRs, runtime_contracts, findings docs
evals/forward_semantic_control/public/v2_phase5/README.md
- The five failure-mode families, geometric construction, and
expected behaviour per mode
- Case schemas (single-step + chained) with field semantics
- How cases were geometrically mined (phase5_mine.py)
- Authoring rules: add cases, never relax assertions
evals/forward_semantic_control/public/v2_phase6_demo/README.md
- The three conditions with case counts and what each proves
- Why the baseline is in-system (not a transformer LLM) — table
- Case schema with the `condition` field
- Authoring rules: surface specific asymmetry, never relax predicate
evals/forward_semantic_control/public/inner_loop_benign/README.md
- Why this corpus exists (replaces adversarial-by-accident v1/dev)
- The Cl(4,1) signature quirk (23/85 tokens with negative
self-cga_inner) and the 0.25 self-score authoring filter
- Expected exhaustion_rate per condition
- How to verify a new case before committing (one-liner snippet)
New contract tests (tests/test_cli_demo.py::TestDemoPreambles + ::TestResultsReadme):
- Phase 6 preamble explains C1/C2/C3 and the in-system baseline rationale
- Phase 5 preamble explains all five families AND that δ is falsifiable
- Preamble suppressed under --json (parseable JSON from byte 0)
- `demo all` runs both preambles + a "what this means" summary
- results/README.md mentions every phase report file
- All three corpus READMEs exist
Tests: 1107 passed, 2 skipped (+8 from preceding baseline).
No mechanism changes — all additions are documentation surface.
Shay