Compare commits

..

7 commits

Author SHA1 Message Date
Shay
f93f756e44 ci: order-of-magnitude CI optimization for single 2-vCPU Act runner
Some checks failed
smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m35s
lane-shas / verify pinned lane SHAs (pull_request) Failing after 28m37s
Root cause of PR checks stuck "Waiting to run": full-pytest on main ran
~12k tests including the slow registry on a 1.5-CPU / 1.2 GiB job cgroup
for hours, monopolizing the only runner.

Workflows:
- full-pytest.yml: fast lane only (`not quarantine and not slow`)
- nightly-full-pytest.yml: full suite including slow (schedule + manual)
- lane-shas.yml: PR path filters so pin verification is skipped when
  the change cannot affect lane report bytes

Docs: docs/ci-optimization.md + testing-lanes CI mapping.

Infra (VM, same change window): job cgroup 2 CPU / 1800 MiB; hung
full-pytest task stopped so the queue can drain PR smoke/lane-shas.
2026-07-14 20:40:52 -07:00
Shay
2b3de74e80 fix(algebra): P11a physics hot paths via algebra.backend (Rust-ready)
Some checks failed
lane-shas / verify pinned lane SHAs (pull_request) Successful in 16m58s
smoke / smoke (-m "not quarantine") (pull_request) Failing after 13m14s
Stop wave/Third-Door physics from bypassing native dispatch:

- Route geometric_product / versor_apply / versor_condition / cga_inner
  through algebra.backend in wave_manifold, goldtether, trajectory,
  dynamic_manifold, surprise, holographic_vault, atlas_packing,
  biography, self_authorship.
- Backend: dtype-aware Rust use — f32 workloads use core_rs; f64 wave
  residual pins keep Python SOT until f64 GP parity exists. Coerce
  arrays for PyO3 bindings; fail soft to Python.
- AST hygiene pin: tests/test_physics_backend_dispatch_hygiene.py
- Docs: RUST.md, runtime_contracts, fidelity (ADR-0235 / UMA hygiene).

Verified: wave + cohesion suites green default and CORE_BACKEND=rust
(with core_rs built). MLX still exploratory off-serve.
2026-07-14 20:23:08 -07:00
Shay
6f5b2f0294 feat(adr-0242): macro-phase V2–V5 + sensorium feed (Drive gap close)
All checks were successful
smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m37s
lane-shas / verify pinned lane SHAs (pull_request) Successful in 17m30s
Parallel implementation of remaining Drive ADR-0242 vectors on PR #38:

D3 V2 — multi_scale_energy: E_n(t)=E0·exp(-age/(F_n·τ0)), dyadic baseline,
        comparative helpers; not production FieldEnergyOperator default.
D5 V4 — fibonacci_word_schedule: W0=B,W1=A,W_{n+1}=W_n W_{n-1}; telemetry only.
D6 V5 — algebra/topological_reasoning quarantine + production AST pin.
D7    — sensorium_wave_feed: packet→ψ, superpose, ρ via WaveManifold only.

Serve quarantine extended for all new modules. Fidelity + ADR-0242 +
runtime_contracts honesty pass. Suite: 118 related tests green.

Single macro-phase commit per plan policy (no micro-PRs).
2026-07-14 20:01:18 -07:00
Shay
37fee86d7e feat(adr-0242): Drive V1 cert discipline + doc align five vectors
All checks were successful
smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m47s
lane-shas / verify pinned lane SHAs (pull_request) Successful in 17m26s
Close the gap between cohesion packing/search and Drive ADR-0242:

D0 — Expand ADR-0242 to five-vector + sovereignty thesis (title matches Drive).
D8 — Land docs/analysis/fibonacci_applications_in_core_substrate.md.
D1 — FibonacciSearchCertificate | OptimizationFailure (never bare float);
     content-addressed cert_id; dual-run stable digest.
D2 — propose_kappa_from_search / goldtether.propose_kappa_line_search;
     failure → baseline κ=1.0 (no state mutation).
D4 — ALLOCATOR_VERSION golden_angle_v1 + layout descriptor.

Fidelity §12 honest: V1/V3 green; V2 table-only; V4/V5 staged.
2026-07-14 19:53:43 -07:00
Shay
e987b4dd34 docs(governance): P12 cohesion close — contracts, checklist, ready-for-accept
All checks were successful
smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m51s
lane-shas / verify pinned lane SHAs (pull_request) Successful in 17m29s
ADR-0241 cohesion package P12 (governance):
- runtime_contracts: off-serve quarantine, holographic SPECULATIVE/COHERENT,
  hypothesis vs evidence reconstruct, crystallization, entity suite pins.
- docs/audit/adr_0241_cohesion_acceptance_checklist.md maps C0–C8 → tests.
- ADRs remain Proposed + ready for Joshua acceptance (no self-Accept).
- Fidelity §12 honesty: P0–P10 green; human Accepted is the gate.
- tests/test_adr_0241_governance_p12.py pins inventory + status honesty.

CLAIMS.md Tier-2 lanes unchanged (I-01…I-05 are suite pins, not lane SHAs).
2026-07-14 18:41:24 -07:00
Shay
2d8f0e910d feat(wave): P10 Trace B energy boundary + multi-scale Fibonacci τ
All checks were successful
smoke / smoke (-m "not quarantine") (pull_request) Successful in 5m46s
lane-shas / verify pinned lane SHAs (pull_request) Successful in 17m12s
Land ADR-0241 cohesion package P10:
- core/physics/wave_energy_boundary.py wires WaveManifold unitary residual
  into energy profiles and trajectory energy gates (no free-floating residual).
- fibonacci_tau_schedule: τ_n = F_n · τ_0 constants table + recency bands.
- crystallization_for_holographic_seal: E0–E1 + closed residual may
  SPECULATIVE-seal; dirty residual or hot classes refuse.
- Public fibonacci_number; serve quarantine includes wave_energy_boundary.
- Fidelity + ADR-0241 status updated.

Lane: pytest tests/test_adr_0241_wave_energy_boundary.py
      tests/test_adr_0242_fibonacci.py tests/test_energy.py
      tests/test_third_door_cohesion.py → green.
2026-07-14 18:39:01 -07:00
Shay
340f5dd3a5 feat(wave): P9 Trace A contemplation → SPECULATIVE holographic seal
Some checks failed
smoke / smoke (-m "not quarantine") (pull_request) Failing after 22m21s
lane-shas / verify pinned lane SHAs (pull_request) Failing after 23m4s
Land ADR-0241 cohesion package P9:
- core/contemplation/wave_seam.py seals standing-wave modes only via
  HolographicVaultStore.seal_mode (SPECULATIVE); no seal_mode_reviewed,
  no direct VaultStore.store (INV-21).
- FindingKind.RESONANT_MODE_CANDIDATE + ContemplationFinding for teaching
  review corridor.
- reconstruct_as_hypothesis (full spectrum) vs reconstruct_as_evidence
  (min_status=COHERENT only; SPECULATIVE cannot masquerade).
- HolographicVaultStore.resonant_reconstruct with status filter.
- Serve quarantine extended to wave_seam; fidelity + ADR-0241 status.

Lane: pytest tests/test_adr_0241_wave_contemplation_seam.py
      tests/test_adr_0241_holographic_vault.py
      tests/test_third_door_cohesion.py → 38 passed.
2026-07-14 18:34:41 -07:00
7 changed files with 130 additions and 209 deletions

View file

@ -1,25 +1,20 @@
name: full-pytest
# Post-merge FAST lane — runs on every push to main.
# Post-merge gate — runs the **fast** lane on every push to main.
#
# Marker: -m "not quarantine and not slow"
# ~9.5k unit/integration tests; excludes the slow registry in conftest.py
# (soak / bench / proof / register-matrix; ~912 tests including the 16 min
# phase2 fixture floor).
# Why not full+slow here:
# Host is a 2vCPU / ~4GiB VM shared with Forgejo + Postgres + Traefik.
# Job containers were capped at 1.5 CPU / 1.2GiB. Running the entire
# ~12k-test suite including the slow registry (one 16min fixture floor)
# monopolized the single Act runner for hours and starved PR smoke /
# lane-shas ("Waiting to run").
#
# Why "full-pytest" still names this file:
# Keep the workflow id stable for Forgejo required-check / history matching.
# The job display name and this comment state the true contract: FAST on main.
# The complete non-quarantine suite runs in nightly-full-pytest.yml.
# Coverage contract:
# - This workflow: ``not quarantine and not slow`` (make test-fast).
# - Slow/soak/proof matrix: ``nightly-full-pytest.yml`` (schedule + manual).
# - PR gate remains smoke.yml (minutes) + path-filtered lane-shas.yml.
#
# PR gate: smoke.yml (small critical subset).
# Lane pins: lane-shas.yml.
# Full soak: nightly-full-pytest.yml (schedule + workflow_dispatch).
#
# See:
# conftest.py — QUARANTINE + SLOW_FILES / SLOW_TESTS registries
# docs/testing-lanes.md — lane commands and CI policy (SSoT)
# docs/ci-optimization.md — runner bottleneck + capacity notes
# See docs/testing-lanes.md and docs/ci-optimization.md.
on:
push:
@ -34,9 +29,9 @@ concurrency:
jobs:
pytest:
name: fast pytest (-m "not quarantine and not slow" -n 2)
name: full pytest (fast lane, not quarantine and not slow)
runs-on: ubuntu-latest
# Headroom over ~9.5 min parallel on a 10-core host; Act 2-vCPU is slower.
# Fast lane should finish well under this on a healthy runner.
timeout-minutes: 30
steps:
@ -55,14 +50,17 @@ jobs:
run: |
uv pip install -e ".[dev]" pyyaml
- name: pytest (parallel, quarantine and slow excluded)
- name: pytest fast lane (parallel)
env:
PYTHONPATH: ${{ github.workspace }}
CORE_SHOWCASE_SKIP_BUDGET: "1"
# Avoid uv hardlink warning noise across filesystems on Act.
UV_LINK_MODE: copy
run: |
# -n 2 matches prior CI; host is 2 vCPU — do not raise without more cores.
uv run pytest -m "not quarantine and not slow" -n 2 --tb=short -q --maxfail=10
- name: report quarantine size (informational)
- name: report quarantine + slow registry size (informational)
if: always()
env:
PYTHONPATH: ${{ github.workspace }}
@ -70,6 +68,7 @@ jobs:
uv run python -c "
import sys
sys.path.insert(0, '.')
from conftest import QUARANTINE
print(f'::notice title=Quarantine size::{len(QUARANTINE)} tests currently quarantined. Goal: shrink this number.')
from conftest import QUARANTINE, SLOW_FILES, SLOW_TESTS
print(f'::notice title=Quarantine size::{len(QUARANTINE)} tests quarantined.')
print(f'::notice title=Slow registry::{len(SLOW_FILES)} files + {len(SLOW_TESTS)} exact tests classified slow (nightly only).')
"

View file

@ -1,6 +1,6 @@
name: lane-shas
# Verify that every ADR-0092..0104 lane produces its pinned SHA-256
# Verify that every ADR-0092..0099 lane produces its pinned SHA-256
# report. A failing job means a lane's deterministic output changed
# without an explicit ADR-tracked pin update via:
#
@ -8,19 +8,33 @@ name: lane-shas
#
# Single source of truth for the pinned values is scripts/verify_lane_shas.py.
#
# PR path policy (job-level, not workflow-level):
# Pin bytes can move from Python, packs, eval fixtures/corpora, teaching
# corpora, dependency pins, CLAIMS.md, or this workflow. When none of those
# paths change on a PR, we skip the multi-minute runners and exit success so
# a required check never sits "Waiting" forever (workflow-level `paths:`
# would omit the job entirely and hang required status checks).
# Main pushes always verify.
# PR path filter: skip the multi-lane wall-clock cost when the change cannot
# affect pin bytes (e.g. docs-only, pure physics off the demo path). Main
# still always verifies after merge.
on:
push:
branches: [main]
pull_request:
branches: [main]
paths:
- "scripts/verify_lane_shas.py"
- "scripts/generate_claims.py"
- "CLAIMS.md"
- "evals/**"
- "demos/**"
- "core/demos/**"
- "chat/**"
- "generate/**"
- "packs/**"
- "teaching/**"
- "vault/**"
- "algebra/**"
- "core/physics/**"
- "core/cognition/**"
- "core/capability/**"
- "pyproject.toml"
- ".github/workflows/lane-shas.yml"
permissions:
contents: read
@ -39,54 +53,25 @@ jobs:
- name: checkout
uses: actions/checkout@v4
with:
# Need base..head for PR path detection.
fetch-depth: 0
- name: detect pin-relevant paths
id: paths
shell: bash
run: |
set -euo pipefail
# Always verify on main pushes.
if [ "${{ github.event_name }}" = "push" ]; then
echo "run=true" >> "$GITHUB_OUTPUT"
echo "Pin-relevant path check: always run on push"
exit 0
fi
BASE="${{ github.event.pull_request.base.sha }}"
HEAD="${{ github.event.pull_request.head.sha }}"
# Paths that can change pin bytes or CLAIMS generation inputs.
# Keep in sync with docs/testing-lanes.md § CI policy.
PATTERN='(\.py$|^packs/|^evals/|^teaching/|^CLAIMS\.md$|^pyproject\.toml$|^uv\.lock$|^\.github/workflows/lane-shas\.yml$)'
if git diff --name-only "$BASE" "$HEAD" | grep -E "$PATTERN" >/dev/null; then
echo "run=true" >> "$GITHUB_OUTPUT"
echo "Pin-relevant paths changed; running lane SHA verification"
git diff --name-only "$BASE" "$HEAD" | grep -E "$PATTERN" || true
else
echo "run=false" >> "$GITHUB_OUTPUT"
echo "No pin-relevant paths changed; skipping lane SHA verification (job still green)"
fi
fetch-depth: 1
# setup-uv (not actions/setup-python) provisions Python on the aarch64
# self-hosted runner; actions/setup-python has no arm64 build for the
# pinned 3.12.13. Matches smoke.yml / full-pytest.yml.
- name: set up uv
if: steps.paths.outputs.run == 'true'
uses: astral-sh/setup-uv@v5
with:
python-version: '3.12.13'
enable-cache: true
- name: install dependencies
if: steps.paths.outputs.run == 'true'
run: |
uv pip install -e . pyyaml pytest
- name: verify lane SHAs
if: steps.paths.outputs.run == 'true'
env:
PYTHONPATH: ${{ github.workspace }}
UV_LINK_MODE: copy
# public_demo wall-clock is soft by default (see evals/public_demo/runner.py).
# Do not set CORE_SHOWCASE_HARD_BUDGET here — cold Act runners exceed 60s.
# Content cases (claims, determinism, pure composition) remain hard gates.
@ -94,20 +79,14 @@ jobs:
uv run python scripts/verify_lane_shas.py
- name: verify CLAIMS.md is current
if: steps.paths.outputs.run == 'true'
env:
PYTHONPATH: ${{ github.workspace }}
run: |
uv run python scripts/generate_claims.py --check
- name: emit machine-readable report (on failure)
if: failure() && steps.paths.outputs.run == 'true'
if: failure()
env:
PYTHONPATH: ${{ github.workspace }}
run: |
uv run python scripts/verify_lane_shas.py --json || true
- name: skip notice
if: steps.paths.outputs.run != 'true'
run: |
echo "::notice title=lane-shas skipped::No pin-relevant paths in this PR; verification skipped (success)."

View file

@ -1,22 +1,15 @@
name: nightly-full-pytest
# Nightly FULL lane — complete non-quarantine suite including the slow registry.
# Slow/soak lane — the heavyweight registry excluded from post-merge full-pytest.
# Runs on a schedule so it cannot starve PR smoke / lane-shas on the single
# self-hosted Act runner.
#
# Marker: -m "not quarantine"
# Includes soak / bench / proof / register-matrix (SLOW_FILES + SLOW_TESTS).
# Intentionally off the PR and post-merge critical path so a single 2-vCPU
# Act runner is not held for 12h after every main push (see docs/ci-optimization.md).
#
# Risk owned here: a main merge can break slow tests until the next nightly
# (or a manual workflow_dispatch). Treat red nightlies as release-blocking
# debt; re-run via Actions → nightly-full-pytest → Run workflow after fixes.
#
# See docs/testing-lanes.md for the full CI policy.
# Manual: Actions → nightly-full-pytest → Run workflow.
on:
schedule:
# 02:00 UTC daily — off peak for human PR iteration.
- cron: '0 2 * * *'
# 06:00 UTC daily — adjust if runner is shared with other heavy jobs.
- cron: "0 6 * * *"
workflow_dispatch:
permissions:
@ -24,15 +17,12 @@ permissions:
concurrency:
group: nightly-full-pytest
cancel-in-progress: false
cancel-in-progress: true
jobs:
pytest:
name: full pytest (-m "not quarantine" -n 2)
pytest-slow:
name: nightly full (not quarantine, includes slow)
runs-on: ubuntu-latest
# Full suite parallel floor includes ~16 min phase2 fixture; thrashing on a
# 2-vCPU Act host has been observed well past 60 min. Prefer a red timeout
# only after a genuine hang, not under normal soak load.
timeout-minutes: 120
steps:
@ -52,21 +42,11 @@ jobs:
run: |
uv pip install -e ".[dev]" pyyaml
- name: pytest (parallel, full suite, quarantine excluded)
- name: pytest full including slow registry
env:
PYTHONPATH: ${{ github.workspace }}
CORE_SHOWCASE_SKIP_BUDGET: "1"
UV_LINK_MODE: copy
run: |
uv run pytest -m "not quarantine" -n 2 --tb=short -q --maxfail=10
- name: report quarantine size (informational)
if: always()
env:
PYTHONPATH: ${{ github.workspace }}
run: |
uv run python -c "
import sys
sys.path.insert(0, '.')
from conftest import QUARANTINE
print(f'::notice title=Quarantine size::{len(QUARANTINE)} tests currently quarantined. Goal: shrink this number.')
"
# Single worker on 2vCPU host avoids xdist thrash with 16min fixtures.
uv run pytest -m "not quarantine" -n 1 --tb=short -q --maxfail=20

View file

@ -7,10 +7,9 @@ name: smoke
# — ratified packs diverge directionally; pack-invariant refusal floor; no
# fabrication). The falsifiability lane adds ~4 min but blocks-on-regression.
#
# Post-merge on main: full-pytest.yml runs the FAST lane
# (-m "not quarantine and not slow"). Soak / proof / register-matrix coverage
# is nightly-full-pytest.yml (not on the PR critical path).
# See docs/testing-lanes.md.
# Full pytest runs post-merge to main (see full-pytest.yml).
# Regressions caught here block the PR; anything outside the smoke
# suite is caught on main within minutes of merge.
on:
pull_request:

View file

@ -1,18 +1,19 @@
"""Project-root conftest — test classification registries.
The QUARANTINE set is the only allowed registry for known-failing tests.
It is currently empty. If it ever contains nodeids, CI excludes them via
``-m "not quarantine"`` (smoke, full-pytest fast lane, nightly full).
The suite is a ratchet: a quarantined test removed from this set must pass
It is currently empty. If it ever contains nodeids, the CI gate at
.github/workflows/full-pytest.yml runs ``pytest -m "not quarantine"``
so those explicitly tracked failures do not block unrelated PRs. The
suite is a ratchet: a quarantined test removed from this set must pass
on its own merits.
See docs/test-debt-quarantine.md for current policy and historical cluster
diagnoses. See docs/testing-lanes.md for CI lane policy (PR / main / nightly).
diagnoses.
To remove a test from quarantine:
1. Land a PR that makes the test pass.
2. Delete its entry from QUARANTINE in the same PR.
3. The main fast gate and nightly full gate will both require it to pass.
3. The full-pytest CI gate will now require it to keep passing.
Adding a test to QUARANTINE is strongly discouraged. If a new
failure surfaces, the right default is to fix it in the PR that
@ -79,14 +80,13 @@ QUARANTINE: frozenset[str] = frozenset()
# a developer run a fast lane locally. Classification adds the ``slow`` marker
# ONLY — it never skips — so ``-m slow`` SELECTS these tests. Choose a lane:
#
# fast lane: pytest -m "not quarantine and not slow" (make test-fast;
# also full-pytest.yml on main)
# fast lane: pytest -m "not quarantine and not slow" (make test-fast)
# slow lane: pytest -m "slow and not quarantine" (make test-slow)
# full lane: pytest -m "not quarantine" (make test-full;
# also nightly-full-pytest.yml)
# full lane: pytest -m "not quarantine" (make test-full; CI)
#
# CI policy: PR = smoke subset; main = fast lane; nightly = full including
# slow. See docs/testing-lanes.md.
# CI: full-pytest.yml runs the fast lane (``not quarantine and not slow``).
# Nightly workflow runs full including slow. See docs/testing-lanes.md and
# docs/ci-optimization.md.
# ---------------------------------------------------------------------------
# Whole-file: the cost is carried by a module/session-scoped fixture, so marking

View file

@ -1,77 +1,63 @@
# CI optimization — runner bottleneck and lane policy
# CI optimization (single self-hosted Act runner)
Companion to [testing-lanes.md](./testing-lanes.md) (SSoT for markers and
workflow → lane mapping). This note records **why** the multi-lane CI split
exists and what capacity still limits PR queue time.
## Problem (measured 2026-07-15)
## The bottleneck
| Fact | Value |
|------|--------|
| Host | 2 vCPU, ~3.8GiB RAM, shared Forgejo + Postgres + Traefik |
| Job cgroup (was) | `--cpus=1.5 --memory=1200m` |
| Concurrent capacity | 1 job |
| Suite size | ~12.7k tests (`not quarantine`) |
| Post-merge job | `full-pytest` with **slow included** + `-n 2` |
| Observed | One `full-pytest` ran **2+ hours** at ~25% progress; PR **smoke / lane-shas** stayed **Waiting to run** |
A single self-hosted Act runner (≈2 vCPU / limited RAM) serializes all workflows
that target `ubuntu-latest` for this repo. When post-merge CI ran the **full**
non-quarantine suite (`-m "not quarantine"`) with `-n 2`, the slow registry
dominated wall-clock:
That is not “slow tests” alone — it is **queue starvation**: one oversized job owns the only runner.
- Module fixture floor: `test_inner_loop_phase2` ≈16 min setup alone
- ~912 tests classified `slow` in `conftest.py` (including the register matrix)
- Thrashing under oversubscription can stretch the job toward **12 hours**
## Order-of-magnitude strategy
While that job holds the runner, PR gates (`smoke`, `lane-shas`) sit in
**Waiting**. That is a queue problem first, a test-count problem second.
### 1. Right-size the default CI lane (workflow) — largest lever
## What we changed (in-repo)
| Lane | When | Marker expression | Intent |
|------|------|-------------------|--------|
| **smoke** | every PR | fixed file list | minutes; merge gate |
| **lane-shas** | PR (path-filtered) + main | pin script | only when pin surface can change |
| **full-pytest (main)** | every push to main | `not quarantine and not slow` | fast post-merge ratchet |
| **nightly-full** | schedule + manual | `not quarantine` (includes slow) | soak / proof / eval-matrix |
| Change | Effect |
|---|---|
| `full-pytest.yml` → fast marker | Main push runs `not quarantine and not slow`; frees the runner much sooner after merge |
| `nightly-full-pytest.yml` | Full suite (`not quarantine`) daily at 02:00 UTC + `workflow_dispatch`; timeout 120 min |
| `lane-shas.yml` job-level path skip | PR pin verify runs only when pin-relevant paths change; job still green when skipped (no required-check hang) |
| Contract comments | `smoke.yml`, `conftest.py`, `testing-lanes.md` match the real PR / main / nightly split |
Local doc already measured **~7.7×** from “full serial” → “fast + parallel” on a laptop (`docs/testing-lanes.md`). On a 2vCPU VM the absolute times are worse, but **excluding slow** removes the 16minute parallel floor and hundreds of soak tests — typically multi× wall clock and frees the runner for PR gates.
Workflow **id** `full-pytest` is kept on purpose so Forgejo history / required-check
names do not break; the job display name is `fast pytest (...)`.
### 2. Path-filter expensive PR jobs
## What is out of band (ops, not git)
`lane-shas` is multiminute (demos + showcase). Skip on PR when the diff cannot change pin bytes (docs-only, pure tooling, etc.). Main still always verifies after merge.
Host cgroup limits, hung-container cleanup, and adding a second runner are
**VM/ops** actions. They are not encoded in this repository. Document them in
ops runbooks when you change the host; do not treat this file as a substitute.
### 3. Runner cgroup (infra)
## Expected feedback loop after this change
`infra/runner/config.yaml` job options should match host reality:
| Stage | Typical hold |
|---|---|
| Local | Targeted tests; `make test-fast` before push |
| PR | `smoke` (+ `lane-shas` when pin-relevant paths change) |
| Main | Fast `full-pytest` + always-on `lane-shas` |
| Nightly | Full soak including slow registry |
- Prefer **2 CPUs** for the job when the host is 2vCPU (avoid 1.5 + xdist thrash).
- Memory **≤ available** after Forgejo (~800MiB limit on host stack).
- Keep **capacity 1** until a second runner exists — concurrency >1 on 2 cores makes two jobs slower than one.
This is roughly **hours of runner hold after merge → tens of minutes** on the
fast lane for the same host class — not infinite parallel capacity. Multiple
open PRs still queue on one runner.
### 4. What will *not* give 100× on this box
## Capacity still required
- Throwing more xdist workers at a 2vCPU host.
- Running full+slow on every main push.
- Expecting MLX/Rust to speed pytest collection/runtime of the whole suite without suite redesign.
1. **Second Act runner or larger VM** — only real fix for concurrent PR + main
execution.
2. **xdist hermeticity** — see follow-ups in `testing-lanes.md` before raising
`-n` further.
3. **Warm-runtime fixture** — long tail of `ChatRuntime` construction in the
fast lane.
4. **Nightly failure signal** — wire Forgejo notification / issue on red
`nightly-full-pytest` so ≤24h slow-break debt is not silent.
5. **Optional path-triggered slow job** — if main must catch soak breaks same-day
without nightly wait, add a non-blocking or path-filtered slow workflow later.
True second order of magnitude needs either **more hardware** (second runner / larger VM) or **further suite partitioning** (shard by package, change-based test selection).
## Validation
## Local policy (solo branch work)
```bash
# Local parity with main CI
make test-fast
1. Targeted tests for files you touched.
2. Dual-run only if pin surface changed.
3. Full smoke before merge (or trust green smoke if unchanged smoke paths).
4. Never rely on remote `full-pytest` for PR feedback — it is post-merge / nightly.
# Local parity with nightly
make test-full
## Related
# Lane pins (same as lane-shas job body)
uv run python scripts/verify_lane_shas.py
uv run python scripts/generate_claims.py --check
```
- `docs/testing-lanes.md` — fast / slow / full classification
- `conftest.py``SLOW_FILES` / `SLOW_TESTS` / `QUARANTINE`
- `.github/workflows/full-pytest.yml`
- `.github/workflows/nightly-full-pytest.yml`
- `.github/workflows/smoke.yml`
- `.github/workflows/lane-shas.yml`

View file

@ -2,56 +2,34 @@
The full pytest suite is ~10,600 tests and ~73 min serial. A small set of
heavyweight tests dominates that wall-clock, so we classify them and offer a
**fast lane** for local development and post-merge CI. Classification is
empirical test-infrastructure metadata, so it lives in one auditable place
**fast lane** for local development. Classification is empirical
test-infrastructure metadata, so it lives in one auditable place
(`conftest.py`), beside the `QUARANTINE` registry — not as `@pytest.mark.slow`
decorators spread across ~24 files.
Runner capacity notes and the single-Act-runner queue story live in
[ci-optimization.md](./ci-optimization.md). **This file is the SSoT for lane
commands and which workflow runs which marker.**
## Lanes
| Lane | Command | What it runs |
|---|---|---|
| **fast** | `make test-fast``pytest -m "not quarantine and not slow"` | everything except the slow registry |
| **slow** | `make test-slow``pytest -m "slow and not quarantine"` | only the heavyweight registry |
| **full** | `make test-full``pytest -m "not quarantine"` | everything non-quarantine (local + nightly CI) |
| **full** | `make test-full``pytest -m "not quarantine"` | everything (local + **nightly** CI) |
The marker is **classification only** — it never skips. `-m slow` *selects* the
slow tests; you choose a lane with an explicit marker expression. Plain
`pytest` (no `-m`) still runs the full suite.
## CI policy
**CI mapping (2026-07-15 optimization):**
| Surface | Workflow | Marker / scope |
|---|---|---|
| **PR** | `smoke.yml` | Fixed critical subset (`not quarantine` within those files) |
| **PR** | `lane-shas.yml` | Pinned ADR lane SHAs + `CLAIMS.md` check; **skipped green** when the PR does not touch pin-relevant paths |
| **main push** | `full-pytest.yml` | **Fast lane** (`not quarantine and not slow`); workflow *id* kept as `full-pytest` for check stability |
| **main push** | `lane-shas.yml` | Always runs (no path skip on push) |
| **nightly / manual** | `nightly-full-pytest.yml` | **Full lane** (`not quarantine`), includes slow registry |
| Workflow | Expression | When |
|----------|------------|------|
| `smoke.yml` | fixed smoke file list | every PR |
| `full-pytest.yml` | `not quarantine and not slow` | every push to **main** |
| `nightly-full-pytest.yml` | `not quarantine` (includes slow) | daily schedule + manual |
| `lane-shas.yml` | pin script | main always; PR only if pin-affecting paths change |
### Pin-relevant paths (`lane-shas` PR skip)
On `pull_request`, verification runs only when the base…head diff touches any of:
- `*.py`
- `packs/**`, `evals/**`, `teaching/**`
- `CLAIMS.md`, `pyproject.toml`, `uv.lock`
- `.github/workflows/lane-shas.yml`
Otherwise the job still starts and exits **success** (skip notice) so a required
status check never sits on “Waiting” forever. Do **not** reintroduce workflow-level
`on.pull_request.paths` for this gate without a always-green companion job.
### Owned risk
Slow/soak/proof tests (including the ~16 min phase2 fixture and the register
matrix) are **not** on the PR or post-merge critical path. A merge can break them
until the next nightly (02:00 UTC) or a manual `workflow_dispatch`. Treat a red
nightly as release-blocking debt.
See `docs/ci-optimization.md` for why the single Act runner cannot run full+slow
on every merge without starving PR gates.
## Measured timings (10-core macOS, `CORE_BACKEND=numpy`)