From c93c1f6ce7404f11f82be58ad66ecd8efd7f36f5 Mon Sep 17 00:00:00 2001 From: Shay Date: Mon, 25 May 2026 06:08:47 -0700 Subject: [PATCH] docs: master plan post-substrate-audit (7 phases + EoD snapshot) (#262) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-of-day handoff doc. Carries strategic plan + active-PR snapshot across the session boundary so morning-self picks up cleanly. Contents: - Three guiding rules (fully-wired definition, thesis check, L10 is load-bearing) - EoD snapshot: PRs in flight (#261 B, gate unopened), 5 W-* closures today, 49-test quarantine registry with cluster breakdown, AM resumption order - 7-phase plan in dependency order: 1. Hygiene & honesty (today) 2. Operator decisions (5 quick calls) 3. Test debt paydown (5 clusters, easiest→hardest) 4. L10 runtime model (THE load-bearing decision) 5. L10 cascade (6 items unlock once L10 commits) 6. 'Decodes, not generates' milestone (Phase 5 enables this) 7. Validation & projection (patents, Rust parity, benchmarks) - Tonight's lessons (4 items including pytest-xdist surprises and the lane subprocess timeout calibration) - Cross-refs to ratchet, registry, scopes, and memory entries --- docs/master-plan-post-substrate-audit.md | 201 +++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 docs/master-plan-post-substrate-audit.md diff --git a/docs/master-plan-post-substrate-audit.md b/docs/master-plan-post-substrate-audit.md new file mode 100644 index 00000000..c3662f2e --- /dev/null +++ b/docs/master-plan-post-substrate-audit.md @@ -0,0 +1,201 @@ +# Master plan — post-substrate-audit + +**Authored:** 2026-05-24 (end-of-day, PST) — call it a night, fresh AM. +**Author intent:** carry the strategic plan + active-PR snapshot across the +session boundary so morning-self picks up cleanly. + +--- + +## Three guiding rules + +1. **"Fully wired"** = every `W-*` entry in [substrate-liveness-ratchet](audit/substrate-liveness-ratchet.md) CLOSED. When the registry is empty, the design executes as written. +2. **Thesis check on every move**: per [[thesis-decoding-not-generating]] — does this teach the engine to find better, or just store another found thing? Only the former clears the bar. +3. **L10 is the load-bearing decision.** It gates W-003, W-007, W-009, W-017, W-018, and the recognizer-storage ADR. Wrong shape locks in wrong architecture for the rest of the cascade. + +--- + +## End-of-day snapshot (2026-05-24) + +### Active in flight + +| PR | Branch | State | Blocker | +|----|--------|-------|---------| +| **#261** (B speedup) | `chore/test-speed-b-extract-lane-runners` | CI rerunning after timeout bump + test_lane_sha_verifier fix | Wait for `verify pinned lane SHAs` to go green | +| **(unopened)** Gate | `ci/full-pytest-gate-quarantine-markers` (pushed, no PR) | Locally verified: 6657 pass / 0 unexpected fail / 76 skip / 49 quarantine | Open PR **after #261 merges** (otherwise gate's CI hits pre-fix lane-shas state) | + +### Closed today (5 W-* + 5 PRs) + +- **W-004** — vault E2 re-thaw (PR #251) +- **W-015** — `_slerp_toward` → rotor-geodesic anchor pull (PR #255) +- **W-016** — vault probe in discovery loop (PR #257) +- **W-011** — recognition refusal propagation (PR #258, paired) +- **W-012** — `InnerLoopExhaustion` caught in ChatRuntime (PR #258, paired) +- Ratchet v5 (PR #260) — registry updated + +### Quarantine registry (49 tests) + +Cluster A (4) — ADR ledger drift, one-token extensions, shape of W-002 +Cluster B (15) — surface decoration drift (`pack-grounded ()` suffix) +Cluster C (27) — lane/runner metric drift (some are real regressions) +Cluster D (2) — CLI/internal API drift +Cluster E (1) — pytest-xdist parallel-execution incompatibility (`test_articulation_bench::test_footprint_emits_samples_and_bounds`) + +All confirmed pre-existing via bisect against `c1a1b7a` (pre-W-*-work commit). Today's 5 closures introduced **zero** new failures. Cluster E is the lone xdist-induced regression — caused by gate's `-n 4` choice, not by today's substantive work. + +### Resumption order in AM + +1. **First**: check PR #261's CI re-run status. If green, merge. If red, debug from CI logs. +2. **Second**: rebase `ci/full-pytest-gate-quarantine-markers` onto post-#261 main; open as PR. +3. **Third**: read this plan top-to-bottom to re-load context. +4. **Fourth**: Phase 2 work — five operator decisions queued (W-006, W-010, W-013, W-014, W-019). Each ~30 min. + +--- + +## Phase 1 — Hygiene & honesty (today, in flight) + +- **B PR #261** — extract `math_teaching_corpus` lane from pytest into CI lane SHAs (−9 min suite time, ~30% reduction) +- **Gate PR** (pending #261) — `conftest.py` QUARANTINE registry of 49 known-failing tests + new `full-pytest.yml` workflow running `pytest -m "not quarantine" -n 4` +- **5 W-* closures** — W-004, W-011, W-012, W-015, W-016 + +**Outcome:** test debt is **visible and ratcheted**; speed recovered; the audit-derived registry is operational. We can ship the next dozen PRs without flying blind. + +**Follow-up queued (task #7):** small PR migrating `lane-shas.yml` from plain `pip` to `uv` to match the gate's precedent. Per [[feedback-use-uv-consistently]]. + +--- + +## Phase 2 — Operator decisions (small, you-decide, fast) + +Five short calls clear five `W-*` entries with minimal-to-no engineering. + +| Item | Decision needed | Resolution paths | +|------|-----------------|------------------| +| **W-006** | Pack readback rules: wire or delete? | (a) wire `packs//readback_rules.py` into surface generation per original intent; (b) accept that `generate/realizer.py` superseded the design and DELETE the dormant modules | +| **W-010** | L4 recognition vocab: intentional token-level or wire L3? | (a) document as intentional in ADR-0143 amendment (no code change); (b) wire `VocabManifold` consumption into `derive_recognizer()` via new ADR (larger) | +| **W-013** | `core/cognition/explain.py`: wire / relocate / delete | (a) wire into `core chat` for "explain this turn" REPL; (b) move to `evals/` if offline audit tool; (c) delete | +| **W-014** | `core/cognition/provenance.py`: wire / relocate / delete | (a) wire into live turn result for per-turn provenance; (b) relocate to `evals/`; (c) leave as-is, document as evals-only | +| **W-019** | `teaching/from_miner.py` + `from_curriculum.py`: CLI / runtime / library? | (a) wire CLI: `core teaching propose --from-miner ` (smallest fix); (b) wire into W-017 promotion pipeline (later, larger); (c) leave as test-live library | + +Each ~30 min thought + small PR. Mostly cleanup-as-you-find calls per [[feedback-cleanup-as-you-find]]. + +--- + +## Phase 3 — Test debt paydown + +Goal: shrink `QUARANTINE` from 49 → 0. One PR per cluster. + +### Sequence (easiest → hardest) + +1. **Cluster A** (4 tests, ADR ledger drift) — one-token extensions, same shape as W-002 (#240). **~1 hour.** +2. **Cluster D** (2 tests, CLI/API drift) — quick reads, quick fixes. **~30 min.** +3. **Cluster E** (1 test, xdist incompatibility) — rewrite to measure only self-allocations OR mark `@pytest.mark.xdist_group("serial")`. **~30 min.** +4. **Cluster B** (15 tests, surface decoration drift) — mechanical: update assertions to use `in` containment or accept the `pack-grounded ()` suffix. **~3-4 hours.** +5. **Cluster C** (27 tests, lane metric drift) — **the dangerous one.** Each test needs investigation: stale threshold (re-pin) vs real quality regression (fix the regression). Some will surface honest engine-quality problems we've been blind to. Expect 1-3 PRs worth of work to surface what's stale vs what's broken. + +By end of Phase 3, the gate enforces zero quarantine and every contract pinned in tests is a contract the engine actually holds. + +--- + +## Phase 4 — THE load-bearing decision: L10 runtime model + +This is the architectural commitment that gates W-003, W-007, W-009, W-017, W-018. +**Until L10 commits, the rest of the audit registry can't sequence.** + +Scope already landed (#236). What's left: + +- **Spike**: prove a long-lived process can hold field state + vault + session continuity without leak/drift across 24+ hours of turns +- **ADR (or cluster)**: pick process shape A/B/C, state partitioning, reboot recovery model, HITL async substrate +- **First implementation**: minimal long-lived runtime that survives the spike's stress test + +This is the **riskiest single decision in the whole plan**. Wrong shape locks in wrong architecture for everything downstream. + +**Discipline note** (per [[feedback-scope-time-is-cheap]]): treat L10's spike with the same rigor as the substrate-liveness audit — multi-resolution scope, agent-cross-checked, before writing any committing ADR. The audit just demonstrated what "do it right" looks like; L10 deserves the same. + +**Open offer**: draft a v1 of the L10 *scope-of-scope* (what questions the spike must answer) once Phase 1 lands. Operator's call to take. + +--- + +## Phase 5 — The L10 cascade (after Phase 4) + +Each is small **once L10 commits process shape**: + +| Item | What lands | Depends on | +|------|-----------|------------| +| **W-003** | `VaultPromotionPolicy` wired into runtime promotion path | L10 process shape | +| **Recognizer-storage ADR** | Where recognizers live across turns/sessions/reboots | L10 + W-003 | +| **W-007** | `DerivedRecognizer` integration into `CognitiveTurnPipeline` | Recognizer-storage ADR | +| **W-009** | HITL async queue (operator reviews while engine serves turns) | L10 + drop-off sibling ADR | +| **W-017** | Automated T1/T2 → T3 promotion | W-009 (W-016 portion already satisfied by #257) | +| **W-018** | Autonomous contemplation loop | L10 | + +When all six land + recognizer-storage answers, **the audit registry is all CLOSED.** The design executes. + +--- + +## Phase 6 — The "decodes, not generates" milestone + +This is where smashing expectations actually lives. With Phase 5 done, CORE demonstrably: + +- **Runs forever** (Phase 4) — capability accumulates across invocations, not per-turn +- **Recognizes deterministically** (W-007) — turn loop dispatches through anti-unifier +- **Promotes evidence automatically** (W-017) — T1 vault → T3 reviewed corpus on HITL ratification +- **Learns from refusals** (W-011 + recognizer wiring) — typed refusals become curriculum candidates +- **Contemplates autonomously** (W-018) — between-turn slack becomes hypothesis exploration + +That set of behaviors is **not what a transformer-based chatbot can do**. That's the architectural distinctiveness moment. Patent territory. + +--- + +## Phase 7 — Validation & projection + +After Phase 6: + +- **Patent filings** — operator deferred until cleanup complete; Phase 6 is the cue +- **Rust backend parity** — CLAUDE.md work-sequencing #5, was waiting on Python semantics being locked by tests; Phase 3 locks them +- **Curriculum expansion** — CLAUDE.md #6; now safe because eval/replay/calibration are deterministic +- **Long-context benchmarks** vs transformer baselines — NIAH probe exists at `evals/long_context_cost/`; concrete "we recall at N=100k where they hallucinate" demonstrations + +--- + +## What "smashing expectations" honestly means + +Two flavors, worth distinguishing: + +1. **Architectural distinctiveness** (Phase 6) — CORE does things transformers structurally can't. Replay-equivalent, identity-preserving, audit-trailed, exact-recall. **This is achievable on the path above.** +2. **Benchmark wins** (GSM8k, etc.) — possible but requires Phase 7's curriculum expansion AND a real claim on what we're testing. Cluster C in Phase 3 will tell us whether current grounding accuracy is actually competitive or whether we've been measuring with stale thresholds. + +**The thesis (decodes, not generates) makes (1) the legitimate target. (2) is downstream validation, not the goal. Aim at (1); (2) follows.** + +--- + +## The one critical risk + +**L10 is the single point of failure.** If we commit to the wrong process shape, every Phase 5 item gets re-done. Treat L10's spike with the same rigor as the substrate-liveness audit — multi-resolution scope, agent-cross-checked, before writing any committing ADR. + +--- + +## Cross-references + +- [substrate-liveness-ratchet](audit/substrate-liveness-ratchet.md) — wiring debt registry (v5) +- [substrate-liveness-registry](audit/substrate-liveness-registry.md) — per-layer audit evidence +- [test-debt-quarantine](test-debt-quarantine.md) — pytest QUARANTINE registry (49 tests, 5 clusters) +- [L10-runtime-model-scope](decisions/L10-runtime-model-scope.md) — gates W-003, W-007, W-009 +- [recognizer-storage-scope](decisions/recognizer-storage-scope.md) — gates W-007 +- [substrate-liveness-audit-scope](decisions/substrate-liveness-audit-scope.md) — defines audit shape + +## Memory cross-references + +- [[thesis-decoding-not-generating]] — load-bearing project thesis +- [[feedback-adr-cross-reference-discipline]] — grep ALL existing ADRs before any new scope +- [[feedback-cleanup-as-you-find]] — applies to Phase 2 W-006/W-013/W-014 +- [[feedback-scope-time-is-cheap]] — applies to Phase 4 L10 +- [[feedback-address-critiques-dont-waive]] — if a critique surfaces in any phase, fix don't waive +- [[feedback-parallel-agent-worktrees]] — for any parallel-agent dispatch +- [[feedback-use-uv-consistently]] — `uv venv` / `uv pip install` / `uv run`, never `python3 -m venv` / `--break-system-packages` +- [[project-engine-identity-candidate]] — DNA-analog concept SHELVED; un-shelve trigger = L10 sub-question 3 (cross-reboot identity verification) + +## Tonight's lessons (in case they surface tomorrow) + +1. **Lane subprocess timeout (300s) was too tight** for any lane that takes more than ~5 min on CI. Bumped to 900s in #261's third commit. Future lane additions should consider their CI-runtime cost. +2. **pytest-xdist `-n 4` surfaces parallel-execution incompatibilities.** First example was `test_articulation_bench` (memory-RSS measurement). Cluster E is the place to track these; consider whether the gate workflow should drop to `-n 2` if more parallel-flake surfaces. +3. **Wrong-branch PR pattern** continues to hit subagents. Seen earlier with #256 (Sonnet's W-016 on wrong branch) and earlier with Gemini's L2/L3/L5/L7. Every parallel-agent brief should emphasize the rebase-onto-current-main step before PR creation. +4. **`/tmp/`** was a workaround for PEP 668 — sourced [[feedback-use-uv-consistently]]. The CI uses uv (the gate workflow already sets the precedent); the existing lane-shas workflow still uses pip and gets migrated in task #7.