310aed9ff0
* docs: consolidate governance anchors and clean up test registries * refactor(cli): decompose cli into dedicated modules * test: fix broken test baselines and formatting * docs: add domain boundary READMEs for governance anchors * test: update baseline for determination lane * test: fix capability_pass expectation * test: fix CORE_SHOWCASE_SKIP_BUDGET enforcement * chore: cleanup CLI extraction and unreachable code
219 lines
14 KiB
Markdown
219 lines
14 KiB
Markdown
# Master plan — post-substrate-audit
|
||
|
||
**Authored:** 2026-05-24 (end-of-day, PST) — call it a night, fresh AM.
|
||
**Author intent:** carry the strategic plan + active-PR snapshot across the
|
||
session boundary so morning-self picks up cleanly.
|
||
|
||
> **2026-05-26 amendment** — Phases 1–5 substantially complete. Math
|
||
> architecture corridor closed end-to-end (Phase A → B → C → D + operator
|
||
> ratification, ADR-0163); first measurable GSM8K lift produced
|
||
> (`correct: 0 → 3, wrong: 0 unchanged`). Workbench surface operational
|
||
> end-to-end (W-026 API + W-027 shell + W-028 chat surface). HITL queue
|
||
> read-only CLI shipped (ADR-0161 Step 1). See
|
||
> [SESSION-2026-05-26-corridor-closure.md](sessions/SESSION-2026-05-26-corridor-closure.md)
|
||
> for the day's full ledger (15 merges + 1 issue). Sections below remain
|
||
> as the historical 2026-05-24 plan; **next-moves are in the session
|
||
> recap, not here**.
|
||
>
|
||
> **2026-07-03 current-state note** — the historical pytest `QUARANTINE`
|
||
> registry described below has been retired. `conftest.py` now has
|
||
> `QUARANTINE = frozenset()`, and `docs/test-debt-quarantine.md` is the
|
||
> current source for quarantine policy.
|
||
|
||
---
|
||
|
||
## Three guiding rules
|
||
|
||
1. **"Fully wired"** = every `W-*` entry in [substrate-liveness-ratchet](audit/substrate-liveness-ratchet.md) CLOSED. When the registry is empty, the design executes as written.
|
||
2. **Thesis check on every move**: per [[thesis-decoding-not-generating]] — does this teach the engine to find better, or just store another found thing? Only the former clears the bar.
|
||
3. **L10 is the load-bearing decision.** It gates W-003, W-007, W-009, W-017, W-018, and the recognizer-storage ADR. Wrong shape locks in wrong architecture for the rest of the cascade.
|
||
|
||
---
|
||
|
||
## End-of-day snapshot (2026-05-24)
|
||
|
||
### Active in flight
|
||
|
||
| PR | Branch | State | Blocker |
|
||
|----|--------|-------|---------|
|
||
| **#261** (B speedup) | `chore/test-speed-b-extract-lane-runners` | CI rerunning after timeout bump + test_lane_sha_verifier fix | Wait for `verify pinned lane SHAs` to go green |
|
||
| **(unopened)** Gate | `ci/full-pytest-gate-quarantine-markers` (pushed, no PR) | Locally verified: 6657 pass / 0 unexpected fail / 76 skip / 49 quarantine | Open PR **after #261 merges** (otherwise gate's CI hits pre-fix lane-shas state) |
|
||
|
||
### Closed today (5 W-* + 5 PRs)
|
||
|
||
- **W-004** — vault E2 re-thaw (PR #251)
|
||
- **W-015** — `_slerp_toward` → rotor-geodesic anchor pull (PR #255)
|
||
- **W-016** — vault probe in discovery loop (PR #257)
|
||
- **W-011** — recognition refusal propagation (PR #258, paired)
|
||
- **W-012** — `InnerLoopExhaustion` caught in ChatRuntime (PR #258, paired)
|
||
- Ratchet v5 (PR #260) — registry updated
|
||
|
||
### Historical quarantine registry (49 tests; now retired)
|
||
|
||
Cluster A (4) — ADR ledger drift, one-token extensions, shape of W-002
|
||
Cluster B (15) — surface decoration drift (`pack-grounded (<pack>)` suffix)
|
||
Cluster C (27) — lane/runner metric drift (some are real regressions)
|
||
Cluster D (2) — CLI/internal API drift
|
||
Cluster E (1) — pytest-xdist parallel-execution incompatibility (`test_articulation_bench::test_footprint_emits_samples_and_bounds`)
|
||
|
||
All confirmed pre-existing via bisect against `c1a1b7a` (pre-W-*-work commit). Today's 5 closures introduced **zero** new failures. Cluster E is the lone xdist-induced regression — caused by gate's `-n 4` choice, not by today's substantive work.
|
||
|
||
### Resumption order in AM
|
||
|
||
1. **First**: check PR #261's CI re-run status. If green, merge. If red, debug from CI logs.
|
||
2. **Second**: rebase `ci/full-pytest-gate-quarantine-markers` onto post-#261 main; open as PR.
|
||
3. **Third**: read this plan top-to-bottom to re-load context.
|
||
4. **Fourth**: Phase 2 work — five operator decisions queued (W-006, W-010, W-013, W-014, W-019). Each ~30 min.
|
||
|
||
---
|
||
|
||
## Phase 1 — Hygiene & honesty (today, in flight)
|
||
|
||
- **B PR #261** — extract `math_teaching_corpus` lane from pytest into CI lane SHAs (−9 min suite time, ~30% reduction)
|
||
- **Gate PR** (pending #261) — historical `conftest.py` QUARANTINE registry of 49 known-failing tests + new `full-pytest.yml` workflow running `pytest -m "not quarantine" -n 4`
|
||
- **5 W-* closures** — W-004, W-011, W-012, W-015, W-016
|
||
|
||
**Outcome:** test debt is **visible and ratcheted**; speed recovered; the audit-derived registry is operational. We can ship the next dozen PRs without flying blind.
|
||
|
||
**Follow-up queued (task #7):** small PR migrating `lane-shas.yml` from plain `pip` to `uv` to match the gate's precedent. Per [[feedback-use-uv-consistently]].
|
||
|
||
---
|
||
|
||
## Phase 2 — Operator decisions (small, you-decide, fast)
|
||
|
||
Five short calls clear five `W-*` entries with minimal-to-no engineering.
|
||
|
||
| Item | Decision needed | Resolution paths |
|
||
|------|-----------------|------------------|
|
||
| **W-006** | Pack readback rules: wire or delete? | (a) wire `packs/<lang>/readback_rules.py` into surface generation per original intent; (b) accept that `generate/realizer.py` superseded the design and DELETE the dormant modules |
|
||
| **W-010** | L4 recognition vocab: intentional token-level or wire L3? | (a) document as intentional in ADR-0143 amendment (no code change); (b) wire `VocabManifold` consumption into `derive_recognizer()` via new ADR (larger) |
|
||
| **W-013** | `core/cognition/explain.py`: wire / relocate / delete | (a) wire into `core chat` for "explain this turn" REPL; (b) move to `evals/` if offline audit tool; (c) delete |
|
||
| **W-014** | `core/cognition/provenance.py`: wire / relocate / delete | (a) wire into live turn result for per-turn provenance; (b) relocate to `evals/`; (c) leave as-is, document as evals-only |
|
||
| **W-019** | `teaching/from_miner.py` + `from_curriculum.py`: CLI / runtime / library? | (a) wire CLI: `core teaching propose --from-miner <dir>` (smallest fix); (b) wire into W-017 promotion pipeline (later, larger); (c) leave as test-live library |
|
||
|
||
Each ~30 min thought + small PR. Mostly cleanup-as-you-find calls per [[feedback-cleanup-as-you-find]].
|
||
|
||
---
|
||
|
||
## Phase 3 — Test debt paydown
|
||
|
||
Historical goal: shrink `QUARANTINE` from 49 -> 0. This is now complete;
|
||
`conftest.py` has an empty registry. The historical cluster sequence remains
|
||
below for audit archaeology.
|
||
|
||
### Sequence (easiest → hardest)
|
||
|
||
1. **Cluster A** (4 tests, ADR ledger drift) — one-token extensions, same shape as W-002 (#240). **~1 hour.**
|
||
2. **Cluster D** (2 tests, CLI/API drift) — quick reads, quick fixes. **~30 min.**
|
||
3. **Cluster E** (1 test, xdist incompatibility) — rewrite to measure only self-allocations OR mark `@pytest.mark.xdist_group("serial")`. **~30 min.**
|
||
4. **Cluster B** (15 tests, surface decoration drift) — mechanical: update assertions to use `in` containment or accept the `pack-grounded (<pack>)` suffix. **~3-4 hours.**
|
||
5. **Cluster C** (27 tests, lane metric drift) — **the dangerous one.** Each test needs investigation: stale threshold (re-pin) vs real quality regression (fix the regression). Some will surface honest engine-quality problems we've been blind to. Expect 1-3 PRs worth of work to surface what's stale vs what's broken.
|
||
|
||
By end of Phase 3, the gate enforces zero quarantine and every contract pinned in tests is a contract the engine actually holds.
|
||
|
||
---
|
||
|
||
## Phase 4 — THE load-bearing decision: L10 runtime model
|
||
|
||
This is the architectural commitment that gates W-003, W-007, W-009, W-017, W-018.
|
||
**Until L10 commits, the rest of the audit registry can't sequence.**
|
||
|
||
Scope already landed (#236). What's left:
|
||
|
||
- **Spike**: prove a long-lived process can hold field state + vault + session continuity without leak/drift across 24+ hours of turns
|
||
- **ADR (or cluster)**: pick process shape A/B/C, state partitioning, reboot recovery model, HITL async substrate
|
||
- **First implementation**: minimal long-lived runtime that survives the spike's stress test
|
||
|
||
This is the **riskiest single decision in the whole plan**. Wrong shape locks in wrong architecture for everything downstream.
|
||
|
||
**Discipline note** (per [[feedback-scope-time-is-cheap]]): treat L10's spike with the same rigor as the substrate-liveness audit — multi-resolution scope, agent-cross-checked, before writing any committing ADR. The audit just demonstrated what "do it right" looks like; L10 deserves the same.
|
||
|
||
**Open offer**: draft a v1 of the L10 *scope-of-scope* (what questions the spike must answer) once Phase 1 lands. Operator's call to take.
|
||
|
||
---
|
||
|
||
## Phase 5 — The L10 cascade (after Phase 4)
|
||
|
||
Each is small **once L10 commits process shape**:
|
||
|
||
| Item | What lands | Depends on |
|
||
|------|-----------|------------|
|
||
| **W-003** | `VaultPromotionPolicy` wired into runtime promotion path | L10 process shape |
|
||
| **Recognizer-storage ADR** | Where recognizers live across turns/sessions/reboots | L10 + W-003 |
|
||
| **W-007** | `DerivedRecognizer` integration into `CognitiveTurnPipeline` | Recognizer-storage ADR |
|
||
| **W-009** | HITL async queue (operator reviews while engine serves turns) | L10 + drop-off sibling ADR |
|
||
| **W-017** | Automated T1/T2 → T3 promotion | W-009 (W-016 portion already satisfied by #257) |
|
||
| **W-018** | Autonomous contemplation loop | L10 |
|
||
|
||
When all six land + recognizer-storage answers, **the audit registry is all CLOSED.** The design executes.
|
||
|
||
---
|
||
|
||
## Phase 6 — The "decodes, not generates" milestone
|
||
|
||
This is where smashing expectations actually lives. With Phase 5 done, CORE demonstrably:
|
||
|
||
- **Runs forever** (Phase 4) — capability accumulates across invocations, not per-turn
|
||
- **Recognizes deterministically** (W-007) — turn loop dispatches through anti-unifier
|
||
- **Promotes evidence automatically** (W-017) — T1 vault → T3 reviewed corpus on HITL ratification
|
||
- **Learns from refusals** (W-011 + recognizer wiring) — typed refusals become curriculum candidates
|
||
- **Contemplates autonomously** (W-018) — between-turn slack becomes hypothesis exploration
|
||
|
||
That set of behaviors is **not what a transformer-based chatbot can do**. That's the architectural distinctiveness moment. Patent territory.
|
||
|
||
---
|
||
|
||
## Phase 7 — Validation & projection
|
||
|
||
After Phase 6:
|
||
|
||
- **Patent filings** — operator deferred until cleanup complete; Phase 6 is the cue
|
||
- **Rust backend parity** — CLAUDE.md work-sequencing #5, was waiting on Python semantics being locked by tests; Phase 3 locks them
|
||
- **Curriculum expansion** — CLAUDE.md #6; now safe because eval/replay/calibration are deterministic
|
||
- **Long-context benchmarks** vs transformer baselines — NIAH probe exists at `evals/long_context_cost/`; concrete "we recall at N=100k where they hallucinate" demonstrations
|
||
|
||
---
|
||
|
||
## What "smashing expectations" honestly means
|
||
|
||
Two flavors, worth distinguishing:
|
||
|
||
1. **Architectural distinctiveness** (Phase 6) — CORE does things transformers structurally can't. Replay-equivalent, identity-preserving, audit-trailed, exact-recall. **This is achievable on the path above.**
|
||
2. **Benchmark wins** (GSM8k, etc.) — possible but requires Phase 7's curriculum expansion AND a real claim on what we're testing. Cluster C in Phase 3 will tell us whether current grounding accuracy is actually competitive or whether we've been measuring with stale thresholds.
|
||
|
||
**The thesis (decodes, not generates) makes (1) the legitimate target. (2) is downstream validation, not the goal. Aim at (1); (2) follows.**
|
||
|
||
---
|
||
|
||
## The one critical risk
|
||
|
||
**L10 is the single point of failure.** If we commit to the wrong process shape, every Phase 5 item gets re-done. Treat L10's spike with the same rigor as the substrate-liveness audit — multi-resolution scope, agent-cross-checked, before writing any committing ADR.
|
||
|
||
---
|
||
|
||
## Cross-references
|
||
|
||
- [substrate-liveness-ratchet](audit/substrate-liveness-ratchet.md) — wiring debt registry (v5)
|
||
- [substrate-liveness-registry](audit/substrate-liveness-registry.md) — per-layer audit evidence
|
||
- [test-debt-quarantine](test-debt-quarantine.md) — current pytest quarantine policy and historical cluster notes
|
||
- [L10-runtime-model-scope](adr/L10-runtime-model-scope.md) — gates W-003, W-007, W-009
|
||
- [recognizer-storage-scope](adr/recognizer-storage-scope.md) — gates W-007
|
||
- [substrate-liveness-audit-scope](adr/substrate-liveness-audit-scope.md) — defines audit shape
|
||
|
||
## Memory cross-references
|
||
|
||
- [[thesis-decoding-not-generating]] — load-bearing project thesis
|
||
- [[feedback-adr-cross-reference-discipline]] — grep ALL existing ADRs before any new scope
|
||
- [[feedback-cleanup-as-you-find]] — applies to Phase 2 W-006/W-013/W-014
|
||
- [[feedback-scope-time-is-cheap]] — applies to Phase 4 L10
|
||
- [[feedback-address-critiques-dont-waive]] — if a critique surfaces in any phase, fix don't waive
|
||
- [[feedback-parallel-agent-worktrees]] — for any parallel-agent dispatch
|
||
- [[feedback-use-uv-consistently]] — `uv venv` / `uv pip install` / `uv run`, never `python3 -m venv` / `--break-system-packages`
|
||
- [[project-engine-identity-candidate]] — DNA-analog concept SHELVED; un-shelve trigger = L10 sub-question 3 (cross-reboot identity verification)
|
||
|
||
## Tonight's lessons (in case they surface tomorrow)
|
||
|
||
1. **Lane subprocess timeout (300s) was too tight** for any lane that takes more than ~5 min on CI. Bumped to 900s in #261's third commit. Future lane additions should consider their CI-runtime cost.
|
||
2. **pytest-xdist `-n 4` surfaces parallel-execution incompatibilities.** First example was `test_articulation_bench` (memory-RSS measurement). Cluster E is the place to track these; consider whether the gate workflow should drop to `-n 2` if more parallel-flake surfaces.
|
||
3. **Wrong-branch PR pattern** continues to hit subagents. Seen earlier with #256 (Sonnet's W-016 on wrong branch) and earlier with Gemini's L2/L3/L5/L7. Every parallel-agent brief should emphasize the rebase-onto-current-main step before PR creation.
|
||
4. **`/tmp/<scratch-venv>`** was a workaround for PEP 668 — sourced [[feedback-use-uv-consistently]]. The CI uses uv (the gate workflow already sets the precedent); the existing lane-shas workflow still uses pip and gets migrated in task #7.
|