core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/master-plan-post-substrate-audit.md

14 KiB
Raw Permalink Blame History

Master plan — post-substrate-audit

Authored: 2026-05-24 (end-of-day, PST) — call it a night, fresh AM. Author intent: carry the strategic plan + active-PR snapshot across the session boundary so morning-self picks up cleanly.

2026-05-26 amendment — Phases 1–5 substantially complete. Math architecture corridor closed end-to-end (Phase A → B → C → D + operator ratification, ADR-0163); first measurable GSM8K lift produced (correct: 0 → 3, wrong: 0 unchanged). Workbench surface operational end-to-end (W-026 API + W-027 shell + W-028 chat surface). HITL queue read-only CLI shipped (ADR-0161 Step 1). See SESSION-2026-05-26-corridor-closure.md for the day's full ledger (15 merges + 1 issue). Sections below remain as the historical 2026-05-24 plan; next-moves are in the session recap, not here.

2026-07-03 current-state note — the historical pytest QUARANTINE registry described below has been retired. conftest.py now has QUARANTINE = frozenset(), and docs/test-debt-quarantine.md is the current source for quarantine policy.

Three guiding rules

  1. "Fully wired" = every W-* entry in substrate-liveness-ratchet CLOSED. When the registry is empty, the design executes as written.
  2. Thesis check on every move: per thesis-decoding-not-generating — does this teach the engine to find better, or just store another found thing? Only the former clears the bar.
  3. L10 is the load-bearing decision. It gates W-003, W-007, W-009, W-017, W-018, and the recognizer-storage ADR. Wrong shape locks in wrong architecture for the rest of the cascade.

End-of-day snapshot (2026-05-24)

Active in flight

PR Branch State Blocker
#261 (B speedup) chore/test-speed-b-extract-lane-runners CI rerunning after timeout bump + test_lane_sha_verifier fix Wait for verify pinned lane SHAs to go green
(unopened) Gate ci/full-pytest-gate-quarantine-markers (pushed, no PR) Locally verified: 6657 pass / 0 unexpected fail / 76 skip / 49 quarantine Open PR after #261 merges (otherwise gate's CI hits pre-fix lane-shas state)

Closed today (5 W-* + 5 PRs)

  • W-004 — vault E2 re-thaw (PR #251)
  • W-015 — _slerp_toward → rotor-geodesic anchor pull (PR #255)
  • W-016 — vault probe in discovery loop (PR #257)
  • W-011 — recognition refusal propagation (PR #258, paired)
  • W-012 — InnerLoopExhaustion caught in ChatRuntime (PR #258, paired)
  • Ratchet v5 (PR #260) — registry updated

Historical quarantine registry (49 tests; now retired)

Cluster A (4) — ADR ledger drift, one-token extensions, shape of W-002 Cluster B (15) — surface decoration drift (pack-grounded (<pack>) suffix) Cluster C (27) — lane/runner metric drift (some are real regressions) Cluster D (2) — CLI/internal API drift Cluster E (1) — pytest-xdist parallel-execution incompatibility (test_articulation_bench::test_footprint_emits_samples_and_bounds)

All confirmed pre-existing via bisect against c1a1b7a (pre-W-*-work commit). Today's 5 closures introduced zero new failures. Cluster E is the lone xdist-induced regression — caused by gate's -n 4 choice, not by today's substantive work.

Resumption order in AM

  1. First: check PR #261's CI re-run status. If green, merge. If red, debug from CI logs.
  2. Second: rebase ci/full-pytest-gate-quarantine-markers onto post-#261 main; open as PR.
  3. Third: read this plan top-to-bottom to re-load context.
  4. Fourth: Phase 2 work — five operator decisions queued (W-006, W-010, W-013, W-014, W-019). Each ~30 min.

Phase 1 — Hygiene & honesty (today, in flight)

  • B PR #261 — extract math_teaching_corpus lane from pytest into CI lane SHAs (−9 min suite time, ~30% reduction)
  • Gate PR (pending #261) — historical conftest.py QUARANTINE registry of 49 known-failing tests + new full-pytest.yml workflow running pytest -m "not quarantine" -n 4
  • 5 W- closures* — W-004, W-011, W-012, W-015, W-016

Outcome: test debt is visible and ratcheted; speed recovered; the audit-derived registry is operational. We can ship the next dozen PRs without flying blind.

Follow-up queued (task #7): small PR migrating lane-shas.yml from plain pip to uv to match the gate's precedent. Per feedback-use-uv-consistently.

Phase 2 — Operator decisions (small, you-decide, fast)

Five short calls clear five W-* entries with minimal-to-no engineering.

Item Decision needed Resolution paths
W-006 Pack readback rules: wire or delete? (a) wire packs/<lang>/readback_rules.py into surface generation per original intent; (b) accept that generate/realizer.py superseded the design and DELETE the dormant modules
W-010 L4 recognition vocab: intentional token-level or wire L3? (a) document as intentional in ADR-0143 amendment (no code change); (b) wire VocabManifold consumption into derive_recognizer() via new ADR (larger)
W-013 core/cognition/explain.py: wire / relocate / delete (a) wire into core chat for "explain this turn" REPL; (b) move to evals/ if offline audit tool; (c) delete
W-014 core/cognition/provenance.py: wire / relocate / delete (a) wire into live turn result for per-turn provenance; (b) relocate to evals/; (c) leave as-is, document as evals-only
W-019 teaching/from_miner.py + from_curriculum.py: CLI / runtime / library? (a) wire CLI: core teaching propose --from-miner <dir> (smallest fix); (b) wire into W-017 promotion pipeline (later, larger); (c) leave as test-live library

Each ~30 min thought + small PR. Mostly cleanup-as-you-find calls per feedback-cleanup-as-you-find.

Phase 3 — Test debt paydown

Historical goal: shrink QUARANTINE from 49 -> 0. This is now complete; conftest.py has an empty registry. The historical cluster sequence remains below for audit archaeology.

Sequence (easiest → hardest)

  1. Cluster A (4 tests, ADR ledger drift) — one-token extensions, same shape as W-002 (#240). ~1 hour.
  2. Cluster D (2 tests, CLI/API drift) — quick reads, quick fixes. ~30 min.
  3. Cluster E (1 test, xdist incompatibility) — rewrite to measure only self-allocations OR mark @pytest.mark.xdist_group("serial"). ~30 min.
  4. Cluster B (15 tests, surface decoration drift) — mechanical: update assertions to use in containment or accept the pack-grounded (<pack>) suffix. ~3-4 hours.
  5. Cluster C (27 tests, lane metric drift) — the dangerous one. Each test needs investigation: stale threshold (re-pin) vs real quality regression (fix the regression). Some will surface honest engine-quality problems we've been blind to. Expect 1-3 PRs worth of work to surface what's stale vs what's broken.

By end of Phase 3, the gate enforces zero quarantine and every contract pinned in tests is a contract the engine actually holds.

Phase 4 — THE load-bearing decision: L10 runtime model

This is the architectural commitment that gates W-003, W-007, W-009, W-017, W-018. Until L10 commits, the rest of the audit registry can't sequence.

Scope already landed (#236). What's left:

  • Spike: prove a long-lived process can hold field state + vault + session continuity without leak/drift across 24+ hours of turns
  • ADR (or cluster): pick process shape A/B/C, state partitioning, reboot recovery model, HITL async substrate
  • First implementation: minimal long-lived runtime that survives the spike's stress test

This is the riskiest single decision in the whole plan. Wrong shape locks in wrong architecture for everything downstream.

Discipline note (per feedback-scope-time-is-cheap): treat L10's spike with the same rigor as the substrate-liveness audit — multi-resolution scope, agent-cross-checked, before writing any committing ADR. The audit just demonstrated what "do it right" looks like; L10 deserves the same.

Open offer: draft a v1 of the L10 scope-of-scope (what questions the spike must answer) once Phase 1 lands. Operator's call to take.

Phase 5 — The L10 cascade (after Phase 4)

Each is small once L10 commits process shape:

Item What lands Depends on
W-003 VaultPromotionPolicy wired into runtime promotion path L10 process shape
Recognizer-storage ADR Where recognizers live across turns/sessions/reboots L10 + W-003
W-007 DerivedRecognizer integration into CognitiveTurnPipeline Recognizer-storage ADR
W-009 HITL async queue (operator reviews while engine serves turns) L10 + drop-off sibling ADR
W-017 Automated T1/T2 → T3 promotion W-009 (W-016 portion already satisfied by #257)
W-018 Autonomous contemplation loop L10

When all six land + recognizer-storage answers, the audit registry is all CLOSED. The design executes.

Phase 6 — The "decodes, not generates" milestone

This is where smashing expectations actually lives. With Phase 5 done, CORE demonstrably:

  • Runs forever (Phase 4) — capability accumulates across invocations, not per-turn
  • Recognizes deterministically (W-007) — turn loop dispatches through anti-unifier
  • Promotes evidence automatically (W-017) — T1 vault → T3 reviewed corpus on HITL ratification
  • Learns from refusals (W-011 + recognizer wiring) — typed refusals become curriculum candidates
  • Contemplates autonomously (W-018) — between-turn slack becomes hypothesis exploration

That set of behaviors is not what a transformer-based chatbot can do. That's the architectural distinctiveness moment. Patent territory.

Phase 7 — Validation & projection

After Phase 6:

  • Patent filings — operator deferred until cleanup complete; Phase 6 is the cue
  • Rust backend parity — CLAUDE.md work-sequencing #5, was waiting on Python semantics being locked by tests; Phase 3 locks them
  • Curriculum expansion — CLAUDE.md #6; now safe because eval/replay/calibration are deterministic
  • Long-context benchmarks vs transformer baselines — NIAH probe exists at evals/long_context_cost/; concrete "we recall at N=100k where they hallucinate" demonstrations

What "smashing expectations" honestly means

Two flavors, worth distinguishing:

  1. Architectural distinctiveness (Phase 6) — CORE does things transformers structurally can't. Replay-equivalent, identity-preserving, audit-trailed, exact-recall. This is achievable on the path above.
  2. Benchmark wins (GSM8k, etc.) — possible but requires Phase 7's curriculum expansion AND a real claim on what we're testing. Cluster C in Phase 3 will tell us whether current grounding accuracy is actually competitive or whether we've been measuring with stale thresholds.

The thesis (decodes, not generates) makes (1) the legitimate target. (2) is downstream validation, not the goal. Aim at (1); (2) follows.

The one critical risk

L10 is the single point of failure. If we commit to the wrong process shape, every Phase 5 item gets re-done. Treat L10's spike with the same rigor as the substrate-liveness audit — multi-resolution scope, agent-cross-checked, before writing any committing ADR.

Cross-references

Memory cross-references

Tonight's lessons (in case they surface tomorrow)

  1. Lane subprocess timeout (300s) was too tight for any lane that takes more than ~5 min on CI. Bumped to 900s in #261's third commit. Future lane additions should consider their CI-runtime cost.
  2. pytest-xdist -n 4 surfaces parallel-execution incompatibilities. First example was test_articulation_bench (memory-RSS measurement). Cluster E is the place to track these; consider whether the gate workflow should drop to -n 2 if more parallel-flake surfaces.
  3. Wrong-branch PR pattern continues to hit subagents. Seen earlier with #256 (Sonnet's W-016 on wrong branch) and earlier with Gemini's L2/L3/L5/L7. Every parallel-agent brief should emphasize the rebase-onto-current-main step before PR creation.
  4. /tmp/<scratch-venv> was a workaround for PEP 668 — sourced feedback-use-uv-consistently. The CI uses uv (the gate workflow already sets the precedent); the existing lane-shas workflow still uses pip and gets migrated in task #7.