diff --git a/docs/audit/L10-L11-lookback-2026-06-05.md b/docs/audit/L10-L11-lookback-2026-06-05.md new file mode 100644 index 00000000..c9cdbc63 --- /dev/null +++ b/docs/audit/L10-L11-lookback-2026-06-05.md @@ -0,0 +1,256 @@ +# L10/L11 Lived-Spine Lookback Review (#567-#573) + +Date: 2026-06-05 +Branch: `audit/l10-l11-lookback` +Base reviewed: `origin/main` at `ea503f6d5177` +Scope: PRs #567-#573, the L10/L11 lived-spine stack. + +## Executive Finding + +The stack landed the intended Shape B+ resume spine: bit-exact field/vault/session +serialization, schema-v2 state loading, reboot-transparent L10 traces, EngineIdentity +lineage stamping, and proposal-only idle learning. The strongest wrong=0 hazards +requested in the brief are solid: `idle_tick()` does not ratify or bypass HITL, +vault restore does not reproject, and persistence restore adds no hidden field +repair at a forbidden site. + +One live crash-consistency hazard remains: the new multi-file checkpoint treats +`manifest.json` as the final commit marker but loads `session_state.json` +unconditionally. A power loss after `session_state.json` is atomically replaced +and before `manifest.json` is replaced can pair an old manifest with newer lived +state on reboot. Fix before the next phase. + +## Hazards + +### H1 - Multi-file checkpoint can restore mixed-generation state + +Category: hazard + +Evidence: +- `ChatRuntime.checkpoint_engine_state()` writes recognizers, candidates, then + `session_state.json`, then `manifest.json` as the stated commit marker + (`chat/runtime.py:779-803`). +- `EngineStateStore.save_session_state()` explicitly says session state is saved + before the manifest so the manifest is the last durable act + (`engine_state/__init__.py:225-236`). +- On load, `_load_engine_state()` reads the manifest turn count, then restores + `session_state.json` whenever `persist_session_state=True` and the file exists; + there is no generation, digest, or turn-count check tying the loaded session + snapshot to the manifest (`chat/runtime.py:719-755`, + `engine_state/__init__.py:238-251`). +- The P4 tests cover an orphan temp file and `os.replace` failure for + `manifest.json`, but they only assert recovered `turn_count` from the manifest; + they do not simulate a newer committed `session_state.json` with an older + manifest (`tests/test_l10_continuity.py:204-238`). + +Impact: +A kill between `save_session_state()` and `save_manifest()` can next-boot into a +hybrid checkpoint: old manifest/turn count plus newer field/vault/referent state. +That violates the L10 P4 claim that recovery lands on a valid prior checkpoint. +It is not a versor-condition failure, but it is a lived-spine continuity and +deterministic replay hazard. + +Fix before next phase: +Make the checkpoint generation atomic at the composed-state level. Minimal repair: +stamp `session_state.json` with the same committed `turn_count`/generation and +load it only when it matches the manifest. Stronger repair: write a generation +directory or content-addressed bundle and have the manifest point to hashes for +recognizers, candidates, and session state. Add a P4 regression test that creates +old manifest plus newer session state and proves restore refuses/falls back to +the valid prior generation. + +## Drift + +### D1 - P5a recall precision did not land + +Category: drift + +The L10 continuity spike design specified P5 as the T-experience gate, including +P5a recall precision@k stability. The landed contract explicitly records P5a as +`not_covered` (`evals/l10_continuity/contract.md:33-38`), while P5b/P5c landed +as anchor/coherence catastrophe checks (`evals/l10_continuity/predicates.py:291-380`). +This is honest in the contract, but it is still a design-vs-implementation cut. + +Fix before next phase: +Do not claim T-experience or long-horizon recall stability from #567-#573 alone. +Add the held-out exact-CGA recall probe set and gate P5a before using the lived +spine as evidence of continuous experiencing memory. + +### D2 - EngineIdentity scope differs from the L10 scoping recommendation + +Category: drift + +The L10 scoping brief recommended an EngineIdentity hash over +`schema_version + canonical_bytes` and a lineage chain recording schema migration. +The shipped EngineIdentity is instead a content hash over ratified identity, +safety, ethics, register, anchor-lens packs plus code revision +(`core/engine_identity.py:1-20`, `core/engine_identity.py:85-127`). +The manifest stamps this identity and its parent (`engine_state/__init__.py:161-189`). + +This shipped model is internally coherent and tested, but it is not the same +identity primitive described in the scoping record. Lived experience remains in +Shape B+ `session_state.json`; the identity hash names the ratified substrate. + +Fix before next phase: +Amend the L10/L11 design record or ADR scope to say EngineIdentity deliberately +means ratified substrate identity, not checkpoint-content identity. If checkpoint +content is still desired, add a separate `checkpoint_identity`/digest and link it +from the manifest rather than overloading EngineIdentity. + +## Gaps + +### G1 - P4 is sampled at the manifest/orphan boundary, not arbitrary interruption + +Category: gap + +The design brief elevated power-loss-at-arbitrary-instruction as a primary driver. +The landed runner simulates an orphan `.manifest.json.*.tmp` and the tests patch +`os.replace` for manifest writes (`evals/l10_continuity/runner.py:107-119`, +`tests/test_l10_continuity.py:204-238`). That is useful, but it does not enumerate +or sample every write boundary in the multi-file checkpoint. + +Fix before next phase: +After H1 is fixed, extend P4 to cover crashes after each durable file in the +checkpoint sequence: recognizers, candidates, session state, and manifest. + +### G2 - Idle learning backpressure remains unresolved + +Category: gap + +The roadmap/scoping notes called out absent-reviewer saturation as a real operator +condition. The proposal path has a pending cap (`DEFAULT_PENDING_CAP = 256`) and +emits a queue-full report when saturated (`teaching/proposals.py:48`, +`teaching/proposals.py:551-586`). `idle_tick()` treats proposal refusals as +non-created proposals and continues (`chat/runtime.py:858-865`). This is safe for +wrong=0, but it means learning can stall silently from the runtime result shape: +`IdleTickResult` has no refused/capacity count (`chat/runtime.py:508-514`). + +Fix before next phase: +Expose refusal counts/reasons in `IdleTickResult` or idle telemetry so operators +can distinguish "no determined candidates" from "HITL queue full." + +## Solid + +### S1 - `idle_tick()` does not bypass HITL + +Category: solid + +`idle_tick()` contemplates pending candidates, calls `propose_from_candidate()`, +and never calls `accept_proposal()` or writes to the corpus (`chat/runtime.py:815-873`). +`propose_from_candidate()` first builds a proposal through `check_eligibility()`, +which rejects `polarity="undetermined"` (`teaching/proposals.py:165-181`), then +records a `created` event and replay evidence; acceptance remains a separate +operator path (`teaching/proposals.py:623-676`). Tests cover undetermined refusal +and determined-but-pending behavior (`tests/test_continuous_learning_idle.py:72-99`). + +Answer to the audit question: an undetermined candidate cannot reach `pending` +through `idle_tick()`. A determined candidate can reach `pending`, but not +`accepted`; that is the intended HITL boundary. + +### S2 - Vault restore does not reproject + +Category: solid + +`VaultStore.from_dict()` decodes persisted arrays, rebuilds `_exact_index`, leaves +`_matrix_cache` derived, and never calls `store()` or `reproject()` +(`vault/store.py:444-469`). The bright-line test asserts byte-identical restored +versors and exact recall (`tests/test_vaultstore_codec.py:41-84`). The existing +live `store()` path still auto-reprojects every N stores, but the restore path is +pure deserialize plus derived-index rebuild. + +Answer to the audit question: engine-state restore does not reproject a vault. + +### S3 - Persistence adds no forbidden field repair + +Category: solid + +The array codec is raw dtype/shape/base64 bytes; `FieldState.from_dict()` rebuilds +from decoded bytes via the constructor, which copies and validates shape/dtype but +does not normalize (`field/state.py:117-132`, `field/state.py:173-183`). +`SessionContext.restore()` composes component restore functions and explicitly +keeps vocab/persona outside session state (`session/context.py:373-405`). +No new persistence load path calls `unitize`, `normalize`, `null_project`, or a +field repair operator. + +Answer to the audit question: no new persistence restore path normalizes or repairs +field state at a forbidden site. + +### S4 - Schema-v2 compatibility/refusal is implemented + +Category: solid + +`_SCHEMA_VERSION = 2` is present (`engine_state/__init__.py:74`). `load_manifest()` +tolerates old/equal versions and refuses newer versions with +`IncompatibleEngineStateError` (`engine_state/__init__.py:191-209`). Tests cover v2 +manifest write and v1 no-session-state fallback +(`tests/test_engine_state_session_persistence.py:40-54`). + +### S5 - `persist_session_state` is default false and opt-in + +Category: solid + +`RuntimeConfig.persist_session_state` defaults to `False` with comments explaining +the per-turn O(n) snapshot cost and always-on-runtime scope (`core/config.py:279-285`). +`ChatRuntime` only loads/saves the lived session snapshot when the flag is true +(`chat/runtime.py:752-755`, `chat/runtime.py:789-795`). The L10 soak runner forces +the flag on for the resume lane (`evals/l10_continuity/runner.py:93-104`). + +### S6 - Referent slot/history aliasing is preserved on restore + +Category: solid + +`ReferentRegistry.to_dict()` persists history as source of truth and slots as +slot-to-history-index so `from_dict()` reuses the same restored `ReferentEntry` +objects (`session/referents.py:173-204`). The test confirms the active slot object +is the same object as the corresponding history entry +(`tests/test_session_context_codec.py:52-63`). + +### S7 - Source-tree invariant scans exclude `.claude/` + +Category: solid + +The whole-tree architectural scans filter `.claude` in both walk/rglob styles +(`tests/test_architectural_invariants.py:240-243`, +`tests/test_architectural_invariants.py:648-658`). This satisfies the audit note +that invariant source-tree scans must exclude agent worktrees. + +## LOC Accounting + +GitHub/merge-commit accounting for #567-#573: + +| PR | Title | Files | Additions | Deletions | Net | +| --- | --- | ---: | ---: | ---: | ---: | +| #567 | L10 continuity spike | 8 | 1305 | 0 | +1305 | +| #568 | Shape B+ Phase A | 4 | 319 | 1 | +318 | +| #569 | Shape B+ Phase B | 2 | 138 | 0 | +138 | +| #570 | Shape B+ Phase C | 8 | 413 | 5 | +408 | +| #571 | Shape B+ Phase D+E | 9 | 215 | 40 | +175 | +| #572 | L11 identity continuity | 10 | 549 | 5 | +544 | +| #573 | Continuous learning idle | 2 | 203 | 0 | +203 | +| Total | | 43 | 3142 | 51 | +3091 | + +This was not a net-removal stack. It added a new eval lane, persistence codecs, +state wiring, identity lineage, and idle-learning tests/substrate. + +## Validation + +Requested baseline: + +```bash +core test --suite smoke -q +``` + +The bare `core` command was not initially on PATH in the fresh worktree. After +running through the local editable project entrypoint: + +```bash +uv run --with-editable . core test --suite smoke -q +``` + +Result: + +```text +87 passed in 116.32s (0:01:56) +``` + +No source code was changed during this audit. Only this report was added.