audit(l10-l11): lookback review lived spine
This commit is contained in:
Shay
parent
ea503f6d51
commit
31cb741005
1 changed files with 256 additions and 0 deletions
256
docs/audit/L10-L11-lookback-2026-06-05.md
Normal file
256
docs/audit/L10-L11-lookback-2026-06-05.md
Normal file
|
|
@ -0,0 +1,256 @@
|
|||
# L10/L11 Lived-Spine Lookback Review (#567-#573)
|
||||
|
||||
Date: 2026-06-05
|
||||
Branch: `audit/l10-l11-lookback`
|
||||
Base reviewed: `origin/main` at `ea503f6d5177`
|
||||
Scope: PRs #567-#573, the L10/L11 lived-spine stack.
|
||||
|
||||
## Executive Finding
|
||||
|
||||
The stack landed the intended Shape B+ resume spine: bit-exact field/vault/session
|
||||
serialization, schema-v2 state loading, reboot-transparent L10 traces, EngineIdentity
|
||||
lineage stamping, and proposal-only idle learning. The strongest wrong=0 hazards
|
||||
requested in the brief are solid: `idle_tick()` does not ratify or bypass HITL,
|
||||
vault restore does not reproject, and persistence restore adds no hidden field
|
||||
repair at a forbidden site.
|
||||
|
||||
One live crash-consistency hazard remains: the new multi-file checkpoint treats
|
||||
`manifest.json` as the final commit marker but loads `session_state.json`
|
||||
unconditionally. A power loss after `session_state.json` is atomically replaced
|
||||
and before `manifest.json` is replaced can pair an old manifest with newer lived
|
||||
state on reboot. Fix before the next phase.
|
||||
|
||||
## Hazards
|
||||
|
||||
### H1 - Multi-file checkpoint can restore mixed-generation state
|
||||
|
||||
Category: hazard
|
||||
|
||||
Evidence:
|
||||
- `ChatRuntime.checkpoint_engine_state()` writes recognizers, candidates, then
|
||||
`session_state.json`, then `manifest.json` as the stated commit marker
|
||||
(`chat/runtime.py:779-803`).
|
||||
- `EngineStateStore.save_session_state()` explicitly says session state is saved
|
||||
before the manifest so the manifest is the last durable act
|
||||
(`engine_state/__init__.py:225-236`).
|
||||
- On load, `_load_engine_state()` reads the manifest turn count, then restores
|
||||
`session_state.json` whenever `persist_session_state=True` and the file exists;
|
||||
there is no generation, digest, or turn-count check tying the loaded session
|
||||
snapshot to the manifest (`chat/runtime.py:719-755`,
|
||||
`engine_state/__init__.py:238-251`).
|
||||
- The P4 tests cover an orphan temp file and `os.replace` failure for
|
||||
`manifest.json`, but they only assert recovered `turn_count` from the manifest;
|
||||
they do not simulate a newer committed `session_state.json` with an older
|
||||
manifest (`tests/test_l10_continuity.py:204-238`).
|
||||
|
||||
Impact:
|
||||
A kill between `save_session_state()` and `save_manifest()` can next-boot into a
|
||||
hybrid checkpoint: old manifest/turn count plus newer field/vault/referent state.
|
||||
That violates the L10 P4 claim that recovery lands on a valid prior checkpoint.
|
||||
It is not a versor-condition failure, but it is a lived-spine continuity and
|
||||
deterministic replay hazard.
|
||||
|
||||
Fix before next phase:
|
||||
Make the checkpoint generation atomic at the composed-state level. Minimal repair:
|
||||
stamp `session_state.json` with the same committed `turn_count`/generation and
|
||||
load it only when it matches the manifest. Stronger repair: write a generation
|
||||
directory or content-addressed bundle and have the manifest point to hashes for
|
||||
recognizers, candidates, and session state. Add a P4 regression test that creates
|
||||
old manifest plus newer session state and proves restore refuses/falls back to
|
||||
the valid prior generation.
|
||||
|
||||
## Drift
|
||||
|
||||
### D1 - P5a recall precision did not land
|
||||
|
||||
Category: drift
|
||||
|
||||
The L10 continuity spike design specified P5 as the T-experience gate, including
|
||||
P5a recall precision@k stability. The landed contract explicitly records P5a as
|
||||
`not_covered` (`evals/l10_continuity/contract.md:33-38`), while P5b/P5c landed
|
||||
as anchor/coherence catastrophe checks (`evals/l10_continuity/predicates.py:291-380`).
|
||||
This is honest in the contract, but it is still a design-vs-implementation cut.
|
||||
|
||||
Fix before next phase:
|
||||
Do not claim T-experience or long-horizon recall stability from #567-#573 alone.
|
||||
Add the held-out exact-CGA recall probe set and gate P5a before using the lived
|
||||
spine as evidence of continuous experiencing memory.
|
||||
|
||||
### D2 - EngineIdentity scope differs from the L10 scoping recommendation
|
||||
|
||||
Category: drift
|
||||
|
||||
The L10 scoping brief recommended an EngineIdentity hash over
|
||||
`schema_version + canonical_bytes` and a lineage chain recording schema migration.
|
||||
The shipped EngineIdentity is instead a content hash over ratified identity,
|
||||
safety, ethics, register, anchor-lens packs plus code revision
|
||||
(`core/engine_identity.py:1-20`, `core/engine_identity.py:85-127`).
|
||||
The manifest stamps this identity and its parent (`engine_state/__init__.py:161-189`).
|
||||
|
||||
This shipped model is internally coherent and tested, but it is not the same
|
||||
identity primitive described in the scoping record. Lived experience remains in
|
||||
Shape B+ `session_state.json`; the identity hash names the ratified substrate.
|
||||
|
||||
Fix before next phase:
|
||||
Amend the L10/L11 design record or ADR scope to say EngineIdentity deliberately
|
||||
means ratified substrate identity, not checkpoint-content identity. If checkpoint
|
||||
content is still desired, add a separate `checkpoint_identity`/digest and link it
|
||||
from the manifest rather than overloading EngineIdentity.
|
||||
|
||||
## Gaps
|
||||
|
||||
### G1 - P4 is sampled at the manifest/orphan boundary, not arbitrary interruption
|
||||
|
||||
Category: gap
|
||||
|
||||
The design brief elevated power-loss-at-arbitrary-instruction as a primary driver.
|
||||
The landed runner simulates an orphan `.manifest.json.*.tmp` and the tests patch
|
||||
`os.replace` for manifest writes (`evals/l10_continuity/runner.py:107-119`,
|
||||
`tests/test_l10_continuity.py:204-238`). That is useful, but it does not enumerate
|
||||
or sample every write boundary in the multi-file checkpoint.
|
||||
|
||||
Fix before next phase:
|
||||
After H1 is fixed, extend P4 to cover crashes after each durable file in the
|
||||
checkpoint sequence: recognizers, candidates, session state, and manifest.
|
||||
|
||||
### G2 - Idle learning backpressure remains unresolved
|
||||
|
||||
Category: gap
|
||||
|
||||
The roadmap/scoping notes called out absent-reviewer saturation as a real operator
|
||||
condition. The proposal path has a pending cap (`DEFAULT_PENDING_CAP = 256`) and
|
||||
emits a queue-full report when saturated (`teaching/proposals.py:48`,
|
||||
`teaching/proposals.py:551-586`). `idle_tick()` treats proposal refusals as
|
||||
non-created proposals and continues (`chat/runtime.py:858-865`). This is safe for
|
||||
wrong=0, but it means learning can stall silently from the runtime result shape:
|
||||
`IdleTickResult` has no refused/capacity count (`chat/runtime.py:508-514`).
|
||||
|
||||
Fix before next phase:
|
||||
Expose refusal counts/reasons in `IdleTickResult` or idle telemetry so operators
|
||||
can distinguish "no determined candidates" from "HITL queue full."
|
||||
|
||||
## Solid
|
||||
|
||||
### S1 - `idle_tick()` does not bypass HITL
|
||||
|
||||
Category: solid
|
||||
|
||||
`idle_tick()` contemplates pending candidates, calls `propose_from_candidate()`,
|
||||
and never calls `accept_proposal()` or writes to the corpus (`chat/runtime.py:815-873`).
|
||||
`propose_from_candidate()` first builds a proposal through `check_eligibility()`,
|
||||
which rejects `polarity="undetermined"` (`teaching/proposals.py:165-181`), then
|
||||
records a `created` event and replay evidence; acceptance remains a separate
|
||||
operator path (`teaching/proposals.py:623-676`). Tests cover undetermined refusal
|
||||
and determined-but-pending behavior (`tests/test_continuous_learning_idle.py:72-99`).
|
||||
|
||||
Answer to the audit question: an undetermined candidate cannot reach `pending`
|
||||
through `idle_tick()`. A determined candidate can reach `pending`, but not
|
||||
`accepted`; that is the intended HITL boundary.
|
||||
|
||||
### S2 - Vault restore does not reproject
|
||||
|
||||
Category: solid
|
||||
|
||||
`VaultStore.from_dict()` decodes persisted arrays, rebuilds `_exact_index`, leaves
|
||||
`_matrix_cache` derived, and never calls `store()` or `reproject()`
|
||||
(`vault/store.py:444-469`). The bright-line test asserts byte-identical restored
|
||||
versors and exact recall (`tests/test_vaultstore_codec.py:41-84`). The existing
|
||||
live `store()` path still auto-reprojects every N stores, but the restore path is
|
||||
pure deserialize plus derived-index rebuild.
|
||||
|
||||
Answer to the audit question: engine-state restore does not reproject a vault.
|
||||
|
||||
### S3 - Persistence adds no forbidden field repair
|
||||
|
||||
Category: solid
|
||||
|
||||
The array codec is raw dtype/shape/base64 bytes; `FieldState.from_dict()` rebuilds
|
||||
from decoded bytes via the constructor, which copies and validates shape/dtype but
|
||||
does not normalize (`field/state.py:117-132`, `field/state.py:173-183`).
|
||||
`SessionContext.restore()` composes component restore functions and explicitly
|
||||
keeps vocab/persona outside session state (`session/context.py:373-405`).
|
||||
No new persistence load path calls `unitize`, `normalize`, `null_project`, or a
|
||||
field repair operator.
|
||||
|
||||
Answer to the audit question: no new persistence restore path normalizes or repairs
|
||||
field state at a forbidden site.
|
||||
|
||||
### S4 - Schema-v2 compatibility/refusal is implemented
|
||||
|
||||
Category: solid
|
||||
|
||||
`_SCHEMA_VERSION = 2` is present (`engine_state/__init__.py:74`). `load_manifest()`
|
||||
tolerates old/equal versions and refuses newer versions with
|
||||
`IncompatibleEngineStateError` (`engine_state/__init__.py:191-209`). Tests cover v2
|
||||
manifest write and v1 no-session-state fallback
|
||||
(`tests/test_engine_state_session_persistence.py:40-54`).
|
||||
|
||||
### S5 - `persist_session_state` is default false and opt-in
|
||||
|
||||
Category: solid
|
||||
|
||||
`RuntimeConfig.persist_session_state` defaults to `False` with comments explaining
|
||||
the per-turn O(n) snapshot cost and always-on-runtime scope (`core/config.py:279-285`).
|
||||
`ChatRuntime` only loads/saves the lived session snapshot when the flag is true
|
||||
(`chat/runtime.py:752-755`, `chat/runtime.py:789-795`). The L10 soak runner forces
|
||||
the flag on for the resume lane (`evals/l10_continuity/runner.py:93-104`).
|
||||
|
||||
### S6 - Referent slot/history aliasing is preserved on restore
|
||||
|
||||
Category: solid
|
||||
|
||||
`ReferentRegistry.to_dict()` persists history as source of truth and slots as
|
||||
slot-to-history-index so `from_dict()` reuses the same restored `ReferentEntry`
|
||||
objects (`session/referents.py:173-204`). The test confirms the active slot object
|
||||
is the same object as the corresponding history entry
|
||||
(`tests/test_session_context_codec.py:52-63`).
|
||||
|
||||
### S7 - Source-tree invariant scans exclude `.claude/`
|
||||
|
||||
Category: solid
|
||||
|
||||
The whole-tree architectural scans filter `.claude` in both walk/rglob styles
|
||||
(`tests/test_architectural_invariants.py:240-243`,
|
||||
`tests/test_architectural_invariants.py:648-658`). This satisfies the audit note
|
||||
that invariant source-tree scans must exclude agent worktrees.
|
||||
|
||||
## LOC Accounting
|
||||
|
||||
GitHub/merge-commit accounting for #567-#573:
|
||||
|
||||
| PR | Title | Files | Additions | Deletions | Net |
|
||||
| --- | --- | ---: | ---: | ---: | ---: |
|
||||
| #567 | L10 continuity spike | 8 | 1305 | 0 | +1305 |
|
||||
| #568 | Shape B+ Phase A | 4 | 319 | 1 | +318 |
|
||||
| #569 | Shape B+ Phase B | 2 | 138 | 0 | +138 |
|
||||
| #570 | Shape B+ Phase C | 8 | 413 | 5 | +408 |
|
||||
| #571 | Shape B+ Phase D+E | 9 | 215 | 40 | +175 |
|
||||
| #572 | L11 identity continuity | 10 | 549 | 5 | +544 |
|
||||
| #573 | Continuous learning idle | 2 | 203 | 0 | +203 |
|
||||
| Total | | 43 | 3142 | 51 | +3091 |
|
||||
|
||||
This was not a net-removal stack. It added a new eval lane, persistence codecs,
|
||||
state wiring, identity lineage, and idle-learning tests/substrate.
|
||||
|
||||
## Validation
|
||||
|
||||
Requested baseline:
|
||||
|
||||
```bash
|
||||
core test --suite smoke -q
|
||||
```
|
||||
|
||||
The bare `core` command was not initially on PATH in the fresh worktree. After
|
||||
running through the local editable project entrypoint:
|
||||
|
||||
```bash
|
||||
uv run --with-editable . core test --suite smoke -q
|
||||
```
|
||||
|
||||
Result:
|
||||
|
||||
```text
|
||||
87 passed in 116.32s (0:01:56)
|
||||
```
|
||||
|
||||
No source code was changed during this audit. Only this report was added.
|
||||
Loading…
Reference in a new issue