b823d85d28
Rank 1 on the adopted docket, and the cheapest item on it: two ratified ADRs
whose records contradicted the code, plus the ADR-0252 obligation that Track
A's ratified NO-GO created. All four land in two files because they are two
files under one authority — splitting them would have bought two reviews of
the same document.
ADR-0146 (R-12a, option A) — the daemon it rejected, shipped.
The Shape-A rejection stands AS REASONING: a background daemon does require
process-supervision infrastructure that Shape B does not. That infrastructure
was subsequently built. chat/always_on_daemon.py implements the three items
"What is NOT in Scope" excludes, each mapped to where:
- cross-process file locking -> advisory fcntl.flock(LOCK_EX|LOCK_NB) at :82,
acquired BEFORE any signal or runtime setup so a second life fails fast
- signal handling -> SIGINT/SIGTERM graceful stop at :145
- daemon synchronization -> the load-time strict-identity guard; the flag is
forced at :48 and a different-identity checkpoint is refused, so a restart
is the same life or it is nothing
Shape B remains the persistence model and the CLI default; the daemon is an
additional shape layered on it, owned by this ADR. A new ADR for a shape that
shipped six weeks earlier adds a document without adding a decision.
The stale bullet gets an INLINE pointer, not only a trailing addendum. H-8's
mechanism is that an authoritative line converts "I should check" into "I
already checked" — the reader who stops at the excluded-scope list is exactly
the reader this gap is about, and they never scroll to the end.
ADR-0252 (R-12b, option A) — the "34 organs" basis footnote.
Footnote at the first use, marking the count as underived: the reproducible
figure is 18 resolve_promotable_* organs, ALL in generate/derivation/ and none
anywhere else, with exactly 32 modules in the package as the likely origin of
"34". The diagnosis, supersession plan and §4 conformance bar are unaffected —
none depends on the count. Option B (replace every "34" with "18") rejected: it
rewrites a ratified document's prose where a note at the point of use is all
that is owed.
ADR-0252 §5 — the verdict banner. NO-GO, ratified. Criterion pre-registered at
299c92be BEFORE the run, run at 797ebad5, artifact digest b3d9d275…, with the
mechanism stated in one measured sentence: the similarity quotient that
delivers attribute-invariance is the same quotient that annihilates structural
contrast. Scope held narrow on purpose — this refutes H1 for point-position
role encodings aligned under similarity, not every Cl(4,1) representation, and
not the symbolic lane. Also records N-8 (two prior GO verdicts on unmerged
branches, both unsound) and G-21.
ADR-0252 §6 — the amendment the NO-GO forces, and the load-bearing half.
§6 retired the surface organs "until the structure-mapper proves out" and named
no replacement for the NO-GO branch. Left alone that is a live instruction that
can never fire — an authoritative-looking dead instrument, the H-9 class
exactly. Amended: the organs are RETAINED, and that is the operative state
rather than a pause in a transition; the retirement condition is re-stated as a
demonstrated replacement passing the §5 criterion, which does not exist. The
"Build (gated on §5 GO)" bullet is marked NOT AUTHORIZED with the reason.
Retiring a working organ on the strength of a hypothesis ABOUT its replacement
is backwards; the hypothesis has now been tested.
Verification — every factual claim re-derived at ca5e614e rather than copied
from the packet: 18 organs, 0 outside generate/derivation/, exactly 32 modules
(tightening the draft's "~32"); lock, signals and identity guard each cited by
line; report digest read back from the artifact file.
Three things caught before landing, recorded because this commit is about
records being wrong.
1. The date. The addendum drafted "landed 2026-06-14". This container's clone
was SHALLOW — 168 commits, roots grafted at 2026-07-19 — so git log
reported 2026-07-20, which is the shallow boundary and not a fact. Neither
figure was verifiable until the clone was deepened to 2340 commits, which
confirmed the packet: 18e25580, 2026-06-14, the same commit introducing
both the lock and the signal handling. The addendum now cites the SHA. A
date that cannot be re-derived is testimony.
2. A wrong section reference — the amendment cited §9 for a sentence in §8.
3. A misquote — "the existing reader" for "the existing 34-organ reader".
Writing a misquote into a ratified document while amending that document for
accuracy would have been this ruling's own failure mode committed inside its
own fix. Both were found by re-reading the amendment against the source it
quotes, which is standing philosophy 7.
Records closed: H-8(a), H-8(b), G-15. Track A's outstanding obligation
discharged. H-8(c)/(d) remain open on R-3 and land with PR-5.
No runtime behavior. No test changes. Two ADR files and four register/plan
files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4
149 lines
10 KiB
Markdown
149 lines
10 KiB
Markdown
# ADR-0146: L10 Shape B Hybrid Engine-State Persistence
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-25
|
|
**Scope doc:** [L10-runtime-model-scope](./L10-runtime-model-scope.md)
|
|
**Related:** ADR-0055 (inter-session memory), ADR-0040 (telemetry), ADR-0057 (proposals), W-008, W-003, W-007, W-009, W-017, W-018.
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
CORE's runtime has historically been session-bounded: every `core` CLI invocation builds a fresh `ChatRuntime` instance, loading packs and teaching corpora anew, while session-state is lost. To realize the vision of a forever-running cognitive engine that accumulates capability over its lifetime, surviving reboots as recovery rather than control flow, CORE requires a defined process and persistence model.
|
|
|
|
The [L10-runtime-model-scope](./L10-runtime-model-scope.md) evaluated three candidate process shapes:
|
|
- **Shape A (Long-lived daemon):** A single persistent daemon process running `cmd_serve`, where CLI invocations act as IPC clients.
|
|
- **Shape B (Hybrid state externalized; CLI restores it):** Engine-state is checkpointed to disk at logical action boundaries, and CLI invocations load and resume this checkpoint.
|
|
- **Shape C (One-shot CLI with audit trail reconstruction):** Every invocation builds state from scratch by replaying the entire append-only audit trail (telemetry JSONL) from inception.
|
|
|
|
### Candidate Evaluation and Rationale
|
|
|
|
- **Shape B (Selected)** is chosen because:
|
|
- It maintains **library-session compatibility** without requiring a background daemon process to be running on the host system.
|
|
- Startup cost is bounded to $O(\text{checkpoint size})$ rather than $O(\text{audit trail size})$, which ensures high performance as the transaction history grows.
|
|
- Approximately 80% of the underlying persistence infrastructure (packs, telemetry, corpus) is already written to disk.
|
|
- High-value engine-state objects, such as `DerivedRecognizer`, are already serializable (via `DerivedRecognizer.to_json() / from_json()`).
|
|
- **Shape A (Rejected)** is rejected because a background daemon process cannot survive host library-session interruptions (such as IDE reloads or parent process terminations) without complex process supervision infrastructure.
|
|
- **Shape C (Rejected)** is rejected because the $O(N)$ rebuild cost to replay the entire audit trail grows without bound over time, violating the performance and efficiency doctrines.
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
Adopt **Shape B: Hybrid engine-state persistence**.
|
|
|
|
At every logical-action boundary (specifically, at the turn boundary in `ChatRuntime.chat()`), the current engine-state is serialized and checkpointed to an `engine_state/` directory in the repository root (or the path specified by the `CORE_ENGINE_STATE_DIR` environment variable). Any subsequent CLI invocation loads this checkpoint, restoring `RecognizerRegistry` and the `DiscoveryCandidate` working set, and continues.
|
|
|
|
Session-state remains ephemeral and is discarded upon turn completion or process exit.
|
|
|
|
---
|
|
|
|
## State Class Assignments
|
|
|
|
The runtime state is partitioned into four distinct classes:
|
|
|
|
| State class | Objects | Persistence |
|
|
| :--- | :--- | :--- |
|
|
| **Session-state** | `session_thread`, current intent, field excitation | Ephemeral — lost on reboot / process exit, no concern. |
|
|
| **Engine-state** | `RecognizerRegistry`, `DiscoveryCandidate` working set | Persistent — written to `engine_state/recognizers.jsonl` and `engine_state/discovery_candidates.jsonl` on turn boundaries. |
|
|
| **Substrate-state** | Ratified packs, teaching corpus, telemetry JSONL, proposal log | Persistent — already on disk; append-only and immutable without operator intervention. |
|
|
| **T1 vault** | `VaultStore` (in-memory deque) | Ephemeral — intentionally ephemeral per ADR-0055 T1; promoted to T3 via HITL. |
|
|
|
|
---
|
|
|
|
## `engine_state/` Directory Specification
|
|
|
|
The checkpoint directory is structured as follows:
|
|
|
|
```text
|
|
engine_state/
|
|
├── manifest.json
|
|
├── recognizers.jsonl
|
|
└── discovery_candidates.jsonl
|
|
```
|
|
|
|
- **`engine_state/recognizers.jsonl`**: One JSON line per registered recognizer, serialized using `DerivedRecognizer.to_json()`.
|
|
- **`engine_state/discovery_candidates.jsonl`**: One JSON line per pending candidate, serialized using `DiscoveryCandidate.as_dict()`. Note that while `as_dict()` is already implemented, a corresponding `from_dict()` (or load path) will be implemented to deserialize candidates.
|
|
- **`engine_state/manifest.json`**: Metadata schema pinning correctness:
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"written_at_revision": "<git-sha>",
|
|
"turn_count": N
|
|
}
|
|
```
|
|
|
|
### File Operations and Invariants:
|
|
- The `engine_state/` directory is created on the first checkpoint. A missing directory represents a clean-slate start and must not raise an error.
|
|
- Unlike substrate-state (which is append-only), **engine-state files are mutable and overwritten** during each checkpoint to reflect the current active working state.
|
|
- Checkpointing must be atomic (e.g., write to temporary file and rename) to prevent corruption if the process is terminated mid-write.
|
|
|
|
---
|
|
|
|
## Checkpoint Protocol
|
|
|
|
The `ChatRuntime` class manages the lifecycle of the engine-state checkpoint:
|
|
|
|
1. **`ChatRuntime.checkpoint_engine_state(path: Path)`**: Called at the turn boundary after a turn completes, but *before* the response is returned to the caller. This serializes and overwrites the files in the target directory.
|
|
2. **`ChatRuntime.load_engine_state(path: Path)`**: Called within `ChatRuntime.__init__` at startup if the `engine_state/` directory exists and the `--no-load-state` CLI flag is not set.
|
|
3. **`--no-load-state` CLI Flag**: An opt-out flag for debugging, testing, or executing clean-slate runs. When set, `load_engine_state` is bypassed.
|
|
|
|
---
|
|
|
|
## Determinism Guarantee
|
|
|
|
To preserve the non-negotiable byte-identical replay contract:
|
|
- Engine state files must be written using canonical JSON serialization: `sort_keys=True`, and tight separators `separators=(",", ":")` with `ensure_ascii=False`.
|
|
- **Round-Trip Invariant:** Loading a checkpoint and immediately re-saving it must produce byte-identical files on disk. Unit and integration tests must pin this round-trip invariant to prevent serialization drift.
|
|
|
|
---
|
|
|
|
## What is NOT in Scope
|
|
|
|
To maintain a narrow and robust focus, the following items are explicitly excluded from this design:
|
|
- **VaultStore persistence:** `VaultStore` remains an ephemeral T1 memory layer per ADR-0055. Permanent memory resides in the T3 teaching corpus and is promoted only via HITL.
|
|
- **Concurrency control:** Since Shape B is single-process and synchronous, cross-process file locking, daemon synchronization, and signal handling are out of scope. — **Superseded for the daemon path only; see the Addendum (2026-07-28) at the end of this ADR. Shape B's own single-process path is unchanged.**
|
|
- **Network surfaces:** The engine remains strictly local-only; no TCP/HTTP servers or sockets are added to support persistence.
|
|
- **Multi-tenancy/multi-instance:** A single repository supports exactly one active engine state checkpoint.
|
|
- **Re-architecting `ChatRuntime`:** The unit of execution is unchanged; `ChatRuntime` merely gains load/save hook methods.
|
|
|
|
---
|
|
|
|
## Unlocks
|
|
|
|
Establishing this hybrid persistence model directly unlocks the following ratchet tasks:
|
|
- **W-003 (`VaultPromotionPolicy` wiring):** The timing for when the active field state crystallizes and promotes candidates is now defined by the turn-boundary checkpoint.
|
|
- **W-007 (DerivedRecognizer integration):** Provides the persistent `RecognizerRegistry` slot that preserves active recognizers across turns.
|
|
- **W-009 (HITL async queue):** The pending `DiscoveryCandidate` working set on disk acts as the async queue state, allowing the operator to review candidates asynchronously.
|
|
- **W-017 / W-018:** Enables autonomous contemplation and automated memory promotion pipelines to check and update persistence boundaries safely.
|
|
|
|
---
|
|
|
|
## Risks and Mitigations
|
|
|
|
- **Serialization Drift:** A stale serializer or added fields on `DerivedRecognizer` or `DiscoveryCandidate` could break reload compatibility.
|
|
- *Mitigation:* Pin round-trip serialization in unit tests. Verify that schema updates include migrations or clear-slate fallbacks.
|
|
- **Stale Checkpoint after Pack Mutation:** If a user checks out a different git revision or modifies packs, the loaded checkpoint might refer to invalid types or mismatching revisions.
|
|
- *Mitigation:* Compare `written_at_revision` in `manifest.json` with the current git SHA. If they mismatch, log a warning but continue startup (do not refuse to start, as a reboot is recovery, not control flow).
|
|
|
|
---
|
|
|
|
## Addendum — 2026-07-28, daemon shape reconciled
|
|
|
|
*Ruled R-12a, option A (`docs/assessment/50-rulings.md`), 2026-07-28 by Joshua Shay. This addendum changes no decision. It stops the record contradicting the code.*
|
|
|
|
**The Shape-A rejection stands as reasoning.** A background daemon requires process-supervision infrastructure that Shape B does not — that was true when written and is true now. What changed is that the infrastructure was subsequently built.
|
|
|
|
`chat/always_on_daemon.py` (landed `18e25580`, 2026-06-14) runs a supervised always-on daemon **over** Shape-B persistence, and implements the three items this ADR's "What is NOT in Scope" places outside it:
|
|
|
|
| Excluded item | Where it is implemented |
|
|
|---|---|
|
|
| cross-process file locking | advisory `fcntl.flock(LOCK_EX \| LOCK_NB)` single-instance lock (`:82`), acquired *before* any signal or runtime setup so a second life fails fast |
|
|
| signal handling | SIGINT/SIGTERM handlers for a graceful stop (`:145`), with `install_signals=False` for tests that drive their own `stop_event` |
|
|
| daemon synchronization | the load-time **strict-identity guard** — `strict_identity_continuity` is forced (`:48`) and a different-identity checkpoint is refused, so a daemon restart is the *same* life or it is nothing |
|
|
|
|
**Shape B remains the persistence model and the CLI's default.** The daemon is an *additional* process shape layered on it, not a replacement, and it is owned by this ADR rather than by a new one — the decision here was never wrong, only its scope sentence went stale. A separate ADR for a shape already shipped would add a document without adding a decision (R-12a option B, rejected).
|
|
|
|
**Scope of the supersession:** the "Concurrency control" bullet under *What is NOT in Scope* no longer describes the daemon path. It continues to describe the single-process CLI path exactly as ratified.
|
|
|
|
*Recorded because a ratified ADR that contradicts running code converts "I should check" into "I already checked" — the H-8 mechanism, catalogued in `docs/assessment/31-hindrance-audit.md`.*
|