core/docs/decisions/ADR-0219-generation-checkpoint-atomicity.md

# ADR-0219 — Generation-dir atomic checkpoint (L10 continuity hardening)

Status: accepted
Date: 2026-06-15
Extends: ADR-0146 (engine-state persistence), ADR-0156 (per-file atomic writes)

## Context

ADR-0156 (W-022) introduced `_atomic_write_text` to guarantee **per-file**
atomicity: each of the four checkpoint files (`recognizers.jsonl`,
`discovery_candidates.jsonl`, `session_state.json`, `manifest.json`) is
written via `temp + os.replace`, so a kill mid-write leaves the prior file
intact.

ADR-0156 §"Out of scope" explicitly excluded **cross-file atomicity**:

> "There is no cross-file consistency guarantee: recognizers and candidates
>  are written in separate atomic ops, so a kill between them leaves the
>  store in a mixed state (e.g., recognizers@N over manifest@N-1)."

The flat layout is therefore not a committed state — it is four
independently-atomic files with no shared commit boundary.  L10's telos
("one continuous life") requires that a reboot always resumes from a
**coherent, complete** checkpoint, not a mixed-generation one.

A second deficiency: `finalize_turn_trace_hash` (ADR-0153) called
`save_discovery_candidates` **outside** the main checkpoint sequence,
creating a second write path into the checkpoint directory without going
through the generation commit.

ADR-0156 also deferred the **parent-directory fsync** after `os.replace`.
POSIX strictly requires fsyncing the directory to make the rename metadata
durable.  This ADR closes that deferral.

## Decision

### Generation-dir model

The checkpoint becomes a **committed generation**: a complete, fsync-ed
`gen-NNNN/` directory pointed to by a single `current` file.  The atomic
replacement of `current` is the only commit boundary.

```
engine_state/
  gen-0041/
    recognizers.jsonl
    discovery_candidates.jsonl
    session_state.json
    manifest.json
  current                    ← one line: "gen-0041"
```

Two-phase commit protocol:
1. **`begin_generation()`** — allocate `gen-NNNN/`, return `(gen_num, gen_dir)`.
2. Write all checkpoint files into `gen_dir` via `EngineStateStore(gen_dir).save_*`.
3. **`commit_generation(gen_num)`** — fsync gen dir → `os.replace(tmp_current, current)` → fsync parent → GC old gens.

A SIGKILL before the `os.replace` in step 3 leaves the prior `current`
intact.  A SIGKILL after commits the new generation.  Incomplete `gen-NNNN/`
directories with no `current` entry are garbage, ignored by all load paths.

### `finalize_turn_trace_hash` write path closed

`finalize_turn_trace_hash` no longer calls `save_discovery_candidates`.
The in-memory candidates carry the updated `source_turn_trace`; they are
persisted atomically as part of the next `checkpoint_engine_state` call.
(If the process dies between back-stamp and next checkpoint, the trace hash
is lost from the candidate but the candidate itself survives in the prior
committed generation — acceptable: the trace hash is audit metadata, not
state-determining.)

### Legacy flat layout — explicit migration

On the first `begin_generation()` call against a flat-layout store (pre-0219
`manifest.json` at root, no `current`):
1. Flat files are copied into `gen-0000/` (fsync-ed).
2. `current` is written pointing to `gen-0000`.
3. The next generation (`gen-0001`) is allocated and returned.

Load methods fall back to the flat root when no `current` exists, so a v1
or v2 checkpoint is readable without a write (the migration only happens at
write time).

### Parent-dir fsync

`commit_generation` fsyncs both the generation directory (content durability)
and the parent directory after the `os.replace` (pointer rename metadata
durability), closing the ADR-0156 deferral.

### GC

`commit_generation` retains the last K=2 committed generations (the current
and its predecessor) and prunes older ones.  K≥2 ensures the predecessor is
available if the committed gen dir is somehow corrupted after commit.
Pruned generation names are logged at DEBUG level.

## Implementation

- `engine_state/__init__.py`: `begin_generation`, `commit_generation`,
  `_current_gen_dir`, `_resolve_dir`, `_gc_old_generations`, `_fsync_dir`.
  All `load_*` methods updated to use `_resolve_dir()`.  `exists()` checks
  both layouts.  `save_*` methods unchanged (write relative to `self.path`,
  so they naturally write into a gen dir when instantiated as
  `EngineStateStore(gen_dir)`).
- `chat/runtime.py`: `checkpoint_engine_state` uses the two-phase commit.
  `finalize_turn_trace_hash` no longer calls `save_discovery_candidates`.
- `evals/l10_continuity/runner.py`: `_inject_orphan_tmp` updated to inject
  both orphan shapes (unreferenced gen dir + torn `current` temp file).

## Acceptance gate

All of the following must hold (proven by `tests/test_adr_0219_generation_checkpoint.py`):
- Crash before pointer swap restores the prior committed generation.
- Crash after pointer swap restores the new generation.
- Incomplete gen dirs (not pointed to by `current`) are ignored on load.
- No load path mixes files across generations.
- Legacy flat layout migrates explicitly on first write; reads cleanly without migration.
- No normalization or repair on restore.
- `versor_condition < 1e-6` throughout (proven by L10 lane).

## Invariants pinned by tests

`tests/test_adr_0219_generation_checkpoint.py`:
- `test_fresh_store_writes_gen0000_and_current` — first checkpoint creates gen-0000 + current.
- `test_second_checkpoint_advances_generation` — gen-0001 replaces gen-0000.
- `test_orphan_gen_dir_ignored_before_pointer_swap` — unreferenced gen-9999 is invisible.
- `test_torn_current_tmp_ignored` — `.current.*.tmp` orphan does not affect load.
- `test_no_cross_generation_mixing` — load always reads from a single gen dir.
- `test_legacy_flat_layout_migrates_on_first_write` — flat state → gen-0000 on write.
- `test_legacy_flat_layout_readable_without_write` — load falls back to flat root.
- `test_commit_point_matches_turn_count` — recovered turn_count == committed turns.
- `test_gc_retains_last_two_generations` — GC prunes only generations outside the window.

## Related

- ADR-0146 — engine-state persistence foundation
- ADR-0156 — per-file atomic writes (extended here to cross-file)
- ADR-0157 — revision-mismatch warning on load (unaffected)
- ADR-0158 — reboot event audit (unaffected)
- L10 continuity hardening brief pack: `docs/handoff/l10-continuity-hardening-briefs-2026-06-15.md`