core/evals/l10_continuity/contract.md
Shay 23bc28caf9 feat(evals): L10 continuity spike — falsifiable long-horizon soak lane
Build evals/l10_continuity/, the empirical gate between the two L10 targets
(T-resume: provable same-life resume; T-experience: continuous experiencing
field-life). Drives the REAL turn loop (ChatRuntime + CognitiveTurnPipeline)
over a deterministic in-vocab corpus, with reboot and orphan-crash legs, and
evaluates falsifiable predicates over recorded evidence. Additive only; no
existing file touched; read-only over the runtime; no serving-path import.

Predicates (each with a *_holds real-soak test AND a *_bites mutation test, per
the CLAUDE.md schema-as-proof discipline):
- P1 closure: versor_condition < 1e-6 every turn (green guard).
- P2a determinism: two independent runtimes -> byte-identical trace_hash.
- P2b reboot transparency (the diagnostic): a reboot never alters pre-reboot
  turns (hard guard); post-reboot transparency is MEASURED and today FALSE --
  the mechanical proof that Shape B (ADR-0146) discards the lived field/vault,
  i.e. "many lives sharing a checkpoint". A pinned test flips if persistence is
  ever added, forcing a doc update so the gap can't close silently.
- P3 bounded resources: vault grows linear-bounded/monotonic (RSS recorded).
- P4 crash recovery: two recoveries from one checkpoint converge (determinism)
  + commit-point/ARIES force boundary (recovered turn_count == committed) +
  atomic-write survives mid-os.replace kill (ADR-0156).
- P5b anchor stability (T-experience crux): field anchors without COLLAPSE
  (dist_to_anchor not -> 0) or FREEZE (turn_movement not -> 0); the long-horizon
  test of the sanctioned _session_anchor_pull (alpha=0.05). Thresholds measured.
- P5c coherence: surfaces stay non-empty and not collapsed to one output, over
  more than one corpus cycle.
- P5a recall precision@k: recorded as not_covered (needs a held-out probe set).

report.py assembles the panel into a structured report with a hardware-stable
deterministic_digest (trace_hash sequence + verdicts; excludes RSS/wall-clock)
as the freeze handle. Run: python -m evals.l10_continuity [n_turns] [reboot_turn].

24 tests pass; adversarially reviewed across 4 lenses (bite-discipline,
invariant/trust-boundary, honesty/determinism, correctness) before landing.
2026-06-05 11:14:17 -07:00

5 KiB
Raw Blame History

L10 Continuity Lane — Contract

Status: spike (falsifiable experiment) · Parent: docs/analysis/L10-continuity-spike-design-2026-06-05.md · Not in default smoke (a soak; run on demand / nightly).

This lane drives the real turn loop (ChatRuntime + CognitiveTurnPipeline) over a deterministic, cyclic, in-vocabulary corpus for N turns, with optional reboot and orphan-crash legs, and evaluates falsifiable predicates over the recorded evidence. It is the empirical gate between the two L10 targets:

  • T-resume — provable same-life resume (determinism + recovery): P1P4.
  • T-experience — a continuous experiencing field-life (content stays meaningful over a long horizon): P5.

Run: PYTHONPATH=. .venv/bin/python -m evals.l10_continuity [n_turns] [reboot_turn]

Predicates

ID Proves Fails loudly when Mutation-verified bite
P1 closure versor_condition < 1e-6 every turn a construction breaks closure (no repair allowed) a record with versor_condition ≥ 1e-6 trips it
P2a determinism two independent runtimes → byte-identical trace_hash sequence the pipeline is nondeterministic a perturbed hash trips it
P2b reboot transparency a reboot never changes turns before the reboot point determinism/state leaks backward across reboot a pre-reboot hash divergence trips it
P3 bounded resources vault grows linear-bounded/monotonic per turn an unbounded cache/store leaks a 10k-entry vault record trips it
P4 recovery determinism two crash-recoveries from one checkpoint converge torn read / nondeterministic boot divergent recovery tails trip it
P4 commit point recovered turn_count == committed turns (ARIES force boundary) the checkpoint isn't the atomic commit boundary None/mismatched count trips it
P5b anchor stability field anchors without collapse (dist_to_anchor↛0) or freeze (turn_movement↛0) the field is swallowed by the attractor, or frozen collapsing distance / zero movement trips it
P5c coherence surfaces stay non-empty and not collapsed to one repeated output the field wanders into noise or freezes onto one output empty / single-surface records trip it

Each predicate has a *_holds test (real soak) and a *_bites test (mutation), per the CLAUDE.md schema-as-proof discipline: a predicate that cannot fail under the violation it nominally catches is decoration, not proof.

Not covered (no silent skips)

  • P5a — recall precision@k stability. Requires a held-out probe set with known-relevant entries and a metric grounded in the vault's actual scoring semantics (the raw recall score is not a clean similarity). Recorded as not_covered in the report; a follow-up increment.

The headline diagnostic (P2b)

Today a reboot restores only recognizers / discovery candidates / turn_count (Shape B, ADR-0146) and discards the lived field / vault / anchor / graph / referents. So post_reboot_transparent == False: a reboot diverges from the uninterrupted run at the first post-reboot turn. This is the mechanical proof of "many lives sharing a checkpoint" and the precise definition of the Shape-B+ persistence work that closes resume-as-same-life. The test_p2b_documents_current_resume_gap test pins this reality and will flip if persistence is later added — forcing a doc update so the gap cannot close silently.

Thresholds (empirical basis, not arbitrary)

All gating thresholds are set from measured real-soak data and deliberately conservative — these are catastrophe gates (collapse / freeze / unbounded leak are yes/no failures of the T-experience claim), not early-warning trend detectors. Gradual-drift detection is a deliberate long-horizon follow-up; tightening toward the healthy band would risk false positives on a different corpus or longer horizon.

Threshold Measured healthy Default floor/ceiling Rationale
P5b dist_to_anchor ~4.06.2 (steady) collapse_floor=1.0 ~75%+ drop toward anchor = pathological collapse
P5b turn_movement median ~1.5 freeze_floor=0.05 ~1/30th of healthy = frozen field
P3 vault growth ~23 entries/turn vault_per_turn_ceiling=4 ~130200% of as-designed writes

P5b vs P5c division of labour: P5b catches the field freezing (movement→0) or collapsing onto the anchor (distance→0); P5c catches the output collapsing to a single repeated surface. The P5c real-soak test runs over more than one corpus cycle so the horizon exercises repetition (a 6-turn run == the cycle length would trivially yield 6 distinct surfaces and prove nothing).

Freeze handle

report.deterministic_digest is a SHA-256 over only hardware-stable evidence (the trace_hash sequence + each predicate's (name, passed) verdict), excluding RSS / wall-clock / raw floats. Pin it once the lane is trusted; a regression flips it.