23bc28caf9
Build evals/l10_continuity/, the empirical gate between the two L10 targets (T-resume: provable same-life resume; T-experience: continuous experiencing field-life). Drives the REAL turn loop (ChatRuntime + CognitiveTurnPipeline) over a deterministic in-vocab corpus, with reboot and orphan-crash legs, and evaluates falsifiable predicates over recorded evidence. Additive only; no existing file touched; read-only over the runtime; no serving-path import. Predicates (each with a *_holds real-soak test AND a *_bites mutation test, per the CLAUDE.md schema-as-proof discipline): - P1 closure: versor_condition < 1e-6 every turn (green guard). - P2a determinism: two independent runtimes -> byte-identical trace_hash. - P2b reboot transparency (the diagnostic): a reboot never alters pre-reboot turns (hard guard); post-reboot transparency is MEASURED and today FALSE -- the mechanical proof that Shape B (ADR-0146) discards the lived field/vault, i.e. "many lives sharing a checkpoint". A pinned test flips if persistence is ever added, forcing a doc update so the gap can't close silently. - P3 bounded resources: vault grows linear-bounded/monotonic (RSS recorded). - P4 crash recovery: two recoveries from one checkpoint converge (determinism) + commit-point/ARIES force boundary (recovered turn_count == committed) + atomic-write survives mid-os.replace kill (ADR-0156). - P5b anchor stability (T-experience crux): field anchors without COLLAPSE (dist_to_anchor not -> 0) or FREEZE (turn_movement not -> 0); the long-horizon test of the sanctioned _session_anchor_pull (alpha=0.05). Thresholds measured. - P5c coherence: surfaces stay non-empty and not collapsed to one output, over more than one corpus cycle. - P5a recall precision@k: recorded as not_covered (needs a held-out probe set). report.py assembles the panel into a structured report with a hardware-stable deterministic_digest (trace_hash sequence + verdicts; excludes RSS/wall-clock) as the freeze handle. Run: python -m evals.l10_continuity [n_turns] [reboot_turn]. 24 tests pass; adversarially reviewed across 4 lenses (bite-discipline, invariant/trust-boundary, honesty/determinism, correctness) before landing.
52 lines
2.2 KiB
Python
52 lines
2.2 KiB
Python
"""The deterministic scripted corpus that drives the L10 continuity soak.
|
|
|
|
The corpus is a fixed, committed sequence of in-vocabulary natural-language
|
|
prompts. Determinism is the point: turn N of a soak is always ``prompt_at(N)``,
|
|
so two independent runs over the same N are byte-identical in their inputs, and
|
|
a reboot leg replays the exact same tail it would have seen uninterrupted.
|
|
|
|
There is NO randomness here — "seeded/replayable" means a fixed cycle, not an
|
|
RNG. The prompts are hand-picked to be resident in the default cognition packs
|
|
so ``ChatRuntime.chat`` never raises ``no in-vocabulary tokens``; the runner
|
|
verifies this at turn 0 and fails loudly if a prompt drifts out of vocabulary.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
# A small, fixed ring of in-vocabulary prompts. Each is a complete cognition
|
|
# turn the default packs can tokenize and ground (mirrors the inputs used by
|
|
# the cognition lane and the ADR-0153 trace-hash tests). The ring is cycled to
|
|
# reach an arbitrary soak length; cycling (not appending novelty) is deliberate
|
|
# — the soak measures whether *repetition over a long horizon* stays closed,
|
|
# deterministic, bounded, and meaningful, not whether novel input is handled.
|
|
_BASE_PROMPTS: tuple[str, ...] = (
|
|
"What causes light?",
|
|
"What is a concept?",
|
|
"Hello.",
|
|
"What causes rain?",
|
|
"What is a principle?",
|
|
"What is memory?",
|
|
)
|
|
|
|
|
|
def base_prompts() -> tuple[str, ...]:
|
|
"""Return the immutable base ring of prompts (a safe copy of the tuple)."""
|
|
return _BASE_PROMPTS
|
|
|
|
|
|
def prompt_at(turn_index: int) -> str:
|
|
"""The prompt for a given 0-based turn index, by cycling the base ring.
|
|
|
|
Deterministic and total: any non-negative ``turn_index`` maps to exactly
|
|
one base prompt, so the corpus is replayable across runs and reboots.
|
|
"""
|
|
if turn_index < 0:
|
|
raise ValueError(f"turn_index must be non-negative, got {turn_index}")
|
|
return _BASE_PROMPTS[turn_index % len(_BASE_PROMPTS)]
|
|
|
|
|
|
def scripted_corpus(n_turns: int) -> tuple[str, ...]:
|
|
"""The first ``n_turns`` prompts of the deterministic soak corpus."""
|
|
if n_turns < 0:
|
|
raise ValueError(f"n_turns must be non-negative, got {n_turns}")
|
|
return tuple(prompt_at(i) for i in range(n_turns))
|