66b8c7c431
A2 of the refined sequencing. Proves (deterministically, not by assertion) what a long-running CORE life costs to persist per turn on a constrained offline device. Measures the Shape B+ checkpoint BYTES per turn (session_state.json) over the real turn loop — bytes, not wall-clock latency (machine-dependent → flaky). Reuses the L10 continuity corpus. Measured cliff: save_session_state re-serializes the FULL snapshot every turn, so per-turn bytes are O(n) in the accumulated life — 3,811 → 88,189 bytes (23x) over 24 turns, ~1.3KB/vault-entry re-written every turn. That blocks continuous-life at the edge. The gate encodes the edge REQUIREMENT (≤16 KiB/turn regardless of session length) as xfail(strict): it fails today (documenting the cliff), runs green in CI, and flips to a hard failure the moment incremental/append-only persistence (O(Δ)/turn) lands — forcing us to retire it. Plus a regression ceiling (passes today) and a determinism check (the byte metric is reproducible → a valid gate). The fix is algorithmic (incremental persistence, Python/Ring-2), NOT a language rewrite. Tagged core/array_codec.py as the locked reference contract for a future gated Ring-1 Zig byte-exact codec (ADR-0196 G0-G8) — step 3, only after the O(Δ) fix and only if this gate proves the codec is still the bottleneck. See contract.md.
74 lines
3.1 KiB
Python
74 lines
3.1 KiB
Python
"""Edge-deployment budget gate (A2) — deterministic per-turn persistence cost.
|
|
|
|
Three obligations:
|
|
- EDGE REQUIREMENT (xfail today, strict): per-turn checkpoint bytes must stay under a
|
|
fixed budget regardless of session length — what a constrained offline device can
|
|
afford for an indefinitely-running life. The current O(n) snapshot breaches it, so
|
|
this is an EXPECTED failure that documents the cliff; it flips to a hard failure
|
|
(xpass, strict) the moment incremental/append-only persistence (O(Δ)/turn) lands,
|
|
forcing us to retire the xfail. This is the gate that makes the fix falsifiable.
|
|
- REGRESSION CEILING (passes today): catches a change that makes the cliff worse.
|
|
- DETERMINISM: the byte metric is reproducible (same corpus → identical series), which
|
|
is why it is a valid gate rather than a flaky wall-clock measurement.
|
|
|
|
Each pipeline turn is ~3s, so the soak runs ONCE (module-scoped) and is kept short —
|
|
the cliff already breaches the 16 KiB edge budget by turn ~4. The full 24-turn measured
|
|
series lives in ``evals/edge_budget/contract.md``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from evals.edge_budget.runner import (
|
|
EDGE_PER_TURN_CEILING_BYTES,
|
|
REGRESSION_PER_TURN_CEILING_BYTES,
|
|
REGRESSION_TOTAL_CEILING_BYTES,
|
|
measure,
|
|
run,
|
|
)
|
|
|
|
_TURNS = 8
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def report() -> dict:
|
|
return run(_TURNS) # one soak, shared across the cost assertions
|
|
|
|
|
|
@pytest.mark.xfail(
|
|
strict=True,
|
|
reason=(
|
|
"O(n) per-turn persistence cliff: save_session_state re-serializes the FULL "
|
|
"snapshot every turn, so per-turn bytes grow with the accumulated life. Flips "
|
|
"green when incremental/append-only persistence lands (O(Δ)/turn). See "
|
|
"evals/edge_budget/contract.md."
|
|
),
|
|
)
|
|
def test_per_turn_checkpoint_cost_is_within_edge_budget(report) -> None:
|
|
# The edge requirement: bounded per-turn write cost on a constrained device.
|
|
assert report["max_per_turn_bytes"] <= EDGE_PER_TURN_CEILING_BYTES, (
|
|
f"per-turn checkpoint peaked at {report['max_per_turn_bytes']} bytes "
|
|
f"(budget {EDGE_PER_TURN_CEILING_BYTES}); growth_ratio={report['growth_ratio']}"
|
|
)
|
|
|
|
|
|
def test_persistence_cost_regression_ceiling(report) -> None:
|
|
# Passes today; guards against making the cliff materially worse before the fix.
|
|
assert report["max_per_turn_bytes"] <= REGRESSION_PER_TURN_CEILING_BYTES
|
|
assert report["total_bytes_written"] <= REGRESSION_TOTAL_CEILING_BYTES
|
|
|
|
|
|
def test_cost_grows_with_accumulated_state_today(report) -> None:
|
|
# Documents the CURRENT defect: per-turn cost is NOT bounded — it tracks vault
|
|
# growth. (When the fix lands this becomes ~flat; update the assertion then.)
|
|
assert report["final_per_turn_bytes"] > report["first_per_turn_bytes"]
|
|
assert report["growth_ratio"] > 1.0
|
|
assert report["edge_budget_met"] is False # the cliff is real, on the record
|
|
|
|
|
|
def test_cost_metric_is_deterministic() -> None:
|
|
# The whole point of measuring BYTES (not latency): reproducible → a valid gate.
|
|
a = [c.checkpoint_bytes for c in measure(2)]
|
|
b = [c.checkpoint_bytes for c in measure(2)]
|
|
assert a == b and len(a) == 2
|