From 9b19085674c4b4dce007b3ce647aa693b8c70dd0 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 28 Jul 2026 07:13:24 +0000 Subject: [PATCH] =?UTF-8?q?feat(evals,tests):=20PR-11=20=E2=80=94=20the=20?= =?UTF-8?q?soak=20becomes=20evidence,=20and=20its=20cadence=20becomes=20a?= =?UTF-8?q?=20pin=20(R-9=20+=20R-2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rank 4 on the docket. The L10 always-on lane has passed a 5000-beat soak since 2026-07-19. That result lived as PROSE IN A CONTRACT FILE — no machine-readable artifact, no run SHA, no rerun path, and a deterministic_digest that report.py computed and nothing pinned. The contract's own closing line asked for the pin ("Pin it once the lane is trusted so a regression flips it") and it was never added. A recorded prose result with no re-derivable artifact is testimony, not evidence, and that file is exactly where the distinction was meant to be enforced. R-2 first, before the run — as the plan required, so the soak measured a decided question rather than deciding one by accident. R-2 was ruled B (beats govern cognition; wall-clock permitted in telemetry only). Measured: this lane carries NO WALL-CLOCK AT ALL — runner.py, predicates.py and report.py contain no time, datetime, or timestamp of any kind. That is stronger than B requires, and it is KEPT rather than "completed", for three reasons now recorded in the contract: 1. B's permissive half is a permission, not an obligation. Adding a field so the code matches a ruling's letter, then adding a pin to guard the field, is a mechanism whose only purpose is to be guarded. 2. "When was this last run?" is already the artifact's COMMIT DATE — immutable, verifiable, stated once. A generated_at inside the report would be a second copy of one fact, drifting the moment a report is regenerated and not committed. That is G-23's defect class, registered this same week. 3. It keeps the digest honestly deterministic. A timestamp would have to be excluded from it — a value in the artifact that the artifact's own integrity check ignores. The contract also records what WOULD change this: a predicate whose claim is about duration rather than sequence (H5's resource-cost leg is the live candidate). B is already ruled, so that needs no new decision — only the predicate that requires it. The re-run: 5000 beats, reboot at 2500, under the POST-R-3 profile. All four predicates pass. Committed as evals/l10_always_on/results/report-5000-6c67e88d.json, digest f814fd97…. The most useful result was not the pass. Every DETERMINISTIC fact reproduced exactly — vault bounded at 6, convergence at beat 1 with a 4999-beat tail, clean reboot with derived learning intact — while the machine-variant float moved from the prose's 1.389e-07 to a worst of 6.177e-08. That is the first real evidence that excluding versor_condition from the digest was correct rather than convenient: a digest covering it would have flipped on a machine change and taught every reader to ignore it. The prose figure is superseded, not contradicted — it was never a reproducible quantity. tests/test_l10_soak_evidence.py, on the gate, guarding two DIFFERENT failure modes that are easy to conflate: REGRESSION — the digest is pinned, plus the horizon (a 24-beat report pinned as though it were the 5000-beat one would pass everything else), the four predicates by name (all_gates_pass alone is satisfied by an empty predicate list), and a vacuity guard on the attested set. STALENESS — the mode nothing guarded, and the one that had ALREADY FIRED. R-9 ruled the cadence change-triggered, not clock-triggered: local-first doctrine makes "nightly" a ruling rather than a cron line, and a nightly that does not run is green for the wrong reason (H-9). So instead of "when did we last run it?", the pin asks the question that matters — DOES THIS EVIDENCE STILL DESCRIBE THE CODE THAT SHIPS? — by holding the SHA-256 of every source the soak attests: chat/always_on.py, chat/always_on_daemon.py, and the three harness modules. That is not hypothetical. It had already fired silently for six weeks: the 2026-07-19 soak ran with accrue_realized_knowledge absent from CONTINUOUS_LIFE_CONFIG_FLAGS, so from R-3 (PR-5, this same day) onward the recorded result attested a process that no longer existed — and nothing in the repository could notice. HAD THIS PIN EXISTED, PR-5 WOULD HAVE FAILED IT. That is precisely the intended behaviour, and it is verified by sabotage: touching always_on_daemon.py turns it red. The failure message says RE-RUN THE SOAK — DO NOT EDIT THE HASHES, because updating a hash to match changed source without re-running converts the pin into a record asserting evidence that does not exist. That is the failure this whole arc is about, committed inside its own remedy. Three sabotages observed red: a touched daemon (the staleness mode), a regressed verdict in the artifact, and a shorter horizon pinned as if it were 5000 beats. H1-H4 promoted onto the gate, and the cheap option was deliberately refused. Measured 20.95s for the four real-soak "holds" and 0.46s for the eight mutation "bites" — 98% of the cost is the holds. Shipping only the bites, or parking the file in a curated-but-unreachable suite, would have satisfied R-9's letter and bought nothing; a suite nobody runs is the non-guarantee this arc spent PR-4 closing. Those four holds are the ONLY tests on any gate that run chat/always_on's run_continuous end-to-end, and that loop just shipped a six-week silent divergence. 21s per push to actually run the process that broke is the most defensible time on this gate. Removed from full_only_baseline.txt in the same edit; the baseline shrank 746 -> 745. Closes G-5. Proof-of-life has moved from prose to committed, pinned, change-guarded evidence. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4 --- core/cli_test.py | 16 ++ docs/assessment/30-gap-register.md | 12 +- docs/assessment/50-execution-plan.md | 8 +- docs/assessment/50-rulings.md | 8 +- evals/l10_always_on/contract.md | 100 +++++++++- .../results/report-5000-6c67e88d.json | 54 ++++++ tests/full_only_baseline.txt | 1 - tests/test_l10_soak_evidence.py | 181 ++++++++++++++++++ 8 files changed, 369 insertions(+), 11 deletions(-) create mode 100644 evals/l10_always_on/results/report-5000-6c67e88d.json create mode 100644 tests/test_l10_soak_evidence.py diff --git a/core/cli_test.py b/core/cli_test.py index b060ca35..1a5c856a 100644 --- a/core/cli_test.py +++ b/core/cli_test.py @@ -156,6 +156,22 @@ TEST_SUITES: dict[str, tuple[str, ...]] = { # ON surface being the only part where staleness is immediately dangerous. # Four sabotages observed red. Pure file parsing, <1s. "tests/test_flag_register.py", + # PR-11 / G-5 / R-9 — the soak's evidence is committed, pinned, and CURRENT. + # Two failure modes, deliberately separated: the digest catches a regression, + # and the attested-source hashes catch STALENESS — evidence that no longer + # describes the code that ships. The second had already fired silently for six + # weeks: the 2026-07-19 soak ran without accrue_realized_knowledge in the + # daemon profile, and nothing anywhere could notice. Had this pin existed, + # PR-5 would have failed it, which is exactly right. <1s. + "tests/test_l10_soak_evidence.py", + # PR-11 / R-9 — the H1-H4 holds+bites pins, promoted onto the gate. + # Measured 20.95s for the four real-soak "holds" and 0.46s for the eight + # mutation "bites". Shipping only the cheap half was considered and rejected: + # these four are the ONLY tests on any gate that run chat/always_on's + # run_continuous loop end-to-end, and that loop just shipped a six-week silent + # divergence. 21s per push to actually run the process that broke is the most + # defensible time on this gate. + "tests/test_l10_always_on_soak.py", # G-22 — CLAIMS.md is a PUBLISHED capability artifact, deterministically # regenerable from the capability ledger + PINNED_SHAS. It published a # superseded evidence digest for deduction_serve_v1 for two days after diff --git a/docs/assessment/30-gap-register.md b/docs/assessment/30-gap-register.md index d61deb27..09300efe 100644 --- a/docs/assessment/30-gap-register.md +++ b/docs/assessment/30-gap-register.md @@ -31,7 +31,7 @@ The comprehension frontier itself, measured. Standing ruling: close fabrications Confirmed at component depth: drive objects exist (`DriveGradientMap` — constructed, never read; `ExertionMeter` — telemetry only); idle mechanisms exist (consolidation, proposal review, contemplation — each flag-gated, each doing one thing); **nothing ranks what matters next**. The daemon heartbeat advances `idle_tick` and nothing more ambitious. This is the AGI-grade conceptual absence: everything CORE does is chosen by the operator. Design work, not a flag flip. **Evidence:** `attention-allocation` + `always-on-process` cards; `02-layer-taxonomy.md` CR-2. · **Authority:** design + ruling (does the L10 process own an agenda, governed by what). -### G-5 · L10 proof debt — the soak ran and passed; what is owed is the artifact, the pinned digest, and a cadence **[AMENDED N-1/N-2/N-4]** +### G-5 · L10 proof debt — **CLOSED 2026-07-28 (PR-11)**; the soak ran and passed, and the artifact, pinned digest and cadence are now committed **[AMENDED N-1/N-2/N-4]** **Layer:** M6 / MV · **Leverage: 5** The always-on process is built; the falsifiable harness (H1–H4, holds/bites pairs, vacuity-guarded) is built; **and it was run.** `evals/l10_always_on/contract.md` §"The measured result" records a **5000-beat soak with a reboot at beat 2500** (landed 2026-07-19 in `aed273b1`) in which all four predicates pass: `versor_condition` flat at `1.389e-07` across all 5000 beats, vault bounded at 6 entries, convergence at beat 1 with the 4999-beat tail at rest, reboot resuming the same life with derived learning intact. @@ -42,6 +42,16 @@ What remains owed is the **ceremony**, and it is this assessment's own §8 maint A recorded prose result with no pinned digest is **testimony, not evidence** — the same failure mode as the map, the ratchet, and the blueprint. +**CLOSED 2026-07-28 — R-9 (A) + R-2 (B), PR-11.** The debt was never the run (N-4): it was the ceremony. Now committed as `evals/l10_always_on/results/report-5000-6c67e88d.json`, digest `f814fd97…` **pinned** by `tests/test_l10_soak_evidence.py` on the gate — the contract's own closing instruction, unmet for nine days, finally met. All four predicates pass at 5000 beats with reboot at 2500. + +**The re-run's most useful result was not the pass.** Every *deterministic* fact reproduced exactly (vault bounded at 6, convergence at beat 1 with a 4999-beat tail, clean reboot with derived learning intact) while the machine-variant float moved from the prose's `1.389e-07` to `6.177e-08` — the first real evidence that excluding `versor_condition` from the digest was correct rather than convenient. + +**The cadence is a pin, not a clock (R-9 A).** It holds the SHA-256 of every source the soak attests and asks *"does this evidence still describe the code that ships?"* rather than *"when did we last run it?"* — because a nightly that does not run is green for the wrong reason (H-9). **It had already fired silently for six weeks:** the 2026-07-19 soak ran without `accrue_realized_knowledge` in the daemon profile, so from R-3 onward it attested a process that no longer existed. Had the pin existed, PR-5 would have failed it. Verified by sabotage: touching `always_on_daemon.py` turns it red. + +**R-2 B ruled and recorded *before* the re-run**, so the soak measured a decided question. Measured result: the lane carries **no wall-clock at all**, in cognition or telemetry — stronger than B requires. Kept rather than "completed": adding a timestamp to match a ruling's letter, then a pin to guard it, is a mechanism whose only purpose is to be guarded, and *"when was this run"* is already the artifact's commit date, stated once. + +**The H1–H4 pins are on the gate**, measured at **+21s** (20.95s holds, 0.46s bites). Shipping only the cheap bites was considered and rejected: the four holds are the only tests on any gate that run `chat/always_on`'s `run_continuous` end-to-end, and that loop just shipped a six-week silent divergence. + **Absorbed 2026-07-28 (R-7, PR-3) — the substrate-liveness ratchet's seven OPEN items.** The ratchet is now historical; every one of its open entries was L10-chained, which is this entry, so they live here rather than in a stale instrument: | Item | What it is | Chain | diff --git a/docs/assessment/50-execution-plan.md b/docs/assessment/50-execution-plan.md index 14df1b77..f5d25ebc 100644 --- a/docs/assessment/50-execution-plan.md +++ b/docs/assessment/50-execution-plan.md @@ -10,12 +10,12 @@ | PR-0 ruling packet (#140) · PR-1 record amendments · R-10 discharged (#138 merged) · **R-12 — both ADR amendments + the §5 verdict banner + the §6 retirement amendment** | PR-2 — CR-1/CR-2 design briefs that do not yet exist (**not** a ruling) | | **Track A — ADR-0252 §5 run to NO-GO**, criterion pre-registered, artifact + digest committed, **verdict ratified 2026-07-28** | PR-8 — a small ADR (H-3/G-20) · PR-10 — a refactor ADR (H-4) | | PR-4 membership + reachability ratchets · PR-6 three doctrine pins · PR-7 M2 trust table · PR-9 count-the-swallow · **PR-5 the flag register (R-3+R-4)** | PR-12 — an ADR **amendment** (R-13 authorizes it; the amendment is still owed) | -| PR-3 + PR-3b (**R-7**, Wave 1) · H-13 (served-marker defect) · H-8e (closure docstring) · G-22 (main was red; `CLAIMS.md` stale) | Track B — the fabrication ADR (standing instruction, unchanged) | -| **Closed:** G-6, G-7, G-8, G-9, G-15, G-22, H-6, H-7, H-8 (**all five instances**), H-11, H-13 · **Answered:** G-1 · **Discharged:** H-10 · **Pinned:** G-23 | Track C — invention, deliberately last | +| PR-3 + PR-3b (**R-7**, Wave 1) · **PR-11 the soak to committed evidence (R-9+R-2)** · H-13 · H-8e · G-22 · G-23 | Track B — the fabrication ADR (standing instruction, unchanged) | +| **Closed:** G-5, G-6, G-7, G-8, G-9, G-15, G-22, H-6, H-7, H-8 (**all five**), H-11, H-13 · **Answered:** G-1 · **Discharged:** H-10 · **Pinned:** G-23 | Track C — invention, deliberately last | **New this arc, not in the original assessment:** **N-8** (the §5 experiment had already returned GO twice, on unmerged branches, both unsound), **N-9** (the gate "drift" was a recorded decision — PR-4's pin 1 **withdrawn** rather than built, R-14 dissolved), **G-21** (the math reader decides **1.0%** of `holdout_dev/v1`), **G-22** (`main` was red before the arc, and `CLAIMS.md` published a superseded evidence digest for two days), **H-13**, **H-14**, **H-8e**. -**What has not moved, stated plainly:** comprehension breadth and proof-of-life. The reader is still 19 constructions wide and decides 1.0% of the held-out corpus; the 5000-beat soak is still prose with no pinned digest. Everything landed this arc is enforcement and evidence machinery. PR-11, PR-12, PR-14 and Track B are where capability moves — and after the rulings, **none of them is blocked on a question any more.** Three are blocked on work. +**What has not moved, stated plainly:** comprehension breadth. The reader is still 19 constructions wide and decides 1.0% of the held-out corpus. *(Proof-of-life has moved: the soak is committed evidence with a pinned digest and a change-triggered cadence as of PR-11.)* Everything landed this arc is enforcement and evidence machinery. PR-11, PR-12, PR-14 and Track B are where capability moves — and after the rulings, **none of them is blocked on a question any more.** Three are blocked on work. **The one strategic finding of the whole arc, stated once here so it is not lost in the registers:** every defect found this arc — G-22, H-13, H-14, N-8, N-9, the fourteen `PENDING` statuses, the four unused suite aliases, the fifteen unreachable ones — is a **record/reality divergence**, not a substrate defect. Nothing found says the geometry is wrong. The recurring class is **mechanisms whose failure state is indistinguishable from their success state**: a suite nobody runs, a pin nobody sees fail, a digest nobody re-derives, a verdict on an unmerged branch, a ruling adopted in conversation. That is what Waves 1–2 are for, and it is why the correct response to "too much going on" is deletion and enforcement rather than redesign. @@ -217,7 +217,7 @@ The packet's original order was by register number. That is the wrong axis: it i | 1 | ~~**R-12**~~ | Two ratified-ADR record amendments — ADR-0146 (daemon) and ADR-0252 ("34 organs" footnote) — **plus** the §5 verdict banner and the §6 retirement-condition amendment, landed in the same edit | **S** | **EXECUTED 2026-07-28.** Zero-risk, changed no decision. Closed **H-8(a)** and **H-8(b)**, and discharged Track A's outstanding obligation. | | 2 | ~~R-7~~ | ~~Register supersession~~ | — | **EXECUTED** — PR-3 + PR-3b, landed 2026-07-28. Listed so the order stays readable. | | 3 | ~~**R-3 + R-4**~~ | PR-5 — the flag register, all 32 booleans classified, profiles as units, the bidirectional pin, `accrue_realized_knowledge` into the daemon profile | **M** | **EXECUTED 2026-07-28.** Closed **G-6, G-8, H-6, H-8(c), H-8(d)** — and **H-8 in full**. The N-6 docstring correction **dissolved**: the docstrings were right and the code was incomplete. **Wave 4's hard gate is lifted.** | -| 4 | **R-9 + R-2** | PR-11 — the L10 soak to a committed artifact with a pinned `deterministic_digest`, the ruled cadence, and R-2's "now" paragraph written into the contract *before* the re-run | **M** | Proof of life. R-2 must land first inside the same PR or the soak answers a design question by accident (the packet's own reasoning for R-2). | +| 4 | ~~**R-9 + R-2**~~ | PR-11 — the soak re-run at 5000 beats under the **corrected** profile, artifact committed, digest pinned, cadence enforced by a source-hash pin, H1–H4 on the gate | **M** | **EXECUTED 2026-07-28.** Closed **G-5**. R-2 landed in the contract *before* the re-run, as designed. The staleness pin **would have caught PR-5** — verified by sabotage. | | 5 | **R-13** | PR-12 — distinct-evidence counting at the seal boundary + the honest re-count of 25 ratified bands, demotions applied, `test_volume_honesty.py` moved in the same PR | **L** | Needs an ADR **amendment** (0175/0263 lineage). R-13 authorizes the demotions; it does not write the amendment. Expect served capability to shrink — that is the mechanism working. | | 6 | **R-8** | PR-14 — the outcome-mix rule, then seal `curriculum_serve_ledger.json`. Four bands qualify on sealing (N-5) | **M** | Depends on PR-12's counting basis. This is the entire throughput frontier, one ruling wide. | | 7 | **R-1 · R-5 · R-6 · R-11** | Posture statements: efferent-action deferral with entry criterion (G-12) · the identity-enforcement discrimination bar (G-11) · non-text ingest deferral with the falsification bench as its standard (G-17) · **R-11 B — measure before gating** | **S** each | All four are prose with a criterion. None blocks anything; leaving them silent is the cost. They go last because they are the only items where lateness is cheap. | diff --git a/docs/assessment/50-rulings.md b/docs/assessment/50-rulings.md index 4eb41e58..6a1f69cf 100644 --- a/docs/assessment/50-rulings.md +++ b/docs/assessment/50-rulings.md @@ -9,7 +9,9 @@ **Also ratified:** the ADR-0252 §5 **NO-GO** verdict. **Execution order (corrected):** R-12 → R-7 → R-3+R-4 → R-9+R-2 → R-13 → R-8 → R-1/R-5/R-6/R-11. -**Executed so far:** R-7 (PR-3, PR-3b) · **R-12** (both ADR amendments + the §5 verdict banner + the §6 retirement-condition amendment) · **R-3 + R-4** (PR-5 — the flag register, the profile mechanism, and the daemon's fourth flag) — all landed 2026-07-28. +**Executed so far:** R-7 (PR-3, PR-3b) · **R-12** (both ADR amendments + the §5 verdict banner + the §6 retirement-condition amendment) · **R-3 + R-4** (PR-5 — the flag register, the profile mechanism, the daemon's fourth flag) · **R-9 + R-2** (PR-11 — the soak's committed artifact, pinned digest, and change-triggered cadence) — all landed 2026-07-28. + +**Remaining:** R-13 → R-8 → R-1/R-5/R-6/R-11. --- @@ -58,7 +60,7 @@ ## R-2 · CR-4 temporal self-location — the stance on "now" -**Status:** **RULED B** — 2026-07-28, Shay (explicit adoption) · *beats govern cognition; wall-clock in telemetry only, pinned* · **Register:** G-13 · **Blocks:** PR-11 (the soak's contract paragraph). +**Status:** **RULED B — EXECUTED 2026-07-28 (PR-11)** · *beats govern cognition; wall-clock permitted in telemetry only.* Recorded in the soak contract **before** the re-run, as designed. Measured: the lane carries **no wall-clock at all** — stronger than B requires, and kept rather than "completed" (see the contract's §"The stance on 'now'") · **Register:** G-13 · **Blocks:** PR-11 (the soak's contract paragraph). **The question.** A continuous life experiences sequence and duration; determinism bans clocks from cognition. Both commitments are correct and their intersection is undesigned. The 24h+ no-drift requirement cannot be *stated* precisely without a stance. @@ -213,7 +215,7 @@ Two flags' documentation describes a production profile the production profile d ## R-9 · The soak's evidence standard and cadence -**Status:** **RULED A** — 2026-07-28, Shay (explicit adoption) · *commit the artifact, pin the digest, change-triggered cadence* · **Register:** G-5 · **Blocks:** PR-11. +**Status:** **RULED A — EXECUTED 2026-07-28 (PR-11)** · *artifact committed, digest pinned, cadence enforced by a source-hash pin rather than a clock; H1–H4 promoted onto the gate at a measured +21s* · **Register:** G-5 · **Blocks:** PR-11. **The question changed shape since the register was written (N-4).** G-5 says *"the soak has never produced an artifact."* It has: `evals/l10_always_on/contract.md` §"The measured result" records a **5000-beat soak with a reboot at beat 2500** (landed 2026-07-19 in `aed273b1`) in which **all four predicates pass** — `versor_condition` flat at `1.389e-07` across all 5000 beats, vault bounded at 6 entries, convergence at beat 1 with the 4999-beat tail at rest, reboot resuming the same life with derived learning intact. diff --git a/evals/l10_always_on/contract.md b/evals/l10_always_on/contract.md index 03449398..f49e549c 100644 --- a/evals/l10_always_on/contract.md +++ b/evals/l10_always_on/contract.md @@ -14,6 +14,46 @@ do **not** prove at horizon. Run: `PYTHONPATH=. .venv/bin/python -m evals.l10_always_on [n_beats] [reboot_beat]` (pass a large `n_beats` — e.g. `100000` — for a true long-horizon soak; default `24`). +## The stance on "now" (R-2, ruled B — recorded here *before* the re-run, deliberately) + +A continuous life experiences sequence and duration; determinism bans clocks from cognition. +Both commitments are correct, and their intersection was undesigned — which meant this lane's +central claim, *"holds over indefinite uptime,"* could not be stated precisely. It is stated +here, before the soak runs, so that the soak measures a decided question rather than deciding +one by accident. + +**Beats are the unit of cognitive time.** "Now" is the current beat index; "before" is a lower +index; "how long" is a beat count. No cognitive path reads a wall clock, and the H1–H4 +predicates are defined over beat indices alone. The Fibonacci recency schedule (τ_n, +ADR-0242) is a constants schedule, not a clock; `turn_count` and idle ticks are ordinals, not +durations. + +**R-2 was ruled B** — *beats govern cognition; a wall-clock stamp is permitted in telemetry +only, never read by any cognitive path.* Measured at 2026-07-28, **this lane carries no +wall-clock at all**, in cognition or telemetry: `runner.py`, `predicates.py` and `report.py` +contain no `time`, `datetime`, or timestamp of any kind. That is stronger than B requires, and +it is kept rather than "completed," for three reasons stated so the absence is not later read +as an oversight: + +1. **B's permissive half is a permission, not an obligation.** Nothing in this lane needs a + duration. Adding a field so that the code matches the letter of a ruling — then adding a + pin to guard the field — is a mechanism whose only purpose is to be guarded. +2. **The cadence's question is already answered elsewhere, exactly once.** *"When was this last + run?"* is the commit date of the committed report artifact: immutable, verifiable, and + single-sourced. A `generated_at` inside the artifact would be a second statement of one + fact, drifting from the first the moment a report is regenerated and not committed — the + same defect this repository registered as G-23 the same week. +3. **It keeps the digest honestly deterministic.** `deterministic_digest` covers hardware-stable + evidence only (per-beat shape + verdicts). A timestamp would have to be excluded from it, + which means shipping a field the digest deliberately does not cover — a value in the + artifact that the artifact's own integrity check ignores. + +**What would change this.** If a predicate is ever added whose claim is genuinely about +*duration* rather than *sequence* — H5's resource-cost leg is the live candidate, since cost +per unit time is not cost per beat — then a wall-clock enters **telemetry**, excluded from the +digest, with a pin asserting no cognitive path reads it. That is B exercised, and B is already +ruled, so it needs no new decision — only the predicate that requires it. + ## Predicates | ID | Proves | Fails loudly when | Mutation-verified bite | @@ -47,5 +87,61 @@ the field-transition walk; this covers indefinite idle uptime). The `deterministic_digest` in the report freezes the per-beat shape (`did_work` / `field_valid` / learning counts / vault size — all deterministic) + the verdicts, excluding -the machine-variant raw `versor_condition` float. Pin it once the lane is trusted so a -regression flips it. +the machine-variant raw `versor_condition` float. ~~Pin it once the lane is trusted so a +regression flips it.~~ **Pinned 2026-07-28 — see below.** + +--- + +## The committed evidence, and the cadence that keeps it true (R-9, ruled A) + +**Artifact:** `evals/l10_always_on/results/report-5000-6c67e88d.json` +**Digest:** `f814fd97f367840b0a9d31a48cb9fb28b2ed01961606f7b1e0b4c3a3593972d6` +**Re-run:** `PYTHONPATH=. uv run python -m evals.l10_always_on 5000 2500` +**Pinned by:** `tests/test_l10_soak_evidence.py`, on the gate. + +For nine days this lane's result was **prose in this file** — a real 5000-beat pass with no +machine-readable artifact, no run SHA, no rerun path, and a digest that `report.py` computed +and nothing pinned. The instruction two paragraphs up asked for the pin and it was never +added. **A recorded prose result with no re-derivable artifact is testimony, not evidence**, +and this file is where that distinction was supposed to be enforced. + +### The 2026-07-28 re-run, and why the prose number moved + +| | recorded 2026-07-19 | re-run 2026-07-28 | +|---|---|---| +| `versor_condition` | "flat at `1.389e-07`" | worst **`6.177e-08`** | +| vault size (bounded) | 6 | **6** | +| convergence beat / tail | 1 / 4999 | **1 / 4999** | +| reboot resume + derived learning | clean | **clean** | +| all four predicates | pass | **pass** | + +**Every deterministic fact reproduced exactly; only the machine-variant float moved.** That is +the first real evidence that excluding `versor_condition` from the digest was the right call — +a digest covering it would have flipped on a machine change and taught every reader to ignore +it. The prose figure `1.389e-07` is therefore **superseded, not contradicted**: it was never a +reproducible quantity, and the artifact records the one that is. + +### The cadence is change-triggered, not clock-triggered + +R-9 ruled the cadence as *"before each arc close, and on any change to the always-on process"* — +**enforced by a pin, not by a schedule.** The local-first doctrine makes "nightly" a ruling +rather than a cron line (`AGENTS.md`: *"when the Mac is asleep or away, the Actions queue +waits… That is fine"*), and a nightly that does not run is green for the wrong reason — H-9's +mechanism exactly. + +So instead of asking *"when did we last run it?"* the pin asks the question that matters: +**does this evidence still describe the code that ships?** It holds the SHA-256 of every source +the soak attests — `chat/always_on.py`, `chat/always_on_daemon.py`, and the three harness +modules — and fails when any of them changes, with the message *re-run the soak; do not edit +the hashes.* + +**This is not a hypothetical guard. It had already fired, silently, for six weeks.** The +2026-07-19 soak ran with `accrue_realized_knowledge` **absent** from +`CONTINUOUS_LIFE_CONFIG_FLAGS`. R-3 added it on 2026-07-28, so from that moment the recorded +result attested a process that no longer existed — and nothing in the repository could notice. +Had this pin existed, PR-5 would have failed it. That is precisely the intended behaviour, and +it is why the cadence is tied to change rather than to a calendar. + +**"When was it last run" is answered by the artifact's commit date** — immutable, verifiable, +and stated once. A `generated_at` field inside the report would be a second copy of that one +fact, drifting the moment a report is regenerated and not committed. diff --git a/evals/l10_always_on/results/report-5000-6c67e88d.json b/evals/l10_always_on/results/report-5000-6c67e88d.json new file mode 100644 index 00000000..3d65b5c3 --- /dev/null +++ b/evals/l10_always_on/results/report-5000-6c67e88d.json @@ -0,0 +1,54 @@ +{ + "n_beats": 5000, + "reboot_beat": 2500, + "all_gates_pass": true, + "deterministic_digest": "f814fd97f367840b0a9d31a48cb9fb28b2ed01961606f7b1e0b4c3a3593972d6", + "predicates": [ + { + "name": "H1_closure", + "passed": true, + "detail": "all 5000 observed beats closed (worst=6.177e-08 < 1e-06)", + "metrics": { + "observed_beats": 5000, + "worst_versor_condition": 6.177101671250627e-08, + "violations": [] + } + }, + { + "name": "H2_bounded_idle", + "passed": true, + "detail": "no idle beat grew the vault (final size 6 over 5000 beats, monotonic=True)", + "metrics": { + "idle_leaks": [], + "final_vault_size": 6, + "vault_monotonic": true + } + }, + { + "name": "H3_convergence", + "passed": true, + "detail": "converged at beat 1; the 4999-beat tail stayed at rest with closure intact", + "metrics": { + "convergence_beat": 1, + "converged_tail_len": 4999, + "reawakenings": [] + } + }, + { + "name": "H4_reboot_resume", + "passed": true, + "detail": "reboot at 2500 resumed the SAME life: identity guard passed, learning survived, 2500 post-reboot beats closed", + "metrics": { + "resumed_cleanly": true, + "learned_fact_survived": true, + "post_reboot_beats": 2500 + } + } + ], + "not_covered": [ + { + "leg": "H5_learning_life_resource_cost", + "reason": "This lane proves the IDLE (converged) life is resource-bounded. The cost of a continuously-LEARNING life under a sustained NEW-fact stream (the full-snapshot checkpoint is O(n^2) in facts; lived_life.json is per-run) is out of scope until an afferent/intake feed and incremental persistence exist; deferred." + } + ] +} diff --git a/tests/full_only_baseline.txt b/tests/full_only_baseline.txt index bc95bb4e..d0db72ff 100644 --- a/tests/full_only_baseline.txt +++ b/tests/full_only_baseline.txt @@ -473,7 +473,6 @@ tests/test_issue_300_versor_margin.py tests/test_kernel_facts.py tests/test_kernel_no_new_legacy_derivation_surfaces.py tests/test_l10_always_on.py -tests/test_l10_always_on_soak.py tests/test_l10_arbitrary_interruption.py tests/test_l10_continuity.py tests/test_labeled_container_subject.py diff --git a/tests/test_l10_soak_evidence.py b/tests/test_l10_soak_evidence.py new file mode 100644 index 00000000..8c8783cc --- /dev/null +++ b/tests/test_l10_soak_evidence.py @@ -0,0 +1,181 @@ +"""PR-11 / G-5 / R-9 — the soak's evidence is committed, pinned, and still current. + +The L10 always-on lane has passed a 5000-beat soak since 2026-07-19. That result +lived as **prose in a contract file** — no machine-readable artifact, no run SHA, no +rerun path, and a ``deterministic_digest`` that ``report.py`` computed and nothing +pinned. The contract's own closing line asked for the pin (*"Pin it once the lane is +trusted so a regression flips it"*) and it was never added. A recorded prose result +with no re-derivable artifact is testimony, not evidence. + +This module makes it evidence, and it guards **two different failure modes** that are +easy to conflate: + +**1. Regression.** The committed report's ``deterministic_digest`` is pinned. If the +per-beat shape or any verdict changes, the digest changes and this fails. The digest +deliberately excludes the machine-variant raw ``versor_condition`` float — and the +2026-07-28 re-run is the first evidence that exclusion was correct: every +*deterministic* fact reproduced exactly (vault bounded at 6, convergence at beat 1 +with a 4999-beat tail, clean reboot resume with learning intact) while the float moved +from the prose's ``1.389e-07`` to ``6.177e-08``. A digest covering it would have +flipped on a machine change and taught everyone to ignore it. + +**2. Staleness — the mode nothing guarded, and the one that already fired.** +R-9 ruled the cadence to be **change-triggered, not clock-triggered**: the local-first +doctrine makes "nightly" a ruling rather than a cron line, because a nightly that does +not run is green for the wrong reason. So instead of asking *"when did we last run +it?"*, this pin asks the question that actually matters — ***does this evidence still +describe the code that ships?*** — by pinning the SHA-256 of every source the soak +attests. Change one, and the pin says the evidence predates the change. + +That is not hypothetical. It had **already happened, silently, for six weeks**: the +2026-07-19 soak ran with ``accrue_realized_knowledge`` absent from +``CONTINUOUS_LIFE_CONFIG_FLAGS``. R-3 added it on 2026-07-28 (PR-5), so the recorded +result stopped describing the shipping configuration — a soak result attesting a +process that no longer existed, with nothing anywhere able to notice. Had this pin +existed, PR-5 would have failed it, which is exactly the intended behaviour: *you +changed the always-on process; the soak evidence is now stale; re-run it.* + +**When this fails, the fix is to re-run the soak — never to edit the constants.** +Updating a hash to match changed source, without re-running, converts this pin into a +record that says evidence exists when it does not. That is the failure this entire arc +is about, and it would be committed inside its own remedy. +""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +_ROOT = Path(__file__).resolve().parents[1] +_REPORT = _ROOT / "evals" / "l10_always_on" / "results" / "report-5000-6c67e88d.json" + +#: The soak's verdict digest at 5000 beats, reboot at 2500, run 2026-07-28 under the +#: post-R-3 continuous-life profile. Covers the per-beat shape + the verdicts. +PINNED_DIGEST = "f814fd97f367840b0a9d31a48cb9fb28b2ed01961606f7b1e0b4c3a3593972d6" + +#: Everything whose change invalidates the evidence above. Two groups, both load-bearing: +#: the always-on process itself (what the soak measures) and the harness (what the digest +#: *means*). A harness change with an unchanged digest would be the more dangerous of the +#: two, because it looks like stability. +ATTESTED_SOURCES: dict[str, str] = { + "chat/always_on.py": "b733571e14fe540519861e9b7f24eb7be84782551dace43dab94cd96adf5d86b", + "chat/always_on_daemon.py": "d69f234fcc15568016056da05a47c91b4abe16a6d8fb87512eec02b5526bc062", + "evals/l10_always_on/runner.py": "9812cd549b41c36a8ee0477f7cb6282d5d9f110f931f3d770457a32aa54119f8", + "evals/l10_always_on/predicates.py": "ef8e44db809e8fd2270ffa0d5ff8890ee09eedfce9f95836ea08d99ff9c12e2f", + "evals/l10_always_on/report.py": "6c203af3eec6d78a1ddbfaadf3c7ad4ba18d0de0cc62a46227f210ca8b7f5277", +} + +_RERUN = "PYTHONPATH=. uv run python -m evals.l10_always_on 5000 2500" + + +def _report() -> dict: + return json.loads(_REPORT.read_text(encoding="utf-8")) + + +def test_the_committed_artifact_exists() -> None: + """No artifact, no evidence. G-5 reopens the moment this file is gone.""" + assert _REPORT.exists(), ( + f"the committed soak report is missing: {_REPORT.relative_to(_ROOT)}. " + f"Re-run and commit it: {_RERUN}" + ) + + +def test_the_artifact_records_the_horizon_it_claims() -> None: + """A 24-beat report pinned as though it were the 5000-beat one would pass every + other assertion here. The horizon *is* the claim.""" + report = _report() + assert report["n_beats"] == 5000, ( + f"the committed report is a {report['n_beats']}-beat run; the contract's claim " + "is the 5000-beat horizon" + ) + assert report["reboot_beat"] == 2500, ( + f"reboot leg at beat {report['reboot_beat']}, not 2500 — H4 proves resume " + "mid-soak, so where the reboot lands is part of the claim" + ) + + +def test_all_four_predicates_passed() -> None: + """H1-H4, by name. ``all_gates_pass`` alone would be satisfied by an empty + predicate list, which is the vacuity this repository keeps finding.""" + report = _report() + outcomes = {p["name"]: p["passed"] for p in report["predicates"]} + assert set(outcomes) == { + "H1_closure", + "H2_bounded_idle", + "H3_convergence", + "H4_reboot_resume", + }, f"the report's predicate set is not H1-H4: {sorted(outcomes)}" + failed = sorted(name for name, passed in outcomes.items() if not passed) + assert not failed, f"the committed soak report records FAILING predicates: {failed}" + assert report["all_gates_pass"] is True + + +def test_the_digest_is_pinned() -> None: + """The contract's own unmet closing instruction, finally met. + + If this fails, the soak's per-beat shape or a verdict changed. That is either a + regression or a deliberate improvement; either way it is a reviewed decision, which + is the whole point of pinning it. + """ + digest = _report()["deterministic_digest"] + assert digest == PINNED_DIGEST, ( + "the committed soak report's deterministic_digest does not match the pin.\n" + f" pinned: {PINNED_DIGEST}\n" + f" in report: {digest}\n" + "The per-beat shape or a verdict moved. Investigate before re-pinning." + ) + + +def test_the_evidence_still_describes_the_code_that_ships() -> None: + """R-9's change-triggered cadence, enforced as a pin rather than a clock. + + This is the assertion that would have caught the six-week divergence described in + this module's docstring. It fails when a source the soak attests has changed since + the soak ran. + """ + drifted: list[str] = [] + missing: list[str] = [] + for rel, expected in sorted(ATTESTED_SOURCES.items()): + path = _ROOT / rel + if not path.exists(): + missing.append(rel) + continue + actual = hashlib.sha256(path.read_bytes()).hexdigest() + if actual != expected: + drifted.append(f"{rel}\n attested {expected[:16]}…\n current {actual[:16]}…") + + assert not missing, ( + f"the soak attests sources that no longer exist: {missing}. The pin cannot " + "describe the current system; re-scope ATTESTED_SOURCES and re-run." + ) + assert not drifted, ( + "the committed soak evidence PREDATES a change to the code it attests, so it no " + "longer describes what ships:\n " + + "\n ".join(drifted) + + f"\n\n RE-RUN THE SOAK — do not edit the hashes:\n {_RERUN}\n" + " then commit the new report, update PINNED_DIGEST and ATTESTED_SOURCES\n" + " together, in one change. Editing a hash without re-running turns this pin\n" + " into a record asserting evidence that does not exist." + ) + + +def test_the_attested_set_is_not_vacuous() -> None: + """Guard the guard. + + An empty or truncated ATTESTED_SOURCES would make the staleness check pass on + nothing while still reporting success — the failure mode this module exists to + close, reproduced inside it. + """ + assert len(ATTESTED_SOURCES) >= 5, ( + f"only {len(ATTESTED_SOURCES)} sources attested; the staleness check is " + "narrower than the lane it guards" + ) + assert "chat/always_on_daemon.py" in ATTESTED_SOURCES, ( + "the daemon module carries CONTINUOUS_LIFE_CONFIG_FLAGS — the exact thing whose " + "silent change made the previous soak result stale. It cannot leave this set." + ) + assert all(len(h) == 64 for h in ATTESTED_SOURCES.values()), ( + "an attested hash is not a full SHA-256; a truncated hash weakens the check " + "silently" + )