From 38ee0854fcd448ace7672d517e93408541a91a10 Mon Sep 17 00:00:00 2001 From: Shay Date: Fri, 24 Jul 2026 16:58:05 -0700 Subject: [PATCH] =?UTF-8?q?fix(ci):=20re-pin=20public=5Fdemo=20lane=20?= =?UTF-8?q?=E2=80=94=20real=20content=20drift,=20not=20the=20env=20flake?= =?UTF-8?q?=20(S2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Archaeology (six worktree measurements) shows public_demo's stale pin (7d8ba0db..., last set 2026-07-14) predates two commits that changed its content deterministically: e7d116c9 (trace-hash format widening, benign) and e0d1b475 (the register-axis regression's introduction, benign for this lane at that point). But starting at 5a343b49 (T13's back-stamp, 2026-07-22) public_demo's own all_claims_supported case genuinely FAILS and stays failing through 5224b5e0 (2026-07-24), fixed only as a side effect of PR #110's unrelated register-axis restoration (f95ac26e) — public_demo embeds the same register-tour scene demo_composition does. The standing "public_demo red = the env-timeout flake, ignore it" memory was true only for the runtime_under_budget case; it caused this real all_claims_supported regression to hide for two days behind the same label. Corrected the memory to triage by which case failed, not by lane name. Re-pinned to da7fad65... (current main, 4/4 passing, verified stable across three independent measurements) and regenerated CLAIMS.md. --- CLAIMS.md | 2 +- .../lane-drift-investigation-2026-07-24.md | 10 ++ .../public-demo-lane-drift-2026-07-24.md | 110 ++++++++++++++++++ evals/public_demo/results/v1_dev.json | 2 +- scripts/verify_lane_shas.py | 2 +- 5 files changed, 123 insertions(+), 3 deletions(-) create mode 100644 docs/research/public-demo-lane-drift-2026-07-24.md diff --git a/CLAIMS.md b/CLAIMS.md index 51904dcb..b7b2182f 100644 --- a/CLAIMS.md +++ b/CLAIMS.md @@ -39,7 +39,7 @@ is a CI failure (`.github/workflows/lane-shas.yml`). | ADR-0095 | `miner_loop_closure` | Miner-sourced proposals route through single reviewed teaching path | `evals/miner_loop_closure/results/v1_dev.json` | `537094fe21d7e6cfbaf42bfc32b82d669fa9bb05a132d2bc93c72b3ceb7762a6` | | ADR-0096 | `fabrication_control_summary` | Phantom endpoints / cross-pack non-bridges / sibling collapses refuse | `evals/fabrication_control/results/v1_summary.json` | `01e1b6b711141f2b4a14551d7df3ea482d8d6dd7b364a25c509f4f8d08cda8a8` | | ADR-0098 | `demo_composition` | Demos compose from shipped modules; no parallel mechanism | `evals/demo_composition/results/v1_dev.json` | `f0611a2ce41721dd40767fc6a83a08470d3c7fd7fc8f1ae8ba003abf8a25ec97` | -| ADR-0099 | `public_demo` | Public showcase runs deterministically under 30s; all claims supported | `evals/public_demo/results/v1_dev.json` | `7d8ba0dbae9287cfe0bf15d231fa78a75abc627121c14900439293e01e1cc1d3` | +| ADR-0099 | `public_demo` | Public showcase runs deterministically under 30s; all claims supported | `evals/public_demo/results/v1_dev.json` | `da7fad654e77aac4573a6fcf6e9eaaf84540be8e135d2e033d9cfd15119df3fc` | | ADR-0104 | `curriculum_loop_closure` | Curriculum-sourced proposals route through single reviewed teaching path | `evals/curriculum_loop_closure/results/v1_dev.json` | `cb94ca0042d78ec2624129ff6493d52e767b69feea32d2997b85d88f1c0883af` | | ADR-0131 | `math_teaching_corpus_v1` | Math teaching corpus replays deterministically; all chains pass exit criterion (correct_rate=1.0, wrong=0) | `evals/math_teaching_corpus/v1/report.json` | `eaf160d145da29f9050ede8d58bf111b0f651dd40aeae9201857d0b97e014dd4` | | ADR-0206 | `deductive_logic_v1` | Propositional entailment scored against an independent truth-table oracle; dev+holdout+external 716/716 correct, wrong=0, refused=0 | `evals/deductive_logic/report.json` | `97a230949016e38d5e3f37a69e4245b320575ee70e5af92ff7607f7b05f74b5f` | diff --git a/docs/research/lane-drift-investigation-2026-07-24.md b/docs/research/lane-drift-investigation-2026-07-24.md index 0e203863..3bd5daae 100644 --- a/docs/research/lane-drift-investigation-2026-07-24.md +++ b/docs/research/lane-drift-investigation-2026-07-24.md @@ -108,6 +108,16 @@ Unchanged: its known failure mode is the env-timeout flake memory and the lane-shas remediation text. Do not re-pin from a laptop run; adjudicate on the Act runner. +**Correction (Tier S, 2026-07-24):** this framing was incomplete. `public_demo` +carries the SAME register-tour scene as `demo_composition` and was +independently root-caused as drifted for the same reasons documented above +(`e7d116c9` benignly, then `e0d1b475`/`5a343b49` as a REAL two-day +`all_claims_supported` regression, fixed by the same `f95ac26e`) — the +env-timeout flake is real but is a different failure mode +(`runtime_under_budget`) than the one that was actually red. Full +archaeology, measured timeline, and the re-pin: +`docs/research/public-demo-lane-drift-2026-07-24.md`. + ## Bonus latent break found while re-pinning `scripts/generate_claims.py` raised on every run since the diff --git a/docs/research/public-demo-lane-drift-2026-07-24.md b/docs/research/public-demo-lane-drift-2026-07-24.md new file mode 100644 index 00000000..b72097d1 --- /dev/null +++ b/docs/research/public-demo-lane-drift-2026-07-24.md @@ -0,0 +1,110 @@ +# `public_demo` Lane Drift — Root-Caused, Re-Pinned + +**Tier S2** (generalization-arc-2026-07-24 §6). Corrects a standing +mischaracterization: `docs/research/lane-drift-investigation-2026-07-24.md` +(the Phase-0.1 investigation this arc opened with) treated `public_demo` as +"not one of the three" drifted lanes, unchanged, its failures explained by +"the known env-timeout flake." That framing is only half right — a +**different, real** flake exists (§4), but the specific drift measured this +session is deterministic content drift with a two-day window of genuine +regression, exactly the class of thing the original investigation found for +`demo_composition`. It was never investigated because every red +`public_demo` in the lane-shas job was reflexively attributed to the flake. + +## 1. Method + +`scripts/verify_lane_shas.py --update` is never used to diagnose drift (it +rewrites every lane and silently drops erroring lanes' pins — the standing +rule three arcs now depend on). Instead: disposable `git worktree`s at six +specific commits, each with its own `uv sync --locked` venv, running +`evals/public_demo/runner.py` directly and reading its printed `sha256` — +the same measurement `verify_lane_shas.py` performs, isolated to one lane +so six historical points could be checked without six full-suite runs. + +## 2. Measured timeline + +| commit | date | change | `public_demo` sha256 | `all_claims_supported` | +|---|---|---|---|---| +| `4b9dbe72` (= `e7d116c9~1`) | pre-2026-07-20 | (pin's origin state) | `7d8ba0db…` **(= stale pin)** | true | +| `e7d116c9` | 2026-07-20 | Cl(4,1) geometric sovereignty convergence — widened `compute_trace_hash` (Phase-B graph-topology inclusion), same commit the ORIGINAL investigation found for `demo_composition`'s drift | `3434f255…` | true | +| `e0d1b475` | 2026-07-20 | #96 fail-closed linguistic governance — introduced the register-axis regression (`_base_runtime_surface` canonical-first) | `228f5287…` | true | +| `5a343b49` | 2026-07-22 | T13 "fix" — back-stamped the REGRESSED surface onto `TurnEvent`, making telemetry consistent with the wrong bytes | `31d9db30…` | **false** | +| `5224b5e0` | 2026-07-24 | (PR #110's base — Band v4-CM merge; regression still present) | `c6596e11…` **(matches BRIEF-S's "register-axis-reverted" baseline exactly)** | **false** | +| `f95ac26e` (→ current `main`) | 2026-07-24 | PR #110's fix — restored response-first surface precedence | `da7fad65…` **(= today, re-pinned below)** | true | + +Each row measured once except `4b9dbe72`/`e7d116c9`/current-`main`, which +independently reproduce values already stated in this session's prior +work — three independent confirmations, not a single sample. + +## 3. Two findings, not one + +### 3.1 Benign digest drift (`7d8ba0db…` → `228f5287…`) + +`e7d116c9` (trace-hash format widening) and `e0d1b475` (register-axis +regression's *introduction*) both changed `public_demo`'s content-derived +digest while `all_claims_supported` stayed **true**. This half matches the +original investigation's framing for `demo_composition`: intentional or at +least non-breaking content change, stale pin, no functional loss. + +### 3.2 A real, two-day regression window (`5a343b49` → `f95ac26e`) + +Starting at `5a343b49` (2026-07-22, the T13 back-stamp), `public_demo`'s own +`all_claims_supported` case **fails** — not a digest mismatch, an actual +`passed: 3/4` with `FAIL all_claims_supported`. It stays failing through +`5224b5e0` (2026-07-24, PR #110's base) and is fixed only as a side effect +of `f95ac26e` (PR #110's register-axis restoration, aimed at +`demo_composition`'s register tour, not `public_demo` specifically — see +[[project-register-axis-serving-regression]]). **`public_demo` embeds a +register-tour-sensitive scene** (`core/demos/showcase.py` imports +`RegisterTourDemo` from `core.demos.tour_adapters`, the same adapter +`demo_composition` uses), so it broke and healed on exactly the same +mechanism, in parallel, unnoticed. + +For roughly two days of active development (Band v2-EN's Phase 1 spine, +the coherence hedge-arm commit, PRs #107–#109), `public_demo` was +genuinely red on every commit in that window. Per the original +investigation's own §"why no gate caught it" for `demo_composition`: *"the +lane-shas CI job is the only gate that runs the demo lanes, and its +failures were conflated with the known `public_demo` env flake."* This is +the same masking, on the fourth lane, undiagnosed until now. + +## 4. The 2026-05-31 env-timeout flake is a SEPARATE, still-real thing + +`feedback-env-core-rs-build-and-public-demo-flake` (memory, 2026-05-31) +documents a genuine wall-clock budget overrun: +`DemoContractError: showcase exceeded ADR-0099 runtime budget: ~46-48s > +30000 ms`, reproduced identically on clean `main`, unrelated to content. +That failure mode is real and this document does not retract it — it is a +different assertion (`runtime_under_budget`, and only under +`CORE_SHOWCASE_HARD_BUDGET=1`; the soft path measured throughout this +session never hard-fails on wall-clock) from the one this document +root-causes (`all_claims_supported`, a content assertion). **The error was +conflating two distinct failure modes under one label** — any future red +`public_demo` should be triaged by WHICH case failed +(`all_claims_supported` vs `runtime_under_budget` vs +`determinism_run_to_run_byte_equality` vs `pure_composition_no_new_mechanism`), +never assumed to be either flake without reading the failing case id. + +## 5. Fix (this branch) + +- `scripts/verify_lane_shas.py`: `public_demo` pin + `7d8ba0dbae9287cf…` → `da7fad654e77aac4…` (single-line, surgical — the + standing rule, not `--update`). +- `evals/public_demo/results/v1_dev.json` regenerated (the committed report + the pin is computed from). +- `CLAIMS.md` regenerated (`scripts/generate_claims.py`, one-line diff — + the `ADR-0099` row's sha256). +- Current `main` (`5fd9a861`) is 4/4 passing, `da7fad65…`, verified stable + across three independent measurement points in this document. + +## 6. Follow-up recorded, not actioned here + +Per the original investigation's own follow-up list: *"document the +lane-shas CI expectation that a red lane job is triaged per-lane, never +assumed to be the public_demo flake"* — §4 above is that documentation. +No CI wiring change is proposed; this is a pin correction plus a corrected +record. + +Relates to [[project-register-axis-serving-regression]], +[[project-generalization-arc]], +`docs/research/lane-drift-investigation-2026-07-24.md`. diff --git a/evals/public_demo/results/v1_dev.json b/evals/public_demo/results/v1_dev.json index 40401e5e..3f7ad619 100644 --- a/evals/public_demo/results/v1_dev.json +++ b/evals/public_demo/results/v1_dev.json @@ -13,7 +13,7 @@ { "case_id": "determinism_run_to_run_byte_equality", "details": { - "sha256": "7f03086d43869979b1bebd15f4f58a8b3887c907f7fbe0e7d4b8cad33b7507f2" + "sha256": "dbef579f2e1342085b9a9699a9b9f7d47d6aeaf519bed681938c7b36c544ba6f" }, "divergence": null, "passed": true diff --git a/scripts/verify_lane_shas.py b/scripts/verify_lane_shas.py index 29b7931a..4de0290c 100644 --- a/scripts/verify_lane_shas.py +++ b/scripts/verify_lane_shas.py @@ -48,7 +48,7 @@ PINNED_SHAS: dict[str, str] = { "domain_contract_validation": "98ace04e3f02bbc5a8ad655bb6593c3f1ee64cb67014f1122fe6c3c85f48d22f", "fabrication_control_summary": "01e1b6b711141f2b4a14551d7df3ea482d8d6dd7b364a25c509f4f8d08cda8a8", "demo_composition": "f0611a2ce41721dd40767fc6a83a08470d3c7fd7fc8f1ae8ba003abf8a25ec97", - "public_demo": "7d8ba0dbae9287cfe0bf15d231fa78a75abc627121c14900439293e01e1cc1d3", + "public_demo": "da7fad654e77aac4573a6fcf6e9eaaf84540be8e135d2e033d9cfd15119df3fc", "math_teaching_corpus_v1": "eaf160d145da29f9050ede8d58bf111b0f651dd40aeae9201857d0b97e014dd4", "deductive_logic_v1": "97a230949016e38d5e3f37a69e4245b320575ee70e5af92ff7607f7b05f74b5f", "deduction_serve_v1": "0b461a5a49c8f8260ca87d0c9c9f9a17232bd1fdedd982e34649eedf9cca30b5",