core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

fix(ci): re-pin public_demo lane — real content drift, not the env flake (S2)

Browse source 
Archaeology (six worktree measurements) shows public_demo's stale pin
(7d8ba0db..., last set 2026-07-14) predates two commits that changed its
content deterministically: e7d116c9 (trace-hash format widening, benign)
and e0d1b475 (the register-axis regression's introduction, benign for
this lane at that point). But starting at 5a343b49 (T13's back-stamp,
2026-07-22) public_demo's own all_claims_supported case genuinely FAILS
and stays failing through 5224b5e0 (2026-07-24), fixed only as a side
effect of PR #110's unrelated register-axis restoration (f95ac26e) —
public_demo embeds the same register-tour scene demo_composition does.

The standing "public_demo red = the env-timeout flake, ignore it" memory
was true only for the runtime_under_budget case; it caused this real
all_claims_supported regression to hide for two days behind the same
label. Corrected the memory to triage by which case failed, not by lane
name. Re-pinned to da7fad65... (current main, 4/4 passing, verified
stable across three independent measurements) and regenerated CLAIMS.md.
This commit is contained in:
Shay 2026-07-24 16:58:05 -07:00
parent 5fd9a861db
commit 38ee0854fc
5 changed files with 123 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CLAIMS.md
View file

@ -39,7 +39,7 @@ is a CI failure (`.github/workflows/lane-shas.yml`).
| ADR-0095 | `miner_loop_closure` | Miner-sourced proposals route through single reviewed teaching path | `evals/miner_loop_closure/results/v1_dev.json` | `537094fe21d7e6cfbaf42bfc32b82d669fa9bb05a132d2bc93c72b3ceb7762a6` |
| ADR-0096 | `fabrication_control_summary` | Phantom endpoints / cross-pack non-bridges / sibling collapses refuse | `evals/fabrication_control/results/v1_summary.json` | `01e1b6b711141f2b4a14551d7df3ea482d8d6dd7b364a25c509f4f8d08cda8a8` |
| ADR-0098 | `demo_composition` | Demos compose from shipped modules; no parallel mechanism | `evals/demo_composition/results/v1_dev.json` | `f0611a2ce41721dd40767fc6a83a08470d3c7fd7fc8f1ae8ba003abf8a25ec97` |
| ADR-0099 | `public_demo` | Public showcase runs deterministically under 30s; all claims supported | `evals/public_demo/results/v1_dev.json` | `7d8ba0dbae9287cfe0bf15d231fa78a75abc627121c14900439293e01e1cc1d3` |
| ADR-0099 | `public_demo` | Public showcase runs deterministically under 30s; all claims supported | `evals/public_demo/results/v1_dev.json` | `da7fad654e77aac4573a6fcf6e9eaaf84540be8e135d2e033d9cfd15119df3fc` |
| ADR-0104 | `curriculum_loop_closure` | Curriculum-sourced proposals route through single reviewed teaching path | `evals/curriculum_loop_closure/results/v1_dev.json` | `cb94ca0042d78ec2624129ff6493d52e767b69feea32d2997b85d88f1c0883af` |
| ADR-0131 | `math_teaching_corpus_v1` | Math teaching corpus replays deterministically; all chains pass exit criterion (correct_rate=1.0, wrong=0) | `evals/math_teaching_corpus/v1/report.json` | `eaf160d145da29f9050ede8d58bf111b0f651dd40aeae9201857d0b97e014dd4` |
| ADR-0206 | `deductive_logic_v1` | Propositional entailment scored against an independent truth-table oracle; dev+holdout+external 716/716 correct, wrong=0, refused=0 | `evals/deductive_logic/report.json` | `97a230949016e38d5e3f37a69e4245b320575ee70e5af92ff7607f7b05f74b5f` |

10
docs/research/lane-drift-investigation-2026-07-24.md
View file

@ -108,6 +108,16 @@ Unchanged: its known failure mode is the env-timeout flake
memory and the lane-shas remediation text. Do not re-pin from a laptop
run; adjudicate on the Act runner.
**Correction (Tier S, 2026-07-24):** this framing was incomplete. `public_demo`
carries the SAME register-tour scene as `demo_composition` and was
independently root-caused as drifted for the same reasons documented above
(`e7d116c9` benignly, then `e0d1b475`/`5a343b49` as a REAL two-day
`all_claims_supported` regression, fixed by the same `f95ac26e`) — the
env-timeout flake is real but is a different failure mode
(`runtime_under_budget`) than the one that was actually red. Full
archaeology, measured timeline, and the re-pin:
`docs/research/public-demo-lane-drift-2026-07-24.md`.
## Bonus latent break found while re-pinning
`scripts/generate_claims.py` raised on every run since the

110
docs/research/public-demo-lane-drift-2026-07-24.md Normal file
View file

@ -0,0 +1,110 @@
# `public_demo` Lane Drift — Root-Caused, Re-Pinned
**Tier S2** (generalization-arc-2026-07-24 §6). Corrects a standing
mischaracterization: `docs/research/lane-drift-investigation-2026-07-24.md`
(the Phase-0.1 investigation this arc opened with) treated `public_demo` as
"not one of the three" drifted lanes, unchanged, its failures explained by
"the known env-timeout flake." That framing is only half right — a
**different, real** flake exists (§4), but the specific drift measured this
session is deterministic content drift with a two-day window of genuine
regression, exactly the class of thing the original investigation found for
`demo_composition`. It was never investigated because every red
`public_demo` in the lane-shas job was reflexively attributed to the flake.
## 1. Method
`scripts/verify_lane_shas.py --update` is never used to diagnose drift (it
rewrites every lane and silently drops erroring lanes' pins — the standing
rule three arcs now depend on). Instead: disposable `git worktree`s at six
specific commits, each with its own `uv sync --locked` venv, running
`evals/public_demo/runner.py` directly and reading its printed `sha256` —
the same measurement `verify_lane_shas.py` performs, isolated to one lane
so six historical points could be checked without six full-suite runs.
## 2. Measured timeline
| commit | date | change | `public_demo` sha256 | `all_claims_supported` |
|---|---|---|---|---|
| `4b9dbe72` (= `e7d116c9~1`) | pre-2026-07-20 | (pin's origin state) | `7d8ba0db…` **(= stale pin)** | true |
| `e7d116c9` | 2026-07-20 | Cl(4,1) geometric sovereignty convergence — widened `compute_trace_hash` (Phase-B graph-topology inclusion), same commit the ORIGINAL investigation found for `demo_composition`'s drift | `3434f255…` | true |
| `e0d1b475` | 2026-07-20 | #96 fail-closed linguistic governance — introduced the register-axis regression (`_base_runtime_surface` canonical-first) | `228f5287…` | true |
| `5a343b49` | 2026-07-22 | T13 "fix" — back-stamped the REGRESSED surface onto `TurnEvent`, making telemetry consistent with the wrong bytes | `31d9db30…` | **false** |
| `5224b5e0` | 2026-07-24 | (PR #110's base — Band v4-CM merge; regression still present) | `c6596e11…` **(matches BRIEF-S's "register-axis-reverted" baseline exactly)** | **false** |
| `f95ac26e` (→ current `main`) | 2026-07-24 | PR #110's fix — restored response-first surface precedence | `da7fad65…` **(= today, re-pinned below)** | true |
Each row measured once except `4b9dbe72`/`e7d116c9`/current-`main`, which
independently reproduce values already stated in this session's prior
work — three independent confirmations, not a single sample.
## 3. Two findings, not one
### 3.1 Benign digest drift (`7d8ba0db…` → `228f5287…`)
`e7d116c9` (trace-hash format widening) and `e0d1b475` (register-axis
regression's *introduction*) both changed `public_demo`'s content-derived
digest while `all_claims_supported` stayed **true**. This half matches the
original investigation's framing for `demo_composition`: intentional or at
least non-breaking content change, stale pin, no functional loss.
### 3.2 A real, two-day regression window (`5a343b49` → `f95ac26e`)
Starting at `5a343b49` (2026-07-22, the T13 back-stamp), `public_demo`'s own
`all_claims_supported` case **fails** — not a digest mismatch, an actual
`passed: 3/4` with `FAIL all_claims_supported`. It stays failing through
`5224b5e0` (2026-07-24, PR #110's base) and is fixed only as a side effect
of `f95ac26e` (PR #110's register-axis restoration, aimed at
`demo_composition`'s register tour, not `public_demo` specifically — see
[[project-register-axis-serving-regression]]). **`public_demo` embeds a
register-tour-sensitive scene** (`core/demos/showcase.py` imports
`RegisterTourDemo` from `core.demos.tour_adapters`, the same adapter
`demo_composition` uses), so it broke and healed on exactly the same
mechanism, in parallel, unnoticed.
For roughly two days of active development (Band v2-EN's Phase 1 spine,
the coherence hedge-arm commit, PRs #107–#109), `public_demo` was
genuinely red on every commit in that window. Per the original
investigation's own §"why no gate caught it" for `demo_composition`: *"the
lane-shas CI job is the only gate that runs the demo lanes, and its
failures were conflated with the known `public_demo` env flake."* This is
the same masking, on the fourth lane, undiagnosed until now.
## 4. The 2026-05-31 env-timeout flake is a SEPARATE, still-real thing
`feedback-env-core-rs-build-and-public-demo-flake` (memory, 2026-05-31)
documents a genuine wall-clock budget overrun:
`DemoContractError: showcase exceeded ADR-0099 runtime budget: ~46-48s >
30000 ms`, reproduced identically on clean `main`, unrelated to content.
That failure mode is real and this document does not retract it — it is a
different assertion (`runtime_under_budget`, and only under
`CORE_SHOWCASE_HARD_BUDGET=1`; the soft path measured throughout this
session never hard-fails on wall-clock) from the one this document
root-causes (`all_claims_supported`, a content assertion). **The error was
conflating two distinct failure modes under one label** — any future red
`public_demo` should be triaged by WHICH case failed
(`all_claims_supported` vs `runtime_under_budget` vs
`determinism_run_to_run_byte_equality` vs `pure_composition_no_new_mechanism`),
never assumed to be either flake without reading the failing case id.
## 5. Fix (this branch)
- `scripts/verify_lane_shas.py`: `public_demo` pin
`7d8ba0dbae9287cf…` → `da7fad654e77aac4…` (single-line, surgical — the
standing rule, not `--update`).
- `evals/public_demo/results/v1_dev.json` regenerated (the committed report
the pin is computed from).
- `CLAIMS.md` regenerated (`scripts/generate_claims.py`, one-line diff —
the `ADR-0099` row's sha256).
- Current `main` (`5fd9a861`) is 4/4 passing, `da7fad65…`, verified stable
across three independent measurement points in this document.
## 6. Follow-up recorded, not actioned here
Per the original investigation's own follow-up list: *"document the
lane-shas CI expectation that a red lane job is triaged per-lane, never
assumed to be the public_demo flake"* — §4 above is that documentation.
No CI wiring change is proposed; this is a pin correction plus a corrected
record.
Relates to [[project-register-axis-serving-regression]],
[[project-generalization-arc]],
`docs/research/lane-drift-investigation-2026-07-24.md`.

2
evals/public_demo/results/v1_dev.json
View file

@ -13,7 +13,7 @@
{
"case_id": "determinism_run_to_run_byte_equality",
"details": {
"sha256": "7f03086d43869979b1bebd15f4f58a8b3887c907f7fbe0e7d4b8cad33b7507f2"
"sha256": "dbef579f2e1342085b9a9699a9b9f7d47d6aeaf519bed681938c7b36c544ba6f"
},
"divergence": null,
"passed": true

2
scripts/verify_lane_shas.py
View file

@ -48,7 +48,7 @@ PINNED_SHAS: dict[str, str] = {
"domain_contract_validation": "98ace04e3f02bbc5a8ad655bb6593c3f1ee64cb67014f1122fe6c3c85f48d22f",
"fabrication_control_summary": "01e1b6b711141f2b4a14551d7df3ea482d8d6dd7b364a25c509f4f8d08cda8a8",
"demo_composition": "f0611a2ce41721dd40767fc6a83a08470d3c7fd7fc8f1ae8ba003abf8a25ec97",
"public_demo": "7d8ba0dbae9287cfe0bf15d231fa78a75abc627121c14900439293e01e1cc1d3",
"public_demo": "da7fad654e77aac4573a6fcf6e9eaaf84540be8e135d2e033d9cfd15119df3fc",
"math_teaching_corpus_v1": "eaf160d145da29f9050ede8d58bf111b0f651dd40aeae9201857d0b97e014dd4",
"deductive_logic_v1": "97a230949016e38d5e3f37a69e4245b320575ee70e5af92ff7607f7b05f74b5f",
"deduction_serve_v1": "0b461a5a49c8f8260ca87d0c9c9f9a17232bd1fdedd982e34649eedf9cca30b5",