From c2892fa466cc872c91574ebdd32c5b1df591cc66 Mon Sep 17 00:00:00 2001 From: Shay Date: Thu, 11 Jun 2026 18:12:00 -0700 Subject: [PATCH] docs(public_demo): add known environment caveat for wall-clock budget flake --- evals/public_demo/contract.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/evals/public_demo/contract.md b/evals/public_demo/contract.md index 23341a88..ae75a8ba 100644 --- a/evals/public_demo/contract.md +++ b/evals/public_demo/contract.md @@ -40,3 +40,27 @@ contract and the underlying adapter byte-equality from ADR-0098). ## Exit code Non-zero on any case whose actual outcome diverges from the case spec. + +## Known Environment Caveat + +**`runtime_under_budget` is environment-sensitive.** + +The pinned artifact (`results/v1_dev.json`) records all 4 cases +passing, including `runtime_under_budget` (`divergence: null`). +However, live CI runs on slower or shared hardware have reproducibly +exceeded the 30s wall-clock budget (observed: 47–48s) during PRs +#684, #685, #686, and #687 gate runs. + +This is a timing flake, not a content regression: + +- All content lanes match their pinned SHAs in every observed run. +- `determinism_run_to_run_byte_equality`, `all_claims_supported`, + and `pure_composition_no_new_mechanism` pass unconditionally. +- The lane was deliberately **not re-pinned** on slower-hardware runs + per standing guidance — the pinned artifact reflects a passing run + on reference dev hardware. + +**Implication for evaluators:** if reproducing this lane in a +constrained or shared environment, a `runtime_under_budget` failure +is an infrastructure signal, not a correctness failure. All +behavioral and compositional invariants remain intact.