docs(public_demo): add known environment caveat for wall-clock budget flake

This commit is contained in:
Shay 2026-06-11 18:12:00 -07:00
parent 28e7324cbe
commit c2892fa466

View file

@ -40,3 +40,27 @@ contract and the underlying adapter byte-equality from ADR-0098).
## Exit code
Non-zero on any case whose actual outcome diverges from the case spec.
## Known Environment Caveat
**`runtime_under_budget` is environment-sensitive.**
The pinned artifact (`results/v1_dev.json`) records all 4 cases
passing, including `runtime_under_budget` (`divergence: null`).
However, live CI runs on slower or shared hardware have reproducibly
exceeded the 30s wall-clock budget (observed: 4748s) during PRs
#684, #685, #686, and #687 gate runs.
This is a timing flake, not a content regression:
- All content lanes match their pinned SHAs in every observed run.
- `determinism_run_to_run_byte_equality`, `all_claims_supported`,
and `pure_composition_no_new_mechanism` pass unconditionally.
- The lane was deliberately **not re-pinned** on slower-hardware runs
per standing guidance — the pinned artifact reflects a passing run
on reference dev hardware.
**Implication for evaluators:** if reproducing this lane in a
constrained or shared environment, a `runtime_under_budget` failure
is an infrastructure signal, not a correctness failure. All
behavioral and compositional invariants remain intact.