From 8f1903e8e73ddba854e5165f9063f06c546af97b Mon Sep 17 00:00:00 2001 From: Shay Date: Wed, 20 May 2026 13:53:13 -0700 Subject: [PATCH] chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(evals, cli): contract standardization + bench --json stdout cleanliness End-of-session shippability pass. Three concrete fixes: 1. core/cli.py — bench --json no longer pollutes stdout Several bench paths call scripts.run_pulse.run_pulse which prints verbose [pulse] traces unconditionally to stdout, breaking jq / programmatic consumers of --json output. New _bench_stdout_guard() redirects stdout → stderr for the duration of the bench run when --json is set. Operator still sees the pulse trace (on stderr), but --json consumers get a clean JSON document on stdout. Applied to all four bench paths: cost, articulation, default suite, and --suite all. Verified: core bench --suite determinism --json now produces parseable JSON; human path still shows 1140 [pulse] lines. 2. evals/{frontier_compare,realizer_guard}/contract.md (new) core/contemplation/contract.md (new) Each new contract follows the established pattern (37 contracts already exist under evals//contract.md): - What it measures - Why it matters (structural win) - How to run - How to read the output - Pass criteria table - When it has failed and why - Runner / module layout Coverage: - frontier_compare: both Lane A (CORE-only suites) and Lane B (cross-provider prompt_battery) with explicit guardrails against mixing — operator asks for the wrong lane combination, runner exits 2 with helpful error. - realizer_guard: C1/C2 articulation safety boundary — synthetic illegal candidates rejected directly by check_surface AND former-bug runtime prompts now produce legal articulations. - contemplation (ADR-0080): not under evals/ since it's runtime infrastructure that consumes eval reports — contract lives at core/contemplation/contract.md. Documents the read-only + SPECULATIVE-only + deterministic-replay invariants and the shared DiscoveryCandidateSink plumbing convergence (ADR-0080). 3. evals/CLAIMS.md — Tier 2 rows added - frontier_compare Lane A: determinism.primary_score, max_versor_condition - frontier_compare Lane B: prompt_battery.primary_score (CORE adapter), cross-provider artifact persistence - realizer_guard: all_claims_supported - contemplation: SPECULATIVE-only invariant, deterministic replay, additive sink path, no pack mutation (all CI-pinned by tests) Verification ------------ $ core test --suite smoke -q 67 passed in 27.22s (no regression) $ uv run pytest -q tests/test_contemplation_loop.py \ tests/test_contemplation_pipeline_convergence.py \ tests/test_frontier_compare_cross_provider.py 27 passed in 4.87s $ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed true (was: JSONDecodeError on prior [pulse] pollution) * feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart Stop-hook caught that #62 only covered contracts — the 929-line report_viewer.html was never audited against the new cross-provider report shape from #61. Two real gaps: 1. Lane-aware observation drawer The drawer hardcoded Lane A (CORE-native) fields: surface, grounding_source, anchor_lens_mode_label, versor_condition. Lane B (cross-provider) observations carry different fields: provider, model, elapsed_ms, error_type, error_message. Loading a cross-provider report rendered only the surface row with empty `grounding` — the provider + model + timing data was unreachable without expanding "Show raw JSON". Fix: detect Lane B (presence of `obs.provider`) and render the appropriate field set. Lane A still renders identically (now also surfaces trace_hash + register_id when present, which were silently buried in the raw JSON before). 2. Pass-rate chart per suite The summary strip showed one aggregate Primary % across all suites, with no way to see WHICH suite is dragging the score. Multi-suite runs (e.g. --suite all) had to expand each panel individually to find the failing one. Fix: new .passrate-chart element below the summary strip, one horizontal bar per suite showing passed/total. All-pass = solid green, all-fail = solid red, partial = green/red split at the pass fraction. CSS only — no new dependencies. 3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar shows the "side-by-side surface evidence across providers" description when loading a Lane B report. Verified -------- - Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54) - One