8f1903e8e7
* chore(evals, cli): contract standardization + bench --json stdout cleanliness
End-of-session shippability pass. Three concrete fixes:
1. core/cli.py — bench --json no longer pollutes stdout
Several bench paths call scripts.run_pulse.run_pulse which prints
verbose [pulse] traces unconditionally to stdout, breaking jq /
programmatic consumers of --json output.
New _bench_stdout_guard() redirects stdout → stderr for the
duration of the bench run when --json is set. Operator still sees
the pulse trace (on stderr), but --json consumers get a clean JSON
document on stdout. Applied to all four bench paths: cost,
articulation, default suite, and --suite all.
Verified: core bench --suite determinism --json now produces
parseable JSON; human path still shows 1140 [pulse] lines.
2. evals/{frontier_compare,realizer_guard}/contract.md (new)
core/contemplation/contract.md (new)
Each new contract follows the established pattern (37 contracts
already exist under evals/<lane>/contract.md):
- What it measures
- Why it matters (structural win)
- How to run
- How to read the output
- Pass criteria table
- When it has failed and why
- Runner / module layout
Coverage:
- frontier_compare: both Lane A (CORE-only suites) and Lane B
(cross-provider prompt_battery) with explicit guardrails
against mixing — operator asks for the wrong lane combination,
runner exits 2 with helpful error.
- realizer_guard: C1/C2 articulation safety boundary — synthetic
illegal candidates rejected directly by check_surface AND
former-bug runtime prompts now produce legal articulations.
- contemplation (ADR-0080): not under evals/ since it's runtime
infrastructure that consumes eval reports — contract lives at
core/contemplation/contract.md. Documents the read-only +
SPECULATIVE-only + deterministic-replay invariants and the
shared DiscoveryCandidateSink plumbing convergence (ADR-0080).
3. evals/CLAIMS.md — Tier 2 rows added
- frontier_compare Lane A: determinism.primary_score, max_versor_condition
- frontier_compare Lane B: prompt_battery.primary_score (CORE adapter),
cross-provider artifact persistence
- realizer_guard: all_claims_supported
- contemplation: SPECULATIVE-only invariant, deterministic replay,
additive sink path, no pack mutation (all CI-pinned by tests)
Verification
------------
$ core test --suite smoke -q
67 passed in 27.22s (no regression)
$ uv run pytest -q tests/test_contemplation_loop.py \
tests/test_contemplation_pipeline_convergence.py \
tests/test_frontier_compare_cross_provider.py
27 passed in 4.87s
$ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed
true (was: JSONDecodeError on prior [pulse] pollution)
* feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart
Stop-hook caught that #62 only covered contracts — the 929-line
report_viewer.html was never audited against the new cross-provider
report shape from #61. Two real gaps:
1. Lane-aware observation drawer
The drawer hardcoded Lane A (CORE-native) fields: surface,
grounding_source, anchor_lens_mode_label, versor_condition.
Lane B (cross-provider) observations carry different fields:
provider, model, elapsed_ms, error_type, error_message.
Loading a cross-provider report rendered only the surface row
with empty `grounding` — the provider + model + timing data
was unreachable without expanding "Show raw JSON".
Fix: detect Lane B (presence of `obs.provider`) and render the
appropriate field set. Lane A still renders identically (now
also surfaces trace_hash + register_id when present, which were
silently buried in the raw JSON before).
2. Pass-rate chart per suite
The summary strip showed one aggregate Primary % across all
suites, with no way to see WHICH suite is dragging the score.
Multi-suite runs (e.g. --suite all) had to expand each panel
individually to find the failing one.
Fix: new .passrate-chart element below the summary strip,
one horizontal bar per suite showing passed/total. All-pass =
solid green, all-fail = solid red, partial = green/red split
at the pass fraction. CSS only — no new dependencies.
3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar
shows the "side-by-side surface evidence across providers"
description when loading a Lane B report.
Verified
--------
- Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54)
- One <script> tag pair preserved
- Generated a real Lane B report via
`python -m evals.frontier_compare --provider core --suite prompt_battery`
for visual confirmation
Out of scope (noted for future PR)
----------------------------------
Sampled 3 `core demo` targets:
- register-tour: clean schema (all_claims_supported, claims, grid)
- audit-tour: both scene_1_* keys AND an empty scenes:[] array — inconsistent
- anti-regression: no all_claims_supported key, uses all_gates_held instead
Demo schema standardization deserves its own PR — operator tooling
would benefit from a uniform top-level success field across demos.
* docs(evals) + chore(demos): systematic audit + uniform success field
Stop-hook caught two real gaps after the contract+UI PR:
- demos had divergent success-field names (all_gates_held vs
learning_loop_closed vs claim_supported vs nested claims_supported)
- no systematic look at the 48 eval directories had been done
Both addressed concretely; remaining work captured in audit doc
rather than vaguely deferred.
1. Demo schema standardization — uniform all_claims_supported field
----------------------------------------------------------------------
All 9 ``core demo`` targets now emit a top-level
``all_claims_supported: bool`` field. Existing per-demo fields
(``all_gates_held``, ``learning_loop_closed``, ``claim_supported``,
nested ``claims_supported``) are preserved for backwards compat —
the new field is an alias derived from the demo's existing success
signal, not a replacement.
Operator tooling and the CI gate can now target
``all_claims_supported`` without knowing each demo's idiomatic
field name.
Files touched:
- evals/anti_regression/run_demo.py — adds AND of all_gates_held +
active_corpus_byte_identical
- evals/learning_loop/run_demo.py — adds AND of learning_loop_closed +
active_corpus_byte_identical
- scripts/publish_pack_measurements.py — adds AND of the three
entries in the nested claims_supported dict
- evals/long_context_cost/comparison_runner.py — adds alias for
claim_supported (singular)
The 5 demos already using ``all_claims_supported`` (audit-tour,
register-tour, anchor-lens-tour, orthogonality-tour, articulation)
are unchanged.
Verified across all 9 demos:
audit-tour : True
register-tour : True
anchor-lens-tour : True
orthogonality-tour : True
pack-measurements : True ← new alias
anti-regression : True ← new alias
learning-loop : True ← new alias
articulation : True
long-context-comparison : True ← new alias
2. docs/EVAL_AUDIT_2026-05-20.md — systematic 48-lane audit
------------------------------------------------------------
Replaces the "future PR" deferral with a concrete document.
Contains:
- Method (what was inspected for each lane).
- Summary (40/48 have contract.md; 18/48 have saved results;
empty results/ ≠ broken — most lanes regenerate on demand).
- Cross-provider relevance triage:
* 9 lanes are cross-provider-relevant and could benefit
from the prompt_battery-style adapter pattern (cognition,
english_fluency_ood, hebrew_fluency, koine_greek_fluency,
grammatical_coverage, inference_closure, multi_step_reasoning,
discourse_paragraph, foundational_*_ood, etc.).
* 29 lanes are CORE-only by design (versor closure, anchor
lens, identity divergence, provenance, etc.) — wiring
providers would be category-erroneous.
- Demo schema standardization status (this PR closes that).
- UI/UX coverage matrix.
- 5 concrete follow-up items, each focused enough for a single
PR, none requiring architectural change.
Regenerated reports
-------------------
evals/long_context_cost/results/comparison_v1.json and
evals/results/phase2_pack_measurements.json now contain the new
all_claims_supported field (auto-regenerated when validating the
schema change).
evals/frontier_compare/results/sample_core_promptbattery.json
added as a reference Lane B report so the new viewer always has
something to load on first open.
8.8 KiB
8.8 KiB
Eval Lane Audit — 2026-05-20
Why this exists
After landing ADR-0080 (contemplation pipeline), ADR-0082 (provider
adapters), and the cross-provider frontier_compare Lane B, a stop
hook caught that "evals/demos brought up-to-date with higher
expectations" was claimed without a systematic look at the 48 eval
directories. This document is that look. It's a prioritization
map, not a fix-everything checklist.
Method
For each evals/<lane>/ directory:
- Contract — does
<lane>/contract.mdexist? - Results — count of JSON files in
<lane>/results/. - CLI — invokable via
core eval <lane>orcore demo <lane>? - Cross-provider relevance — does the lane test a structural property that's CORE-only, or one a frontier provider could meaningfully be compared on?
Summary
- 48 eval directories (excluding
lab/,reports/,results/,__pycache__/). - 40 have
contract.md(83%). Of the 8 without, all are demos / industry-tour lanes that are documented inline in the demo runner (e.g.evals/audit_tour/run_tour.pycarries its own scene-level docs). Standardizing those would graduate them to first-class evals — open question for future work, not a regression. - 18 have ≥ 1 saved results file, 30 have zero. Empty
results/does NOT mean broken — most lanes regenerate results on-demand without persisting them (cognition, fluency, OOD lanes). Only the lanes that publish reproducible numbers to CLAIMS.md should persist.
Cross-provider relevance triage
The user's instruction was to wire competitor providers for relevant benchmarks. Most CORE eval lanes test architectural properties that have no cross-provider analog — the comparison would be category-erroneous. This section makes the split explicit.
Cross-provider-relevant (provider adapter would compare meaningfully)
| Lane | Why relevant | Wired today? |
|---|---|---|
frontier_compare Lane B (prompt_battery) |
Designed for it (ADR-0082) | ✅ yes, this session |
cognition |
Definition/intent quality on shared prompts | ❌ no — would need adapter abstraction |
english_fluency_ood |
Surface fluency on out-of-domain prompts | ❌ no |
hebrew_fluency, koine_greek_fluency |
Language coverage | ❌ no |
elementary_mathematics_ood, foundational_biology_ood, foundational_physics_ood, classical_literature_ood |
Knowledge breadth on OOD prompts | ❌ no |
grammatical_coverage |
Surface grammar quality | ❌ no |
inference_closure, multi_step_reasoning |
Multi-step reasoning closure | ❌ no |
discourse_paragraph |
Discourse-level coherence | ❌ no |
Wiring plan for these: mirror the prompt_battery shape from
frontier_compare/cross_provider.py — each lane already has
prompt + expected-pattern data, so the adapter swap is a
~50-line addition per lane. This is genuine future work, not
done this session.
CORE-only by design (provider comparison would be category-erroneous)
These lanes measure architectural invariants no transformer can structurally satisfy. Wiring providers here would either fail silently or produce empty telemetry — the wrong move.
| Lane | What it tests (uniquely CORE) |
|---|---|
frontier_compare Lane A (determinism, truth_lock, axis_orthogonality) |
trace_hash invariance, versor_condition, anchor-lens engagement |
adversarial_identity |
identity-axis rejection under prompt injection |
anti_regression |
three-gate defense against learning harmful chains |
articulation_of_status |
SPECULATIVE-marker articulation |
calibration |
refusal calibration tied to EpistemicStatus |
cold_start_grounding |
first-turn grounding without vault priming |
compositionality |
pack-grounded composition determinism |
compound_intent_decomposition |
intent classifier decomposition |
contradiction_detection |
CONTESTED transition on paired contradictions |
conversational_thread_coherence |
thread-anaphora over ChatRuntime |
cross_domain_transfer |
pack-bridged cross-domain grounding |
deterministic_fluency |
byte-identical replay across runs |
forward_semantic_control |
rotor-application correctness |
identity_divergence |
identity-pack swap divergence shape |
introspection |
self-report against actual runtime state |
learning_loop |
cold-turn → discovery → propose → accept loop |
long_context_cost |
exact CGA recall at N tokens (the structural-asymmetry claim) |
monotonic_learning |
no-regression invariant under teaching |
multi_agent_composition |
cross-runtime composition determinism |
multi_sentence_response |
discourse-planner spine |
provenance |
trace_hash + term provenance completeness |
realizer_guard |
C1/C2 articulation-legality boundary |
refusal_calibration |
EpistemicStatus-gated refusal markers |
sample_efficiency |
teaching-cost per added capability |
self_consistency_over_time |
session-state consistency across turns |
symbolic_logic |
rotor-composition correctness on logic prompts |
teaching_injection_resistance |
identity-adjacent injection rejection |
walkthrough_chain |
NARRATIVE intent multi-chain composition |
warmed_session_consistency |
session-warm vs cold-start equivalence |
zero_code_domain_acquisition |
pack-only domain capability |
Demo schema standardization
All 9 core demo targets now emit a uniform all_claims_supported: bool top-level field (this session). Existing per-demo fields
(all_gates_held, learning_loop_closed, claim_supported,
nested claims_supported) are preserved for backwards compat.
Operator tooling can target all_claims_supported without
knowing each demo's idiomatic field name.
Verified:
audit-tour : all_claims_supported = True
register-tour : all_claims_supported = True
anchor-lens-tour : all_claims_supported = True
orthogonality-tour : all_claims_supported = True
pack-measurements : all_claims_supported = True
anti-regression : all_claims_supported = True
learning-loop : all_claims_supported = True
articulation : all_claims_supported = True
long-context-comparison : all_claims_supported = True
UI/UX coverage
| Surface | State |
|---|---|
evals/frontier_compare/ui/report_viewer.html |
✅ Lane-aware drawer + pass-rate chart (this session) |
| Other lane HTML viewers | ❌ none exist; lanes produce JSON consumed by core eval text output |
core eval <lane> text output |
⚠ derives summary from JSON but no charts; CLAIMS.md is the cross-lane dashboard |
core demo text output |
⚠ each demo prints its own summary block; no unified format |
Practical recommendation: the right next UI investment is a
single multi-lane dashboard that loads any lane's
results/*.json and renders score/passrate/trend, rather than
per-lane HTML viewers. The frontier_compare/ui/report_viewer.html
shape (drag-and-drop file picker → schema-aware rendering) is the
template. Not done this session.
What was done this session (concrete)
| # | Status | Scope |
|---|---|---|
| 55 | merged | ADR-0080 read-only contemplation boundary |
| 57 | merged | φ separation probe (falsified across 8 variants) |
| 58 | merged | Pipeline convergence: shared sink, separate schemas |
| 59 | merged | Renamed dev's ADR-0081 → ADR-0082 |
| 60 | merged | Fixed INV-02 + wired core contemplation subcommand |
| 61 | merged | ADR-0082 providers wired into frontier_compare runner |
| 62 | open | Contracts + bench --json cleanliness + Lane B viewer + pass-rate chart + this audit + demo schema standardization |
What remains (concrete, not vague)
- Cross-provider wiring for the 9 cross-provider-relevant lanes listed
above. Per-lane work, ~50 lines each. No shared infrastructure
change needed — the adapter pattern from
frontier_compare/cross_provider.pyis reusable. - Multi-lane dashboard that loads any
results/*.jsonfrom any lane. Single HTML file, drag-and-drop, schema-aware rendering. - Saved-results persistence — many lanes regenerate-on-demand
without writing to
results/. CLAIMS.md numbers cannot be replay-verified without persistence. Per-lane CLI option to--save(some lanes already have this; standardize). - Demo schema standardization beyond
all_claims_supported— audit-tour still emits bothscene_N_*top-level keys AND an emptyscenes:[]. Pick one shape per demo, deprecate the other. - Contracts for the 8 demo lanes without
contract.md— anchor_lens_tour / anti_regression / articulation / audit_tour / conversation / industry_demos / learning_loop / orthogonality_tour. Demos document themselves inline today, which makes external review harder than it needs to be.
Each is a focused PR. None require architectural change.