* chore(evals, cli): contract standardization + bench --json stdout cleanliness
End-of-session shippability pass. Three concrete fixes:
1. core/cli.py — bench --json no longer pollutes stdout
Several bench paths call scripts.run_pulse.run_pulse which prints
verbose [pulse] traces unconditionally to stdout, breaking jq /
programmatic consumers of --json output.
New _bench_stdout_guard() redirects stdout → stderr for the
duration of the bench run when --json is set. Operator still sees
the pulse trace (on stderr), but --json consumers get a clean JSON
document on stdout. Applied to all four bench paths: cost,
articulation, default suite, and --suite all.
Verified: core bench --suite determinism --json now produces
parseable JSON; human path still shows 1140 [pulse] lines.
2. evals/{frontier_compare,realizer_guard}/contract.md (new)
core/contemplation/contract.md (new)
Each new contract follows the established pattern (37 contracts
already exist under evals/<lane>/contract.md):
- What it measures
- Why it matters (structural win)
- How to run
- How to read the output
- Pass criteria table
- When it has failed and why
- Runner / module layout
Coverage:
- frontier_compare: both Lane A (CORE-only suites) and Lane B
(cross-provider prompt_battery) with explicit guardrails
against mixing — operator asks for the wrong lane combination,
runner exits 2 with helpful error.
- realizer_guard: C1/C2 articulation safety boundary — synthetic
illegal candidates rejected directly by check_surface AND
former-bug runtime prompts now produce legal articulations.
- contemplation (ADR-0080): not under evals/ since it's runtime
infrastructure that consumes eval reports — contract lives at
core/contemplation/contract.md. Documents the read-only +
SPECULATIVE-only + deterministic-replay invariants and the
shared DiscoveryCandidateSink plumbing convergence (ADR-0080).
3. evals/CLAIMS.md — Tier 2 rows added
- frontier_compare Lane A: determinism.primary_score, max_versor_condition
- frontier_compare Lane B: prompt_battery.primary_score (CORE adapter),
cross-provider artifact persistence
- realizer_guard: all_claims_supported
- contemplation: SPECULATIVE-only invariant, deterministic replay,
additive sink path, no pack mutation (all CI-pinned by tests)
Verification
------------
$ core test --suite smoke -q
67 passed in 27.22s (no regression)
$ uv run pytest -q tests/test_contemplation_loop.py \
tests/test_contemplation_pipeline_convergence.py \
tests/test_frontier_compare_cross_provider.py
27 passed in 4.87s
$ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed
true (was: JSONDecodeError on prior [pulse] pollution)
* feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart
Stop-hook caught that #62 only covered contracts — the 929-line
report_viewer.html was never audited against the new cross-provider
report shape from #61. Two real gaps:
1. Lane-aware observation drawer
The drawer hardcoded Lane A (CORE-native) fields: surface,
grounding_source, anchor_lens_mode_label, versor_condition.
Lane B (cross-provider) observations carry different fields:
provider, model, elapsed_ms, error_type, error_message.
Loading a cross-provider report rendered only the surface row
with empty `grounding` — the provider + model + timing data
was unreachable without expanding "Show raw JSON".
Fix: detect Lane B (presence of `obs.provider`) and render the
appropriate field set. Lane A still renders identically (now
also surfaces trace_hash + register_id when present, which were
silently buried in the raw JSON before).
2. Pass-rate chart per suite
The summary strip showed one aggregate Primary % across all
suites, with no way to see WHICH suite is dragging the score.
Multi-suite runs (e.g. --suite all) had to expand each panel
individually to find the failing one.
Fix: new .passrate-chart element below the summary strip,
one horizontal bar per suite showing passed/total. All-pass =
solid green, all-fail = solid red, partial = green/red split
at the pass fraction. CSS only — no new dependencies.
3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar
shows the "side-by-side surface evidence across providers"
description when loading a Lane B report.
Verified
--------
- Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54)
- One <script> tag pair preserved
- Generated a real Lane B report via
`python -m evals.frontier_compare --provider core --suite prompt_battery`
for visual confirmation
Out of scope (noted for future PR)
----------------------------------
Sampled 3 `core demo` targets:
- register-tour: clean schema (all_claims_supported, claims, grid)
- audit-tour: both scene_1_* keys AND an empty scenes:[] array — inconsistent
- anti-regression: no all_claims_supported key, uses all_gates_held instead
Demo schema standardization deserves its own PR — operator tooling
would benefit from a uniform top-level success field across demos.
* docs(evals) + chore(demos): systematic audit + uniform success field
Stop-hook caught two real gaps after the contract+UI PR:
- demos had divergent success-field names (all_gates_held vs
learning_loop_closed vs claim_supported vs nested claims_supported)
- no systematic look at the 48 eval directories had been done
Both addressed concretely; remaining work captured in audit doc
rather than vaguely deferred.
1. Demo schema standardization — uniform all_claims_supported field
----------------------------------------------------------------------
All 9 ``core demo`` targets now emit a top-level
``all_claims_supported: bool`` field. Existing per-demo fields
(``all_gates_held``, ``learning_loop_closed``, ``claim_supported``,
nested ``claims_supported``) are preserved for backwards compat —
the new field is an alias derived from the demo's existing success
signal, not a replacement.
Operator tooling and the CI gate can now target
``all_claims_supported`` without knowing each demo's idiomatic
field name.
Files touched:
- evals/anti_regression/run_demo.py — adds AND of all_gates_held +
active_corpus_byte_identical
- evals/learning_loop/run_demo.py — adds AND of learning_loop_closed +
active_corpus_byte_identical
- scripts/publish_pack_measurements.py — adds AND of the three
entries in the nested claims_supported dict
- evals/long_context_cost/comparison_runner.py — adds alias for
claim_supported (singular)
The 5 demos already using ``all_claims_supported`` (audit-tour,
register-tour, anchor-lens-tour, orthogonality-tour, articulation)
are unchanged.
Verified across all 9 demos:
audit-tour : True
register-tour : True
anchor-lens-tour : True
orthogonality-tour : True
pack-measurements : True ← new alias
anti-regression : True ← new alias
learning-loop : True ← new alias
articulation : True
long-context-comparison : True ← new alias
2. docs/EVAL_AUDIT_2026-05-20.md — systematic 48-lane audit
------------------------------------------------------------
Replaces the "future PR" deferral with a concrete document.
Contains:
- Method (what was inspected for each lane).
- Summary (40/48 have contract.md; 18/48 have saved results;
empty results/ ≠ broken — most lanes regenerate on demand).
- Cross-provider relevance triage:
* 9 lanes are cross-provider-relevant and could benefit
from the prompt_battery-style adapter pattern (cognition,
english_fluency_ood, hebrew_fluency, koine_greek_fluency,
grammatical_coverage, inference_closure, multi_step_reasoning,
discourse_paragraph, foundational_*_ood, etc.).
* 29 lanes are CORE-only by design (versor closure, anchor
lens, identity divergence, provenance, etc.) — wiring
providers would be category-erroneous.
- Demo schema standardization status (this PR closes that).
- UI/UX coverage matrix.
- 5 concrete follow-up items, each focused enough for a single
PR, none requiring architectural change.
Regenerated reports
-------------------
evals/long_context_cost/results/comparison_v1.json and
evals/results/phase2_pack_measurements.json now contain the new
all_claims_supported field (auto-regenerated when validating the
schema change).
evals/frontier_compare/results/sample_core_promptbattery.json
added as a reference Lane B report so the new viewer always has
something to load on first open.
#58 shipped providers.py + model_registry.py for cross-provider
benchmarking but never connected them to runner.py — the adapters
sat unused. This PR wires them through with a clear lane split.
Why a new suite instead of refactoring existing ones
-----------------------------------------------------
The three existing suites (determinism / truth_lock / axis_orthogonality)
pull CORE-only telemetry: trace_hash, versor_condition, register_id,
register_variant_id, anchor_lens_id, register_canonical_surface.
None of those fields can come from OpenAI / Anthropic / Ollama.
Forcing those suites cross-provider would silently produce reports
where the cross-provider rows have empty telemetry — a worse failure
mode than not running them at all. So the routing is explicit:
CORE-only suites → --provider must be 'core'
Cross-provider suites → any provider; CORE is one adapter among many
Operator asks for the wrong combo → loud error with the right alternative.
New module: evals/frontier_compare/cross_provider.py
-----------------------------------------------------
- ProviderObservation dataclass — provider-agnostic observation shape
(prompt, surface, provider, model, elapsed_ms, error fields). No
CORE-internal telemetry expected.
- run_prompt_battery(adapter, *, cfg) → SuiteReport reusing existing
CaseResult / SuiteReport shapes so the report viewer renders both
lanes without schema branching.
- _PROMPT_BATTERY: 7 fixed cases spanning definition / cause /
verification / comparison / procedure / unknown intent shapes.
Stable case_ids so future re-runs against the same provider produce
diffable JSON.
- Per-case 'passed' is loose by design (non-empty surface, no
exception). Cross-provider quality is for human review — not for
the runner to silently score.
Updated CLI: evals/frontier_compare/__main__.py
-----------------------------------------------
- --provider {core, openai, anthropic, ollama} (default: core)
- --model <id> (validated via require_model_card)
- --env-file <path> (default: ./.env)
- Auto-persist non-CORE runs to
evals/frontier_compare/results/<provider>_<model>_<utc>.json
even when --report is omitted. API calls are rate-limited / paid;
losing the artifact is costly.
- Existing CORE-native behavior unchanged when --provider not set.
Results directory: evals/frontier_compare/results/
--------------------------------------------------
Created with .gitkeep — matches the convention used by other lanes
(evals/long_context_cost/results/, evals/koine_greek_fluency/results/,
etc.). Distinct from reports/ which .gitignore excludes for
transient debug output.
Tests: tests/test_frontier_compare_cross_provider.py (9 cases)
--------------------------------------------------------------
- prompt_battery runs with CORE adapter (no API needed)
- adapter exceptions recorded as failed observations, never propagated
- empty surfaces flagged distinctly from adapter errors
- CLI default runs CORE-native (no breaking change)
- CLI prompt_battery with --provider core routes through cross-provider path
- CLI rejects CORE-only suite + non-CORE provider with operator-helpful error
- --help surfaces both suite families
- unregistered model is rejected before any benchmark cycles burn
- ProviderObservation.succeeded handles error / empty / whitespace cases
Live evidence
-------------
$ core test --suite smoke -q
67 passed in 26.55s (no regression)
$ python -m evals.frontier_compare --provider core --suite prompt_battery --json
model=core-native mode=core suite=prompt_battery passed=True score=1.000
[definition_truth ] PASS Truth is a claim or state grounded by evidence...
[definition_knowledge ] PASS Knowledge is justified understanding grounded...
[cause_understanding ] PASS understanding — teaching-grounded (cognition_chains_v1)...
[verification_evidence ] PASS evidence — teaching-grounded (cognition_chains_v1)...
[comparison_knowledge_wisdom ] PASS knowledge contrasts with wisdom...
[procedure_recall ] PASS To recall means to retrieve a stored state from memory...
[unknown_term ] PASS I haven't learned 'xylomorphic' yet...
$ python -m evals.frontier_compare --provider openai --suite determinism
error: suite 'determinism' is CORE-only; pass --suite prompt_battery
(the cross-provider suite) when --provider='openai'.
.gitignore: adds frontier_wave1.json (stray report file repeatedly
written by ad-hoc test invocations).
Resolves a same-day numbering collision: the prior session produced
ADR-0080 + ADR-0081 (geometric stress field, falsified) in
docs/decisions/ while the frontier-provider-adapters work was
authored as ADR-0081 in a newly-created docs/adr/ directory,
unaware of the concurrent track.
This commit takes the minimum-blast-radius fix:
- docs/adr/ADR-0081-...md → docs/adr/ADR-0082-...md
- Update title header to ADR-0082, add "Renumbered from" breadcrumb
- Update the two source-file docstrings that cite the ADR number
(providers.py, model_registry.py)
The "two ADR directories" question (docs/adr/ vs docs/decisions/)
is NOT resolved here — docs/adr/ now has exactly one entry, while
docs/decisions/ is the canonical location per CLAUDE.md. A future
PR should either consolidate or document the split; this commit
just unblocks the immediate naming collision.
Out of scope:
- Consolidating directories
- Renumbering anything in docs/decisions/
- Re-numbering on the dev's local main (already pulled into this branch)
