* chore(evals, cli): contract standardization + bench --json stdout cleanliness
End-of-session shippability pass. Three concrete fixes:
1. core/cli.py — bench --json no longer pollutes stdout
Several bench paths call scripts.run_pulse.run_pulse which prints
verbose [pulse] traces unconditionally to stdout, breaking jq /
programmatic consumers of --json output.
New _bench_stdout_guard() redirects stdout → stderr for the
duration of the bench run when --json is set. Operator still sees
the pulse trace (on stderr), but --json consumers get a clean JSON
document on stdout. Applied to all four bench paths: cost,
articulation, default suite, and --suite all.
Verified: core bench --suite determinism --json now produces
parseable JSON; human path still shows 1140 [pulse] lines.
2. evals/{frontier_compare,realizer_guard}/contract.md (new)
core/contemplation/contract.md (new)
Each new contract follows the established pattern (37 contracts
already exist under evals/<lane>/contract.md):
- What it measures
- Why it matters (structural win)
- How to run
- How to read the output
- Pass criteria table
- When it has failed and why
- Runner / module layout
Coverage:
- frontier_compare: both Lane A (CORE-only suites) and Lane B
(cross-provider prompt_battery) with explicit guardrails
against mixing — operator asks for the wrong lane combination,
runner exits 2 with helpful error.
- realizer_guard: C1/C2 articulation safety boundary — synthetic
illegal candidates rejected directly by check_surface AND
former-bug runtime prompts now produce legal articulations.
- contemplation (ADR-0080): not under evals/ since it's runtime
infrastructure that consumes eval reports — contract lives at
core/contemplation/contract.md. Documents the read-only +
SPECULATIVE-only + deterministic-replay invariants and the
shared DiscoveryCandidateSink plumbing convergence (ADR-0080).
3. evals/CLAIMS.md — Tier 2 rows added
- frontier_compare Lane A: determinism.primary_score, max_versor_condition
- frontier_compare Lane B: prompt_battery.primary_score (CORE adapter),
cross-provider artifact persistence
- realizer_guard: all_claims_supported
- contemplation: SPECULATIVE-only invariant, deterministic replay,
additive sink path, no pack mutation (all CI-pinned by tests)
Verification
------------
$ core test --suite smoke -q
67 passed in 27.22s (no regression)
$ uv run pytest -q tests/test_contemplation_loop.py \
tests/test_contemplation_pipeline_convergence.py \
tests/test_frontier_compare_cross_provider.py
27 passed in 4.87s
$ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed
true (was: JSONDecodeError on prior [pulse] pollution)
* feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart
Stop-hook caught that #62 only covered contracts — the 929-line
report_viewer.html was never audited against the new cross-provider
report shape from #61. Two real gaps:
1. Lane-aware observation drawer
The drawer hardcoded Lane A (CORE-native) fields: surface,
grounding_source, anchor_lens_mode_label, versor_condition.
Lane B (cross-provider) observations carry different fields:
provider, model, elapsed_ms, error_type, error_message.
Loading a cross-provider report rendered only the surface row
with empty `grounding` — the provider + model + timing data
was unreachable without expanding "Show raw JSON".
Fix: detect Lane B (presence of `obs.provider`) and render the
appropriate field set. Lane A still renders identically (now
also surfaces trace_hash + register_id when present, which were
silently buried in the raw JSON before).
2. Pass-rate chart per suite
The summary strip showed one aggregate Primary % across all
suites, with no way to see WHICH suite is dragging the score.
Multi-suite runs (e.g. --suite all) had to expand each panel
individually to find the failing one.
Fix: new .passrate-chart element below the summary strip,
one horizontal bar per suite showing passed/total. All-pass =
solid green, all-fail = solid red, partial = green/red split
at the pass fraction. CSS only — no new dependencies.
3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar
shows the "side-by-side surface evidence across providers"
description when loading a Lane B report.
Verified
--------
- Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54)
- One <script> tag pair preserved
- Generated a real Lane B report via
`python -m evals.frontier_compare --provider core --suite prompt_battery`
for visual confirmation
Out of scope (noted for future PR)
----------------------------------
Sampled 3 `core demo` targets:
- register-tour: clean schema (all_claims_supported, claims, grid)
- audit-tour: both scene_1_* keys AND an empty scenes:[] array — inconsistent
- anti-regression: no all_claims_supported key, uses all_gates_held instead
Demo schema standardization deserves its own PR — operator tooling
would benefit from a uniform top-level success field across demos.
* docs(evals) + chore(demos): systematic audit + uniform success field
Stop-hook caught two real gaps after the contract+UI PR:
- demos had divergent success-field names (all_gates_held vs
learning_loop_closed vs claim_supported vs nested claims_supported)
- no systematic look at the 48 eval directories had been done
Both addressed concretely; remaining work captured in audit doc
rather than vaguely deferred.
1. Demo schema standardization — uniform all_claims_supported field
----------------------------------------------------------------------
All 9 ``core demo`` targets now emit a top-level
``all_claims_supported: bool`` field. Existing per-demo fields
(``all_gates_held``, ``learning_loop_closed``, ``claim_supported``,
nested ``claims_supported``) are preserved for backwards compat —
the new field is an alias derived from the demo's existing success
signal, not a replacement.
Operator tooling and the CI gate can now target
``all_claims_supported`` without knowing each demo's idiomatic
field name.
Files touched:
- evals/anti_regression/run_demo.py — adds AND of all_gates_held +
active_corpus_byte_identical
- evals/learning_loop/run_demo.py — adds AND of learning_loop_closed +
active_corpus_byte_identical
- scripts/publish_pack_measurements.py — adds AND of the three
entries in the nested claims_supported dict
- evals/long_context_cost/comparison_runner.py — adds alias for
claim_supported (singular)
The 5 demos already using ``all_claims_supported`` (audit-tour,
register-tour, anchor-lens-tour, orthogonality-tour, articulation)
are unchanged.
Verified across all 9 demos:
audit-tour : True
register-tour : True
anchor-lens-tour : True
orthogonality-tour : True
pack-measurements : True ← new alias
anti-regression : True ← new alias
learning-loop : True ← new alias
articulation : True
long-context-comparison : True ← new alias
2. docs/EVAL_AUDIT_2026-05-20.md — systematic 48-lane audit
------------------------------------------------------------
Replaces the "future PR" deferral with a concrete document.
Contains:
- Method (what was inspected for each lane).
- Summary (40/48 have contract.md; 18/48 have saved results;
empty results/ ≠ broken — most lanes regenerate on demand).
- Cross-provider relevance triage:
* 9 lanes are cross-provider-relevant and could benefit
from the prompt_battery-style adapter pattern (cognition,
english_fluency_ood, hebrew_fluency, koine_greek_fluency,
grammatical_coverage, inference_closure, multi_step_reasoning,
discourse_paragraph, foundational_*_ood, etc.).
* 29 lanes are CORE-only by design (versor closure, anchor
lens, identity divergence, provenance, etc.) — wiring
providers would be category-erroneous.
- Demo schema standardization status (this PR closes that).
- UI/UX coverage matrix.
- 5 concrete follow-up items, each focused enough for a single
PR, none requiring architectural change.
Regenerated reports
-------------------
evals/long_context_cost/results/comparison_v1.json and
evals/results/phase2_pack_measurements.json now contain the new
all_claims_supported field (auto-regenerated when validating the
schema change).
evals/frontier_compare/results/sample_core_promptbattery.json
added as a reference Lane B report so the new viewer always has
something to load on first open.
benchmarks/cost.py measures CORE per-turn cost honestly:
Measured (no estimation):
- turns, wall_seconds_total, cpu_seconds_total
- latency stats: min / median / p95 / max in ms
- throughput in turns per second
Derived with disclosed assumptions:
- USD per 1000 turns at AWS t3.medium on-demand
($0.0416/hr, source cited in CloudReference.source_note)
- Frontier pricing comparison: Anthropic Claude Sonnet 4.5 /
Haiku 4.5 and OpenAI GPT-4o, public per-token rates with
source notes, derived using a conservative 20-in / 40-out
tokens-per-turn assumption.
Explicitly NOT reported:
- Joules per turn. Honest energy measurement requires RAPL
(Linux) or IOKit/powermetrics (macOS) with privileged access
that a plain Python process cannot get. Reporting a fabricated
figure from a hand-waved TDP would violate "speculation is not
evidence." cpu_seconds_total is the available proxy.
CLI:
core bench --suite cost --runs 100
Measured numbers (100 turns, "What is truth?", warmup 5):
median latency: 444.88 ms
p95 latency: 447.10 ms
throughput: 2.61 turns/s
$/1000 turns: $0.0044
vs frontier: 48–149× cheaper depending on provider
CLAIMS.md Tier 4 cost/latency rows updated with real numbers
replacing TBDs. evals/reports/cost_latest.json committed as the
captured baseline.
Verified: smoke (67), bench --suite cost CLI works.
contradiction_detection: 0.50 → 1.00 contradiction_flag_rate,
1.00 → 0.00 false_flag_rate. Lane graduates overall.
TeachingStore.add now runs a coherence checker on every new proposal.
Two detection paths, both require subject token overlap:
Typed path — both new and prior parse to triples with the same
relation. Tails must differ in negation/opposition polarity AND
share ≥1 content token. Catches (truth, is, coherence) ↔
(truth, is, not coherence).
Text fallback — at least one side failed to parse a triple (e.g.
relation predicate "depends" not in the cognition pack lexicon
yet). Raw correction texts must differ in polarity AND share ≥2
non-discourse content tokens. ≥2 threshold prevents
single-shared-subject false positives on unrelated corrections.
Catches "meaning depends on use" vs "meaning is independent of use".
On detection, BOTH proposals (new and conflicting prior) transition
to EpistemicStatus.CONTESTED. ADR-0021: CONTESTED is not admissible
as evidence until a coherence judgment ratifies one direction or
falsifies the other.
Runner side: v1 versor-spike heuristic retired. The new CONTESTED
signal is the only one that drives `flagged`. versor_delta retained
in the record for telemetry.
CLAIMS.md Tier 4.5 contradiction rows CLOSED — completes the
truth-seeking schema arc. All red Tier 4.5 rows from the audit are
now green. docs/truth_seeking_schema.md §"Contradiction detection
is not implemented" closed.
Verified: smoke (67), teaching (17), cognition (121), runtime (19),
architectural invariants (40) — all green.
Two Tier 4.5 lanes graduate to passing:
refusal_calibration: 0.00 → 1.00 refusal_rate, 0.00 fabrication,
1.00 in_grounding_answer_rate.
- chat/runtime.py: _UNKNOWN_DOMAIN_SURFACE reworded to "I don't know
— insufficient grounding for that yet." (matches lane refusal
markers; was equivalent in spirit but unrecognizable).
- evals/refusal_calibration/runner.py: per-case `prime` field replays
brief priming turns before the probe. Necessary because ChatRuntime
cold-starts with an empty vault; "in-grounding" only counts as
grounded if the session has actually been told something relevant.
Previous 1.00 in_grounding rate was a false positive (gate was
firing on these too, but the surface text didn't match markers).
articulation_of_status: 0.00 → 1.00 speculative_articulation, 0.60
→ 0.00 false_certainty.
- core/cognition/pipeline.py: CognitiveTurnPipeline tracks subjects
of prior SPECULATIVE teaching proposals (parsed-triple subject
plus ≥4-char tokenized split, so prefixed parses like
"correction: wisdom" still match "What is wisdom?"). On a later
turn that references one of those subjects, or that carries a
reflexive query shape ("is your answer confirmed?", "has this
been reviewed?"), prepends "(speculative, not yet reviewed)" to
the surface. Teach turn itself does not self-mark; only
subsequent probes do.
Lane contracts updated to reflect graduation. CLAIMS.md Tier 4.5
rows for both lanes now CLOSED. docs/truth_seeking_schema.md
§Realizer-side surface gaps closed and rewritten.
Verified: smoke (67), cognition (121), runtime (19), teaching (17),
architectural invariants (40) — all green.
Categorizes every production vault.recall() callsite as RECOGNITION,
EVIDENCE_TELEMETRY, or EVIDENCE_USER_FACING. Adds INV-24 architectural
invariant (TestINV24VaultRecallRegistry, 3 tests) that forces any new
callsite to declare its role and requires EVIDENCE_USER_FACING sites to
pass min_status=COHERENT.
Audit findings:
- chat/runtime.py:330 → RECOGNITION (gate decision input)
- vault/decompose.py:121 → RECOGNITION (grade-decomposed gate fallback)
- generate/stream.py:147 → EVIDENCE_TELEMETRY (walk_surface per runtime contract)
- No EVIDENCE_USER_FACING sites exist today — user-facing surface comes from
pack-grounded realize(proposition, vocab), not vault.recall.
Why this closes Leak C: the write-side fix already stamps SPECULATIVE on
self-stored propositions; the read-side audit confirms no inference path
treats them as ratified evidence. If a future change routes the
generation walk into the user-facing surface, INV-24 forces the
recategorization to be explicit.
CLAIMS.md Tier 4.5 Leak C row now CLOSED. docs/truth_seeking_schema.md
§Leak C updated with full audit categorization.
Verified: smoke (67), cognition (121), runtime (19), all architectural
invariants (40) — green.
Audit of the one-mutation-path invariant (ADR-0021 §3) found three leaks
where pack authority or session-state writes could substitute for coherence
judgment. All three landed fixes or partial closures in this push.
Leaks closed:
- Leak A: pack vocab defaulted to COHERENT — flipped to SPECULATIVE in
language_packs/{compiler,schema}.py; docstring corrected to align with
ADR-0021 (it was rationalizing the leak).
- Leak B: vault.recall was epistemic-blind — VaultStore.store() now stamps
every entry with EpistemicStatus (default SPECULATIVE); recall(min_status=)
filters to admissible-as-evidence tier. All 4 vault-write sites updated.
- Leak C (write-side): generate/proposition.py:198 stored articulated
propositions unmarked — now stamps SPECULATIVE, breaking the
fabrication-feedback loop in principle. Read-side audit of 5 call sites
is the residual.
New architectural invariants (tests/test_architectural_invariants.py):
- INV-21: one-mutation-path allowlist (caught Leak C on first run)
- INV-22: pack lexicon default is SPECULATIVE (Leak A guard)
- INV-23: vault recall epistemic-aware (Leak B guard)
New eval lanes:
- teaching_injection_resistance — ships GREEN at 1.00/1.00/0 (the
structural anti-injection claim is real and measurable)
- refusal_calibration — honest gap: 0% refusal, 0% fabrication
- contradiction_detection — honest gap: 50% flag via versor-delta heuristic,
100% false-positive; motivates the proper coherence-checker
- articulation_of_status — honest gap: 0% speculative articulation, 60%
false certainty; output-side leak surface
New benchmarks:
- benchmarks/footprint.py — total deployed runtime is 7.06 MiB
(109,358x smaller than Llama 3.1 405B, runs offline, no GPU)
- benchmarks/learning_curve.py — monotonic + replay-deterministic curve
per lane
Documentation:
- docs/truth_seeking_schema.md — foundational architectural commitment,
five rules, mapped to human failure modes, leaks published openly
- evals/CLAIMS.md — five-tier public claims doc; Tier 4.5 publishes
known gaps with named fixes; verification contract at top
- README.md — new pillar between algebraic substrate and language pillar
Includes in-flight formation pipeline scaffolding (formation/, tests/formation/,
docs/formation_pipeline_plan.md) and minor CLI/contracts/gitignore edits
that were already in the working tree at session start.
Verification: 798 passed, 2 skipped, 1 deselected (pre-existing pack-count
test drift unrelated to schema changes).
Shay