Three load-bearing pieces:
1. ADR-0086 — UNKNOWN-intent pack-resident token surface
New deterministic composer `pack_grounded_unknown_surface` in
chat/pack_grounding.py. When intent classification returns UNKNOWN
but the prompt contains pack-resident lemmas (via cross-pack
resolver), surface those lemmas with their semantic_domains
instead of falling to the bare _UNKNOWN_DOMAIN_SURFACE. Wired
into chat/runtime.py::_maybe_pack_grounded_surface as the
last typed-intent branch before the OOV fallback. Null-lift
invariant pinned: fully-OOV prompts still emit the universal
disclosure byte-identically. Closes four cognition-eval term
misses: unknown_logos_019 (public), unknown_evidence_042 (dev),
unknown_spirit_041 + unknown_word_018 (holdout). Side effect:
evals/results/phase2_pack_measurements.json refusal_rate drops
from 0.25 → 0.125 across all three identity packs (no longer
refusing on these prompts).
2. ADR-0087 — PROCEDURE selector + trailing-clause subject echo
Two coupled changes in chat/pack_grounding.py:
(a) Numeric-determiner downrank in _extract_procedure_topic_lemma:
tokens whose primary semantic_domain starts with
"quantitative.numeric." are demoted; non-numeric resident
candidates always win. So "compare two terms" anchors on
`compare` not `two`.
(b) Trailing clause echoes the full normalized subject_text
rather than just the selected lemma, so OOV head nouns like
"terms" reach the surface even when only the procedure verb
is pack-resident. Closes procedure_compare_011.
3. 100-register catalog
New packs/register/_catalog.json — canonical machine-readable
spec for all 100 registers (7 currently-ratified + 93 drafted)
organized into 9 voice groups (depth/tone/stance/posture/domain/
cultural/affective/functional/composite). Each entry is a
complete production input — realizer_overrides, marker palettes
(openings/transitions/closings), depth_preference, description,
author_notes. All realizer_overrides use only legal keys per
scripts/ratify_register_packs.py::_KNOWN_OVERRIDE_KEYS.
Companion packs/register/CATALOG.md documents the production
loop: materialize → widen REGISTER_IDS → ratify → smoke.
Cognition-eval lifts (all three splits):
public: term_capture 91.7% → 100.0% (+8.3pp)
holdout: term_capture 83.3% → 100.0% (+16.7pp)
dev: term_capture 78.6% → 100.0% (+21.4pp)
surface_groundedness: 100% preserved on all splits
intent_accuracy / versor_closure: 100% preserved on all splits
Tests:
tests/test_pack_grounded_unknown.py — 14 tests (composer
direct + runtime engagement + null-lift invariant)
tests/test_adr_0087_procedure_selector.py — 12 tests (selector
numeric downrank + trailing-clause echo + regression guard)
Existing test suites unaffected — cognition lane 120 passed / 1
skipped both before and after. Full lane net −3 failures vs
pristine main (39 → 36 — none introduced).
* chore(evals, cli): contract standardization + bench --json stdout cleanliness
End-of-session shippability pass. Three concrete fixes:
1. core/cli.py — bench --json no longer pollutes stdout
Several bench paths call scripts.run_pulse.run_pulse which prints
verbose [pulse] traces unconditionally to stdout, breaking jq /
programmatic consumers of --json output.
New _bench_stdout_guard() redirects stdout → stderr for the
duration of the bench run when --json is set. Operator still sees
the pulse trace (on stderr), but --json consumers get a clean JSON
document on stdout. Applied to all four bench paths: cost,
articulation, default suite, and --suite all.
Verified: core bench --suite determinism --json now produces
parseable JSON; human path still shows 1140 [pulse] lines.
2. evals/{frontier_compare,realizer_guard}/contract.md (new)
core/contemplation/contract.md (new)
Each new contract follows the established pattern (37 contracts
already exist under evals/<lane>/contract.md):
- What it measures
- Why it matters (structural win)
- How to run
- How to read the output
- Pass criteria table
- When it has failed and why
- Runner / module layout
Coverage:
- frontier_compare: both Lane A (CORE-only suites) and Lane B
(cross-provider prompt_battery) with explicit guardrails
against mixing — operator asks for the wrong lane combination,
runner exits 2 with helpful error.
- realizer_guard: C1/C2 articulation safety boundary — synthetic
illegal candidates rejected directly by check_surface AND
former-bug runtime prompts now produce legal articulations.
- contemplation (ADR-0080): not under evals/ since it's runtime
infrastructure that consumes eval reports — contract lives at
core/contemplation/contract.md. Documents the read-only +
SPECULATIVE-only + deterministic-replay invariants and the
shared DiscoveryCandidateSink plumbing convergence (ADR-0080).
3. evals/CLAIMS.md — Tier 2 rows added
- frontier_compare Lane A: determinism.primary_score, max_versor_condition
- frontier_compare Lane B: prompt_battery.primary_score (CORE adapter),
cross-provider artifact persistence
- realizer_guard: all_claims_supported
- contemplation: SPECULATIVE-only invariant, deterministic replay,
additive sink path, no pack mutation (all CI-pinned by tests)
Verification
------------
$ core test --suite smoke -q
67 passed in 27.22s (no regression)
$ uv run pytest -q tests/test_contemplation_loop.py \
tests/test_contemplation_pipeline_convergence.py \
tests/test_frontier_compare_cross_provider.py
27 passed in 4.87s
$ core bench --suite determinism --json 2>/dev/null | jq .results[0].passed
true (was: JSONDecodeError on prior [pulse] pollution)
* feat(evals/ui): report viewer renders Lane B cross-provider + pass-rate chart
Stop-hook caught that #62 only covered contracts — the 929-line
report_viewer.html was never audited against the new cross-provider
report shape from #61. Two real gaps:
1. Lane-aware observation drawer
The drawer hardcoded Lane A (CORE-native) fields: surface,
grounding_source, anchor_lens_mode_label, versor_condition.
Lane B (cross-provider) observations carry different fields:
provider, model, elapsed_ms, error_type, error_message.
Loading a cross-provider report rendered only the surface row
with empty `grounding` — the provider + model + timing data
was unreachable without expanding "Show raw JSON".
Fix: detect Lane B (presence of `obs.provider`) and render the
appropriate field set. Lane A still renders identically (now
also surfaces trace_hash + register_id when present, which were
silently buried in the raw JSON before).
2. Pass-rate chart per suite
The summary strip showed one aggregate Primary % across all
suites, with no way to see WHICH suite is dragging the score.
Multi-suite runs (e.g. --suite all) had to expand each panel
individually to find the failing one.
Fix: new .passrate-chart element below the summary strip,
one horizontal bar per suite showing passed/total. All-pass =
solid green, all-fail = solid red, partial = green/red split
at the pass fraction. CSS only — no new dependencies.
3. SUITE_PREAMBLES gains the prompt_battery entry so the sidebar
shows the "side-by-side surface evidence across providers"
description when loading a Lane B report.
Verified
--------
- Brace/paren/div balance unchanged (308/308 / 380/380 / 54/54)
- One <script> tag pair preserved
- Generated a real Lane B report via
`python -m evals.frontier_compare --provider core --suite prompt_battery`
for visual confirmation
Out of scope (noted for future PR)
----------------------------------
Sampled 3 `core demo` targets:
- register-tour: clean schema (all_claims_supported, claims, grid)
- audit-tour: both scene_1_* keys AND an empty scenes:[] array — inconsistent
- anti-regression: no all_claims_supported key, uses all_gates_held instead
Demo schema standardization deserves its own PR — operator tooling
would benefit from a uniform top-level success field across demos.
* docs(evals) + chore(demos): systematic audit + uniform success field
Stop-hook caught two real gaps after the contract+UI PR:
- demos had divergent success-field names (all_gates_held vs
learning_loop_closed vs claim_supported vs nested claims_supported)
- no systematic look at the 48 eval directories had been done
Both addressed concretely; remaining work captured in audit doc
rather than vaguely deferred.
1. Demo schema standardization — uniform all_claims_supported field
----------------------------------------------------------------------
All 9 ``core demo`` targets now emit a top-level
``all_claims_supported: bool`` field. Existing per-demo fields
(``all_gates_held``, ``learning_loop_closed``, ``claim_supported``,
nested ``claims_supported``) are preserved for backwards compat —
the new field is an alias derived from the demo's existing success
signal, not a replacement.
Operator tooling and the CI gate can now target
``all_claims_supported`` without knowing each demo's idiomatic
field name.
Files touched:
- evals/anti_regression/run_demo.py — adds AND of all_gates_held +
active_corpus_byte_identical
- evals/learning_loop/run_demo.py — adds AND of learning_loop_closed +
active_corpus_byte_identical
- scripts/publish_pack_measurements.py — adds AND of the three
entries in the nested claims_supported dict
- evals/long_context_cost/comparison_runner.py — adds alias for
claim_supported (singular)
The 5 demos already using ``all_claims_supported`` (audit-tour,
register-tour, anchor-lens-tour, orthogonality-tour, articulation)
are unchanged.
Verified across all 9 demos:
audit-tour : True
register-tour : True
anchor-lens-tour : True
orthogonality-tour : True
pack-measurements : True ← new alias
anti-regression : True ← new alias
learning-loop : True ← new alias
articulation : True
long-context-comparison : True ← new alias
2. docs/EVAL_AUDIT_2026-05-20.md — systematic 48-lane audit
------------------------------------------------------------
Replaces the "future PR" deferral with a concrete document.
Contains:
- Method (what was inspected for each lane).
- Summary (40/48 have contract.md; 18/48 have saved results;
empty results/ ≠ broken — most lanes regenerate on demand).
- Cross-provider relevance triage:
* 9 lanes are cross-provider-relevant and could benefit
from the prompt_battery-style adapter pattern (cognition,
english_fluency_ood, hebrew_fluency, koine_greek_fluency,
grammatical_coverage, inference_closure, multi_step_reasoning,
discourse_paragraph, foundational_*_ood, etc.).
* 29 lanes are CORE-only by design (versor closure, anchor
lens, identity divergence, provenance, etc.) — wiring
providers would be category-erroneous.
- Demo schema standardization status (this PR closes that).
- UI/UX coverage matrix.
- 5 concrete follow-up items, each focused enough for a single
PR, none requiring architectural change.
Regenerated reports
-------------------
evals/long_context_cost/results/comparison_v1.json and
evals/results/phase2_pack_measurements.json now contain the new
all_claims_supported field (auto-regenerated when validating the
schema change).
evals/frontier_compare/results/sample_core_promptbattery.json
added as a reference Lane B report so the new viewer always has
something to load on first open.
`core demo pack-measurements` reproduces refusal_rate = 0.25 across
all three identity packs (default_general_v1, precision_first_v1,
generosity_first_v1). The committed baseline was 1.0, dating to the
ADR-0043 original commit (4ba1ef2); the runtime has evolved through
ADR-0048..0072 since then and the report file fell out of sync.
Evidence
- `python -m core.cli demo pack-measurements --json` reproduces 0.25
deterministically on the current main.
- tests/test_pack_measurements_phase2.py — all 6 pass; tests pin
structural invariants (pack_invariant_gate=True, fabrication=0.0,
refusal_rate ∈ [0,1]), not the specific value.
- report-level `claims_supported` still True; the pack-measurements
demo still PASSes in `core demo all`.
Other fields unchanged:
- fabrication_rate : 0.0
- out_of_grounding_count : 8
- pack_invariant_gate : True
- identity_divergence : distinct_rate 0.8 across pack pairs
No code change. Pure artifact refresh.
Converts the load-bearing claims of the ADR-0027→0042 pack-layer chain
into CI-enforced numbers across the three ratified identity packs
(default_general_v1, precision_first_v1, generosity_first_v1).
Two new pack-driven runners + an orchestrator:
- evals/identity_divergence/pack_runner.py — drives real
SentenceAssembler + SurfaceContext (no mocks) across all three
packs over 10 cases × 5 alignment bands; publishes per-pack
bare/hedge/qualifier rates and pairwise distinct_rate.
- evals/refusal_calibration/pack_runner.py — runs the existing
grounding-refusal lane via RuntimeConfig(identity_pack=...);
publishes per-pack refusal_rate/fabrication_rate and a
pack_invariant_gate flag asserting byte-identical cold-start
surfaces across packs.
- scripts/publish_pack_measurements.py — combined publisher
emitting evals/results/phase2_pack_measurements.json.
Baseline numbers (2026-05-17):
- precision_first hedge_rate=0.60, qualifier_rate=0.20
- generosity_first hedge_rate=0.20, qualifier_rate=0.00
- default_general hedge_rate=0.40, qualifier_rate=0.00
- pairwise distinct_rate ∈ [0.40, 0.80]
- refusal_rate=1.00, fabrication_rate=0.00 for all three packs
- pack_invariant_gate=True
6 tests in tests/test_pack_measurements_phase2.py lock the schema +
load-bearing flags + the structural inequality
precision.hedge_rate > generosity.hedge_rate. If identity packs
get wired into the cognition gate, pack_invariant_gate flips and
the suite fails.
ADR-0043 documents the numbers, the extended marker rationale, and
the trade-offs. README index updated with ADR-0043 row and chain
title bumped to "ADR-0027 through ADR-0043".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Shay