Sharpens the measurement layer to match the runtime spine landed in
07fefb9 / 7af7892 / 4e3ddee. Pure eval/benchmark/holdout work —
no runtime or planner code changed.
New isolation lanes
-------------------
* ``evals/compound_intent_decomposition/`` — single-purpose lane for
the new ``classify_compound_intent`` decomposer. Metrics:
``decomposition_accuracy``, ``atom_precision``, ``subject_accuracy``.
Public: ``decomposition=1.0`` on 4e3ddee.
* ``evals/walkthrough_chain/`` — single-purpose lane for the new
WALKTHROUGH sequential teaching-chain walk. Metrics:
``path_exact_rate``, ``anchor_rate``, ``min_hop_rate``, ``bounded_rate``.
Public: ``path_exact=1.0`` on 4e3ddee.
Without these, regressions in compound decomposition or the
walkthrough walk would show up as noise in ``multi_sentence_response``.
Each capability now has a single load-bearing metric on its own lane.
Cold-start lane sharpened
-------------------------
* ``evals/cold_start_grounding/public/v1/cases.jsonl`` extended with
expository, compound, and walkthrough cases (48 total cases across
19 categories including new ``expository_definition``,
``compound_definition_cause``, ``walkthrough_definition``).
* ``evals/cold_start_grounding/runner.py`` uses
``classify_compound_intent(...).primary`` for compound subject
scoring — previously misattributed subjects on multi-part prompts.
Holdouts for the long-span lanes
--------------------------------
Until now only the cognition lane had a holdout split. Adding
holdouts to the long-span lanes gives the planner work somewhere to
fail honestly when we widen:
* ``evals/cold_start_grounding/holdouts/v1/cases.jsonl`` (5 cases)
* ``evals/multi_sentence_response/holdouts/v1/cases.jsonl`` (5 cases)
* ``evals/conversational_thread_coherence/holdouts/v1/cases.jsonl`` (3 cases)
* ``evals/warmed_session_consistency/holdouts/v1/cases.jsonl`` (2 cases)
Discourse-planner-on bench sub-bench
------------------------------------
* ``benchmarks/articulation.py`` adds a planner-on sub-bench that
reports ``articulate_sentence_rate`` alongside the existing
throughput metrics. Baselines articulation under load before any
follow-up touches ``compute_trace_hash``.
Test coverage
-------------
* ``tests/test_compound_walkthrough_eval_lanes.py`` — new file pinning
the two new lane runners.
* ``tests/test_articulation_bench.py``, ``tests/test_cold_start_grounding_lane.py``,
``tests/test_intent_explain_paragraph.py``,
``tests/test_response_mode_classifier.py`` — updated for new cases
and assertions.
Validation
----------
* 152/152 active tests pass on the listed surfaces (2 skipped).
* smoke suite 67/67.
* cognition eval byte-identical: public 100/100/91.7/100.
* multi_sentence flag_on: articulate=1.0, disclosure=0.0, unarticulate=0.0
* compound_intent_decomp public: decomposition=1.0
* walkthrough_chain public: path_exact=1.0
* cold_start_grounding public (48 cases): intent=1.0, grounding=1.0, subject=1.0
Phase 5 (ADR-0067 follow-up):
teaching/cross_pack_supersede.py — supersede_cross_pack_chain()
CLI: core teaching supersede ... --cross-pack
--subject-pack-id ... --object-pack-id ...
Strict per-chain residency, anti-leakage, byte-identical rollback
on any post-append re-load failure. 9 new tests.
Articulation benchmark suite (Phase 4 capability proof):
benchmarks/articulation.py — 5 sub-benches
[1] breadth — every intent shape (9 + OOV + cross-pack)
[2] determinism — N reruns / unique-surface count
[3] footprint — psutil RSS profile across T turns
[4] cross-topic — thread context across mixed subjects
[5] ollama-compare — opt-in side-by-side with local Ollama
CLI: core bench --suite articulation
--runs N (det rerun count)
--turns N (footprint sample window)
--ollama-model MODEL --ollama-reruns N
Full operator preamble + JSON report path.
10 new tests cover the bench shape (psutil import-skipped).
Documentation:
benchmarks/README.md — full operator manual: catalogue of every
bench suite, how to read good/neutral/bad results for each sub-
bench, why CORE vs Ollama comparisons are valid on the
determinism axis and not on linguistic quality, workflow guide.
README.md — articulation bench listed in the live-demo grid and
quick-start examples.
Reference run (llama3:8b, 100 turns, 5 reruns):
determinism_all_identical=True
per-turn ΔRSS ≈ 23 KiB
CORE byte_identical_on_every_prompt=True
Ollama unique_surfaces≥2 on every prompt
Verification:
18 new tests pass
Full lane: 2116 passed, 2 skipped, 0 failed in 2:38
Shay