Sharpens the measurement layer to match the runtime spine landed in
07fefb9 / 7af7892 / 4e3ddee. Pure eval/benchmark/holdout work —
no runtime or planner code changed.
New isolation lanes
-------------------
* ``evals/compound_intent_decomposition/`` — single-purpose lane for
the new ``classify_compound_intent`` decomposer. Metrics:
``decomposition_accuracy``, ``atom_precision``, ``subject_accuracy``.
Public: ``decomposition=1.0`` on 4e3ddee.
* ``evals/walkthrough_chain/`` — single-purpose lane for the new
WALKTHROUGH sequential teaching-chain walk. Metrics:
``path_exact_rate``, ``anchor_rate``, ``min_hop_rate``, ``bounded_rate``.
Public: ``path_exact=1.0`` on 4e3ddee.
Without these, regressions in compound decomposition or the
walkthrough walk would show up as noise in ``multi_sentence_response``.
Each capability now has a single load-bearing metric on its own lane.
Cold-start lane sharpened
-------------------------
* ``evals/cold_start_grounding/public/v1/cases.jsonl`` extended with
expository, compound, and walkthrough cases (48 total cases across
19 categories including new ``expository_definition``,
``compound_definition_cause``, ``walkthrough_definition``).
* ``evals/cold_start_grounding/runner.py`` uses
``classify_compound_intent(...).primary`` for compound subject
scoring — previously misattributed subjects on multi-part prompts.
Holdouts for the long-span lanes
--------------------------------
Until now only the cognition lane had a holdout split. Adding
holdouts to the long-span lanes gives the planner work somewhere to
fail honestly when we widen:
* ``evals/cold_start_grounding/holdouts/v1/cases.jsonl`` (5 cases)
* ``evals/multi_sentence_response/holdouts/v1/cases.jsonl`` (5 cases)
* ``evals/conversational_thread_coherence/holdouts/v1/cases.jsonl`` (3 cases)
* ``evals/warmed_session_consistency/holdouts/v1/cases.jsonl`` (2 cases)
Discourse-planner-on bench sub-bench
------------------------------------
* ``benchmarks/articulation.py`` adds a planner-on sub-bench that
reports ``articulate_sentence_rate`` alongside the existing
throughput metrics. Baselines articulation under load before any
follow-up touches ``compute_trace_hash``.
Test coverage
-------------
* ``tests/test_compound_walkthrough_eval_lanes.py`` — new file pinning
the two new lane runners.
* ``tests/test_articulation_bench.py``, ``tests/test_cold_start_grounding_lane.py``,
``tests/test_intent_explain_paragraph.py``,
``tests/test_response_mode_classifier.py`` — updated for new cases
and assertions.
Validation
----------
* 152/152 active tests pass on the listed surfaces (2 skipped).
* smoke suite 67/67.
* cognition eval byte-identical: public 100/100/91.7/100.
* multi_sentence flag_on: articulate=1.0, disclosure=0.0, unarticulate=0.0
* compound_intent_decomp public: decomposition=1.0
* walkthrough_chain public: path_exact=1.0
* cold_start_grounding public (48 cases): intent=1.0, grounding=1.0, subject=1.0
Extends ``generate/intent.py:_RULES`` with three new expository
patterns so the upstream subject-extraction gap that the dedup
revealed is closed:
* ``^explain\s+`` → DEFINITION
* ``^(write|compose|draft) (a )?(short|brief)?
paragraph (about|on)\s+`` → DEFINITION
* ``^paragraph (about|on)\s+`` → DEFINITION
Rules placed AFTER the NARRATIVE family so ``Tell me about X`` and
``Describe X`` continue to route to NARRATIVE. Subject extraction
re-uses ``_normalize_subject`` so articles and trailing punctuation
are stripped: ``Explain the parent.`` → subject ``parent``.
``ResponseMode`` is untouched and remains orthogonal: the same prompts
still classify as ``EXPLAIN`` / ``PARAGRAPH`` independently.
20 new tests pin: each rule's expected subject, response-mode
preservation, NARRATIVE/EXAMPLE/existing-DEFINITION rules unchanged.
Lane re-measurement (multi_sentence_response, 21 cases):
flag off: multi=0.1429, primed_multi=0.0000, conn=0.5385, grounded=0.8571
flag on : multi=0.9048, primed_multi=1.0000, conn=0.8462, grounded=0.8571
Combined lift over the original (pre-wiring) baseline:
* multi_sentence_rate: +70pp on the substantive predicate
* primed_multi_sentence_rate: +50pp (0.5 → 1.0 post-classifier)
* connective_present_rate: +74pp (0.10 → 0.85)
* grounded_rate: +39pp (0.47 → 0.86)
Cognition eval byte-identical: public 100/100/91.7/100, holdout
100/100/83.3/100 — these prompts aren't in cognition cases, and the
new rules don't perturb any rule that fires for cognition prompts.
Conversational thread coherence unchanged.
docs/evals/discourse_runtime_baseline_2026-05-19.md updated with the
full delta table; the planner is now load-bearing across the warm
and cold pack/teaching paths and the lane measures real capability
rather than punctuation artifacts.
Shay