Shay e985790a03 feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench

Sharpens the measurement layer to match the runtime spine landed in
07fefb9 / 7af7892 / 4e3ddee.  Pure eval/benchmark/holdout work —
no runtime or planner code changed.

New isolation lanes
-------------------

* ``evals/compound_intent_decomposition/`` — single-purpose lane for
  the new ``classify_compound_intent`` decomposer.  Metrics:
  ``decomposition_accuracy``, ``atom_precision``, ``subject_accuracy``.
  Public: ``decomposition=1.0`` on 4e3ddee.
* ``evals/walkthrough_chain/`` — single-purpose lane for the new
  WALKTHROUGH sequential teaching-chain walk.  Metrics:
  ``path_exact_rate``, ``anchor_rate``, ``min_hop_rate``, ``bounded_rate``.
  Public: ``path_exact=1.0`` on 4e3ddee.

Without these, regressions in compound decomposition or the
walkthrough walk would show up as noise in ``multi_sentence_response``.
Each capability now has a single load-bearing metric on its own lane.

Cold-start lane sharpened
-------------------------

* ``evals/cold_start_grounding/public/v1/cases.jsonl`` extended with
  expository, compound, and walkthrough cases (48 total cases across
  19 categories including new ``expository_definition``,
  ``compound_definition_cause``, ``walkthrough_definition``).
* ``evals/cold_start_grounding/runner.py`` uses
  ``classify_compound_intent(...).primary`` for compound subject
  scoring — previously misattributed subjects on multi-part prompts.

Holdouts for the long-span lanes
--------------------------------

Until now only the cognition lane had a holdout split.  Adding
holdouts to the long-span lanes gives the planner work somewhere to
fail honestly when we widen:

* ``evals/cold_start_grounding/holdouts/v1/cases.jsonl`` (5 cases)
* ``evals/multi_sentence_response/holdouts/v1/cases.jsonl`` (5 cases)
* ``evals/conversational_thread_coherence/holdouts/v1/cases.jsonl`` (3 cases)
* ``evals/warmed_session_consistency/holdouts/v1/cases.jsonl`` (2 cases)

Discourse-planner-on bench sub-bench
------------------------------------

* ``benchmarks/articulation.py`` adds a planner-on sub-bench that
  reports ``articulate_sentence_rate`` alongside the existing
  throughput metrics.  Baselines articulation under load before any
  follow-up touches ``compute_trace_hash``.

Test coverage
-------------

* ``tests/test_compound_walkthrough_eval_lanes.py`` — new file pinning
  the two new lane runners.
* ``tests/test_articulation_bench.py``, ``tests/test_cold_start_grounding_lane.py``,
  ``tests/test_intent_explain_paragraph.py``,
  ``tests/test_response_mode_classifier.py`` — updated for new cases
  and assertions.

Validation
----------

* 152/152 active tests pass on the listed surfaces (2 skipped).
* smoke suite 67/67.
* cognition eval byte-identical: public 100/100/91.7/100.
* multi_sentence flag_on: articulate=1.0, disclosure=0.0, unarticulate=0.0
* compound_intent_decomp public: decomposition=1.0
* walkthrough_chain public: path_exact=1.0
* cold_start_grounding public (48 cases): intent=1.0, grounding=1.0, subject=1.0

2026-05-19 12:42:55 -07:00

716 B

Raw Blame History

Compound Intent Decomposition

Lane: compound_intent_decomposition

Scores whether a compound conversational prompt is decomposed into the intended semantic atoms before generation. This lane is structural: it does not grade paragraph fluency or final surface length.

Case Schema

{
  "id": "compound_truth_001",
  "prompt": "What is truth, and why does it matter?",
  "expected_atoms": [
    {"intent": "definition", "subject": "truth"},
    {"intent": "cause", "subject": "truth"}
  ]
}

Metrics

decomposition_accuracy: exact ordered atom match.
atom_precision: expected atoms found in the same position.
subject_accuracy: expected subjects recovered in the same position.

716 B Raw Blame History

Compound Intent Decomposition

Case Schema

Metrics

716 B

Raw Blame History