core/evals/multi_sentence_response
Shay 07fefb923c feat(evals): articulate/disclosure/unarticulate partition
Tightens the multi_sentence_response lane predicates so OOV
invitations and refusal disclosures can no longer be counted as
articulate capability.  Three new metrics partition the case space:

  articulate_sentence_rate  - >=2 sentences AND grounded in
                              {pack, teaching}.  Real capability.
  disclosure_sentence_rate  - >=2 sentences AND grounded in
                              {oov, refusal, none}.  Structural
                              multi-sentence from disclosure templates.
  unarticulate_rate         - <2 sentences regardless of source.

The three sum to 1.0 (modulo rounding) by construction.  The
doctrine-correct headline is now ``articulate_sentence_rate``;
``multi_sentence_rate`` is kept as a continuity metric only.

2 new tests pin: (a) the three-way partition is total and disjoint
(articulate + disclosure + unarticulate == 1.0); (b) OOV/refusal
disclosure surfaces contribute to disclosure_sentence_rate but
never to articulate_sentence_rate.

Live A/B on 21 cases under the new partition:

  flag off: articulate=0.0952, disclosure=0.0476, unarticulate=0.8571
  flag on : articulate=0.8571, disclosure=0.0476, unarticulate=0.0952

Planner lift is +76pp on articulate.  Disclosure stays flat across
the flag (the planner gate correctly leaves disclosure surfaces
alone).  The remaining 9.5pp unarticulate flag-on is the genuine
miss list (walkthrough + compound prompts) that the next two
landings will target.

contract.md updated to make articulate_sentence_rate the headline
and to document the partition explicitly.

cognition eval byte-identical: public 100/100/91.7/100.
smoke suite 67/67.
2026-05-19 12:13:44 -07:00
..
dev feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
public/v1 feat(evals): priming_prompts on multi_sentence_response lane 2026-05-19 11:51:21 -07:00
contract.md feat(evals): articulate/disclosure/unarticulate partition 2026-05-19 12:13:44 -07:00
runner.py feat(evals): articulate/disclosure/unarticulate partition 2026-05-19 12:13:44 -07:00