core/evals/multi_sentence_response
Shay 9367209d04 feat(evals): priming_prompts on multi_sentence_response lane
Option 1 of the lane-isolation work after the 8d1aeec predicate
refinement.  Adds optional ``priming_prompts: [str, ...]`` to each
case in ``multi_sentence_response``.  The runner runs priming prompts
on the same ``ChatRuntime`` instance before the scored prompt and
discards their responses; only the scored prompt is measured.

This isolates code paths (notably the discourse planner hook) that
engage only on the warm pack/teaching path from cold-start one-shot
paths.  Cold-start measurement is preserved: cases without
``priming_prompts`` (or with an empty list) keep the old behavior.

New metric ``primed_multi_sentence_rate`` reports only on primed
cases.  ``primed`` is also exposed per-case in case_details.

Six primed cases added to ``public/v1/cases.jsonl`` (Explain truth /
Tell about truth / Explain knowledge / Tell about light / Tell about
parent / Write a short paragraph about truth).  Each is the cold-
start variant of an existing case plus a single "What is X?"
priming prompt.

3 new tests:
* Priming prompts run in order on the same runtime before the
  scored prompt; primed=True on the result.
* Default cold-start behavior: no priming key OR empty list ⇒
  primed=False; aggregate untouched.
* ``primed_multi_sentence_rate`` separates from aggregate so
  cold cases never inflate/depress the warm-path metric.

A/B measurement on the live runtime (21 cases):
  flag off: multi=0.1429, primed_multi=0.0000, primed_cases=6
  flag on : multi=0.2857, primed_multi=0.5000, primed_cases=6

Lift is real and exclusively on the substrate the planner can
actually serve (teaching-grounded narrative).  The three primed
"Explain X" and "Write a short paragraph about X" cases stay
vault-grounded (Explain / Write are not DEFINITION / NARRATIVE
intents and so don't fire pack-grounded warm), so they don't lift.
That gap is what option 2 will close.

contract.md updated to document priming and the new metric.
2026-05-19 11:51:21 -07:00
..
dev feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
public/v1 feat(evals): priming_prompts on multi_sentence_response lane 2026-05-19 11:51:21 -07:00
contract.md feat(evals): priming_prompts on multi_sentence_response lane 2026-05-19 11:51:21 -07:00
runner.py feat(evals): priming_prompts on multi_sentence_response lane 2026-05-19 11:51:21 -07:00