feat(evals): articulate/disclosure/unarticulate partition
Tightens the multi_sentence_response lane predicates so OOV
invitations and refusal disclosures can no longer be counted as
articulate capability. Three new metrics partition the case space:
articulate_sentence_rate - >=2 sentences AND grounded in
{pack, teaching}. Real capability.
disclosure_sentence_rate - >=2 sentences AND grounded in
{oov, refusal, none}. Structural
multi-sentence from disclosure templates.
unarticulate_rate - <2 sentences regardless of source.
The three sum to 1.0 (modulo rounding) by construction. The
doctrine-correct headline is now ``articulate_sentence_rate``;
``multi_sentence_rate`` is kept as a continuity metric only.
2 new tests pin: (a) the three-way partition is total and disjoint
(articulate + disclosure + unarticulate == 1.0); (b) OOV/refusal
disclosure surfaces contribute to disclosure_sentence_rate but
never to articulate_sentence_rate.
Live A/B on 21 cases under the new partition:
flag off: articulate=0.0952, disclosure=0.0476, unarticulate=0.8571
flag on : articulate=0.8571, disclosure=0.0476, unarticulate=0.0952
Planner lift is +76pp on articulate. Disclosure stays flat across
the flag (the planner gate correctly leaves disclosure surfaces
alone). The remaining 9.5pp unarticulate flag-on is the genuine
miss list (walkthrough + compound prompts) that the next two
landings will target.
contract.md updated to make articulate_sentence_rate the headline
and to document the partition explicitly.
cognition eval byte-identical: public 100/100/91.7/100.
smoke suite 67/67.
This commit is contained in:
parent
6dd8efe7b3
commit
07fefb923c
3 changed files with 153 additions and 10 deletions
|
|
@ -30,13 +30,28 @@ as the *only* multi-sentence-capable code path.
|
||||||
## Scoring rubric
|
## Scoring rubric
|
||||||
|
|
||||||
```text
|
```text
|
||||||
multi_sentence_rate = cases_with_>=2_sentences / total_cases
|
articulate_sentence_rate = cases with >=2 sentences AND grounded in {pack, teaching} / total
|
||||||
non_fragment_rate = cases_where_every_sentence_>=4_tokens / total_cases
|
disclosure_sentence_rate = cases with >=2 sentences AND grounded in {oov, refusal, none} / total
|
||||||
connective_present_rate = cases_with_connective / cases_expecting_connective
|
unarticulate_rate = cases with <2 sentences / total
|
||||||
primed_cases = cases_where_priming_prompts_engaged
|
multi_sentence_rate = cases_with_>=2_sentences / total_cases # continuity metric
|
||||||
primed_multi_sentence_rate = primed_cases_with_>=2_sentences / primed_cases
|
non_fragment_rate = cases_where_every_sentence_>=4_tokens / total_cases
|
||||||
|
connective_present_rate = cases_with_connective / cases_expecting_connective
|
||||||
|
primed_cases = cases_where_priming_prompts_engaged
|
||||||
|
primed_multi_sentence_rate = primed_cases_with_>=2_sentences / primed_cases
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Doctrine-correct headline:** `articulate_sentence_rate`.
|
||||||
|
|
||||||
|
`multi_sentence_rate` is kept for continuity but is misleading on its own:
|
||||||
|
OOV teaching-invitation surfaces ("I don't know that yet — can you teach
|
||||||
|
me?") and refusal disclosures ("I don't know — insufficient grounding
|
||||||
|
for that yet.") are categorically multi-sentence by template, not by
|
||||||
|
articulation. They count toward `disclosure_sentence_rate`, never
|
||||||
|
`articulate_sentence_rate`.
|
||||||
|
|
||||||
|
The decomposition is total:
|
||||||
|
`articulate + disclosure + unarticulate = 1.0` (modulo rounding).
|
||||||
|
|
||||||
## Priming (warm-path measurement)
|
## Priming (warm-path measurement)
|
||||||
|
|
||||||
A case may carry an optional `priming_prompts: [str, ...]` array. The
|
A case may carry an optional `priming_prompts: [str, ...]` array. The
|
||||||
|
|
|
||||||
|
|
@ -163,13 +163,45 @@ def run_lane(cases: list[dict[str, Any]], config: Any = None) -> LaneReport:
|
||||||
if conn_expected else 1.0
|
if conn_expected else 1.0
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# ``multi_sentence_rate`` historically counted any case with ≥ 2
|
||||||
|
# sentences regardless of grounding source. That admitted OOV
|
||||||
|
# teaching invitations and refusal disclosures into the headline
|
||||||
|
# capability number — fixed here by splitting into three honest
|
||||||
|
# buckets:
|
||||||
|
#
|
||||||
|
# articulate_sentence_rate — ≥2 sentences AND grounded in pack
|
||||||
|
# or teaching (real capability).
|
||||||
|
# disclosure_sentence_rate — ≥2 sentences but grounded in oov,
|
||||||
|
# refusal, none, etc. (structural
|
||||||
|
# multi-sentence from disclosure
|
||||||
|
# templates, not articulation).
|
||||||
|
# unarticulate_rate — <2 sentences regardless of source.
|
||||||
|
#
|
||||||
|
# ``multi_sentence_rate`` is retained as a continuity metric. The
|
||||||
|
# doctrine-correct headline is ``articulate_sentence_rate``.
|
||||||
|
_DISCLOSURE_SOURCES = {"oov", "refusal", "none"}
|
||||||
|
articulate = sum(
|
||||||
|
1 for r in results
|
||||||
|
if r.sentence_count >= 2
|
||||||
|
and r.grounding_source in {"pack", "teaching"}
|
||||||
|
)
|
||||||
|
disclosure = sum(
|
||||||
|
1 for r in results
|
||||||
|
if r.sentence_count >= 2
|
||||||
|
and r.grounding_source in _DISCLOSURE_SOURCES
|
||||||
|
)
|
||||||
|
unarticulate = sum(1 for r in results if r.sentence_count < 2)
|
||||||
|
|
||||||
metrics: dict[str, Any] = {
|
metrics: dict[str, Any] = {
|
||||||
"cases": total,
|
"cases": total,
|
||||||
"multi_sentence_rate": round(multi / total, 4) if total else 0.0,
|
"multi_sentence_rate": round(multi / total, 4) if total else 0.0,
|
||||||
"non_fragment_rate": round(non_frag / total, 4) if total else 0.0,
|
"articulate_sentence_rate": round(articulate / total, 4) if total else 0.0,
|
||||||
"grounded_rate": round(grounded / total, 4) if total else 0.0,
|
"disclosure_sentence_rate": round(disclosure / total, 4) if total else 0.0,
|
||||||
"subject_named_rate": round(named / total, 4) if total else 0.0,
|
"unarticulate_rate": round(unarticulate / total, 4) if total else 0.0,
|
||||||
"connective_present_rate": conn_rate,
|
"non_fragment_rate": round(non_frag / total, 4) if total else 0.0,
|
||||||
|
"grounded_rate": round(grounded / total, 4) if total else 0.0,
|
||||||
|
"subject_named_rate": round(named / total, 4) if total else 0.0,
|
||||||
|
"connective_present_rate": conn_rate,
|
||||||
}
|
}
|
||||||
|
|
||||||
primed_results = [r for r in results if r.primed]
|
primed_results = [r for r in results if r.primed]
|
||||||
|
|
|
||||||
|
|
@ -173,6 +173,102 @@ def test_priming_default_is_cold_start(monkeypatch) -> None:
|
||||||
assert detail["primed"] is False
|
assert detail["primed"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_articulate_disclosure_unarticulate_partition(monkeypatch) -> None:
|
||||||
|
"""``articulate + disclosure + unarticulate`` must equal 1.0 modulo
|
||||||
|
rounding. No case can contribute to more than one bucket.
|
||||||
|
|
||||||
|
Articulate: ≥2 sentences AND grounding in {pack, teaching}.
|
||||||
|
Disclosure: ≥2 sentences AND grounding in {oov, refusal, none}.
|
||||||
|
Unarticulate: <2 sentences (regardless of source).
|
||||||
|
"""
|
||||||
|
|
||||||
|
class _FakeResponse:
|
||||||
|
def __init__(self, surface: str, source: str) -> None:
|
||||||
|
self.surface = surface
|
||||||
|
self.grounding_source = source
|
||||||
|
|
||||||
|
plan = iter([
|
||||||
|
# case 0: articulate (pack + ≥2 sentences)
|
||||||
|
_FakeResponse(
|
||||||
|
"Truth is X. Furthermore, truth belongs to cognition.truth.",
|
||||||
|
"pack",
|
||||||
|
),
|
||||||
|
# case 1: articulate (teaching + ≥2 sentences)
|
||||||
|
_FakeResponse(
|
||||||
|
"Light reveals truth. In turn, truth grounds knowledge.",
|
||||||
|
"teaching",
|
||||||
|
),
|
||||||
|
# case 2: disclosure (oov + ≥2 sentences)
|
||||||
|
_FakeResponse(
|
||||||
|
"I don't know that yet. Can you teach me?",
|
||||||
|
"oov",
|
||||||
|
),
|
||||||
|
# case 3: disclosure (none + ≥2 sentences)
|
||||||
|
_FakeResponse(
|
||||||
|
"I don't know. Insufficient grounding for that yet.",
|
||||||
|
"none",
|
||||||
|
),
|
||||||
|
# case 4: unarticulate (single sentence, regardless of source)
|
||||||
|
_FakeResponse("Truth.", "vault"),
|
||||||
|
])
|
||||||
|
|
||||||
|
class _FakeRuntime:
|
||||||
|
def __init__(self, config=None): # noqa: ARG002
|
||||||
|
pass
|
||||||
|
|
||||||
|
def chat(self, prompt: str) -> _FakeResponse: # noqa: ARG002
|
||||||
|
return next(plan)
|
||||||
|
|
||||||
|
monkeypatch.setattr(runner, "ChatRuntime", _FakeRuntime)
|
||||||
|
cases = [
|
||||||
|
{"id": f"c{i}", "category": "x", "prompt": "p",
|
||||||
|
"subject_lemma": "", "expects_connective": False}
|
||||||
|
for i in range(5)
|
||||||
|
]
|
||||||
|
metrics = run_lane(cases).metrics
|
||||||
|
|
||||||
|
assert metrics["articulate_sentence_rate"] == 0.4 # 2/5
|
||||||
|
assert metrics["disclosure_sentence_rate"] == 0.4 # 2/5
|
||||||
|
assert metrics["unarticulate_rate"] == 0.2 # 1/5
|
||||||
|
# Partition is total — must sum to 1.0 modulo rounding.
|
||||||
|
total = (
|
||||||
|
metrics["articulate_sentence_rate"]
|
||||||
|
+ metrics["disclosure_sentence_rate"]
|
||||||
|
+ metrics["unarticulate_rate"]
|
||||||
|
)
|
||||||
|
assert abs(total - 1.0) < 1e-9
|
||||||
|
|
||||||
|
|
||||||
|
def test_disclosure_never_inflates_articulate(monkeypatch) -> None:
|
||||||
|
"""OOV invitations and refusal disclosures must never contribute to
|
||||||
|
``articulate_sentence_rate`` even when they are multi-sentence by
|
||||||
|
template.
|
||||||
|
"""
|
||||||
|
|
||||||
|
class _FakeResponse:
|
||||||
|
surface = "I don't know that yet. Can you teach me?"
|
||||||
|
grounding_source = "oov"
|
||||||
|
|
||||||
|
class _FakeRuntime:
|
||||||
|
def __init__(self, config=None): # noqa: ARG002
|
||||||
|
pass
|
||||||
|
|
||||||
|
def chat(self, prompt: str) -> _FakeResponse: # noqa: ARG002
|
||||||
|
return _FakeResponse()
|
||||||
|
|
||||||
|
monkeypatch.setattr(runner, "ChatRuntime", _FakeRuntime)
|
||||||
|
cases = [
|
||||||
|
{"id": "oov_case", "category": "x", "prompt": "p",
|
||||||
|
"subject_lemma": "", "expects_connective": False}
|
||||||
|
]
|
||||||
|
metrics = run_lane(cases).metrics
|
||||||
|
|
||||||
|
# Multi-sentence is True (continuity metric), but articulate is False.
|
||||||
|
assert metrics["multi_sentence_rate"] == 1.0
|
||||||
|
assert metrics["articulate_sentence_rate"] == 0.0
|
||||||
|
assert metrics["disclosure_sentence_rate"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
def test_primed_multi_sentence_rate_separates_from_aggregate(monkeypatch) -> None:
|
def test_primed_multi_sentence_rate_separates_from_aggregate(monkeypatch) -> None:
|
||||||
"""The ``primed_multi_sentence_rate`` metric reports only on cases
|
"""The ``primed_multi_sentence_rate`` metric reports only on cases
|
||||||
that actually exercised priming, so cold-start cases never inflate
|
that actually exercised priming, so cold-start cases never inflate
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue