a733fc5737
Recovers the multi-word-NP cases the reader previously refused, by adopting ONE
principled canonicalization contract (evals/comprehension/CANONICALIZATION.md) that
the reader AND the gold lanes both follow — so a committed answer can only match
gold or refuse, never silently mean something else.
Contract: a noun-phrase slot -> tokens lowercased, joined with "_"; a plural class
slot singularizes its head first ("metal objects"->"metal_object",
"North station"->"north_station", "Level one"->"level_one"). JOIN is chosen over
head-word-only ("metal objects"->"metal") because head-word-only is
information-destroying — it collapses "metal objects" and "metal tools" into one
false identity, itself a wrong=0 hazard.
reader.py: slot-based templates chunk multi-token NPs (_chunk / _chunk_class
replace the single-token _one / _one_class). Reserved-function-word guard fires only
INSIDE a multi-token slot (a lone "A" item is content, not the article). Still
parse-or-refuse: reserved-word leaks ("Compare beta with beta in the same order"),
non-pluralizable class heads (adjectival "trained"), and the ambiguous adjacent
two-NP subset query ("Are all <Xs> <Ys>?") all REFUSE.
gold (the contract update, logic-preserving — only term NAMES change):
- sy-v1-0008: metal/soft -> metal_object/soft_object (was head-word-only)
- to-v1-0005: red -> red_rank (was head-word-only)
- to-v1-0004: prose made internally consistent ("is after", "north station") +
north -> north_station (original prose used "North station" in the fact but
"north" in the query — a latent inconsistency)
- to-v1-0007: already conformed (level_one…), no change
Gold-only integrity runners stay 8/8 both lanes (structure+query+gold consistent).
Scores: set_membership 8/8, syllogism 6/8->7/8, total_ordering 4/8->7/8, all
wrong=0. Capability index re-frozen: score 0.814356 -> 0.917231, breadth 6,
wrong_total 0, digest 13d7db6c…
Tests: reader chunking + refusal tests; multi-word generative round-trip added to
the wrong=0 property suite (verified to BITE under a head-word-only mutation —
collapsed ids produce a wrong verdict the test catches); pinned counts updated.
100 comprehension/capability targeted + 87 smoke green.
34 lines
1.3 KiB
Python
34 lines
1.3 KiB
Python
"""Phase 2a-r3 — end-to-end: the comprehension reader scored on syllogism.
|
|
|
|
prose -> comprehend -> to_syllogism -> INDEPENDENT oracle -> answer vs gold.
|
|
The load-bearing invariant: wrong == 0 (the reader refuses rather than emit a
|
|
structure that yields a wrong answer). Coverage is allowed to be partial — a
|
|
refusal is honest, a wrong commit is not.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from evals.comprehension.syllogism_runner import run
|
|
|
|
|
|
def test_comprehension_syllogism_wrong_is_zero() -> None:
|
|
report = run()
|
|
assert report["wrong"] == 0, report["wrongs"]
|
|
|
|
|
|
def test_comprehension_syllogism_has_real_coverage() -> None:
|
|
report = run()
|
|
assert report["correct"] > 0
|
|
assert report["correct"] + report["refused"] == report["total"]
|
|
|
|
|
|
def test_comprehension_syllogism_pinned_counts() -> None:
|
|
# Pins the lane: 7 read end-to-end (Barbara/Celarent/Darii/Ferio/Datisi, the
|
|
# invalid undistributed-middle which the oracle correctly rejects, and the
|
|
# multi-word Ferio "metal objects"/"soft objects" now chunked by the
|
|
# canonicalization contract); 1 refused — the existential-import conclusion
|
|
# ("some trained people exist") which is out of the categorical grammar.
|
|
report = run()
|
|
assert report["correct"] == 7
|
|
assert report["refused"] == 1
|
|
assert report["total"] == 8
|