core/evals/relational/v1/refusals.jsonl
Shay 2a3f422783 feat(measure): put the relational reader (#596) on the capability index
A1 of the refined sequencing — the binary-relation reader was inert w.r.t. the
yardstick (contributing 0). This adds a comprehension_relational_predicate domain:
binary-relation prose scored against hand-authored independent gold (predicate,
subject, object) triples — INV-25 independent / INV-27 reader-disjoint (the reader
never produced the gold). Index breadth 8->9, capability_score 0.937258->0.944030,
wrong_total still 0; baseline.json re-frozen to digest 1ea91c1e.

Rigor split: the index lane is POSITIVE-ONLY (clean coverage, consistent with the
other 8 lanes — mixing adversarial refuse-cases into the coverage denominator would
make 'added capability' read as a score drop). The #596 fabrication-catch lives in a
dedicated falsification test (evals/relational/v1/refusals.jsonl): the trailing-
qualifier / dangling-copula / negation / verb-form cases MUST refuse — bites if the
reader ever fabricates. Honest coverage gap recorded: overlaps_event has no copular
surface form (verb-form 'A overlaps B' refuses), so 17 positives cover 15/16 predicates.
2026-06-06 10:09:15 -07:00

9 lines
1.1 KiB
JSON

{"id": "ref-001", "text": "Carol is the sibling of Dan during school.", "why": "trailing temporal qualifier — the #596 fabrication class (sibling_of(carol, dan_during_school))"}
{"id": "ref-002", "text": "Alice is the parent of Bob during the trip.", "why": "the asymmetric twin of ref-001 — both must refuse identically now"}
{"id": "ref-003", "text": "Five is equal to ten during testing.", "why": "trailing qualifier on a comparison"}
{"id": "ref-004", "text": "X is left of Y inside Z.", "why": "two relational structures in one clause"}
{"id": "ref-005", "text": "Monday before Friday is.", "why": "dangling copula — not a fact"}
{"id": "ref-006", "text": "Alice is not the parent of Bob.", "why": "negation is out of grammar — never reads as a positive"}
{"id": "ref-007", "text": "Alice greets Bob.", "why": "no relational connective / no copula"}
{"id": "ref-008", "text": "The weather is nice today.", "why": "no relational template at all"}
{"id": "ref-009", "text": "Sunrise overlaps dawn.", "why": "verb-form relation, no copula — the honest overlaps_event coverage gap"}