core/evals/comprehension
Shay 2a3f422783 feat(measure): put the relational reader (#596) on the capability index
A1 of the refined sequencing — the binary-relation reader was inert w.r.t. the
yardstick (contributing 0). This adds a comprehension_relational_predicate domain:
binary-relation prose scored against hand-authored independent gold (predicate,
subject, object) triples — INV-25 independent / INV-27 reader-disjoint (the reader
never produced the gold). Index breadth 8->9, capability_score 0.937258->0.944030,
wrong_total still 0; baseline.json re-frozen to digest 1ea91c1e.

Rigor split: the index lane is POSITIVE-ONLY (clean coverage, consistent with the
other 8 lanes — mixing adversarial refuse-cases into the coverage denominator would
make 'added capability' read as a score drop). The #596 fabrication-catch lives in a
dedicated falsification test (evals/relational/v1/refusals.jsonl): the trailing-
qualifier / dangling-copula / negation / verb-form cases MUST refuse — bites if the
reader ever fabricates. Honest coverage gap recorded: overlaps_event has no copular
surface form (verb-form 'A overlaps B' refuses), so 17 positives cover 15/16 predicates.
2026-06-06 10:09:15 -07:00
..
__init__.py feat(comprehend): general comprehension reader + set_membership end-to-end (Phase 2a-r1/r2) 2026-06-05 16:32:34 -07:00
CANONICALIZATION.md feat(comprehend): multi-word NP chunking under a canonicalization contract 2026-06-05 22:47:34 -07:00
propositional_runner.py feat(comprehend): propositional-logic comprehension (4th domain, flagship oracle) 2026-06-05 23:24:54 -07:00
relational_metric_runner.py feat(comprehend): arithmetic word-problems via binding_graph (5th domain, real admissibility) 2026-06-06 00:43:16 -07:00
relational_predicate_runner.py feat(measure): put the relational reader (#596) on the capability index 2026-06-06 10:09:15 -07:00
set_membership_runner.py feat(comprehend): general comprehension reader + set_membership end-to-end (Phase 2a-r1/r2) 2026-06-05 16:32:34 -07:00
syllogism_runner.py feat(comprehend): complete 3-domain comprehension organ (syllogism + total_ordering) 2026-06-05 21:02:43 -07:00
total_ordering_runner.py feat(comprehend): complete 3-domain comprehension organ (syllogism + total_ordering) 2026-06-05 21:02:43 -07:00