2a3f422783
A1 of the refined sequencing — the binary-relation reader was inert w.r.t. the yardstick (contributing 0). This adds a comprehension_relational_predicate domain: binary-relation prose scored against hand-authored independent gold (predicate, subject, object) triples — INV-25 independent / INV-27 reader-disjoint (the reader never produced the gold). Index breadth 8->9, capability_score 0.937258->0.944030, wrong_total still 0; baseline.json re-frozen to digest 1ea91c1e. Rigor split: the index lane is POSITIVE-ONLY (clean coverage, consistent with the other 8 lanes — mixing adversarial refuse-cases into the coverage denominator would make 'added capability' read as a score drop). The #596 fabrication-catch lives in a dedicated falsification test (evals/relational/v1/refusals.jsonl): the trailing- qualifier / dangling-copula / negation / verb-form cases MUST refuse — bites if the reader ever fabricates. Honest coverage gap recorded: overlaps_event has no copular surface form (verb-form 'A overlaps B' refuses), so 17 positives cover 15/16 predicates.
86 lines
1.8 KiB
JSON
86 lines
1.8 KiB
JSON
{
|
|
"capability_score": 0.94403,
|
|
"coverage_geomean": 0.94403,
|
|
"coverage_micro": 0.993835,
|
|
"accuracy_micro": 1.0,
|
|
"breadth": 9,
|
|
"min_domain_coverage": 0.833333,
|
|
"wrong_total": 0,
|
|
"assert_mode_valid": true,
|
|
"deterministic_digest": "1ea91c1eefd6a0c4739e6fc97a40d8522d9b0758b4531bce448ec98e25789ccf",
|
|
"domains": [
|
|
{
|
|
"domain": "comprehension_propositional",
|
|
"correct": 12,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_relational_metric",
|
|
"correct": 15,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_relational_predicate",
|
|
"correct": 17,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_set_membership",
|
|
"correct": 8,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_syllogism",
|
|
"correct": 7,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.875,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "comprehension_total_ordering",
|
|
"correct": 7,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.875,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "deductive_logic",
|
|
"correct": 716,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"coverage": 1.0,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "dimensional",
|
|
"correct": 10,
|
|
"wrong": 0,
|
|
"refused": 2,
|
|
"coverage": 0.833333,
|
|
"accuracy": 1.0
|
|
},
|
|
{
|
|
"domain": "relational_metric",
|
|
"correct": 14,
|
|
"wrong": 0,
|
|
"refused": 1,
|
|
"coverage": 0.933333,
|
|
"accuracy": 1.0
|
|
}
|
|
],
|
|
"not_covered": []
|
|
}
|