a6403edcd9
* feat(eval): ProofWriter-OWA refusal-floor lane (B1) — independent oracle, measure-only Mastery-v2 Step-3 Brief 1. Proves the open-world soundness floor: determine() never asserts a query True when the open-world truth is Unknown or False. Hardens "unknown != false" before the transitive-chain (B2) and closed-world (B4) work can stress it. - evals/proofwriter_owa/oracle.py: a SEPARATE minimal OWA label oracle (its own parser + reasoner + inverse/symmetric tables), importing NONE of determine, comprehend/ MeaningGraph realization, or production predicate-entailment helpers (INV-25/27 — the gold producer is disjoint from the solver). Grammar: member/subset/is-a closure, explicit negation (No X is a Y -> disjointness, the source of gold-False), and #775 inverse/symmetric ONE-HOP relational rules. No transitive relational chains. - evals/proofwriter_owa/fixtures.jsonl: 19 hand-authored ProofWriter-OWA-style items (9 True / 7 Unknown / 3 False), SHA-pinned. Gold computed by the oracle; the fixture's hand-authored `expected` pins the oracle (so the oracle is itself verified). - evals/proofwriter_owa/score.py: runs the production path (comprehend/comprehend_relational -> realize -> determine) vs the oracle gold; wrong = determine asserted True on a non-True gold. - tests/test_proofwriter_owa_lane.py: SHA pin; oracle==expected; wrong==0; every serving_support gold-True determines True (no coverage gaps); no answer=False (INV-30). Result: 19 items -> 9 correct (all serving_support True), 10 refused (Unknown/False), 0 wrong. Measure-only: no engine code touched; deliberately NOT a capability-index domain (a refusal floor has low coverage and would drag coverage_geomean). INV-30 green; smoke 99/99. Note: no live ProofWriter dataset access in this environment, so items are hand-authored in OWA style (provenance.md cites ProofWriter V2020.12.3 / arXiv:2012.13048 for semantics, attribution only) — but the GOLD is oracle-computed and verified, not hand-asserted. * harden(eval): OWA lane CLI fails on coverage_gaps too (review #779) score.py::main() exited nonzero only on wrong>0; it now ALSO exits nonzero when coverage_gaps is non-empty (serving-supported gold-True items that refused), matching the acceptance rule the tests already enforce and the module docstring. Both failure modes print to stderr; exit is 1 if either fires. Verified non-vacuously (injected gap -> exit 1). Lane test 5/5 green.
19 lines
3.6 KiB
JSON
19 lines
3.6 KiB
JSON
{"id": "owa-001", "facts": ["Rex is a dog.", "All dogs are mammals."], "query": "Is Rex a mammal?", "expected": "True", "serving_support": true, "note": "member then subset (member o subset)"}
|
|
{"id": "owa-002", "facts": ["Rex is a dog.", "All dogs are mammals.", "All mammals are animals."], "query": "Is Rex an animal?", "expected": "True", "serving_support": true, "note": "member o subset o subset"}
|
|
{"id": "owa-003", "facts": ["All dogs are mammals."], "query": "Are all dogs mammals?", "expected": "True", "serving_support": true, "note": "direct subset"}
|
|
{"id": "owa-004", "facts": ["All dogs are mammals.", "All mammals are animals."], "query": "Are all dogs animals?", "expected": "True", "serving_support": true, "note": "subset o subset"}
|
|
{"id": "owa-005", "facts": ["Truth is a concept."], "query": "Is truth a concept?", "expected": "True", "serving_support": true, "note": "direct member"}
|
|
{"id": "owa-006", "facts": ["Alice is the parent of Bob."], "query": "Is Alice the parent of Bob?", "expected": "True", "serving_support": true, "note": "relational direct"}
|
|
{"id": "owa-007", "facts": ["Bob is the child of Alice."], "query": "Is Alice the parent of Bob?", "expected": "True", "serving_support": true, "note": "relational inverse one-hop (#775)"}
|
|
{"id": "owa-008", "facts": ["Alice is the sibling of Bob."], "query": "Is Bob the sibling of Alice?", "expected": "True", "serving_support": true, "note": "relational symmetric one-hop (#775)"}
|
|
{"id": "owa-009", "facts": ["Bob is less than Alice."], "query": "Is Alice greater than Bob?", "expected": "True", "serving_support": true, "note": "relational inverse order one-hop (#775)"}
|
|
{"id": "owa-010", "facts": ["All dogs are mammals."], "query": "Are all mammals dogs?", "expected": "Unknown", "serving_support": false, "note": "reverse subset is not entailed"}
|
|
{"id": "owa-011", "facts": ["Rex is a dog.", "All cats are mammals."], "query": "Is Rex a mammal?", "expected": "Unknown", "serving_support": false, "note": "no chain: dog has no superclass given"}
|
|
{"id": "owa-012", "facts": ["Alice is the sibling of Bob."], "query": "Is Alice the parent of Bob?", "expected": "Unknown", "serving_support": false, "note": "cross-predicate: sibling does not imply parent"}
|
|
{"id": "owa-013", "facts": ["Alice is less than Bob."], "query": "Is Bob less than Alice?", "expected": "Unknown", "serving_support": false, "note": "asymmetric: a<b does not give b<a"}
|
|
{"id": "owa-014", "facts": [], "query": "Is Alice the sibling of Bob?", "expected": "Unknown", "serving_support": false, "note": "ungrounded: no told fact"}
|
|
{"id": "owa-015", "facts": ["Rex is a dog."], "query": "Is Spot a dog?", "expected": "Unknown", "serving_support": false, "note": "unrelated subject: Spot ungrounded"}
|
|
{"id": "owa-016", "facts": ["Socrates is a man.", "Man is a species."], "query": "Is Socrates a species?", "expected": "Unknown", "serving_support": false, "note": "member o member fallacy: instance-of is not transitive"}
|
|
{"id": "owa-017", "facts": ["Felix is a cat.", "All cats are mammals.", "No mammal is a reptile."], "query": "Is Felix a reptile?", "expected": "False", "serving_support": false, "note": "negation entailed via subset + disjointness"}
|
|
{"id": "owa-018", "facts": ["Tweety is a bird.", "No bird is a mammal."], "query": "Is Tweety a mammal?", "expected": "False", "serving_support": false, "note": "negation entailed: direct disjoint membership"}
|
|
{"id": "owa-019", "facts": ["All sparrows are birds.", "No bird is a mammal."], "query": "Are all sparrows mammals?", "expected": "False", "serving_support": false, "note": "negation entailed: subset into a disjoint class"}
|