e831ed2615
Phase 2a r2/r3/r4 of the redefined plan: the general comprehension reader now
reads THREE independent-gold reasoning domains end-to-end (prose -> MeaningGraph
-> projection -> independent oracle -> answer vs gold), all wrong=0, and all
three are wired into the capability index.
reader.py — new domain-agnostic templates (function words + order; parse-or-refuse):
- categorical E/I/O: "no Xs are Ys"->disjoint, "some Xs are Ys"->intersects,
"some Xs are not Ys"->some_not (A "all Xs are Ys"->subset already existed)
- "therefore <categorical>" -> conclusion QUERY (same neutral predicate vocab)
- comparative facts: "<X> [is] <comp> [than] <Y>" -> less(...), closed
less/greater comparator lexicon, elided-copula support
- sort query ("sort ascending|descending", "... order from <low> to <high>")
and compare query ("compare <X> with <Y>")
- clause-splitting on commas / leading and|or for multi-clause sentences
projectors.py — to_syllogism (premises + validity conclusion, finite-model size 3)
and to_total_ordering (less-facts + sort/compare). Both return None when nothing
is honestly askable of their oracle (caller treats as refusal).
capability_index — wire 3 comprehension lanes into ADAPTERS; re-freeze baseline
breadth 3->6, capability_score 0.919641->0.814356 (geomean falls BY DESIGN as
honest partial-coverage domains join; wrong_total stays 0). digest 0a98b9b4...
Scores: set_membership 8/8, syllogism 6/8, total_ordering 4/8 — all wrong=0.
Multi-word NP handling is DEFERRED on purpose, not missed: the gold lanes
canonicalize multi-word NPs three contradictory ways ("North station"->"north",
"Level one"->"level_one", "metal objects"->"metal"), so no single general rule is
wrong=0-safe. The reader refuses multi-word NPs until the gold lanes carry a
canonicalization contract. Every refusal is a genuine harder phenomenon
(multi-word NP, adjectival predicate, trailing tokens) — never a readable case
silently dropped.
Tests: reader templates, projector unit tests, syllogism/total_ordering
end-to-end wrong=0 with pinned counts, capability breadth 3->6. 138 targeted +
87 smoke green. Lane SHAs 8/9 (sole miss = public_demo env wall-clock flake).
70 lines
2.1 KiB
Python
70 lines
2.1 KiB
Python
"""Score the general comprehension reader on the syllogism gold lane.
|
|
|
|
prose -> comprehend() -> to_syllogism() -> independent oracle -> answer vs gold.
|
|
A refusal (unreadable prose, unprojectable, or oracle-refused projection) is NOT a
|
|
wrong; only a committed answer that disagrees with gold is wrong (must stay 0).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from typing import Any
|
|
|
|
from evals.syllogism.oracle import OracleError, oracle_answer
|
|
from evals.syllogism.runner import _load_cases
|
|
from generate.meaning_graph.projectors import to_syllogism
|
|
from generate.meaning_graph.reader import Refusal, comprehend
|
|
|
|
|
|
def run() -> dict[str, Any]:
|
|
cases = _load_cases()
|
|
correct = wrong = refused = 0
|
|
wrongs: list[dict[str, Any]] = []
|
|
|
|
for case in cases:
|
|
comp = comprehend(case["text"])
|
|
if isinstance(comp, Refusal):
|
|
refused += 1
|
|
continue
|
|
projected = to_syllogism(comp)
|
|
if projected is None:
|
|
refused += 1
|
|
continue
|
|
structure, query = projected
|
|
try:
|
|
got = oracle_answer(structure, query)
|
|
except OracleError:
|
|
refused += 1
|
|
continue
|
|
if got == case.get("gold"):
|
|
correct += 1
|
|
else:
|
|
wrong += 1
|
|
wrongs.append(
|
|
{"id": case.get("id"), "got": got, "gold": case.get("gold"), "text": case["text"]}
|
|
)
|
|
|
|
return {
|
|
"domain": "comprehension_syllogism",
|
|
"total": len(cases),
|
|
"correct": correct,
|
|
"wrong": wrong,
|
|
"refused": refused,
|
|
"wrongs": wrongs,
|
|
"counts": {"correct": correct, "wrong": wrong, "refused": refused},
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
report = run()
|
|
print(json.dumps({k: v for k, v in report.items() if k != "wrongs"}, indent=2, sort_keys=True))
|
|
if report["wrong"]:
|
|
print("WRONG > 0 — comprehension produced a wrong committed answer:", file=sys.stderr)
|
|
print(json.dumps(report["wrongs"], indent=2), file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|