f66f2ee47f
Adds comprehension_propositional — the comprehension organ now reads the classic
propositional ARGUMENT FORMS end-to-end into the flagship deductive_logic ROBDD
oracle (the most robustly independent gold in the repo). The neutral MeaningGraph
now feeds FOUR independent oracles (set-membership, syllogism-validity,
total-ordering, propositional-entailment) from one interlingua — the Option-B
interlingua thesis validated.
reader.py: propositional templates (atoms are chunked NP ids; fits the existing
entities + n-ary relations + negation model — NO interlingua change, propositional
is not arithmetic-quantities):
- "if <P> then <Q>" -> implies(P, Q)
- "not <P>" -> asserted(P, negated=True)
- "<P> or <Q>" -> or(P, Q)
- "<P>" (single token) -> asserted(P) (bare-atom, single-token only to
keep the parse-or-refuse floor)
- "therefore <prop>" -> query of the same predicate
Relations now carry a negated flag end-to-end (asserted negation).
projectors.py: to_deductive_logic serializes propositional relations/query into
formula strings (keyword operators the oracle tokenizer accepts); returns None
(refusal) unless the comprehension is purely propositional, so categorical/ordering
comprehensions never leak into the entailment oracle.
evals: new evals/propositional_logic/v1 (12 cases — modus ponens/tollens,
hypothetical & disjunctive syllogism, the affirming-consequent / denying-antecedent
fallacies which the oracle marks "unknown"; gold = oracle verdict) + gold-only
runner + evals/comprehension/propositional_runner.py. Oracle "refused" (formula
unevaluable) is treated as a decline, never a wrong.
Scores: comprehension_propositional 12/12 wrong=0 (full coverage); no regression on
the 3 existing lanes (8/8, 7/8, 7/8). Capability index breadth 6->7, score
0.917231 -> 0.928622, wrong_total 0, digest 51df7bba…
Tests: reader propositional templates; to_deductive_logic projector tests;
end-to-end full-coverage wrong=0; propositional generative round-trip added to the
wrong=0 property suite (verified to BITE under a reversed-implies mutation);
capability breadth 6->7. 115 targeted + 87 smoke green. Lane SHAs 8/9 (sole miss =
public_demo env wall-clock flake; deductive_logic_v1 unchanged).
71 lines
2.2 KiB
Python
71 lines
2.2 KiB
Python
"""Score the general comprehension reader on the propositional_logic gold lane.
|
|
|
|
prose -> comprehend() -> to_deductive_logic() -> independent ROBDD oracle -> verdict
|
|
vs gold. A refusal (unreadable prose, unprojectable, or an oracle "refused" because
|
|
the projected formula could not be evaluated) is NOT a wrong; only a committed
|
|
verdict that disagrees with gold is wrong (must stay 0).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from typing import Any
|
|
|
|
from evals.deductive_logic.oracle import REFUSED, oracle_entailment
|
|
from evals.propositional_logic.runner import _load_cases
|
|
from generate.meaning_graph.projectors import to_deductive_logic
|
|
from generate.meaning_graph.reader import Refusal, comprehend
|
|
|
|
|
|
def run() -> dict[str, Any]:
|
|
cases = _load_cases()
|
|
correct = wrong = refused = 0
|
|
wrongs: list[dict[str, Any]] = []
|
|
|
|
for case in cases:
|
|
comp = comprehend(case["text"])
|
|
if isinstance(comp, Refusal):
|
|
refused += 1
|
|
continue
|
|
projected = to_deductive_logic(comp)
|
|
if projected is None:
|
|
refused += 1
|
|
continue
|
|
premises, query = projected
|
|
got = oracle_entailment(premises, query)
|
|
if got == REFUSED:
|
|
# the projected formula could not be evaluated -> decline, not a wrong
|
|
refused += 1
|
|
continue
|
|
if got == case.get("gold"):
|
|
correct += 1
|
|
else:
|
|
wrong += 1
|
|
wrongs.append(
|
|
{"id": case.get("id"), "got": got, "gold": case.get("gold"), "text": case["text"]}
|
|
)
|
|
|
|
return {
|
|
"domain": "comprehension_propositional",
|
|
"total": len(cases),
|
|
"correct": correct,
|
|
"wrong": wrong,
|
|
"refused": refused,
|
|
"wrongs": wrongs,
|
|
"counts": {"correct": correct, "wrong": wrong, "refused": refused},
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
report = run()
|
|
print(json.dumps({k: v for k, v in report.items() if k != "wrongs"}, indent=2, sort_keys=True))
|
|
if report["wrong"]:
|
|
print("WRONG > 0 — comprehension produced a wrong committed answer:", file=sys.stderr)
|
|
print(json.dumps(report["wrongs"], indent=2), file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|