f66f2ee47f
Adds comprehension_propositional — the comprehension organ now reads the classic
propositional ARGUMENT FORMS end-to-end into the flagship deductive_logic ROBDD
oracle (the most robustly independent gold in the repo). The neutral MeaningGraph
now feeds FOUR independent oracles (set-membership, syllogism-validity,
total-ordering, propositional-entailment) from one interlingua — the Option-B
interlingua thesis validated.
reader.py: propositional templates (atoms are chunked NP ids; fits the existing
entities + n-ary relations + negation model — NO interlingua change, propositional
is not arithmetic-quantities):
- "if <P> then <Q>" -> implies(P, Q)
- "not <P>" -> asserted(P, negated=True)
- "<P> or <Q>" -> or(P, Q)
- "<P>" (single token) -> asserted(P) (bare-atom, single-token only to
keep the parse-or-refuse floor)
- "therefore <prop>" -> query of the same predicate
Relations now carry a negated flag end-to-end (asserted negation).
projectors.py: to_deductive_logic serializes propositional relations/query into
formula strings (keyword operators the oracle tokenizer accepts); returns None
(refusal) unless the comprehension is purely propositional, so categorical/ordering
comprehensions never leak into the entailment oracle.
evals: new evals/propositional_logic/v1 (12 cases — modus ponens/tollens,
hypothetical & disjunctive syllogism, the affirming-consequent / denying-antecedent
fallacies which the oracle marks "unknown"; gold = oracle verdict) + gold-only
runner + evals/comprehension/propositional_runner.py. Oracle "refused" (formula
unevaluable) is treated as a decline, never a wrong.
Scores: comprehension_propositional 12/12 wrong=0 (full coverage); no regression on
the 3 existing lanes (8/8, 7/8, 7/8). Capability index breadth 6->7, score
0.917231 -> 0.928622, wrong_total 0, digest 51df7bba…
Tests: reader propositional templates; to_deductive_logic projector tests;
end-to-end full-coverage wrong=0; propositional generative round-trip added to the
wrong=0 property suite (verified to BITE under a reversed-implies mutation);
capability breadth 6->7. 115 targeted + 87 smoke green. Lane SHAs 8/9 (sole miss =
public_demo env wall-clock flake; deductive_logic_v1 unchanged).
59 lines
1.7 KiB
Python
59 lines
1.7 KiB
Python
"""Gold-only runner for the staged propositional-logic lane.
|
|
|
|
The independent arbiter is ``evals.deductive_logic.oracle.oracle_entailment`` (the
|
|
ROBDD-matched brute-force entailment checker). This runner verifies the committed
|
|
gold is exactly that oracle's verdict on each case's ``premises``/``query`` formula
|
|
strings — so the gold cannot drift from the independent oracle.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
from evals.deductive_logic.oracle import oracle_entailment
|
|
|
|
_CASES = Path(__file__).resolve().parent / "v1" / "cases.jsonl"
|
|
|
|
|
|
def _load_cases(path: Path = _CASES) -> list[dict[str, Any]]:
|
|
return [
|
|
json.loads(line)
|
|
for line in path.read_text(encoding="utf-8").splitlines()
|
|
if line.strip()
|
|
]
|
|
|
|
|
|
def run(path: Path = _CASES) -> dict[str, Any]:
|
|
cases = _load_cases(path)
|
|
failures: list[str] = []
|
|
for case in cases:
|
|
got = oracle_entailment(tuple(case["premises"]), case["query"])
|
|
if got != case.get("gold"):
|
|
failures.append(f"{case['id']}: oracle={got!r} != gold={case.get('gold')!r}")
|
|
|
|
correct = len(cases) - len(failures)
|
|
return {
|
|
"domain": "propositional_logic",
|
|
"total": len(cases),
|
|
"correct": correct,
|
|
"wrong": 0,
|
|
"refused": 0,
|
|
"gold_integrity_failures": failures,
|
|
"counts": {"correct": correct, "wrong": 0, "refused": 0},
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
report = run()
|
|
print(json.dumps(report, indent=2, sort_keys=True))
|
|
if report["gold_integrity_failures"]:
|
|
print("GOLD INTEGRITY FAILURE", file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|