fdfb71f59b
core chat now decides Aristotelian syllogisms end-to-end -- the marquee
'basic logical work' widening. Closes Phase 0's fork: the ONLY categorical
decider was evals/syllogism/oracle.py (the sealed independence oracle serving
must not import; INV-25). This ships a production decider.
generate/proof_chain/categorical.py lowers a categorical argument to the
propositional regime (one Boolean atom per term-membership profile) and rides
the already-verified ROBDD engine -- no new decision procedure, soundness
inherits from the flagship. Sound AND complete for the modern/Boolean reading
(Darapti + existential-import-only forms correctly INVALID), with no reliance
on a lucky domain size. Independent of the eval oracle by mechanism (ROBDD-
over-profiles vs brute-force finite-model enumeration); proven by case-for-case
agreement with it across the whole syllogism gold lane.
New: generate/proof_chain/categorical.py, render_syllogism (deterministic
valid/invalid/inconsistent templates), CATEGORICAL shape-band, arena
categorical band (4 valid forms + 2 invalid, by-construction gold cross-checked
vs the independent syllogism oracle), tests/test_categorical_decider.py (8
tests incl. the independence-by-agreement proof).
Changed: chat/deduction_surface.py (categorical branch, license-gated via a
shared _license_gate helper; propositional path byte-identical), arena
DeductionSolver dual-path in lock-step with the composer, re-sealed ledger (5
bands, categorical earns SERVE at 0.99087 wrong=0), Phase-2 lane decide()
mirrors the dual path with the 3 categorical cases reclassified to their true
valid/invalid verdicts + 1 invalid case (n 27->28), report+SHA regenerated.
Honest wrinkles: irregular plurals ('fish') decline with unknown_morphology
(a reader limit, correctly not a wrong); surfaces read in singularized terms
('all whale are animal') -- the exact ids the engine reasoned over, no cosmetic
re-pluralization that could drift from the decision.
[Verification]: smoke 180 passed; cognition 122 passed/1 skipped; core test
--suite deductive 45 passed; test_categorical_decider 8 passed; practice
runner 5 bands all SERVE wrong=0; deduction_serve lane SHA regenerates to the
committed pin.
41 lines
1.6 KiB
Python
41 lines
1.6 KiB
Python
"""Deduction-serve lane — wrong=0 gate over the committed v1 corpus.
|
|
|
|
Scores the PRODUCTION serving decider end-to-end from raw text (the same
|
|
pipeline ``chat/deduction_surface.py`` runs), distinct from the bare-engine
|
|
``evals/deductive_logic`` lane and the reader-fidelity
|
|
``evals/comprehension/propositional_runner.py`` lane. See
|
|
``evals/deduction_serve/contract.md`` for the full contract.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from evals.deduction_serve.runner import _ROOT, _load, build_report
|
|
|
|
_V1 = _ROOT / "v1" / "cases.jsonl"
|
|
|
|
|
|
def test_v1_lane_wrong_is_zero() -> None:
|
|
report = build_report(_load(_V1))
|
|
assert report["n"] == 28
|
|
assert report["counts"]["wrong"] == 0
|
|
assert report["all_cases_correct"] is True
|
|
# the sizeable, honest signal: non-trivial committed classes covered,
|
|
# including the categorical band (valid + invalid) added in Phase 4.
|
|
cbg = report["correct_by_gold"]
|
|
assert cbg.get("entailed", 0) >= 5
|
|
assert cbg.get("refuted", 0) >= 5
|
|
assert cbg.get("unknown", 0) >= 5
|
|
assert cbg.get("declined", 0) >= 5
|
|
assert cbg.get("valid", 0) >= 3
|
|
assert cbg.get("invalid", 0) >= 1
|
|
|
|
|
|
def test_runner_treats_wrong_verdict_as_the_only_real_failure() -> None:
|
|
"""A committed disagreement with gold is ``wrong``; a decline never is,
|
|
even when gold expected a definite verdict (that's a miss, tracked via
|
|
``all_cases_correct``, not conflated with a confabulation)."""
|
|
report = build_report([
|
|
{"id": "bad", "text": "If p then q. p. Therefore q.", "gold": "unknown"}
|
|
])
|
|
assert report["counts"]["wrong"] == 1
|
|
assert report["all_cases_correct"] is False
|