1ff06726a6
* feat(determine): add transitive strict-order relational inference Add sound transitive closure for declared strict-order relational predicates — less_than, greater_than, before_event, after_event, and ONLY these. A query p(a, c) determines True when a same-predicate chain p(a, b), p(b, c), ... over the predicate's OWN realized edges is found (BFS reachability) and the proof_chain ROBDD verifies the transitive entailment (search-then-verify, through the UNCHANGED evaluate_entailment via a new lower_transitive_chain). Mirrors _determine_subsumption. Scope / firewall: - TRANSITIVE_PREDICATES is closed and default-off; sibling_of, parent_of, left_of/right_of, inside_of, during_event, overlaps_event are EXCLUDED. - Same-predicate edges only — no transitive-through-inverse, no cross-predicate (triple firewall: recall filter + lowering re-reject + ROBDD re-verify). - Open-world: asserts only answer=True via the shared _relational_determined surface (rule="transitive"), so INV-30 stays at exactly 3 Determined sites and no answer=False is added. One-hop inverse/symmetric (#775) is unchanged. FrameVerdict / closed-world (ADR-0222) is untouched. wrong=0 bite (unit + lane + independent oracle): non-transitive-predicate chains, non-admitted spatial chains, mixed-predicate chains, disjoint chains, reflexive cycles, and inverse+transitive composition all REFUSE. Measurement: new evals/relational_transitive lane with an INDEPENDENT BFS transitive-closure oracle (imports no engine module; INV-25/27), cross-checked both directions (positives oracle-True, confusers oracle-False). Capability-index breadth 10 -> 11, wrong_total 0, deterministic digest re-frozen. Cross-PR reconcile: migrated the one-hop confuser rinf-ref-001 (a less_than chain, which now correctly determines transitively) to a B2 positive; repurposed its unit test to a non-transitive parent_of chain; corrected a pre-existing stale breadth==9 assertion (#775 added a 10th domain but never updated test_capability_index). Verified in the worktree: determine + transitive/one-hop lanes + capability baseline/index + ProofWriter-OWA floor + INV-30/29/21/02 + full smoke — all green. A 3-skeptic read-only adversarial review found no wrong=0 leak. * test(determine): B2 hardening — TRANSITIVE_PREDICATES pin + lower_transitive defensive + budget Add the required B2 hardening tests before merge: - test_transitive_predicates_closed_and_excludes: the table is EXACTLY {less_than, greater_than, before_event, after_event}, every member is in RELATIONAL_PREDICATES, and every deliberately-excluded predicate (sibling_of, spouse_of, parent_of, child_of, left_of, right_of, inside_of, during_event, overlaps_event) is explicitly absent. (Closes the gap where relational.py's comment claimed this test existed but it did not.) - tests/test_composition_lower_transitive.py: direct defensive tests on lower_transitive_chain — empty path, mislabeled (cross-predicate) edge, wrong arity, non-contiguous path, and path-not-reaching-target all lower to None (refuse); plus exact 2-hop and 3-hop theory pins. - test_edge_budget_exhaustion_refuses: above _TRANSITIVE_EDGE_BUDGET the transitive search declines (a safe coverage refusal), never proves.
81 lines
3 KiB
Python
81 lines
3 KiB
Python
"""Lane test — one-hop relational inference (capability-index lane + wrong=0 bite).
|
|
|
|
The positive lane (cases.jsonl) is the capability-index coverage; the confuser lane
|
|
(refusals.jsonl) is the wrong=0 bite — every confuser MUST refuse, or the inverse/
|
|
symmetric rule over-fired. Fixture SHAs are pinned so the gold cannot drift silently.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from chat.runtime import ChatRuntime
|
|
from evals.comprehension.relational_inference_runner import _load_cases, run
|
|
from generate.determine import Determined, determine
|
|
from generate.meaning_graph.relational import (
|
|
comprehend_relational,
|
|
load_relational_pack_lemmas,
|
|
)
|
|
from generate.realize import realize_comprehension
|
|
from session.context import SessionContext
|
|
|
|
_V1 = Path(__file__).resolve().parent.parent / "evals" / "relational_inference" / "v1"
|
|
_CASES_SHA = "03310ecc3ab7b7bf26f0a1779709bf468e888bbca1e53a71cba1d269b5c1dd71"
|
|
_REFUSALS_SHA = "35cf8369ec0ba80a4e90ecc88de73dafac660ac5982288f49c3aeeb3e5123a16"
|
|
_HIGH = 10**9
|
|
|
|
|
|
def _sha(p: Path) -> str:
|
|
return hashlib.sha256(p.read_bytes()).hexdigest()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def vocab_persona():
|
|
rt = ChatRuntime(no_load_state=True)
|
|
return rt._context.vocab, rt._context.persona
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def pack():
|
|
return load_relational_pack_lemmas()
|
|
|
|
|
|
def test_fixture_shas_pinned() -> None:
|
|
assert _sha(_V1 / "cases.jsonl") == _CASES_SHA
|
|
assert _sha(_V1 / "refusals.jsonl") == _REFUSALS_SHA
|
|
|
|
|
|
def test_positive_lane_wrong_zero_and_covers() -> None:
|
|
r = run()
|
|
assert r["wrong"] == 0, r["wrongs"]
|
|
assert r["correct"] > 0 # coverage > 0 — else the index geomean zeroes the score
|
|
assert r["correct"] + r["wrong"] + r["refused"] == r["total"]
|
|
|
|
|
|
def test_confusers_all_refuse(vocab_persona, pack) -> None:
|
|
# wrong=0 BITE: a Determined on ANY confuser means the one-hop rule over-fired
|
|
# (an asymmetric self-converse, a cross-predicate, an object mismatch, transitive-
|
|
# through-inverse, or an ungrounded guess). All must refuse. (The same-predicate
|
|
# strict-order transitive chain that used to live here MIGRATED to a determination
|
|
# under B2 — see evals/relational_transitive — so it is no longer a confuser.)
|
|
vocab, persona = vocab_persona
|
|
asserted = []
|
|
for case in _load_cases(_V1 / "refusals.jsonl"):
|
|
ctx = SessionContext(
|
|
vocab=vocab, persona=persona, vault_reproject_interval=_HIGH
|
|
)
|
|
for fact in case["facts"]:
|
|
realize_comprehension(comprehend_relational(fact, pack), ctx)
|
|
res = determine(comprehend_relational(case["query"], pack), ctx)
|
|
if isinstance(res, Determined):
|
|
asserted.append(
|
|
{
|
|
"id": case["id"],
|
|
"why": case["why"],
|
|
"got": [res.answer, res.predicate, res.subject, res.object],
|
|
}
|
|
)
|
|
assert not asserted, f"confuser(s) asserted (rule over-fired): {asserted}"
|