1ff06726a6
* feat(determine): add transitive strict-order relational inference Add sound transitive closure for declared strict-order relational predicates — less_than, greater_than, before_event, after_event, and ONLY these. A query p(a, c) determines True when a same-predicate chain p(a, b), p(b, c), ... over the predicate's OWN realized edges is found (BFS reachability) and the proof_chain ROBDD verifies the transitive entailment (search-then-verify, through the UNCHANGED evaluate_entailment via a new lower_transitive_chain). Mirrors _determine_subsumption. Scope / firewall: - TRANSITIVE_PREDICATES is closed and default-off; sibling_of, parent_of, left_of/right_of, inside_of, during_event, overlaps_event are EXCLUDED. - Same-predicate edges only — no transitive-through-inverse, no cross-predicate (triple firewall: recall filter + lowering re-reject + ROBDD re-verify). - Open-world: asserts only answer=True via the shared _relational_determined surface (rule="transitive"), so INV-30 stays at exactly 3 Determined sites and no answer=False is added. One-hop inverse/symmetric (#775) is unchanged. FrameVerdict / closed-world (ADR-0222) is untouched. wrong=0 bite (unit + lane + independent oracle): non-transitive-predicate chains, non-admitted spatial chains, mixed-predicate chains, disjoint chains, reflexive cycles, and inverse+transitive composition all REFUSE. Measurement: new evals/relational_transitive lane with an INDEPENDENT BFS transitive-closure oracle (imports no engine module; INV-25/27), cross-checked both directions (positives oracle-True, confusers oracle-False). Capability-index breadth 10 -> 11, wrong_total 0, deterministic digest re-frozen. Cross-PR reconcile: migrated the one-hop confuser rinf-ref-001 (a less_than chain, which now correctly determines transitively) to a B2 positive; repurposed its unit test to a non-transitive parent_of chain; corrected a pre-existing stale breadth==9 assertion (#775 added a 10th domain but never updated test_capability_index). Verified in the worktree: determine + transitive/one-hop lanes + capability baseline/index + ProofWriter-OWA floor + INV-30/29/21/02 + full smoke — all green. A 3-skeptic read-only adversarial review found no wrong=0 leak. * test(determine): B2 hardening — TRANSITIVE_PREDICATES pin + lower_transitive defensive + budget Add the required B2 hardening tests before merge: - test_transitive_predicates_closed_and_excludes: the table is EXACTLY {less_than, greater_than, before_event, after_event}, every member is in RELATIONAL_PREDICATES, and every deliberately-excluded predicate (sibling_of, spouse_of, parent_of, child_of, left_of, right_of, inside_of, during_event, overlaps_event) is explicitly absent. (Closes the gap where relational.py's comment claimed this test existed but it did not.) - tests/test_composition_lower_transitive.py: direct defensive tests on lower_transitive_chain — empty path, mislabeled (cross-predicate) edge, wrong arity, non-contiguous path, and path-not-reaching-target all lower to None (refuse); plus exact 2-hop and 3-hop theory pins. - test_edge_budget_exhaustion_refuses: above _TRANSITIVE_EDGE_BUDGET the transitive search declines (a safe coverage refusal), never proves.
102 lines
3.4 KiB
Python
102 lines
3.4 KiB
Python
"""Score the transitive strict-order relational DETERMINE capability on independent gold.
|
|
|
|
told fact(s) -> ``determine(query)`` vs gold derived from each case's ``query_edge``
|
|
(``[predicate, subject, object]``). The gold is CROSS-CHECKED against the independent
|
|
transitive-closure oracle (``evals.relational_transitive.oracle``), which is disjoint from
|
|
``generate.determine`` (INV-25 / INV-27) — the engine never produced it.
|
|
|
|
A refusal is a COVERAGE miss (``refused``), never a wrong; only a ``Determined`` that
|
|
disagrees with gold — wrong predicate/subject/object, ``answer`` not True, or ``rule`` not
|
|
``"transitive"`` — is ``wrong``, and wrong must stay 0. The confusers that MUST refuse live
|
|
in ``v1/refusals.jsonl`` + the dedicated lane test. This is the positive-coverage lane that
|
|
puts transitive relational inference ON the capability index (breadth 10->11).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
from chat.runtime import ChatRuntime
|
|
from generate.determine import Determined, determine
|
|
from generate.meaning_graph.relational import (
|
|
comprehend_relational,
|
|
load_relational_pack_lemmas,
|
|
)
|
|
from generate.realize import realize_comprehension
|
|
from session.context import SessionContext
|
|
|
|
_CASES = Path(__file__).resolve().parent / "v1" / "cases.jsonl"
|
|
_HIGH = 10**9
|
|
|
|
|
|
def _load_cases(path: Path) -> list[dict[str, Any]]:
|
|
return [
|
|
json.loads(line)
|
|
for line in path.read_text(encoding="utf-8").splitlines()
|
|
if line.strip()
|
|
]
|
|
|
|
|
|
def run(path: Path = _CASES) -> dict[str, Any]:
|
|
cases = _load_cases(path)
|
|
pack = load_relational_pack_lemmas()
|
|
rt = ChatRuntime(no_load_state=True)
|
|
vocab, persona = rt._context.vocab, rt._context.persona
|
|
|
|
correct = wrong = refused = 0
|
|
wrongs: list[dict[str, Any]] = []
|
|
|
|
for case in cases:
|
|
ctx = SessionContext(
|
|
vocab=vocab, persona=persona, vault_reproject_interval=_HIGH
|
|
)
|
|
for fact in case["facts"]:
|
|
realize_comprehension(comprehend_relational(fact, pack), ctx)
|
|
res = determine(comprehend_relational(case["query"], pack), ctx)
|
|
if not isinstance(res, Determined):
|
|
refused += 1 # coverage miss, not a wrong
|
|
continue
|
|
predicate, subject, obj = case["query_edge"]
|
|
got = [res.answer, res.predicate, res.subject, res.object, res.rule]
|
|
gold = [True, predicate, subject, obj, "transitive"]
|
|
if got == gold:
|
|
correct += 1
|
|
else:
|
|
wrong += 1
|
|
wrongs.append({"id": case.get("id"), "got": got, "gold": gold})
|
|
|
|
return {
|
|
"domain": "comprehension_relational_transitive",
|
|
"total": len(cases),
|
|
"correct": correct,
|
|
"wrong": wrong,
|
|
"refused": refused,
|
|
"wrongs": wrongs,
|
|
"counts": {"correct": correct, "wrong": wrong, "refused": refused},
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
report = run()
|
|
print(
|
|
json.dumps(
|
|
{k: v for k, v in report.items() if k != "wrongs"},
|
|
indent=2,
|
|
sort_keys=True,
|
|
)
|
|
)
|
|
if report["wrong"]:
|
|
print(
|
|
"WRONG > 0 — transitive relational inference produced a wrong determination:",
|
|
file=sys.stderr,
|
|
)
|
|
print(json.dumps(report["wrongs"], indent=2), file=sys.stderr)
|
|
return 1
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
raise SystemExit(main())
|