core/evals/relational_transitive/v1/refusals.jsonl
Shay 1ff06726a6
feat(determine): add transitive strict-order relational inference (#781)
* feat(determine): add transitive strict-order relational inference

Add sound transitive closure for declared strict-order relational predicates —
less_than, greater_than, before_event, after_event, and ONLY these. A query
p(a, c) determines True when a same-predicate chain p(a, b), p(b, c), ... over the
predicate's OWN realized edges is found (BFS reachability) and the proof_chain ROBDD
verifies the transitive entailment (search-then-verify, through the UNCHANGED
evaluate_entailment via a new lower_transitive_chain). Mirrors _determine_subsumption.

Scope / firewall:
- TRANSITIVE_PREDICATES is closed and default-off; sibling_of, parent_of,
  left_of/right_of, inside_of, during_event, overlaps_event are EXCLUDED.
- Same-predicate edges only — no transitive-through-inverse, no cross-predicate
  (triple firewall: recall filter + lowering re-reject + ROBDD re-verify).
- Open-world: asserts only answer=True via the shared _relational_determined
  surface (rule="transitive"), so INV-30 stays at exactly 3 Determined sites and
  no answer=False is added. One-hop inverse/symmetric (#775) is unchanged.
  FrameVerdict / closed-world (ADR-0222) is untouched.

wrong=0 bite (unit + lane + independent oracle): non-transitive-predicate chains,
non-admitted spatial chains, mixed-predicate chains, disjoint chains, reflexive
cycles, and inverse+transitive composition all REFUSE.

Measurement: new evals/relational_transitive lane with an INDEPENDENT BFS
transitive-closure oracle (imports no engine module; INV-25/27), cross-checked both
directions (positives oracle-True, confusers oracle-False). Capability-index breadth
10 -> 11, wrong_total 0, deterministic digest re-frozen.

Cross-PR reconcile: migrated the one-hop confuser rinf-ref-001 (a less_than chain,
which now correctly determines transitively) to a B2 positive; repurposed its unit
test to a non-transitive parent_of chain; corrected a pre-existing stale breadth==9
assertion (#775 added a 10th domain but never updated test_capability_index).

Verified in the worktree: determine + transitive/one-hop lanes + capability
baseline/index + ProofWriter-OWA floor + INV-30/29/21/02 + full smoke — all green.
A 3-skeptic read-only adversarial review found no wrong=0 leak.

* test(determine): B2 hardening — TRANSITIVE_PREDICATES pin + lower_transitive defensive + budget

Add the required B2 hardening tests before merge:

- test_transitive_predicates_closed_and_excludes: the table is EXACTLY
  {less_than, greater_than, before_event, after_event}, every member is in
  RELATIONAL_PREDICATES, and every deliberately-excluded predicate (sibling_of,
  spouse_of, parent_of, child_of, left_of, right_of, inside_of, during_event,
  overlaps_event) is explicitly absent. (Closes the gap where relational.py's
  comment claimed this test existed but it did not.)

- tests/test_composition_lower_transitive.py: direct defensive tests on
  lower_transitive_chain — empty path, mislabeled (cross-predicate) edge, wrong
  arity, non-contiguous path, and path-not-reaching-target all lower to None
  (refuse); plus exact 2-hop and 3-hop theory pins.

- test_edge_budget_exhaustion_refuses: above _TRANSITIVE_EDGE_BUDGET the transitive
  search declines (a safe coverage refusal), never proves.
2026-06-15 14:20:30 -07:00

7 lines
2.5 KiB
JSON

{"id": "rtrans-ref-001", "facts": ["Alice is the sibling of Bob.", "Bob is the sibling of Carol."], "query": "Is Alice the sibling of Carol?", "edges": [["sibling_of", "alice", "bob"], ["sibling_of", "bob", "carol"]], "query_edge": ["sibling_of", "alice", "carol"], "why": "sibling_of is symmetric, not transitive — a sib b, b sib c does NOT give a sib c; must refuse"}
{"id": "rtrans-ref-002", "facts": ["Alice is the parent of Bob.", "Bob is the parent of Carol."], "query": "Is Alice the parent of Carol?", "edges": [["parent_of", "alice", "bob"], ["parent_of", "bob", "carol"]], "query_edge": ["parent_of", "alice", "carol"], "why": "parent ∘ parent = grandparent ≠ parent — parent_of is not transitive; must refuse"}
{"id": "rtrans-ref-003", "facts": ["The box is left of the cup.", "The cup is left of the lamp."], "query": "Is the box left of the lamp?", "edges": [["left_of", "box", "cup"], ["left_of", "cup", "lamp"]], "query_edge": ["left_of", "box", "lamp"], "why": "left_of is not admitted as transitive (needs an explicit shared-frame / total-order proof); must refuse"}
{"id": "rtrans-ref-004", "facts": ["Alice is less than Bob.", "Bob is before Carol."], "query": "Is Alice less than Carol?", "edges": [["less_than", "alice", "bob"], ["before_event", "bob", "carol"]], "query_edge": ["less_than", "alice", "carol"], "why": "mixed predicates do not compose — the less_than search walks only less_than edges; carol unreachable; must refuse"}
{"id": "rtrans-ref-005", "facts": ["Alice is less than Bob.", "Bob is less than Alice."], "query": "Is Alice less than Alice?", "edges": [["less_than", "alice", "bob"], ["less_than", "bob", "alice"]], "query_edge": ["less_than", "alice", "alice"], "why": "strict orders are irreflexive — a reflexive query (start == target) refuses; the conflicting cycle must not fabricate a result"}
{"id": "rtrans-ref-006", "facts": ["Alice is less than Bob.", "Carol is less than Dan."], "query": "Is Alice less than Dan?", "edges": [["less_than", "alice", "bob"], ["less_than", "carol", "dan"]], "query_edge": ["less_than", "alice", "dan"], "why": "disjoint chains — no contiguous path from alice to dan (missing middle); must refuse"}
{"id": "rtrans-ref-007", "facts": ["Bob is less than Alice.", "Carol is less than Bob."], "query": "Is Alice greater than Carol?", "edges": [["less_than", "bob", "alice"], ["less_than", "carol", "bob"]], "query_edge": ["greater_than", "alice", "carol"], "why": "inverse+transitive composition is out of scope for B2 — greater_than has no edges to chain and one-hop inverse is a single hop; must refuse"}