feat: one-hop sound relational entailment (inverse/symmetric) + capability-index lane (#775)
* feat(determine): one-hop sound relational entailment — inverse/converse + symmetric Mastery-v2 Step 3 lead capability (core). DETERMINE could perceive relational structure (16 predicates, breadth-complete) but could not derive the simplest entailed relational facts — it read only the stored direction. This adds two SOUND one-hop rules that read a stored edge in its other lawful direction: INVERSE/converse greater_than(a,b) <= told less_than(b,a) SYMMETRIC sibling_of(b,a) <= told sibling_of(a,b) Strictly scoped (per review): OPEN-WORLD (asserts only True, never False — the answer=False path stays unbuilt, INV-30's firewall holds), ONE hop (NO transitive chaining), DECLARED rules only. Ontology/metadata-driven, not prose intuition: - generate/meaning_graph/relational.py: declarative algebra tables — _INVERSE_PAIRS (the converse edges; the pack carries no inverse metadata) + INVERSE_OF (derived, an involution) + SYMMETRIC_PREDICATES (mirrors the pack's graph.edge.symmetric tag). load_relational_pack_symmetric() reads the pack ontology; a test pins the constant equal to it (no silent divergence). - generate/determine/determine.py: _relational_one_hop() between direct entailment and transitive subsumption; new Determined.rule provenance field (direct/inverse/ symmetric/subsumption) — replay-safe (render reads only basis; trace hashes surface). wrong=0 confuser block proves the rule cannot over-fire: less_than is not self-inverse, sibling_of does not imply parent_of, greater_than does not imply equal_to, NO transitive chain (direct or through-inverse), and no answer=False is ever emitted. An obsolete pin (test_symmetric_converse_is_not_faked, which asserted the pre-rule direct-only floor) is updated to the new sound behavior and a preserved asymmetric-converse bite. Tests: test_determine_relational_inference.py (13, the capability contract) + updated test_relational_reader.py. Broad relational/determine/grounding + capability_index baseline sweep: 114 green; baseline digest UNCHANGED (this is a determination capability; the comprehension lanes are untouched). REMAINING for the full lead PR (next): the measurement lane — an independent-oracle evals/relational_inference lane + a capability_index adapter + baseline re-freeze (breadth 9->10, wrong_total still 0), so the capability registers on the yardstick. INTEGRATION NOTE (cross-PR): this adds 2 Determined() construction sites (now 4: direct/inverse/symmetric/subsumption, ALL answer=True). When rebased onto main with INV-30 (PR #770), update INV-30's test_determine_construction_sites_are_visible count 2 -> 4 and confirm all four assert True. Merge order: #770 -> this. * feat(capability-index): relational-inference lane — breadth 9->10, wrong=0 Mastery-v2 Step 3 lead, measurement half. Puts the one-hop relational-inference capability ON the capability index with an independent-oracle lane: - evals/relational_inference/v1/: cases.jsonl (13 positive: 8 inverse + 5 symmetric, authored from the relation algebra INDEPENDENTLY of determine/relational per INV-25/27), refusals.jsonl (8 confusers that MUST refuse), provenance.md (incl. the honest overlaps_event reader-surface coverage gap). - evals/comprehension/relational_inference_runner.py: told fact(s) -> determine(query) vs gold; refusal = coverage miss, disagreement = wrong (structurally 0 on positives). - evals/capability_index/adapters.py: comprehension_relational_inference_result added to ADAPTERS. - baseline.json re-frozen: breadth 9 -> 10, capability_score 0.94403 -> 0.949483, wrong_total still 0, digest deliberately changed (35dea2b2...). - tests/test_relational_inference_lane.py: SHA-pinned gold, positive wrong=0 + coverage>0, and the confuser wrong=0 BITE (every confuser must refuse). Also reconciles INV-30 (now in main via #770): determine.py grew from 2 to 3 Determined() construction sites (direct, the shared relational one-hop constructor, transitive subsumption) — ALL answer=True. test_determine_construction_sites_are_visible updated 2 -> 3 (correcting the prior commit note's overcount of 4; inverse and symmetric share the single _relational_determined constructor, so it is ONE new site). Also fixes a .gitignore gap: ADR-0219 gen-dir checkpoints (engine_state/current, engine_state/gen-*/) were not ignored (only the old flat engine_state/*.json patterns were), so runtime checkpoints could be committed by accident. Verification: new lane 13/0/0 coverage 1.0; capability_baseline digest matches the re-freeze; INV-30 + architectural invariants green; smoke 99/99. * docs+test: reconcile stale relational-inference docstrings; assert rule provenance (review #775) Addresses two review-requested cleanups on #775: 1. Stale docstrings. generate/meaning_graph/relational.py said the symmetric converse is "a sound-but-incomplete refusal at DETERMINE" and there is "NO transitive/ symmetric/rule inference"; generate/determine/determine.py said "symmetric-converse questions are Undetermined", "no transitive/symmetric/rule inference", and "asserts only answer=True on a direct hit". All false after the one-hop relational algebra landed. Updated: the reader does direct-reading only; DETERMINE applies declared one-hop inverse/converse + pack-declared symmetric (plus the existing member/subset subsumption), still open-world / never answer=False; transitive relational closure, negation, and closed-world falsehood remain out of scope. 2. Rule-provenance gold now meaningful. The relational_inference runner compared only (answer, predicate, subject, object), ignoring res.rule though the gold carries "rule": "inverse"/"symmetric". Added res.rule to the got/gold tuple so a "right answer via the wrong rule path" can no longer pass silently. Lane still 13/0/0; baseline counts (and digest) unchanged. Verification: runner 13/0/0; lane + capability_baseline + unit + INV-30 green (22).
This commit is contained in:
parent
512453b6fc
commit
6f70e834ff
12 changed files with 615 additions and 36 deletions
|
|
@ -84,13 +84,22 @@ def comprehension_relational_predicate_result() -> DomainResult:
|
||||||
return DomainResult("comprehension_relational_predicate", c, w, r)
|
return DomainResult("comprehension_relational_predicate", c, w, r)
|
||||||
|
|
||||||
|
|
||||||
|
def comprehension_relational_inference_result() -> DomainResult:
|
||||||
|
from evals.comprehension.relational_inference_runner import run
|
||||||
|
|
||||||
|
c, w, r = _counts(run())
|
||||||
|
return DomainResult("comprehension_relational_inference", c, w, r)
|
||||||
|
|
||||||
|
|
||||||
#: The reasoning domains currently composed into the index (self-loading lanes).
|
#: The reasoning domains currently composed into the index (self-loading lanes).
|
||||||
#: The six ``comprehension_*`` lanes score the GENERAL comprehension reader; the
|
#: The six ``comprehension_*`` lanes score the GENERAL comprehension reader; the
|
||||||
#: relational_metric one reads arithmetic prose into the binding-graph quantity
|
#: relational_metric one reads arithmetic prose into the binding-graph quantity
|
||||||
#: substrate (admissibility-checked) and projects to the arithmetic oracle, and the
|
#: substrate (admissibility-checked) and projects to the arithmetic oracle, and the
|
||||||
#: relational_predicate one (#596) reads binary-relation prose into pack-named
|
#: relational_predicate one (#596) reads binary-relation prose into pack-named
|
||||||
#: predicates — so the index now measures comprehension breadth across categorical,
|
#: predicates, and the relational_inference one DETERMINES one-hop inverse/symmetric
|
||||||
#: ordering, propositional, quantitative, AND relational reasoning.
|
#: entailments (mastery-v2 Step 3) — so the index measures comprehension breadth across
|
||||||
|
#: categorical, ordering, propositional, quantitative, relational-reading, AND
|
||||||
|
#: relational-inference reasoning.
|
||||||
ADAPTERS = (
|
ADAPTERS = (
|
||||||
deductive_logic_result,
|
deductive_logic_result,
|
||||||
relational_metric_result,
|
relational_metric_result,
|
||||||
|
|
@ -101,6 +110,7 @@ ADAPTERS = (
|
||||||
comprehension_propositional_result,
|
comprehension_propositional_result,
|
||||||
comprehension_relational_metric_result,
|
comprehension_relational_metric_result,
|
||||||
comprehension_relational_predicate_result,
|
comprehension_relational_predicate_result,
|
||||||
|
comprehension_relational_inference_result,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,13 @@
|
||||||
{
|
{
|
||||||
"capability_score": 0.94403,
|
"capability_score": 0.949483,
|
||||||
"coverage_geomean": 0.94403,
|
"coverage_geomean": 0.949483,
|
||||||
"coverage_micro": 0.993835,
|
"coverage_micro": 0.993932,
|
||||||
"accuracy_micro": 1.0,
|
"accuracy_micro": 1.0,
|
||||||
"breadth": 9,
|
"breadth": 10,
|
||||||
"min_domain_coverage": 0.833333,
|
"min_domain_coverage": 0.833333,
|
||||||
"wrong_total": 0,
|
"wrong_total": 0,
|
||||||
"assert_mode_valid": true,
|
"assert_mode_valid": true,
|
||||||
"deterministic_digest": "1ea91c1eefd6a0c4739e6fc97a40d8522d9b0758b4531bce448ec98e25789ccf",
|
"deterministic_digest": "35dea2b23084de9d56bd97225ccee6d9c07e650551559dd26c4fed7324806ffc",
|
||||||
"domains": [
|
"domains": [
|
||||||
{
|
{
|
||||||
"domain": "comprehension_propositional",
|
"domain": "comprehension_propositional",
|
||||||
|
|
@ -17,6 +17,14 @@
|
||||||
"coverage": 1.0,
|
"coverage": 1.0,
|
||||||
"accuracy": 1.0
|
"accuracy": 1.0
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"domain": "comprehension_relational_inference",
|
||||||
|
"correct": 13,
|
||||||
|
"wrong": 0,
|
||||||
|
"refused": 0,
|
||||||
|
"coverage": 1.0,
|
||||||
|
"accuracy": 1.0
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"domain": "comprehension_relational_metric",
|
"domain": "comprehension_relational_metric",
|
||||||
"correct": 15,
|
"correct": 15,
|
||||||
|
|
|
||||||
106
evals/comprehension/relational_inference_runner.py
Normal file
106
evals/comprehension/relational_inference_runner.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
"""Score the one-hop relational-inference DETERMINE capability on independent gold.
|
||||||
|
|
||||||
|
told fact(s) -> ``determine(query)`` vs hand-authored gold ``(predicate, subject,
|
||||||
|
object)``. The gold is authored by reading the inverse/symmetric relational algebra,
|
||||||
|
INDEPENDENTLY of ``generate.determine`` / ``generate.meaning_graph.relational``
|
||||||
|
(INV-25 / INV-27) — the engine never produced it.
|
||||||
|
|
||||||
|
A refusal is a COVERAGE miss (``refused``), never a wrong; only a ``Determined`` that
|
||||||
|
disagrees with gold — wrong predicate/subject/object, or ``answer`` not True — is
|
||||||
|
``wrong``, and wrong must stay 0. The confuser cases that MUST refuse live in
|
||||||
|
``refusals.jsonl`` + the dedicated lane test. This is the positive-coverage lane that
|
||||||
|
puts the one-hop relational-inference capability ON the capability index (breadth 9->10).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from chat.runtime import ChatRuntime
|
||||||
|
from generate.determine import Determined, determine
|
||||||
|
from generate.meaning_graph.relational import (
|
||||||
|
comprehend_relational,
|
||||||
|
load_relational_pack_lemmas,
|
||||||
|
)
|
||||||
|
from generate.realize import realize_comprehension
|
||||||
|
from session.context import SessionContext
|
||||||
|
|
||||||
|
_CASES = (
|
||||||
|
Path(__file__).resolve().parent.parent
|
||||||
|
/ "relational_inference"
|
||||||
|
/ "v1"
|
||||||
|
/ "cases.jsonl"
|
||||||
|
)
|
||||||
|
_HIGH = 10**9
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cases(path: Path = _CASES) -> list[dict[str, Any]]:
|
||||||
|
return [
|
||||||
|
json.loads(line)
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines()
|
||||||
|
if line.strip()
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def run(path: Path = _CASES) -> dict[str, Any]:
|
||||||
|
cases = _load_cases(path)
|
||||||
|
pack = load_relational_pack_lemmas()
|
||||||
|
rt = ChatRuntime(no_load_state=True)
|
||||||
|
vocab, persona = rt._context.vocab, rt._context.persona
|
||||||
|
|
||||||
|
correct = wrong = refused = 0
|
||||||
|
wrongs: list[dict[str, Any]] = []
|
||||||
|
|
||||||
|
for case in cases:
|
||||||
|
ctx = SessionContext(
|
||||||
|
vocab=vocab, persona=persona, vault_reproject_interval=_HIGH
|
||||||
|
)
|
||||||
|
for fact in case["facts"]:
|
||||||
|
realize_comprehension(comprehend_relational(fact, pack), ctx)
|
||||||
|
res = determine(comprehend_relational(case["query"], pack), ctx)
|
||||||
|
if not isinstance(res, Determined):
|
||||||
|
refused += 1 # coverage miss, not a wrong
|
||||||
|
continue
|
||||||
|
got = [res.answer, res.predicate, res.subject, res.object, res.rule]
|
||||||
|
gold = [True, case["predicate"], case["subject"], case["object"], case["rule"]]
|
||||||
|
if got == gold:
|
||||||
|
correct += 1
|
||||||
|
else:
|
||||||
|
wrong += 1
|
||||||
|
wrongs.append({"id": case.get("id"), "got": got, "gold": gold})
|
||||||
|
|
||||||
|
return {
|
||||||
|
"domain": "comprehension_relational_inference",
|
||||||
|
"total": len(cases),
|
||||||
|
"correct": correct,
|
||||||
|
"wrong": wrong,
|
||||||
|
"refused": refused,
|
||||||
|
"wrongs": wrongs,
|
||||||
|
"counts": {"correct": correct, "wrong": wrong, "refused": refused},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
report = run()
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{k: v for k, v in report.items() if k != "wrongs"},
|
||||||
|
indent=2,
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if report["wrong"]:
|
||||||
|
print(
|
||||||
|
"WRONG > 0 — relational inference produced a wrong determination:",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
print(json.dumps(report["wrongs"], indent=2), file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
13
evals/relational_inference/v1/cases.jsonl
Normal file
13
evals/relational_inference/v1/cases.jsonl
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
{"id": "rinf-001", "facts": ["Bob is less than Alice."], "query": "Is Alice greater than Bob?", "predicate": "greater_than", "subject": "alice", "object": "bob", "rule": "inverse"}
|
||||||
|
{"id": "rinf-002", "facts": ["Alice is greater than Bob."], "query": "Is Bob less than Alice?", "predicate": "less_than", "subject": "bob", "object": "alice", "rule": "inverse"}
|
||||||
|
{"id": "rinf-003", "facts": ["Alice is the parent of Bob."], "query": "Is Bob the child of Alice?", "predicate": "child_of", "subject": "bob", "object": "alice", "rule": "inverse"}
|
||||||
|
{"id": "rinf-004", "facts": ["Bob is the child of Alice."], "query": "Is Alice the parent of Bob?", "predicate": "parent_of", "subject": "alice", "object": "bob", "rule": "inverse"}
|
||||||
|
{"id": "rinf-005", "facts": ["The box is left of the cup."], "query": "Is the cup right of the box?", "predicate": "right_of", "subject": "cup", "object": "box", "rule": "inverse"}
|
||||||
|
{"id": "rinf-006", "facts": ["The cup is right of the box."], "query": "Is the box left of the cup?", "predicate": "left_of", "subject": "box", "object": "cup", "rule": "inverse"}
|
||||||
|
{"id": "rinf-007", "facts": ["Dawn is before noon."], "query": "Is noon after dawn?", "predicate": "after_event", "subject": "noon", "object": "dawn", "rule": "inverse"}
|
||||||
|
{"id": "rinf-008", "facts": ["Noon is after dawn."], "query": "Is dawn before noon?", "predicate": "before_event", "subject": "dawn", "object": "noon", "rule": "inverse"}
|
||||||
|
{"id": "rinf-009", "facts": ["Alice is the sibling of Bob."], "query": "Is Bob the sibling of Alice?", "predicate": "sibling_of", "subject": "bob", "object": "alice", "rule": "symmetric"}
|
||||||
|
{"id": "rinf-010", "facts": ["Alice is the spouse of Bob."], "query": "Is Bob the spouse of Alice?", "predicate": "spouse_of", "subject": "bob", "object": "alice", "rule": "symmetric"}
|
||||||
|
{"id": "rinf-011", "facts": ["Gold is equal to silver."], "query": "Is silver equal to gold?", "predicate": "equal_to", "subject": "silver", "object": "gold", "rule": "symmetric"}
|
||||||
|
{"id": "rinf-012", "facts": ["Gold is distinct from silver."], "query": "Is silver distinct from gold?", "predicate": "distinct_from", "subject": "silver", "object": "gold", "rule": "symmetric"}
|
||||||
|
{"id": "rinf-013", "facts": ["The room is adjacent to the hall."], "query": "Is the hall adjacent to the room?", "predicate": "adjacent_to", "subject": "hall", "object": "room", "rule": "symmetric"}
|
||||||
35
evals/relational_inference/v1/provenance.md
Normal file
35
evals/relational_inference/v1/provenance.md
Normal file
|
|
@ -0,0 +1,35 @@
|
||||||
|
# evals/relational_inference/v1 — provenance
|
||||||
|
|
||||||
|
Independent gold for the one-hop relational-inference DETERMINE capability
|
||||||
|
(mastery-v2 Step 3). The gold `(predicate, subject, object)` triples are authored by
|
||||||
|
reading the relational predicate algebra (inverse/converse + symmetric)
|
||||||
|
**independently** of `generate/determine/determine.py` and
|
||||||
|
`generate/meaning_graph/relational.py` (INV-25 / INV-27) — the reader/determiner never
|
||||||
|
produced them.
|
||||||
|
|
||||||
|
## cases.jsonl — positive coverage (13)
|
||||||
|
|
||||||
|
Each case: told fact(s) → `determine(query)` should assert the entailed relation.
|
||||||
|
|
||||||
|
- **8 INVERSE/converse** — both directions of `less_than`/`greater_than`,
|
||||||
|
`parent_of`/`child_of`, `left_of`/`right_of`, `before_event`/`after_event`.
|
||||||
|
- **5 SYMMETRIC** — `sibling_of`, `spouse_of`, `equal_to`, `distinct_from`,
|
||||||
|
`adjacent_to`, each told one direction and queried the other.
|
||||||
|
|
||||||
|
A refusal here is a COVERAGE miss (counted `refused`), never a wrong.
|
||||||
|
|
||||||
|
## refusals.jsonl — confusers that MUST refuse (8)
|
||||||
|
|
||||||
|
Transitive (direct + through-inverse), asymmetric self-converse (`less_than`,
|
||||||
|
`parent_of`), cross-predicate (`sibling_of` ≠ `parent_of`, `greater_than` ≠
|
||||||
|
`equal_to`), object mismatch, and ungrounded. A `Determined` on any of these is a
|
||||||
|
wrong=0 breach — the lane test fails loudly.
|
||||||
|
|
||||||
|
## Known coverage gap (honest)
|
||||||
|
|
||||||
|
`overlaps_event` (symmetric, pack-declared `graph.edge.symmetric`) is **excluded** from
|
||||||
|
the positive gold: the relational reader requires a copula ("X is _connective_ Y"), but
|
||||||
|
`overlaps` is a finite verb — "The meeting overlaps the lunch" refuses
|
||||||
|
(`no_relational_template`) and "is overlaps" is ungrammatical. This is a **reader
|
||||||
|
surface limitation, not a determination gap** (the symmetric rule would fire if a fact
|
||||||
|
were stored). A finite-verb relational surface is brief-pack follow-up work.
|
||||||
8
evals/relational_inference/v1/refusals.jsonl
Normal file
8
evals/relational_inference/v1/refusals.jsonl
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
{"id": "rinf-ref-001", "facts": ["Alice is less than Bob.", "Bob is less than Carol."], "query": "Is Alice less than Carol?", "why": "transitive chain — out of scope (one hop only); must refuse"}
|
||||||
|
{"id": "rinf-ref-002", "facts": ["Bob is less than Alice.", "Carol is less than Bob."], "query": "Is Alice greater than Carol?", "why": "transitive through the inverse rule — must refuse (one hop)"}
|
||||||
|
{"id": "rinf-ref-003", "facts": ["Alice is less than Bob."], "query": "Is Bob less than Alice?", "why": "less_than is asymmetric — a<b does NOT entail b<a; must refuse"}
|
||||||
|
{"id": "rinf-ref-004", "facts": ["Alice is the parent of Bob."], "query": "Is Bob the parent of Alice?", "why": "parent_of is not symmetric (its converse is child_of); must refuse"}
|
||||||
|
{"id": "rinf-ref-005", "facts": ["Alice is the sibling of Bob."], "query": "Is Alice the parent of Bob?", "why": "cross-predicate — sibling_of does not imply parent_of; must refuse"}
|
||||||
|
{"id": "rinf-ref-006", "facts": ["Alice is greater than Bob."], "query": "Is Alice equal to Bob?", "why": "cross-predicate — greater_than does not imply equal_to; must refuse"}
|
||||||
|
{"id": "rinf-ref-007", "facts": ["Bob is less than Alice."], "query": "Is Alice greater than Carol?", "why": "object mismatch — inverse edge is (alice,bob) not (alice,carol); must refuse"}
|
||||||
|
{"id": "rinf-ref-008", "facts": [], "query": "Is Alice the sibling of Bob?", "why": "ungrounded — no told fact; absence never asserts (open-world); must refuse"}
|
||||||
|
|
@ -9,11 +9,12 @@ what I was told (unverified)". Until COHERENT promotion exists (out of scope), D
|
||||||
produces only ``as_told`` assertions or ``Undetermined`` refusals. No estimation, no
|
produces only ``as_told`` assertions or ``Undetermined`` refusals. No estimation, no
|
||||||
corpus mutation (teaching stays HITL proposal-only).
|
corpus mutation (teaching stays HITL proposal-only).
|
||||||
|
|
||||||
wrong=0 / soundness (open-world): D0 asserts an answer ONLY when the asked relation
|
wrong=0 / soundness (open-world): D0 asserts an answer ONLY when the asked relation is
|
||||||
is DIRECTLY entailed by a realized fact. Absence of a fact never refutes it
|
SOUNDLY entailed by realized facts — directly, by ONE-HOP relational algebra
|
||||||
(open-world), so D0 never asserts an answer from missing knowledge — it refuses
|
(inverse/converse + pack-declared symmetric), or by transitive member/subset
|
||||||
(``Undetermined``). It asserts only ``answer=True`` on a direct hit; it never asserts
|
subsumption. Absence of a fact never refutes it (open-world), so D0 never asserts an
|
||||||
False.
|
answer from missing knowledge — it refuses (``Undetermined``). It asserts only
|
||||||
|
``answer=True``; it never asserts False.
|
||||||
|
|
||||||
Supported predicates are a CLOSED set for which DIRECT entailment is sound — a
|
Supported predicates are a CLOSED set for which DIRECT entailment is sound — a
|
||||||
realized ground fact ``p(subject, target)`` answers the asked ``p(subject, target)``:
|
realized ground fact ``p(subject, target)`` answers the asked ``p(subject, target)``:
|
||||||
|
|
@ -22,9 +23,12 @@ predicates of ``en_core_relational_predicates_v1`` (``parent_of``, ``less_than``
|
||||||
``left_of``, ``before_event`` …; see ``generate.meaning_graph.relational``). The
|
``left_of``, ``before_event`` …; see ``generate.meaning_graph.relational``). The
|
||||||
categorical (``subset``/``disjoint`` …) and propositional (``implies``/``or`` …)
|
categorical (``subset``/``disjoint`` …) and propositional (``implies``/``or`` …)
|
||||||
predicates are deliberately EXCLUDED — their truth is not a stored-pair lookup, so
|
predicates are deliberately EXCLUDED — their truth is not a stored-pair lookup, so
|
||||||
admitting them would be unsound. Negated questions, symmetric-converse questions, and
|
admitting them would be unsound. Negated questions and any predicate outside the closed
|
||||||
any predicate outside the closed set are an honest ``Undetermined`` — D0 ships no
|
set are an honest ``Undetermined``. Beyond direct entailment, D0 applies two SOUND
|
||||||
entailment path it cannot exercise (no transitive/symmetric/rule inference).
|
extensions over the realized facts — ONE-HOP relational algebra (inverse/converse +
|
||||||
|
pack-declared symmetric; see ``generate.meaning_graph.relational``) and transitive
|
||||||
|
member/subset SUBSUMPTION — each search-then-verified, never closed-world, never
|
||||||
|
``answer=False``.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -33,7 +37,11 @@ from dataclasses import dataclass
|
||||||
|
|
||||||
from generate.composition import LogicChainPlan, lower_logic_chain
|
from generate.composition import LogicChainPlan, lower_logic_chain
|
||||||
from generate.meaning_graph.reader import Comprehension, Refusal
|
from generate.meaning_graph.reader import Comprehension, Refusal
|
||||||
from generate.meaning_graph.relational import RELATIONAL_PREDICATES
|
from generate.meaning_graph.relational import (
|
||||||
|
INVERSE_OF,
|
||||||
|
RELATIONAL_PREDICATES,
|
||||||
|
SYMMETRIC_PREDICATES,
|
||||||
|
)
|
||||||
from generate.realize import RealizedRecord, recall_realized
|
from generate.realize import RealizedRecord, recall_realized
|
||||||
from session.context import SessionContext
|
from session.context import SessionContext
|
||||||
from teaching.epistemic import ADMISSIBLE_AS_EVIDENCE, EpistemicStatus
|
from teaching.epistemic import ADMISSIBLE_AS_EVIDENCE, EpistemicStatus
|
||||||
|
|
@ -77,6 +85,12 @@ class Determined:
|
||||||
subject: str
|
subject: str
|
||||||
object: str
|
object: str
|
||||||
grounds: tuple[RealizedRecord, ...]
|
grounds: tuple[RealizedRecord, ...]
|
||||||
|
#: Which sound rule produced the answer — provenance for audit/replay. One of
|
||||||
|
#: ``direct`` (a stored fact of the asked predicate AND direction), ``inverse`` /
|
||||||
|
#: ``symmetric`` (a one-hop relational-algebra rule reading the stored edge in its
|
||||||
|
#: other lawful direction), or ``subsumption`` (transitive is-a chaining). It does
|
||||||
|
#: NOT affect the surface — ``render_determination`` reads only ``basis``.
|
||||||
|
rule: str = "direct"
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
@dataclass(frozen=True, slots=True)
|
||||||
|
|
@ -140,6 +154,14 @@ def determine(
|
||||||
grounds=(grounding,),
|
grounds=(grounding,),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# 1b. RELATIONAL one-hop entailment: a SOUND inverse/converse or symmetric rule
|
||||||
|
# reads a stored edge in its OTHER lawful direction. Open-world (asserts only True),
|
||||||
|
# ONE hop (no transitive chaining), structurally sound by construction — the same
|
||||||
|
# discipline as direct entailment; never False, never an undeclared rule.
|
||||||
|
relational = _relational_one_hop(ctx, predicate, subject, target)
|
||||||
|
if relational is not None:
|
||||||
|
return relational
|
||||||
|
|
||||||
# 2. TRANSITIVE subsumption (C): when direct entailment misses, a member/subset
|
# 2. TRANSITIVE subsumption (C): when direct entailment misses, a member/subset
|
||||||
# query may still hold by SOUND is-a chaining (member∘subset, subset∘subset) decided
|
# query may still hold by SOUND is-a chaining (member∘subset, subset∘subset) decided
|
||||||
# by the sound+complete proof_chain ROBDD — NEVER member∘member.
|
# by the sound+complete proof_chain ROBDD — NEVER member∘member.
|
||||||
|
|
@ -153,6 +175,57 @@ def determine(
|
||||||
return Undetermined("ungrounded" if not direct else "not_entailed")
|
return Undetermined("ungrounded" if not direct else "not_entailed")
|
||||||
|
|
||||||
|
|
||||||
|
def _find_relational_edge(
|
||||||
|
ctx: SessionContext, predicate: str, a: str, b: str
|
||||||
|
) -> RealizedRecord | None:
|
||||||
|
"""A realized ``predicate(a, b)`` fact (exact structural recall, R1a), or ``None``."""
|
||||||
|
facts = recall_realized(ctx, subject=a, predicate=predicate)
|
||||||
|
return next((f for f in facts if f.relation_arguments == (a, b)), None)
|
||||||
|
|
||||||
|
|
||||||
|
def _relational_one_hop(
|
||||||
|
ctx: SessionContext, predicate: str, subject: str, target: str
|
||||||
|
) -> Determined | None:
|
||||||
|
"""Answer ``predicate(subject, target)`` by ONE sound relational-algebra rule, or
|
||||||
|
``None`` (the caller then refuses, open-world). Two rules, each reading a stored edge
|
||||||
|
in its OTHER lawful direction:
|
||||||
|
|
||||||
|
INVERSE/converse ``p(subject, target)`` <= stored ``inverse(p)(target, subject)``
|
||||||
|
SYMMETRIC ``p(subject, target)`` <= stored ``p(target, subject)``
|
||||||
|
|
||||||
|
NEVER transitive (one hop only), NEVER False (open-world), NEVER an undeclared rule:
|
||||||
|
inverse fires only for a declared converse pair and symmetric only for a pack-declared
|
||||||
|
symmetric predicate — so ``less_than`` is not self-inverse and ``parent_of`` is not
|
||||||
|
symmetric.
|
||||||
|
"""
|
||||||
|
inverse = INVERSE_OF.get(predicate)
|
||||||
|
if inverse is not None:
|
||||||
|
edge = _find_relational_edge(ctx, inverse, target, subject)
|
||||||
|
if edge is not None:
|
||||||
|
return _relational_determined(predicate, subject, target, edge, "inverse")
|
||||||
|
if predicate in SYMMETRIC_PREDICATES:
|
||||||
|
edge = _find_relational_edge(ctx, predicate, target, subject)
|
||||||
|
if edge is not None:
|
||||||
|
return _relational_determined(predicate, subject, target, edge, "symmetric")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _relational_determined(
|
||||||
|
predicate: str, subject: str, target: str, ground: RealizedRecord, rule: str
|
||||||
|
) -> Determined:
|
||||||
|
"""A one-hop relational ``Determined`` — answer True, basis from the ground's
|
||||||
|
standing (as_told today), the single stored edge as grounds, the rule recorded."""
|
||||||
|
return Determined(
|
||||||
|
answer=True,
|
||||||
|
basis=_basis((ground,)),
|
||||||
|
predicate=predicate,
|
||||||
|
subject=subject,
|
||||||
|
object=target,
|
||||||
|
grounds=(ground,),
|
||||||
|
rule=rule,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _subset_path(
|
def _subset_path(
|
||||||
start: str, target: str, supers: dict[str, list[tuple[str, RealizedRecord]]]
|
start: str, target: str, supers: dict[str, list[tuple[str, RealizedRecord]]]
|
||||||
) -> tuple[RealizedRecord, ...] | None:
|
) -> tuple[RealizedRecord, ...] | None:
|
||||||
|
|
@ -272,4 +345,5 @@ def _verify_subsumption(
|
||||||
subject=subject,
|
subject=subject,
|
||||||
object=target,
|
object=target,
|
||||||
grounds=grounds,
|
grounds=grounds,
|
||||||
|
rule="subsumption",
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -23,10 +23,12 @@ Fail-closed (wrong=0 at the comprehension layer):
|
||||||
("north station") canonicalizes per the join contract; an argument that still holds
|
("north station") canonicalizes per the join contract; an argument that still holds
|
||||||
relational structure refuses rather than fabricate a compound entity.
|
relational structure refuses rather than fabricate a compound entity.
|
||||||
|
|
||||||
Scope — ground binary relations, DIRECT reading only. NO transitive/symmetric/rule
|
Scope — this reader grounds stated binary relations in the STATED DIRECTION only; it
|
||||||
inference: a symmetric lemma (``sibling_of``, ``equal_to``, ``adjacent_to`` …) reads
|
performs no inference itself. DETERMINE then applies declared ONE-HOP relational algebra
|
||||||
ONLY the stated direction; the converse is a sound-but-incomplete refusal at DETERMINE,
|
over the realized facts — inverse/converse pairs and pack-declared symmetric predicates
|
||||||
never a fabricated assertion. Negation is out of grammar (a negated surface refuses).
|
(``INVERSE_OF`` / ``SYMMETRIC_PREDICATES``) — so a symmetric lemma's converse and an
|
||||||
|
inverse pair NOW determine soundly. Transitive relational closure, negation (out of
|
||||||
|
grammar — a negated surface refuses), and closed-world falsehood remain out of scope.
|
||||||
|
|
||||||
This reader is invoked EXPLICITLY (the pack is loaded by the caller, not default-mounted);
|
This reader is invoked EXPLICITLY (the pack is loaded by the caller, not default-mounted);
|
||||||
it does not perturb ``comprehend``'s templates or their wrong=0 tests.
|
it does not perturb ``comprehend``'s templates or their wrong=0 tests.
|
||||||
|
|
@ -86,6 +88,56 @@ _CONNECTIVE_TO_LEMMA: dict[tuple[str, ...], str] = {
|
||||||
#: for direct relational entailment. (The realized fact's predicate is one of these.)
|
#: for direct relational entailment. (The realized fact's predicate is one of these.)
|
||||||
RELATIONAL_PREDICATES = frozenset(_CONNECTIVE_TO_LEMMA.values())
|
RELATIONAL_PREDICATES = frozenset(_CONNECTIVE_TO_LEMMA.values())
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Relational algebra — the SOUND one-hop rules DETERMINE may apply. Each rule
|
||||||
|
# reads a stored fact in its OTHER lawful direction; open-world (asserts only
|
||||||
|
# True), ONE hop (no transitive chaining), structurally sound by construction.
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
#: INVERSE/converse pairs: ``p(a, b)`` holds iff ``q(b, a)`` holds. The pack carries
|
||||||
|
#: no inverse metadata (only the ``graph.edge.symmetric`` tag), so the converse edges
|
||||||
|
#: are declared here as a closed table; a test pins every member to
|
||||||
|
#: ``RELATIONAL_PREDICATES`` so a typo cannot mint an unknown predicate.
|
||||||
|
_INVERSE_PAIRS: frozenset[frozenset[str]] = frozenset({
|
||||||
|
frozenset({"parent_of", "child_of"}),
|
||||||
|
frozenset({"less_than", "greater_than"}),
|
||||||
|
frozenset({"left_of", "right_of"}),
|
||||||
|
frozenset({"before_event", "after_event"}),
|
||||||
|
})
|
||||||
|
|
||||||
|
#: ``lemma -> its converse lemma`` (both directions), derived from ``_INVERSE_PAIRS``.
|
||||||
|
#: A predicate NOT in this map has no converse — so ``less_than`` is never self-inverse.
|
||||||
|
INVERSE_OF: dict[str, str] = {
|
||||||
|
lemma: other
|
||||||
|
for pair in _INVERSE_PAIRS
|
||||||
|
for lemma, other in (tuple(pair), tuple(pair)[::-1])
|
||||||
|
}
|
||||||
|
|
||||||
|
#: The semantic-domain tag the relational pack uses to mark a symmetric predicate.
|
||||||
|
_SYMMETRIC_DOMAIN_TAG = "graph.edge.symmetric"
|
||||||
|
|
||||||
|
#: SYMMETRIC predicates: ``p(a, b)`` holds iff ``p(b, a)`` holds. This MIRRORS the pack
|
||||||
|
#: ontology (the ``graph.edge.symmetric`` tag, see ``load_relational_pack_symmetric``);
|
||||||
|
#: a test pins the two equal so the table can never silently diverge from the pack. A
|
||||||
|
#: predicate NOT here is asymmetric — so ``parent_of`` is never read in both directions.
|
||||||
|
SYMMETRIC_PREDICATES: frozenset[str] = frozenset({
|
||||||
|
"sibling_of", "spouse_of", "equal_to",
|
||||||
|
"distinct_from", "adjacent_to", "overlaps_event",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def load_relational_pack_symmetric() -> frozenset[str]:
|
||||||
|
"""The symmetric-predicate lemmas as DECLARED by the loaded pack ontology (the
|
||||||
|
``graph.edge.symmetric`` semantic-domain tag). ``SYMMETRIC_PREDICATES`` is pinned
|
||||||
|
equal to this by a test — the pack is the source of truth, the constant is the
|
||||||
|
runtime-cheap mirror (no per-determination pack load)."""
|
||||||
|
return frozenset(
|
||||||
|
entry.lemma
|
||||||
|
for entry in load_pack_entries(RELATIONAL_PACK_ID)
|
||||||
|
if _SYMMETRIC_DOMAIN_TAG in entry.semantic_domains
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
#: The article stripped from an argument slot's edges ("the box" -> "box"). The
|
#: The article stripped from an argument slot's edges ("the box" -> "box"). The
|
||||||
#: copula ("is") is NOT stripped here — it is enforced STRUCTURALLY (adjacent to the
|
#: copula ("is") is NOT stripped here — it is enforced STRUCTURALLY (adjacent to the
|
||||||
#: connective) so a dangling copula ("Monday before Friday is.") cannot read as a fact.
|
#: connective) so a dangling copula ("Monday before Friday is.") cannot read as a fact.
|
||||||
|
|
|
||||||
|
|
@ -2137,18 +2137,20 @@ class TestINV30OpenWorldDetermineNeverAssertsFalse:
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_determine_construction_sites_are_visible(self):
|
def test_determine_construction_sites_are_visible(self):
|
||||||
"""30c — the scan actually sees the open-world gear's two `Determined`
|
"""30c — the scan actually sees the open-world gear's three `Determined`
|
||||||
sites (direct entailment + transitive subsumption), and both assert
|
sites (direct entailment, one-hop relational inverse/symmetric, and
|
||||||
True. If the count drops the scan went blind, not clean; if `ok` drops
|
transitive subsumption), and all assert True. If the count drops the scan
|
||||||
the firewall was breached at its own source."""
|
went blind, not clean; if `ok` drops the firewall was breached at its own
|
||||||
|
source. (Grew 2 -> 3 when one-hop relational inference landed; both new
|
||||||
|
rules share the single `_relational_determined` constructor.)"""
|
||||||
determine_path = (
|
determine_path = (
|
||||||
PROJECT_ROOT_FOR_INV21 / "generate" / "determine" / "determine.py"
|
PROJECT_ROOT_FOR_INV21 / "generate" / "determine" / "determine.py"
|
||||||
)
|
)
|
||||||
constructions = _determined_constructions(
|
constructions = _determined_constructions(
|
||||||
ast.parse(determine_path.read_text())
|
ast.parse(determine_path.read_text())
|
||||||
)
|
)
|
||||||
assert len(constructions) == 2, (
|
assert len(constructions) == 3, (
|
||||||
"Expected exactly the two open-world `Determined(answer=True)` "
|
"Expected exactly the three open-world `Determined(answer=True)` "
|
||||||
f"construction sites in determine.py; saw {len(constructions)}. "
|
f"construction sites in determine.py; saw {len(constructions)}. "
|
||||||
"If the gear legitimately changed shape, update this count — but "
|
"If the gear legitimately changed shape, update this count — but "
|
||||||
"confirm every site still asserts only `answer=True`."
|
"confirm every site still asserts only `answer=True`."
|
||||||
|
|
|
||||||
183
tests/test_determine_relational_inference.py
Normal file
183
tests/test_determine_relational_inference.py
Normal file
|
|
@ -0,0 +1,183 @@
|
||||||
|
"""DETERMINE — one-hop sound relational entailment (mastery-v2 Step 3 lead).
|
||||||
|
|
||||||
|
A relational query may hold by ONE sound predicate-algebra rule that reads a stored
|
||||||
|
edge in its OTHER lawful direction:
|
||||||
|
|
||||||
|
INVERSE/converse greater_than(a, b) <= told less_than(b, a)
|
||||||
|
SYMMETRIC sibling_of(b, a) <= told sibling_of(a, b)
|
||||||
|
|
||||||
|
Scope is deliberately narrow: OPEN-WORLD (asserts only True, never False), ONE hop
|
||||||
|
(NO transitive chaining), DECLARED rules only (less_than is NOT self-inverse;
|
||||||
|
parent_of is NOT symmetric). Closed-world / assert-False / FrameVerdict are out of
|
||||||
|
scope (a later slice). The wrong=0 discipline lives in the confuser block below.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from chat.runtime import ChatRuntime
|
||||||
|
from generate.determine import Determined, Undetermined, determine
|
||||||
|
from generate.meaning_graph.relational import (
|
||||||
|
INVERSE_OF,
|
||||||
|
RELATIONAL_PREDICATES,
|
||||||
|
SYMMETRIC_PREDICATES,
|
||||||
|
comprehend_relational,
|
||||||
|
load_relational_pack_lemmas,
|
||||||
|
load_relational_pack_symmetric,
|
||||||
|
)
|
||||||
|
from generate.realize import realize_comprehension
|
||||||
|
from session.context import SessionContext
|
||||||
|
|
||||||
|
_HIGH = 10**9
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def vocab_persona():
|
||||||
|
rt = ChatRuntime(no_load_state=True)
|
||||||
|
return rt._context.vocab, rt._context.persona
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def pack():
|
||||||
|
return load_relational_pack_lemmas()
|
||||||
|
|
||||||
|
|
||||||
|
def _ctx(vocab_persona) -> SessionContext:
|
||||||
|
vocab, persona = vocab_persona
|
||||||
|
return SessionContext(vocab=vocab, persona=persona, vault_reproject_interval=_HIGH)
|
||||||
|
|
||||||
|
|
||||||
|
def _tell(text: str, ctx: SessionContext, pack):
|
||||||
|
return realize_comprehension(comprehend_relational(text, pack), ctx)
|
||||||
|
|
||||||
|
|
||||||
|
def _ask(text: str, ctx: SessionContext, pack):
|
||||||
|
return determine(comprehend_relational(text, pack), ctx)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Positive — the engine now derives the simplest entailed relational facts
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_inverse_converse_admits_true(vocab_persona, pack) -> None:
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Bob is less than Alice.", ctx, pack) # less_than(bob, alice)
|
||||||
|
res = _ask("Is Alice greater than Bob?", ctx, pack) # greater_than(alice, bob)
|
||||||
|
assert isinstance(res, Determined)
|
||||||
|
assert res.answer is True and res.basis == "as_told" and res.rule == "inverse"
|
||||||
|
assert res.predicate == "greater_than"
|
||||||
|
assert res.subject == "alice" and res.object == "bob"
|
||||||
|
# grounds = the single stored converse edge
|
||||||
|
assert len(res.grounds) == 1 and res.grounds[0].relation_predicate == "less_than"
|
||||||
|
|
||||||
|
|
||||||
|
def test_symmetric_admits_true(vocab_persona, pack) -> None:
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is the sibling of Bob.", ctx, pack) # sibling_of(alice, bob)
|
||||||
|
res = _ask("Is Bob the sibling of Alice?", ctx, pack) # sibling_of(bob, alice)
|
||||||
|
assert isinstance(res, Determined) and res.answer is True
|
||||||
|
assert res.rule == "symmetric" and res.predicate == "sibling_of"
|
||||||
|
assert res.subject == "bob" and res.object == "alice"
|
||||||
|
|
||||||
|
|
||||||
|
def test_direct_stored_direction_still_determines(vocab_persona, pack) -> None:
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is the sibling of Bob.", ctx, pack)
|
||||||
|
res = _ask("Is Alice the sibling of Bob?", ctx, pack) # the stored direction
|
||||||
|
assert isinstance(res, Determined) and res.answer is True and res.rule == "direct"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# wrong=0 confuser block — the rule must not over-fire
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_less_than_is_not_self_inverse(vocab_persona, pack) -> None:
|
||||||
|
"""less_than is asymmetric: a<b does NOT entail b<a (its converse is greater_than)."""
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is less than Bob.", ctx, pack) # less_than(alice, bob)
|
||||||
|
res = _ask("Is Bob less than Alice?", ctx, pack) # less_than(bob, alice)?
|
||||||
|
assert isinstance(res, Undetermined)
|
||||||
|
|
||||||
|
|
||||||
|
def test_sibling_does_not_imply_parent(vocab_persona, pack) -> None:
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is the sibling of Bob.", ctx, pack)
|
||||||
|
res = _ask("Is Alice the parent of Bob?", ctx, pack)
|
||||||
|
assert isinstance(res, Undetermined)
|
||||||
|
|
||||||
|
|
||||||
|
def test_greater_than_does_not_imply_equal(vocab_persona, pack) -> None:
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is greater than Bob.", ctx, pack)
|
||||||
|
res = _ask("Is Alice equal to Bob?", ctx, pack)
|
||||||
|
assert isinstance(res, Undetermined)
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_transitive_chain_direct(vocab_persona, pack) -> None:
|
||||||
|
"""One hop only: a<b and b<c must NOT entail a<c (transitive is a later slice)."""
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is less than Bob.", ctx, pack)
|
||||||
|
_tell("Bob is less than Carol.", ctx, pack)
|
||||||
|
res = _ask("Is Alice less than Carol?", ctx, pack)
|
||||||
|
assert isinstance(res, Undetermined)
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_transitive_chain_through_inverse(vocab_persona, pack) -> None:
|
||||||
|
"""The inverse rule is also one hop: it must not become a transitive bridge."""
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Bob is less than Alice.", ctx, pack) # => greater_than(alice, bob) by inverse
|
||||||
|
_tell("Carol is less than Bob.", ctx, pack) # => greater_than(bob, carol) by inverse
|
||||||
|
res = _ask("Is Alice greater than Carol?", ctx, pack) # needs transitive — refuse
|
||||||
|
assert isinstance(res, Undetermined)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unsupported_predicate_refuses(vocab_persona, pack) -> None:
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
# No told fact at all → ungrounded refusal, never a guess.
|
||||||
|
res = _ask("Is Alice the sibling of Bob?", ctx, pack)
|
||||||
|
assert isinstance(res, Undetermined)
|
||||||
|
|
||||||
|
|
||||||
|
def test_never_asserts_false(vocab_persona, pack) -> None:
|
||||||
|
"""Open-world: every determination is True-or-refuse; answer=False is unreachable
|
||||||
|
(INV-30 is the structural guarantee — this is the behavioral echo on this path)."""
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is less than Bob.", ctx, pack)
|
||||||
|
for q in (
|
||||||
|
"Is Bob less than Alice?",
|
||||||
|
"Is Alice greater than Bob?",
|
||||||
|
"Is Alice equal to Bob?",
|
||||||
|
"Is Alice the parent of Bob?",
|
||||||
|
"Is Alice less than Bob?",
|
||||||
|
):
|
||||||
|
res = _ask(q, ctx, pack)
|
||||||
|
if isinstance(res, Determined):
|
||||||
|
assert res.answer is True
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Ontology pins — the algebra cannot silently diverge from the pack / vocab
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_symmetric_table_matches_pack_ontology(pack) -> None:
|
||||||
|
"""SYMMETRIC_PREDICATES MUST equal the pack's graph.edge.symmetric declarations —
|
||||||
|
the pack is the source of truth; the constant is the runtime-cheap mirror."""
|
||||||
|
assert SYMMETRIC_PREDICATES == load_relational_pack_symmetric()
|
||||||
|
|
||||||
|
|
||||||
|
def test_algebra_members_are_relational_predicates() -> None:
|
||||||
|
"""Every inverse/symmetric lemma is a real reader predicate — a typo cannot mint
|
||||||
|
an unknown predicate into the determination path."""
|
||||||
|
for lemma in set(INVERSE_OF) | SYMMETRIC_PREDICATES:
|
||||||
|
assert lemma in RELATIONAL_PREDICATES
|
||||||
|
|
||||||
|
|
||||||
|
def test_inverse_is_an_involution() -> None:
|
||||||
|
"""inverse(inverse(p)) == p, and no predicate is its own inverse (asymmetry)."""
|
||||||
|
for lemma, other in INVERSE_OF.items():
|
||||||
|
assert lemma != other
|
||||||
|
assert INVERSE_OF[other] == lemma
|
||||||
79
tests/test_relational_inference_lane.py
Normal file
79
tests/test_relational_inference_lane.py
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
"""Lane test — one-hop relational inference (capability-index lane + wrong=0 bite).
|
||||||
|
|
||||||
|
The positive lane (cases.jsonl) is the capability-index coverage; the confuser lane
|
||||||
|
(refusals.jsonl) is the wrong=0 bite — every confuser MUST refuse, or the inverse/
|
||||||
|
symmetric rule over-fired. Fixture SHAs are pinned so the gold cannot drift silently.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from chat.runtime import ChatRuntime
|
||||||
|
from evals.comprehension.relational_inference_runner import _load_cases, run
|
||||||
|
from generate.determine import Determined, determine
|
||||||
|
from generate.meaning_graph.relational import (
|
||||||
|
comprehend_relational,
|
||||||
|
load_relational_pack_lemmas,
|
||||||
|
)
|
||||||
|
from generate.realize import realize_comprehension
|
||||||
|
from session.context import SessionContext
|
||||||
|
|
||||||
|
_V1 = Path(__file__).resolve().parent.parent / "evals" / "relational_inference" / "v1"
|
||||||
|
_CASES_SHA = "03310ecc3ab7b7bf26f0a1779709bf468e888bbca1e53a71cba1d269b5c1dd71"
|
||||||
|
_REFUSALS_SHA = "f137bbe03b91301d9aa802ab9274c7d43ff9c3966207a8ba5e5f8fc19f0f7243"
|
||||||
|
_HIGH = 10**9
|
||||||
|
|
||||||
|
|
||||||
|
def _sha(p: Path) -> str:
|
||||||
|
return hashlib.sha256(p.read_bytes()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def vocab_persona():
|
||||||
|
rt = ChatRuntime(no_load_state=True)
|
||||||
|
return rt._context.vocab, rt._context.persona
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def pack():
|
||||||
|
return load_relational_pack_lemmas()
|
||||||
|
|
||||||
|
|
||||||
|
def test_fixture_shas_pinned() -> None:
|
||||||
|
assert _sha(_V1 / "cases.jsonl") == _CASES_SHA
|
||||||
|
assert _sha(_V1 / "refusals.jsonl") == _REFUSALS_SHA
|
||||||
|
|
||||||
|
|
||||||
|
def test_positive_lane_wrong_zero_and_covers() -> None:
|
||||||
|
r = run()
|
||||||
|
assert r["wrong"] == 0, r["wrongs"]
|
||||||
|
assert r["correct"] > 0 # coverage > 0 — else the index geomean zeroes the score
|
||||||
|
assert r["correct"] + r["wrong"] + r["refused"] == r["total"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_confusers_all_refuse(vocab_persona, pack) -> None:
|
||||||
|
# wrong=0 BITE: a Determined on ANY confuser means the one-hop rule over-fired
|
||||||
|
# (a transitive chain, an asymmetric self-converse, a cross-predicate, an object
|
||||||
|
# mismatch, or an ungrounded guess). All must refuse.
|
||||||
|
vocab, persona = vocab_persona
|
||||||
|
asserted = []
|
||||||
|
for case in _load_cases(_V1 / "refusals.jsonl"):
|
||||||
|
ctx = SessionContext(
|
||||||
|
vocab=vocab, persona=persona, vault_reproject_interval=_HIGH
|
||||||
|
)
|
||||||
|
for fact in case["facts"]:
|
||||||
|
realize_comprehension(comprehend_relational(fact, pack), ctx)
|
||||||
|
res = determine(comprehend_relational(case["query"], pack), ctx)
|
||||||
|
if isinstance(res, Determined):
|
||||||
|
asserted.append(
|
||||||
|
{
|
||||||
|
"id": case["id"],
|
||||||
|
"why": case["why"],
|
||||||
|
"got": [res.answer, res.predicate, res.subject, res.object],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert not asserted, f"confuser(s) asserted (rule over-fired): {asserted}"
|
||||||
|
|
@ -225,7 +225,7 @@ def test_distinct_relations_about_one_subject_stay_distinct(vocab_persona) -> No
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# wrong=0 BITE — direct entailment only; no fabricated inference
|
# wrong=0 BITE — direct + DECLARED one-hop relational rules only; no fabrication
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -237,19 +237,28 @@ def test_present_but_non_entailing_refuses(vocab_persona) -> None:
|
||||||
assert isinstance(res, Undetermined) and res.reason == "not_entailed"
|
assert isinstance(res, Undetermined) and res.reason == "not_entailed"
|
||||||
|
|
||||||
|
|
||||||
def test_symmetric_converse_is_not_faked(vocab_persona) -> None:
|
def test_symmetric_converse_now_soundly_determines(vocab_persona) -> None:
|
||||||
# sibling_of is symmetric in the world, but D0 does DIRECT reading only: the stored
|
# sibling_of IS symmetric (pack-declared ``graph.edge.symmetric``): the converse now
|
||||||
# direction determines; the converse is a sound-but-incomplete refusal, NOT a faked
|
# SOUNDLY determines via the one-hop symmetric rule (mastery-v2 Step 3) — a DECLARED
|
||||||
# assertion. (If this flips to Determined, symmetric inference was smuggled in.)
|
# sound rule, not a fake. The stored direction still determines directly. (Was
|
||||||
|
# test_symmetric_converse_is_not_faked, which pinned the pre-rule direct-only floor.)
|
||||||
ctx = _ctx(vocab_persona)
|
ctx = _ctx(vocab_persona)
|
||||||
_tell("Alice is the sibling of Bob.", ctx)
|
_tell("Alice is the sibling of Bob.", ctx)
|
||||||
assert isinstance(_ask("Is Alice the sibling of Bob?", ctx), Determined) # stored dir
|
direct = _ask("Is Alice the sibling of Bob?", ctx)
|
||||||
|
assert isinstance(direct, Determined) and direct.rule == "direct" # stored dir
|
||||||
converse = _ask("Is Bob the sibling of Alice?", ctx)
|
converse = _ask("Is Bob the sibling of Alice?", ctx)
|
||||||
# The bite is that the converse does NOT assert. Its reason is "ungrounded" (no
|
assert isinstance(converse, Determined) and converse.answer is True
|
||||||
# sibling_of fact has "bob" as subject) rather than "not_entailed" — either way it
|
assert converse.rule == "symmetric"
|
||||||
# is a refusal, proving symmetric inference was not smuggled in.
|
|
||||||
|
|
||||||
|
def test_asymmetric_converse_is_not_faked(vocab_persona) -> None:
|
||||||
|
# The wrong=0 bite, preserved: an ASYMMETRIC predicate's same-predicate converse is
|
||||||
|
# NEVER faked. parent_of is not symmetric — parent_of(alice, bob) does NOT entail
|
||||||
|
# parent_of(bob, alice). (Its sound converse is the DIFFERENT predicate child_of.)
|
||||||
|
ctx = _ctx(vocab_persona)
|
||||||
|
_tell("Alice is the parent of Bob.", ctx)
|
||||||
|
converse = _ask("Is Bob the parent of Alice?", ctx)
|
||||||
assert isinstance(converse, Undetermined)
|
assert isinstance(converse, Undetermined)
|
||||||
assert converse.reason in {"ungrounded", "not_entailed"}
|
|
||||||
|
|
||||||
|
|
||||||
def test_ungrounded_relation_refuses(vocab_persona) -> None:
|
def test_ungrounded_relation_refuses(vocab_persona) -> None:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue