6f70e834ff
* feat(determine): one-hop sound relational entailment — inverse/converse + symmetric Mastery-v2 Step 3 lead capability (core). DETERMINE could perceive relational structure (16 predicates, breadth-complete) but could not derive the simplest entailed relational facts — it read only the stored direction. This adds two SOUND one-hop rules that read a stored edge in its other lawful direction: INVERSE/converse greater_than(a,b) <= told less_than(b,a) SYMMETRIC sibling_of(b,a) <= told sibling_of(a,b) Strictly scoped (per review): OPEN-WORLD (asserts only True, never False — the answer=False path stays unbuilt, INV-30's firewall holds), ONE hop (NO transitive chaining), DECLARED rules only. Ontology/metadata-driven, not prose intuition: - generate/meaning_graph/relational.py: declarative algebra tables — _INVERSE_PAIRS (the converse edges; the pack carries no inverse metadata) + INVERSE_OF (derived, an involution) + SYMMETRIC_PREDICATES (mirrors the pack's graph.edge.symmetric tag). load_relational_pack_symmetric() reads the pack ontology; a test pins the constant equal to it (no silent divergence). - generate/determine/determine.py: _relational_one_hop() between direct entailment and transitive subsumption; new Determined.rule provenance field (direct/inverse/ symmetric/subsumption) — replay-safe (render reads only basis; trace hashes surface). wrong=0 confuser block proves the rule cannot over-fire: less_than is not self-inverse, sibling_of does not imply parent_of, greater_than does not imply equal_to, NO transitive chain (direct or through-inverse), and no answer=False is ever emitted. An obsolete pin (test_symmetric_converse_is_not_faked, which asserted the pre-rule direct-only floor) is updated to the new sound behavior and a preserved asymmetric-converse bite. Tests: test_determine_relational_inference.py (13, the capability contract) + updated test_relational_reader.py. Broad relational/determine/grounding + capability_index baseline sweep: 114 green; baseline digest UNCHANGED (this is a determination capability; the comprehension lanes are untouched). REMAINING for the full lead PR (next): the measurement lane — an independent-oracle evals/relational_inference lane + a capability_index adapter + baseline re-freeze (breadth 9->10, wrong_total still 0), so the capability registers on the yardstick. INTEGRATION NOTE (cross-PR): this adds 2 Determined() construction sites (now 4: direct/inverse/symmetric/subsumption, ALL answer=True). When rebased onto main with INV-30 (PR #770), update INV-30's test_determine_construction_sites_are_visible count 2 -> 4 and confirm all four assert True. Merge order: #770 -> this. * feat(capability-index): relational-inference lane — breadth 9->10, wrong=0 Mastery-v2 Step 3 lead, measurement half. Puts the one-hop relational-inference capability ON the capability index with an independent-oracle lane: - evals/relational_inference/v1/: cases.jsonl (13 positive: 8 inverse + 5 symmetric, authored from the relation algebra INDEPENDENTLY of determine/relational per INV-25/27), refusals.jsonl (8 confusers that MUST refuse), provenance.md (incl. the honest overlaps_event reader-surface coverage gap). - evals/comprehension/relational_inference_runner.py: told fact(s) -> determine(query) vs gold; refusal = coverage miss, disagreement = wrong (structurally 0 on positives). - evals/capability_index/adapters.py: comprehension_relational_inference_result added to ADAPTERS. - baseline.json re-frozen: breadth 9 -> 10, capability_score 0.94403 -> 0.949483, wrong_total still 0, digest deliberately changed (35dea2b2...). - tests/test_relational_inference_lane.py: SHA-pinned gold, positive wrong=0 + coverage>0, and the confuser wrong=0 BITE (every confuser must refuse). Also reconciles INV-30 (now in main via #770): determine.py grew from 2 to 3 Determined() construction sites (direct, the shared relational one-hop constructor, transitive subsumption) — ALL answer=True. test_determine_construction_sites_are_visible updated 2 -> 3 (correcting the prior commit note's overcount of 4; inverse and symmetric share the single _relational_determined constructor, so it is ONE new site). Also fixes a .gitignore gap: ADR-0219 gen-dir checkpoints (engine_state/current, engine_state/gen-*/) were not ignored (only the old flat engine_state/*.json patterns were), so runtime checkpoints could be committed by accident. Verification: new lane 13/0/0 coverage 1.0; capability_baseline digest matches the re-freeze; INV-30 + architectural invariants green; smoke 99/99. * docs+test: reconcile stale relational-inference docstrings; assert rule provenance (review #775) Addresses two review-requested cleanups on #775: 1. Stale docstrings. generate/meaning_graph/relational.py said the symmetric converse is "a sound-but-incomplete refusal at DETERMINE" and there is "NO transitive/ symmetric/rule inference"; generate/determine/determine.py said "symmetric-converse questions are Undetermined", "no transitive/symmetric/rule inference", and "asserts only answer=True on a direct hit". All false after the one-hop relational algebra landed. Updated: the reader does direct-reading only; DETERMINE applies declared one-hop inverse/converse + pack-declared symmetric (plus the existing member/subset subsumption), still open-world / never answer=False; transitive relational closure, negation, and closed-world falsehood remain out of scope. 2. Rule-provenance gold now meaningful. The relational_inference runner compared only (answer, predicate, subject, object), ignoring res.rule though the gold carries "rule": "inverse"/"symmetric". Added res.rule to the got/gold tuple so a "right answer via the wrong rule path" can no longer pass silently. Lane still 13/0/0; baseline counts (and digest) unchanged. Verification: runner 13/0/0; lane + capability_baseline + unit + INV-30 green (22).
79 lines
2.8 KiB
Python
79 lines
2.8 KiB
Python
"""Lane test — one-hop relational inference (capability-index lane + wrong=0 bite).
|
|
|
|
The positive lane (cases.jsonl) is the capability-index coverage; the confuser lane
|
|
(refusals.jsonl) is the wrong=0 bite — every confuser MUST refuse, or the inverse/
|
|
symmetric rule over-fired. Fixture SHAs are pinned so the gold cannot drift silently.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from chat.runtime import ChatRuntime
|
|
from evals.comprehension.relational_inference_runner import _load_cases, run
|
|
from generate.determine import Determined, determine
|
|
from generate.meaning_graph.relational import (
|
|
comprehend_relational,
|
|
load_relational_pack_lemmas,
|
|
)
|
|
from generate.realize import realize_comprehension
|
|
from session.context import SessionContext
|
|
|
|
_V1 = Path(__file__).resolve().parent.parent / "evals" / "relational_inference" / "v1"
|
|
_CASES_SHA = "03310ecc3ab7b7bf26f0a1779709bf468e888bbca1e53a71cba1d269b5c1dd71"
|
|
_REFUSALS_SHA = "f137bbe03b91301d9aa802ab9274c7d43ff9c3966207a8ba5e5f8fc19f0f7243"
|
|
_HIGH = 10**9
|
|
|
|
|
|
def _sha(p: Path) -> str:
|
|
return hashlib.sha256(p.read_bytes()).hexdigest()
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def vocab_persona():
|
|
rt = ChatRuntime(no_load_state=True)
|
|
return rt._context.vocab, rt._context.persona
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def pack():
|
|
return load_relational_pack_lemmas()
|
|
|
|
|
|
def test_fixture_shas_pinned() -> None:
|
|
assert _sha(_V1 / "cases.jsonl") == _CASES_SHA
|
|
assert _sha(_V1 / "refusals.jsonl") == _REFUSALS_SHA
|
|
|
|
|
|
def test_positive_lane_wrong_zero_and_covers() -> None:
|
|
r = run()
|
|
assert r["wrong"] == 0, r["wrongs"]
|
|
assert r["correct"] > 0 # coverage > 0 — else the index geomean zeroes the score
|
|
assert r["correct"] + r["wrong"] + r["refused"] == r["total"]
|
|
|
|
|
|
def test_confusers_all_refuse(vocab_persona, pack) -> None:
|
|
# wrong=0 BITE: a Determined on ANY confuser means the one-hop rule over-fired
|
|
# (a transitive chain, an asymmetric self-converse, a cross-predicate, an object
|
|
# mismatch, or an ungrounded guess). All must refuse.
|
|
vocab, persona = vocab_persona
|
|
asserted = []
|
|
for case in _load_cases(_V1 / "refusals.jsonl"):
|
|
ctx = SessionContext(
|
|
vocab=vocab, persona=persona, vault_reproject_interval=_HIGH
|
|
)
|
|
for fact in case["facts"]:
|
|
realize_comprehension(comprehend_relational(fact, pack), ctx)
|
|
res = determine(comprehend_relational(case["query"], pack), ctx)
|
|
if isinstance(res, Determined):
|
|
asserted.append(
|
|
{
|
|
"id": case["id"],
|
|
"why": case["why"],
|
|
"got": [res.answer, res.predicate, res.subject, res.object],
|
|
}
|
|
)
|
|
assert not asserted, f"confuser(s) asserted (rule over-fired): {asserted}"
|