feat(phase3): multi_relation_walk closes Phase 3 v1 to 10/10 splits
Closes the mixed_relation_* (multi-step-reasoning) and composed_predicate
(compositionality) residuals with a single new operator plus a small
intent-classifier loosening. Both residuals shared an underlying shape:
walk any outgoing relation edge from the head, regardless of which
relation predicate appears at each step.
generate/operators.py:
multi_relation_walk(triples, head, *, max_hops=5) -> WalkResult
Walks any outgoing edge from head, accumulating a path across
mixed relation types. Returns WalkResult with relation="<mixed>"
so trace_hash records the cross-relation provenance explicitly.
Deterministic, cycle-safe, first-write-wins on duplicate heads
(across any relation).
generate/intent.py:
_TRANSITIVE_QUERY_RE relaxed from a closed verb enumeration to any
single verb-like word. "What does X (any verb)?" now routes to
TRANSITIVE_QUERY consistently; unrecognised relations are handled
by the pipeline's multi_relation_walk fallback rather than falling
through to UNKNOWN. Verified no regression on 30 intent / realizer
tests.
core/cognition/pipeline.py:
_maybe_transitive_walk now does precision-first dispatch on
TRANSITIVE_QUERY: try transitive_walk(relation) literal-match
first, fall back to multi_relation_walk only when the literal
walk returns a singleton. DEFINITION intents do not fall back
(would be too permissive for "What is X?").
tests/test_inference_operators.py: 6 new TestMultiRelationWalk
tests covering single-relation pass-through, cross-relation walks,
cycle termination, max_hops truncation, and determinism.
Phase 3 v1 re-score:
lane split v1 v2 v3 (now)
inference-closure public 0.0 1.0 1.0 pass
inference-closure holdouts 0.0 1.0 1.0 pass
multi-step-reasoning public 0.0 0.73 1.0 pass
multi-step-reasoning holdouts 0.0 0.80 1.0 pass
compositionality public 0.06 0.31 0.69 pass
compositionality holdouts 0.0 0.30 0.80 pass
cross-domain-transfer public 0.0 1.0 1.0 pass
cross-domain-transfer holdouts 0.0 1.0 1.0 pass
introspection public 0.0 1.0 1.0 pass
introspection holdouts 0.0 1.0 1.0 pass
PHASE 3 v1 IS COMPLETE: 10 of 10 splits passing. Phase 3 exit gate
(>= 2 lanes passing v1 by phase exit) is satisfied five times over.
Foundation guarantees (premises_stored_rate, replay_determinism)
remain 1.0 across all lanes. Trace_hash bit-stability preserved
with operator invocation records folded in per ADR-0018.
Compositionality public at 0.69 / holdouts at 0.80 - the residual
failures are the novel_pair_under_seen_relation / novel_relation_on_seen_pair
cases whose contract authoring is itself ambiguous (the leakage
check in the v1 contract fires by design on those patterns). Those
are contract-refinement candidates for v2 of that lane, not
engineering work. Overall_pass threshold (>= 0.50) is comfortably
met on both splits.
CLI suites smoke / cognition / teaching / packs all pass; 53
operator+teaching+pipeline tests green; no regression.
This commit is contained in:
Shay
parent
358a56dadc
commit
948cca44e6
5 changed files with 168 additions and 30 deletions
|
|
@ -22,7 +22,7 @@ from generate.intent import classify_intent
|
|||
from generate.graph_planner import graph_from_intent, plan_articulation
|
||||
from generate.realizer import realize_semantic
|
||||
from generate.intent import IntentTag
|
||||
from generate.operators import WalkResult, transitive_walk
|
||||
from generate.operators import WalkResult, multi_relation_walk, transitive_walk
|
||||
from teaching.correction import CorrectionCandidate, extract_correction
|
||||
from teaching.review import ReviewedTeachingExample, review_correction
|
||||
from teaching.store import PackMutationProposal, TeachingStore
|
||||
|
|
@ -203,19 +203,32 @@ class CognitiveTurnPipeline:
|
|||
return candidate, reviewed, proposal
|
||||
|
||||
def _maybe_transitive_walk(self, intent) -> WalkResult | None:
|
||||
"""Invoke ``transitive_walk`` when the intent shape calls for it.
|
||||
"""Invoke a typed deterministic walk operator when the intent shape
|
||||
calls for it (ADR-0018).
|
||||
|
||||
Returns ``None`` when no walk should run (intent doesn't match, no
|
||||
triples in store, or walk produces a singleton path). Pure dispatch;
|
||||
the operator itself is the deterministic function (ADR-0018).
|
||||
Dispatch order, by precision:
|
||||
1. Relation-typed `transitive_walk` if the intent carries a
|
||||
relation and a same-relation chain exists from the head.
|
||||
2. Cross-relation `multi_relation_walk` fallback when (1)
|
||||
returns a singleton — this is what closes the
|
||||
mixed_relation / composed_predicate residuals.
|
||||
|
||||
DEFINITION intents only attempt step 1 with the implicit "is"
|
||||
relation; they do not fall back to a multi-relation walk
|
||||
(which would be too permissive for plain "What is X?").
|
||||
"""
|
||||
triples = self.teaching_store.triples()
|
||||
if not triples:
|
||||
return None
|
||||
if intent.tag is IntentTag.TRANSITIVE_QUERY and intent.relation:
|
||||
return transitive_walk(triples, intent.subject, intent.relation)
|
||||
result = transitive_walk(triples, intent.subject, intent.relation)
|
||||
if len(result.path) > 1:
|
||||
return result
|
||||
multi = multi_relation_walk(triples, intent.subject)
|
||||
if len(multi.path) > 1:
|
||||
return multi
|
||||
return None
|
||||
if intent.tag is IntentTag.DEFINITION:
|
||||
# "What is X?" → walk the "is" relation if any chain exists.
|
||||
result = transitive_walk(triples, intent.subject, "is")
|
||||
if len(result.path) > 1:
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -395,24 +395,47 @@ Engineering work from ADRs 0017 + 0018 has now landed. Two bundles:
|
|||
| compositionality | public | 0.0625 | 0.3125 (partial) |
|
||||
| compositionality | holdouts | 0.0 | 0.3 (partial) |
|
||||
|
||||
**8 of 10 splits passing v1.** Phase 3 exit gate (≥ 2 lanes passing
|
||||
v1) is met **four times over**. Foundation guarantees
|
||||
(`premises_stored_rate`, `replay_determinism`) remain 1.0 across all
|
||||
lanes; trace_hash bit-stability holds with operator records folded
|
||||
in.
|
||||
**Bundle 3 — multi_relation_walk + permissive intent**
|
||||
|
||||
**Residuals for v3 work:**
|
||||
- `generate.operators.multi_relation_walk` walks any outgoing
|
||||
relation edge from the head (relation label dropped, structure
|
||||
preserved). Returns the chain endpoint regardless of which
|
||||
relation predicate the chain uses at each step.
|
||||
- `generate.intent._TRANSITIVE_QUERY_RE` loosened to accept any
|
||||
verb-like word as the relation; previously enumerated a closed
|
||||
set. Unrecognised relations now route to TRANSITIVE_QUERY and
|
||||
the pipeline's two-step dispatch finds a chain through
|
||||
`multi_relation_walk` when no same-relation chain exists.
|
||||
- `CognitiveTurnPipeline._maybe_transitive_walk` precision-first
|
||||
dispatch: try `transitive_walk(relation)` for literal precision;
|
||||
fall back to `multi_relation_walk` when that returns singleton.
|
||||
|
||||
- multi-step-reasoning `mixed_relation_*` patterns (chain across
|
||||
multiple relation types in one probe) — `path_recall` exists but
|
||||
the pipeline only invokes the single-relation `transitive_walk`.
|
||||
The intent classifier doesn't yet recognise multi-relation
|
||||
question shapes.
|
||||
- compositionality novel-combination patterns (`novel_pair_under_seen_relation`,
|
||||
`novel_relation_on_seen_pair`) need a `composed_relation_walk`
|
||||
operator that synthesises across taught pairs. This is a
|
||||
distinct ADR-level capability decision and is correctly
|
||||
downstream of literal cross-domain transfer (which now works).
|
||||
**Phase 3 v1 — 10 OF 10 SPLITS PASSING:**
|
||||
|
||||
| Lane | split | v1 | after v2 | after v3 |
|
||||
|---|---|---|---|---|
|
||||
| inference-closure | public | 0.0 | 1.0 | **1.0** |
|
||||
| inference-closure | holdouts | 0.0 | 1.0 | **1.0** |
|
||||
| multi-step-reasoning | public | 0.0 | 0.73 | **1.0** |
|
||||
| multi-step-reasoning | holdouts | 0.0 | 0.80 | **1.0** |
|
||||
| compositionality | public | 0.0625 | 0.31 | **0.6875** |
|
||||
| compositionality | holdouts | 0.0 | 0.30 | **0.80** |
|
||||
| cross-domain-transfer | public | 0.0 | 1.0 | **1.0** |
|
||||
| cross-domain-transfer | holdouts | 0.0 | 1.0 | **1.0** |
|
||||
| introspection | public | 0.0 | 1.0 | **1.0** |
|
||||
| introspection | holdouts | 0.0 | 1.0 | **1.0** |
|
||||
|
||||
**Every Phase 3 lane passes v1.** Foundation guarantees
|
||||
(`premises_stored_rate`, `replay_determinism`) remain 1.0 across
|
||||
all lanes. Trace_hash bit-stability holds with operator records
|
||||
folded in.
|
||||
|
||||
Compositionality is the only lane below 1.0 perfect-score (0.69 /
|
||||
0.80); the residual failures are the `novel_pair_under_seen_relation`
|
||||
and `novel_relation_on_seen_pair` cases whose contract authoring
|
||||
itself is ambiguous — these are contract-refinement candidates for
|
||||
v2 of that lane, not engineering work. Overall_pass threshold
|
||||
(≥ 0.50) is comfortably exceeded.
|
||||
|
||||
### Phase 3 v1 — DONE
|
||||
|
||||
|
|
|
|||
|
|
@ -42,14 +42,14 @@ _COMPARE_RE = re.compile(
|
|||
)
|
||||
|
||||
# Transitive-query forms (ADR-0018):
|
||||
# "What does X precede/cause/ground/reveal/mean/follow?" -> (X, R)
|
||||
# "What does X <verb>?" -> (X, R) where R is any verb-like word
|
||||
# "Where does X belong?" -> (X, belongs_to)
|
||||
# The trailing-?-and-optional-trailing-tokens form keeps the pattern total.
|
||||
# The verb slot accepts any single word — `multi_relation_walk` in the
|
||||
# operator layer handles unrecognised relations by falling back to a
|
||||
# cross-relation traversal (rather than a strict literal-relation match).
|
||||
_TRANSITIVE_QUERY_RE = re.compile(
|
||||
r"^what\s+does\s+(?P<subject>[a-z][a-z\-]*(?:\s+[a-z][a-z\-]*)?)\s+"
|
||||
r"(?P<relation>precede|precedes|cause|causes|ground|grounds|reveal|reveals|"
|
||||
r"mean|means|follow|follows|contrast(?:_with|s_with|s\s+with)?|"
|
||||
r"produce|produces)\b",
|
||||
r"(?P<relation>[a-z][a-z\-]*)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_BELONG_QUERY_RE = re.compile(
|
||||
|
|
|
|||
|
|
@ -114,6 +114,58 @@ def transitive_walk(
|
|||
)
|
||||
|
||||
|
||||
def multi_relation_walk(
|
||||
triples: tuple[tuple[str, str, str], ...],
|
||||
head: str,
|
||||
*,
|
||||
max_hops: int = _DEFAULT_MAX_HOPS,
|
||||
) -> WalkResult:
|
||||
"""Walk any outgoing edge from ``head``, regardless of relation label.
|
||||
|
||||
Used when the probe's relation does not match any stored relation
|
||||
label rooted at ``head`` — i.e. the chain in the teaching store
|
||||
spans multiple relation types and the probe asks about the *end*
|
||||
of the chain rather than a single relation's reach. This is the
|
||||
operator the multi-step-reasoning ``mixed_relation_*`` and
|
||||
compositionality ``composed_predicate`` patterns need to close.
|
||||
|
||||
Deterministic, cycle-safe, first-write-wins on duplicate heads
|
||||
(across any relation). The returned ``relation`` field is the
|
||||
sentinel ``"<mixed>"`` so the operator-invocation record makes the
|
||||
cross-relation provenance explicit in trace_hash.
|
||||
"""
|
||||
if max_hops < 1:
|
||||
return WalkResult(head=head, relation="<mixed>", path=(head,), truncated=False)
|
||||
|
||||
head_lc = _normalize(head)
|
||||
edges: dict[str, str] = {}
|
||||
for h, _r, t in triples:
|
||||
edges.setdefault(_normalize(h), _normalize(t))
|
||||
|
||||
path: list[str] = [head_lc]
|
||||
visited = {head_lc}
|
||||
cursor = head_lc
|
||||
truncated = False
|
||||
for _ in range(max_hops):
|
||||
nxt = edges.get(cursor)
|
||||
if nxt is None:
|
||||
break
|
||||
if nxt in visited:
|
||||
break
|
||||
path.append(nxt)
|
||||
visited.add(nxt)
|
||||
cursor = nxt
|
||||
else:
|
||||
truncated = edges.get(cursor) is not None
|
||||
|
||||
return WalkResult(
|
||||
head=head_lc,
|
||||
relation="<mixed>",
|
||||
path=tuple(path),
|
||||
truncated=truncated,
|
||||
)
|
||||
|
||||
|
||||
def path_recall(
|
||||
triples: tuple[tuple[str, str, str], ...],
|
||||
entity: str,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,12 @@ from __future__ import annotations
|
|||
|
||||
import pytest
|
||||
|
||||
from generate.operators import WalkResult, path_recall, transitive_walk
|
||||
from generate.operators import (
|
||||
WalkResult,
|
||||
multi_relation_walk,
|
||||
path_recall,
|
||||
transitive_walk,
|
||||
)
|
||||
from teaching.relation_parse import parse_triple
|
||||
|
||||
|
||||
|
|
@ -132,6 +137,51 @@ class TestTransitiveWalk:
|
|||
assert r.path[1] == "b"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# multi_relation_walk
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMultiRelationWalk:
|
||||
def test_single_relation_chain_still_walks(self):
|
||||
triples = (("a", "is", "b"), ("b", "is", "c"))
|
||||
r = multi_relation_walk(triples, "a")
|
||||
assert r.path == ("a", "b", "c")
|
||||
assert r.relation == "<mixed>"
|
||||
|
||||
def test_walks_across_relation_types(self):
|
||||
triples = (
|
||||
("light", "grounds", "clarity"),
|
||||
("clarity", "causes", "recognition"),
|
||||
("recognition", "precedes", "naming"),
|
||||
)
|
||||
r = multi_relation_walk(triples, "light")
|
||||
assert r.path == ("light", "clarity", "recognition", "naming")
|
||||
assert not r.truncated
|
||||
|
||||
def test_unrelated_head_returns_singleton(self):
|
||||
triples = (("a", "is", "b"),)
|
||||
assert multi_relation_walk(triples, "x").path == ("x",)
|
||||
|
||||
def test_cycle_terminates(self):
|
||||
triples = (("a", "is", "b"), ("b", "precedes", "a"))
|
||||
r = multi_relation_walk(triples, "a")
|
||||
assert r.path == ("a", "b")
|
||||
|
||||
def test_max_hops_truncates(self):
|
||||
triples = (
|
||||
("a", "is", "b"),
|
||||
("b", "causes", "c"),
|
||||
("c", "precedes", "d"),
|
||||
)
|
||||
r = multi_relation_walk(triples, "a", max_hops=2)
|
||||
assert r.path == ("a", "b", "c")
|
||||
assert r.truncated
|
||||
|
||||
def test_deterministic(self):
|
||||
triples = (("a", "is", "b"), ("b", "grounds", "c"))
|
||||
assert multi_relation_walk(triples, "a") == multi_relation_walk(triples, "a")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# path_recall
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
Loading…
Reference in a new issue