feat(realize): R1 — structural identity & recall (relation-space + span-free idempotency)

R0 keyed a realized fact by its subject's field versor, which is NOT injective:
two facts about one subject embed to byte-identical versors and collide at inf on
metric recall (proven). R1 adds the missing structural key.

- RealizedRecord/metadata carry ordered `relation_arguments` (the relation-space
  key R0's sorted `entity_names` discards) and a span-free `structure_key`.
- `recall_realized(ctx, subject=/predicate=/content_hash=/structure_key=/
  structure_kind=/entity=)` retrieves realized facts by EXACT structural metadata
  (no metric / ANN), via a new read-only `VaultStore.iter_metadata()` accessor.
- Idempotency now dedups on the span-free `structure_key`, so the same proposition
  told from a different source/offset collapses (R0's span-inclusive content_hash
  could not). Guarded by an ambiguous-entity-name refusal — a wrong=0 defense,
  since `Entity.name` is non-unique in the model (only `entity_id` is enforced).
- `content_hash` retained for provenance + replay_hash; `vault_index` pinned to the
  live deque position.

Design adversarially verified (docs/analysis/REALIZE-R1-DETERMINE-scope-2026-06-06.md);
the false "established pattern" private-access comment is removed in favor of the
public accessor. wrong=0 + versor_condition<1e-6 + exact CGA recall preserved;
vault/store.py adds only a read-only accessor (no normalization). Green: 23 realize
+ 110 invariant/vault + 90 smoke; ruff check clean.
This commit is contained in:
Shay 2026-06-06 05:52:49 -07:00
parent 16ba2771bb
commit ef3181aa01
6 changed files with 606 additions and 14 deletions

View file

@ -0,0 +1,246 @@
# REALIZE R1 + DETERMINE (Step 4) — scope, verified against the substrate
**Date:** 2026-06-06 · **Predecessor:** `REALIZE-scope-2026-06-06.md` (R0, #589/#590 merged at `16ba2771`).
This scopes the arc the goal names: R1 (relation-space recall, span-free
idempotency, the binding_graph/arithmetic realize path), the OOV-grounding
substrate question, and DETERMINE (Step 4). Every claim below was checked
against the merged `origin/main` source and, where noted, against an empirical
in-tree run (not a sandbox trace).
---
## 0. The unifying finding — the field versor is not an injective key for realized knowledge
R0 stored a realized fact as `(versor, metadata)` where
`versor = probe_ingest([subject_token]).F`. That conflates two distinct keys:
1. **Field placement** (the versor) — *associative*: "what is near this in
concept space." Good for resonance, bad for identity.
2. **Structural identity** (`content_hash` / `structure_canonical` /
`entity_names` / `relation_predicate`) — *exact, injective, reboot-stable by
construction* (sha256 + canonical strings). This is what makes facts
distinct and recallable-as-themselves.
The field versor collides on at least **three** axes (all verified):
- **Same subject, different relation.** "Truth is a concept." and "Truth is
beautiful." both embed `probe_ingest(["truth"])` → the **same** versor. On
recall both score `inf` (exact self-match) → indistinguishable. *(R0's
acknowledged recall-collision.)*
- **Different OOV entities → one versor.** Empirically (in-tree, separate
processes, `PYTHONHASHSEED` ∈ {0,1,random}): `zelophehad`,
`photosynthesis`, `unbridgeable` — three unrelated words — all ground to the
**byte-identical** versor (`ἀποκρίνομαι`'s root-versor). Root cause: the
mounted morphology has **20 entries** (Greek/Hebrew cognition roots), so every
arbitrary English token falls through `_best_decomposition`'s affinity
fallback (`ingest/gate.py`) and collapses onto whichever root wins the tie.
- **(In-vocab is injective per-subject** but still collides per-relation, axis 1.)
**The fix that unifies R1-recall and OOV:** recall realized knowledge by its
**structural key** (exact metadata match), not solely by the field metric. The
field versor stays as deterministic associative placement. Structural recall is
**exact and deterministic** — it adds no cosine / HNSW / ANN, so it honors the
CLAUDE.md exact-recall invariant. `vault/store.py` is untouched (it stays a
forbidden-normalization site); structural recall is a read-only scan over
realized metadata, layered in `generate/realize/`.
## 1. OOV-grounding: the prior claim is refuted — it is deterministic, not non-deterministic
R0's recorded finding ("OOV grounding is NON-deterministic across reboots") is
**wrong** and is corrected here. Evidence (`/tmp/oov_proc.py`, faithful in-tree,
`no_load_state`):
- `probe_ingest([oov]).F` is **byte-identical across separate processes** and
across `PYTHONHASHSEED` ∈ {0, 1, random}. In-vocab (`truth`) and OOV (`rhea`,
`zelophehad`, …) are all deterministic.
- The latent hazard that *could* cause non-determinism — `_DECOMPOSITION_CACHE`
keyed by `id(vocab)` (an `id()`-reuse-under-GC hazard) — is benign for
identical-config runtimes (the cached value is a decomposition *string*; the
versor is recomputed from the current vocab's roots, which are pack-identical).
So the real blocker for "accept arbitrary entities" is **non-injectivity**, not
non-determinism. And non-injectivity is **tolerable once recall is structural**:
distinct OOV facts stay distinct by `content_hash` even when they field-collide.
The honest, documented limit: *field-metric / associative recall* of OOV-named
facts is degenerate until the morphology substrate (or a content-addressed
entity identity) improves — but *structural* recall and reboot-stable storage
are exact today.
**Correction (from design-review): what actually carries reboot-stability.** The
versor placement is `probe_ingest([token])`, which `realize.py` itself documents
as composing with the *current session state* — so it is deterministic *given
the session state*, NOT a pure function of the subject, and NOT guaranteed to
re-derive identically in a fresh session at a different turn. Reboot-stability of
an already-realized fact rests on **Shape B+ snapshot** persisting the exact
versor *bytes* (`vault.to_dict`/`from_dict`, no reprojection on load), restored
bit-exactly — NOT on re-deriving the same versor. The cross-process determinism
finding above is a *bonus* (re-telling the same fact in a fresh, empty session
lands at the same place), not the load-bearing guarantee. Correctness of recall
rests on the **structural key**, not the versor, in every slice below.
## 2. The slices (dependency-ordered, each a small load-bearing PR; wrong=0 + versor_condition<1e-6 + exact recall preserved throughout)
### R1a — Structural recall (the keystone)
`generate/realize/recall.py::recall_realized(ctx, *, subject=None,
predicate=None, content_hash=None, structure_kind=None) -> tuple[RealizedRecord, ...]`.
A read-only, exact scan over `ctx.vault._metadata` entries with
`kind == "realized"`, matching the given structural predicates (subject ∈
`entity_names`, `relation_predicate ==`, `content_hash ==`, `structure_kind ==`),
returned in deterministic (vault-index) order. No metric, no vault mutation.
- **Falsifiable test:** realize two facts about "truth" with different
predicates → `recall_realized(subject="truth")` returns **both** distinctly;
`recall_realized(predicate=P1)` returns only the first. Assert that the raw
`vault.recall(probe_ingest(["truth"]).F)` collides (both `inf`) — proving
structural recall is *strictly necessary*, not decoration.
### R1b — Span-free idempotency
`content_hash = sha256_of(graph.to_canonical_string())` is **span-inclusive**
(`MeaningGraph.to_canonical_string()` emits `span={source_id}[start:end]` per
entity/relation — verified). So the same fact at a different offset does **not**
dedup. Add a span-free `structure_key` = sha256 of an ordered, span-free
projection: `(predicate, negated, tuple(argument-entity-names in arg order),
sorted other-entity-names)`. Dedup on `structure_key`; retain `content_hash`
(span-inclusive) for provenance.
- **Falsifiable test:** "Truth is a concept." at offset 0 and the same relation
inside a longer carrier sentence (different span) → second realize returns
`created=False`; vault length stays 1. A genuinely different fact (different
predicate/args) is **not** collapsed.
### R1c — binding_graph / arithmetic realize
Extend `realize_comprehension` (or a sibling `realize_quantitative`) to accept a
`QuantComprehension` (`generate/quantitative_comprehension.py`):
`structure_kind = "binding_graph"`, `structure_canonical =
binding_graph.to_canonical_string()`, versor = deterministic placement
(`probe_ingest` of the asked/first entity token; OOV-safe per §1), recalled
**structurally**. Eligibility: a fact-bearing `QuantComprehension` (≥1
`BoundFact`); a pure query realizes nothing (it is recall, not intake), a
`Refusal` realizes nothing.
- **Falsifiable test:** `comprehend_quantitative("alice has 3 coins.")`
realize → `recall_realized(structure_kind="binding_graph", subject="alice")`
returns the record; snapshot→reboot→recall is byte-exact; SPECULATIVE.
### OOV — lift the in-vocab gate (enabled by R1a)
Remove R0's `index_of` in-vocab gate; allow OOV subjects. Storage + structural
recall are exact and reboot-stable (§1). Add a cross-process determinism
guarantee test. Document the field-metric-collision limit in the record/docs
(honest, not papered over). Correct the refuted memory claim.
### D0 — DETERMINE (Step 4): reason over realized structure → assert / refuse
`generate/determine/determine.py::determine(question: Comprehension | Refusal,
ctx) -> Determined(answer, basis, predicate, grounds) | Undetermined(reason)`.
Take a *query-bearing* comprehension; structurally recall (R1a) the realized
facts that could ground it; run a **named entailment predicate** over the
realized structure — for R0/R1 relations, **direct structural entailment** (the
asked relation is exactly a realized relation; transitive subsumption is a later
extension). **Assert** the answer iff the asked relation is structurally
entailed by a realized fact; **refuse** (`Undetermined`) otherwise.
**Honesty correction (from design-review): as-told, never "verified".** Every
realizable record is **SPECULATIVE**, and `ADMISSIBLE_AS_EVIDENCE = {COHERENT}`
(teaching/epistemic.py) — so a realized record is admissible only as a
*candidate*, never as *evidence*. D0 therefore carries the grounding's epistemic
**basis** forward: a determination grounded in SPECULATIVE records is
`Determined(answer=…, basis="as_told", …)` — "based on what I was told
(unverified)" — and **never** a "verified" claim. COHERENT promotion is
out-of-scope, so D0 produces only `as_told` or `Undetermined` (honest and
non-vacuous). No estimation. No corpus mutation (teaching stays HITL
proposal-only). wrong=0: never assert an un-entailed claim, never upgrade
SPECULATIVE to verified.
- **Falsifiable tests (must bite):** realize "Truth is a concept."; ask "Is
Truth a concept?" → `Determined(answer=True, basis="as_told", predicate=
"member", grounds=[that record])`. **Present-but-non-entailing:** ask "Is
Truth a number?" (a record about *truth* exists, but `member(truth, number)`
is NOT realized) → `Undetermined("not_entailed")`. Ask before realizing →
`Undetermined("ungrounded")`. A `Refusal` question → `Undetermined`. **Mutation
check:** delete the grounding record → `Determined` flips to `Undetermined`
(proves the verdict is entailment, not "a record about the subject exists").
## 3. Sequencing, hazards, invariants
Order: **R1a → R1b → R1c → OOV → D0** (R1a unblocks recall for R1c/OOV/D0; R1b is
independent/small). Each slice:
- preserves **wrong=0** (every ineligible/ungrounded input realizes/determines
NOTHING — typed `NotRealized` / `Undetermined`, never a coerced write/assert);
- preserves **versor_condition < 1e-6** (no new embedder; `probe_ingest` only,
closure stays `algebra/versor.py`'s);
- preserves **exact CGA recall** (structural recall is exact metadata match, not
a metric approximation; `vault/store.py` untouched);
- keeps **EpistemicStatus honesty** (SPECULATIVE only; COHERENT is never a
default — promotion stays out of scope, a later slice);
- adds no **parallel learning path** (DETERMINE asserts/refuses; it does not
ratify or mutate corpus — teaching stays HITL proposal-only);
- holds **INV-21** (only allowlisted writers call `VaultStore.store`; R1c's
writer is allowlisted) and does not regress the capability-index anchor
(breadth 8, wrong 0, `50e0675b`).
## 3.5 Verified adjustments (adversarial design-review, 2026-06-06)
Six skeptic agents tried to refute each slice against the merged source +
CLAUDE.md. All six returned **adjust** (none blocked, none rubber-stamped).
Resolutions, each re-grounded in source:
- **R1a — public accessor, not `_metadata`.** `realize.py` is the *only*
external reader of the private `vault._metadata` (the R0 "established pattern"
comment is false). → Add a read-only `VaultStore.iter_metadata()` yielding
`(index, metadata)` in deque order (non-mutating, not a normalization site);
`recall_realized` and the R0 dedup loop consume it; delete the false comment.
`vault_index` is the **live** deque position (authoritative in the unbounded
session tier; provenance-only under bounded eviction — pinned, not left
ambiguous). *Invariant risk: none — exact metadata equality is the opposite of
approximate recall.*
- **R1b — entity *identity*, plus a wrong=0 refusal.** `Entity.name` is
non-unique in the model (only `entity_id` is enforced unique; today the reader
sets `entity_id == name`, reader.py:364, so the hazard is latent, not live).
A name-keyed `structure_key` could collapse a converse/homonym fact →
drops a distinct fact (wrong=0). → `structure_key` keys on the ordered
argument **entity-ids** (`== name` today, span-free), AND realize **refuses**
a graph with duplicate entity names: `NotRealized("ambiguous_entity_names")`
— the defense-in-depth guard CLAUDE.md asks for *now*, before a future reader
makes it live.
- **R1c — honest versor, real guard.** The versor rule is restated per §1
(deterministic-given-session-state; the sumquery `"total"` is a synthesized
OOV name with maximal collision — documented like §0, since **structural
recall carries correctness**). The "fact-bearing" gate is vacuous at the
realize layer (every `QuantComprehension` has ≥1 fact + one query by
construction); the real wrong=0 filter is `isinstance(comp, Refusal) →
NotRealized` (and `comprehend_quantitative` already refuses factless input
upstream). *No wrong=0 admission breach: every equation passed
`check_admissibility`; a `Refusal` short-circuits.*
- **OOV — correct the side-effect contract.** `probe_ingest` of an OOV subject
is **not** side-effect-free: it mutates the shared vocab via
`insert_transient` (gate.py:252). → Correct realize.py's "side-effect-free"
comment; document the transient as session-scoped, **excluded from the
snapshot** (snapshot serializes vault+state, *not* vocab — verified
session/context.py:345), and re-derived on reboot. The determinism test bites
on two axes: cross-process byte-identity AND intra-session path-independence
(probe equals cold-reground whether or not grounded earlier). *Invariant risk:
none — a non-versor construction raises and is caught → `NotRealized`.*
- **D0 — as-told, never verified** (see the corrected design above).
- **Cross-cutting — anchor partition proven.** The capability-index eval imports
**no** `vault` / `ChatRuntime` / `realize` / `SessionContext` / `probe_ingest`
(grep of `evals/capability_index/` is empty) — it scores comprehension against
oracles and never realizes or recalls from a session vault. So realize-writes
**cannot** regress the anchor (50e0675b) — partition by construction, not
assertion. Runtime metric-recall pollution (OOV facts clustering at the
root-versor) is a separate, bounded note: realized entries are SPECULATIVE +
`kind`-tagged, so the evidence path (`min_status=COHERENT`) already excludes
them; default (candidate) recall includes them as the session's own memory by
design. Sequencing R1a→R1b→R1c→OOV→D0 is confirmed sound.
## 4. Explicitly out of scope (later)
COHERENT promotion (contradiction-free coherence judgment), teaching-loop
proposal generation from realized facts, trace-folding, content-addressed
entity identity / morphology-substrate expansion (the deeper OOV-injectivity
fix), LEARNED ESTIMATION (Step 6), the always-on process (L10).

View file

@ -6,10 +6,12 @@ from generate.realize.realize import (
RealizedRecord,
realize_comprehension,
)
from generate.realize.recall import recall_realized
__all__ = [
"NotRealized",
"Realized",
"RealizedRecord",
"realize_comprehension",
"recall_realized",
]

View file

@ -9,11 +9,16 @@ A realized record is NOT a new store — it is a structured vault entry
stamping, and bit-exact Shape B+ persistence for free (see
``docs/analysis/REALIZE-scope-2026-06-06.md``).
Slice R0 deliberately boring (everything else grows from this without corrupting the
field): one told fact, SPECULATIVE only, in-vocab subject only, single non-negated
declarative relation, dedup by content hash. Explicitly OUT of R0: COHERENT
promotion, teaching-loop proposals, the quantitative ``binding_graph`` path,
trace-folding, and relation-space recall.
Slice R0 was deliberately boring (everything else grows from this without corrupting
the field): one told fact, SPECULATIVE only, in-vocab subject only, single non-negated
declarative relation. Slice R1 (this module + ``recall.py``) adds the relation-space
KEY R0 lacked: ordered ``relation_arguments`` + a span-free ``structure_key`` so
distinct facts about one subject stay distinct (they collide on the field versor), and
``recall_realized`` retrieves them by exact structural metadata. Dedup is now span-free
(``structure_key``), guarded by an ambiguous-entity-name refusal (wrong=0). Explicitly
still OUT: COHERENT promotion, teaching-loop proposals, the quantitative
``binding_graph`` path (R1c), trace-folding (see
``docs/analysis/REALIZE-R1-DETERMINE-scope-2026-06-06.md``).
wrong=0 at this layer: a ``Refusal`` (or any ineligible / ungrounded input) realizes
NOTHING it is returned as ``NotRealized(reason)``, never coerced into a vault entry.
@ -47,12 +52,23 @@ class RealizedRecord:
structure_kind: str
structure_canonical: str
relation_predicate: str
#: Ordered argument identities (subject first) — the relation-space key R0's
#: sorted ``entity_names`` discards. Distinct facts about one subject differ
#: here even though they collide on the field versor.
relation_arguments: tuple[str, ...]
entity_names: tuple[str, ...]
source_span: str
content_hash: str
#: Span-FREE structural identity (predicate + negated + ordered argument
#: ids). Idempotency dedups on this, so the same proposition told from a
#: different source/offset collapses (which the span-inclusive ``content_hash``
#: could not). ``content_hash`` is retained for provenance + ``replay_hash``.
structure_key: str
replay_hash: str
epistemic_status: str
tier: str
#: LIVE deque position at recall/write time — authoritative in the unbounded
#: session tier; provenance-only under bounded-vault eviction.
vault_index: int
@ -97,6 +113,16 @@ def realize_comprehension(
if not rel.arguments:
return NotRealized("empty_relation") # defensive — arity>=1 by construction
# wrong=0 defense (R1b): the MeaningGraph model permits distinct entities to
# share a name (only ``entity_id`` is enforced unique; today the reader sets
# entity_id == name, so this is latent, not live). A name-keyed structural
# identity would collapse a converse/homonym fact, dropping a genuinely
# distinct proposition — so refuse the ambiguous input now, before a future
# reader makes it load-bearing.
names = [e.name for e in graph.entities]
if len(set(names)) != len(names):
return NotRealized("ambiguous_entity_names")
subject_id = rel.arguments[0]
entity = next((e for e in graph.entities if e.entity_id == subject_id), None)
if entity is None:
@ -121,9 +147,25 @@ def realize_comprehension(
return NotRealized("grounding_failed")
versor = np.asarray(field_state.F, dtype=np.float32)
# Ordered argument identities (subject first). entity_id == name in today's
# reader; the map keeps this correct if a future reader separates them.
name_of = {e.entity_id: e.name for e in graph.entities}
relation_arguments = tuple(name_of.get(a, a) for a in rel.arguments)
structure_canonical = graph.to_canonical_string()
source_span = rel.span.to_canonical_string()
content_hash = sha256_of(structure_canonical)
content_hash = sha256_of(structure_canonical) # span-INCLUSIVE: provenance
# Span-FREE structural identity: the asserted proposition, free of source
# offsets. Idempotency dedups on this, so the same fact told from a different
# source/offset collapses (the span-inclusive content_hash could not).
structure_key = sha256_of(
{
"structure_kind": _STRUCTURE_KIND_MEANING_GRAPH,
"predicate": rel.predicate,
"negated": rel.negated,
"arguments": list(relation_arguments),
}
)
status = EpistemicStatus.SPECULATIVE # COHERENT is never a default (ADR-0021)
replay_hash = sha256_of(
{
@ -134,14 +176,13 @@ def realize_comprehension(
)
entity_names = tuple(sorted(e.name for e in graph.entities))
# Idempotency: a re-told fact (same content_hash) does not grow the vault. Dedup
# is exact-canonical (the content_hash is span-INCLUSIVE), so the SAFE direction
# only — it never drops a genuinely distinct fact; it may fail to collapse the
# same fact re-stated at a different offset (a span-free key is a later R1 option).
# ``vault._metadata`` is read directly here (no public iterator exists yet);
# this is the established project pattern and an intentional boundary for R0.
for idx, meta in enumerate(ctx.vault._metadata):
if meta.get("kind") == "realized" and meta.get("content_hash") == content_hash:
# Idempotency: a re-told fact (same span-free structure_key) does not grow the
# vault. Refusing duplicate entity names above keeps the name-keyed key
# injective, so dedup only ever collapses the genuinely-same proposition —
# never drops a distinct one. ``iter_metadata`` is the public read-only
# accessor; ``idx`` is the LIVE deque position.
for idx, meta in ctx.vault.iter_metadata():
if meta.get("kind") == "realized" and meta.get("structure_key") == structure_key:
return Realized(record=_record_from_metadata(meta, idx), created=False)
metadata = {
@ -149,9 +190,11 @@ def realize_comprehension(
"structure_kind": _STRUCTURE_KIND_MEANING_GRAPH,
"structure_canonical": structure_canonical,
"relation_predicate": rel.predicate,
"relation_arguments": list(relation_arguments),
"entity_names": list(entity_names),
"source_span": source_span,
"content_hash": content_hash,
"structure_key": structure_key,
"replay_hash": replay_hash,
"tier": "session",
}
@ -161,9 +204,11 @@ def realize_comprehension(
structure_kind=_STRUCTURE_KIND_MEANING_GRAPH,
structure_canonical=structure_canonical,
relation_predicate=rel.predicate,
relation_arguments=relation_arguments,
entity_names=entity_names,
source_span=source_span,
content_hash=content_hash,
structure_key=structure_key,
replay_hash=replay_hash,
epistemic_status=status.value,
tier="session",
@ -179,9 +224,11 @@ def _record_from_metadata(meta: dict, idx: int) -> RealizedRecord:
structure_kind=meta.get("structure_kind", _STRUCTURE_KIND_MEANING_GRAPH),
structure_canonical=meta.get("structure_canonical", ""),
relation_predicate=meta.get("relation_predicate", ""),
relation_arguments=tuple(meta.get("relation_arguments", [])),
entity_names=tuple(meta.get("entity_names", [])),
source_span=meta.get("source_span", ""),
content_hash=meta.get("content_hash", ""),
structure_key=meta.get("structure_key", ""),
replay_hash=meta.get("replay_hash", ""),
epistemic_status=meta.get("epistemic_status", "speculative"),
tier=meta.get("tier", "session"),

View file

@ -0,0 +1,62 @@
"""REALIZE R1 — structural recall of realized knowledge.
The field versor is NOT an injective key: two facts about the same subject embed
to the byte-identical versor, so the metric reader (``vault.recall``) returns
both at ``inf`` and cannot tell them apart. ``recall_realized`` retrieves
realized facts by their EXACT structural metadata subject (the first relation
argument), predicate, content_hash, span-free structure_key, or structure_kind
an exact, deterministic equality scan (no cosine / HNSW / ANN). It reads through
the public ``VaultStore.iter_metadata`` accessor and never mutates the vault.
"""
from __future__ import annotations
from session.context import SessionContext
from .realize import RealizedRecord, _record_from_metadata
def recall_realized(
ctx: SessionContext,
*,
subject: str | None = None,
predicate: str | None = None,
content_hash: str | None = None,
structure_key: str | None = None,
structure_kind: str | None = None,
entity: str | None = None,
) -> tuple[RealizedRecord, ...]:
"""Return realized records matching ALL provided structural filters.
Filters are conjunctive; ``None`` filters are ignored, so ``recall_realized(ctx)``
returns every realized record (in live deque order).
- ``subject`` the relation's first argument (the subject) equals this.
- ``predicate`` the relation predicate equals this.
- ``content_hash`` exact span-inclusive identity (disambiguates a single fact).
- ``structure_key`` exact span-FREE identity (same proposition, any source).
- ``structure_kind`` the substrate (``meaning_graph`` / ``binding_graph``).
- ``entity`` this name appears among the fact's entities (any role).
The scan is exact and order-preserving; it makes no metric call and does not
mutate the vault.
"""
out: list[RealizedRecord] = []
for idx, meta in ctx.vault.iter_metadata():
if meta.get("kind") != "realized":
continue
args = meta.get("relation_arguments", [])
if subject is not None and (not args or args[0] != subject):
continue
if predicate is not None and meta.get("relation_predicate") != predicate:
continue
if content_hash is not None and meta.get("content_hash") != content_hash:
continue
if structure_key is not None and meta.get("structure_key") != structure_key:
continue
if structure_kind is not None and meta.get("structure_kind") != structure_kind:
continue
if entity is not None and entity not in meta.get("entity_names", []):
continue
out.append(_record_from_metadata(meta, idx))
return tuple(out)

View file

@ -0,0 +1,220 @@
"""REALIZE slice R1 — structural identity & recall.
The R0 store keys a realized fact by its subject's FIELD versor. That versor is
not an injective identity: two facts about the same subject embed to the
*byte-identical* versor, so metric recall returns both at score ``inf`` and
cannot tell them apart (proven by ``test_metric_recall_collides_on_same_subject``).
R1 adds the missing key: a realized fact carries its ordered
``relation_arguments`` and a span-free ``structure_key``, and
``recall_realized`` retrieves facts by their EXACT structural metadata
(subject / predicate / content_hash / structure_key) exact and deterministic,
no metric, no vault mutation. Span-free idempotency dedups the same proposition
told from a different source/offset (which R0's span-inclusive content_hash
missed).
"""
from __future__ import annotations
import pytest
from algebra.versor import versor_condition
from chat.runtime import ChatRuntime
from generate.meaning_graph.reader import comprehend
from generate.realize import (
NotRealized,
Realized,
RealizedRecord,
realize_comprehension,
recall_realized,
)
from session.context import SessionContext
_HIGH_INTERVAL = 10**9 # never auto-reproject in these tests
@pytest.fixture(scope="module")
def vocab_persona():
rt = ChatRuntime(no_load_state=True)
return rt._context.vocab, rt._context.persona
def _ctx(vocab_persona) -> SessionContext:
vocab, persona = vocab_persona
return SessionContext(vocab=vocab, persona=persona, vault_reproject_interval=_HIGH_INTERVAL)
def _realize(text: str, ctx: SessionContext, source_id: str = "input"):
return realize_comprehension(comprehend(text, source_id=source_id), ctx)
# --------------------------------------------------------------------------- #
# The problem: the field versor is NOT an injective key (collision is real)
# --------------------------------------------------------------------------- #
def test_metric_recall_collides_on_same_subject(vocab_persona) -> None:
"""Two distinct facts about the same subject store at the byte-identical
versor, so metric recall returns both at ``inf`` structurally blind.
This is the falsifiable justification for structural recall."""
ctx = _ctx(vocab_persona)
a = _realize("Truth is a concept.", ctx)
b = _realize("Truth is a thought.", ctx)
assert isinstance(a, Realized) and isinstance(b, Realized)
assert a.record.content_hash != b.record.content_hash # genuinely distinct facts
# byte-identical stored versors -> the metric cannot separate them
assert ctx.vault._versors[0].tobytes() == ctx.vault._versors[1].tobytes()
hits = ctx.vault.recall(ctx.probe_ingest(["truth"]).F, top_k=5)
realized = [h for h in hits if h["metadata"].get("kind") == "realized"]
assert len(realized) == 2
assert all(h["score"] == float("inf") for h in realized) # both collide at inf
# --------------------------------------------------------------------------- #
# Structural recall — exact metadata match, deterministic, distinct
# --------------------------------------------------------------------------- #
def test_structural_recall_by_subject_returns_all_same_subject_facts(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
_realize("Truth is a concept.", ctx)
_realize("Truth is a thought.", ctx)
_realize("Knowledge is a concept.", ctx) # different subject
got = recall_realized(ctx, subject="truth")
assert all(isinstance(r, RealizedRecord) for r in got)
assert len(got) == 2 # both truth facts, NOT the knowledge fact
hashes = {r.content_hash for r in got}
assert len(hashes) == 2 # distinctly recalled
assert all(r.relation_arguments[0] == "truth" for r in got)
def test_structural_recall_by_content_hash_disambiguates(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
a = _realize("Truth is a concept.", ctx)
_realize("Truth is a thought.", ctx)
assert isinstance(a, Realized)
got = recall_realized(ctx, content_hash=a.record.content_hash)
assert len(got) == 1
assert got[0].content_hash == a.record.content_hash
assert "concept" in got[0].entity_names
def test_structural_recall_by_predicate(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
_realize("Truth is a concept.", ctx)
_realize("Truth is a thought.", ctx)
assert len(recall_realized(ctx, predicate="member")) == 2
assert recall_realized(ctx, predicate="no_such_predicate") == ()
def test_structural_recall_conjoins_filters(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
_realize("Truth is a concept.", ctx)
_realize("Truth is a thought.", ctx)
_realize("Knowledge is a concept.", ctx)
got = recall_realized(ctx, subject="truth", predicate="member")
assert len(got) == 2
assert recall_realized(ctx, subject="knowledge", predicate="member")[0].relation_arguments == (
"knowledge",
"concept",
)
def test_structural_recall_empty_on_no_match(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
_realize("Truth is a concept.", ctx)
assert recall_realized(ctx, subject="nonexistent") == ()
assert recall_realized(ctx) == recall_realized(ctx, subject=None) # no filter -> all realized
def test_structural_recall_survives_reboot(vocab_persona) -> None:
vocab, persona = vocab_persona
ctx = _ctx(vocab_persona)
a = _realize("Truth is a concept.", ctx)
assert isinstance(a, Realized)
rebooted = SessionContext(vocab=vocab, persona=persona, vault_reproject_interval=_HIGH_INTERVAL)
rebooted.restore(ctx.snapshot())
got = recall_realized(rebooted, subject="truth")
assert len(got) == 1
assert got[0].content_hash == a.record.content_hash
assert got[0].relation_arguments == ("truth", "concept")
# field versor still valid after the round trip (recall is structural, but the
# stored placement must remain a versor)
v = rebooted.vault._versors[got[0].vault_index]
assert versor_condition(v) < 1e-6
def test_record_carries_ordered_relation_arguments(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
a = _realize("Truth is a concept.", ctx)
assert isinstance(a, Realized)
# ordered (subject, object) — NOT the sorted entity_names
assert a.record.relation_arguments == ("truth", "concept")
assert a.record.entity_names == ("concept", "truth") # sorted, unchanged from R0
# --------------------------------------------------------------------------- #
# Span-free idempotency — the same proposition from a different source dedups
# --------------------------------------------------------------------------- #
def test_structure_key_is_span_free_but_content_hash_is_not(vocab_persona) -> None:
a_ctx = _ctx(vocab_persona)
b_ctx = _ctx(vocab_persona)
ra = _realize("Truth is a concept.", a_ctx, source_id="docA")
rb = _realize("Truth is a concept.", b_ctx, source_id="docB")
assert isinstance(ra, Realized) and isinstance(rb, Realized)
# same proposition, different provenance:
assert ra.record.structure_key == rb.record.structure_key # span/source-free
assert ra.record.content_hash != rb.record.content_hash # span-INCLUSIVE differs
def test_span_free_idempotency_dedups_across_source(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
first = _realize("Truth is a concept.", ctx, source_id="docA")
second = _realize("Truth is a concept.", ctx, source_id="docB")
assert isinstance(first, Realized) and first.created is True
# R0's content_hash dedup would MISS this (different span -> different hash);
# the span-free structure_key catches it.
assert isinstance(second, Realized) and second.created is False
assert len(ctx.vault._metadata) == 1
def test_distinct_facts_not_collapsed_by_structure_key(vocab_persona) -> None:
ctx = _ctx(vocab_persona)
_realize("Truth is a concept.", ctx)
_realize("Truth is a thought.", ctx) # same subject+predicate, different object
assert len(ctx.vault._metadata) == 2 # NOT collapsed
keys = {m["structure_key"] for m in ctx.vault._metadata}
assert len(keys) == 2
def test_ineligible_input_still_realizes_nothing(vocab_persona) -> None:
# R1 does not loosen R0's wrong=0 floor.
ctx = _ctx(vocab_persona)
assert isinstance(_realize("Is truth a concept?", ctx), NotRealized)
assert isinstance(_realize("The weather is nice today.", ctx), NotRealized)
assert len(ctx.vault._metadata) == 0
def test_duplicate_entity_names_refused(vocab_persona) -> None:
"""wrong=0 defense: the model permits distinct entity_ids to share a NAME.
A name-keyed structure_key would collapse a converse/homonym fact onto an
existing one (dropping a distinct proposition), so realize refuses the
ambiguous graph rather than risk the silent drop. Latent in today's reader
(entity_id == name), but proven to bite via a hand-built graph."""
from generate.meaning_graph.model import Entity, MeaningGraph, MeaningSpan, Relation
from generate.meaning_graph.reader import Comprehension
span = MeaningSpan(source_id="input", start=0, end=10, text="dup vs dup")
graph = MeaningGraph(
entities=(
Entity(entity_id="dup_1", name="dup", span=span),
Entity(entity_id="dup_2", name="dup", span=span),
),
relations=(Relation(predicate="member", arguments=("dup_1", "dup_2"), span=span),),
)
ctx = _ctx(vocab_persona)
res = realize_comprehension(Comprehension(meaning_graph=graph, queries=()), ctx)
assert isinstance(res, NotRealized) and res.reason == "ambiguous_entity_names"
assert len(ctx.vault._metadata) == 0

View file

@ -405,6 +405,21 @@ class VaultStore:
self._matrix_cache = np.asarray(self._versors, dtype=np.float32)
return self._matrix_cache
def iter_metadata(self):
"""Yield ``(index, metadata)`` for every stored entry, in deque order.
A read-only, non-mutating view over the stored metadata NOT a
normalization site (``vault/store.py`` is forbidden from repairing field
state; this only reads). ``index`` is the LIVE deque position, the same
integer ``recall``/``recall_batch`` return; it is authoritative in the
unbounded session tier and provenance-only under bounded-vault eviction.
The yielded dicts are the live metadata objects (not copies) callers
read them; they must not mutate. Lets structured consumers (e.g. realized-
knowledge recall) scan by metadata without reaching into ``_metadata``.
"""
for i, meta in enumerate(self._metadata):
yield i, meta
@property
def reproject_interval(self) -> int:
return self._reproject_interval