core/docs/analysis/REALIZE-R1-DETERMINE-scope-2026-06-06.md

# REALIZE R1 + DETERMINE (Step 4) — scope, verified against the substrate

**Date:** 2026-06-06 · **Predecessor:** `REALIZE-scope-2026-06-06.md` (R0, #589/#590 merged at `16ba2771`).

This scopes the arc the goal names: R1 (relation-space recall, span-free
idempotency, the binding_graph/arithmetic realize path), the OOV-grounding
substrate question, and DETERMINE (Step 4). Every claim below was checked
against the merged `origin/main` source and, where noted, against an empirical
in-tree run (not a sandbox trace).

---

## 0. The unifying finding — the field versor is not an injective key for realized knowledge

R0 stored a realized fact as `(versor, metadata)` where
`versor = probe_ingest([subject_token]).F`. That conflates two distinct keys:

1. **Field placement** (the versor) — *associative*: "what is near this in
   concept space." Good for resonance, bad for identity.
2. **Structural identity** (`content_hash` / `structure_canonical` /
   `entity_names` / `relation_predicate`) — *exact, injective, reboot-stable by
   construction* (sha256 + canonical strings). This is what makes facts
   distinct and recallable-as-themselves.

The field versor collides on at least **three** axes (all verified):

- **Same subject, different relation.** "Truth is a concept." and "Truth is
  beautiful." both embed `probe_ingest(["truth"])` → the **same** versor. On
  recall both score `inf` (exact self-match) → indistinguishable. *(R0's
  acknowledged recall-collision.)*
- **Different OOV entities → one versor.** Empirically (in-tree, separate
  processes, `PYTHONHASHSEED` ∈ {0,1,random}): `zelophehad`,
  `photosynthesis`, `unbridgeable` — three unrelated words — all ground to the
  **byte-identical** versor (`ἀποκρίνομαι`'s root-versor). Root cause: the
  mounted morphology has **20 entries** (Greek/Hebrew cognition roots), so every
  arbitrary English token falls through `_best_decomposition`'s affinity
  fallback (`ingest/gate.py`) and collapses onto whichever root wins the tie.
- **(In-vocab is injective per-subject** but still collides per-relation, axis 1.)

**The fix that unifies R1-recall and OOV:** recall realized knowledge by its
**structural key** (exact metadata match), not solely by the field metric. The
field versor stays as deterministic associative placement. Structural recall is
**exact and deterministic** — it adds no cosine / HNSW / ANN, so it honors the
CLAUDE.md exact-recall invariant. `vault/store.py` is untouched (it stays a
forbidden-normalization site); structural recall is a read-only scan over
realized metadata, layered in `generate/realize/`.

## 1. OOV-grounding: the prior claim is refuted — it is deterministic, not non-deterministic

> **Update (#591 shipped):** the §0/§1 non-injectivity finding below
> (`zelophehad`/`photosynthesis`/`unbridgeable` → one root-versor; field-metric
> recall "degenerate") describes the **pre-#591** substrate. #591 (`ingest/gate.py`
> content-derived `_token_spin_delta`, merged) makes OOV placement **injective by
> token content** (verified live: those three now produce three distinct versors), so
> the gate-lift no longer rests only on structural recall — though structural recall
> remains what *carries* correctness.

R0's recorded finding ("OOV grounding is NON-deterministic across reboots") is
**wrong** and is corrected here. Evidence (`/tmp/oov_proc.py`, faithful in-tree,
`no_load_state`):

- `probe_ingest([oov]).F` is **byte-identical across separate processes** and
  across `PYTHONHASHSEED` ∈ {0, 1, random}. In-vocab (`truth`) and OOV (`rhea`,
  `zelophehad`, …) are all deterministic.
- The latent hazard that *could* cause non-determinism — `_DECOMPOSITION_CACHE`
  keyed by `id(vocab)` (an `id()`-reuse-under-GC hazard) — is benign for
  identical-config runtimes (the cached value is a decomposition *string*; the
  versor is recomputed from the current vocab's roots, which are pack-identical).

So the real blocker for "accept arbitrary entities" is **non-injectivity**, not
non-determinism. And non-injectivity is **tolerable once recall is structural**:
distinct OOV facts stay distinct by `content_hash` even when they field-collide.
The honest, documented limit: *field-metric / associative recall* of OOV-named
facts is degenerate until the morphology substrate (or a content-addressed
entity identity) improves — but *structural* recall and reboot-stable storage
are exact today.

**Correction (from design-review): what actually carries reboot-stability.** The
versor placement is `probe_ingest([token])`, which `realize.py` itself documents
as composing with the *current session state* — so it is deterministic *given
the session state*, NOT a pure function of the subject, and NOT guaranteed to
re-derive identically in a fresh session at a different turn. Reboot-stability of
an already-realized fact rests on **Shape B+ snapshot** persisting the exact
versor *bytes* (`vault.to_dict`/`from_dict`, no reprojection on load), restored
bit-exactly — NOT on re-deriving the same versor. The cross-process determinism
finding above is a *bonus* (re-telling the same fact in a fresh, empty session
lands at the same place), not the load-bearing guarantee. Correctness of recall
rests on the **structural key**, not the versor, in every slice below.

## 2. The slices (dependency-ordered, each a small load-bearing PR; wrong=0 + versor_condition<1e-6 + exact recall preserved throughout)

### R1a — Structural recall (the keystone)

`generate/realize/recall.py::recall_realized(ctx, *, subject=None,
predicate=None, content_hash=None, structure_kind=None) -> tuple[RealizedRecord, ...]`.
A read-only, exact scan over `ctx.vault._metadata` entries with
`kind == "realized"`, matching the given structural predicates (subject ∈
`entity_names`, `relation_predicate ==`, `content_hash ==`, `structure_kind ==`),
returned in deterministic (vault-index) order. No metric, no vault mutation.

- **Falsifiable test:** realize two facts about "truth" with different
  predicates → `recall_realized(subject="truth")` returns **both** distinctly;
  `recall_realized(predicate=P1)` returns only the first. Assert that the raw
  `vault.recall(probe_ingest(["truth"]).F)` collides (both `inf`) — proving
  structural recall is *strictly necessary*, not decoration.

### R1b — Span-free idempotency

`content_hash = sha256_of(graph.to_canonical_string())` is **span-inclusive**
(`MeaningGraph.to_canonical_string()` emits `span={source_id}[start:end]` per
entity/relation — verified). So the same fact at a different offset does **not**
dedup. Add a span-free `structure_key` = sha256 of an ordered, span-free
projection: `(predicate, negated, tuple(argument-entity-names in arg order),
sorted other-entity-names)`. Dedup on `structure_key`; retain `content_hash`
(span-inclusive) for provenance.

- **Falsifiable test:** "Truth is a concept." at offset 0 and the same relation
  inside a longer carrier sentence (different span) → second realize returns
  `created=False`; vault length stays 1. A genuinely different fact (different
  predicate/args) is **not** collapsed.

### R1c — binding_graph / arithmetic realize

Extend `realize_comprehension` (or a sibling `realize_quantitative`) to accept a
`QuantComprehension` (`generate/quantitative_comprehension.py`):
`structure_kind = "binding_graph"`, `structure_canonical =
binding_graph.to_canonical_string()`, versor = deterministic placement
(`probe_ingest` of the asked/first entity token; OOV-safe per §1), recalled
**structurally**. Eligibility: a fact-bearing `QuantComprehension` (≥1
`BoundFact`); a pure query realizes nothing (it is recall, not intake), a
`Refusal` realizes nothing.

- **Falsifiable test:** `comprehend_quantitative("alice has 3 coins.")` →
  realize → `recall_realized(structure_kind="binding_graph", subject="alice")`
  returns the record; snapshot→reboot→recall is byte-exact; SPECULATIVE.

### OOV — lift the in-vocab gate (enabled by R1a)

Remove R0's `index_of` in-vocab gate; allow OOV subjects. Storage + structural
recall are exact and reboot-stable (§1). Add a cross-process determinism
guarantee test. Document the field-metric-collision limit in the record/docs
(honest, not papered over). Correct the refuted memory claim.

### D0 — DETERMINE (Step 4): reason over realized structure → assert / refuse

`generate/determine/determine.py::determine(question: Comprehension | Refusal,
ctx) -> Determined(answer, basis, predicate, grounds) | Undetermined(reason)`.
Take a *query-bearing* comprehension; structurally recall (R1a) the realized
facts that could ground it; run a **named entailment predicate** over the
realized structure — for R0/R1 relations, **direct structural entailment** (the
asked relation is exactly a realized relation; transitive subsumption is a later
extension). **Assert** the answer iff the asked relation is structurally
entailed by a realized fact; **refuse** (`Undetermined`) otherwise.

**Honesty correction (from design-review): as-told, never "verified".** Every
realizable record is **SPECULATIVE**, and `ADMISSIBLE_AS_EVIDENCE = {COHERENT}`
(teaching/epistemic.py) — so a realized record is admissible only as a
*candidate*, never as *evidence*. D0 therefore carries the grounding's epistemic
**basis** forward: a determination grounded in SPECULATIVE records is
`Determined(answer=…, basis="as_told", …)` — "based on what I was told
(unverified)" — and **never** a "verified" claim. COHERENT promotion is
out-of-scope, so D0 produces only `as_told` or `Undetermined` (honest and
non-vacuous). No estimation. No corpus mutation (teaching stays HITL
proposal-only). wrong=0: never assert an un-entailed claim, never upgrade
SPECULATIVE to verified.

- **Falsifiable tests (must bite):** realize "Truth is a concept."; ask "Is
  Truth a concept?" → `Determined(answer=True, basis="as_told", predicate=
  "member", grounds=[that record])`. **Present-but-non-entailing:** ask "Is
  Truth a number?" (a record about *truth* exists, but `member(truth, number)`
  is NOT realized) → `Undetermined("not_entailed")`. Ask before realizing →
  `Undetermined("ungrounded")`. A `Refusal` question → `Undetermined`. **Mutation
  check:** delete the grounding record → `Determined` flips to `Undetermined`
  (proves the verdict is entailment, not "a record about the subject exists").

## 3. Sequencing, hazards, invariants

Order: **R1a → R1b → R1c → OOV → D0** (R1a unblocks recall for R1c/OOV/D0; R1b is
independent/small). Each slice:

- preserves **wrong=0** (every ineligible/ungrounded input realizes/determines
  NOTHING — typed `NotRealized` / `Undetermined`, never a coerced write/assert);
- preserves **versor_condition < 1e-6** (no new embedder; `probe_ingest` only,
  closure stays `algebra/versor.py`'s);
- preserves **exact CGA recall** (structural recall is exact metadata match, not
  a metric approximation; `vault/store.py` untouched);
- keeps **EpistemicStatus honesty** (SPECULATIVE only; COHERENT is never a
  default — promotion stays out of scope, a later slice);
- adds no **parallel learning path** (DETERMINE asserts/refuses; it does not
  ratify or mutate corpus — teaching stays HITL proposal-only);
- holds **INV-21** (only allowlisted writers call `VaultStore.store`; R1c's
  writer is allowlisted) and does not regress the capability-index anchor
  (breadth 8, wrong 0, `50e0675b`).

## 3.5 Verified adjustments (adversarial design-review, 2026-06-06)

Six skeptic agents tried to refute each slice against the merged source +
CLAUDE.md. All six returned **adjust** (none blocked, none rubber-stamped).
Resolutions, each re-grounded in source:

- **R1a — public accessor, not `_metadata`.** `realize.py` is the *only*
  external reader of the private `vault._metadata` (the R0 "established pattern"
  comment is false). → Add a read-only `VaultStore.iter_metadata()` yielding
  `(index, metadata)` in deque order (non-mutating, not a normalization site);
  `recall_realized` and the R0 dedup loop consume it; delete the false comment.
  `vault_index` is the **live** deque position (authoritative in the unbounded
  session tier; provenance-only under bounded eviction — pinned, not left
  ambiguous). *Invariant risk: none — exact metadata equality is the opposite of
  approximate recall.*
- **R1b — entity *identity*, plus a wrong=0 refusal.** `Entity.name` is
  non-unique in the model (only `entity_id` is enforced unique; today the reader
  sets `entity_id == name`, reader.py:364, so the hazard is latent, not live).
  A name-keyed `structure_key` could collapse a converse/homonym fact →
  drops a distinct fact (wrong=0). → `structure_key` keys on the ordered
  argument **entity-ids** (`== name` today, span-free), AND realize **refuses**
  a graph with duplicate entity names: `NotRealized("ambiguous_entity_names")`
  — the defense-in-depth guard CLAUDE.md asks for *now*, before a future reader
  makes it live.
- **R1c — honest versor, real guard.** The versor rule is restated per §1
  (deterministic-given-session-state; the sumquery `"total"` is a synthesized
  OOV name with maximal collision — documented like §0, since **structural
  recall carries correctness**). The "fact-bearing" gate is vacuous at the
  realize layer (every `QuantComprehension` has ≥1 fact + one query by
  construction); the real wrong=0 filter is `isinstance(comp, Refusal) →
  NotRealized` (and `comprehend_quantitative` already refuses factless input
  upstream). *No wrong=0 admission breach: every equation passed
  `check_admissibility`; a `Refusal` short-circuits.*
- **OOV — correct the side-effect contract.** `probe_ingest` of an OOV subject
  is **not** side-effect-free: it mutates the shared vocab via
  `insert_transient` (gate.py:252). → Correct realize.py's "side-effect-free"
  comment; document the transient as session-scoped, **excluded from the
  snapshot** (snapshot serializes vault+state, *not* vocab — verified
  session/context.py:345), and re-derived on reboot. The determinism test bites
  on two axes: cross-process byte-identity AND intra-session path-independence
  (probe equals cold-reground whether or not grounded earlier). *Invariant risk:
  none — a non-versor construction raises and is caught → `NotRealized`.*
- **D0 — as-told, never verified** (see the corrected design above).
- **Cross-cutting — anchor partition proven.** The capability-index eval imports
  **no** `vault` / `ChatRuntime` / `realize` / `SessionContext` / `probe_ingest`
  (grep of `evals/capability_index/` is empty) — it scores comprehension against
  oracles and never realizes or recalls from a session vault. So realize-writes
  **cannot** regress the anchor (50e0675b) — partition by construction, not
  assertion. Runtime metric-recall pollution (OOV facts clustering at the
  root-versor) is a separate, bounded note: realized entries are SPECULATIVE +
  `kind`-tagged, so the evidence path (`min_status=COHERENT`) already excludes
  them; default (candidate) recall includes them as the session's own memory by
  design. Sequencing R1a→R1b→R1c→OOV→D0 is confirmed sound.

## 4. Explicitly out of scope (later)

COHERENT promotion (contradiction-free coherence judgment), teaching-loop
proposal generation from realized facts, trace-folding, content-addressed
entity identity / morphology-substrate expansion (the deeper OOV-injectivity
fix), LEARNED ESTIMATION (Step 6), the always-on process (L10).