# ADR-0180: Delta-CRDT Sharded Substrate for Multimodal Concurrency

**Status:** Proposed
**Date:** 2026-05-29
**Authors:** Joshua M. Shay, Core R&D Engine
**Domains:** `core-rs/src/vault.rs`, `sensorium/`, `field/`

## 1. Context & Problem Statement

With the introduction of continuous, high-density sensory modalities
(specifically Native Geometric Vision and Kinematics, forthcoming ADRs), the
ingestion rate of the system shifts from discrete textual tokens to sustained
60+ FPS high-dimensional streams.

The supreme architectural invariant of `core` is **Modality Blindness**: all
senses must project into a singular, unified Conformal Geometric Algebra
Cl(4,1) manifold to achieve Holonomy Resonance (cross-modal unification
without late-fusion neural networks).

However, enforcing this single geometric truth creates a brutal mechanical
bottleneck: **Global Lock Contention**. If the Vision, Audio, and Text adapters
attempt to concurrently mutate a globally shared `Vault` (the epistemic state)
guarded by a standard Mutex or RwLock, the resulting thread contention will
completely choke the M-Series Unified Memory Architecture (UMA) and Ryzen
threading topologies. We risk sacrificing mechanical sympathy for mathematical
elegance.

## 1.5 What's Being Sharded: The Existing Python Ingest Path

Before specifying the CRDT substrate, this section names the single-threaded
Python pipeline that §2 makes concurrent, so the proof obligation in §4.3 can
be grounded against actual code rather than an abstract baseline. Per
CLAUDE.md work-sequencing item 5 ("Rust backend parity only after Python
semantics are locked by tests"), the contracts in §1.5.2 must be covered by
Python tests on `main` before any change to `core-rs/src/vault.rs` lands.

### 1.5.1 Current Ingest Pipeline (single modality, single thread)

```text
surface signal S
  → sensorium/protocol.py :: ProjectionHead.project(S)       # Logos recovery boundary
  → ingest/gate.py                                            # raw-input normalization (allowed)
  → field/state.py :: F                                       # current field state
  → field/operators.py :: versor_apply(V, F) = V * F * rev(V) # algebra-owned transition
  → field/propagate.py                                        # diffusion step
  → vault/decompose.py + vault/store.py                       # exact CGA recall write
  → core/cognition/trace.py :: compute_trace_hash(...)        # deterministic replay anchor
```

The `ProjectionHead` protocol (`sensorium/protocol.py:CL41_DIM = 32`) is
already the modality boundary this ADR claims to defend. The Modality enum
already covers `TEXT | VISION | AUDIO | MOTOR`. §2's "Modality Blindness"
should be re-stated in terms of the existing **Logos-recovery boundary** —
that is the load-bearing CORE concept; "Modality Blindness" is a synonym
worth either dropping or anchoring to `ProjectionHead` in §1.

### 1.5.2 Ordering Properties Under the Current Path

| Step | Operation | Commutative | Associative | Idempotent | Notes |
|---|---|---|---|---|---|
| `ProjectionHead.project` | S → (32,) | n/a | n/a | yes | pure function on S |
| `ingest/gate.py` | normalize → F₀ | no | no | no | site of CGA construction; ordering-dependent |
| `versor_apply` | V · F · rev(V) | **no** | yes | no | non-commutative sandwich |
| `field/propagate` | F → F′ | depends on operator | depends | no | linear-blend diffusion (Threshold 1) |
| `vault/store.write` | append (F, provenance) | **yes** | yes | yes | exact CGA recall; semilattice-eligible |
| `compute_trace_hash` | reduce → bytes | no | no | yes | order-sensitive by construction |

The semilattice claim in §2.2 holds **only** at the `vault/store` layer —
not at `versor_apply` and not at `compute_trace_hash`. The CRDT substrate
must therefore shard *write-accumulation*, not the full ingest path. Any
operation upstream of `vault/store` that the substrate parallelizes must
either (a) be proven order-invariant on its inputs, or (b) carry an explicit
serialization barrier.

### 1.5.3 Trace-Hash Inputs That Must Survive Sharding

`compute_trace_hash` (per `core/cognition/trace.py:27`) currently hashes a
payload that includes `admissibility_trace_hash` among other fields. For the
proof obligation `hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)` to
be checkable:

1. The set of `(F, provenance)` tuples written to the Vault must be identical
   between the sequential and concurrent runs — *as a set*, not as a sequence.
2. The trace-hash reduction must consume vault state in a content-addressed
   order (e.g. sorted by a deterministic key on the multivector + provenance),
   not in wall-clock arrival order. The merge kernel in §2.2 currently
   describes time-driven flushes ("every 16ms"); §4.3 cannot hold under that
   policy unless the *hashing* step re-sorts.
3. `admissibility_trace_hash` and any other upstream-of-Vault hash inputs
   must be computed on the serialized portion of the path (§1.5.2 row 2-4),
   not on the sharded portion.

### 1.5.4 Pre-Refactor Test Obligations (Python-side, on `main`)

Before any code in `core-rs/src/vault.rs` changes, the following must exist
as Python tests and be green on `main`:

- **T-1**  Set-equality of vault writes under shuffled single-thread ingest:
  for any ingest sequence `[s₁, …, sₙ]` and any permutation `π`, the
  resulting `vault.store` contents are equal as sets.
- **T-2**  `compute_trace_hash` invariance under set-equal vault states with
  identical upstream serialized prefixes. If this fails today, §4.3 cannot
  hold and the reduction step needs a content-addressed sort first.
- **T-3**  `versor_apply` non-commutativity is asserted (negative test): if a
  future refactor accidentally makes it commutative, it will be caught here
  rather than masked by the CRDT substrate.
- **T-4**  `ProjectionHead.project` purity: same `S` → byte-identical `(32,)`
  output across repeated calls, across threads, across processes.

T-1 and T-2 are the load-bearing ones. T-3 and T-4 are guards against silent
drift.

### 1.5.5 What Stays Out of Scope of This ADR

- **Approximate recall.** CLAUDE.md §Core Primitives is non-negotiable: exact
  CGA recall, no HNSW/ANN/cosine. The CRDT merge produces *eventually-exact*
  recall, never approximate. The sub-50ms window in §3.2 is a **latency**
  window for write-visibility, not a **fidelity** window — once merged, recall
  is exact byte-for-byte.
- **Hidden background execution.** The "Merge Kernel" in §2.2 must be an
  explicitly-mounted runtime component with a named owner and observable
  state, not a daemon thread. CLAUDE.md §Security forbids hidden background
  execution; the kernel must surface its pending-delta count in
  telemetry/`TurnEvent` for replay evidence.
- **MLX/UMA hardware optimization.** §2.3's zero-copy MLX handshake is a
  follow-up ADR; it is mentioned here as horizon-setting only. The CRDT
  substrate itself must work on a pure-CPU Rust path first.

### 1.5.6 Cross-References

- ADR-0054 (Vault Recall Indexing + Batching) — the matrix-cache contract the
  sharded path must preserve at the read side.
- `docs/runtime_contracts.md` — the response/telemetry/memory/identity
  contracts that §3.1's "zero modification of `anti_unifier` and `carrier`"
  claim is being measured against.
- CLAUDE.md §Normalization Rules — `ingest/gate.py` remains the **only**
  allowed pre-Vault normalization site; the CRDT substrate must not introduce
  per-shard normalizers.

## 2. Decision: Logical Unity, Physical Sharding

We will resolve this tension by decoupling the logical manifold from the
physical memory layout. The manifold remains singular and mathematically
continuous, but the underlying Rust substrate will heavily shard the ingestion
pathways.

We adopt an architecture based on **Delta-State CRDTs (Conflict-Free Replicated
Data Types)** acting over lock-free, thread-local arenas, resolved via
asynchronous Semilattice Joins.

### 2.1 Thread-Local Sensory Arenas

* **Deprecation of Direct Global Writes:** Adapters (`sensorium/adapters/*`)
  are strictly forbidden from writing directly to the global `epistemic_state`.
* **Local Delta Caches:** Each active modality adapter is assigned a
  thread-local memory arena in `core-rs`. As dense geometric primitives
  (spheres, lines, motors) are generated by the `ProjectionHead`, they are
  written lock-free into this local cache.

### 2.2 The Semilattice Join (CRDT Merge)

* The geometric `Field` operates as an additive accumulation of knowledge. It
  mathematically satisfies the properties of a **Join Semilattice**
  (commutativity, associativity, and idempotence of state integration).
* At predefined intervals (e.g., every 16ms to match 60fps, or at semantic
  chunk boundaries), the local thread generates a `Delta` — a snapshot of the
  newly ingested multivectors.
* A background, lock-free **Merge Kernel** sweeps these Deltas and folds them
  into the global `Vault` using atomic compare-and-swap (CAS) operations or
  unified memory tensor reductions via MLX.

### 2.3 Nanospin Orchestration & Zero-Copy Symbiosis

* **Python/Rust Boundary:** Python will write raw sensory arrays into
  lock-free Ring Buffers (e.g., `crossbeam` channels).
* **Rust Worker Pool:** A pool of pinned Rust worker threads will continuously
  poll these buffers using **nanospin** loops to avoid OS context-switching
  latency.
* **MLX UMA Handshake:** For cross-modal resonance (calculating
  `cga_inner(text_F, vision_F)` across millions of points), Rust will pass raw
  memory pointers (`&[f32]`) directly to the MLX Neural Engine. MLX will
  perform the massively parallel tensor reduction without ever copying the data
  across a PCIe bus.

## 3. Consequences

### 3.1 Positive Impacts (The Exploits)

* **Zero Ingestion Contention:** The vision pipeline can run at the maximum
  framerate allowable by the neural backbone without ever being blocked by
  textual or auditory processing.
* **Hardware Saturation:** Safely maximizes utilization of Apple Silicon
  unified memory and multi-core Ryzen architectures.
* **Mathematical Purity Maintained:** The `anti_unifier` and `carrier` logic
  requires zero modification. They simply operate on a slightly delayed,
  eventually-consistent global state.

### 3.2 Negative Impacts (The Risks)

* **Eventual Consistency Latency:** There will be a sub-50ms window where a
  visual primitive exists in the local Delta Cache but has not yet merged into
  the global Vault. During this micro-window, cross-modal resonance with a
  simultaneous text token cannot occur.
* **Memory Overhead:** Maintaining the local arenas and managing the Delta
  garbage collection increases baseline memory footprint.

## 4. Execution Plan & Proof Obligations

1. **`core-rs` Mutation:** Refactor `core-rs/src/vault.rs` to implement the
   `LocalArena` struct and the `SemilatticeDelta` trait.
2. **MLX Integration:** Define the zero-copy C-FFI boundary between the Rust
   arenas and the MLX distance-calculation tensors.
3. **Trace Invariance Proof:** Extend `evals/` to prove that
   `hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)`. The order of
   asynchronous merging must not alter the final unified geometric topography.
