diff --git a/docs/decisions/ADR-0180-crdt-sharded-vault-concurrency.md b/docs/decisions/ADR-0180-crdt-sharded-vault-concurrency.md new file mode 100644 index 00000000..14a320a7 --- /dev/null +++ b/docs/decisions/ADR-0180-crdt-sharded-vault-concurrency.md @@ -0,0 +1,215 @@ +# ADR-0180: Delta-CRDT Sharded Substrate for Multimodal Concurrency + +**Status:** Proposed +**Date:** 2026-05-29 +**Authors:** Joshua M. Shay, Core R&D Engine +**Domains:** `core-rs/src/vault.rs`, `sensorium/`, `field/` + +## 1. Context & Problem Statement + +With the introduction of continuous, high-density sensory modalities +(specifically Native Geometric Vision and Kinematics, forthcoming ADRs), the +ingestion rate of the system shifts from discrete textual tokens to sustained +60+ FPS high-dimensional streams. + +The supreme architectural invariant of `core` is **Modality Blindness**: all +senses must project into a singular, unified Conformal Geometric Algebra +Cl(4,1) manifold to achieve Holonomy Resonance (cross-modal unification +without late-fusion neural networks). + +However, enforcing this single geometric truth creates a brutal mechanical +bottleneck: **Global Lock Contention**. If the Vision, Audio, and Text adapters +attempt to concurrently mutate a globally shared `Vault` (the epistemic state) +guarded by a standard Mutex or RwLock, the resulting thread contention will +completely choke the M-Series Unified Memory Architecture (UMA) and Ryzen +threading topologies. We risk sacrificing mechanical sympathy for mathematical +elegance. + +## 1.5 What's Being Sharded: The Existing Python Ingest Path + +Before specifying the CRDT substrate, this section names the single-threaded +Python pipeline that §2 makes concurrent, so the proof obligation in §4.3 can +be grounded against actual code rather than an abstract baseline. Per +CLAUDE.md work-sequencing item 5 ("Rust backend parity only after Python +semantics are locked by tests"), the contracts in §1.5.2 must be covered by +Python tests on `main` before any change to `core-rs/src/vault.rs` lands. + +### 1.5.1 Current Ingest Pipeline (single modality, single thread) + +```text +surface signal S + → sensorium/protocol.py :: ProjectionHead.project(S) # Logos recovery boundary + → ingest/gate.py # raw-input normalization (allowed) + → field/state.py :: F # current field state + → field/operators.py :: versor_apply(V, F) = V * F * rev(V) # algebra-owned transition + → field/propagate.py # diffusion step + → vault/decompose.py + vault/store.py # exact CGA recall write + → core/cognition/trace.py :: compute_trace_hash(...) # deterministic replay anchor +``` + +The `ProjectionHead` protocol (`sensorium/protocol.py:CL41_DIM = 32`) is +already the modality boundary this ADR claims to defend. The Modality enum +already covers `TEXT | VISION | AUDIO | MOTOR`. §2's "Modality Blindness" +should be re-stated in terms of the existing **Logos-recovery boundary** — +that is the load-bearing CORE concept; "Modality Blindness" is a synonym +worth either dropping or anchoring to `ProjectionHead` in §1. + +### 1.5.2 Ordering Properties Under the Current Path + +| Step | Operation | Commutative | Associative | Idempotent | Notes | +|---|---|---|---|---|---| +| `ProjectionHead.project` | S → (32,) | n/a | n/a | yes | pure function on S | +| `ingest/gate.py` | normalize → F₀ | no | no | no | site of CGA construction; ordering-dependent | +| `versor_apply` | V · F · rev(V) | **no** | yes | no | non-commutative sandwich | +| `field/propagate` | F → F′ | depends on operator | depends | no | linear-blend diffusion (Threshold 1) | +| `vault/store.write` | append (F, provenance) | **yes** | yes | yes | exact CGA recall; semilattice-eligible | +| `compute_trace_hash` | reduce → bytes | no | no | yes | order-sensitive by construction | + +The semilattice claim in §2.2 holds **only** at the `vault/store` layer — +not at `versor_apply` and not at `compute_trace_hash`. The CRDT substrate +must therefore shard *write-accumulation*, not the full ingest path. Any +operation upstream of `vault/store` that the substrate parallelizes must +either (a) be proven order-invariant on its inputs, or (b) carry an explicit +serialization barrier. + +### 1.5.3 Trace-Hash Inputs That Must Survive Sharding + +`compute_trace_hash` (per `core/cognition/trace.py:27`) currently hashes a +payload that includes `admissibility_trace_hash` among other fields. For the +proof obligation `hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)` to +be checkable: + +1. The set of `(F, provenance)` tuples written to the Vault must be identical + between the sequential and concurrent runs — *as a set*, not as a sequence. +2. The trace-hash reduction must consume vault state in a content-addressed + order (e.g. sorted by a deterministic key on the multivector + provenance), + not in wall-clock arrival order. The merge kernel in §2.2 currently + describes time-driven flushes ("every 16ms"); §4.3 cannot hold under that + policy unless the *hashing* step re-sorts. +3. `admissibility_trace_hash` and any other upstream-of-Vault hash inputs + must be computed on the serialized portion of the path (§1.5.2 row 2-4), + not on the sharded portion. + +### 1.5.4 Pre-Refactor Test Obligations (Python-side, on `main`) + +Before any code in `core-rs/src/vault.rs` changes, the following must exist +as Python tests and be green on `main`: + +- **T-1** Set-equality of vault writes under shuffled single-thread ingest: + for any ingest sequence `[s₁, …, sₙ]` and any permutation `π`, the + resulting `vault.store` contents are equal as sets. +- **T-2** `compute_trace_hash` invariance under set-equal vault states with + identical upstream serialized prefixes. If this fails today, §4.3 cannot + hold and the reduction step needs a content-addressed sort first. +- **T-3** `versor_apply` non-commutativity is asserted (negative test): if a + future refactor accidentally makes it commutative, it will be caught here + rather than masked by the CRDT substrate. +- **T-4** `ProjectionHead.project` purity: same `S` → byte-identical `(32,)` + output across repeated calls, across threads, across processes. + +T-1 and T-2 are the load-bearing ones. T-3 and T-4 are guards against silent +drift. + +### 1.5.5 What Stays Out of Scope of This ADR + +- **Approximate recall.** CLAUDE.md §Core Primitives is non-negotiable: exact + CGA recall, no HNSW/ANN/cosine. The CRDT merge produces *eventually-exact* + recall, never approximate. The sub-50ms window in §3.2 is a **latency** + window for write-visibility, not a **fidelity** window — once merged, recall + is exact byte-for-byte. +- **Hidden background execution.** The "Merge Kernel" in §2.2 must be an + explicitly-mounted runtime component with a named owner and observable + state, not a daemon thread. CLAUDE.md §Security forbids hidden background + execution; the kernel must surface its pending-delta count in + telemetry/`TurnEvent` for replay evidence. +- **MLX/UMA hardware optimization.** §2.3's zero-copy MLX handshake is a + follow-up ADR; it is mentioned here as horizon-setting only. The CRDT + substrate itself must work on a pure-CPU Rust path first. + +### 1.5.6 Cross-References + +- ADR-0054 (Vault Recall Indexing + Batching) — the matrix-cache contract the + sharded path must preserve at the read side. +- `docs/runtime_contracts.md` — the response/telemetry/memory/identity + contracts that §3.1's "zero modification of `anti_unifier` and `carrier`" + claim is being measured against. +- CLAUDE.md §Normalization Rules — `ingest/gate.py` remains the **only** + allowed pre-Vault normalization site; the CRDT substrate must not introduce + per-shard normalizers. + +## 2. Decision: Logical Unity, Physical Sharding + +We will resolve this tension by decoupling the logical manifold from the +physical memory layout. The manifold remains singular and mathematically +continuous, but the underlying Rust substrate will heavily shard the ingestion +pathways. + +We adopt an architecture based on **Delta-State CRDTs (Conflict-Free Replicated +Data Types)** acting over lock-free, thread-local arenas, resolved via +asynchronous Semilattice Joins. + +### 2.1 Thread-Local Sensory Arenas + +* **Deprecation of Direct Global Writes:** Adapters (`sensorium/adapters/*`) + are strictly forbidden from writing directly to the global `epistemic_state`. +* **Local Delta Caches:** Each active modality adapter is assigned a + thread-local memory arena in `core-rs`. As dense geometric primitives + (spheres, lines, motors) are generated by the `ProjectionHead`, they are + written lock-free into this local cache. + +### 2.2 The Semilattice Join (CRDT Merge) + +* The geometric `Field` operates as an additive accumulation of knowledge. It + mathematically satisfies the properties of a **Join Semilattice** + (commutativity, associativity, and idempotence of state integration). +* At predefined intervals (e.g., every 16ms to match 60fps, or at semantic + chunk boundaries), the local thread generates a `Delta` — a snapshot of the + newly ingested multivectors. +* A background, lock-free **Merge Kernel** sweeps these Deltas and folds them + into the global `Vault` using atomic compare-and-swap (CAS) operations or + unified memory tensor reductions via MLX. + +### 2.3 Nanospin Orchestration & Zero-Copy Symbiosis + +* **Python/Rust Boundary:** Python will write raw sensory arrays into + lock-free Ring Buffers (e.g., `crossbeam` channels). +* **Rust Worker Pool:** A pool of pinned Rust worker threads will continuously + poll these buffers using **nanospin** loops to avoid OS context-switching + latency. +* **MLX UMA Handshake:** For cross-modal resonance (calculating + `cga_inner(text_F, vision_F)` across millions of points), Rust will pass raw + memory pointers (`&[f32]`) directly to the MLX Neural Engine. MLX will + perform the massively parallel tensor reduction without ever copying the data + across a PCIe bus. + +## 3. Consequences + +### 3.1 Positive Impacts (The Exploits) + +* **Zero Ingestion Contention:** The vision pipeline can run at the maximum + framerate allowable by the neural backbone without ever being blocked by + textual or auditory processing. +* **Hardware Saturation:** Safely maximizes utilization of Apple Silicon + unified memory and multi-core Ryzen architectures. +* **Mathematical Purity Maintained:** The `anti_unifier` and `carrier` logic + requires zero modification. They simply operate on a slightly delayed, + eventually-consistent global state. + +### 3.2 Negative Impacts (The Risks) + +* **Eventual Consistency Latency:** There will be a sub-50ms window where a + visual primitive exists in the local Delta Cache but has not yet merged into + the global Vault. During this micro-window, cross-modal resonance with a + simultaneous text token cannot occur. +* **Memory Overhead:** Maintaining the local arenas and managing the Delta + garbage collection increases baseline memory footprint. + +## 4. Execution Plan & Proof Obligations + +1. **`core-rs` Mutation:** Refactor `core-rs/src/vault.rs` to implement the + `LocalArena` struct and the `SemilatticeDelta` trait. +2. **MLX Integration:** Define the zero-copy C-FFI boundary between the Rust + arenas and the MLX distance-calculation tensors. +3. **Trace Invariance Proof:** Extend `evals/` to prove that + `hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)`. The order of + asynchronous merging must not alter the final unified geometric topography. diff --git a/docs/handoff/PARALLEL-WORK-PLAN-2026-05-29.md b/docs/handoff/PARALLEL-WORK-PLAN-2026-05-29.md new file mode 100644 index 00000000..209b69d7 --- /dev/null +++ b/docs/handoff/PARALLEL-WORK-PLAN-2026-05-29.md @@ -0,0 +1,90 @@ +# Parallel work plan — post-GB-3a (2026-05-29) + +Dispatch-ready briefs for operators (Opus / Sonnet / Gemini) to run **in parallel +with** Claude's serial Gap-B line. Each brief touches a **disjoint file set** so +PRs cannot conflict. One sub-phase = one PR. Do **not** merge — Claude verifies +(lane-SHA + CLI) and merges. + +> **The hard constraint (learned this session):** GB-3b → GB-4 → GB-5 all edit the +> **same** files (`compose.py`, `clauses.py`). They are **serial** — never dispatch +> two of them in parallel. The four ADR-0179 EX PRs each rewrote one file off main +> and cost a full reconciliation. Parallelism is only safe across the *disjoint* +> surfaces below. + +The `wrong=0` rule, the sealed/serving boundary, and "no grammar templates" +(ADR-0165) from `docs/handoff/CHATGPT-REMOTE-BRIEF.md` §1/§2/§6 apply verbatim to +all three. Never touch `chat/**`, `generate/math_roundtrip.py`, `algebra/**`, +`field/**`, `vault/**`, CI files, or any currently-green test. + +--- + +## Track A — CP-1 cue-precision ledger substrate (ADR-0177) · Opus/Sonnet + +**Files (only):** new package `generate/cue_precision/` (e.g. `ledger.py`) + new +`tests/test_adr_0177_cp1_ledger.py`. **Read:** `docs/decisions/ADR-0177-cue-precision-learning.md` +§"Sub-phases" CP-1, and `core/reliability_gate/ledger.py` (mirror its `ClassTally` +counts-only, refusals-excluded discipline). + +**Scope:** the `(cue, op, unit_shape)` ledger + credit assignment **mechanism only** +— record per-pattern committed/correct counts from gold-checked practice chains. +**Do not wire it into the gate or any scorer** (that is CP-2). No reliability-based +resolution (CP-3). It is inert substrate with tests, exactly like +`core/reliability_gate/` shipped before its consumer. + +**Acceptance:** deterministic; counts-only (no float thresholds invented); +refusals never counted as commitments; serving untouched; the package is imported +by nothing outside its own tests. + +**Dispatch:** `Read docs/handoff/PARALLEL-WORK-PLAN-2026-05-29.md, do Track A (CP-1 ledger substrate) exactly as written. New generate/cue_precision/ package + its test only. Inert (no gate/scorer wiring). One PR. Do not merge.` + +--- + +## Track B — scale the sealed practice case set (ADR-0163 §F) · Sonnet/Codex + +**Files (only):** `evals/gsm8k_math/practice/**` data/case files (and a loader if +one is needed) — **not** `train_sample/v1/cases.jsonl` (that is the pinned serving +lane; do not touch it). **Read:** `evals/gsm8k_math/README.md`, +`evals/gsm8k_math/practice/v1/runner.py`. + +**Scope:** add more GSM8K practice cases (additive only) so the practice/learning +signal has volume — 50 is mechanism-demonstration, not enough for cue-precision to +learn. Pure data + scoring; the sealed lane gold-checks attempts, so more cases +can only add eliminations/correct, never a *served* wrong. + +**Acceptance:** deterministic ordering; cases are well-formed (id/question/answer); +the serving `train_sample` lane and its pinned SHA are **untouched**; `build_search_report` +still runs. State the new case count in the PR body. + +**Dispatch:** `Read docs/handoff/PARALLEL-WORK-PLAN-2026-05-29.md, do Track B (scale the practice case set) exactly as written. Only evals/gsm8k_math/practice/** ; never touch train_sample. Additive. One PR. Do not merge.` + +--- + +## Track C — EX-3 redo: tight multi-word units (ADR-0179) · Sonnet (or Gemini research) + +**Files (only):** `generate/derivation/extract.py` + `tests/test_adr_0179_extract.py`. +**Read:** `docs/handoff/AUDIT-ADR-0179-EX-RECONCILE.md` §"Why EX-3 was deferred" +(the exact trap), and the `extract.py` module docstring. + +**Scope:** a **tight** multi-word-unit rule that does **not** cross connectives and +does **not** regress GB-2. The deferred greedy version read `"6 apples and 4 apples"` +as unit `"apples and"` and broke `test_same_unit_list_sums`. The replacement must: +(a) keep `"12 jumping jacks"` → `"jumping jacks"` only where tight; (b) leave +`"6 apples and 4 apples"` as two `apples` quantities; (c) keep **every** test in +`test_adr_0178_gb1_clauses.py`, `test_adr_0178_gb2_compose.py`, +`test_adr_0178_gb3_referent_guard.py`, `test_adr_0179_extract.py` green. If no rule +satisfies all of (a)–(c) without a grammar template, **write a note and ship no +code** — a refusal is fine. + +**Acceptance:** all listed test files stay green; lexeme-level only (ADR-0165); +serving untouched. *Safe to run alongside GB-3b: Gap-B edits `compose.py`/`clauses.py`, +not `extract.py`.* + +**Dispatch:** `Read docs/handoff/PARALLEL-WORK-PLAN-2026-05-29.md, do Track C (tight multi-word units) exactly as written. Only extract.py + its test. Must not regress any GB-1/2/3 test. If impossible without a grammar template, write a note and ship no code. One PR. Do not merge.` + +--- + +## What Claude keeps (serial, do NOT dispatch) + +GB-3b (cross-clause chaining consuming GB-1, referent-safe) → GB-4 (held +hypotheses + eliminate) → GB-5 (DAG/0033-class). All on `compose.py`/`clauses.py`, +all `wrong=0`-critical. One at a time.