ADR-0197: vision compiler over Delta-CRDT (docs only) (#513)
This commit is contained in:
Shay
GitHub
parent
1c56ac710e
commit
81062bb0fb
3 changed files with 643 additions and 0 deletions
188
docs/decisions/ADR-0197-vision-compiler-delta-crdt.md
Normal file
188
docs/decisions/ADR-0197-vision-compiler-delta-crdt.md
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
# ADR-0197: CORE-native Vision Compiler over the Delta-CRDT Substrate
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-31
|
||||
**Authors:** Joshua M. Shay, Core R&D Engine
|
||||
**Domains:** `sensorium/vision/`, `sensorium/adapters/vision.py`, `packs/vision/`, `core-rs/src/vault.rs` (read-only contract), `evals/vision_sensorium/`
|
||||
**Depends on:** ADR-0013 (Sensorium Multimodal Protocol), ADR-0180 (Delta-CRDT Sharded Substrate), ADR-0181 (Audio Compiler — structural precedent)
|
||||
**Companion docs:** [vision-compiler-spec.md](../plans/vision-compiler-spec.md) *(to be written)*, [vision-compiler-eval-plan.md](../plans/vision-compiler-eval-plan.md) *(to be written)*
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & Problem Statement
|
||||
|
||||
`sensorium/protocol.py` already fixes the contract vision must satisfy:
|
||||
|
||||
- `ProjectionHead.project(S) -> (32,) float32` is the **Logos-recovery boundary** (`CL41_DIM = 32`). A visual scene is recovered as words in the one manifold; once it crosses the boundary the field has no concept of "vision."
|
||||
- `Modality.VISION` already exists in the enum but has **no adapter and no compiler** — `sensorium/adapters/` ships `text.py` and `audio.py` only.
|
||||
- `ModalityPack` enforces gate/checksum invariants at construction; `ModalityRegistry.mount()` runs the unitarity check and `project()` refuses a closed gate.
|
||||
|
||||
ADR-0013 (Accepted) requires every new modality to cross this boundary *before* it reaches `ingest/gate.py`, and forbids touching `ingest/`, `field/`, `generate/`, `vault/`, `vocab/` to add one. Vision must therefore **arrive already compiled** into a `(32,)` Cl(4,1) versor.
|
||||
|
||||
ADR-0180 (Proposed) introduces the Delta-CRDT sharded substrate **explicitly to absorb the continuous, high-density streams that audio and vision produce** — vision is named, not incidental — and imposes the same hard constraint audio answered (§1.5.2): any operation upstream of `vault/store` that the substrate parallelizes must either be proven order-invariant on its inputs, or carry an explicit serialization barrier.
|
||||
|
||||
ADR-0181 (Proposed) is the structural precedent. This ADR deliberately inherits its skeleton — pipeline shape, checksum chain, merge-key triple, serialization-barrier discipline, PR ladder, gate-closed default, and the **"compiler, not embedding bridge"** doctrine. What it does **not** inherit is audio's canonical ordering or blade semantics: those are temporal and acoustic. Vision's manifest is **spatial and scale-structured**, and that single difference is the load-bearing redesign of §2.1 and §2.6.
|
||||
|
||||
The problem this ADR solves: **how does vision enter CORE as a lawful, deterministic, replayable modality that is also a well-behaved Delta-CRDT delta producer** — without violating the no-core-mutation rule, the exact-recall rule, or ADR-0180's order-invariance obligation.
|
||||
|
||||
The wrong answer is an embedding bridge (ViT / CLIP / DINOv2 / SAM as substrate). The temptation is far stronger here than for audio because pretrained vision encoders are excellent — and that is exactly why it must be resisted. An opaque latent cannot be checksummed, cannot be replayed bit-for-bit, and cannot supply the content-addressed merge key ADR-0180 §1.5.3 demands. The right answer is a **deterministic visual compiler**.
|
||||
|
||||
## 2. Decision
|
||||
|
||||
We will build `vision_core_v1` as a **deterministic visual compiler** under `sensorium/vision/`, lowering a canonical image (or single video frame) through a typed `VisionIR` into a `(32,)` float32 Cl(4,1) versor, and we will make its **spatial chunk boundary the Delta-CRDT delta boundary**. Learned models (ViT, CLIP, DINOv2, SAM, depth/flow estimators) are admitted only as subordinate teacher/shadow lanes, never as the substrate.
|
||||
|
||||
The compilation pipeline (detailed in the companion spec) mirrors audio's shape with spatial substitutions:
|
||||
|
||||
```text
|
||||
image / single video frame
|
||||
→ canonicalizer (fixed colorspace + gamma, fixed resampling grid, source+canonical sha256)
|
||||
→ spatial grid (tile lattice × scale pyramid — the analog of audio's frame grid)
|
||||
→ visual lexer (orientation energy, spatial-frequency bands, luminance/chroma stats,
|
||||
corner/blob onsets, region boundaries)
|
||||
→ typed VisionIR (regions/segments, contour arcs, salient-object events, texture atoms,
|
||||
content anchors)
|
||||
→ operator registry (pack-local blade aliases + quantized theta rules)
|
||||
→ rotor lowering (elliptic bivector rotors only in v1)
|
||||
→ versor composition (geometric_product → unitize_versor → versor_condition < 1e-6)
|
||||
→ (32,) float32 == one VisionCompilationUnit
|
||||
```
|
||||
|
||||
Normalization sites stay inside CLAUDE.md's allowlist: quantization and resampling live in the vision pack/compiler construction boundary, `unitize_versor` is algebra-owned, and **no hot-path drift repair is added**.
|
||||
|
||||
### 2.1 The optimal mapping to Delta-CRDT (the load-bearing decision)
|
||||
|
||||
ADR-0180 §1.5.2 gives vision the same binary choice it gave audio: prove order-invariance, or carry a serialization barrier. We answer both, at two granularities — but the *within-chunk ordering is the part that differs from audio and must be designed, not copied*.
|
||||
|
||||
| Granularity | Operation | CRDT treatment | Why |
|
||||
|---|---|---|---|
|
||||
| **Within a chunk** (one tile, or one image at one scale) | `compile_events` (rotor chaining via `geometric_product`) | **Serialization barrier** | The sandwich product is non-commutative. In-chunk composition runs serially, single-threaded, in **canonical spatial order**, inside one thread-local arena. |
|
||||
| **Across tiles / scales / frames** | merge of `VisionCompilationUnit`s into the Vault | **Order-invariant delta** | Each unit is `(versor, provenance)` written at the `vault/store` layer — the only semilattice-eligible layer. Merge is commutative, associative, idempotent. |
|
||||
|
||||
**The redesign vs ADR-0181.** Audio's canonical within-chunk order is the temporal hop index — total and obvious. Vision has no time axis inside a frame, so the canonical order must be an **explicit, deterministic, resolution-independent spatial total order over the IR events**. The v1 proposal: order by `(scale_level, morton_code(tile_row, tile_col), event_category_precedence, stable_event_id)`, where `morton_code` is a Z-order curve over the fixed tile lattice. This makes the in-chunk fold reproducible and gives proof obligation V-4 something concrete to assert against. **This ordering rule is the single most important thing to red-line in this ADR** (see §2.6).
|
||||
|
||||
### 2.2 Content-addressed merge key from the checksum chain
|
||||
|
||||
Identical to ADR-0181 §2.2 in structure; only the layer names change:
|
||||
|
||||
```text
|
||||
source_sha256 → canonical_sha256 → tile_stream_sha256 → ir_sha256
|
||||
→ pack_manifest_sha256 → projection_sha256
|
||||
```
|
||||
|
||||
The **merge key for a vision delta is `(canonical_sha256, ir_sha256, projection_sha256)`** — the same triple `AudioCompilationUnit.merge_key` exposes, so the existing merge kernel and `core-rs` content-addressed sort apply unchanged. Consequences carry over verbatim: idempotence is structural (identical canonical pixels under an identical pack → identical key → CRDT join deduplicates), the content-addressed sort is free, and `projection_sha256` is computed behind the serialization barrier on the serialized in-chunk composition (ADR-0180 §1.5.3 point 3).
|
||||
|
||||
### 2.3 Physical sharding mirrors the visual domain's natural concurrency
|
||||
|
||||
ADR-0180 §2.1 assigns each active adapter a thread-local arena and forbids `sensorium/adapters/*` from writing global `epistemic_state` directly. For vision this is semantically aligned, not merely mechanical: **independent spatial regions, scale levels, and successive video frames are genuinely concurrent visual streams.** Each gets its own arena; each `VisualEvent` retains its tile/scale coordinates so the merge reconstructs spatial layout without a global lock during ingestion. The substrate's physical sharding is a faithful image of the visual source's structure — a tiled image is *already* a set of independent deltas.
|
||||
|
||||
### 2.4 Eventual-consistency window is safe for vision
|
||||
|
||||
Same argument as ADR-0181 §2.4. `ProjectionHead.project` is pure on the signal (ADR-0180 T-4): the vision compiler reads no cross-modal or global state during compilation, so a delayed merge cannot change what it produces. Each unit retains its own spatial coordinates, so cross-modal resonance re-anchors on merged state after the sub-50ms window closes; recall remains exact byte-for-byte once merged, never approximate.
|
||||
|
||||
### 2.5 Gate-closed by default
|
||||
|
||||
`vision_core_v1` mounts with `gate_engaged = false` until the eval gates in the companion plan pass. A closed gate makes `ModalityRegistry.project("vision_core_v1", …)` raise — vision contributes no deltas to any arena until determinism, checksum, unitarity, and mount-validation gates are green. This reuses existing registry enforcement; no new gating machinery is added.
|
||||
|
||||
### 2.6 OPEN DESIGN QUESTION — blade semantics for vision (red-line target)
|
||||
|
||||
The `(32,)` output shape and the unitarity check are **rigid contract** and not open. What is open — and what must be settled in the companion spec before PR-2 — is the **geometric meaning of the blades for a visual signal**. Audio used "elliptic bivector rotors only in v1" over an acoustic event vocabulary; vision cannot inherit that assignment because the underlying structure is spatial.
|
||||
|
||||
Cl(4,1) gives us (per `algebra/cl41.py`): grade-0 (scalar, idx 0), grade-1 (5 vectors, idx 1–5), grade-2 (10 bivectors, idx 6–15), grade-3 (10 trivectors, idx 16–25), grade-4 (5, idx 26–30), grade-5 (pseudoscalar, idx 31).
|
||||
|
||||
Candidate v1 assignment **(to be red-lined, not yet decided):**
|
||||
- **grade-1 vectors** → canonical spatial/scale frame (2 image axes + scale axis + 2 CGA null directions for position encoding, reusing the CGA machinery in `algebra/cga.py`).
|
||||
- **grade-2 bivectors** → oriented local structure: orientation energy and spatial-frequency bands lowered as elliptic rotors (the closest direct analog to audio's prosody arcs).
|
||||
- **grade-3 / grade-4** → compositional/region relationships (contour closure, containment, figure–ground).
|
||||
- **scalar / pseudoscalar** → global luminance/saliency magnitude and chirality.
|
||||
|
||||
Open sub-questions for review:
|
||||
1. Is `S` a whole canonical frame, a single tile, or a region crop? (Affects whether one image is one unit or many.) Proposed: **one tile at one scale = one chunk = one unit**; whole-image versor is their merged contribution.
|
||||
2. Should position be carried in CGA null vectors (conformal embedding) or quantized into the operator registry's theta rules, as audio quantizes pitch?
|
||||
3. Are elliptic rotors sufficient for v1, or does figure–ground require a hyperbolic/parabolic generator? (Default: elliptic-only in v1, matching ADR-0181.)
|
||||
4. What is the resolution-independence guarantee for `morton_code` ordering across canonical grid sizes within one pack version?
|
||||
|
||||
## 3. Consequences
|
||||
|
||||
### 3.1 Positive
|
||||
|
||||
- **Second concrete exerciser of ADR-0180**, and the first *spatial* one — it stresses the substrate's order-invariance proof against a non-temporal canonical order, which audio never exercised.
|
||||
- **Order-invariance is proven, not hoped.** §2.1/§2.2 supply a concrete serialization barrier and the same content-addressed key, closing ADR-0180 §1.5.2 for the vision path.
|
||||
- **No core mutation.** Everything new lives under `sensorium/vision/`, `packs/vision/`, `tests/`, `evals/`. `ingest/`, `field/`, `generate/`, `vault/`, `vocab/` are untouched (ADR-0013).
|
||||
- **Substrate reuse is total.** The merge-key triple is byte-identical in shape to audio's, so `core-rs` merge/dedup, trace hygiene, and the Python arena mirror all apply without substrate changes.
|
||||
- **Trace hygiene composes.** Turn traces record `(canonical_sha256, ir_sha256, projection_sha256)` and pack IDs — never raw pixels.
|
||||
|
||||
### 3.2 Negative / Risks
|
||||
|
||||
- **Embedding-bridge temptation (primary risk).** Pretrained vision encoders are so strong that reviewers will be tempted to admit one as substrate. The doctrine line must hold: substrate is the deterministic compiler; learned models are teacher/shadow lanes only.
|
||||
- **Semantic underreach (v1).** The compiler captures layout, structure, orientation, and salient regions better than fine-grained object identity. Acceptable: caption/detector teachers backfill identity while the substrate stays native (mirrors audio's transcript-teacher posture).
|
||||
- **Canonicalization brittleness.** Colorspace, gamma, and resampling kernel must be pinned; an unpinned resize is the vision analog of audio's unpinned FIR and will break A-1/V-1 determinism. Frozen in the pack manifest.
|
||||
- **Spatial quantization regime.** Tile size, scale-pyramid depth, and orientation bins frozen in the manifest; `basis_version` is part of the merge key's pack-manifest leg, so projections stay comparable across versions or are explicitly incomparable.
|
||||
- **Streaming seam artifacts (video).** Cross-frame continuity, optical-flow state, and temporal dedup are deferred to a streaming phase; **v1 is single-frame / whole-image, offline.**
|
||||
- **Licensing contamination.** Any GPL/non-commercial reference detector is an oracle only, never a runtime dependency.
|
||||
|
||||
## 4. Execution Plan & Proof Obligations
|
||||
|
||||
### 4.1 PR stack (additive, doctrine-first)
|
||||
|
||||
| PR | Scope | Gate |
|
||||
|---|---|---|
|
||||
| **PR-1 (this)** | ADR-0197 + vision-compiler-spec + eval plan (docs only) | review + blade-semantics red-line (§2.6) resolved |
|
||||
| **PR-2** | Deterministic substrate: `sensorium/vision/{types,canonical,checksum,resample,grid,lexer,parser,operators,compiler,trace}.py` | determinism + versor unit tests |
|
||||
| **PR-3** | Pack artifacts `packs/vision/vision_core_v1/*` + `VisionProjectionHead` adapter + mount tests | mount/gate/checksum gates |
|
||||
| **PR-4** | `evals/vision_sensorium/` fixtures, expected IR, expected projection hashes | full eval-gate table |
|
||||
| **PR-5** | Delta-CRDT wiring: `VisionCompilationUnit` → thread-local arena → merge key, behind ADR-0180's substrate | sequential==concurrent trace-hash proof |
|
||||
| **PR-6** | Teacher/shadow lanes (ViT/CLIP/DINOv2/SAM/depth) behind optional extras | teachers admitted only as typed hints |
|
||||
|
||||
PR-5 must not start until ADR-0180's §1.5.4 obligations (T-1…T-4) are green on `main`. PR-2 must not start until §2.6 is resolved.
|
||||
|
||||
### 4.2 Vision-specific proof obligations (extend ADR-0180 §1.5.4; mirror ADR-0181 §4.2)
|
||||
|
||||
Each must **fail loudly** under the violation it names:
|
||||
|
||||
- **V-1 (determinism).** Same canonical pixels + same pack ⇒ byte-identical `(32,)`, across repeated calls, threads, and processes. Fails on any non-determinism (dict ordering, unpinned resize/colorspace, float reduction order).
|
||||
- **V-2 (set-equality of merges).** A set of `VisionCompilationUnit`s folds to the same Vault state regardless of arena flush order (permutation invariance). Fails if a delta's contribution is order-sensitive at the merge layer.
|
||||
- **V-3 (content-addressed key).** Trace-hash over vision deltas is invariant under set-equal Vault states when keyed by `(canonical_sha256, ir_sha256, projection_sha256)`. Fails if the reduction consumes deltas in arrival order.
|
||||
- **V-4 (serialization barrier + canonical spatial order).** In-chunk `compile_events` is asserted order-sensitive (negative test): swapping two events in canonical spatial order changes the versor. **Plus the vision-specific clause:** the canonical spatial order (§2.1) is itself deterministic and stable under the pack's fixed grid — re-tiling the same canonical image yields the same event order.
|
||||
- **V-5 (versor condition).** Every emitted unit satisfies `versor_condition(v) < 1e-6`; threshold never weakened to pass.
|
||||
- **V-6 (trace hygiene).** No raw pixel bytes appear in any `TurnEvent`/Vault record; only the three hashes + pack IDs + optional teacher provenance.
|
||||
|
||||
### 4.3 The strict compilation invariant
|
||||
|
||||
```text
|
||||
same canonical image bytes
|
||||
+ same compiler version
|
||||
+ same pack manifest (incl. basis_version, tile/scale/orientation regime)
|
||||
+ same operator registry
|
||||
+ same canonical spatial ordering rule
|
||||
= same VisionIR
|
||||
= same versor
|
||||
= same projection hash
|
||||
= same CRDT merge key
|
||||
= identical post-merge Vault contribution (idempotent under re-ingest)
|
||||
```
|
||||
|
||||
## 5. Alternatives Considered
|
||||
|
||||
- **Embedding-first projector (ViT / CLIP / DINOv2 as substrate).** Fast and semantically rich, but opaque; cannot be replayed bit-for-bit and cannot supply ADR-0180 §1.5.3's content-addressed key. Rejected as substrate; retained as teacher/shadow.
|
||||
- **Segmentation-first (SAM as substrate).** Produces good region proposals but is non-deterministic across versions and gives no checksummable key. Rejected as substrate; useful as a teacher for region anchors.
|
||||
- **Patch-token codec (VQ-GAN / image tokenizer).** Strategically interesting for a future *generation/output* lane (and a likely touch-point with the eventual MOTOR decoder work); poor first substrate for an epistemically explicit engine. Deferred to a shadow/output lane, mirroring ADR-0181's EnCodec deferral.
|
||||
- **Vision as a downstream cognition mutation.** Violates ADR-0013's no-core-mutation rule and ADR-0180's "adapters never write global state directly" rule. Rejected.
|
||||
|
||||
## 6. Cross-References
|
||||
|
||||
- ADR-0013 — projection boundary; no-core-mutation constraint; Logos-recovery framing.
|
||||
- ADR-0180 — Delta-CRDT substrate; §1.5.2 order-invariance (closed here for vision via §2.1), §1.5.3 content-addressed merge key, §1.5.4 T-1…T-4 (vision analogs in §4.2), §1.5.5 trace hygiene.
|
||||
- ADR-0181 — Audio compiler; the structural precedent this ADR mirrors. Divergences are confined to canonical ordering (§2.1) and blade semantics (§2.6).
|
||||
- ADR-0054 — Vault recall indexing/batching; the read-side contract merged vision deltas must preserve (exact CGA recall).
|
||||
- CLAUDE.md §Normalization Rules — quantization/resampling confined to pack/compiler construction; `unitize_versor` algebra-owned; no hot-path repair.
|
||||
- `sensorium/protocol.py`, `sensorium/registry.py` — the `ProjectionHead` / `ModalityPack` / `ModalityRegistry` contracts this ADR implements a vision instance of.
|
||||
|
||||
---
|
||||
|
||||
### Open questions for red-line (consolidated)
|
||||
|
||||
1. **§2.6 blade assignment** — confirm or revise the grade→meaning mapping. *Blocks PR-2.*
|
||||
2. **§2.1 canonical spatial order** — accept `(scale_level, morton_code, precedence, stable_id)` or propose an alternative deterministic total order. *Blocks V-4.*
|
||||
3. **Unit granularity** — is one tile-at-one-scale the chunk (proposed), or the whole frame?
|
||||
4. **Position encoding** — CGA null vectors vs. quantized operator theta rules.
|
||||
5. **Companion specs** — confirm `vision-compiler-spec.md` + `vision-compiler-eval-plan.md` as the PR-1 deliverables alongside this ADR.
|
||||
137
docs/plans/vision-compiler-eval-plan.md
Normal file
137
docs/plans/vision-compiler-eval-plan.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Vision Compiler Eval Plan — `vision_core_v1`
|
||||
|
||||
**Companion to:** [ADR-0197](../decisions/ADR-0197-vision-compiler-delta-crdt.md),
|
||||
[vision-compiler-spec.md](./vision-compiler-spec.md)
|
||||
**Status:** Proposed (PR-1 docs)
|
||||
|
||||
This plan defines the seeding corpus, the acceptance gates that lift `vision_core_v1` from gate-closed to gate-engaged, the Delta-CRDT proof obligations, and the teacher-migration policy. It is the last PR-1 companion ADR-0197 names; it is what PR-2's "determinism + versor tests" gate is measured against.
|
||||
|
||||
---
|
||||
|
||||
## 1. Seeding corpus — visual atoms, not captions
|
||||
|
||||
Small, curated, checksum-locked. Four tiers:
|
||||
|
||||
- **Tier A — structural atoms:** flat field, oriented edge (each orientation bin), corner/junction, blob/region onset, open vs. closed contour, low- vs. high-spatial-frequency band.
|
||||
- **Tier B — region & figure–ground:** salient figure on ground, background texture, occlusion boundary, symmetry, periodic repetition, scale-change across pyramid levels.
|
||||
- **Tier C — color/material/lighting:** hue/saturation regimes, high vs. low luminance contrast, shadow/highlight, specular vs. matte, gradient ramp vs. hard step.
|
||||
- **Tier D — alignment anchors:** caption hypothesis, detector box, segment mask, OCR text span, linked text surface (only when alignment is trustworthy).
|
||||
|
||||
Tier D is auxiliary; object/lexical semantics is a later enrichment step. Synthetic fixtures (oriented sinusoidal gratings, checkerboards, luminance gradients, single-dot impulses, step edges, solid color fields, controlled occlusions) drive first-pass determinism; checksum-locked real fixtures cover what synthesis does poorly (natural texture, soft occlusion, cluttered figure–ground).
|
||||
|
||||
## 2. Acceptance gates (gate-engaged criteria)
|
||||
|
||||
| Gate | Pass criterion | Obligation |
|
||||
|---|---|---|
|
||||
| Projection shape | exactly `(32,)` | V-1 |
|
||||
| Projection dtype | exactly `float32` | V-1 |
|
||||
| Compiler replay | bit-identical on same platform/build | V-1 |
|
||||
| Cross-platform stability | equal after quantization, within declared numeric tolerance | V-1 |
|
||||
| Canonical-ordering stability | re-tiling the same canonical image yields the same event order | V-4 |
|
||||
| `versor_condition` | `< 1e-6` (never weakened) | V-5 |
|
||||
| Canonical checksum stability | 100% on fixture corpus (colorspace + resize pinned) | V-1 |
|
||||
| Gate closure | projection blocked when `gate_engaged = false` | — |
|
||||
| Mount validation | bad checksum or bad unitarity blocks pack mount | — |
|
||||
| Trace hygiene | no raw pixel bytes in any turn trace | V-6 |
|
||||
| IR replay | `VisionIR -> versor` replays identically from stored IR | V-1 |
|
||||
|
||||
These mirror the existing modality-test posture (`sensorium/registry.py` mount/gate enforcement, `ModalityPack.__post_init__` invariants) — no new gating machinery.
|
||||
|
||||
## 3. Delta-CRDT proof obligations (ADR-0197 §4.2)
|
||||
|
||||
Each must be able to **fail loudly** under the violation it names (CLAUDE.md §Schema-Defined Proof Obligations — no decorative tests):
|
||||
|
||||
| ID | Obligation | Fails if… | ADR-0180 analog |
|
||||
|---|---|---|---|
|
||||
| **V-1** | Determinism: same canonical pixels + pack ⇒ byte-identical `(32,)` across calls/threads/processes | dict ordering, unpinned resize/colorspace, or float reduction order leaks in | T-4 |
|
||||
| **V-2** | Set-equality of merges: a set of tile units folds to the same Vault state for any arena flush permutation | a delta's contribution is order-sensitive at the merge layer | T-1 |
|
||||
| **V-3** | Content-addressed trace-hash: invariant under set-equal Vault states when keyed by `(canonical, ir, projection)` sha | the reduction consumes deltas in arrival order | T-2 |
|
||||
| **V-4** | Serialization barrier **+ canonical spatial order**: in-chunk `compile_events` is order-sensitive (negative test), and the §6 spatial order is itself deterministic and re-tiling-stable | swapping two spatial-order events fails to change the versor, **or** re-tiling the same image reorders events | T-3 |
|
||||
| **V-5** | `versor_condition < 1e-6` on every emitted unit | the threshold is weakened to pass | — |
|
||||
| **V-6** | Trace hygiene: no pixels in any `TurnEvent`/Vault record | raw pixel data leaks into a delta's provenance | §1.5.5 |
|
||||
|
||||
V-4 carries one clause audio's A-4 did not: vision has no temporal axis, so the canonical *spatial* order (vision-compiler-spec §6) is the thing that makes the fold reproducible, and it must be tested as such.
|
||||
|
||||
### 3.1 The sequential==concurrent proof (PR-5 acceptance)
|
||||
|
||||
The load-bearing test for the CRDT mapping:
|
||||
|
||||
```text
|
||||
ingest tiles [t1, t2, t3] sequentially → vault_seq, trace_hash_seq
|
||||
ingest the same tiles across N arenas merged → vault_conc, trace_hash_conc
|
||||
assert set(vault_seq) == set(vault_conc) # V-2
|
||||
assert trace_hash_seq == trace_hash_conc # V-3
|
||||
assert re-ingesting any tile is a no-op on the vault # idempotence (ADR-0197 §4.3)
|
||||
```
|
||||
|
||||
This is the vision instance of ADR-0180 §4.3's `hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)` and must pass before PR-5 merges. Because tiles are spatially independent, this proof is the first *non-temporal* exerciser of the substrate's order-invariance (ADR-0197 §3.1).
|
||||
|
||||
### 3.2 Pytest skeleton
|
||||
|
||||
```python
|
||||
def test_vision_projection_is_deterministic(vision_fixture, vision_pack): # V-1
|
||||
v1 = vision_pack.projection.project(vision_fixture)
|
||||
v2 = vision_pack.projection.project(vision_fixture)
|
||||
assert v1.shape == (32,)
|
||||
assert v1.dtype.name == "float32"
|
||||
assert np.array_equal(v1, v2)
|
||||
|
||||
def test_vision_pack_gate_blocks_projection(vision_pack_closed, vision_fixture): # gate closure
|
||||
with pytest.raises(Exception):
|
||||
_ = vision_pack_closed.projection.project(vision_fixture)
|
||||
|
||||
def test_vision_ir_replay_matches_original(vision_fixture, compiler): # IR replay
|
||||
unit = compiler.compile_image(vision_fixture)
|
||||
replay = compiler.compile_ir(unit.vision_ir)
|
||||
assert np.array_equal(unit.versor, replay.versor)
|
||||
assert unit.ir_sha256 == replay.ir_sha256
|
||||
|
||||
def test_chunk_composition_is_order_sensitive(two_events, compiler): # V-4 barrier
|
||||
a = compiler.compile_events([two_events[0], two_events[1]])
|
||||
b = compiler.compile_events([two_events[1], two_events[0]])
|
||||
assert not np.array_equal(a, b)
|
||||
|
||||
def test_canonical_spatial_order_is_retiling_stable(vision_fixture, compiler): # V-4 spatial clause
|
||||
order_a = compiler.canonical_event_order(vision_fixture)
|
||||
order_b = compiler.canonical_event_order(compiler.recanonicalize(vision_fixture))
|
||||
assert [e.stable_event_id for e in order_a] == [e.stable_event_id for e in order_b]
|
||||
|
||||
def test_merge_is_permutation_invariant(units): # V-2
|
||||
import itertools
|
||||
states = {fold_into_vault(p) for p in itertools.islice(_perms(units), 8)}
|
||||
assert len(states) == 1
|
||||
```
|
||||
|
||||
## 4. Teacher / shadow lane policy (PR-6)
|
||||
|
||||
Teachers label or align; they **never** define the substrate and **never** fold embeddings into the main versor path. They are admitted only through typed, versioned, checksummed hints stored as `content_anchors` / `evidence_ids` in the IR.
|
||||
|
||||
| Source | Best role in CORE | Why not the substrate |
|
||||
|---|---|---|
|
||||
| **CLIP** | coarse scene/object labels, image-text alignment prototypes | embedding model; opaque latent violates the design goal |
|
||||
| **DINOv2** | self-supervised region/correspondence hints | dense embedding; not a checksummable typed visual IR |
|
||||
| **ViT (supervised)** | classification evidence, weak object labels | latent features, no content-addressed key |
|
||||
| **SAM** | region/segment proposals as `region` anchors | non-deterministic across versions; gives no checksummable key |
|
||||
| **Depth Anything / RAFT** | depth/flow evidence lanes (and the seam for future video) | estimators, not a CORE-native deterministic compiler |
|
||||
| **Tesseract / OCR** | text-span anchors when glyphs are present | text extraction, wrong primary ontology for vision |
|
||||
|
||||
Migration policy, verbatim (identical to audio):
|
||||
|
||||
```text
|
||||
Use teachers to label or align.
|
||||
Never let teachers define the substrate.
|
||||
Never fold teacher embeddings directly into the main versor path.
|
||||
Only admit teacher outputs through typed, versioned, checksumed hints.
|
||||
```
|
||||
|
||||
This is the line ADR-0197 §3.2 names as the primary risk: pretrained vision encoders are strong enough to tempt substrate use. They stay in this table, never on the versor path.
|
||||
|
||||
## 5. Phased sequence (priority order, no calendar)
|
||||
|
||||
1. **Doctrine** — ADR + spec + eval plan locked (PR-1, this).
|
||||
2. **Deterministic substrate** — canonicalizer, checksums, resample, grid, lexer, parser, operators, compiler (PR-2).
|
||||
3. **Governance** — pack artifacts, adapter, mount/gate/checksum tests (PR-3).
|
||||
4. **Evaluation** — fixtures, expected IR, expected projection hashes, gate table (PR-4).
|
||||
5. **Delta-CRDT wiring** — arena + merge key + sequential==concurrent proof (PR-5), gated on ADR-0180 §1.5.4 (T-1…T-4) green on `main`.
|
||||
6. **Auxiliary lanes** — CLIP/DINOv2/SAM/depth/OCR teachers behind optional extras (PR-6).
|
||||
7. **Streaming** — stateful incremental compiler preserving continuity across frames/seams (later; v1 is single-frame/whole-image, offline).
|
||||
318
docs/plans/vision-compiler-spec.md
Normal file
318
docs/plans/vision-compiler-spec.md
Normal file
|
|
@ -0,0 +1,318 @@
|
|||
# Vision Compiler Spec — `vision_core_v1`
|
||||
|
||||
**Companion to:** [ADR-0197](../decisions/ADR-0197-vision-compiler-delta-crdt.md)
|
||||
**Status:** Proposed (PR-1 docs)
|
||||
**Scope:** the deterministic substrate (PR-2/PR-3) and its Delta-CRDT delta interface (PR-5).
|
||||
|
||||
This spec fixes the typed IR, the operator/manifest format, the numeric determinism rules, and the `VisionCompilationUnit` → Delta-CRDT delta contract. It is implementation-facing; the *why* lives in ADR-0197, the *acceptance* lives in the eval plan. It also **resolves the two ADR-0197 red-line blockers** — §2.1 canonical spatial ordering and §2.6 blade semantics — see §6 and §7 below.
|
||||
|
||||
---
|
||||
|
||||
## 0. Resolutions of the ADR-0197 open questions
|
||||
|
||||
| ADR-0197 open Q | Resolution in this spec |
|
||||
|---|---|
|
||||
| **#1 unit granularity** | **One tile at one scale level = one chunk = one `VisionCompilationUnit`.** The whole-image versor is the *merged* contribution of its tile units, never a separate object. |
|
||||
| **#2 position encoding** | **v1: position is carried in IR tile/scale coordinates and modulates rotor `theta`; layout is preserved by the canonical spatial order (§6).** CGA conformal *translators* (parabolic, `n_inf` generator) are deferred to v2. Rationale: keeps v1 elliptic-only and audio-parallel, and avoids unproven parabolic numerics on the hot path. |
|
||||
| **#3 elliptic sufficiency** | **Elliptic bivector rotors only in v1** (square = −1), matching ADR-0181. Figure–ground is a grade-2 saliency rotor (`B_SALIENCE`), not a boost. |
|
||||
| **#4 morton resolution-independence** | **Resolved by construction:** tiling happens *after* canonicalization to the pack's fixed grid (§3), so `morton_code` runs over fixed normalized tile indices and is identical regardless of source resolution. |
|
||||
|
||||
The CGA machinery in `algebra/cga.py` (`embed_point`, `cga_inner`) remains the **recall-side** distance metric for merged vision deltas in the Vault (ADR-0054) — exactly as for audio. This spec governs only how the **compile-side** versor is *built* (elliptic-rotor composition); it does not change how versors are *compared* at recall.
|
||||
|
||||
## 1. Two-clock architecture
|
||||
|
||||
A low-level **spatial clock** measures pixel facts; a higher-level **visual-grammar clock** emits typed events. The primary path is fully deterministic; learned systems are confined to auxiliary evidence lanes (PR-6).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Image bytes / single video frame] --> B[Canonicalizer<br/>fixed colorspace + gamma + grid + checksums]
|
||||
B --> C[Spatial grid<br/>tile lattice × scale pyramid]
|
||||
C --> D[Visual lexer<br/>orientation energy, spatial-freq bands,<br/>luma/chroma stats, corner/blob onset, region boundaries]
|
||||
D --> E[Typed VisionIR parser<br/>regions/segments, contour arcs,<br/>salient-object events, texture atoms, anchors]
|
||||
E --> F[Canonical spatial ordering<br/>scale → morton → precedence → stable_id]
|
||||
F --> G[Operator registry<br/>pack manifest + blade aliases + theta rules]
|
||||
G --> H[Rotor lowering]
|
||||
H --> I[Versor composition<br/>unitize_versor + versor_condition]
|
||||
I --> J["(32,) float32 — one VisionCompilationUnit"]
|
||||
E --> K[Vision evidence trace<br/>hashes, teacher provenance, pack IDs]
|
||||
J --> L[Thread-local arena<br/>ADR-0180 §2.1]
|
||||
L --> M[Semilattice merge<br/>keyed by content-addressed sha]
|
||||
```
|
||||
|
||||
## 2. Typed VisionIR
|
||||
|
||||
The IR is built from **typed regions and events**, never from raw pixels or feature maps. Detector/caption hypotheses may exist only as auxiliary content anchors, never as the sole meaning of the image.
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
import numpy as np
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VisionImage:
|
||||
pixels: np.ndarray # canonical linear-light float32, shape (H, W, C)
|
||||
grid_h: int # canonical tile rows
|
||||
grid_w: int # canonical tile cols
|
||||
scale_levels: int
|
||||
source_sha256: str
|
||||
canonical_sha256: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class TileCoord:
|
||||
scale_level: int # 0 = finest
|
||||
tile_row: int
|
||||
tile_col: int
|
||||
|
||||
@property
|
||||
def morton(self) -> int: # Z-order interleave of (tile_row, tile_col)
|
||||
r, c, code, bit = self.tile_row, self.tile_col, 0, 0
|
||||
while (r >> bit) or (c >> bit):
|
||||
code |= ((r >> bit) & 1) << (2 * bit)
|
||||
code |= ((c >> bit) & 1) << (2 * bit + 1)
|
||||
bit += 1
|
||||
return code
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VisualToken:
|
||||
kind: Literal[
|
||||
"flat", "edge", "corner", "blob", "texture",
|
||||
"orient_bin", "freq_bin", "chroma_bin",
|
||||
]
|
||||
coord: TileCoord
|
||||
value_q: tuple[int, ...] # canonical quantized payload
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VisualEvent:
|
||||
event_type: str
|
||||
coord: TileCoord
|
||||
attrs: tuple[tuple[str, int | str], ...] # quantized ints / short strings
|
||||
evidence_ids: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VisionIR:
|
||||
regions: tuple[VisualEvent, ...]
|
||||
contour_arcs: tuple[VisualEvent, ...]
|
||||
orient_events: tuple[VisualEvent, ...]
|
||||
texture_atoms: tuple[VisualEvent, ...]
|
||||
salient_events: tuple[VisualEvent, ...]
|
||||
content_anchors: tuple[VisualEvent, ...]
|
||||
ir_sha256: str
|
||||
```
|
||||
|
||||
### 2.1 The compilation unit (the CRDT delta)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VisionCompilationUnit:
|
||||
canonical_sha256: str
|
||||
ir_sha256: str
|
||||
pack_id: str
|
||||
pack_manifest_sha256: str
|
||||
projection_sha256: str
|
||||
coord: TileCoord # tile/scale this unit covers
|
||||
versor: np.ndarray # (32,) float32
|
||||
versor_condition: float
|
||||
|
||||
@property
|
||||
def merge_key(self) -> tuple[str, str, str]:
|
||||
# ADR-0197 §2.2 — same triple as AudioCompilationUnit.
|
||||
return (self.canonical_sha256, self.ir_sha256, self.projection_sha256)
|
||||
```
|
||||
|
||||
`VisionCompilationUnit` is the single object the vision adapter writes into its thread-local arena (ADR-0180 §2.1). It carries no pixels (ADR-0197 §3.1 / ADR-0180 §1.5.5). `coord` is retained so the merge can reconstruct spatial layout without a global lock (ADR-0197 §2.3).
|
||||
|
||||
## 3. Canonical signal formation
|
||||
|
||||
- Internal representation: **linear-light float**, fixed colorspace (sRGB primaries, gamma linearized to a pinned LUT), alpha dropped, fixed canonical resolution grid. Original-source bytes preserved separately for provenance.
|
||||
- Resampling: **pinned separable kernel** (fixed Lanczos-3 coefficients), generated **once**, stored as a pack artifact (`resample_kernel_v1.npy`) and checksummed in the manifest. The runtime never relies on library defaults — this is the vision analog of audio's pinned FIR (ADR-0197 §3.2).
|
||||
- Tiling happens **after** canonical resize, so the tile lattice and `morton` order are resolution-independent (resolves ADR-0197 #4).
|
||||
|
||||
## 4. Visual lexer
|
||||
|
||||
Operates on **measured facts**, not semantic guesses. Default tile 16×16 px at the finest scale; a fixed 3-level Gaussian/Laplacian pyramid. Each tile yields quantized descriptors: dominant gradient-orientation histogram bin, oriented-energy bin per spatial-frequency band (low/high), local luminance-contrast bin, quantized hue/saturation regime, corner/blob response bins, texture periodicity/entropy bin, flat-region flag.
|
||||
|
||||
## 5. Parser → typed events
|
||||
|
||||
Promotes lexer output into typed regions/events. Preserves the distinctions a downstream reader needs: a hard oriented edge vs. a soft luminance ramp, a closed contour vs. a dangling one, a salient figure vs. background texture. "Unstructured texture" is the fallback only when a more specific parse is impossible — the visual analog of audio's "chaotic noise" fallback.
|
||||
|
||||
## 6. Canonical spatial ordering (resolves ADR-0197 §2.1)
|
||||
|
||||
The in-chunk fold is a **serialization barrier** (ADR-0197 §2.1); it requires a deterministic total order over the chunk's `VisualEvent`s. The order is:
|
||||
|
||||
```text
|
||||
key(event) = (coord.scale_level,
|
||||
coord.morton,
|
||||
event_precedence[category(event.event_type)],
|
||||
stable_event_id)
|
||||
```
|
||||
|
||||
- `coord.morton` is the Z-order interleave in `TileCoord.morton` — a space-filling curve that keeps spatially-near events adjacent in the fold.
|
||||
- `event_precedence` is a fixed list in the manifest (§6.1 `[ordering]`).
|
||||
- `stable_event_id` is the content hash of the event's quantized attrs (final tiebreak; never wall-clock).
|
||||
|
||||
**This order is the thing V-4 asserts against** (ADR-0197 §4.2): re-tiling the same canonical image yields the same order, and swapping two events changes the versor.
|
||||
|
||||
## 7. Operator registry (pack-local blade aliases)
|
||||
|
||||
Because the `(32,)` boundary is fixed but no canonical *semantic* blade map is exposed, v1 uses **pack-local, versioned, checksummed blade aliases**, identical in discipline to ADR-0181 §6. v1 uses **elliptic bivector operators only** (square = −1), so every rotor is `R = cos(θ/2) + B·sin(θ/2)`. Parabolic (CGA translator) and hyperbolic operators are deferred to v2.
|
||||
|
||||
Position does **not** get its own geometric generator in v1 (resolves ADR-0197 #2): tile/scale coordinates modulate `theta` and order the fold; they are not embedded as a CGA point on the compile path.
|
||||
|
||||
| Visual atom family | Measured source | Alias | Default blade index | Theta rule |
|
||||
|---|---|---|---|---|
|
||||
| Oriented edge energy | gradient-orientation histogram | `B_ORIENT` | 6 | `q(base + g1·orient_q + g2·scale_level)` |
|
||||
| Low spatial-frequency band | low bandpass energy | `B_FREQ_LOW` | 7 | `q(base + g3·energy_q)` |
|
||||
| High spatial-frequency band | high bandpass energy | `B_FREQ_HIGH` | 8 | `q(base + g4·energy_q)` |
|
||||
| Corner / junction | corner response bin | `B_CORNER` | 9 | `q(base + g5·corner_q)` |
|
||||
| Blob / region onset | blob detector bin | `B_BLOB` | 10 | `q(base + g6·blob_q)` |
|
||||
| Contour closure | boundary continuity | `B_CONTOUR` | 11 | `q(base + g7·closure_q)` |
|
||||
| Luminance contrast | local contrast bin | `B_CONTRAST` | 12 | `q(base + g8·contrast_q)` |
|
||||
| Chroma / color regime | quantized hue/sat bin | `B_CHROMA` | 13 | `q(base + g9·hue_q + g10·sat_q)` |
|
||||
| Texture regularity | periodicity / entropy bin | `B_TEXTURE` | 14 | `q(base + g11·texture_q)` |
|
||||
| Saliency / figure–ground | center-surround salience | `B_SALIENCE` | 15 | `q(base + g12·salience_q)` |
|
||||
|
||||
Indices are **reasonable defaults, not metaphysical claims** about Cl(4,1) (verbatim the ADR-0181 §6 stance). The contract is that the mapping is explicit, versioned, checksummed, and frozen in the manifest. `B_SALIENCE` is the figure–ground atom that ADR-0197 §2.6 declined to model with a boost.
|
||||
|
||||
### 7.1 Minimal manifest (`packs/vision/vision_core_v1/manifest.toml`)
|
||||
|
||||
```toml
|
||||
pack_id = "vision_core_v1"
|
||||
modality = "vision"
|
||||
cl41_dim = 32
|
||||
compiler_version = "0.1.0"
|
||||
basis_version = "vision-basis-v1"
|
||||
|
||||
[canonical]
|
||||
colorspace = "srgb_linear"
|
||||
gamma_lut = "gamma_lut_v1.npy"
|
||||
tile_px = 16
|
||||
scale_levels = 3
|
||||
output_dtype = "float32"
|
||||
internal_dtype = "float64"
|
||||
|
||||
[resampling]
|
||||
algorithm = "separable_lanczos3"
|
||||
kernel_path = "resample_kernel_v1.npy"
|
||||
kernel_sha256 = "sha256:REPLACE_ME"
|
||||
|
||||
[gating]
|
||||
gate_engaged = false
|
||||
checksum_verified = false
|
||||
versor_condition_max = 1.0e-6
|
||||
|
||||
[ordering]
|
||||
event_precedence = ["region", "contour", "orient", "texture", "salient", "content_anchor"]
|
||||
```
|
||||
|
||||
### 7.2 Operator row (`operators.jsonl`)
|
||||
|
||||
```json
|
||||
{
|
||||
"operator_id": "vision.orient.edge_energy.v1",
|
||||
"event_type": "orient.edge_energy",
|
||||
"blade_alias": "B_ORIENT",
|
||||
"blade_index": 6,
|
||||
"rotor_kind": "elliptic",
|
||||
"base_theta_q": 48,
|
||||
"gain_rules": {"orient_q": 3, "scale_level": 2, "confidence_q": 1},
|
||||
"theta_clip_q": 384,
|
||||
"version": "1"
|
||||
}
|
||||
```
|
||||
|
||||
## 8. Numeric determinism
|
||||
|
||||
Rule (verbatim from ADR-0181 §7): **quantize before semantics, normalize after composition.** Quantization regime (frozen in manifest): orientation in 16 ordinal bins, oriented energy in log bins, contrast in dB-like bins, hue/sat in fixed ordinal bins, all confidences in uint8. After quantization, compute in float64, compose sparse rotors in canonical spatial order, call algebra-owned `unitize_versor`, cast to float32 **only** at the output boundary.
|
||||
|
||||
```python
|
||||
import math
|
||||
import numpy as np
|
||||
|
||||
def quantize_theta(theta: float, step: float = 1.0 / 1024.0) -> float:
|
||||
return round(theta / step) * step
|
||||
|
||||
def build_elliptic_rotor(blade_index: int, theta: float) -> np.ndarray:
|
||||
out = np.zeros(32, dtype=np.float64)
|
||||
half = quantize_theta(theta) / 2.0
|
||||
out[0] = math.cos(half)
|
||||
out[blade_index] = math.sin(half)
|
||||
return out
|
||||
|
||||
def compile_events(events, registry, geometric_product, unitize_versor, versor_condition):
|
||||
# SERIALIZATION BARRIER (ADR-0197 §2.1): in-chunk composition is order-sensitive,
|
||||
# single-threaded, in CANONICAL SPATIAL ORDER (§6). The substrate never parallelizes this.
|
||||
v = np.zeros(32, dtype=np.float64)
|
||||
v[0] = 1.0
|
||||
for ev in events: # MUST already be in §6 canonical spatial order
|
||||
spec = registry[ev.event_type]
|
||||
theta = spec.theta_from_event(ev) # deterministic, quantized inputs only
|
||||
r = build_elliptic_rotor(spec.blade_index, theta)
|
||||
v = geometric_product(v, r)
|
||||
v = unitize_versor(v)
|
||||
if versor_condition(v) >= 1e-6:
|
||||
raise ValueError("vision compilation failed versor check")
|
||||
return v.astype(np.float32)
|
||||
```
|
||||
|
||||
`geometric_product`, `unitize_versor`, `versor_condition` are imported from `algebra/`; the vision compiler adds **no** new normalization function. `embed_point`/`cga_inner` are used only on the Vault recall side, never here.
|
||||
|
||||
## 9. Repo-facing adapter (`sensorium/adapters/vision.py`)
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
import numpy as np
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class VisionProjectionHead:
|
||||
compiler: "VisionCompiler"
|
||||
modality = ... # Modality.VISION
|
||||
|
||||
@property
|
||||
def embedding_dim(self) -> int:
|
||||
return 32
|
||||
|
||||
def project(self, image: "VisionImage") -> np.ndarray:
|
||||
# One image projects to its merged tile-unit versor (granularity resolution #1):
|
||||
out = self.compiler.compile_image(image).versor
|
||||
if out.shape != (32,):
|
||||
raise ValueError(f"expected (32,), got {out.shape}")
|
||||
if out.dtype != np.float32:
|
||||
raise TypeError(f"expected float32, got {out.dtype}")
|
||||
return out
|
||||
|
||||
def project_batch(self, images: list["VisionImage"]) -> np.ndarray:
|
||||
return np.stack([self.project(im) for im in images], axis=0)
|
||||
|
||||
def verify_unitarity(self, image: "VisionImage") -> bool:
|
||||
return self.compiler.compile_image(image).versor_condition < 1e-6
|
||||
```
|
||||
|
||||
The adapter is thin and pack-governed; it satisfies the `ProjectionHead` protocol in `sensorium/protocol.py` and is mounted as `ModalityPack(modality_type=Modality.VISION, gate_engaged=False)` until the eval gates pass.
|
||||
|
||||
## 10. Delta-CRDT delta interface (PR-5)
|
||||
|
||||
The vision adapter **never** writes the global `epistemic_state` (ADR-0180 §2.1). Instead:
|
||||
|
||||
1. `compile_tile()` produces one `VisionCompilationUnit` per tile/scale (the §8 serialization barrier runs here).
|
||||
2. Each unit is written lock-free into the adapter's **thread-local arena**. Independent regions/scales/frames each have their own arena (ADR-0197 §2.3).
|
||||
3. The **Merge Kernel** (ADR-0180 §2.2, explicitly mounted, not a daemon) folds pending units into the Vault ordered by `unit.merge_key`. Duplicate keys deduplicate (idempotence).
|
||||
4. The kernel surfaces its pending-delta count in `TurnEvent` for replay evidence (ADR-0180 §1.5.5).
|
||||
|
||||
The per-tile Vault contribution is `(versor, provenance)` where provenance = `{merge_key, pack_id, pack_manifest_sha256, coord}` — content-addressed, no pixels.
|
||||
|
||||
## 11. File plan (PR-2 … PR-6)
|
||||
|
||||
```text
|
||||
sensorium/vision/{__init__,types,canonical,checksum,resample,grid,lexer,parser,operators,compiler,trace,fixtures,teachers}.py
|
||||
sensorium/adapters/vision.py
|
||||
packs/vision/vision_core_v1/{manifest.toml,basis_map.json,operators.jsonl,atoms.jsonl,prototypes.jsonl,resample_kernel_v1.npy,gamma_lut_v1.npy,checksums.json}
|
||||
tests/test_vision_{image,resample,grid,lexer,parser,ordering,pack_manifest,sensorium_mount,trace,crdt_delta}.py
|
||||
evals/vision_sensorium/{fixtures/*.png,manifest.json,expected_ir.jsonl,expected_projection_hashes.json}
|
||||
```
|
||||
Loading…
Reference in a new issue