ADR-0013: `sensorium/` Multimodal Protocol Layer

Status: Accepted
Date: 2026-05-13

Context

CORE is currently text-only. The vocabulary manifold holds null vectors for text tokens. The ingest/gate.py accepts token sequences. The field, propagation, vault, and generate layers have no concept of modality — they operate on (32,) multivectors regardless of where those multivectors came from.

This is the correct architecture. The field should not know or care about modality. The question is: how does a vision signal, an audio waveform, or a motor pose become a (32,) multivector in the first place?

The core_sensorium package in core-ai solved this problem for the Cl(3,0) (2, 2) complex multivector substrate. The solution — a typed modality protocol with a ProjectionHead at the inward boundary — is architecturally sound and translates directly to the Cl(4,1) (32,) substrate.

Decision

Add a sensorium/ package that sits upstream of core_ingest/ as the modality-to-manifold conversion layer. ingest/gate.py, field/, generate/, and vault/ are not modified.

The Pipeline Position

raw modality signal
    → sensorium/adapters/<modality>.py     # ProjectionHead: signal → (32,) multivector
    → sensorium/pack.py                    # ModalityPack mounts the adapter
    → core_ingest/                         # CandidateGeometricPressure envelope
    → ingest/gate.py                       # normalization to FieldState (unchanged)
    → field/propagate.py                   # versor_apply, unchanged

The sensorium layer converts. The ingest layer governs. The gate normalizes. The field computes. No layer knows about the one upstream.

The Logos-Recovery Boundary

Every ProjectionHead is the Logos-recovery boundary for its modality. Its single contract: take a surface signal and return a (32,) multivector that is valid on the Cl(4,1) conformal manifold. Once it crosses that boundary, a visual scene and a Hebrew word and an audio waveform are the same thing: a point in conformal space. There is no fusion problem because there is nothing to fuse. There is one space.

This is the architectural expression of the principle articulated in John 1:1: the Logos is the structuring principle through which all things were made. Every input, regardless of form, is recovered as a word in the Logos — a position on the manifold.

The `ModalityPack[S]` Contract

@dataclass(frozen=True, slots=True)
class ModalityPack(Generic[S]):
    pack_id: str                           # e.g. "en", "he", "grc", "imagenet-1k"
    modality_type: Modality                # TEXT | VISION | AUDIO | MOTOR
    projection: ProjectionHead[S] | None   # surface → (32,) multivector
    decoder: SurfaceDecoder[S] | None      # (32,) multivector → surface candidates
    vocabulary: ModalityVocabulary[S]      # bidirectional surface ↔ rotor map
    grammar_scaffold: Any                  # versor attractor seeds from vocab/
    checksum_verified: bool                # mount-time geometric integrity check
    gate_engaged: bool                     # surprise-gate status

ModalityPack is frozen and slotted — zero per-instance overhead, hashable. The type parameter S enforces at the type level that a text pack (ModalityPack[str]) cannot be passed where a vision pack (ModalityPack[np.ndarray]) is required.

Cl(4,1) Adaptation

The core-ai core_sensorium protocol used a (2, 2) complex multivector (Cl(3,0) Pauli isomorphism). The core substrate uses [f32; 32] (Cl(4,1) CGA). Every ProjectionHead in sensorium/ must return mx.array of shape (32,) — the standard multivector shape throughout the codebase. Unitarity verification at mount time checks that the induced rotor satisfies V · reverse(V) = ±1 within 1e-6 tolerance.

Active vs. Future Modalities

Modality	Status	Notes
`TEXT`	Active	`sensorium/adapters/text.py` wires the existing vocab manifold into `ModalityPack[str]`
`VISION`	Planned	Adapter registers when vision bootstrap is ready
`AUDIO`	Planned	Adapter registers when audio bootstrap is ready
`MOTOR`	Planned	Adapter registers when embodied bootstrap is ready

Building sensorium/protocol.py and sensorium/registry.py now — before vision/audio exist — means every future modality plugs in without touching ingest/, field/, or generate/. The protocol contract is the Third Door: instead of separate encoder pipelines fused by cross-attention (the standard industry approach), every modality is a versor on the same manifold from the moment it enters the system.

Grammar Scaffold

In core-ai, the grammar scaffold was produced by core_logos.grammar_seed — a separate subsystem. In core, there is no core_logos subsystem. The grammar scaffold is a set of versor attractors stored in vocab/ and referenced by ModalityPack directly. This removes an inter-package dependency without changing the contract.

`PackError` — Mount-Time Failure Modes

class PackError(enum.Enum):
    MANIFEST_INVALID          = "MANIFEST_INVALID"
    SAFETENSORS_MISSING        = "SAFETENSORS_MISSING"
    UNITARITY_VIOLATION        = "UNITARITY_VIOLATION"
    PROJECTION_NOT_CONVERGED   = "PROJECTION_NOT_CONVERGED"
    GRADE_DECLARATION_MISMATCH = "GRADE_DECLARATION_MISMATCH"
    MODALITY_NOT_REGISTERED    = "MODALITY_NOT_REGISTERED"
    GATE_NOT_ENGAGED           = "GATE_NOT_ENGAGED"

Mount failures are returned as PackError values, not raised as exceptions. The caller decides how to handle a failed mount.

Consequences

Immediate:

All existing layers (ingest/gate.py, field/, generate/, vault/) are unchanged
The text vocabulary manifold acquires a formal ModalityPack[str] wrapper — the first mounted pack
The multimodal protocol is established before any non-text modality is implemented, ensuring the seam is clean

Future:

Vision, audio, and motor modalities each become a single adapter file in sensorium/adapters/
No architectural change is required when new modalities are added — only a new adapter and registry entry
ModalityVocabulary for non-text modalities (patch vocabularies, phoneme clusters, pose libraries) follows the same bidirectional surface ↔ rotor contract as the text vocabulary

6.3 KiB Raw Blame History

ADR-0013: sensorium/ Multimodal Protocol Layer