diff --git a/docs/decisions/ADR-0181-audio-compiler-delta-crdt.md b/docs/decisions/ADR-0181-audio-compiler-delta-crdt.md new file mode 100644 index 00000000..85eccf3c --- /dev/null +++ b/docs/decisions/ADR-0181-audio-compiler-delta-crdt.md @@ -0,0 +1,254 @@ +# ADR-0181: CORE-native Audio Compiler over the Delta-CRDT Substrate + +**Status:** Proposed +**Date:** 2026-05-29 +**Authors:** Joshua M. Shay, Core R&D Engine +**Domains:** `sensorium/audio/`, `sensorium/adapters/audio.py`, `packs/audio/`, `core-rs/src/vault.rs` (read-only contract), `evals/audio_sensorium/` +**Depends on:** ADR-0013 (Sensorium Multimodal Protocol), ADR-0180 (Delta-CRDT Sharded Substrate) +**Companion docs:** [audio-compiler-spec.md](../plans/audio-compiler-spec.md), [audio-compiler-eval-plan.md](../plans/audio-compiler-eval-plan.md) + +--- + +## 1. Context & Problem Statement + +CORE's sensory boundary is already lawful and text-only-in-practice. `sensorium/protocol.py` +fixes the contract: + +- `ProjectionHead.project(S) -> (32,) float32` is the **Logos-recovery boundary** + (`CL41_DIM = 32`). +- `Modality.AUDIO` already exists in the enum but has no adapter + (`sensorium/adapters/` ships `text.py` only). +- `ModalityPack` enforces gate/checksum invariants at construction; `ModalityRegistry.mount()` + runs the unitarity check and `project()` refuses a closed gate. + +ADR-0013 (Accepted) requires every new modality to cross this boundary *before* it reaches +`ingest/gate.py`, and forbids touching `ingest/`, `field/`, `generate/`, `vault/`, `vocab/` +to add one. Audio must therefore **arrive already compiled** into a `(32,)` Cl(4,1) multivector. + +ADR-0180 (Proposed) introduces the **Delta-CRDT sharded substrate** precisely to absorb the +continuous, high-density streams that audio and vision produce — naming AUDIO explicitly — and +flags one hard constraint we must satisfy (§1.5.2): + +> The semilattice claim holds **only** at the `vault/store` layer — not at `versor_apply` +> and not at `compute_trace_hash`. … Any operation upstream of `vault/store` that the +> substrate parallelizes must either (a) be proven order-invariant on its inputs, or +> (b) carry an explicit serialization barrier. + +The problem this ADR solves: **how does audio enter CORE as a lawful, deterministic, +replayable modality that is also a well-behaved Delta-CRDT delta producer** — without +violating the no-core-mutation rule, the exact-recall rule, or ADR-0180's order-invariance +obligation? + +The wrong answer is an embedding bridge (CLAP/EnCodec/Whisper as substrate). It produces an +opaque latent that cannot be checksummed, cannot be replayed bit-for-bit, and cannot supply +the content-addressed merge key ADR-0180 §1.5.3 demands. The right answer is a **compiler**. + +## 2. Decision + +We will build `audio_core_v1` as a **deterministic auditory compiler** under `sensorium/audio/`, +lowering canonical waveform input through a typed `AudioIR` into a `(32,)` float32 Cl(4,1) +versor, and we will make its **chunk boundary the Delta-CRDT delta boundary**. Learned models +(Whisper, NeMo, CLAP, EnCodec) are admitted only as subordinate teacher/shadow lanes, never as +the substrate. + +The compilation pipeline (detailed in the spec) is: + +```text +waveform / live stream + → canonicalizer (mono, fixed sample rate, source+canonical sha256) + → frame grid (20 ms window / 10 ms hop) + → acoustic lexer (energy, voicing, onset, pitch candidates, spectral bands, pauses) + → typed AudioIR (speech/pause spans, prosody arcs, turn/overlap events, non-speech atoms) + → operator registry (pack-local blade aliases + quantized theta rules) + → rotor lowering (elliptic bivector rotors only in v1) + → versor composition (geometric_product → unitize_versor → versor_condition < 1e-6) + → (32,) float32 == one AudioCompilationUnit +``` + +Normalization sites stay inside CLAUDE.md's allowlist: quantization and FIR resampling live in +the audio pack/compiler construction boundary, `unitize_versor` is algebra-owned, and **no +hot-path drift repair is added**. CGA null vectors are preserved as null vectors. + +### 2.1 The optimal mapping to Delta-CRDT (the load-bearing decision) + +ADR-0180 §1.5.2 gives audio a binary choice for everything the substrate parallelizes: +**prove order-invariance, or carry a serialization barrier.** The audio compiler answers +*both*, at two different granularities: + +| Granularity | Operation | CRDT treatment | Why | +|---|---|---|---| +| **Within a chunk** | `compile_events` (rotor chaining via `geometric_product`) | **Serialization barrier** | The sandwich product is non-commutative (ADR-0180 §1.5.2 row 3). In-chunk composition runs serially, single-threaded, in canonical event order, inside one thread-local arena. | +| **Across chunks / streams** | merge of `AudioCompilationUnit`s into the Vault | **Order-invariant delta** | Each unit is `(versor, provenance)` written at the `vault/store` layer — the *only* semilattice-eligible layer (ADR-0180 §1.5.2 row 5). Merge is commutative, associative, idempotent. | + +This is the synthesis ADR-0180 was waiting for: **the audio chunk boundary is the natural +delta boundary** named in ADR-0180 §2.2 ("at semantic chunk boundaries"). The compiler does the +order-sensitive work behind the barrier and emits only order-invariant deltas across it. The +substrate never parallelizes a non-commutative operation. + +### 2.2 Content-addressed merge key from the checksum chain + +ADR-0180 §1.5.3 requires the trace-hash reduction to consume vault state in a +**content-addressed order**, not wall-clock arrival order, or `hash(Sequential_Ingest) == +hash(Concurrent_CRDT_Ingest)` cannot hold. The audio compiler already produces exactly this key +as a byproduct of its layered checksum chain: + +```text +source_sha256 → canonical_sha256 → token_stream_sha256 → ir_sha256 + → pack_manifest_sha256 → projection_sha256 +``` + +The **merge key for an audio delta is `(canonical_sha256, ir_sha256, projection_sha256)`**. +Consequences: + +- **Idempotence is structural, not asserted.** Identical canonical bytes under an identical + pack produce an identical key; the CRDT join deduplicates them. This satisfies the + idempotence leg of the join semilattice (ADR-0180 §2.2) by construction. +- **The content-addressed sort is free.** The merge kernel orders pending audio deltas by this + key. No re-sort pass is needed at hash time for the audio portion of the path. +- **`projection_sha256` is computed behind the serialization barrier** (§2.1), i.e. on the + serialized in-chunk composition — satisfying ADR-0180 §1.5.3 point 3 (upstream-of-Vault + hashes must be computed on the serialized portion). + +### 2.3 Physical sharding mirrors the audio domain's natural concurrency + +ADR-0180 §2.1 assigns each active modality adapter a thread-local arena and forbids +`sensorium/adapters/*` from writing the global `epistemic_state` directly. For audio this is not +just a mechanical convenience — it is **semantically aligned**: overlapping speakers, +interruptions, and turn boundaries (the `B_OVERLAP`, `B_TURN` atoms in the operator table) are +*literally* concurrent auditory streams. Each concurrent stream gets its own arena; the +`AuditoryEvent` timing (`start_hop`/`end_hop`) is preserved inside each unit so the merge can +reconstruct overlap relationships without a global lock during ingestion. The substrate's +physical sharding is therefore a faithful image of the audio source's structure, not an +impedance mismatch. + +### 2.4 Eventual-consistency window is safe for audio + +ADR-0180 §3.2 documents a sub-50ms window where a delta sits in the local arena before merge. +Audio is robust to this because: + +- `ProjectionHead.project` is **pure on the signal** (ADR-0180 T-4) — the audio compiler never + reads cross-modal or global state during compilation, so a delayed merge cannot change what it + produces. +- Each unit retains its own intra-stream event timing, so cross-modal resonance re-anchors on + merged state after the window closes; recall remains **exact byte-for-byte once merged** + (ADR-0180 §1.5.5), never approximate. + +### 2.5 Gate-closed by default + +`audio_core_v1` mounts with `gate_engaged = false` until the eval gates in the companion plan +pass. A closed gate makes `ModalityRegistry.project("audio_core_v1", …)` raise — audio cannot +contribute deltas to any arena until determinism, checksum, unitarity, and mount-validation +gates are green. This reuses the existing registry enforcement; no new gating machinery is +added. + +## 3. Consequences + +### 3.1 Positive + +- **First concrete exerciser of ADR-0180.** Audio is the first continuous modality to land on + the Delta-CRDT substrate, converting ADR-0180's proof obligation `hash(Sequential) == + hash(Concurrent)` from abstract to testable against real delta producers. +- **Order-invariance is proven, not hoped.** §2.1/§2.2 give a concrete serialization barrier and + a content-addressed key, closing ADR-0180 §1.5.2's open constraint for the audio path. +- **No core mutation.** Everything new lives under `sensorium/audio/`, `packs/audio/`, + `tests/`, `evals/`. `ingest/`, `field/`, `generate/`, `vault/`, `vocab/` are untouched + (ADR-0013), and `anti_unifier`/`carrier` need no changes (ADR-0180 §3.1). +- **Trace hygiene composes.** Turn traces record `(canonical_sha256, ir_sha256, + projection_sha256)` and pack IDs — never raw PCM (ADR-0180 §1.5.5). + +### 3.2 Negative / Risks + +- **Semantic underreach (v1).** The compiler captures prosody, turn dynamics, and salient + non-speech events better than lexical content. Acceptable: transcript teachers backfill + lexical evidence while the substrate stays native. +- **Pack ontology drift.** If blade aliases or operator gains change freely, projections become + incomparable across versions. Mitigation: the pack is versioned, checksummed, and gate-closed; + `basis_version` is part of the merge key's pack-manifest leg. +- **Over/under-quantization.** Too-coarse bins flatten meaning; too-fine bins make replay + brittle. Mitigation: quantization regime is frozen in the manifest and covered by + cross-platform stability gates. +- **Streaming seam artifacts.** Stateful resampling/pitch/overlap must preserve continuity + across chunk boundaries. Deferred to the streaming phase; v1 is offline/whole-buffer. +- **Licensing contamination.** openSMILE's OSS build is non-commercial; it is a reference oracle + only, never a runtime dependency. + +## 4. Execution Plan & Proof Obligations + +### 4.1 PR stack (additive, doctrine-first) + +| PR | Scope | Gate | +|---|---|---| +| **PR-1 (this)** | ADR-0181 + compiler spec + eval plan (docs only) | review | +| **PR-2** | Deterministic substrate: `sensorium/audio/{types,canonical,checksum,resample,frames,lexer,parser,operators,compiler,trace}.py` | determinism + versor unit tests | +| **PR-3** | Pack artifacts `packs/audio/audio_core_v1/*` + `AudioProjectionHead` adapter + mount tests | mount/gate/checksum gates | +| **PR-4** | `evals/audio_sensorium/` fixtures, expected IR, expected projection hashes | full eval-gate table | +| **PR-5** | Delta-CRDT wiring: `AudioCompilationUnit` → thread-local arena → merge key, behind ADR-0180's substrate | sequential==concurrent trace-hash proof | +| **PR-6** | Teacher/shadow lanes (Whisper/NeMo/CLAP/EnCodec) behind optional extras | teachers admitted only as typed hints | + +PR-5 must not start until ADR-0180's own §1.5.4 obligations (T-1…T-4) are green on `main` — +the audio delta path rides on them. + +### 4.2 Audio-specific proof obligations (extend ADR-0180 §1.5.4) + +Per CLAUDE.md §Schema-Defined Proof Obligations, each must be able to **fail loudly** under the +violation it names: + +- **A-1 (determinism / ADR-0180 T-4 analog).** Same canonical bytes + same pack ⇒ byte-identical + `(32,)`, across repeated calls, threads, and processes. Fails if any non-determinism (dict + ordering, unpinned FIR, float reduction order) leaks in. +- **A-2 (set-equality of merges / ADR-0180 T-1 analog).** A set of `AudioCompilationUnit`s folds + to the same Vault state regardless of arena flush order (permutation invariance). Fails if a + delta's contribution is order-sensitive at the merge layer. +- **A-3 (content-addressed key / ADR-0180 T-2 analog).** Trace-hash over audio deltas is + invariant under set-equal Vault states when keyed by `(canonical_sha256, ir_sha256, + projection_sha256)`. Fails if the reduction consumes deltas in arrival order. +- **A-4 (serialization barrier).** In-chunk `compile_events` is asserted order-sensitive + (negative test): swapping two events in canonical order changes the versor. This guards the + barrier in §2.1 from silently becoming commutative and masking a real ordering bug. +- **A-5 (versor condition).** Every emitted unit satisfies `versor_condition(v) < 1e-6`; the + threshold is never weakened to pass. +- **A-6 (trace hygiene).** No raw waveform bytes appear in any `TurnEvent`/Vault record; only + the three hashes + pack IDs + optional teacher provenance. + +### 4.3 The strict compilation invariant + +```text +same canonical audio bytes + + same compiler version + + same pack manifest (incl. basis_version) + + same operator registry += same AudioIR += same versor += same projection hash += same CRDT merge key += identical post-merge Vault contribution (idempotent under re-ingest) +``` + +The final clause is the ADR-0181 addition to the PDF's original invariant: determinism is not +only replayability, it is **CRDT idempotence** — the property that makes audio safe to shard. + +## 5. Alternatives Considered + +- **Transcript-first cascade (Whisper as substrate).** Chunked, text-intermediate; discards the + auditory structure (the "No." vs "No?" vs whispered "no" distinction) and produces no + checksummable, content-addressed key. Rejected as substrate; retained as teacher. +- **Embedding-first projector (CLAP).** Fast but opaque; cannot be replayed bit-for-bit and + cannot supply ADR-0180 §1.5.3's content-addressed merge key. Rejected as substrate. +- **Codec-token front end (EnCodec / Moshi-like).** Strategically interesting for future + full-duplex output; poor first substrate for an epistemically explicit engine. Deferred to a + shadow/output lane. +- **Audio as a downstream cognition mutation.** Violates ADR-0013's no-core-mutation rule and + ADR-0180's "adapters never write global state directly" rule. Rejected. + +## 6. Cross-References + +- ADR-0013 — projection boundary; no-core-mutation constraint. +- ADR-0180 — Delta-CRDT substrate; §1.5.2 order-invariance constraint (closed here for audio), + §1.5.3 content-addressed merge key, §1.5.4 T-1…T-4 (audio analogs in §4.2), §1.5.5 trace + hygiene & no hidden background execution. +- ADR-0054 — Vault recall indexing/batching; the read-side contract the merged audio deltas must + preserve (exact CGA recall). +- CLAUDE.md §Normalization Rules — quantization/FIR confined to pack/compiler construction; + `unitize_versor` algebra-owned; no hot-path repair. +- `sensorium/protocol.py`, `sensorium/registry.py` — the `ProjectionHead`/`ModalityPack`/ + `ModalityRegistry` contracts this ADR implements an audio instance of. diff --git a/docs/plans/audio-compiler-eval-plan.md b/docs/plans/audio-compiler-eval-plan.md new file mode 100644 index 00000000..48fc0ad1 --- /dev/null +++ b/docs/plans/audio-compiler-eval-plan.md @@ -0,0 +1,144 @@ +# Audio Compiler Eval Plan — `audio_core_v1` + +**Companion to:** [ADR-0181](../decisions/ADR-0181-audio-compiler-delta-crdt.md), +[audio-compiler-spec.md](./audio-compiler-spec.md) +**Status:** Proposed (PR-1 docs) + +This plan defines the seeding corpus, the acceptance gates that lift `audio_core_v1` from +gate-closed to gate-engaged, the Delta-CRDT proof obligations, and the teacher-migration policy. + +--- + +## 1. Seeding corpus — auditory atoms, not transcripts + +Small, curated, checksum-locked. Four tiers: + +- **Tier A — speech & turn atoms:** silence, short/long pause, voiced/unvoiced speech, onset, + offset, interruption, overlap, rising/falling contour, emphatic energy. +- **Tier B — prosody & affect:** calm, urgent, uncertain, whispered, shouted, sorrowful, joyful, + irritated, sarcastic-like contours where reliably labelable. +- **Tier C — non-speech events:** laughter, sigh, cry, alarm, knock, impact, keyboard, water, + traffic, animal call, music bed, broadband noise. +- **Tier D — alignment anchors:** spoken phrase, timestamp span, transcript hypothesis, + speaker/channel metadata, linked text surface (only when alignment is trustworthy). + +Tier D is auxiliary; lexical semantics is a later enrichment step. Synthetic fixtures (sine +bursts, chirps, impulses, periodic voiced surrogates, silence, controlled overlaps) drive +first-pass determinism; checksum-locked real fixtures cover what synthesis does poorly +(laughter, sigh, whisper, interruption, yes/no rising-pitch questions). + +## 2. Acceptance gates (gate-engaged criteria) + +| Gate | Pass criterion | +|---|---| +| Projection shape | exactly `(32,)` | +| Projection dtype | exactly `float32` | +| Compiler replay | bit-identical on same platform/build | +| Cross-platform stability | equal after quantization, within declared numeric tolerance | +| `versor_condition` | `< 1e-6` (never weakened) | +| Canonical checksum stability | 100% on fixture corpus | +| Gate closure | projection blocked when `gate_engaged = false` | +| Mount validation | bad checksum or bad unitarity blocks pack mount | +| Trace hygiene | no raw waveform bytes in any turn trace | +| IR replay | `AudioIR -> versor` replays identically from stored IR | + +These mirror the existing modality-test posture (`sensorium/registry.py` mount/gate enforcement, +`ModalityPack.__post_init__` invariants). + +## 3. Delta-CRDT proof obligations (ADR-0181 §4.2) + +Each must be able to **fail loudly** under the violation it names (CLAUDE.md §Schema-Defined +Proof Obligations — no decorative tests): + +| ID | Obligation | Fails if… | ADR-0180 analog | +|---|---|---|---| +| **A-1** | Determinism: same bytes + pack ⇒ byte-identical `(32,)` across calls/threads/processes | dict ordering, unpinned FIR, or float reduction order leaks in | T-4 | +| **A-2** | Set-equality of merges: a set of units folds to the same Vault state for any arena flush permutation | a delta's contribution is order-sensitive at the merge layer | T-1 | +| **A-3** | Content-addressed trace-hash: invariant under set-equal Vault states when keyed by `(canonical, ir, projection)` sha | the reduction consumes deltas in arrival order | T-2 | +| **A-4** | Serialization barrier: in-chunk `compile_events` is order-sensitive (negative test) | swapping two canonical-order events fails to change the versor | T-3 | +| **A-5** | `versor_condition < 1e-6` on every emitted unit | the threshold is weakened to pass | — | +| **A-6** | Trace hygiene: no PCM in any `TurnEvent`/Vault record | raw waveform leaks into a delta's provenance | §1.5.5 | + +### 3.1 The sequential==concurrent proof (PR-5 acceptance) + +The load-bearing test for the CRDT mapping: + +```text +ingest fixtures [c1, c2, c3] sequentially → vault_seq, trace_hash_seq +ingest the same fixtures across N arenas merged → vault_conc, trace_hash_conc +assert set(vault_seq) == set(vault_conc) # A-2 +assert trace_hash_seq == trace_hash_conc # A-3 +assert re-ingesting any fixture is a no-op on the vault # idempotence (ADR-0181 §4.3) +``` + +This is the audio instance of ADR-0180 §4.3's `hash(Sequential_Ingest) == +hash(Concurrent_CRDT_Ingest)` and must pass before PR-5 merges. + +### 3.2 Pytest skeleton + +```python +def test_audio_projection_is_deterministic(audio_fixture, audio_pack): # A-1 + v1 = audio_pack.projection.project(audio_fixture) + v2 = audio_pack.projection.project(audio_fixture) + assert v1.shape == (32,) + assert v1.dtype.name == "float32" + assert np.array_equal(v1, v2) + +def test_audio_pack_gate_blocks_projection(audio_pack_closed, audio_fixture): # gate closure + with pytest.raises(Exception): + _ = audio_pack_closed.projection.project(audio_fixture) + +def test_audio_ir_replay_matches_original(audio_fixture, compiler): # IR replay + unit = compiler.compile(audio_fixture) + replay = compiler.compile_ir(unit.audio_ir) + assert np.array_equal(unit.versor, replay.versor) + assert unit.ir_sha256 == replay.ir_sha256 + +def test_chunk_composition_is_order_sensitive(two_events, compiler): # A-4 barrier + a = compiler.compile_events([two_events[0], two_events[1]]) + b = compiler.compile_events([two_events[1], two_events[0]]) + assert not np.array_equal(a, b) + +def test_merge_is_permutation_invariant(units): # A-2 + import itertools, random + states = {fold_into_vault(p) for p in itertools.islice(_perms(units), 8)} + assert len(states) == 1 +``` + +## 4. Teacher / shadow lane policy (PR-6) + +Teachers label or align; they **never** define the substrate and **never** fold embeddings into +the main versor path. They are admitted only through typed, versioned, checksummed hints stored +as `content_anchors` / `evidence_ids` in the IR. + +| Source | Best role in CORE | Why not the substrate | +|---|---|---| +| **Whisper** | offline transcript evidence, weak lexical labels, language ID | chunked, text-intermediate; loses native auditory structure | +| **NeMo Parakeet** | timestamp/alignment teacher, live evidence lane | ASR, not a CORE-native auditory compiler | +| **NeMo Canary** | streaming multilingual transcript/translation evidence | useful evidence, wrong primary ontology | +| **CLAP** | coarse sound-event labels, audio-text alignment prototypes | embedding model; opaque latent violates the design goal | +| **EnCodec** | reconstruction shadow lane, transport, future speech-to-speech output | codec tokens ≠ lawful typed auditory operators | +| **Moshi** | latency / full-duplex reference target | codec-token speech modeling, not deterministic IR→versor | +| **openSMILE / eGeMAPS** | reference feature catalog / offline oracle | OSS build is **non-commercial**; reference only, never a runtime dep unless licensing cleared | + +Migration policy, verbatim: + +```text +Use teachers to label or align. +Never let teachers define the substrate. +Never fold teacher embeddings directly into the main versor path. +Only admit teacher outputs through typed, versioned, checksumed hints. +``` + +## 5. Phased sequence (priority order, no calendar) + +1. **Doctrine** — ADR + spec + eval plan locked (PR-1, this). +2. **Deterministic substrate** — canonicalizer, checksums, resample, lexer, parser, operators, + compiler (PR-2). +3. **Governance** — pack artifacts, adapter, mount/gate/checksum tests (PR-3). +4. **Evaluation** — fixtures, expected IR, expected projection hashes, gate table (PR-4). +5. **Delta-CRDT wiring** — arena + merge key + sequential==concurrent proof (PR-5), gated on + ADR-0180 §1.5.4 (T-1…T-4) green on `main`. +6. **Auxiliary lanes** — Whisper/NeMo/CLAP/EnCodec teachers behind optional extras (PR-6). +7. **Streaming** — stateful incremental compiler preserving continuity across chunk seams + (later; v1 is offline/whole-buffer). diff --git a/docs/plans/audio-compiler-spec.md b/docs/plans/audio-compiler-spec.md new file mode 100644 index 00000000..fe64e807 --- /dev/null +++ b/docs/plans/audio-compiler-spec.md @@ -0,0 +1,321 @@ +# Audio Compiler Spec — `audio_core_v1` + +**Companion to:** [ADR-0181](../decisions/ADR-0181-audio-compiler-delta-crdt.md) +**Status:** Proposed (PR-1 docs) +**Scope:** the deterministic substrate (PR-2/PR-3) and its Delta-CRDT delta interface (PR-5). + +This spec fixes the typed intermediate representation, the operator/manifest format, the numeric +determinism rules, and the `AudioCompilationUnit` → Delta-CRDT delta contract. It is +implementation-facing; the *why* lives in ADR-0181, the *acceptance* lives in the eval plan. + +--- + +## 1. Two-clock architecture + +A low-level **acoustic clock** measures signal facts; a higher-level **auditory grammar clock** +emits typed events. The primary path is fully deterministic; learned systems are confined to +auxiliary evidence lanes (PR-6). + +```mermaid +flowchart LR + A[Waveform bytes / live stream] --> B[Canonicalizer
mono + fixed SR + checksums] + B --> C[Frame grid
20 ms window / 10 ms hop] + C --> D[Acoustic lexer
energy, voicing, onset,
pitch candidates, spectral bands, pauses] + D --> E[Typed AudioIR parser
speech/pause spans, prosody arcs,
turn/overlap events, non-speech atoms, anchors] + E --> F[Canonical ordering
quantization + stable serialization] + F --> G[Operator registry
pack manifest + blade aliases + theta rules] + G --> H[Rotor lowering] + H --> I[Versor composition
unitize_versor + versor_condition] + I --> J["(32,) float32 — one AudioCompilationUnit"] + E --> K[Audio evidence trace
hashes, teacher provenance, pack IDs] + J --> L[Thread-local arena
ADR-0180 §2.1] + L --> M[Semilattice merge
keyed by content-addressed sha] +``` + +## 2. Typed AudioIR + +The IR is built from **typed spans and events**, never from raw frames or mel bins. Transcript +anchors may exist only as auxiliary content hypotheses, never as the sole meaning of the audio. + +```python +from __future__ import annotations +from dataclasses import dataclass +from typing import Literal +import numpy as np + + +@dataclass(frozen=True, slots=True) +class AudioSignal: + samples: np.ndarray # canonical mono float32 + sample_rate: int # 24_000 + start_ms: int + end_ms: int + source_sha256: str + canonical_sha256: str + + +@dataclass(frozen=True, slots=True) +class PitchCandidate: + cents_q: int # quantized cents + prob_q: int # 0..255 + + +@dataclass(frozen=True, slots=True) +class AudioToken: + kind: Literal[ + "silence", "voiced", "unvoiced", "onset", + "energy_bin", "pitch_candidates", "spectral_bin", + ] + start_hop: int + end_hop: int + value_q: tuple[int, ...] # canonical quantized payload + + +@dataclass(frozen=True, slots=True) +class AuditoryEvent: + event_type: str + start_hop: int + end_hop: int + attrs: tuple[tuple[str, int | str], ...] + evidence_ids: tuple[str, ...] + + +@dataclass(frozen=True, slots=True) +class AudioIR: + speech_spans: tuple[AuditoryEvent, ...] + pause_spans: tuple[AuditoryEvent, ...] + prosody_arcs: tuple[AuditoryEvent, ...] + turn_events: tuple[AuditoryEvent, ...] + non_speech_events: tuple[AuditoryEvent, ...] + content_anchors: tuple[AuditoryEvent, ...] + ir_sha256: str +``` + +### 2.1 The compilation unit (the CRDT delta) + +```python +@dataclass(frozen=True, slots=True) +class AudioCompilationUnit: + canonical_sha256: str + ir_sha256: str + pack_id: str + pack_manifest_sha256: str + projection_sha256: str + versor: np.ndarray # (32,) float32 + versor_condition: float + + @property + def merge_key(self) -> tuple[str, str, str]: + # ADR-0181 §2.2 — content-addressed CRDT merge / dedup key. + return (self.canonical_sha256, self.ir_sha256, self.projection_sha256) +``` + +`AudioCompilationUnit` is the single object the audio adapter writes into its thread-local arena +(ADR-0180 §2.1). It carries no PCM (ADR-0181 §3.1 / ADR-0180 §1.5.5). + +## 3. Canonical signal formation + +- Internal processing: **mono, 24 kHz, float**; original-source bytes preserved separately for + provenance; a derived 16 kHz stream is produced **only** for teacher ASR (PR-6). +- Resampling: **pinned polyphase FIR** (SciPy `resample_poly` semantics — zero-phase, + odd-length symmetric filter). The FIR coefficients are generated **once**, stored as a pack + artifact (`resample_fir_v1.npy`), and checksummed in the manifest. The runtime never relies on + library defaults. + +## 4. Acoustic lexer + +Operates on **measured facts**, not semantic guesses. Default frame 20 ms / hop 10 ms. Each +frame yields quantized descriptors: RMS/log-energy bin, voiced/unvoiced flag, candidate F0 set +(pYIN-style: multiple candidates with probabilities **before** Viterbi smoothing), onset +strength bin, coarse spectral centroid/tilt bin, zero-crossing regime, pause classification. + +## 5. Parser → typed events + +Promotes lexer output into typed spans/events. Preserves the distinction between "No.", "No?", +shouted "No!", whispered "no", and silent hesitation. Non-speech atoms (laughter, alarm, impact, +music, broadband noise) are first-class; "chaotic noise" is the fallback only when a more +specific parse is impossible. + +## 6. Operator registry (pack-local blade aliases) + +Because the `(32,)` boundary is fixed but no canonical *semantic* blade map is exposed, v1 uses +**pack-local, versioned, checksummed blade aliases**. v1 uses **elliptic bivector operators +only** (square = −1), so every rotor uses the numerically well-behaved +`R = cos(θ/2) + B·sin(θ/2)`. Hyperbolic/boost-like operators are deferred. + +| Auditory atom family | Measured source | Alias | Default blade index | Theta rule | +|---|---|---|---|---| +| Speech present | voiced frames / harmonic ratio | `B_SPEECH` | 6 | `q(base + g1·voiced_ratio_q)` | +| Short pause | pause duration | `B_PAUSE_SHORT` | 7 | `q(base + g2·dur_hops)` | +| Long pause | pause duration | `B_PAUSE_LONG` | 8 | `q(base + g3·dur_hops)` | +| Rising final contour | final F0 slope | `B_PITCH_RISE` | 9 | `q(base + g4·slope_q)` | +| Falling final contour | final F0 slope | `B_PITCH_FALL` | 10 | `q(base + g5·abs(slope_q))` | +| Emphasis / force | energy delta | `B_EMPHASIS` | 11 | `q(base + g6·delta_db_q)` | +| Hesitation / uncertainty | filled pause + low-conf contour | `B_HESITATION` | 12 | `q(base + g7·hesitation_q)` | +| Turn boundary | silent gap + local reset | `B_TURN` | 13 | `q(base + g8·boundary_q)` | +| Overlap / interruption | simultaneous speech / abrupt cut | `B_OVERLAP` | 14 | `q(base + g9·overlap_q)` | +| Alert-like event | salience / alarm morphology | `B_ALERT` | 15 | `q(base + g10·salience_q)` | +| Laughter | periodic burst pattern | `B_LAUGH` | 16 | `q(base + g11·laugh_q)` | +| Cry / distress | voicing + modulation profile | `B_DISTRESS` | 17 | `q(base + g12·distress_q)` | +| Music / tonal bed | stable harmonic bed | `B_MUSIC` | 18 | `q(base + g13·tonal_q)` | +| Chaotic broadband noise | flat/noisy spectrum | `B_NOISE` | 19 | `q(base + g14·noise_q)` | + +Indices are **reasonable defaults, not metaphysical claims** about Cl(4,1). The contract is that +the mapping is explicit, versioned, checksummed, and frozen in the manifest. `B_OVERLAP` and +`B_TURN` are the atoms that motivate per-stream arenas in ADR-0181 §2.3. + +### 6.1 Minimal manifest (`packs/audio/audio_core_v1/manifest.toml`) + +```toml +pack_id = "audio_core_v1" +modality = "audio" +cl41_dim = 32 +compiler_version = "0.1.0" +basis_version = "audio-basis-v1" + +[canonical] +sample_rate = 24000 +channels = 1 +frame_ms = 20 +hop_ms = 10 +output_dtype = "float32" +internal_dtype = "float64" + +[resampling] +algorithm = "polyphase_fir" +fir_path = "resample_fir_v1.npy" +fir_sha256 = "sha256:REPLACE_ME" +padtype = "constant" +cval = 0.0 + +[gating] +gate_engaged = false +checksum_verified = false +versor_condition_max = 1.0e-6 + +[ordering] +event_precedence = ["channel", "pause", "speech", "prosody", "turn", "non_speech", "content_anchor"] +``` + +### 6.2 Operator row (`operators.jsonl`) + +```json +{ + "operator_id": "audio.prosody.question_contour.v1", + "event_type": "prosody.question_contour", + "blade_alias": "B_PITCH_RISE", + "blade_index": 9, + "rotor_kind": "elliptic", + "base_theta_q": 64, + "gain_rules": {"slope_q": 3, "final_energy_q": 1, "confidence_q": 1}, + "theta_clip_q": 384, + "version": "1" +} +``` + +## 7. Numeric determinism + +Rule: **quantize before semantics, normalize after composition.** Raw float measurements are too +unstable to hash. Quantization regime (frozen in manifest): boundaries in hop units, log energy +in 1 dB bins, F0 in 25-cent bins, pitch slope in coarse cents-per-100 ms bins, spectral shape in +fixed ordinal bins, all confidences in uint8. After quantization, compute in float64, compose +sparse rotors in canonical order, call algebra-owned `unitize_versor`, cast to float32 **only** +at the output boundary. + +```python +import math +import numpy as np + +EPS = 1e-12 + +def quantize_theta(theta: float, step: float = 1.0 / 1024.0) -> float: + return round(theta / step) * step + +def build_elliptic_rotor(blade_index: int, theta: float) -> np.ndarray: + out = np.zeros(32, dtype=np.float64) + half = quantize_theta(theta) / 2.0 + out[0] = math.cos(half) + out[blade_index] = math.sin(half) + return out + +def compile_events(events, registry, geometric_product, unitize_versor, versor_condition): + # SERIALIZATION BARRIER (ADR-0181 §2.1): in-chunk composition is order-sensitive, + # single-threaded, canonical order. The substrate never parallelizes this loop. + v = np.zeros(32, dtype=np.float64) + v[0] = 1.0 + for ev in events: # must already be in canonical order + spec = registry[ev.event_type] + theta = spec.theta_from_event(ev) # deterministic, quantized inputs only + r = build_elliptic_rotor(spec.blade_index, theta) + v = geometric_product(v, r) + v = unitize_versor(v) + if versor_condition(v) >= 1e-6: + raise ValueError("audio compilation failed versor check") + return v.astype(np.float32) +``` + +`geometric_product`, `unitize_versor`, `versor_condition` are imported from `algebra/` — the +audio compiler adds **no** new normalization function. + +## 8. Repo-facing adapter (`sensorium/adapters/audio.py`) + +```python +from __future__ import annotations +from dataclasses import dataclass +import numpy as np + +@dataclass(frozen=True, slots=True) +class AudioProjectionHead: + compiler: "AudioCompiler" + modality = ... # Modality.AUDIO + + @property + def embedding_dim(self) -> int: + return 32 + + def project(self, signal: "AudioSignal") -> np.ndarray: + unit = self.compiler.compile(signal) + out = unit.versor + if out.shape != (32,): + raise ValueError(f"expected (32,), got {out.shape}") + if out.dtype != np.float32: + raise TypeError(f"expected float32, got {out.dtype}") + return out + + def project_batch(self, signals: list["AudioSignal"]) -> np.ndarray: + return np.stack([self.project(s) for s in signals], axis=0) + + def verify_unitarity(self, signal: "AudioSignal") -> bool: + return self.compiler.compile(signal).versor_condition < 1e-6 +``` + +The adapter is thin and pack-governed; it satisfies the `ProjectionHead` protocol in +`sensorium/protocol.py` and is mounted as a `ModalityPack(modality_type=Modality.AUDIO, +gate_engaged=False)` until the eval gates pass. + +## 9. Delta-CRDT delta interface (PR-5) + +The audio adapter **never** writes the global `epistemic_state` (ADR-0180 §2.1). Instead: + +1. `compile()` produces one `AudioCompilationUnit` per canonical chunk (the serialization + barrier of §7 runs here). +2. The unit is written lock-free into the adapter's **thread-local arena**. Concurrent streams + (overlap/interruption) each have their own arena (ADR-0181 §2.3). +3. The **Merge Kernel** (ADR-0180 §2.2, an explicitly-mounted component, not a daemon — ADR-0180 + §1.5.5) folds pending units into the global Vault ordered by `unit.merge_key`. Duplicate keys + are deduplicated (idempotence). +4. The kernel surfaces its pending-delta count in `TurnEvent` for replay evidence (ADR-0180 + §1.5.5). + +The per-chunk Vault contribution is `(versor, provenance)` where provenance = +`{merge_key, pack_id, pack_manifest_sha256}` — content-addressed, no PCM. + +## 10. File plan (PR-2 … PR-6) + +```text +sensorium/audio/{__init__,types,canonical,checksum,resample,frames,lexer,parser,operators,compiler,trace,fixtures,teachers}.py +sensorium/adapters/audio.py +packs/audio/audio_core_v1/{manifest.toml,basis_map.json,operators.jsonl,atoms.jsonl,prototypes.jsonl,resample_fir_v1.npy,checksums.json} +tests/test_audio_{signal,resample,lexer,parser,compiler,pack_manifest,sensorium_mount,trace,crdt_delta}.py +evals/audio_sensorium/{fixtures/*.wav,manifest.json,expected_ir.jsonl,expected_projection_hashes.json} +```