core/docs/plans/audio-compiler-spec.md

Audio Compiler Spec — audio_core_v1

Companion to: ADR-0181 Status: Proposed (PR-1 docs) Scope: the deterministic substrate (PR-2/PR-3) and its Delta-CRDT delta interface (PR-5).

This spec fixes the typed intermediate representation, the operator/manifest format, the numeric determinism rules, and the AudioCompilationUnit → Delta-CRDT delta contract. It is implementation-facing; the why lives in ADR-0181, the acceptance lives in the eval plan.

---

1. Two-clock architecture

A low-level acoustic clock measures signal facts; a higher-level auditory grammar clock emits typed events. The primary path is fully deterministic; learned systems are confined to auxiliary evidence lanes (PR-6).

flowchart LR
  A[Waveform bytes / live stream] --> B[Canonicalizer<br/>mono + fixed SR + checksums]
  B --> C[Frame grid<br/>20 ms window / 10 ms hop]
  C --> D[Acoustic lexer<br/>energy, voicing, onset,<br/>pitch candidates, spectral bands, pauses]
  D --> E[Typed AudioIR parser<br/>speech/pause spans, prosody arcs,<br/>turn/overlap events, non-speech atoms, anchors]
  E --> F[Canonical ordering<br/>quantization + stable serialization]
  F --> G[Operator registry<br/>pack manifest + blade aliases + theta rules]
  G --> H[Rotor lowering]
  H --> I[Versor composition<br/>unitize_versor + versor_condition]
  I --> J["(32,) float32 — one AudioCompilationUnit"]
  E --> K[Audio evidence trace<br/>hashes, teacher provenance, pack IDs]
  J --> L[Thread-local arena<br/>ADR-0180 §2.1]
  L --> M[Semilattice merge<br/>keyed by content-addressed sha]

2. Typed AudioIR

The IR is built from typed spans and events, never from raw frames or mel bins. Transcript anchors may exist only as auxiliary content hypotheses, never as the sole meaning of the audio.

from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
import numpy as np


@dataclass(frozen=True, slots=True)
class AudioSignal:
    samples: np.ndarray        # canonical mono float32
    sample_rate: int           # 24_000
    start_ms: int
    end_ms: int
    source_sha256: str
    canonical_sha256: str


@dataclass(frozen=True, slots=True)
class PitchCandidate:
    cents_q: int               # quantized cents
    prob_q: int                # 0..255


@dataclass(frozen=True, slots=True)
class AudioToken:
    kind: Literal[
        "silence", "voiced", "unvoiced", "onset",
        "energy_bin", "pitch_candidates", "spectral_bin",
    ]
    start_hop: int
    end_hop: int
    value_q: tuple[int, ...]   # canonical quantized payload


@dataclass(frozen=True, slots=True)
class AuditoryEvent:
    event_type: str
    start_hop: int
    end_hop: int
    attrs: tuple[tuple[str, int | str], ...]
    evidence_ids: tuple[str, ...]


@dataclass(frozen=True, slots=True)
class AudioIR:
    speech_spans:     tuple[AuditoryEvent, ...]
    pause_spans:      tuple[AuditoryEvent, ...]
    prosody_arcs:     tuple[AuditoryEvent, ...]
    turn_events:      tuple[AuditoryEvent, ...]
    non_speech_events: tuple[AuditoryEvent, ...]
    content_anchors:  tuple[AuditoryEvent, ...]
    ir_sha256:        str

2.1 The compilation unit (the CRDT delta)

@dataclass(frozen=True, slots=True)
class AudioCompilationUnit:
    canonical_sha256:  str
    ir_sha256:         str
    pack_id:           str
    pack_manifest_sha256: str
    projection_sha256: str
    versor:            np.ndarray   # (32,) float32
    versor_condition:  float

    @property
    def merge_key(self) -> tuple[str, str, str]:
        # ADR-0181 §2.2 — content-addressed CRDT merge / dedup key.
        return (self.canonical_sha256, self.ir_sha256, self.projection_sha256)

AudioCompilationUnit is the single object the audio adapter writes into its thread-local arena (ADR-0180 §2.1). It carries no PCM (ADR-0181 §3.1 / ADR-0180 §1.5.5).

3. Canonical signal formation

• Internal processing: mono, 24 kHz, float; original-source bytes preserved separately for provenance; a derived 16 kHz stream is produced only for teacher ASR (PR-6).
• Resampling: pinned polyphase FIR (SciPy resample_poly semantics — zero-phase, odd-length symmetric filter). The FIR coefficients are generated once, stored as a pack artifact (resample_fir_v1.npy), and checksummed in the manifest. The runtime never relies on library defaults.

4. Acoustic lexer

Operates on measured facts, not semantic guesses. Default frame 20 ms / hop 10 ms. Each frame yields quantized descriptors: RMS/log-energy bin, voiced/unvoiced flag, candidate F0 set (pYIN-style: multiple candidates with probabilities before Viterbi smoothing), onset strength bin, coarse spectral centroid/tilt bin, zero-crossing regime, pause classification.

5. Parser → typed events

Promotes lexer output into typed spans/events. Preserves the distinction between "No.", "No?", shouted "No!", whispered "no", and silent hesitation. Non-speech atoms (laughter, alarm, impact, music, broadband noise) are first-class; "chaotic noise" is the fallback only when a more specific parse is impossible.

6. Operator registry (pack-local blade aliases)

Because the (32,) boundary is fixed but no canonical semantic blade map is exposed, v1 uses pack-local, versioned, checksummed blade aliases. v1 uses elliptic bivector operators only (square = −1), so every rotor uses the numerically well-behaved R = cos(θ/2) + B·sin(θ/2). Hyperbolic/boost-like operators are deferred.

Auditory atom family	Measured source	Alias	Default blade index	Theta rule
Speech present	voiced frames / harmonic ratio	B_SPEECH	6	q(base + g1·voiced_ratio_q)
Short pause	pause duration	B_PAUSE_SHORT	7	q(base + g2·dur_hops)
Long pause	pause duration	B_PAUSE_LONG	8	q(base + g3·dur_hops)
Rising final contour	final F0 slope	B_PITCH_RISE	9	q(base + g4·slope_q)
Falling final contour	final F0 slope	B_PITCH_FALL	10	q(base + g5·abs(slope_q))
Emphasis / force	energy delta	B_EMPHASIS	11	q(base + g6·delta_db_q)
Hesitation / uncertainty	filled pause + low-conf contour	B_HESITATION	12	q(base + g7·hesitation_q)
Turn boundary	silent gap + local reset	B_TURN	13	q(base + g8·boundary_q)
Overlap / interruption	simultaneous speech / abrupt cut	B_OVERLAP	14	q(base + g9·overlap_q)
Alert-like event	salience / alarm morphology	B_ALERT	15	q(base + g10·salience_q)
Laughter	periodic burst pattern	B_LAUGH	16	q(base + g11·laugh_q)
Cry / distress	voicing + modulation profile	B_DISTRESS	17	q(base + g12·distress_q)
Music / tonal bed	stable harmonic bed	B_MUSIC	18	q(base + g13·tonal_q)
Chaotic broadband noise	flat/noisy spectrum	B_NOISE	19	q(base + g14·noise_q)

Indices are reasonable defaults, not metaphysical claims about Cl(4,1). The contract is that the mapping is explicit, versioned, checksummed, and frozen in the manifest. B_OVERLAP and B_TURN are the atoms that motivate per-stream arenas in ADR-0181 §2.3.

6.1 Minimal manifest (packs/audio/audio_core_v1/manifest.toml)

pack_id = "audio_core_v1"
modality = "audio"
cl41_dim = 32
compiler_version = "0.1.0"
basis_version = "audio-basis-v1"

[canonical]
sample_rate = 24000
channels = 1
frame_ms = 20
hop_ms = 10
output_dtype = "float32"
internal_dtype = "float64"

[resampling]
algorithm = "polyphase_fir"
fir_path = "resample_fir_v1.npy"
fir_sha256 = "sha256:REPLACE_ME"
padtype = "constant"
cval = 0.0

[gating]
gate_engaged = false
checksum_verified = false
versor_condition_max = 1.0e-6

[ordering]
event_precedence = ["channel", "pause", "speech", "prosody", "turn", "non_speech", "content_anchor"]

6.2 Operator row (operators.jsonl)

{
  "operator_id": "audio.prosody.question_contour.v1",
  "event_type": "prosody.question_contour",
  "blade_alias": "B_PITCH_RISE",
  "blade_index": 9,
  "rotor_kind": "elliptic",
  "base_theta_q": 64,
  "gain_rules": {"slope_q": 3, "final_energy_q": 1, "confidence_q": 1},
  "theta_clip_q": 384,
  "version": "1"
}

7. Numeric determinism

Rule: quantize before semantics, normalize after composition. Raw float measurements are too unstable to hash. Quantization regime (frozen in manifest): boundaries in hop units, log energy in 1 dB bins, F0 in 25-cent bins, pitch slope in coarse cents-per-100 ms bins, spectral shape in fixed ordinal bins, all confidences in uint8. After quantization, compute in float64, compose sparse rotors in canonical order, call algebra-owned unitize_versor, cast to float32 only at the output boundary.

import math
import numpy as np

EPS = 1e-12

def quantize_theta(theta: float, step: float = 1.0 / 1024.0) -> float:
    return round(theta / step) * step

def build_elliptic_rotor(blade_index: int, theta: float) -> np.ndarray:
    out = np.zeros(32, dtype=np.float64)
    half = quantize_theta(theta) / 2.0
    out[0] = math.cos(half)
    out[blade_index] = math.sin(half)
    return out

def compile_events(events, registry, geometric_product, unitize_versor, versor_condition):
    # SERIALIZATION BARRIER (ADR-0181 §2.1): in-chunk composition is order-sensitive,
    # single-threaded, canonical order. The substrate never parallelizes this loop.
    v = np.zeros(32, dtype=np.float64)
    v[0] = 1.0
    for ev in events:                          # must already be in canonical order
        spec = registry[ev.event_type]
        theta = spec.theta_from_event(ev)      # deterministic, quantized inputs only
        r = build_elliptic_rotor(spec.blade_index, theta)
        v = geometric_product(v, r)
        v = unitize_versor(v)
    if versor_condition(v) >= 1e-6:
        raise ValueError("audio compilation failed versor check")
    return v.astype(np.float32)

geometric_product, unitize_versor, versor_condition are imported from algebra/ — the audio compiler adds no new normalization function.

8. Repo-facing adapter (sensorium/adapters/audio.py)

from __future__ import annotations
from dataclasses import dataclass
import numpy as np

@dataclass(frozen=True, slots=True)
class AudioProjectionHead:
    compiler: "AudioCompiler"
    modality = ...                # Modality.AUDIO

    @property
    def embedding_dim(self) -> int:
        return 32

    def project(self, signal: "AudioSignal") -> np.ndarray:
        unit = self.compiler.compile(signal)
        out = unit.versor
        if out.shape != (32,):
            raise ValueError(f"expected (32,), got {out.shape}")
        if out.dtype != np.float32:
            raise TypeError(f"expected float32, got {out.dtype}")
        return out

    def project_batch(self, signals: list["AudioSignal"]) -> np.ndarray:
        return np.stack([self.project(s) for s in signals], axis=0)

    def verify_unitarity(self, signal: "AudioSignal") -> bool:
        return self.compiler.compile(signal).versor_condition < 1e-6

The adapter is thin and pack-governed; it satisfies the ProjectionHead protocol in sensorium/protocol.py and is mounted as a ModalityPack(modality_type=Modality.AUDIO, gate_engaged=False) until the eval gates pass.

9. Delta-CRDT delta interface (PR-5)

The audio adapter never writes the global epistemic_state (ADR-0180 §2.1). Instead:

1. compile() produces one AudioCompilationUnit per canonical chunk (the serialization barrier of §7 runs here).
2. The unit is written lock-free into the adapter's thread-local arena. Concurrent streams (overlap/interruption) each have their own arena (ADR-0181 §2.3).
3. The Merge Kernel (ADR-0180 §2.2, an explicitly-mounted component, not a daemon — ADR-0180 §1.5.5) folds pending units into the global Vault ordered by unit.merge_key. Duplicate keys are deduplicated (idempotence).
4. The kernel surfaces its pending-delta count in TurnEvent for replay evidence (ADR-0180 §1.5.5).

The per-chunk Vault contribution is (versor, provenance) where provenance = {merge_key, pack_id, pack_manifest_sha256} — content-addressed, no PCM.

10. File plan (PR-2 … PR-6)

sensorium/audio/{__init__,types,canonical,checksum,resample,frames,lexer,parser,operators,compiler,trace,fixtures,teachers}.py
sensorium/adapters/audio.py
packs/audio/audio_core_v1/{manifest.toml,basis_map.json,operators.jsonl,atoms.jsonl,prototypes.jsonl,resample_fir_v1.npy,checksums.json}
tests/test_audio_{signal,resample,lexer,parser,compiler,pack_manifest,sensorium_mount,trace,crdt_delta}.py
evals/audio_sensorium/{fixtures/*.wav,manifest.json,expected_ir.jsonl,expected_projection_hashes.json}