docs: audio section pipeline overview & primer

A from-the-ground-up reference for the audio modality: waveform → canonicalize → frames → acoustic lexer → typed AudioIR → operators/rotors → (32,) versor, with every spec/constant grounded in the code (PR-2..PR-6 stack). Includes the three-way clarification of learned models: embeddings (never), audio specs (CORE's own DSP, not any model), text transcript (Whisper/NeMo as typed labels only). Documents the substrate-vs-evidence split, the verbatim teacher policy, current inert/gated status, and the 'scrutinise the consumer' flag. Cross-links ADR-0180/0181 + spec + eval-plan. Lives at docs/audio_pipeline_overview.md.
2026-05-29 13:47:31 -07:00 · 2026-05-29 13:47:31 -07:00 · ae48a3cbf9
commit ae48a3cbf9
parent f017785a6d
1 changed files with 355 additions and 0 deletions
--- a/docs/audio_pipeline_overview.md
+++ b/docs/audio_pipeline_overview.md
@ -0,0 +1,355 @@
+# Audio Section — Pipeline Overview & Primer
+
+A from-the-ground-up reference for CORE's audio modality: how a waveform becomes
+a lawful Cl(4,1) versor, what every spec means, and exactly where (and why)
+learned models like Whisper are — and are **not** — involved.
+
+**Audience:** anyone re-familiarising with audio/DSP or onboarding to the audio
+section. No one knows everything; this is the single greppable source of truth.
+
+**Authoritative sources this summarises** (read these for the binding contract):
+- [`docs/decisions/ADR-0181-audio-compiler-delta-crdt.md`](decisions/ADR-0181-audio-compiler-delta-crdt.md) — the decision.
+- [`docs/plans/audio-compiler-spec.md`](plans/audio-compiler-spec.md) — the compiler spec.
+- [`docs/plans/audio-compiler-eval-plan.md`](plans/audio-compiler-eval-plan.md) — eval gates + teacher policy.
+- Code: `sensorium/audio/{canonical,frames,lexer,parser,operators,compiler,checksum,arena,teachers}.py`.
+
+> All numbers below are pulled from the code as of the ADR-0181 PR-2…PR-6 stack
+> (merged to `main`). If you change a constant, update this doc in the same PR.
+
+---
+
+## 0. The one big idea
+
+Most systems turn audio into an **embedding** (an opaque vector from a neural
+net). CORE refuses that. It treats audio like a **compiler treats source code**:
+raw waveform → measured acoustic facts → a *typed intermediate representation
+(IR)* → a lawful geometric object (a 32-dim Cl(4,1) versor). Every step is
+deterministic and checksummed, so the same bytes always produce the same result,
+bit-for-bit. Learned models may only *annotate*, never *define*.
+
+```mermaid
+flowchart LR
+    W[waveform] --> C[canonicalize<br/>mono 24kHz f32]
+    C --> F[frame grid<br/>20ms / 10ms]
+    F --> L[acoustic lexer<br/>energy / voicing / pitch]
+    L --> P[parser → typed AudioIR<br/>speech / pause / prosody / turn]
+    P --> O[operators → rotors]
+    O --> V[(32,) versor<br/>SUBSTRATE]
+    W -. optional .-> T[Whisper / NeMo<br/>transcript]
+    T -. typed label .-> P
+    P -.-> AN[content.* anchors<br/>EVIDENCE — never touches versor]
+```
+
+---
+
+## 1. Audio file fundamentals (the refresher)
+
+A digital audio file is just **amplitude samples over time**. The specs that
+define one:
+
+| Property | What it means | CORE's canonical choice |
+|---|---|---|
+| **Sample rate** | samples per second (Hz) | **24,000 Hz** (`CANONICAL_SAMPLE_RATE`) |
+| **Channels** | mono (1), stereo (2), … | **mono** (multi-channel is averaged down) |
+| **Sample format / bit depth** | how each sample is stored (int16, float32…) | **float32**, range ~[−1, 1] |
+| **Encoding** | PCM (raw) vs compressed (MP3/AAC) | PCM-equivalent float array |
+
+- **Nyquist limit:** a 24 kHz sample rate represents frequencies up to **12 kHz**
+  (half the rate). Speech energy is mostly below 8 kHz, so 24 kHz comfortably
+  captures speech + sibilance. (Teacher ASR gets a *derived* 16 kHz stream
+  because models like Whisper expect 16 kHz.)
+- **Why no `.wav` files in the repo:** tests synthesize signals from parameter
+  specs (`evals/audio_sensorium/fixtures.json`) instead of committing binary
+  blobs. A spec like `{tone, 150 Hz, 300 ms}` is diffable and greppable; a `.wav`
+  is opaque. The signal is a pure function of the spec, so "what's pinned is
+  exactly what's tested." `numpy` `PCG64` RNG + cast-to-float32-at-the-boundary
+  makes it bit-reproducible across machines.
+
+---
+
+## 2. Stage 1 — Canonicalize (`canonical.py`)
+
+Turns *whatever arrived* into the one canonical form everything else assumes:
+
+1. **Downmix to mono** — average across channels (handles `(N,)`, `(N,C)`,
+   `(C,N)`; channel axis = the smaller dimension).
+2. **Resample to 24 kHz if needed** — using a **pinned polyphase FIR filter**
+   from the pack (not an ad-hoc resampler — determinism). Same-rate input is an
+   exact passthrough.
+3. **Hash twice for provenance:**
+   - `source_sha256` = hash of the original bytes as received.
+   - `canonical_sha256` = hash of the canonical float32 image.
+
+Output is an `AudioSignal` (samples + sample_rate + start/end ms + the two
+hashes). **No raw PCM travels further than this** in the trace — only hashes.
+
+> *FIR = Finite Impulse Response filter; polyphase = an efficient way to resample
+> by rational ratios. "Pinned" = the exact filter taps are frozen pack data so
+> resampling is byte-identical everywhere.*
+
+---
+
+## 3. Stage 2 — Frame grid (`frames.py`)
+
+Audio is non-stationary (it changes constantly), so we analyse it in short
+overlapping chunks called **frames**:
+
+- **Window = 20 ms** (`FRAME_MS`) → 480 samples at 24 kHz.
+- **Hop = 10 ms** (`HOP_MS`) → 240 samples → **50% overlap** between frames.
+- The last partial frame is **zero-padded** to a full window, so the grid is a
+  pure function of (length, rate, frame_ms, hop_ms).
+
+A "hop" is the time unit the rest of the pipeline counts in: **hop index `i` =
+the chunk starting at `i × 10 ms`.** So 300 ms of audio ≈ 30 hops.
+
+---
+
+## 4. Stage 3 — Acoustic lexer (`lexer.py`) — *the actual audio features*
+
+This is where DSP happens. For **each frame** it measures facts and
+**quantizes** them (so the token stream hashes deterministically — "quantize
+before semantics"). Per hop it emits one primary classification token plus
+descriptors:
+
+### 4a. Energy (loudness)
+`log_energy_db = 20·log10(RMS)` — RMS is root-mean-square amplitude. Quantized to
+**1 dB integer bins**. Frames quieter than **−55 dB** (`SILENCE_DB`) → `silence`.
+
+### 4b. Voicing — voiced vs unvoiced
+Uses **Zero-Crossing Rate (ZCR)** = fraction of adjacent samples where the signal
+flips sign. Vowels (voiced) are quasi-periodic → **low ZCR**; fricatives/noise →
+**high ZCR**.
+- `voiced` if `ZCR ≤ 0.20` (`VOICED_ZCR_MAX`) **and** `dB ≥ −45` (`VOICED_MIN_DB`).
+- Otherwise (loud but noisy) → `unvoiced`, and it records a **spectral centroid
+  bin** (16 bins) = the "center of mass" of the spectrum (bright vs dull), from a
+  Hanning-windowed FFT.
+
+### 4c. Pitch (F0) — only for voiced frames
+Estimates **fundamental frequency** via **autocorrelation** (pYIN-style: find the
+lag where the signal best correlates with a delayed copy of itself; that lag =
+the pitch period).
+- Search range **50–500 Hz** (`F0_MIN/MAX_HZ`) — the human voice range.
+- Pitch in **cents**, not Hz: `cents = 1200·log2(Hz / 55)`, referenced to **55 Hz
+  (note A1)**, quantized to **25-cent bins**. Cents are a log scale — equal
+  *musical* intervals are equal cent-distances, which is how prosody works
+  perceptually.
+- Keeps the **top 2 candidates** as `(cents_q, prob_q)` pairs, `prob_q` ∈ 0–255
+  = peak strength/confidence.
+
+> Net: each hop becomes integer tokens like `energy_bin(−9)`, `voiced(dB, zcr)`,
+> `pitch_candidates(cents_q, prob_q, …)`.
+
+---
+
+## 5. Stage 4 — Parser → typed AudioIR (`parser.py`)
+
+Collapses runs of like frames into **typed spans/events** (never per-frame
+noise). Six event families:
+
+| Family | Built from | Example event types |
+|---|---|---|
+| **speech_spans** | runs of `voiced` | `speech.voiced` |
+| **pause_spans** | runs of `silence` | `pause.short`, `pause.long` (≥ **30 hops = 300 ms**) |
+| **prosody_arcs** | F0 slope / energy delta over a voiced span | `prosody.rise`, `prosody.fall`, `prosody.emphasis` (≥ **6 dB** swing) |
+| **turn_events** | a long pause | `turn.boundary` |
+| **non_speech_events** | runs of `unvoiced` | `nonspeech.noise` |
+| **content_anchors** | **teacher hints only** (PR-6) | `content.transcript`, … |
+
+Prosody logic: compare F0 at the end vs start of a voiced span — rising ≥ 1
+cent-bin → `prosody.rise` (question-like), falling → `prosody.fall`
+(statement-like). The whole IR is hashed → `ir_sha256`.
+
+---
+
+## 6. Stage 5 — Operators → rotors → versor (`operators.py`, `compiler.py`)
+
+Each event type maps to a **declared rotor** (a rotation in the geometric
+algebra), not an opaque vector:
+
+- v1 uses **elliptic bivector rotors only** — 6 planes in Cl(4,1) (blade indices
+  6,7,8,10,11,13) that square to −1, giving the well-behaved
+  `R = cos(θ/2) + B·sin(θ/2)`. This guarantees the composition is always a **unit
+  versor** (`versor_condition < 1e-6` without weakening the threshold).
+- The angle `θ_q` is an **integer** = `base_theta_q + Σ(gain × quantized_attr)`,
+  clipped. E.g. `prosody.rise` (plane 10, base 64) adds `3 × slope_q`.
+  `THETA_STEP = π/512` (1024 steps span the circle).
+- `compile_events` folds the canonically-ordered events into one versor by
+  repeated geometric product + unitize. **Events with no operator (like
+  `content.*` teacher hints) are skipped** — that's why teachers can't change the
+  versor.
+- Result = a **(32,) float32 multivector** — the single object that crosses into
+  CORE's field/vault, exactly like a text token does.
+
+The operator table (the `B_*` aliases) is the audio "phonology": `B_PAUSE_LONG`,
+`B_SPEECH`, `B_PITCH_RISE`, `B_PITCH_FALL`, `B_EMPHASIS`, `B_TURN`, `B_NOISE`.
+
+---
+
+## 7. The checksum chain & merge key (`checksum.py`)
+
+Every link is content-addressed:
+
+```text
+source_sha256 → canonical_sha256 → token_stream_sha256 → ir_sha256
+              → pack_manifest_sha256 → projection_sha256
+```
+
+The **merge key = `(canonical_sha256, ir_sha256, projection_sha256)`** — what the
+Delta-CRDT (`arena.py`, PR-5) uses to dedup/order. Teacher hints move only the
+`ir_sha256` leg (evidence), never `projection_sha256` (substrate).
+
+---
+
+## 8. Concurrency — the Delta-CRDT arena (`arena.py`, PR-5)
+
+Each compiled chunk is one `AudioCompilationUnit` (the delta). Units accumulate
+in a thread-local, share-nothing `AudioArena`; the merge kernel folds arena
+snapshots into one **content-addressed, deduplicated, totally-ordered** set keyed
+by `merge_key`. The merge is permutation- and duplicate-invariant, so
+`hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)` — the proof obligation
+of [ADR-0180](decisions/ADR-0180-crdt-sharded-vault-concurrency.md). The Python
+layer mirrors the Rust `LocalArena`/`SemilatticeDelta`/`merge_kernel`
+(`core-rs/src/vault.rs`) so they stay in parity when the binding lands.
+
+---
+
+## 9. Where learned models fit — and where they emphatically do not
+
+This is the most-asked question, so it gets its own section.
+
+### Three things a model *could* hand you
+
+| What | Used in CORE? | Where it comes from |
+|---|---|---|
+| **Embeddings** (opaque latent vectors) | ❌ **Never** — explicitly rejected | the whole "no embedding bridge" rejection (CLAP/EnCodec as substrate) |
+| **Audio specs** (pitch, energy, voicing, pauses, prosody, turns) | ✅ Yes | **CORE's own deterministic DSP compiler** (`lexer.py`/`parser.py`) — *not any learned model* |
+| **Text transcript** (the words) | ✅ (intended, as evidence) | **Whisper / NeMo** — admitted as a typed *label* |
+
+### Concretely
+
+- **The audio specs do NOT come from Whisper.** Every acoustic fact — framing,
+  energy, ZCR voicing, F0/pitch, spectral centroid, pause/turn detection — is
+  measured by CORE's own lawful DSP. Whisper has zero involvement. That's the
+  native substrate, and it stands alone.
+- **Whisper's only intended job is audio → words.** It would emit a **text
+  transcript** (+ timestamps/language ID), attached to a time span as a
+  `content.transcript` anchor — a **lexical label / evidence** in the IR. It fills
+  the one gap DSP can't: *what words were said*, vs *how they were said*.
+- **We take Whisper's discrete text output, never its internals.** Whisper is
+  itself a neural net full of embeddings — but CORE ingests only its **emitted
+  string + timestamps** (a typed, checksummed hint), never its latent vectors. A
+  word like `"home"` as a label, not a 768-dim vector.
+- **CLAP is the one to watch.** Its natural output *is* embeddings + audio-text
+  alignment. The eval plan admits CLAP only for **coarse text labels**
+  ("laughter", "alarm") and rejects its embeddings. The rule across all teachers:
+  **words/labels in, vectors never.**
+
+### The two lanes never mix
+
+```text
+waveform ─┬─► CORE DSP compiler ──► AudioIR specs (pitch/energy/pauses/turns) ──► versor   [SUBSTRATE]
+          │
+          └─► Whisper ──► "are you coming home?" + timestamps ──► content.transcript label  [EVIDENCE]
+                                                                  (never touches the versor)
+```
+
+### Teacher policy, verbatim (eval-plan §4)
+
+```text
+Use teachers to label or align.
+Never let teachers define the substrate.
+Never fold teacher embeddings directly into the main versor path.
+Only admit teacher outputs through typed, versioned, checksumed hints.
+```
+
+### Current status (important)
+
+As of the merged PR-6: the teacher lanes are **declared, gated behind optional
+extras, and inert.** `load_teacher("whisper")` raises `TeacherUnavailable` —
+**no real model is wired or imported.** The only working teacher is the
+deterministic `StubTranscriptTeacher` (no weights), used to prove the contract.
+The structural guarantee — a teacher hint leaves the versor/`projection_sha256`
+**byte-identical** — is enforced by a failable test
+(`tests/test_audio_teachers.py::test_teacher_hint_does_not_change_versor`).
+
+The real doctrinal commitment is **not** admitting a teacher; it's whenever
+someone builds the **consumer** that reads teacher hints into comprehension. No
+such consumer exists yet. That is the PR to scrutinise hard.
+
+---
+
+## 10. Specs quick-reference (all from the code)
+
+| Spec | Value | Source |
+|---|---|---|
+| Canonical sample rate | 24,000 Hz (Nyquist 12 kHz) | `canonical.py` |
+| Format | mono float32 | `canonical.py` |
+| Frame / hop | 20 ms (480 smp) / 10 ms (240 smp), 50% overlap | `frames.py` |
+| Silence threshold | −55 dB | `lexer.py` |
+| Voiced criteria | ZCR ≤ 0.20 and dB ≥ −45 | `lexer.py` |
+| F0 range / ref / bin | 50–500 Hz / 55 Hz (A1) / 25 cents | `lexer.py` |
+| Spectral centroid bins | 16 | `lexer.py` |
+| Long pause / turn | ≥ 30 hops (300 ms) | `parser.py` |
+| Emphasis threshold | ≥ 6 dB intra-span swing | `parser.py` |
+| Rotor type | elliptic bivector, 6 planes (6,7,8,10,11,13) | `operators.py` |
+| θ resolution | π/512 per step, 1024 steps | `operators.py` |
+| Output | (32,) float32, `versor_condition < 1e-6` | `compiler.py` |
+| Merge key | `(canonical_sha256, ir_sha256, projection_sha256)` | `checksum.py` |
+
+---
+
+## 11. The test fixtures, acoustically (`evals/audio_sensorium/`)
+
+Signals are synthesized from `fixtures.json` (no `.wav` blobs). Three primitives:
+`_tone` (sine + optional linear F0 sweep), `_silence` (zeros), `_noise` (seeded
+Gaussian) — all 24 kHz mono float32.
+
+| Fixture | Synthesis | What the lexer/parser extracts |
+|---|---|---|
+| `silence_500ms` | 500 ms zeros | 50 hops `silence` → `pause.long` + `turn.boundary` |
+| `rise_question` | 300 ms sine, 150 Hz **sweeping +90 → 240 Hz**, amp 0.5 | low ZCR + loud → `speech.voiced`; rising F0 → `prosody.rise` |
+| `fall_statement` | 300 ms sine, 230 Hz **sweeping −90 → 140 Hz**, amp 0.5 | `speech.voiced` + `prosody.fall` |
+| `noise_burst` | 300 ms Gaussian noise (seed 7, amp 0.3) | high ZCR → `unvoiced` → `nonspeech.noise` |
+| `speech_then_pause` | 300 ms 150 Hz tone + 400 ms silence | `speech.voiced` then `pause.long` + `turn.boundary` |
+
+A 0.5-amplitude sine has RMS ≈ 0.354 → ≈ **−9 dB** (well above −45) and very low
+ZCR → reliably "voiced." Gaussian noise crosses zero constantly → high ZCR →
+"unvoiced." The fixtures are designed so the parser's *accuracy* is checkable,
+not just its determinism.
+
+### What the tests actually assert
+
+`tests/test_audio_*.py` (in the PR smoke gate): exact `(32,)` float32 shape;
+`versor_condition < 1e-6`; bit-identical replay; frozen `canonical_sha256` /
+`ir_sha256` pins; IR-replay equality; **parser accuracy** (`event_type_counts`
+match the designed parse); cross-platform versor stability within `atol=1e-6`;
+trace hygiene (no PCM); gate-closure; sequential==concurrent merge; and
+teacher-shadow invariance. They prove the path is **deterministic, replayable,
+checksummed, lawfully shaped, and parses the intended structure** — they do *not*
+test real-world speech, accents, noise robustness, or transcription accuracy
+(that needs real corpora + the deferred teacher adapters).
+
+---
+
+## 12. Mini-glossary
+
+**PCM** — raw uncompressed samples. **RMS** — root-mean-square amplitude
+(loudness). **dB** — log loudness scale. **ZCR** — zero-crossing rate (voicing
+proxy). **F0 / pitch** — fundamental frequency. **cents** — log pitch unit (100
+cents = 1 semitone). **spectral centroid** — frequency "center of mass"
+(brightness). **Nyquist** — max representable freq = ½ sample rate. **frame/hop**
+— short analysis window / step between windows. **rotor/versor** — geometric-
+algebra rotation operator / the unit object it produces. **FIR** — finite impulse
+response (resampling) filter. **Delta-CRDT** — conflict-free replicated data type;
+order-invariant merge.
+
+---
+
+## 13. Where to go next
+
+- Run the audio tests: `core test -- tests/test_audio_*.py -q` (or plain
+  `uv run pytest tests/test_audio_*.py -q`).
+- Compiler internals & rationale: [`docs/plans/audio-compiler-spec.md`](plans/audio-compiler-spec.md).
+- Eval gates & teacher policy: [`docs/plans/audio-compiler-eval-plan.md`](plans/audio-compiler-eval-plan.md).
+- The decision & trade-offs: [`docs/decisions/ADR-0181-audio-compiler-delta-crdt.md`](decisions/ADR-0181-audio-compiler-delta-crdt.md).
+- The concurrency substrate: [`docs/decisions/ADR-0180-crdt-sharded-vault-concurrency.md`](decisions/ADR-0180-crdt-sharded-vault-concurrency.md).