diff --git a/docs/audio_pipeline_overview.md b/docs/audio_pipeline_overview.md new file mode 100644 index 00000000..b2319eb1 --- /dev/null +++ b/docs/audio_pipeline_overview.md @@ -0,0 +1,355 @@ +# Audio Section — Pipeline Overview & Primer + +A from-the-ground-up reference for CORE's audio modality: how a waveform becomes +a lawful Cl(4,1) versor, what every spec means, and exactly where (and why) +learned models like Whisper are — and are **not** — involved. + +**Audience:** anyone re-familiarising with audio/DSP or onboarding to the audio +section. No one knows everything; this is the single greppable source of truth. + +**Authoritative sources this summarises** (read these for the binding contract): +- [`docs/decisions/ADR-0181-audio-compiler-delta-crdt.md`](decisions/ADR-0181-audio-compiler-delta-crdt.md) — the decision. +- [`docs/plans/audio-compiler-spec.md`](plans/audio-compiler-spec.md) — the compiler spec. +- [`docs/plans/audio-compiler-eval-plan.md`](plans/audio-compiler-eval-plan.md) — eval gates + teacher policy. +- Code: `sensorium/audio/{canonical,frames,lexer,parser,operators,compiler,checksum,arena,teachers}.py`. + +> All numbers below are pulled from the code as of the ADR-0181 PR-2…PR-6 stack +> (merged to `main`). If you change a constant, update this doc in the same PR. + +--- + +## 0. The one big idea + +Most systems turn audio into an **embedding** (an opaque vector from a neural +net). CORE refuses that. It treats audio like a **compiler treats source code**: +raw waveform → measured acoustic facts → a *typed intermediate representation +(IR)* → a lawful geometric object (a 32-dim Cl(4,1) versor). Every step is +deterministic and checksummed, so the same bytes always produce the same result, +bit-for-bit. Learned models may only *annotate*, never *define*. + +```mermaid +flowchart LR + W[waveform] --> C[canonicalize
mono 24kHz f32] + C --> F[frame grid
20ms / 10ms] + F --> L[acoustic lexer
energy / voicing / pitch] + L --> P[parser → typed AudioIR
speech / pause / prosody / turn] + P --> O[operators → rotors] + O --> V[(32,) versor
SUBSTRATE] + W -. optional .-> T[Whisper / NeMo
transcript] + T -. typed label .-> P + P -.-> AN[content.* anchors
EVIDENCE — never touches versor] +``` + +--- + +## 1. Audio file fundamentals (the refresher) + +A digital audio file is just **amplitude samples over time**. The specs that +define one: + +| Property | What it means | CORE's canonical choice | +|---|---|---| +| **Sample rate** | samples per second (Hz) | **24,000 Hz** (`CANONICAL_SAMPLE_RATE`) | +| **Channels** | mono (1), stereo (2), … | **mono** (multi-channel is averaged down) | +| **Sample format / bit depth** | how each sample is stored (int16, float32…) | **float32**, range ~[−1, 1] | +| **Encoding** | PCM (raw) vs compressed (MP3/AAC) | PCM-equivalent float array | + +- **Nyquist limit:** a 24 kHz sample rate represents frequencies up to **12 kHz** + (half the rate). Speech energy is mostly below 8 kHz, so 24 kHz comfortably + captures speech + sibilance. (Teacher ASR gets a *derived* 16 kHz stream + because models like Whisper expect 16 kHz.) +- **Why no `.wav` files in the repo:** tests synthesize signals from parameter + specs (`evals/audio_sensorium/fixtures.json`) instead of committing binary + blobs. A spec like `{tone, 150 Hz, 300 ms}` is diffable and greppable; a `.wav` + is opaque. The signal is a pure function of the spec, so "what's pinned is + exactly what's tested." `numpy` `PCG64` RNG + cast-to-float32-at-the-boundary + makes it bit-reproducible across machines. + +--- + +## 2. Stage 1 — Canonicalize (`canonical.py`) + +Turns *whatever arrived* into the one canonical form everything else assumes: + +1. **Downmix to mono** — average across channels (handles `(N,)`, `(N,C)`, + `(C,N)`; channel axis = the smaller dimension). +2. **Resample to 24 kHz if needed** — using a **pinned polyphase FIR filter** + from the pack (not an ad-hoc resampler — determinism). Same-rate input is an + exact passthrough. +3. **Hash twice for provenance:** + - `source_sha256` = hash of the original bytes as received. + - `canonical_sha256` = hash of the canonical float32 image. + +Output is an `AudioSignal` (samples + sample_rate + start/end ms + the two +hashes). **No raw PCM travels further than this** in the trace — only hashes. + +> *FIR = Finite Impulse Response filter; polyphase = an efficient way to resample +> by rational ratios. "Pinned" = the exact filter taps are frozen pack data so +> resampling is byte-identical everywhere.* + +--- + +## 3. Stage 2 — Frame grid (`frames.py`) + +Audio is non-stationary (it changes constantly), so we analyse it in short +overlapping chunks called **frames**: + +- **Window = 20 ms** (`FRAME_MS`) → 480 samples at 24 kHz. +- **Hop = 10 ms** (`HOP_MS`) → 240 samples → **50% overlap** between frames. +- The last partial frame is **zero-padded** to a full window, so the grid is a + pure function of (length, rate, frame_ms, hop_ms). + +A "hop" is the time unit the rest of the pipeline counts in: **hop index `i` = +the chunk starting at `i × 10 ms`.** So 300 ms of audio ≈ 30 hops. + +--- + +## 4. Stage 3 — Acoustic lexer (`lexer.py`) — *the actual audio features* + +This is where DSP happens. For **each frame** it measures facts and +**quantizes** them (so the token stream hashes deterministically — "quantize +before semantics"). Per hop it emits one primary classification token plus +descriptors: + +### 4a. Energy (loudness) +`log_energy_db = 20·log10(RMS)` — RMS is root-mean-square amplitude. Quantized to +**1 dB integer bins**. Frames quieter than **−55 dB** (`SILENCE_DB`) → `silence`. + +### 4b. Voicing — voiced vs unvoiced +Uses **Zero-Crossing Rate (ZCR)** = fraction of adjacent samples where the signal +flips sign. Vowels (voiced) are quasi-periodic → **low ZCR**; fricatives/noise → +**high ZCR**. +- `voiced` if `ZCR ≤ 0.20` (`VOICED_ZCR_MAX`) **and** `dB ≥ −45` (`VOICED_MIN_DB`). +- Otherwise (loud but noisy) → `unvoiced`, and it records a **spectral centroid + bin** (16 bins) = the "center of mass" of the spectrum (bright vs dull), from a + Hanning-windowed FFT. + +### 4c. Pitch (F0) — only for voiced frames +Estimates **fundamental frequency** via **autocorrelation** (pYIN-style: find the +lag where the signal best correlates with a delayed copy of itself; that lag = +the pitch period). +- Search range **50–500 Hz** (`F0_MIN/MAX_HZ`) — the human voice range. +- Pitch in **cents**, not Hz: `cents = 1200·log2(Hz / 55)`, referenced to **55 Hz + (note A1)**, quantized to **25-cent bins**. Cents are a log scale — equal + *musical* intervals are equal cent-distances, which is how prosody works + perceptually. +- Keeps the **top 2 candidates** as `(cents_q, prob_q)` pairs, `prob_q` ∈ 0–255 + = peak strength/confidence. + +> Net: each hop becomes integer tokens like `energy_bin(−9)`, `voiced(dB, zcr)`, +> `pitch_candidates(cents_q, prob_q, …)`. + +--- + +## 5. Stage 4 — Parser → typed AudioIR (`parser.py`) + +Collapses runs of like frames into **typed spans/events** (never per-frame +noise). Six event families: + +| Family | Built from | Example event types | +|---|---|---| +| **speech_spans** | runs of `voiced` | `speech.voiced` | +| **pause_spans** | runs of `silence` | `pause.short`, `pause.long` (≥ **30 hops = 300 ms**) | +| **prosody_arcs** | F0 slope / energy delta over a voiced span | `prosody.rise`, `prosody.fall`, `prosody.emphasis` (≥ **6 dB** swing) | +| **turn_events** | a long pause | `turn.boundary` | +| **non_speech_events** | runs of `unvoiced` | `nonspeech.noise` | +| **content_anchors** | **teacher hints only** (PR-6) | `content.transcript`, … | + +Prosody logic: compare F0 at the end vs start of a voiced span — rising ≥ 1 +cent-bin → `prosody.rise` (question-like), falling → `prosody.fall` +(statement-like). The whole IR is hashed → `ir_sha256`. + +--- + +## 6. Stage 5 — Operators → rotors → versor (`operators.py`, `compiler.py`) + +Each event type maps to a **declared rotor** (a rotation in the geometric +algebra), not an opaque vector: + +- v1 uses **elliptic bivector rotors only** — 6 planes in Cl(4,1) (blade indices + 6,7,8,10,11,13) that square to −1, giving the well-behaved + `R = cos(θ/2) + B·sin(θ/2)`. This guarantees the composition is always a **unit + versor** (`versor_condition < 1e-6` without weakening the threshold). +- The angle `θ_q` is an **integer** = `base_theta_q + Σ(gain × quantized_attr)`, + clipped. E.g. `prosody.rise` (plane 10, base 64) adds `3 × slope_q`. + `THETA_STEP = π/512` (1024 steps span the circle). +- `compile_events` folds the canonically-ordered events into one versor by + repeated geometric product + unitize. **Events with no operator (like + `content.*` teacher hints) are skipped** — that's why teachers can't change the + versor. +- Result = a **(32,) float32 multivector** — the single object that crosses into + CORE's field/vault, exactly like a text token does. + +The operator table (the `B_*` aliases) is the audio "phonology": `B_PAUSE_LONG`, +`B_SPEECH`, `B_PITCH_RISE`, `B_PITCH_FALL`, `B_EMPHASIS`, `B_TURN`, `B_NOISE`. + +--- + +## 7. The checksum chain & merge key (`checksum.py`) + +Every link is content-addressed: + +```text +source_sha256 → canonical_sha256 → token_stream_sha256 → ir_sha256 + → pack_manifest_sha256 → projection_sha256 +``` + +The **merge key = `(canonical_sha256, ir_sha256, projection_sha256)`** — what the +Delta-CRDT (`arena.py`, PR-5) uses to dedup/order. Teacher hints move only the +`ir_sha256` leg (evidence), never `projection_sha256` (substrate). + +--- + +## 8. Concurrency — the Delta-CRDT arena (`arena.py`, PR-5) + +Each compiled chunk is one `AudioCompilationUnit` (the delta). Units accumulate +in a thread-local, share-nothing `AudioArena`; the merge kernel folds arena +snapshots into one **content-addressed, deduplicated, totally-ordered** set keyed +by `merge_key`. The merge is permutation- and duplicate-invariant, so +`hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest)` — the proof obligation +of [ADR-0180](decisions/ADR-0180-crdt-sharded-vault-concurrency.md). The Python +layer mirrors the Rust `LocalArena`/`SemilatticeDelta`/`merge_kernel` +(`core-rs/src/vault.rs`) so they stay in parity when the binding lands. + +--- + +## 9. Where learned models fit — and where they emphatically do not + +This is the most-asked question, so it gets its own section. + +### Three things a model *could* hand you + +| What | Used in CORE? | Where it comes from | +|---|---|---| +| **Embeddings** (opaque latent vectors) | ❌ **Never** — explicitly rejected | the whole "no embedding bridge" rejection (CLAP/EnCodec as substrate) | +| **Audio specs** (pitch, energy, voicing, pauses, prosody, turns) | ✅ Yes | **CORE's own deterministic DSP compiler** (`lexer.py`/`parser.py`) — *not any learned model* | +| **Text transcript** (the words) | ✅ (intended, as evidence) | **Whisper / NeMo** — admitted as a typed *label* | + +### Concretely + +- **The audio specs do NOT come from Whisper.** Every acoustic fact — framing, + energy, ZCR voicing, F0/pitch, spectral centroid, pause/turn detection — is + measured by CORE's own lawful DSP. Whisper has zero involvement. That's the + native substrate, and it stands alone. +- **Whisper's only intended job is audio → words.** It would emit a **text + transcript** (+ timestamps/language ID), attached to a time span as a + `content.transcript` anchor — a **lexical label / evidence** in the IR. It fills + the one gap DSP can't: *what words were said*, vs *how they were said*. +- **We take Whisper's discrete text output, never its internals.** Whisper is + itself a neural net full of embeddings — but CORE ingests only its **emitted + string + timestamps** (a typed, checksummed hint), never its latent vectors. A + word like `"home"` as a label, not a 768-dim vector. +- **CLAP is the one to watch.** Its natural output *is* embeddings + audio-text + alignment. The eval plan admits CLAP only for **coarse text labels** + ("laughter", "alarm") and rejects its embeddings. The rule across all teachers: + **words/labels in, vectors never.** + +### The two lanes never mix + +```text +waveform ─┬─► CORE DSP compiler ──► AudioIR specs (pitch/energy/pauses/turns) ──► versor [SUBSTRATE] + │ + └─► Whisper ──► "are you coming home?" + timestamps ──► content.transcript label [EVIDENCE] + (never touches the versor) +``` + +### Teacher policy, verbatim (eval-plan §4) + +```text +Use teachers to label or align. +Never let teachers define the substrate. +Never fold teacher embeddings directly into the main versor path. +Only admit teacher outputs through typed, versioned, checksumed hints. +``` + +### Current status (important) + +As of the merged PR-6: the teacher lanes are **declared, gated behind optional +extras, and inert.** `load_teacher("whisper")` raises `TeacherUnavailable` — +**no real model is wired or imported.** The only working teacher is the +deterministic `StubTranscriptTeacher` (no weights), used to prove the contract. +The structural guarantee — a teacher hint leaves the versor/`projection_sha256` +**byte-identical** — is enforced by a failable test +(`tests/test_audio_teachers.py::test_teacher_hint_does_not_change_versor`). + +The real doctrinal commitment is **not** admitting a teacher; it's whenever +someone builds the **consumer** that reads teacher hints into comprehension. No +such consumer exists yet. That is the PR to scrutinise hard. + +--- + +## 10. Specs quick-reference (all from the code) + +| Spec | Value | Source | +|---|---|---| +| Canonical sample rate | 24,000 Hz (Nyquist 12 kHz) | `canonical.py` | +| Format | mono float32 | `canonical.py` | +| Frame / hop | 20 ms (480 smp) / 10 ms (240 smp), 50% overlap | `frames.py` | +| Silence threshold | −55 dB | `lexer.py` | +| Voiced criteria | ZCR ≤ 0.20 and dB ≥ −45 | `lexer.py` | +| F0 range / ref / bin | 50–500 Hz / 55 Hz (A1) / 25 cents | `lexer.py` | +| Spectral centroid bins | 16 | `lexer.py` | +| Long pause / turn | ≥ 30 hops (300 ms) | `parser.py` | +| Emphasis threshold | ≥ 6 dB intra-span swing | `parser.py` | +| Rotor type | elliptic bivector, 6 planes (6,7,8,10,11,13) | `operators.py` | +| θ resolution | π/512 per step, 1024 steps | `operators.py` | +| Output | (32,) float32, `versor_condition < 1e-6` | `compiler.py` | +| Merge key | `(canonical_sha256, ir_sha256, projection_sha256)` | `checksum.py` | + +--- + +## 11. The test fixtures, acoustically (`evals/audio_sensorium/`) + +Signals are synthesized from `fixtures.json` (no `.wav` blobs). Three primitives: +`_tone` (sine + optional linear F0 sweep), `_silence` (zeros), `_noise` (seeded +Gaussian) — all 24 kHz mono float32. + +| Fixture | Synthesis | What the lexer/parser extracts | +|---|---|---| +| `silence_500ms` | 500 ms zeros | 50 hops `silence` → `pause.long` + `turn.boundary` | +| `rise_question` | 300 ms sine, 150 Hz **sweeping +90 → 240 Hz**, amp 0.5 | low ZCR + loud → `speech.voiced`; rising F0 → `prosody.rise` | +| `fall_statement` | 300 ms sine, 230 Hz **sweeping −90 → 140 Hz**, amp 0.5 | `speech.voiced` + `prosody.fall` | +| `noise_burst` | 300 ms Gaussian noise (seed 7, amp 0.3) | high ZCR → `unvoiced` → `nonspeech.noise` | +| `speech_then_pause` | 300 ms 150 Hz tone + 400 ms silence | `speech.voiced` then `pause.long` + `turn.boundary` | + +A 0.5-amplitude sine has RMS ≈ 0.354 → ≈ **−9 dB** (well above −45) and very low +ZCR → reliably "voiced." Gaussian noise crosses zero constantly → high ZCR → +"unvoiced." The fixtures are designed so the parser's *accuracy* is checkable, +not just its determinism. + +### What the tests actually assert + +`tests/test_audio_*.py` (in the PR smoke gate): exact `(32,)` float32 shape; +`versor_condition < 1e-6`; bit-identical replay; frozen `canonical_sha256` / +`ir_sha256` pins; IR-replay equality; **parser accuracy** (`event_type_counts` +match the designed parse); cross-platform versor stability within `atol=1e-6`; +trace hygiene (no PCM); gate-closure; sequential==concurrent merge; and +teacher-shadow invariance. They prove the path is **deterministic, replayable, +checksummed, lawfully shaped, and parses the intended structure** — they do *not* +test real-world speech, accents, noise robustness, or transcription accuracy +(that needs real corpora + the deferred teacher adapters). + +--- + +## 12. Mini-glossary + +**PCM** — raw uncompressed samples. **RMS** — root-mean-square amplitude +(loudness). **dB** — log loudness scale. **ZCR** — zero-crossing rate (voicing +proxy). **F0 / pitch** — fundamental frequency. **cents** — log pitch unit (100 +cents = 1 semitone). **spectral centroid** — frequency "center of mass" +(brightness). **Nyquist** — max representable freq = ½ sample rate. **frame/hop** +— short analysis window / step between windows. **rotor/versor** — geometric- +algebra rotation operator / the unit object it produces. **FIR** — finite impulse +response (resampling) filter. **Delta-CRDT** — conflict-free replicated data type; +order-invariant merge. + +--- + +## 13. Where to go next + +- Run the audio tests: `core test -- tests/test_audio_*.py -q` (or plain + `uv run pytest tests/test_audio_*.py -q`). +- Compiler internals & rationale: [`docs/plans/audio-compiler-spec.md`](plans/audio-compiler-spec.md). +- Eval gates & teacher policy: [`docs/plans/audio-compiler-eval-plan.md`](plans/audio-compiler-eval-plan.md). +- The decision & trade-offs: [`docs/decisions/ADR-0181-audio-compiler-delta-crdt.md`](decisions/ADR-0181-audio-compiler-delta-crdt.md). +- The concurrency substrate: [`docs/decisions/ADR-0180-crdt-sharded-vault-concurrency.md`](decisions/ADR-0180-crdt-sharded-vault-concurrency.md).