docs(adr-0181): propose CORE-native audio compiler over Delta-CRDT substrate (#462)
Maps the AssetOverflow audio-compiler proposal onto the existing sensorium ProjectionHead boundary (ADR-0013) and the Delta-CRDT sharded substrate (ADR-0180). Core mapping decision: the audio chunk boundary IS the CRDT delta boundary. In-chunk rotor composition (compile_events) is the explicit serialization barrier ADR-0180 §1.5.2 requires for non-commutative versor_apply; the resulting AudioCompilationUnit is the order-invariant delta written at the only semilattice-eligible layer (vault/store). The compiler's checksum chain supplies the content-addressed merge key (canonical, ir, projection sha256) that ADR-0180 §1.5.3 demands, making idempotence structural and the sequential==concurrent trace-hash proof checkable. Adds: - docs/decisions/ADR-0181-audio-compiler-delta-crdt.md (decision + mapping) - docs/plans/audio-compiler-spec.md (typed AudioIR, operator table, manifest, numeric determinism, delta interface) - docs/plans/audio-compiler-eval-plan.md (corpus, gates, A-1..A-6 CRDT proof obligations, teacher-migration policy) Docs only; no runtime/core mutation. PR-2..PR-6 substrate work sequenced in the ADR, with PR-5 gated on ADR-0180 §1.5.4 (T-1..T-4) green on main.
This commit is contained in:
Shay
GitHub
parent
de6df1edc9
commit
fd96074230
3 changed files with 719 additions and 0 deletions
254
docs/decisions/ADR-0181-audio-compiler-delta-crdt.md
Normal file
254
docs/decisions/ADR-0181-audio-compiler-delta-crdt.md
Normal file
|
|
@ -0,0 +1,254 @@
|
|||
# ADR-0181: CORE-native Audio Compiler over the Delta-CRDT Substrate
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-29
|
||||
**Authors:** Joshua M. Shay, Core R&D Engine
|
||||
**Domains:** `sensorium/audio/`, `sensorium/adapters/audio.py`, `packs/audio/`, `core-rs/src/vault.rs` (read-only contract), `evals/audio_sensorium/`
|
||||
**Depends on:** ADR-0013 (Sensorium Multimodal Protocol), ADR-0180 (Delta-CRDT Sharded Substrate)
|
||||
**Companion docs:** [audio-compiler-spec.md](../plans/audio-compiler-spec.md), [audio-compiler-eval-plan.md](../plans/audio-compiler-eval-plan.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & Problem Statement
|
||||
|
||||
CORE's sensory boundary is already lawful and text-only-in-practice. `sensorium/protocol.py`
|
||||
fixes the contract:
|
||||
|
||||
- `ProjectionHead.project(S) -> (32,) float32` is the **Logos-recovery boundary**
|
||||
(`CL41_DIM = 32`).
|
||||
- `Modality.AUDIO` already exists in the enum but has no adapter
|
||||
(`sensorium/adapters/` ships `text.py` only).
|
||||
- `ModalityPack` enforces gate/checksum invariants at construction; `ModalityRegistry.mount()`
|
||||
runs the unitarity check and `project()` refuses a closed gate.
|
||||
|
||||
ADR-0013 (Accepted) requires every new modality to cross this boundary *before* it reaches
|
||||
`ingest/gate.py`, and forbids touching `ingest/`, `field/`, `generate/`, `vault/`, `vocab/`
|
||||
to add one. Audio must therefore **arrive already compiled** into a `(32,)` Cl(4,1) multivector.
|
||||
|
||||
ADR-0180 (Proposed) introduces the **Delta-CRDT sharded substrate** precisely to absorb the
|
||||
continuous, high-density streams that audio and vision produce — naming AUDIO explicitly — and
|
||||
flags one hard constraint we must satisfy (§1.5.2):
|
||||
|
||||
> The semilattice claim holds **only** at the `vault/store` layer — not at `versor_apply`
|
||||
> and not at `compute_trace_hash`. … Any operation upstream of `vault/store` that the
|
||||
> substrate parallelizes must either (a) be proven order-invariant on its inputs, or
|
||||
> (b) carry an explicit serialization barrier.
|
||||
|
||||
The problem this ADR solves: **how does audio enter CORE as a lawful, deterministic,
|
||||
replayable modality that is also a well-behaved Delta-CRDT delta producer** — without
|
||||
violating the no-core-mutation rule, the exact-recall rule, or ADR-0180's order-invariance
|
||||
obligation?
|
||||
|
||||
The wrong answer is an embedding bridge (CLAP/EnCodec/Whisper as substrate). It produces an
|
||||
opaque latent that cannot be checksummed, cannot be replayed bit-for-bit, and cannot supply
|
||||
the content-addressed merge key ADR-0180 §1.5.3 demands. The right answer is a **compiler**.
|
||||
|
||||
## 2. Decision
|
||||
|
||||
We will build `audio_core_v1` as a **deterministic auditory compiler** under `sensorium/audio/`,
|
||||
lowering canonical waveform input through a typed `AudioIR` into a `(32,)` float32 Cl(4,1)
|
||||
versor, and we will make its **chunk boundary the Delta-CRDT delta boundary**. Learned models
|
||||
(Whisper, NeMo, CLAP, EnCodec) are admitted only as subordinate teacher/shadow lanes, never as
|
||||
the substrate.
|
||||
|
||||
The compilation pipeline (detailed in the spec) is:
|
||||
|
||||
```text
|
||||
waveform / live stream
|
||||
→ canonicalizer (mono, fixed sample rate, source+canonical sha256)
|
||||
→ frame grid (20 ms window / 10 ms hop)
|
||||
→ acoustic lexer (energy, voicing, onset, pitch candidates, spectral bands, pauses)
|
||||
→ typed AudioIR (speech/pause spans, prosody arcs, turn/overlap events, non-speech atoms)
|
||||
→ operator registry (pack-local blade aliases + quantized theta rules)
|
||||
→ rotor lowering (elliptic bivector rotors only in v1)
|
||||
→ versor composition (geometric_product → unitize_versor → versor_condition < 1e-6)
|
||||
→ (32,) float32 == one AudioCompilationUnit
|
||||
```
|
||||
|
||||
Normalization sites stay inside CLAUDE.md's allowlist: quantization and FIR resampling live in
|
||||
the audio pack/compiler construction boundary, `unitize_versor` is algebra-owned, and **no
|
||||
hot-path drift repair is added**. CGA null vectors are preserved as null vectors.
|
||||
|
||||
### 2.1 The optimal mapping to Delta-CRDT (the load-bearing decision)
|
||||
|
||||
ADR-0180 §1.5.2 gives audio a binary choice for everything the substrate parallelizes:
|
||||
**prove order-invariance, or carry a serialization barrier.** The audio compiler answers
|
||||
*both*, at two different granularities:
|
||||
|
||||
| Granularity | Operation | CRDT treatment | Why |
|
||||
|---|---|---|---|
|
||||
| **Within a chunk** | `compile_events` (rotor chaining via `geometric_product`) | **Serialization barrier** | The sandwich product is non-commutative (ADR-0180 §1.5.2 row 3). In-chunk composition runs serially, single-threaded, in canonical event order, inside one thread-local arena. |
|
||||
| **Across chunks / streams** | merge of `AudioCompilationUnit`s into the Vault | **Order-invariant delta** | Each unit is `(versor, provenance)` written at the `vault/store` layer — the *only* semilattice-eligible layer (ADR-0180 §1.5.2 row 5). Merge is commutative, associative, idempotent. |
|
||||
|
||||
This is the synthesis ADR-0180 was waiting for: **the audio chunk boundary is the natural
|
||||
delta boundary** named in ADR-0180 §2.2 ("at semantic chunk boundaries"). The compiler does the
|
||||
order-sensitive work behind the barrier and emits only order-invariant deltas across it. The
|
||||
substrate never parallelizes a non-commutative operation.
|
||||
|
||||
### 2.2 Content-addressed merge key from the checksum chain
|
||||
|
||||
ADR-0180 §1.5.3 requires the trace-hash reduction to consume vault state in a
|
||||
**content-addressed order**, not wall-clock arrival order, or `hash(Sequential_Ingest) ==
|
||||
hash(Concurrent_CRDT_Ingest)` cannot hold. The audio compiler already produces exactly this key
|
||||
as a byproduct of its layered checksum chain:
|
||||
|
||||
```text
|
||||
source_sha256 → canonical_sha256 → token_stream_sha256 → ir_sha256
|
||||
→ pack_manifest_sha256 → projection_sha256
|
||||
```
|
||||
|
||||
The **merge key for an audio delta is `(canonical_sha256, ir_sha256, projection_sha256)`**.
|
||||
Consequences:
|
||||
|
||||
- **Idempotence is structural, not asserted.** Identical canonical bytes under an identical
|
||||
pack produce an identical key; the CRDT join deduplicates them. This satisfies the
|
||||
idempotence leg of the join semilattice (ADR-0180 §2.2) by construction.
|
||||
- **The content-addressed sort is free.** The merge kernel orders pending audio deltas by this
|
||||
key. No re-sort pass is needed at hash time for the audio portion of the path.
|
||||
- **`projection_sha256` is computed behind the serialization barrier** (§2.1), i.e. on the
|
||||
serialized in-chunk composition — satisfying ADR-0180 §1.5.3 point 3 (upstream-of-Vault
|
||||
hashes must be computed on the serialized portion).
|
||||
|
||||
### 2.3 Physical sharding mirrors the audio domain's natural concurrency
|
||||
|
||||
ADR-0180 §2.1 assigns each active modality adapter a thread-local arena and forbids
|
||||
`sensorium/adapters/*` from writing the global `epistemic_state` directly. For audio this is not
|
||||
just a mechanical convenience — it is **semantically aligned**: overlapping speakers,
|
||||
interruptions, and turn boundaries (the `B_OVERLAP`, `B_TURN` atoms in the operator table) are
|
||||
*literally* concurrent auditory streams. Each concurrent stream gets its own arena; the
|
||||
`AuditoryEvent` timing (`start_hop`/`end_hop`) is preserved inside each unit so the merge can
|
||||
reconstruct overlap relationships without a global lock during ingestion. The substrate's
|
||||
physical sharding is therefore a faithful image of the audio source's structure, not an
|
||||
impedance mismatch.
|
||||
|
||||
### 2.4 Eventual-consistency window is safe for audio
|
||||
|
||||
ADR-0180 §3.2 documents a sub-50ms window where a delta sits in the local arena before merge.
|
||||
Audio is robust to this because:
|
||||
|
||||
- `ProjectionHead.project` is **pure on the signal** (ADR-0180 T-4) — the audio compiler never
|
||||
reads cross-modal or global state during compilation, so a delayed merge cannot change what it
|
||||
produces.
|
||||
- Each unit retains its own intra-stream event timing, so cross-modal resonance re-anchors on
|
||||
merged state after the window closes; recall remains **exact byte-for-byte once merged**
|
||||
(ADR-0180 §1.5.5), never approximate.
|
||||
|
||||
### 2.5 Gate-closed by default
|
||||
|
||||
`audio_core_v1` mounts with `gate_engaged = false` until the eval gates in the companion plan
|
||||
pass. A closed gate makes `ModalityRegistry.project("audio_core_v1", …)` raise — audio cannot
|
||||
contribute deltas to any arena until determinism, checksum, unitarity, and mount-validation
|
||||
gates are green. This reuses the existing registry enforcement; no new gating machinery is
|
||||
added.
|
||||
|
||||
## 3. Consequences
|
||||
|
||||
### 3.1 Positive
|
||||
|
||||
- **First concrete exerciser of ADR-0180.** Audio is the first continuous modality to land on
|
||||
the Delta-CRDT substrate, converting ADR-0180's proof obligation `hash(Sequential) ==
|
||||
hash(Concurrent)` from abstract to testable against real delta producers.
|
||||
- **Order-invariance is proven, not hoped.** §2.1/§2.2 give a concrete serialization barrier and
|
||||
a content-addressed key, closing ADR-0180 §1.5.2's open constraint for the audio path.
|
||||
- **No core mutation.** Everything new lives under `sensorium/audio/`, `packs/audio/`,
|
||||
`tests/`, `evals/`. `ingest/`, `field/`, `generate/`, `vault/`, `vocab/` are untouched
|
||||
(ADR-0013), and `anti_unifier`/`carrier` need no changes (ADR-0180 §3.1).
|
||||
- **Trace hygiene composes.** Turn traces record `(canonical_sha256, ir_sha256,
|
||||
projection_sha256)` and pack IDs — never raw PCM (ADR-0180 §1.5.5).
|
||||
|
||||
### 3.2 Negative / Risks
|
||||
|
||||
- **Semantic underreach (v1).** The compiler captures prosody, turn dynamics, and salient
|
||||
non-speech events better than lexical content. Acceptable: transcript teachers backfill
|
||||
lexical evidence while the substrate stays native.
|
||||
- **Pack ontology drift.** If blade aliases or operator gains change freely, projections become
|
||||
incomparable across versions. Mitigation: the pack is versioned, checksummed, and gate-closed;
|
||||
`basis_version` is part of the merge key's pack-manifest leg.
|
||||
- **Over/under-quantization.** Too-coarse bins flatten meaning; too-fine bins make replay
|
||||
brittle. Mitigation: quantization regime is frozen in the manifest and covered by
|
||||
cross-platform stability gates.
|
||||
- **Streaming seam artifacts.** Stateful resampling/pitch/overlap must preserve continuity
|
||||
across chunk boundaries. Deferred to the streaming phase; v1 is offline/whole-buffer.
|
||||
- **Licensing contamination.** openSMILE's OSS build is non-commercial; it is a reference oracle
|
||||
only, never a runtime dependency.
|
||||
|
||||
## 4. Execution Plan & Proof Obligations
|
||||
|
||||
### 4.1 PR stack (additive, doctrine-first)
|
||||
|
||||
| PR | Scope | Gate |
|
||||
|---|---|---|
|
||||
| **PR-1 (this)** | ADR-0181 + compiler spec + eval plan (docs only) | review |
|
||||
| **PR-2** | Deterministic substrate: `sensorium/audio/{types,canonical,checksum,resample,frames,lexer,parser,operators,compiler,trace}.py` | determinism + versor unit tests |
|
||||
| **PR-3** | Pack artifacts `packs/audio/audio_core_v1/*` + `AudioProjectionHead` adapter + mount tests | mount/gate/checksum gates |
|
||||
| **PR-4** | `evals/audio_sensorium/` fixtures, expected IR, expected projection hashes | full eval-gate table |
|
||||
| **PR-5** | Delta-CRDT wiring: `AudioCompilationUnit` → thread-local arena → merge key, behind ADR-0180's substrate | sequential==concurrent trace-hash proof |
|
||||
| **PR-6** | Teacher/shadow lanes (Whisper/NeMo/CLAP/EnCodec) behind optional extras | teachers admitted only as typed hints |
|
||||
|
||||
PR-5 must not start until ADR-0180's own §1.5.4 obligations (T-1…T-4) are green on `main` —
|
||||
the audio delta path rides on them.
|
||||
|
||||
### 4.2 Audio-specific proof obligations (extend ADR-0180 §1.5.4)
|
||||
|
||||
Per CLAUDE.md §Schema-Defined Proof Obligations, each must be able to **fail loudly** under the
|
||||
violation it names:
|
||||
|
||||
- **A-1 (determinism / ADR-0180 T-4 analog).** Same canonical bytes + same pack ⇒ byte-identical
|
||||
`(32,)`, across repeated calls, threads, and processes. Fails if any non-determinism (dict
|
||||
ordering, unpinned FIR, float reduction order) leaks in.
|
||||
- **A-2 (set-equality of merges / ADR-0180 T-1 analog).** A set of `AudioCompilationUnit`s folds
|
||||
to the same Vault state regardless of arena flush order (permutation invariance). Fails if a
|
||||
delta's contribution is order-sensitive at the merge layer.
|
||||
- **A-3 (content-addressed key / ADR-0180 T-2 analog).** Trace-hash over audio deltas is
|
||||
invariant under set-equal Vault states when keyed by `(canonical_sha256, ir_sha256,
|
||||
projection_sha256)`. Fails if the reduction consumes deltas in arrival order.
|
||||
- **A-4 (serialization barrier).** In-chunk `compile_events` is asserted order-sensitive
|
||||
(negative test): swapping two events in canonical order changes the versor. This guards the
|
||||
barrier in §2.1 from silently becoming commutative and masking a real ordering bug.
|
||||
- **A-5 (versor condition).** Every emitted unit satisfies `versor_condition(v) < 1e-6`; the
|
||||
threshold is never weakened to pass.
|
||||
- **A-6 (trace hygiene).** No raw waveform bytes appear in any `TurnEvent`/Vault record; only
|
||||
the three hashes + pack IDs + optional teacher provenance.
|
||||
|
||||
### 4.3 The strict compilation invariant
|
||||
|
||||
```text
|
||||
same canonical audio bytes
|
||||
+ same compiler version
|
||||
+ same pack manifest (incl. basis_version)
|
||||
+ same operator registry
|
||||
= same AudioIR
|
||||
= same versor
|
||||
= same projection hash
|
||||
= same CRDT merge key
|
||||
= identical post-merge Vault contribution (idempotent under re-ingest)
|
||||
```
|
||||
|
||||
The final clause is the ADR-0181 addition to the PDF's original invariant: determinism is not
|
||||
only replayability, it is **CRDT idempotence** — the property that makes audio safe to shard.
|
||||
|
||||
## 5. Alternatives Considered
|
||||
|
||||
- **Transcript-first cascade (Whisper as substrate).** Chunked, text-intermediate; discards the
|
||||
auditory structure (the "No." vs "No?" vs whispered "no" distinction) and produces no
|
||||
checksummable, content-addressed key. Rejected as substrate; retained as teacher.
|
||||
- **Embedding-first projector (CLAP).** Fast but opaque; cannot be replayed bit-for-bit and
|
||||
cannot supply ADR-0180 §1.5.3's content-addressed merge key. Rejected as substrate.
|
||||
- **Codec-token front end (EnCodec / Moshi-like).** Strategically interesting for future
|
||||
full-duplex output; poor first substrate for an epistemically explicit engine. Deferred to a
|
||||
shadow/output lane.
|
||||
- **Audio as a downstream cognition mutation.** Violates ADR-0013's no-core-mutation rule and
|
||||
ADR-0180's "adapters never write global state directly" rule. Rejected.
|
||||
|
||||
## 6. Cross-References
|
||||
|
||||
- ADR-0013 — projection boundary; no-core-mutation constraint.
|
||||
- ADR-0180 — Delta-CRDT substrate; §1.5.2 order-invariance constraint (closed here for audio),
|
||||
§1.5.3 content-addressed merge key, §1.5.4 T-1…T-4 (audio analogs in §4.2), §1.5.5 trace
|
||||
hygiene & no hidden background execution.
|
||||
- ADR-0054 — Vault recall indexing/batching; the read-side contract the merged audio deltas must
|
||||
preserve (exact CGA recall).
|
||||
- CLAUDE.md §Normalization Rules — quantization/FIR confined to pack/compiler construction;
|
||||
`unitize_versor` algebra-owned; no hot-path repair.
|
||||
- `sensorium/protocol.py`, `sensorium/registry.py` — the `ProjectionHead`/`ModalityPack`/
|
||||
`ModalityRegistry` contracts this ADR implements an audio instance of.
|
||||
144
docs/plans/audio-compiler-eval-plan.md
Normal file
144
docs/plans/audio-compiler-eval-plan.md
Normal file
|
|
@ -0,0 +1,144 @@
|
|||
# Audio Compiler Eval Plan — `audio_core_v1`
|
||||
|
||||
**Companion to:** [ADR-0181](../decisions/ADR-0181-audio-compiler-delta-crdt.md),
|
||||
[audio-compiler-spec.md](./audio-compiler-spec.md)
|
||||
**Status:** Proposed (PR-1 docs)
|
||||
|
||||
This plan defines the seeding corpus, the acceptance gates that lift `audio_core_v1` from
|
||||
gate-closed to gate-engaged, the Delta-CRDT proof obligations, and the teacher-migration policy.
|
||||
|
||||
---
|
||||
|
||||
## 1. Seeding corpus — auditory atoms, not transcripts
|
||||
|
||||
Small, curated, checksum-locked. Four tiers:
|
||||
|
||||
- **Tier A — speech & turn atoms:** silence, short/long pause, voiced/unvoiced speech, onset,
|
||||
offset, interruption, overlap, rising/falling contour, emphatic energy.
|
||||
- **Tier B — prosody & affect:** calm, urgent, uncertain, whispered, shouted, sorrowful, joyful,
|
||||
irritated, sarcastic-like contours where reliably labelable.
|
||||
- **Tier C — non-speech events:** laughter, sigh, cry, alarm, knock, impact, keyboard, water,
|
||||
traffic, animal call, music bed, broadband noise.
|
||||
- **Tier D — alignment anchors:** spoken phrase, timestamp span, transcript hypothesis,
|
||||
speaker/channel metadata, linked text surface (only when alignment is trustworthy).
|
||||
|
||||
Tier D is auxiliary; lexical semantics is a later enrichment step. Synthetic fixtures (sine
|
||||
bursts, chirps, impulses, periodic voiced surrogates, silence, controlled overlaps) drive
|
||||
first-pass determinism; checksum-locked real fixtures cover what synthesis does poorly
|
||||
(laughter, sigh, whisper, interruption, yes/no rising-pitch questions).
|
||||
|
||||
## 2. Acceptance gates (gate-engaged criteria)
|
||||
|
||||
| Gate | Pass criterion |
|
||||
|---|---|
|
||||
| Projection shape | exactly `(32,)` |
|
||||
| Projection dtype | exactly `float32` |
|
||||
| Compiler replay | bit-identical on same platform/build |
|
||||
| Cross-platform stability | equal after quantization, within declared numeric tolerance |
|
||||
| `versor_condition` | `< 1e-6` (never weakened) |
|
||||
| Canonical checksum stability | 100% on fixture corpus |
|
||||
| Gate closure | projection blocked when `gate_engaged = false` |
|
||||
| Mount validation | bad checksum or bad unitarity blocks pack mount |
|
||||
| Trace hygiene | no raw waveform bytes in any turn trace |
|
||||
| IR replay | `AudioIR -> versor` replays identically from stored IR |
|
||||
|
||||
These mirror the existing modality-test posture (`sensorium/registry.py` mount/gate enforcement,
|
||||
`ModalityPack.__post_init__` invariants).
|
||||
|
||||
## 3. Delta-CRDT proof obligations (ADR-0181 §4.2)
|
||||
|
||||
Each must be able to **fail loudly** under the violation it names (CLAUDE.md §Schema-Defined
|
||||
Proof Obligations — no decorative tests):
|
||||
|
||||
| ID | Obligation | Fails if… | ADR-0180 analog |
|
||||
|---|---|---|---|
|
||||
| **A-1** | Determinism: same bytes + pack ⇒ byte-identical `(32,)` across calls/threads/processes | dict ordering, unpinned FIR, or float reduction order leaks in | T-4 |
|
||||
| **A-2** | Set-equality of merges: a set of units folds to the same Vault state for any arena flush permutation | a delta's contribution is order-sensitive at the merge layer | T-1 |
|
||||
| **A-3** | Content-addressed trace-hash: invariant under set-equal Vault states when keyed by `(canonical, ir, projection)` sha | the reduction consumes deltas in arrival order | T-2 |
|
||||
| **A-4** | Serialization barrier: in-chunk `compile_events` is order-sensitive (negative test) | swapping two canonical-order events fails to change the versor | T-3 |
|
||||
| **A-5** | `versor_condition < 1e-6` on every emitted unit | the threshold is weakened to pass | — |
|
||||
| **A-6** | Trace hygiene: no PCM in any `TurnEvent`/Vault record | raw waveform leaks into a delta's provenance | §1.5.5 |
|
||||
|
||||
### 3.1 The sequential==concurrent proof (PR-5 acceptance)
|
||||
|
||||
The load-bearing test for the CRDT mapping:
|
||||
|
||||
```text
|
||||
ingest fixtures [c1, c2, c3] sequentially → vault_seq, trace_hash_seq
|
||||
ingest the same fixtures across N arenas merged → vault_conc, trace_hash_conc
|
||||
assert set(vault_seq) == set(vault_conc) # A-2
|
||||
assert trace_hash_seq == trace_hash_conc # A-3
|
||||
assert re-ingesting any fixture is a no-op on the vault # idempotence (ADR-0181 §4.3)
|
||||
```
|
||||
|
||||
This is the audio instance of ADR-0180 §4.3's `hash(Sequential_Ingest) ==
|
||||
hash(Concurrent_CRDT_Ingest)` and must pass before PR-5 merges.
|
||||
|
||||
### 3.2 Pytest skeleton
|
||||
|
||||
```python
|
||||
def test_audio_projection_is_deterministic(audio_fixture, audio_pack): # A-1
|
||||
v1 = audio_pack.projection.project(audio_fixture)
|
||||
v2 = audio_pack.projection.project(audio_fixture)
|
||||
assert v1.shape == (32,)
|
||||
assert v1.dtype.name == "float32"
|
||||
assert np.array_equal(v1, v2)
|
||||
|
||||
def test_audio_pack_gate_blocks_projection(audio_pack_closed, audio_fixture): # gate closure
|
||||
with pytest.raises(Exception):
|
||||
_ = audio_pack_closed.projection.project(audio_fixture)
|
||||
|
||||
def test_audio_ir_replay_matches_original(audio_fixture, compiler): # IR replay
|
||||
unit = compiler.compile(audio_fixture)
|
||||
replay = compiler.compile_ir(unit.audio_ir)
|
||||
assert np.array_equal(unit.versor, replay.versor)
|
||||
assert unit.ir_sha256 == replay.ir_sha256
|
||||
|
||||
def test_chunk_composition_is_order_sensitive(two_events, compiler): # A-4 barrier
|
||||
a = compiler.compile_events([two_events[0], two_events[1]])
|
||||
b = compiler.compile_events([two_events[1], two_events[0]])
|
||||
assert not np.array_equal(a, b)
|
||||
|
||||
def test_merge_is_permutation_invariant(units): # A-2
|
||||
import itertools, random
|
||||
states = {fold_into_vault(p) for p in itertools.islice(_perms(units), 8)}
|
||||
assert len(states) == 1
|
||||
```
|
||||
|
||||
## 4. Teacher / shadow lane policy (PR-6)
|
||||
|
||||
Teachers label or align; they **never** define the substrate and **never** fold embeddings into
|
||||
the main versor path. They are admitted only through typed, versioned, checksummed hints stored
|
||||
as `content_anchors` / `evidence_ids` in the IR.
|
||||
|
||||
| Source | Best role in CORE | Why not the substrate |
|
||||
|---|---|---|
|
||||
| **Whisper** | offline transcript evidence, weak lexical labels, language ID | chunked, text-intermediate; loses native auditory structure |
|
||||
| **NeMo Parakeet** | timestamp/alignment teacher, live evidence lane | ASR, not a CORE-native auditory compiler |
|
||||
| **NeMo Canary** | streaming multilingual transcript/translation evidence | useful evidence, wrong primary ontology |
|
||||
| **CLAP** | coarse sound-event labels, audio-text alignment prototypes | embedding model; opaque latent violates the design goal |
|
||||
| **EnCodec** | reconstruction shadow lane, transport, future speech-to-speech output | codec tokens ≠ lawful typed auditory operators |
|
||||
| **Moshi** | latency / full-duplex reference target | codec-token speech modeling, not deterministic IR→versor |
|
||||
| **openSMILE / eGeMAPS** | reference feature catalog / offline oracle | OSS build is **non-commercial**; reference only, never a runtime dep unless licensing cleared |
|
||||
|
||||
Migration policy, verbatim:
|
||||
|
||||
```text
|
||||
Use teachers to label or align.
|
||||
Never let teachers define the substrate.
|
||||
Never fold teacher embeddings directly into the main versor path.
|
||||
Only admit teacher outputs through typed, versioned, checksumed hints.
|
||||
```
|
||||
|
||||
## 5. Phased sequence (priority order, no calendar)
|
||||
|
||||
1. **Doctrine** — ADR + spec + eval plan locked (PR-1, this).
|
||||
2. **Deterministic substrate** — canonicalizer, checksums, resample, lexer, parser, operators,
|
||||
compiler (PR-2).
|
||||
3. **Governance** — pack artifacts, adapter, mount/gate/checksum tests (PR-3).
|
||||
4. **Evaluation** — fixtures, expected IR, expected projection hashes, gate table (PR-4).
|
||||
5. **Delta-CRDT wiring** — arena + merge key + sequential==concurrent proof (PR-5), gated on
|
||||
ADR-0180 §1.5.4 (T-1…T-4) green on `main`.
|
||||
6. **Auxiliary lanes** — Whisper/NeMo/CLAP/EnCodec teachers behind optional extras (PR-6).
|
||||
7. **Streaming** — stateful incremental compiler preserving continuity across chunk seams
|
||||
(later; v1 is offline/whole-buffer).
|
||||
321
docs/plans/audio-compiler-spec.md
Normal file
321
docs/plans/audio-compiler-spec.md
Normal file
|
|
@ -0,0 +1,321 @@
|
|||
# Audio Compiler Spec — `audio_core_v1`
|
||||
|
||||
**Companion to:** [ADR-0181](../decisions/ADR-0181-audio-compiler-delta-crdt.md)
|
||||
**Status:** Proposed (PR-1 docs)
|
||||
**Scope:** the deterministic substrate (PR-2/PR-3) and its Delta-CRDT delta interface (PR-5).
|
||||
|
||||
This spec fixes the typed intermediate representation, the operator/manifest format, the numeric
|
||||
determinism rules, and the `AudioCompilationUnit` → Delta-CRDT delta contract. It is
|
||||
implementation-facing; the *why* lives in ADR-0181, the *acceptance* lives in the eval plan.
|
||||
|
||||
---
|
||||
|
||||
## 1. Two-clock architecture
|
||||
|
||||
A low-level **acoustic clock** measures signal facts; a higher-level **auditory grammar clock**
|
||||
emits typed events. The primary path is fully deterministic; learned systems are confined to
|
||||
auxiliary evidence lanes (PR-6).
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Waveform bytes / live stream] --> B[Canonicalizer<br/>mono + fixed SR + checksums]
|
||||
B --> C[Frame grid<br/>20 ms window / 10 ms hop]
|
||||
C --> D[Acoustic lexer<br/>energy, voicing, onset,<br/>pitch candidates, spectral bands, pauses]
|
||||
D --> E[Typed AudioIR parser<br/>speech/pause spans, prosody arcs,<br/>turn/overlap events, non-speech atoms, anchors]
|
||||
E --> F[Canonical ordering<br/>quantization + stable serialization]
|
||||
F --> G[Operator registry<br/>pack manifest + blade aliases + theta rules]
|
||||
G --> H[Rotor lowering]
|
||||
H --> I[Versor composition<br/>unitize_versor + versor_condition]
|
||||
I --> J["(32,) float32 — one AudioCompilationUnit"]
|
||||
E --> K[Audio evidence trace<br/>hashes, teacher provenance, pack IDs]
|
||||
J --> L[Thread-local arena<br/>ADR-0180 §2.1]
|
||||
L --> M[Semilattice merge<br/>keyed by content-addressed sha]
|
||||
```
|
||||
|
||||
## 2. Typed AudioIR
|
||||
|
||||
The IR is built from **typed spans and events**, never from raw frames or mel bins. Transcript
|
||||
anchors may exist only as auxiliary content hypotheses, never as the sole meaning of the audio.
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
import numpy as np
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AudioSignal:
|
||||
samples: np.ndarray # canonical mono float32
|
||||
sample_rate: int # 24_000
|
||||
start_ms: int
|
||||
end_ms: int
|
||||
source_sha256: str
|
||||
canonical_sha256: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PitchCandidate:
|
||||
cents_q: int # quantized cents
|
||||
prob_q: int # 0..255
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AudioToken:
|
||||
kind: Literal[
|
||||
"silence", "voiced", "unvoiced", "onset",
|
||||
"energy_bin", "pitch_candidates", "spectral_bin",
|
||||
]
|
||||
start_hop: int
|
||||
end_hop: int
|
||||
value_q: tuple[int, ...] # canonical quantized payload
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AuditoryEvent:
|
||||
event_type: str
|
||||
start_hop: int
|
||||
end_hop: int
|
||||
attrs: tuple[tuple[str, int | str], ...]
|
||||
evidence_ids: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AudioIR:
|
||||
speech_spans: tuple[AuditoryEvent, ...]
|
||||
pause_spans: tuple[AuditoryEvent, ...]
|
||||
prosody_arcs: tuple[AuditoryEvent, ...]
|
||||
turn_events: tuple[AuditoryEvent, ...]
|
||||
non_speech_events: tuple[AuditoryEvent, ...]
|
||||
content_anchors: tuple[AuditoryEvent, ...]
|
||||
ir_sha256: str
|
||||
```
|
||||
|
||||
### 2.1 The compilation unit (the CRDT delta)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AudioCompilationUnit:
|
||||
canonical_sha256: str
|
||||
ir_sha256: str
|
||||
pack_id: str
|
||||
pack_manifest_sha256: str
|
||||
projection_sha256: str
|
||||
versor: np.ndarray # (32,) float32
|
||||
versor_condition: float
|
||||
|
||||
@property
|
||||
def merge_key(self) -> tuple[str, str, str]:
|
||||
# ADR-0181 §2.2 — content-addressed CRDT merge / dedup key.
|
||||
return (self.canonical_sha256, self.ir_sha256, self.projection_sha256)
|
||||
```
|
||||
|
||||
`AudioCompilationUnit` is the single object the audio adapter writes into its thread-local arena
|
||||
(ADR-0180 §2.1). It carries no PCM (ADR-0181 §3.1 / ADR-0180 §1.5.5).
|
||||
|
||||
## 3. Canonical signal formation
|
||||
|
||||
- Internal processing: **mono, 24 kHz, float**; original-source bytes preserved separately for
|
||||
provenance; a derived 16 kHz stream is produced **only** for teacher ASR (PR-6).
|
||||
- Resampling: **pinned polyphase FIR** (SciPy `resample_poly` semantics — zero-phase,
|
||||
odd-length symmetric filter). The FIR coefficients are generated **once**, stored as a pack
|
||||
artifact (`resample_fir_v1.npy`), and checksummed in the manifest. The runtime never relies on
|
||||
library defaults.
|
||||
|
||||
## 4. Acoustic lexer
|
||||
|
||||
Operates on **measured facts**, not semantic guesses. Default frame 20 ms / hop 10 ms. Each
|
||||
frame yields quantized descriptors: RMS/log-energy bin, voiced/unvoiced flag, candidate F0 set
|
||||
(pYIN-style: multiple candidates with probabilities **before** Viterbi smoothing), onset
|
||||
strength bin, coarse spectral centroid/tilt bin, zero-crossing regime, pause classification.
|
||||
|
||||
## 5. Parser → typed events
|
||||
|
||||
Promotes lexer output into typed spans/events. Preserves the distinction between "No.", "No?",
|
||||
shouted "No!", whispered "no", and silent hesitation. Non-speech atoms (laughter, alarm, impact,
|
||||
music, broadband noise) are first-class; "chaotic noise" is the fallback only when a more
|
||||
specific parse is impossible.
|
||||
|
||||
## 6. Operator registry (pack-local blade aliases)
|
||||
|
||||
Because the `(32,)` boundary is fixed but no canonical *semantic* blade map is exposed, v1 uses
|
||||
**pack-local, versioned, checksummed blade aliases**. v1 uses **elliptic bivector operators
|
||||
only** (square = −1), so every rotor uses the numerically well-behaved
|
||||
`R = cos(θ/2) + B·sin(θ/2)`. Hyperbolic/boost-like operators are deferred.
|
||||
|
||||
| Auditory atom family | Measured source | Alias | Default blade index | Theta rule |
|
||||
|---|---|---|---|---|
|
||||
| Speech present | voiced frames / harmonic ratio | `B_SPEECH` | 6 | `q(base + g1·voiced_ratio_q)` |
|
||||
| Short pause | pause duration | `B_PAUSE_SHORT` | 7 | `q(base + g2·dur_hops)` |
|
||||
| Long pause | pause duration | `B_PAUSE_LONG` | 8 | `q(base + g3·dur_hops)` |
|
||||
| Rising final contour | final F0 slope | `B_PITCH_RISE` | 9 | `q(base + g4·slope_q)` |
|
||||
| Falling final contour | final F0 slope | `B_PITCH_FALL` | 10 | `q(base + g5·abs(slope_q))` |
|
||||
| Emphasis / force | energy delta | `B_EMPHASIS` | 11 | `q(base + g6·delta_db_q)` |
|
||||
| Hesitation / uncertainty | filled pause + low-conf contour | `B_HESITATION` | 12 | `q(base + g7·hesitation_q)` |
|
||||
| Turn boundary | silent gap + local reset | `B_TURN` | 13 | `q(base + g8·boundary_q)` |
|
||||
| Overlap / interruption | simultaneous speech / abrupt cut | `B_OVERLAP` | 14 | `q(base + g9·overlap_q)` |
|
||||
| Alert-like event | salience / alarm morphology | `B_ALERT` | 15 | `q(base + g10·salience_q)` |
|
||||
| Laughter | periodic burst pattern | `B_LAUGH` | 16 | `q(base + g11·laugh_q)` |
|
||||
| Cry / distress | voicing + modulation profile | `B_DISTRESS` | 17 | `q(base + g12·distress_q)` |
|
||||
| Music / tonal bed | stable harmonic bed | `B_MUSIC` | 18 | `q(base + g13·tonal_q)` |
|
||||
| Chaotic broadband noise | flat/noisy spectrum | `B_NOISE` | 19 | `q(base + g14·noise_q)` |
|
||||
|
||||
Indices are **reasonable defaults, not metaphysical claims** about Cl(4,1). The contract is that
|
||||
the mapping is explicit, versioned, checksummed, and frozen in the manifest. `B_OVERLAP` and
|
||||
`B_TURN` are the atoms that motivate per-stream arenas in ADR-0181 §2.3.
|
||||
|
||||
### 6.1 Minimal manifest (`packs/audio/audio_core_v1/manifest.toml`)
|
||||
|
||||
```toml
|
||||
pack_id = "audio_core_v1"
|
||||
modality = "audio"
|
||||
cl41_dim = 32
|
||||
compiler_version = "0.1.0"
|
||||
basis_version = "audio-basis-v1"
|
||||
|
||||
[canonical]
|
||||
sample_rate = 24000
|
||||
channels = 1
|
||||
frame_ms = 20
|
||||
hop_ms = 10
|
||||
output_dtype = "float32"
|
||||
internal_dtype = "float64"
|
||||
|
||||
[resampling]
|
||||
algorithm = "polyphase_fir"
|
||||
fir_path = "resample_fir_v1.npy"
|
||||
fir_sha256 = "sha256:REPLACE_ME"
|
||||
padtype = "constant"
|
||||
cval = 0.0
|
||||
|
||||
[gating]
|
||||
gate_engaged = false
|
||||
checksum_verified = false
|
||||
versor_condition_max = 1.0e-6
|
||||
|
||||
[ordering]
|
||||
event_precedence = ["channel", "pause", "speech", "prosody", "turn", "non_speech", "content_anchor"]
|
||||
```
|
||||
|
||||
### 6.2 Operator row (`operators.jsonl`)
|
||||
|
||||
```json
|
||||
{
|
||||
"operator_id": "audio.prosody.question_contour.v1",
|
||||
"event_type": "prosody.question_contour",
|
||||
"blade_alias": "B_PITCH_RISE",
|
||||
"blade_index": 9,
|
||||
"rotor_kind": "elliptic",
|
||||
"base_theta_q": 64,
|
||||
"gain_rules": {"slope_q": 3, "final_energy_q": 1, "confidence_q": 1},
|
||||
"theta_clip_q": 384,
|
||||
"version": "1"
|
||||
}
|
||||
```
|
||||
|
||||
## 7. Numeric determinism
|
||||
|
||||
Rule: **quantize before semantics, normalize after composition.** Raw float measurements are too
|
||||
unstable to hash. Quantization regime (frozen in manifest): boundaries in hop units, log energy
|
||||
in 1 dB bins, F0 in 25-cent bins, pitch slope in coarse cents-per-100 ms bins, spectral shape in
|
||||
fixed ordinal bins, all confidences in uint8. After quantization, compute in float64, compose
|
||||
sparse rotors in canonical order, call algebra-owned `unitize_versor`, cast to float32 **only**
|
||||
at the output boundary.
|
||||
|
||||
```python
|
||||
import math
|
||||
import numpy as np
|
||||
|
||||
EPS = 1e-12
|
||||
|
||||
def quantize_theta(theta: float, step: float = 1.0 / 1024.0) -> float:
|
||||
return round(theta / step) * step
|
||||
|
||||
def build_elliptic_rotor(blade_index: int, theta: float) -> np.ndarray:
|
||||
out = np.zeros(32, dtype=np.float64)
|
||||
half = quantize_theta(theta) / 2.0
|
||||
out[0] = math.cos(half)
|
||||
out[blade_index] = math.sin(half)
|
||||
return out
|
||||
|
||||
def compile_events(events, registry, geometric_product, unitize_versor, versor_condition):
|
||||
# SERIALIZATION BARRIER (ADR-0181 §2.1): in-chunk composition is order-sensitive,
|
||||
# single-threaded, canonical order. The substrate never parallelizes this loop.
|
||||
v = np.zeros(32, dtype=np.float64)
|
||||
v[0] = 1.0
|
||||
for ev in events: # must already be in canonical order
|
||||
spec = registry[ev.event_type]
|
||||
theta = spec.theta_from_event(ev) # deterministic, quantized inputs only
|
||||
r = build_elliptic_rotor(spec.blade_index, theta)
|
||||
v = geometric_product(v, r)
|
||||
v = unitize_versor(v)
|
||||
if versor_condition(v) >= 1e-6:
|
||||
raise ValueError("audio compilation failed versor check")
|
||||
return v.astype(np.float32)
|
||||
```
|
||||
|
||||
`geometric_product`, `unitize_versor`, `versor_condition` are imported from `algebra/` — the
|
||||
audio compiler adds **no** new normalization function.
|
||||
|
||||
## 8. Repo-facing adapter (`sensorium/adapters/audio.py`)
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
import numpy as np
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AudioProjectionHead:
|
||||
compiler: "AudioCompiler"
|
||||
modality = ... # Modality.AUDIO
|
||||
|
||||
@property
|
||||
def embedding_dim(self) -> int:
|
||||
return 32
|
||||
|
||||
def project(self, signal: "AudioSignal") -> np.ndarray:
|
||||
unit = self.compiler.compile(signal)
|
||||
out = unit.versor
|
||||
if out.shape != (32,):
|
||||
raise ValueError(f"expected (32,), got {out.shape}")
|
||||
if out.dtype != np.float32:
|
||||
raise TypeError(f"expected float32, got {out.dtype}")
|
||||
return out
|
||||
|
||||
def project_batch(self, signals: list["AudioSignal"]) -> np.ndarray:
|
||||
return np.stack([self.project(s) for s in signals], axis=0)
|
||||
|
||||
def verify_unitarity(self, signal: "AudioSignal") -> bool:
|
||||
return self.compiler.compile(signal).versor_condition < 1e-6
|
||||
```
|
||||
|
||||
The adapter is thin and pack-governed; it satisfies the `ProjectionHead` protocol in
|
||||
`sensorium/protocol.py` and is mounted as a `ModalityPack(modality_type=Modality.AUDIO,
|
||||
gate_engaged=False)` until the eval gates pass.
|
||||
|
||||
## 9. Delta-CRDT delta interface (PR-5)
|
||||
|
||||
The audio adapter **never** writes the global `epistemic_state` (ADR-0180 §2.1). Instead:
|
||||
|
||||
1. `compile()` produces one `AudioCompilationUnit` per canonical chunk (the serialization
|
||||
barrier of §7 runs here).
|
||||
2. The unit is written lock-free into the adapter's **thread-local arena**. Concurrent streams
|
||||
(overlap/interruption) each have their own arena (ADR-0181 §2.3).
|
||||
3. The **Merge Kernel** (ADR-0180 §2.2, an explicitly-mounted component, not a daemon — ADR-0180
|
||||
§1.5.5) folds pending units into the global Vault ordered by `unit.merge_key`. Duplicate keys
|
||||
are deduplicated (idempotence).
|
||||
4. The kernel surfaces its pending-delta count in `TurnEvent` for replay evidence (ADR-0180
|
||||
§1.5.5).
|
||||
|
||||
The per-chunk Vault contribution is `(versor, provenance)` where provenance =
|
||||
`{merge_key, pack_id, pack_manifest_sha256}` — content-addressed, no PCM.
|
||||
|
||||
## 10. File plan (PR-2 … PR-6)
|
||||
|
||||
```text
|
||||
sensorium/audio/{__init__,types,canonical,checksum,resample,frames,lexer,parser,operators,compiler,trace,fixtures,teachers}.py
|
||||
sensorium/adapters/audio.py
|
||||
packs/audio/audio_core_v1/{manifest.toml,basis_map.json,operators.jsonl,atoms.jsonl,prototypes.jsonl,resample_fir_v1.npy,checksums.json}
|
||||
tests/test_audio_{signal,resample,lexer,parser,compiler,pack_manifest,sensorium_mount,trace,crdt_delta}.py
|
||||
evals/audio_sensorium/{fixtures/*.wav,manifest.json,expected_ir.jsonl,expected_projection_hashes.json}
|
||||
```
|
||||
Loading…
Reference in a new issue