core/docs/plans/audio-compiler-eval-plan.md

Audio Compiler Eval Plan — audio_core_v1

Companion to: ADR-0181, audio-compiler-spec.md Status: Proposed (PR-1 docs)

This plan defines the seeding corpus, the acceptance gates that lift audio_core_v1 from gate-closed to gate-engaged, the Delta-CRDT proof obligations, and the teacher-migration policy.

---

1. Seeding corpus — auditory atoms, not transcripts

Small, curated, checksum-locked. Four tiers:

• Tier A — speech & turn atoms: silence, short/long pause, voiced/unvoiced speech, onset, offset, interruption, overlap, rising/falling contour, emphatic energy.
• Tier B — prosody & affect: calm, urgent, uncertain, whispered, shouted, sorrowful, joyful, irritated, sarcastic-like contours where reliably labelable.
• Tier C — non-speech events: laughter, sigh, cry, alarm, knock, impact, keyboard, water, traffic, animal call, music bed, broadband noise.
• Tier D — alignment anchors: spoken phrase, timestamp span, transcript hypothesis, speaker/channel metadata, linked text surface (only when alignment is trustworthy).

Tier D is auxiliary; lexical semantics is a later enrichment step. Synthetic fixtures (sine bursts, chirps, impulses, periodic voiced surrogates, silence, controlled overlaps) drive first-pass determinism; checksum-locked real fixtures cover what synthesis does poorly (laughter, sigh, whisper, interruption, yes/no rising-pitch questions).

2. Acceptance gates (gate-engaged criteria)

Gate	Pass criterion
Projection shape	exactly (32,)
Projection dtype	exactly float32
Compiler replay	bit-identical on same platform/build
Cross-platform stability	equal after quantization, within declared numeric tolerance
versor_condition	< 1e-6 (never weakened)
Canonical checksum stability	100% on fixture corpus
Gate closure	projection blocked when gate_engaged = false
Mount validation	bad checksum or bad unitarity blocks pack mount
Trace hygiene	no raw waveform bytes in any turn trace
IR replay	AudioIR -> versor replays identically from stored IR

These mirror the existing modality-test posture (sensorium/registry.py mount/gate enforcement, ModalityPack.__post_init__ invariants).

3. Delta-CRDT proof obligations (ADR-0181 §4.2)

Each must be able to fail loudly under the violation it names (CLAUDE.md §Schema-Defined Proof Obligations — no decorative tests):

ID	Obligation	Fails if…	ADR-0180 analog
A-1	Determinism: same bytes + pack ⇒ byte-identical (32,) across calls/threads/processes	dict ordering, unpinned FIR, or float reduction order leaks in	T-4
A-2	Set-equality of merges: a set of units folds to the same Vault state for any arena flush permutation	a delta's contribution is order-sensitive at the merge layer	T-1
A-3	Content-addressed trace-hash: invariant under set-equal Vault states when keyed by (canonical, ir, projection) sha	the reduction consumes deltas in arrival order	T-2
A-4	Serialization barrier: in-chunk compile_events is order-sensitive (negative test)	swapping two canonical-order events fails to change the versor	T-3
A-5	versor_condition < 1e-6 on every emitted unit	the threshold is weakened to pass	—
A-6	Trace hygiene: no PCM in any TurnEvent/Vault record	raw waveform leaks into a delta's provenance	§1.5.5

3.1 The sequential==concurrent proof (PR-5 acceptance)

The load-bearing test for the CRDT mapping:

ingest fixtures [c1, c2, c3] sequentially      → vault_seq,  trace_hash_seq
ingest the same fixtures across N arenas merged → vault_conc, trace_hash_conc
assert set(vault_seq) == set(vault_conc)                       # A-2
assert trace_hash_seq == trace_hash_conc                       # A-3
assert re-ingesting any fixture is a no-op on the vault        # idempotence (ADR-0181 §4.3)

This is the audio instance of ADR-0180 §4.3's hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest) and must pass before PR-5 merges.

3.2 Pytest skeleton

def test_audio_projection_is_deterministic(audio_fixture, audio_pack):       # A-1
    v1 = audio_pack.projection.project(audio_fixture)
    v2 = audio_pack.projection.project(audio_fixture)
    assert v1.shape == (32,)
    assert v1.dtype.name == "float32"
    assert np.array_equal(v1, v2)

def test_audio_pack_gate_blocks_projection(audio_pack_closed, audio_fixture):  # gate closure
    with pytest.raises(Exception):
        _ = audio_pack_closed.projection.project(audio_fixture)

def test_audio_ir_replay_matches_original(audio_fixture, compiler):            # IR replay
    unit = compiler.compile(audio_fixture)
    replay = compiler.compile_ir(unit.audio_ir)
    assert np.array_equal(unit.versor, replay.versor)
    assert unit.ir_sha256 == replay.ir_sha256

def test_chunk_composition_is_order_sensitive(two_events, compiler):           # A-4 barrier
    a = compiler.compile_events([two_events[0], two_events[1]])
    b = compiler.compile_events([two_events[1], two_events[0]])
    assert not np.array_equal(a, b)

def test_merge_is_permutation_invariant(units):                                # A-2
    import itertools, random
    states = {fold_into_vault(p) for p in itertools.islice(_perms(units), 8)}
    assert len(states) == 1

4. Teacher / shadow lane policy (PR-6)

Teachers label or align; they never define the substrate and never fold embeddings into the main versor path. They are admitted only through typed, versioned, checksummed hints stored as content_anchors / evidence_ids in the IR.

Source	Best role in CORE	Why not the substrate
Whisper	offline transcript evidence, weak lexical labels, language ID	chunked, text-intermediate; loses native auditory structure
NeMo Parakeet	timestamp/alignment teacher, live evidence lane	ASR, not a CORE-native auditory compiler
NeMo Canary	streaming multilingual transcript/translation evidence	useful evidence, wrong primary ontology
CLAP	coarse sound-event labels, audio-text alignment prototypes	embedding model; opaque latent violates the design goal
EnCodec	reconstruction shadow lane, transport, future speech-to-speech output	codec tokens ≠ lawful typed auditory operators
Moshi	latency / full-duplex reference target	codec-token speech modeling, not deterministic IR→versor
openSMILE / eGeMAPS	reference feature catalog / offline oracle	OSS build is non-commercial; reference only, never a runtime dep unless licensing cleared

Migration policy, verbatim:

Use teachers to label or align.
Never let teachers define the substrate.
Never fold teacher embeddings directly into the main versor path.
Only admit teacher outputs through typed, versioned, checksumed hints.

5. Phased sequence (priority order, no calendar)

1. Doctrine — ADR + spec + eval plan locked (PR-1, this).
2. Deterministic substrate — canonicalizer, checksums, resample, lexer, parser, operators, compiler (PR-2).
3. Governance — pack artifacts, adapter, mount/gate/checksum tests (PR-3).
4. Evaluation — fixtures, expected IR, expected projection hashes, gate table (PR-4).
5. Delta-CRDT wiring — arena + merge key + sequential==concurrent proof (PR-5), gated on ADR-0180 §1.5.4 (T-1…T-4) green on main.
6. Auxiliary lanes — Whisper/NeMo/CLAP/EnCodec teachers behind optional extras (PR-6).
7. Streaming — stateful incremental compiler preserving continuity across chunk seams (later; v1 is offline/whole-buffer).