diff --git a/README.md b/README.md index 2db2e305..158beb88 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,20 @@ When facing a design decision, the world offers two visible options: use what al --- +## Native Substrate Direction — Python, Rust, Zig + +CORE is not moving toward a wholesale Zig rewrite. The architecture is moving toward a stricter native-substrate boundary: + +- **Python** remains the semantic source of truth: cognition runtime, teaching/review workflows, pack ratification, eval harnesses, and Workbench/operator tooling. +- **Rust** remains the incumbent native algebra backend: Cl(4,1) products, versor operations, CGA inner product, exact recall, and diffusion surfaces already proven by parity gates. +- **Zig** is a candidate material for the next native substrate layer: Delta-CRDT arenas/deltas/merge kernels, deterministic modality compilers such as `audio_core_v1`, stable C ABI surfaces, edge-native ingestion, and selected exact recall challenge kernels only after parity and benchmark proof. + +The rule is component law, not language preference. Zig may enter where explicit allocation, deterministic buffer ownership, C ABI clarity, and edge-native deployment materially strengthen CORE. Zig must not replace review-gated semantics, introduce approximate recall, hide repair in native code, or turn teacher/shadow models into substrate. + +Decision package: [`docs/zig/README.md`](docs/zig/README.md). Adoption gates: [`docs/zig/adoption-gates.md`](docs/zig/adoption-gates.md). + +--- + ## The Truth-Seeking Schema Co-equal with the algebraic substrate. CORE's epistemic schema is a foundational architectural commitment: every claim that enters the runtime field carries a typed position in a revision graph (`SPECULATIVE`, `COHERENT`, `CONTESTED`, `FALSIFIED`); coherence — not source authority — is the only admission signal; no claim is ever locked, even when COHERENT; identity cannot be rewritten by content; and exactly one mutation path admits knowledge, enforced by a CI-level architectural-invariant test. diff --git a/docs/decisions/ADR-0196-native-substrate-language-doctrine.md b/docs/decisions/ADR-0196-native-substrate-language-doctrine.md new file mode 100644 index 00000000..20b32e7e --- /dev/null +++ b/docs/decisions/ADR-0196-native-substrate-language-doctrine.md @@ -0,0 +1,114 @@ +# ADR-0196 — Native Substrate Language Doctrine (Python / Rust / Zig) + +**Status:** Accepted. +**Date:** 2026-05-31. + +## Decision + +CORE is **not** moving toward a wholesale Zig rewrite. This ADR ratifies the +component-by-component native-substrate doctrine recorded in +[`docs/zig/`](../zig/README.md) as binding architectural direction. + +The governing conclusion: + +- **Python** remains the semantic source of truth — cognition runtime, + teaching/review workflows, pack ratification, eval harnesses, + Workbench/operator tooling, and all fast-changing cognition semantics. +- **Rust** (`core-rs`) remains the incumbent native algebra backend — Cl(4,1) + geometric product, versor apply/closure, CGA inner product, exact recall, and + diffusion — already parity-gated and opt-in via `CORE_BACKEND=rust` + (`algebra/backend.py`). +- **Zig** is a *candidate* material for the Ring 1 native substrate layer only: + Delta-CRDT arenas/deltas/merge kernels, deterministic modality compilers + (e.g. `audio_core_v1`), stable C ABI surfaces, edge-native ingestion, and + selected exact-recall challenge kernels — and only after parity and benchmark + proof. + +The architectural question is therefore not *"should CORE be rewritten in +Zig?"* (the answer is no) but *"which substrate components require Zig's +explicit allocation, C ABI clarity, edge-native build story, and deterministic +buffer ownership strongly enough to justify a new native lane?"* + +## Ring architecture + +| Ring | Layer | Material | +|---|---|---| +| Ring 3 | Operator / Workbench / Review | Python + TypeScript | +| Ring 2 | Semantic cognition runtime | **Python — source of truth** | +| Ring 1 | Native substrate services | **Rust incumbent + Zig candidate** | +| Ring 0 | Hardware / memory substrate | CPU, UMA, future MLX/Metal | + +Zig belongs in Ring 1 where its properties matter. It does not belong in Ring 2 +merely because the project is getting large. + +## Adoption gates (binding) + +No Zig component is promoted without passing the G0–G8 ladder defined in +[`docs/zig/adoption-gates.md`](../zig/adoption-gates.md): + +```text +G0 doctrine +G1 reference contract locked +G2 prototype behind explicit selector +G3 parity proof +G4 determinism proof +G5 mechanical advantage proof +G6 operational proof +G7 supported backend +G8 default backend, only by later ADR +``` + +Default-by-availability is forbidden: a present Zig library must never change +runtime behavior. Backend selection is always explicit and trace-visible. + +## Decision rules (binding) + +1. **Python semantics lock first.** A Zig implementation must trail a locked + reference contract; it may accelerate or package behavior, never reinterpret + it. +2. **No approximate substrate.** No ANN/HNSW, cosine fallback, opaque + embeddings, hot-path repair, or arrival-order-dependent merge. +3. **Explicit ownership or no adoption.** Every component names input/output + buffers, allocator ownership, failure ownership, lifecycle, determinism + contract, and fallback. +4. **No hidden background work.** Native workers exist only as explicitly + mounted components with observable state. +5. **Component migration, not language migration.** Language count is not the + enemy; boundary ambiguity is. + +## Non-goals + +This ADR does **not** authorize any Zig implementation. It does not authorize +porting `chat.runtime`, teaching/review, pack ratification, the eval framework, +the Workbench API, identity/safety/ethics policy, or the natural-language +realizer to any native language. Those are semantic/governance layers where +Python remains the right material. + +## Consequences + +- The doctrine package under `docs/zig/**` is now binding. Future Zig PRs must + cite the gate they are clearing and the reference contract they trail. +- The first gate instantiation is the Delta-CRDT substrate's **G1 + reference-contract lock**, handled under **ADR-0180** (whose gate obligations + this ADR governs). That lock corresponds to the **ZC-0** slice in + [`docs/zig/crdt-substrate/implementation-slices.md`](../zig/crdt-substrate/implementation-slices.md). +- **ZC-1 and beyond (any Zig code) remain unauthorized** until ADR-0180's + reference contract is Accepted and a separate ADR clears gate G2. + +## Alternatives considered + +- **Wholesale Zig rewrite** — rejected. It would make CORE's semantic and + governance layers harder, not truer; it contradicts the *decoding, not + generating* thesis and the explicit *"not an infrastructure playground"* + stance. +- **Status quo (Rust-only, no doctrine)** — rejected. Leaving the + native-substrate boundary undefined risks ad-hoc Zig adoption later, when the + system is harder to re-materialize around the correct substrate. + +## References + +- [`docs/zig/README.md`](../zig/README.md) — decision package overview. +- [`docs/zig/adoption-gates.md`](../zig/adoption-gates.md) — G0–G8 proof ladder. +- [`docs/zig/core-native-system/README.md`](../zig/core-native-system/README.md) — ring architecture and native boundary. +- [`docs/zig/runtime-ffi/README.md`](../zig/runtime-ffi/README.md) — C ABI / backend selection / memory ownership. +- ADR-0180 (CRDT substrate · G1 lock), ADR-0181 (audio compiler), ADR-0019 / ADR-0020 / ADR-0054 (recall & algebra anchors). diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 95a4979f..708f2d69 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -239,6 +239,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0181](ADR-0181-audio-compiler-delta-crdt.md) | CORE-native Audio Compiler over the Delta-CRDT Substrate | Proposed | | [ADR-0182](ADR-0182-cross-composer-disagreement-pooling.md) | Cross-composer disagreement pooling: refuse distractor-quantity confusers without a reactive cue rule | Proposed (spec only — no code). Follow-on to | | [ADR-0183](ADR-0183-lawful-audio-lexeme-path.md) | Lawful Audio→Lexeme Path (stub) | Proposed (stub — placeholder to record the fork; not yet a full design) | +| [ADR-0196](ADR-0196-native-substrate-language-doctrine.md) | Native Substrate Language Doctrine (Python / Rust / Zig) | Accepted (2026-05-31) | --- diff --git a/docs/zig/README.md b/docs/zig/README.md new file mode 100644 index 00000000..3a213f35 --- /dev/null +++ b/docs/zig/README.md @@ -0,0 +1,169 @@ +# Zig Native Substrate Decision Package + +**Status:** Decision package / doctrine draft +**Scope:** `docs/zig/**` only +**Purpose:** Decide where Zig belongs in CORE before the system becomes harder to re-materialize around the correct native substrate. + +This directory does **not** authorize a wholesale rewrite of CORE in Zig. + +It establishes a component-by-component doctrine for when Zig is the right material, when Rust remains the right material, and when Python must remain the semantic source of truth. The goal is consistency without premature ossification: use the right tool where the physics of the component demands it, and refuse novelty where it would fragment the architecture. + +--- + +## Governing conclusion + +Zig is not the language for the whole CORE runtime. + +Zig is a strong candidate for the **native substrate layer**: + +```text +Native substrate: + - Delta-CRDT arena / delta / merge kernel + - modality compilers, starting with audio_core_v1 + - C ABI surfaces for edge/local/native runtime integration + - selected exact recall / batch recall kernels, if parity and benchmark gates prove advantage + - future no-Python edge ingestion components + +Python semantic runtime: + - pack authoring and ratification + - teaching / review / proposal workflows + - eval lanes and proof obligations + - Workbench API and operator tooling + - fast-changing cognition semantics + +Rust native backend: + - existing proven algebra hot paths + - PyO3/NumPy acceleration surfaces already parity-gated + - current `core-rs` kernels until Zig wins by evidence +``` + +The correct architectural question is therefore not: + +```text +Should CORE be rewritten in Zig? +``` + +It is: + +```text +Which substrate components require Zig's explicit allocation, C ABI clarity, +edge-native build story, and deterministic buffer ownership strongly enough to +justify a new native lane? +``` + +--- + +## Directory map + +| Directory | Component | Decision posture | +|---|---|---| +| [`crdt-substrate/`](crdt-substrate/) | Delta-CRDT arenas, deltas, merge kernel, content ordering | Zig is a first-class candidate. Prototype before replacing Rust. | +| [`audio-compiler/`](audio-compiler/) | Deterministic audio compiler, chunk units, checksums, native audio lexer | Zig is strongly preferred for long-term native substrate if Python spec is locked first. | +| [`core-native-system/`](core-native-system/) | Native CORE boundary and what belongs outside Python | Define a ring architecture; no full runtime rewrite. | +| [`runtime-ffi/`](runtime-ffi/) | C ABI, backend selection, memory ownership, error surfaces | Zig must enter through stable FFI, not through ad-hoc bindings. | +| [`algebra-kernels/`](algebra-kernels/) | Cl(4,1), recall, diffusion, batch recall | Rust remains incumbent; Zig may challenge only through parity and benchmark gates. | +| [`adoption-gates.md`](adoption-gates.md) | Cross-cutting proof gates | No Zig component is promoted without deterministic, semantic, and operational proof. | + +--- + +## Decision rules + +### Rule 1 — Python semantics lock first + +Any Zig implementation must trail a locked semantic contract. If Python currently defines the reference behavior, Zig may accelerate or package that behavior but may not reinterpret it. + +Zig work is invalid if it starts by asking the runtime to trust a new behavior because it is native. + +### Rule 2 — no approximate substrate + +Zig must not introduce approximate recall, embedding fallback, lossy indexing, or hidden normalization. Any native component must preserve CORE's exactness doctrine. + +Permitted: + +```text +same semantics, better mechanical substrate +same inputs, same outputs, lower latency +same deltas, stable content-addressed order +same compiler IR, stable projection hash +``` + +Forbidden: + +```text +approximate nearest neighbor +cosine fallback +opaque embedding substrate +hot-path repair +arrival-order dependent merge +runtime mutation hidden behind native worker threads +``` + +### Rule 3 — explicit ownership or no adoption + +Zig is only worth adding where explicit ownership matters. Every proposed Zig component must name: + +1. input buffers, +2. output buffers, +3. allocator ownership, +4. failure ownership, +5. lifecycle / free semantics, +6. determinism contract, +7. Python/Rust fallback behavior. + +### Rule 4 — no hidden background work + +Native workers may exist only as explicitly mounted runtime components with observable state. The merge kernel, audio compiler, and future native ingestion services must surface pending counts, errors, and replay evidence. There must be no invisible daemon that mutates truth-bearing state. + +### Rule 5 — component migration, not language migration + +CORE may become multi-native: + +```text +Python: source of semantic truth and proof workflows +Rust: incumbent acceleration backend +Zig: substrate candidate for CRDT/modality/edge-native surfaces +TypeScript/React: Workbench UI +``` + +This is acceptable only if each boundary is crisp. Language count is not the enemy; boundary ambiguity is. + +--- + +## Promotion statuses + +Use these statuses in future Zig docs and PRs: + +| Status | Meaning | +|---|---| +| `doctrine` | Decision only. No implementation authorized. | +| `prototype` | Experimental implementation allowed behind an explicit flag. | +| `parity-candidate` | Has reference parity tests; not default. | +| `bench-candidate` | Has parity and benchmark evidence; still explicit opt-in. | +| `supported-backend` | Supported but not necessarily default. | +| `default-backend` | May become default only after an ADR accepts replay, parity, fallback, ops, and CI evidence. | + +No document in `docs/zig/**` should silently jump a component beyond these statuses. + +--- + +## Current recommendation + +Priority order: + +1. **CRDT substrate Zig prototype** — arena, delta, join, merge, content hash. +2. **Audio compiler Zig substrate prototype** — canonical PCM, frame grid, deterministic lexer, AudioIR hash, projection hash boundary. +3. **Runtime FFI contract** — caller-owned buffers, typed errors, no hidden allocation surprises. +4. **Batch recall challenge** — optional Zig/Rust comparison for `vault_recall_batch` only after parity tests exist. +5. **Standalone edge-native ingestion runner** — future, after the above contracts stabilize. + +Non-priority: + +```text +chat/runtime.py rewrite +teaching/review rewrite +Workbench rewrite +eval harness rewrite +pack authoring rewrite +``` + +Those are semantic and governance-heavy surfaces; Zig would make them harder, not truer. diff --git a/docs/zig/adoption-gates.md b/docs/zig/adoption-gates.md new file mode 100644 index 00000000..cde37ff3 --- /dev/null +++ b/docs/zig/adoption-gates.md @@ -0,0 +1,182 @@ +# Zig Adoption Gates + +**Status:** doctrine +**Applies to:** all future Zig work in CORE + +Zig may enter CORE only through evidence. This page defines the proof ladder for any Zig component. + +--- + +## Gate ladder + +```text +G0 doctrine +G1 reference contract locked +G2 prototype behind explicit selector +G3 parity proof +G4 determinism proof +G5 mechanical advantage proof +G6 operational proof +G7 supported backend +G8 default backend, only by later ADR +``` + +No component skips a gate because it is small, native, or easy to write. + +--- + +## G0 — Doctrine + +Before implementation, document: + +- exact component boundary; +- current source of truth; +- why Zig is being considered; +- why Python or Rust may be insufficient for this boundary; +- non-goals; +- fallback path; +- proof obligations. + +The document must name the governing ADRs/specs. Current anchors include ADR-0013, ADR-0019, ADR-0020, ADR-0054, ADR-0180, and ADR-0181. + +--- + +## G1 — Reference contract locked + +A Zig implementation must trail a locked reference contract. + +Valid references: + +- tested Python behavior; +- tested Rust behavior; +- an ADR/spec/eval plan with executable acceptance gates; +- fixture corpus plus expected hashes. + +Invalid references: + +- performance hope; +- preference for a language; +- untested native behavior; +- implementation-first semantics. + +--- + +## G2 — Explicit prototype selector + +A Zig prototype must be unreachable unless deliberately selected. + +Acceptable selectors: + +```text +CORE_NATIVE_BACKEND=zig +CORE_CRDT_BACKEND=zig +CORE_AUDIO_COMPILER_BACKEND=zig +core doctor --zig +``` + +A local Zig library being present must not automatically alter runtime behavior. + +--- + +## G3 — Parity + +Parity means the Zig component preserves the reference contract. + +| Component | Required parity | +|---|---| +| CRDT merge | same canonical order, same dedup result, same merge hash for all input permutations | +| Audio compiler | same AudioIR hash, projection hash, merge key, and declared `(32,)` output contract | +| Algebra kernel | bit-identical result, or ADR-approved numeric tolerance | +| Batch recall | identical top-k indices, scores, and tie-break behavior | +| FFI boundary | same typed errors at Python surface | + +The parity gate must fail loudly if ordering, hashing, closure, or recall semantics drift. + +--- + +## G4 — Determinism + +A Zig component must prove repeatability across: + +- repeated calls; +- reordered inputs when order-invariance is claimed; +- concurrent calls when thread-safety is claimed; +- cold and warm startup; +- supported target platforms. + +CRDT minimum: + +```text +join(a, b) == join(b, a) +join(join(a, b), c) == join(a, join(b, c)) +join(a, a) == a +hash(merge(permutation(deltas))) == hash(merge(deltas)) +``` + +Audio minimum: + +```text +same canonical bytes ++ same compiler version ++ same pack manifest ++ same operator registry += same AudioIR += same projection hash += same CRDT merge key +``` + +--- + +## G5 — Mechanical advantage + +Zig must prove at least one real advantage: + +- clearer C ABI; +- simpler edge-native deployment; +- lower memory footprint; +- lower latency on representative workloads; +- explicit allocation/lifecycle control; +- reduced binding overhead; +- safer ownership at a native boundary. + +Performance data without parity does not count. + +--- + +## G6 — Operational proof + +A supported Zig backend needs: + +- build command; +- diagnostic command; +- backend selector; +- ABI version check; +- memory free contract; +- error contract; +- fallback behavior; +- CI or local verification command. + +--- + +## G7/G8 — Supported and default + +Supported backend status requires parity, determinism, operational proof, and documented advantage. + +Default backend status requires a separate ADR. Default-by-availability is not allowed. + +--- + +## Review checklist + +Every Zig PR must answer: + +1. What behavior is being ported or introduced? +2. Where is the reference contract? +3. How is Zig selected? +4. What is the fallback? +5. Who owns each buffer? +6. How are allocations released? +7. What errors are surfaced? +8. What proves deterministic replay? +9. What benchmark or deployment evidence justifies the component? +10. What test fails if semantics drift? diff --git a/docs/zig/algebra-kernels/README.md b/docs/zig/algebra-kernels/README.md new file mode 100644 index 00000000..0f73b86c --- /dev/null +++ b/docs/zig/algebra-kernels/README.md @@ -0,0 +1,152 @@ +# Zig Guidance — Algebra and Recall Kernels + +**Status:** doctrine / challenge candidate +**Component:** Cl(4,1), exact recall, batch recall, graph diffusion + +Rust is the incumbent native backend for algebra and exact recall. Zig may challenge specific kernels only through parity and benchmark evidence. + +--- + +## Current incumbent + +The current `core-rs` backend already covers: + +- Cl(4,1) geometric product; +- versor apply / closure path; +- CGA inner product; +- exact vault recall; +- exponential-map unitization; +- graph diffusion. + +It is built as a PyO3/NumPy extension, uses Rayon for parallel recall, and has parity tests across multiple algebra surfaces. + +Therefore, Zig should not start by replacing these surfaces. It should start by challenging uncovered or ABI-sensitive surfaces. + +--- + +## Best Zig algebra candidate: `vault_recall_batch` + +`vault_recall_batch` is currently Python-canonical and has no Rust binding. It is a good challenge kernel because: + +- the semantics are already precise; +- scoring is exact CGA diagonal-metric accumulation; +- top-k ordering is deterministic; +- batch input/output maps naturally to caller-owned buffers; +- C ABI may be cleaner than a PyO3-only path; +- performance can be measured without touching semantic runtime. + +### Candidate ABI + +```c +int core_recall_batch_f32( + const float* matrix, /* N x 32, row-major */ + unsigned long n, + const float* queries, /* B x 32, row-major */ + unsigned long b, + unsigned long top_k, + unsigned int* out_indices, /* B x top_k */ + float* out_scores, /* B x top_k */ + CoreNativeError* err +); +``` + +The caller owns output buffers. Zig writes into them. No allocation is required in the hot path except optional scratch passed explicitly by later ABI versions. + +--- + +## Required recall invariants + +Any Zig recall kernel must preserve: + +```text +exact CGA metric +component-serial accumulation if bit-identity requires it +descending score order +ascending index tie-break unless the store layer supplies a content-addressed order +top_k truncation semantics +empty input behavior +shape errors as typed failures +no approximate search +``` + +Forbidden: + +```text +cosine fallback +ANN/HNSW/FAISS-like index +learned projection +score normalization +hidden reordering +GPU nondeterminism without proof +``` + +--- + +## Geometric product / versor apply + +These are not first Zig targets. + +They may become candidates only if: + +- the Rust path becomes difficult to build/deploy; +- C ABI is needed by non-Python hosts; +- Zig produces equal behavior with simpler lifecycle; +- performance exceeds Rust on representative workloads; +- parity tests prove closure behavior exactly. + +The closure invariant remains: + +```text +versor_condition(F) < 1e-6 +``` + +No native path may hide violations by repair outside the approved closure boundary. + +--- + +## Graph diffusion + +Graph diffusion is a possible Zig candidate after CRDT and audio work. + +It is suitable because: + +- input/output buffers are regular arrays; +- edge lists are compact; +- memory ownership can be explicit; +- a C ABI could support edge-native runtimes. + +It is not urgent because the Rust implementation already removed major marshalling overhead through zero-copy NumPy views. + +--- + +## Unitize / expmap + +Unitization can be native, but it is dangerous to duplicate casually. Any Zig implementation must be tested against the existing algebra contract and audio compiler closure gates. + +Required: + +- same boost/rotation plane semantics; +- same f32/f64 strategy as the reference surface; +- same behavior on near-zero input; +- same post-condition threshold; +- no hidden fallback rotor unless the reference does the same. + +--- + +## Promotion sequence for any Zig algebra kernel + +1. Identify exact reference surface. +2. Add parity tests independent of Zig. +3. Add Zig implementation behind explicit selector. +4. Run parity fixtures. +5. Add benchmark fixtures. +6. Compare Rust, Python, and Zig. +7. Promote only if Zig wins a named mechanical criterion. + +--- + +## Decision posture + +For algebra kernels, Zig is a challenger, not the incumbent. + +The correct first Zig algebra target is not the full Cl(4,1) backend. It is `vault_recall_batch` or another uncovered, buffer-regular, ABI-friendly kernel. diff --git a/docs/zig/audio-compiler/README.md b/docs/zig/audio-compiler/README.md new file mode 100644 index 00000000..9743980d --- /dev/null +++ b/docs/zig/audio-compiler/README.md @@ -0,0 +1,297 @@ +# Zig Guidance — Audio Compiler + +**Status:** doctrine / prototype candidate +**Component:** `audio_core_v1` deterministic compiler substrate +**Primary governing ADR:** ADR-0181 + +The audio compiler is the second strongest Zig candidate after the CRDT substrate, and may become the best long-term fit once the Python reference spec is locked. + +Audio in CORE must not enter as an opaque embedding. It must enter as a deterministic compiler product: canonical waveform bytes become typed AudioIR, then lawful operators, then a `(32,) float32` versor plus content-addressed provenance. + +--- + +## Why this component is suitable for Zig + +The audio compiler is streaming, buffer-heavy, deterministic, and edge-native. That maps well to Zig. + +Zig is suitable for: + +- canonical PCM handling; +- fixed sample-rate frame grids; +- deterministic quantization; +- acoustic lexer passes; +- typed event serialization; +- checksum chain construction; +- pack-local read-only operator tables; +- chunk-to-compilation-unit emission; +- C ABI handoff to CRDT arenas; +- future no-Python streaming compilation. + +Zig is not suitable for replacing the teacher/shadow lanes. Whisper, NeMo, CLAP, EnCodec, and similar tools remain optional Python-side evidence lanes. They label or align; they never define the substrate. + +--- + +## Existing contract to preserve + +ADR-0181 requires: + +```text +waveform / live stream + -> canonicalizer + -> frame grid + -> acoustic lexer + -> typed AudioIR + -> operator registry + -> rotor lowering + -> versor composition + -> (32,) float32 AudioCompilationUnit +``` + +The chunk boundary is the Delta-CRDT delta boundary: + +```text +inside chunk: serial rotor composition, order-sensitive +across chunks: order-invariant CRDT deltas +``` + +The native compiler must never parallelize the non-commutative `compile_events` loop. It may parallelize independent chunks only after each chunk has its own serialization barrier. + +--- + +## What should be in Zig + +A Zig audio compiler lane should include: + +```text +core-zig/src/audio/signal.zig +core-zig/src/audio/canonical.zig +core-zig/src/audio/resample.zig +core-zig/src/audio/frames.zig +core-zig/src/audio/lexer.zig +core-zig/src/audio/ir.zig +core-zig/src/audio/operators.zig +core-zig/src/audio/project.zig +core-zig/src/audio/checksum.zig +core-zig/src/audio/ffi.zig +include/core_audio.h +``` + +### Canonicalizer + +Responsibilities: + +- accept caller-owned PCM-like input; +- downmix to mono if policy allows; +- resample to the declared rate using frozen coefficients; +- emit source and canonical hashes; +- produce canonical sample buffer with declared dtype and sample rate. + +Forbidden: + +- relying on platform resampler defaults; +- hidden model inference; +- lossy semantic classification; +- leaking raw PCM past the trace boundary. + +### Frame grid + +Responsibilities: + +- deterministic window length; +- deterministic hop length; +- deterministic zero padding; +- no wall-clock dependent chunking; +- explicit continuity state for streaming phase. + +### Acoustic lexer + +Responsibilities: + +- measure facts, not guess semantics; +- quantize before semantic promotion; +- emit integer-like token payloads; +- avoid unstable float hashes; +- keep all thresholds pack-owned and versioned. + +### AudioIR serialization + +Responsibilities: + +- canonical event order; +- stable field ordering; +- no map/dict iteration dependence; +- stable hash independent of allocation address; +- content anchors as evidence only. + +### Operator lowering + +Responsibilities: + +- read pack-local, checksummed blade aliases; +- use declared elliptic rotor rules for v1; +- quantize theta before rotor construction; +- compose in canonical event order; +- call algebra-owned `unitize_versor` semantics or an approved native equivalent; +- verify `versor_condition < 1e-6` without weakening. + +--- + +## AudioCompilationUnit contract + +The Zig compiler should produce a C-compatible equivalent of: + +```text +canonical_sha256 +ir_sha256 +pack_id +pack_manifest_sha256 +projection_sha256 +versor: [32]f32 +versor_condition +merge_key = (canonical_sha256, ir_sha256, projection_sha256) +``` + +The unit is the only audio object that may enter a CRDT arena. + +No raw PCM should appear in the Vault record or TurnEvent trace. Provenance is hash-based: + +```text +{ merge_key, pack_id, pack_manifest_sha256 } +``` + +--- + +## Minimum C ABI sketch + +```c +typedef struct CoreAudioCompiler CoreAudioCompiler; +typedef struct CoreAudioUnit CoreAudioUnit; + +typedef struct CoreAudioCompileInput { + const float* samples; + unsigned long sample_count; + unsigned int sample_rate; + unsigned int channels; + unsigned long start_ms; + unsigned long end_ms; +} CoreAudioCompileInput; + +CoreAudioCompiler* core_audio_compiler_open( + const char* manifest_path, + CoreAudioError* err +); + +void core_audio_compiler_free(CoreAudioCompiler* compiler); + +int core_audio_compile( + const CoreAudioCompiler* compiler, + const CoreAudioCompileInput* input, + CoreAudioUnit** out_unit, + CoreAudioError* err +); + +void core_audio_unit_free(CoreAudioUnit* unit); + +int core_audio_unit_get_versor( + const CoreAudioUnit* unit, + float out_versor[32], + CoreAudioError* err +); + +int core_audio_unit_get_merge_key( + const CoreAudioUnit* unit, + unsigned char out_canonical_sha256[32], + unsigned char out_ir_sha256[32], + unsigned char out_projection_sha256[32], + CoreAudioError* err +); +``` + +The final ABI must be ratified before implementation is treated as supported. + +--- + +## What should remain Python first + +The Python implementation/spec should remain first for: + +- IR schema design; +- fixture authoring; +- expected hash generation; +- pack manifest evolution; +- eval harness; +- teacher/shadow lanes; +- operator table experimentation; +- Workbench display. + +Zig should not become the place where audio ontology is invented. It should become the place where a locked ontology is compiled with mechanical discipline. + +--- + +## Required proof obligations + +### A-Z1 — Canonical determinism + +Same source bytes and same pack produce same canonical hash, frame tokens, IR hash, projection hash, and versor. + +### A-Z2 — Serialization barrier + +Swapping two order-sensitive in-chunk events must change the compiled versor. This negative test proves the compiler has not accidentally treated non-commutative composition as a CRDT merge. + +### A-Z3 — Merge-key idempotence + +Re-ingesting the same AudioCompilationUnit must be a no-op after CRDT merge. + +### A-Z4 — Trace hygiene + +No raw PCM bytes may appear past compiler-local buffers. + +### A-Z5 — Teacher isolation + +Teacher hints may affect content/evidence anchors, but must not mutate `projection_sha256` unless a separate ADR explicitly changes substrate policy. + +### A-Z6 — Closure invariant + +Every emitted unit must satisfy: + +```text +versor_condition(v) < 1e-6 +``` + +No threshold weakening is allowed. + +--- + +## Migration sequence + +1. Keep Python `sensorium/audio/**` as the reference implementation. +2. Freeze fixtures and expected hashes. +3. Build Zig canonicalizer + frame grid first. +4. Add lexer and IR serialization. +5. Add operator lowering after Python semantics are stable. +6. Emit `AudioCompilationUnit` through C ABI. +7. Wire to CRDT arena only behind explicit selector. +8. Promote only after parity, determinism, and sequential==concurrent proofs pass. + +--- + +## Non-goals + +Zig audio must not: + +- make ASR the substrate; +- use CLAP/EnCodec embeddings as the projection; +- hide stochastic model calls inside compile; +- add normalization outside the allowed compiler boundary; +- bypass `ModalityPack` gate closure; +- write the global Vault directly; +- compile audio into any shape other than `(32,) float32` at the projection boundary; +- skip the CRDT merge-key discipline. + +--- + +## Recommendation + +Make Zig a first-class experimental lane for `audio_core_v1` after Python PR-2/PR-4 semantics and fixtures are locked. + +Do not make Zig the first implementation of the audio ontology. Make it the mechanically superior implementation of a proven compiler contract. diff --git a/docs/zig/audio-compiler/implementation-slices.md b/docs/zig/audio-compiler/implementation-slices.md new file mode 100644 index 00000000..1ddde2c7 --- /dev/null +++ b/docs/zig/audio-compiler/implementation-slices.md @@ -0,0 +1,200 @@ +# Zig Audio Compiler — Implementation Slices + +**Status:** doctrine / implementation plan +**Depends on:** `docs/zig/audio-compiler/README.md` + +This document breaks a future Zig audio compiler into small, reviewable slices. + +Zig must not become the place where the audio ontology is invented. Python/spec/fixture work remains the reference first. Zig becomes valuable only after that reference is stable. + +--- + +## ZA-0 — Reference lock + +Purpose: lock the Python audio compiler behavior before native port. + +Required reference surfaces: + +- canonical signal formation; +- frame grid; +- acoustic lexer; +- typed AudioIR; +- operator registry; +- projection hash; +- `AudioCompilationUnit` merge key; +- trace hygiene; +- teacher isolation. + +Exit gate: + +```text +Python fixtures and expected hashes are green. +No Zig implementation starts by defining semantics. +``` + +--- + +## ZA-1 — Canonicalizer and frame grid + +Purpose: port deterministic buffer mechanics first. + +Candidate files: + +```text +core-zig/src/audio/canonical.zig +core-zig/src/audio/frames.zig +core-zig/src/audio/checksum.zig +``` + +Required checks: + +- same canonical hash as reference fixtures; +- same frame count; +- same padding behavior; +- same source/canonical hash semantics; +- no raw PCM exported beyond the compiler-local boundary. + +--- + +## ZA-2 — Acoustic lexer + +Purpose: port measured fact extraction. + +Candidate file: + +```text +core-zig/src/audio/lexer.zig +``` + +Required checks: + +- same energy bins; +- same voiced/unvoiced classification; +- same pause/onset classification; +- same pitch candidate quantization where implemented; +- stable token stream hash. + +The lexer measures facts. It must not infer semantic truth or call teacher models. + +--- + +## ZA-3 — AudioIR serialization + +Purpose: produce canonical typed IR from tokens. + +Candidate file: + +```text +core-zig/src/audio/ir.zig +``` + +Required checks: + +- same event families as reference; +- same canonical event ordering; +- same serialized field order; +- same `ir_sha256`; +- content anchors remain evidence, not substrate. + +--- + +## ZA-4 — Operators and projection + +Purpose: lower AudioIR to one `(32,) float32` unit. + +Candidate files: + +```text +core-zig/src/audio/operators.zig +core-zig/src/audio/project.zig +``` + +Required checks: + +- same pack manifest interpretation; +- same operator lookup; +- same theta quantization; +- canonical event-order composition; +- same projection hash; +- `versor_condition < 1e-6`; +- negative test proves in-chunk event order matters. + +The compiler must never parallelize the non-commutative in-chunk composition loop. + +--- + +## ZA-5 — AudioCompilationUnit C ABI + +Purpose: expose compiled units to Python/CRDT without Python object semantics. + +Candidate files: + +```text +core-zig/include/core_audio.h +core-zig/src/audio/ffi.zig +core_native/audio_zig.py +``` + +Required checks: + +- unit exposes `[32]f32`; +- unit exposes canonical, IR, and projection hash legs; +- unit exposes pack/projection metadata needed for provenance; +- no raw PCM exported; +- Zig-owned unit has a matching free function; +- invalid input produces typed error. + +--- + +## ZA-6 — CRDT handoff + +Purpose: write compiled units into thread-local arenas behind explicit selector. + +Depends on: + +- audio compiler parity; +- CRDT backend parity, or the current reference CRDT path. + +Required checks: + +- one compiled unit writes to one local arena; +- duplicate unit dedups by merge key; +- arena flush order does not change final Vault contribution; +- sequential vs concurrent proof passes; +- pending-delta count is observable. + +--- + +## ZA-7 — Streaming phase + +Purpose: preserve continuity across chunk seams. + +Deferred until offline/whole-buffer v1 is locked. + +Streaming must add: + +- explicit stream state object; +- deterministic carry-over buffers; +- stable chunk IDs; +- seam tests; +- clear reset behavior; +- no hidden global state. + +Exit gate: + +```text +streamed chunks produce the same result as whole-buffer mode where equivalence is claimed. +``` + +--- + +## Stop conditions + +Stop or revert the Zig lane if: + +- parity fails and requires semantic changes; +- teacher outputs begin changing substrate projection; +- trace hygiene is weakened; +- raw PCM leaks into Vault/TurnEvent records; +- closure thresholds are weakened; +- backend selection becomes automatic by library presence. diff --git a/docs/zig/core-native-system/README.md b/docs/zig/core-native-system/README.md new file mode 100644 index 00000000..9db5a19b --- /dev/null +++ b/docs/zig/core-native-system/README.md @@ -0,0 +1,207 @@ +# Zig Guidance — CORE Native System + +**Status:** doctrine +**Component:** native substrate boundary across CORE + +This document defines what "CORE native" means if Zig is introduced. + +It does not mean rewriting the cognitive runtime in Zig. It means establishing a disciplined native substrate layer below the Python semantic runtime and beside the existing Rust backend. + +--- + +## Ring architecture + +CORE should be understood as rings: + +```text +Ring 3 — Operator / Workbench / Review + Python + TypeScript + CLI, Workbench, evals, review queues, demos + +Ring 2 — Semantic cognition runtime + Python + chat runtime, teaching, packs, proposition/realizer logic, trace policy + +Ring 1 — Native substrate services + Rust incumbent + Zig candidates + exact recall, algebra kernels, CRDT merge, modality compilers, FFI + +Ring 0 — Hardware / memory substrate + CPU, UMA, future MLX/Metal integration, filesystem artifacts +``` + +Zig belongs in Ring 1 where its properties matter. It does not belong in Ring 2 merely because the project is getting large. + +--- + +## What CORE-native should mean + +A component is CORE-native if it: + +- accepts or emits lawful `(32,)` substrate values; +- preserves exact replay semantics; +- owns a deterministic buffer-level transform; +- has a stable ABI or runtime boundary; +- has a precise error contract; +- can be evaluated independently; +- does not invent semantic truth outside packs/review. + +Examples: + +```text +CRDT merge kernel +Audio compiler substrate +Batch recall kernel +Native projection compiler +Native edge ingestion runner +``` + +Non-examples: + +```text +Teaching review policy +Identity pack doctrine +Surface realizer wording +Workbench UX state +Domain curriculum planning +``` + +--- + +## Language placement + +| Layer | Preferred material | Why | +|---|---|---| +| Pack/review/evals | Python | semantic iteration, proof harness, fixtures, operator workflows | +| Current algebra hot path | Rust incumbent | already parity-gated, PyO3/NumPy integrated, Rayon parallelism | +| CRDT substrate | Zig candidate | explicit allocator, C ABI, content-addressed memory law | +| Audio compiler | Zig candidate after Python spec | streaming deterministic compiler, edge-native C ABI | +| Workbench UI | TypeScript/React | browser-native UI | +| Standalone edge runner | Zig candidate later | no-Python deployment and explicit runtime ownership | + +--- + +## Native substrate package shape + +If Zig lands, it should be grouped as a native substrate package, not scattered across semantics: + +```text +core-zig/ + build.zig + include/ + core_native.h + core_crdt.h + core_audio.h + core_recall.h + src/ + crdt/ + audio/ + recall/ + ffi/ + version.zig + tests/ +``` + +No Zig code should live under `chat/`, `teaching/`, `generate/`, `packs/`, or `workbench/` unless a later ADR narrows a specific native subcomponent. + +--- + +## Backend selection doctrine + +The native system must be explicitly selected. + +Recommended selectors: + +```text +CORE_NATIVE_BACKEND=python +CORE_NATIVE_BACKEND=rust +CORE_NATIVE_BACKEND=zig +CORE_CRDT_BACKEND=rust|zig|python +CORE_AUDIO_COMPILER_BACKEND=python|zig +``` + +The active backend should be trace-visible for any runtime path where it can affect performance or replay evidence. + +Forbidden: + +```text +load Zig because the library exists +switch backend mid-turn +use native backend without recording backend identity +silently fall back after partial mutation +``` + +Fallback is good. Silent semantic fallback is not. + +--- + +## What should be native first + +Priority order: + +1. CRDT arena/delta/merge. +2. Audio canonicalizer/frame/lexer/IR hash. +3. AudioCompilationUnit C ABI and CRDT handoff. +4. `vault_recall_batch` challenge implementation. +5. Standalone edge ingestion runner. + +This order follows the actual pressure points: continuous modalities, exact concurrent ingestion, and deterministic native compilation. + +--- + +## What should not be native now + +Do not port: + +- `chat.runtime`; +- teaching proposals/review; +- pack ratification; +- eval framework; +- Workbench API; +- identity/safety/ethics policy; +- natural language realizer logic. + +These are not slow buffer kernels. They are semantic/governance layers where Python remains the right material. + +--- + +## Native runtime invariants + +Every native CORE component must preserve: + +```text +versor_condition(F) < 1e-6 where it emits field state +no approximate recall +no hidden normalization +no hidden global mutation +no substrate embeddings +no raw sensory payload in trace/Vault records unless explicitly allowed by ADR +content-addressed ordering where concurrency can reorder arrival +``` + +--- + +## Edge-native horizon + +A future edge runner may be Zig-native if it has a narrow, frozen contract: + +```text +load signed/checksummed packs +mount selected compiler(s) +accept local signal chunks +compile to AudioCompilationUnit / VisionCompilationUnit / MotorUnit +push into local arena +explicitly merge_tick +publish replay evidence +serve health/status +``` + +That runner should not own teaching review, pack mutation, cloud sync, or semantic expansion. + +--- + +## Decision posture + +Zig should be treated as a **native substrate material**, not a general project language. + +The masterpiece standard is not one-language purity. It is boundary purity: every layer made from the correct material, with no confusion about what it is allowed to know or mutate. diff --git a/docs/zig/crdt-substrate/README.md b/docs/zig/crdt-substrate/README.md new file mode 100644 index 00000000..728760fd --- /dev/null +++ b/docs/zig/crdt-substrate/README.md @@ -0,0 +1,277 @@ +# Zig Guidance — Delta-CRDT Substrate + +**Status:** doctrine / prototype candidate +**Component:** Delta-CRDT arena, delta, join, merge kernel, content-addressed ordering +**Primary governing ADR:** ADR-0180 + +This is the strongest near-term Zig candidate in CORE. + +The CRDT substrate is not semantic cognition. It is the mechanical substrate for concurrent modality write accumulation: thread-local arenas emit canonical deltas, and an explicitly mounted merge kernel folds those deltas into a global Vault-visible state without making final recall approximate or arrival-order dependent. + +--- + +## Why this component is suitable for Zig + +The CRDT substrate wants properties Zig is good at providing: + +- explicit allocator ownership; +- small native surface area; +- deterministic byte-level ordering; +- stable C ABI; +- edge-native build story; +- predictable memory layout; +- no garbage collector; +- no hidden runtime; +- precise caller-owned buffer contracts. + +This is not a broad language preference. It is a substrate fit. + +--- + +## Existing contract to preserve + +ADR-0180 states that the semilattice claim holds only at the `vault/store` layer, not at `versor_apply`, not at `field/propagate`, and not at trace reduction. The CRDT substrate must shard write accumulation only. + +Required law: + +```text +ProjectionHead.project pure signal projection +versor_apply non-commutative, do not shard unless serialized +field propagation not a CRDT merge +vault/store.write semilattice-eligible write accumulation +recall result ordering content-addressed, not arrival-addressed +``` + +The existing Rust substrate already implements the conceptual shape: + +```text +ArenaEntry { versor: [f32; 32], provenance: Vec } +Delta { entries: sorted/deduped ArenaEntry list } +LocalArena { thread-local write cache } +merge_kernel(deltas) -> canonical Delta +``` + +Zig may challenge this substrate only by preserving the same laws. + +--- + +## What should be in Zig + +A Zig CRDT lane should include: + +```text +core-zig/src/crdt/entry.zig +core-zig/src/crdt/arena.zig +core-zig/src/crdt/delta.zig +core-zig/src/crdt/merge.zig +core-zig/src/crdt/hash.zig +core-zig/src/crdt/ffi.zig +include/core_crdt.h +``` + +### `ArenaEntry` + +Should represent one content-addressed write: + +```text +versor: [32]f32 +provenance: []const u8 +``` + +The content key is: + +```text +raw IEEE-754 f32 bits of all 32 components ++ provenance bytes +``` + +The substrate must not compare floats numerically for canonical ordering. It must compare bytes/bits to avoid platform-dependent `NaN`, `-0.0`, and `+0.0` behavior. + +### `LocalArena` + +Should be thread-local and share-nothing. + +Allowed: + +```text +push local entries +snapshot to Delta +clear/drain if explicitly requested by owner +report pending count +``` + +Forbidden: + +```text +write global Vault +spawn hidden merge thread +normalize entries +repair bad versors +infer epistemic status +call object-store/S3/cloud code +``` + +### `Delta` + +A `Delta` must be canonical: + +```text +entries sorted by content key +byte-identical duplicates removed +stable independent of insertion order +``` + +### `merge_kernel` + +The merge kernel must be explicit, not hidden. + +Required behavior: + +```text +input: list of Deltas +output: one canonical Delta containing their union +law: permutation-invariant +law: duplicate-delta idempotent +law: content-addressed ordering only +``` + +--- + +## Minimum C ABI sketch + +```c +typedef struct CoreCrdtArena CoreCrdtArena; +typedef struct CoreCrdtDelta CoreCrdtDelta; + +typedef struct CoreCrdtError { + int code; + char message[256]; +} CoreCrdtError; + +CoreCrdtArena* core_crdt_arena_new(void); +void core_crdt_arena_free(CoreCrdtArena* arena); + +int core_crdt_arena_push( + CoreCrdtArena* arena, + const float versor[32], + const unsigned char* provenance, + unsigned long provenance_len, + CoreCrdtError* err +); + +int core_crdt_arena_snapshot( + const CoreCrdtArena* arena, + CoreCrdtDelta** out_delta, + CoreCrdtError* err +); + +void core_crdt_delta_free(CoreCrdtDelta* delta); + +int core_crdt_delta_join( + const CoreCrdtDelta* a, + const CoreCrdtDelta* b, + CoreCrdtDelta** out_delta, + CoreCrdtError* err +); + +int core_crdt_merge_kernel( + const CoreCrdtDelta* const* deltas, + unsigned long delta_count, + CoreCrdtDelta** out_delta, + CoreCrdtError* err +); + +int core_crdt_delta_hash( + const CoreCrdtDelta* delta, + unsigned char out_sha256[32], + CoreCrdtError* err +); +``` + +This is illustrative, not final. The final ABI must be ratified before implementation is considered supported. + +--- + +## What must stay out of Zig + +The CRDT substrate must not own: + +- epistemic status promotion; +- review decisions; +- teaching proposal admission; +- pack ratification; +- semantic normalization; +- recall scoring policy beyond canonical store order; +- object-store sync; +- hidden background execution; +- modality compilation semantics, except when called as a separate compiler component. + +The CRDT substrate is storage-order law, not cognition. + +--- + +## Required proof obligations + +### C-1 — Commutativity + +```text +join(a, b) == join(b, a) +``` + +### C-2 — Associativity + +```text +join(join(a, b), c) == join(a, join(b, c)) +``` + +### C-3 — Idempotence + +```text +join(a, a) == a +``` + +### C-4 — Permutation invariance + +```text +merge_kernel([d1, d2, d3]) == merge_kernel(any_permutation([d1, d2, d3])) +``` + +### C-5 — Duplicate re-ingest is no-op + +```text +merge(existing, already_seen_delta) == existing +``` + +### C-6 — Recall order invariance + +Equal-score recall must not depend on wall-clock arrival. The canonical merged order must be content-addressed so tie breaks are replay-stable. + +### C-7 — No hidden global mutation + +Tests must prove that pushing to an arena never mutates the global Vault. Only explicit merge publication may alter the global visible state. + +--- + +## Migration sequence + +1. Keep Rust implementation as incumbent. +2. Add Python tests that pin CRDT laws against the current behavior. +3. Add Zig prototype behind `CORE_CRDT_BACKEND=zig`. +4. Compare Zig against Rust/Python reference on canonical fixtures. +5. Add benchmark only after parity passes. +6. Promote only if Zig wins on ABI, deployment, memory, or performance. + +--- + +## Promotion criteria + +Zig should become the preferred CRDT substrate only if it demonstrates: + +- equal or stronger deterministic law coverage; +- simpler C ABI for modality compilers and edge runtime; +- explicit allocation lifecycle that is easier to audit than the incumbent; +- no loss in exact recall behavior; +- observable merge state; +- clean failure when absent or stale. + +Until then, Zig remains a prototype lane, not a replacement. diff --git a/docs/zig/crdt-substrate/implementation-slices.md b/docs/zig/crdt-substrate/implementation-slices.md new file mode 100644 index 00000000..d0d5c3b8 --- /dev/null +++ b/docs/zig/crdt-substrate/implementation-slices.md @@ -0,0 +1,181 @@ +# Zig CRDT Substrate — Implementation Slices + +**Status:** doctrine / implementation plan +**Depends on:** `docs/zig/crdt-substrate/README.md` + +This document breaks the Zig CRDT candidate into small, reviewable slices. + +No slice should touch production runtime behavior until the explicit backend selector and parity gates exist. + +--- + +## ZC-0 — Contract pinning + +Purpose: prove the current CRDT law before adding Zig. + +Deliverables: + +```text +tests/test_crdt_semilattice_contract.py +tests/test_crdt_content_ordering.py +tests/test_crdt_no_global_write_from_arena.py +``` + +Required checks: + +- join commutativity; +- join associativity; +- join idempotence; +- stable content ordering; +- duplicate deduplication; +- push-to-arena does not mutate Vault; +- canonical merge independent of arena flush order. + +Exit gate: Python/Rust reference behavior is locked. + +--- + +## ZC-1 — Zig skeleton + +Purpose: create an inert Zig crate/library with no runtime wiring. + +Deliverables: + +```text +core-zig/build.zig +core-zig/src/version.zig +core-zig/src/crdt/entry.zig +core-zig/src/crdt/delta.zig +core-zig/src/crdt/arena.zig +core-zig/src/crdt/merge.zig +core-zig/src/crdt/ffi.zig +core-zig/include/core_crdt.h +``` + +Allowed: + +- local Zig unit tests; +- no Python runtime dispatch; +- no core CLI integration except maybe future `core doctor --zig` planning. + +Exit gate: Zig library builds and tests locally. + +--- + +## ZC-2 — C ABI self-test + +Purpose: prove the FFI is stable before Python binding. + +Deliverables: + +```text +core-zig/tests/crdt_ffi_test.zig +core-zig/examples/crdt_smoke.c +``` + +Required checks: + +- create/free arena; +- push entries; +- snapshot delta; +- join deltas; +- merge many deltas; +- hash delta; +- reject bad pointers/lengths gracefully where possible. + +Exit gate: C ABI round-trip passes without Python. + +--- + +## ZC-3 — Python binding behind selector + +Purpose: expose Zig CRDT only when explicitly requested. + +Deliverables: + +```text +core_native/zig_loader.py +core_native/crdt_zig.py +tests/test_crdt_zig_binding.py +``` + +Selector: + +```text +CORE_CRDT_BACKEND=zig +``` + +Required checks: + +- missing library produces clear unavailable status; +- bad ABI version is rejected; +- errors become typed Python errors; +- no automatic selection by importability. + +Exit gate: Python can call Zig CRDT functions in isolation. + +--- + +## ZC-4 — Parity fixtures + +Purpose: prove Zig equals the reference. + +Deliverables: + +```text +tests/test_crdt_zig_parity.py +``` + +Required checks: + +- same merged entries as reference; +- same hash as reference; +- same behavior under input permutations; +- same dedup behavior; +- same provenance distinction behavior; +- same handling of `-0.0`, `+0.0`, and NaN bit patterns if fixtures include them. + +Exit gate: parity green. + +--- + +## ZC-5 — Benchmark and memory profile + +Purpose: prove Zig has a reason to exist. + +Deliverables: + +```text +benchmarks/crdt_merge.py +bench_reports/crdt_merge_zig.json +``` + +Metrics: + +- latency by entry count; +- latency by delta count; +- memory allocation profile; +- peak RSS if measured from Python; +- merge hash equality check included before timing is trusted. + +Exit gate: Zig wins a named mechanical criterion or remains prototype-only. + +--- + +## ZC-6 — Runtime integration proposal + +Purpose: decide whether to wire Zig into actual modality ingestion. + +Allowed only after ZC-0 through ZC-5. + +Deliverables: + +```text +ADR: Zig CRDT backend promotion +runtime integration plan +telemetry fields +fallback plan +CI lane +``` + +No production runtime integration is authorized before this slice. diff --git a/docs/zig/runtime-ffi/README.md b/docs/zig/runtime-ffi/README.md new file mode 100644 index 00000000..fd665497 --- /dev/null +++ b/docs/zig/runtime-ffi/README.md @@ -0,0 +1,214 @@ +# Zig Guidance — Runtime FFI Boundary + +**Status:** doctrine +**Component:** C ABI and runtime backend boundary + +Zig should enter CORE through stable FFI surfaces, not through ad-hoc runtime imports. This document defines the shape of that boundary. + +--- + +## Why FFI discipline matters + +The risk of adding Zig is not the language itself. The risk is boundary confusion: + +```text +Who owns this buffer? +Who frees this allocation? +Is this function allowed to mutate global state? +Is the backend selected intentionally? +Are errors typed or just logged? +Can replay identify which backend ran? +``` + +If those questions are ambiguous, Zig should not be merged. + +--- + +## ABI principles + +### 1. C ABI first + +All Zig substrate components should expose a C ABI first. + +Python, Rust, Swift, or future edge binaries may bind to that ABI, but the ABI itself should not depend on Python object semantics. + +### 2. Explicit versioning + +Every library should expose: + +```c +unsigned int core_native_abi_version(void); +const char* core_native_build_id(void); +``` + +Runtime bindings must reject unknown ABI versions. + +### 3. Caller-owned inputs + +Input buffers should usually be caller-owned. Zig must treat them as borrowed unless the API explicitly copies them. + +### 4. Explicit output ownership + +If Zig allocates output, Zig must provide the matching free function. + +Pattern: + +```c +int component_do_work(..., ComponentOutput** out, ComponentError* err); +void component_output_free(ComponentOutput* out); +``` + +No output allocation should be freed by Python, Rust, or libc unless the ABI explicitly says so. + +### 5. Typed error surface + +Do not print errors from Zig as the only failure signal. Return structured error codes and bounded messages. + +Example: + +```c +typedef struct CoreNativeError { + int code; + char message[256]; +} CoreNativeError; +``` + +Python bindings should translate these to typed Python exceptions. + +--- + +## Backend selection + +Native backends must be selected explicitly. + +Recommended environment variables: + +```text +CORE_NATIVE_BACKEND=python|rust|zig +CORE_CRDT_BACKEND=python|rust|zig +CORE_AUDIO_COMPILER_BACKEND=python|zig +CORE_RECALL_BACKEND=python|rust|zig +``` + +Rules: + +- absence of Zig library must not break the Python path; +- presence of Zig library must not silently change runtime behavior; +- backend identity must be visible in diagnostics; +- backend identity should be recorded in replay evidence where behavior/performance could matter. + +--- + +## Memory ownership table + +| Case | Preferred ownership | +|---|---| +| input NumPy matrix | Python owns; pass pointer/shape/stride only after contiguity check | +| audio input samples | caller owns unless canonicalizer explicitly copies | +| CRDT provenance bytes | Zig may copy into arena-owned storage | +| CRDT delta output | Zig owns; caller frees through Zig free function | +| AudioCompilationUnit output | Zig owns; caller frees through Zig free function | +| error message | caller passes error struct pointer; Zig writes bounded message | +| returned slices | avoid raw borrowed slices unless lifetime is obvious and documented | + +--- + +## Shape and dtype validation + +All FFI surfaces must validate shape and dtype before computation. + +Examples: + +```text +versor: exactly 32 f32 values +matrix: N x 32 f32, row-major contiguous +queries: B x 32 f32, row-major contiguous +audio samples: declared dtype, channel count, sample rate, length +edges: E x 2 i32 if graph diffusion is used +``` + +Invalid shape is a typed error, not undefined behavior. + +--- + +## Forbidden FFI behaviors + +Zig FFI must not: + +- hold Python object pointers; +- assume NumPy memory remains valid after caller scope ends; +- spawn hidden background threads; +- mutate global CORE state; +- allocate without a free path; +- panic across FFI boundary; +- return borrowed memory with unclear lifetime; +- reinterpret non-contiguous buffers as contiguous; +- hide fallback after partial mutation; +- suppress errors and continue. + +--- + +## Python binding shape + +Python binding modules should be thin and boring: + +```text +core_native/zig_loader.py +core_native/crdt_zig.py +core_native/audio_zig.py +core_native/recall_zig.py +``` + +Responsibilities: + +- locate library; +- check ABI version; +- check buffer contiguity; +- call C ABI; +- translate errors; +- free outputs; +- expose Pythonic wrappers; +- never define semantics not present in the reference contract. + +--- + +## Diagnostic command + +A future CLI diagnostic should report: + +```text +core doctor --zig + library found: yes/no + ABI version: ... + build id: ... + enabled selectors: ... + CRDT backend available: yes/no + audio compiler backend available: yes/no + recall backend available: yes/no + last self-test: pass/fail +``` + +If a user asks for a required Zig backend and it is unavailable, fail fast with a clear error. + +--- + +## Runtime trace policy + +Any turn/event affected by Zig-backed native behavior should be able to report: + +```text +native_backend: zig +native_component: crdt|audio|recall|algebra +abi_version: N +build_id: sha/hash/version +``` + +This does not mean every trace needs verbose native metadata. But replay-critical paths must not hide which substrate executed. + +--- + +## Recommendation + +Add the FFI contract before adding serious Zig code. + +A clean FFI boundary lets CORE evaluate Zig without infecting Python semantics, Rust parity work, or Workbench/operator flows.