core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/Yellowpaper.md

17 KiB
Raw Blame History

The CORE Yellowpaper

Formal Specification of the Cl(4,1) Versor Engine

Companion to the Whitepaper. All conceptual foundations and design philosophy are in docs/Whitepaper.md. This document is the mathematical and implementation specification.

I. The Mathematical Foundation

1. Why Cl(4,1)

The original CORE architecture used Cl(3,0) — the geometric algebra of 3D Euclidean space. Cl(3,0) has 8 basis elements (scalar, 3 vectors, 3 bivectors, 1 pseudoscalar) and maps onto 2×2 complex matrices via the Pauli isomorphism.

Cl(4,1) is the Conformal Geometric Algebra (CGA) of 3D Euclidean space. It has 32 basis elements and signature (4,1): four positive directions e1, e2, e3, e4 and one negative direction e5. The CGA extension adds two null basis vectors:

o = (e5 - e4) / 2        # origin point
∞ = e5 + e4              # point at infinity

The key identity that motivates the upgrade:

In Cl(4,1), a Euclidean point p = (x,y,z) embeds as a null vector:

P = p + (1/2)|p|² ∞ + o

and satisfies:

P · P = 0

All conformal transformations (rotations, translations, dilations, inversions) are versors in Cl(4,1). In Cl(3,0), translations required special handling outside the algebra. In Cl(4,1), translations are versors — the algebra is fully closed over all conformal motions.

2. Basis Structure

Cl(4,1) has 2^5 = 32 basis blades organized by grade:

Grade Count Basis elements Interpretation
0 1 1 Scalar
1 5 e1, e2, e3, e4, e5 Vectors
2 10 e12, e13, e14, e15, e23, e24, e25, e34, e35, e45 Bivectors
3 10 e123, e124, e125, e134, e135, e145, e234, e235, e245, e345 Trivectors
4 5 e1234, e1235, e1245, e1345, e2345 Quadvectors
5 1 e12345 Pseudoscalar

Metric (signature (4,1)):

e1² = e2² = e3² = e4² = +1
e5² = -1
ei · ej = 0  for i ≠ j

The geometric product multiplication table is a 32×32 signed permutation matrix, computed once at startup and stored in a OnceLock<Table> in core-rs/src/cl41.rs.

3. Representation in Code

All multivectors are represented as [f32; 32] arrays. The index mapping is fixed:

index 0:  scalar (grade 0)
index 1-5:  grade-1 components (e1, e2, e3, e4, e5)
index 6-15: grade-2 components
index 16-25: grade-3 components
index 26-30: grade-4 components
index 31:  pseudoscalar (grade 5)

This layout is fixed at the Rust layer and mirrored in the Python algebra modules. All Python–Rust interchange uses this same 32-element f32 array.

II. The Versor Engine — Core Invariant

The Versor Condition

A multivector V ∈ Cl(4,1) is a versor if and only if:

V · reverse(V) = ±1

Where reverse(V) reverses the order of every basis blade product:

  • Grade 0: unchanged (sign +1)
  • Grade 1: unchanged (sign +1)
  • Grade 2: sign −1
  • Grade 3: sign −1
  • Grade 4: sign +1
  • Grade 5: sign +1

The Sandwich Product

The unique allowed field transition is:

F_new = V · F · reverse(V)

This is the versor sandwich product. Its properties:

  • If V is a versor and F is a versor, then F_new is a versor (algebraic closure)
  • Preserves grade structure under any conformal transformation
  • Reversal is free: reverse(V) is computed by sign-flipping grade-2 and grade-3 components in-place

Verification

versor_condition(F) = ||F · reverse(F) - 1||_F

This scalar is zero on the versor manifold. It is computed:

  1. Exactly once at the injection gate on every input
  2. In tests only — never in the propagation hot path

Tolerance: versor_condition(F) < 1e-6 for acceptance.

III. Conformal Geometric Algebra (CGA) Distance

The Null Cone

A vector X ∈ Cl(4,1) is null if:

X · X = 0

All embedded Euclidean points live on the null cone. The conformal embedding of point p = (x,y,z):

P = xe1 + ye2 + ze3 + (1/2)|p|² e4 + e5

(Using the compact basis e4=∞, e5=o convention.) This satisfies P·P = 0 by construction.

The Distance Identity

For null vectors X, Y representing Euclidean points:

X · Y = -(1/2) d(X, Y)²

Where d(X,Y) is Euclidean distance and · denotes the grade-0 scalar part of the geometric product.

This identity makes the CGA inner product the exact conformal distance. It is the foundation of vault recall.

Vault Recall

Given a query versor Q and a vault of stored versors {V_i}:

best_match = argmax_i { Q · V_i }

This is implemented as a parallel scan in core-rs/src/vault.rs via Rayon. The scan is:

  • Exact (not approximate)
  • Allocation-free per worker thread
  • GIL-releasing (Rayon runs outside Python)
  • O(N) where N = vault size

No ANN index is used. No approximate neighbor structure is maintained. No index rebuild is required on vault growth.

Null Cone Drift

Over long sessions, stored versors can drift off the null cone due to floating-point accumulation. The null_project() function in core-rs/src/cga.rs resets them:

X ← X / sqrt(|X · reverse(X)|)

This is called as VaultStore.reproject() every N turns. It is not drift correction in the sense of the deleted monitor stack — it is a periodic renormalization required by finite-precision arithmetic on any manifold, and it costs a single division per stored versor.

IV. Holonomy Encoding

Holonomy is the accumulated geometric transformation from traversing a closed path in the vocabulary manifold. It is used to encode prompt context as a single versor that captures the path-dependent structure of the input.

Forward walk over word versors w_0, ..., w_n:

F = normalize(w_0 · w_1 · ... · w_n)

Reverse walk with damping (1-α):

R = normalize((1-α) · reverse(w_n) · ... · reverse(w_0))

Holonomy:

H = normalize(F · R)

Where α ∈ [0,1] is the blend factor (default 0.5). The holonomy versor encodes not just which words appeared, but the order in which they appeared and the curvature of the path they traced.

Implementation: core-rs/src/holonomy.rs — the entire computation is a single allocation-free Rust function. At 100-token inputs, this replaces 200+ Python dispatch calls with a single call crossing the PyO3 boundary.

Boundedness invariant:

||H||_F ∈ [0.5, 2.0]  for any prompt length

Verified in tests/test_holonomy.py via property-based testing with Hypothesis.

V. The Vocabulary Manifold

The vocabulary manifold is a finite set of null vectors {v_w} ⊂ Cl(4,1), one per token w in the vocabulary.

Construction: Each word w is embedded as a null vector via the CGA point embedding:

  1. Obtain a 3D semantic coordinate p_w (from a frozen static embedding or from the manifold’s coordinate frame)
  2. Embed: v_w = p_w_x·e1 + p_w_y·e2 + p_w_z·e3 + (1/2)|p_w|²·e4 + e5
  3. Verify: v_w · v_w = 0 (null condition)

Token projection: At each generation step:

next_token = argmin_w { d_CGA(F_current, v_w) }
               = argmax_w { F_current · v_w }

This is a nearest-null-vector scan. For vocabularies up to ~50,000 tokens it is computed in a single vectorized MLX pass.

VI. The Sensorium — Modality Protocol Specification

The sensorium/ layer converts any surface signal into a (32,) Cl(4,1) multivector before it reaches ingest/gate.py. Every ProjectionHead is the Logos-recovery boundary for its modality.

Modality Enum

class Modality(enum.Enum):
    TEXT   = "text"
    VISION = "vision"
    AUDIO  = "audio"
    MOTOR  = "motor"

New modalities must be added here AND register a projection head in sensorium/registry.py before any pack can mount.

ProjectionHead[S, F] Protocol

class ProjectionHead(Protocol[S, F]):
    modality: Modality
    embedding_dim: int  # must be 32 for Cl(4,1)

    def project(self, signal: S) -> mx.array:         # shape (32,)
    def project_batch(self, signals: list[S]) -> mx.array:  # shape (N, 32)
    def verify_unitarity(self, sample: S) -> bool
        # True iff V · reverse(V) = ±1 within 1e-6

Note: core-ai used shape (2, 2) complex (Cl(3,0) Pauli isomorphism). core uses shape (32,) f32 (Cl(4,1) canonical layout).

ModalityPack[S] Dataclass

@dataclass(frozen=True, slots=True)
class ModalityPack(Generic[S]):
    pack_id: str                          # "en", "he", "grc", "imagenet-1k", ...
    modality_type: Modality
    projection: ProjectionHead[S] | None  # None for articulation-only modalities
    decoder: SurfaceDecoder[S] | None     # None for perception-only modalities
    vocabulary: ModalityVocabulary[S]     # bidirectional surface ↔ rotor map
    grammar_scaffold: Any                 # versor attractors from vocab/
    checksum_verified: bool
    gate_engaged: bool = True

Frozen + slotted: zero per-instance dict overhead, hashable. Type-parameterised: ModalityPack[str] and ModalityPack[np.ndarray] are not interchangeable at the type level.

Mount-Time Failure Modes

Error Meaning
MANIFEST_INVALID Pack manifest fails integrity check
UNITARITY_VIOLATION Projection head produces non-unitary rotor
PROJECTION_NOT_CONVERGED Projection head did not converge during validation
GRADE_DECLARATION_MISMATCH Declared grades do not match produced grades
MODALITY_NOT_REGISTERED Modality not in sensorium/registry.py
GATE_NOT_ENGAGED Surprise-gate not active (non-text modality during seeding)

Active Modalities

Pack ID Modality Surface type S Status
en TEXT str Active
he TEXT str Active (Hebrew depth corpus)
grc TEXT str Active (Koine Greek depth corpus)
vision adapters VISION np.ndarray Planned
audio adapters AUDIO np.ndarray Planned
motor adapters MOTOR np.ndarray Planned

See ADR-0013 for the full protocol specification.

VII. The core_ingest Governance Layer — Pre-Gate Specification

The core_ingest/ layer wraps upstream of ingest/gate.py. The gate is not modified.

DeterminismClass

Class Meaning Auto-Accept Eligible?
D0 Fully deterministic, pinned inputs and code ✅
D1 Deterministic with pinned external artifact ✅
D2 Nondeterministic but replay-captured ❌
D3 External unpinned model or API ❌
D4 Human / operator proposal ❌

A D2–D4 frontend is structurally forbidden from claiming AUTO_ACCEPT_ELIGIBLE. Enforced in CandidateGeometricPressure.__post_init__.

CandidateGeometricPressure Content-Addressing

pressure_id  = SHA-256(full canonical packet)    # structural deduplication
semantic_key = SHA-256(kind + modality + lemma + subject + verb + object + payload)
                                                  # convergent-evidence detection

Two packets with the same semantic_key assert the same claim from different provenance sources. Convergence is tracked by the IngestCompiler and surfaced as a confidence signal to downstream consumers.

Three-Gate Validation Flow

CandidateGeometricPressure batch
    → ProvenanceGate    # SourceSpan integrity, SHA-256 of source material
    → SemanticGate      # span completeness, balanced delimiters, non-empty
    → GovernanceGate    # ReviewLevel, DeterminismClass, ReviewDecision overrides
    → ValidationReport  # per-packet disposition
    → LearningArtifact  # accepted packets → train/ export path

StructuralSegmenter — Why, Not What

LLM extraction was rejected: a language model upstream of the gate is a D3 nondeterministic oracle whose semantic projections would be silently embedded in the field state. The StructuralSegmenter carves at form boundaries only — the meaning of a span stays inside the field where it belongs. Biblical texts (Hebrew, Koine Greek) are D0 by construction: canonical verse boundaries are fixed. See ADR-0012.

VIII. Persona as CGA Motor

A CGA motor is a versor that encodes a screw motion: a combined rotation and translation in conformal space.

M = T · R

Where T is a translator versor and R is a rotor. Every motor satisfies the versor condition by construction.

Persona application:

F_biased = M · F · reverse(M)

This rotates and translates the field state within the conformal manifold, biasing generation toward the persona’s characteristic region of the vocabulary manifold. It is a single versor product — algebraically closed, no weight overlay, no post-hoc bias vector.

Motor composition:

M_combined = M_2 · M_1

Personas compose. Two persona motors can be combined into a single motor before application. The composition is also a versor.

IX. The Three-Language Contract

Layer Language Entry point Invariant
Orchestration Python session/context.py Reads and writes FieldState. Never calls algebra directly — always via algebra/backend.py.
Backend dispatch Python algebra/backend.py Single switch: core_rs if available, pure Python fallback.
Algebra kernel Rust (PyO3) core-rs/src/lib.rs [f32; 32] in, [f32; 32] out. No heap allocation in hot path. All errors are thiserror named variants.
Tensor ops MLX field/propagate.py Used for batched matmul and field tensor operations. Stays in UMA.

Zero-copy contract:

  • Python passes numpy arrays to Rust via PyO3 buffer protocol
  • Rust reads into [f32; 32] stack arrays — one copy from Python heap to Rust stack
  • Rust returns new [f32; 32] as numpy array — one copy from Rust stack to Python heap
  • No intermediate heap allocation in the Rust kernel

GIL contract:

  • vault_recall (Rayon parallel scan) releases the GIL before entering Rayon and reacquires after
  • All other Rust functions hold the GIL for the duration of the call (fast enough that release is not worth the overhead)

X. Verification Invariants (The Implementation Gate)

These are testable predicates. Every invariant has a corresponding test in tests/.

Invariant Expression Tolerance Test file
Versor closure ||F·reverse(F) - 1||_F < 1e-6 test_versor_closure.py
Null cone ||X·X|| for all vault entries < 1e-6 test_null_cone.py
Holonomy boundedness ||H||_F [0.5, 2.0] test_holonomy.py
Motor condition ||M·reverse(M) - 1||_F < 1e-6 (in test_versor_closure.py)
CGA distance symmetry cga_inner(X,Y) == cga_inner(Y,X) exact test_cga.py
Vault recall self recall(V_i, top_k=1)[0] == i exact test_vault_recall.py
Projection unitarity ||V·reverse(V) - 1||_F (sensorium mount) < 1e-6 test_sensorium_mount.py
Ingest D-class gate D2–D4 ↛ AUTO_ACCEPT_ELIGIBLE (construction) exact test_core_ingest.py

These are structural contracts, not regression tests. A failing invariant means the algebra is broken, not the behavior.

XI. The Rust Acceleration Contract

Performance-critical operations in Rust:

Operation Complexity Why Rust
geometric_product O(32²) = 1024 MADs Called 2-3× per versor_apply; autovectorized at opt-level=3
versor_apply 3× geometric_product No allocation; entire sandwich product in one stack frame
cga_inner O(32) Called every token decode and every vault recall
vault_recall O(N × 32) Rayon parallel scan across N stored versors
holonomy_encode O(2L × 32²) 2L products for L-token prompt; replaces 2L Python dispatch calls
propagate_batch O(B × 32²) B parallel versor_apply for beam search

Build:

cd core-rs
maturin develop --release
cargo test

XII. What Was Deleted and Why

The formal record is in docs/DELETION_LOG.md. The summary:

Deleted subsystem Algebraic reason
spectral_normalize() (5/6 call sites) Compensated for rotor drift in an unclosed operation. Versor sandwich product does not drift.
grade_guard.py Grade purity is a consequence of versor products, not a condition to be checked.
_maybe_correct_field() Drift correction requires an unclosed operation upstream. The operation was closed instead.
RotorDriftTelemetry Measures a symptom. The symptom was eliminated.
HippocampusIndex (ANN) CGA inner product is exact. Approximate indexing introduced error into an analytically exact operation.
_compute_g3_energy() Pseudoscalar accumulation is impossible when all transitions are versor products.
_stabilize_post_turn_g3() Followed from the above.

CORE Yellowpaper — Versor Engine Edition. For the architectural vision, origin story, seven axioms, and three pillars, see docs/Whitepaper.md. For agent instructions and invariant enforcement, see CLAUDE.md.