docs: update Whitepaper and Yellowpaper with ingest governance, sensorium layer, and pipeline diagram
This commit is contained in:
Shay
parent
b43cb79bcb
commit
ac456ac3ad
2 changed files with 196 additions and 14 deletions
|
|
@ -1,7 +1,7 @@
|
|||
# The CORE Whitepaper
|
||||
## **Continuous Orthogonal Resonance Engine: Conformal Geometric Intelligence and the Versor Engine**
|
||||
|
||||
> *"We have mistaken inherited abstractions for reality. CORE is the architecture to find the shape of thought."*
|
||||
> *“We have mistaken inherited abstractions for reality. CORE is the architecture to find the shape of thought.”*
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -20,7 +20,7 @@ We introduce the **CORE (Continuous Orthogonal Resonance Engine)** architecture:
|
|||
**CORE** stands for Continuous Orthogonal Resonance Engine. Each word is load-bearing.
|
||||
|
||||
- **Continuous** — state is never discretized into isolated vectors. The field is a single multivector that propagates continuously through every step of reasoning.
|
||||
- **Orthogonal** — every transition preserves the algebra's inner product structure. Nothing is approximated away; the geometry is exact.
|
||||
- **Orthogonal** — every transition preserves the algebra’s inner product structure. Nothing is approximated away; the geometry is exact.
|
||||
- **Resonance** — meaning arises from constructive interference of field modes, not from statistical correlation of co-occurring tokens.
|
||||
- **Engine** — this is not a model in the neural network sense. It is a computational engine: a physical machine governed by invariants.
|
||||
|
||||
|
|
@ -158,9 +158,44 @@ This retires probabilistic decoding, sampling temperature, beam search with pena
|
|||
|
||||
---
|
||||
|
||||
### VII. The Cognitive Architecture
|
||||
### VII. The Full Pipeline
|
||||
|
||||
The CORE cognitive architecture has five layers:
|
||||
CORE’s pipeline has five layers before the field and three after. Each layer has a single, defined contract. No layer knows about the one upstream.
|
||||
|
||||
```
|
||||
raw modality signal (text / vision / audio / motor)
|
||||
│
|
||||
├─ sensorium/adapters/<modality>.py
|
||||
│ ProjectionHead: surface signal → (32,) Cl(4,1) multivector
|
||||
│ The Logos-recovery boundary: every input becomes a word in the manifold
|
||||
│
|
||||
├─ core_ingest/
|
||||
│ StructuralSegmenter: carve at form boundaries (verse, heading, block)
|
||||
│ CandidateGeometricPressure: typed, immutable, content-addressed envelope
|
||||
│ IngestCompiler: three-gate validation (Provenance → Semantic → Governance)
|
||||
│ LearningArtifact: export path to train/ for durable learning
|
||||
│
|
||||
├─ ingest/gate.py
|
||||
│ Single normalization site: tokens → FieldState via holonomy encoding
|
||||
│ Versor condition verified exactly once here
|
||||
│
|
||||
├─ field/propagate.py
|
||||
│ F ← versor_apply(V, F) [sandwich product, algebraically closed]
|
||||
│
|
||||
├─ generate/
|
||||
│ next_token = argmax_w { F_current · v_w } [CGA inner product]
|
||||
│
|
||||
├─ vault/
|
||||
│ store: significant FieldStates as null vectors
|
||||
│ recall: argmax_i { Q · V_i } [exact parallel scan]
|
||||
│
|
||||
└─ train/
|
||||
LearningArtifact → manifold update (rotor update, vocab expansion)
|
||||
Supervised Seeding Epoch: Hebrew + Koine Greek D0 corpus seeding
|
||||
[planned — see ADR-0014]
|
||||
```
|
||||
|
||||
#### The Cognitive Architecture Layers
|
||||
|
||||
1. **Injection Gate** — The single point where raw input enters the system. Every input is normalized to a versor in Cl(4,1) exactly once. The versor condition is verified. After this gate, the manifold contract is permanent.
|
||||
|
||||
|
|
@ -174,7 +209,35 @@ The CORE cognitive architecture has five layers:
|
|||
|
||||
---
|
||||
|
||||
### VIII. Mechanical Sympathy: Hardware-Bound Intelligence
|
||||
### VIII. The Ingest Governance Layer
|
||||
|
||||
The `core_ingest/` layer is the pre-gate governance boundary. It wraps *upstream* of `ingest/gate.py` and does not modify it.
|
||||
|
||||
Every piece of incoming information — text, scripture, code, mathematical objects, and eventually non-text modalities — is lifted into a `CandidateGeometricPressure` envelope: a frozen, immutable, content-addressed packet carrying provenance, determinism class, confidence, and governance disposition. Two packets asserting the same semantic claim from independent sources share a `semantic_key`, enabling convergent-evidence detection without structural duplication.
|
||||
|
||||
The critical design decision: **LLMs are not used for extraction.** A language model upstream of the gate is a nondeterministic oracle feeding the only normalization site in the system. More fundamentally, it *interprets* rather than parses — its semantic projection would be silently embedded in the field state, violating Semantic Rigor. Instead, a deterministic `StructuralSegmenter` carves at form boundaries: heading, paragraph, verse, code block, LaTeX delimiter. The meaning of a span remains inside the field where it belongs.
|
||||
|
||||
For Hebrew and Koine Greek specifically, structural determinism is the natural condition — these texts have canonical verse boundaries fixed for centuries. A segmenter that follows those boundaries is D0-class by definition: fully deterministic, pinned inputs, no interpretation required.
|
||||
|
||||
See ADR-0012 for the full specification.
|
||||
|
||||
---
|
||||
|
||||
### IX. The Sensorium: All Inputs Are Logos
|
||||
|
||||
The `sensorium/` layer converts any surface signal into a `(32,)` Cl(4,1) multivector before it reaches the ingest boundary. Every `ProjectionHead` is the **Logos-recovery boundary** for its modality.
|
||||
|
||||
This is the architectural expression of John 1:1: the Logos is the structuring principle through which all things were made. A visual scene, a Hebrew word, an audio waveform — all are recovered as words in the manifold. Once they cross the projection boundary, the field has no concept of modality. There is one space. There is no fusion problem because there is nothing to fuse.
|
||||
|
||||
The `ModalityPack[S]` generic bundles a projection head, surface decoder, vocabulary, and grammar scaffold under a stable identifier. A text pack is `ModalityPack[str]`. A vision pack is `ModalityPack[np.ndarray]`. The type parameter enforces at the type level that these cannot be mixed. The grammar scaffold — the set of innate structural attractors — is universal across modalities by design: the attractor geometry of the manifold is the same regardless of what kind of surface signal arrived.
|
||||
|
||||
Current modality status: TEXT is active. VISION, AUDIO, and MOTOR are planned; their adapters register when their bootstrap corpora are ready. Adding a modality requires one adapter file in `sensorium/adapters/` and a registry entry. No existing layer is touched.
|
||||
|
||||
See ADR-0013 for the full specification.
|
||||
|
||||
---
|
||||
|
||||
### X. Mechanical Sympathy: Hardware-Bound Intelligence
|
||||
|
||||
An architecture that fights its underlying silicon is a failed synthesis. CORE is designed for the **Unified Memory Architecture (UMA)** of Apple Silicon.
|
||||
|
||||
|
|
@ -187,7 +250,7 @@ The three-language implementation maps directly onto three execution domains: Py
|
|||
|
||||
---
|
||||
|
||||
### IX. The Deletion Philosophy
|
||||
### XI. The Deletion Philosophy
|
||||
|
||||
The Versor Engine was not built by adding subsystems. It was built by deleting them.
|
||||
|
||||
|
|
@ -199,7 +262,7 @@ The `docs/DELETION_LOG.md` records every deleted subsystem and the algebraic rea
|
|||
|
||||
---
|
||||
|
||||
### X. Extensions
|
||||
### XII. Extensions
|
||||
|
||||
**CORE-Logos** — The language articulation subsystem. Specified in the companion Yellow and White Addenda inherited from `core-ai`. The Logos defines the vocabulary manifold, the token projection law, the holonomy encoder, and the termination condition.
|
||||
|
||||
|
|
|
|||
|
|
@ -209,7 +209,7 @@ Verified in `tests/test_holonomy.py` via property-based testing with Hypothesis.
|
|||
The vocabulary manifold is a finite set of null vectors {v_w} ⊂ Cl(4,1), one per token w in the vocabulary.
|
||||
|
||||
**Construction:** Each word w is embedded as a null vector via the CGA point embedding:
|
||||
1. Obtain a 3D semantic coordinate p_w (from a frozen static embedding or from the manifold's coordinate frame)
|
||||
1. Obtain a 3D semantic coordinate p_w (from a frozen static embedding or from the manifold’s coordinate frame)
|
||||
2. Embed: `v_w = p_w_x·e1 + p_w_y·e2 + p_w_z·e3 + (1/2)|p_w|²·e4 + e5`
|
||||
3. Verify: `v_w · v_w = 0` (null condition)
|
||||
|
||||
|
|
@ -223,7 +223,124 @@ This is a nearest-null-vector scan. For vocabularies up to ~50,000 tokens it is
|
|||
|
||||
---
|
||||
|
||||
### VI. Persona as CGA Motor
|
||||
### VI. The Sensorium — Modality Protocol Specification
|
||||
|
||||
The `sensorium/` layer converts any surface signal into a `(32,)` Cl(4,1) multivector before it reaches `ingest/gate.py`. Every `ProjectionHead` is the Logos-recovery boundary for its modality.
|
||||
|
||||
#### `Modality` Enum
|
||||
|
||||
```python
|
||||
class Modality(enum.Enum):
|
||||
TEXT = "text"
|
||||
VISION = "vision"
|
||||
AUDIO = "audio"
|
||||
MOTOR = "motor"
|
||||
```
|
||||
|
||||
New modalities must be added here AND register a projection head in `sensorium/registry.py` before any pack can mount.
|
||||
|
||||
#### `ProjectionHead[S, F]` Protocol
|
||||
|
||||
```python
|
||||
class ProjectionHead(Protocol[S, F]):
|
||||
modality: Modality
|
||||
embedding_dim: int # must be 32 for Cl(4,1)
|
||||
|
||||
def project(self, signal: S) -> mx.array: # shape (32,)
|
||||
def project_batch(self, signals: list[S]) -> mx.array: # shape (N, 32)
|
||||
def verify_unitarity(self, sample: S) -> bool
|
||||
# True iff V · reverse(V) = ±1 within 1e-6
|
||||
```
|
||||
|
||||
Note: `core-ai` used shape `(2, 2)` complex (Cl(3,0) Pauli isomorphism). `core` uses shape `(32,)` f32 (Cl(4,1) canonical layout).
|
||||
|
||||
#### `ModalityPack[S]` Dataclass
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ModalityPack(Generic[S]):
|
||||
pack_id: str # "en", "he", "grc", "imagenet-1k", ...
|
||||
modality_type: Modality
|
||||
projection: ProjectionHead[S] | None # None for articulation-only modalities
|
||||
decoder: SurfaceDecoder[S] | None # None for perception-only modalities
|
||||
vocabulary: ModalityVocabulary[S] # bidirectional surface ↔ rotor map
|
||||
grammar_scaffold: Any # versor attractors from vocab/
|
||||
checksum_verified: bool
|
||||
gate_engaged: bool = True
|
||||
```
|
||||
|
||||
Frozen + slotted: zero per-instance dict overhead, hashable. Type-parameterised: `ModalityPack[str]` and `ModalityPack[np.ndarray]` are not interchangeable at the type level.
|
||||
|
||||
#### Mount-Time Failure Modes
|
||||
|
||||
| Error | Meaning |
|
||||
|---|---|
|
||||
| `MANIFEST_INVALID` | Pack manifest fails integrity check |
|
||||
| `UNITARITY_VIOLATION` | Projection head produces non-unitary rotor |
|
||||
| `PROJECTION_NOT_CONVERGED` | Projection head did not converge during validation |
|
||||
| `GRADE_DECLARATION_MISMATCH` | Declared grades do not match produced grades |
|
||||
| `MODALITY_NOT_REGISTERED` | Modality not in `sensorium/registry.py` |
|
||||
| `GATE_NOT_ENGAGED` | Surprise-gate not active (non-text modality during seeding) |
|
||||
|
||||
#### Active Modalities
|
||||
|
||||
| Pack ID | Modality | Surface type `S` | Status |
|
||||
|---|---|---|---|
|
||||
| `en` | TEXT | `str` | Active |
|
||||
| `he` | TEXT | `str` | Active (Hebrew depth corpus) |
|
||||
| `grc` | TEXT | `str` | Active (Koine Greek depth corpus) |
|
||||
| vision adapters | VISION | `np.ndarray` | Planned |
|
||||
| audio adapters | AUDIO | `np.ndarray` | Planned |
|
||||
| motor adapters | MOTOR | `np.ndarray` | Planned |
|
||||
|
||||
See ADR-0013 for the full protocol specification.
|
||||
|
||||
---
|
||||
|
||||
### VII. The `core_ingest` Governance Layer — Pre-Gate Specification
|
||||
|
||||
The `core_ingest/` layer wraps upstream of `ingest/gate.py`. The gate is not modified.
|
||||
|
||||
#### `DeterminismClass`
|
||||
|
||||
| Class | Meaning | Auto-Accept Eligible? |
|
||||
|---|---|---|
|
||||
| D0 | Fully deterministic, pinned inputs and code | ✅ |
|
||||
| D1 | Deterministic with pinned external artifact | ✅ |
|
||||
| D2 | Nondeterministic but replay-captured | ❌ |
|
||||
| D3 | External unpinned model or API | ❌ |
|
||||
| D4 | Human / operator proposal | ❌ |
|
||||
|
||||
A D2–D4 frontend is structurally forbidden from claiming `AUTO_ACCEPT_ELIGIBLE`. Enforced in `CandidateGeometricPressure.__post_init__`.
|
||||
|
||||
#### `CandidateGeometricPressure` Content-Addressing
|
||||
|
||||
```
|
||||
pressure_id = SHA-256(full canonical packet) # structural deduplication
|
||||
semantic_key = SHA-256(kind + modality + lemma + subject + verb + object + payload)
|
||||
# convergent-evidence detection
|
||||
```
|
||||
|
||||
Two packets with the same `semantic_key` assert the same claim from different provenance sources. Convergence is tracked by the `IngestCompiler` and surfaced as a confidence signal to downstream consumers.
|
||||
|
||||
#### Three-Gate Validation Flow
|
||||
|
||||
```
|
||||
CandidateGeometricPressure batch
|
||||
→ ProvenanceGate # SourceSpan integrity, SHA-256 of source material
|
||||
→ SemanticGate # span completeness, balanced delimiters, non-empty
|
||||
→ GovernanceGate # ReviewLevel, DeterminismClass, ReviewDecision overrides
|
||||
→ ValidationReport # per-packet disposition
|
||||
→ LearningArtifact # accepted packets → train/ export path
|
||||
```
|
||||
|
||||
#### `StructuralSegmenter` — Why, Not What
|
||||
|
||||
LLM extraction was rejected: a language model upstream of the gate is a D3 nondeterministic oracle whose semantic projections would be silently embedded in the field state. The `StructuralSegmenter` carves at *form* boundaries only — the meaning of a span stays inside the field where it belongs. Biblical texts (Hebrew, Koine Greek) are D0 by construction: canonical verse boundaries are fixed. See ADR-0012.
|
||||
|
||||
---
|
||||
|
||||
### VIII. Persona as CGA Motor
|
||||
|
||||
A CGA **motor** is a versor that encodes a screw motion: a combined rotation and translation in conformal space.
|
||||
|
||||
|
|
@ -238,7 +355,7 @@ Persona application:
|
|||
F_biased = M · F · reverse(M)
|
||||
```
|
||||
|
||||
This rotates and translates the field state within the conformal manifold, biasing generation toward the persona's characteristic region of the vocabulary manifold. It is a single versor product — algebraically closed, no weight overlay, no post-hoc bias vector.
|
||||
This rotates and translates the field state within the conformal manifold, biasing generation toward the persona’s characteristic region of the vocabulary manifold. It is a single versor product — algebraically closed, no weight overlay, no post-hoc bias vector.
|
||||
|
||||
**Motor composition:**
|
||||
```
|
||||
|
|
@ -249,7 +366,7 @@ Personas compose. Two persona motors can be combined into a single motor before
|
|||
|
||||
---
|
||||
|
||||
### VII. The Three-Language Contract
|
||||
### IX. The Three-Language Contract
|
||||
|
||||
| Layer | Language | Entry point | Invariant |
|
||||
|---|---|---|---|
|
||||
|
|
@ -270,7 +387,7 @@ Personas compose. Two persona motors can be combined into a single motor before
|
|||
|
||||
---
|
||||
|
||||
### VIII. Verification Invariants (The Implementation Gate)
|
||||
### X. Verification Invariants (The Implementation Gate)
|
||||
|
||||
These are testable predicates. Every invariant has a corresponding test in `tests/`.
|
||||
|
||||
|
|
@ -282,12 +399,14 @@ These are testable predicates. Every invariant has a corresponding test in `test
|
|||
| Motor condition | `\|\|M·reverse(M) - 1\|\|_F` | < 1e-6 | (in `test_versor_closure.py`) |
|
||||
| CGA distance symmetry | `cga_inner(X,Y) == cga_inner(Y,X)` | exact | `test_cga.py` |
|
||||
| Vault recall self | `recall(V_i, top_k=1)[0] == i` | exact | `test_vault_recall.py` |
|
||||
| Projection unitarity | `\|\|V·reverse(V) - 1\|\|_F` (sensorium mount) | < 1e-6 | `test_sensorium_mount.py` |
|
||||
| Ingest D-class gate | D2–D4 ↛ AUTO_ACCEPT_ELIGIBLE (construction) | exact | `test_core_ingest.py` |
|
||||
|
||||
These are structural contracts, not regression tests. A failing invariant means the algebra is broken, not the behavior.
|
||||
|
||||
---
|
||||
|
||||
### IX. The Rust Acceleration Contract
|
||||
### XI. The Rust Acceleration Contract
|
||||
|
||||
**Performance-critical operations in Rust:**
|
||||
|
||||
|
|
@ -309,7 +428,7 @@ cargo test
|
|||
|
||||
---
|
||||
|
||||
### X. What Was Deleted and Why
|
||||
### XII. What Was Deleted and Why
|
||||
|
||||
The formal record is in `docs/DELETION_LOG.md`. The summary:
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue