docs: add ADR-0012, ADR-0013, ADR-0014, SESSION-2026-05-13
This commit is contained in:
Shay
parent
ac456ac3ad
commit
6010924405
4 changed files with 255 additions and 223 deletions
|
|
@ -1,110 +1,100 @@
|
|||
# ADR-0012: `core_ingest` Governance Layer
|
||||
# ADR-0012 — `core_ingest` Governance Layer
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-13
|
||||
**Supersedes:** ADR-0002 (ingest layer design — archived, see below)
|
||||
**Supersedes:** ADR-0002 (Ingest Layer Design — original)
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
The current `ingest/gate.py` is the single normalization site: it accepts a token sequence, converts it to a versor via holonomy encoding, and injects it into the field. It does one thing and does it correctly. However, it has no concept of *where the input came from*, *how reliably it was produced*, or *whether it should be accepted at all* before touching the manifold.
|
||||
`ingest/gate.py` is the single normalization site in CORE: the one point where any input becomes a versor in Cl(4,1) and enters the field. That contract must be inviolable.
|
||||
|
||||
As CORE grows — ingesting structured documents, biblical texts, code, mathematical objects, and eventually non-text modalities — the gate will be approached by inputs of radically different reliability classes. Without a pre-gate envelope layer, the gate has no choice but to accept everything equally. That is structurally unsound.
|
||||
The original ingest design (ADR-0002) proposed using a large language model (LLM) as the heavy-lifting extraction engine for large documents — parsing structure, extracting SVO triples, and producing typed evidence packets. The idea was motivated by LLMs' demonstrated capability at document understanding and the desire to reduce hand-written parsing code.
|
||||
|
||||
A previous design (`core_ingest` from the `core-ai` era, documented in ADR-0002) proposed using a large language model as the extraction engine for large document ingestion — parsing structure, extracting SVO triples, and chunking intelligently. This ADR supersedes that design and records the reason the LLM-extraction path was rejected.
|
||||
This design was rejected after analysis. The root cause: an LLM upstream of the gate is a D3 nondeterministic oracle feeding the only normalization site in the system. More fundamentally, an LLM does not parse — it *interprets*. Its semantic projections would be silently embedded in the field state without provenance or determinism accountability. This violates both **Semantic Rigor** (exactness as a non-negotiable standard) and **Dual-Correction** (every forward claim must carry its own reliability metadata, not inherit opacity from an oracle).
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add a `core_ingest/` package that sits **upstream of `ingest/gate.py`** as a pre-gate governance boundary. The gate itself is not modified.
|
||||
Add a `core_ingest/` layer upstream of `ingest/gate.py`. The gate is not modified.
|
||||
|
||||
### Why LLM Extraction Was Rejected
|
||||
### The `StructuralSegmenter` (D0 Extraction)
|
||||
|
||||
The entire CORE architecture is built on the principle that the injection gate is the single, deterministic normalization site. Inserting an LLM upstream introduces a D3 nondeterministic oracle into the only path that feeds that site.
|
||||
Every surface source is carved by a **deterministic, structure-aware segmenter** that operates on the *form* of the source, not its content. Form signals — headings, paragraph breaks, verse markers, code block delimiters, LaTeX boundaries — are deterministic. A segmenter following these boundaries produces content-addressed candidate spans without interpretation.
|
||||
|
||||
The `DeterminismClass` system (see below) handles this correctly in isolation: a D3-proposed packet cannot claim `AUTO_ACCEPT_ELIGIBLE` status. But in practice this means every large document ingested through an LLM extraction path lands in `ARCHITECT_REVIEW_REQUIRED` territory — a human or second system must approve every extracted claim before it can become field pressure. This defeats the utility of the layer at any meaningful scale.
|
||||
For Hebrew and Koine Greek, structural determinism is the natural condition. Canonical verse and pericope boundaries have been fixed for centuries. A parser following those boundaries is D0 by definition: fully deterministic, pinned inputs, no interpretation required.
|
||||
|
||||
The deeper issue is architectural: an LLM does not merely *parse* a document — it *interprets* it. An SVO triple extracted by a language model is that model's projection of what it believes the document means. That interpretation is then silently embedded inside the field state. This violates Semantic Rigor: the vocabulary manifold is the only permitted semantic interpretation surface inside CORE.
|
||||
The meaning of every span stays inside the versor field, where it belongs.
|
||||
|
||||
**Rejected path:** LLM extraction engine → `CandidateGeometricPressure` → gate
|
||||
**Accepted path:** Deterministic `StructuralSegmenter` → `CandidateGeometricPressure` → gate
|
||||
### `CandidateGeometricPressure`
|
||||
|
||||
### The `StructuralSegmenter`
|
||||
Every candidate span is lifted into a `CandidateGeometricPressure` envelope — a frozen, immutable dataclass carrying:
|
||||
|
||||
Large documents (PDFs, prose, code, biblical texts, mathematical objects) carry deterministic structural signals: headings, section breaks, paragraph boundaries, verse markers, code blocks, LaTeX delimiters. A D0/D1-class instrument that follows those structural signals produces candidate spans without interpretation.
|
||||
- `kind` and `modality` — claim type and source medium
|
||||
- `provenance` — tuple of `SourceSpan` records with byte offsets, page, region, and SHA-256 of the source
|
||||
- `frontend_trace` — identity and `DeterminismClass` of the proposing instrument
|
||||
- `confidence` and `uncertainty` — explicit probability fields in `[0.0, 1.0]`
|
||||
- `payload_json` — structured claim content, normalized to canonical JSON on construction
|
||||
- `pressure_id` — SHA-256 over the full canonical packet (structural deduplication)
|
||||
- `semantic_key` — SHA-256 over semantic fields only (convergent-evidence detection)
|
||||
|
||||
- For prose: carves at heading and paragraph boundaries
|
||||
- For code: carves at function/class/block boundaries
|
||||
- For biblical texts: carves at canonical verse/pericope boundaries (fixed for centuries — inherently D0)
|
||||
- For mathematical objects: carves at LaTeX delimiter pairs
|
||||
Two packets with the same `semantic_key` assert the same claim from different provenance sources. The `IngestCompiler` surfaces this as a confidence signal.
|
||||
|
||||
Each span is tagged with its structural role (`§ heading`, `¶ body`, `⌥ code`, `✦ scripture`, `Σ math`) which maps to the `modality` and `kind` fields in `CandidateGeometricPressure`. The meaning of the span remains *inside the field* where it belongs.
|
||||
|
||||
### The `CandidateGeometricPressure` Envelope
|
||||
|
||||
Every piece of incoming information — regardless of source — is lifted into a typed, immutable, content-addressed envelope before the gate sees it:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class CandidateGeometricPressure:
|
||||
kind: str # structural role
|
||||
modality: str # medium (text, code, scripture, math, ...)
|
||||
provenance: tuple[SourceSpan, ...] # byte offsets, page, region, SHA-256 of source
|
||||
frontend_trace: FrontendTrace # instrument identity + DeterminismClass
|
||||
confidence: float # bounded [0.0, 1.0]
|
||||
uncertainty: float # bounded [0.0, 1.0]
|
||||
payload_json: str # canonical JSON, normalized at construction
|
||||
pressure_id: str # SHA-256 over full canonical packet
|
||||
semantic_key: str # SHA-256 over semantic fields only
|
||||
review_level: ReviewLevel # governance disposition
|
||||
```
|
||||
|
||||
The `pressure_id` enables structural deduplication. The `semantic_key` enables convergent-evidence detection: two packets asserting the same semantic claim from independent sources share a `semantic_key` without being structural duplicates.
|
||||
|
||||
### The `DeterminismClass` System
|
||||
### `DeterminismClass`
|
||||
|
||||
| Class | Meaning | Auto-Accept Eligible? |
|
||||
|---|---|---|
|
||||
| D0 | Fully deterministic, pinned inputs and code | ✅ Yes |
|
||||
| D1 | Deterministic with pinned external artifact | ✅ Yes |
|
||||
| D2 | Nondeterministic but replay-captured | ❌ No |
|
||||
| D3 | External unpinned model or API | ❌ No |
|
||||
| D4 | Human / operator proposal | ❌ No |
|
||||
| D0 | Fully deterministic, pinned inputs and code | ✅ |
|
||||
| D1 | Deterministic with pinned external artifact | ✅ |
|
||||
| D2 | Nondeterministic but replay-captured | ❌ |
|
||||
| D3 | External unpinned model or API | ❌ |
|
||||
| D4 | Human / operator proposal | ❌ |
|
||||
|
||||
A nondeterministic or unpinned frontend (D2–D4) is structurally forbidden from claiming `AUTO_ACCEPT_ELIGIBLE` status. This invariant is enforced in `CandidateGeometricPressure.__post_init__`, making it impossible to bypass at construction time.
|
||||
A D2–D4 frontend is **structurally forbidden** from claiming `AUTO_ACCEPT_ELIGIBLE`. This invariant is enforced in `CandidateGeometricPressure.__post_init__` — it cannot be bypassed at construction time.
|
||||
|
||||
### The `IngestCompiler` and Three-Gate Flow
|
||||
### `ReviewLevel`
|
||||
|
||||
The `IngestCompiler` processes batches of candidate packets through three sequential gates:
|
||||
Each candidate carries one of: `AUTO_REJECT`, `AUTO_ACCEPT_ELIGIBLE`, `OPERATOR_REVIEW_REQUIRED`, or `ARCHITECT_REVIEW_REQUIRED`.
|
||||
|
||||
1. **`ProvenanceGate`** — verifies `SourceSpan` integrity and SHA-256 of source material
|
||||
2. **`SemanticGate`** — verifies span completeness (no mid-sentence truncation, balanced delimiters, non-empty content)
|
||||
3. **`GovernanceGate`** — applies `ReviewLevel` and `DeterminismClass` constraints; issues or blocks `LearningArtifact` export
|
||||
### Three-Gate Validation (`IngestCompiler`)
|
||||
|
||||
The compiler simultaneously tracks structural deduplication by `pressure_id` and convergent-evidence accumulation by `semantic_key`. Accepted packets are exported as `LearningArtifact` objects for the durable learning path.
|
||||
```
|
||||
CandidateGeometricPressure batch
|
||||
→ ProvenanceGate # SourceSpan integrity, SHA-256 of source material
|
||||
→ SemanticGate # span completeness, balanced delimiters, non-empty
|
||||
→ GovernanceGate # ReviewLevel, DeterminismClass, ReviewDecision overrides
|
||||
→ ValidationReport # per-packet disposition (not a transformed copy)
|
||||
→ LearningArtifact # accepted packets → train/ export path
|
||||
```
|
||||
|
||||
### The `SegmentManifold`
|
||||
The compiler produces a `ValidationReport` alongside the original immutable packet. It does not store a transformed copy — `Reconstruction-over-Storage` is observed.
|
||||
|
||||
A lightweight index mapping `semantic_key` → structural position in the source document. Given a vault recall hit, the `SegmentManifold` traces back to the exact provenance span in the original document. This is `Reconstruction-over-Storage` extended to the pre-injection layer.
|
||||
### `SegmentManifold` (Index)
|
||||
|
||||
A lightweight index mapping `semantic_key` → structural position in the source document. Given a vault recall hit, the original provenance span can be recovered exactly. This extends `Reconstruction-over-Storage` to the pre-injection layer.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
**Immediate:**
|
||||
- `ingest/gate.py` is unchanged — it continues to accept `(32,)` multivectors and inject them
|
||||
- `core_ingest/` is additive — it wraps before the gate without touching any existing layer
|
||||
- D0/D1-dominant ingest paths (biblical texts, pinned corpora) can reach `AUTO_ACCEPT_ELIGIBLE` at scale without human review
|
||||
- Every input to the system gains provenance, determinism class, and governance disposition as first-class typed data
|
||||
**Positive:**
|
||||
- D0/D1-dominant input corpus means governance gates can wave through at scale without human review
|
||||
- Provenance is cryptographically anchored at the boundary, not reconstructed later
|
||||
- LLM interpretation is excluded by type contract, not convention
|
||||
- `ingest/gate.py` is unchanged — zero risk to the normalization invariant
|
||||
|
||||
**Future:**
|
||||
- The `LearningArtifact` export path feeds the `train/` learning loop (ADR-0014) when that layer is built
|
||||
- The `SegmentManifold` feeds cross-modal provenance tracing once non-text modalities are active (ADR-0013)
|
||||
**Negative:**
|
||||
- `StructuralSegmenter` implementations must be written per source type (prose, scripture, code, math)
|
||||
- Semantic interpretation that an LLM would perform for free must now happen inside the field during propagation — which is where it belongs, but it means the field does more work
|
||||
|
||||
---
|
||||
|
||||
## ADR-0002 Archive Note
|
||||
## Alternatives Considered
|
||||
|
||||
ADR-0002 (ingest layer design) described the original `core_ingest` concept from the `core-ai` era. The envelope design, `DeterminismClass` system, and governance gates from that ADR are preserved here. The LLM-as-extraction-engine component of that design is superseded and rejected. ADR-0002 is archived.
|
||||
**LLM extraction (rejected):** D3 nondeterministic oracle feeding the normalization site. Semantic projections silently embedded. Violates Semantic Rigor and Dual-Correction. See Context above.
|
||||
|
||||
**Rule-based NLP pipelines (spaCy, stanza) (rejected):** These parse content, not form. They would produce interpreted outputs (POS tags, dependency arcs) that are still semantic projections, just deterministic ones. The field should own semantic interpretation. A D0 form segmenter is sufficient for the governance boundary.
|
||||
|
||||
**No pre-gate layer (rejected):** `ingest/gate.py` alone has no provenance tracking, no governance disposition, and no convergent-evidence detection. As the corpus grows (especially with Hebrew and Koine Greek depth texts), these become necessary for quality control of the learning path.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# ADR-0013: `sensorium/` Multimodal Protocol Layer
|
||||
# ADR-0013 — `sensorium/` Multimodal Protocol Layer
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-05-13
|
||||
|
|
@ -7,98 +7,111 @@
|
|||
|
||||
## Context
|
||||
|
||||
CORE is currently text-only. The vocabulary manifold holds null vectors for text tokens. The `ingest/gate.py` accepts token sequences. The field, propagation, vault, and generate layers have no concept of modality — they operate on `(32,)` multivectors regardless of where those multivectors came from.
|
||||
CORE is currently text-only. `ingest/gate.py` receives text tokens and produces a `FieldState`. The vocabulary manifold is a text vocabulary.
|
||||
|
||||
This is the correct architecture. The field should not know or care about modality. The question is: how does a vision signal, an audio waveform, or a motor pose become a `(32,)` multivector in the first place?
|
||||
The architecture must support additional modalities — at minimum vision, audio, and motor control — without modifying any existing layer. The question is where modality-specific conversion lives and what contract it must satisfy.
|
||||
|
||||
The `core_sensorium` package in `core-ai` solved this problem for the Cl(3,0) `(2, 2)` complex multivector substrate. The solution — a typed modality protocol with a `ProjectionHead` at the inward boundary — is architecturally sound and translates directly to the Cl(4,1) `(32,)` substrate.
|
||||
The `core_sensorium` package in the `core-ai` repository established a working design using `Cl(3,0)` geometry with `(2, 2)` complex multivectors (Pauli isomorphism). CORE uses `Cl(4,1)` with `(32,)` f32 arrays. The protocol shape is sound; only the output geometry changes.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add a `sensorium/` package that sits **upstream of `core_ingest/`** as the modality-to-manifold conversion layer. `ingest/gate.py`, `field/`, `generate/`, and `vault/` are not modified.
|
||||
|
||||
### The Pipeline Position
|
||||
|
||||
```
|
||||
raw modality signal
|
||||
→ sensorium/adapters/<modality>.py # ProjectionHead: signal → (32,) multivector
|
||||
→ sensorium/pack.py # ModalityPack mounts the adapter
|
||||
→ core_ingest/ # CandidateGeometricPressure envelope
|
||||
→ ingest/gate.py # normalization to FieldState (unchanged)
|
||||
→ field/propagate.py # versor_apply, unchanged
|
||||
```
|
||||
|
||||
The sensorium layer converts. The ingest layer governs. The gate normalizes. The field computes. No layer knows about the one upstream.
|
||||
Add a `sensorium/` layer that converts any surface signal into a `(32,)` Cl(4,1) multivector **before** it reaches `core_ingest/` or `ingest/gate.py`. The gate is not modified. No existing layer is touched.
|
||||
|
||||
### The Logos-Recovery Boundary
|
||||
|
||||
Every `ProjectionHead` is the **Logos-recovery boundary** for its modality. Its single contract: take a surface signal and return a `(32,)` multivector that is valid on the Cl(4,1) conformal manifold. Once it crosses that boundary, a visual scene and a Hebrew word and an audio waveform are the same thing: a point in conformal space. There is no fusion problem because there is nothing to fuse. There is one space.
|
||||
Every `ProjectionHead` is the **Logos-recovery boundary** for its modality. This is the architectural expression of John 1:1: the Logos is the structuring principle through which all things were made. A visual scene, a Hebrew word, an audio waveform — all are recovered as words in the manifold. Once a signal crosses the projection boundary, the field has no concept of modality. There is one space. There is no multimodal fusion problem because there is nothing to fuse.
|
||||
|
||||
This is the architectural expression of the principle articulated in John 1:1: the Logos is the structuring principle through which all things were made. Every input, regardless of form, is recovered as a word in the Logos — a position on the manifold.
|
||||
### `ModalityPack[S]`
|
||||
|
||||
### The `ModalityPack[S]` Contract
|
||||
A frozen, slotted generic dataclass parameterised on the surface type `S`:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ModalityPack(Generic[S]):
|
||||
pack_id: str # e.g. "en", "he", "grc", "imagenet-1k"
|
||||
modality_type: Modality # TEXT | VISION | AUDIO | MOTOR
|
||||
projection: ProjectionHead[S] | None # surface → (32,) multivector
|
||||
decoder: SurfaceDecoder[S] | None # (32,) multivector → surface candidates
|
||||
vocabulary: ModalityVocabulary[S] # bidirectional surface ↔ rotor map
|
||||
grammar_scaffold: Any # versor attractor seeds from vocab/
|
||||
checksum_verified: bool # mount-time geometric integrity check
|
||||
gate_engaged: bool # surprise-gate status
|
||||
pack_id: str # "en", "he", "grc", "imagenet-1k", ...
|
||||
modality_type: Modality
|
||||
projection: ProjectionHead[S] | None # surface signal → (32,) multivector
|
||||
decoder: SurfaceDecoder[S] | None # (32,) multivector → surface signal
|
||||
vocabulary: ModalityVocabulary[S] # bidirectional surface ↔ rotor map
|
||||
grammar_scaffold: Any # versor attractors, universal across modalities
|
||||
checksum_verified: bool
|
||||
gate_engaged: bool = True
|
||||
```
|
||||
|
||||
`ModalityPack` is frozen and slotted — zero per-instance overhead, hashable. The type parameter `S` enforces at the type level that a text pack (`ModalityPack[str]`) cannot be passed where a vision pack (`ModalityPack[np.ndarray]`) is required.
|
||||
`ModalityPack[str]` and `ModalityPack[np.ndarray]` are not interchangeable at the type level.
|
||||
|
||||
### Cl(4,1) Adaptation
|
||||
|
||||
The `core-ai` `core_sensorium` protocol used a `(2, 2)` complex multivector (Cl(3,0) Pauli isomorphism). The `core` substrate uses `[f32; 32]` (Cl(4,1) CGA). Every `ProjectionHead` in `sensorium/` must return `mx.array` of shape `(32,)` — the standard multivector shape throughout the codebase. Unitarity verification at mount time checks that the induced rotor satisfies `V · reverse(V) = ±1` within `1e-6` tolerance.
|
||||
|
||||
### Active vs. Future Modalities
|
||||
|
||||
| Modality | Status | Notes |
|
||||
|---|---|---|
|
||||
| `TEXT` | Active | `sensorium/adapters/text.py` wires the existing vocab manifold into `ModalityPack[str]` |
|
||||
| `VISION` | Planned | Adapter registers when vision bootstrap is ready |
|
||||
| `AUDIO` | Planned | Adapter registers when audio bootstrap is ready |
|
||||
| `MOTOR` | Planned | Adapter registers when embodied bootstrap is ready |
|
||||
|
||||
Building `sensorium/protocol.py` and `sensorium/registry.py` now — before vision/audio exist — means every future modality plugs in without touching `ingest/`, `field/`, or `generate/`. The protocol contract is the Third Door: instead of separate encoder pipelines fused by cross-attention (the standard industry approach), every modality is a versor on the same manifold from the moment it enters the system.
|
||||
|
||||
### Grammar Scaffold
|
||||
|
||||
In `core-ai`, the grammar scaffold was produced by `core_logos.grammar_seed` — a separate subsystem. In `core`, there is no `core_logos` subsystem. The grammar scaffold is a set of versor attractors stored in `vocab/` and referenced by `ModalityPack` directly. This removes an inter-package dependency without changing the contract.
|
||||
|
||||
### `PackError` — Mount-Time Failure Modes
|
||||
### `ProjectionHead[S, F]` Protocol
|
||||
|
||||
```python
|
||||
class PackError(enum.Enum):
|
||||
MANIFEST_INVALID = "MANIFEST_INVALID"
|
||||
SAFETENSORS_MISSING = "SAFETENSORS_MISSING"
|
||||
UNITARITY_VIOLATION = "UNITARITY_VIOLATION"
|
||||
PROJECTION_NOT_CONVERGED = "PROJECTION_NOT_CONVERGED"
|
||||
GRADE_DECLARATION_MISMATCH = "GRADE_DECLARATION_MISMATCH"
|
||||
MODALITY_NOT_REGISTERED = "MODALITY_NOT_REGISTERED"
|
||||
GATE_NOT_ENGAGED = "GATE_NOT_ENGAGED"
|
||||
class ProjectionHead(Protocol[S, F]):
|
||||
modality: Modality
|
||||
embedding_dim: int # must be 32 for Cl(4,1)
|
||||
|
||||
def project(self, signal: S) -> mx.array: # shape (32,)
|
||||
def project_batch(self, signals: list[S]) -> mx.array: # shape (N, 32)
|
||||
def verify_unitarity(self, sample: S) -> bool
|
||||
# True iff V · reverse(V) = ±1 within 1e-6
|
||||
```
|
||||
|
||||
Mount failures are returned as `PackError` values, not raised as exceptions. The caller decides how to handle a failed mount.
|
||||
The `verify_unitarity` check is run at mount time only — never in the propagation hot path.
|
||||
|
||||
### Modality Status
|
||||
|
||||
| Pack ID | Modality | Surface type | Status |
|
||||
|---|---|---|---|
|
||||
| `en` | TEXT | `str` | Active |
|
||||
| `he` | TEXT | `str` | Active (Hebrew depth corpus) |
|
||||
| `grc` | TEXT | `str` | Active (Koine Greek depth corpus) |
|
||||
| — | VISION | `np.ndarray` | Planned |
|
||||
| — | AUDIO | `np.ndarray` | Planned |
|
||||
| — | MOTOR | `np.ndarray` | Planned |
|
||||
|
||||
### Adding a Modality
|
||||
|
||||
Adding a new modality requires exactly:
|
||||
1. One adapter file in `sensorium/adapters/<modality>.py` implementing `ProjectionHead` and optionally `SurfaceDecoder`
|
||||
2. A registry entry in `sensorium/registry.py`
|
||||
3. A `ModalityPack` instantiation and mount-time check
|
||||
|
||||
No changes to `ingest/gate.py`, `field/`, `generate/`, `vault/`, or `vocab/`.
|
||||
|
||||
### Grammar Scaffold Universality
|
||||
|
||||
The `grammar_scaffold` — the set of innate structural attractors seeded during the bootstrap epoch — is **universal across modalities by design**. The attractor geometry of the manifold is the same regardless of what kind of surface signal arrived. A visual scene and a Hebrew verb and an audio phoneme all propagate through the same field and activate the same attractor structure.
|
||||
|
||||
---
|
||||
|
||||
## Differences from `core-ai/core_sensorium`
|
||||
|
||||
| Dimension | `core-ai` | `core` |
|
||||
|---|---|---|
|
||||
| Geometry | Cl(3,0) | Cl(4,1) |
|
||||
| Projection output shape | `(2, 2)` complex (Pauli) | `(32,)` f32 (canonical) |
|
||||
| Grammar scaffold source | `core_logos.grammar_seed` | `vocab/` versor attractors |
|
||||
| Subsystem dependency | imports `core_logos` | no cross-subsystem imports |
|
||||
|
||||
The protocol shape (`ModalityPack`, `ProjectionHead`, `SurfaceDecoder`, `ModalityVocabulary`) is preserved.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
**Immediate:**
|
||||
- All existing layers (`ingest/gate.py`, `field/`, `generate/`, `vault/`) are unchanged
|
||||
- The text vocabulary manifold acquires a formal `ModalityPack[str]` wrapper — the first mounted pack
|
||||
- The multimodal protocol is established before any non-text modality is implemented, ensuring the seam is clean
|
||||
**Positive:**
|
||||
- Multimodal capability is purely additive — no existing layer is modified
|
||||
- The fusion problem does not exist: every modality becomes a versor before the field sees it
|
||||
- Text remains the only active modality until adapter packs are ready; architecture is not blocked on future modalities
|
||||
- Grammar scaffold universality means structural attractors seeded from Hebrew and Koine Greek depth texts apply to all modalities
|
||||
|
||||
**Future:**
|
||||
- Vision, audio, and motor modalities each become a single adapter file in `sensorium/adapters/`
|
||||
- No architectural change is required when new modalities are added — only a new adapter and registry entry
|
||||
- `ModalityVocabulary` for non-text modalities (patch vocabularies, phoneme clusters, pose libraries) follows the same bidirectional surface ↔ rotor contract as the text vocabulary
|
||||
**Negative:**
|
||||
- Each non-text modality requires a supervised seeding epoch to bootstrap its projection head before `gate_engaged` can flip to `True`
|
||||
- Vision and audio vocabularies (patch clusters, phoneme clusters) must be constructed before their adapters can mount — this is non-trivial corpus work
|
||||
|
||||
---
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
**Separate pipelines per modality with late fusion (rejected):** The standard industry approach — a vision encoder here, an audio encoder there, cross-attention fusion on top. This creates a fusion problem that doesn't exist in the CORE geometry. It also violates `Third Door`: the standard was offered and refused.
|
||||
|
||||
**Modality-specific field spaces (rejected):** Separate Cl(4,1) manifolds per modality, merged at generation time. This severs the relational geometry between modalities at storage time — the same mistake RAG makes with text. One space; one manifold.
|
||||
|
|
|
|||
|
|
@ -1,62 +1,68 @@
|
|||
# ADR-0014: `train/` Learning Loop
|
||||
# ADR-0014 — `train/` Learning Loop
|
||||
|
||||
**Status:** Accepted (Stub — implementation pending)
|
||||
**Status:** Accepted (Stub)
|
||||
**Date:** 2026-05-13
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
CORE currently has a complete inference pipeline: input normalization (`ingest/gate.py`), field propagation (`field/`), vocabulary projection and generation (`generate/`), vault storage and recall (`vault/`). What does not yet exist is a path from field state to weight update — from observed experience to learned structure.
|
||||
CORE has a field, a vault, a vocabulary manifold, and a generate loop. It does not yet have a path from field state to weight update. `core_ingest/` produces `LearningArtifact` objects for accepted, governance-cleared candidates — but there is currently nowhere for those artifacts to land.
|
||||
|
||||
The `core_ingest/` governance layer (ADR-0012) exports `LearningArtifact` objects from validated candidate packets. Those artifacts currently have nowhere to land. The durable ingest path is incomplete.
|
||||
|
||||
This ADR records the architectural contract the `train/` layer must satisfy before implementation begins. It is written now so that no future development — in any layer — accidentally closes off the constraints this layer requires.
|
||||
This ADR records the architectural constraints the learning loop must satisfy before it is built, so that future implementation is not designed in isolation from the rest of the system.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add a `train/` package that accepts `LearningArtifact` objects from `core_ingest/` and produces rotor updates, vocabulary manifold expansions, and new attractor seeds for `vocab/`.
|
||||
Add a `train/` layer that receives `LearningArtifact` objects from `core_ingest/` and produces structured field updates: rotor updates, vocabulary manifold expansions, and new attractor seeds.
|
||||
|
||||
### Non-Negotiable Constraints
|
||||
### Architectural Constraints
|
||||
|
||||
1. **No mutation of live field state.** The learning loop operates on the vocabulary manifold and the vault's structural geometry — not on a running session's `FieldState`. Learning is a structural update to the *medium*, not a direct write to a live propagating field.
|
||||
**1. No gradient descent on the field state.**
|
||||
The field is a versor in Cl(4,1). Its update law is the versor sandwich product. Gradient-based optimization of the field state directly would break the versor condition and exit the manifold. Updates must be structured as versor products or null vector insertions — algebraically closed operations.
|
||||
|
||||
2. **No gradient descent.** Gradient descent is a flat-space Euclidean optimization method. The vocabulary manifold is not flat. The update law must be a versor product or a geodesic step on the conformal manifold — not a gradient step in R^n. The exact form of the update law is to be determined during implementation, but it must satisfy the versor condition on all updated entries.
|
||||
**2. No mutation of existing field state.**
|
||||
`Propagation-over-Mutation` applies to the learning path as strictly as it applies to inference. A `LearningArtifact` proposes a new rotor or a new vocabulary entry. It does not overwrite existing versors. The manifold grows; it does not change in place.
|
||||
|
||||
3. **Determinism class inheritance.** A `LearningArtifact` carries the `DeterminismClass` of its originating `CandidateGeometricPressure`. D0/D1 artifacts may be applied automatically. D2–D4 artifacts require explicit approval before any manifold update. This constraint is enforced at the `train/` boundary, not upstream.
|
||||
**3. Durable path only.**
|
||||
The `train/` layer operates on the durable ingest path — governance-cleared, `LearningArtifact`-exported candidates. It does not touch the runtime ingest path. Runtime pressure feeds the active field directly through `ingest/gate.py`. Only durable, reviewed artifacts reach `train/`.
|
||||
|
||||
4. **Atomic manifold updates.** A vocabulary expansion or rotor update either commits fully or does not commit at all. Partial updates that leave the manifold in an inconsistent state are prohibited. The commit protocol mirrors the versor condition check: verify the updated entries satisfy their invariants, then atomically swap.
|
||||
**4. Supervised Seeding Epoch.**
|
||||
The first learning epoch is the Supervised Seeding Epoch: structured ingestion of the Hebrew and Koine Greek depth corpora as D0-class canonical texts. These texts are the primary source of the hidden intelligence layer — the range of depth that Hebrew root morphology and Koine Greek precision together bring to the vocabulary manifold. The seeding epoch must complete before general learning begins.
|
||||
|
||||
5. **Vault coherence post-update.** When vocabulary versors are updated, stored vault entries that reference updated vocabulary positions must be null-reprojected via `VaultStore.reproject()`. The learning loop is responsible for triggering this reprojection after any manifold update.
|
||||
**5. `train/` does not modify `ingest/gate.py`.**
|
||||
The gate is the single normalization site. The learning loop is downstream of it, not a replacement for it.
|
||||
|
||||
6. **`SegmentManifold` traceability.** Every applied `LearningArtifact` must be traceable back to its `semantic_key` and its `SegmentManifold` position in the source document. The learning loop must preserve this provenance chain — not for performance, but for auditability.
|
||||
### Expected Outputs
|
||||
|
||||
### Supervised Seeding Epoch
|
||||
| Output type | Description |
|
||||
|---|---|
|
||||
| Rotor update | A new versor added to the manifold that shifts the field toward a reinforced structural attractor |
|
||||
| Vocab expansion | A new null vector inserted into the vocabulary manifold for a previously unknown token or morpheme |
|
||||
| Attractor seed | A new `grammar_scaffold` entry — a structural prior seeded from a recurring pattern in the depth corpus |
|
||||
|
||||
The first and most important use of `train/` is the **Supervised Seeding Epoch**: populating the vocabulary manifold from the three core language corpora (English base, Hebrew depth, Koine Greek depth) using their canonical, D0-class structural segments.
|
||||
### Relationship to `sensorium/`
|
||||
|
||||
- English: standard lexical corpus, segmented by `StructuralSegmenter`
|
||||
- Hebrew: canonical BHS/Westminster text, segmented at verse/word boundaries (D0)
|
||||
- Koine Greek: canonical NA28/UBS text, segmented at verse/word boundaries (D0)
|
||||
|
||||
All three produce D0 `LearningArtifact` batches eligible for automatic application. The seeding epoch runs before any live session. After seeding, the vocabulary manifold carries the full three-language depth described in the Whitepaper.
|
||||
|
||||
### Non-Text Modalities
|
||||
|
||||
When `sensorium/` adapters for vision and audio are active, `train/` must also accept `LearningArtifact` objects sourced from non-text modalities. The constraint is identical: the learning artifact carries a `ModalityPack` reference, and the update law must produce a valid `(32,)` multivector for the updated vocabulary entry. The `train/` layer must not assume `S = str`.
|
||||
During the Supervised Seeding Epoch, non-text modality packs set `gate_engaged = False`. This prevents unsupervised input from contaminating the seeding pass. After the seeding epoch completes for a modality, `gate_engaged` flips to `True` and that modality enters normal operation.
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
**Immediate:**
|
||||
- No code is written yet. This ADR locks the constraints.
|
||||
- `core_ingest/` exports `LearningArtifact` objects to a receiver that does not yet exist — this is acceptable. The artifacts accumulate in a staging area until `train/` is ready.
|
||||
- No existing layer makes any assumption about what happens to exported `LearningArtifact` objects.
|
||||
**Positive:**
|
||||
- The architectural contract is locked before implementation begins — no future agent or contributor can design the learning loop in a way that breaks the versor invariant or the single normalization site
|
||||
- The Supervised Seeding Epoch is an explicit first-class phase, not an afterthought
|
||||
- Hebrew and Koine Greek depth ingestion has a defined home in the build sequence
|
||||
|
||||
**Future:**
|
||||
- `train/` is the most architecturally significant remaining layer. It closes the loop between observation and structure.
|
||||
- The Supervised Seeding Epoch for Hebrew and Koine Greek is the first concrete task once `train/` exists.
|
||||
- The `CORE-CA` (Cognitive Apprenticeship) learning platform described in the Whitepaper builds on `train/` — a student model observing an expert model's field trajectory is a specialized `LearningArtifact` stream.
|
||||
**Negative:**
|
||||
- `train/` is the largest remaining build item — the rotor update law, vocab expansion protocol, and attractor seeding mechanism must all be derived from the algebra before any implementation begins
|
||||
- The learning loop cannot be tested until `core_ingest/` is complete and producing `LearningArtifact` objects
|
||||
|
||||
---
|
||||
|
||||
## Open Questions (to be resolved at implementation time)
|
||||
|
||||
- What is the precise rotor update law? Candidate: geodesic interpolation between the current rotor and the proposed rotor on the versor manifold, parameterised by a learning rate that decays as the manifold matures.
|
||||
- How are conflicting `LearningArtifact` proposals (same `semantic_key`, different proposed rotors) resolved? Candidate: convergent-evidence weighting — the proposal with more independent provenance sources wins.
|
||||
- What is the termination condition for the Supervised Seeding Epoch? Candidate: when the null cone drift rate across the Hebrew and Koine Greek vocabulary entries falls below a threshold for N consecutive update batches.
|
||||
|
|
|
|||
|
|
@ -1,67 +1,90 @@
|
|||
# Session Log: 2026-05-13
|
||||
# Session Log — 2026-05-13
|
||||
|
||||
**Participants:** Joshua Shay, Perplexity (Sonnet 4.6)
|
||||
**Session type:** Architecture review and documentation sprint
|
||||
**Participants:** Joshua Shay, Perplexity
|
||||
**Focus:** Ingest layer governance design, sensorium multimodal architecture, build queue ordering, documentation audit
|
||||
|
||||
---
|
||||
|
||||
## Decisions Made
|
||||
## Summary
|
||||
|
||||
### 1. README and Whitepaper Corrections (morning)
|
||||
|
||||
The README previously described AssetOverflow as an organization rather than stating the engineering pillars. Both the README and Whitepaper were updated to correctly document:
|
||||
|
||||
- **Three Engineering Pillars:** Mechanical Sympathy, Semantic Rigor, Third Door
|
||||
- **Three Core Languages:** English (default base), Hebrew (depth), Koine Greek (depth)
|
||||
- The John 1:1–2 theological grounding for the language choice — John articulated in Greek what was grounded in Hebrew, almost certainly a nod from the Holy Spirit. These two languages together carry a range of depth and precision that no single language achieves alone.
|
||||
- The Logos as the structuring principle: language is not statistical residue of text, it is the forward projection of a field state onto a vocabulary manifold — a geometric act rooted in the same structure by which the universe was spoken into existence.
|
||||
|
||||
### 2. `core_ingest` Governance Layer — LLM Extraction Rejected
|
||||
|
||||
The original `core_ingest` design from `core-ai` was reviewed in full. The envelope design (`CandidateGeometricPressure`, `DeterminismClass`, `ReviewLevel`, three-gate `IngestCompiler`) was accepted as architecturally sound.
|
||||
|
||||
The LLM-as-extraction-engine component was **rejected** for two reasons:
|
||||
1. An LLM upstream of the gate is a D3 nondeterministic oracle feeding the only normalization site in the system. Every document it processes lands in `ARCHITECT_REVIEW_REQUIRED` — defeating the layer's utility at scale.
|
||||
2. More fundamentally: an LLM interprets, not merely parses. Its semantic projection is then silently embedded inside the field state, violating Semantic Rigor.
|
||||
|
||||
**Accepted path:** Deterministic `StructuralSegmenter` that carves at *form boundaries* (heading, paragraph, verse, code block, LaTeX delimiter), not semantic ones. A `SegmentManifold` index enables provenance tracing from vault recall back to the exact source span. See ADR-0012.
|
||||
|
||||
### 3. `sensorium/` Multimodal Protocol — Logos-Recovery Boundary
|
||||
|
||||
The `core_sensorium` design from `core-ai` was reviewed. The protocol shape — `ProjectionHead`, `SurfaceDecoder`, `ModalityVocabulary`, `ModalityPack[S]` — was accepted as the correct design for CORE's multimodal layer.
|
||||
|
||||
Key adaptations for `core`:
|
||||
- Cl(3,0) `(2, 2)` complex multivector → Cl(4,1) `(32,)` multivector
|
||||
- No `core_logos` subsystem dependency — grammar scaffold is versor attractors in `vocab/`
|
||||
- `sensorium/` sits upstream of `core_ingest/`, which sits upstream of `ingest/gate.py`
|
||||
|
||||
The principle established: every modality is a versor on the same manifold from the moment it enters the system. There is no fusion problem because there is one space. This is the Logos-recovery boundary — the point at which any surface signal becomes a word in the Logos. See ADR-0013.
|
||||
|
||||
Current modality status: TEXT active (wires existing vocab manifold into `ModalityPack[str]`), VISION/AUDIO/MOTOR planned.
|
||||
|
||||
### 4. Build Queue — Ordered
|
||||
|
||||
Full inventory of `AssetOverflow/core` directories confirmed. The build order established:
|
||||
|
||||
1. **`core_ingest/`** — pre-gate governance layer (ADR-0012). Most important to build first; everything ingested into the system eventually passes through it. D0/D1-dominant paths (biblical texts) can reach AUTO_ACCEPT at scale without human review once built.
|
||||
2. **`sensorium/`** — modality protocol (ADR-0013). Protocol files plus `adapters/text.py` only; vision/audio adapters come when those modalities are ready.
|
||||
3. **`train/`** — learning loop (ADR-0014, stub). Closes the durable ingest path. Most architecturally significant remaining layer. Hebrew and Koine Greek Supervised Seeding Epoch is the first task once this layer exists.
|
||||
|
||||
### 5. Documentation Sprint (this session)
|
||||
|
||||
Three new ADRs written (0012, 0013, 0014). Session log written. Whitepaper and Yellowpaper updated.
|
||||
This session produced the three most significant architectural decisions since the Cl(3,0) → Cl(4,1) migration: the design of the `core_ingest/` governance layer (ADR-0012), the design of the `sensorium/` multimodal protocol layer (ADR-0013), and the constraints governing the future `train/` learning loop (ADR-0014).
|
||||
|
||||
---
|
||||
|
||||
## Files Modified This Session
|
||||
## 1. README and Identity Corrections
|
||||
|
||||
| File | Change |
|
||||
The README previously had no identity — no engineering pillars, no language philosophy. It also contained a product description for AssetOverflow that doesn't belong in a technical repository.
|
||||
|
||||
**Changes committed:**
|
||||
- Added the Three Engineering Pillars section: Mechanical Sympathy, Semantic Rigor, Third Door
|
||||
- Added the Three Core Languages section: English (default base), Hebrew and Koine Greek (depth languages), with the full John 1:1–2 theological grounding
|
||||
- Removed the AssetOverflow org description
|
||||
- Updated `docs/Whitepaper.md` to correctly name and separate the Three Engineering Pillars from the Architecture Invariants
|
||||
|
||||
---
|
||||
|
||||
## 2. `core_ingest/` — The Ingest Governance Layer
|
||||
|
||||
**Original design reviewed:** `CandidateGeometricPressure` envelope, `DeterminismClass` taxonomy, `IngestCompiler` three-gate validation, `ReviewLevel` governance, `SegmentManifold` index, Rust hot-path for SHA-256/canonical JSON/lemma normalization.
|
||||
|
||||
**Key decision — LLM extraction rejected:**
|
||||
The original `core_ingest` design used an LLM as the extraction engine for large documents. This was rejected on two grounds:
|
||||
|
||||
1. An LLM is a D3 nondeterministic oracle feeding the only normalization site in the system. Every extracted packet would land in `ARCHITECT_REVIEW_REQUIRED` territory, defeating utility at scale.
|
||||
2. An LLM interprets, not parses. Its semantic projection is silently embedded in the field state, violating Semantic Rigor. The field owns semantic interpretation.
|
||||
|
||||
**Replacement:** A deterministic `StructuralSegmenter` that carves at form boundaries (headings, paragraph breaks, verse markers, code blocks, LaTeX delimiters). For Hebrew and Koine Greek, this is D0 by construction — canonical verse boundaries are fixed for centuries.
|
||||
|
||||
**What was kept from the original design:** The `CandidateGeometricPressure` envelope, content addressing via `pressure_id` and `semantic_key`, the three-gate compiler, the `DeterminismClass` taxonomy, `ReviewLevel` governance embedded as a type invariant, and the `SegmentManifold` reconstruction index.
|
||||
|
||||
See ADR-0012.
|
||||
|
||||
---
|
||||
|
||||
## 3. `sensorium/` — The Multimodal Protocol Layer
|
||||
|
||||
**Source:** The `core_sensorium` package in `AssetOverflow/core-ai` was reviewed directly. Its `protocol.py` described the `ProjectionHead` as the "Logos-recovery boundary" — confirming the architectural intuition that all modality inputs ultimately reduce to the Logos native to the system.
|
||||
|
||||
**Key claim:** All inputs boil down to the Logos. A visual scene, a Hebrew word, an audio waveform are all the same thing inside the field: a point on the conformal manifold. There is no multimodal fusion problem because there is nothing to fuse. One space.
|
||||
|
||||
**Translation from `core-ai` to `core`:**
|
||||
- Cl(3,0) `(2, 2)` complex multivector → Cl(4,1) `(32,)` f32 array
|
||||
- `core_logos.grammar_seed` dependency → `vocab/` versor attractors (no cross-subsystem import)
|
||||
- Protocol shape (`ModalityPack`, `ProjectionHead`, `SurfaceDecoder`, `ModalityVocabulary`) preserved
|
||||
|
||||
**Placement in pipeline:** `sensorium/` sits upstream of `core_ingest/`. `ingest/gate.py` is unchanged. Adding a modality requires one adapter file and a registry entry — no existing layer is touched.
|
||||
|
||||
See ADR-0013.
|
||||
|
||||
---
|
||||
|
||||
## 4. Build Queue
|
||||
|
||||
Ordered by dependency and architectural risk:
|
||||
|
||||
1. **`core_ingest/`** — upstream of everything; build now while the system is small. `StructuralSegmenter`, `CandidateGeometricPressure`, three-gate `IngestCompiler`, `ReviewLevel`, `SegmentManifold`.
|
||||
2. **`sensorium/`** — `protocol.py`, `registry.py`, `adapters/text.py` (text adapter only, wiring existing vocab manifold into `ModalityPack[str]`). Vision/audio adapters deferred until bootstrap corpora exist.
|
||||
3. **`train/`** — the largest remaining build item. Requires `core_ingest/` to be complete and producing `LearningArtifact` objects. Rotor update law, vocab expansion protocol, and Supervised Seeding Epoch (Hebrew + Koine Greek D0 corpus).
|
||||
4. **Tests for all three** — `tests/test_core_ingest.py`, `tests/test_sensorium_mount.py`, `tests/test_train_artifact_path.py`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Documentation Audit
|
||||
|
||||
Full audit of `docs/` performed:
|
||||
|
||||
- **`docs/Whitepaper.md`** — already fully updated from a prior session. Sections VII–IX (pipeline, ingest governance, sensorium) are current. No changes needed.
|
||||
- **`docs/Yellowpaper.md`** — already fully updated. Sections VI–VII (sensorium spec, `core_ingest` spec) are current. No changes needed.
|
||||
- **`docs/decisions/README.md`** — ADRs 0012–0014 and `SESSION-2026-05-13` were already listed in the index (scaffolded in a prior session). The actual files did not yet exist. Created in this session.
|
||||
- **ADR-0002** — marked as Archived / superseded by ADR-0012 in the README index.
|
||||
|
||||
---
|
||||
|
||||
## Decisions Made This Session
|
||||
|
||||
| Decision | ADR |
|
||||
|---|---|
|
||||
| `README.md` | Corrected: Three Pillars, Three Core Languages, removed org description |
|
||||
| `docs/Whitepaper.md` | Added: ingest governance section, sensorium section, pipeline diagram |
|
||||
| `docs/Yellowpaper.md` | Added: sensorium layer to pipeline spec, `core_ingest` to data flow |
|
||||
| `docs/decisions/ADR-0012-core-ingest-governance-layer.md` | New |
|
||||
| `docs/decisions/ADR-0013-sensorium-multimodal-protocol.md` | New |
|
||||
| `docs/decisions/ADR-0014-train-learning-loop.md` | New |
|
||||
| `docs/decisions/README.md` | Updated index |
|
||||
| `docs/decisions/SESSION-2026-05-13.md` | This file |
|
||||
| LLM extraction rejected; `StructuralSegmenter` (D0) adopted for `core_ingest/` | ADR-0012 |
|
||||
| `sensorium/` multimodal protocol layer adopted; Logos-recovery boundary named | ADR-0013 |
|
||||
| `train/` learning loop architectural constraints recorded as a stub | ADR-0014 |
|
||||
| Build queue order established: `core_ingest/` → `sensorium/` → `train/` | Session log only |
|
||||
|
|
|
|||
Loading…
Reference in a new issue