377201015f
- Add docs/decisions/README.md: ADR format guide and index - Add docs/decisions/ADR-0001-vocab-layer-invariants.md - Add docs/decisions/ADR-0002-ingest-layer-design.md - Add docs/decisions/ADR-0003-coordinate-system-dissolution.md - Add docs/decisions/ADR-0004-rotor-as-operator-not-property.md - Add docs/decisions/SESSION-2026-05-12.md: full timestamped session log
84 lines
3.9 KiB
Markdown
84 lines
3.9 KiB
Markdown
# ADR-0002: Ingest Layer Architecture
|
||
|
||
**Date:** 2026-05-12
|
||
**Status:** Accepted
|
||
|
||
## Context
|
||
|
||
CORE needs a boundary that converts external information — text, code, scripture,
|
||
mathematical objects, audio — into CORE-native pressure before it enters the
|
||
field. The initial `core_ingest` design (from `core-ai`) proposed using a modern
|
||
LLM as the extraction engine for large document ingestion, on the basis that
|
||
current LLMs are strong at structured extraction from long documents.
|
||
|
||
The question was whether to port this design into `AssetOverflow/core`, scrap
|
||
it, or revise it.
|
||
|
||
## Decision
|
||
|
||
Port the structural elements of `core_ingest` into the `ingest/` layer, but
|
||
**replace the LLM extraction engine with a deterministic StructuralSegmenter**.
|
||
|
||
What is retained:
|
||
- `CandidateGeometricPressure` as the canonical pre-injection envelope
|
||
- Dual-path architecture: runtime ingest (transient) vs. durable ingest (governed)
|
||
- Content addressing via SHA-256 `pressure_id` and `semantic_key`
|
||
- `IngestCompiler` with three sequential gates: Provenance, Semantic, Governance
|
||
- `DeterminismClass` (D0–D4) and `ReviewLevel` embedded in packet type contracts
|
||
- `LearningArtifact` as the durable export form
|
||
|
||
What is rejected:
|
||
- LLM as extraction engine for document segmentation and SVO triple extraction
|
||
|
||
What is added:
|
||
- `StructuralSegmenter`: a D0/D1-class instrument per modality that segments
|
||
documents at *form* boundaries (headings, verse markers, code delimiters,
|
||
LaTeX boundaries) rather than semantic ones. Interpretation happens inside
|
||
the field during propagation, not before injection.
|
||
- `SegmentManifold`: lightweight index mapping `semantic_key` → structural
|
||
position in the source document, enabling provenance reconstruction.
|
||
|
||
## Rationale
|
||
|
||
Using an LLM as extractor introduces a **D3 (external unpinned) oracle** at the
|
||
only normalization site in the system. D3 packets cannot claim
|
||
`AUTO_ACCEPT_ELIGIBLE` status — enforced by the type contract — which means
|
||
every LLM-extracted claim requires human review before becoming pressure. This
|
||
defeats the utility at scale.
|
||
|
||
More fundamentally: an LLM doesn’t parse — it *interprets*. Its projection of
|
||
what a document means becomes silently embedded in the field state. That
|
||
violates **Semantic Rigor** (we own our semantics) and **Third Door** (we don’t
|
||
use external models to define our own representations).
|
||
|
||
Structural segmentation is deterministic and model-free. For Hebrew and Koine
|
||
Greek specifically, canonical verse/pericope boundaries are fixed and centuries
|
||
old — a D0 parser by definition.
|
||
|
||
Serves **Propagation-over-Mutation**: incoming claims don’t modify state;
|
||
they are proposed, validated, and either accepted as `LearningArtifact` objects
|
||
or rejected with a full audit trail.
|
||
|
||
Serves **Reconstruction-over-Storage**: the `SegmentManifold` stores enough
|
||
structured state to trace any vault recall back to its exact source provenance
|
||
span without storing full document copies.
|
||
|
||
## Consequences
|
||
|
||
- **Easier:** D0/D1 instruments dominate the ingest path, meaning the governance
|
||
gates can wave through the majority of ingested content at scale without
|
||
human review.
|
||
- **Harder:** Modality-specific `StructuralSegmenter` implementations must be
|
||
built per content type. This is the right cost.
|
||
- **Forbidden:** LLMs, external NLP models, or any D3/D4 instrument producing
|
||
`AUTO_ACCEPT_ELIGIBLE` packets — the `__post_init__` invariant on
|
||
`CandidateGeometricPressure` makes this structurally impossible.
|
||
|
||
## Alternatives Considered
|
||
|
||
- **LLM extraction with human review of all output:** Rejected. Scales to zero.
|
||
- **General-purpose NLP library (spaCy, stanza) for SVO extraction:** Rejected.
|
||
External libraries define the semantics. Third Door.
|
||
- **Scrap ingest layer entirely:** Rejected. The boundary is necessary. Without
|
||
a governed injection point, external information enters the field without
|
||
provenance, confidence, or governance metadata.
|