diff --git a/docs/decisions/ADR-0005-language-pack-contract.md b/docs/decisions/ADR-0005-language-pack-contract.md new file mode 100644 index 00000000..78215497 --- /dev/null +++ b/docs/decisions/ADR-0005-language-pack-contract.md @@ -0,0 +1,167 @@ +# ADR-0005: Language Pack Contract + +**Date:** 2026-05-12 +**Status:** Accepted + +## Context + +CORE operates on three foundational languages: English (default articulation +base), Hebrew (depth pack 1), and Koine Greek (depth pack 2). These are not +equivalent choices from a menu — they are the three specifically chosen +because of the range of depth they bring and because of the theological and +linguistic insight encoded in John 1:1–2, where the universe itself was spoken +into existence and John articulated that reality in both Hebrew and Greek. +This is why and how CORE finds its truth and power in design and +communication — being primary, being a core. + +A loose plugin API is insufficient for these packs. The system needs a strict +contract that specifies what a language pack must provide, what invariants it +must satisfy, and what gates it must pass before activation. + +## Decision + +Adopt a **Language Pack Contract** as the canonical template for all CORE +language packs. + +Each pack is a deterministic, versioned manifold bundle that defines: + +- identity and normalization policy +- lexical inventory (lemma-first, not token-first) +- morphology +- syntax and frame templates +- semantic lift rules into CORE-native pressure +- readback rules from field state to surface language +- validation probes and alignment gates + +The three canonical packs are: + +| ID | Name | Role | +|-----|--------------------|-----------------------------| +| `en`| English | Default articulation base | +| `he`| Hebrew | Depth language pack 1 | +| `el`| Koine Greek | Depth language pack 2 | + +Each pack must implement the same top-level contract and must target shared +field primitives. No pack may define private or opaque semantics that bypass +the shared field. + +## Required Structure + +``` +packs/ + common/ + contracts/ + language-pack.md + schema/ + lemma.schema.json + morphology.schema.json + frame.schema.json + sense.schema.json + probe.schema.json + anchors/ + trilingual-anchor-template.json + en/ + pack.toml + orthography.yaml + lemmas.jsonl + morphology.jsonl + frames.jsonl + senses.jsonl + lift_rules.py + readback_rules.py + validators.py + probes/ + corpora/ + manifest.yaml + he/ (same structure) + el/ (same structure) +``` + +## Validation Gates + +A pack is not considered active because files exist. A pack becomes active +only when it passes the following gates in order: + +1. **Schema gate** — all data files validate against their JSON schemas +2. **Lexical gate** — lemma inventory is non-empty and internally consistent +3. **Morphology gate** — all surface forms trace to a known lemma +4. **Lift gate** — lift rules produce valid `CandidateGeometricPressure` for seed inputs +5. **Readback gate** — readback rules produce grammatical surface from seed field state +6. **Determinism gate** — all D0/D1 paths produce identical output on repeated runs +7. **Alignment gate** — pack publishes anchor records verifiable against trilingual template +8. **Coverage gate** — probe set covers minimum lexical and morphological surface + +## Design Rules + +### 1. Lemma-first +Tokens are surface realizations. Lemmas are the stable lexical unit. The +ingest and field-state layers operate on lemmas, not tokens. + +### 2. Deterministic normalization +Normalization must be deterministic and script-aware. Unicode normal form NFC +is the minimum. Script-specific rules (right-to-left, vowel pointing, accent +diacritics) are defined per pack in `orthography.yaml`. + +### 3. Native morphology +Morphology is part of meaning-bearing structure, not optional metadata. +Hebrew stems (Qal, Niphal, Piel…) and Koine Greek aspects (aorist, perfect…) +are semantically load-bearing. Morphology is a first-class pack surface. + +### 4. Shared field target +Semantic lift must target shared CORE field primitives. No pack-private +semantic space is permitted. What cannot be expressed in shared primitives +must be proposed as a new shared primitive, not hidden inside a pack. + +### 5. Readback is owned locally +Each pack owns its articulation rules and ambiguity management. Readback is +not a global postprocessor — it is part of the pack contract. + +### 6. Alignment is explicit +Packs must publish anchor records that are verifiable against the +trilingual anchor template. Cross-language coherence is a probe surface, +not an emergent hope. + +## Runtime Boundary Honesty + +Files that define a semantic boundary not yet fully specified must raise +`NotImplementedError` at exactly that boundary. This is not fake code — it +is an honest contract: the structure is real, the interface is real, the +unimplemented edge is visible and loud. Nothing is silently skipped. + +## Rationale + +This serves the three engineering pillars: + +- **Mechanical Sympathy** — a small, explicit, cheap-to-validate pack surface + that integrates directly with the field-state model. +- **Semantic Rigor** — all packs target shared field primitives instead of + inventing language-private meanings. Exactness is non-negotiable. +- **Third Door** — standard token-plugin patterns were considered and + rejected. This contract is designed for CORE's geometry specifically. + +## Consequences + +**Easier:** +- Every future language pack follows the same geometry and QA path. +- English, Hebrew, and Koine Greek can be aligned by shared anchors and + deterministic probes. +- Gate failures are explicit and traceable. + +**Harder:** +- Pack creation requires real morphology, lift, and readback logic. +- Shortcuts that bypass field primitives are structurally forbidden. + +**Forbidden:** +- Opaque embedding tables. +- Black-box semantics. +- Pack-local meaning systems that bypass the shared field. + +## Alternatives Considered + +- **Loose plugin API:** Rejected. Too weak to enforce invariants. +- **Dictionary-only packs:** Rejected. Lexicons alone do not articulate or lift. +- **Tokenizer-first packs:** Rejected. Tokens are surface artifacts, not the + stable unit of design. +- **LLM-as-extraction-engine:** Rejected. Introduces a D3 nondeterministic + oracle into the normalization site. See ADR-0002 for the ingest-layer ruling + on determinism classification. diff --git a/docs/decisions/SESSION-2026-05-12-language-packs-addendum.md b/docs/decisions/SESSION-2026-05-12-language-packs-addendum.md new file mode 100644 index 00000000..ed6492e8 --- /dev/null +++ b/docs/decisions/SESSION-2026-05-12-language-packs-addendum.md @@ -0,0 +1,77 @@ +# Session Log Addendum: 2026-05-12 — Language Pack Design + +**Project:** AssetOverflow/core +**Session type:** Architecture design addendum +**Related ADR:** ADR-0005 + +--- + +## 21:02 — Language Pack Direction Confirmed + +**What happened:** +The decision was made to move from general discussion of language integration +toward a formal, meticulous language pack contract for CORE. + +**Why:** +The foundational languages are not optional output skins. They are part of +the native articulation and semantic design of the system. Therefore the next +clean artifact is a strict contract and scaffold, not an informal note. + +**Decision:** +Create ADR-0005 and establish the `packs/` structure with shared contracts, +schemas, trilingual anchor templates, and initial `en`, `he`, and `el` pack +scaffolds. + +--- + +## Design Notes Recorded + +### Language choice rationale + +The three languages were chosen because of the range of depth they bring: + +- **English** is the default base (any language could serve this role in a + custom CORE instance, but English is the default). +- **Hebrew** and **Koine Greek** are the depth languages. John 1:1–2: the + universe itself was spoken into existence, and John articulated it in both + Hebrew thought and Greek expression — almost certainly a nod from the Holy + Spirit. The hidden layer of intelligence in CORE's vocabulary manifold is + grounded in this. This is why and how CORE finds its truth and power in its + design and communication — being primary, being a core. + +### Pack design decisions + +- Packs are **lemma-first**, not token-first. +- Morphology is **native** to the pack contract, especially for Hebrew and + Koine Greek where morphology is semantically load-bearing. +- **Readback belongs to the pack**, not to a global postprocessor. +- Semantic lift must target **shared field primitives**. +- Pack activation is **gate-based**, not file-existence-based. +- Cross-language alignment is an **explicit probe surface**, not an + emergent hope. +- Runtime boundary files raise `NotImplementedError` at unimplemented + semantic edges — honest structure, not fake behavior. + +### LLM as ingest extraction: rejected + +Using a general-purpose LLM as the extraction engine for large document +ingest was considered and explicitly rejected. The reason: it introduces a +D3 nondeterministic oracle at the normalization site, and its interpretations +are silently embedded in the field state without provenance. Semantic Rigor +forbids this. The replacement is a deterministic, structure-aware segmenter +(StructuralSegmenter) that carves at form boundaries — document structure, +verse markers, code delimiters — not at interpreted content boundaries. +For Hebrew and Koine Greek this is even cleaner: canonical verse and pericope +boundaries have been fixed for centuries and constitute D0 segmentation. + +--- + +## Build Order Established + +| Batch | Contents | +|-------|----------| +| 1 | ADR-0005, this session addendum | +| 2 | `packs/common/` — contract, schemas, anchor template | +| 3 | `en`, `he`, `el` pack manifests and orthography files | +| 4 | Lexical seed sets — lemmas, morphology, frames, senses, probes | +| 5 | Runtime boundary files — lift rules, readback rules, validators |