diff --git a/docs/decisions/recognizer-storage-scope.md b/docs/decisions/recognizer-storage-scope.md new file mode 100644 index 00000000..8bda6564 --- /dev/null +++ b/docs/decisions/recognizer-storage-scope.md @@ -0,0 +1,427 @@ +# Scope: Derived Recognizer Storage + +**Status:** Draft v2 / scope-only (not a decision yet — prerequisite for one) +**Date:** 2026-05-24 (v1: initial draft; v2: substrate-liveness audit + HITL machinery cross-reference + forever-running prerequisite reframe + substrate-destination sketch + LOW corrections) +**Author:** CORE agents +**Anchor:** [thesis-decoding-not-generating](../../../.claude/projects/-Users-kaizenpro-Projects-core/memory/thesis-decoding-not-generating.md) (memory) +**Companions:** [teaching-derived-recognition-scope](./teaching-derived-recognition-scope.md), [epistemic-state-taxonomy-scope](./epistemic-state-taxonomy-scope.md) +**Substrate (load-bearing):** ADR-0006 (field energy operator), ADR-0014 (vault promotion policy), ADR-0144 (proposition-graph epistemic carrier) +**Destination:** Mechanism C-class substrate residency (deferred — gated on text-embedding scope) + +--- + +## Why this document exists + +ADR-0144 added the `EpistemicGraph` carrier and ADR-0143 / the recognition +spike (PRs #224, #226) shipped the deterministic anti-unifier and multi- +resolution decoding. The `DerivedRecognizer` defined in +`recognition/anti_unifier.py` is a frozen serializable dataclass that the +pipeline accepts as a parameter (`core/cognition/pipeline.py:121`). Nothing +in main constructs, persists, or shares one across sessions. + +The teaching-derived-recognition scope deferred the storage question with +three candidates (pack / vault / substrate). That candidate set was drafted +without acknowledging the existing thermodynamic substrate already +implemented under ADR-0006 + ADR-0014. **This scope corrects that +omission.** The storage question collapses once the existing substrate is +in view. + +This document defines the question. The answer belongs to the ADR that +follows. + +--- + +## What the existing substrate already provides + +The thermodynamic / crystallization dynamics the project committed to +exist as designed-and-partially-implemented architecture. The honest +status follows. + +### Substrate liveness audit + +| Concern | Mechanism | Status | Verification | +|---|---|---|---| +| Region excitation | Injection pressure raises `EnergyClass` | **Live** | `FieldEnergyOperator` called by `ingest/gate.py:292`, `field/propagate.py:26`, `language_packs/compiler.py:72` | +| Cooling | `recency = clamp(activation_count,0,8)/8 * exp(-age/12)` contribution | **Live** | `core/physics/energy.py:94`, exercised via callers above | +| Crystallization gate | E0/E1 = `vault_candidate`; promotion requires `coherence_residual ≤ residual_threshold` (default 0.05) | **Dormant** | `VaultPromotionPolicy` defined in `core/physics/learning.py`, but `core.physics.learning` is imported by **no module outside `core/physics/`** as of 2026-05-24 | +| Re-activation on recall | Vault recall transiently raises region to E2, then cools | **Specified, not verified live** | Documented in ADR-0006 §"Integration Points"; no code path traced from vault recall back to energy re-injection | + +This is a meaningful correction to v1, which described the entire lattice +as live. The honest read: the **energy half** of the substrate is wired +and load-bearing; the **promotion / crystallization half** is spec-in-code +that nothing calls. Vault content today exists in the vault because +something else put it there, not because the promotion gate evaluated it. + +### What this means for the recognizer-storage ADR + +The ADR that follows this scope must deliver **both**: + +1. **Wire the dormant promotion path** so it actually decides what + crystallizes (this is a load-bearing change in its own right, not a + recognizer-specific concern), AND +2. **Extend the promotion path for recognizers as content type** + (recognizer-specific measurements per the section below). + +Treating (1) as "already in place" is the v1 overclaim this revision +corrects. The recognizer-storage ADR may legitimately decide to scope (1) +into a sibling ADR — but it cannot assume (1) is already done. + +### What the substrate does not yet specify (independent of liveness) + +- **Recognizers as content type.** ADR-0006/0014 were written for field + regions excited by injection pressure. A derived recognizer is a typed + pattern, not a field region. The mapping is open whether or not the + promotion path is wired. +- **Drop-off / deprecation.** Promotion (when implemented) is + one-directional. The substrate has no policy for *removing* crystallized + content that has been at E0 long enough to be deemed deprecated. + +These two gaps plus the wiring debt are the load-bearing unknowns this +scope frames. + +--- + +## Prerequisite: forever-running runtime + +**Forever-running engine, with reboot as a recovery event, not a +control-flow boundary.** Modern computers reboot occasionally; that does +not justify frontloading work into startup or treating session boundaries +as architectural primitives. The engine's capability state is meant to +accumulate over its lifetime and survive reboot without being rebuilt from +scratch. + +**Honest status of this prerequisite in the current codebase.** +CORE today is session-bounded: `core chat` is a CLI invocation; each call +builds a fresh `ChatRuntime` instance (`chat/runtime.py`); packs and +teaching corpora are loaded fresh per invocation; vault is on-disk and +persists across invocations but is reloaded each time. There is no +long-lived process. The forever-running runtime is **the destination +architecture**, not a current property — and it is its own scope, not +something the recognizer-storage ADR can or should land alone. + +Naming it as a prerequisite (not an assumption) means: the +recognizer-storage ADR is gated on the runtime-model scope existing and +committing to forever-running. If that scope lands later or commits +differently, this scope's framing has to be revisited. + +**Operational consequences for storage** (assuming the prerequisite holds): + +- Anything cheap to rebuild from primitives MAY be in-memory only; reboot + cost is bounded. +- Anything expensive to rebuild (HITL-ratified accumulated capability) + MUST persist; reboot reloads, does not re-derive. +- "Re-derive on demand per session" is not a valid pattern — there is no + per-session demand boundary. +- HITL is the narrow async entrypoint for ratification, never bypassed, + never required for runtime continuation. + +--- + +## The reframed question + +> **How does a derived recognizer participate in the existing +> field-energy / vault-promotion lattice, and what extension is needed +> to support HITL-gated drop-off?** + +Not "where does it live" — that question dissolves once the substrate is +in view. The answer is "in the same place every other crystallizable +artifact lives, subject to the same thermodynamic dynamics, plus one +extension." + +--- + +## Three measurements that need definition + +The existing substrate operates on field regions via three measurements: +excitation, coherence residual, and cool-down. For recognizers to +participate, the recognizer-specific analogs of each must be defined. + +### Measurement 1 — Recognizer excitation + +**Natural map:** a `recognize()` call that returns an admission outcome +raises the recognizer's energy class; refusals do not. + +**Framing note — temporal direction.** Field-region excitation in +ADR-0006 is driven by *inputs the region receives* (injection pressure). +The "admission raises energy" framing above treats *outputs the +recognizer produces* as the excitation signal — opposite temporal +direction. The spike should validate this is the right map: if +recognizer-as-input-consumer ("the recognizer was applied to this +input") is the better analog of "the region received injection," then +attempted-application is the natural metric, not admission. Both +candidates below; the framing direction is itself one of the open +questions. + +**Open question:** does *attempted* application (regardless of outcome) +count as excitation, or only admission? Attempted-counts model favors +recognizers that *get tried* often (good for discovery); admission-counts +model favors recognizers that *succeed* often (good for reliability). +Spike should measure both and pick the one whose cool-down dynamics match +the operator's intuition. + +### Measurement 2 — Recognizer coherence residual + +**Non-obvious.** For a field region, coherence residual is geometric (the +operator's output deviation from coherence). A typed pattern with slots +has no such geometry. + +**Candidate definitions** (spike must pick or compose): + +- **Anti-unifier stability under teaching addition.** Add one held-out + example to the teaching set, re-derive, measure structural delta. Low + delta = high coherence. Implementable; spike-testable. +- **Admit/refuse precision against held-out.** Measure false-admit vs. + false-refuse rate against a separate held-out set. Operator-meaningful + but requires the held-out set to be HITL-curated. +- **Multi-resolution agreement.** A recognizer admits at chunk level; + does word-level resolution produce a consistent feature bundle? Chunk- + word agreement = high coherence. + +**Risk:** if no candidate is well-defined, recognizers cannot be promoted +via the existing `VaultPromotionPolicy` and need a parallel promotion +policy. The spike must surface this rather than paper over it. + +### Measurement 3 — Recognizer promotion criteria + +**Natural extension of `VaultPromotionPolicy.decide()`.** Same shape +(cooled + coherent ⇒ promote), parameterized by content type. Either: + +- **Single policy, content-type-aware.** Extend `VaultPromotionPolicy` + with a `content_type` parameter; per-type thresholds. +- **Sibling policy.** New `RecognizerPromotionPolicy` with the same + decision shape. Less coupling, more duplication. + +ADR decides. Scope does not commit. + +--- + +## The drop-off extension (sibling ADR) + +The existing substrate promotes to vault but does not deprecate out of it. +A forever-running learning system needs drop-off, or it accumulates +unbounded crystallized state. + +### Cross-reference to existing HITL machinery + +The project already has a HITL ratification path for teaching artifacts: + +- **ADR-0057** (teaching-chain proposal + review + replay-equivalence + gate) establishes the shape: engine proposes → automated gate + (replay-equivalence) → operator review → ratified-or-rejected → + append-only proposal log. +- **`teaching/store.py`** implements `PackMutationProposal` and + `TeachingStore` for the proposal-and-storage half. +- **`teaching/review.py`** implements `ReviewOutcome` (`ACCEPTED` / + `REJECTED_IDENTITY` / `REJECTED_EMPTY`) and `review_correction()` for + the review half. +- **ADR-0055 / ADR-0056** (inter-session memory + contemplation) supply + the proposal-generation half from the discovery side. + +Drop-off should **reuse this machinery wherever the shape fits**, not +invent a parallel review path. The shape that fits: + +- **Proposal generation:** new — recency-driven (engine notices a + crystallized recognizer that hasn't been excited for N turns), not + correction-driven. A `DeprecationCandidate` analogous to + `PackMutationProposal` but flowing in the deletion direction. +- **Automated gate:** new — *not* replay-equivalence (deletion is not a + surface change in the same way). Candidate gates: "zero admissions on + any held-out corpus we still have," "no recognizer-composition + references from active recognizers." The drop-off ADR specifies. +- **Operator review:** **reuse** — extend `ReviewOutcome` enum with + `ACCEPTED_FOR_DELETION` (or a parallel `DeprecationReviewOutcome` enum + if mixing creates ambiguity), reuse `review_*` plumbing where the + signatures match. +- **Append-only log:** **reuse** — the same append-only discipline + ADR-0057 establishes for proposals applies to deprecation decisions. + +The drop-off ADR's load-bearing originality is the *trigger* (recency) +and the *gate* (not replay-equivalence). The *review-and-log* half is a +small extension to existing machinery, not a parallel path. + +### Proposed shape (not committed) + +1. Crystallized recognizers carry a recency telemetry field (last + excitation timestamp + cumulative excitation count). +2. A `DeprecationPolicy` analog of `VaultPromotionPolicy` evaluates + crystallized entries against a deprecation threshold (e.g., zero + excitations for N turns AND zero admissions for M turns). +3. Deprecation candidates surface as `DeprecationCandidate` proposals + into the existing teaching-review machinery (extended where needed). +4. Once HITL ratifies deletion, the entry is removed from the persistence + layer and the decision is recorded in the append-only log. + +**Non-negotiable:** HITL is the narrow entrypoint, never bypassed. The +engine MAY propose deletion. The engine MAY NOT delete autonomously. + +**HITL latency as load-bearing constraint.** In a forever-running engine +the operator is not always available. The drop-off ADR must specify what +happens when deprecation candidates accumulate faster than HITL can +review them: a queue cap, a rate limit on proposal generation, an +operator alert, or a no-op (let the queue grow). This is more than +backpressure plumbing — it is a real constraint on whether the system +can self-regulate memory growth without an operator in the loop. + +This extension is sized for its own ADR (parallel to ADR-0014, building +on ADR-0057). The storage scope names it but does not specify it. + +--- + +## What this scope explicitly rejects + +- **Recognizer-specific pack format** (`en_math_recognizers_v1`). Packs + are cold startup-loaded artifacts; conflating a *capability* (live, + accumulating) with a *pack entry* (frozen, manifested) breaks the + forever-running model. +- **Vault as recognizer container without acknowledging the substrate.** + An earlier draft of this scope proposed "vault with fixed lookup mode" + — that was an independent reinvention of crystallization without + recognizing ADR-0006/0014. The recognizer is *crystallized via the + existing dynamics*, not stored as a vault content entry with a parallel + lookup mode. +- **Per-session re-derivation.** No session boundary in the forever- + running model. Re-derivation on reboot is acceptable for the in-memory + registry of *not-yet-crystallized* recognizers; crystallized + recognizers reload from disk. +- **Approximate match for recognizer lookup.** Exact structural match + only. The hot/cold distinction is a *storage* property, not a *match* + property; both hot and cold paths return byte-identical structural + matches. + +--- + +## Substrate-resident destination — sketch (illustrative, not committed) + +The scope names substrate-residency (Mechanism C-class) as the +destination once text-embedding scope lands. To keep that destination +honest rather than an IOU, a one-paragraph sketch of what it could +plausibly look like: + +A substrate-resident recognizer is a structural feature of Engine A +itself — not a stored pattern that the engine *consults*, but an +algebraic operator the engine *is*. Two non-exclusive candidates worth +spike-testing once text-embedding exists: + +- **Recognizer as versor.** Anti-unification produces a versor `V_R` + whose sandwich application `V_R · F · reverse(V_R)` on a field state + `F` derived from text returns a state whose grade/null-cone + characteristics indicate admission. Refusal is failure of versor + closure on the input region. Lookup is geometric application, not + table search. +- **Recognizer as null-cone region.** The recognizer defines a region + in the null cone; admission is exact intersection of the input + region with the recognizer region (per ADR-0006's exact-CGA + discipline, no approximate hull). Hot vs. cold becomes a property of + *which regions are currently materialized* in the runtime manifold + versus serialized to vault for re-thaw. + +Both shapes preserve byte-identical replay, both admit exact match (no +approximate predicates), both compose via existing CGA primitives. The +sketch is illustrative; the actual ADR for substrate-residency will live +downstream of text-embedding scope and will revise this gesture against +whatever embedding semantics actually land. + +The point of including the sketch in *this* scope is to prevent +"substrate-resident" from becoming a forever-deferred destination with +no shape. Naming a shape — even tentatively — keeps the destination +honest. + +--- + +## What this scope does NOT commit + +- **Which coherence-residual definition wins.** Three candidates; spike + decides. +- **Single-policy vs. sibling-policy promotion.** ADR decides. +- **Drop-off thresholds.** Sibling ADR decides. +- **In-memory registry data structure.** Implementation detail; the scope + commits to "recognizers participate in the lattice," not to a specific + Python class. +- **Substrate-resident form.** Mechanism C-class, deferred until text- + embedding scope. Named here as the destination so it remains explicit + rather than forgotten. +- **Cross-process / multi-tenant concerns.** Out of scope; assume single + forever-running engine instance for now. + +--- + +## Determinism requirements (non-negotiable) + +The recognizer-storage layer must preserve: + +1. **Byte-identical replay.** Same teaching examples + same input stream + ⇒ same admissions, same refusals, same feature bundles, same trace + hashes. +2. **Reboot-equivalent state.** State after `(boot, run N turns, reboot, + reload)` is byte-identical to state after `(boot, run N turns)` minus + process-memory artifacts. +3. **No drift repair, no hot-path normalization, no approximate match.** + Per CLAUDE.md's normalization rules — the storage layer is a forbidden + site for these patterns. + +--- + +## Risks the spike / first ADR must surface + +- **No clean coherence-residual for typed patterns.** If none of the three + candidate definitions survives spike measurement, recognizers cannot + cleanly join the existing promotion lattice and need a parallel one. + Surface honestly; do not paper over. +- **Cold-path latency (general vault concern, not recognizer-specific).** + Reading a crystallized artifact from disk on re-thaw inherits whatever + performance characteristics current exact-CGA vault recall already + has. Recognizers do not introduce a new latency class — they inherit + the existing one. Surface here only because if vault recall is slow + enough to matter for *content*, it will matter equally for + *recognizers*. Measure once; the measurement applies to both. +- **HITL queue backpressure.** If recognizers crystallize faster than + HITL can ratify deprecation, the registry grows unbounded. The drop-off + ADR must specify a backpressure policy (queue cap? rate limit? + operator alert?). +- **Excitation counting under teaching corpus changes.** If teaching + examples change, recognizer identity changes (`teaching_set_id` + hashes the tokens). Excitation history attached to the old + `teaching_set_id` does not transfer. Acceptable cost or load-bearing + problem? Spike measures. + +--- + +## Open questions for follow-up scopes + +- **Recognizer composition.** When two recognizers admit the same input + with overlapping but distinct feature bundles, what is the composition + rule? (Intersection? Union? HITL arbitration?) +- **Cross-recognizer interference.** Does excitation of recognizer A + affect recognizer B's energy if A and B share teaching examples? Field + regions interact geometrically; recognizers' analog is undefined. +- **Substrate migration path.** When text-embedding scope lands and + Mechanism C becomes viable, what is the migration discipline for + crystallized recognizers? In-place re-encoding? Parallel substrate- + resident copies? +- **Multi-recognizer admission.** Today the pipeline accepts one + `DerivedRecognizer`. The forever-running engine has many. Dispatch / + ranking / fall-through is downstream of this scope but tightly + coupled. + +--- + +## Summary + +The recognizer-storage question reframes once ADR-0006 and ADR-0014 are +in view. Recognizers are a new content type that should participate in +the existing thermodynamic lattice — but with the honest caveat that +only the *energy half* of that lattice is live today; the +*promotion-to-vault gate* is spec-in-code that nothing currently calls. + +The recognizer-storage ADR therefore has to deliver three things, not +one: wire the dormant promotion path, define the three recognizer- +specific measurements (excitation counting, coherence residual, +promotion criteria), and name the HITL-gated drop-off extension as a +sibling ADR that reuses ADR-0057's review machinery rather than +inventing a parallel one. Substrate-residency (sketched, not committed) +is the destination once text-embedding scope lands. Forever-running +runtime is a *prerequisite* (its own scope), not a current property. + +The scope's commitment is to **the question reframed against the +existing substrate, with the substrate's actual liveness honestly +audited**. Answers belong to the spike and the ADR that follows. diff --git a/docs/decisions/teaching-derived-recognition-scope.md b/docs/decisions/teaching-derived-recognition-scope.md index 44234c20..01c22295 100644 --- a/docs/decisions/teaching-derived-recognition-scope.md +++ b/docs/decisions/teaching-derived-recognition-scope.md @@ -564,3 +564,23 @@ and on the epistemic-state audit succeeding. This document does not propose a decision. It defines the question. Per [[feedback-scope-time-is-cheap]]: scope time is cheap. If a fourth buried assumption is hiding here, surface it before the spike commits. + +--- + +## Amendment (2026-05-24) — Connection to existing thermodynamic substrate + +The "Storage layer question (deferred)" section above lists three +candidates (pack / vault / substrate) without acknowledging the existing +thermodynamic substrate already implemented under **ADR-0006** (field +energy operator) and **ADR-0014** (vault promotion policy). That omission +biases the candidate set: all three options are framed as new +infrastructure, when in fact the project already operates a thawed ↔ +crystallized lattice with exponential cooling decay, coherence-residual +gated promotion, and transient re-thaw on recall. + +The storage question is taken up properly in [recognizer-storage-scope]( +./recognizer-storage-scope.md), which reframes it as: *how does a derived +recognizer participate in the existing field-energy / vault-promotion +lattice, and what extension is needed to support HITL-gated drop-off?* +The three-candidate framing in this document is preserved for history; +the recognizer-storage scope supersedes it.