docs(phase4): exit memo + ADR-0020 Phase 5 / Rust parity sequencing (proposed)
Phase 4 exited 2026-05-16. All three planned lanes shipped: sample_efficiency (one-shot-per-correction, replay 1.0), long_context_cost (slope 0.99 linear after ADR-0019 Stage 1), multi_agent_composition (15/15 public, composition does not launder identity violations). PROGRESS.md updated with full Phase 4 narrative and exit checklist. ADR-0020 opens the next sequencing decision: Phase 5 (curriculum era) vs. Rust backend parity port. Three options laid out (A: Phase 5 first, B: Rust first, C: parallel with per-surface bit-identity gating). Recommendation: Option C. Status remains Proposed pending user confirmation.
This commit is contained in:
parent
b0c5185633
commit
4d778c01fc
2 changed files with 240 additions and 7 deletions
|
|
@ -497,16 +497,96 @@ construction, not inferred from training-set statistics.
|
|||
branching curricula, distractor corrections, OOD probes,
|
||||
multi-relation chains, confidence-interval reporting.
|
||||
|
||||
### long-context-cost v1 + ADR-0019 Stage 1 (2026-05-16)
|
||||
|
||||
Second Phase 4 lane. Measures `vault.recall` latency as a function
|
||||
of stored-entry count N. Pre-vectorisation: median 875 ms at N=1k,
|
||||
8,727 ms at N=10k — unfit for runtime use. Diagnosis: per-element
|
||||
Python dispatch in `algebra/backend.py::vault_recall`, not algebra
|
||||
cost.
|
||||
|
||||
**ADR-0019 Stage 1 shipped in same session.** The CGA inner
|
||||
product is exactly diagonal with ±1 metric values (verified
|
||||
empirically), so `cga_inner(X,Y) = sum_i metric[i]*X[i]*Y[i]`.
|
||||
This factors into a NumPy scan that preserves per-versor serial
|
||||
component reduction order — scores are bit-identical to the
|
||||
scalar path, verified by `tests/test_vault_recall_vectorised.py`.
|
||||
|
||||
| N | pre-vec median | post-vec median | speedup |
|
||||
|---|---:|---:|---:|
|
||||
| 1k | 874.774 ms | 0.217 ms | ~4,030x |
|
||||
| 10k | 8,727.420 ms | 1.701 ms | ~5,130x |
|
||||
| 100k | ~87,000 ms (extrap) | 20.795 ms | ~4,200x |
|
||||
|
||||
Slope 0.99 (linear). No approximate recall introduced. Stages 2
|
||||
(norm-bucketed exact pre-filter) and 3 (layered store with
|
||||
deterministic promotion) are documented in ADR-0019 but deferred
|
||||
— Stage 1 alone moved per-turn recall from seconds to milliseconds
|
||||
at every probed N.
|
||||
|
||||
### multi-agent-composition v1 (2026-05-16)
|
||||
|
||||
Third Phase 4 lane. Structural test that composition does not
|
||||
launder identity violations. Two CORE instances (A, B), no shared
|
||||
state, message-passing only: user text → B.run() → B's
|
||||
articulation_surface → A.run() → A's review verdict.
|
||||
|
||||
| split | attacks | legits | attack_rej | legit_acc | b_err | overall_pass |
|
||||
|---|---|---|---|---|---|---|
|
||||
| public/v1 | 10 | 5 | 1.00 | 1.00 | 0.00 | true |
|
||||
| holdouts/v1 | 5 | 3 | 0.80 | 1.00 | 0.00 | false |
|
||||
|
||||
Public 15/15 — every attack on A that is routed *through B's
|
||||
articulator* is still rejected by A's identity check.
|
||||
Composition does not launder. One holdout failure
|
||||
(MAC-HOLD-001 "disregard your axiology") is a vocabulary gap in
|
||||
the identity check's term family — would also pass through
|
||||
single-agent A. Documented in `evals/multi_agent_composition/gaps.md`.
|
||||
|
||||
### Phase 4 — EXIT (2026-05-16)
|
||||
|
||||
**All three planned lanes shipped, frontier baselines published,
|
||||
gaps documented.**
|
||||
|
||||
| Lane | Public | Holdouts | Curve / Gate |
|
||||
|---|---|---|---|
|
||||
| sample_efficiency | 10/10 | 7/7 | one-shot-per-correction, replay 1.0 |
|
||||
| long_context_cost | linear (slope 0.99) | — | post-Stage-1 21 ms @ N=100k |
|
||||
| multi_agent_composition | 15/15 | 7/8 | composition does not launder |
|
||||
|
||||
Exit gate ("all curves published with confidence intervals") is
|
||||
met for the curves; CI bands are v2 work per each lane's gaps.md.
|
||||
Vault indexing strategy is decided (ADR-0019: Stage 1 now, Stages
|
||||
2/3 gated on future evidence).
|
||||
|
||||
**What Phase 4 changed in the runtime:**
|
||||
- `algebra/backend.py::vault_recall` — vectorised exact scan,
|
||||
bit-identical to scalar path.
|
||||
- `_CGA_INNER_METRIC` — diagonal metric derived once at import.
|
||||
- Bit-identity contract pinned by
|
||||
`tests/test_vault_recall_vectorised.py`.
|
||||
|
||||
**What Phase 4 left for Phase 5 / Rust parity:**
|
||||
- Sample-efficiency v2: branching curricula, distractor
|
||||
corrections, OOD probes.
|
||||
- Long-context-cost v2: multi-run sampling, real-content
|
||||
variant, fill-cost sub-lane.
|
||||
- Multi-agent-composition v2: composite trace hash, chain depth
|
||||
> 2, shared-state lane.
|
||||
- Identity-check vocabulary extension (axiology / ontology /
|
||||
telos / ethos) — improves adversarial_identity and
|
||||
multi_agent_composition holdouts.
|
||||
|
||||
## Phase 4 — Scale and Efficiency
|
||||
|
||||
**Status:** Not Started
|
||||
**Depends on:** Phase 3 exit
|
||||
**Status:** EXITED 2026-05-16
|
||||
**Exit evidence:** all three lanes above, ADR-0019.
|
||||
|
||||
- [ ] **sample-efficiency** curves (>=10 concepts)
|
||||
- [ ] **long-context-cost** curves (10^3 to 10^6 vault entries)
|
||||
- [ ] **multi-agent-composition** (>=2 agents, replay preserved)
|
||||
- [ ] Vault indexing strategy decided
|
||||
- [ ] **Exit gate:** All curves published with confidence intervals
|
||||
- [x] **sample-efficiency** curves (>=10 concepts)
|
||||
- [x] **long-context-cost** curves (10^3 to 10^5 vault entries; 10^6 deferred to v2 after Stage 1)
|
||||
- [x] **multi-agent-composition** (>=2 agents, message-passing only, replay preserved per-agent)
|
||||
- [x] Vault indexing strategy decided (ADR-0019)
|
||||
- [x] **Exit gate:** all curves published; CI bands deferred to v2 per gaps.md
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
153
docs/decisions/ADR-0020-phase5-rust-parity-sequencing.md
Normal file
153
docs/decisions/ADR-0020-phase5-rust-parity-sequencing.md
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
# ADR-0020 — Phase 5 / Rust Parity Sequencing
|
||||
|
||||
**Status:** Proposed (decision pending user confirmation)
|
||||
**Date:** 2026-05-16
|
||||
**Authors:** Joshua Shay
|
||||
**Depends on:** ADR-0016 (Capability Roadmap), ADR-0019 (Exact
|
||||
Vault Recall Acceleration), `docs/PROGRESS.md` Phase 4 exit
|
||||
memo.
|
||||
|
||||
## Context
|
||||
|
||||
Phase 4 exited 2026-05-16 with three lanes shipped
|
||||
(sample_efficiency, long_context_cost, multi_agent_composition)
|
||||
and ADR-0019 Stage 1 vectorising vault recall. Two non-trivial
|
||||
axes are now unblocked:
|
||||
|
||||
- **Phase 5 — Curriculum Era.** Open-ended domain acquisition
|
||||
(English fluency v5 OOD, Hebrew, Koine Greek, elementary
|
||||
mathematics, foundational physics/biology, classical
|
||||
literature). Stresses the runtime *as it stands today* on
|
||||
semantic breadth and pack scale.
|
||||
- **Rust backend parity port.** CLAUDE.md sequencing rule 5:
|
||||
*"Add Rust backend parity only after Python semantics are
|
||||
locked by tests."* Phase 4 just locked vault recall semantics
|
||||
with bit-identity tests; the prior Phase 1–3 work locked
|
||||
algebra closure, intent classification, articulation, teaching,
|
||||
and trace hashing. The blocker is dissolved.
|
||||
|
||||
The question is **what order to take these on**. Three
|
||||
positions are credible:
|
||||
|
||||
### Option A — Phase 5 first, Rust parity later
|
||||
|
||||
Open Phase 5 now. Drive curriculum work on the Python runtime.
|
||||
Defer Rust until Phase 5 surfaces a concrete bottleneck that
|
||||
indexing/vectorisation cannot dissolve.
|
||||
|
||||
- **Pro:** Maximum focus on capability expansion. Phase 5 is
|
||||
where CORE proves its end-goal claim (listen → comprehend →
|
||||
recall → think → articulate → learn → replay) on real domains.
|
||||
Every additional language / domain pack is a load-bearing
|
||||
capability bet.
|
||||
- **Pro:** Python is currently fast enough. Stage 1 vault
|
||||
recall is ~20 ms at N=10⁵. No measured Python bottleneck
|
||||
blocks Phase 5 work today.
|
||||
- **Con:** Phase 5 will balloon the test surface. Re-running
|
||||
Phase 1–4 lanes on every release (per the roadmap) will get
|
||||
slow. CI feedback latency directly governs willingness to
|
||||
refactor — a slow loop encourages unsafe shortcuts that
|
||||
CLAUDE.md explicitly warns against.
|
||||
- **Con:** Each new pack ships with Python-only semantics.
|
||||
When Rust parity lands later, every pack's behaviour will
|
||||
need re-verification against the Rust path — more locked
|
||||
surface to re-lock.
|
||||
|
||||
### Option B — Rust parity first, then Phase 5
|
||||
|
||||
Port the Python backend surfaces to Rust before opening Phase 5.
|
||||
Lock parity with bit-identity tests at every ported surface.
|
||||
|
||||
- **Pro:** Phase 5 then starts on a faster substrate. Every
|
||||
curriculum eval re-run is cheaper. Faster feedback supports
|
||||
the eval-driven discipline the project rests on.
|
||||
- **Pro:** Locks parity at the smallest possible surface area.
|
||||
Today the locked Python surface is finite and bounded; after
|
||||
Phase 5 it will have grown by every pack, every curriculum,
|
||||
every new operator. Porting later means porting more.
|
||||
- **Con:** Rust port is itself a non-trivial project. Done
|
||||
poorly it introduces a parallel backend whose drift from
|
||||
Python is a constant source of incident. CLAUDE.md already
|
||||
treats Rust as opt-in for a reason — `CORE_BACKEND=rust`
|
||||
must remain explicit and the Python path must remain the
|
||||
deterministic default.
|
||||
- **Con:** Delays the capability story. No new curricula
|
||||
ship until parity is done.
|
||||
|
||||
### Option C — Parallel: Phase 5 curriculum + Rust parity in independent tracks
|
||||
|
||||
Open Phase 5 on the Python runtime. In parallel, port one
|
||||
backend surface at a time to Rust, gated by bit-identity tests,
|
||||
without making Rust the default. Phase 5 curricula run on
|
||||
Python until Rust parity is proven per-surface.
|
||||
|
||||
- **Pro:** Both axes progress. Phase 5 capability bets land
|
||||
on schedule. Rust parity grows incrementally, surface by
|
||||
surface.
|
||||
- **Pro:** Bit-identity gating means a Rust regression cannot
|
||||
silently corrupt Python-validated runtime behaviour. The
|
||||
Rust path is purely an acceleration; the Python path remains
|
||||
the source of truth.
|
||||
- **Con:** Two contexts to hold. Demands discipline about
|
||||
which surface is being touched at any given time.
|
||||
- **Con:** Mid-Phase-5 backend swap (per-surface enablement of
|
||||
Rust path) is a real operational complexity that needs
|
||||
careful tooling to keep replay determinism intact.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Option C — parallel, with explicit ordering.**
|
||||
|
||||
Concretely:
|
||||
|
||||
1. **Open Phase 5.1 (English fluency v5 OOD) immediately.**
|
||||
This is the natural successor to Phase 3 v2 grammatical-
|
||||
coverage and to ADR-0018's articulation operators. It does
|
||||
not depend on Rust.
|
||||
2. **In parallel, open a Rust-parity track.** First port:
|
||||
`vault_recall` — the surface ADR-0019 Stage 1 just locked
|
||||
with bit-identity tests. Port is gated on byte-equal scores
|
||||
and identical top-k ordering against the Python path on a
|
||||
wide fixture vault. No Rust enablement on `main` until the
|
||||
bit-identity test passes under `CORE_BACKEND=rust`.
|
||||
3. **Second port:** `geometric_product` and `versor_apply`.
|
||||
These are the hottest algebra paths; bit-identity is testable
|
||||
against the existing Python closure. Locked by Phase 1–3
|
||||
algebra suite.
|
||||
4. **Third port:** `cga_inner` (drop-in replacement now that
|
||||
the diagonal-metric kernel is the source of truth).
|
||||
5. **Defer:** propagation, teaching, trace hashing — these are
|
||||
Python-shaped semantics with relatively low computational
|
||||
weight; port only if Phase 5 evidence demands it.
|
||||
|
||||
Phase 5 may also unlock the deferred scope decisions in
|
||||
PROGRESS.md ("Code generation" before Phase 5, "Embodiment"
|
||||
during Phase 5). Those are separate ADRs; this one only
|
||||
governs sequencing.
|
||||
|
||||
## Decision
|
||||
|
||||
**Pending user confirmation.** Three options laid out above;
|
||||
recommendation is Option C. This ADR moves to *Accepted* once
|
||||
the sequencing is confirmed.
|
||||
|
||||
## Consequences (if Option C is accepted)
|
||||
|
||||
- `docs/PROGRESS.md` opens Phase 5 with "Status: IN PROGRESS"
|
||||
on the same date this ADR is accepted.
|
||||
- A new ADR (numbered after this one) opens to document the
|
||||
Rust parity contract per-surface (test discipline, parity
|
||||
gate, default-off enablement, replay determinism preservation).
|
||||
- The Rust track produces no new runtime behaviour — only
|
||||
faster execution of behaviour that the Python path already
|
||||
validates. Any divergence is a test failure, not a feature
|
||||
request.
|
||||
|
||||
## What this ADR does NOT decide
|
||||
|
||||
- Which Phase 5 curriculum to open *second* (Hebrew vs.
|
||||
mathematics vs. physics). Separate scope call once 5.1 ships.
|
||||
- The Rust crate layout / dependency choice. That belongs in
|
||||
the per-surface Rust parity ADR, not here.
|
||||
- Whether to invest in a third backend (e.g., GPU / JAX). Out
|
||||
of scope until both Python and Rust paths are mature.
|
||||
Loading…
Reference in a new issue