b40422e9db
Closes the last open Rust parity gate from ADR-0020. Kernel: new versor_apply_closed_f64 in core-rs/src/versor.rs performs the full sandwich V·F·rev(V) + closure in f64, mirroring Python's algebra.versor.versor_apply + _close_applied_versor exactly: - no null-vector early branch (Python doesn't have one) - unitize_versor with dense-support seed fallback gate - post-unitize versor_condition < 1e-6 recheck - seed_to_rotor on failure, passthrough as last resort PyO3 binding: versor_apply_with_closure_f64 accepts/returns float64 arrays through new extract_f64_slice / f64_array_to_numpy helpers. algebra/backend.py::versor_apply routes through it under CORE_BACKEND=rust. Parity gate re-enabled (was skipped pending this port). 8/8 bit- identical across normalized hot-path + identity-versor cases. Bench (5000 iters, runtime hot path): python: 213.0 us/call rust: 7.4 us/call → 28.8x speedup All lanes green: algebra 132 (was 124+8skip), smoke 54, runtime 19, cognition 57, teaching 17, packs 6. Cognition eval 100% across all metrics. PROGRESS.md updated: versor_apply marked passing; Phase 5 Rust parity track now 5/5 surfaces gated and enabled.
182 lines
8.3 KiB
Markdown
182 lines
8.3 KiB
Markdown
# ADR-0020 — Phase 5 / Rust Parity Sequencing
|
||
|
||
**Status:** Accepted (2026-05-16)
|
||
**Date:** 2026-05-16
|
||
**Authors:** Joshua Shay
|
||
**Depends on:** ADR-0016 (Capability Roadmap), ADR-0019 (Exact
|
||
Vault Recall Acceleration), `docs/PROGRESS.md` Phase 4 exit
|
||
memo.
|
||
|
||
## Context
|
||
|
||
Phase 4 exited 2026-05-16 with three lanes shipped
|
||
(sample_efficiency, long_context_cost, multi_agent_composition)
|
||
and ADR-0019 Stage 1 vectorising vault recall. Two non-trivial
|
||
axes are now unblocked:
|
||
|
||
- **Phase 5 — Curriculum Era.** Open-ended domain acquisition
|
||
(English fluency v5 OOD, Hebrew, Koine Greek, elementary
|
||
mathematics, foundational physics/biology, classical
|
||
literature). Stresses the runtime *as it stands today* on
|
||
semantic breadth and pack scale.
|
||
- **Rust backend parity port.** CLAUDE.md sequencing rule 5:
|
||
*"Add Rust backend parity only after Python semantics are
|
||
locked by tests."* Phase 4 just locked vault recall semantics
|
||
with bit-identity tests; the prior Phase 1–3 work locked
|
||
algebra closure, intent classification, articulation, teaching,
|
||
and trace hashing. The blocker is dissolved.
|
||
|
||
The question is **what order to take these on**. Three
|
||
positions are credible:
|
||
|
||
### Option A — Phase 5 first, Rust parity later
|
||
|
||
Open Phase 5 now. Drive curriculum work on the Python runtime.
|
||
Defer Rust until Phase 5 surfaces a concrete bottleneck that
|
||
indexing/vectorisation cannot dissolve.
|
||
|
||
- **Pro:** Maximum focus on capability expansion. Phase 5 is
|
||
where CORE proves its end-goal claim (listen → comprehend →
|
||
recall → think → articulate → learn → replay) on real domains.
|
||
Every additional language / domain pack is a load-bearing
|
||
capability bet.
|
||
- **Pro:** Python is currently fast enough. Stage 1 vault
|
||
recall is ~20 ms at N=10⁵. No measured Python bottleneck
|
||
blocks Phase 5 work today.
|
||
- **Con:** Phase 5 will balloon the test surface. Re-running
|
||
Phase 1–4 lanes on every release (per the roadmap) will get
|
||
slow. CI feedback latency directly governs willingness to
|
||
refactor — a slow loop encourages unsafe shortcuts that
|
||
CLAUDE.md explicitly warns against.
|
||
- **Con:** Each new pack ships with Python-only semantics.
|
||
When Rust parity lands later, every pack's behaviour will
|
||
need re-verification against the Rust path — more locked
|
||
surface to re-lock.
|
||
|
||
### Option B — Rust parity first, then Phase 5
|
||
|
||
Port the Python backend surfaces to Rust before opening Phase 5.
|
||
Lock parity with bit-identity tests at every ported surface.
|
||
|
||
- **Pro:** Phase 5 then starts on a faster substrate. Every
|
||
curriculum eval re-run is cheaper. Faster feedback supports
|
||
the eval-driven discipline the project rests on.
|
||
- **Pro:** Locks parity at the smallest possible surface area.
|
||
Today the locked Python surface is finite and bounded; after
|
||
Phase 5 it will have grown by every pack, every curriculum,
|
||
every new operator. Porting later means porting more.
|
||
- **Con:** Rust port is itself a non-trivial project. Done
|
||
poorly it introduces a parallel backend whose drift from
|
||
Python is a constant source of incident. CLAUDE.md already
|
||
treats Rust as opt-in for a reason — `CORE_BACKEND=rust`
|
||
must remain explicit and the Python path must remain the
|
||
deterministic default.
|
||
- **Con:** Delays the capability story. No new curricula
|
||
ship until parity is done.
|
||
|
||
### Option C — Parallel: Phase 5 curriculum + Rust parity in independent tracks
|
||
|
||
Open Phase 5 on the Python runtime. In parallel, port one
|
||
backend surface at a time to Rust, gated by bit-identity tests,
|
||
without making Rust the default. Phase 5 curricula run on
|
||
Python until Rust parity is proven per-surface.
|
||
|
||
- **Pro:** Both axes progress. Phase 5 capability bets land
|
||
on schedule. Rust parity grows incrementally, surface by
|
||
surface.
|
||
- **Pro:** Bit-identity gating means a Rust regression cannot
|
||
silently corrupt Python-validated runtime behaviour. The
|
||
Rust path is purely an acceleration; the Python path remains
|
||
the source of truth.
|
||
- **Con:** Two contexts to hold. Demands discipline about
|
||
which surface is being touched at any given time.
|
||
- **Con:** Mid-Phase-5 backend swap (per-surface enablement of
|
||
Rust path) is a real operational complexity that needs
|
||
careful tooling to keep replay determinism intact.
|
||
|
||
## Recommendation
|
||
|
||
**Option C — parallel, with explicit ordering.**
|
||
|
||
Concretely:
|
||
|
||
1. **Open Phase 5.1 (English fluency v5 OOD) immediately.**
|
||
This is the natural successor to Phase 3 v2 grammatical-
|
||
coverage and to ADR-0018's articulation operators. It does
|
||
not depend on Rust.
|
||
2. **In parallel, open a Rust-parity track.** First port:
|
||
`vault_recall` — the surface ADR-0019 Stage 1 just locked
|
||
with bit-identity tests. Port is gated on byte-equal scores
|
||
and identical top-k ordering against the Python path on a
|
||
wide fixture vault. No Rust enablement on `main` until the
|
||
bit-identity test passes under `CORE_BACKEND=rust`.
|
||
3. **Second port:** `geometric_product` and `versor_apply`.
|
||
These are the hottest algebra paths; bit-identity is testable
|
||
against the existing Python closure. Locked by Phase 1–3
|
||
algebra suite.
|
||
4. **Third port:** `cga_inner` (drop-in replacement now that
|
||
the diagonal-metric kernel is the source of truth).
|
||
5. **Defer:** propagation, teaching, trace hashing — these are
|
||
Python-shaped semantics with relatively low computational
|
||
weight; port only if Phase 5 evidence demands it.
|
||
|
||
Phase 5 may also unlock the deferred scope decisions in
|
||
PROGRESS.md ("Code generation" before Phase 5, "Embodiment"
|
||
during Phase 5). Those are separate ADRs; this one only
|
||
governs sequencing.
|
||
|
||
## Decision
|
||
|
||
**Option C — parallel, with explicit ordering.** Confirmed
|
||
2026-05-16.
|
||
|
||
## Consequences
|
||
|
||
- `docs/PROGRESS.md` opens Phase 5 with "Status: IN PROGRESS"
|
||
on the same date this ADR is accepted.
|
||
- A new ADR (numbered after this one) opens to document the
|
||
Rust parity contract per-surface (test discipline, parity
|
||
gate, default-off enablement, replay determinism preservation).
|
||
- The Rust track produces no new runtime behaviour — only
|
||
faster execution of behaviour that the Python path already
|
||
validates. Any divergence is a test failure, not a feature
|
||
request.
|
||
|
||
## Parity status (live)
|
||
|
||
| Surface | Gate file | Status | Dispatch |
|
||
|----------------------|---------------------------------------------------------|-------------------|-------------------------|
|
||
| `vault_recall` | `tests/test_vault_recall_rust_parity.py` | passing | enabled |
|
||
| `cga_inner` | `tests/test_cga_inner_rust_parity.py` | passing | enabled |
|
||
| `geometric_product` | `tests/test_geometric_product_rust_parity.py` | passing | enabled |
|
||
| `versor_condition` | `tests/test_versor_condition_rust_parity.py` | passing (after f64 fold fix) | enabled |
|
||
| `versor_apply` | `tests/test_versor_apply_rust_parity.py` | passing (f64 port, 29× over Python) | enabled |
|
||
|
||
### `versor_apply` f64 parity port — landed
|
||
|
||
The Rust `versor_apply_closed` diverged from Python on two axes
|
||
caught by the bit-identity gate:
|
||
|
||
1. **Precision** — Rust folded the sandwich (V·F·rev(V)) in f32;
|
||
Python in f64.
|
||
2. **Closure structure** — Rust had a null-vector early branch and no
|
||
post-unitize `versor_condition < 1e-6` recheck; Python is the
|
||
inverse (no null branch; recheck + seed-rotor fallback).
|
||
|
||
Resolution: a new `versor_apply_closed_f64` in `core-rs/src/versor.rs`
|
||
performs the full sandwich + closure in f64, mirroring Python's
|
||
`_close_applied_versor` exactly (no null branch, post-unitize
|
||
condition recheck, seed-rotor fallback). Exposed through PyO3 as
|
||
`versor_apply_with_closure_f64`; `algebra/backend.py::versor_apply`
|
||
routes through it under `CORE_BACKEND=rust`. Parity gate
|
||
re-enabled — 8/8 bit-identical. Bench: **29× over Python**
|
||
(213µs → 7.4µs per call on the runtime hot path).
|
||
|
||
## What this ADR does NOT decide
|
||
|
||
- Which Phase 5 curriculum to open *second* (Hebrew vs.
|
||
mathematics vs. physics). Separate scope call once 5.1 ships.
|
||
- The Rust crate layout / dependency choice. That belongs in
|
||
the per-surface Rust parity ADR, not here.
|
||
- Whether to invest in a third backend (e.g., GPU / JAX). Out
|
||
of scope until both Python and Rust paths are mature.
|