core/docs/decisions/ADR-0054-vault-recall-indexing-batching.md

ADR-0054 — Vault Recall: Matrix-Cache Indexing + Batched API; Holdout Split Wired

Status: Accepted Date: 2026-05-18 Author: Shay

---

Context

Two doctrine-aligned items from CLAUDE.md were still open after ADR-0053:

1. CLAUDE.md item #4 — "Add exact vault recall indexing/batching without approximate search." ADR-0019 Stage 1 vectorised the single-query CGA scan inside algebra.backend.vault_recall, but the deque → ndarray conversion still happened on every recall, and there was no batched-query API. Repeated recalls against a slowly-growing vault paid the conversion cost each call.
2. Holdouts not in the official eval runner. The cognition lane has had a 19-case plaintext holdout file (evals/cognition/holdouts/cases_plaintext.jsonl) since the lane was set up, but core eval cognition --split accepted only dev and public. Holdout numbers existed only via ad-hoc scripts spawned during ADR-0053.

Both items are minimal-doctrine work: no algebra change, no new approximation, no new normalisation, no hot-path repair. Bundled together because both touch the validation/eval surface.

---

Decision

Part 1 — Vault recall indexing + batching

VaultStore matrix cache (vault/store.py).

A lazily-built _matrix_cache: np.ndarray | None is held on the store. It is None initially and after any mutation; the first recall after a mutation rebuilds it via np.asarray(self._versors, dtype=np.float32). Invalidation hooks:

• store() — always invalidates (append shifts the deque view).
• reproject() — invalidates (every entry replaced).
• _rebuild_index() — invalidates (called on max-entries eviction).

The cache is read-only from the recall path; vault_recall receives it via a new optional prebuilt_matrix= kwarg and skips the deque → ndarray conversion when supplied. No shared mutable state is held across calls — the matrix is the same buffer between recalls only while no mutation has happened.

Batched recall (algebra.backend.vault_recall_batch).

New function with signature vault_recall_batch(matrix, queries, top_k) -> list[list[(int, float)]]. Accepts (N, D) matrix and (B, D) (or (D,)) queries, returns one ranked list per query. Scoring uses the same diagonal CGA metric and accumulates in component-serial order:

scores = np.zeros((B, N), dtype=np.float32)
for i in range(D):
    scores += (_CGA_INNER_METRIC[i] * M[:, i])[None, :] * Q[:, i, None]

Folding component-by-component preserves bit-identity with the single-query path's float32 addition order. Tiebreak rule (descending score, ascending index) is identical.

VaultStore.recall_batch.

Public sibling to recall. Same per-query semantics — exact-self- match promotion via the byte-key index, optional min_status filter, score=+inf for exact hits — but the underlying scoring scan is a single component-serial sweep over the cached matrix.

Part 2 — Wire --split holdout

evals/framework.py:

• LaneInfo.holdout_cases_path(version) resolves the first existing of holdouts/cases.jsonl, holdouts/cases_plaintext.jsonl, holdouts/<version>/cases.jsonl. Sealed (*.age) holdouts are not decrypted here — that path stays in evals.holdout_runner.run_holdout, which enforces aggregate-only output by trust-boundary contract.
• run_lane(split="holdout") reads that path and dispatches to the lane's run_lane(cases, config=...) like any other split.

core/cli.py:

• --split argparse choices extended to {"dev", "public", "holdout"}.
• Example added to EPILOG.

---

Why this is doctrine-aligned

• No approximate search. Both the matrix cache and vault_recall_batch are indexing/vectorisation changes only; scoring arithmetic is unchanged.
• No hidden normalisation, no hot-path repair. The cache is invalidated, not "auto-rebuilt to fix drift." reproject() was already the canonical drift-repair path; this ADR only invalidates the cache when it runs.
• No shared mutable state across recalls. The cache buffer is read by vault_recall via a kwarg; nothing in the recall path mutates it. Mutation paths (store / reproject / eviction) clear it explicitly.
• versor_condition < 1e-6 invariant untouched. No field is constructed, normalised, or transformed.
• Holdouts run via the same harness as dev/public. No parallel scoring path was added; the trust boundary on sealed holdouts is preserved by routing plaintext through the standard runner and leaving the encrypted path to holdout_runner.

---

Characterisation

Vault recall — bit-identity gate

tests/test_vault_recall_indexing_batch.py adds 21 tests. The batched path is verified bit-identical to per-query vault_recall across three seeds × 7 queries × N=137 — every index sequence and every float32 score matches exactly.

The pre-existing tests/test_vault_recall_vectorised.py (ADR-0019 Stage 1 gate) continues to pass — the single-query path is unchanged when no prebuilt_matrix is passed.

Eval lanes — first official holdout run

core eval cognition --split holdout
  cases                : 19
  intent_accuracy      : 100.0%
  surface_groundedness :  94.7%
  term_capture_rate    :  70.8%
  versor_closure_rate  : 100.0%

core eval cognition --split dev
  cases                : 13
  intent_accuracy      : 100.0%
  surface_groundedness : 100.0%
  term_capture_rate    :  78.6%
  versor_closure_rate  : 100.0%

core eval cognition --split public
  cases                : 13
  intent_accuracy      : 100.0%
  surface_groundedness : 100.0%
  term_capture_rate    :  91.7%
  versor_closure_rate  : 100.0%

The single surface_groundedness miss on holdouts is the predicted correction_truth_040 case — see ADR-0053 scope-limits. Term capture on holdouts is the next-cheapest pull (echo the corrected- subject lemma in the CORRECTION acknowledgement), candidate for a follow-up ADR.

Lanes (all green)

core test --suite smoke         67 passed
core test --suite cognition    121 passed
core test --suite runtime       19 passed
core test --suite teaching      17 passed
core test --suite packs          6 passed
core test --suite algebra      132 passed

---

Consequences

What changes

• algebra/backend.py gains vault_recall_batch and an optional prebuilt_matrix= kwarg on vault_recall.
• vault/store.py gains a lazy matrix cache, cache-invalidation hooks on mutation paths, and a recall_batch method.
• evals/framework.py gains LaneInfo.holdout_cases_path and a "holdout" branch in run_lane.
• core/cli.py --split now accepts "holdout".

What does not change

• Single-query vault_recall semantics — same scores, same order, same Rust dispatch.
• ADR-0019 Stage 1 bit-identity contract — still gated.
• versor_condition < 1e-6 invariant unaffected.
• Encrypted holdout decryption — still owned by evals.holdout_runner.run_holdout; aggregate-only output contract preserved.
• All five core lanes remain green.
• Cognition eval numbers on dev/public unchanged from ADR-0053.

Scope limits

• No Rust binding for vault_recall_batch yet. Python is the canonical path; a Rust batched binding can be added under a separate ADR with a parity gate analogous to ADR-0019.
• Holdout case_details are written when run via --split holdout because the standard LaneResult.case_details carries the lane runner's output. The trust-boundary doctrine in evals/holdout_runner.py applies to sealed (encrypted) holdouts — the cognition holdout file is plaintext-in-tree by intent (development), so writing details is consistent. Once a sealed cognition holdout exists, callers must use holdout_runner.run_holdout (aggregate-only) instead of framework.run_lane.

---

Cross-References

• ADR-0019 — Stage 1 vectorised single-query path this ADR builds on (if a file by that name does not exist, the contract lives in tests/test_vault_recall_vectorised.py).
• ADR-0053 — last cognition lane work; its scope-limits section predicted the holdout number.

---

Verification

tests/test_vault_recall_indexing_batch.py — 21 tests, all green
tests/test_eval_holdout_split.py          — 10 tests, all green
tests/test_vault_recall_vectorised.py     —  4 tests still green
tests/test_vault_recall_rust_parity.py    —  pre-existing parity gate still green

The non-negotiable field invariant (versor_condition(F) < 1e-6) is preserved: this ADR adds an indexing cache, a batched scan function, and a CLI flag — no algebra change, no field construction, no normalisation.