core/evals/long_context_cost/contract.md

long-context-cost eval lane

What it measures

Per-turn recall latency as a function of vault size. The roadmap flags this as the test of whether vault/store.py is sufficient at scale: if recall is super-linear in N (the number of stored entries), an indexing strategy is required. Per CLAUDE.md, the strategy must be exact — B-tree / suffix array / signature- based bucketing — not approximate recall (HNSW / ANN are forbidden on the runtime path).

Setup

For each vault size N in {10³, 10⁴, 10⁵, 10⁶}:

1. Populate a fresh VaultStore with N synthetic versors of shape (32,) float32. Reprojection is disabled during fill so the benchmark measures recall cost, not fill cost.
2. Run K independent recall queries against the populated vault. Each query is a fresh random versor; the timing budget excludes query generation.
3. Record per-query latency.

Per N the lane reports:

• latency_median_ms
• latency_p95_ms
• latency_max_ms
• latency_mean_ms

Aggregate analysis:

• log-log linear fit over (N, median latency). Slope ≈ 1 indicates linear cost (the expected baseline for exact O(N) scan); slope

  1 indicates super-linear behaviour that requires intervention.

• asymptotic_class — "linear" if slope ∈ [0.85, 1.15], else the inferred class label.

Phase 4 discipline

This is a quantitative-curve lane. No pass/fail threshold beyond the structural gates:

• Every recall must return without exception.
• Recall must not silently degrade (no zero-result returns at scale when the same query at smaller N produced results).

Curve interpretation is left to the reader. The lane's job is to publish the curve and recommend an indexing strategy if super- linear scaling is detected.

Indexing strategy decision

Per CLAUDE.md ("Vault recall is exact and deterministic. Do not add cosine similarity, HNSW, ANN indexes, or approximate recall to the runtime path"), any indexing introduced must be an exact acceleration. Options on the table if the curve calls for it:

• Norm-bucketed pre-filter. Pre-compute the L2 norm of each stored versor; for a query of norm q, only scan buckets whose norm lies in [q - δ, q + δ] for a bound δ derived from the inner-product threshold. Exact: no candidate is dropped that could possibly beat the cutoff.
• Blade-signature index. Group versors by which grade-1 blades dominate; query searches the K buckets whose signature overlaps the query's. Exact: full scan within the candidate set.
• Layered store with promotion. Recently-stored versors live in a fast in-memory tier; older versors in a slower exact-scan tier. Promotion is deterministic by access pattern.

v1 reports the measurement and recommends one of these (or "linear scan is acceptable") with evidence.

Anti-overfitting

• Synthetic versors are generated from a fixed seed for reproducibility.
• The same query distribution is used across all N (queries are not tailored to vault contents).
• K queries are sampled IID; we report variability via p95 and max, not just mean.

Replay determinism

Latency itself is hardware- and load-dependent and is not reproducible bit-for-bit. What is reproducible is the shape of the curve and the asymptotic class. The lane reports the curve from a single run; multi-run statistical analysis is v2 work.

What this lane does NOT measure

• Fill cost (storing N entries) — separate concern, separately bounded by reproject interval policy.
• Memory footprint at N=10⁶ — separate concern.
• Recall quality on realistic vault contents (covered indirectly by every other lane that exercises real cognition pack content).

The lane is narrow: recall latency as a function of vault size.