core/docs/plans/vision-compiler-eval-plan.md

Vision Compiler Eval Plan — vision_core_v1

Companion to: ADR-0197, vision-compiler-spec.md Status: Proposed (PR-1 docs)

This plan defines the seeding corpus, the acceptance gates that lift vision_core_v1 from gate-closed to gate-engaged, the Delta-CRDT proof obligations, and the teacher-migration policy. It is the last PR-1 companion ADR-0197 names; it is what PR-2's "determinism + versor tests" gate is measured against.

---

1. Seeding corpus — visual atoms, not captions

Small, curated, checksum-locked. Four tiers:

• Tier A — structural atoms: flat field, oriented edge (each orientation bin), corner/junction, blob/region onset, open vs. closed contour, low- vs. high-spatial-frequency band.
• Tier B — region & figure–ground: salient figure on ground, background texture, occlusion boundary, symmetry, periodic repetition, scale-change across pyramid levels.
• Tier C — color/material/lighting: hue/saturation regimes, high vs. low luminance contrast, shadow/highlight, specular vs. matte, gradient ramp vs. hard step.
• Tier D — alignment anchors: caption hypothesis, detector box, segment mask, OCR text span, linked text surface (only when alignment is trustworthy).

Tier D is auxiliary; object/lexical semantics is a later enrichment step. Synthetic fixtures (oriented sinusoidal gratings, checkerboards, luminance gradients, single-dot impulses, step edges, solid color fields, controlled occlusions) drive first-pass determinism; checksum-locked real fixtures cover what synthesis does poorly (natural texture, soft occlusion, cluttered figure–ground).

2. Acceptance gates (gate-engaged criteria)

Gate	Pass criterion	Obligation
Projection shape	exactly (32,)	V-1
Projection dtype	exactly float32	V-1
Compiler replay	bit-identical on same platform/build	V-1
Cross-platform stability	equal after quantization, within declared numeric tolerance	V-1
Canonical-ordering stability	re-tiling the same canonical image yields the same event order	V-4
versor_condition	< 1e-6 (never weakened)	V-5
Canonical checksum stability	100% on fixture corpus (colorspace + resize pinned)	V-1
Gate closure	projection blocked when gate_engaged = false	—
Mount validation	bad checksum or bad unitarity blocks pack mount	—
Trace hygiene	no raw pixel bytes in any turn trace	V-6
IR replay	VisionIR -> versor replays identically from stored IR	V-1

These mirror the existing modality-test posture (sensorium/registry.py mount/gate enforcement, ModalityPack.__post_init__ invariants) — no new gating machinery.

3. Delta-CRDT proof obligations (ADR-0197 §4.2)

Each must be able to fail loudly under the violation it names (CLAUDE.md §Schema-Defined Proof Obligations — no decorative tests):

ID	Obligation	Fails if…	ADR-0180 analog
V-1	Determinism: same canonical pixels + pack ⇒ byte-identical (32,) across calls/threads/processes	dict ordering, unpinned resize/colorspace, or float reduction order leaks in	T-4
V-2	Set-equality of merges: a set of tile units folds to the same Vault state for any arena flush permutation	a delta's contribution is order-sensitive at the merge layer	T-1
V-3	Content-addressed trace-hash: invariant under set-equal Vault states when keyed by (canonical, ir, projection) sha	the reduction consumes deltas in arrival order	T-2
V-4	Serialization barrier + canonical spatial order: in-chunk compile_events is order-sensitive (negative test), and the §6 spatial order is itself deterministic and re-tiling-stable	swapping two spatial-order events fails to change the versor, or re-tiling the same image reorders events	T-3
V-5	versor_condition < 1e-6 on every emitted unit	the threshold is weakened to pass	—
V-6	Trace hygiene: no pixels in any TurnEvent/Vault record	raw pixel data leaks into a delta's provenance	§1.5.5

V-4 carries one clause audio's A-4 did not: vision has no temporal axis, so the canonical spatial order (vision-compiler-spec §6) is the thing that makes the fold reproducible, and it must be tested as such.

3.1 The sequential==concurrent proof (PR-5 acceptance)

The load-bearing test for the CRDT mapping:

ingest tiles [t1, t2, t3] sequentially          → vault_seq,  trace_hash_seq
ingest the same tiles across N arenas merged     → vault_conc, trace_hash_conc
assert set(vault_seq) == set(vault_conc)                       # V-2
assert trace_hash_seq == trace_hash_conc                       # V-3
assert re-ingesting any tile is a no-op on the vault           # idempotence (ADR-0197 §4.3)

This is the vision instance of ADR-0180 §4.3's hash(Sequential_Ingest) == hash(Concurrent_CRDT_Ingest) and must pass before PR-5 merges. Because tiles are spatially independent, this proof is the first non-temporal exerciser of the substrate's order-invariance (ADR-0197 §3.1).

3.2 Pytest skeleton

def test_vision_projection_is_deterministic(vision_tile, vision_pack):          # V-1
    v1 = vision_pack.projection.project(vision_tile)
    v2 = vision_pack.projection.project(vision_tile)
    assert v1.shape == (32,)
    assert v1.dtype.name == "float32"
    assert np.array_equal(v1, v2)

def test_vision_pack_gate_blocks_projection(vision_pack_closed, vision_tile):   # gate closure
    with pytest.raises(Exception):
        _ = vision_pack_closed.projection.project(vision_tile)

def test_vision_ir_replay_matches_original(vision_tile, compiler):               # IR replay
    unit = compiler.compile_tile(vision_tile)
    replay = compiler.compile_ir(unit.vision_ir)
    assert np.array_equal(unit.versor, replay.versor)
    assert unit.ir_sha256 == replay.ir_sha256

def test_chunk_composition_is_order_sensitive(two_events, compiler):             # V-4 barrier
    a = compiler.compile_events([two_events[0], two_events[1]])
    b = compiler.compile_events([two_events[1], two_events[0]])
    assert not np.array_equal(a, b)

def test_canonical_spatial_order_is_retiling_stable(vision_fixture, compiler):   # V-4 spatial clause
    order_a = compiler.canonical_event_order(vision_fixture)
    order_b = compiler.canonical_event_order(compiler.recanonicalize(vision_fixture))
    assert [e.stable_event_id for e in order_a] == [e.stable_event_id for e in order_b]

def test_merge_is_permutation_invariant(units):                                 # V-2
    import itertools
    states = {fold_into_vault(p) for p in itertools.islice(_perms(units), 8)}
    assert len(states) == 1

4. Teacher / shadow lane policy (PR-6)

Teachers label or align; they never define the substrate and never fold embeddings into the main versor path. They are admitted only through typed, versioned, checksummed hints stored as content_anchors / evidence_ids in the IR.

Source	Best role in CORE	Why not the substrate
CLIP	coarse scene/object labels, image-text alignment prototypes	embedding model; opaque latent violates the design goal
DINOv2	self-supervised region/correspondence hints	dense embedding; not a checksummable typed visual IR
ViT (supervised)	classification evidence, weak object labels	latent features, no content-addressed key
SAM	region/segment proposals as region anchors	non-deterministic across versions; gives no checksummable key
Depth Anything / RAFT	depth/flow evidence lanes (and the seam for future video)	estimators, not a CORE-native deterministic compiler
Tesseract / OCR	text-span anchors when glyphs are present	text extraction, wrong primary ontology for vision

Migration policy, verbatim (identical to audio):

Use teachers to label or align.
Never let teachers define the substrate.
Never fold teacher embeddings directly into the main versor path.
Only admit teacher outputs through typed, versioned, checksumed hints.

This is the line ADR-0197 §3.2 names as the primary risk: pretrained vision encoders are strong enough to tempt substrate use. They stay in this table, never on the versor path.

5. Phased sequence (priority order, no calendar)

1. Doctrine — ADR + spec + eval plan locked (PR-1, this).
2. Deterministic substrate — canonicalizer, checksums, resample, grid, lexer, parser, operators, compiler (PR-2).
3. Governance — pack artifacts, adapter, mount/gate/checksum tests (PR-3).
4. Evaluation — fixtures, expected IR, expected projection hashes, gate table (PR-4).
5. Delta-CRDT wiring — arena + merge key + sequential==concurrent proof (PR-5), gated on ADR-0180 §1.5.4 (T-1…T-4) green on main.
6. Auxiliary lanes — CLIP/DINOv2/SAM/depth/OCR teachers behind optional extras (PR-6).
7. Streaming — stateful incremental compiler preserving continuity across frames/seams (later; v1 is single-frame/whole-image, offline).