core/docs/research/sme-experiment-verdict-797ebad5.md

ADR-0252 §5 — the structure-mapping experiment, run to a verdict

Date: 2026-07-28 · Runner: Opus 5 (Track A, docs/assessment/50-execution-plan.md §6) Base: forgejo/main @ 797ebad5 · Criterion: pre-registered at 299c92be, before the run Artifact: evals/structure_mapping/adr0252_s5/results/report-797ebad5.json deterministic_digest: b3d9d27592e213104c51bcace7415fd819d722a1b06f7d0a5ca65bd5a234e4ec Reproduce: uv run python -m evals.structure_mapping.adr0252_s5.experiment

---

VERDICT: NO-GO

Both embedding variants fail. Structure-sensitivity (§5.3c) fails at every attribute weight tested — it is not a knife-edge on one constant. Per ADR-0252 §5.4, "a well-controlled NO-GO is a full-credit result": it refutes geometric SME for comprehension in the embedding class tested and redirects stage 2 to a different representation.

For ratification. This document authorizes nothing on its own.

---

1. What was already there, and why it could not stand

The plan recorded §5 as "authorized, scaffolded, and unrun." It is scaffolded and authorized. It is not unrun — it has been run twice, and both runs returned GO on unmerged branches:

Attempt	Branch @ SHA	Claimed	Why the claim does not hold
1	rnd/structure-mapping-experiment @ fc9d0c14	GO	Disowned by its own successor: the S1–S4 label entered the embedding via hand-typed "canonical" matrices, so all cases of a structure were the same byte-array. Procrustes on identical matrices is trivially zero.
2	rnd/sme-experiment-v2 @ 96e5f468	GO	Genuinely blind — and then except ValueError: res = 1000.0. Its rule cross_mean > same_mean*2 and same_mean < 1.0 is satisfied by that sentinel alone. A solver exception is not a distance.

Attempt 2 carries four further defects, each of which independently voids it:

1. Wrong branch of the instrument. It passed mixed versor/point sequences to conformal_procrustes, which dispatches those to field conjugacy W F W⁻¹ — the branch that raises field conjugacy versor not closed. Its entire separability signal is that raise.
2. Corpus provenance. Its doc says "51 actual problem graphs from the holdout_dev/v1 corpus." 5 of 51 carry holdout ids; 46 carry bare integers (1, 2, 9, … 2087) matching nothing in that corpus, or gma-* ids belonging to the public lane. §5.6's discipline is "holdout_dev/v1 only", and §5.1 permits synthetic variants only "marked as such." Nothing is marked.
3. Duplicate data. At least three exact-duplicate graphs under different ids (1≡51, 2≡60, 17≡46) and three colliding ids (gma-115, gma-121, gma-135 each appear twice with different graphs, so its id → label dict silently drops one of each pair). Duplicate graphs align at residual 0 by construction — attempt 1's tautology re-imported through the data instead of through the embedding.
4. Unreproducible. Its scripts/extract_sme_corpus.py is cited as an artifact and was never committed. It also scored only the first 20 rows.

None of this was caught by re-reading the documents. It was caught by reading the script, the corpus file, and the ids.

2. The coverage fact that shaped this run

Measured at 797ebad5: the serving reader returns a selected graph for 5 of the 500 holdout_dev/v1 cases — 1.0% — and all five carry the same relational skeleton (compare_multiplicative; they are exactly the organ cohort in evals/structure_mapping/scoring/labels.py). On the public lane the same pipeline yields 24/150.

So §5.1's corpus — "minimum four structures with clean labels … drawn from holdout_dev/v1" — is not extractable today. That is almost certainly why attempt 2 reached for other lanes; the mistake was doing it silently.

This run uses §5.1's own escape hatch, visibly: 5 real cases extracted through parse_and_solve, 48 synthetic from four declared templates, every one carrying provenance="synthetic" and pinned by tests/test_adr_0252_s5_blindness.py.

Consequence, stated rather than buried: this verdict speaks about the geometry, not about coverage. §5.5's compression estimate ("how many holdout families collapse to how few canonical structures") is not answerable at 1.0% reader coverage and was not attempted.

3. The instrument, characterised before use

conformal_procrustes has two branches and they are not interchangeable:

Input	Branch	Behaviour
(5,K) grade-1 point cloud	Kabsch + Umeyama	never raises; invariant to rotation, translation and uniform scale
mixed versor/point sequence	field conjugacy	raises on unlike inputs

Measured on dummy clouds before any corpus was embedded — identical 1.8e-16, rotated 7.8e-16, translated 3.9e-15, uniformly scaled ×2.5 1.4e-14, non-similar z-stretch 5.9e-1, unrelated 1.1e0. This run uses the first branch. Non-convergence rate: 0.0% across 1378 pairs in both variants — no pair was excluded, and no exception contributed to any number below.

4. Results

53 cases, 1378 pairs, 334 same-structure / 1044 cross-structure.

	RS-A (ATTR_SCALE=0)	RS-B (ATTR_SCALE=0.02)
(a) Separability	PASS — AUC 1.0000, τ=0.0935, 100% / 100%, margin +0.0867	FAIL — AUC 0.8297, 75.1% / 75.7%, margin −0.4757
(b) Attribute-invariance	PASS, but analytic — all 12 variants at exactly 0.000000 because attributes are not in the geometry	FAIL — median 0.0331 (ratio 0.107); the ×3 rescale pairs land at 0.093–0.227
(c) Structure-sensitivity	FAIL — add-vs-subtract = 0.000000	FAIL — S1-vs-S2 = 0.1764 < τ=0.2225
(d) Systematicity (reported, not gating)	chain↔chain 0.0000, chain↔flat 0.8703	chain↔chain 0.0448, chain↔flat 0.8661
Verdict	NO-GO	NO-GO

4.1 Why RS-A's decisive control fails, exactly

The two graphs in the add-vs-subtract minimal pair differ in exactly one field — one entity, identical numbers, add against subtract. Their configurations are:

subtract : (1,0,0)  (2,0,-1)  (3,0,0)
add      : (1,0,0)  (2,0,+1)  (3,0,0)

These are related by a 180° rotation about e1 — a proper rotation, det = +1. Kabsch aligns them exactly, so the residual is not merely small, it is 0.0.

The similarity quotient that would give attribute-invariance is the same quotient that annihilates the structural contrast. A geometry blind to rotation cannot distinguish "gain" from "loss" when their only encoded difference is a signed offset on one axis.

This much is repairable by constants: with asymmetric kind heights (subtract → 7.0) the three minimal pairs measure 0.162 / 1.940 / 0.217, all above τ. That repair is a diagnostic and does not change this verdict — the criterion was fixed in advance and a re-run under a repaired scheme requires its own pre-registration. It is recorded here because concealing it would make the NO-GO look deeper than it is.

4.2 What is not repairable by constants — the deeper result

Sweeping ATTR_SCALE under the same thresholds (committed as diagnostic_sweep in the report):

ATTR_SCALE	AUC	(a)	(b)	(c)	rescale median
0.0	1.0000	PASS	PASS	FAIL	0.00000
0.001	1.0000	PASS	PASS	FAIL	0.00615
0.002	1.0000	PASS	PASS	FAIL	0.01245
0.005	1.0000	PASS	PASS	FAIL	0.03259
0.01	0.9742	FAIL	PASS	FAIL	0.06682
0.02	0.8297	FAIL	FAIL	FAIL	0.13505
0.05	0.6888	FAIL	FAIL	FAIL	0.40610
0.1	0.6694	FAIL	FAIL	FAIL	1.16296

Read down the table: every regime in which the SME property survives is a regime in which attributes contribute essentially nothing. Where attributes are geometrically negligible (≤0.005, rescale residual ≤0.033), separability is perfect — but "invariance" there is a restatement of blindness. The moment attributes carry real weight, separability collapses monotonically (1.00 → 0.97 → 0.83 → 0.69) and invariance degrades with it.

There is no setting at which the geometry both sees surface attributes and factors them out. That is the SME signature §5.3b was written to detect, and it is absent — not narrowly missed.

The mechanism is structural rather than incidental: Procrustes measures a metric distance between point configurations modulo similarity. Any attribute that moves a point either moves it within the similarity orbit — in which case it was never carrying information the aligner could use — or outside it, in which case invariance fails by construction. There is no third option in this class of embedding.

5. Scope of the refutation — stated narrowly on purpose

What this refutes. H1 for the class of embeddings that encode role-structure as point positions in Cl(4,1) and align them with conformal Procrustes under similarity. The §4.2 argument is about the similarity quotient itself, so it generalises across position-encoding schemes in this class, not just the particular constants used here.

What this does not refute. That some Cl(4,1) representation could carry relational structure — e.g. one where relations are versors composed rather than points positioned, or where alignment is a non-metric admissibility check rather than a Procrustes residual. Those are different experiments and would need their own pre-registrations.

What it does not touch at all. Whether the symbolic structure-mapping already in evals/structure_mapping/ works. That lane is untouched here.

6. What follows

Under ADR-0252 §5.4 a NO-GO "cleanly refutes geometric SME for comprehension and redirects stage 2 to a different representation," and §5.4 pre-declares this full credit. Three consequences for the registers, offered for ruling and not taken:

1. G-1 can close with a recorded verdict, and the §6 build authorization question is settled in the direction of "not on this representation."
2. Track B's widening is unblocked in shape (50-execution-plan.md §7's hard gate: "Track B widening ⟸ Track A verdict"). The verdict says widening proceeds by more constructions, not by geometric structure-mapping.
3. H-10's mispricing is confirmed and discharged. The experiment cost hours, not an arc, and it governs the comprehension paradigm — exactly as H-10 said.

A fourth item is new and is the one worth your attention most: the 1.0% reader coverage of holdout_dev/v1 (§2) is a sharper measurement of the comprehension frontier than G-3's construction count, on a corpus the project already treats as its held-out standard. It belongs in the gap register on its own.

---

Every number above is produced by uv run python -m evals.structure_mapping.adr0252_s5.experiment at 797ebad5 and is reproduced by the committed report artifact and its digest. The criterion constants in experiment.py are unchanged since 299c92be, which carried them before the run and carried no results; the only post-run edit to that file is the additive diagnostic_sweep block, and git log -p shows it.

Provenance note on that SHA. The pre-registration commit was authored as dfc394d2 and re-created as 299c92be when the branch was re-signed before landing on main — same tree, same message, same ordering, an SSH signature added. The claim the verdict rests on is unchanged and independently checkable at 299c92be: that commit contains the thresholds and no results. It is recorded here rather than left as a silent SHA change, because a document citing a commit that is not in its own repository's history is the exact failure this verdict's §1 catalogues.