Track A of docs/assessment/50-execution-plan.md §6, executed under its own
protocol. Criterion pre-registered at dfc394d2 (that commit carries the
thresholds as importable constants and NO results); this commit carries the run.
VERDICT: NO-GO. Full credit by ADR-0252 §5.4's own terms.
docs/research/sme-experiment-verdict-797ebad5.md
evals/structure_mapping/adr0252_s5/results/report-797ebad5.json
deterministic_digest b3d9d27592e213104c51bcace7415fd819d722a1b06f7d0a5ca65bd5a234e4ec
53 cases, 1378 pairs, 0.0% non-convergence — no exception contributed to any
number. RS-A (structure-only) separates perfectly, AUC 1.0000, margin +0.0867,
and fails structure-sensitivity. RS-B (attribute-bearing) fails all three.
The mechanism, measured rather than argued: the similarity quotient that would
deliver attribute-invariance is the same quotient that annihilates structural
contrast. An add-vs-subtract minimal pair — one entity, identical numbers, one
relation kind changed — aligns at residual exactly 0.0, its two configurations
being related by a PROPER rotation about e1. Sweeping the attribute weight
(committed as diagnostic_sweep), structure-sensitivity fails at every setting,
and every regime where the SME property survives is a regime where attributes
contribute nothing: AUC 1.00 -> 0.97 -> 0.83 -> 0.69 as they begin to matter.
Scope stated narrowly: this refutes H1 for embeddings that encode role-structure
as point positions aligned by conformal Procrustes under similarity. The
argument is about the quotient, so it generalises across that class; it does not
refute every Cl(4,1) representation, and it does not touch the symbolic
structure-mapping lane already in evals/structure_mapping/.
Two findings that were on nobody's list:
N-8 — the experiment was NOT unrun. rnd/sme-experiment-v2 @ 96e5f468 is titled
"Verdict: GO" and rnd/structure-mapping-experiment @ fc9d0c14 carries an earlier
one. Neither survives inspection: attempt 1 leaked the S1-S4 label into the
embedding; attempt 2 was blind and then took its separability from
`except ValueError: res = 1000.0` — a solver exception counted as a distance —
on a corpus 46/51 of which is outside holdout_dev/v1 with nothing marked, with
duplicate graphs, colliding ids, and an extractor that was never committed. The
register, the plan and the ADR all read the same absence and inherited the same
error for nine days. A branch tip is not a record.
G-21 — the math reader returns a selected graph for 5 of 500 holdout_dev/v1
cases (1.0%), all one skeleton; 24/150 on the public lane. That is why §5.1's
four-structure corpus is not extractable, and it is a sharper measurement of the
comprehension frontier than G-3's construction count.
Registers updated: G-1 carries the verdict, H-10 confirmed and discharged, G-21
added, plan §0 gains N-8, Track A closed, synthesis frontier 2 rewritten.
Pins: tests/test_adr_0252_s5_blindness.py, each observed red before green
(label leak and duplicate-graph sabotage both caught).
Off-serving — evals/structure_mapping/adr0252_s5/ is imported by no serving
path, emits no answers, and changes no flag.
[Verification]: 797ebad5 + this branch — `uv run core test --suite smoke -q`
641 passed in 199.76s; `uv run ruff check` clean; report regenerated after lint
with an unchanged digest.
Track A, per 50-execution-plan.md §6: "state the GO/NO-GO criterion in writing
BEFORE running (the ADR's §5 conformance bar, not a criterion chosen after
seeing results)." This commit carries the criterion and the code and NO results.
Why the protocol is load-bearing here rather than ceremonial: the §5 experiment
is not unrun. It has been run twice and returned GO twice, on unmerged branches,
and neither run's rule was binding.
- rnd/structure-mapping-experiment @ fc9d0c14 — disowned by its own successor:
the S1-S4 label entered the embedding, so Procrustes ran on identical
matrices per structure.
- rnd/sme-experiment-v2 @ 96e5f468 — blind, and then measured separability
with `except ValueError: res = 1000.0`. Its GO test cross_mean > 2*same_mean
is satisfied by that sentinel alone. A solver exception is not a distance.
Instrument characterised before use, on dummy clouds: conformal_procrustes has
two branches. The (5,K) Kabsch+Umeyama path never raises and is invariant to
rotation, translation AND uniform scale (identical 1.8e-16, rotated 7.8e-16,
translated 3.9e-15, scaled x2.5 1.4e-14) while separating non-similar
deformation (5.9e-1) and unrelated clouds (1.1e0). The mixed versor/point
sequence path performs field conjugacy and raises. Attempt 2 used the second;
this uses the first, and treats a raise as non-convergence — excluded, reported
as a rate, never counted as separation.
Corpus fact that forced the design, measured at 797ebad5: the serving reader
returns a selected graph for 5 of the 500 holdout_dev/v1 cases (1.0%), all with
the same skeleton. §5.1's four-structure corpus is not extractable from
holdout_dev/v1 today, so §5.1's own escape hatch is used and MARKED — 5 real
cases, 48 synthetic from four declared templates. A verdict from this corpus
speaks about the geometry, not about coverage; §5.5's compression estimate is
not answerable at 1.0% coverage and is not attempted.
Off-serving. Imported by no serving path; no answers emitted.
Increment 2 for ADR-0252 Track B. Extends Increment 1's structure-mapping
slice with pure-family canonicals S2–S4, overlapping-waves selector, and a
structure-mapping-owned pure-S1 text extract that recovers four real
holdout cases the serving reader misses (0148, 0228, 0234, 0441).
Bar results (command-backed via scripts/measure_trackb_inc2.py):
- generalization ratio S1 = 9.0 (9 holdout cases / 1 template)
- coverage gain organ 5 → trackb 9, wrong=0
- selector routes S1/S2/S3; refuses empty; surface≠structure
Off-serving: no organ retirement, serving reader untouched. S2–S4 holdout
ratios are 0 (parser frontier). S3/S4 emit refuses at multi-register scope
without weakening the three-gate wrong=0 path.
[Verification]: Smoke suite passed locally (~132s, 176 passed);
trackb unit tests 35 passed; measure_trackb_inc2 all modes green.
ADR-0252 §5 geometric SME is NO-GO; this is Track B Increment 1.
Adds role-predicate conversion from MathProblemGraph, S1 canonical skeleton,
blind symbolic mapper (match/refuse + binding), and solve via classical
verify plus multi-register certificate. Research report and holdout measures
included. Serving reader unchanged; no S2–S4 generalization.
