core/docs/research/trackb-s1-slice-2026-07-20.md

Track B, Increment 1 — Symbolic Structure-Mapping S1 vertical slice

Date: 2026-07-20
Branch: feat/trackb-symbolic-sme-s1
Paradigm: ADR-0252 (Expert Structure-Mapping on a Predictive-Processing Substrate)
§5 status: NO-GO for geometric SME (conformal_procrustes is Kabsch/Umeyama and presupposes role correspondence). PRs #83 and #84 closed as superseded.
Scope: S1 only (compare-multiplicative-then-total). Off-serving. No S2–S4. No serving cutover.

---

1. What shipped

Piece	Path	Role
Role-predicate schema	generate/structure_mapping/role_predicate.py	Typed contain / transfer / compare / rate / total
Converter	generate/structure_mapping/convert.py	Deterministic MathProblemGraph → RoleGraph
S1 canonical	generate/structure_mapping/canonicals.py	compare(b,a,k) ∧ contain(a,…) ∧ total(a,b,sum) — role vars only
Symbolic mapper	generate/structure_mapping/mapper.py	Relational match → binding or refuse; no gold labels
Solve corridor	generate/structure_mapping/solve_s1.py	Binding → graph → classical solve+verify and multi-register certificate; emit only if both agree
Scoring labels (isolated)	evals/structure_mapping/scoring/	Gold S1 ids for scoring only; forbidden from mapper imports
Measure script	scripts/measure_trackb_s1_slice.py	Blind holdout / FP / surface-variant / vs-organ
Unit tests	tests/test_trackb_s1_structure_mapping.py	16 tests driving shipped code

Serving reader (parse_and_solve dispatch, resolve_promotable_*, chat/runtime.py) untouched (diff bytes vs main = 0).

---

2. Discipline checks

Blindness

• Mapper signature: map_to_s1(role_graph: RoleGraph) — no label parameter.
• Static audit: generate/structure_mapping/ does not import evals.structure_mapping.scoring or load holdout_dev_v1_labels.jsonl.
• Labels enter only after map decisions, in the measure script / scorer.

No crash-as-signal

• Solve path catches MultiRegisterError / classical failures and returns refuse outcomes; exceptions are never treated as residual scores.

wrong=0

• Emit only when classical verifier passes and multi-register chain is certified and answers agree within relative 1e-6.

---

3. Measurements (command + real output)

3a. Parse→role coverage on S1 holdout cohort

$ uv run python scripts/measure_trackb_s1_slice.py --mode coverage
=== parse→role coverage on S1 holdout ===
gsm8k-holdout-dev-v1-0101 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0108 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0268 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0411 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0453 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None

Coverage: 5/5 S1 organ cases parse → role graph with {compare, contain, total} and map to S1.

3b. Right-for-right-reason (S1 holdout)

$ uv run python scripts/measure_trackb_s1_slice.py --mode right-reason
=== S1 right-for-right-reason (holdout_dev/v1 S1 cohort) ===
CASE gsm8k-holdout-dev-v1-0101 emit=9.0 gold=9.0 match=True wrong_flag=False derivation='S1: Frankie = 2.0 × Eduardo; Eduardo = 3.0; total = Eduardo + Frankie = 3.0 + 2.0*3.0 = 9.0' mr_cert=True classical_v=True binding={'a': 'Eduardo', 'b': 'Frankie', 'k': 2.0, 'a_value': 3.0, 'unit': 'lasts', 'canonical_id': 'S1'} ...
CASE gsm8k-holdout-dev-v1-0108 emit=110.0 gold=110.0 match=True ... derivation='S1: Dana Point Beach = 4.0 × Newport Beach; ...'
CASE gsm8k-holdout-dev-v1-0268 emit=16.0 gold=16.0 match=True ... derivation='S1: Bruce Scored 4 Goals While Michael = 3.0 × Bruce; ...'
CASE gsm8k-holdout-dev-v1-0411 emit=52.0 gold=52.0 match=True ... derivation='S1: male students = 3.0 × female students; ...'
CASE gsm8k-holdout-dev-v1-0453 emit=28.0 gold=28.0 match=True ... derivation='S1: Olaf = 3.0 × dad; dad = 7.0; total = dad + Olaf = 7.0 + 3.0*7.0 = 28.0' ...
SUMMARY right-reason: correct=5 wrong=0 refused=0 n=5

Result: correct=5 wrong=0 refused=0. Every emit carries a derivation of the form b = k×a; total = a+b, with multi-register certificate + classical verify both true.

Note on residuals: early runs emitted multi-register decoded floats (~8.999999982). That is geometric relaxation noise, not a zero residual trophy. The ship path requires MR certification then emits the classically verified scalar once corridors agree (relative 1e-6). Both corridors remain mandatory.

3c. False-positive / separability

$ uv run python scripts/measure_trackb_s1_slice.py --mode false-positive
=== A: holdout_dev/v1 non-S1 cases (parse→map) ===
SUMMARY holdout-non-S1: fp=0 tn=495 n_non_s1=495 parse_fail_among_non_s1=495 fp_rate=0.000000
NOTE: holdout_dev/v1 currently yields selected_graph on only the 5 S1 organ cases; non-S1 holdout FP is vacuous at the graph layer when parse_fail=495. Layer B/C are the real separability tests.
=== B: synthetic non-S1 graphs ===
  transfer: mapped_s1=False refuse=non_s1_has_transfer
  contain_total_no_compare: mapped_s1=False refuse=compare_count_not_one:0
  compare_no_total: mapped_s1=False refuse=missing_total_query
  compare_with_b_seed: mapped_s1=False refuse=b_has_independent_contain_not_pure_s1
  apply_rate: mapped_s1=False refuse=non_s1_has_rate
SUMMARY synthetic: fp=0 tn=5 n=5 fp_rate=0.000000
=== C: other gsm8k_math corpora non-compare graphs ===
SUMMARY other-corpus non-S1 graphs: fp=0 tn=135 n=135 fp_rate=0.000000

Honest reading:

• Layer A (holdout non-S1): vacuous at graph level — the serving reader produces selected_graph only for the five S1 cases on this split. Reporting fp_rate=0 here without the NOTE would be the same class of cheat as prior geometric “GO”s. We refuse that claim.
• Layer B (synthetic): 0/5 false positives on transfer / additive-total / compare-without-total / dual-seed compare / rate.
• Layer C (other corpora, 135 non-compare graphs): 0/135 false positives (mostly transfers from public/).

3d. Surface-variant generalization (re-parse)

$ uv run python scripts/measure_trackb_s1_slice.py --mode surface-variant
BASE_ID gsm8k-holdout-dev-v1-0108
BASE_TEXT Dana Point beach has four times the number of sharks as Newport Beach. If Newport Beach has 22 sharks, how many sharks are there in total on the two beaches?
VARIANT_TEXT Cedar Cove beach has five times the number of dolphins as Harbor Beach. If Harbor Beach has 11 dolphins, how many dolphins are there in total on the two beaches?
BASE_MAP mapped_s1=True binding k=4.0 a_value=22.0 ...
VARIANT_MAP mapped_s1=True binding={'a': 'Harbor Beach', 'b': 'Cedar Cove Beach', 'k': 5.0, 'a_value': 11.0, 'unit': 'dolphins', ...}
VARIANT_SOLVE emitted=True answer=66.0 expected=66.0 derivation='S1: Cedar Cove Beach = 5.0 × Harbor Beach; Harbor Beach = 11.0; total = ... = 66.0'
SUMMARY surface-variant: map=True k_ok=True a_ok=True names_ok=True ans_ok=True

Re-parsed renamed/re-numbered text still maps to S1 with the new bindings and solves correctly. Driven by relations, not surface tokens.

3e. Side-by-side vs S1 surface organ

$ uv run python scripts/measure_trackb_s1_slice.py --mode vs-organ
case_id                               organ     trackb       gold organ_ok    tb_ok    map
gsm8k-holdout-dev-v1-0101               9.0        9.0        9.0     True     True   True
gsm8k-holdout-dev-v1-0108             110.0      110.0      110.0     True     True   True
gsm8k-holdout-dev-v1-0268              16.0       16.0       16.0     True     True   True
gsm8k-holdout-dev-v1-0411              52.0       52.0       52.0     True     True   True
gsm8k-holdout-dev-v1-0453              28.0       28.0       28.0     True     True   True

Track B matches the organ on all five S1 cases (answer + map). Organ path remains the serving parse_and_solve; Track B is a parallel off-serving structure-map → certificate path.

---

4. Unit tests

$ uv run pytest tests/test_trackb_s1_structure_mapping.py -v
16 passed in 0.62s

Includes real holdout texts, refuse paths, blindness import audit, and dual-corridor emit.

---

5. Baselines (serving invariant)

Pre-edit (worktree at main tip 1ccef491, before Track B code):

holdout report (committed): {'correct': 5, 'refused': 495, 'wrong': 0} n=500
uv run core test --suite smoke -q → 176 passed in 134.05s

Post-edit: same smoke suite (off-serving code only; serving modules diff empty). Holdout serving numbers unchanged by construction (no live path edits).

---

6. Pure-S1 integrity fix (reviewer-critical)

An early mapper draft accepted totals that included {a,b} among other parts and only blocked an independent contain on b. Combined with rebuild-from-binding solve, that could emit a certified wrong answer (drop third entity):

# A=5, C=100, B=2×A, total all → classical 115; buggy pure rebuild → 15
$ uv run python -c '...'  # see measure-s1-pure-gate-repro
map=StructureMapRefuse reason=total_parts_not_exactly_a_b
emit=False ans=None refuse=structure_map_refuse:total_parts_not_exactly_a_b mr=False
REPRO FIXED

Fix (shipped):

1. map_to_s1 requires total part-set exactly {a,b} (total_parts_not_exactly_a_b).
2. Any contain on an entity other than a is refused (extra_contain_not_pure_s1:* / b_has_independent_contain_not_pure_s1).
3. try_s1_structure_map_and_solve(graph=…) additionally requires classical solve of the original graph to agree with the pure-S1 rebuild before emit (fail-closed backstop).

Unit tests: test_mapper_refuses_total_superset_with_extra_entity, test_mapper_refuses_extra_contain_even_if_total_were_ab_only, test_multi_entity_slice_never_emits_wrong_certified_answer, test_multi_entity_small_c_same_trap (20 tests total).

7. Explicit non-claims / STOP gate

• Not claimed: geometric SME revival; S2–S4 coverage; serving cutover; sealed-test movement.
• Not claimed: holdout non-S1 graph-level separability beyond the vacuous parse layer — only synthetic + other-corpus graphs.
• Not claimed: case 0148 (classic “twice the total…”) — reader currently refuses parse; out of converter reach until parse coverage grows.
• STOP: human reviewer verifies this slice against the code before any generalization.

---

8. Inventory: proven vs assumed

Item	Status
S1 map+solve on organ cohort (5)	Proven (commands above)
wrong=0 on emitted path	Proven
Surface-variant re-parse	Proven
FP on synthetic + 135 non-compare graphs	Proven
Holdout non-S1 graph FP	Unclaimed (vacuous parse layer)
Generalization to S2–S4	Not in scope
Serving supersede of organ	Not in scope