# Track B, Increment 1 — Symbolic Structure-Mapping S1 vertical slice

**Date**: 2026-07-20  
**Branch**: `feat/trackb-symbolic-sme-s1`  
**Paradigm**: ADR-0252 (Expert Structure-Mapping on a Predictive-Processing Substrate)  
**§5 status**: NO-GO for geometric SME (`conformal_procrustes` is Kabsch/Umeyama and presupposes role correspondence). PRs #83 and #84 closed as superseded.  
**Scope**: S1 only (compare-multiplicative-then-total). Off-serving. No S2–S4. No serving cutover.

---

## 1. What shipped

| Piece | Path | Role |
| --- | --- | --- |
| Role-predicate schema | `generate/structure_mapping/role_predicate.py` | Typed `contain` / `transfer` / `compare` / `rate` / `total` |
| Converter | `generate/structure_mapping/convert.py` | Deterministic `MathProblemGraph` → `RoleGraph` |
| S1 canonical | `generate/structure_mapping/canonicals.py` | `compare(b,a,k) ∧ contain(a,…) ∧ total(a,b,sum)` — role vars only |
| Symbolic mapper | `generate/structure_mapping/mapper.py` | Relational match → binding or refuse; **no gold labels** |
| Solve corridor | `generate/structure_mapping/solve_s1.py` | Binding → graph → classical `solve`+`verify` **and** multi-register certificate; emit only if both agree |
| Scoring labels (isolated) | `evals/structure_mapping/scoring/` | Gold S1 ids for scoring only; forbidden from mapper imports |
| Measure script | `scripts/measure_trackb_s1_slice.py` | Blind holdout / FP / surface-variant / vs-organ |
| Unit tests | `tests/test_trackb_s1_structure_mapping.py` | 16 tests driving shipped code |

Serving reader (`parse_and_solve` dispatch, `resolve_promotable_*`, `chat/runtime.py`) **untouched** (diff bytes vs main = 0).

---

## 2. Discipline checks

### Blindness
- Mapper signature: `map_to_s1(role_graph: RoleGraph)` — no label parameter.
- Static audit: `generate/structure_mapping/` does not import `evals.structure_mapping.scoring` or load `holdout_dev_v1_labels.jsonl`.
- Labels enter only after map decisions, in the measure script / scorer.

### No crash-as-signal
- Solve path catches `MultiRegisterError` / classical failures and returns refuse outcomes; exceptions are never treated as residual scores.

### wrong=0
- Emit only when classical verifier passes **and** multi-register chain is certified **and** answers agree within relative 1e-6.

---

## 3. Measurements (command + real output)

### 3a. Parse→role coverage on S1 holdout cohort

```text
$ uv run python scripts/measure_trackb_s1_slice.py --mode coverage
=== parse→role coverage on S1 holdout ===
gsm8k-holdout-dev-v1-0101 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0108 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0268 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0411 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
gsm8k-holdout-dev-v1-0453 parsed=True kinds=['compare', 'contain', 'total'] mapped_s1=True refuse=None
```

Coverage: **5/5** S1 organ cases parse → role graph with `{compare, contain, total}` and map to S1.

### 3b. Right-for-right-reason (S1 holdout)

```text
$ uv run python scripts/measure_trackb_s1_slice.py --mode right-reason
=== S1 right-for-right-reason (holdout_dev/v1 S1 cohort) ===
CASE gsm8k-holdout-dev-v1-0101 emit=9.0 gold=9.0 match=True wrong_flag=False derivation='S1: Frankie = 2.0 × Eduardo; Eduardo = 3.0; total = Eduardo + Frankie = 3.0 + 2.0*3.0 = 9.0' mr_cert=True classical_v=True binding={'a': 'Eduardo', 'b': 'Frankie', 'k': 2.0, 'a_value': 3.0, 'unit': 'lasts', 'canonical_id': 'S1'} ...
CASE gsm8k-holdout-dev-v1-0108 emit=110.0 gold=110.0 match=True ... derivation='S1: Dana Point Beach = 4.0 × Newport Beach; ...'
CASE gsm8k-holdout-dev-v1-0268 emit=16.0 gold=16.0 match=True ... derivation='S1: Bruce Scored 4 Goals While Michael = 3.0 × Bruce; ...'
CASE gsm8k-holdout-dev-v1-0411 emit=52.0 gold=52.0 match=True ... derivation='S1: male students = 3.0 × female students; ...'
CASE gsm8k-holdout-dev-v1-0453 emit=28.0 gold=28.0 match=True ... derivation='S1: Olaf = 3.0 × dad; dad = 7.0; total = dad + Olaf = 7.0 + 3.0*7.0 = 28.0' ...
SUMMARY right-reason: correct=5 wrong=0 refused=0 n=5
```

**Result**: `correct=5 wrong=0 refused=0`. Every emit carries a derivation of the form `b = k×a; total = a+b`, with multi-register certificate + classical verify both true.

> Note on residuals: early runs emitted multi-register decoded floats (~`8.999999982`). That is geometric relaxation noise, **not** a zero residual trophy. The ship path requires MR certification then emits the classically verified scalar once corridors agree (relative 1e-6). Both corridors remain mandatory.

### 3c. False-positive / separability

```text
$ uv run python scripts/measure_trackb_s1_slice.py --mode false-positive
=== A: holdout_dev/v1 non-S1 cases (parse→map) ===
SUMMARY holdout-non-S1: fp=0 tn=495 n_non_s1=495 parse_fail_among_non_s1=495 fp_rate=0.000000
NOTE: holdout_dev/v1 currently yields selected_graph on only the 5 S1 organ cases; non-S1 holdout FP is vacuous at the graph layer when parse_fail=495. Layer B/C are the real separability tests.
=== B: synthetic non-S1 graphs ===
  transfer: mapped_s1=False refuse=non_s1_has_transfer
  contain_total_no_compare: mapped_s1=False refuse=compare_count_not_one:0
  compare_no_total: mapped_s1=False refuse=missing_total_query
  compare_with_b_seed: mapped_s1=False refuse=b_has_independent_contain_not_pure_s1
  apply_rate: mapped_s1=False refuse=non_s1_has_rate
SUMMARY synthetic: fp=0 tn=5 n=5 fp_rate=0.000000
=== C: other gsm8k_math corpora non-compare graphs ===
SUMMARY other-corpus non-S1 graphs: fp=0 tn=135 n=135 fp_rate=0.000000
```

**Honest reading**:
- Layer A (holdout non-S1): **vacuous at graph level** — the serving reader produces `selected_graph` only for the five S1 cases on this split. Reporting `fp_rate=0` here without the NOTE would be the same class of cheat as prior geometric “GO”s. We refuse that claim.
- Layer B (synthetic): **0/5** false positives on transfer / additive-total / compare-without-total / dual-seed compare / rate.
- Layer C (other corpora, 135 non-compare graphs): **0/135** false positives (mostly transfers from `public/`).

### 3d. Surface-variant generalization (re-parse)

```text
$ uv run python scripts/measure_trackb_s1_slice.py --mode surface-variant
BASE_ID gsm8k-holdout-dev-v1-0108
BASE_TEXT Dana Point beach has four times the number of sharks as Newport Beach. If Newport Beach has 22 sharks, how many sharks are there in total on the two beaches?
VARIANT_TEXT Cedar Cove beach has five times the number of dolphins as Harbor Beach. If Harbor Beach has 11 dolphins, how many dolphins are there in total on the two beaches?
BASE_MAP mapped_s1=True binding k=4.0 a_value=22.0 ...
VARIANT_MAP mapped_s1=True binding={'a': 'Harbor Beach', 'b': 'Cedar Cove Beach', 'k': 5.0, 'a_value': 11.0, 'unit': 'dolphins', ...}
VARIANT_SOLVE emitted=True answer=66.0 expected=66.0 derivation='S1: Cedar Cove Beach = 5.0 × Harbor Beach; Harbor Beach = 11.0; total = ... = 66.0'
SUMMARY surface-variant: map=True k_ok=True a_ok=True names_ok=True ans_ok=True
```

Re-parsed renamed/re-numbered text still maps to S1 with the **new** bindings and solves correctly. Driven by relations, not surface tokens.

### 3e. Side-by-side vs S1 surface organ

```text
$ uv run python scripts/measure_trackb_s1_slice.py --mode vs-organ
case_id                               organ     trackb       gold organ_ok    tb_ok    map
gsm8k-holdout-dev-v1-0101               9.0        9.0        9.0     True     True   True
gsm8k-holdout-dev-v1-0108             110.0      110.0      110.0     True     True   True
gsm8k-holdout-dev-v1-0268              16.0       16.0       16.0     True     True   True
gsm8k-holdout-dev-v1-0411              52.0       52.0       52.0     True     True   True
gsm8k-holdout-dev-v1-0453              28.0       28.0       28.0     True     True   True
```

Track B matches the organ on all five S1 cases (answer + map). Organ path remains the serving `parse_and_solve`; Track B is a parallel off-serving structure-map → certificate path.

---

## 4. Unit tests

```text
$ uv run pytest tests/test_trackb_s1_structure_mapping.py -v
16 passed in 0.62s
```

Includes real holdout texts, refuse paths, blindness import audit, and dual-corridor emit.

---

## 5. Baselines (serving invariant)

Pre-edit (worktree at main tip `1ccef491`, before Track B code):

```text
holdout report (committed): {'correct': 5, 'refused': 495, 'wrong': 0} n=500
uv run core test --suite smoke -q → 176 passed in 134.05s
```

Post-edit: same smoke suite (off-serving code only; serving modules diff empty). Holdout serving numbers unchanged by construction (no live path edits).

---

## 6. Pure-S1 integrity fix (reviewer-critical)

An early mapper draft accepted totals that *included* `{a,b}` among other parts
and only blocked an independent contain on `b`. Combined with rebuild-from-binding
solve, that could emit a certified **wrong** answer (drop third entity):

```text
# A=5, C=100, B=2×A, total all → classical 115; buggy pure rebuild → 15
$ uv run python -c '...'  # see measure-s1-pure-gate-repro
map=StructureMapRefuse reason=total_parts_not_exactly_a_b
emit=False ans=None refuse=structure_map_refuse:total_parts_not_exactly_a_b mr=False
REPRO FIXED
```

**Fix (shipped):**
1. `map_to_s1` requires total part-set **exactly** `{a,b}` (`total_parts_not_exactly_a_b`).
2. Any `contain` on an entity other than `a` is refused (`extra_contain_not_pure_s1:*` / `b_has_independent_contain_not_pure_s1`).
3. `try_s1_structure_map_and_solve(graph=…)` additionally requires classical solve of the **original** graph to agree with the pure-S1 rebuild before emit (fail-closed backstop).

Unit tests: `test_mapper_refuses_total_superset_with_extra_entity`,
`test_mapper_refuses_extra_contain_even_if_total_were_ab_only`,
`test_multi_entity_slice_never_emits_wrong_certified_answer`,
`test_multi_entity_small_c_same_trap` (20 tests total).

## 7. Explicit non-claims / STOP gate

- **Not claimed**: geometric SME revival; S2–S4 coverage; serving cutover; sealed-test movement.
- **Not claimed**: holdout non-S1 graph-level separability beyond the vacuous parse layer — only synthetic + other-corpus graphs.
- **Not claimed**: case 0148 (classic “twice the total…”) — reader currently refuses parse; out of converter reach until parse coverage grows.
- **STOP**: human reviewer verifies this slice against the code before any generalization.

---

## 8. Inventory: proven vs assumed

| Item | Status |
| --- | --- |
| S1 map+solve on organ cohort (5) | **Proven** (commands above) |
| wrong=0 on emitted path | **Proven** |
| Surface-variant re-parse | **Proven** |
| FP on synthetic + 135 non-compare graphs | **Proven** |
| Holdout non-S1 graph FP | **Unclaimed** (vacuous parse layer) |
| Generalization to S2–S4 | **Not in scope** |
| Serving supersede of organ | **Not in scope** |
