core/docs/research/trackb-inc2-generalization-2026-07-20.md

# Track B, Increment 2 — Generalization bar (S1–S4, selector, coverage)

**Date**: 2026-07-20  
**Branch**: `feat/trackb-symbolic-sme-s2s4`  
**Paradigm**: ADR-0252  
**Prerequisite**: Increment 1 (PR #85) merged; §5 geometric SME remains NO-GO  
**Scope**: Off-serving. S1–S4 canonicals + selector + SM-owned pure-S1 extract. No serving cutover. No organ retirement in this PR.

---

## 1. The bar (ADR-0252 §4 / Increment 2)

| # | Requirement | Result |
| --- | --- | --- |
| 1 | Generalization ratio > 1 on ≥1 canonical (real holdout IDs) | **PASS** — S1 ratio = **9.0** (9 cases / 1 template) |
| 2 | Coverage gain **or** organ retirement | **PASS** — coverage gain: **+4** holdout cases (organ 5 → trackb 9), `wrong=0` |
| 3 | Selector works (S1–S4 held; ties refuse; surface≠structure) | **PASS** — routing table below |

Honest negatives (not failures of the bar):
- S2/S3/S4 holdout ratios = 0 — **parser frontier**: organ yields graphs on only 5/500 holdout cases (all S1).
- S3/S4 multi-register corridor cannot certify pure rebuilds yet (`apply_rate` / `compare_additive` out of scope) — map succeeds, emit refuses (gate held, not weakened).
- Organ retirement **not** performed (off-serving path; coverage-gain chosen).

---

## 2. What shipped

| Piece | Path |
| --- | --- |
| Canonicals S1–S4 | `generate/structure_mapping/canonicals.py` |
| Role schema + `compare_add` | `generate/structure_mapping/role_predicate.py` |
| Converter (incl. additive compare) | `generate/structure_mapping/convert.py` |
| Pure-family mappers S1–S4 | `generate/structure_mapping/mapper.py` |
| Overlapping-waves selector | `generate/structure_mapping/selector.py` |
| Generalized 3-gate solve | `generate/structure_mapping/solve.py` |
| SM-owned pure-S1 text extract | `generate/structure_mapping/text_extract.py` |
| Pipeline (organ parse → extract fallback) | `generate/structure_mapping/pipeline.py` |
| Measure script | `scripts/measure_trackb_inc2.py` |
| Tests | `tests/test_trackb_inc2_structure_mapping.py` (15) + Inc1 suite (20) |

Serving reader (`parse_and_solve` dispatch, organs, `chat/runtime.py`) **untouched**.

---

## 3. Measurements (command + real output)

### 3a. Generalization-ratio table

```text
$ uv run python scripts/measure_trackb_inc2.py --mode ratio
=== generalization ratio (holdout_dev/v1, command-backed) ===
CASE gsm8k-holdout-dev-v1-0101 sid=S1 emit=9.0 gold=9.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0108 sid=S1 emit=110.0 gold=110.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0148 sid=S1 emit=1500.0 gold=1500.0 right=True src=sm_extract pat=s1_p3_day_pair ...
CASE gsm8k-holdout-dev-v1-0228 sid=S1 emit=600.0 gold=600.0 right=True src=sm_extract pat=s1_p2_b_seeded ...
CASE gsm8k-holdout-dev-v1-0234 sid=S1 emit=240.0 gold=240.0 right=True src=sm_extract pat=s1_p2_deaf_blind ...
CASE gsm8k-holdout-dev-v1-0268 sid=S1 emit=16.0 gold=16.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0411 sid=S1 emit=52.0 gold=52.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0441 sid=S1 emit=75.0 gold=75.0 right=True src=sm_extract pat=s1_p2_b_seeded ...
CASE gsm8k-holdout-dev-v1-0453 sid=S1 emit=28.0 gold=28.0 right=True src=organ ...
--- ratio table ---
canonical=S1 templates=1 cases_carried=9 ratio=9.000 ids=[...9 holdout ids...]
canonical=S2 templates=1 cases_carried=0 ratio=0.000 ids=[]
canonical=S3 templates=1 cases_carried=0 ratio=0.000 ids=[]
canonical=S4 templates=1 cases_carried=0 ratio=0.000 ids=[]
SUMMARY active_canonicals=1 total_cases=9 aggregate_ratio=9.000
```

All nine S1 carries include the 3-part proof (right answer, derivation `b=k×a; total=a+b`, multi-register + classical cert). Surface-variant re-extract of the inverted-seed form still maps (Ava/Ben → 48).

### 3b. Coverage gain

```text
$ uv run python scripts/measure_trackb_inc2.py --mode coverage-gain
organ_correct=5 trackb_correct=9 newly_solved=4
  NEW gsm8k-holdout-dev-v1-0148 ans=1500.0 gold=1500.0 sid=S1 extract=s1_p3_day_pair
  NEW gsm8k-holdout-dev-v1-0228 ans=600.0 gold=600.0 sid=S1 extract=s1_p2_b_seeded
  NEW gsm8k-holdout-dev-v1-0234 ans=240.0 gold=240.0 sid=S1 extract=s1_p2_deaf_blind
  NEW gsm8k-holdout-dev-v1-0441 ans=75.0 gold=75.0 sid=S1 extract=s1_p2_b_seeded
```

These four are **real surface-distinct holdout cases**, not synthetic clones. The SM-owned extract layer is structure-mapping-owned (regex pure-family templates) and runs only when the serving reader refuses — it does not modify the live reader.

### 3c. Selector

```text
$ uv run python scripts/measure_trackb_inc2.py --mode selector
CASE real_s1_0411 expect=S1 got=S1 refused=False ok=True
CASE real_s2_public_transfer expect=S2 got=S2 refused=False ok=True
CASE struct_s2_not_s1 expect=S2 got=S2 refused=False ok=True
CASE struct_s1 expect=S1 got=S1 refused=False ok=True
CASE empty_ops_refuse expect=None got=None refused=True reason=no_family_matched ok=True
CASE struct_s3_rate expect=S3 got=S3 refused=False ok=True
```

Transfer graphs route to S2 even when the surface text could mention multiplicative language elsewhere; empty ops refuse. Ranking uses fixed systematicity scores, not entity names or numbers.

### 3d. Parser frontier

```text
organ_selected_graph_count=5/500
role_kind_multiset_counts={'compare,contain,total': 5}
--- case 0148 ---
organ_parsed=False
sm_pipeline emit=True answer=1500.0 gold=1500.0 src=sm_extract sid=S1
```

**0148** (the original S1 family namesake) is a **parser-frontier refusal** on the serving reader; the SM-owned extract recovers it. Holdout still has zero organ graphs for transfer/rate/additive-compare families — that is the next structural bottleneck, not the mapper.

### 3e. S2 on real public transfers (not holdout; engineering evidence only)

```text
SUMMARY s2_public_sample: ok=5 scanned_transferish=20
  S2 gma-102 ans=54.0 ... transfer 7 Eve→David ...
```

S2 pure-family map+solve works on real public transfer graphs. Not counted in the holdout ratio table (discipline: holdout IDs only).

### 3f. Right-reason summary

```text
SUMMARY right-reason: correct=9 wrong=0 refused=0
SUMMARY surface-variant: ok=True
```

### 3g. Full holdout wrong=0 gate (all 500 — not curated subset)

Regression: a loose inverted-seed regex matched multi-entity zoo case **0361**
(`John has 8 more pandas…`) as pure S1 and the corridor emitted `1.125` vs
gold `114.0`. Fixed by pure-family surface gates (require explicit multiplicative
language; refuse bare `N more` / multi-clause / multi-numeric surfaces).

```text
$ uv run python scripts/measure_trackb_inc2.py --mode full-holdout-wrong0
=== full holdout_dev/v1 wrong=0 gate (all 500 cases) ===
SUMMARY full-holdout: emit_ok=9 wrong=0 refused=491 n=500
PASS: wrong=0 absolute on full holdout_dev/v1
```

---

## 4. Discipline checks

- **Blind**: mappers/selector take only `RoleGraph`; no label parameter. Static audit: `generate/structure_mapping/` does not import scoring labels.
- **wrong=0**: emit only if classical verify **and** multi-register cert **and** agreement; original-graph backstop when a graph is supplied.
- **No crash-as-signal**: corridor exceptions → refuse outcomes.
- **Off-serving**: no organ retirement; no `parse_and_solve` dispatch edits.

---

## 5. Deviations / frontiers (honest)

1. **S2–S4 holdout ratio = 0** until the serving reader (or a reviewed SM extract family) produces pure transfer/rate/additive graphs on holdout. Do not manufacture synthetic holdout clones for ratio.
2. **S3/S4 solve emit** blocked by multi-register scope (`apply_rate`, `compare_additive` not Tier-2a). Gates not weakened — refuse.
3. **SM extract is narrow** (regex pure-S1 templates only, with fail-closed purity gates). It is not a general parser; expanding it is a deliberate future increment. Overmatch risk is gated by `mode full-holdout-wrong0` on all 500 cases.
4. **Organ retirement** deferred; coverage-gain path met bar 2 without serving risk.
5. **0361 pure-S1 overmatch** (fixed in follow-up commit): optional `times` in P2 regex treated additive `8 more` as multiplicative; purity gate + required `times as many` closed it. Full-holdout scan is now mandatory before claiming wrong=0.

---

## 6. STOP

One PR (base `main`). Reviewer verifies:
- blindness + pure-family gates,
- ratio > 1 cases are real holdout IDs (not clones),
- `wrong=0` three-gate path,
- serving reader unchanged.

No S2–S4 serving cutover until a later ruling. No sealed-test run.