core/docs/research/trackb-inc2-generalization-2026-07-20.md

Track B, Increment 2 — Generalization bar (S1–S4, selector, coverage)

Date: 2026-07-20
Branch: feat/trackb-symbolic-sme-s2s4
Paradigm: ADR-0252
Prerequisite: Increment 1 (PR #85) merged; §5 geometric SME remains NO-GO
Scope: Off-serving. S1–S4 canonicals + selector + SM-owned pure-S1 extract. No serving cutover. No organ retirement in this PR.

---

1. The bar (ADR-0252 §4 / Increment 2)

#	Requirement	Result
1	Generalization ratio > 1 on ≥1 canonical (real holdout IDs)	PASS — S1 ratio = 9.0 (9 cases / 1 template)
2	Coverage gain or organ retirement	PASS — coverage gain: +4 holdout cases (organ 5 → trackb 9), wrong=0
3	Selector works (S1–S4 held; ties refuse; surface≠structure)	PASS — routing table below

Honest negatives (not failures of the bar):

• S2/S3/S4 holdout ratios = 0 — parser frontier: organ yields graphs on only 5/500 holdout cases (all S1).
• S3/S4 multi-register corridor cannot certify pure rebuilds yet (apply_rate / compare_additive out of scope) — map succeeds, emit refuses (gate held, not weakened).
• Organ retirement not performed (off-serving path; coverage-gain chosen).

---

2. What shipped

Piece	Path
Canonicals S1–S4	generate/structure_mapping/canonicals.py
Role schema + compare_add	generate/structure_mapping/role_predicate.py
Converter (incl. additive compare)	generate/structure_mapping/convert.py
Pure-family mappers S1–S4	generate/structure_mapping/mapper.py
Overlapping-waves selector	generate/structure_mapping/selector.py
Generalized 3-gate solve	generate/structure_mapping/solve.py
SM-owned pure-S1 text extract	generate/structure_mapping/text_extract.py
Pipeline (organ parse → extract fallback)	generate/structure_mapping/pipeline.py
Measure script	scripts/measure_trackb_inc2.py
Tests	tests/test_trackb_inc2_structure_mapping.py (15) + Inc1 suite (20)

Serving reader (parse_and_solve dispatch, organs, chat/runtime.py) untouched.

---

3. Measurements (command + real output)

3a. Generalization-ratio table

$ uv run python scripts/measure_trackb_inc2.py --mode ratio
=== generalization ratio (holdout_dev/v1, command-backed) ===
CASE gsm8k-holdout-dev-v1-0101 sid=S1 emit=9.0 gold=9.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0108 sid=S1 emit=110.0 gold=110.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0148 sid=S1 emit=1500.0 gold=1500.0 right=True src=sm_extract pat=s1_p3_day_pair ...
CASE gsm8k-holdout-dev-v1-0228 sid=S1 emit=600.0 gold=600.0 right=True src=sm_extract pat=s1_p2_b_seeded ...
CASE gsm8k-holdout-dev-v1-0234 sid=S1 emit=240.0 gold=240.0 right=True src=sm_extract pat=s1_p2_deaf_blind ...
CASE gsm8k-holdout-dev-v1-0268 sid=S1 emit=16.0 gold=16.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0411 sid=S1 emit=52.0 gold=52.0 right=True src=organ ...
CASE gsm8k-holdout-dev-v1-0441 sid=S1 emit=75.0 gold=75.0 right=True src=sm_extract pat=s1_p2_b_seeded ...
CASE gsm8k-holdout-dev-v1-0453 sid=S1 emit=28.0 gold=28.0 right=True src=organ ...
--- ratio table ---
canonical=S1 templates=1 cases_carried=9 ratio=9.000 ids=[...9 holdout ids...]
canonical=S2 templates=1 cases_carried=0 ratio=0.000 ids=[]
canonical=S3 templates=1 cases_carried=0 ratio=0.000 ids=[]
canonical=S4 templates=1 cases_carried=0 ratio=0.000 ids=[]
SUMMARY active_canonicals=1 total_cases=9 aggregate_ratio=9.000

All nine S1 carries include the 3-part proof (right answer, derivation b=k×a; total=a+b, multi-register + classical cert). Surface-variant re-extract of the inverted-seed form still maps (Ava/Ben → 48).

3b. Coverage gain

$ uv run python scripts/measure_trackb_inc2.py --mode coverage-gain
organ_correct=5 trackb_correct=9 newly_solved=4
  NEW gsm8k-holdout-dev-v1-0148 ans=1500.0 gold=1500.0 sid=S1 extract=s1_p3_day_pair
  NEW gsm8k-holdout-dev-v1-0228 ans=600.0 gold=600.0 sid=S1 extract=s1_p2_b_seeded
  NEW gsm8k-holdout-dev-v1-0234 ans=240.0 gold=240.0 sid=S1 extract=s1_p2_deaf_blind
  NEW gsm8k-holdout-dev-v1-0441 ans=75.0 gold=75.0 sid=S1 extract=s1_p2_b_seeded

These four are real surface-distinct holdout cases, not synthetic clones. The SM-owned extract layer is structure-mapping-owned (regex pure-family templates) and runs only when the serving reader refuses — it does not modify the live reader.

3c. Selector

$ uv run python scripts/measure_trackb_inc2.py --mode selector
CASE real_s1_0411 expect=S1 got=S1 refused=False ok=True
CASE real_s2_public_transfer expect=S2 got=S2 refused=False ok=True
CASE struct_s2_not_s1 expect=S2 got=S2 refused=False ok=True
CASE struct_s1 expect=S1 got=S1 refused=False ok=True
CASE empty_ops_refuse expect=None got=None refused=True reason=no_family_matched ok=True
CASE struct_s3_rate expect=S3 got=S3 refused=False ok=True

Transfer graphs route to S2 even when the surface text could mention multiplicative language elsewhere; empty ops refuse. Ranking uses fixed systematicity scores, not entity names or numbers.

3d. Parser frontier

organ_selected_graph_count=5/500
role_kind_multiset_counts={'compare,contain,total': 5}
--- case 0148 ---
organ_parsed=False
sm_pipeline emit=True answer=1500.0 gold=1500.0 src=sm_extract sid=S1

0148 (the original S1 family namesake) is a parser-frontier refusal on the serving reader; the SM-owned extract recovers it. Holdout still has zero organ graphs for transfer/rate/additive-compare families — that is the next structural bottleneck, not the mapper.

3e. S2 on real public transfers (not holdout; engineering evidence only)

SUMMARY s2_public_sample: ok=5 scanned_transferish=20
  S2 gma-102 ans=54.0 ... transfer 7 Eve→David ...

S2 pure-family map+solve works on real public transfer graphs. Not counted in the holdout ratio table (discipline: holdout IDs only).

3f. Right-reason summary

SUMMARY right-reason: correct=9 wrong=0 refused=0
SUMMARY surface-variant: ok=True

3g. Full holdout wrong=0 gate (all 500 — not curated subset)

Regression: a loose inverted-seed regex matched multi-entity zoo case 0361 (John has 8 more pandas…) as pure S1 and the corridor emitted 1.125 vs gold 114.0. Fixed by pure-family surface gates (require explicit multiplicative language; refuse bare N more / multi-clause / multi-numeric surfaces).

$ uv run python scripts/measure_trackb_inc2.py --mode full-holdout-wrong0
=== full holdout_dev/v1 wrong=0 gate (all 500 cases) ===
SUMMARY full-holdout: emit_ok=9 wrong=0 refused=491 n=500
PASS: wrong=0 absolute on full holdout_dev/v1

---

4. Discipline checks

• Blind: mappers/selector take only RoleGraph; no label parameter. Static audit: generate/structure_mapping/ does not import scoring labels.
• wrong=0: emit only if classical verify and multi-register cert and agreement; original-graph backstop when a graph is supplied.
• No crash-as-signal: corridor exceptions → refuse outcomes.
• Off-serving: no organ retirement; no parse_and_solve dispatch edits.

---

5. Deviations / frontiers (honest)

1. S2–S4 holdout ratio = 0 until the serving reader (or a reviewed SM extract family) produces pure transfer/rate/additive graphs on holdout. Do not manufacture synthetic holdout clones for ratio.
2. S3/S4 solve emit blocked by multi-register scope (apply_rate, compare_additive not Tier-2a). Gates not weakened — refuse.
3. SM extract is narrow (regex pure-S1 templates only, with fail-closed purity gates). It is not a general parser; expanding it is a deliberate future increment. Overmatch risk is gated by mode full-holdout-wrong0 on all 500 cases.
4. Organ retirement deferred; coverage-gain path met bar 2 without serving risk.
5. 0361 pure-S1 overmatch (fixed in follow-up commit): optional times in P2 regex treated additive 8 more as multiplicative; purity gate + required times as many closed it. Full-holdout scan is now mandatory before claiming wrong=0.

---

6. STOP

One PR (base main). Reviewer verifies:

• blindness + pure-family gates,
• ratio > 1 cases are real holdout IDs (not clones),
• wrong=0 three-gate path,
• serving reader unchanged.

No S2–S4 serving cutover until a later ruling. No sealed-test run.