core/docs/analysis/gsm8k-capability-paradigm-sprint10-lookback-2026-06-17.md
Shay a37f390009
feat(derivation): capability paradigm sprint 10 frontier lift (#823)
* feat(derivation): capability paradigm sprint 10 frontier lift

Add Gate A2o affine_comparative_inversion_total (0009) and Gate A2p
sequential_comparative_scale (0006). Reject wholesale multiplicative_aggregate
and defer 0013 piecewise calendar until month day-count grounds in text.

Serving: 21/29/0 → 23/27/0, wrong=0 preserved. report.json and sealed
artifacts untouched.

* fix(gsm8k): bind affine inversion total question subject

* fix(gsm8k): bind sequential scale page question subject

* test(gsm8k): cover sprint10 subject binding

* fix(gsm8k): license affine inversion aggregate units

* fix(gsm8k): bind sequential scale factors to reading chain

* test(gsm8k): cover sprint10 aggregate and scale confusers
2026-06-18 12:15:52 -07:00

185 lines
No EOL
6.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# GSM8K Capability Paradigm Sprint 10 — Lookback (2026-06-17)
## 1. Starting baseline
| Metric | Value |
|--------|------:|
| correct | **21** |
| refused | **29** |
| wrong | **0** |
After #822 (Sprint 9 post-merge hardening). Prerequisite satisfied.
## 2. Experience Flywheel output summary
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint10.json`
| Field | Value |
|-------|------:|
| retained_record_count | 50 |
| lift_refused_to_correct | **0** |
| promotion_candidates | all `blocked_by_wrong_risk` or diagnostic hold |
Top signals:
- `multiplicative_aggregate:multiplicative_aggregation` — 0006, 0013 joint refusal; **0 sealed-wrong** on surface tag but wholesale MA blocked by 0025/0047 neighbors.
- Singletons 0004/0007/0009 — `joint_sealed_no_resolution`; no `candidate_family` cluster.
- Blocked: DCS (0032, 0047), currency (0019, 0028), sealed_elimination (0011, 0026).
## 3. Scout/frontier/microscope output summary
| Regime | correct | refused | wrong |
|--------|--------:|--------:|------:|
| Serving | 21 | 29 | 0 |
| Sealed (resolve_pooled) | 3 | 39 | 8 |
- `lift_refused_to_correct`: **0**
- Dominant bottleneck: joint refusal (23/50)
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
Microscope: 0006/0013 tagged `multiplicative_aggregate`; 0004 fraction surface; 0007 unbound_target; 0009 unbound_target.
## 4. Candidate families ranked
| Rank | Family | Cases | Lift | Wrong-risk | Verdict |
|------|--------|-------|------|------------|---------|
| 1 | `piecewise_daily_hours_total` | 0013 | +1 | medium | **Deferred** — June=30 not grounded in problem text |
| 2 | `sequential_comparative_scale` | 0006 | +1 | medium | **Selected (A2p)** |
| 3 | `affine_comparative_inversion_total` | 0009 | +1 | lowmedium | **Selected (A2o)** |
| 4 | `multiplicative_aggregate` wholesale | 0006, 0013 | +2? | **high** (0025, 0047) | **Rejected** |
| 5 | `nested_fraction_partition` | 0004 | +1 | high (0026) | **Rejected** |
| 6 | `loose_inventory_boxing` | 0007 | +1 | high (0047) | **Rejected** |
| — | DCS / currency / sealed_elimination | — | — | blocked | **Deferred** |
## 5. Blocked families and why
| Family | Blocked because |
|--------|-----------------|
| `multiplicative_aggregate` wholesale | 0025, 0047 sealed-wrong share MA surface; 0006/0013 not end-to-end MA |
| `piecewise_daily_hours_total` | 0013 lacks explicit month day-count token (30 never appears) |
| `nested_fraction_partition` | 0026 sealed-wrong shares fraction chain |
| `loose_inventory_boxing` | 0047 sealed-wrong shares divisive packing |
| DCS / currency / sealed_elimination | unchanged blocked set |
## 6. Research gate decision
**Proceed with dual narrow organs** — not wholesale `multiplicative_aggregate`.
Arena subagents converged:
- 0006 and 0013 do **not** share one safe organ (Sprint 6 decomposition reaffirmed).
- 0013 blocked on calendar grounding for wrong=0 self-verification.
- 0009 clears negative-evidence audit (no sealed-wrong neighbor bleed).
- 0006 admits typed sequential scale chain with body-scoped completeness.
## 7. Selected paradigm
**Dual-organ Sprint 10** (independently narrow, independently verified):
1. **Gate A2o `affine_comparative_inversion_total`**`N more A than M×B` + conditional given + `total` aggregate question.
2. **Gate A2p `sequential_comparative_scale`** — initial `pages` + ordered `times longer` / `previous length` scale chain.
Rejected: broad MA parser, piecewise calendar without grounded days, 0004/0007 singletons, product_bridge, DCS injector.
## 8. Typed chains for target cases
**0009 (gold 185):** `chickens = (15010)/4 = 35``total = 150+35 = 185`.
**0006 (gold 480):** `8×5×3×4 = 480` (running page-length scale chain; age/year scaffolding excluded from completeness).
**0013 (deferred):** needs `days_half = 30×½` but **30 is not in problem text**.
## 9. Implemented organs
| Module | Promotion entry |
|--------|-----------------|
| `generate/derivation/affine_comparative_inversion_total.py` | `resolve_promotable_affine_comparative_inversion_total` |
| `generate/derivation/sequential_comparative_scale.py` | `resolve_promotable_sequential_comparative_scale` |
Wired in `generate/math_candidate_graph.py` after Gate A2n.
## 10. Before/after score
| Metric | Before | After | Δ |
|--------|-------:|------:|--:|
| correct | 21 | **23** | +2 |
| refused | 29 | **27** | 2 |
| wrong | 0 | **0** | 0 |
**State A achieved:** 23/27/0 (target was 23+/27-/0).
## 11. Newly solved IDs
**0006**, **0009**
## 12. Preserved solved IDs
0001, 0002, 0003, 0005, 0008, 0010, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046
## 13. Wrong=0 proof
- Full train_sample replay: 23 / 27 / 0
- `tests/test_math_candidate_graph_sprint10_frontier_lift.py::TestTrainSampleScore`
- Holdout_dev: wrong=0 (no new admissions required)
- Smoke suite: 108 passed
- Regression lane: 272+ passed (excluding pre-existing `report.json` historical pin drift)
## 14. Confuser matrix
| Confuser | Organ | Result |
|----------|-------|--------|
| Actor mismatch in conditional | A2o | refuse |
| No conditional given | A2o | refuse |
| Yield / non-nested comparative | A2o | refuse |
| Asks age not pages | A2p | refuse |
| Doubled weight (holdout shape) | A2p | refuse |
| Single scale step only | A2p | refuse |
| MA 0025 basket shape | A2p | refuse |
| MA 0047 macaroon shape | A2p | refuse |
| 0013 calendar without day token | piecewise (deferred) | refuse |
## 15. Sibling generality proof
- A2o sibling: `6 more apples than 3× oranges` + conditional → `(426)/3+42`
- A2p sibling: `12 pages` + `3× longer` + `2× longer` + `5× previous``12×3×2×5`
## 16. Sealed-wrong negative evidence used
- Rejected wholesale MA: 0025 (group under-count), 0047 (remainder vs total)
- Rejected 0004: 0026 fraction/money chain
- Rejected 0007: 0047 divisive packing
- A2o/A2p confuser tests include 0025/0047 surface shapes
## 17. Composition-validation pin changes
**None.** No new cv rows; organs gated by self-verification only.
## 18. report.json untouched
Yes.
## 19. Sealed artifacts untouched
Yes.
## 20. No case-id logic
Yes.
## 21. No hardcoded answer
Yes.
## 22. No direct-answer shortcut
Yes.
## 23. Sprint 11 recommendation
**Primary:** Ratify `piecewise_daily_hours_total` only after calendar grounding policy — either require explicit month day-count in problem text or a pinned, tested month-length table with confusers (0014 single-rate, 0017 tariff, MA opener misparse).
**Secondary:** Revisit 0013 jointly with temporal exemplar `ta-v1-0001` injection design — not recognizer widening.
**Non-goals:** DCS (0032/0047), currency (0019/0028), sealed_elimination (0011/0026), 0004 nested fraction until 0026 confuser matrix, 0007 boxing until 0047 confuser matrix.
**Experience Flywheel:** Continue compaction; no serving promotion from flywheel alone until `lift_refused_to_correct > 0` with sealed-wrong clearance.