core/docs/research/increment-1-band-plan-2026-07-19.md
Shay c217af31ee docs(reader-arc): band-solve=0 outcome + minimal-convertible-band measurement
Records the increment-1 outcome straight (band-solve=0, ~30-band bet
falsified; foundations = banked byproduct not success) + the minimal-band
measurement: no small band converts; taxonomy not at bedrock (GSM8K statements
are per-sentence multi-capability); multi-compound highest-leverage (70/106
tractable needed-sets) but insufficient alone; q:complex biggest intractable
wall (~35 cases). Increment-2 recommendation = CASE-FIRST (target specific
closest cases, build their exact end-to-end needs) not capability-first, since
the taxonomy fragments. Foundations verified full-500 wrong=0, smoke 176.
2026-07-18 21:37:16 -07:00

196 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Increment-1 Band Plan — reader arc (RULED — APPROVED TO BUILD)
## RULING (Josh, PR #79, 2026-07-19) — approved to build under these
- **Q1 depletion → REFUSE, confirmed.** Count depletion-containing band cases as *expected haircut*, not bugs.
- **Q2 existential → admit CONTAINER-BOUND only, refuse container-less, confirmed.** If tune surfaces a
container-bound case that is actually a partitioned total, refuse that sub-form too.
- **Q3 q:difference → DEFER the capability, CLOSE the hazard now.** Its 23 cases likely strand on
`multi-compound` (increment 2), so building the direction driver now is risk without in-band yield —
defer to where its cases convert, and **verify the co-occurrence to confirm**. BUT make the existing
partial Pattern-B ("how many more") path **fail-closed immediately**: if it can emit a guessed-direction
difference today, that is a latent wrong=0 hazard to close regardless. **Increment 1 then carries EXACTLY
ONE wrong=0 driver (#78).**
- **Sharpening 1** — allowlist-miss refusals are recorded as **curriculum** (unknown-verb refusals feed the
practice/coverage backlog, not treated as failures).
- **Sharpening 2** — the sha256 tune/measure split stays **FIXED across increments 13** (the ~40 bar is
cumulative; `evals/gsm8k_math/holdout_dev/v1/split.py` is frozen).
- **Sharpening 3** — the **haircut factor** ships with the triple in the final report.
The §3/§4/§6 design below is unchanged EXCEPT: q:difference (item 4) becomes *hazard-close only*, not a
built capability; increment 1 carries one wrong=0 driver (#78), not two.
**Status**: RULED — approved to build (measure-once). Design-first gate cleared (same gate as PR #76).
---
## BUILD OUTCOME (2026-07-19) — band-solve=0; the ~30-band bet is FALSIFIED
The headline stays band-solve=0. The two foundations are **built and verified**; they are the banked
byproduct, **not** a success against increment 1's conversion goal.
### Shipped (verified, wrong=0-safe)
The **#78 positive-polarity allowlist substrate** — `NEUTRAL_COUNT_VERBS` (`math_roundtrip.py`, extends
`ADD_VERBS` with curated neutral production verbs), shared by the seed matcher (acquisition path) and the
comparison `_comparison_anchor_verb()`. Depletion/transfer verbs refuse by **positive determination**
(not a blocklist). Verified: production injects / depletion refuses; **71 pinned comparative tests pass**;
**full-500 wrong=0** (tune 261 + measure 239); PARSED unchanged (tune 3, measure 2 — no regression, no
conversion); **smoke 176 green**. Merge gate met: full-500 wrong=0 + refusal histogram stable-or-explained.
### band-solve = 0, measured FOUR ways
yield harness, band map, band oracle (rewrite-and-run), capability histogram — all agree: the scoped band
`{seed + forward-comparison + q:simple/summation}` converts **0** tune cases. Its otherwise-in-band cases
**strand on capabilities outside the band**: 25 need multi-compound, 17 need compare-additive. §5 named
this exact risk; it landed.
### Minimal-convertible-band measurement (Josh's option-3 spec)
- **No small band converts.** The band oracle converts 0 with `{seed + comparison + q}`.
- **Taxonomy is NOT at bedrock.** The refined atomic classifier's "smallest tractable set" `{seed, q:simple}`
(4 cases) dissolved on inspection — every one actually needs rate / compare / currency / copula
(e.g. "James **spends 30 minutes twice a day** on meditation" = rate + temporal + unit-conversion in ONE
sentence). **Real GSM8K statements are individually multi-capability**; the conjunction recurs *within*
a sentence, so a capability list keeps fragmenting.
- **multi-compound is the single highest-leverage capability** — on the critical path for **70/106 (66%)**
of tractable needed-sets — but insufficient alone (its cases also need rate/currency/etc.).
- **q:complex is the biggest intractable wall** (~35+ cases gated on it: 3 size-1, 18+14 size-2 needed-sets).
- **Closest cases** (0000/0001/0148/0082) all need complex compare forms (chained / mass-noun entities /
"than" + aggregate reference) + summation/difference — still multi-capability.
### Implication for increment 2 (the first-conversion increment) — CASE-FIRST, not capability-first
Because the taxonomy fragments, no small capability band is a real floor. The path to the first conversion
is **case-first**: pick 23 specific closest tune cases, hand-enumerate their *exact* end-to-end needs
(however many capabilities), build precisely that, convert them, measure. The "minimal convertible band" is
not a small capability set — it is whatever a chosen handful of real cases need, built end-to-end. This is
the recommendation carried to Josh with the foundations PR.
**Date**: 2026-07-19
**Base**: `forgejo/main @ e1eb2a5c` (worktree `core-wt-seed`, branch `feat/seeding-injection` → to be
repurposed to the band).
**Model context**: unit of work = case-BAND (coverage is a per-case conjunction; see
`compare-increment-funnel-2026-07-18.md` MEASURED OUTCOME + the band map). Done-when = band-solve wrong=0,
NOT injection coverage.
---
## §1 The band (approved)
**`{ seed-simple + comparison (#78 polarity) + question-arithmetic (simple / summation / difference) }`**
The tractable spine's *foundational core* — the two capabilities every future band reuses
(question-arithmetic + #78 polarity) plus the two most common statement types (seed, comparison), adding
**zero new hard capabilities beyond #78** (which is already scoped). Band-map upper bound ≈ **30 cases**
(size 34). Deliberately the ~30 band, not the 42 band — see §5.
## §2 What EXISTS today (grounding — increment 1 EXTENDS, does not rebuild)
- **Compare COMPILER**: merged (PR #77) — `compare_multiplicative` executes wrong=0 (27/27). The reader
frame was reverted to the `_comparison_anchor_verb()` whitelist (issue #78 open).
- **Question layer, PARTIALLY built** (`generate/math_candidate_parser.py`, `CandidateUnknown`):
- `q:summation` — aggregate-cue vocab {in total, altogether, combined, together, in all} → `entity=None`
→ the 2b summation compiler. **Works today** (0101 solved via compare + summation).
- `q:simple` — "How many `<unit>` does `<Entity>` have?" → single-register decode. Works.
- `q:difference` — PARTIAL: "how many more `<X>`" Pattern-B sets `peer_count`; **direction is not yet a
positively-determined, fail-closed decision** (§4 driver 2).
- `extract_conditional_op_question_candidates` exists (some q:complex conditional forms).
- **Seed injector** (`inject_discrete_count_statement` + `_try_extract_discrete_count_anchor`): whitelists
possession {has/have/had} → `CandidateInitial`, acquisition {collected/received/bought/got} →
`CandidateOperation(add)`; **refuses every other verb and all existential "There are N X"** (fail-closed).
So the question-arithmetic *foundation largely exists*; increment 1 **completes** it (q:difference
direction) and adds the **seed + comparison** statement readers on the shared **#78 polarity substrate**.
## §3 The gap increment 1 builds
1. **#78 polarity substrate (the shared foundation).** Positive polarity determination = a
**neutral-polarity ALLOWLIST**, fail-closed (NOT the reverted fail-open blocklist):
- `_NEUTRAL_SEED_VERBS` (production/possession/acquisition: made/makes, baked, grew, scored, wrote,
caught, taught, built, planted, picked, earned, harvested, found, drew, cooked, produced, + existing
has/have/had/collected/received/bought/got) → seed (`InitialPossession` or from-zero `add`).
- `_DEPLETION_VERBS` (lost/spent/gave/sold/used/donated/ate/dropped/paid/lent) → **refuse** in increment 1
(a depletion is a delta over prior state, not a seed; modelling it is out of scope — fail-closed).
- **Unknown verb → refuse.** Allowlist, positively determined. This *closes issue #78* for both seed
and comparison, built once.
2. **Seed-simple injection** — extend the base injector via the substrate + a new **existential** path
("There are `N` `X` [in/on the `C`]" → `InitialPossession(entity=C-or-collective, N X)`; zero polarity risk).
3. **Comparison frame-anchoring** — re-land the verb-free frame (the reverted `c3aed13b`), but the verb
slot is gated by the **same `_NEUTRAL_SEED_VERBS` allowlist** (fail-closed) rather than a blocklist.
Depletion/unknown verb in a frame → refuse. Reuses the merged compare compiler.
4. **q:difference direction (§4 driver 2)** — complete the Pattern-B path: "how many **more** A than B" →
signed `A B`; "how many **fewer/less**" → `B A`; **ambiguous/unsupplied direction → refuse**.
5. **q:simple / q:summation** — verify coverage on the band's cases; extend minimally only where a target
case needs it (measured on tune).
## §4 The TWO wrong=0 direction drivers (Josh point 4 — positively determined, fail-closed)
- **Driver 1 — statement polarity (#78):** neutral-seed allowlist vs depletion. Seed vs deplete decided
by verb class, fail-closed on unknown.
- **Driver 2 — question q:difference direction:** "more…than" → `AB`, "fewer…than" → `BA`, ambiguous →
**refuse**. Same discipline as #78 and the inverse-compare lesson resurfacing in the question layer. A
wrong direction here is a coherent-but-inverted answer on the run-once arbiter — the exact fail-open
hazard we refused to ship in #77.
## §5 Deliberate smaller-band rationale (Josh point 1 — recorded so it is NOT read as under-ambition)
The map says a "real bar" needs size-4/~42; this band is the ~30, size-34 one, **by design**:
- **Increment 1's job is the FIRST end-to-end reader conversion the arc has ever landed.** Corridor
real-reach is still 5/500 — *all* from the compiler tier; **no reader increment has converted a single
case.** The bar for increment 1 is crossing zero, on foundations.
- It ships the **two foundations every future band reuses** (question-arithmetic completion + #78 polarity)
and adds **zero new hard capabilities beyond #78** (already scoped).
- The 42 band adds **multi-compound** — a *new* hard capability. Stacking a new hard capability onto the
first-ever conversion attempt raises the odds increment 1 *also* lands ~0. **Smaller-but-foundational
maximizes the probability of the critical first win**, and every capability it ships is reused, never
redone.
## §6 Done-when (Josh point 2 — reset for the band model; X=15-per-layer retired)
Increment 1 is DONE when:
1. **band-solve > 0 wrong=0** on the disjoint measure split — the first real reader conversion; AND
2. both foundations **validated and reuse-ready**: #78 polarity allowlist (fail-closed, wrong=0 on the
depletion confusers) + q:difference direction (fail-closed on ambiguity); AND
3. wrong=0 held on tune + measure + smoke; cross-family parses stable-or-explained.
The **~40 "real bar" is CUMULATIVE across increments 13** (foundation → +multi-compound ≈ 42 →
+currency/rate ≈ 63), NOT increment 1 alone. Forcing one increment to clear a bar the map says needs five
capabilities would recreate the unsatisfiable-threshold problem at band scale.
## §7 Measurement (Josh point 3 + discipline)
- **Develop-on-tune / measure-once.** wrong=0 verified on tune throughout; measure split run ONCE at the end.
- **HAIRCUT FACTOR = a named increment-1 output**: `real band-solve ÷ ~30 upper bound`. This ratio — the
single most valuable number beyond the conversions themselves — recalibrates the projected yield of every
future band. Reported explicitly with the triple.
- Funnel re-run after each layer; refusal-histogram pinned + asserted stable-or-explained; cross-family
parses unchanged-or-better; the depletion-confuser suite proves the polarity allowlist fail-closed.
## §8 Out of scope + roadmap boundary
- **Intractable-now (52%, 135/258):** `q:complex` (101 — conditional/averaging/multi-step), percent,
simultaneous-equations, fraction/partitive. Explicitly OUT.
- **Increment 23 (the cumulative bar):** `multi-compound` (→42), `currency/rate` (→63).
- Tractable holdout ceiling ≈ **48%** until the q:complex decomposition (§9) revises it.
## §9 Scheduled NEXT (not increment-1 scope — Josh's forward directive)
**Decompose `q:complex` (101 cases) by sub-type BEFORE treating 48% as a fixed ceiling.** Hypothesis:
"averaging" = sum-over-count and "simple multi-step" = a turn program — both may be **corridor-tractable
with the compiler we already have**, hiding inside the "intractable" 101. If even 3040 are reachable, the
ceiling moves materially. Measure before planning around 48% as fixed. Does NOT touch increment-1 scope.
Roadmap boundary to rule on in principle: when the tractable bands are exhausted (~48% reached), the
serve / practice / tackle-intractable fork opens. The conjunction structure is a wall for the practice lane
*too* (reward = full-conjunction case-solve is near-zero until a closure happens by luck), so the ceiling
argues for **continued design-first**, not switching.
## §10 Open questions for the ruling
- **Q1 — depletion verbs: refuse vs model-as-subtract in increment 1?** Plan says refuse (fail-closed,
smallest scope). Modelling "X had M, lost N" needs prior-state + subtract-operation wiring — proposed
for a later increment. Confirm refuse-only for now.
- **Q2 — existential entity binding.** "There are N X in the C" → entity = C (container). Container-less
("There are N X") → collective entity from the noun, or refuse? Plan leans refuse when no container/subject
grounds the register (fail-closed), admit only container-bound. Confirm.
- **Q3 — is q:difference in-scope for increment 1, or deferred to keep the band at seed+comparison+q:simple/summation?**
Plan includes it (23 cases, and it completes the question-arithmetic foundation), but it adds the 2nd
polarity driver. If you'd rather minimize the first-conversion risk surface, q:difference can drop to
increment 1.5. Recommend keeping it (foundational, and its cases are otherwise stranded).