Records the increment-1 outcome straight (band-solve=0, ~30-band bet falsified; foundations = banked byproduct not success) + the minimal-band measurement: no small band converts; taxonomy not at bedrock (GSM8K statements are per-sentence multi-capability); multi-compound highest-leverage (70/106 tractable needed-sets) but insufficient alone; q:complex biggest intractable wall (~35 cases). Increment-2 recommendation = CASE-FIRST (target specific closest cases, build their exact end-to-end needs) not capability-first, since the taxonomy fragments. Foundations verified full-500 wrong=0, smoke 176.
14 KiB
Increment-1 Band Plan — reader arc (RULED — APPROVED TO BUILD)
RULING (Josh, PR #79, 2026-07-19) — approved to build under these
- Q1 depletion → REFUSE, confirmed. Count depletion-containing band cases as expected haircut, not bugs.
- Q2 existential → admit CONTAINER-BOUND only, refuse container-less, confirmed. If tune surfaces a container-bound case that is actually a partitioned total, refuse that sub-form too.
- Q3 q:difference → DEFER the capability, CLOSE the hazard now. Its 23 cases likely strand on
multi-compound(increment 2), so building the direction driver now is risk without in-band yield — defer to where its cases convert, and verify the co-occurrence to confirm. BUT make the existing partial Pattern-B ("how many more") path fail-closed immediately: if it can emit a guessed-direction difference today, that is a latent wrong=0 hazard to close regardless. Increment 1 then carries EXACTLY ONE wrong=0 driver (#78). - Sharpening 1 — allowlist-miss refusals are recorded as curriculum (unknown-verb refusals feed the practice/coverage backlog, not treated as failures).
- Sharpening 2 — the sha256 tune/measure split stays FIXED across increments 1–3 (the ~40 bar is
cumulative;
evals/gsm8k_math/holdout_dev/v1/split.pyis frozen). - Sharpening 3 — the haircut factor ships with the triple in the final report.
The §3/§4/§6 design below is unchanged EXCEPT: q:difference (item 4) becomes hazard-close only, not a built capability; increment 1 carries one wrong=0 driver (#78), not two.
Status: RULED — approved to build (measure-once). Design-first gate cleared (same gate as PR #76).
BUILD OUTCOME (2026-07-19) — band-solve=0; the ~30-band bet is FALSIFIED
The headline stays band-solve=0. The two foundations are built and verified; they are the banked byproduct, not a success against increment 1's conversion goal.
Shipped (verified, wrong=0-safe)
The #78 positive-polarity allowlist substrate — NEUTRAL_COUNT_VERBS (math_roundtrip.py, extends
ADD_VERBS with curated neutral production verbs), shared by the seed matcher (acquisition path) and the
comparison _comparison_anchor_verb(). Depletion/transfer verbs refuse by positive determination
(not a blocklist). Verified: production injects / depletion refuses; 71 pinned comparative tests pass;
full-500 wrong=0 (tune 261 + measure 239); PARSED unchanged (tune 3, measure 2 — no regression, no
conversion); smoke 176 green. Merge gate met: full-500 wrong=0 + refusal histogram stable-or-explained.
band-solve = 0, measured FOUR ways
yield harness, band map, band oracle (rewrite-and-run), capability histogram — all agree: the scoped band
{seed + forward-comparison + q:simple/summation} converts 0 tune cases. Its otherwise-in-band cases
strand on capabilities outside the band: 25 need multi-compound, 17 need compare-additive. §5 named
this exact risk; it landed.
Minimal-convertible-band measurement (Josh's option-3 spec)
- No small band converts. The band oracle converts 0 with
{seed + comparison + q}. - Taxonomy is NOT at bedrock. The refined atomic classifier's "smallest tractable set"
{seed, q:simple}(4 cases) dissolved on inspection — every one actually needs rate / compare / currency / copula (e.g. "James spends 30 minutes twice a day on meditation" = rate + temporal + unit-conversion in ONE sentence). Real GSM8K statements are individually multi-capability; the conjunction recurs within a sentence, so a capability list keeps fragmenting. - multi-compound is the single highest-leverage capability — on the critical path for 70/106 (66%) of tractable needed-sets — but insufficient alone (its cases also need rate/currency/etc.).
- q:complex is the biggest intractable wall (~35+ cases gated on it: 3 size-1, 18+14 size-2 needed-sets).
- Closest cases (0000/0001/0148/0082) all need complex compare forms (chained / mass-noun entities / "than" + aggregate reference) + summation/difference — still multi-capability.
Implication for increment 2 (the first-conversion increment) — CASE-FIRST, not capability-first
Because the taxonomy fragments, no small capability band is a real floor. The path to the first conversion
is case-first: pick 2–3 specific closest tune cases, hand-enumerate their exact end-to-end needs
(however many capabilities), build precisely that, convert them, measure. The "minimal convertible band" is
not a small capability set — it is whatever a chosen handful of real cases need, built end-to-end. This is
the recommendation carried to Josh with the foundations PR.
Date: 2026-07-19
Base: forgejo/main @ e1eb2a5c (worktree core-wt-seed, branch feat/seeding-injection → to be
repurposed to the band).
Model context: unit of work = case-BAND (coverage is a per-case conjunction; see
compare-increment-funnel-2026-07-18.md MEASURED OUTCOME + the band map). Done-when = band-solve wrong=0,
NOT injection coverage.
§1 The band (approved)
{ seed-simple + comparison (#78 polarity) + question-arithmetic (simple / summation / difference) }
The tractable spine's foundational core — the two capabilities every future band reuses (question-arithmetic + #78 polarity) plus the two most common statement types (seed, comparison), adding zero new hard capabilities beyond #78 (which is already scoped). Band-map upper bound ≈ 30 cases (size 3–4). Deliberately the ~30 band, not the 42 band — see §5.
§2 What EXISTS today (grounding — increment 1 EXTENDS, does not rebuild)
- Compare COMPILER: merged (PR #77) —
compare_multiplicativeexecutes wrong=0 (27/27). The reader frame was reverted to the_comparison_anchor_verb()whitelist (issue #78 open). - Question layer, PARTIALLY built (
generate/math_candidate_parser.py,CandidateUnknown):q:summation— aggregate-cue vocab {in total, altogether, combined, together, in all} →entity=None→ the 2b summation compiler. Works today (0101 solved via compare + summation).q:simple— "How many<unit>does<Entity>have?" → single-register decode. Works.q:difference— PARTIAL: "how many more<X>" Pattern-B setspeer_count; direction is not yet a positively-determined, fail-closed decision (§4 driver 2).extract_conditional_op_question_candidatesexists (some q:complex conditional forms).
- Seed injector (
inject_discrete_count_statement+_try_extract_discrete_count_anchor): whitelists possession {has/have/had} →CandidateInitial, acquisition {collected/received/bought/got} →CandidateOperation(add); refuses every other verb and all existential "There are N X" (fail-closed).
So the question-arithmetic foundation largely exists; increment 1 completes it (q:difference direction) and adds the seed + comparison statement readers on the shared #78 polarity substrate.
§3 The gap increment 1 builds
- #78 polarity substrate (the shared foundation). Positive polarity determination = a
neutral-polarity ALLOWLIST, fail-closed (NOT the reverted fail-open blocklist):
_NEUTRAL_SEED_VERBS(production/possession/acquisition: made/makes, baked, grew, scored, wrote, caught, taught, built, planted, picked, earned, harvested, found, drew, cooked, produced, + existing has/have/had/collected/received/bought/got) → seed (InitialPossessionor from-zeroadd)._DEPLETION_VERBS(lost/spent/gave/sold/used/donated/ate/dropped/paid/lent) → refuse in increment 1 (a depletion is a delta over prior state, not a seed; modelling it is out of scope — fail-closed).- Unknown verb → refuse. Allowlist, positively determined. This closes issue #78 for both seed and comparison, built once.
- Seed-simple injection — extend the base injector via the substrate + a new existential path
("There are
NX[in/on theC]" →InitialPossession(entity=C-or-collective, N X); zero polarity risk). - Comparison frame-anchoring — re-land the verb-free frame (the reverted
c3aed13b), but the verb slot is gated by the same_NEUTRAL_SEED_VERBSallowlist (fail-closed) rather than a blocklist. Depletion/unknown verb in a frame → refuse. Reuses the merged compare compiler. - q:difference direction (§4 driver 2) — complete the Pattern-B path: "how many more A than B" →
signed
A − B; "how many fewer/less" →B − A; ambiguous/unsupplied direction → refuse. - q:simple / q:summation — verify coverage on the band's cases; extend minimally only where a target case needs it (measured on tune).
§4 The TWO wrong=0 direction drivers (Josh point 4 — positively determined, fail-closed)
- Driver 1 — statement polarity (#78): neutral-seed allowlist vs depletion. Seed vs deplete decided by verb class, fail-closed on unknown.
- Driver 2 — question q:difference direction: "more…than" →
A−B, "fewer…than" →B−A, ambiguous → refuse. Same discipline as #78 and the inverse-compare lesson resurfacing in the question layer. A wrong direction here is a coherent-but-inverted answer on the run-once arbiter — the exact fail-open hazard we refused to ship in #77.
§5 Deliberate smaller-band rationale (Josh point 1 — recorded so it is NOT read as under-ambition)
The map says a "real bar" needs size-4/~42; this band is the ~30, size-3–4 one, by design:
- Increment 1's job is the FIRST end-to-end reader conversion the arc has ever landed. Corridor real-reach is still 5/500 — all from the compiler tier; no reader increment has converted a single case. The bar for increment 1 is crossing zero, on foundations.
- It ships the two foundations every future band reuses (question-arithmetic completion + #78 polarity) and adds zero new hard capabilities beyond #78 (already scoped).
- The 42 band adds multi-compound — a new hard capability. Stacking a new hard capability onto the first-ever conversion attempt raises the odds increment 1 also lands ~0. Smaller-but-foundational maximizes the probability of the critical first win, and every capability it ships is reused, never redone.
§6 Done-when (Josh point 2 — reset for the band model; X=15-per-layer retired)
Increment 1 is DONE when:
- band-solve > 0 wrong=0 on the disjoint measure split — the first real reader conversion; AND
- both foundations validated and reuse-ready: #78 polarity allowlist (fail-closed, wrong=0 on the depletion confusers) + q:difference direction (fail-closed on ambiguity); AND
- wrong=0 held on tune + measure + smoke; cross-family parses stable-or-explained.
The ~40 "real bar" is CUMULATIVE across increments 1–3 (foundation → +multi-compound ≈ 42 → +currency/rate ≈ 63), NOT increment 1 alone. Forcing one increment to clear a bar the map says needs five capabilities would recreate the unsatisfiable-threshold problem at band scale.
§7 Measurement (Josh point 3 + discipline)
- Develop-on-tune / measure-once. wrong=0 verified on tune throughout; measure split run ONCE at the end.
- HAIRCUT FACTOR = a named increment-1 output:
real band-solve ÷ ~30 upper bound. This ratio — the single most valuable number beyond the conversions themselves — recalibrates the projected yield of every future band. Reported explicitly with the triple. - Funnel re-run after each layer; refusal-histogram pinned + asserted stable-or-explained; cross-family parses unchanged-or-better; the depletion-confuser suite proves the polarity allowlist fail-closed.
§8 Out of scope + roadmap boundary
- Intractable-now (52%, 135/258):
q:complex(101 — conditional/averaging/multi-step), percent, simultaneous-equations, fraction/partitive. Explicitly OUT. - Increment 2–3 (the cumulative bar):
multi-compound(→42),currency/rate(→63). - Tractable holdout ceiling ≈ 48% until the q:complex decomposition (§9) revises it.
§9 Scheduled NEXT (not increment-1 scope — Josh's forward directive)
Decompose q:complex (101 cases) by sub-type BEFORE treating 48% as a fixed ceiling. Hypothesis:
"averaging" = sum-over-count and "simple multi-step" = a turn program — both may be corridor-tractable
with the compiler we already have, hiding inside the "intractable" 101. If even 30–40 are reachable, the
ceiling moves materially. Measure before planning around 48% as fixed. Does NOT touch increment-1 scope.
Roadmap boundary to rule on in principle: when the tractable bands are exhausted (~48% reached), the serve / practice / tackle-intractable fork opens. The conjunction structure is a wall for the practice lane too (reward = full-conjunction case-solve is near-zero until a closure happens by luck), so the ceiling argues for continued design-first, not switching.
§10 Open questions for the ruling
- Q1 — depletion verbs: refuse vs model-as-subtract in increment 1? Plan says refuse (fail-closed, smallest scope). Modelling "X had M, lost N" needs prior-state + subtract-operation wiring — proposed for a later increment. Confirm refuse-only for now.
- Q2 — existential entity binding. "There are N X in the C" → entity = C (container). Container-less ("There are N X") → collective entity from the noun, or refuse? Plan leans refuse when no container/subject grounds the register (fail-closed), admit only container-bound. Confirm.
- Q3 — is q:difference in-scope for increment 1, or deferred to keep the band at seed+comparison+q:simple/summation? Plan includes it (23 cases, and it completes the question-arithmetic foundation), but it adds the 2nd polarity driver. If you'd rather minimize the first-conversion risk surface, q:difference can drop to increment 1.5. Recommend keeping it (foundational, and its cases are otherwise stranded).