From c88e52c6a98f2ffc955e227ae297c2093358241a Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 18 Jul 2026 15:54:43 -0700 Subject: [PATCH 1/2] docs(reader-arc): compare_multiplicative increment plan (increment 1, for ruling) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First increment of the reader design-first arc (dev2 spike §7). Design-first, for ruling before build. Ground truth (5 real holdout_dev compare parses): one entity seeded, the other DEFINED by actor = factor x reference (Comparison, direction times/fraction); unknown = total -> existing 2b summation closes them once compare is solvable. Compiler tier: cross-register affine definition actor := factor x reference (dilate reference STATE by -ln factor, write to actor register). Two new bits designed explicitly: reads-one-register-writes-another; actor is DEFINED not seeded (compare introduces the register). NO conservation pin (compare creates, doesn't move — transfers conserve, compare doesn't). fraction direction folded in; compare_additive is a later increment. Immediate effect: corridor real-reach 0/500 -> the compare parses that already exist (loop-works proof). Reader grammar family: reader emits compare for ~5/500 (narrow phrasings); incidence ~20% (~100/500) -> broaden compare templates; most of the delta. Measurement: holdout_dev only, wrong=0 floor, sealed untouched, tuned/measured DISJOINT split, delta vs X=15. Risks: 'N times more' ambiguity, garbled entity spans, ordering, defined-register-in-summation. Nothing built; increment PR ships after this plan is ruled on. --- ...ultiplicative-increment-plan-2026-07-18.md | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 docs/research/compare-multiplicative-increment-plan-2026-07-18.md diff --git a/docs/research/compare-multiplicative-increment-plan-2026-07-18.md b/docs/research/compare-multiplicative-increment-plan-2026-07-18.md new file mode 100644 index 00000000..3ee4c129 --- /dev/null +++ b/docs/research/compare-multiplicative-increment-plan-2026-07-18.md @@ -0,0 +1,103 @@ +# Increment Plan — `compare_multiplicative` (reader arc, increment 1) + +**Status**: PLAN — design-first, for ruling before any build +**Date**: 2026-07-18 +**Base**: `forgejo/main @ dd1e4670` (dev2 spike + errata merged, #75) +**Branch**: `feat/compare-mult-increment-plan` +**Arc**: reader, design-first — per-family increments in lockstep +(`docs/research/dev2-frontier-measurement-spike-2026-07-18.md` §7). This is increment 1; +`compare_multiplicative` jumps the queue because the reader already emits real parses for it. + +--- + +## 1. The increment (both halves, one PR) + +Per §7.1 the increment lands a **grammar family in Reader A + its matching compiler tier, in +lockstep**, as one smoke-gated PR carrying a measured `holdout_dev` delta, `wrong=0` held, sealed test +untouched. The two halves cannot move separately — proof below: the 5 real compare parses exist today +and die at the compiler (corridor 0/500). + +## 2. Ground truth — the 5 real compare parses (holdout_dev) + +Every one has the same shape: **one entity seeded, the other *defined* by the comparison, unknown = +total.** + +| id | shape | gold | +| :--- | :--- | :--- | +| 0101 | Eduardo=3; Frankie = 2×Eduardo; total | 9 | +| 0108 | Newport=22; Dana = 4×Newport; total | 110 | +| 0411 | female=13; male = 3×female; total | 52 | +| 0453 | dad=7; Olaf = 3×dad; total | 28 | +| 0268 | Bruce=4; Michael = 3×Bruce; total | 16 | + +`compare_multiplicative` op carries `Comparison(reference_actor, factor, direction ∈ {times,fraction})`: +**`actor_quantity = factor × reference_actor_quantity`** (ADR-0123). All 5 have `unknown.entity = None` +→ the existing **certified summation turn** (ADR-0250 2b) closes them once the compare op is solvable. + +## 3. Compiler tier — `compare_multiplicative` (evals/multi_register_program.py) + +A **cross-register affine definition**: `actor := factor × reference`. Reuses the quantity kernel and +the multi-register framework; two things are new and must be designed explicitly: + +- **Reads one register, writes another.** Today's per-register ops read+write the same register. + Compile the op to: transport the **reference register's field STATE** by a dilator + (`dilate(ref_state, −ln factor)`), relax to it, and write the result into the **actor** register. + Anti-hollow: transport the reference *state*, never a decoded value. +- **The actor is *defined*, not seeded.** In all 5, the actor has no `initial_state` possession — the + comparison introduces the register. The executor must admit a register first written by a compare op + (today it seeds only from `initial_state`). +- **No conservation pin.** A transfer *moves* quantity (Σ conserved, hard-reject); a comparison + *creates* a related quantity (Σ is not conserved by design). The conservation pin applies to + transfers only; compare ops are exempt — stated, not silently skipped. +- `direction='fraction'` (factor < 1, "half as many as") is the same dilator with `factor < 1` — folded + in. `compare_additive` (`more`/`fewer`, a translator) is a **later** increment (frequency order §7.2). + +This is the whole compiler half: a new op branch that dilates a reference register into an actor +register, plus admitting compare-defined registers. The certified record chain and the existing +summation turn carry the rest unchanged. Expected immediate effect: **corridor real-reach 0/500 → the +compare parses that already exist** — the §7.2 "loop works end-to-end on official data" proof. + +## 4. Reader grammar family — broaden `compare_multiplicative` parsing + +Today the reader emits `compare_multiplicative` for ~5/500 — narrow phrasings only ("double what", +"four times the number of", "three times as many/more"). Frontier incidence (spike §3) puts +`compare_multiplicative` markers in **~20% of official problems (~100/500)**, so the reader captures +~5% of the compare surface. The grammar family broadens the compare templates (e.g. "twice as … as", +"N times as much/many as", "N times the number/amount of", "half/a third as many as") — measured by how +many more official problems parse to a correct compare graph. Reader work lives in the +`generate/math_candidate_*` parser; this half is where most of the coverage delta comes from. + +**Honest scope:** this increment solves problems whose only frontier kind is `compare_multiplicative` +(plus basic ops + summation). Multi-frontier problems (spike §3: 16/100 carry ≥2 markers) wait for their +other families' increments — recorded, not counted here. + +## 5. Measurement protocol (§7.5 constraints, verbatim) + +- **Signal = official `holdout_dev` only** — never the authored corpus. +- **`wrong=0` is the floor** — a compare parse counts only if `compile → execute → gold` agrees. +- **Sealed test (1,319) untouched.** +- **Disjoint split:** the reader grammar is tuned against one `holdout_dev` sub-split and the increment's + delta is measured on a **held-out disjoint** sub-split — "better" can only mean "generalizes." +- **Delta reported** = new-correct on the measured split; the increment must clear **X = 15** (spike + §7.3, pending Shay's ruling) or it triggers a practice-lane ruling. Compare incidence (~20%) means a + paying reader family should convert well above 15. + +## 6. Risks / open decisions + +1. **"N times more than"** is linguistically ambiguous (N× vs (N+1)×); the reader currently reads it as + N× and gold agreed on the sampled 5, but the grammar family must pin one reading and measure, not + assume. +2. **Garbled entity spans** (0268's actor is a run-on string) — cosmetic today (answer still correct), + but the reader family should tighten entity extraction; a wrong *entity binding* could mis-route a + compare in a 3-entity problem. +3. **Compare-then-op ordering** — if a defined register is later transferred/scaled, the compare must + resolve first; the executor already runs ops in story order, so this is a test to pin, not new code. +4. **Register-introduced-by-compare** interacts with the total summation (which sums all registers) — + confirm the defined register is included in the sum (it must be). + +## 7. Increment PR (what actually ships, after this plan is ruled on) + +One smoke-gated PR: the compiler tier (§3) + the reader grammar family (§4) + tests +(the 5 real parses solved end-to-end wrong=0; compare-defines-register; no-conservation-on-compare; +fraction direction; the disjoint-split holdout delta) + the measured `holdout_dev` delta recorded +against X. Nothing else in it. From 09172da20478e6ec4914deec0ce9507ebb4f0aee Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 18 Jul 2026 16:07:41 -0700 Subject: [PATCH 2/2] docs(reader-arc): fold PR #76 amendments into compare increment plan; X=15 ratified MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four amendments (Shay, #76 review) folded: 1. Summation registers-driven (NEW CODE): summation orders by program.seeds (initial_state) today, so a compare-DEFINED register (no seed) is silently excluded from the total — all 5 real cases are totals. Fix: sum all registers at solve time, deterministic order (seed order then definition order); compile-side validation admits defined entities as answer targets with unit propagation from the reference. Pin: defined register in the certified sum. 2. Compare record spec: record identifies the reference entity + binds the reference state digest via the existing conditional operand_source_digest (non-compare/non-summation digests unchanged); deterministic re-verification reconstructs the dilation's source register from records alone. 3. Tune/measure split pinned deterministically (sha256(id)-ordered) FIRST, before any grammar work, documented + fixed in advance. 4. 'N times more than' pinned to N× as a disclosed convention; gold mismatches tracked as recorded findings, never silent wrongs. X = 15 ratified as the arc escalation threshold (spike §7.3 updated). §6 risks reclassified (risk 4 -> new code §3.1; risk 1 -> pinned convention §4). Build proceeds under this ruling — no separate re-ruling. --- ...ultiplicative-increment-plan-2026-07-18.md | 78 +++++++++++++------ ...2-frontier-measurement-spike-2026-07-18.md | 6 +- 2 files changed, 59 insertions(+), 25 deletions(-) diff --git a/docs/research/compare-multiplicative-increment-plan-2026-07-18.md b/docs/research/compare-multiplicative-increment-plan-2026-07-18.md index 3ee4c129..262d47b8 100644 --- a/docs/research/compare-multiplicative-increment-plan-2026-07-18.md +++ b/docs/research/compare-multiplicative-increment-plan-2026-07-18.md @@ -1,6 +1,7 @@ # Increment Plan — `compare_multiplicative` (reader arc, increment 1) -**Status**: PLAN — design-first, for ruling before any build +**Status**: APPROVED with four amendments (PR #76 ruling, folded below); **X = 15 ratified** as the +arc's escalation threshold. Build proceeds under this ruling — no separate re-ruling. **Date**: 2026-07-18 **Base**: `forgejo/main @ dd1e4670` (dev2 spike + errata merged, #75) **Branch**: `feat/compare-mult-increment-plan` @@ -52,10 +53,26 @@ the multi-register framework; two things are new and must be designed explicitly - `direction='fraction'` (factor < 1, "half as many as") is the same dilator with `factor < 1` — folded in. `compare_additive` (`more`/`fewer`, a translator) is a **later** increment (frequency order §7.2). -This is the whole compiler half: a new op branch that dilates a reference register into an actor -register, plus admitting compare-defined registers. The certified record chain and the existing -summation turn carry the rest unchanged. Expected immediate effect: **corridor real-reach 0/500 → the -compare parses that already exist** — the §7.2 "loop works end-to-end on official data" proof. +Expected immediate effect: **corridor real-reach 0/500 → the compare parses that already exist** — the +§7.2 "loop works end-to-end on official data" proof. Two amendments (PR #76) make this correct rather +than "unchanged": + +### 3.1 Summation becomes registers-driven (AMENDMENT 1 — new code, not a confirmation) +The certified summation turn today orders by `program.seeds`, which come from `initial_state`. A +compare-**defined** register has **no seed**, so as currently coded it would be **silently excluded +from the total** — and all 5 real cases are totals. Fix: the summation sums **all registers present at +solve time**, in a **deterministically pinned order** (seed order, then definition order). Compile-side +validation must admit **defined entities as answer targets** (a total or a concrete unknown may be a +compare-defined register), with **unit propagation from the reference** (the defined register inherits +the reference's unit). Pin test: a compare-defined register appears in the certified sum. + +### 3.2 The compare record (AMENDMENT 2 — specify before build) +Cross-register provenance must live in the chain so deterministic re-verification can reconstruct +**which register fed the dilation from records alone** (same reasoning as the summation-ruling). The +`compare_multiplicative` record: identifies the **reference entity**, and binds the **reference state's +digest** via the existing conditional `operand_source_digest` (present only on cross-register records, +so non-compare/non-summation record digests stay byte-identical). Verification re-derives the source +register from the record and checks the dilation reproduces. ## 4. Reader grammar family — broaden `compare_multiplicative` parsing @@ -67,6 +84,12 @@ Today the reader emits `compare_multiplicative` for ~5/500 — narrow phrasings many more official problems parse to a correct compare graph. Reader work lives in the `generate/math_candidate_*` parser; this half is where most of the coverage delta comes from. +**"N times more than" — pinned convention (AMENDMENT 4).** The grammar family reads "N times more +than X" as **N×X** (not (N+1)×X), fixed as an **explicit disclosed convention** — the ambiguity is +real, so the reading is a stated choice, not an assumption. Every gold mismatch attributable to this +reading is **tracked as a recorded finding** (never a silent wrong); if the corpus systematically means +(N+1)×, the record will show it and the convention flips on evidence. + **Honest scope:** this increment solves problems whose only frontier kind is `compare_multiplicative` (plus basic ops + summation). Multi-frontier problems (spike §3: 16/100 carry ≥2 markers) wait for their other families' increments — recorded, not counted here. @@ -76,28 +99,39 @@ other families' increments — recorded, not counted here. - **Signal = official `holdout_dev` only** — never the authored corpus. - **`wrong=0` is the floor** — a compare parse counts only if `compile → execute → gold` agrees. - **Sealed test (1,319) untouched.** -- **Disjoint split:** the reader grammar is tuned against one `holdout_dev` sub-split and the increment's - delta is measured on a **held-out disjoint** sub-split — "better" can only mean "generalizes." -- **Delta reported** = new-correct on the measured split; the increment must clear **X = 15** (spike - §7.3, pending Shay's ruling) or it triggers a practice-lane ruling. Compare incidence (~20%) means a - paying reader family should convert well above 15. +- **Disjoint split, pinned deterministically FIRST (AMENDMENT 3).** Before any grammar work, the 500 + `holdout_dev` cases are partitioned by a **deterministic protocol** — `sha256(id)`-ordered assignment + (e.g. even/odd rank → tune / measure), documented in the plan and **fixed in advance**. The reader + grammar is tuned only against the *tune* split; the increment's delta is measured only on the disjoint + *measure* split. Disjointness is only as strong as its enforcement, so it is committed before a line of + grammar changes — "better" can only mean "generalizes." +- **Delta reported** = new-correct on the measured split; the increment must clear **X = 15** (ratified, + spike §7.3) or it triggers a practice-lane ruling. Compare incidence (~20%) means a paying reader + family should convert well above 15. -## 6. Risks / open decisions +## 6. Risks / open decisions (post-amendment) -1. **"N times more than"** is linguistically ambiguous (N× vs (N+1)×); the reader currently reads it as - N× and gold agreed on the sampled 5, but the grammar family must pin one reading and measure, not - assume. +1. **"N times more than"** — resolved to a pinned N× convention with tracked mismatches (§4, AMENDMENT 4). 2. **Garbled entity spans** (0268's actor is a run-on string) — cosmetic today (answer still correct), but the reader family should tighten entity extraction; a wrong *entity binding* could mis-route a - compare in a 3-entity problem. + compare in a 3-entity problem. Remaining open risk. 3. **Compare-then-op ordering** — if a defined register is later transferred/scaled, the compare must resolve first; the executor already runs ops in story order, so this is a test to pin, not new code. -4. **Register-introduced-by-compare** interacts with the total summation (which sums all registers) — - confirm the defined register is included in the sum (it must be). +4. **Register-introduced-by-compare in the total** — reclassified: this is **new code**, not a + confirmation (§3.1, AMENDMENT 1). Summation is registers-driven with a pinned order; a defined + register that is the answer target propagates its unit from the reference. -## 7. Increment PR (what actually ships, after this plan is ruled on) +## 7. Increment PR (what ships next, under this ruling) -One smoke-gated PR: the compiler tier (§3) + the reader grammar family (§4) + tests -(the 5 real parses solved end-to-end wrong=0; compare-defines-register; no-conservation-on-compare; -fraction direction; the disjoint-split holdout delta) + the measured `holdout_dev` delta recorded -against X. Nothing else in it. +One smoke-gated PR: the compiler tier (§3, including §3.1 registers-driven summation + §3.2 compare +record) + the reader grammar family (§4) + tests + the measured delta. Test set: +- the 5 real parses solved end-to-end, `wrong=0`; +- **compare-defined register appears in the certified sum** (§3.1 pin); +- defined-entity answer target with unit propagated from the reference (§3.1); +- **compare record re-verification** — source register reconstructed from records alone (§3.2); +- no-conservation-on-compare; `fraction` direction; +- N× convention pinned, mismatches surfaced as findings (§4); +- the deterministic tune/measure split fixed before grammar work (§5, AMENDMENT 3); +- the measured `holdout_dev` **measure-split** delta recorded against **X = 15**. + +Nothing else in it. Build proceeds now under this ruling. diff --git a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md index 998826bf..1305eb07 100644 --- a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md +++ b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md @@ -139,9 +139,9 @@ the `wrong=0` floor held, and the sealed test untouched. official data. 2. Then strict §3 frequency order: **rate (31) → fraction (28) → compare_additive (9) → partition (1)**. -### 7.3 Escalation threshold X — PROPOSED (Shay rules with the merge) -Fixed now, before anyone is attached, so "switch when design stops working" cannot drift. -**Proposed X = 15 new-correct on `holdout_dev` per full family increment.** When a completed increment +### 7.3 Escalation threshold X — **RATIFIED (Shay, 2026-07-18, PR #76): X = 15** +Fixed before anyone is attached, so "switch when design stops working" cannot drift. +**X = 15 new-correct on `holdout_dev` per full family increment.** When a completed increment buys fewer than X, escalate to a practice-lane ruling. Rationale: these families carry 20–31% incidence (~100–155/500 problems each); a design that is paying converts far more than 15, so falling below 15/500 (3%) means design has stopped efficiently converting known incidence — a number trips the