From ab5c663eb321ca9ca2923cc1bc7aef833489cb85 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 18 Jul 2026 15:24:18 -0700 Subject: [PATCH 1/3] =?UTF-8?q?docs(research):=20dev-holdout-2=20/=20front?= =?UTF-8?q?ier-measurement=20spike=20=E2=80=94=20verdict?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Headline correction: evals/gsm8k_math/dev + public are CORE-AUTHORED GSM8K-style (ADR-0119.2), NOT real GSM8K. The compiler's 200/200 is on authored, mostly depth-1, frontier-free problems — a mechanism validation, not real-GSM8K capability. ADR-0249/0250 'real GSM8K' language needs correcting. Draw-protocol trace: train_sample + holdout_dev + sealed-test = real GSM8K (deterministic seeded draws); dev/public = authored. Real measurement (end-to-end on 500 real holdout_dev): Reader A parses 5/500 (1%, refuses 495); my compiler solves 0/500; the 5 parsed are all compare_multiplicative (frontier), refused by the affine compiler. Corroborates holdout_dev's historical 'real GSM8K capability 0%'. Verdict: (a) DOMINANT — frontier + real GSM8K are READER-gated (99% parse-refusal); (b) confirmed as authoring bias in dev/public; (c) refuted — frontier is common, not rare. Sequencing: reader is the biggest bucket; more compiler tiers = capability without inputs; serve-landing premature. Reader-practice-loop NAMED as leading next-arc (reader = first learnable component; attempt-parse->compile->execute-> gold-check, wrong=0-gated on held-out real data; generalization-preserving). Public-seal approved with authored-not-real + composition (depth/kinds) disclosure. Spike verdict for ruling; nothing built. --- ...2-frontier-measurement-spike-2026-07-18.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 docs/research/dev2-frontier-measurement-spike-2026-07-18.md diff --git a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md new file mode 100644 index 00000000..b500a992 --- /dev/null +++ b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md @@ -0,0 +1,117 @@ +# dev-holdout-2 / Frontier-Measurement Spike — verdict + +**Status**: SPIKE VERDICT — for ruling before any build +**Date**: 2026-07-18 +**Base**: `forgejo/main @ b87cd46c` (ADR-0249 + ADR-0250 Accepted; compiler solves the authored dev/public corpus) +**Branch**: `feat/dev2-frontier-spike` +**Charge (Shay)**: public-seal + floors; the Reader A answer; the raw-incidence measurement with an +(a)/(b)/(c) verdict; the draw-protocol trace — before building anything. + +--- + +## 0. Headline correction (load-bearing — read first) + +**`evals/gsm8k_math/dev` (50) and `public` (150) are NOT real GSM8K.** ADR-0119.2 authors them as +200 **CORE-original, GSM8K-*style*** problems written in CORE's own vocabulary/grammar to a chosen +operation distribution (case notes: "Generated public case exercising divide operations"). My +ADR-0249/0250 evidence called these "the real GSM8K dev holdout" — that is inaccurate and should be +corrected to "CORE-authored GSM8K-style." The compiler's verified result is **200/200 wrong=0 on +authored, grammar-matched, mostly depth-1 problems** — a real validation of the *mechanism*, not of +real-GSM8K capability. + +## 1. Draw-protocol trace (Shay's ask: was the draw annotability-conditioned?) + +| Slice | Real GSM8K? | How drawn | +| :--- | :--- | :--- | +| `train_sample` (50) | **Real** | `openai/gsm8k` main/train, rev `740312a…`; deterministic `sha256(f"{i}:SALT")` rank, top 50. What the grammar was hand-built against (overfit). | +| `holdout_dev/v1` (500) | **Real** | train split minus train_sample, sorted by `sha256(question)`, first 500. "Real GSM8K CORE was NOT built on." | +| sealed test (1,319) | **Real** | GSM8K test split, sealed (`.age`). | +| **`dev` (50), `public` (150)** | **Authored** | **CORE-authored** (ADR-0119.2), CORE grammar, chosen op/depth distribution. | + +So (b) is not subtle "selection bias in a real draw" — it is **authoring**: dev/public are synthetic +and their frontier-freeness is an authoring choice, carrying zero information about real GSM8K's +distribution. The real draws (train_sample, holdout_dev) are unbiased and were the right place to +measure all along. + +## 2. The real measurement (end-to-end on 500 real holdout_dev cases) + +Pipeline: real text → `parse_and_solve` (Reader A) → my compiler (Tier-1/Tier-2) → execute → gold. + +| Metric | Value | +| :--- | :--- | +| Reader A parses a graph | **5 / 500 (1%)** — refuses **495** | +| Reader A own solver correct | 5 / 500 (1%) | +| **My compiler solved** | **0 / 500** | +| …compiler wrong | 0 (fail-closed held) | +| The 5 parsed graphs' op kinds | **all `compare_multiplicative`** (a frontier kind) | + +Corroborates the holdout_dev README's 2026-06-04 baseline ("real GSM8K capability 0%"). The compiler +solves 0 real cases — not because it is wrong, but because the reader feeds it only 5 graphs and those +5 are a frontier kind the affine compiler refuses. + +## 3. The (a)/(b)/(c) verdict + +- **(a) — DOMINANT. The frontier (and real GSM8K generally) is reader-gated.** Reader A refuses + 495/500 real problems at parse. The binding constraint is not the compiler's affine scope; it is + that real GSM8K text does not become graphs at all. This is a *mechanism* gap in the reader. +- **(b) — CONFIRMED for dev/public, but as authoring, not draw-filtering.** The frontier-free authored + corpus tells us nothing about real GSM8K. Every "real GSM8K" coverage claim on dev/public needs the + denominator correction to "authored corpus." +- **(c) — REFUTED as the binding issue.** Frontier kinds are not rare in real GSM8K: the *very first* + problems that parse are `compare_multiplicative`, and real problems visibly use comparison / rate / + fraction language ("twice as many as", "half of"). The frontier is common; the reader just can't + reach it. + +## 4. Public-seal — approved, with the disclosure sharpened + +Sealing/pinning `public` (150) buys a **narrow authored regression floor**: 150 cases, **1 op per +case (depth-1)**, **two kinds only** (divide 53 + transfer 97), all CORE-authored. Pin dev-1 (50/50) +and public (150/150) as regression floors, but the coverage record MUST carry, per slice: +- an **op-kind histogram** and **ops-per-case depth**, and +- the **authored-not-real** provenance flag. + +Honest headline: **"200/200 wrong=0 on CORE-authored GSM8K-style (depth/kinds disclosed); real-GSM8K +compiler coverage = 0/500 (reader-gated)."** Anything shorter inflates. + +## 5. Sequencing verdict (three branches, resolved by the data) + +- big coherent bucket → build that tier; +- long-tail fragmentation → serve landing (§3.7 + scoped `generate/`); +- **frontier-rare-in-raw-GSM8K → serve landing** — **does NOT apply**: the frontier is common, not rare. + +The data selects a branch the skeleton didn't enumerate: **the biggest bucket is the reader itself** +(495/500). Building more compiler tiers is capability with no inputs; serve-landing is premature +(there is ~nothing real to serve). The decidable next move is to **make the reader parse real GSM8K**. + +## 6. Reader-practice-loop — named, not started (per the charge) + +The Reader-A answer is **no** — it cannot turn real GSM8K text into graphs at scale — so by the +standing agreement this moves the reader-practice-loop from "eventually" to the **leading next-arc +candidate**. The reader is the first genuinely *learnable* component in the stack: `attempt-parse → +compile → execute → gold-check` is a practice loop with a **decidable, wrong=0-gated reward** on a +held-out real set (`holdout_dev`, never the authored corpus). Crucially, this is the +**generalization-preserving** way to grow real capability — it optimizes parsing *coverage* against +held-out real data, the opposite of authoring more grammar-matched problems (which is the overfit +trap). It is named here for the post-spike ruling; not started. + +## 7. The generalization reframe (Shay, live) + +"We don't want overfitting; we want generalization" is right, and the compiler honors it: it is a +**general mechanism** (versor transport + chained certified relaxation), not memorized answers — +which is why it holds wrong=0 across authored problems it never saw. But the measurement forces a +distinction: **mechanism-generality ≠ real-problem coverage.** The mechanism is general yet starved — +1% reader coverage, and its affine scope misses the compare/rate/fraction the little real data uses. +"Generalized problem-solving on real inputs" (the original done-when) is therefore **not yet shown**; +what is shown is a general composition substrate validated on clean inputs. The reader-practice-loop +is exactly how that value (generalize, don't overfit) becomes real-problem coverage. + +## 8. For the ruling (three arcs on the table, none started) + +1. **Reader-practice-loop** — grow real-GSM8K parse coverage off 1%, wrong=0-gated on held-out real + data. Strongest by the measurement. +2. **More compiler tiers** (compare/rate/fraction) — de-prioritized: the reader feeds them almost + nothing; capability without inputs. +3. **Serve-landing** (§3.7 + `generate/`) — premature until there is real capability to serve. + +Recommendation: reader-practice-loop. Plus the cheap housekeeping: pin the two authored floors +(composition-disclosed) and correct the "real GSM8K" language in the ADR-0249/0250 evidence. From 639616b3e50851b9616a271ffbb3bd5559096426 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 18 Jul 2026 15:37:01 -0700 Subject: [PATCH 2/3] docs(dev2-spike): errata + two-instrument reframe + incidence + reader-arc design Folds in Shay's rulings on the spike: - ERRATA sections added to ADR-0249, ADR-0250, and both acceptance-evidence docs (statuses stay Accepted): corpus label 'real GSM8K dev holdout' -> 'CORE-authored GSM8K-style corpus (ADR-0119.2)'. Mechanism claims (wrong=0, PARITY, conservation, atomicity, chain-of-custody, 200/200) all stand; reframe not retraction. Covers the #74 review comment that repeated the label. (Also fixed ADR-0250 evidence status -> RATIFIED, which the ratification housekeeping missed.) - Two instruments named + never-blended scoreboard: authored = mechanism-correctness (200/200 stands as that); official holdout = generalization (5/500 symbolic / 0/500 corridor). public/dev-1 floors relabeled MECHANISM floors, composition-disclosed. - Censored-incidence fix: parse-INDEPENDENT tagging of 100 official cases (documented heuristic). 70/100 carry frontier markers (rate 31, fraction 28, compare_mult 20) -> (c) refuted, frontier COMMON not rare; the table is the reader-arc curriculum by real-world frequency. - Reader-practice-loop design with the four non-negotiable constraints (real held-out data only; wrong=0 gates consolidation; sealed test untouched; holdout_dev split disjoint from the measured metric). Sequencing: capability->practice->calibration-> serve; reader is the critical path for all three; serve-landing de-prioritized (0/500). Spike verdict for ruling; nothing built. --- ...miltonian-compiler-composition-frontier.md | 18 +++ .../ADR-0250-tier2-multi-entity-arithmetic.md | 16 ++ docs/handoff/ADR-0249-Acceptance-Evidence.md | 5 + docs/handoff/ADR-0250-Acceptance-Evidence.md | 7 +- ...2-frontier-measurement-spike-2026-07-18.md | 150 ++++++++++++------ 5 files changed, 143 insertions(+), 53 deletions(-) diff --git a/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md b/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md index 68731a32..e62478cf 100644 --- a/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md +++ b/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md @@ -9,6 +9,24 @@ --- +## ERRATA (2026-07-18) — corpus mislabel corrected; status stays Accepted + +This ADR and its evidence repeatedly call `evals/gsm8k_math/dev` the **"real GSM8K dev holdout."** +That label is wrong: per ADR-0119.2, `dev` (and `public`) are **CORE-authored, GSM8K-*style*** problems +in CORE's own grammar — **not** real GSM8K. Real GSM8K lives in `train_sample`, `holdout_dev/v1`, and +the sealed test. Wherever this ADR says "real GSM8K dev holdout," read **"CORE-authored GSM8K-style +corpus (ADR-0119.2)."** + +**Nothing about the mechanism is retracted.** Certified relaxation, conservation, atomicity, +chain-of-custody, `wrong=0`, and PARITY against a symbolic fold of the *same* compiled program all +stand — the authored corpus is the **mechanism-correctness instrument**, and the compiler's 200/200 on +it is exactly that evidence. What falls is only the corpus label. Measured real-GSM8K reach (spike +`docs/research/dev2-frontier-measurement-spike-2026-07-18.md`): reader parses 5/500, corridor solves +0/500 — reader-gated. This erratum records that we caught our own mislabel via measurement; that is the +honest-measurement doctrine working, not a defect hidden. + +--- + ## 1. What this ADR is — and the thesis The generalized-lift instrument (ADR-0246 seam S4) recorded a single blocking diff --git a/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md b/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md index d4131c17..e1db0c8c 100644 --- a/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md +++ b/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md @@ -9,6 +9,22 @@ --- +## ERRATA (2026-07-18) — corpus mislabel corrected; status stays Accepted + +This ADR calls the authored corpus the **"entire real GSM8K dev holdout."** That label is wrong: per +ADR-0119.2, `dev` (50) and `public` (150) are **CORE-authored, GSM8K-*style*** problems, **not** real +GSM8K. Read every "real GSM8K" reference here as **"CORE-authored GSM8K-style corpus (ADR-0119.2)."** + +**Nothing about the Tier-2 mechanism is retracted** — the multi-register model, coupled-translator +transfers, the relative conservation hard-reject, prepare→validate→commit atomicity, the certified +summation turn, and the chain-of-custody all stand; the authored corpus is the mechanism-correctness +instrument and the 50/50 (200/200 across both ADRs) is exactly that evidence. Only the corpus label +falls. Measured real-GSM8K reach (spike `docs/research/dev2-frontier-measurement-spike-2026-07-18.md`): +reader parses 5/500, corridor 0/500 — reader-gated. Recorded here as the honest-measurement doctrine +catching its own mislabel. + +--- + ## 1. What this ADR is ADR-0249 closed Tier-1 (single-accumulator affine arithmetic) and recorded the 24 multi-entity diff --git a/docs/handoff/ADR-0249-Acceptance-Evidence.md b/docs/handoff/ADR-0249-Acceptance-Evidence.md index fbbbb762..bfdeba34 100644 --- a/docs/handoff/ADR-0249-Acceptance-Evidence.md +++ b/docs/handoff/ADR-0249-Acceptance-Evidence.md @@ -1,5 +1,10 @@ # ADR-0249 — Acceptance Evidence +> **ERRATA (2026-07-18):** "real GSM8K dev holdout" in this doc = **CORE-authored GSM8K-style corpus +> (ADR-0119.2)**, the mechanism-correctness instrument — not real GSM8K. Mechanism results (wrong=0, +> PARITY, 200/200) stand; only the corpus label is corrected. Measured real-GSM8K reach: reader 5/500, +> corridor 0/500 (reader-gated). See ADR-0249 §ERRATA + `docs/research/dev2-frontier-measurement-spike-2026-07-18.md`. + **Status**: RATIFIED — ADR-0249 Accepted 2026-07-18 by Joshua Shay for commit `220039f7` **Date**: 2026-07-18 **ADR**: `docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md` diff --git a/docs/handoff/ADR-0250-Acceptance-Evidence.md b/docs/handoff/ADR-0250-Acceptance-Evidence.md index 15a17d2a..bd69d9af 100644 --- a/docs/handoff/ADR-0250-Acceptance-Evidence.md +++ b/docs/handoff/ADR-0250-Acceptance-Evidence.md @@ -1,6 +1,11 @@ # ADR-0250 — Acceptance Evidence -**Status**: Evidence pack for Shay's ruling (no self-Accept) +> **ERRATA (2026-07-18):** "real GSM8K dev holdout" in this doc = **CORE-authored GSM8K-style corpus +> (ADR-0119.2)**, the mechanism-correctness instrument — not real GSM8K. Mechanism results (wrong=0, +> full-holdout 50/50) stand; only the corpus label is corrected. Measured real-GSM8K reach: reader +> 5/500, corridor 0/500 (reader-gated). See ADR-0250 §ERRATA + `docs/research/dev2-frontier-measurement-spike-2026-07-18.md`. + +**Status**: RATIFIED — ADR-0250 Accepted 2026-07-18 by Joshua Shay **Date**: 2026-07-18 **ADR**: `docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md` **Design record**: `docs/research/tier2-multi-entity-arithmetic-spike-2026-07-18.md` diff --git a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md index b500a992..c0c81395 100644 --- a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md +++ b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md @@ -49,69 +49,115 @@ Corroborates the holdout_dev README's 2026-06-04 baseline ("real GSM8K capabilit solves 0 real cases — not because it is wrong, but because the reader feeds it only 5 graphs and those 5 are a frontier kind the affine compiler refuses. -## 3. The (a)/(b)/(c) verdict +## 3. Censored-incidence fix — the parse-independent measurement -- **(a) — DOMINANT. The frontier (and real GSM8K generally) is reader-gated.** Reader A refuses - 495/500 real problems at parse. The binding constraint is not the compiler's affine scope; it is - that real GSM8K text does not become graphs at all. This is a *mechanism* gap in the reader. -- **(b) — CONFIRMED for dev/public, but as authoring, not draw-filtering.** The frontier-free authored - corpus tells us nothing about real GSM8K. Every "real GSM8K" coverage claim on dev/public needs the - denominator correction to "authored corpus." -- **(c) — REFUTED as the binding issue.** Frontier kinds are not rare in real GSM8K: the *very first* - problems that parse are `compare_multiplicative`, and real problems visibly use comparison / rate / - fraction language ("twice as many as", "half of"). The frontier is common; the reader just can't - reach it. +The 500-run's op-kind incidence (§2) is **censored data**: it reads kinds only from the 5 graphs the +reader produced, so it says nothing about what the 495 refusals *need*. To answer (b)-vs-(c) honestly, +frontier incidence must be measured **around** the refusing reader. Protocol: the first **100** of the +sha256-ordered holdout_dev (deterministic), classified by **keyword/pattern markers in the raw text** +— parse-independent, multi-label, imperfect (a marker need not mean the op is required; over/under-count +possible), but the *scale* is robust to that noise. -## 4. Public-seal — approved, with the disclosure sharpened +| Frontier kind (marker) | Incidence /100 | +| :--- | :--- | +| `apply_rate` ("per", "each … costs", "every …") | **31** | +| `fraction_portion` ("half", "third", "%", "a/b") | **28** | +| `compare_multiplicative` ("twice", "N times as many") | **20** | +| `compare_additive` ("more/fewer than") | 9 | +| `unit_partition` ("split into", "divided among") | 1 | +| **basic-only (no frontier marker)** | **30 / 100** | +| **≥1 frontier marker** | **70 / 100** | -Sealing/pinning `public` (150) buys a **narrow authored regression floor**: 150 cases, **1 op per -case (depth-1)**, **two kinds only** (divide 53 + transfer 97), all CORE-authored. Pin dev-1 (50/50) -and public (150/150) as regression floors, but the coverage record MUST carry, per slice: -- an **op-kind histogram** and **ops-per-case depth**, and -- the **authored-not-real** provenance flag. +## 4. The (a)/(b)/(c) verdict (de-censored) -Honest headline: **"200/200 wrong=0 on CORE-authored GSM8K-style (depth/kinds disclosed); real-GSM8K -compiler coverage = 0/500 (reader-gated)."** Anything shorter inflates. +- **(a) — DOMINANT. Real GSM8K is reader-gated.** Reader A refuses 495/500 at parse; text does not + become graphs. A *mechanism* gap in the reader, not the compiler's affine scope. +- **(b) — CONFIRMED as authoring bias.** dev/public are authored frontier-free; they carry zero + information about real GSM8K's distribution. Every "real GSM8K" claim on them needs the correction. +- **(c) — REFUTED, now with parse-independent evidence.** The frontier is **common, not rare**: + **70/100** official problems carry frontier markers; rate + fraction alone appear in ~55%. Widening + off-serving capability against this frontier does NOT fail the decidable-done-when test — the reader + reaching it does. -## 5. Sequencing verdict (three branches, resolved by the data) +**Bonus: this table is the reader arc's curriculum, ordered by real-world frequency** — rate (31) → +fraction (28) → compare_multiplicative (20) → compare_additive (9) → partition (1). Practice earns the +most real coverage by attacking rate/fraction/comparison first. -- big coherent bucket → build that tier; -- long-tail fragmentation → serve landing (§3.7 + scoped `generate/`); -- **frontier-rare-in-raw-GSM8K → serve landing** — **does NOT apply**: the frontier is common, not rare. +## 5. Two instruments, cleanly named, both kept (ruling) -The data selects a branch the skeleton didn't enumerate: **the biggest bucket is the reader itself** -(495/500). Building more compiler tiers is capability with no inputs; serve-landing is premature -(there is ~nothing real to serve). The decidable next move is to **make the reader parse real GSM8K**. +The finding is a **reframe, not a retraction** — nothing measured is thrown away, it is relabeled. -## 6. Reader-practice-loop — named, not started (per the charge) +- **Authored corpus = the mechanism-correctness instrument.** A controlled curriculum that proved the + compiler end-to-end. Its **200/200 wrong=0 stands, as exactly that** — evidence the mechanism + (versor transport + chained certified relaxation + conservation + atomicity + chain-of-custody) is + correct on clean inputs. +- **Official holdout = the generalization instrument.** `holdout_dev` (500), sealed test (1,319) — + real GSM8K, never authored. -The Reader-A answer is **no** — it cannot turn real GSM8K text into graphs at scale — so by the -standing agreement this moves the reader-practice-loop from "eventually" to the **leading next-arc -candidate**. The reader is the first genuinely *learnable* component in the stack: `attempt-parse → -compile → execute → gold-check` is a practice loop with a **decidable, wrong=0-gated reward** on a -held-out real set (`holdout_dev`, never the authored corpus). Crucially, this is the -**generalization-preserving** way to grow real capability — it optimizes parsing *coverage* against -held-out real data, the opposite of authoring more grammar-matched problems (which is the overfit -trap). It is named here for the post-spike ruling; not started. +**The scoreboard, never blended:** -## 7. The generalization reframe (Shay, live) +| Instrument | Slice | Score | Reads as | +| :--- | :--- | :--- | :--- | +| Mechanism-correctness | authored (200) | **200/200 wrong=0** | the compiler is correct | +| Generalization | official holdout_dev (500) | **5/500 Reader-A-symbolic · 0/500 corridor** | real reach is ~0 | -"We don't want overfitting; we want generalization" is right, and the compiler honors it: it is a -**general mechanism** (versor transport + chained certified relaxation), not memorized answers — -which is why it holds wrong=0 across authored problems it never saw. But the measurement forces a -distinction: **mechanism-generality ≠ real-problem coverage.** The mechanism is general yet starved — -1% reader coverage, and its affine scope misses the compare/rate/fraction the little real data uses. -"Generalized problem-solving on real inputs" (the original done-when) is therefore **not yet shown**; -what is shown is a general composition substrate validated on clean inputs. The reader-practice-loop -is exactly how that value (generalize, don't overfit) becomes real-problem coverage. +Seal `public` and pin both floors (dev-1 50/50, public 150/150) — but **label them mechanism floors, +not coverage claims**, each carrying an op-kind histogram + ops-per-case depth + the authored-not-real +flag (public: depth-1, divide 53 + transfer 97). A regression floor that guards mechanism correctness; +it says nothing about real-GSM8K coverage, and the record must say so. -## 8. For the ruling (three arcs on the table, none started) +## 6. Sequencing verdict (resolved by the data) -1. **Reader-practice-loop** — grow real-GSM8K parse coverage off 1%, wrong=0-gated on held-out real - data. Strongest by the measurement. -2. **More compiler tiers** (compare/rate/fraction) — de-prioritized: the reader feeds them almost - nothing; capability without inputs. -3. **Serve-landing** (§3.7 + `generate/`) — premature until there is real capability to serve. +The three-branch skeleton doesn't fire: the frontier is common (§3), so "frontier-rare → serve +landing" is out; and the biggest bucket isn't a compiler tier — **it's the reader** (495/500). Stated +plainly: **corridor real end-to-end capability is 0/500, so serve-landing of corridor arithmetic loses +priority — there is nothing real-reaching to serve.** §3.7 calibration stays queued; the **reader +practice volume is what eventually generates §3.7's calibration data.** Order unchanged: +**capability → practice → calibration → serve** — and the reader arc is now the **critical path for all +three**. -Recommendation: reader-practice-loop. Plus the cheap housekeeping: pin the two authored floors -(composition-disclosed) and correct the "real GSM8K" language in the ADR-0249/0250 evidence. +## 7. The next arc — reader practice loop (named for ruling; not started) + +99% parse-refusal makes the trigger a measured fact, not a judgment call. The reader is the first +genuinely *learnable* component: `attempt-parse → compile → execute → gold-check` is a practice loop +with a **decidable, wrong=0-gated reward**. Non-negotiable design constraints for the proposal: + +1. **Practice signal is real held-out official data only** — never the authored corpus. (Training + against authored grammar is exactly how the overfit happened; §3's 200/200-vs-5/500 gap is its + fingerprint.) +2. **`wrong=0` gates what consolidates** — a parse/graph is only kept if execute+gold-check is right, + never merely if it parsed. +3. **The sealed test (1,319) is never touched** — final exam only. +4. **`holdout_dev` splits** so practiced-against cases are **disjoint** from the measured dev metric + — "getting better" can only mean "generalizes to real problems," never "fit the metric set." + +Curriculum order = the §3 real-world frequency: rate → fraction → compare_multiplicative → … +The done-when is a rising `correct` on the disjoint held-out real split **with `wrong=0`** — the honest +metric the holdout lane was built to keep in front of us. + +## 8. Errata proposal — ADR-0249 / ADR-0250 (statuses stay Accepted) + +A prominent **errata section** (not silent rewording) is added to both ADRs and their acceptance +evidence, and to the PR #74 review record: the mechanism claims all survive; the single correction is +the corpus label — **"real GSM8K dev holdout" → "CORE-authored GSM8K-style corpus (ADR-0119.2)."** The +record thus shows we caught our own mislabel, which is the honest-measurement doctrine working, not a +failure to hide. (Errata text lands in this PR alongside the spike.) + +## 9. The generalization reframe (Shay, live) + +"Generalize, don't overfit" is right, and the compiler honors it: it is a general *mechanism*, not +memorized answers. The measurement only forces one distinction — **mechanism-generality ≠ real-problem +coverage.** The mechanism is general but starved: 1% reader reach, affine scope. The reader practice +loop is precisely how that value becomes real coverage without re-introducing the overfit. + +## 10. For the ruling (nothing started) + +1. **Reader-practice-loop** — RECOMMENDED. Critical path for capability → practice → calibration → + serve. Grow real-GSM8K parse coverage off 1%, wrong=0-gated on a disjoint held-out real split; + curriculum ordered by §3 frequency (rate → fraction → compare). +2. **More compiler tiers** (compare/rate/fraction) — de-prioritized: the reader feeds them ~nothing. +3. **Serve-landing** (§3.7 + `generate/`) — premature; corridor real reach is 0/500. + +Delivered in this PR for the ruling: the errata sections (§8), the relabeled seal + two mechanism +floors composition-disclosed (§5), the parse-independent incidence tagging (§3), and the reader-arc +design with the four non-negotiable constraints (§7). **Nothing builds before the ruling on this PR.** From 25762614c02b76eb4e56c313ff4b031f9b342edd Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 18 Jul 2026 15:48:13 -0700 Subject: [PATCH 3/3] =?UTF-8?q?docs(dev2-spike):=20restructure=20per=20rul?= =?UTF-8?q?ing=20=E2=80=94=20reader=20design-first,=20practice=20=3D=20bac?= =?UTF-8?q?kup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Supersedes the practice-loop recommendation (§7/§10): the arc is the reader, DESIGN-FIRST — per-family increments, grammar family + compiler tier in LOCKSTEP (§2 proves they can't move separately: the 5 real compare_multiplicative parses die at the affine compiler today). Each increment = own smoke-gated PR with a measured holdout_dev delta, wrong=0 floor, sealed test untouched. - Increment order: compare_multiplicative FIRST (reader already emits it → 0/500 to >0 immediately, cheapest end-to-end proof), then §3 frequency order (rate → fraction → compare_additive → partition). - Escalation threshold X PROPOSED = 15 new-correct/holdout_dev per full family increment (Shay rules with the merge); below it → practice-lane ruling. Rationale: families carry 20-31% incidence, a paying design converts far more. - Practice loop demoted to PAPER-DESIGN backup (§7.4) with leverage analysis: verifier = Tier-1/2 compiler stack, audit = certificate chains, consolidation = prepare→validate→commit generalized, hygiene = ADR-0119 sealing; genuinely new = candidate generator + consolidation store. Four §7 constraints carry over verbatim. - Design-first work IS the backup's curriculum (§7.6): refusal taxonomies, tagged incidence, labeled (parse,compile,execute,gold) tuples are artifacts. §5 floors/scoreboard + §6 sequencing affirmed unchanged; errata merge with this. --- ...2-frontier-measurement-spike-2026-07-18.md | 88 ++++++++++++++----- 1 file changed, 64 insertions(+), 24 deletions(-) diff --git a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md index c0c81395..998826bf 100644 --- a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md +++ b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md @@ -116,24 +116,61 @@ practice volume is what eventually generates §3.7's calibration data.** Order u **capability → practice → calibration → serve** — and the reader arc is now the **critical path for all three**. -## 7. The next arc — reader practice loop (named for ruling; not started) +## 7. The arc (RULED) — reader, design-first: per-family increments in lockstep -99% parse-refusal makes the trigger a measured fact, not a judgment call. The reader is the first -genuinely *learnable* component: `attempt-parse → compile → execute → gold-check` is a practice loop -with a **decidable, wrong=0-gated reward**. Non-negotiable design constraints for the proposal: +Superseding §10's practice-loop recommendation: the frontier being **common, formulaic, and +frequency-ordered** (§3) argues *for* designed grammar-family extensions, not against — a formulaic +frontier is exactly what designed grammar eats efficiently, each increment cheaply measurable. +**Learning earns its complexity only when design stops paying — decided by number, not feel.** The +practice loop is the paper-designed backup (§7.4), not the arc. -1. **Practice signal is real held-out official data only** — never the authored corpus. (Training - against authored grammar is exactly how the overfit happened; §3's 200/200-vs-5/500 gap is its - fingerprint.) -2. **`wrong=0` gates what consolidates** — a parse/graph is only kept if execute+gold-check is right, - never merely if it parsed. -3. **The sealed test (1,319) is never touched** — final exam only. -4. **`holdout_dev` splits** so practiced-against cases are **disjoint** from the measured dev metric - — "getting better" can only mean "generalizes to real problems," never "fit the metric set." +### 7.1 Increment = a grammar family + its compiler tier, landed together +Reader and compiler move in **lockstep**. §2 is the proof they cannot move separately: the 5 real +`compare_multiplicative` parses **die at the affine compiler today** (reader emits, compiler refuses → +0/500). A reader family with no compiler tier produces graphs nothing solves; a compiler tier with no +reader family has no inputs — that is the *only* thing §10.2 de-prioritizes. **Paired, they are the +arc.** Each increment is its own smoke-gated PR carrying a measured `holdout_dev` delta (new correct), +the `wrong=0` floor held, and the sealed test untouched. -Curriculum order = the §3 real-world frequency: rate → fraction → compare_multiplicative → … -The done-when is a rising `correct` on the disjoint held-out real split **with `wrong=0`** — the honest -metric the holdout lane was built to keep in front of us. +### 7.2 Increment order +1. **`compare_multiplicative` FIRST** — jumps the §3 frequency queue because the reader *already emits + real parses for it* (the 5/500), so pairing a compare compiler tier moves corridor real-reach from + **0/500 → >0 immediately** — the cheapest possible end-to-end proof the increment loop works on + official data. +2. Then strict §3 frequency order: **rate (31) → fraction (28) → compare_additive (9) → partition (1)**. + +### 7.3 Escalation threshold X — PROPOSED (Shay rules with the merge) +Fixed now, before anyone is attached, so "switch when design stops working" cannot drift. +**Proposed X = 15 new-correct on `holdout_dev` per full family increment.** When a completed increment +buys fewer than X, escalate to a practice-lane ruling. Rationale: these families carry 20–31% incidence +(~100–155/500 problems each); a design that is paying converts far more than 15, so falling below +15/500 (3%) means design has stopped efficiently converting known incidence — a number trips the +switch, not a feeling. + +### 7.4 Backup lane — the practice loop (PAPER DESIGN only; starts warm if X trips) +Same decidable loop (`attempt-parse → compile → execute → gold-check`, wrong=0-gated). Its leverage — +how it reuses what these arcs already built, so only two things are genuinely new: +- **Verifier** = the Tier-1/Tier-2 compiler stack: a candidate parse is correct **iff** + `compile → execute → gold` agrees. No separate oracle. +- **Audit** = the certificate chains (`RelaxationCertificate`, `TurnRecord`, chain-of-custody). +- **Consolidation gating** = `prepare → validate → commit` generalized: a candidate consolidates only + on full success. +- **Holdout hygiene** = ADR-0119 sealing (split discipline; sealed test untouched). +- **Genuinely new** = a **candidate generator** (proposes parses/grammar variants) + a **consolidation + store** (what survives wrong=0). That is the whole delta the lane adds. + +### 7.5 Non-negotiable constraints (bind the increments now; carry to the lane verbatim) +1. Eval/practice signal is **real held-out official data only** — never the authored corpus (§3's + 200/200-vs-5/500 gap is the overfit fingerprint). +2. `wrong=0` gates what consolidates / lands. +3. The sealed test (1,319) is **never touched** — final exam only. +4. `holdout_dev` **splits**: tuned/practiced-against cases disjoint from the measured metric split — + "getting better" can only mean "generalizes to real problems." + +### 7.6 Design-first work IS the backup's curriculum (record, don't discard) +Every increment's refusal taxonomy, tagged incidence, and labeled `(parse, compile, execute, gold)` +tuples are **practice-lane artifacts**, not throwaway — the labeled data + failure map a candidate +generator would train against. Design-first pre-pays the backup's cold-start. ## 8. Errata proposal — ADR-0249 / ADR-0250 (statuses stay Accepted) @@ -150,14 +187,17 @@ memorized answers. The measurement only forces one distinction — **mechanism-g coverage.** The mechanism is general but starved: 1% reader reach, affine scope. The reader practice loop is precisely how that value becomes real coverage without re-introducing the overfit. -## 10. For the ruling (nothing started) +## 10. Ruling outcome (Shay, on this PR) -1. **Reader-practice-loop** — RECOMMENDED. Critical path for capability → practice → calibration → - serve. Grow real-GSM8K parse coverage off 1%, wrong=0-gated on a disjoint held-out real split; - curriculum ordered by §3 frequency (rate → fraction → compare). -2. **More compiler tiers** (compare/rate/fraction) — de-prioritized: the reader feeds them ~nothing. -3. **Serve-landing** (§3.7 + `generate/`) — premature; corridor real reach is 0/500. +The arc is the **reader, design-first**: per-family increments — a grammar family in Reader A plus its +matching compiler tier, landed **in lockstep** (§7) — starting with **`compare_multiplicative`**, then +strict §3 frequency order (rate → fraction → compare_additive → partition). Each increment is its own +smoke-gated PR with a measured `holdout_dev` delta, `wrong=0` held, sealed test untouched. -Delivered in this PR for the ruling: the errata sections (§8), the relabeled seal + two mechanism -floors composition-disclosed (§5), the parse-independent incidence tagging (§3), and the reader-arc -design with the four non-negotiable constraints (§7). **Nothing builds before the ruling on this PR.** +The **practice loop is the paper-designed backup** (§7.4), not the arc; it starts warm only if a full +increment buys **fewer than X = 15 new-correct on `holdout_dev`** (proposed §7.3, Shay rules with the +merge). §5 (mechanism floors + never-blended scoreboard), §6 sequencing +(capability → practice → calibration → serve; serve-landing deferred at 0/500), the errata (§8), and +the four non-negotiable constraints (§7.5) are affirmed and merge with this. + +**Next: the `compare_multiplicative` increment plan as the arc's first PR. Nothing else starts.**