From 639616b3e50851b9616a271ffbb3bd5559096426 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 18 Jul 2026 15:37:01 -0700 Subject: [PATCH] docs(dev2-spike): errata + two-instrument reframe + incidence + reader-arc design Folds in Shay's rulings on the spike: - ERRATA sections added to ADR-0249, ADR-0250, and both acceptance-evidence docs (statuses stay Accepted): corpus label 'real GSM8K dev holdout' -> 'CORE-authored GSM8K-style corpus (ADR-0119.2)'. Mechanism claims (wrong=0, PARITY, conservation, atomicity, chain-of-custody, 200/200) all stand; reframe not retraction. Covers the #74 review comment that repeated the label. (Also fixed ADR-0250 evidence status -> RATIFIED, which the ratification housekeeping missed.) - Two instruments named + never-blended scoreboard: authored = mechanism-correctness (200/200 stands as that); official holdout = generalization (5/500 symbolic / 0/500 corridor). public/dev-1 floors relabeled MECHANISM floors, composition-disclosed. - Censored-incidence fix: parse-INDEPENDENT tagging of 100 official cases (documented heuristic). 70/100 carry frontier markers (rate 31, fraction 28, compare_mult 20) -> (c) refuted, frontier COMMON not rare; the table is the reader-arc curriculum by real-world frequency. - Reader-practice-loop design with the four non-negotiable constraints (real held-out data only; wrong=0 gates consolidation; sealed test untouched; holdout_dev split disjoint from the measured metric). Sequencing: capability->practice->calibration-> serve; reader is the critical path for all three; serve-landing de-prioritized (0/500). Spike verdict for ruling; nothing built. --- ...miltonian-compiler-composition-frontier.md | 18 +++ .../ADR-0250-tier2-multi-entity-arithmetic.md | 16 ++ docs/handoff/ADR-0249-Acceptance-Evidence.md | 5 + docs/handoff/ADR-0250-Acceptance-Evidence.md | 7 +- ...2-frontier-measurement-spike-2026-07-18.md | 150 ++++++++++++------ 5 files changed, 143 insertions(+), 53 deletions(-) diff --git a/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md b/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md index 68731a32..e62478cf 100644 --- a/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md +++ b/docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md @@ -9,6 +9,24 @@ --- +## ERRATA (2026-07-18) — corpus mislabel corrected; status stays Accepted + +This ADR and its evidence repeatedly call `evals/gsm8k_math/dev` the **"real GSM8K dev holdout."** +That label is wrong: per ADR-0119.2, `dev` (and `public`) are **CORE-authored, GSM8K-*style*** problems +in CORE's own grammar — **not** real GSM8K. Real GSM8K lives in `train_sample`, `holdout_dev/v1`, and +the sealed test. Wherever this ADR says "real GSM8K dev holdout," read **"CORE-authored GSM8K-style +corpus (ADR-0119.2)."** + +**Nothing about the mechanism is retracted.** Certified relaxation, conservation, atomicity, +chain-of-custody, `wrong=0`, and PARITY against a symbolic fold of the *same* compiled program all +stand — the authored corpus is the **mechanism-correctness instrument**, and the compiler's 200/200 on +it is exactly that evidence. What falls is only the corpus label. Measured real-GSM8K reach (spike +`docs/research/dev2-frontier-measurement-spike-2026-07-18.md`): reader parses 5/500, corridor solves +0/500 — reader-gated. This erratum records that we caught our own mislabel via measurement; that is the +honest-measurement doctrine working, not a defect hidden. + +--- + ## 1. What this ADR is — and the thesis The generalized-lift instrument (ADR-0246 seam S4) recorded a single blocking diff --git a/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md b/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md index d4131c17..e1db0c8c 100644 --- a/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md +++ b/docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md @@ -9,6 +9,22 @@ --- +## ERRATA (2026-07-18) — corpus mislabel corrected; status stays Accepted + +This ADR calls the authored corpus the **"entire real GSM8K dev holdout."** That label is wrong: per +ADR-0119.2, `dev` (50) and `public` (150) are **CORE-authored, GSM8K-*style*** problems, **not** real +GSM8K. Read every "real GSM8K" reference here as **"CORE-authored GSM8K-style corpus (ADR-0119.2)."** + +**Nothing about the Tier-2 mechanism is retracted** — the multi-register model, coupled-translator +transfers, the relative conservation hard-reject, prepare→validate→commit atomicity, the certified +summation turn, and the chain-of-custody all stand; the authored corpus is the mechanism-correctness +instrument and the 50/50 (200/200 across both ADRs) is exactly that evidence. Only the corpus label +falls. Measured real-GSM8K reach (spike `docs/research/dev2-frontier-measurement-spike-2026-07-18.md`): +reader parses 5/500, corridor 0/500 — reader-gated. Recorded here as the honest-measurement doctrine +catching its own mislabel. + +--- + ## 1. What this ADR is ADR-0249 closed Tier-1 (single-accumulator affine arithmetic) and recorded the 24 multi-entity diff --git a/docs/handoff/ADR-0249-Acceptance-Evidence.md b/docs/handoff/ADR-0249-Acceptance-Evidence.md index fbbbb762..bfdeba34 100644 --- a/docs/handoff/ADR-0249-Acceptance-Evidence.md +++ b/docs/handoff/ADR-0249-Acceptance-Evidence.md @@ -1,5 +1,10 @@ # ADR-0249 — Acceptance Evidence +> **ERRATA (2026-07-18):** "real GSM8K dev holdout" in this doc = **CORE-authored GSM8K-style corpus +> (ADR-0119.2)**, the mechanism-correctness instrument — not real GSM8K. Mechanism results (wrong=0, +> PARITY, 200/200) stand; only the corpus label is corrected. Measured real-GSM8K reach: reader 5/500, +> corridor 0/500 (reader-gated). See ADR-0249 §ERRATA + `docs/research/dev2-frontier-measurement-spike-2026-07-18.md`. + **Status**: RATIFIED — ADR-0249 Accepted 2026-07-18 by Joshua Shay for commit `220039f7` **Date**: 2026-07-18 **ADR**: `docs/adr/ADR-0249-reader-hamiltonian-compiler-composition-frontier.md` diff --git a/docs/handoff/ADR-0250-Acceptance-Evidence.md b/docs/handoff/ADR-0250-Acceptance-Evidence.md index 15a17d2a..bd69d9af 100644 --- a/docs/handoff/ADR-0250-Acceptance-Evidence.md +++ b/docs/handoff/ADR-0250-Acceptance-Evidence.md @@ -1,6 +1,11 @@ # ADR-0250 — Acceptance Evidence -**Status**: Evidence pack for Shay's ruling (no self-Accept) +> **ERRATA (2026-07-18):** "real GSM8K dev holdout" in this doc = **CORE-authored GSM8K-style corpus +> (ADR-0119.2)**, the mechanism-correctness instrument — not real GSM8K. Mechanism results (wrong=0, +> full-holdout 50/50) stand; only the corpus label is corrected. Measured real-GSM8K reach: reader +> 5/500, corridor 0/500 (reader-gated). See ADR-0250 §ERRATA + `docs/research/dev2-frontier-measurement-spike-2026-07-18.md`. + +**Status**: RATIFIED — ADR-0250 Accepted 2026-07-18 by Joshua Shay **Date**: 2026-07-18 **ADR**: `docs/adr/ADR-0250-tier2-multi-entity-arithmetic.md` **Design record**: `docs/research/tier2-multi-entity-arithmetic-spike-2026-07-18.md` diff --git a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md index b500a992..c0c81395 100644 --- a/docs/research/dev2-frontier-measurement-spike-2026-07-18.md +++ b/docs/research/dev2-frontier-measurement-spike-2026-07-18.md @@ -49,69 +49,115 @@ Corroborates the holdout_dev README's 2026-06-04 baseline ("real GSM8K capabilit solves 0 real cases — not because it is wrong, but because the reader feeds it only 5 graphs and those 5 are a frontier kind the affine compiler refuses. -## 3. The (a)/(b)/(c) verdict +## 3. Censored-incidence fix — the parse-independent measurement -- **(a) — DOMINANT. The frontier (and real GSM8K generally) is reader-gated.** Reader A refuses - 495/500 real problems at parse. The binding constraint is not the compiler's affine scope; it is - that real GSM8K text does not become graphs at all. This is a *mechanism* gap in the reader. -- **(b) — CONFIRMED for dev/public, but as authoring, not draw-filtering.** The frontier-free authored - corpus tells us nothing about real GSM8K. Every "real GSM8K" coverage claim on dev/public needs the - denominator correction to "authored corpus." -- **(c) — REFUTED as the binding issue.** Frontier kinds are not rare in real GSM8K: the *very first* - problems that parse are `compare_multiplicative`, and real problems visibly use comparison / rate / - fraction language ("twice as many as", "half of"). The frontier is common; the reader just can't - reach it. +The 500-run's op-kind incidence (§2) is **censored data**: it reads kinds only from the 5 graphs the +reader produced, so it says nothing about what the 495 refusals *need*. To answer (b)-vs-(c) honestly, +frontier incidence must be measured **around** the refusing reader. Protocol: the first **100** of the +sha256-ordered holdout_dev (deterministic), classified by **keyword/pattern markers in the raw text** +— parse-independent, multi-label, imperfect (a marker need not mean the op is required; over/under-count +possible), but the *scale* is robust to that noise. -## 4. Public-seal — approved, with the disclosure sharpened +| Frontier kind (marker) | Incidence /100 | +| :--- | :--- | +| `apply_rate` ("per", "each … costs", "every …") | **31** | +| `fraction_portion` ("half", "third", "%", "a/b") | **28** | +| `compare_multiplicative` ("twice", "N times as many") | **20** | +| `compare_additive` ("more/fewer than") | 9 | +| `unit_partition` ("split into", "divided among") | 1 | +| **basic-only (no frontier marker)** | **30 / 100** | +| **≥1 frontier marker** | **70 / 100** | -Sealing/pinning `public` (150) buys a **narrow authored regression floor**: 150 cases, **1 op per -case (depth-1)**, **two kinds only** (divide 53 + transfer 97), all CORE-authored. Pin dev-1 (50/50) -and public (150/150) as regression floors, but the coverage record MUST carry, per slice: -- an **op-kind histogram** and **ops-per-case depth**, and -- the **authored-not-real** provenance flag. +## 4. The (a)/(b)/(c) verdict (de-censored) -Honest headline: **"200/200 wrong=0 on CORE-authored GSM8K-style (depth/kinds disclosed); real-GSM8K -compiler coverage = 0/500 (reader-gated)."** Anything shorter inflates. +- **(a) — DOMINANT. Real GSM8K is reader-gated.** Reader A refuses 495/500 at parse; text does not + become graphs. A *mechanism* gap in the reader, not the compiler's affine scope. +- **(b) — CONFIRMED as authoring bias.** dev/public are authored frontier-free; they carry zero + information about real GSM8K's distribution. Every "real GSM8K" claim on them needs the correction. +- **(c) — REFUTED, now with parse-independent evidence.** The frontier is **common, not rare**: + **70/100** official problems carry frontier markers; rate + fraction alone appear in ~55%. Widening + off-serving capability against this frontier does NOT fail the decidable-done-when test — the reader + reaching it does. -## 5. Sequencing verdict (three branches, resolved by the data) +**Bonus: this table is the reader arc's curriculum, ordered by real-world frequency** — rate (31) → +fraction (28) → compare_multiplicative (20) → compare_additive (9) → partition (1). Practice earns the +most real coverage by attacking rate/fraction/comparison first. -- big coherent bucket → build that tier; -- long-tail fragmentation → serve landing (§3.7 + scoped `generate/`); -- **frontier-rare-in-raw-GSM8K → serve landing** — **does NOT apply**: the frontier is common, not rare. +## 5. Two instruments, cleanly named, both kept (ruling) -The data selects a branch the skeleton didn't enumerate: **the biggest bucket is the reader itself** -(495/500). Building more compiler tiers is capability with no inputs; serve-landing is premature -(there is ~nothing real to serve). The decidable next move is to **make the reader parse real GSM8K**. +The finding is a **reframe, not a retraction** — nothing measured is thrown away, it is relabeled. -## 6. Reader-practice-loop — named, not started (per the charge) +- **Authored corpus = the mechanism-correctness instrument.** A controlled curriculum that proved the + compiler end-to-end. Its **200/200 wrong=0 stands, as exactly that** — evidence the mechanism + (versor transport + chained certified relaxation + conservation + atomicity + chain-of-custody) is + correct on clean inputs. +- **Official holdout = the generalization instrument.** `holdout_dev` (500), sealed test (1,319) — + real GSM8K, never authored. -The Reader-A answer is **no** — it cannot turn real GSM8K text into graphs at scale — so by the -standing agreement this moves the reader-practice-loop from "eventually" to the **leading next-arc -candidate**. The reader is the first genuinely *learnable* component in the stack: `attempt-parse → -compile → execute → gold-check` is a practice loop with a **decidable, wrong=0-gated reward** on a -held-out real set (`holdout_dev`, never the authored corpus). Crucially, this is the -**generalization-preserving** way to grow real capability — it optimizes parsing *coverage* against -held-out real data, the opposite of authoring more grammar-matched problems (which is the overfit -trap). It is named here for the post-spike ruling; not started. +**The scoreboard, never blended:** -## 7. The generalization reframe (Shay, live) +| Instrument | Slice | Score | Reads as | +| :--- | :--- | :--- | :--- | +| Mechanism-correctness | authored (200) | **200/200 wrong=0** | the compiler is correct | +| Generalization | official holdout_dev (500) | **5/500 Reader-A-symbolic · 0/500 corridor** | real reach is ~0 | -"We don't want overfitting; we want generalization" is right, and the compiler honors it: it is a -**general mechanism** (versor transport + chained certified relaxation), not memorized answers — -which is why it holds wrong=0 across authored problems it never saw. But the measurement forces a -distinction: **mechanism-generality ≠ real-problem coverage.** The mechanism is general yet starved — -1% reader coverage, and its affine scope misses the compare/rate/fraction the little real data uses. -"Generalized problem-solving on real inputs" (the original done-when) is therefore **not yet shown**; -what is shown is a general composition substrate validated on clean inputs. The reader-practice-loop -is exactly how that value (generalize, don't overfit) becomes real-problem coverage. +Seal `public` and pin both floors (dev-1 50/50, public 150/150) — but **label them mechanism floors, +not coverage claims**, each carrying an op-kind histogram + ops-per-case depth + the authored-not-real +flag (public: depth-1, divide 53 + transfer 97). A regression floor that guards mechanism correctness; +it says nothing about real-GSM8K coverage, and the record must say so. -## 8. For the ruling (three arcs on the table, none started) +## 6. Sequencing verdict (resolved by the data) -1. **Reader-practice-loop** — grow real-GSM8K parse coverage off 1%, wrong=0-gated on held-out real - data. Strongest by the measurement. -2. **More compiler tiers** (compare/rate/fraction) — de-prioritized: the reader feeds them almost - nothing; capability without inputs. -3. **Serve-landing** (§3.7 + `generate/`) — premature until there is real capability to serve. +The three-branch skeleton doesn't fire: the frontier is common (§3), so "frontier-rare → serve +landing" is out; and the biggest bucket isn't a compiler tier — **it's the reader** (495/500). Stated +plainly: **corridor real end-to-end capability is 0/500, so serve-landing of corridor arithmetic loses +priority — there is nothing real-reaching to serve.** §3.7 calibration stays queued; the **reader +practice volume is what eventually generates §3.7's calibration data.** Order unchanged: +**capability → practice → calibration → serve** — and the reader arc is now the **critical path for all +three**. -Recommendation: reader-practice-loop. Plus the cheap housekeeping: pin the two authored floors -(composition-disclosed) and correct the "real GSM8K" language in the ADR-0249/0250 evidence. +## 7. The next arc — reader practice loop (named for ruling; not started) + +99% parse-refusal makes the trigger a measured fact, not a judgment call. The reader is the first +genuinely *learnable* component: `attempt-parse → compile → execute → gold-check` is a practice loop +with a **decidable, wrong=0-gated reward**. Non-negotiable design constraints for the proposal: + +1. **Practice signal is real held-out official data only** — never the authored corpus. (Training + against authored grammar is exactly how the overfit happened; §3's 200/200-vs-5/500 gap is its + fingerprint.) +2. **`wrong=0` gates what consolidates** — a parse/graph is only kept if execute+gold-check is right, + never merely if it parsed. +3. **The sealed test (1,319) is never touched** — final exam only. +4. **`holdout_dev` splits** so practiced-against cases are **disjoint** from the measured dev metric + — "getting better" can only mean "generalizes to real problems," never "fit the metric set." + +Curriculum order = the §3 real-world frequency: rate → fraction → compare_multiplicative → … +The done-when is a rising `correct` on the disjoint held-out real split **with `wrong=0`** — the honest +metric the holdout lane was built to keep in front of us. + +## 8. Errata proposal — ADR-0249 / ADR-0250 (statuses stay Accepted) + +A prominent **errata section** (not silent rewording) is added to both ADRs and their acceptance +evidence, and to the PR #74 review record: the mechanism claims all survive; the single correction is +the corpus label — **"real GSM8K dev holdout" → "CORE-authored GSM8K-style corpus (ADR-0119.2)."** The +record thus shows we caught our own mislabel, which is the honest-measurement doctrine working, not a +failure to hide. (Errata text lands in this PR alongside the spike.) + +## 9. The generalization reframe (Shay, live) + +"Generalize, don't overfit" is right, and the compiler honors it: it is a general *mechanism*, not +memorized answers. The measurement only forces one distinction — **mechanism-generality ≠ real-problem +coverage.** The mechanism is general but starved: 1% reader reach, affine scope. The reader practice +loop is precisely how that value becomes real coverage without re-introducing the overfit. + +## 10. For the ruling (nothing started) + +1. **Reader-practice-loop** — RECOMMENDED. Critical path for capability → practice → calibration → + serve. Grow real-GSM8K parse coverage off 1%, wrong=0-gated on a disjoint held-out real split; + curriculum ordered by §3 frequency (rate → fraction → compare). +2. **More compiler tiers** (compare/rate/fraction) — de-prioritized: the reader feeds them ~nothing. +3. **Serve-landing** (§3.7 + `generate/`) — premature; corridor real reach is 0/500. + +Delivered in this PR for the ruling: the errata sections (§8), the relabeled seal + two mechanism +floors composition-disclosed (§5), the parse-independent incidence tagging (§3), and the reader-arc +design with the four non-negotiable constraints (§7). **Nothing builds before the ruling on this PR.**