# dev-holdout-2 / Frontier-Measurement Spike — verdict

**Status**: SPIKE VERDICT — for ruling before any build
**Date**: 2026-07-18
**Base**: `forgejo/main @ b87cd46c` (ADR-0249 + ADR-0250 Accepted; compiler solves the authored dev/public corpus)
**Branch**: `feat/dev2-frontier-spike`
**Charge (Shay)**: public-seal + floors; the Reader A answer; the raw-incidence measurement with an
(a)/(b)/(c) verdict; the draw-protocol trace — before building anything.

---

## 0. Headline correction (load-bearing — read first)

**`evals/gsm8k_math/dev` (50) and `public` (150) are NOT real GSM8K.** ADR-0119.2 authors them as
200 **CORE-original, GSM8K-*style*** problems written in CORE's own vocabulary/grammar to a chosen
operation distribution (case notes: "Generated public case exercising divide operations"). My
ADR-0249/0250 evidence called these "the real GSM8K dev holdout" — that is inaccurate and should be
corrected to "CORE-authored GSM8K-style." The compiler's verified result is **200/200 wrong=0 on
authored, grammar-matched, mostly depth-1 problems** — a real validation of the *mechanism*, not of
real-GSM8K capability.

## 1. Draw-protocol trace (Shay's ask: was the draw annotability-conditioned?)

| Slice | Real GSM8K? | How drawn |
| :--- | :--- | :--- |
| `train_sample` (50) | **Real** | `openai/gsm8k` main/train, rev `740312a…`; deterministic `sha256(f"{i}:SALT")` rank, top 50. What the grammar was hand-built against (overfit). |
| `holdout_dev/v1` (500) | **Real** | train split minus train_sample, sorted by `sha256(question)`, first 500. "Real GSM8K CORE was NOT built on." |
| sealed test (1,319) | **Real** | GSM8K test split, sealed (`.age`). |
| **`dev` (50), `public` (150)** | **Authored** | **CORE-authored** (ADR-0119.2), CORE grammar, chosen op/depth distribution. |

So (b) is not subtle "selection bias in a real draw" — it is **authoring**: dev/public are synthetic
and their frontier-freeness is an authoring choice, carrying zero information about real GSM8K's
distribution. The real draws (train_sample, holdout_dev) are unbiased and were the right place to
measure all along.

## 2. The real measurement (end-to-end on 500 real holdout_dev cases)

Pipeline: real text → `parse_and_solve` (Reader A) → my compiler (Tier-1/Tier-2) → execute → gold.

| Metric | Value |
| :--- | :--- |
| Reader A parses a graph | **5 / 500 (1%)** — refuses **495** |
| Reader A own solver correct | 5 / 500 (1%) |
| **My compiler solved** | **0 / 500** |
| …compiler wrong | 0 (fail-closed held) |
| The 5 parsed graphs' op kinds | **all `compare_multiplicative`** (a frontier kind) |

Corroborates the holdout_dev README's 2026-06-04 baseline ("real GSM8K capability 0%"). The compiler
solves 0 real cases — not because it is wrong, but because the reader feeds it only 5 graphs and those
5 are a frontier kind the affine compiler refuses.

## 3. Censored-incidence fix — the parse-independent measurement

The 500-run's op-kind incidence (§2) is **censored data**: it reads kinds only from the 5 graphs the
reader produced, so it says nothing about what the 495 refusals *need*. To answer (b)-vs-(c) honestly,
frontier incidence must be measured **around** the refusing reader. Protocol: the first **100** of the
sha256-ordered holdout_dev (deterministic), classified by **keyword/pattern markers in the raw text**
— parse-independent, multi-label, imperfect (a marker need not mean the op is required; over/under-count
possible), but the *scale* is robust to that noise.

| Frontier kind (marker) | Incidence /100 |
| :--- | :--- |
| `apply_rate` ("per", "each … costs", "every …") | **31** |
| `fraction_portion` ("half", "third", "%", "a/b") | **28** |
| `compare_multiplicative` ("twice", "N times as many") | **20** |
| `compare_additive` ("more/fewer than") | 9 |
| `unit_partition` ("split into", "divided among") | 1 |
| **basic-only (no frontier marker)** | **30 / 100** |
| **≥1 frontier marker** | **70 / 100** |

## 4. The (a)/(b)/(c) verdict (de-censored)

- **(a) — DOMINANT. Real GSM8K is reader-gated.** Reader A refuses 495/500 at parse; text does not
  become graphs. A *mechanism* gap in the reader, not the compiler's affine scope.
- **(b) — CONFIRMED as authoring bias.** dev/public are authored frontier-free; they carry zero
  information about real GSM8K's distribution. Every "real GSM8K" claim on them needs the correction.
- **(c) — REFUTED, now with parse-independent evidence.** The frontier is **common, not rare**:
  **70/100** official problems carry frontier markers; rate + fraction alone appear in ~55%. Widening
  off-serving capability against this frontier does NOT fail the decidable-done-when test — the reader
  reaching it does.

**Bonus: this table is the reader arc's curriculum, ordered by real-world frequency** — rate (31) →
fraction (28) → compare_multiplicative (20) → compare_additive (9) → partition (1). Practice earns the
most real coverage by attacking rate/fraction/comparison first.

## 5. Two instruments, cleanly named, both kept (ruling)

The finding is a **reframe, not a retraction** — nothing measured is thrown away, it is relabeled.

- **Authored corpus = the mechanism-correctness instrument.** A controlled curriculum that proved the
  compiler end-to-end. Its **200/200 wrong=0 stands, as exactly that** — evidence the mechanism
  (versor transport + chained certified relaxation + conservation + atomicity + chain-of-custody) is
  correct on clean inputs.
- **Official holdout = the generalization instrument.** `holdout_dev` (500), sealed test (1,319) —
  real GSM8K, never authored.

**The scoreboard, never blended:**

| Instrument | Slice | Score | Reads as |
| :--- | :--- | :--- | :--- |
| Mechanism-correctness | authored (200) | **200/200 wrong=0** | the compiler is correct |
| Generalization | official holdout_dev (500) | **5/500 Reader-A-symbolic · 0/500 corridor** | real reach is ~0 |

Seal `public` and pin both floors (dev-1 50/50, public 150/150) — but **label them mechanism floors,
not coverage claims**, each carrying an op-kind histogram + ops-per-case depth + the authored-not-real
flag (public: depth-1, divide 53 + transfer 97). A regression floor that guards mechanism correctness;
it says nothing about real-GSM8K coverage, and the record must say so.

## 6. Sequencing verdict (resolved by the data)

The three-branch skeleton doesn't fire: the frontier is common (§3), so "frontier-rare → serve
landing" is out; and the biggest bucket isn't a compiler tier — **it's the reader** (495/500). Stated
plainly: **corridor real end-to-end capability is 0/500, so serve-landing of corridor arithmetic loses
priority — there is nothing real-reaching to serve.** §3.7 calibration stays queued; the **reader
practice volume is what eventually generates §3.7's calibration data.** Order unchanged:
**capability → practice → calibration → serve** — and the reader arc is now the **critical path for all
three**.

## 7. The arc (RULED) — reader, design-first: per-family increments in lockstep

Superseding §10's practice-loop recommendation: the frontier being **common, formulaic, and
frequency-ordered** (§3) argues *for* designed grammar-family extensions, not against — a formulaic
frontier is exactly what designed grammar eats efficiently, each increment cheaply measurable.
**Learning earns its complexity only when design stops paying — decided by number, not feel.** The
practice loop is the paper-designed backup (§7.4), not the arc.

### 7.1 Increment = a grammar family + its compiler tier, landed together
Reader and compiler move in **lockstep**. §2 is the proof they cannot move separately: the 5 real
`compare_multiplicative` parses **die at the affine compiler today** (reader emits, compiler refuses →
0/500). A reader family with no compiler tier produces graphs nothing solves; a compiler tier with no
reader family has no inputs — that is the *only* thing §10.2 de-prioritizes. **Paired, they are the
arc.** Each increment is its own smoke-gated PR carrying a measured `holdout_dev` delta (new correct),
the `wrong=0` floor held, and the sealed test untouched.

### 7.2 Increment order
1. **`compare_multiplicative` FIRST** — jumps the §3 frequency queue because the reader *already emits
   real parses for it* (the 5/500), so pairing a compare compiler tier moves corridor real-reach from
   **0/500 → >0 immediately** — the cheapest possible end-to-end proof the increment loop works on
   official data.
2. Then strict §3 frequency order: **rate (31) → fraction (28) → compare_additive (9) → partition (1)**.

### 7.3 Escalation threshold X — **RATIFIED (Shay, 2026-07-18, PR #76): X = 15**
Fixed before anyone is attached, so "switch when design stops working" cannot drift.
**X = 15 new-correct on `holdout_dev` per full family increment.** When a completed increment
buys fewer than X, escalate to a practice-lane ruling. Rationale: these families carry 20–31% incidence
(~100–155/500 problems each); a design that is paying converts far more than 15, so falling below
15/500 (3%) means design has stopped efficiently converting known incidence — a number trips the
switch, not a feeling.

### 7.4 Backup lane — the practice loop (PAPER DESIGN only; starts warm if X trips)
Same decidable loop (`attempt-parse → compile → execute → gold-check`, wrong=0-gated). Its leverage —
how it reuses what these arcs already built, so only two things are genuinely new:
- **Verifier** = the Tier-1/Tier-2 compiler stack: a candidate parse is correct **iff**
  `compile → execute → gold` agrees. No separate oracle.
- **Audit** = the certificate chains (`RelaxationCertificate`, `TurnRecord`, chain-of-custody).
- **Consolidation gating** = `prepare → validate → commit` generalized: a candidate consolidates only
  on full success.
- **Holdout hygiene** = ADR-0119 sealing (split discipline; sealed test untouched).
- **Genuinely new** = a **candidate generator** (proposes parses/grammar variants) + a **consolidation
  store** (what survives wrong=0). That is the whole delta the lane adds.

### 7.5 Non-negotiable constraints (bind the increments now; carry to the lane verbatim)
1. Eval/practice signal is **real held-out official data only** — never the authored corpus (§3's
   200/200-vs-5/500 gap is the overfit fingerprint).
2. `wrong=0` gates what consolidates / lands.
3. The sealed test (1,319) is **never touched** — final exam only.
4. `holdout_dev` **splits**: tuned/practiced-against cases disjoint from the measured metric split —
   "getting better" can only mean "generalizes to real problems."

### 7.6 Design-first work IS the backup's curriculum (record, don't discard)
Every increment's refusal taxonomy, tagged incidence, and labeled `(parse, compile, execute, gold)`
tuples are **practice-lane artifacts**, not throwaway — the labeled data + failure map a candidate
generator would train against. Design-first pre-pays the backup's cold-start.

## 8. Errata proposal — ADR-0249 / ADR-0250 (statuses stay Accepted)

A prominent **errata section** (not silent rewording) is added to both ADRs and their acceptance
evidence, and to the PR #74 review record: the mechanism claims all survive; the single correction is
the corpus label — **"real GSM8K dev holdout" → "CORE-authored GSM8K-style corpus (ADR-0119.2)."** The
record thus shows we caught our own mislabel, which is the honest-measurement doctrine working, not a
failure to hide. (Errata text lands in this PR alongside the spike.)

## 9. The generalization reframe (Shay, live)

"Generalize, don't overfit" is right, and the compiler honors it: it is a general *mechanism*, not
memorized answers. The measurement only forces one distinction — **mechanism-generality ≠ real-problem
coverage.** The mechanism is general but starved: 1% reader reach, affine scope. The reader practice
loop is precisely how that value becomes real coverage without re-introducing the overfit.

## 10. Ruling outcome (Shay, on this PR)

The arc is the **reader, design-first**: per-family increments — a grammar family in Reader A plus its
matching compiler tier, landed **in lockstep** (§7) — starting with **`compare_multiplicative`**, then
strict §3 frequency order (rate → fraction → compare_additive → partition). Each increment is its own
smoke-gated PR with a measured `holdout_dev` delta, `wrong=0` held, sealed test untouched.

The **practice loop is the paper-designed backup** (§7.4), not the arc; it starts warm only if a full
increment buys **fewer than X = 15 new-correct on `holdout_dev`** (proposed §7.3, Shay rules with the
merge). §5 (mechanism floors + never-blended scoreboard), §6 sequencing
(capability → practice → calibration → serve; serve-landing deferred at 0/500), the errata (§8), and
the four non-negotiable constraints (§7.5) are affirmed and merge with this.

**Next: the `compare_multiplicative` increment plan as the arc's first PR. Nothing else starts.**