docs: the honest real-GSM8K capability measurement (0% sound; composer 17% wrong on held-out)

The result that should have been visible for weeks. Measured on the new holdout_dev
lane (500 real held-out cases) + sealed test:
- real capability 0% (train_sample's 4 correct generalize to 0/500 held-out)
- resolve_pooled (the composition substrate) commits 17.4% WRONG on held-out
- NO structural gate (step/pool/op) separates its 2 correct from 87 wrong -> the
  composer cannot tell right readings from wrong; gating it is overfitting
- the only wrong=0 policy on real GSM8K is refusal (current serving 0/0/500)

Real lift needs a verifiable reading mechanism, not composer tuning. Non-serving (docs).

docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md

@@ -0,0 +1,87 @@
+<!-- CANONICAL | real-gsm8k-capability-measurement-2026-06-04.md | 2026-06-04 | measurement (Opus) lane | the honest measurement of real GSM8K capability + why the composer is unsound | verified against origin/main 43169979 + held-out dev lane + operator sealed runs -->
+
+# Real GSM8K Capability — the honest measurement (and why "wrong=0" was a 50-case illusion)
+
+This is the result the project should have been seeing for weeks. It was hidden because
+development was scored on a 50-case sample CORE was built against, never on held-out data.
+Now measured on 500 held-out cases (the new `holdout_dev` lane) and the sealed 1,319.
+
+## 1. The numbers — one scorer, three sets (2026-06-04)
+
+| Set | correct | wrong | refused | what it is |
+|---|---:|---:|---:|---|
+| `train_sample` (50) | 4 | 0 | 46 | the data CORE was **built on** — overfit |
+| `holdout_dev` (500) | **0** | 0 | 500 | real GSM8K **never built-against** — the honest baseline |
+| sealed test (1,319) | **0** | 0 | 1,319 | the final exam |
+
+**Real GSM8K capability is 0%.** The 4 train "correct" generalise to **zero** of 500
+held-out cases. Serving holds `wrong=0` **only because it refuses everything real** — the
+one genuine, defensible property (zero confabulation), not an accuracy.
+
+## 2. The deeper finding — the composer is *unsound*, not just narrow
+
+The serving path refuses real cases (0/0/500). The obvious "lift" is to wire the
+**derivation composer** (`resolve_pooled` — the ADR-0178/0207 substrate, the reader the
+whole composition program was built around as `wrong=0`-safe). Measured on held-out:
+
+> **`resolve_pooled` on the 500: 2 correct / 87 WRONG / 411 refused — 17.4% wrong.**
+
+The composer's self-verification gate (grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness)
+**is not `wrong=0`-safe on real data.** "wrong=0" held only on the 50 train cases it was
+tuned to. On real GSM8K the composer confabulates 17%. This is *why* `product_bridge`
+(which wraps `resolve_pooled`) committed 0/5 on the sealed set — it was exposing a leaky
+reader through a narrow whitelist.
+
+## 3. Why no gate fixes it — correct and wrong are structurally identical
+
+Characterising all 89 composer commits on held-out (2 correct, 87 wrong):
+
+- **Step count** does not separate them: wrong commits span 1–5 steps; the 2 correct are at
+  2 and 4 — inside the wrong distribution.
+- **Pool size** does not separate them: 54 of 87 wrong are *lone chains* (pool = 1, no rival
+  to trigger the disagreement refusal); the 2 correct also have pool ∈ {1, 2}.
+- **Op-sequence** does not separate them: the wrong commits are dominated by multiply-chains
+  (`multiply,multiply` ×28, `multiply` ×25, triple+ ×22); the 2 correct are *also*
+  multiply-chains.
+
+**The composer cannot distinguish its correct readings from its wrong ones.** The 2 correct
+are indistinguishable from the 87 wrong by every structural feature available to a gate.
+Therefore **no principled gate admits the correct while refusing the wrong** — any gate that
+keeps the 2 would be overfit to those exact cases (the same disease that produced the
+breach). The disagreement rule, the substrate's main `wrong=0` defence, fails on the 54
+lone-chain wrongs by construction.
+
+## 4. What this means (honest, load-bearing)
+
+1. **The only `wrong=0`-safe policy on real GSM8K is refusal.** Current serving (0/0/500) is
+   that policy. It is the honest floor, and the zero-confabulation discipline is real.
+2. **Incremental composer tuning is not a path to sound lift.** The composer's readings are
+   not verifiable as built; committing them leaks ~17% wrong. Tightening to dodge specific
+   held-out wrongs is overfitting.
+3. **Real lift requires a fundamentally more *verifiable* reading mechanism** — one that can
+   prove a multi-step reading correct (or refuse), rather than search-and-self-verify, which
+   the data shows is indistinguishable between right and wrong. That is deep research, not a
+   knob.
+4. **The instrument now exists.** Every future capability claim is measured on `holdout_dev`
+   (correct must rise with `wrong=0`) and confirmed on the sealed test. No train_sample
+   number is ever the score again.
+
+## 5. The honest external framing (for any outward material)
+
+CORE on real GSM8K: **0% coverage, 0 confabulations.** It refuses what it cannot verify; it
+does not guess. That is the truthful claim — not an accuracy number. The composition
+substrate is a research direction with a measured 17% confabulation rate on real data, which
+is *why it is not wired to serving*. Progress is now measurable honestly for the first time.
+
+## Reproduce
+
+```bash
+PYTHONPATH=. .venv/bin/python -m evals.gsm8k_math.holdout_dev.v1.runner   # 0/0/500
+# composer-unsoundness:
+PYTHONPATH=. .venv/bin/python -c "import json;from generate.derivation.pool import resolve_pooled;d=[json.loads(l) for l in open('evals/gsm8k_math/holdout_dev/v1/cases.jsonl')];c=w=0
+for x in d:
+ r=resolve_pooled(x['problem'])
+ if r: c+= abs(r.answer-x['expected_answer'])<1e-6; w+= abs(r.answer-x['expected_answer'])>=1e-6
+print('composer correct',c,'wrong',w)"
+# sealed (operator only, key required): 0/0/1319
+```