From c2e74fe2fd9e2ec2a0c847ed1f00e93b91e2ae90 Mon Sep 17 00:00:00 2001 From: Shay Date: Thu, 4 Jun 2026 02:42:14 -0700 Subject: [PATCH] docs: the honest real-GSM8K capability measurement (0% sound; composer 17% wrong on held-out) The result that should have been visible for weeks. Measured on the new holdout_dev lane (500 real held-out cases) + sealed test: - real capability 0% (train_sample's 4 correct generalize to 0/500 held-out) - resolve_pooled (the composition substrate) commits 17.4% WRONG on held-out - NO structural gate (step/pool/op) separates its 2 correct from 87 wrong -> the composer cannot tell right readings from wrong; gating it is overfitting - the only wrong=0 policy on real GSM8K is refusal (current serving 0/0/500) Real lift needs a verifiable reading mechanism, not composer tuning. Non-serving (docs). --- ...gsm8k-capability-measurement-2026-06-04.md | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md diff --git a/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md b/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md new file mode 100644 index 00000000..231ee703 --- /dev/null +++ b/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md @@ -0,0 +1,87 @@ + + +# Real GSM8K Capability — the honest measurement (and why "wrong=0" was a 50-case illusion) + +This is the result the project should have been seeing for weeks. It was hidden because +development was scored on a 50-case sample CORE was built against, never on held-out data. +Now measured on 500 held-out cases (the new `holdout_dev` lane) and the sealed 1,319. + +## 1. The numbers — one scorer, three sets (2026-06-04) + +| Set | correct | wrong | refused | what it is | +|---|---:|---:|---:|---| +| `train_sample` (50) | 4 | 0 | 46 | the data CORE was **built on** — overfit | +| `holdout_dev` (500) | **0** | 0 | 500 | real GSM8K **never built-against** — the honest baseline | +| sealed test (1,319) | **0** | 0 | 1,319 | the final exam | + +**Real GSM8K capability is 0%.** The 4 train "correct" generalise to **zero** of 500 +held-out cases. Serving holds `wrong=0` **only because it refuses everything real** — the +one genuine, defensible property (zero confabulation), not an accuracy. + +## 2. The deeper finding — the composer is *unsound*, not just narrow + +The serving path refuses real cases (0/0/500). The obvious "lift" is to wire the +**derivation composer** (`resolve_pooled` — the ADR-0178/0207 substrate, the reader the +whole composition program was built around as `wrong=0`-safe). Measured on held-out: + +> **`resolve_pooled` on the 500: 2 correct / 87 WRONG / 411 refused — 17.4% wrong.** + +The composer's self-verification gate (grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness) +**is not `wrong=0`-safe on real data.** "wrong=0" held only on the 50 train cases it was +tuned to. On real GSM8K the composer confabulates 17%. This is *why* `product_bridge` +(which wraps `resolve_pooled`) committed 0/5 on the sealed set — it was exposing a leaky +reader through a narrow whitelist. + +## 3. Why no gate fixes it — correct and wrong are structurally identical + +Characterising all 89 composer commits on held-out (2 correct, 87 wrong): + +- **Step count** does not separate them: wrong commits span 1–5 steps; the 2 correct are at + 2 and 4 — inside the wrong distribution. +- **Pool size** does not separate them: 54 of 87 wrong are *lone chains* (pool = 1, no rival + to trigger the disagreement refusal); the 2 correct also have pool ∈ {1, 2}. +- **Op-sequence** does not separate them: the wrong commits are dominated by multiply-chains + (`multiply,multiply` ×28, `multiply` ×25, triple+ ×22); the 2 correct are *also* + multiply-chains. + +**The composer cannot distinguish its correct readings from its wrong ones.** The 2 correct +are indistinguishable from the 87 wrong by every structural feature available to a gate. +Therefore **no principled gate admits the correct while refusing the wrong** — any gate that +keeps the 2 would be overfit to those exact cases (the same disease that produced the +breach). The disagreement rule, the substrate's main `wrong=0` defence, fails on the 54 +lone-chain wrongs by construction. + +## 4. What this means (honest, load-bearing) + +1. **The only `wrong=0`-safe policy on real GSM8K is refusal.** Current serving (0/0/500) is + that policy. It is the honest floor, and the zero-confabulation discipline is real. +2. **Incremental composer tuning is not a path to sound lift.** The composer's readings are + not verifiable as built; committing them leaks ~17% wrong. Tightening to dodge specific + held-out wrongs is overfitting. +3. **Real lift requires a fundamentally more *verifiable* reading mechanism** — one that can + prove a multi-step reading correct (or refuse), rather than search-and-self-verify, which + the data shows is indistinguishable between right and wrong. That is deep research, not a + knob. +4. **The instrument now exists.** Every future capability claim is measured on `holdout_dev` + (correct must rise with `wrong=0`) and confirmed on the sealed test. No train_sample + number is ever the score again. + +## 5. The honest external framing (for any outward material) + +CORE on real GSM8K: **0% coverage, 0 confabulations.** It refuses what it cannot verify; it +does not guess. That is the truthful claim — not an accuracy number. The composition +substrate is a research direction with a measured 17% confabulation rate on real data, which +is *why it is not wired to serving*. Progress is now measurable honestly for the first time. + +## Reproduce + +```bash +PYTHONPATH=. .venv/bin/python -m evals.gsm8k_math.holdout_dev.v1.runner # 0/0/500 +# composer-unsoundness: +PYTHONPATH=. .venv/bin/python -c "import json;from generate.derivation.pool import resolve_pooled;d=[json.loads(l) for l in open('evals/gsm8k_math/holdout_dev/v1/cases.jsonl')];c=w=0 +for x in d: + r=resolve_pooled(x['problem']) + if r: c+= abs(r.answer-x['expected_answer'])<1e-6; w+= abs(r.answer-x['expected_answer'])>=1e-6 +print('composer correct',c,'wrong',w)" +# sealed (operator only, key required): 0/0/1319 +```