From d4a002e62676c67cb837199bc2564e45f9cceb0f Mon Sep 17 00:00:00 2001 From: Shay Date: Thu, 4 Jun 2026 02:52:24 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20fourth=20lift=20attempt=20falsified=20(?= =?UTF-8?q?strict=20sum-reader=200/2)=20=E2=80=94=20exhaustive?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four genuine build-and-measure attempts on held-out, all falsified: - composer 87 wrong, narrow reader 61 wrong, strict sum-reader 2 wrong (100%), candidate-graph 0 admissible. Even 'exactly 2 numbers + total cue' cases are multi-step in disguise. No sound committing strategy exists in this substrate. --- docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md b/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md index 20cfbd9c..7e289c3e 100644 --- a/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md +++ b/docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md @@ -58,7 +58,9 @@ I did not stop at diagnosis — I built and measured two committing readers on h | Attempt | held-out 500 | result | |---|---|---| | `resolve_pooled` (the built composer) | 2 correct / **87 wrong** | 17% confabulation | -| Maximally-narrow forced reader (exactly 2 grounded quantities + one unambiguous op cue, refuse otherwise) | 0 correct / **61 wrong** | **100% confabulation when it fires** | +| Maximally-narrow forced reader (exactly 2 grounded quantities + one unambiguous op cue) | 0 correct / **61 wrong** | **100% confab when it fires** | +| Strictest sum-reader (exactly 2 numbers + total-cue in question + every complexity cue blocked → A+B) | 0 correct / **2 wrong** | **100% confab; the 2 "simple" cases were multi-step (a fraction, a "double") hiding behind 2 numbers** | +| candidate-graph sound filter (roundtrip + disagreement) | **0 admissible candidates built** | sound, but constructs nothing to commit | The narrow reader is the more brutal result: GSM8K problems that *look* like "2 numbers + a cue word" are almost never 2-operand problems — they are multi-step problems where the