From 6f2053bcb7cd806646bafefbaeb54c2ee430aec5 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 6 Jun 2026 16:09:41 -0700 Subject: [PATCH] =?UTF-8?q?docs(analysis):=20VERIFIED=20producer=20?= =?UTF-8?q?=E2=80=94=20validate-first=20verdict=20is=20DO=20NOT=20BUILD=20?= =?UTF-8?q?(comprehension-bound)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scoping + holdout validation of the VERIFIED canonical-comparison producer (the ADR-0206 §5 math-serving widening's gate). Records the empirical verdict so it is not re-chased: on holdout_dev the fold-derivation reader is 2 correct / 87 wrong on candidate-graph-refused cases, and a pure-chain certifier would admit 37 wrong (a wrong=0 breach) — the train_sample 3/7 was overfit. math_verifier.verify is solver-replay soundness, not correctness; the R1 graph reader is nested (0 flips). Math serving is comprehension-bound; the math seam correctly stays inert; the real lever is the general COMPREHEND organ, not a narrow GSM8K certifier. No code change. --- ...canonical-comparison-scoping-2026-06-06.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/analysis/VERIFIED-canonical-comparison-scoping-2026-06-06.md b/docs/analysis/VERIFIED-canonical-comparison-scoping-2026-06-06.md index 982a5a11..1444c8e7 100644 --- a/docs/analysis/VERIFIED-canonical-comparison-scoping-2026-06-06.md +++ b/docs/analysis/VERIFIED-canonical-comparison-scoping-2026-06-06.md @@ -64,3 +64,31 @@ should target the *narrowest* shape the reader extracts reliably, proving the me end-to-end (build → emit VERIFIED → seam widens → wrong=0 holds on holdout) before widening the shape coverage. This is the "checkable-conclusion domains" direction ([[project-self-check-soundness-not-correctness]]) made concrete for math serving. + +## Empirical verdict (2026-06-06) — DO NOT BUILD the fold-reader certifier + +A validate-first probe killed the back-substitution / pure-chain-certifier idea on the +independent holdout, BEFORE any build: + +- The serving `verify` (`math_verifier.verify`) is **solver-replay soundness**, not + correctness — it proves the solver executed the graph faithfully, NOT that the *parse* + (text→graph) is right. wrong=0 holds by the candidate-graph parser's conservative refusal. +- **No independent second reader helps.** On the refused set the R1 graph reader covers + `0/44` (train_sample) — it is *nested* in candidate-graph, not complementary. Cross-reader + agreement → 0 flips. +- The fold-derivation reader IS complementary but **unsafe**: on `holdout_dev` it answers + 89 of the 495 refused cases at **2 correct / 87 WRONG**. The train_sample looked like + 3 correct / 7 wrong — **overfit**. +- A pure-chain certifier (admit when no unhandled-structure cue: profit/per/%/more-than/…) + splits the holdout fold-answers into **1 correct / 37 WRONG (admit)** vs 1/50 (refuse). + It would **admit 37 wrong answers** — a wrong=0 breach. The mis-reads carry no shallow + structural signature; separating the 2 correct from the 87 wrong *is* the comprehension + problem. A certifier strict enough to reject all 87 rejects the 2 too. + +**Conclusion:** the VERIFIED-for-arithmetic producer via the existing readers is **not +buildable at wrong=0**. Math serving is **comprehension-bound**: the candidate-graph parser +refuses what it cannot model, and the only complementary reader is ~98% wrong on those. The +ADR-0206 §5 math seam correctly stays **inert**; the real lever is the general COMPREHEND +organ (helps every domain), not a narrow GSM8K certifier. Re-open only if a genuinely +complementary, independently-validated reader lands. Probe: `resolve_pooled` vs +`_score_one_candidate_graph` over `evals/gsm8k_math/holdout_dev/v1/cases.jsonl`.