From 47a406b59034a830f5ee42147ade71253e9a78fb Mon Sep 17 00:00:00 2001 From: Shay Date: Fri, 29 May 2026 13:23:02 -0700 Subject: [PATCH] =?UTF-8?q?docs(adr-0163-f2):=20clarify=20`refuse`=20is=20?= =?UTF-8?q?maturity-relative=20=E2=80=94=20confusers=20are=20solvable=20co?= =?UTF-8?q?verage=20targets=20that=20graduate=20(#478)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A confuser's `expected: refuse` does not mean unanswerable. Every confuser carries its true gold (`answer_numeric`) and is a solvable coverage target; `refuse` means "the reader can't yet comprehend this category, so refusing is the honest outcome — committing a WRONG value is the defect." Example: 0001 "buys a toy for 30 coins … how many left?" is plain 50-30=20; it is refuse only because today's reader takes `buys` as a gain. Adds §2.1 (graduation protocol): when a general mechanism reads a category correctly (validated on train_sample + the category, wrong=0 preserved), those cases graduate refuse -> solve and committing their gold becomes a win. Reframes the `spurious` verdict as "solved-before-graduation" (a graduation signal, with pair-consistency as the genuine-reading-vs-surface-match discriminator), not automatically a defect. Notes that no v1 category is degenerate, and how a truly-unanswerable case would be labelled (answer_numeric: null + degenerate: true) so the two senses of "refuse" stay distinguishable. Spec only; no corpus/runner change (today's commits are wrong-reading, so spurious stays flagged until a category genuinely graduates). --- .../ADR-0163-F2-confuser-corpus-spec.md | 49 ++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0163-F2-confuser-corpus-spec.md b/docs/decisions/ADR-0163-F2-confuser-corpus-spec.md index bc5e3d99..28b57142 100644 --- a/docs/decisions/ADR-0163-F2-confuser-corpus-spec.md +++ b/docs/decisions/ADR-0163-F2-confuser-corpus-spec.md @@ -41,6 +41,7 @@ lane: |---|---|---| | **`wrong` on confusers** | the reader committed an answer a confuser was built to trip | **must be 0** | | **`refused` on confusers** | the reader declined (the correct response to a not-yet-readable shape) | expected-high, honest frontier | +| **`spurious` on confusers** | committed the *correct* value on a `refuse` case (right answer, but the bar was "don't commit yet") | see §2.1 — a *graduation signal*, not automatically a defect | | **`solved` on the paired positives** | the reader genuinely read the near-identical solvable case | real-capability signal | The headline number is **wrong=0**, never "how many solved." A change that solves @@ -48,6 +49,49 @@ more confusers is only progress if it does so by a **general mechanism** that al keeps `wrong=0` — and is validated on the held-out real lane, not by passing these specific rows. +## 2.1 `refuse` is maturity-relative — every confuser is a solvable coverage target + +**Amendment (2026-05-29).** A confuser's `expected: refuse` does **not** mean the +problem is unanswerable or missing information. Every confuser carries its **true +gold** in `answer_numeric` and is, ultimately, a **solvable coverage target**. The +`refuse` label is **relative to what the reader can currently comprehend**: it means +*"the engine cannot yet read this category, so refusing is the honest outcome — +committing a **wrong** value is the defect."* + +The clearest example is `confuser-v1-0001`: + +> "Dan has 50 coins. He buys a toy **for** 30 coins. How many coins does Dan have +> **left**?" — gold **20**. + +This is a plain subtraction (`50 − 30 = 20`; "left" even names the operation). It is +labelled `refuse` only because today's reader has `buys` in its *gain* lexeme set and +would commit `50 + 30 = 80`. A mature reader **should solve it to 20** — at which +point its honest label is `solve`, not `refuse`. + +**Graduation protocol.** When a *general* mechanism teaches the reader to read a +category correctly — validated on `train_sample` + the category, with `wrong=0` +preserved (the §3/§8 guardrails) — those cases **graduate** `refuse → solve`, and +committing their gold becomes a *win*. Graduation is the intended success path; it +does **not** contradict "never optimise `refused` down by patches" (§2), because the +bar is comprehension proven generally, not a row passed by vocab fitting. + +**`spurious` reframed.** A correct-valued commit on a `refuse` case is scored +`spurious` (right value, but the case said "don't commit yet"). That is +**"solved-before-graduation" — a signal to re-examine the case for graduation, not +automatically a defect.** The **pair-consistency** check is the discriminator: a +reader that solves the twin *and* commits the confuser by the same surface cue is +surface-matching (a real tell); a reader that commits the correct gold *while still +discriminating the pair* has genuinely read it, and the case should graduate. Until a +category graduates, `spurious` stays flagged (conservative — wrong=0 first). + +**No category in v1 is degenerate.** Every current confuser is a solvable coverage +target (disguised-polarity, pseudo-accumulation/fractions, multi-referent, +multi-actor-pronoun, distractor, temporal-scope, comparative-referent, unit-confuser +all have a definite gold reading). A genuinely *unanswerable* case (truly +insufficient/contradictory information) — if ever added — would be labelled +`expected: refuse` with `answer_numeric: null` and a `degenerate: true` flag, so the +two senses of "refuse" (not-yet-readable vs unanswerable) stay distinguishable. + ## 3. Anti-overfitting design rules (the load-bearing part) 1. **Minimal pairs.** Every confuser ships with a near-identical *solvable* twin @@ -68,7 +112,10 @@ specific rows. (below), with varied surface forms, so "passing the category" can't be one rule. 5. **Expected-behavior labelling.** Each case is labelled `expected: refuse | solve`. Most confusers are `refuse` (the reader can't yet read them and must not guess) — - so the corpus is primarily a *refusal-correctness* test. + so the corpus is primarily a *refusal-correctness* test. The `refuse` label is + **maturity-relative**, not a claim of unanswerability: every confuser carries its + true gold and graduates `refuse → solve` once a general mechanism reads its + category (see §2.1). ## 4. Confuser categories (grounded in observed real misfires)