# ADR-0163-F2 — Confuser Corpus: a discrimination probe, not a coverage target

**Status:** Proposed (spec only — no code). Follow-on to ADR-0163 §F (the Track-B
additive practice scale). Corrects the *metric*, not just the data.

> **One line.** A small, curated set of **hard negatives and near-miss confusers**
> whose purpose is to measure whether the reader **refuses what it cannot yet read**
> — success is `wrong = 0` on the confusers, *not* a higher solved count.

---

## 1. Why this exists (the failure it fixes)

ADR-0163 §F added 150 templated additive practice cases. GB-3b flipped 55–96 of
them — but on **real GSM8K (`train_sample`) accumulation fired on only 2/50, 1
correct / 1 wrong**. The synthetic flips did not generalise. The canonical misfire:

> `train_sample-0002`: *"Jan buys 1000 feet of cable. She splits it into 25-foot
> sections. She gives 1/4 to a friend. She puts half in storage. How much does she
> keep?"* (gold 15) — read as `buys 1000 … gives … to` → **996**.

The templated corpus has two defects that *cause* overfitting (memory:
`feedback-synthetic-corpus-overfitting-trap`):

1. **Redundancy** — the 46 gain cases are one template × 46 noun/number swaps, so a
   surface-cue matcher "passes" without comprehension and the CP ledger fills with
   tautological confirmations.
2. **No hard negatives** — there is no problem that *looks* like accumulation but
   isn't (a fraction problem, a disguised-polarity verb, a second actor). The
   confusers are exactly what teach a reader **where to refuse**, and a corpus of
   clean templates contains none.

This spec builds the missing half: the cases that make being wrong *informative*.

## 2. The reframe — what "success" means here

This corpus is a **discrimination probe**, scored the opposite way from a coverage
lane:

| metric | meaning | bar |
|---|---|---|
| **`wrong` on confusers** | the reader committed an answer a confuser was built to trip | **must be 0** |
| **`refused` on confusers** | the reader declined (the correct response to a not-yet-readable shape) | expected-high, honest frontier |
| **`spurious` on confusers** | committed the *correct* value on a `refuse` case (right answer, but the bar was "don't commit yet") | see §2.1 — a *graduation signal*, not automatically a defect |
| **`solved` on the paired positives** | the reader genuinely read the near-identical solvable case | real-capability signal |

The headline number is **wrong=0**, never "how many solved." A change that solves
more confusers is only progress if it does so by a **general mechanism** that also
keeps `wrong=0` — and is validated on the held-out real lane, not by passing these
specific rows.

## 2.1 `refuse` is maturity-relative — every confuser is a solvable coverage target

**Amendment (2026-05-29).** A confuser's `expected: refuse` does **not** mean the
problem is unanswerable or missing information. Every confuser carries its **true
gold** in `answer_numeric` and is, ultimately, a **solvable coverage target**. The
`refuse` label is **relative to what the reader can currently comprehend**: it means
*"the engine cannot yet read this category, so refusing is the honest outcome —
committing a **wrong** value is the defect."*

The clearest example is `confuser-v1-0001`:

> "Dan has 50 coins. He buys a toy **for** 30 coins. How many coins does Dan have
> **left**?" — gold **20**.

This is a plain subtraction (`50 − 30 = 20`; "left" even names the operation). It is
labelled `refuse` only because today's reader has `buys` in its *gain* lexeme set and
would commit `50 + 30 = 80`. A mature reader **should solve it to 20** — at which
point its honest label is `solve`, not `refuse`.

**Graduation protocol.** When a *general* mechanism teaches the reader to read a
category correctly — validated on `train_sample` + the category, with `wrong=0`
preserved (the §3/§8 guardrails) — those cases **graduate** `refuse → solve`, and
committing their gold becomes a *win*. Graduation is the intended success path; it
does **not** contradict "never optimise `refused` down by patches" (§2), because the
bar is comprehension proven generally, not a row passed by vocab fitting.

**`spurious` reframed.** A correct-valued commit on a `refuse` case is scored
`spurious` (right value, but the case said "don't commit yet"). That is
**"solved-before-graduation" — a signal to re-examine the case for graduation, not
automatically a defect.** The **pair-consistency** check is the discriminator: a
reader that solves the twin *and* commits the confuser by the same surface cue is
surface-matching (a real tell); a reader that commits the correct gold *while still
discriminating the pair* has genuinely read it, and the case should graduate. Until a
category graduates, `spurious` stays flagged (conservative — wrong=0 first).

**No category in v1 is degenerate.** Every current confuser is a solvable coverage
target (disguised-polarity, pseudo-accumulation/fractions, multi-referent,
multi-actor-pronoun, distractor, temporal-scope, comparative-referent, unit-confuser
all have a definite gold reading). A genuinely *unanswerable* case (truly
insufficient/contradictory information) — if ever added — would be labelled
`expected: refuse` with `answer_numeric: null` and a `degenerate: true` flag, so the
two senses of "refuse" (not-yet-readable vs unanswerable) stay distinguishable.

## 3. Anti-overfitting design rules (the load-bearing part)

1. **Minimal pairs.** Every confuser ships with a near-identical *solvable* twin
   that shares its surface cues but has a different correct reading. A reader that
   passes by surface cue alone gets the pair wrong; only a reader that reads the
   *structure* separates them. Example pair:
   - solvable: *"Anna has 25 stickers. She gives 10 **more** to her collection…"* → +10
   - confuser: *"Anna has 25 stickers. She gives 10 **to a friend**…"* → −10
2. **Real-sourced, not invented templates.** Each case is a paraphrase of (or
   structurally drawn from) a real GSM8K problem; record the source id. No new
   generative template that a matcher could learn.
3. **Held-out by contract.** The corpus is a **probe, never a training-to-fit
   target.** If the reader is changed so a confuser stops misfiring, that change
   MUST be a general mechanism (e.g. completeness counting the fractions it
   currently misses), proven not to regress `train_sample`, and the confuser MUST be
   one of *many* of its category so the fix can't be memorisation.
4. **Diversity quota per category.** ≥ ~8 distinct cases per confuser category
   (below), with varied surface forms, so "passing the category" can't be one rule.
5. **Expected-behavior labelling.** Each case is labelled `expected: refuse | solve`.
   Most confusers are `refuse` (the reader can't yet read them and must not guess) —
   so the corpus is primarily a *refusal-correctness* test. The `refuse` label is
   **maturity-relative**, not a claim of unanswerability: every confuser carries its
   true gold and graduates `refuse → solve` once a general mechanism reads its
   category (see §2.1).

## 4. Confuser categories (grounded in observed real misfires)

| category | what it trips | example (confuser) | correct | expected |
|---|---|---|---|---|
| **disguised-polarity verb** | gain verb that is actually a spend | "He buys a gift **for** 60" (coins) | −60 | refuse |
| **pseudo-accumulation / fractions** | has/gives surface over a fraction-division problem | the 0002 cable problem | (multi-step) | refuse |
| **multi-referent (H1)** | two actors, same unit | "Alice has 6. **Tom** has 2. How many does Alice have?" | 6 | refuse |
| **multi-actor pronoun (ADR-0174)** | ambiguous "he/she" antecedent | "Sam and Tom shopped. **He** bought 5 more." | (ambiguous) | refuse |
| **distractor quantity** | a number not in the computation | "for **3** days", "**25**-foot sections" | (varies) | refuse-unless-consumed |
| **temporal/question-scope (H3)** | question asks *before* a stated change | "…gave 3 away. How many **before** giving any?" | pre-change | refuse |
| **comparative-referent (H2)** | comparative binds to the non-asked actor | "**Tom** picked twice as many. How many did **Alice** pick?" | Alice's | refuse |
| **unit confuser** | different-unit quantities that look summable | "6 **boxes** and 50 **apples**" | (not a sum) | refuse |
| **genuine positive (paired twin)** | the solvable near-identical case | "She gives 10 **more**…" | +10 | solve |

Categories are derived from the actual hazards already proven live (GB-3a H1/H2/H3,
the 0002 misfire, the `buys-a-gift-for` polarity flip) — not invented.

## 5. Schema (`evals/gsm8k_math/confusers/v1/cases.jsonl`)

```json
{
  "case_id": "confuser-v1-0001",
  "question": "…",
  "answer_numeric": 15,
  "category": "pseudo-accumulation",
  "surface_trap": "buys/gives-to over a fraction-division problem",
  "expected": "refuse",
  "pair_id": "confuser-v1-0002",
  "source": "gsm8k-train-sample-v1-0002 (paraphrase)",
  "notes": "fractions 1/4, half and 25-foot sections are unconsumed -> must refuse"
}
```

Deterministic ordering; `pair_id` links minimal pairs; `source` records provenance.

## 6. The runner / how it scores (`confusers/v1/runner.py`)

For each composer under test (accumulation, multiplicative, chain), report **per
category**: `solved-correct / refused / wrong`, plus the **pair-consistency** check
(did the reader separate each minimal pair, or pass both by surface cue?). The gate:

- **`wrong == 0` across all confusers** — the hard bar (a confuser answered is a
  defect, regardless of value).
- **pair-consistency** — a pair where the reader solves the twin but *also* answers
  the confuser (same surface, wrong) is a surface-matching tell → fail.
- `refused` count is reported as the honest frontier, never optimised down by patches.

## 7. How the CP ledger uses it (the real learning value)

Run the general composers over the confusers and `record_case` by gold. The
**wrong attempts on confusers become the negative samples** the templated corpus
lacked — they push the offending `(cue, op, unit_shape)` reliabilities *down*,
which is exactly the discrimination signal CP-2b needs. Positive twins push the
genuine cues up. This is "compare and contrast" with hard negatives, not redundant
template confirmations.

## 8. Construction process (and its guardrails)

1. Start from the proven hazards + real `train_sample` misfires; paraphrase real
   problems into confuser/twin pairs.
2. Curate by hand (small, ≤ ~80 cases); no generative template.
3. Each row carries category + surface_trap + expected + source + pair_id.
4. **Guardrail:** building this corpus is *not* an invitation to grow reader vocab
   to pass it. The corpus is frozen as a probe; reader changes are judged on
   `train_sample` + pair-consistency, never on raw confuser solved-count.

## 9. Acceptance (Proposed → Accepted)

1. The corpus lands (~8/ category, all paired where applicable) with the schema +
   a runner reporting per-category `solved/refused/wrong` + pair-consistency.
2. The current composers score **`wrong = 0`** on it (they should *refuse* nearly
   all — that is the correct, honest result today).
3. A documented baseline (mostly-refuse) is recorded; future reader work is measured
   against it as a **regression gate** (`wrong` may never rise) and a generalisation
   signal (`solved` may rise only via general mechanisms that hold on `train_sample`).

## 10. Non-goals / explicit guardrails

- **Not** a coverage lane; **not** scored by solved-count.
- **Not** to be expanded with templates or used to tune cue vocab reactively.
- **Not** a replacement for `train_sample` as the real-capability measure — it is the
  *refusal-correctness* complement to it.

See [[thesis-decoding-not-generating]],
`feedback-synthetic-corpus-overfitting-trap`, ADR-0178 (GB-3a hazards),
ADR-0177 (CP ledger consumes the hard negatives).