docs(adr-0163-f2): confuser corpus spec — a discrimination probe, not a coverage target (#468)
Follow-on to ADR-0163 §F that corrects the metric exposed by the overfitting finding (96/150 synthetic flips vs 1/50 real; the 0002 cable/fraction problem read as accumulation -> 996). Specs a small, hand-curated, real-sourced set of hard negatives + near-miss confusers (disguised-polarity verbs, pseudo-accumulation fractions, multi-referent/multi-actor, distractor quantities, temporal/question- scope, comparative-referent, unit confusers) with minimal-pair construction. Scored the opposite way from a coverage lane: success = wrong=0 on confusers + pair-consistency + honest refusal frontier, NEVER solved-count. Held-out by contract (not a training-to-fit target); the CP ledger consumes the wrong attempts as the hard negatives the templated corpus lacked. Guardrails forbid reactive vocab growth and template expansion.
This commit is contained in:
Shay
GitHub
parent
6a4d356ce9
commit
31d9cb429d
1 changed files with 158 additions and 0 deletions
158
docs/decisions/ADR-0163-F2-confuser-corpus-spec.md
Normal file
158
docs/decisions/ADR-0163-F2-confuser-corpus-spec.md
Normal file
|
|
@ -0,0 +1,158 @@
|
||||||
|
# ADR-0163-F2 — Confuser Corpus: a discrimination probe, not a coverage target
|
||||||
|
|
||||||
|
**Status:** Proposed (spec only — no code). Follow-on to ADR-0163 §F (the Track-B
|
||||||
|
additive practice scale). Corrects the *metric*, not just the data.
|
||||||
|
|
||||||
|
> **One line.** A small, curated set of **hard negatives and near-miss confusers**
|
||||||
|
> whose purpose is to measure whether the reader **refuses what it cannot yet read**
|
||||||
|
> — success is `wrong = 0` on the confusers, *not* a higher solved count.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why this exists (the failure it fixes)
|
||||||
|
|
||||||
|
ADR-0163 §F added 150 templated additive practice cases. GB-3b flipped 55–96 of
|
||||||
|
them — but on **real GSM8K (`train_sample`) accumulation fired on only 2/50, 1
|
||||||
|
correct / 1 wrong**. The synthetic flips did not generalise. The canonical misfire:
|
||||||
|
|
||||||
|
> `train_sample-0002`: *"Jan buys 1000 feet of cable. She splits it into 25-foot
|
||||||
|
> sections. She gives 1/4 to a friend. She puts half in storage. How much does she
|
||||||
|
> keep?"* (gold 15) — read as `buys 1000 … gives … to` → **996**.
|
||||||
|
|
||||||
|
The templated corpus has two defects that *cause* overfitting (memory:
|
||||||
|
`feedback-synthetic-corpus-overfitting-trap`):
|
||||||
|
|
||||||
|
1. **Redundancy** — the 46 gain cases are one template × 46 noun/number swaps, so a
|
||||||
|
surface-cue matcher "passes" without comprehension and the CP ledger fills with
|
||||||
|
tautological confirmations.
|
||||||
|
2. **No hard negatives** — there is no problem that *looks* like accumulation but
|
||||||
|
isn't (a fraction problem, a disguised-polarity verb, a second actor). The
|
||||||
|
confusers are exactly what teach a reader **where to refuse**, and a corpus of
|
||||||
|
clean templates contains none.
|
||||||
|
|
||||||
|
This spec builds the missing half: the cases that make being wrong *informative*.
|
||||||
|
|
||||||
|
## 2. The reframe — what "success" means here
|
||||||
|
|
||||||
|
This corpus is a **discrimination probe**, scored the opposite way from a coverage
|
||||||
|
lane:
|
||||||
|
|
||||||
|
| metric | meaning | bar |
|
||||||
|
|---|---|---|
|
||||||
|
| **`wrong` on confusers** | the reader committed an answer a confuser was built to trip | **must be 0** |
|
||||||
|
| **`refused` on confusers** | the reader declined (the correct response to a not-yet-readable shape) | expected-high, honest frontier |
|
||||||
|
| **`solved` on the paired positives** | the reader genuinely read the near-identical solvable case | real-capability signal |
|
||||||
|
|
||||||
|
The headline number is **wrong=0**, never "how many solved." A change that solves
|
||||||
|
more confusers is only progress if it does so by a **general mechanism** that also
|
||||||
|
keeps `wrong=0` — and is validated on the held-out real lane, not by passing these
|
||||||
|
specific rows.
|
||||||
|
|
||||||
|
## 3. Anti-overfitting design rules (the load-bearing part)
|
||||||
|
|
||||||
|
1. **Minimal pairs.** Every confuser ships with a near-identical *solvable* twin
|
||||||
|
that shares its surface cues but has a different correct reading. A reader that
|
||||||
|
passes by surface cue alone gets the pair wrong; only a reader that reads the
|
||||||
|
*structure* separates them. Example pair:
|
||||||
|
- solvable: *"Anna has 25 stickers. She gives 10 **more** to her collection…"* → +10
|
||||||
|
- confuser: *"Anna has 25 stickers. She gives 10 **to a friend**…"* → −10
|
||||||
|
2. **Real-sourced, not invented templates.** Each case is a paraphrase of (or
|
||||||
|
structurally drawn from) a real GSM8K problem; record the source id. No new
|
||||||
|
generative template that a matcher could learn.
|
||||||
|
3. **Held-out by contract.** The corpus is a **probe, never a training-to-fit
|
||||||
|
target.** If the reader is changed so a confuser stops misfiring, that change
|
||||||
|
MUST be a general mechanism (e.g. completeness counting the fractions it
|
||||||
|
currently misses), proven not to regress `train_sample`, and the confuser MUST be
|
||||||
|
one of *many* of its category so the fix can't be memorisation.
|
||||||
|
4. **Diversity quota per category.** ≥ ~8 distinct cases per confuser category
|
||||||
|
(below), with varied surface forms, so "passing the category" can't be one rule.
|
||||||
|
5. **Expected-behavior labelling.** Each case is labelled `expected: refuse | solve`.
|
||||||
|
Most confusers are `refuse` (the reader can't yet read them and must not guess) —
|
||||||
|
so the corpus is primarily a *refusal-correctness* test.
|
||||||
|
|
||||||
|
## 4. Confuser categories (grounded in observed real misfires)
|
||||||
|
|
||||||
|
| category | what it trips | example (confuser) | correct | expected |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **disguised-polarity verb** | gain verb that is actually a spend | "He buys a gift **for** 60" (coins) | −60 | refuse |
|
||||||
|
| **pseudo-accumulation / fractions** | has/gives surface over a fraction-division problem | the 0002 cable problem | (multi-step) | refuse |
|
||||||
|
| **multi-referent (H1)** | two actors, same unit | "Alice has 6. **Tom** has 2. How many does Alice have?" | 6 | refuse |
|
||||||
|
| **multi-actor pronoun (ADR-0174)** | ambiguous "he/she" antecedent | "Sam and Tom shopped. **He** bought 5 more." | (ambiguous) | refuse |
|
||||||
|
| **distractor quantity** | a number not in the computation | "for **3** days", "**25**-foot sections" | (varies) | refuse-unless-consumed |
|
||||||
|
| **temporal/question-scope (H3)** | question asks *before* a stated change | "…gave 3 away. How many **before** giving any?" | pre-change | refuse |
|
||||||
|
| **comparative-referent (H2)** | comparative binds to the non-asked actor | "**Tom** picked twice as many. How many did **Alice** pick?" | Alice's | refuse |
|
||||||
|
| **unit confuser** | different-unit quantities that look summable | "6 **boxes** and 50 **apples**" | (not a sum) | refuse |
|
||||||
|
| **genuine positive (paired twin)** | the solvable near-identical case | "She gives 10 **more**…" | +10 | solve |
|
||||||
|
|
||||||
|
Categories are derived from the actual hazards already proven live (GB-3a H1/H2/H3,
|
||||||
|
the 0002 misfire, the `buys-a-gift-for` polarity flip) — not invented.
|
||||||
|
|
||||||
|
## 5. Schema (`evals/gsm8k_math/confusers/v1/cases.jsonl`)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"case_id": "confuser-v1-0001",
|
||||||
|
"question": "…",
|
||||||
|
"answer_numeric": 15,
|
||||||
|
"category": "pseudo-accumulation",
|
||||||
|
"surface_trap": "buys/gives-to over a fraction-division problem",
|
||||||
|
"expected": "refuse",
|
||||||
|
"pair_id": "confuser-v1-0002",
|
||||||
|
"source": "gsm8k-train-sample-v1-0002 (paraphrase)",
|
||||||
|
"notes": "fractions 1/4, half and 25-foot sections are unconsumed -> must refuse"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Deterministic ordering; `pair_id` links minimal pairs; `source` records provenance.
|
||||||
|
|
||||||
|
## 6. The runner / how it scores (`confusers/v1/runner.py`)
|
||||||
|
|
||||||
|
For each composer under test (accumulation, multiplicative, chain), report **per
|
||||||
|
category**: `solved-correct / refused / wrong`, plus the **pair-consistency** check
|
||||||
|
(did the reader separate each minimal pair, or pass both by surface cue?). The gate:
|
||||||
|
|
||||||
|
- **`wrong == 0` across all confusers** — the hard bar (a confuser answered is a
|
||||||
|
defect, regardless of value).
|
||||||
|
- **pair-consistency** — a pair where the reader solves the twin but *also* answers
|
||||||
|
the confuser (same surface, wrong) is a surface-matching tell → fail.
|
||||||
|
- `refused` count is reported as the honest frontier, never optimised down by patches.
|
||||||
|
|
||||||
|
## 7. How the CP ledger uses it (the real learning value)
|
||||||
|
|
||||||
|
Run the general composers over the confusers and `record_case` by gold. The
|
||||||
|
**wrong attempts on confusers become the negative samples** the templated corpus
|
||||||
|
lacked — they push the offending `(cue, op, unit_shape)` reliabilities *down*,
|
||||||
|
which is exactly the discrimination signal CP-2b needs. Positive twins push the
|
||||||
|
genuine cues up. This is "compare and contrast" with hard negatives, not redundant
|
||||||
|
template confirmations.
|
||||||
|
|
||||||
|
## 8. Construction process (and its guardrails)
|
||||||
|
|
||||||
|
1. Start from the proven hazards + real `train_sample` misfires; paraphrase real
|
||||||
|
problems into confuser/twin pairs.
|
||||||
|
2. Curate by hand (small, ≤ ~80 cases); no generative template.
|
||||||
|
3. Each row carries category + surface_trap + expected + source + pair_id.
|
||||||
|
4. **Guardrail:** building this corpus is *not* an invitation to grow reader vocab
|
||||||
|
to pass it. The corpus is frozen as a probe; reader changes are judged on
|
||||||
|
`train_sample` + pair-consistency, never on raw confuser solved-count.
|
||||||
|
|
||||||
|
## 9. Acceptance (Proposed → Accepted)
|
||||||
|
|
||||||
|
1. The corpus lands (~8/ category, all paired where applicable) with the schema +
|
||||||
|
a runner reporting per-category `solved/refused/wrong` + pair-consistency.
|
||||||
|
2. The current composers score **`wrong = 0`** on it (they should *refuse* nearly
|
||||||
|
all — that is the correct, honest result today).
|
||||||
|
3. A documented baseline (mostly-refuse) is recorded; future reader work is measured
|
||||||
|
against it as a **regression gate** (`wrong` may never rise) and a generalisation
|
||||||
|
signal (`solved` may rise only via general mechanisms that hold on `train_sample`).
|
||||||
|
|
||||||
|
## 10. Non-goals / explicit guardrails
|
||||||
|
|
||||||
|
- **Not** a coverage lane; **not** scored by solved-count.
|
||||||
|
- **Not** to be expanded with templates or used to tune cue vocab reactively.
|
||||||
|
- **Not** a replacement for `train_sample` as the real-capability measure — it is the
|
||||||
|
*refusal-correctness* complement to it.
|
||||||
|
|
||||||
|
See [[thesis-decoding-not-generating]],
|
||||||
|
`feedback-synthetic-corpus-overfitting-trap`, ADR-0178 (GB-3a hazards),
|
||||||
|
ADR-0177 (CP ledger consumes the hard negatives).
|
||||||
Loading…
Reference in a new issue