core/docs/sessions/2026-05-29-comprehension-and-overfitting-discipline.md

# Session 2026-05-29 (pt. 2) — Comprehension chaining, and the overfitting course-correction

**Status:** paused (clean). Continues
[SESSION-2026-05-29 — the multi-step build arc](./SESSION-2026-05-29-multistep-build-arc.md).
**Headline:** Reconciled a wave of remote/operator work, built the cue-precision
ledger + its measurement, shipped the first real *comprehension* reading
(single-referent accumulation), then — prompted by a timely caution — proved that
the synthetic corpus was over-rewarding surface matching, tore down the overfit
work, and built a **confuser corpus** that scores refusal instead of flips.

`serving stayed 3/47/0 byte-identical the entire session.`

---

## TL;DR

1. **Reconciled the remote/ChatGPT extraction work** — integrated EX-1/EX-4/EX-5
   into one coherent extractor (#455), merged the GB-1/GB-2 audit (#450), fixed a
   stale decimal test, closed the superseded/off-brief PRs, and verified+merged the
   parallel operator PRs (CP-1 #458, Track-B scale #459, Track-C EX-3 deferral #460).
2. **GB-3a referent guard** (#456) — the mandated lookback proved the GB-2 hazards
   H1/H2/H3 were *live*; clause-scoped the composer so multi-clause/referent sums
   refuse. The ADR-0174 multi-actor hazard's defensive fix, finally built.
3. **CP-2a cue-precision training + measurement** (#461, + the function-word unit
   filter) — and the load-bearing finding: **no cue is reliable yet** (every pattern
   floors at ~0.0), so CP-2b (trust) is blocked on *candidate generation*, not the
   ledger. Cue-precision and structure are coupled; structure comes first.
4. **GB-3b.1 accumulation** (#464 scope, #465 impl) — the first cross-clause
   comprehension reading (`Sam has 14. He buys 9 more.` → 23). Practice additive
   lane **0 → 55 correct, 0 new wrong**.
5. **The course-correction** — GB-3b.2 (multi-change + vocab growth) reached 96/150
   synthetic but **1/50 on real GSM8K, +1 wrong** (the 0002 cable/fraction problem
   read as accumulation → 996). Recognised as overfitting, **torn down unshipped**,
   lesson recorded in memory.
6. **Confuser corpus** (#468 spec, #471 corpus) — a discrimination probe scored the
   opposite way: `wrong → 0` + pair-consistency, not flip-count. Baseline surfaced
   **7 real defects + 4 surface-match tells** the templated lane had hidden.

---

## The arc, and why each step happened

### Reconciliation (the cost of a contested working tree)

The session opened reviewing remote ChatGPT work. The four EX PRs each rewrote the
same `extract.py` off `main`, so they conflicted pairwise — integration, not merge.
EX-3 was **deferred** (its greedy multi-word unit regresses GB-2 and, on the redo,
hits a *second* trap — postmodifier adjectives like `25 years old`). EX-4's "unblocks
0024" claim was a fabricated-input overclaim; its own audit (#450) admitted 0024
stays blocked. Lesson reinforced: **design against the real corpus, not paraphrases.**

A mid-session hazard: multiple operators (and Claude) ran `git` in the *same*
working directory, which silently wiped uncommitted work. Recovered everything
(it was all in PRs); adopted **dedicated worktrees** for the rest of the session.

### The measurement that set the route

CP-2a trained the CP-1 ledger over 200 sealed cases and reported per-pattern
reliability. Every `(cue, op, unit_shape)` floored at ~0.0 — the blunt search's
readings are almost always wrong vs gold, so the conservative floor correctly trusts
nothing. **This is the microscope working:** it said the lever is not "trust good
cues" (there are none) but "make the readings less crude" → GB-3b structure first.

### The first real comprehension flip — and its honest ceiling

GB-3b.1 reads single-referent accumulation: anchor on the actor's quantity, apply a
grounded `±M` per change clause (`buys`/`more` → +, `gives…to`/`eats` → −), refuse on
a new named actor (the H1 hazard) or ambiguous polarity. Practice additive: **0 → 55
correct, 0 new wrong.** Honest calibration recorded at the time: those 55 are
*curated* cases; `train_sample` (real GSM8K) is the hard bar (48/50 multi-step mixed).

### The course-correction (the most important part)

Pushing GB-3b.2 (multi-change + reactive verb-vocab) reached 96/150 — but the
generalisation check told the truth: **real GSM8K moved 1/50, with a new wrong.**
`train_sample-0002` ("buys 1000 feet… splits into 25-foot sections… gives 1/4 away…")
was read as `buys…gives` accumulation → **996** (gold 15). The synthetic corpus was
rewarding surface-cue matching. Per the user's caution, GB-3b.2 was **torn down**,
and the lesson written to memory (`feedback-synthetic-corpus-overfitting-trap`):

> Positive+negative samples are the ledger's fuel — but only from a *general* reader
> on a *diverse* corpus. Correct-because-genuinely-read = signal;
> correct-because-the-rule-was-fit-to-it = noise. Track-B's templates have no hard
> negatives, so they can't teach refusal and they tempt fitting the reader to them.

### The confuser corpus (the cure)

Built the missing half: ~30 hand-curated, real-sourced cases across the proven
misfire categories (disguised-polarity, pseudo-accumulation/fractions, multi-referent,
multi-actor-pronoun, distractor-quantity, temporal-scope, comparative-referent,
unit-confuser) + genuine-positive minimal-pair twins. Scored **opposite** to a
coverage lane: the bar is `wrong → 0` (answering a confuser is a defect regardless
of value) + **pair-consistency** (solving a twin but answering its confuser = a
surface-matching tell). Baseline: `7 solved / 15 refused / 7 WRONG / 1 spurious`,
4 pair-tells. The 7 wrong are the sealed-composer defects the synthetic flips hid —
now named and **pinned as a no-regression gate** (never reactively patched).

---

## Shipped this session (all serving `3/47/0` byte-identical, lane-SHA 8/8)

| PR | What |
|----|------|
| #455 | Reconcile EX-1/EX-4/EX-5 into one extractor (+ stale-test fix) |
| #450 | GB-1/GB-2 lookback audit (merged) |
| #456 | GB-3a clause-scoped referent guard (H1/H2/H3 refuse) |
| #458 | CP-1 cue-precision ledger substrate (inert) |
| #459 | Track-B: 150 additive practice cases |
| #460 | Track-C: EX-3 second-deferral pin |
| #461 | CP-2a ledger training + measurement + function-word unit filter |
| #464 / #465 | GB-3b scope + GB-3b.1 accumulation (practice 0→55) |
| #468 / #471 | Confuser corpus spec + corpus v1 (baseline: 7 defects surfaced) |

**Torn down unshipped:** GB-3b.2 (overfit multi-change/vocab).
**New substrate on main:** `core/reliability_gate/`, `generate/cue_precision/`,
`generate/derivation/{accumulate,compose,clauses,…}`, `evals/gsm8k_math/{practice,confusers}/`.

## The honest frontier (next, when resumed)

`train_sample` (real GSM8K) is the only capability metric; the synthetic lanes are
mechanism-demos. The confuser baseline names the defects, and the fixes are
**general mechanisms, not reactive patches**, each validated on `train_sample` +
the probe (`wrong` must drop, never rise):

- **extraction completeness** — the gate must *see* the fractions/`25-foot`/distractor
  it currently misses, so pseudo-accumulation & distractor cases refuse (the same
  lever behind 0002→996). Highest leverage.
- **question-time reading** (temporal-scope / H3).
- **referent binding** (comparative-referent / H2, multi-referent).
- **CP-2b** only after the above give the ledger reliable cues to trust.

## Discipline notes (durable)

- Serving wrong=0 is sacrosanct; all new readers are *sealed* until a Phase-5
  ratification. Progress shows as practice `correct` rising with `wrong` at 0.
- Synthetic flip-counts are not capability. Measure on real GSM8K; reward refusal.
- Dedicated worktrees for concurrent work; never two operators in one checkout.