docs: session wrap-up (comprehension chaining + overfitting discipline) + CLAUDE.md substrate (#472)
- New session record SESSION-2026-05-29-comprehension-and-overfitting-discipline.md:
the reconciliation wave, CP-1/CP-2a (no cue reliable yet -> structure-first),
GB-3a referent guard, GB-3b.1 accumulation (practice 0->55), the GB-3b.2
overfitting course-correction (96/150 synthetic vs 1/50 real -> torn down), and
the confuser corpus (baseline: 7 defects surfaced). Serving 3/47/0 throughout.
- Lands the prior dangling SESSION-2026-05-29-multistep-build-arc.md (was untracked).
- CLAUDE.md "Current Key Modules": adds the sealed GSM8K comprehension substrate
(core/reliability_gate, generate/derivation, generate/cue_precision,
evals/gsm8k_math {train_sample,practice,confusers}, the lane-SHA gate).
This commit is contained in:
Shay
GitHub
parent
a67b475e59
commit
d95f8a99e0
3 changed files with 238 additions and 0 deletions
|
|
@ -104,6 +104,14 @@ runtime path. Vault recall is exact and deterministic.
|
|||
- `calibration/*` — bounded replay-based calibration.
|
||||
- `docs/runtime_contracts.md` — response, telemetry, memory, identity, and testing contracts.
|
||||
|
||||
### GSM8K math comprehension substrate (sealed; serving stays `3/47/0` until ratified)
|
||||
|
||||
- `core/reliability_gate/` — calibrated-learning ledger + gate (ADR-0175): `ClassTally` counts, `conservative_floor` (one-sided Wilson, N_MIN=10), θ ceilings.
|
||||
- `generate/derivation/` — the comprehension composer: `extract.py` (lexeme quantity extraction, EX-1/4/5 + function-word unit filter), `clauses.py` (GB-1 segmentation), `compose.py` (GB-2a list-sum + GB-3a clause-scoped referent guard), `accumulate.py` (GB-3b.1 single-referent gain/loss chaining), `multistep.py`/`search.py` (bounded search), `verify.py` (the wrong=0 self-verification gate: grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness).
|
||||
- `generate/cue_precision/` — `(cue, op, unit_shape)` reliability ledger + trainer (ADR-0177 CP-1/CP-2a); inert (consulted by no serving/gate path yet).
|
||||
- `evals/gsm8k_math/` — `train_sample/` (real GSM8K, the capability metric), `practice/` (sealed attempt-and-eliminate lane + ADR-0163-F additive set), `confusers/` (ADR-0163-F2 discrimination probe — scored by `wrong→0` + pair-consistency, NOT flip-count).
|
||||
- `scripts/verify_lane_shas.py`, `scripts/generate_claims.py --check` — the serving-frozen gate (pinned eval-lane SHAs + `CLAIMS.md`).
|
||||
|
||||
## Efficiency and Performance Doctrine
|
||||
|
||||
Performance is part of correctness for this project because slow feedback hides
|
||||
|
|
|
|||
|
|
@ -0,0 +1,137 @@
|
|||
# Session 2026-05-29 (pt. 2) — Comprehension chaining, and the overfitting course-correction
|
||||
|
||||
**Status:** paused (clean). Continues
|
||||
[SESSION-2026-05-29 — the multi-step build arc](./SESSION-2026-05-29-multistep-build-arc.md).
|
||||
**Headline:** Reconciled a wave of remote/operator work, built the cue-precision
|
||||
ledger + its measurement, shipped the first real *comprehension* reading
|
||||
(single-referent accumulation), then — prompted by a timely caution — proved that
|
||||
the synthetic corpus was over-rewarding surface matching, tore down the overfit
|
||||
work, and built a **confuser corpus** that scores refusal instead of flips.
|
||||
|
||||
`serving stayed 3/47/0 byte-identical the entire session.`
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. **Reconciled the remote/ChatGPT extraction work** — integrated EX-1/EX-4/EX-5
|
||||
into one coherent extractor (#455), merged the GB-1/GB-2 audit (#450), fixed a
|
||||
stale decimal test, closed the superseded/off-brief PRs, and verified+merged the
|
||||
parallel operator PRs (CP-1 #458, Track-B scale #459, Track-C EX-3 deferral #460).
|
||||
2. **GB-3a referent guard** (#456) — the mandated lookback proved the GB-2 hazards
|
||||
H1/H2/H3 were *live*; clause-scoped the composer so multi-clause/referent sums
|
||||
refuse. The ADR-0174 multi-actor hazard's defensive fix, finally built.
|
||||
3. **CP-2a cue-precision training + measurement** (#461, + the function-word unit
|
||||
filter) — and the load-bearing finding: **no cue is reliable yet** (every pattern
|
||||
floors at ~0.0), so CP-2b (trust) is blocked on *candidate generation*, not the
|
||||
ledger. Cue-precision and structure are coupled; structure comes first.
|
||||
4. **GB-3b.1 accumulation** (#464 scope, #465 impl) — the first cross-clause
|
||||
comprehension reading (`Sam has 14. He buys 9 more.` → 23). Practice additive
|
||||
lane **0 → 55 correct, 0 new wrong**.
|
||||
5. **The course-correction** — GB-3b.2 (multi-change + vocab growth) reached 96/150
|
||||
synthetic but **1/50 on real GSM8K, +1 wrong** (the 0002 cable/fraction problem
|
||||
read as accumulation → 996). Recognised as overfitting, **torn down unshipped**,
|
||||
lesson recorded in memory.
|
||||
6. **Confuser corpus** (#468 spec, #471 corpus) — a discrimination probe scored the
|
||||
opposite way: `wrong → 0` + pair-consistency, not flip-count. Baseline surfaced
|
||||
**7 real defects + 4 surface-match tells** the templated lane had hidden.
|
||||
|
||||
---
|
||||
|
||||
## The arc, and why each step happened
|
||||
|
||||
### Reconciliation (the cost of a contested working tree)
|
||||
|
||||
The session opened reviewing remote ChatGPT work. The four EX PRs each rewrote the
|
||||
same `extract.py` off `main`, so they conflicted pairwise — integration, not merge.
|
||||
EX-3 was **deferred** (its greedy multi-word unit regresses GB-2 and, on the redo,
|
||||
hits a *second* trap — postmodifier adjectives like `25 years old`). EX-4's "unblocks
|
||||
0024" claim was a fabricated-input overclaim; its own audit (#450) admitted 0024
|
||||
stays blocked. Lesson reinforced: **design against the real corpus, not paraphrases.**
|
||||
|
||||
A mid-session hazard: multiple operators (and Claude) ran `git` in the *same*
|
||||
working directory, which silently wiped uncommitted work. Recovered everything
|
||||
(it was all in PRs); adopted **dedicated worktrees** for the rest of the session.
|
||||
|
||||
### The measurement that set the route
|
||||
|
||||
CP-2a trained the CP-1 ledger over 200 sealed cases and reported per-pattern
|
||||
reliability. Every `(cue, op, unit_shape)` floored at ~0.0 — the blunt search's
|
||||
readings are almost always wrong vs gold, so the conservative floor correctly trusts
|
||||
nothing. **This is the microscope working:** it said the lever is not "trust good
|
||||
cues" (there are none) but "make the readings less crude" → GB-3b structure first.
|
||||
|
||||
### The first real comprehension flip — and its honest ceiling
|
||||
|
||||
GB-3b.1 reads single-referent accumulation: anchor on the actor's quantity, apply a
|
||||
grounded `±M` per change clause (`buys`/`more` → +, `gives…to`/`eats` → −), refuse on
|
||||
a new named actor (the H1 hazard) or ambiguous polarity. Practice additive: **0 → 55
|
||||
correct, 0 new wrong.** Honest calibration recorded at the time: those 55 are
|
||||
*curated* cases; `train_sample` (real GSM8K) is the hard bar (48/50 multi-step mixed).
|
||||
|
||||
### The course-correction (the most important part)
|
||||
|
||||
Pushing GB-3b.2 (multi-change + reactive verb-vocab) reached 96/150 — but the
|
||||
generalisation check told the truth: **real GSM8K moved 1/50, with a new wrong.**
|
||||
`train_sample-0002` ("buys 1000 feet… splits into 25-foot sections… gives 1/4 away…")
|
||||
was read as `buys…gives` accumulation → **996** (gold 15). The synthetic corpus was
|
||||
rewarding surface-cue matching. Per the user's caution, GB-3b.2 was **torn down**,
|
||||
and the lesson written to memory (`feedback-synthetic-corpus-overfitting-trap`):
|
||||
|
||||
> Positive+negative samples are the ledger's fuel — but only from a *general* reader
|
||||
> on a *diverse* corpus. Correct-because-genuinely-read = signal;
|
||||
> correct-because-the-rule-was-fit-to-it = noise. Track-B's templates have no hard
|
||||
> negatives, so they can't teach refusal and they tempt fitting the reader to them.
|
||||
|
||||
### The confuser corpus (the cure)
|
||||
|
||||
Built the missing half: ~30 hand-curated, real-sourced cases across the proven
|
||||
misfire categories (disguised-polarity, pseudo-accumulation/fractions, multi-referent,
|
||||
multi-actor-pronoun, distractor-quantity, temporal-scope, comparative-referent,
|
||||
unit-confuser) + genuine-positive minimal-pair twins. Scored **opposite** to a
|
||||
coverage lane: the bar is `wrong → 0` (answering a confuser is a defect regardless
|
||||
of value) + **pair-consistency** (solving a twin but answering its confuser = a
|
||||
surface-matching tell). Baseline: `7 solved / 15 refused / 7 WRONG / 1 spurious`,
|
||||
4 pair-tells. The 7 wrong are the sealed-composer defects the synthetic flips hid —
|
||||
now named and **pinned as a no-regression gate** (never reactively patched).
|
||||
|
||||
---
|
||||
|
||||
## Shipped this session (all serving `3/47/0` byte-identical, lane-SHA 8/8)
|
||||
|
||||
| PR | What |
|
||||
|----|------|
|
||||
| #455 | Reconcile EX-1/EX-4/EX-5 into one extractor (+ stale-test fix) |
|
||||
| #450 | GB-1/GB-2 lookback audit (merged) |
|
||||
| #456 | GB-3a clause-scoped referent guard (H1/H2/H3 refuse) |
|
||||
| #458 | CP-1 cue-precision ledger substrate (inert) |
|
||||
| #459 | Track-B: 150 additive practice cases |
|
||||
| #460 | Track-C: EX-3 second-deferral pin |
|
||||
| #461 | CP-2a ledger training + measurement + function-word unit filter |
|
||||
| #464 / #465 | GB-3b scope + GB-3b.1 accumulation (practice 0→55) |
|
||||
| #468 / #471 | Confuser corpus spec + corpus v1 (baseline: 7 defects surfaced) |
|
||||
|
||||
**Torn down unshipped:** GB-3b.2 (overfit multi-change/vocab).
|
||||
**New substrate on main:** `core/reliability_gate/`, `generate/cue_precision/`,
|
||||
`generate/derivation/{accumulate,compose,clauses,…}`, `evals/gsm8k_math/{practice,confusers}/`.
|
||||
|
||||
## The honest frontier (next, when resumed)
|
||||
|
||||
`train_sample` (real GSM8K) is the only capability metric; the synthetic lanes are
|
||||
mechanism-demos. The confuser baseline names the defects, and the fixes are
|
||||
**general mechanisms, not reactive patches**, each validated on `train_sample` +
|
||||
the probe (`wrong` must drop, never rise):
|
||||
|
||||
- **extraction completeness** — the gate must *see* the fractions/`25-foot`/distractor
|
||||
it currently misses, so pseudo-accumulation & distractor cases refuse (the same
|
||||
lever behind 0002→996). Highest leverage.
|
||||
- **question-time reading** (temporal-scope / H3).
|
||||
- **referent binding** (comparative-referent / H2, multi-referent).
|
||||
- **CP-2b** only after the above give the ledger reliable cues to trust.
|
||||
|
||||
## Discipline notes (durable)
|
||||
|
||||
- Serving wrong=0 is sacrosanct; all new readers are *sealed* until a Phase-5
|
||||
ratification. Progress shows as practice `correct` rising with `wrong` at 0.
|
||||
- Synthetic flip-counts are not capability. Measure on real GSM8K; reward refusal.
|
||||
- Dedicated worktrees for concurrent work; never two operators in one checkout.
|
||||
93
docs/sessions/SESSION-2026-05-29-multistep-build-arc.md
Normal file
93
docs/sessions/SESSION-2026-05-29-multistep-build-arc.md
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
# Session 2026-05-29 — The Multi-Step Build Arc: from "another matcher" to a comprehension composer
|
||||
|
||||
**Status:** In progress (EX-2 landing) ✓ through GB-2
|
||||
**Companion:** [SESSION-2026-05-28 — risk/reward learning architecture](./SESSION-2026-05-28-risk-reward-learning-architecture.md) (the design discussion that opened this arc)
|
||||
**Headline:** Took the GSM8K work from "build another recognizer per shape" to a **safe, inspectable, comprehension-guided multi-step solver** — and, just as importantly, proved *with measurement* exactly where coverage is gated (extraction richness × cue precision × scale), so the remaining roadmap is honest rather than hopeful.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Across one long session we:
|
||||
|
||||
1. **Shipped ADR-0175 Phases 1–3b** (the two-regime calibrated-learning substrate) and **Phase 5a** (retired the inert reader; net −1,038 LOC; serving byte-identical at `3/47/0`).
|
||||
2. **Did a lookback review** of the 5-PR stack (no live hazards; recorded drift); landed it.
|
||||
3. **Built the multi-step machine** end-to-end: MS-1 (question-targeting) → MS-2 (chain model + comparatives) → MS-3 (target-guided search) → GB-1 (clause segmentation) → GB-2 (list-sum-then-scale) → EX-2 (decimal grounding).
|
||||
4. **Scoped the full remaining roadmap** as ADRs 0176–0179.
|
||||
5. Kept **`wrong=0` on serving (`3/47/0`) byte-identical the entire time** — every attempt-and-eliminate ran in the sealed practice lane.
|
||||
|
||||
The arc's intellectual payoff: each phase's *measurement* told us the next lever, and they converged on a clean dependency chain rather than a guessing game.
|
||||
|
||||
---
|
||||
|
||||
## The converged architecture (what we now believe)
|
||||
|
||||
```
|
||||
ADR-0175 Two-regime calibrated learning serving wrong=0 | sealed practice attempt+eliminate
|
||||
ADR-0176 Multi-step grounded search extract → chain → self-verify gate (grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness)
|
||||
ADR-0178 Comprehension-guided structure read clause-by-clause; structure-from-reading, not enumeration
|
||||
ADR-0177 Cue-precision learning trust gate + search prune (closes "self-verification necessary-not-sufficient")
|
||||
ADR-0179 Extraction richness feed the composer real quantities (the unblock prerequisite)
|
||||
× SCALE (ADR-0163 §F) to compound
|
||||
```
|
||||
|
||||
**The honest dependency chain (the session's central finding):**
|
||||
> extraction richness (0179) → unblocks the built structure (MS-3/GB-2) → produces gold-matching chains → feeds cue-precision (0177) → coverage; all gated by `wrong=0` and amplified by scale.
|
||||
|
||||
Coverage does not come from more search — it comes from *reading the structure* and *feeding it real quantities*, with the self-verification gate keeping every attempt honest.
|
||||
|
||||
---
|
||||
|
||||
## What shipped to `main`
|
||||
|
||||
| PR | What | Result |
|
||||
|---|---|---|
|
||||
| #430 | Phase 5a — retire inert reader | net −1,038 LOC; `3/47/0` byte-identical |
|
||||
| #432 | Phase 1 — reliability ledger + gate (`core/reliability_gate/`) | inert substrate; invariants #3/#4 proven |
|
||||
| #433 | Phase 2 — sealed practice lane (`evals/gsm8k_math/practice/v1/`) | diagnosis: 35 skill / 12 knowledge / 0 ambiguity |
|
||||
| #434 | Phase 3a — self-verification gate (`generate/derivation/`) | invariant #2 proven (spurious refuses) |
|
||||
| #435 | Phase 3b — multiplicative search | first attempts live; the "necessary-not-sufficient" finding |
|
||||
| #436 | self-verification **completeness** clause | practice wrongs 9→2 |
|
||||
| #438 | `en_core_comparatives_v1` pack | irreducible world-facts (twice→×2, half→×0.5) |
|
||||
| #439 | MS-1 — question-targeting (`Target`) | search pruning + stopping signal |
|
||||
| #440 | MS-2 — chain model (text + comparative operands) | 0024/0033 mixed chains self-verify |
|
||||
| #441 | MS-3 — target-guided bounded search | flips 0021; honest 4/9/37 practice |
|
||||
| #444 | GB-1 — clause segmentation + clause-local sub-derivation | per-clause structure |
|
||||
| #445 | GB-2 — same-unit list-sum-then-scale | clean-case capability; extraction-gated on practice |
|
||||
| #447 | EX-2 — bare-decimal grounding (shared primitive) | **0003 unblocked** (+1 practice flip); serving byte-identical |
|
||||
|
||||
**Proposed (doc) PRs:** #437 (ADR-0176), #442 (ADR-0177), #443 (ADR-0178), #446 (ADR-0179).
|
||||
|
||||
---
|
||||
|
||||
## The findings the microscope produced (in order)
|
||||
|
||||
1. **Per-shape matchers can't compound** — 79% need multiplication, 0% single-step; phrasings are unbounded. (Killed the "another matcher" path.)
|
||||
2. **Self-verification is necessary but not sufficient** (3b) — 9/13 self-verified attempts were wrong; the gap is *which composition*, not *can we multiply*.
|
||||
3. **Completeness** catches multi-step-incomplete attempts (correct first-steps mistaken for answers) — wrongs 9→2.
|
||||
4. **Two gaps**: A = cue→op precision; B = compositional structure. Gap B dominates.
|
||||
5. **Structure-from-reading** (Gap B) resolves the rich-search-vs-uniqueness tension: every gold case fits a sequential clause read.
|
||||
6. **Cue-precision is bottlenecked** — it can only learn from gold-matching chains (~4/50 on blunt shapes); it's the trust substrate + prune, not the coverage unlock.
|
||||
7. **Extraction is the wall** — non-uniform units, decimals, word-numbers, multi-word units block the built machinery. (EX-2 proved it: fixing decimal grounding flipped 0003.)
|
||||
|
||||
Each finding came with a deterministic trace — the microscope, exactly as the design predicted.
|
||||
|
||||
---
|
||||
|
||||
## wrong=0 discipline held throughout
|
||||
|
||||
- Serving `3/47/0` byte-identical across all 13 PRs — verified at each landing (and EX-2, the one shared-primitive touch, gated by the pinned lane-SHA check).
|
||||
- Every attempt-and-eliminate ran in the **sealed practice lane**; practice "wrongs" are gold-checked eliminations (learning signal), never served.
|
||||
- Invariants proven by failing-under-violation tests at each phase (#1 seal, #2 spurious-refusal, #3 determinism, #4 no-self-authorization, completeness, target-match).
|
||||
- A 5-PR **lookback review** caught and recorded drift before it compounded.
|
||||
|
||||
---
|
||||
|
||||
## Next levers (honest, in order)
|
||||
|
||||
1. **Finish EX (extraction richness)** — EX-1 word-numbers, EX-3 multi-word units, **EX-4 list-unit inheritance** (unblocks 0024's same-unit list). EX-2 (decimals) just proved the pattern.
|
||||
2. **Cue-precision (ADR-0177)** — once richer extraction produces more gold-matching chains, the ledger has signal to learn from.
|
||||
3. **Scale (ADR-0163 §F)** — volume is what makes the learning *compound*; 50 cases is mechanism-demonstration.
|
||||
4. **CI-hosted contemplation loops** (recorded) — run the sealed practice loop on CI minutes, mobile-triggered, once the learning config stabilizes.
|
||||
|
||||
The eval headline (`3/47/0`) moves once extraction + structure + cue-precision compose on enough volume — and the substrate to make that lift *trustworthy* is now built and proven.
|
||||
Loading…
Reference in a new issue