Lands the honest GSM8K iteration instrument and the first verifiable
reader, both measured on real held-out data + the sealed test.
THE INSTRUMENT (the durable win):
- evals/gsm8k_math/holdout_dev/v1: 500 real GSM8K cases CORE was NOT
built against. The 50-case train_sample is overfit and hid a sealed
wrong=0 breach for weeks. This lane is the un-gameable dev signal.
- test_holdout_dev_lane.py: wrong==0 floor (forever) + baseline snapshot.
THE READER (a safe pattern, not yet a sealed lift):
- generate/derivation/ratio_chain.py: structurally-forced, verifiable
chained-ratio reader. Refuse-preferring.
- Measured: held-out 500 -> 1 correct / 0 wrong; SEALED 1,319 ->
0 correct / 0 WRONG. It holds the prime directive (zero confab) but
its held-out hit did not reproduce on the sealed set, so it is a
proof-of-concept of the verifiable-reader pattern, not a real
capability gain. Documented as such; not oversold.
EVIDENCE:
- docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md: the
honest finding (real capability 0%, composer 17% wrong on held-out,
no separating gate) + why verifiable readers are the path.
- Regression: 851 passed, 0 failed. train_sample wrong=0.
wrong=0 confirmed on the sealed 1,319 before this touched serving.
The 2026-06-04 sealed-breach post-mortem proved the 50-case train_sample has ZERO
predictive validity (its 4 "correct" are overfit; they hid a 5-wrong sealed breach).
This adds the instrument we never had: 500 real GSM8K cases CORE was NOT built on —
the train split minus the 50 train_sample, deterministic sha256(question) sort.
Same scorer as train_sample + the sealed lane, so the three are directly comparable:
train_sample(50): 4/0/46 holdout_dev(500): 0/0/500 sealed test(1319): 0/0/1319
Real GSM8K capability is 0%. The 4 train "correct" generalize to NOT ONE of 500
held-out cases. wrong=0 holds (refuses, never confabulates).
- evals/gsm8k_math/holdout_dev/v1/: cases.jsonl (500), runner, report (0/0/500), README.
- tests/test_holdout_dev_lane.py: floor (wrong==0, forever) + baseline snapshot (0/500).
Discipline: iterate here (open, large enough to resist trivial overfit); the sealed
test stays the final arbiter. wrong=0 is the floor; correct rising is the goal;
"refuse everything" is the FAILING baseline to beat, not a pass. Non-serving (eval only).
