Lands the honest GSM8K iteration instrument and the first verifiable
reader, both measured on real held-out data + the sealed test.
THE INSTRUMENT (the durable win):
- evals/gsm8k_math/holdout_dev/v1: 500 real GSM8K cases CORE was NOT
built against. The 50-case train_sample is overfit and hid a sealed
wrong=0 breach for weeks. This lane is the un-gameable dev signal.
- test_holdout_dev_lane.py: wrong==0 floor (forever) + baseline snapshot.
THE READER (a safe pattern, not yet a sealed lift):
- generate/derivation/ratio_chain.py: structurally-forced, verifiable
chained-ratio reader. Refuse-preferring.
- Measured: held-out 500 -> 1 correct / 0 wrong; SEALED 1,319 ->
0 correct / 0 WRONG. It holds the prime directive (zero confab) but
its held-out hit did not reproduce on the sealed set, so it is a
proof-of-concept of the verifiable-reader pattern, not a real
capability gain. Documented as such; not oversold.
EVIDENCE:
- docs/analysis/real-gsm8k-capability-measurement-2026-06-04.md: the
honest finding (real capability 0%, composer 17% wrong on held-out,
no separating gate) + why verifiable readers are the path.
- Regression: 851 passed, 0 failed. train_sample wrong=0.
wrong=0 confirmed on the sealed 1,319 before this touched serving.
6e0d114726