core/evals/gsm8k_math/holdout_dev
Shay cd0df2019f test(reader-arc): pin deterministic tune/measure split BEFORE grammar work
AMENDMENT 3 (PR #76): the reader grammar family is developed against the TUNE
split only; increment deltas are measured on the disjoint MEASURE split only.
Split = pure function of case id (measure iff sha256(id) even, else tune), so
disjointness is enforced by construction and cannot drift. Fixed 2026-07-18,
never re-seeded. Sizes: tune=261, measure=239. This commit lands BEFORE any
grammar-tuning commit — the git history is the proof the split was pinned in
advance (criterion 3). wrong=0 is verified across the full 500 regardless of
split; only the coverage delta is attributed to the measure split.
2026-07-18 16:39:04 -07:00
..
v1 test(reader-arc): pin deterministic tune/measure split BEFORE grammar work 2026-07-18 16:39:04 -07:00
__init__.py feat(eval): held-out dev lane — the honest iteration metric (real capability = 0) 2026-06-04 02:30:42 -07:00