test(reader-arc): pin deterministic tune/measure split BEFORE grammar work
AMENDMENT 3 (PR #76): the reader grammar family is developed against the TUNE split only; increment deltas are measured on the disjoint MEASURE split only. Split = pure function of case id (measure iff sha256(id) even, else tune), so disjointness is enforced by construction and cannot drift. Fixed 2026-07-18, never re-seeded. Sizes: tune=261, measure=239. This commit lands BEFORE any grammar-tuning commit — the git history is the proof the split was pinned in advance (criterion 3). wrong=0 is verified across the full 500 regardless of split; only the coverage delta is attributed to the measure split.
This commit is contained in:
parent
9136c1819b
commit
cd0df2019f
1 changed files with 25 additions and 0 deletions
25
evals/gsm8k_math/holdout_dev/v1/split.py
Normal file
25
evals/gsm8k_math/holdout_dev/v1/split.py
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
"""Deterministic tune / measure split of the official holdout_dev/v1 (ADR-0250 reader arc).
|
||||
|
||||
Pinned **before any grammar work** (compare increment plan §5, AMENDMENT 3): the reader grammar
|
||||
family is developed against the **tune** split only; every increment's coverage delta is measured on
|
||||
the disjoint **measure** split only. The assignment is a pure function of the case id, so it is
|
||||
reproducible and cannot drift — disjointness is enforced by construction, not by discipline.
|
||||
|
||||
Rule (fixed 2026-07-18, never re-seeded): a case is ``measure`` iff ``sha256(id)`` is even, else
|
||||
``tune`` (≈ 50 / 50). `wrong=0` is verified across the **full 500** regardless of split; only the
|
||||
*coverage delta* is attributed to the measure split.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
|
||||
__all__ = ["split_of", "TUNE", "MEASURE"]
|
||||
|
||||
TUNE = "tune"
|
||||
MEASURE = "measure"
|
||||
|
||||
|
||||
def split_of(case_id: str) -> str:
|
||||
"""Return ``"tune"`` or ``"measure"`` for ``case_id`` — deterministic, id-only."""
|
||||
digest = int(hashlib.sha256(case_id.encode("utf-8")).hexdigest(), 16)
|
||||
return MEASURE if digest % 2 == 0 else TUNE
|
||||
Loading…
Reference in a new issue