# ADR-0119.7 — Sealed GSM8K Test Set as gsm8k_math Holdout **Status:** Accepted **Date:** 2026-05-23 **Author:** CORE agents + reviewers **Depends on:** ADR-0105 (sealed holdout encryption), ADR-0114, ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.3 --- ## Context Phase 5.7 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). Closes the last substrate sub-phase before ADR-0120 (first `expert` promotion contract) becomes feasible. ADR-0114 §Phase 5 designated the real GSM8K test set as a **sealed holdout** — never read during development. This ADR ships that seal: the 1,319 GSM8K test cases are encrypted to the recipient established under ADR-0119.1 and live on disk at `evals/gsm8k_math/holdouts/v1/cases.jsonl.age`. Plaintext never touches disk; the seal is one-way until a release event signed by a reviewer opens the lane. This ADR also surfaces the **first honest measurement of CORE against real GSM8K**: | Cases | Correct | Wrong | Refused | |---|---|---|---| | 1,319 | 0 | 0 | 1,319 | The parser grammar covers zero GSM8K test problems today. That is the truthful gap. **`wrong == 0` discipline holds against the external corpus** — CORE refuses, never confabulates. ADR-0114a Obligation #4 holds without modification. --- ## Decision ### Source Hugging Face `openai/gsm8k` (config `"main"`, `test` split, 1,319 rows). Loaded once at seal time via `datasets.load_dataset`. The plaintext is held in memory only during encryption — never written to disk in the working tree. ### Schema (per case) ```text id "gsm8k-test-NNNN" (zero-padded) problem the GSM8K question, verbatim expected_answer the integer or float after "#### N" in the answer field (commas stripped; e.g. "2,125" → 2125) expected_unit "" (sentinel — runner skips unit-comparison; gates on answer value alone) ``` GSM8K answers carry no unit (they're pure numbers), so the lane runner (ADR-0119.3) was extended in this PR with the rule: > "An empty `expected_unit` means the case carries no unit-level > expectation; the runner skips the unit comparison and grades on > answer value alone." This is a minimal extension to ADR-0119.3's outcome rules; the `wrong == 0` discipline is preserved. ### Sealing `scripts/seal_gsm8k_test.py` performs the conversion + encryption: 1. Loads GSM8K test split via `datasets` 2. Strips the worked-solution portion of each answer; keeps only the final numeric (after `####`) 3. Builds the in-memory JSONL plaintext 4. Reads the recipient key from `docs/holdout_recipients.txt` (established under ADR-0119.1 — same recipient as fab_control; single repo-level key) 5. Encrypts via `pyrage` (X25519 age recipient) 6. Writes the ciphertext to `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` (only the ciphertext ever exists on disk) The script refuses to overwrite the test-seal path with a train-derived seal (`--split train` writes to a sibling path with a distinct name). ### Seal discipline (per ADR-0119) The development team **MUST** operate blind to the sealed contents until a release event signed by a reviewer opens the lane: - The plaintext does not appear in the working tree - No CI workflow may set `CORE_HOLDOUT_KEY` - The recipient identity is stored OUTSIDE the repo (per ADR-0119.1, at `~/.config/core/holdout_keys/repo_holdout.txt`) - Tests requiring decryption skip (not fail) without the env var - Re-encryption / re-seal events go through their own ADR amendment ### First measurement on main Running the lane runner (ADR-0119.3) against the sealed file (decrypted under `CORE_HOLDOUT_KEY`): ```text cases_total: 1319 correct: 0 wrong: 0 refused: 1319 correct_rate: 0.0 wrong_rate: 0.0 refused_rate: 1.0 wrong_count_is_zero: True overall_pass: True ``` **This is the load-bearing first number.** It says exactly how far CORE's grammar is from GSM8K coverage. ADR-0120 (first `expert` promotion contract) will require a minimum `correct_rate` — and that threshold will tell us how much parser expansion is required before the lane can promote `mathematics_logic` to `expert`. --- ## ADR-0114a obligation roll-up after Phase 5.7 | # | Obligation | Status | |---|---|---| | 1 | Sealed-holdout discipline | ✓ fab_control (0119.1) + **gsm8k_math (this ADR)** | | 2 | OOD surface variation | ✓ ADR-0118a | | 3 | Replay-equal trace | ✓ ADR-0117 | | 4 | Typed refusal + wrong==0 | ✓ ADR-0116 + 0119.3; **holds against real GSM8K** | | 5 | Reasoning-isolation perturbation suite | ✓ ADR-0125 | | 6 | Compositional-depth curve harness | ✓ ADR-0119.6 | | 7 | Frontier-baseline comparison | ✓ ADR-0119.4 | | 8 | Adversarial generation; wrong==0 | ✓ ADR-0119.5 | | 9 | Determinism | ✓ | | 10 | Operation provenance via pack | ✓ ADR-0116 | **All 10 obligations now discharged on main for the gsm8k_math lane.** ADR-0120 can compose them into the first `expert` promotion contract. --- ## Invariants ### `adr_0119_7_sealed_file_present_and_age_formatted` `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` exists, is non-empty, and starts with the `age-encryption.org/` header. ### `adr_0119_7_no_plaintext_companion_files` No sibling plaintext files (`cases.jsonl`, `cases_plaintext.jsonl`, `cases-train.jsonl`, etc.) exist in the holdout directory. The seal is one-way: only ciphertext on disk. ### `adr_0119_7_seal_decrypts_to_documented_schema` Decrypting with the registered identity yields ≥ 1,000 cases, each matching the documented schema (id, problem, expected_answer, expected_unit==""). ### `adr_0119_7_runner_holds_wrong_zero_against_real_gsm8k` Running the lane runner against the decrypted sealed file produces `wrong == 0`. CORE refuses what it cannot grammar-handle; it does not confabulate. ADR-0114a Obligation #4 holds against the external corpus. ### `adr_0119_7_skip_without_key` Tests that require decryption SKIP (do not fail) when `CORE_HOLDOUT_KEY` is unset. Matches ADR-0105 dev-mode discipline; CI runs do not need the key. --- ## Acceptance evidence - `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` (420,486 bytes) encrypted to the ADR-0119.1 recipient - `scripts/seal_gsm8k_test.py` reproduces the seal from `datasets.load_dataset("openai/gsm8k", "main")["test"]` - `evals/gsm8k_math/runner.py` extended with empty-unit semantics: empty `expected_unit` skips unit-comparison - `tests/test_adr_0119_7_sealed_gsm8k.py` (6 cases) green with `CORE_HOLDOUT_KEY` set; 3 pass + 3 skip without - First measurement recorded above - ADR linked from `docs/decisions/README.md` --- ## Consequences - **Phase 5 is now substrate-complete.** All 8 sub-phases (5.1..5.8) have landed or are in flight; the only remaining piece for ADR-0120 is the numeric threshold-setting work (depth-curve ε, minimum correct_rate, etc.) — that's a single ADR's job, not a multi-phase arc. - **The first honest CORE-vs-real-GSM8K number is on the books.** 0/1319 correct is a real measurement, not a placeholder. Future parser-expansion work has a concrete target — every lift moves that number. - **`wrong == 0` is now a discharged invariant against an external test set.** A future parser regression that introduces confabulation will be caught by ADR-0119.7's runner-against-sealed test — the lane gate refuses the regression at promotion time. - **The seal is one-way under standard operation.** No CI workflow reads `CORE_HOLDOUT_KEY`; only an explicit release event (with the env var set) decrypts the test. The standard development loop stays blind to the contents. --- ## Out of scope - The `expert` ledger tier definition and promotion contract (ADR-0120) — composes ADR-0114a obligations with numeric thresholds. - Parser-grammar expansion to handle rate/comparison/percentage GSM8K problems. Future ADR(s); each expansion is its own scope-bounded unit. - A second sealed corpus (e.g. MATH, AIME) — would be a new lane under a future ADR. - Multi-reviewer threshold signing (currently single-recipient seal; open candidate frontier item from ADR-0105). - Auto-re-sealing on dataset updates. The current ADR seals one snapshot; if GSM8K is updated upstream a future re-seal ADR documents the change.