core/docs/decisions/ADR-0119.7-sealed-gsm8k-test.md

ADR-0119.7 — Sealed GSM8K Test Set as gsm8k_math Holdout

Status: Accepted Date: 2026-05-23 Author: CORE agents + reviewers Depends on: ADR-0105 (sealed holdout encryption), ADR-0114, ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.3

---

Context

Phase 5.7 of ADR-0119. Closes the last substrate sub-phase before ADR-0120 (first expert promotion contract) becomes feasible.

ADR-0114 §Phase 5 designated the real GSM8K test set as a sealed holdout — never read during development. This ADR ships that seal: the 1,319 GSM8K test cases are encrypted to the recipient established under ADR-0119.1 and live on disk at evals/gsm8k_math/holdouts/v1/cases.jsonl.age. Plaintext never touches disk; the seal is one-way until a release event signed by a reviewer opens the lane.

This ADR also surfaces the first honest measurement of CORE against real GSM8K:

Cases	Correct	Wrong	Refused
1,319	0	0	1,319

The parser grammar covers zero GSM8K test problems today. That is the truthful gap. wrong == 0 discipline holds against the external corpus — CORE refuses, never confabulates. ADR-0114a Obligation #4 holds without modification.

---

Decision

Source

Hugging Face openai/gsm8k (config "main", test split, 1,319 rows). Loaded once at seal time via datasets.load_dataset. The plaintext is held in memory only during encryption — never written to disk in the working tree.

Schema (per case)

id              "gsm8k-test-NNNN"  (zero-padded)
problem         the GSM8K question, verbatim
expected_answer the integer or float after "#### N" in the answer field
                (commas stripped; e.g. "2,125" → 2125)
expected_unit   ""  (sentinel — runner skips unit-comparison; gates on
                     answer value alone)

GSM8K answers carry no unit (they're pure numbers), so the lane runner (ADR-0119.3) was extended in this PR with the rule:

"An empty expected_unit means the case carries no unit-level expectation; the runner skips the unit comparison and grades on answer value alone."

This is a minimal extension to ADR-0119.3's outcome rules; the wrong == 0 discipline is preserved.

Sealing

scripts/seal_gsm8k_test.py performs the conversion + encryption:

1. Loads GSM8K test split via datasets
2. Strips the worked-solution portion of each answer; keeps only the final numeric (after ####)
3. Builds the in-memory JSONL plaintext
4. Reads the recipient key from docs/holdout_recipients.txt (established under ADR-0119.1 — same recipient as fab_control; single repo-level key)
5. Encrypts via pyrage (X25519 age recipient)
6. Writes the ciphertext to evals/gsm8k_math/holdouts/v1/cases.jsonl.age (only the ciphertext ever exists on disk)

The script refuses to overwrite the test-seal path with a train-derived seal (--split train writes to a sibling path with a distinct name).

Seal discipline (per ADR-0119)

The development team MUST operate blind to the sealed contents until a release event signed by a reviewer opens the lane:

• The plaintext does not appear in the working tree
• No CI workflow may set CORE_HOLDOUT_KEY
• The recipient identity is stored OUTSIDE the repo (per ADR-0119.1, at ~/.config/core/holdout_keys/repo_holdout.txt)
• Tests requiring decryption skip (not fail) without the env var
• Re-encryption / re-seal events go through their own ADR amendment

First measurement on main

Running the lane runner (ADR-0119.3) against the sealed file (decrypted under CORE_HOLDOUT_KEY):

cases_total: 1319
correct:     0
wrong:       0
refused:     1319
correct_rate: 0.0
wrong_rate:   0.0
refused_rate: 1.0
wrong_count_is_zero: True
overall_pass: True

This is the load-bearing first number. It says exactly how far CORE's grammar is from GSM8K coverage. ADR-0120 (first expert promotion contract) will require a minimum correct_rate — and that threshold will tell us how much parser expansion is required before the lane can promote mathematics_logic to expert.

---

ADR-0114a obligation roll-up after Phase 5.7

#	Obligation	Status
1	Sealed-holdout discipline	✓ fab_control (0119.1) + gsm8k_math (this ADR)
2	OOD surface variation	✓ ADR-0118a
3	Replay-equal trace	✓ ADR-0117
4	Typed refusal + wrong==0	✓ ADR-0116 + 0119.3; holds against real GSM8K
5	Reasoning-isolation perturbation suite	✓ ADR-0125
6	Compositional-depth curve harness	✓ ADR-0119.6
7	Frontier-baseline comparison	✓ ADR-0119.4
8	Adversarial generation; wrong==0	✓ ADR-0119.5
9	Determinism	✓
10	Operation provenance via pack	✓ ADR-0116

All 10 obligations now discharged on main for the gsm8k_math lane. ADR-0120 can compose them into the first expert promotion contract.

---

Invariants

adr_0119_7_sealed_file_present_and_age_formatted

evals/gsm8k_math/holdouts/v1/cases.jsonl.age exists, is non-empty, and starts with the age-encryption.org/ header.

adr_0119_7_no_plaintext_companion_files

No sibling plaintext files (cases.jsonl, cases_plaintext.jsonl, cases-train.jsonl, etc.) exist in the holdout directory. The seal is one-way: only ciphertext on disk.

adr_0119_7_seal_decrypts_to_documented_schema

Decrypting with the registered identity yields ≥ 1,000 cases, each matching the documented schema (id, problem, expected_answer, expected_unit=="").

adr_0119_7_runner_holds_wrong_zero_against_real_gsm8k

Running the lane runner against the decrypted sealed file produces wrong == 0. CORE refuses what it cannot grammar-handle; it does not confabulate. ADR-0114a Obligation #4 holds against the external corpus.

adr_0119_7_skip_without_key

Tests that require decryption SKIP (do not fail) when CORE_HOLDOUT_KEY is unset. Matches ADR-0105 dev-mode discipline; CI runs do not need the key.

---

Acceptance evidence

• evals/gsm8k_math/holdouts/v1/cases.jsonl.age (420,486 bytes) encrypted to the ADR-0119.1 recipient
• scripts/seal_gsm8k_test.py reproduces the seal from datasets.load_dataset("openai/gsm8k", "main")["test"]
• evals/gsm8k_math/runner.py extended with empty-unit semantics: empty expected_unit skips unit-comparison
• tests/test_adr_0119_7_sealed_gsm8k.py (6 cases) green with CORE_HOLDOUT_KEY set; 3 pass + 3 skip without
• First measurement recorded above
• ADR linked from docs/decisions/README.md

---

Consequences

• Phase 5 is now substrate-complete. All 8 sub-phases (5.1..5.8) have landed or are in flight; the only remaining piece for ADR-0120 is the numeric threshold-setting work (depth-curve ε, minimum correct_rate, etc.) — that's a single ADR's job, not a multi-phase arc.
• The first honest CORE-vs-real-GSM8K number is on the books. 0/1319 correct is a real measurement, not a placeholder. Future parser-expansion work has a concrete target — every lift moves that number.
• wrong == 0 is now a discharged invariant against an external test set. A future parser regression that introduces confabulation will be caught by ADR-0119.7's runner-against-sealed test — the lane gate refuses the regression at promotion time.
• The seal is one-way under standard operation. No CI workflow reads CORE_HOLDOUT_KEY; only an explicit release event (with the env var set) decrypts the test. The standard development loop stays blind to the contents.

---

Out of scope

• The expert ledger tier definition and promotion contract (ADR-0120) — composes ADR-0114a obligations with numeric thresholds.
• Parser-grammar expansion to handle rate/comparison/percentage GSM8K problems. Future ADR(s); each expansion is its own scope-bounded unit.
• A second sealed corpus (e.g. MATH, AIME) — would be a new lane under a future ADR.
• Multi-reviewer threshold signing (currently single-recipient seal; open candidate frontier item from ADR-0105).
• Auto-re-sealing on dataset updates. The current ADR seals one snapshot; if GSM8K is updated upstream a future re-seal ADR documents the change.