core/docs/decisions/ADR-0119.7-sealed-gsm8k-test.md

# ADR-0119.7 — Sealed GSM8K Test Set as gsm8k_math Holdout

**Status:** Accepted
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Depends on:** ADR-0105 (sealed holdout encryption), ADR-0114, ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.3

---

## Context

Phase 5.7 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). Closes
the last substrate sub-phase before ADR-0120 (first `expert` promotion
contract) becomes feasible.

ADR-0114 §Phase 5 designated the real GSM8K test set as a **sealed
holdout** — never read during development. This ADR ships that seal:
the 1,319 GSM8K test cases are encrypted to the recipient established
under ADR-0119.1 and live on disk at
`evals/gsm8k_math/holdouts/v1/cases.jsonl.age`. Plaintext never
touches disk; the seal is one-way until a release event signed by a
reviewer opens the lane.

This ADR also surfaces the **first honest measurement of CORE against
real GSM8K**:

| Cases | Correct | Wrong | Refused |
|---|---|---|---|
| 1,319 | 0 | 0 | 1,319 |

The parser grammar covers zero GSM8K test problems today. That is
the truthful gap. **`wrong == 0` discipline holds against the
external corpus** — CORE refuses, never confabulates. ADR-0114a
Obligation #4 holds without modification.

---

## Decision

### Source

Hugging Face `openai/gsm8k` (config `"main"`, `test` split, 1,319 rows).
Loaded once at seal time via `datasets.load_dataset`. The plaintext is
held in memory only during encryption — never written to disk in the
working tree.

### Schema (per case)

```text
id              "gsm8k-test-NNNN"  (zero-padded)
problem         the GSM8K question, verbatim
expected_answer the integer or float after "#### N" in the answer field
                (commas stripped; e.g. "2,125" → 2125)
expected_unit   ""  (sentinel — runner skips unit-comparison; gates on
                     answer value alone)
```

GSM8K answers carry no unit (they're pure numbers), so the lane runner
(ADR-0119.3) was extended in this PR with the rule:

> "An empty `expected_unit` means the case carries no unit-level
> expectation; the runner skips the unit comparison and grades on
> answer value alone."

This is a minimal extension to ADR-0119.3's outcome rules; the
`wrong == 0` discipline is preserved.

### Sealing

`scripts/seal_gsm8k_test.py` performs the conversion + encryption:

1. Loads GSM8K test split via `datasets`
2. Strips the worked-solution portion of each answer; keeps only the
   final numeric (after `####`)
3. Builds the in-memory JSONL plaintext
4. Reads the recipient key from `docs/holdout_recipients.txt`
   (established under ADR-0119.1 — same recipient as fab_control;
   single repo-level key)
5. Encrypts via `pyrage` (X25519 age recipient)
6. Writes the ciphertext to
   `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` (only the ciphertext
   ever exists on disk)

The script refuses to overwrite the test-seal path with a
train-derived seal (`--split train` writes to a sibling path with a
distinct name).

### Seal discipline (per ADR-0119)

The development team **MUST** operate blind to the sealed contents
until a release event signed by a reviewer opens the lane:

- The plaintext does not appear in the working tree
- No CI workflow may set `CORE_HOLDOUT_KEY`
- The recipient identity is stored OUTSIDE the repo (per
  ADR-0119.1, at `~/.config/core/holdout_keys/repo_holdout.txt`)
- Tests requiring decryption skip (not fail) without the env var
- Re-encryption / re-seal events go through their own ADR amendment

### First measurement on main

Running the lane runner (ADR-0119.3) against the sealed file
(decrypted under `CORE_HOLDOUT_KEY`):

```text
cases_total: 1319
correct:     0
wrong:       0
refused:     1319
correct_rate: 0.0
wrong_rate:   0.0
refused_rate: 1.0
wrong_count_is_zero: True
overall_pass: True
```

**This is the load-bearing first number.** It says exactly how far
CORE's grammar is from GSM8K coverage. ADR-0120 (first `expert`
promotion contract) will require a minimum `correct_rate` — and that
threshold will tell us how much parser expansion is required before
the lane can promote `mathematics_logic` to `expert`.

---

## ADR-0114a obligation roll-up after Phase 5.7

| # | Obligation | Status |
|---|---|---|
| 1 | Sealed-holdout discipline | ✓ fab_control (0119.1) + **gsm8k_math (this ADR)** |
| 2 | OOD surface variation | ✓ ADR-0118a |
| 3 | Replay-equal trace | ✓ ADR-0117 |
| 4 | Typed refusal + wrong==0 | ✓ ADR-0116 + 0119.3; **holds against real GSM8K** |
| 5 | Reasoning-isolation perturbation suite | ✓ ADR-0125 |
| 6 | Compositional-depth curve harness | ✓ ADR-0119.6 |
| 7 | Frontier-baseline comparison | ✓ ADR-0119.4 |
| 8 | Adversarial generation; wrong==0 | ✓ ADR-0119.5 |
| 9 | Determinism | ✓ |
| 10 | Operation provenance via pack | ✓ ADR-0116 |

**All 10 obligations now discharged on main for the gsm8k_math lane.**
ADR-0120 can compose them into the first `expert` promotion contract.

---

## Invariants

### `adr_0119_7_sealed_file_present_and_age_formatted`

`evals/gsm8k_math/holdouts/v1/cases.jsonl.age` exists, is non-empty,
and starts with the `age-encryption.org/` header.

### `adr_0119_7_no_plaintext_companion_files`

No sibling plaintext files (`cases.jsonl`, `cases_plaintext.jsonl`,
`cases-train.jsonl`, etc.) exist in the holdout directory. The seal
is one-way: only ciphertext on disk.

### `adr_0119_7_seal_decrypts_to_documented_schema`

Decrypting with the registered identity yields ≥ 1,000 cases, each
matching the documented schema (id, problem, expected_answer,
expected_unit=="").

### `adr_0119_7_runner_holds_wrong_zero_against_real_gsm8k`

Running the lane runner against the decrypted sealed file produces
`wrong == 0`. CORE refuses what it cannot grammar-handle; it does
not confabulate. ADR-0114a Obligation #4 holds against the external
corpus.

### `adr_0119_7_skip_without_key`

Tests that require decryption SKIP (do not fail) when
`CORE_HOLDOUT_KEY` is unset. Matches ADR-0105 dev-mode discipline;
CI runs do not need the key.

---

## Acceptance evidence

- `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` (420,486 bytes)
  encrypted to the ADR-0119.1 recipient
- `scripts/seal_gsm8k_test.py` reproduces the seal from
  `datasets.load_dataset("openai/gsm8k", "main")["test"]`
- `evals/gsm8k_math/runner.py` extended with empty-unit
  semantics: empty `expected_unit` skips unit-comparison
- `tests/test_adr_0119_7_sealed_gsm8k.py` (6 cases) green with
  `CORE_HOLDOUT_KEY` set; 3 pass + 3 skip without
- First measurement recorded above
- ADR linked from `docs/decisions/README.md`

---

## Consequences

- **Phase 5 is now substrate-complete.** All 8 sub-phases (5.1..5.8)
  have landed or are in flight; the only remaining piece for ADR-0120
  is the numeric threshold-setting work (depth-curve ε, minimum
  correct_rate, etc.) — that's a single ADR's job, not a multi-phase
  arc.
- **The first honest CORE-vs-real-GSM8K number is on the books.**
  0/1319 correct is a real measurement, not a placeholder. Future
  parser-expansion work has a concrete target — every lift moves
  that number.
- **`wrong == 0` is now a discharged invariant against an external
  test set.** A future parser regression that introduces
  confabulation will be caught by ADR-0119.7's runner-against-sealed
  test — the lane gate refuses the regression at promotion time.
- **The seal is one-way under standard operation.** No CI workflow
  reads `CORE_HOLDOUT_KEY`; only an explicit release event (with
  the env var set) decrypts the test. The standard development loop
  stays blind to the contents.

---

## Out of scope

- The `expert` ledger tier definition and promotion contract
  (ADR-0120) — composes ADR-0114a obligations with numeric thresholds.
- Parser-grammar expansion to handle rate/comparison/percentage GSM8K
  problems. Future ADR(s); each expansion is its own scope-bounded
  unit.
- A second sealed corpus (e.g. MATH, AIME) — would be a new lane
  under a future ADR.
- Multi-reviewer threshold signing (currently single-recipient seal;
  open candidate frontier item from ADR-0105).
- Auto-re-sealing on dataset updates. The current ADR seals one
  snapshot; if GSM8K is updated upstream a future re-seal ADR
  documents the change.