The 1,319 GSM8K test cases are now sealed at
evals/gsm8k_math/holdouts/v1/cases.jsonl.age, age-encrypted to the
ADR-0119.1 recipient. Plaintext never touched disk in the working
tree; only ciphertext is committed.
First honest CORE-vs-real-GSM8K measurement
cases_total: 1319
correct: 0
wrong: 0 ← ADR-0114a Obligation #4 holds against external corpus
refused: 1319
overall_pass: True
Zero confabulation. Parser refuses what it can't grammar-handle; the
"wrong == 0" discipline survives the move from CORE-original cases
to a real public benchmark. The 0/1319 correct rate is the truthful
gap that ADR-0120's threshold work will quantify.
What landed
scripts/seal_gsm8k_test.py
- Loads GSM8K via datasets.load_dataset("openai/gsm8k", "main")
- Strips worked-solution prose; extracts final-answer integer/float
after "####" (handles "2,125" → 2125 thousands-separator)
- Reads recipient from docs/holdout_recipients.txt (single repo key
per ADR-0119.1)
- Encrypts via pyrage; writes only ciphertext
- Refuses to overwrite test path with train-derived seal
evals/gsm8k_math/runner.py
- Empty expected_unit (sentinel) skips unit-comparison; grades on
answer value alone. Required because GSM8K answers carry no unit
structurally. wrong-zero discipline preserved.
tests/test_adr_0119_7_sealed_gsm8k.py — 6 invariants:
1. sealed file present + age-formatted
2. no plaintext companion files (sibling-leak guard)
3. decrypted JSONL matches documented schema
4. runner against decrypted suite produces wrong==0
5. tests skip (not fail) when CORE_HOLDOUT_KEY unset
6. case ids match "gsm8k-test-NNNN" pattern
Defensive gitignore: plaintext patterns under
evals/gsm8k_math/holdouts/v1/ are explicitly excluded.
ADR-0114a obligation roll-up
10/10 discharged for the gsm8k_math lane:
#1 ✓ sealed-holdout (fab_control + GSM8K test)
#2..#10 ✓ as before
Phase 5 status: 5.1..5.7 done; 5.8 in flight (PR #149). After 5.8
merges, ADR-0120 (first expert promotion contract) becomes
feasible.
Test plan
- pytest tests/test_adr_0119_7_sealed_gsm8k.py with CORE_HOLDOUT_KEY → 6/6
- pytest without CORE_HOLDOUT_KEY → 3 pass + 3 skip
- core test --suite smoke -q → 67/67
- CLAIMS.md regenerated (no diff)
- HF token NEVER in repo (saved at ~/.cache/huggingface/token, mode 600)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
38872f825a