38872f825a
The 1,319 GSM8K test cases are now sealed at evals/gsm8k_math/holdouts/v1/cases.jsonl.age, age-encrypted to the ADR-0119.1 recipient. Plaintext never touched disk in the working tree; only ciphertext is committed. First honest CORE-vs-real-GSM8K measurement cases_total: 1319 correct: 0 wrong: 0 ← ADR-0114a Obligation #4 holds against external corpus refused: 1319 overall_pass: True Zero confabulation. Parser refuses what it can't grammar-handle; the "wrong == 0" discipline survives the move from CORE-original cases to a real public benchmark. The 0/1319 correct rate is the truthful gap that ADR-0120's threshold work will quantify. What landed scripts/seal_gsm8k_test.py - Loads GSM8K via datasets.load_dataset("openai/gsm8k", "main") - Strips worked-solution prose; extracts final-answer integer/float after "####" (handles "2,125" → 2125 thousands-separator) - Reads recipient from docs/holdout_recipients.txt (single repo key per ADR-0119.1) - Encrypts via pyrage; writes only ciphertext - Refuses to overwrite test path with train-derived seal evals/gsm8k_math/runner.py - Empty expected_unit (sentinel) skips unit-comparison; grades on answer value alone. Required because GSM8K answers carry no unit structurally. wrong-zero discipline preserved. tests/test_adr_0119_7_sealed_gsm8k.py — 6 invariants: 1. sealed file present + age-formatted 2. no plaintext companion files (sibling-leak guard) 3. decrypted JSONL matches documented schema 4. runner against decrypted suite produces wrong==0 5. tests skip (not fail) when CORE_HOLDOUT_KEY unset 6. case ids match "gsm8k-test-NNNN" pattern Defensive gitignore: plaintext patterns under evals/gsm8k_math/holdouts/v1/ are explicitly excluded. ADR-0114a obligation roll-up 10/10 discharged for the gsm8k_math lane: #1 ✓ sealed-holdout (fab_control + GSM8K test) #2..#10 ✓ as before Phase 5 status: 5.1..5.7 done; 5.8 in flight (PR #149). After 5.8 merges, ADR-0120 (first expert promotion contract) becomes feasible. Test plan - pytest tests/test_adr_0119_7_sealed_gsm8k.py with CORE_HOLDOUT_KEY → 6/6 - pytest without CORE_HOLDOUT_KEY → 3 pass + 3 skip - core test --suite smoke -q → 67/67 - CLAIMS.md regenerated (no diff) - HF token NEVER in repo (saved at ~/.cache/huggingface/token, mode 600) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
38 lines
855 B
Text
38 lines
855 B
Text
__pycache__/
|
|
.pytest_cache/
|
|
.hypothesis/
|
|
*.pyc
|
|
.DS_Store
|
|
|
|
*.egg-info/
|
|
traces/
|
|
.formation_cache/
|
|
|
|
core-rs/target/
|
|
core-rs/Cargo.lock
|
|
|
|
uv.lock
|
|
|
|
# Environment secrets — never commit real keys
|
|
.env
|
|
.env.local
|
|
.env.*.local
|
|
|
|
# Benchmark report output
|
|
reports/
|
|
frontier_wave1.json
|
|
|
|
# Claude Code local session artifacts (per-developer, never tracked)
|
|
.claude/
|
|
|
|
# Runnable audit-passed showcases (ADR-0112 + ADR-0113) are generated on
|
|
# demand from the signed claim + on-disk lane result files. The inputs
|
|
# are committed; the renders are not.
|
|
evals/audit_passed/
|
|
evals/expert_demos/
|
|
|
|
# ADR-0119.7 — GSM8K sealed-holdout discipline. Only the encrypted
|
|
# .age file may be tracked; plaintext companions must never land in git.
|
|
evals/gsm8k_math/holdouts/v1/cases.jsonl
|
|
evals/gsm8k_math/holdouts/v1/cases_plaintext.jsonl
|
|
evals/gsm8k_math/holdouts/v1/cases-train*
|