257fd4503d
* feat(evals): add pyrage dependency * feat(evals): add sealed holdout path resolution * feat(evals): implement sealed holdout decryption * feat(evals): add sealed holdout CLI * test(evals): add sealed holdout encryption tests * docs(decisions): add ADR-0105 sealed holdout encryption * feat(evals): route holdout split through sealed decryptor * docs(decisions): add ADR-0105 index entry * chore: restore project description * fix(evals): use pyrage Identity.from_str and pin curriculum SHA - holdout_runner: pyrage exposes Identity.from_str, not from_file; parse identity file by line and pass list[Identity] into decrypt(). Restores PR 108's sealed-holdout test suite to green. - verify_lane_shas: realign curriculum_loop_closure pin with the actual deterministic runner output (carryover from PR 107).
54 lines
1.8 KiB
Markdown
54 lines
1.8 KiB
Markdown
# ADR-0105 — Sealed Holdout Encryption via age
|
|
|
|
Status: Accepted (2026-05-22)
|
|
|
|
## Context
|
|
|
|
Eval holdouts exist to measure generalization beyond public and development
|
|
splits. Plaintext holdouts inside the repository violate the intended trust
|
|
boundary because:
|
|
|
|
- case content is inspectable by contributors and automation,
|
|
- eval leakage becomes irreversible once committed,
|
|
- downstream tooling can accidentally consume holdout content.
|
|
|
|
Prior ADRs established SHA-pinned eval provenance and curriculum ratification,
|
|
but the holdout layer remained scaffolded.
|
|
|
|
## Decision
|
|
|
|
CORE adopts recipient-based `age` encryption for sealed holdouts.
|
|
|
|
Implementation requirements:
|
|
|
|
1. Holdouts are committed as `*.age` ciphertext files.
|
|
2. Decryption identities are supplied via `CORE_HOLDOUT_KEY`.
|
|
3. If an identity is explicitly supplied, decryption failures are fail-closed.
|
|
4. Plaintext fallback is permitted only for local development when no key is
|
|
configured.
|
|
5. Decrypted content must remain memory-only and never be written back into the
|
|
repository working tree.
|
|
6. Holdout sealing uses recipient-only encryption via `pyrage`.
|
|
|
|
## Consequences
|
|
|
|
Positive:
|
|
|
|
- reduces accidental eval leakage,
|
|
- preserves aggregate-only scoring semantics,
|
|
- allows public repository structure without exposing hidden eval content,
|
|
- keeps holdout management deterministic and scriptable.
|
|
|
|
Negative:
|
|
|
|
- contributors now require explicit identities for sealed evaluation,
|
|
- CI workflows must manage holdout identities securely,
|
|
- local plaintext workflows become transitional-only.
|
|
|
|
## Acceptance Gates
|
|
|
|
- `tests/test_holdout_encryption.py` passes.
|
|
- `scripts/seal_holdouts.py --dry-run` discovers seal targets correctly.
|
|
- Wrong identities fail closed.
|
|
- Dev fallback works only when no key is configured.
|
|
- Existing holdouts are resealed as `.age` artifacts.
|