core/docs/adr/ADR-0105-sealed-holdout-encryption.md

ADR-0105 — Sealed Holdout Encryption via age

Status: Accepted (2026-05-22)

Context

Eval holdouts exist to measure generalization beyond public and development splits. Plaintext holdouts inside the repository violate the intended trust boundary because:

• case content is inspectable by contributors and automation,
• eval leakage becomes irreversible once committed,
• downstream tooling can accidentally consume holdout content.

Prior ADRs established SHA-pinned eval provenance and curriculum ratification, but the holdout layer remained scaffolded.

Decision

CORE adopts recipient-based age encryption for sealed holdouts.

Implementation requirements:

1. Holdouts are committed as *.age ciphertext files.
2. Decryption identities are supplied via CORE_HOLDOUT_KEY.
3. If an identity is explicitly supplied, decryption failures are fail-closed.
4. Plaintext fallback is permitted only for local development when no key is configured.
5. Decrypted content must remain memory-only and never be written back into the repository working tree.
6. Holdout sealing uses recipient-only encryption via pyrage.

Consequences

Positive:

• reduces accidental eval leakage,
• preserves aggregate-only scoring semantics,
• allows public repository structure without exposing hidden eval content,
• keeps holdout management deterministic and scriptable.

Negative:

• contributors now require explicit identities for sealed evaluation,
• CI workflows must manage holdout identities securely,
• local plaintext workflows become transitional-only.

Acceptance Gates

• tests/test_holdout_encryption.py passes.
• scripts/seal_holdouts.py --dry-run discovers seal targets correctly.
• Wrong identities fail closed.
• Dev fallback works only when no key is configured.
• Existing holdouts are resealed as .age artifacts.