54e6bfc0d0
Implements the 4-phase documentation reorganization master plan. - Consolidation: Merged brief/, handoff/, planning/, and decisions/ into briefs/, handoffs/, plans/, and adr/ respectively (101 ADRs relocated) - Root Cleanup: Relocated HANDOFF-gpt55-*.md and key top-level docs (runtime_contracts.md, etc.) to canonical folders. Added superseded alerts. - Indices & Navigation: Created docs/README.md navigation document, docs/sessions/README.md index, docs/adr/README.md index - Note: Also includes prior commit adding ADR-0200+ corpus hygiene governance (ADR-0225, dependency map, backfilled cross-references)
54 lines
1.8 KiB
Markdown
54 lines
1.8 KiB
Markdown
# ADR-0105 — Sealed Holdout Encryption via age
|
|
|
|
Status: Accepted (2026-05-22)
|
|
|
|
## Context
|
|
|
|
Eval holdouts exist to measure generalization beyond public and development
|
|
splits. Plaintext holdouts inside the repository violate the intended trust
|
|
boundary because:
|
|
|
|
- case content is inspectable by contributors and automation,
|
|
- eval leakage becomes irreversible once committed,
|
|
- downstream tooling can accidentally consume holdout content.
|
|
|
|
Prior ADRs established SHA-pinned eval provenance and curriculum ratification,
|
|
but the holdout layer remained scaffolded.
|
|
|
|
## Decision
|
|
|
|
CORE adopts recipient-based `age` encryption for sealed holdouts.
|
|
|
|
Implementation requirements:
|
|
|
|
1. Holdouts are committed as `*.age` ciphertext files.
|
|
2. Decryption identities are supplied via `CORE_HOLDOUT_KEY`.
|
|
3. If an identity is explicitly supplied, decryption failures are fail-closed.
|
|
4. Plaintext fallback is permitted only for local development when no key is
|
|
configured.
|
|
5. Decrypted content must remain memory-only and never be written back into the
|
|
repository working tree.
|
|
6. Holdout sealing uses recipient-only encryption via `pyrage`.
|
|
|
|
## Consequences
|
|
|
|
Positive:
|
|
|
|
- reduces accidental eval leakage,
|
|
- preserves aggregate-only scoring semantics,
|
|
- allows public repository structure without exposing hidden eval content,
|
|
- keeps holdout management deterministic and scriptable.
|
|
|
|
Negative:
|
|
|
|
- contributors now require explicit identities for sealed evaluation,
|
|
- CI workflows must manage holdout identities securely,
|
|
- local plaintext workflows become transitional-only.
|
|
|
|
## Acceptance Gates
|
|
|
|
- `tests/test_holdout_encryption.py` passes.
|
|
- `scripts/seal_holdouts.py --dry-run` discovers seal targets correctly.
|
|
- Wrong identities fail closed.
|
|
- Dev fallback works only when no key is configured.
|
|
- Existing holdouts are resealed as `.age` artifacts.
|