core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
core/docs/adr/ADR-0105-sealed-holdout-encryption.md
Shay 54e6bfc0d0
docs: reorganize docs landscape 
Implements the 4-phase documentation reorganization master plan.

- Consolidation: Merged brief/, handoff/, planning/, and decisions/ into briefs/, handoffs/, plans/, and adr/ respectively (101 ADRs relocated)
- Root Cleanup: Relocated HANDOFF-gpt55-*.md and key top-level docs (runtime_contracts.md, etc.) to canonical folders. Added superseded alerts.
- Indices & Navigation: Created docs/README.md navigation document, docs/sessions/README.md index, docs/adr/README.md index
- Note: Also includes prior commit adding ADR-0200+ corpus hygiene governance (ADR-0225, dependency map, backfilled cross-references)
2026-06-30 16:59:36 -07:00

1.8 KiB
Raw Blame History

ADR-0105 — Sealed Holdout Encryption via age

Status: Accepted (2026-05-22)

Context

Eval holdouts exist to measure generalization beyond public and development splits. Plaintext holdouts inside the repository violate the intended trust boundary because:

  • case content is inspectable by contributors and automation,
  • eval leakage becomes irreversible once committed,
  • downstream tooling can accidentally consume holdout content.

Prior ADRs established SHA-pinned eval provenance and curriculum ratification, but the holdout layer remained scaffolded.

Decision

CORE adopts recipient-based age encryption for sealed holdouts.

Implementation requirements:

  1. Holdouts are committed as *.age ciphertext files.
  2. Decryption identities are supplied via CORE_HOLDOUT_KEY.
  3. If an identity is explicitly supplied, decryption failures are fail-closed.
  4. Plaintext fallback is permitted only for local development when no key is configured.
  5. Decrypted content must remain memory-only and never be written back into the repository working tree.
  6. Holdout sealing uses recipient-only encryption via pyrage.

Consequences

Positive:

  • reduces accidental eval leakage,
  • preserves aggregate-only scoring semantics,
  • allows public repository structure without exposing hidden eval content,
  • keeps holdout management deterministic and scriptable.

Negative:

  • contributors now require explicit identities for sealed evaluation,
  • CI workflows must manage holdout identities securely,
  • local plaintext workflows become transitional-only.

Acceptance Gates

  • tests/test_holdout_encryption.py passes.
  • scripts/seal_holdouts.py --dry-run discovers seal targets correctly.
  • Wrong identities fail closed.
  • Dev fallback works only when no key is configured.
  • Existing holdouts are resealed as .age artifacts.