feat(evals): ADR-0105 — sealed holdout encryption via age (#108 )

* feat(evals): add pyrage dependency

* feat(evals): add sealed holdout path resolution

* feat(evals): implement sealed holdout decryption

* feat(evals): add sealed holdout CLI

* test(evals): add sealed holdout encryption tests

* docs(decisions): add ADR-0105 sealed holdout encryption

* feat(evals): route holdout split through sealed decryptor

* docs(decisions): add ADR-0105 index entry

* chore: restore project description

* fix(evals): use pyrage Identity.from_str and pin curriculum SHA

- holdout_runner: pyrage exposes Identity.from_str, not from_file; parse
  identity file by line and pass list[Identity] into decrypt(). Restores
  PR 108's sealed-holdout test suite to green.
- verify_lane_shas: realign curriculum_loop_closure pin with the actual
  deterministic runner output (carryover from PR 107).

2026-05-22 10:09:43 -07:00

1.8 KiB

Raw Blame History

ADR-0105 — Sealed Holdout Encryption via age

Status: Accepted (2026-05-22)

Context

Eval holdouts exist to measure generalization beyond public and development splits. Plaintext holdouts inside the repository violate the intended trust boundary because:

case content is inspectable by contributors and automation,
eval leakage becomes irreversible once committed,
downstream tooling can accidentally consume holdout content.

Prior ADRs established SHA-pinned eval provenance and curriculum ratification, but the holdout layer remained scaffolded.

Decision

CORE adopts recipient-based age encryption for sealed holdouts.

Implementation requirements:

Holdouts are committed as *.age ciphertext files.
Decryption identities are supplied via CORE_HOLDOUT_KEY.
If an identity is explicitly supplied, decryption failures are fail-closed.
Plaintext fallback is permitted only for local development when no key is configured.
Decrypted content must remain memory-only and never be written back into the repository working tree.
Holdout sealing uses recipient-only encryption via pyrage.

Consequences

Positive:

reduces accidental eval leakage,
preserves aggregate-only scoring semantics,
allows public repository structure without exposing hidden eval content,
keeps holdout management deterministic and scriptable.

Negative:

contributors now require explicit identities for sealed evaluation,
CI workflows must manage holdout identities securely,
local plaintext workflows become transitional-only.

Acceptance Gates

tests/test_holdout_encryption.py passes.
scripts/seal_holdouts.py --dry-run discovers seal targets correctly.
Wrong identities fail closed.
Dev fallback works only when no key is configured.
Existing holdouts are resealed as .age artifacts.

1.8 KiB Raw Blame History