core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/decisions/ADR-0119.1-sealed-holdout-fabrication-control.md

3.5 KiB
Raw Blame History

ADR-0119.1 — Seal fabrication_control Holdout (ADR-0105 Amendment)

Status: Accepted Date: 2026-05-23 Author: CORE agents + reviewers Amends: ADR-0105 Depends on: ADR-0105, ADR-0114a

Context

Under ADR-0105, CORE established recipient-based age encryption for sealed holdouts to prevent leakage. Plaintext fallback was allowed temporarily for local development.

ADR-0114a (Obligation #1) mandates that every lane migrate its holdout from plaintext fallback to an age-encrypted seal to satisfy anti-overfitting rules. This ADR implements this migration for the first lane: fabrication_control (the smallest and simplest lane).

Decision

We seal the fabrication_control holdout using the age format keyed to a dedicated public key recipient.

Configuration Details

  • Sealed Lane: fabrication_control
  • Recipient Public Key: age1zt3ud2skl4y8qt306dlux0d3g68tt2trsur64yuh7gv4dag5qg5su9jllz
  • Private Identity Location: The private identity key is expected to live outside the repository at ~/.config/core/holdout_keys/repo_holdout.txt.
  • CI / Execution Setup: A fresh CI run or local test suite obtains access to this identity by setting the CORE_HOLDOUT_KEY environment variable pointing to the absolute path of the identity file.

Actions Taken

  1. Generated a new age x25519 keypair for the repository.
  2. Encrypted the original holdout file evals/fabrication_control/cases/holdout.jsonl to evals/fabrication_control/holdouts/v1/cases.jsonl.age using the public key recipient.
  3. Removed the unencrypted evals/fabrication_control/cases/holdout.jsonl from the repository.
  4. Documented the recipient in docs/holdout_recipients.txt.
  5. Exposed run_lane in the fabrication_control runner evals/fabrication_control/runner.py to support the generic framework execution model, and updated _run_split to decrypt the holdout when split is holdout.

Invariants

  • Running fabrication_control holdout split without the environment variable CORE_HOLDOUT_KEY set raises a typed EnvironmentError and fails closed.
  • Decrypting the sealed cases.jsonl.age using the repository private key reproduces the original holdout cases byte-for-byte.

Acceptance Evidence

Accepted when:

  • The .age file exists at evals/fabrication_control/holdouts/v1/cases.jsonl.age.
  • The tests in tests/test_adr_0119_1_sealed_holdout.py pass cleanly, validating:
    • Encryption format and presence of header in cases.jsonl.age.
    • Byte-equal reproduction of holdout cases upon decryption.
    • Fail-closed behavior (EnvironmentError raised) when key is missing.
    • Successful execution and reproduction of the prior metrics (refusal_recall == 1.0, fabrication_rate == 0.0) when key is provided.

Consequences

  • The fabrication_control lane holdout is fully sealed, fulfilling Obligation #1 of ADR-0114a.
  • Plaintext leaks of fabrication_control holdout are eliminated from the current tree.

Out of Scope

  • Migrating other evaluation lanes' holdouts (reserved for subsequent ADRs).