66 lines
3.5 KiB
Markdown
66 lines
3.5 KiB
Markdown
# ADR-0119.1 — Seal fabrication_control Holdout (ADR-0105 Amendment)
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-23
|
|
**Author:** CORE agents + reviewers
|
|
**Amends:** ADR-0105
|
|
**Depends on:** ADR-0105, ADR-0114a
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
Under [ADR-0105](file:///Users/kaizenpro/Projects/core/docs/decisions/ADR-0105-sealed-holdout-encryption.md), CORE established recipient-based `age` encryption for sealed holdouts to prevent leakage. Plaintext fallback was allowed temporarily for local development.
|
|
|
|
[ADR-0114a](file:///Users/kaizenpro/Projects/core/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md) (Obligation #1) mandates that every lane migrate its holdout from plaintext fallback to an `age`-encrypted seal to satisfy anti-overfitting rules. This ADR implements this migration for the first lane: `fabrication_control` (the smallest and simplest lane).
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
We seal the `fabrication_control` holdout using the `age` format keyed to a dedicated public key recipient.
|
|
|
|
### Configuration Details
|
|
- **Sealed Lane:** `fabrication_control`
|
|
- **Recipient Public Key:** `age1zt3ud2skl4y8qt306dlux0d3g68tt2trsur64yuh7gv4dag5qg5su9jllz`
|
|
- **Private Identity Location:** The private identity key is expected to live outside the repository at `~/.config/core/holdout_keys/repo_holdout.txt`.
|
|
- **CI / Execution Setup:** A fresh CI run or local test suite obtains access to this identity by setting the `CORE_HOLDOUT_KEY` environment variable pointing to the absolute path of the identity file.
|
|
|
|
### Actions Taken
|
|
1. Generated a new age x25519 keypair for the repository.
|
|
2. Encrypted the original holdout file `evals/fabrication_control/cases/holdout.jsonl` to `evals/fabrication_control/holdouts/v1/cases.jsonl.age` using the public key recipient.
|
|
3. Removed the unencrypted `evals/fabrication_control/cases/holdout.jsonl` from the repository.
|
|
4. Documented the recipient in [docs/holdout_recipients.txt](file:///Users/kaizenpro/Projects/core/docs/holdout_recipients.txt).
|
|
5. Exposed `run_lane` in the `fabrication_control` runner `evals/fabrication_control/runner.py` to support the generic framework execution model, and updated `_run_split` to decrypt the holdout when split is `holdout`.
|
|
|
|
---
|
|
|
|
## Invariants
|
|
|
|
- Running `fabrication_control` holdout split without the environment variable `CORE_HOLDOUT_KEY` set raises a typed `EnvironmentError` and fails closed.
|
|
- Decrypting the sealed `cases.jsonl.age` using the repository private key reproduces the original holdout cases byte-for-byte.
|
|
|
|
---
|
|
|
|
## Acceptance Evidence
|
|
|
|
Accepted when:
|
|
- The `.age` file exists at `evals/fabrication_control/holdouts/v1/cases.jsonl.age`.
|
|
- The tests in [tests/test_adr_0119_1_sealed_holdout.py](file:///Users/kaizenpro/Projects/core/tests/test_adr_0119_1_sealed_holdout.py) pass cleanly, validating:
|
|
- Encryption format and presence of header in `cases.jsonl.age`.
|
|
- Byte-equal reproduction of holdout cases upon decryption.
|
|
- Fail-closed behavior (`EnvironmentError` raised) when key is missing.
|
|
- Successful execution and reproduction of the prior metrics (`refusal_recall == 1.0`, `fabrication_rate == 0.0`) when key is provided.
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
- The `fabrication_control` lane holdout is fully sealed, fulfilling Obligation #1 of [ADR-0114a](file:///Users/kaizenpro/Projects/core/docs/decisions/ADR-0114a-anti-overfitting-proof-obligations.md).
|
|
- Plaintext leaks of `fabrication_control` holdout are eliminated from the current tree.
|
|
|
|
---
|
|
|
|
## Out of Scope
|
|
|
|
- Migrating other evaluation lanes' holdouts (reserved for subsequent ADRs).
|