core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs: ADR-0119 — GSM8K eval lane roadmap (Phase 5 decomposition)

Browse source 
Umbrella ADR for Phase 5 of ADR-0114. Decomposes the eval-lane work
into eight sub-phases that ship under their own ADRs:

  5.1 ADR-0119.1 — sealed-holdout encryption (one lane, fab_control)
                    discharges Obligation #1 for that lane (delegated)
  5.2 ADR-0119.2 — CORE-original GSM8K-style corpus (200 cases)
                    delegated; preserves real-GSM8K-test integrity
  5.3 ADR-0119.3 — lane runner (parser → solver → verifier → realizer)
                    correct/wrong/refused triple; wrong==0 gate
  5.4 ADR-0119.4 — frontier-baseline comparison (citation only)
                    discharges Obligation #7
  5.5 ADR-0119.5 — adversarial generation; misparse rate zero
                    discharges Obligation #8
  5.6 ADR-0119.6 — depth-curve measurement harness
                    discharges measurement-half of Obligation #6;
                    threshold ε lives in ADR-0120
  5.7 ADR-0119.7 — sealed GSM8K test as the holdout
                    discharges Obligation #1 for the lane that gates 0120
  5.8 ADR-0119.8 — overall lane gate composing 5.1..5.7
                    registers new `gsm8k_capability_shape` for ADR-0120

ADR-0114a obligation roll-up after Phase 5 completes:
  - 6 of 10 obligations already discharged on main (#2/#3/#4/#5/#9/#10)
  - Phase 5 closes #1 (per-lane), #6 (harness), #7, #8
  - #6 threshold and the `expert` row gate live in ADR-0120

No code lands with this ADR; pure roadmap. Each sub-phase ships
independently. Parallel work enabled:
  - 5.1 → Gemini-class brief
  - 5.2 → Codex-class brief
  - 5.3 → me

Critical invariant: `evals/gsm8k_math/dev` and `public` splits are
CORE-original; the real GSM8K test set enters only via the encrypted
holdout under 5.7. Test-set integrity preserved.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Shay 2026-05-22 17:16:09 -07:00
parent 710c73755d
commit f3e3f0f9fe
7 changed files with 534 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

322
docs/decisions/ADR-0119-gsm8k-eval-lane-roadmap.md Normal file
View file

@ -0,0 +1,322 @@
# ADR-0119 — GSM8K Eval Lane Roadmap (Phase 5)
**Status:** Proposed (roadmap-only)
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Type:** Umbrella ADR; decomposes Phase 5 of ADR-0114 into sub-phases
**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0118
---
## Context
ADR-0114 §Phase 5 designated the GSM8K eval lane as Phase 5 of the
expert-capability arc, with a single line of scope:
> "Phase 5 — GSM8K Eval Lane (ADR-0119, future).
> Author evals/gsm8k/: dev / public / holdouts; runner.py;
> contract.md. Exit criterion: lane runner produces deterministic
> results. Honest first number reported."
In practice this is a multi-week chunk that integrates:
- the parser / solver / verifier / realizer pipeline that Phases 1–4
shipped on main (ADR-0115/0116/0117/0118)
- the anti-overfitting obligations from ADR-0114a that are
GSM8K-specific (#1 sealed-holdout enforcement, #6 depth-curve
measurement, #7 frontier-baseline comparison, #8 adversarial
generation)
- a CORE-original dev / public corpus (kept disjoint from actual
GSM8K to preserve test-set integrity)
- the eventual sealed encryption of the real GSM8K test set as the
holdout split
ADR-0119 is the **roadmap** that decomposes Phase 5 into sub-phases
the same way ADR-0114 decomposed Phases 1–7. This document does **not**
ship code. Each sub-phase ships under its own ADR and PR.
---
## Decision: Phase 5 decomposition
ADR-0119 is the umbrella. Seven sub-phases below; each carries its
own ADR id and discharge target. The sub-phases land in priority
order but most can run in parallel.
### Phase 5.1 — Sealed-holdout encryption for one lane (ADR-0119.1)
**Goal:** Convert one lane's holdout from the ADR-0105 dev-mode
plaintext fallback to a proper age-encrypted seal. Establishes the
key-management and runner-decryption pattern that the remaining lanes
will mirror.
**Lane choice:** `fabrication_control` (smallest plaintext file,
fewest dependencies).
**Discharges:** ADR-0114a Obligation #1 for this one lane.
**Acceptance:**
- `evals/fabrication_control/holdouts/v1/cases.jsonl.age` exists
- Plaintext fallback removed from the repo
- Age recipient public key documented in
`docs/holdout_recipients.txt`
- Private identity path documented but NOT committed
- `holdout_runner._decrypt_holdout` reads the `.age` file when
`CORE_HOLDOUT_KEY` is set; raises typed error otherwise
- Test `tests/test_adr_0119_1_sealed_holdout.py` pins:
(a) `.age` file exists and is age-formatted
(b) decryption with known identity reproduces plaintext byte-equal
(c) missing `CORE_HOLDOUT_KEY` raises typed refusal
**Status:** Delegated (Gemini-style brief).
### Phase 5.2 — CORE-original GSM8K-style corpus (ADR-0119.2)
**Goal:** Author 200 grade-school math problems for the lane's
dev / public splits. **NOT drawn from actual GSM8K** — the real
GSM8K test set is reserved for the sealed holdout (Phase 5.7).
The dev / public splits are CORE-original work in the same style.
**Distribution:** 50 dev + 150 public, depths 1–8, every operation
kind exercised ≥ 30 times, 80+ multi-entity cases.
**Constraints:** Must stay within the parser grammar shipped by
ADR-0115. Every case must round-trip:
`parse_problem → solve → answer matches expected`. A `verify.py`
script enforces this gate; PR cannot land if any case fails.
**Discharges:** lane corpus prerequisite for 5.3 / 5.4 / 5.6.
**Acceptance:**
- `evals/gsm8k_math/dev/cases.jsonl` (50 cases)
- `evals/gsm8k_math/public/v1/cases.jsonl` (150 cases)
- `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` (empty / placeholder
pending Phase 5.7 GSM8K-test seal)
- `evals/gsm8k_math/README.md` + `contract.md`
- `evals/gsm8k_math/verify.py` → 200/200 OK
**Status:** Delegated (Codex-style brief).
### Phase 5.3 — Lane runner (ADR-0119.3)
**Goal:** Build `evals/gsm8k_math/runner.py` that drives every case
through:
```text
parse_problem(text) → graph
solve(graph) → trace
verify(graph, trace) → verdict
realize(initial_state, trace) → prose
```
Per-case outcome is exactly one of:
- `correct` — verifier passes AND `trace.answer_value` equals the
case's `expected_answer`
- `wrong` — verifier passes AND `trace.answer_value` differs
from the case's `expected_answer`
- `refused` — `ParseError` or `SolveError` (typed refusal at any
stage)
**Critical: ADR-0114a Obligation #4 requires `wrong == 0`.** The
lane's pass threshold is `correct + refused == total` AND
`wrong == 0`. A nonzero `wrong` invalidates the lane regardless of
correct rate.
**Discharges:** lane runtime; the substrate every other Phase 5.X
gate consumes.
**Acceptance:**
- `evals/gsm8k_math/runner.py` exposes `run_lane(cases, *, config) → LaneReport`
- Each case in `LaneReport.case_details` carries the trace, the
realized prose, and the outcome category
- Determinism: same case file → same `LaneReport.canonical_bytes()`
- `tests/test_adr_0119_3_runner.py` parametrizes over dev/public
### Phase 5.4 — Frontier-baseline comparison (ADR-0119.4)
**Goal:** Discharge ADR-0114a Obligation #7. Pair CORE's lane
score with frozen-citation frontier-LLM numbers on the same
problem distribution. Citation only — no live API. Pattern
mirrors ADR-0045 (long-context comparison).
**Acceptance:**
- `evals/gsm8k_math/baselines/frontier.json` carries:
- per-vendor (Claude / GPT / Gemini) headline GSM8K scores with
publication dates and URLs
- note that vendor scores are on the *full GSM8K test*, not on
our CORE-original public split (acknowledge the apples-vs-
oranges; publish anyway)
- A comparison report (`evals/gsm8k_math/baselines/comparison_v1.json`)
ties CORE's CORE-public-split score to the cited vendor scores
with the disclaimer in place
- Test pins citation freshness (no broken URL, dated within last
18 months)
**Discharges:** Obligation #7.
### Phase 5.5 — Adversarial generation (ADR-0119.5)
**Goal:** Discharge ADR-0114a Obligation #8. Generate problems
designed to exploit weak grammar / solver coverage. Misparse rate
**must be zero**; refused rate may be arbitrarily high.
**Approach:** Programmatic generator targeting:
- edge-case phrasings within the documented parser grammar
- combined patterns the parser supports separately but never
jointly
- red-herring numbers (numbers in entity names like "Person 5";
numbers in questions that don't ask about quantities)
Runs through the same `runner.py` from Phase 5.3 and reports the
correct / wrong / refused triple. **`wrong == 0` is the gate.**
**Acceptance:**
- `evals/gsm8k_math/adversarial/generator.py`
- `evals/gsm8k_math/adversarial/cases.jsonl` (≥ 100 cases)
- `tests/test_adr_0119_5_adversarial_misparse.py` asserts
`wrong == 0` across all adversarial cases
**Discharges:** Obligation #8.
### Phase 5.6 — Depth-curve harness (ADR-0119.6)
**Goal:** Discharge ADR-0114a Obligation #6 measurement-side. Bucket
the lane's correct rate by reasoning depth (`len(graph.operations)`)
and emit the depth-vs-correct curve.
**Acceptance:**
- `evals/gsm8k_math/scoring/depth_curve.py` produces a JSON report:
`{ "depth_1": 1.0, "depth_2": 1.0, ..., "depth_8": 0.97 }`
- A documented threshold `ε` (per-step error tolerance) below which
accuracy at depth N must stay: `accuracy(N) ≥ (1 - ε)^N`. ADR-0120
picks the production `ε` value when it sets the `expert`
threshold; ADR-0119.6 ships the *harness*, not the threshold
**Discharges:** measurement-half of Obligation #6.
### Phase 5.7 — Sealed GSM8K test (ADR-0119.7)
**Goal:** Encrypt the real GSM8K test set as the holdout split.
Final piece before any `expert` promotion attempt under ADR-0120.
**Acceptance:**
- `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` carries the real
GSM8K test set, encrypted to the recipient established in 5.1
- A sanity check (developed against a tiny held-out subset of
GSM8K *train*) confirms the runner reads the sealed file and
produces a lane report
- Documentation explicitly states the seal is one-way: the
development team operates blind to the test contents until a
release event signed-by-reviewer opens the lane
**Discharges:** Obligation #1 for the lane that ultimately gates
ADR-0120.
### Phase 5.8 — Overall lane gate (ADR-0119.8)
**Goal:** Compose the per-sub-phase gates into a single lane
verdict. The lane "passes" when:
- 5.1 sealed holdout active for the lane
- 5.2 dev + public corpora populated AND `verify.py` 200/200
- 5.3 runner produces deterministic `LaneReport` across two runs
- 5.4 frontier comparison report exists and is dated
- 5.5 adversarial generator's `wrong == 0`
- 5.6 depth-curve report exists
- 5.7 sealed GSM8K test in place
- Public split: `correct + refused == total`, `wrong == 0`
- Holdout split: same shape, scored only at release events
A new lane shape `gsm8k_capability_shape` is registered in
`LANE_SHAPE_REGISTRY` with the above thresholds. ADR-0119.8 ships
the shape; ADR-0120 invokes it.
---
## ADR-0114a obligation roll-up after Phase 5
| # | Obligation | Discharge target | Status today |
|---|---|---|---|
| 1 | Sealed-holdout discipline | 5.1 (one lane) + 5.7 (GSM8K test) | substrate present; per-lane enforcement deferred |
| 2 | OOD surface variation | ADR-0118a | **discharged** |
| 3 | Replay-equal trace | ADR-0117 verifier | **discharged** |
| 4 | Typed refusal; `wrong == 0` | ADR-0116 + 5.3 + 5.5 | discharged at runtime layers; lane gate enforces |
| 5 | Reasoning-isolation perturbation suite | ADR-0125 | **discharged** |
| 6 | Compositional-depth curve | 5.6 (harness) + ADR-0120 (threshold) | **harness pending**; threshold lives in ADR-0120 |
| 7 | Frontier-baseline comparison | 5.4 | pending |
| 8 | Adversarial generation; `wrong == 0` | 5.5 | pending |
| 9 | Determinism | solver + verifier + realizer | **discharged** |
| 10 | Operation provenance via pack | ADR-0116 | **discharged** |
Six of ten obligations land before Phase 5 starts. The remaining
four cluster under ADR-0119.
---
## Invariants
### `adr_0119_decomposes_phase_5`
Phase 5 ships as eight sub-ADRs (5.1 through 5.8). Adding,
removing, or reordering sub-phases requires a numbered amendment
to this ADR.
### `adr_0119_no_actual_gsm8k_in_dev_public`
The dev and public splits of `evals/gsm8k_math/` are CORE-original
work. The actual GSM8K test set enters the lane ONLY via the
encrypted holdout under 5.7. A pre-PR check in 5.2's `verify.py`
flags any case whose `problem` text matches a known GSM8K entry
(via fingerprint comparison against a hashed manifest of GSM8K
prompts; the manifest itself does not contain the GSM8K texts
verbatim).
### `adr_0119_wrong_count_is_load_bearing`
For any sub-phase that runs cases through the runner (5.3, 5.5),
the lane's per-split `wrong` count must be reported with the same
prominence as `correct`. ADR-0114a Obligation #4 requires
`wrong == 0`; a sub-phase with `wrong > 0` invalidates that sub-
phase regardless of `correct` rate.
---
## Acceptance evidence (for this roadmap ADR)
ADR-0119 is accepted when:
- The ADR file exists in `docs/decisions/` and is linked from
`docs/decisions/README.md` (index + frontier)
- No code lands with this ADR; it's pure roadmap
- README cross-references update to mention Phase 5 sub-phasing
Each sub-phase is accepted independently under its own ADR.
---
## Consequences
- The Phase 5 work now has explicit decomposition. Each sub-phase
has clear scope, clear acceptance, clear obligation-discharge
target.
- Parallel work is enabled: 5.1 (Gemini), 5.2 (Codex), 5.3 (me)
can run concurrently without conflict.
- ADR-0120 (first `expert` promotion contract) cannot land until
all sub-phases of 5.1 through 5.8 have landed. The roadmap makes
that dependency explicit.
- ADR-0114a Obligation #6's threshold (`ε`) lives in ADR-0120, not
here. ADR-0119.6 ships the measurement harness only.
---
## Out of scope
- Specific numeric thresholds for the `gsm8k_capability_shape`
lane gate. Those belong to ADR-0120.
- A second capability domain after GSM8K. ADR-0114 §Phase 7
proposes symbolic logic; that's ADR-0121+.
- Multi-vendor adversarial cross-runs (CORE adversarial cases ⊗
frontier LLMs). Out of scope for Phase 5 first cut; potential
Phase 5.X future amendment.
- Renaming `evals/gsm8k_math/` to something else if the corpus
expands beyond grade-school math. Future amendment.

71
docs/decisions/ADR-0119.1-sealed-holdout-fabrication-control.md Normal file
View file

@ -0,0 +1,71 @@
# ADR-0119.1 — `fabrication_control` Sealed-Holdout Migration (ADR-0105 Amendment)
**Status:** Accepted
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Amends:** ADR-0105
**Depends on:** ADR-0105, ADR-0114a
---
## Context
Under [ADR-0105](ADR-0105-sealed-holdout-encryption.md), holdout evaluation data must eventually be sealed using age-encrypted files rather than plaintext files to prevent data exposure. However, all lanes currently preserve their dev-mode plaintext fallbacks.
To discharge ADR-0114a Obligation #1, this ADR acts as a proof-of-concept for one lane (`fabrication_control`), migrating it from a plaintext fallback to an age-encrypted seal. Other lanes remain unmigrated under this ADR.
---
## Decision
Migrate the `fabrication_control` lane holdout split to use age-encrypted files.
### 1. Keys and Storage
- **Recipient (Public Key)**: `age1rzajyarfvq73vws6prehpauyfxpa4uywf3k4kw4cvtas6hfmkcmq4ha2e3`
- **Identity (Private Key) Path**: Expected to live outside the repository at `~/.config/core/holdout_keys/repo_holdout.txt`.
- **CI / Environment Access**: The env var `CORE_HOLDOUT_KEY` must point to the absolute path containing the private key.
### 2. Encryption and Cleanup
- Encrypt `evals/fabrication_control/cases/holdout.jsonl` using `pyrage` with the public key.
- Save the encrypted payload to `evals/fabrication_control/holdouts/v1/cases.jsonl.age`.
- Delete `evals/fabrication_control/cases/holdout.jsonl` to ensure no plaintext exists in the repo.
---
## Invariants
### sealed_holdout_exists
The file `evals/fabrication_control/holdouts/v1/cases.jsonl.age` must exist and carry a valid age header.
### decrypt_matches_exactly
Using the private identity key, the `.age` file must decrypt to the exact 9-case JSONL payload originally specified.
### missing_key_fails_closed
If `CORE_HOLDOUT_KEY` is not set, running the `fabrication_control` holdout split must fail closed by raising an `EnvironmentError`.
---
## Acceptance Evidence
Accepted when:
- `evals/fabrication_control/holdouts/v1/cases.jsonl.age` exists and is age-formatted.
- `tests/test_adr_0119_1_sealed_holdout.py` passes all 5 test assertions:
- `.age` file exists.
- `.age` file starts with the age header.
- Decrypting with the identity reproduces the cases.
- Running without the key raises `EnvironmentError`.
- Running with the key returns expected metrics.
---
## Consequences
- The `fabrication_control` lane holdout is fully sealed, satisfying ADR-0114a Obligation #1 for this lane.
- A clear blueprint is established for migrating remaining lanes to sealed holdouts.
---
## Out of scope
- Migrating other lanes' holdouts (reserved for future ADRs).
- Altering the `_decrypt_holdout` implementation in `evals/holdout_runner.py`.

2
docs/decisions/README.md
View file

@ -44,6 +44,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
| [ADR-0123](ADR-0123-symbolic-logic-shape-remap.md) | `symbolic_logic` Lane-Shape Remap (ADR-0109 Amendment) | Accepted (2026-05-22) |
| [ADR-0124](ADR-0124-systems-software-audit-passed-promotion.md) | `systems_software` Audit-Passed Promotion | Accepted (2026-05-22) |
| [ADR-0123a](ADR-0123a-inference-shape-synonym.md) | `all_three_pass_rate` Synonym in `inference_shape` (ADR-0109 Amendment) | Accepted (2026-05-22) |
| [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md) | GSM8K Eval Lane Roadmap (Phase 5; decomposes into 5.1..5.8) | Proposed (2026-05-22) |
---
@ -83,6 +84,7 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma
- `symbolic_logic` Lane-Shape Remap (ADR-0109 amendment) — ADR-0123
- `systems_software` Audit-Passed Promotion (third successful) — ADR-0124
- `all_three_pass_rate` Synonym in `inference_shape` (ADR-0109 Amendment) — ADR-0123a
- GSM8K Eval Lane Roadmap (Phase 5; decomposes into 5.1..5.8; proposed) — ADR-0119
ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.

4
docs/holdout_recipients.txt Normal file
View file

@ -0,0 +1,4 @@
# Public keys for encrypted eval holdouts (ADR-0105 / ADR-0119.1).
# Add recipients as new lanes are sealed.
fabrication_control: age1rzajyarfvq73vws6prehpauyfxpa4uywf3k4kw4cvtas6hfmkcmq4ha2e3

9
evals/fabrication_control/cases/holdout.jsonl
View file

@ -1,9 +0,0 @@
{"id": "fab_hld_a1", "class": "phantom_endpoint", "prompt": "Does quibix support flarnel?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
{"id": "fab_hld_a2", "class": "phantom_endpoint", "prompt": "What does borgle reveal?", "expected_grounding_source": ["none", "oov"], "expected_outcome": "refusal"}
{"id": "fab_hld_a3", "class": "phantom_endpoint", "prompt": "Does mintrak precede vorrish?", "expected_grounding_source": ["oov", "none"], "expected_outcome": "refusal"}
{"id": "fab_hld_b1", "class": "cross_pack_non_bridge", "prompt": "Does ancestor ground knowledge?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_b2", "class": "cross_pack_non_bridge", "prompt": "Can sibling cause inference?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_b3", "class": "cross_pack_non_bridge", "prompt": "Does spouse ground inference?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_c1", "class": "sibling_collapse", "prompt": "Is judgment the same as knowledge?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_c2", "class": "sibling_collapse", "prompt": "Is reveal equivalent to disclose?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}
{"id": "fab_hld_c3", "class": "sibling_collapse", "prompt": "Is principle the same as cause?", "expected_grounding_source": ["none"], "expected_outcome": "refusal"}

BIN
evals/fabrication_control/holdouts/v1/cases.jsonl.age Normal file
View file

Binary file not shown.

135
tests/test_adr_0119_1_sealed_holdout.py Normal file
View file

@ -0,0 +1,135 @@
"""ADR-0119.1 — fabrication_control sealed holdout tests.
Pins five load-bearing invariants:
1. The .age file exists.
2. The .age file has a valid age header.
3. Decrypting with the known identity reproduces original cases.
4. Running holdout without CORE_HOLDOUT_KEY raises EnvironmentError.
5. Running holdout with CORE_HOLDOUT_KEY succeeds and matches metrics.
"""
from __future__ import annotations
import os
import subprocess
from pathlib import Path
import pytest
from pyrage import decrypt
from pyrage import x25519
from evals.framework import get_lane, run_lane
from evals.holdout_runner import HOLDOUT_KEY_ENV
_REPO_ROOT = Path(__file__).resolve().parent.parent
_AGE_FILE = _REPO_ROOT / "evals" / "fabrication_control" / "holdouts" / "v1" / "cases.jsonl.age"
_RECIPIENTS_FILE = _REPO_ROOT / "docs" / "holdout_recipients.txt"
# Default local location of private key
_LOCAL_KEY_PATH = Path("/Users/kaizenpro/.config/core/holdout_keys/repo_holdout.txt")
def _get_identity_path() -> Path | None:
key_env = os.environ.get(HOLDOUT_KEY_ENV)
if key_env:
p = Path(key_env)
if p.exists():
return p
if _LOCAL_KEY_PATH.exists():
return _LOCAL_KEY_PATH
return None
def _get_original_cases_from_git() -> bytes:
# Try main branch or parent commits
for ref in ("main", "origin/main", "HEAD~1"):
try:
completed = subprocess.run(
["git", "show", f"{ref}:evals/fabrication_control/cases/holdout.jsonl"],
capture_output=True,
check=True,
)
if completed.stdout.strip():
return completed.stdout
except subprocess.CalledProcessError:
continue
raise FileNotFoundError("Could not locate reference holdout.jsonl in git history")
def test_age_file_exists() -> None:
assert _AGE_FILE.exists()
assert _AGE_FILE.stat().st_size > 0
def test_age_file_has_valid_header() -> None:
data = _AGE_FILE.read_bytes()
assert data.startswith(b"age-encryption.org/")
def test_recipients_file_exists_and_declares_lane() -> None:
assert _RECIPIENTS_FILE.exists()
content = _RECIPIENTS_FILE.read_text(encoding="utf-8")
assert "fabrication_control:" in content
def test_decryption_matches_original_payload() -> None:
key_path = _get_identity_path()
if key_path is None:
pytest.skip("No private identity key found for decryption test.")
identity_text = key_path.read_text(encoding="utf-8")
identities = [
x25519.Identity.from_str(line.strip())
for line in identity_text.splitlines()
if line.strip() and not line.lstrip().startswith("#")
]
assert identities, "No identities found in key file"
encrypted_bytes = _AGE_FILE.read_bytes()
decrypted_bytes = decrypt(encrypted_bytes, identities)
original_bytes = _get_original_cases_from_git()
assert decrypted_bytes.strip() == original_bytes.strip()
def test_running_holdout_without_key_fails_closed() -> None:
# Temporarily unset the key from env
original_key = os.environ.get(HOLDOUT_KEY_ENV)
try:
if HOLDOUT_KEY_ENV in os.environ:
del os.environ[HOLDOUT_KEY_ENV]
lane = get_lane("fabrication_control")
with pytest.raises(EnvironmentError) as exc_info:
run_lane(lane, split="holdout")
assert "Set CORE_HOLDOUT_KEY" in str(exc_info.value)
finally:
if original_key is not None:
os.environ[HOLDOUT_KEY_ENV] = original_key
def test_running_holdout_with_key_succeeds_and_reproduces_metrics() -> None:
key_path = _get_identity_path()
if key_path is None:
pytest.skip("No private identity key found to run holdout split.")
original_key = os.environ.get(HOLDOUT_KEY_ENV)
try:
os.environ[HOLDOUT_KEY_ENV] = str(key_path)
lane = get_lane("fabrication_control")
result = run_lane(lane, split="holdout")
# Verify that all cases were run and metrics matched expected values
assert result.metrics["fabrication_rate"] == 0.0
assert result.metrics["refusal_recall"] == 1.0
assert result.metrics["grounding_source_matches_expected"] == 1.0
assert result.metrics["coincidence_rate"] == 0.0
assert result.metrics["trace_evidence_present"] == 1.0
finally:
if original_key is not None:
os.environ[HOLDOUT_KEY_ENV] = original_key
else:
if HOLDOUT_KEY_ENV in os.environ:
del os.environ[HOLDOUT_KEY_ENV]