From b2b3678debc69556ce0bf2817d27a3c67cef97c0 Mon Sep 17 00:00:00 2001 From: Shay Date: Wed, 3 Jun 2026 16:00:35 -0700 Subject: [PATCH 1/3] docs: mark ADR-0174 Phase 5a shipped in composition-capability-scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 5a (retire the inert GSM8K scoring-path comprehension-reader dispatch, -1,038 LOC) landed in 3fd3172, an ancestor of HEAD 8327c6b, but composition-capability-scope.md still listed it as remaining work -- drift that mis-scoped a downstream brief. Re-anchor the canonical doc to the tree: - §1: the lifecycle.py GSM8K-scoring dispatch row now reads "retired in Phase 5a (3fd3172)" instead of "inert ... retirable in 5a". - §2: Phase 5a marked SHIPPED with a runway-status banner; the §9 completeness-gate precondition marked LANDED (PR #534, e1bcdf6). - §4: sequencing restructured -- items 0 (§9 precondition), 1 (gating analysis), 3 (5a) are DONE; the live runway starts at item 2 (the <=20-case validation sub-corpus), then Phase 5b. - §9: status banner noting the recommended guard landed (#534); the "RED on main" statements are the pre-landing measurement, kept for provenance. Docs-only. No code/eval/serving change. Baseline re-verified at HEAD: train_sample 6/44/0 (all 44 refusals at branches_enumerated=0), completeness-guard suite 21 passed. --- docs/analysis/composition-capability-scope.md | 50 +++++++++++++------ 1 file changed, 35 insertions(+), 15 deletions(-) diff --git a/docs/analysis/composition-capability-scope.md b/docs/analysis/composition-capability-scope.md index 77c77d52..f35e659b 100644 --- a/docs/analysis/composition-capability-scope.md +++ b/docs/analysis/composition-capability-scope.md @@ -39,7 +39,7 @@ solver**, not at the operation layer. |---|---| | Held-hypothesis reader (P1–P4: state, constraint-propagation, lookback, contemplate) | **Shipped, wired into serving** candidate-graph | | `HYPOTHESIS_CAP`, vault>packs>audit precedence | already set/enforced (0174 OQ#1/#3 moot) | -| `lifecycle.py` GSM8K-scoring dispatch | **inert** (admits 0/50) — retirable in 5a | +| `lifecycle.py` GSM8K-scoring dispatch | **retired in Phase 5a** (commit `3fd3172`, ancestor of HEAD `8327c6b`) — was inert (admitted 0/50) | | `lifecycle.py` / `audit.py` reader surface | **load-bearing** for the ADR-0172 teaching corridor — **keep** | | Solver operation kinds | **8 already exist** (verified by discovery): `add`, `subtract`, `transfer`, `multiply`, `divide`, `apply_rate`, `compare_additive`, `compare_multiplicative`. Missing arithmetic op kinds is **not** the blocker. | | **Live audit of all 44 refusals (ran `parse_and_solve` on `main`)** | **44/44 refuse at `branches_enumerated = 0`** — i.e. *upstream of the solver*. No branch is built, so the 8 operation kinds are **never reached**. `rate`/`ratio` also exist as first-class `SEMANTIC_ROLES`/`QUESTION_FORMS` (`binding_graph/model.py:42–65`); only `percent`/`accumulate` lack a named op-kind — moot while nothing reaches the solver. | @@ -47,14 +47,22 @@ solver**, not at the operation layer. ## 2. The actual remaining work (ADR-0174 amended Phase 5) -### Phase 5a — retire the inert parallel parser (structural, ~0 lift) -Retire the GSM8K-scoring-only inert dispatch (`_try_comprehension_reader` / -`_try_reader_for_question`) + adapter + `use_reader` plumbing. Net **−1,038 LOC** -(code + tests). **Keep** `lifecycle.py`/`audit.py` — their reader *refusals* feed -the ADR-0172 math-contemplation teaching corridor (`teaching/math_*`, -`evals/flywheel_demo`, `core/cli.py`). Low-risk; single-path serving. -Acceptance: 6/44/0 byte-identical, capability-axis lanes 100% `wrong=0`, -pinned-SHAs pass. +> **Runway status (2026-06-03):** Phase 5a **SHIPPED** in commit `3fd3172` +> (ancestor of HEAD `8327c6b`); the §9 completeness-gate precondition **LANDED** +> in PR #534 (`e1bcdf6`). The live runway now starts at the ≤20-case validation +> sub-corpus (§4 item 2), then **Phase 5b** (emission/representation) — the only +> remaining capability work. + +### Phase 5a — retire the inert parallel parser (SHIPPED in `3fd3172`) +**Done.** Commit `3fd3172` (ancestor of HEAD `8327c6b`) retired the +GSM8K-scoring-only inert dispatch (`_try_comprehension_reader` / +`_try_reader_for_question`) + adapter (`lifecycle_runtime_adapter.py`) + +`use_reader` / `comprehension_reader_questions` plumbing. Net **−1,038 LOC** +(code + tests). `lifecycle.py`/`audit.py` were **kept** — their reader *refusals* +feed the ADR-0172 math-contemplation teaching corridor (`teaching/math_*`, +`evals/flywheel_demo`, `core/cli.py`). Single-path serving. +Acceptance **met and re-verified at HEAD:** train_sample 6/44/0 byte-identical +(all 44 refusals at `branches_enumerated=0`), completeness-guard suite green. ### Phase 5b — emission / representation buildout (semantic — the real lift) "This is where `correct` climbs toward 25. It is not a refactor." **Verified @@ -79,8 +87,9 @@ ops behind the same admissibility predicates is lower-risk than new solver math. **5b first PR (smallest verified-tractable slice):** pick the emission failure mode with the highest count whose representation is bounded — and prove it reaches branch -enumeration on ≥1 case without breaking `wrong=0`. **But the §9 completeness-gate -precondition lands before any emission PR.** +enumeration on ≥1 case without breaking `wrong=0`. The §9 completeness-gate +precondition **landed in PR #534** (`e1bcdf6`); its driver +(`tests/test_candidate_graph_completeness_guard.py`) is green at HEAD. ## 3. Test anchors (reusable, relabeled to 5a/5b) 1. **Serving gate (`gsm8k_math`):** 5a byte-identical 6/44/0; 5b climbs @@ -94,13 +103,17 @@ precondition lands before any emission PR.** hard negatives are the first deposits into it. Author before 5b measurement. ## 4. Corrected sequencing -0. **Precondition (§9):** harden the comparative-multiplicative completeness tripwire so a dropped `×` clause refuses (as `twice` does). 5b emission on these shapes is unsafe until this lands. Driver test is FINAL and RED on `main` (see §9). + +> **Status (2026-06-03):** items 0, 1, and 3 are **DONE**. The live runway starts +> at item 2 (the sub-corpus), then item 4 (Phase 5b). + +0. **Precondition (§9) — LANDED (PR #534, `e1bcdf6`).** The comparative-multiplicative completeness tripwire now refuses a dropped `×` clause (as `twice` does). The driver `tests/test_candidate_graph_completeness_guard.py` pins the §9 hard-negative matrix (10 no-ref ` times` cases + with-ref must-solve + must-still-refuse controls) and is **green** at HEAD `8327c6b`. The "RED on `main`" claim in §9 is the pre-landing measurement. 1. **5b gating analysis — done (§8):** the **emission/representation** audit ran and the 32 no-injection cases are sub-classified by representation. Highest-count bounded gap = **R1 derived/intermediate symbol (24/44)**. -2. **Author the ≤20-case multi-step validation sub-corpus** (test anchor for 5b). -3. **Phase 5a** (optional, can run in parallel — structural, low-risk). +2. **Author the ≤20-case multi-step validation sub-corpus** (test anchor for 5b) — **next open step.** +3. **Phase 5a — SHIPPED (`3fd3172`, ancestor of HEAD `8327c6b`):** the inert scoring-path dispatch was retired (−1,038 LOC); `lifecycle.py`/`audit.py` kept for the ADR-0172 corridor. 4. **Phase 5b** — build the **emitter/representation** for R1 (derived-symbol) on the §8 near-pure exemplars (prove ≥1 case reaches `branches_enumerated > 0` and admits), each behind §3 gates **and the §9 precondition**. Not new operations — the 8 exist and are unreached. -(Do **not** "land Phase 1" — shipped. Do **not** treat P3/P4 as the lever — shipped, metric flat.) +(Do **not** "land Phase 1" — shipped. Do **not** treat P3/P4 as the lever — shipped, metric flat. Do **not** re-do Phase 5a — shipped in `3fd3172`.) ## 5. Relation to cross-subject testing 5b maturing the math `DomainSolver`'s emission/representation is still the precondition @@ -159,6 +172,13 @@ four near-pure exemplars above — move them from refused to admitted, `wrong=0` ## 9. ⚠ Latent `wrong=0` hazard surfaced by the live audit (gate gap) +> **Status (2026-06-03):** the guard recommended below **LANDED in PR #534** +> (`e1bcdf6`). Its hard-negative matrix lives in +> `tests/test_candidate_graph_completeness_guard.py` (10 no-ref ` times` cases +> + with-ref must-solve + must-still-refuse controls) and is **green** at HEAD +> `8327c6b` (21 passed). The "RED on `main`" statements below are the point-in-time +> pre-landing measurement, kept for provenance. + Contrastive probes (not in the 50-case sample) surface a reproducible **admitted-wrong** path. **Use the parseable aggregate question form `"...do they have together?"`** — the short form `"How many apples together?"` refuses upstream at question-parse From a8027ca34d3a1e9ad669919317bb98569a76c735 Mon Sep 17 00:00:00 2001 From: Shay Date: Wed, 3 Jun 2026 16:30:29 -0700 Subject: [PATCH 2/3] test(gsm8k): add composition validation corpus --- .../composition_validation/v1/cases.jsonl | 20 ++++++ .../composition_validation/v1/contract.md | 62 +++++++++++++++++++ 2 files changed, 82 insertions(+) create mode 100644 evals/gsm8k_math/composition_validation/v1/cases.jsonl create mode 100644 evals/gsm8k_math/composition_validation/v1/contract.md diff --git a/evals/gsm8k_math/composition_validation/v1/cases.jsonl b/evals/gsm8k_math/composition_validation/v1/cases.jsonl new file mode 100644 index 00000000..11756b69 --- /dev/null +++ b/evals/gsm8k_math/composition_validation/v1/cases.jsonl @@ -0,0 +1,20 @@ +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0001","composition":"R1","gate":"5b-R1","gold":64,"note":"Derived keyboard cost = 3 * mouse cost, then total accessories = keyboard + mouse.","question":"Fabian bought a brand new computer mouse and keyboard to be able to work from home. The cost of the keyboard was three times greater than the cost of the mouse. If the mouse cost $16, how much did Fabian spent on his new accessories?","source":"gsm8k_train_sample:0029","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0002","composition":"R1","gate":"5b-R1","gold":400,"note":"Derived second-floor count = 3 * first-floor count, then total floors.","question":"In a building, there are a hundred ladies on the first-floor studying. There are three times that many girls at a party being held on the second floor of the building. How many ladies are on the two floors in total?","source":"gsm8k_train_sample:0038","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0003","composition":"R1","gate":"5b-R1","gold":9,"note":"Derived total beads from two products, then divide by beads per bracelet.","question":"Marnie makes bead bracelets. She bought 5 bags of 50 beads and 2 bags of 100 beads. If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?","source":"gsm8k_train_sample:0008","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0004","composition":"R1","gate":"5b-R1","gold":3840,"note":"Chained intermediate symbols: combined IG+Facebook, Twitter, TikTok, YouTube, then total.","question":"Malcolm has 240 followers on Instagram and 500 followers on Facebook. The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined. Meanwhile, the number of followers he has on TikTok is 3 times the number of followers is has on Twitter, and he has 510 more followers on Youtube than he has on TikTok. How many followers does Malcolm have on all his social media?","source":"gsm8k_train_sample:0027","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss.","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0006","composition":"R5","gate":"5b-R5","gold":14,"note":"Multi-step duration: round-trip drive time, scalar beach time, total trip time.","question":"Jake decides to go to the beach for a fun day. It is a 2-hour drive each way. He then spends 2.5 times at long at the beach as his total driving time. How much time does the trip take?","source":"gsm8k_train_sample:0030","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0007","composition":"R6","gate":"5b-R6","gold":21,"note":"Fraction mutation: final temperature = 3/4 of current, decrease = current - final.","question":"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?","source":"gsm8k_train_sample:0005","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0008","composition":"R6","gate":"5b-R6","gold":15,"note":"Percent partition: girls/boys split, percent of each, then total owners.","question":"A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?","source":"gsm8k_train_sample:0046","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":1,"baseline_verdict":"refuse","case_id":"cv-0009","composition":"compare_mult","gate":"5b-compare-mult","gold":60,"note":"Referenced multiplicative comparison should eventually solve; today it is a completeness-guard refusal.","question":"Ivan has 20 dice. Jerry has twice as many dice as Ivan. How many dice do they have altogether?","source":"guard:real_gsm8k_605","target_verdict":"solve"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0010","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has 3 times as many apples. How many apples do they have together?","source":"guard:n_times_without_reference:3_as_many","target_verdict":"refuse"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0011","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has 5 times as many apples. How many apples do they have together?","source":"guard:n_times_without_reference:5_as_many","target_verdict":"refuse"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0012","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has three times more apples. How many apples do they have together?","source":"guard:n_times_without_reference:three_more","target_verdict":"refuse"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0013","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has 3 times the number of apples. How many apples do they have together?","source":"guard:n_times_without_reference:3_the_number_of","target_verdict":"refuse"} +{"baseline_answer":null,"baseline_branches_enumerated":1,"baseline_verdict":"refuse","case_id":"cv-0014","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; completeness guard currently catches the unconsumed scalar.","question":"Tom has 7 apples. Jerry has twice as many apples. How many apples do they have together?","source":"guard:existing_multiplier_refusal:twice_no_reference","target_verdict":"refuse"} +{"baseline_answer":null,"baseline_branches_enumerated":1,"baseline_verdict":"refuse","case_id":"cv-0015","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; completeness guard currently catches the unconsumed scalar.","question":"Tom has 7 apples. Jerry has double the apples. How many apples do they have together?","source":"guard:existing_multiplier_refusal:double_no_reference","target_verdict":"refuse"} +{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0016","composition":"world_knowledge","gate":"permanent","gold":72,"note":"Requires unstated legs-per-animal world knowledge; remains outside the decidable regime for this corpus.","question":"Over several years, Daniel has adopted any stray animals he sees on the side of the road. He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat. All of the animals are perfectly healthy. In total, how many legs do his animals have?","source":"gsm8k_train_sample:0040","target_verdict":"refuse"} +{"baseline_answer":438,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0017","composition":"compare_mult_additive","gate":"baseline","gold":438,"note":"Existing graph-path correct case; completeness guard must not over-refuse it.","question":"Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday. Brooke does three times as many jumping jacks as Sidney. How many jumping jacks did Brooke do?","source":"gsm8k_train_sample:0024","target_verdict":"solve"} +{"baseline_answer":28,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0018","composition":"compare_mult","gate":"baseline","gold":28,"note":"Referenced multiplicative comparison already solves; protects against over-broad no-reference refusal.","question":"Tom has 7 apples. Jerry has 3 times as many apples as Tom. How many apples do they have together?","source":"guard:n_times_with_reference:3_as_many_as_tom","target_verdict":"solve"} +{"baseline_answer":1200,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0019","composition":"additive","gate":"baseline","gold":1200,"note":"Stated-sum contrastive control for the derived-symbol Nicole/Cindy form.","question":"Nicole has 400 cards. Cindy has 800 cards. How many cards do they have together?","source":"analysis:composition-capability-scope:R1_contrastive_control","target_verdict":"solve"} +{"baseline_answer":450,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0020","composition":"product_additive","gate":"baseline","gold":450,"note":"Existing guarded product bridge correct case; protects current 6/44/0 lift.","question":"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. How much total weight does he move?","source":"gsm8k_train_sample:0021","target_verdict":"solve"} diff --git a/evals/gsm8k_math/composition_validation/v1/contract.md b/evals/gsm8k_math/composition_validation/v1/contract.md new file mode 100644 index 00000000..0e163711 --- /dev/null +++ b/evals/gsm8k_math/composition_validation/v1/contract.md @@ -0,0 +1,62 @@ +# GSM8K Composition Validation v1 + +This corpus is a Phase 5b measurement instrument, not a serving lane and not an +emission implementation. It pins a small set of real and guard-seeded GSM8K +composition shapes so each future slice is measured against two coupled facts: +coverage may rise only while `wrong == 0` remains true. + +## Scope + +- 20 cases total. +- Positive cases use `target_verdict = solve` and a non-baseline `gate`; they + refuse today and should flip only when that gate's capability slice lands. +- Hard-negative cases use `target_verdict = refuse` and `gate = permanent`; they + must continue refusing because the input is incomplete or outside the current + decidable regime. +- Baseline controls use `gate = baseline`; they already solve and must keep + solving to the exact gold answer. + +## Schema + +Each JSONL row has these fields: + +- `case_id`: stable corpus-local id. +- `source`: exact source pointer (`gsm8k_train_sample:*`, `guard:*`, or + `analysis:*`). +- `question`: prompt passed to `generate.math_candidate_graph.parse_and_solve`. +- `gold`: numeric gold answer, or `null` for permanent hard-negatives where no + complete reading exists. +- `target_verdict`: `solve` or `refuse`. +- `composition`: coarse composition family. +- `gate`: `baseline`, `permanent`, or a future capability gate such as `5b-R1`. +- `baseline_verdict`: tree-verified current verdict at corpus creation. +- `baseline_answer`: current answer when `baseline_verdict = solve`, otherwise + `null`. +- `baseline_branches_enumerated`: current `CandidateGraphResult` branch count. +- `note`: short reason the case is included. + +## Invariants + +1. Wrong-zero firewall: every case must either refuse or answer exactly `gold`; + a non-gold admission is a failure. +2. Null-gold hard-negatives: when `gold` is `null`, any admission is a failure. +3. Baseline controls: every `gate = baseline` row must solve to `gold` at every + step. +4. Permanent refusals: every `gate = permanent` row must refuse until a future + reviewed ADR explicitly changes its decidable regime. +5. Progress metric: count rows where `target_verdict = solve` and the current + answer equals `gold`; this starts at the baseline controls and may increase + only when invariants 1-4 still pass. + +## Baseline + +At creation on `origin/main` lineage after PR #534 plus the docs-only runway +correction, the intended baseline is: + +- 4 solve +- 16 refuse +- 0 wrong + +The corpus deliberately includes both future positives and permanent +hard-negatives. A future Phase 5b slice should update only the measured current +result, not the row's baseline fields. From c89adc65472f1c98956ae91949ed83d061287185 Mon Sep 17 00:00:00 2001 From: Shay Date: Wed, 3 Jun 2026 18:40:37 -0700 Subject: [PATCH 3/3] test(gsm8k): executable invariants for composition validation corpus The composition_validation/v1 corpus shipped as data + a prose contract with no executing test -- decoration until a test can fail (CLAUDE.md, Schema-Defined Proof Obligations). Add the load-bearing gate: - forever-invariants: wrong=0 firewall (admit => ==gold; null-gold => refuse), baseline-control regression net, permanent-refusal permanence, and frozen baseline-field/tree match for the non-positive rows. - current snapshot: aggregate 4 solve / 16 refuse / 0 wrong -- the single assertion a Phase 5b slice updates when it flips a positive. Future positives (5b-* gates) are checked by the firewall only, so a refuse->solve flip at 5b stays green without rewriting frozen rows. Verified: 44 passed; falsified (corrupting a control's gold fails the firewall + control + snapshot tests); completeness guard still 21 passed. --- tests/test_composition_validation_corpus.py | 177 ++++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 tests/test_composition_validation_corpus.py diff --git a/tests/test_composition_validation_corpus.py b/tests/test_composition_validation_corpus.py new file mode 100644 index 00000000..b55617a3 --- /dev/null +++ b/tests/test_composition_validation_corpus.py @@ -0,0 +1,177 @@ +"""Executable invariants for the Phase 5b composition validation sub-corpus. + +The corpus (``evals/gsm8k_math/composition_validation/v1/cases.jsonl``) is the +ADR-0174 OQ#5 measurement instrument. A JSONL of cases plus a prose contract is +only *decoration* until a test can fail under the violations it names +(CLAUDE.md, "Schema-Defined Proof Obligations"). This module is that test. + +It enforces two kinds of obligation: + +* **Forever-invariants** (never need updating as Phase 5b lands): + - wrong=0 firewall: a case may only refuse or answer exactly ``gold``; + a ``gold == null`` case may only refuse. + - regression net: every ``gate == "baseline"`` row keeps solving to ``gold``. + - permanence: every ``gate == "permanent"`` row keeps refusing. + - frozen baseline snapshot for those non-positive rows matches the live tree. + +* **Current snapshot** (the one assertion a Phase 5b slice updates when it + flips a positive): the aggregate is ``4 solve / 16 refuse / 0 wrong`` today. + +A future positive (``gate`` like ``5b-R1``) is *expected* to flip +refuse -> solve when its slice lands; that flip must still satisfy the firewall, +so it is checked by the firewall test, not pinned by a per-row baseline match. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from generate.math_candidate_graph import parse_and_solve + +_REPO_ROOT = Path(__file__).resolve().parents[1] +_CASES_PATH = ( + _REPO_ROOT / "evals" / "gsm8k_math" / "composition_validation" / "v1" / "cases.jsonl" +) + +_REQUIRED_FIELDS = frozenset( + { + "case_id", + "source", + "question", + "gold", + "target_verdict", + "composition", + "gate", + "baseline_verdict", + "baseline_answer", + "baseline_branches_enumerated", + "note", + } +) + +_EXPECTED_TOTAL = 20 +_EXPECTED_BASELINE_CONTROLS = 4 +_EXPECTED_PERMANENT = 7 +_EXPECTED_FUTURE_POSITIVE = 9 + + +def _load_cases() -> list[dict]: + rows = [ + json.loads(line) + for line in _CASES_PATH.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + return rows + + +_CASES = _load_cases() +_IDS = [c["case_id"] for c in _CASES] + + +def _num_eq(a, b) -> bool: + if a is None or b is None: + return a is b + return abs(float(a) - float(b)) < 1e-6 + + +def test_corpus_structure() -> None: + """20 unique, fully-specified rows with the expected gate partition.""" + assert len(_CASES) == _EXPECTED_TOTAL + assert len(set(_IDS)) == _EXPECTED_TOTAL, "duplicate case_id" + for c in _CASES: + missing = _REQUIRED_FIELDS - c.keys() + assert not missing, f"{c.get('case_id')} missing fields: {missing}" + assert c["target_verdict"] in {"solve", "refuse"} + # gate ↔ target_verdict consistency + if c["gate"] == "baseline": + assert c["target_verdict"] == "solve" + if c["target_verdict"] == "refuse": + assert c["gate"] == "permanent", ( + f"{c['case_id']}: refuse target must be a permanent gate" + ) + + baseline = [c for c in _CASES if c["gate"] == "baseline"] + permanent = [c for c in _CASES if c["gate"] == "permanent"] + future = [c for c in _CASES if c["gate"].startswith("5b-")] + assert len(baseline) == _EXPECTED_BASELINE_CONTROLS + assert len(permanent) == _EXPECTED_PERMANENT + assert len(future) == _EXPECTED_FUTURE_POSITIVE + assert len(baseline) + len(permanent) + len(future) == _EXPECTED_TOTAL + + +@pytest.mark.parametrize("case", _CASES, ids=_IDS) +def test_wrong_zero_firewall(case: dict) -> None: + """Forever: a case refuses or answers exactly gold; null-gold ⇒ refuse.""" + res = parse_and_solve(case["question"]) + if res.is_admitted: + assert case["gold"] is not None, ( + f"{case['case_id']}: admitted {res.answer!r} but gold is null" + ) + assert _num_eq(res.answer, case["gold"]), ( + f"{case['case_id']}: WRONG — admitted {res.answer!r} != gold {case['gold']!r}" + ) + + +@pytest.mark.parametrize( + "case", [c for c in _CASES if c["gate"] == "baseline"], ids=[c["case_id"] for c in _CASES if c["gate"] == "baseline"] +) +def test_baseline_controls_still_solve(case: dict) -> None: + """Forever regression net: baseline controls solve to gold.""" + res = parse_and_solve(case["question"]) + assert res.is_admitted, f"{case['case_id']}: control regressed to refuse" + assert _num_eq(res.answer, case["gold"]) + + +@pytest.mark.parametrize( + "case", [c for c in _CASES if c["gate"] == "permanent"], ids=[c["case_id"] for c in _CASES if c["gate"] == "permanent"] +) +def test_permanent_refusals_still_refuse(case: dict) -> None: + """Forever: permanent hard-negatives never admit.""" + res = parse_and_solve(case["question"]) + assert not res.is_admitted, ( + f"{case['case_id']}: permanent refusal admitted {res.answer!r}" + ) + + +@pytest.mark.parametrize( + "case", + [c for c in _CASES if c["gate"] in {"baseline", "permanent"}], + ids=[c["case_id"] for c in _CASES if c["gate"] in {"baseline", "permanent"}], +) +def test_frozen_baseline_fields_match_tree(case: dict) -> None: + """Non-positive rows never drift, so their recorded baseline snapshot must + still equal the live result. (Future positives are intentionally excluded — + their snapshot is what a 5b slice flips.)""" + res = parse_and_solve(case["question"]) + live_verdict = "solve" if res.is_admitted else "refuse" + assert live_verdict == case["baseline_verdict"] + assert res.branches_enumerated == case["baseline_branches_enumerated"] + if res.is_admitted: + assert _num_eq(res.answer, case["baseline_answer"]) + else: + assert case["baseline_answer"] is None + + +def test_current_baseline_snapshot() -> None: + """Current aggregate is 4 solve / 16 refuse / 0 wrong. + + This is the single assertion a Phase 5b slice updates when it flips a + positive (refuse -> solve); the forever-invariants above do not change. + """ + solve = refuse = wrong = 0 + for case in _CASES: + res = parse_and_solve(case["question"]) + if res.is_admitted: + solve += 1 + if case["gold"] is None or not _num_eq(res.answer, case["gold"]): + wrong += 1 + else: + refuse += 1 + assert wrong == 0 + assert (solve, refuse) == (4, 16), ( + f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b " + f"slice landed, update this expectation and the affected rows' " + f"baseline fields in lockstep" + )