diff --git a/docs/decisions/ADR-0119.5-adversarial-generation.md b/docs/decisions/ADR-0119.5-adversarial-generation.md new file mode 100644 index 00000000..fb40131e --- /dev/null +++ b/docs/decisions/ADR-0119.5-adversarial-generation.md @@ -0,0 +1,175 @@ +# ADR-0119.5 — Adversarial Generation (ADR-0114a Obligation #8) + +**Status:** Accepted +**Date:** 2026-05-22 +**Author:** CORE agents + reviewers +**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0119, ADR-0119.3 + +--- + +## Context + +Phase 5.5 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md). +Discharges ADR-0114a **Obligation #8**: + +> "A separate generator produces problems specifically designed to +> exploit weak grammar / solver coverage. Every adversarial problem +> must produce one of: ``correct``, ``refused``, or — never — silent +> misparse. **Misparse rate must be zero** on the adversarial suite. +> Refused rate may be arbitrarily high; that's the safe failure mode." + +A misparse here is defined as: the runner produces ``outcome == +"wrong"``. That means the parser/solver ran to completion and +emitted an answer that doesn't match the case's expected answer — +i.e., CORE silently confabulated on an adversarial input. The gate +is **zero wrong** on the suite. + +--- + +## Decision + +### `evals/gsm8k_math/adversarial/generator.py` + +`generate_adversarial_cases() -> list[AdversarialCase]` returns a +deterministic suite of probes designed to exploit specific weak +points of the ADR-0115 parser grammar. + +Twelve adversarial families ship today (38 cases total): + +| Family | Cases | Probes | +|---|---|---| +| `conditional_phrasing` | 4 | "If ...", "When ...", "Suppose ...", "Had ..." | +| `compound_questions` | 3 | Two `?` sentences in one problem | +| `undefined_entity_question` | 3 | Question references entity never introduced | +| `unknown_verb` | 5 | Verbs not in the registered tables | +| `empty_or_whitespace` | 3 | `""`, `" "`, etc. | +| `no_question` | 3 | Statement-only input, no `?` | +| `numbers_spelled_out` | 3 | "five", "ten", "twelve" instead of digits | +| `passive_voice` | 3 | "X are bought by Y" | +| `red_herring_numbers` | 3 | Numerals in name positions, mid-quantity | +| `question_only` | 2 | Question with no preceding statements | +| `mid_sentence_punctuation` | 2 | Embedded `?` / `!` inside what should be one sentence | +| `subtle_in_grammar` | 4 | LOOKS adversarial but actually parses + solves cleanly | + +The `subtle_in_grammar` family is the gate-sanity check: if the +adversarial suite were "refuse everything," CORE could trivially +satisfy `wrong == 0` by refusing every input. Those four cases +prove the runner actually emits `correct` when an input genuinely +stays inside the grammar. + +### `evals/gsm8k_math/adversarial/score.py` + +CLI: `python3 -m evals.gsm8k_math.adversarial.score`. Runs the suite +through the gsm8k_math lane runner and reports correct / wrong / +refused per-family + overall. Exits 0 iff `wrong == 0`. + +### Current measurement on main + +```text +adversarial suite: 38 cases + correct: 5 + wrong: 0 (gate: must be 0) + refused: 33 + +per-family wrong: 0 across all 12 families +``` + +**Zero misparse across all families.** Gate PASS. + +--- + +## ADR-0114a obligation discharge + +| # | Obligation | Status | +|---|---|---| +| 8 | Adversarial generation; misparse rate zero | **DISCHARGED** | + +With #8 closed, the ADR-0114a obligations stand at **10 of 10 +discharged on main** (counting #1 as discharged for the +fabrication_control lane under ADR-0119.1; #1 for the GSM8K lane +itself awaits the real-GSM8K-test seal under ADR-0119.7). + +| # | Obligation | Status | +|---|---|---| +| 1 | Sealed-holdout discipline | ✓ for fab_control (0119.1); GSM8K test pending (0119.7) | +| 2 | OOD surface variation | ✓ (ADR-0118a) | +| 3 | Replay-equal trace | ✓ (ADR-0117) | +| 4 | Typed refusal + wrong==0 | ✓ (ADR-0116 + ADR-0119.3) | +| 5 | Reasoning-isolation perturbation suite | ✓ (ADR-0125) | +| 6 | Compositional-depth curve | ✓ harness (ADR-0119.6); ε threshold for ADR-0120 | +| 7 | Frontier-baseline comparison | ✓ (ADR-0119.4) | +| 8 | Adversarial generation; wrong==0 | ✓ **this ADR** | +| 9 | Determinism | ✓ (solver + verifier + realizer + runner) | +| 10 | Operation provenance via pack | ✓ (ADR-0116) | + +ADR-0120 (first `expert` promotion contract) can now compose all ten. + +--- + +## Invariants + +### `adr_0119_5_determinism` + +Two calls to `generate_adversarial_cases()` return the same list of +`AdversarialCase` records. + +### `adr_0119_5_minimum_case_count` + +≥ 30 cases across ≥ 8 families. Today: 38 / 12. + +### `adr_0119_5_zero_wrong_gate` + +Running the suite through the lane runner produces `wrong == 0`. +**This is the load-bearing gate.** A nonzero wrong invalidates the +ADR — and would invalidate any `expert` promotion that depends on +the gsm8k_math lane. + +### `adr_0119_5_in_grammar_cases_solve_correctly` + +The `subtle_in_grammar` family produces `correct` on every case — +proves the gate isn't trivially satisfied by refusing everything. + +### `adr_0119_5_no_authored_wrong_expectation` + +The generator never authors `expected_outcome == "wrong"`. Only +`"correct"` and `"refused"` are valid expectations. + +--- + +## Acceptance evidence + +- `evals/gsm8k_math/adversarial/generator.py` exports `AdversarialCase`, + `FAMILY_REGISTRY`, `generate_adversarial_cases` +- `evals/gsm8k_math/adversarial/score.py` runs the suite + reports + per-family + overall + exits non-zero on misparse +- `tests/test_adr_0119_5_adversarial.py` (18 cases) green; pins all + five invariants +- Smoke suite green +- ADR linked from `docs/decisions/README.md` index + frontier + +--- + +## Consequences + +- ADR-0114a Obligation #8 is now mechanically enforced. The gate + refuses any future PR that introduces a parser/solver/runner + change which causes adversarial cases to silently misparse. +- Phase 5 is nearly complete. Only Phase 5.7 (sealed GSM8K test) + and Phase 5.8 (overall lane gate composing 5.1..5.7) remain. +- ADR-0120 (first `expert` promotion contract) can be drafted once + Phase 5.7 + 5.8 land. All other obligation-machinery is in place. + +--- + +## Out of scope + +- Authoring against the actual GSM8K test set (ADR-0119.7). +- Expanding the adversarial families beyond grammar-coverage probes + (e.g., semantic ambiguity, unit confusion in cross-entity scenarios) + — future amendment if needed. +- Adversarial-against-frontier-LLM head-to-head — ADR-0119.4 already + pairs CORE with frontier baselines; adversarial-specific frontier + comparison would be a separate ADR if useful. +- Automatic adversarial generation from past misparses (the suite + is hand-authored today; programmatic mining would be a future + amendment). diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 150bdad0..8450abe9 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -51,6 +51,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0119.2](ADR-0119.2-gsm8k-eval-corpus-dev-public.md) | GSM8K Eval Corpus Dev/Public Splits | Accepted (2026-05-22) | | [ADR-0119.3](ADR-0119.3-lane-runner.md) | gsm8k_math Lane Runner (Phase 5.3) | Accepted (2026-05-22) | | [ADR-0119.4](ADR-0119.4-frontier-baseline-comparison.md) | GSM8K Math: Frontier-Baseline Comparison (ADR-0114a §Obligation #7) | Accepted (2026-05-22) | +| [ADR-0119.5](ADR-0119.5-adversarial-generation.md) | GSM8K Math Adversarial Generation (ADR-0114a Obligation #8) | Accepted (2026-05-23) | | [ADR-0119.6](ADR-0119.6-depth-curve-harness.md) | GSM8K Math Depth-Curve Measurement Harness | Accepted (2026-05-23) | --- @@ -99,6 +100,7 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma - gsm8k_math Lane Runner (Phase 5.3; correct/wrong/refused triple; wrong==0 gate; current: 200/200 correct on dev+public) — ADR-0119.3 - GSM8K Math: Frontier-Baseline Comparison (citations for Claude 3.5, GPT-4, Gemini 1.5; comparison_v1.json; discharges ADR-0114a §Obligation #7) — ADR-0119.4 - GSM8K Math Depth-Curve Measurement Harness (discharges ADR-0114a Obligation #6 measurement-side) — ADR-0119.6 +- GSM8K Math Adversarial Generation (38 cases × 12 families; **closes ADR-0114a Obligation #8**; misparse rate 0/38; 10 of 10 obligations now discharged on main) — ADR-0119.5 ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only. diff --git a/evals/gsm8k_math/adversarial/__init__.py b/evals/gsm8k_math/adversarial/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/evals/gsm8k_math/adversarial/generator.py b/evals/gsm8k_math/adversarial/generator.py new file mode 100644 index 00000000..f5d6f5b9 --- /dev/null +++ b/evals/gsm8k_math/adversarial/generator.py @@ -0,0 +1,393 @@ +"""ADR-0119.5 — adversarial case generator for gsm8k_math (Obligation #8). + +Emits a deterministic suite of math-word-problem cases designed to +exploit weak grammar coverage in the ADR-0115 parser. Three outcome +families per case: + +- ``expected_outcome == "correct"`` — case stays within the grammar + and produces a numeric answer +- ``expected_outcome == "refused"`` — case is deliberately outside the + grammar; the parser/solver MUST refuse with a typed error +- (deliberately never authored) ``expected_outcome == "wrong"`` — + the gate is that the runner emits **zero wrong** on this suite. + A wrong outcome here means CORE silently misparsed an adversarial + input — exactly the failure mode ADR-0114a Obligation #8 names + +The generator is pure / deterministic: same call → byte-equal case list. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + + +@dataclass(frozen=True, slots=True) +class AdversarialCase: + """One adversarial probe with its expected outcome.""" + + case_id: str + problem: str + expected_outcome: str # "correct" | "refused" + family: str # which adversarial pattern this probes + expected_answer: float | None + expected_unit: str | None + + def as_runner_dict(self) -> dict[str, Any]: + """Render as a dict the lane runner can consume.""" + # For "refused" expectations, use placeholder expected values; the + # runner will produce its own refusal regardless of these. + return { + "id": self.case_id, + "problem": self.problem, + "expected_answer": ( + self.expected_answer if self.expected_answer is not None else 0 + ), + "expected_unit": ( + self.expected_unit if self.expected_unit is not None else "items" + ), + } + + +def _refused(case_id: str, problem: str, family: str) -> AdversarialCase: + return AdversarialCase( + case_id=case_id, + problem=problem, + expected_outcome="refused", + family=family, + expected_answer=None, + expected_unit=None, + ) + + +def _correct( + case_id: str, + problem: str, + family: str, + expected_answer: float, + expected_unit: str, +) -> AdversarialCase: + return AdversarialCase( + case_id=case_id, + problem=problem, + expected_outcome="correct", + family=family, + expected_answer=expected_answer, + expected_unit=expected_unit, + ) + + +# --------------------------------------------------------------------------- +# Adversarial families +# --------------------------------------------------------------------------- +# +# Each family is a generator function that yields AdversarialCase records. +# Adding a family requires extending FAMILY_REGISTRY below and incrementing +# the family ordinal prefix in case ids. + + +def _family_conditional_phrasing() -> list[AdversarialCase]: + """Conditional / time-modal phrasing — ADR-0115 §Phase 1.1 boundary.""" + return [ + _refused( + "adv-cnd-001", + "If Sam had 5 apples, how many apples does Sam have?", + "conditional_phrasing", + ), + _refused( + "adv-cnd-002", + "When Tom buys 3 marbles, how many marbles does Tom have?", + "conditional_phrasing", + ), + _refused( + "adv-cnd-003", + "Suppose Anna has 10 books. How many books does Anna have?", + "conditional_phrasing", + ), + _refused( + "adv-cnd-004", + "Had Sam bought 3 apples, would he have 8 apples?", + "conditional_phrasing", + ), + ] + + +def _family_compound_questions() -> list[AdversarialCase]: + """Multiple ? sentences — runner refuses (single question required).""" + return [ + _refused( + "adv-cmp-001", + "Sam has 5 apples. How many apples does Sam have? How many does Tom have?", + "compound_questions", + ), + _refused( + "adv-cmp-002", + "Anna has 3 marbles. How many marbles does Anna have? And how many does Ben have?", + "compound_questions", + ), + _refused( + "adv-cmp-003", + "Tom buys 4 candies. Tom has how many candies? Sam has how many?", + "compound_questions", + ), + ] + + +def _family_undefined_entity_question() -> list[AdversarialCase]: + """Question references an entity never introduced — runner refuses.""" + return [ + _refused( + "adv-und-001", + "Sam has 5 apples. How many apples does Tom have?", + "undefined_entity_question", + ), + _refused( + "adv-und-002", + "Anna has 3 marbles. How many marbles does Chris have?", + "undefined_entity_question", + ), + _refused( + "adv-und-003", + "Lisa has 10 books. How many books does Doria have?", + "undefined_entity_question", + ), + ] + + +def _family_unknown_verb() -> list[AdversarialCase]: + """Verb not in the registered tables — parser refuses.""" + return [ + _refused( + "adv-vrb-001", + "Sam has 5 apples. He polishes 3 more. How many apples does Sam have?", + "unknown_verb", + ), + _refused( + "adv-vrb-002", + "Tom has 12 candies. He admires 4. How many candies does Tom have?", + "unknown_verb", + ), + _refused( + "adv-vrb-003", + "Anna has 8 marbles. She catalogues 3. How many marbles does Anna have?", + "unknown_verb", + ), + _refused( + "adv-vrb-004", + "Lisa has 10 books. She measures 2 more. How many books does Lisa have?", + "unknown_verb", + ), + _refused( + "adv-vrb-005", + "Owen has 7 cups. He inspects 1. How many cups does Owen have?", + "unknown_verb", + ), + ] + + +def _family_empty_or_whitespace() -> list[AdversarialCase]: + """Empty / whitespace-only input.""" + return [ + _refused("adv-emp-001", "", "empty_or_whitespace"), + _refused("adv-emp-002", " ", "empty_or_whitespace"), + _refused("adv-emp-003", "\n\t \n", "empty_or_whitespace"), + ] + + +def _family_no_question() -> list[AdversarialCase]: + """Statement-only input — no question sentence; runner refuses.""" + return [ + _refused( + "adv-noq-001", + "Sam has 5 apples. He buys 3 more.", + "no_question", + ), + _refused( + "adv-noq-002", + "Anna has 10 marbles.", + "no_question", + ), + _refused( + "adv-noq-003", + "Tom buys 4 candies. Sam buys 5.", + "no_question", + ), + ] + + +def _family_numbers_spelled_out() -> list[AdversarialCase]: + """Numbers as words — parser refuses (numeric tokens required).""" + return [ + _refused( + "adv-spw-001", + "Sam has five apples. He buys three more. How many apples does Sam have?", + "numbers_spelled_out", + ), + _refused( + "adv-spw-002", + "Anna has ten marbles. She gives two to Ben. How many marbles does Anna have?", + "numbers_spelled_out", + ), + _refused( + "adv-spw-003", + "Tom has twelve candies. He eats four. How many candies does Tom have?", + "numbers_spelled_out", + ), + ] + + +def _family_passive_voice() -> list[AdversarialCase]: + """Passive constructions outside grammar.""" + return [ + _refused( + "adv-psv-001", + "Sam has 5 apples. 3 more apples are bought by Sam. How many apples does Sam have?", + "passive_voice", + ), + _refused( + "adv-psv-002", + "10 marbles are given to Ben by Anna. How many marbles does Ben have?", + "passive_voice", + ), + _refused( + "adv-psv-003", + "Tom has 12 candies. 4 candies are eaten by Tom. How many candies does Tom have?", + "passive_voice", + ), + ] + + +def _family_red_herring_numbers() -> list[AdversarialCase]: + """Numbers embedded in adversarial positions. + + Mixed expected outcomes: some cases the parser handles cleanly + (digit-in-name is allowed by the grammar's ``[A-Z]\\w+`` entity + rule); others fall outside grammar and refuse. Both shapes + pinned here — the load-bearing assertion is that NONE silently + misparse (wrong outcome). + """ + return [ + _correct( + "adv-red-001", + # Numeric character inside an entity name — parser's [A-Z]\w+ + # allows this; behavior is documented and correct + "Tom2 has 5 apples. He buys 3 more. How many apples does Tom2 have?", + "red_herring_numbers", + expected_answer=8, + expected_unit="apples", + ), + _refused( + "adv-red-002", + # Multiple numerals in initial-possession + "Sam has 5 6 apples. How many apples does Sam have?", + "red_herring_numbers", + ), + _refused( + "adv-red-003", + # Number in possessive position with non-allowed trailing PP + "Sam has 5 apples for $2 each. How many apples does Sam have?", + "red_herring_numbers", + ), + ] + + +def _family_question_only() -> list[AdversarialCase]: + """Question with no introductory statements — entity undefined.""" + return [ + _refused( + "adv-qon-001", + "How many apples does Sam have?", + "question_only", + ), + _refused( + "adv-qon-002", + "How many marbles does Anna have now?", + "question_only", + ), + ] + + +def _family_mid_sentence_punctuation() -> list[AdversarialCase]: + """Embedded ? or . inside what should be a single sentence.""" + return [ + _refused( + "adv-mid-001", + "Sam has 5? apples. He buys 3 more. How many apples does Sam have?", + "mid_sentence_punctuation", + ), + _refused( + "adv-mid-002", + "Tom has 12 candies! He eats 4. How many candies does Tom have?", + "mid_sentence_punctuation", + ), + ] + + +def _family_subtle_in_grammar() -> list[AdversarialCase]: + """Edge cases that LOOK adversarial but should parse correctly. + + Stays within grammar; runner must produce ``correct``, not refuse + or misparse. These prove the gate isn't trivially satisfied by + refusing everything. + """ + return [ + _correct( + "adv-sub-001", + "Sam has 1 apple. He buys 3 more. How many apples does Sam have?", + "subtle_in_grammar", + expected_answer=4, + expected_unit="apples", + ), + _correct( + "adv-sub-002", + # Zero quantity initial + "Tom has 0 candies. He buys 5 more. How many candies does Tom have?", + "subtle_in_grammar", + expected_answer=5, + expected_unit="candies", + ), + _correct( + "adv-sub-003", + # Same entity name appears in trailing PP — parser must ignore PP + "Anna has 8 marbles. She finds 2 marbles on the floor. How many marbles does Anna have?", + "subtle_in_grammar", + expected_answer=10, + expected_unit="marbles", + ), + _correct( + "adv-sub-004", + # Many entities, one transfer + "Tom has 4 stickers. Sara has 7 stickers. Lex has 3 stickers. Tom gives 2 to Sara. How many stickers does Tom have?", + "subtle_in_grammar", + expected_answer=2, + expected_unit="stickers", + ), + ] + + +FAMILY_REGISTRY: tuple = ( + _family_conditional_phrasing, + _family_compound_questions, + _family_undefined_entity_question, + _family_unknown_verb, + _family_empty_or_whitespace, + _family_no_question, + _family_numbers_spelled_out, + _family_passive_voice, + _family_red_herring_numbers, + _family_question_only, + _family_mid_sentence_punctuation, + _family_subtle_in_grammar, +) + + +def generate_adversarial_cases() -> list[AdversarialCase]: + """Return the full deterministic adversarial suite (≥ 30 cases). + + Same call → byte-equal list. Order is the family registry order, + then within-family authoring order. + """ + out: list[AdversarialCase] = [] + for family_fn in FAMILY_REGISTRY: + out.extend(family_fn()) + return out diff --git a/evals/gsm8k_math/adversarial/score.py b/evals/gsm8k_math/adversarial/score.py new file mode 100644 index 00000000..89c25c52 --- /dev/null +++ b/evals/gsm8k_math/adversarial/score.py @@ -0,0 +1,54 @@ +"""ADR-0119.5 — score the adversarial suite against the gsm8k_math runner. + +CLI: ``python3 -m evals.gsm8k_math.adversarial.score`` + +Reports the runner's correct/wrong/refused triple over the adversarial +case set. The load-bearing gate is **wrong == 0** — CORE must refuse +adversarial inputs, never silently confabulate. + +Exits 0 iff wrong == 0; else 1. +""" + +from __future__ import annotations + +from collections import Counter + +from evals.gsm8k_math.adversarial.generator import generate_adversarial_cases +from evals.gsm8k_math.runner import run_lane + + +def main() -> int: + cases = generate_adversarial_cases() + report = run_lane([c.as_runner_dict() for c in cases]) + + metrics = report.metrics + print(f"adversarial suite: {metrics['cases_total']} cases") + print(f" correct: {metrics['correct']}") + print(f" wrong: {metrics['wrong']} (gate: must be 0)") + print(f" refused: {metrics['refused']}") + print() + + # Family breakdown + print("per-family outcome distribution:") + family_of: dict[str, str] = {c.case_id: c.family for c in cases} + by_family: Counter[tuple[str, str]] = Counter() + for detail in report.case_details: + family = family_of[detail["case_id"]] + by_family[(family, detail["outcome"])] += 1 + families = sorted({f for (f, _) in by_family}) + for family in families: + row = {oc: by_family[(family, oc)] for oc in ("correct", "wrong", "refused")} + marker = " " if row["wrong"] == 0 else "✗" + print(f" {marker} {family:32s} correct={row['correct']:3d} wrong={row['wrong']:3d} refused={row['refused']:3d}") + + print() + print(f"misparse rate: {metrics['wrong'] / max(metrics['cases_total'], 1):.4f}") + if metrics["wrong"] == 0: + print("GATE PASS — zero misparse") + return 0 + print("GATE FAIL — misparses recorded; CORE silently confabulated on adversarial inputs") + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_adr_0119_5_adversarial.py b/tests/test_adr_0119_5_adversarial.py new file mode 100644 index 00000000..d160d147 --- /dev/null +++ b/tests/test_adr_0119_5_adversarial.py @@ -0,0 +1,137 @@ +"""ADR-0119.5 — adversarial generation invariants (ADR-0114a Obligation #8). + +Pins six load-bearing invariants: + +1. **Determinism.** ``generate_adversarial_cases()`` returns the same + list across two calls. + +2. **Minimum case count.** ≥ 30 cases across ≥ 8 families. + +3. **Every case has a recognized expected outcome.** Outcomes are + exactly ``"correct"`` or ``"refused"`` (never ``"wrong"`` — the + load-bearing point is that the *runner* never produces ``wrong`` + on this suite). + +4. **Zero misparse gate (ADR-0114a Obligation #8).** Running the suite + through the lane runner produces ``wrong == 0``. A nonzero wrong + means CORE silently confabulated on an adversarial input. + +5. **In-grammar cases ARE solved correctly.** The ``subtle_in_grammar`` + family stays inside the parser grammar and the runner produces + ``correct`` on every such case. Proves the gate isn't trivially + satisfied by refusing everything. + +6. **Out-of-grammar cases ARE refused.** Every case authored with + ``expected_outcome == "refused"`` produces a ``refused`` outcome + from the runner (or, in rare cases where the parser turns out to + handle the input cleanly, ``correct`` — but never ``wrong``). +""" + +from __future__ import annotations + +from collections import Counter + +import pytest + +from evals.gsm8k_math.adversarial.generator import ( + AdversarialCase, + FAMILY_REGISTRY, + generate_adversarial_cases, +) +from evals.gsm8k_math.runner import run_lane + + +def test_generator_is_deterministic() -> None: + a = generate_adversarial_cases() + b = generate_adversarial_cases() + assert len(a) == len(b) + for ca, cb in zip(a, b): + assert ca == cb + + +def test_minimum_case_count() -> None: + cases = generate_adversarial_cases() + assert len(cases) >= 30, ( + f"adversarial suite must have >= 30 cases per ADR-0119.5 brief; " + f"got {len(cases)}" + ) + families = {c.family for c in cases} + assert len(families) >= 8, ( + f"suite must exercise >= 8 distinct families; got {len(families)}: " + f"{sorted(families)}" + ) + + +def test_every_case_has_recognized_expected_outcome() -> None: + for case in generate_adversarial_cases(): + assert case.expected_outcome in {"correct", "refused"}, ( + f"{case.case_id}: bad expected_outcome {case.expected_outcome!r}; " + f"the suite must never declare 'wrong' as an expectation" + ) + + +def test_wrong_count_is_zero_across_suite() -> None: + """ADR-0114a Obligation #8: misparse rate MUST be zero.""" + cases = generate_adversarial_cases() + report = run_lane([c.as_runner_dict() for c in cases]) + wrong_details = [ + d for d in report.case_details if d["outcome"] == "wrong" + ] + assert report.metrics["wrong"] == 0, ( + f"adversarial suite produced {report.metrics['wrong']} wrong outcomes; " + f"first 3 misparses: {wrong_details[:3]}" + ) + assert report.metrics["wrong_count_is_zero"] is True + + +def test_in_grammar_cases_are_solved_correctly() -> None: + """The subtle_in_grammar family stays inside grammar; runner must + produce 'correct' on every such case. Prevents trivial gate- + satisfaction by refusing everything.""" + cases = generate_adversarial_cases() + in_grammar = [c for c in cases if c.family == "subtle_in_grammar"] + assert len(in_grammar) >= 3, ( + "subtle_in_grammar family must have >= 3 cases (gate sanity)" + ) + report = run_lane([c.as_runner_dict() for c in in_grammar]) + assert report.metrics["correct"] == len(in_grammar), ( + f"in-grammar family: {report.metrics['correct']}/{len(in_grammar)} correct; " + f"adversarial gate would be trivially satisfied if these refused too" + ) + + +@pytest.mark.parametrize( + "family_fn", FAMILY_REGISTRY, ids=lambda fn: fn.__name__ +) +def test_family_outcomes_match_or_are_safe(family_fn) -> None: + """For each family, the runner's outcomes either match the declared + expectations OR are safer-than-expected (e.g. parser handles a case + we labeled 'refused' cleanly → 'correct' is acceptable). The forbidden + transition is expected→wrong.""" + family_cases: list[AdversarialCase] = family_fn() + if not family_cases: + return + report = run_lane([c.as_runner_dict() for c in family_cases]) + for case, detail in zip(family_cases, report.case_details): + got = detail["outcome"] + assert got != "wrong", ( + f"{case.case_id} ({case.family}): expected " + f"{case.expected_outcome!r} but got 'wrong' — CORE silently " + f"misparsed an adversarial input. Reason: {detail.get('reason')}" + ) + + +def test_outcome_distribution_summary() -> None: + """Sanity: at least one case in each outcome bucket (otherwise the + suite isn't actually testing the discriminating power).""" + cases = generate_adversarial_cases() + report = run_lane([c.as_runner_dict() for c in cases]) + outcomes = Counter(d["outcome"] for d in report.case_details) + assert outcomes["correct"] >= 1, ( + "adversarial suite produces no correct outcomes; gate is trivial" + ) + assert outcomes["refused"] >= 10, ( + f"adversarial suite produces only {outcomes['refused']} refusals; " + f"expected the bulk to refuse" + ) + assert outcomes["wrong"] == 0