# ADR-0119.5 — Adversarial Generation (ADR-0114a Obligation #8)

**Status:** Accepted
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0119, ADR-0119.3

---

## Context

Phase 5.5 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md).
Discharges ADR-0114a **Obligation #8**:

> "A separate generator produces problems specifically designed to
> exploit weak grammar / solver coverage. Every adversarial problem
> must produce one of: ``correct``, ``refused``, or — never — silent
> misparse. **Misparse rate must be zero** on the adversarial suite.
> Refused rate may be arbitrarily high; that's the safe failure mode."

A misparse here is defined as: the runner produces ``outcome ==
"wrong"``. That means the parser/solver ran to completion and
emitted an answer that doesn't match the case's expected answer —
i.e., CORE silently confabulated on an adversarial input. The gate
is **zero wrong** on the suite.

---

## Decision

### `evals/gsm8k_math/adversarial/generator.py`

`generate_adversarial_cases() -> list[AdversarialCase]` returns a
deterministic suite of probes designed to exploit specific weak
points of the ADR-0115 parser grammar.

Twelve adversarial families ship today (38 cases total):

| Family | Cases | Probes |
|---|---|---|
| `conditional_phrasing` | 4 | "If ...", "When ...", "Suppose ...", "Had ..." |
| `compound_questions` | 3 | Two `?` sentences in one problem |
| `undefined_entity_question` | 3 | Question references entity never introduced |
| `unknown_verb` | 5 | Verbs not in the registered tables |
| `empty_or_whitespace` | 3 | `""`, `"   "`, etc. |
| `no_question` | 3 | Statement-only input, no `?` |
| `numbers_spelled_out` | 3 | "five", "ten", "twelve" instead of digits |
| `passive_voice` | 3 | "X are bought by Y" |
| `red_herring_numbers` | 3 | Numerals in name positions, mid-quantity |
| `question_only` | 2 | Question with no preceding statements |
| `mid_sentence_punctuation` | 2 | Embedded `?` / `!` inside what should be one sentence |
| `subtle_in_grammar` | 4 | LOOKS adversarial but actually parses + solves cleanly |

The `subtle_in_grammar` family is the gate-sanity check: if the
adversarial suite were "refuse everything," CORE could trivially
satisfy `wrong == 0` by refusing every input. Those four cases
prove the runner actually emits `correct` when an input genuinely
stays inside the grammar.

### `evals/gsm8k_math/adversarial/score.py`

CLI: `python3 -m evals.gsm8k_math.adversarial.score`. Runs the suite
through the gsm8k_math lane runner and reports correct / wrong /
refused per-family + overall. Exits 0 iff `wrong == 0`.

### Current measurement on main

```text
adversarial suite: 38 cases
  correct: 5
  wrong:   0  (gate: must be 0)
  refused: 33

per-family wrong: 0 across all 12 families
```

**Zero misparse across all families.** Gate PASS.

---

## ADR-0114a obligation discharge

| # | Obligation | Status |
|---|---|---|
| 8 | Adversarial generation; misparse rate zero | **DISCHARGED** |

With #8 closed, the ADR-0114a obligations stand at **10 of 10
discharged on main** (counting #1 as discharged for the
fabrication_control lane under ADR-0119.1; #1 for the GSM8K lane
itself awaits the real-GSM8K-test seal under ADR-0119.7).

| # | Obligation | Status |
|---|---|---|
| 1 | Sealed-holdout discipline | ✓ for fab_control (0119.1); GSM8K test pending (0119.7) |
| 2 | OOD surface variation | ✓ (ADR-0118a) |
| 3 | Replay-equal trace | ✓ (ADR-0117) |
| 4 | Typed refusal + wrong==0 | ✓ (ADR-0116 + ADR-0119.3) |
| 5 | Reasoning-isolation perturbation suite | ✓ (ADR-0125) |
| 6 | Compositional-depth curve | ✓ harness (ADR-0119.6); ε threshold for ADR-0120 |
| 7 | Frontier-baseline comparison | ✓ (ADR-0119.4) |
| 8 | Adversarial generation; wrong==0 | ✓ **this ADR** |
| 9 | Determinism | ✓ (solver + verifier + realizer + runner) |
| 10 | Operation provenance via pack | ✓ (ADR-0116) |

ADR-0120 (first `expert` promotion contract) can now compose all ten.

---

## Invariants

### `adr_0119_5_determinism`

Two calls to `generate_adversarial_cases()` return the same list of
`AdversarialCase` records.

### `adr_0119_5_minimum_case_count`

≥ 30 cases across ≥ 8 families. Today: 38 / 12.

### `adr_0119_5_zero_wrong_gate`

Running the suite through the lane runner produces `wrong == 0`.
**This is the load-bearing gate.** A nonzero wrong invalidates the
ADR — and would invalidate any `expert` promotion that depends on
the gsm8k_math lane.

### `adr_0119_5_in_grammar_cases_solve_correctly`

The `subtle_in_grammar` family produces `correct` on every case —
proves the gate isn't trivially satisfied by refusing everything.

### `adr_0119_5_no_authored_wrong_expectation`

The generator never authors `expected_outcome == "wrong"`. Only
`"correct"` and `"refused"` are valid expectations.

---

## Acceptance evidence

- `evals/gsm8k_math/adversarial/generator.py` exports `AdversarialCase`,
  `FAMILY_REGISTRY`, `generate_adversarial_cases`
- `evals/gsm8k_math/adversarial/score.py` runs the suite + reports
  per-family + overall + exits non-zero on misparse
- `tests/test_adr_0119_5_adversarial.py` (18 cases) green; pins all
  five invariants
- Smoke suite green
- ADR linked from `docs/decisions/README.md` index + frontier

---

## Consequences

- ADR-0114a Obligation #8 is now mechanically enforced. The gate
  refuses any future PR that introduces a parser/solver/runner
  change which causes adversarial cases to silently misparse.
- Phase 5 is nearly complete. Only Phase 5.7 (sealed GSM8K test)
  and Phase 5.8 (overall lane gate composing 5.1..5.7) remain.
- ADR-0120 (first `expert` promotion contract) can be drafted once
  Phase 5.7 + 5.8 land. All other obligation-machinery is in place.

---

## Out of scope

- Authoring against the actual GSM8K test set (ADR-0119.7).
- Expanding the adversarial families beyond grammar-coverage probes
  (e.g., semantic ambiguity, unit confusion in cross-entity scenarios)
  — future amendment if needed.
- Adversarial-against-frontier-LLM head-to-head — ADR-0119.4 already
  pairs CORE with frontier baselines; adversarial-specific frontier
  comparison would be a separate ADR if useful.
- Automatic adversarial generation from past misparses (the suite
  is hand-authored today; programmatic mining would be a future
  amendment).