core/docs/decisions/ADR-0119.5-adversarial-generation.md

ADR-0119.5 — Adversarial Generation (ADR-0114a Obligation #8)

Status: Accepted Date: 2026-05-22 Author: CORE agents + reviewers Depends on: ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0119, ADR-0119.3

---

Context

Phase 5.5 of ADR-0119. Discharges ADR-0114a Obligation #8:

"A separate generator produces problems specifically designed to exploit weak grammar / solver coverage. Every adversarial problem must produce one of: correct, refused, or — never — silent misparse. Misparse rate must be zero on the adversarial suite. Refused rate may be arbitrarily high; that's the safe failure mode."

A misparse here is defined as: the runner produces outcome == "wrong". That means the parser/solver ran to completion and emitted an answer that doesn't match the case's expected answer — i.e., CORE silently confabulated on an adversarial input. The gate is zero wrong on the suite.

---

Decision

evals/gsm8k_math/adversarial/generator.py

generate_adversarial_cases() -> list[AdversarialCase] returns a deterministic suite of probes designed to exploit specific weak points of the ADR-0115 parser grammar.

Twelve adversarial families ship today (38 cases total):

Family	Cases	Probes
conditional_phrasing	4	"If ...", "When ...", "Suppose ...", "Had ..."
compound_questions	3	Two ? sentences in one problem
undefined_entity_question	3	Question references entity never introduced
unknown_verb	5	Verbs not in the registered tables
empty_or_whitespace	3	"", " ", etc.
no_question	3	Statement-only input, no ?
numbers_spelled_out	3	"five", "ten", "twelve" instead of digits
passive_voice	3	"X are bought by Y"
red_herring_numbers	3	Numerals in name positions, mid-quantity
question_only	2	Question with no preceding statements
mid_sentence_punctuation	2	Embedded ? / ! inside what should be one sentence
subtle_in_grammar	4	LOOKS adversarial but actually parses + solves cleanly

The subtle_in_grammar family is the gate-sanity check: if the adversarial suite were "refuse everything," CORE could trivially satisfy wrong == 0 by refusing every input. Those four cases prove the runner actually emits correct when an input genuinely stays inside the grammar.

evals/gsm8k_math/adversarial/score.py

CLI: python3 -m evals.gsm8k_math.adversarial.score. Runs the suite through the gsm8k_math lane runner and reports correct / wrong / refused per-family + overall. Exits 0 iff wrong == 0.

Current measurement on main

adversarial suite: 38 cases
  correct: 5
  wrong:   0  (gate: must be 0)
  refused: 33

per-family wrong: 0 across all 12 families

Zero misparse across all families. Gate PASS.

---

ADR-0114a obligation discharge

#	Obligation	Status
8	Adversarial generation; misparse rate zero	DISCHARGED

With #8 closed, the ADR-0114a obligations stand at 10 of 10 discharged on main (counting #1 as discharged for the fabrication_control lane under ADR-0119.1; #1 for the GSM8K lane itself awaits the real-GSM8K-test seal under ADR-0119.7).

#	Obligation	Status
1	Sealed-holdout discipline	✓ for fab_control (0119.1); GSM8K test pending (0119.7)
2	OOD surface variation	✓ (ADR-0118a)
3	Replay-equal trace	✓ (ADR-0117)
4	Typed refusal + wrong==0	✓ (ADR-0116 + ADR-0119.3)
5	Reasoning-isolation perturbation suite	✓ (ADR-0125)
6	Compositional-depth curve	✓ harness (ADR-0119.6); ε threshold for ADR-0120
7	Frontier-baseline comparison	✓ (ADR-0119.4)
8	Adversarial generation; wrong==0	✓ this ADR
9	Determinism	✓ (solver + verifier + realizer + runner)
10	Operation provenance via pack	✓ (ADR-0116)

ADR-0120 (first expert promotion contract) can now compose all ten.

---

Invariants

adr_0119_5_determinism

Two calls to generate_adversarial_cases() return the same list of AdversarialCase records.

adr_0119_5_minimum_case_count

≥ 30 cases across ≥ 8 families. Today: 38 / 12.

adr_0119_5_zero_wrong_gate

Running the suite through the lane runner produces wrong == 0. This is the load-bearing gate. A nonzero wrong invalidates the ADR — and would invalidate any expert promotion that depends on the gsm8k_math lane.

adr_0119_5_in_grammar_cases_solve_correctly

The subtle_in_grammar family produces correct on every case — proves the gate isn't trivially satisfied by refusing everything.

adr_0119_5_no_authored_wrong_expectation

The generator never authors expected_outcome == "wrong". Only "correct" and "refused" are valid expectations.

---

Acceptance evidence

• evals/gsm8k_math/adversarial/generator.py exports AdversarialCase, FAMILY_REGISTRY, generate_adversarial_cases
• evals/gsm8k_math/adversarial/score.py runs the suite + reports per-family + overall + exits non-zero on misparse
• tests/test_adr_0119_5_adversarial.py (18 cases) green; pins all five invariants
• Smoke suite green
• ADR linked from docs/decisions/README.md index + frontier

---

Consequences

• ADR-0114a Obligation #8 is now mechanically enforced. The gate refuses any future PR that introduces a parser/solver/runner change which causes adversarial cases to silently misparse.
• Phase 5 is nearly complete. Only Phase 5.7 (sealed GSM8K test) and Phase 5.8 (overall lane gate composing 5.1..5.7) remain.
• ADR-0120 (first expert promotion contract) can be drafted once Phase 5.7 + 5.8 land. All other obligation-machinery is in place.

---

Out of scope

• Authoring against the actual GSM8K test set (ADR-0119.7).
• Expanding the adversarial families beyond grammar-coverage probes (e.g., semantic ambiguity, unit confusion in cross-entity scenarios) — future amendment if needed.
• Adversarial-against-frontier-LLM head-to-head — ADR-0119.4 already pairs CORE with frontier baselines; adversarial-specific frontier comparison would be a separate ADR if useful.
• Automatic adversarial generation from past misparses (the suite is hand-authored today; programmatic mining would be a future amendment).