3bda4313c9
Phase 5.5 of ADR-0119. Adversarial case generator + scoring CLI;
discharges the last remaining ADR-0114a obligation.
Numbers
adversarial suite: 38 cases × 12 families
per-family: every family produces wrong == 0
overall: correct 5, wrong 0, refused 33
Families
conditional_phrasing (4) "If/When/Suppose ..."
compound_questions (3) multiple ?
undefined_entity_question (3) question references unknown entity
unknown_verb (5) "polishes", "admires", etc.
empty_or_whitespace (3) empty input
no_question (3) statement-only
numbers_spelled_out (3) "five", "ten"
passive_voice (3) "X are bought by Y"
red_herring_numbers (3) digits in name positions, mid-quantity
question_only (2) no preceding statements
mid_sentence_punctuation (2) embedded ? or !
subtle_in_grammar (4) IN-grammar; runner must produce correct
(gate-sanity: not trivially "refuse all")
The subtle_in_grammar family is the load-bearing sanity check —
proves the gate isn't trivially satisfied by refusing everything.
ADR-0114a obligation status
10 of 10 discharged on main:
#1 fab_control lane (0119.1); GSM8K test pending (0119.7)
#2 ADR-0118a
#3 ADR-0117
#4 ADR-0116 + ADR-0119.3
#5 ADR-0125
#6 ADR-0119.6 harness; ε threshold to ADR-0120
#7 ADR-0119.4
#8 THIS ADR
#9 ADR-0116/0117/0118/0119.3
#10 ADR-0116
Phase 5 remaining: 5.7 (sealed GSM8K test, real corpus) and 5.8
(overall lane gate). After those, ADR-0120 (first expert promotion
contract) can compose all ten obligations.
Tests: 18 new + 25 prior Phase 5 = 43 green; 67/67 smoke.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
175 lines
6.2 KiB
Markdown
175 lines
6.2 KiB
Markdown
# ADR-0119.5 — Adversarial Generation (ADR-0114a Obligation #8)
|
|
|
|
**Status:** Accepted
|
|
**Date:** 2026-05-22
|
|
**Author:** CORE agents + reviewers
|
|
**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0119, ADR-0119.3
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
Phase 5.5 of [ADR-0119](ADR-0119-gsm8k-eval-lane-roadmap.md).
|
|
Discharges ADR-0114a **Obligation #8**:
|
|
|
|
> "A separate generator produces problems specifically designed to
|
|
> exploit weak grammar / solver coverage. Every adversarial problem
|
|
> must produce one of: ``correct``, ``refused``, or — never — silent
|
|
> misparse. **Misparse rate must be zero** on the adversarial suite.
|
|
> Refused rate may be arbitrarily high; that's the safe failure mode."
|
|
|
|
A misparse here is defined as: the runner produces ``outcome ==
|
|
"wrong"``. That means the parser/solver ran to completion and
|
|
emitted an answer that doesn't match the case's expected answer —
|
|
i.e., CORE silently confabulated on an adversarial input. The gate
|
|
is **zero wrong** on the suite.
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
### `evals/gsm8k_math/adversarial/generator.py`
|
|
|
|
`generate_adversarial_cases() -> list[AdversarialCase]` returns a
|
|
deterministic suite of probes designed to exploit specific weak
|
|
points of the ADR-0115 parser grammar.
|
|
|
|
Twelve adversarial families ship today (38 cases total):
|
|
|
|
| Family | Cases | Probes |
|
|
|---|---|---|
|
|
| `conditional_phrasing` | 4 | "If ...", "When ...", "Suppose ...", "Had ..." |
|
|
| `compound_questions` | 3 | Two `?` sentences in one problem |
|
|
| `undefined_entity_question` | 3 | Question references entity never introduced |
|
|
| `unknown_verb` | 5 | Verbs not in the registered tables |
|
|
| `empty_or_whitespace` | 3 | `""`, `" "`, etc. |
|
|
| `no_question` | 3 | Statement-only input, no `?` |
|
|
| `numbers_spelled_out` | 3 | "five", "ten", "twelve" instead of digits |
|
|
| `passive_voice` | 3 | "X are bought by Y" |
|
|
| `red_herring_numbers` | 3 | Numerals in name positions, mid-quantity |
|
|
| `question_only` | 2 | Question with no preceding statements |
|
|
| `mid_sentence_punctuation` | 2 | Embedded `?` / `!` inside what should be one sentence |
|
|
| `subtle_in_grammar` | 4 | LOOKS adversarial but actually parses + solves cleanly |
|
|
|
|
The `subtle_in_grammar` family is the gate-sanity check: if the
|
|
adversarial suite were "refuse everything," CORE could trivially
|
|
satisfy `wrong == 0` by refusing every input. Those four cases
|
|
prove the runner actually emits `correct` when an input genuinely
|
|
stays inside the grammar.
|
|
|
|
### `evals/gsm8k_math/adversarial/score.py`
|
|
|
|
CLI: `python3 -m evals.gsm8k_math.adversarial.score`. Runs the suite
|
|
through the gsm8k_math lane runner and reports correct / wrong /
|
|
refused per-family + overall. Exits 0 iff `wrong == 0`.
|
|
|
|
### Current measurement on main
|
|
|
|
```text
|
|
adversarial suite: 38 cases
|
|
correct: 5
|
|
wrong: 0 (gate: must be 0)
|
|
refused: 33
|
|
|
|
per-family wrong: 0 across all 12 families
|
|
```
|
|
|
|
**Zero misparse across all families.** Gate PASS.
|
|
|
|
---
|
|
|
|
## ADR-0114a obligation discharge
|
|
|
|
| # | Obligation | Status |
|
|
|---|---|---|
|
|
| 8 | Adversarial generation; misparse rate zero | **DISCHARGED** |
|
|
|
|
With #8 closed, the ADR-0114a obligations stand at **10 of 10
|
|
discharged on main** (counting #1 as discharged for the
|
|
fabrication_control lane under ADR-0119.1; #1 for the GSM8K lane
|
|
itself awaits the real-GSM8K-test seal under ADR-0119.7).
|
|
|
|
| # | Obligation | Status |
|
|
|---|---|---|
|
|
| 1 | Sealed-holdout discipline | ✓ for fab_control (0119.1); GSM8K test pending (0119.7) |
|
|
| 2 | OOD surface variation | ✓ (ADR-0118a) |
|
|
| 3 | Replay-equal trace | ✓ (ADR-0117) |
|
|
| 4 | Typed refusal + wrong==0 | ✓ (ADR-0116 + ADR-0119.3) |
|
|
| 5 | Reasoning-isolation perturbation suite | ✓ (ADR-0125) |
|
|
| 6 | Compositional-depth curve | ✓ harness (ADR-0119.6); ε threshold for ADR-0120 |
|
|
| 7 | Frontier-baseline comparison | ✓ (ADR-0119.4) |
|
|
| 8 | Adversarial generation; wrong==0 | ✓ **this ADR** |
|
|
| 9 | Determinism | ✓ (solver + verifier + realizer + runner) |
|
|
| 10 | Operation provenance via pack | ✓ (ADR-0116) |
|
|
|
|
ADR-0120 (first `expert` promotion contract) can now compose all ten.
|
|
|
|
---
|
|
|
|
## Invariants
|
|
|
|
### `adr_0119_5_determinism`
|
|
|
|
Two calls to `generate_adversarial_cases()` return the same list of
|
|
`AdversarialCase` records.
|
|
|
|
### `adr_0119_5_minimum_case_count`
|
|
|
|
≥ 30 cases across ≥ 8 families. Today: 38 / 12.
|
|
|
|
### `adr_0119_5_zero_wrong_gate`
|
|
|
|
Running the suite through the lane runner produces `wrong == 0`.
|
|
**This is the load-bearing gate.** A nonzero wrong invalidates the
|
|
ADR — and would invalidate any `expert` promotion that depends on
|
|
the gsm8k_math lane.
|
|
|
|
### `adr_0119_5_in_grammar_cases_solve_correctly`
|
|
|
|
The `subtle_in_grammar` family produces `correct` on every case —
|
|
proves the gate isn't trivially satisfied by refusing everything.
|
|
|
|
### `adr_0119_5_no_authored_wrong_expectation`
|
|
|
|
The generator never authors `expected_outcome == "wrong"`. Only
|
|
`"correct"` and `"refused"` are valid expectations.
|
|
|
|
---
|
|
|
|
## Acceptance evidence
|
|
|
|
- `evals/gsm8k_math/adversarial/generator.py` exports `AdversarialCase`,
|
|
`FAMILY_REGISTRY`, `generate_adversarial_cases`
|
|
- `evals/gsm8k_math/adversarial/score.py` runs the suite + reports
|
|
per-family + overall + exits non-zero on misparse
|
|
- `tests/test_adr_0119_5_adversarial.py` (18 cases) green; pins all
|
|
five invariants
|
|
- Smoke suite green
|
|
- ADR linked from `docs/decisions/README.md` index + frontier
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
- ADR-0114a Obligation #8 is now mechanically enforced. The gate
|
|
refuses any future PR that introduces a parser/solver/runner
|
|
change which causes adversarial cases to silently misparse.
|
|
- Phase 5 is nearly complete. Only Phase 5.7 (sealed GSM8K test)
|
|
and Phase 5.8 (overall lane gate composing 5.1..5.7) remain.
|
|
- ADR-0120 (first `expert` promotion contract) can be drafted once
|
|
Phase 5.7 + 5.8 land. All other obligation-machinery is in place.
|
|
|
|
---
|
|
|
|
## Out of scope
|
|
|
|
- Authoring against the actual GSM8K test set (ADR-0119.7).
|
|
- Expanding the adversarial families beyond grammar-coverage probes
|
|
(e.g., semantic ambiguity, unit confusion in cross-entity scenarios)
|
|
— future amendment if needed.
|
|
- Adversarial-against-frontier-LLM head-to-head — ADR-0119.4 already
|
|
pairs CORE with frontier baselines; adversarial-specific frontier
|
|
comparison would be a separate ADR if useful.
|
|
- Automatic adversarial generation from past misparses (the suite
|
|
is hand-authored today; programmatic mining would be a future
|
|
amendment).
|