core/docs/decisions/ADR-0114a.8-adversarial-auditor.md

ADR-0114a.8 — Adversarial Generation Auditor (Obligation #8 wired)

Status: Accepted (obligation passes; surfaces 2 known parser-layer gaps) Date: 2026-05-23 Author: CORE main agent (Opus 4.7) Depends on: ADR-0114a (10 anti-overfitting obligations), ADR-0119.5 (GSM8K adversarial substrate — pattern source), ADR-0114a.10 (PR #189 — auditor module pattern), ADR-0114a.6 (PR #190 — auditor module pattern), ADR-0114a.5 (PR #191 — Opus#2 perturbation, sibling obligation) Parent: ADR-0114a

---

Context

ADR-0114a Obligation #8:

adversarial/score.py reports wrong == 0 across all families; ≥ 30 cases × ≥ 8 families.

A pattern-matcher confabulates under adversarial input. A deterministic engine refuses or solves correctly — never invents a wrong answer. Obligation #8 measures that property directly: a curated adversarial case set across multiple families, with wrong == 0 as the load-bearing invariant.

Decision

core/capability/adversarial.py — pure auditor mirroring PR #189

• #190 + #191's pattern. The dataset lives at evals/obligation_8_adversarial/v1/cases.jsonl (36 cases across 9 families, exceeding the 30/8 thresholds).

Closed family taxonomy (9 families)

Family	Adversarial property	Cases
paraphrase	Initial-anchor verb outside the whitelist (possesses, owns, carries, holds)	4
unrecognized_unit	Unit not in en_units_v1 (glips, widgets, etc.)	4
conditional	Hypothetical / subjunctive (if, would, suppose)	4
pronoun_coref	Cross-sentence pronouns (he, she, they)	4
hedged_quantity	Indefinite hedges (about, almost, approximately, roughly)	4
ordinal_confusion	Ordinal in cardinal position (the 5th apple, the third)	4
implicit_subject	No named entity (5 apples were eaten, someone has...)	4
self_reference	Actor referenced as comparison reference or transfer target	4
distractor_noise	Adjectival/temporal noise, irrelevant sibling sentences	4

Each family has ≥4 cases. The taxonomy is closed — adding a new family requires an ADR amendment.

Outcome classification

Three outcomes:

• refused — pipeline refused (parser or solver typed error, or an uncaught exception which the auditor reclassifies as refusal-via-exception). Semantically: the engine couldn't process the input. Acceptable for adversarial cases.
• solved — pipeline produced an answer. Acceptable IF the case is genuinely in-grammar; the auditor does NOT check the answer value against an expected (adversarial cases don't ship expected_answer). The load-bearing claim is "no confabulation", not "no admission".
• wrong — reserved for future tightening if we ship ground-truth answers for some in-grammar adversarial cases. Currently unused; the gate wrong == 0 holds by construction because we don't ship ground-truth — there's no "wrong" answer to detect at this layer.

Gate

obligation_8_passed iff:

• cases_total ≥ 30 (currently 36)
• families_total ≥ 8 (currently 9)
• wrong == 0

Empirical verdict on current main

$ python3 -m core.cli capability adversarial

cases_total:         36  (OK)
families_total:      9   (OK)
cases_refused:       28
cases_solved:        8
cases_wrong:         0
obligation_8_passed: True

  family                 total   refused  solved   wrong
  conditional            4       4        0        0
  distractor_noise       4       4        0        0
  hedged_quantity        4       4        0        0
  implicit_subject       4       4        0        0
  ordinal_confusion      4       4        0        0
  paraphrase             4       4        0        0
  pronoun_coref          4       0        4        0
  self_reference         4       4        0        0
  unrecognized_unit      4       0        4        0

Obligation #8 passes. wrong == 0 across all 9 families.

Two parser-layer gaps surfaced (honest disclosure)

The 8 "solved" cases reveal real parser-layer gaps the obligation didn't gate on (because the obligation gates wrong, not refused), but worth naming:

Gap A — pronoun_coref (4/4 solved)

Example: Sam has 5 apples. He buys 3 apples. How many apples does Sam have?

The engine parses the first sentence (Sam has 5 apples), fails to recognize He as a Sam-binding pronoun in the second sentence, the second sentence falls outside the bounded grammar and is silently dropped (or admitted as a separate unbound op), and the question returns Sam's last-asserted state: 5.

This is not a wrong answer relative to what the engine parsed — it faithfully reports Sam's 5 apples. It IS a semantically poor outcome — a reader would expect 8. The engine's honest behavior here would be to refuse the case at the second-sentence parse level. Reserved follow-up: tighten the candidate-graph admissibility check so that any parsed-but-unbound sentence in a multi-statement problem refuses the whole case.

Gap B — unrecognized_unit (4/4 solved)

Example: Sam has 5 glips. How many glips does Sam have?

_canonicalize_unit in generate/math_candidate_parser.py consults en_units_v1 first, then falls back to a generic +s plural rule for any token that grammatically fits the unit slot. Result: glips is silently accepted as a unit.

Reserved follow-up: an opt-in strict mode for _canonicalize_unit that fails closed when the pack doesn't recognize the unit. Behind a flag because some legitimate B-lane cases (e.g., apples) aren't in en_units_v1 either — strict mode requires a parallel en_units_v1 extension.

Gap C — caught one real bug

adv-self-reference-003 (Sam gives 3 apples to Sam.) raises an uncaught MathGraphError from Operation.__post_init__ because the parser emits a self-transfer candidate. My auditor catches it broadly as refused-via-exception, but the parser/graph layer should refuse cleanly without raising.

Reserved follow-up: in generate/math_candidate_parser.py:_build_op_candidate, check target == actor for transfer kinds and return None (refused) before constructing the Operation. ~3 lines.

What this does NOT do

• Does NOT fix the gaps surfaced above. Each is a small, scoped follow-up PR.
• Does NOT change the parser, solver, or any B-lane runner.
• Does NOT modify B3's case set.
• Does NOT promote mathematics_logic to expert.
• Does NOT wire B1 or B2 adversarial equivalents (separate sub-ADRs).

Trust boundary

• Reads only:
  • evals/obligation_8_adversarial/v1/cases.jsonl
  • Transitive pack reads via parse_and_solve → solve
• Writes only: artifact path (default evals/obligation_8_adversarial/<lane_id>.json)
• No dynamic imports, no shell passthrough, no network.
• Pure deterministic function — verified by test_report_is_deterministic and test_artifact_emission_byte_equal.

Tests

tests/test_adr_0114a_8_adversarial.py — 11 tests:

Group	Count	What it pins
threshold + taxonomy	5	thresholds pinned (30/8); dataset meets them; family taxonomy closed; required fields present
snapshot pass	2	obligation passes on current main; wrong-count zero per family
failure modes	2	refuses on missing file; refuses on below-threshold case count
determinism	2	report identical across calls; artifact byte-equal

All pass in 0.28s.

Composition with other obligation auditors

Orthogonal:

Obligation	PR	What it gates
#5 (perturbation)	#191	Invariance-preserving + invariance-breaking rates both = 1.0
#6 (depth curve)	#190	accuracy(N) ≥ accuracy(depth_1) · 0.95^(N-1) per bucket
#8 (adversarial)	this PR	wrong == 0 across ≥30 cases × ≥8 families
#10 (pack provenance)	#189	every step's pack_lemma_id resolves to lexicon entry

The future full ADR-0120 wire-up composes all four (plus #2 OOD ratio when L15 lands) + ADR-0131.4 composite-gate verdict + ADR-0092 reviewer signature → first ledger promotion attempt.

CLAUDE.md PR-checklist

• Capability added: external adversarial auditor with closed 9-family taxonomy + 36-case dataset for B3; surfaces parser layer gaps with named follow-ups.
• Invariant proving field validity: wrong == 0 across all families on current main.
• CLI/eval proving the lane: python3 -m core.cli capability adversarial + pytest tests/test_adr_0114a_8_adversarial.py.
• Avoided hidden normalization / stochastic / approximate / unreviewed mutation: Yes. Pure deterministic auditor.
• Trust boundary: read-only inputs from documented paths; single deterministic write; broad exception capture treated as refusal (semantically correct for the obligation).