core/docs/decisions/ADR-0114a.5-perturbation-suite.md

ADR-0114a.5 — Reasoning-Isolation Perturbation Suite (Obligation #5, B3)

Status: Accepted
Date: 2026-05-23
Parent ADR: ADR-0114a (math-capability promotion obligations)
Methodology: ADR-0125 (reasoning-isolation perturbation methodology)
Auditor pattern: PR #189 (core/capability/pack_provenance.py — validate_lane)
B3 substrate: ADR-0131.3 (bounded-grammar word-problem benchmark)

---

Context

ADR-0114a Obligation #5 requires a programmatic perturbation suite that separates concept-stable reasoning from surface pattern matching. Perturbations are either invariance-preserving (answer must not change) or invariance-breaking (answer must change by a predicted delta).

ADR-0125 discharged this obligation for the GSM8K-style parser-dev lane. This ADR discharges it for the B3 bounded-grammar lane — the lane whose cases live in evals/math_bounded_grammar/v1/cases.jsonl and are gated by ADR-0131's composite-expert promotion check.

The perturbation suite operates on problem text only — it never modifies the parser, solver, graph schema, or authored cases. Both rate thresholds are ε = 0 (binary, per ADR-0120 §"Threshold rationale"): invariance-preserving rate == 1.0 AND invariance-breaking rate == 1.0.

---

Decision

core/capability/perturbation_b3.py exports:

generate_b3_perturbations(case_id, problem, expected_answer, expected_unit)
    -> list[B3Perturbation]

skip_reasons_b3(case_id, problem, expected_answer, expected_unit)
    -> dict[str, str]

validate_perturbation_suite(lane_id, cases_path) -> PerturbationReport

and the frozen, slotted B3Perturbation and PerturbationReport records. All generators are pure: same inputs → same output set, byte-equal.

---

Closed Perturbation Taxonomy

Invariance-Preserving Transforms

Transform	Description	Skip condition
entity_rename_v1	Rename entities via pool row 1: Sam→Alex, Tom→Carol, Bob→David	No B3 entity matched
entity_rename_v2	Rename via pool row 2: Sam→Pat, Tom→Robin, Bob→Jordan	No B3 entity matched
entity_rename_v3	Rename via pool row 3: Sam→Quinn, Tom→Morgan, Bob→Blake	No B3 entity matched
unit_synonym	Replace count unit consistently (apples↔oranges, dollars↔cents)	Unit has no synonym, or problem contains rate $N syntax
commutative_reorder	Swap consecutive initial-possession sentences for a single entity with two distinct units	No single-entity multi-unit consecutive init pair found (all current B3 cases)

Entity substitution pool (complete)

Source	v1 target	v2 target	v3 target
Sam	Alex	Pat	Quinn
Tom	Carol	Robin	Morgan
Bob	David	Jordan	Blake
Birds	Birds (unchanged — collective noun)	Birds	Birds

Unit synonym pool (pack-aligned against en_units_v1)

Source	Target
apples	oranges
oranges	apples
dollars	cents
cents	dollars

Synonyms are verified parseable by _canonical_unit at generation time. Rate cases (those containing $N literal syntax) are excluded because the rate-declaration pattern hardcodes $; substituting the unit alone would produce a malformed rate sentence.

Invariance-Breaking Transforms

Transform	Description	Predicted delta	Skip condition
value_replacement_init	Replace first initial-possession value with value + 2	new_answer − original_answer	No initial-possession value found; or replacement yields same answer
value_replacement_op	Replace first operation numeric value with value + 2	new_answer − original_answer	No add/subtract operation with distinct numeric operand found; or replacement yields same answer
op_verb_flip	Swap first add/subtract verb to its conjugate (buys↔loses family)	new_answer − original_answer	Operation verb is multiply/divide/transfer/compare (not in flip table)

Verb flip table (complete closed set)

Source	Target
buys	loses
loses	buys
gets	eats
eats	gets
receives	loses
earns	spends
spends	earns
finds	loses
adds	loses
sells	gets
donates	gets
uses	gets
drops	gets
removes	gets
sends	buys

For invariance-breaking transforms, the predicted expected answer is computed by running parse_problem + solve on the perturbed text at generation time and stored in B3Perturbation.expected_answer. The scorer re-runs the pipeline and verifies the stored value is reproduced exactly.

---

Acceptance Evidence

Accepted when:

• core capability perturbation exits 0 with:
  • preserving_rate: 1.0000 (130/130 applicable variants)
  • breaking_rate: 1.0000 (68/68 applicable variants)
• tests/test_adr_0114a_5_perturbation.py 15/15 green
• B3 axis lane report unchanged (wrong == 0 preserved)
• ADR linked from docs/decisions/README.md

Current run (2026-05-23, main B3 at ADR-0131.3)

lane_id:                B3_bounded_grammar
cases_total:            50
cases_expected_correct: 35
preserving:             130/130 = 1.0000
breaking:               68/68  = 1.0000
obligation_5_passed:    True

Skip summary (documented, not silent):

Transform	Cases skipped
commutative_reorder	35 (all solved_correct — no single-entity multi-unit init state exists yet)
op_verb_flip	15 (multiply/divide/compare/transfer cases)
value_replacement_init	7 (initial-possession value not parseable in isolation, or cancels)
value_replacement_op	15 (no distinct operation numeric operand)
unit_synonym	7 (rate-declaration $ syntax cases)
entity_rename_v1/v2/v3	1 each (b3-013 "Birds" collective noun maps to itself — no substitution)

---

ADR-0114a Obligation Discharge Summary

Obligation #5 requirement	Status under ADR-0114a.5
Invariance-preserving rate == 1.0	Discharged — 130/130 = 1.0000
Invariance-breaking rate == 1.0	Discharged — 68/68 = 1.0000
Closed perturbation taxonomy	Discharged — all transforms documented above
Deterministic / byte-equal report	Discharged — SHA-256 digest pinned in CI
Refusal for missing lane file	Discharged — typed PerturbationReport with obligation_5_passed=False

---

Consequences

• ADR-0114a Obligation #5 is now discharged for B3. B1 and B2 equivalents are deferred to separate sub-ADRs (mirroring PR #189's "B3 only, others deferred" structure).
• The perturbation suite operates exclusively on input-text strings; it never touches the parser, solver, graph schema, or authored cases. The wrong == 0 firewall on the B3 axis lane is unaffected.
• commutative_reorder generates 0 variants for all current B3 cases (no single-entity multi-unit initial state exists yet). The transform is fully implemented and covered by a unit test; future cases that meet the criterion will be automatically scored.
• Solver bugs revealed by perturbation are out of scope for this ADR. If a perturbed problem reveals a solver defect, file a follow-up ADR; do NOT patch the solver from inside this PR.

---

Out of Scope

• B1 (symbolic equivalence) and B2 (teaching corpus) perturbation suites.
• Cross-sentence semantic perturbations (pronoun rewrites, paraphrasing) — those are coverage axes, not reasoning isolation.
• ADR-0114a obligations #2 (OOD), #6 (depth curve), #8 (adversarial).
• Composite-gate wiring changes (ADR-0131.4 already shipped).
• LLMs, sampling, stochastic generation, or approximate recall.