core/docs/decisions/ADR-0114a.5-perturbation-suite.md

# ADR-0114a.5 — Reasoning-Isolation Perturbation Suite (Obligation #5, B3)

**Status:** Accepted  
**Date:** 2026-05-23  
**Parent ADR:** ADR-0114a (math-capability promotion obligations)  
**Methodology:** ADR-0125 (reasoning-isolation perturbation methodology)  
**Auditor pattern:** PR #189 (`core/capability/pack_provenance.py` — `validate_lane`)  
**B3 substrate:** ADR-0131.3 (bounded-grammar word-problem benchmark)

---

## Context

ADR-0114a Obligation #5 requires a programmatic perturbation suite that
separates concept-stable reasoning from surface pattern matching.
Perturbations are either **invariance-preserving** (answer must not change)
or **invariance-breaking** (answer must change by a predicted delta).

ADR-0125 discharged this obligation for the GSM8K-style parser-dev lane.
This ADR discharges it for the **B3 bounded-grammar lane** — the lane
whose cases live in `evals/math_bounded_grammar/v1/cases.jsonl` and are
gated by ADR-0131's composite-expert promotion check.

The perturbation suite operates on **problem text only** — it never
modifies the parser, solver, graph schema, or authored cases. Both rate
thresholds are ε = 0 (binary, per ADR-0120 §"Threshold rationale"):
invariance-preserving rate == 1.0 AND invariance-breaking rate == 1.0.

---

## Decision

`core/capability/perturbation_b3.py` exports:

```text
generate_b3_perturbations(case_id, problem, expected_answer, expected_unit)
    -> list[B3Perturbation]

skip_reasons_b3(case_id, problem, expected_answer, expected_unit)
    -> dict[str, str]

validate_perturbation_suite(lane_id, cases_path) -> PerturbationReport
```

and the frozen, slotted `B3Perturbation` and `PerturbationReport` records.
All generators are pure: same inputs → same output set, byte-equal.

---

## Closed Perturbation Taxonomy

### Invariance-Preserving Transforms

| Transform | Description | Skip condition |
|---|---|---|
| `entity_rename_v1` | Rename entities via pool row 1: Sam→Alex, Tom→Carol, Bob→David | No B3 entity matched |
| `entity_rename_v2` | Rename via pool row 2: Sam→Pat, Tom→Robin, Bob→Jordan | No B3 entity matched |
| `entity_rename_v3` | Rename via pool row 3: Sam→Quinn, Tom→Morgan, Bob→Blake | No B3 entity matched |
| `unit_synonym` | Replace count unit consistently (apples↔oranges, dollars↔cents) | Unit has no synonym, or problem contains rate `$N` syntax |
| `commutative_reorder` | Swap consecutive initial-possession sentences for a single entity with two distinct units | No single-entity multi-unit consecutive init pair found (all current B3 cases) |

#### Entity substitution pool (complete)

| Source | v1 target | v2 target | v3 target |
|---|---|---|---|
| Sam | Alex | Pat | Quinn |
| Tom | Carol | Robin | Morgan |
| Bob | David | Jordan | Blake |
| Birds | Birds (unchanged — collective noun) | Birds | Birds |

#### Unit synonym pool (pack-aligned against en_units_v1)

| Source | Target |
|---|---|
| apples | oranges |
| oranges | apples |
| dollars | cents |
| cents | dollars |

Synonyms are verified parseable by `_canonical_unit` at generation time.
Rate cases (those containing `$N` literal syntax) are excluded because the
rate-declaration pattern hardcodes `$`; substituting the unit alone would
produce a malformed rate sentence.

### Invariance-Breaking Transforms

| Transform | Description | Predicted delta | Skip condition |
|---|---|---|---|
| `value_replacement_init` | Replace first initial-possession value with value + 2 | new_answer − original_answer | No initial-possession value found; or replacement yields same answer |
| `value_replacement_op` | Replace first operation numeric value with value + 2 | new_answer − original_answer | No add/subtract operation with distinct numeric operand found; or replacement yields same answer |
| `op_verb_flip` | Swap first add/subtract verb to its conjugate (buys↔loses family) | new_answer − original_answer | Operation verb is multiply/divide/transfer/compare (not in flip table) |

#### Verb flip table (complete closed set)

| Source | Target |
|---|---|
| buys | loses |
| loses | buys |
| gets | eats |
| eats | gets |
| receives | loses |
| earns | spends |
| spends | earns |
| finds | loses |
| adds | loses |
| sells | gets |
| donates | gets |
| uses | gets |
| drops | gets |
| removes | gets |
| sends | buys |

For invariance-breaking transforms, the predicted expected answer is
computed by running `parse_problem` + `solve` on the perturbed text at
generation time and stored in `B3Perturbation.expected_answer`. The scorer
re-runs the pipeline and verifies the stored value is reproduced exactly.

---

## Acceptance Evidence

Accepted when:

- `core capability perturbation` exits 0 with:
  - `preserving_rate: 1.0000` (130/130 applicable variants)
  - `breaking_rate: 1.0000` (68/68 applicable variants)
- `tests/test_adr_0114a_5_perturbation.py` 15/15 green
- B3 axis lane report unchanged (`wrong == 0` preserved)
- ADR linked from `docs/decisions/README.md`

### Current run (2026-05-23, main B3 at ADR-0131.3)

```
lane_id:                B3_bounded_grammar
cases_total:            50
cases_expected_correct: 35
preserving:             130/130 = 1.0000
breaking:               68/68  = 1.0000
obligation_5_passed:    True
```

**Skip summary** (documented, not silent):

| Transform | Cases skipped |
|---|---|
| commutative_reorder | 35 (all solved_correct — no single-entity multi-unit init state exists yet) |
| op_verb_flip | 15 (multiply/divide/compare/transfer cases) |
| value_replacement_init | 7 (initial-possession value not parseable in isolation, or cancels) |
| value_replacement_op | 15 (no distinct operation numeric operand) |
| unit_synonym | 7 (rate-declaration `$` syntax cases) |
| entity_rename_v1/v2/v3 | 1 each (b3-013 "Birds" collective noun maps to itself — no substitution) |

---

## ADR-0114a Obligation Discharge Summary

| Obligation #5 requirement | Status under ADR-0114a.5 |
|---|---|
| Invariance-preserving rate == 1.0 | Discharged — 130/130 = 1.0000 |
| Invariance-breaking rate == 1.0 | Discharged — 68/68 = 1.0000 |
| Closed perturbation taxonomy | Discharged — all transforms documented above |
| Deterministic / byte-equal report | Discharged — SHA-256 digest pinned in CI |
| Refusal for missing lane file | Discharged — typed `PerturbationReport` with `obligation_5_passed=False` |

---

## Consequences

- ADR-0114a Obligation #5 is now discharged for B3. B1 and B2 equivalents
  are deferred to separate sub-ADRs (mirroring PR #189's "B3 only, others
  deferred" structure).
- The perturbation suite operates exclusively on input-text strings; it
  never touches the parser, solver, graph schema, or authored cases.
  The `wrong == 0` firewall on the B3 axis lane is unaffected.
- `commutative_reorder` generates 0 variants for all current B3 cases
  (no single-entity multi-unit initial state exists yet). The transform
  is fully implemented and covered by a unit test; future cases that meet
  the criterion will be automatically scored.
- Solver bugs revealed by perturbation are out of scope for this ADR.
  If a perturbed problem reveals a solver defect, file a follow-up ADR;
  do NOT patch the solver from inside this PR.

---

## Out of Scope

- B1 (symbolic equivalence) and B2 (teaching corpus) perturbation suites.
- Cross-sentence semantic perturbations (pronoun rewrites, paraphrasing) —
  those are coverage axes, not reasoning isolation.
- ADR-0114a obligations #2 (OOD), #6 (depth curve), #8 (adversarial).
- Composite-gate wiring changes (ADR-0131.4 already shipped).
- LLMs, sampling, stochastic generation, or approximate recall.