# ADR-0125 — Reasoning-Isolation Perturbation Suite **Status:** Accepted **Date:** 2026-05-22 **Author:** CORE agents + reviewers **Depends on:** ADR-0114a, ADR-0115, ADR-0116, ADR-0118a --- ## Context ADR-0114a Obligation #5 requires a programmatic perturbation suite that separates concept-stable reasoning from surface pattern matching. Perturbations are either invariance-preserving, where the answer must not change, or invariance-breaking, where CORE must produce the predicted changed graph/trace result. ADR-0118a already covered OOD surface variation for entity renaming, unit renaming, and linear number scaling. This ADR adds the semantic suite over the same GSM8K-style parser development lane without changing the parser, solver, graph schema, or authored dev cases. --- ## Decision `generate/perturbation_suite.py` exports: ```text generate_perturbations(problem, ground_truth_graph, *, seed) ``` and the frozen, slotted `Perturbation` record. The generator is pure and deterministic: same problem, graph, and seed produce byte-equal perturbation records. The suite reuses ADR-0118a registry and rendering helpers for entity and unit relabeling. It adds semantic transforms that stay inside the ADR-0115 Phase 1.1 pattern registry: | Transform | Kind | Behavior | |---|---|---| | `rename_entities` | invariance-preserving | Relabel every entity through the ADR-0118a OOD registry. | | `rename_units` | invariance-preserving | Relabel every unit through the ADR-0118a OOD registry. | | `reorder_independent_initial_possessions` | invariance-preserving | Reverse two or more independent initial possession sentences. | | `reorder_independent_operations` | invariance-preserving | Reverse operations only when their affected `(entity, unit)` state sets are pairwise disjoint. | | `replace_verb_with_synonym` | invariance-preserving | Replace the first add/subtract/transfer verb with a different parser-registry synonym of the same kind. | | `add_zero_quantity_entity` | invariance-preserving | Add an unused registry entity with zero of the queried unit. | | `swap_non_commuting_operations` | invariance-breaking | Swap two same-state operations when the replay trace changes; expected answer and trace hash are computed from the swapped graph. | `scale_numbers_by_k` is not duplicated here. ADR-0118a owns that Obligation #2 transform and already pins the linear scaling ratio. `evals/gsm8k_parser_dev/perturbation_score.py` scores the live parser+solver against generated perturbations, reports explicit skip counts, prints per-transform ratios, and exits `0` iff both aggregate invariance classes score 100%. --- ## Invariants ### `adr_0125_generator_determinism` Two calls with the same problem, graph, and seed produce byte-equal serialized perturbation records. ### `adr_0125_preserving_answer_stability` Every invariance-preserving perturbation solves to the original answer value. ### `adr_0125_breaking_predictable_result` Every invariance-breaking perturbation solves to the expected answer and the predicted trace hash computed from the perturbed graph. ### `adr_0125_parser_registry_boundary` All rendered perturbations stay inside the documented parser pattern registry: direct initial possessions, supported operation verbs, supported transfer syntax, supported multiply/divide syntax, and one supported question. ### `adr_0125_skips_are_explicit` Inapplicable transforms are skipped with reported reasons. They are not counted as silent failures or fabricated successes. --- ## ADR-0114a Obligation Discharge Summary This ADR closes ADR-0114a Obligation #5 for the GSM8K-style parser dev lane by making reasoning-isolation perturbations executable and scored through the same parser+solver contract used for public cases. | Obligation #5 transform | Status under ADR-0125 | |---|---| | Rename all entities | Discharged here by reuse of ADR-0118a helpers | | Rename all units | Discharged here by reuse of ADR-0118a helpers | | Multiply all numbers by `k` | Discharged by ADR-0118a, not duplicated | | Reorder independent sentences | Discharged for independent initial possessions; independent operations implemented with 0 applicable current dev cases | | Swap order of non-commuting operations | Discharged with predicted answer + trace-hash check | | Replace verb with synonym in registry | Discharged | The current 50-case dev split has no pairwise-disjoint operation cases, so `reorder_independent_operations` reports `0/0` applicable and `50/50` skipped. The transform is implemented and covered by a synthetic unit test; future dev/holdout cases that contain independent operations will be scored by the same gate. --- ## Acceptance Evidence Accepted when: - `generate/perturbation_suite.py` exports `Perturbation` and `generate_perturbations` - `evals/gsm8k_parser_dev/perturbation_score.py` runs as `python3 -m evals.gsm8k_parser_dev.perturbation_score` - `tests/test_perturbation_suite.py` is green - Smoke suite is green - The perturbation scorer reports: - `add_zero_quantity_entity`: 50/50 = 1.0000 - `rename_entities`: 50/50 = 1.0000 - `rename_units`: 50/50 = 1.0000 - `reorder_independent_initial_possessions`: 21/21 = 1.0000 - `reorder_independent_operations`: 0/0 = n/a, 50 skipped - `replace_verb_with_synonym`: 36/36 = 1.0000 - `swap_non_commuting_operations`: 17/17 = 1.0000 - invariance-preserving: 207/207 = 1.0000 - invariance-breaking: 17/17 = 1.0000 - ADR linked from `docs/decisions/README.md` index and frontier --- ## Consequences - ADR-0114a Obligation #5 now has a deterministic local score lane for applicable GSM8K-style dev perturbations. - The scorer distinguishes semantic invariance from source-order graph identity: reordering may change tuple order in `MathProblemGraph`, but the answer invariant is still checked through the solver. - Trace-changing swaps are first-class evidence even when the final numeric answer remains equal. - Independent-operation coverage is explicit rather than implied; the current public dev set has no applicable pairwise-disjoint operations. --- ## Out of Scope - Number scaling, which remains owned by ADR-0118a. - Parser, solver, graph-schema, or dev-case expansion. - New constructions outside ADR-0115 Phase 1.1. - Holdout scoring. The generator is holdout-ready, but holdout access remains governed by ADR-0114a Obligation #1. - LLMs, sampling, stochastic generation, approximate recall, or unreviewed mutation.