core/docs/decisions/ADR-0125-reasoning-isolation-perturbation-suite.md

ADR-0125 — Reasoning-Isolation Perturbation Suite

Status: Accepted Date: 2026-05-22 Author: CORE agents + reviewers Depends on: ADR-0114a, ADR-0115, ADR-0116, ADR-0118a

---

Context

ADR-0114a Obligation #5 requires a programmatic perturbation suite that separates concept-stable reasoning from surface pattern matching. Perturbations are either invariance-preserving, where the answer must not change, or invariance-breaking, where CORE must produce the predicted changed graph/trace result.

ADR-0118a already covered OOD surface variation for entity renaming, unit renaming, and linear number scaling. This ADR adds the semantic suite over the same GSM8K-style parser development lane without changing the parser, solver, graph schema, or authored dev cases.

---

Decision

generate/perturbation_suite.py exports:

generate_perturbations(problem, ground_truth_graph, *, seed)

and the frozen, slotted Perturbation record. The generator is pure and deterministic: same problem, graph, and seed produce byte-equal perturbation records.

The suite reuses ADR-0118a registry and rendering helpers for entity and unit relabeling. It adds semantic transforms that stay inside the ADR-0115 Phase 1.1 pattern registry:

Transform	Kind	Behavior
rename_entities	invariance-preserving	Relabel every entity through the ADR-0118a OOD registry.
rename_units	invariance-preserving	Relabel every unit through the ADR-0118a OOD registry.
reorder_independent_initial_possessions	invariance-preserving	Reverse two or more independent initial possession sentences.
reorder_independent_operations	invariance-preserving	Reverse operations only when their affected (entity, unit) state sets are pairwise disjoint.
replace_verb_with_synonym	invariance-preserving	Replace the first add/subtract/transfer verb with a different parser-registry synonym of the same kind.
add_zero_quantity_entity	invariance-preserving	Add an unused registry entity with zero of the queried unit.
swap_non_commuting_operations	invariance-breaking	Swap two same-state operations when the replay trace changes; expected answer and trace hash are computed from the swapped graph.

scale_numbers_by_k is not duplicated here. ADR-0118a owns that Obligation #2 transform and already pins the linear scaling ratio.

evals/gsm8k_parser_dev/perturbation_score.py scores the live parser+solver against generated perturbations, reports explicit skip counts, prints per-transform ratios, and exits 0 iff both aggregate invariance classes score 100%.

---

Invariants

adr_0125_generator_determinism

Two calls with the same problem, graph, and seed produce byte-equal serialized perturbation records.

adr_0125_preserving_answer_stability

Every invariance-preserving perturbation solves to the original answer value.

adr_0125_breaking_predictable_result

Every invariance-breaking perturbation solves to the expected answer and the predicted trace hash computed from the perturbed graph.

adr_0125_parser_registry_boundary

All rendered perturbations stay inside the documented parser pattern registry: direct initial possessions, supported operation verbs, supported transfer syntax, supported multiply/divide syntax, and one supported question.

adr_0125_skips_are_explicit

Inapplicable transforms are skipped with reported reasons. They are not counted as silent failures or fabricated successes.

---

ADR-0114a Obligation Discharge Summary

This ADR closes ADR-0114a Obligation #5 for the GSM8K-style parser dev lane by making reasoning-isolation perturbations executable and scored through the same parser+solver contract used for public cases.

Obligation #5 transform	Status under ADR-0125
Rename all entities	Discharged here by reuse of ADR-0118a helpers
Rename all units	Discharged here by reuse of ADR-0118a helpers
Multiply all numbers by k	Discharged by ADR-0118a, not duplicated
Reorder independent sentences	Discharged for independent initial possessions; independent operations implemented with 0 applicable current dev cases
Swap order of non-commuting operations	Discharged with predicted answer + trace-hash check
Replace verb with synonym in registry	Discharged

The current 50-case dev split has no pairwise-disjoint operation cases, so reorder_independent_operations reports 0/0 applicable and 50/50 skipped. The transform is implemented and covered by a synthetic unit test; future dev/holdout cases that contain independent operations will be scored by the same gate.

---

Acceptance Evidence

Accepted when:

• generate/perturbation_suite.py exports Perturbation and generate_perturbations
• evals/gsm8k_parser_dev/perturbation_score.py runs as python3 -m evals.gsm8k_parser_dev.perturbation_score
• tests/test_perturbation_suite.py is green
• Smoke suite is green
• The perturbation scorer reports:
  • add_zero_quantity_entity: 50/50 = 1.0000
  • rename_entities: 50/50 = 1.0000
  • rename_units: 50/50 = 1.0000
  • reorder_independent_initial_possessions: 21/21 = 1.0000
  • reorder_independent_operations: 0/0 = n/a, 50 skipped
  • replace_verb_with_synonym: 36/36 = 1.0000
  • swap_non_commuting_operations: 17/17 = 1.0000
  • invariance-preserving: 207/207 = 1.0000
  • invariance-breaking: 17/17 = 1.0000
• ADR linked from docs/decisions/README.md index and frontier

---

Consequences

• ADR-0114a Obligation #5 now has a deterministic local score lane for applicable GSM8K-style dev perturbations.
• The scorer distinguishes semantic invariance from source-order graph identity: reordering may change tuple order in MathProblemGraph, but the answer invariant is still checked through the solver.
• Trace-changing swaps are first-class evidence even when the final numeric answer remains equal.
• Independent-operation coverage is explicit rather than implied; the current public dev set has no applicable pairwise-disjoint operations.

---

Out of Scope

• Number scaling, which remains owned by ADR-0118a.
• Parser, solver, graph-schema, or dev-case expansion.
• New constructions outside ADR-0115 Phase 1.1.
• Holdout scoring. The generator is holdout-ready, but holdout access remains governed by ADR-0114a Obligation #1.
• LLMs, sampling, stochastic generation, approximate recall, or unreviewed mutation.