# ADR-0114a.2 — OOD-Ratio Auditor (Obligation #2 wired for B3) **Status:** Accepted **Date:** 2026-05-23 **Author:** CORE agents **Depends on:** ADR-0114a (anti-overfitting obligations), ADR-0118a (OOD methodology), ADR-0131.3 (B3 grammar contract), PR #189 (ADR-0114a.10 pack-provenance auditor pattern) --- ## Context ADR-0114a Obligation #2 requires: > `ood_score.py`'s **OOD/public ratio for the domain's lanes ≥ 0.95**. The spirit: a pattern-matcher overfits to surface distributions in the public split and falls off when surface varies; a deterministic reasoner stays approximately flat. The 0.95 ratio means OOD accuracy must be at least 95% of public accuracy. ADR-0118a established the methodological blueprint for OOD generation: variants hold the underlying mathematical graph constant while rotating surface form across three classes (entity names, unit nouns, sentence ordering). This ADR applies that methodology to the B3 bounded-grammar lane (ADR-0131.3), which is the lane whose pipeline (parser → graph → solver → verifier) exercises the end-to-end math candidate graph contract. --- ## Decision ### OOD case set **`evals/obligation_2_ood_ratio/v1/cases.jsonl`** — 35 OOD cases, each a surface-varied sibling of one of B3's 35 `solved_correct` public cases (b3-001 through b3-035). Every case carries a `public_sibling_case_id` field for audit traceability. **Three surface-variation families:** | Family | Description | Coverage | |---|---|---| | Entity-name | `Sam` rotated through closed pool; two-entity cases rotate both names | 33 of 35 cases (all named-entity shapes) | | Unit-noun | Count-noun unit rotated through closed pool when the sibling's unit is in the substitution set | 19 of 35 cases | | there_are_count | Unit-noun substituted (unit acts as implicit entity; no named entity) | 2 of 35 cases | **Closed substitution pools (pinned in this ADR):** Entity-name pool (10 names): ``` Maya, Liam, Noah, Diana, Felix, Priya, Omar, Rosa, Jun, Kai ``` These names do not appear in B3's public cases (`Sam`, `Tom` are the only names in B3). Each OOD case substitutes at least one name from this pool. Unit-noun substitution pool (7 count nouns, all allowed by the B3 parser): ``` oranges, marbles, pencils, books, stamps, coins, balls ``` These are in the parser's `allowed_count_nouns` set (math_parser.py). Units NOT in the substitution pool (kept unchanged): `dollars`, `feet`, `hours`, `cookies` — these are currency, physical dimension, or time units where substitution would produce semantically incoherent rate problems. **Grammar contract:** every OOD case is parseable by the B3 bounded grammar (ADR-0131.3 `grammar.md`). Out-of-grammar forms are adversarial territory (obligation #8), not OOD territory. No case in this set is expected to refuse. **No semantic perturbations:** values, operations, and mathematical invariants are identical to the public sibling. OOD ≠ perturbation (obligation #5's domain). ### OOD runner **`evals/obligation_2_ood_ratio/v1/runner.py`** — runs OOD cases through the B3 parser → solver → verifier pipeline; emits `report.json` mirroring B3's runner shape. All cases carry `expected == "solved_correct"` because OOD cases are surface-varied siblings of B3's solved_correct public cases only. ### Auditor **`core/capability/ood_ratio.py`** — external auditor independent of the runner. Reads B3's public `report.json` and the OOD lane `report.json`, computes: ``` ood_ratio = ood_accuracy / public_accuracy ``` Emits `OodRatioReport` with two separate gates: 1. **Ratio gate**: `ood_ratio >= 0.95` — `obligation_2_ratio_satisfied` 2. **Wrong-zero gate**: `ood_wrong == 0` — `obligation_2_wrong_zero` Both must hold for `obligation_2_passed = True`. Gate threshold `0.95` is pinned; changing it requires a new ADR. Refusal conditions: missing report files, `public_accuracy == 0` (no baseline). Module shape mirrors `core/capability/pack_provenance.py` (ADR-0114a.10, PR #189) and `evals/gsm8k_math/scoring/depth_curve.py` (ADR-0114a.6, PR #190). ### CLI **`core capability ood-ratio`** — regenerates the OOD runner report, runs the auditor, writes `evals/obligation_2_ood_ratio/.json`. Exit 0 iff both gates pass. --- ## Invariants ### `obligation_2_ratio_gate_pinned` The gate threshold is 0.95 in `core/capability/ood_ratio.OOD_RATIO_GATE`. Changing it requires a new ADR. ### `obligation_2_ood_cases_in_grammar` Every case in `evals/obligation_2_ood_ratio/v1/cases.jsonl` is parseable by `generate.math_parser.parse_problem` without `ParseError`. ### `obligation_2_sibling_traceability` Every OOD case's `public_sibling_case_id` resolves to a real case in `evals/math_bounded_grammar/v1/cases.jsonl`. ### `obligation_2_entity_pool` Every OOD named-entity case (all shapes except `there_are_count`) uses at least one entity name from the closed pool above; no public B3 sibling uses pool names. ### `obligation_2_unit_pool` OOD cases whose public sibling's unit appears in the substitution pool use a different unit than their sibling. ### `obligation_2_wrong_zero_gate` Separate from the ratio gate: `ood_wrong == 0` is independently required. ### `obligation_2_determinism` Same OOD cases + same report files produce byte-equal auditor output. --- ## Acceptance evidence Accepted when: - `evals/obligation_2_ood_ratio/v1/cases.jsonl` contains 35 cases (≥ 30) - `python3 -m evals.obligation_2_ood_ratio.v1.runner` exits 0 with `wrong == 0` - `python3 -m core.cli capability ood-ratio` exits 0 with `ratio >= 0.95` - `tests/test_adr_0114a_2_ood_ratio.py` is green (16 tests) - B3 public lane (50/50) unchanged - Obligation #10 pack-provenance auditor still passes **Measured results on this PR:** | Metric | Value | |---|---| | B3 public accuracy | 50/50 = 1.0000 | | OOD accuracy | 35/35 = 1.0000 | | OOD/public ratio | 1.0000 | | wrong (OOD) | 0 | | `obligation_2_passed` | True | **Shape family breakdown:** | Shape | OOD cases | Entity change | Unit change | |---|---|---|---| | canonical_has_buys | 16 | ✓ | mixed (count nouns only) | | transfer | 2 | ✓ (both actors) | ✓ | | multiply | 2 | ✓ | ✓ | | divide | 1 | ✓ | ✓ | | apply_rate | 3 | ✓ | ✓ (item unit) | | compare_additive | 2 | ✓ (both actors) | ✓ | | compare_multiplicative | 7 | ✓ (both actors) | mixed | | there_are_count | 2 | unit-as-entity | ✓ | | substance_qualifier | 1 | ✓ | — (feet, not in pool) | --- ## Obligation discharge summary (for B3) | Obligation | Status under ADR-0114a.2 | |---|---| | #2 OOD surface variation ratio | **Discharged for B3** | | All others | See ADR-0114a, ADR-0118a, ADR-0114a.10 | --- ## Consequences - ADR-0114a Obligation #2 is now executable evidence for B3: a deterministic local command (`core capability ood-ratio`) produces the ratio. - The substitution pools are frozen; extending them requires a new ADR or a PR that updates this ADR's pool tables and re-runs the tests. - B1 and B2 OOD equivalents are deferred to separate sub-ADRs (mirror PR #189's structure) because those lanes have different pipeline paths. - The wrong-zero gate catches any case where a surface substitution accidentally produces a valid but incorrectly-solved problem (a dataset authoring error, not an architecture failure). --- ## Out of scope - B1 (symbolic equivalence) + B2 (teaching corpus) OOD equivalents. - Cross-grammar fuzzing and paraphrases (adversarial, obligation #8). - Reasoning-isolation perturbations (obligation #5). - Compositional-depth curve (obligation #6). - Composite-gate or promotion-gate wiring.