# ADR-0114a.2 — OOD-Ratio Auditor (Obligation #2 wired for B3)

**Status:** Accepted
**Date:** 2026-05-23
**Author:** CORE agents
**Depends on:** ADR-0114a (anti-overfitting obligations), ADR-0118a (OOD methodology), ADR-0131.3 (B3 grammar contract), PR #189 (ADR-0114a.10 pack-provenance auditor pattern)

---

## Context

ADR-0114a Obligation #2 requires:

> `ood_score.py`'s **OOD/public ratio for the domain's lanes ≥ 0.95**.

The spirit: a pattern-matcher overfits to surface distributions in the public split
and falls off when surface varies; a deterministic reasoner stays approximately flat.
The 0.95 ratio means OOD accuracy must be at least 95% of public accuracy.

ADR-0118a established the methodological blueprint for OOD generation: variants hold
the underlying mathematical graph constant while rotating surface form across three
classes (entity names, unit nouns, sentence ordering). This ADR applies that
methodology to the B3 bounded-grammar lane (ADR-0131.3), which is the lane whose
pipeline (parser → graph → solver → verifier) exercises the end-to-end math
candidate graph contract.

---

## Decision

### OOD case set

**`evals/obligation_2_ood_ratio/v1/cases.jsonl`** — 35 OOD cases, each a
surface-varied sibling of one of B3's 35 `solved_correct` public cases (b3-001
through b3-035).  Every case carries a `public_sibling_case_id` field for audit
traceability.

**Three surface-variation families:**

| Family | Description | Coverage |
|---|---|---|
| Entity-name | `Sam` rotated through closed pool; two-entity cases rotate both names | 33 of 35 cases (all named-entity shapes) |
| Unit-noun | Count-noun unit rotated through closed pool when the sibling's unit is in the substitution set | 19 of 35 cases |
| there_are_count | Unit-noun substituted (unit acts as implicit entity; no named entity) | 2 of 35 cases |

**Closed substitution pools (pinned in this ADR):**

Entity-name pool (10 names):
```
Maya, Liam, Noah, Diana, Felix, Priya, Omar, Rosa, Jun, Kai
```
These names do not appear in B3's public cases (`Sam`, `Tom` are the only names
in B3). Each OOD case substitutes at least one name from this pool.

Unit-noun substitution pool (7 count nouns, all allowed by the B3 parser):
```
oranges, marbles, pencils, books, stamps, coins, balls
```
These are in the parser's `allowed_count_nouns` set (math_parser.py).
Units NOT in the substitution pool (kept unchanged): `dollars`, `feet`, `hours`,
`cookies` — these are currency, physical dimension, or time units where
substitution would produce semantically incoherent rate problems.

**Grammar contract:** every OOD case is parseable by the B3 bounded grammar
(ADR-0131.3 `grammar.md`). Out-of-grammar forms are adversarial territory
(obligation #8), not OOD territory. No case in this set is expected to refuse.

**No semantic perturbations:** values, operations, and mathematical invariants
are identical to the public sibling. OOD ≠ perturbation (obligation #5's domain).

### OOD runner

**`evals/obligation_2_ood_ratio/v1/runner.py`** — runs OOD cases through the
B3 parser → solver → verifier pipeline; emits `report.json` mirroring B3's
runner shape. All cases carry `expected == "solved_correct"` because OOD cases
are surface-varied siblings of B3's solved_correct public cases only.

### Auditor

**`core/capability/ood_ratio.py`** — external auditor independent of the runner.
Reads B3's public `report.json` and the OOD lane `report.json`, computes:

```
ood_ratio = ood_accuracy / public_accuracy
```

Emits `OodRatioReport` with two separate gates:

1. **Ratio gate**: `ood_ratio >= 0.95` — `obligation_2_ratio_satisfied`
2. **Wrong-zero gate**: `ood_wrong == 0` — `obligation_2_wrong_zero`

Both must hold for `obligation_2_passed = True`.

Gate threshold `0.95` is pinned; changing it requires a new ADR.

Refusal conditions: missing report files, `public_accuracy == 0` (no baseline).

Module shape mirrors `core/capability/pack_provenance.py` (ADR-0114a.10, PR #189)
and `evals/gsm8k_math/scoring/depth_curve.py` (ADR-0114a.6, PR #190).

### CLI

**`core capability ood-ratio`** — regenerates the OOD runner report, runs the
auditor, writes `evals/obligation_2_ood_ratio/<lane_id>.json`. Exit 0 iff both
gates pass.

---

## Invariants

### `obligation_2_ratio_gate_pinned`

The gate threshold is 0.95 in `core/capability/ood_ratio.OOD_RATIO_GATE`.
Changing it requires a new ADR.

### `obligation_2_ood_cases_in_grammar`

Every case in `evals/obligation_2_ood_ratio/v1/cases.jsonl` is parseable by
`generate.math_parser.parse_problem` without `ParseError`.

### `obligation_2_sibling_traceability`

Every OOD case's `public_sibling_case_id` resolves to a real case in
`evals/math_bounded_grammar/v1/cases.jsonl`.

### `obligation_2_entity_pool`

Every OOD named-entity case (all shapes except `there_are_count`) uses at least
one entity name from the closed pool above; no public B3 sibling uses pool names.

### `obligation_2_unit_pool`

OOD cases whose public sibling's unit appears in the substitution pool use a
different unit than their sibling.

### `obligation_2_wrong_zero_gate`

Separate from the ratio gate: `ood_wrong == 0` is independently required.

### `obligation_2_determinism`

Same OOD cases + same report files produce byte-equal auditor output.

---

## Acceptance evidence

Accepted when:

- `evals/obligation_2_ood_ratio/v1/cases.jsonl` contains 35 cases (≥ 30)
- `python3 -m evals.obligation_2_ood_ratio.v1.runner` exits 0 with `wrong == 0`
- `python3 -m core.cli capability ood-ratio` exits 0 with `ratio >= 0.95`
- `tests/test_adr_0114a_2_ood_ratio.py` is green (16 tests)
- B3 public lane (50/50) unchanged
- Obligation #10 pack-provenance auditor still passes

**Measured results on this PR:**

| Metric | Value |
|---|---|
| B3 public accuracy | 50/50 = 1.0000 |
| OOD accuracy | 35/35 = 1.0000 |
| OOD/public ratio | 1.0000 |
| wrong (OOD) | 0 |
| `obligation_2_passed` | True |

**Shape family breakdown:**

| Shape | OOD cases | Entity change | Unit change |
|---|---|---|---|
| canonical_has_buys | 16 | ✓ | mixed (count nouns only) |
| transfer | 2 | ✓ (both actors) | ✓ |
| multiply | 2 | ✓ | ✓ |
| divide | 1 | ✓ | ✓ |
| apply_rate | 3 | ✓ | ✓ (item unit) |
| compare_additive | 2 | ✓ (both actors) | ✓ |
| compare_multiplicative | 7 | ✓ (both actors) | mixed |
| there_are_count | 2 | unit-as-entity | ✓ |
| substance_qualifier | 1 | ✓ | — (feet, not in pool) |

---

## Obligation discharge summary (for B3)

| Obligation | Status under ADR-0114a.2 |
|---|---|
| #2 OOD surface variation ratio | **Discharged for B3** |
| All others | See ADR-0114a, ADR-0118a, ADR-0114a.10 |

---

## Consequences

- ADR-0114a Obligation #2 is now executable evidence for B3: a deterministic
  local command (`core capability ood-ratio`) produces the ratio.
- The substitution pools are frozen; extending them requires a new ADR or a PR
  that updates this ADR's pool tables and re-runs the tests.
- B1 and B2 OOD equivalents are deferred to separate sub-ADRs (mirror PR #189's
  structure) because those lanes have different pipeline paths.
- The wrong-zero gate catches any case where a surface substitution accidentally
  produces a valid but incorrectly-solved problem (a dataset authoring error, not
  an architecture failure).

---

## Out of scope

- B1 (symbolic equivalence) + B2 (teaching corpus) OOD equivalents.
- Cross-grammar fuzzing and paraphrases (adversarial, obligation #8).
- Reasoning-isolation perturbations (obligation #5).
- Compositional-depth curve (obligation #6).
- Composite-gate or promotion-gate wiring.
