core/docs/decisions/ADR-0114a.2-ood-ratio-auditor.md

ADR-0114a.2 — OOD-Ratio Auditor (Obligation #2 wired for B3)

Status: Accepted Date: 2026-05-23 Author: CORE agents Depends on: ADR-0114a (anti-overfitting obligations), ADR-0118a (OOD methodology), ADR-0131.3 (B3 grammar contract), PR #189 (ADR-0114a.10 pack-provenance auditor pattern)

---

Context

ADR-0114a Obligation #2 requires:

ood_score.py's OOD/public ratio for the domain's lanes ≥ 0.95.

The spirit: a pattern-matcher overfits to surface distributions in the public split and falls off when surface varies; a deterministic reasoner stays approximately flat. The 0.95 ratio means OOD accuracy must be at least 95% of public accuracy.

ADR-0118a established the methodological blueprint for OOD generation: variants hold the underlying mathematical graph constant while rotating surface form across three classes (entity names, unit nouns, sentence ordering). This ADR applies that methodology to the B3 bounded-grammar lane (ADR-0131.3), which is the lane whose pipeline (parser → graph → solver → verifier) exercises the end-to-end math candidate graph contract.

---

Decision

OOD case set

evals/obligation_2_ood_ratio/v1/cases.jsonl — 35 OOD cases, each a surface-varied sibling of one of B3's 35 solved_correct public cases (b3-001 through b3-035). Every case carries a public_sibling_case_id field for audit traceability.

Three surface-variation families:

Family	Description	Coverage
Entity-name	Sam rotated through closed pool; two-entity cases rotate both names	33 of 35 cases (all named-entity shapes)
Unit-noun	Count-noun unit rotated through closed pool when the sibling's unit is in the substitution set	19 of 35 cases
there_are_count	Unit-noun substituted (unit acts as implicit entity; no named entity)	2 of 35 cases

Closed substitution pools (pinned in this ADR):

Entity-name pool (10 names):

Maya, Liam, Noah, Diana, Felix, Priya, Omar, Rosa, Jun, Kai

These names do not appear in B3's public cases (Sam, Tom are the only names in B3). Each OOD case substitutes at least one name from this pool.

Unit-noun substitution pool (7 count nouns, all allowed by the B3 parser):

oranges, marbles, pencils, books, stamps, coins, balls

These are in the parser's allowed_count_nouns set (math_parser.py). Units NOT in the substitution pool (kept unchanged): dollars, feet, hours, cookies — these are currency, physical dimension, or time units where substitution would produce semantically incoherent rate problems.

Grammar contract: every OOD case is parseable by the B3 bounded grammar (ADR-0131.3 grammar.md). Out-of-grammar forms are adversarial territory (obligation #8), not OOD territory. No case in this set is expected to refuse.

No semantic perturbations: values, operations, and mathematical invariants are identical to the public sibling. OOD ≠ perturbation (obligation #5's domain).

OOD runner

evals/obligation_2_ood_ratio/v1/runner.py — runs OOD cases through the B3 parser → solver → verifier pipeline; emits report.json mirroring B3's runner shape. All cases carry expected == "solved_correct" because OOD cases are surface-varied siblings of B3's solved_correct public cases only.

Auditor

core/capability/ood_ratio.py — external auditor independent of the runner. Reads B3's public report.json and the OOD lane report.json, computes:

ood_ratio = ood_accuracy / public_accuracy

Emits OodRatioReport with two separate gates:

1. Ratio gate: ood_ratio >= 0.95 — obligation_2_ratio_satisfied
2. Wrong-zero gate: ood_wrong == 0 — obligation_2_wrong_zero

Both must hold for obligation_2_passed = True.

Gate threshold 0.95 is pinned; changing it requires a new ADR.

Refusal conditions: missing report files, public_accuracy == 0 (no baseline).

Module shape mirrors core/capability/pack_provenance.py (ADR-0114a.10, PR #189) and evals/gsm8k_math/scoring/depth_curve.py (ADR-0114a.6, PR #190).

CLI

core capability ood-ratio — regenerates the OOD runner report, runs the auditor, writes evals/obligation_2_ood_ratio/<lane_id>.json. Exit 0 iff both gates pass.

---

Invariants

obligation_2_ratio_gate_pinned

The gate threshold is 0.95 in core/capability/ood_ratio.OOD_RATIO_GATE. Changing it requires a new ADR.

obligation_2_ood_cases_in_grammar

Every case in evals/obligation_2_ood_ratio/v1/cases.jsonl is parseable by generate.math_parser.parse_problem without ParseError.

obligation_2_sibling_traceability

Every OOD case's public_sibling_case_id resolves to a real case in evals/math_bounded_grammar/v1/cases.jsonl.

obligation_2_entity_pool

Every OOD named-entity case (all shapes except there_are_count) uses at least one entity name from the closed pool above; no public B3 sibling uses pool names.

obligation_2_unit_pool

OOD cases whose public sibling's unit appears in the substitution pool use a different unit than their sibling.

obligation_2_wrong_zero_gate

Separate from the ratio gate: ood_wrong == 0 is independently required.

obligation_2_determinism

Same OOD cases + same report files produce byte-equal auditor output.

---

Acceptance evidence

Accepted when:

• evals/obligation_2_ood_ratio/v1/cases.jsonl contains 35 cases (≥ 30)
• python3 -m evals.obligation_2_ood_ratio.v1.runner exits 0 with wrong == 0
• python3 -m core.cli capability ood-ratio exits 0 with ratio >= 0.95
• tests/test_adr_0114a_2_ood_ratio.py is green (16 tests)
• B3 public lane (50/50) unchanged
• Obligation #10 pack-provenance auditor still passes

Measured results on this PR:

Metric	Value
B3 public accuracy	50/50 = 1.0000
OOD accuracy	35/35 = 1.0000
OOD/public ratio	1.0000
wrong (OOD)	0
obligation_2_passed	True

Shape family breakdown:

Shape	OOD cases	Entity change	Unit change
canonical_has_buys	16	✓	mixed (count nouns only)
transfer	2	✓ (both actors)	✓
multiply	2	✓	✓
divide	1	✓	✓
apply_rate	3	✓	✓ (item unit)
compare_additive	2	✓ (both actors)	✓
compare_multiplicative	7	✓ (both actors)	mixed
there_are_count	2	unit-as-entity	✓
substance_qualifier	1	✓	— (feet, not in pool)

---

Obligation discharge summary (for B3)

Obligation	Status under ADR-0114a.2
#2 OOD surface variation ratio	Discharged for B3
All others	See ADR-0114a, ADR-0118a, ADR-0114a.10

---

Consequences

• ADR-0114a Obligation #2 is now executable evidence for B3: a deterministic local command (core capability ood-ratio) produces the ratio.
• The substitution pools are frozen; extending them requires a new ADR or a PR that updates this ADR's pool tables and re-runs the tests.
• B1 and B2 OOD equivalents are deferred to separate sub-ADRs (mirror PR #189's structure) because those lanes have different pipeline paths.
• The wrong-zero gate catches any case where a surface substitution accidentally produces a valid but incorrectly-solved problem (a dataset authoring error, not an architecture failure).

---

Out of scope

• B1 (symbolic equivalence) + B2 (teaching corpus) OOD equivalents.
• Cross-grammar fuzzing and paraphrases (adversarial, obligation #8).
• Reasoning-isolation perturbations (obligation #5).
• Compositional-depth curve (obligation #6).
• Composite-gate or promotion-gate wiring.