# ADR-0119.2 — GSM8K Eval Corpus Dev/Public Splits

**Status:** Accepted
**Date:** 2026-05-22
**Author:** CORE agents + reviewers
**Depends on:** ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0119

---

## Context

ADR-0114 §Phase 5 designates the actual GSM8K test set as a **SEALED HOLDOUT** — to never be read during development to prevent contamination and overfitting. Consequently, we cannot use GSM8K problems directly for our own dev/public splits.

To measure mathematical reasoning capabilities during development, we require original, GSM8K-style math problems that use our own vocabulary and grammar (defined in ADR-0115). This ADR authors 200 CORE-original math problems (50 for dev, 150 for public splits) to serve as local benchmarks.

---

## Decision

We have authored and verified 200 GSM8K-style math word problems using the CORE vocabulary and grammar:
- **`evals/gsm8k_math/dev/cases.jsonl`**: 50 cases (`gma-001` through `gma-050`).
- **`evals/gsm8k_math/public/v1/cases.jsonl`**: 150 cases (`gma-101` through `gma-250`).
- **`evals/gsm8k_math/holdouts/v1/cases.jsonl`** / **`cases_plaintext.jsonl`**: Left empty; these will contain the sealed holdout in a future phase.

### Scope Expansion
Compared to the Phase 1.1 seed cases, the authored set expands scope as follows:
- **Reasoning depth**: 1-8 steps (vs Phase 1.1's 1-3).
- **Multi-entity complexity**: Up to 5 entities with realistic interactions (transfers, divides, multiplies, etc.).
- **Supported operations**: add, subtract, transfer, multiply, divide.

### Constraints Enforced
All cases enforce the rules specified in Phase 1.1:
- No conditional / time-modal phrasing
- No rate/per-unit pricing requiring inference
- No compound questions / multiple unknowns
- No implicit-entity / generic plural
- No metaphor or mixed units within one entity
- No numeric magnitude beyond integer scope
- No non-CORE-current-grammar constructions

### Verification Script
We added `evals/gsm8k_math/verify.py` which loads all 200 cases and verifies that:
1. `parse_problem(case["problem"])` successfully parses to a graph.
2. The parsed graph matches the ground-truth graph byte-for-byte.
3. The solver yields `answer_value == expected_answer`.
4. The solver yields `answer_unit == expected_unit`.

---

## Invariants

### `adr_0119_2_sealed_holdout_untouched`
The actual GSM8K holdout test set remains completely unread and sealed. No GSM8K test cases overlap with this original corpus.

### `adr_0119_2_perfect_parser_solver_resonance`
All 200 cases parse and solve to the expected answer and unit exactly.

### `adr_0119_2_distribution_balanced`
The authored corpus strictly adheres to the target depth and operation distributions:
- Depth 1: 20 cases
- Depth 2-3: 60 cases
- Depth 4-5: 60 cases
- Depth 6-8: 60 cases
- Each operation appears at least 30 times.
- At least 80 cases contain 2+ entities.

---

## Acceptance Evidence

This ADR is accepted when:
- `verify.py` completes successfully and prints `200/200 OK` with the correct distributions:
  ```text
  --- Verification Stats ---
  Cases parsed and solved successfully: 200/200
  Depth distribution: {1: 20, '2-3': 60, '4-5': 60, '6-8': 60}
  Operation distribution: {'transfer': 109, 'divide': 155, 'multiply': 193, 'subtract': 165, 'add': 237}
  Multi-entity cases: 111

  200/200 OK
  ```

---

## Consequences

- The `gsm8k_math` evaluation lane has high-coverage dev and public splits, unblocking solver benchmarking.
- The sealed holdout remains uncontaminated.
- Regression testing of math solving capabilities is now locally automated.