ADR-0119.2 — GSM8K Eval Corpus Dev/Public Splits

Status: Accepted Date: 2026-05-22 Author: CORE agents + reviewers Depends on: ADR-0114, ADR-0114a, ADR-0115, ADR-0116, ADR-0117, ADR-0119

Context

ADR-0114 §Phase 5 designates the actual GSM8K test set as a SEALED HOLDOUT — to never be read during development to prevent contamination and overfitting. Consequently, we cannot use GSM8K problems directly for our own dev/public splits.

To measure mathematical reasoning capabilities during development, we require original, GSM8K-style math problems that use our own vocabulary and grammar (defined in ADR-0115). This ADR authors 200 CORE-original math problems (50 for dev, 150 for public splits) to serve as local benchmarks.

Decision

We have authored and verified 200 GSM8K-style math word problems using the CORE vocabulary and grammar:

evals/gsm8k_math/dev/cases.jsonl: 50 cases (gma-001 through gma-050).
evals/gsm8k_math/public/v1/cases.jsonl: 150 cases (gma-101 through gma-250).
evals/gsm8k_math/holdouts/v1/cases.jsonl / cases_plaintext.jsonl: Left empty; these will contain the sealed holdout in a future phase.

Scope Expansion

Compared to the Phase 1.1 seed cases, the authored set expands scope as follows:

Reasoning depth: 1-8 steps (vs Phase 1.1's 1-3).
Multi-entity complexity: Up to 5 entities with realistic interactions (transfers, divides, multiplies, etc.).
Supported operations: add, subtract, transfer, multiply, divide.

Constraints Enforced

All cases enforce the rules specified in Phase 1.1:

No conditional / time-modal phrasing
No rate/per-unit pricing requiring inference
No compound questions / multiple unknowns
No implicit-entity / generic plural
No metaphor or mixed units within one entity
No numeric magnitude beyond integer scope
No non-CORE-current-grammar constructions

Verification Script

We added evals/gsm8k_math/verify.py which loads all 200 cases and verifies that:

parse_problem(case["problem"]) successfully parses to a graph.
The parsed graph matches the ground-truth graph byte-for-byte.
The solver yields answer_value == expected_answer.
The solver yields answer_unit == expected_unit.

Invariants

`adr_0119_2_sealed_holdout_untouched`

The actual GSM8K holdout test set remains completely unread and sealed. No GSM8K test cases overlap with this original corpus.

`adr_0119_2_perfect_parser_solver_resonance`

All 200 cases parse and solve to the expected answer and unit exactly.

`adr_0119_2_distribution_balanced`

The authored corpus strictly adheres to the target depth and operation distributions:

Depth 1: 20 cases
Depth 2-3: 60 cases
Depth 4-5: 60 cases
Depth 6-8: 60 cases
Each operation appears at least 30 times.
At least 80 cases contain 2+ entities.

Acceptance Evidence

This ADR is accepted when:

verify.py completes successfully and prints 200/200 OK with the correct distributions:

--- Verification Stats ---
Cases parsed and solved successfully: 200/200
Depth distribution: {1: 20, '2-3': 60, '4-5': 60, '6-8': 60}
Operation distribution: {'transfer': 109, 'divide': 155, 'multiply': 193, 'subtract': 165, 'add': 237}
Multi-entity cases: 111

200/200 OK

Consequences

The gsm8k_math evaluation lane has high-coverage dev and public splits, unblocking solver benchmarking.
The sealed holdout remains uncontaminated.
Regression testing of math solving capabilities is now locally automated.

3.5 KiB Raw Blame History