`gsm8k_math` — Curated Eval Split for the GSM8K Evaluation Lane

Status: ADR-0119.2. 200 cases authored. Schema source of truth: generate/math_problem_graph.py (typed dataclasses). Format: JSONL — one case per line.

Why this set is not drawn from GSM8K

The GSM8K eval lane (ADR-0119) treats the actual GSM8K corpus as a sealed holdout test set. To preserve that integrity, we author this dataset independently in the same style as GSM8K (grade-school word problems with integer answers and 1-8 reasoning steps) but using our own vocabulary and grammar, ensuring zero overlap with the sealed holdout.

The dataset measures the solver pipeline (parser → solver → verifier → realizer). A correctly-parsed and solved problem is one whose parser output matches the ground-truth graph byte-for-byte and solves to the expected answer and unit.

Case schema

Each line is one JSON object:

{
  "id": "gma-NNN",
  "problem": "<the natural-language word problem>",
  "expected_answer": <integer>,
  "expected_unit": "<unit string>",
  "ground_truth_graph": {
    "entities": ["<entity_1>", "<entity_2>", ...],
    "initial_state": [
      {"entity": "<entity>", "quantity": {"unit": "<unit>", "value": <number>}},
      ...
    ],
    "operations": [
      {"actor": "<entity>", "kind": "<add|subtract|transfer|multiply|divide>",
       "operand": {"unit": "<unit>", "value": <number>},
       "target": "<entity>"  /* required when kind=transfer; omitted otherwise */},
      ...
    ],
    "unknown": {"entity": "<entity>" | null, "unit": "<unit>"}
  },
  "patterns": ["<pattern_tag_1>", "<pattern_tag_2>", ...],
  "notes": "<authoring rationale>"
}

Field rules

id — gma-NNN where:
- gma-001 ... gma-050 are for the dev split.
- gma-101 ... gma-250 are for the public split.
problem — one or more complete English sentences ending in a question. Use Title-Cased proper names for entities ("Sam", "Anna's Toy Box"). Be consistent: the same entity always spelled the same way in problem and ground_truth_graph.entities.
expected_answer — the integer answer to the question.
expected_unit — the unit string the answer is in. Must match ground_truth_graph.unknown.unit byte-for-byte.
ground_truth_graph.entities — tuple in order of first introduction in the problem text. Not alphabetical. No duplicates.
ground_truth_graph.initial_state — every entity that starts the problem with a known quantity. Empty list is legal if no initial possessions are asserted (rare).
ground_truth_graph.operations — in source-text order. Empty list is legal (e.g. multi-entity sum questions with no mutations).
ground_truth_graph.unknown.entity — set to the entity the question asks about, or null if the question asks for a total across all entities ("How many ... in total?"; "How many do they have altogether?").
patterns — tag list naming the constructions used. See Pattern registry below.
notes — author-supplied one-sentence rationale. Read by future reviewers when the parser fails this case.

Canonicalization rules

Units — lowercase, plural form ("apples", "candies", "dollars", "hours"). Use "dollars" for "$" quantities; the parser is expected to rewrite the "$" surface to the canonical unit.
Entities — preserve capitalization as written. Do not lowercase.
Numbers — integers when the text shows integers.
Operation kinds — exactly one of add, subtract, transfer, multiply, divide. Choose the one closest to the verb in the text:
- "buys / gets / receives / earns / finds / adds" → add
- "eats / loses / sells / spends / drops / uses / removes" → subtract
- "gives / sends / hands / passes / mails / transfers" → transfer (and set target)
- "doubles / triples / Nx as many" → multiply
- "splits evenly into N / N% of / shares equally with N people" → divide

Scope limits (ADR-0119.2)

The parser and solver handle the following patterns and no others. Cases violating these constraints are out of scope:

NO Time-modal / conditional phrasing ("If Sam had 5 apples, ...") — out of scope. Use direct declarative phrasing only.
NO Rate/per-unit pricing requiring inference ("Each apple costs $2. Sam buys 4. How much does he spend?") — out of scope. A simpler variant ("Sam spends $8 on apples. How much does he have left?") IS in scope.
NO Multi-clause / compound-question problems ("How many does Sam have, and how many does Tom have?") — out of scope. One unknown per case.
NO Implicit-entity / generic plural ("There are 5 boys. Each has 2 apples.") — out of scope. Use named entities.
NO Comparative phrasing without explicit numbers ("Sam has twice as many as Tom") — out of scope. Use numeric multipliers only ("Sam has 2 times 3 apples").
NO metaphor or mixed units within one entity — out of scope. Keep units consistent.
NO numeric magnitude beyond integer scope — out of scope. Only use integers.

Pattern registry

When tagging a case under patterns, draw from this list.

Pattern tag	Construction	Example
`initial_has`	" has ."	"Sam has 5 apples."
`initial_there_are`	"There are ." (no entity; rare)	"There are 12 candies on the table."
`operation_buy_more`	" buys more."	"He buys 3 more."
`operation_get_more`	" gets more ."	"She gets 4 more pencils."
`operation_find_adds`	" finds ."	"Sam finds 2 apples on the path."
`operation_eat_loses`	" eats ."	"Tom eats 4 candies."
`operation_lose_loses`	" loses ."	"Anna loses 3 marbles."
`operation_sell_loses`	" sells ."	"Lisa sells 2 books."
`operation_donate_loses`	" donates ."	"Lisa donates 3 books."
`operation_use_loses`	" uses ."	"He uses 2 sheets of paper."
`operation_give_transfer`	" gives to ."	"Anna gives 3 marbles to Ben."
`operation_send_transfer`	" sends to ."	"Tom sends 4 letters to Sara."
`operation_double`	" doubles ..."	"Sam doubles his savings."
`operation_triple`	" triples ..."	"Sam triples his stickers."
`operation_split_divide`	"splits/shares evenly"	"They split 12 candies evenly."
`question_how_many_entity`	"How many does have?"	"How many apples does Sam have?"
`question_how_many_left`	"How many ... left?"	"How many candies does Tom have left?"
`question_how_many_total`	"How many ... in total?" / "altogether"	"How many stickers do they have in total?"
`question_how_many_now`	"How many ... now?"	"How many marbles does Anna have now?"

6.8 KiB Raw Blame History

gsm8k_math — Curated Eval Split for the GSM8K Evaluation Lane