core/evals/gsm8k_math

gsm8k_math — Curated Eval Split for the GSM8K Evaluation Lane

Status: ADR-0119.2. 200 cases authored. Schema source of truth: generate/math_problem_graph.py (typed dataclasses). Format: JSONL — one case per line.

Why this set is not drawn from GSM8K

The GSM8K eval lane (ADR-0119) treats the actual GSM8K corpus as a sealed holdout test set. To preserve that integrity, we author this dataset independently in the same style as GSM8K (grade-school word problems with integer answers and 1-8 reasoning steps) but using our own vocabulary and grammar, ensuring zero overlap with the sealed holdout.

The dataset measures the solver pipeline (parser → solver → verifier → realizer). A correctly-parsed and solved problem is one whose parser output matches the ground-truth graph byte-for-byte and solves to the expected answer and unit.

Case schema

Each line is one JSON object:

{
  "id": "gma-NNN",
  "problem": "<the natural-language word problem>",
  "expected_answer": <integer>,
  "expected_unit": "<unit string>",
  "ground_truth_graph": {
    "entities": ["<entity_1>", "<entity_2>", ...],
    "initial_state": [
      {"entity": "<entity>", "quantity": {"unit": "<unit>", "value": <number>}},
      ...
    ],
    "operations": [
      {"actor": "<entity>", "kind": "<add|subtract|transfer|multiply|divide>",
       "operand": {"unit": "<unit>", "value": <number>},
       "target": "<entity>"  /* required when kind=transfer; omitted otherwise */},
      ...
    ],
    "unknown": {"entity": "<entity>" | null, "unit": "<unit>"}
  },
  "patterns": ["<pattern_tag_1>", "<pattern_tag_2>", ...],
  "notes": "<authoring rationale>"
}

Field rules

• id — gma-NNN where:
  • gma-001 ... gma-050 are for the dev split.
  • gma-101 ... gma-250 are for the public split.
• problem — one or more complete English sentences ending in a question. Use Title-Cased proper names for entities ("Sam", "Anna's Toy Box"). Be consistent: the same entity always spelled the same way in problem and ground_truth_graph.entities.
• expected_answer — the integer answer to the question.
• expected_unit — the unit string the answer is in. Must match ground_truth_graph.unknown.unit byte-for-byte.
• ground_truth_graph.entities — tuple in order of first introduction in the problem text. Not alphabetical. No duplicates.
• ground_truth_graph.initial_state — every entity that starts the problem with a known quantity. Empty list is legal if no initial possessions are asserted (rare).
• ground_truth_graph.operations — in source-text order. Empty list is legal (e.g. multi-entity sum questions with no mutations).
• ground_truth_graph.unknown.entity — set to the entity the question asks about, or null if the question asks for a total across all entities ("How many ... in total?"; "How many do they have altogether?").
• patterns — tag list naming the constructions used. See Pattern registry below.
• notes — author-supplied one-sentence rationale. Read by future reviewers when the parser fails this case.

Canonicalization rules

• Units — lowercase, plural form ("apples", "candies", "dollars", "hours"). Use "dollars" for "$" quantities; the parser is expected to rewrite the "$" surface to the canonical unit.
• Entities — preserve capitalization as written. Do not lowercase.
• Numbers — integers when the text shows integers.
• Operation kinds — exactly one of add, subtract, transfer, multiply, divide. Choose the one closest to the verb in the text:
  • "buys / gets / receives / earns / finds / adds" → add
  • "eats / loses / sells / spends / drops / uses / removes" → subtract
  • "gives / sends / hands / passes / mails / transfers" → transfer (and set target)
  • "doubles / triples / Nx as many" → multiply
  • "splits evenly into N / N% of / shares equally with N people" → divide

Scope limits (ADR-0119.2)

The parser and solver handle the following patterns and no others. Cases violating these constraints are out of scope:

• NO Time-modal / conditional phrasing ("If Sam had 5 apples, ...") — out of scope. Use direct declarative phrasing only.
• NO Rate/per-unit pricing requiring inference ("Each apple costs $2. Sam buys 4. How much does he spend?") — out of scope. A simpler variant ("Sam spends $8 on apples. How much does he have left?") IS in scope.
• NO Multi-clause / compound-question problems ("How many does Sam have, and how many does Tom have?") — out of scope. One unknown per case.
• NO Implicit-entity / generic plural ("There are 5 boys. Each has 2 apples.") — out of scope. Use named entities.
• NO Comparative phrasing without explicit numbers ("Sam has twice as many as Tom") — out of scope. Use numeric multipliers only ("Sam has 2 times 3 apples").
• NO metaphor or mixed units within one entity — out of scope. Keep units consistent.
• NO numeric magnitude beyond integer scope — out of scope. Only use integers.

Pattern registry

When tagging a case under patterns, draw from this list.

Pattern tag	Construction	Example
initial_has	" has ."	"Sam has 5 apples."
initial_there_are	"There are ." (no entity; rare)	"There are 12 candies on the table."
operation_buy_more	" buys more."	"He buys 3 more."
operation_get_more	" gets more ."	"She gets 4 more pencils."
operation_find_adds	" finds ."	"Sam finds 2 apples on the path."
operation_eat_loses	" eats ."	"Tom eats 4 candies."
operation_lose_loses	" loses ."	"Anna loses 3 marbles."
operation_sell_loses	" sells ."	"Lisa sells 2 books."
operation_donate_loses	" donates ."	"Lisa donates 3 books."
operation_use_loses	" uses ."	"He uses 2 sheets of paper."
operation_give_transfer	" gives to ."	"Anna gives 3 marbles to Ben."
operation_send_transfer	" sends to ."	"Tom sends 4 letters to Sara."
operation_double	" doubles ..."	"Sam doubles his savings."
operation_triple	" triples ..."	"Sam triples his stickers."
operation_split_divide	"splits/shares evenly"	"They split 12 candies evenly."
question_how_many_entity	"How many does have?"	"How many apples does Sam have?"
question_how_many_left	"How many ... left?"	"How many candies does Tom have left?"
question_how_many_total	"How many ... in total?" / "altogether"	"How many stickers do they have in total?"
question_how_many_now	"How many ... now?"	"How many marbles does Anna have now?"