core/evals/gsm8k_parser_dev/README.md

gsm8k_parser_dev — Curated Dev Set for the ADR-0115 Math Problem Parser

Status: ADR-0115 Phase 1.1 (initial seed). 5 of 50 target cases authored. Schema source of truth: generate/math_problem_graph.py (typed dataclasses). Format: JSONL — one case per line.

Why this dev set is not drawn from GSM8K

The eventual GSM8K eval lane (ADR-0119) treats the actual GSM8K corpus as sealed test material. To preserve that integrity we author this dev set independently in the same style as GSM8K (grade-school word problems with integer answers and 1-8 reasoning steps) but with no overlap.

The dev set measures the parser, not the difficulty of the problem. A correctly-parsed problem is one whose parser(problem.text) == problem.ground_truth_graph byte-equal.

Case schema

Each line is one JSON object:

{
  "id": "gpd-NNN",
  "problem": "<the natural-language word problem>",
  "expected_answer": <integer or float>,
  "expected_unit": "<unit string>",
  "ground_truth_graph": {
    "entities": ["<entity_1>", "<entity_2>", ...],
    "initial_state": [
      {"entity": "<entity>", "quantity": {"unit": "<unit>", "value": <number>}},
      ...
    ],
    "operations": [
      {"actor": "<entity>", "kind": "<add|subtract|transfer|multiply|divide>",
       "operand": {"unit": "<unit>", "value": <number>},
       "target": "<entity>"  /* required when kind=transfer; omitted otherwise */},
      ...
    ],
    "unknown": {"entity": "<entity>" | null, "unit": "<unit>"}
  },
  "patterns": ["<pattern_tag_1>", "<pattern_tag_2>", ...],
  "notes": "<authoring rationale>"
}

Field rules

• id — gpd-NNN zero-padded to 3 digits, sequential across the file.
• problem — one or more complete English sentences ending in a question. Use Title-Cased proper names for entities ("Sam", "Anna's Toy Box"). Be consistent: the same entity always spelled the same way in problem and ground_truth_graph.entities.
• expected_answer — the integer (or float) answer to the question.
• expected_unit — the unit string the answer is in. Must match ground_truth_graph.unknown.unit byte-for-byte.
• ground_truth_graph.entities — tuple in order of first introduction in the problem text. Not alphabetical. No duplicates.
• ground_truth_graph.initial_state — every entity that starts the problem with a known quantity. Empty list is legal if no initial possessions are asserted (rare).
• ground_truth_graph.operations — in source-text order. Empty list is legal (e.g. multi-entity sum questions with no mutations).
• ground_truth_graph.unknown.entity — set to the entity the question asks about, or null if the question asks for a total across all entities ("How many ... in total?"; "How many do they have altogether?").
• patterns — tag list naming the constructions used. See Pattern registry below.
• notes — author-supplied one-sentence rationale. Read by future reviewers when the parser fails this case.

Canonicalization rules

• Units — lowercase, plural form ("apples", "candies", "dollars", "hours"). Use "dollars" for "$" quantities; the parser is expected to rewrite the "$" surface to the canonical unit.
• Entities — preserve capitalization as written. Do not lowercase.
• Numbers — integers when the text shows integers. Use floats only if the problem text mentions fractional units explicitly (rare in grade-school problems).
• Operation kinds — exactly one of add, subtract, transfer, multiply, divide. Choose the one closest to the verb in the text:
  • "buys / gets / receives / earns / finds / adds" → add
  • "eats / loses / sells / spends / drops / uses / removes" → subtract
  • "gives / sends / hands / passes / mails / transfers" → transfer (and set target)
  • "doubles / triples / Nx as many" → multiply
  • "splits evenly into N / N% of / shares equally with N people" → divide

What this dev set does NOT cover (Phase 1.1 scope)

The parser landing under ADR-0115 will handle the following patterns and no others. Cases violating these constraints belong to a later phase and should not appear in this file:

• Time-modal / conditional phrasing ("If Sam had 5 apples, ...") — out of scope for Phase 1.1. Use direct declarative phrasing only.
• Rate/per-unit pricing requiring inference ("Each apple costs $2. Sam buys 4. How much does he spend?") — out of scope. A simpler variant ("Sam spends $8 on apples. How much does he have left?") IS in scope.
• Multi-clause / compound-question problems ("How many does Sam have, and how many does Tom have?") — out of scope. One unknown per case.
• Implicit-entity / generic plural ("There are 5 boys. Each has 2 apples.") — out of scope. Use named entities.
• Comparative phrasing without explicit numbers ("Sam has twice as many as Tom") — out of scope. Use numeric multipliers only ("Sam has 2 times 3 apples").

These exclusions are not permanent — Phase 1.2+ will lift them under their own ADRs.

Pattern registry

When tagging a case under patterns, draw from this list. Add new tags only when authoring a case that uses a construction not yet covered; update the parser's pattern table at the same time.

Pattern tag	Construction	Example
initial_has	" has ."	"Sam has 5 apples."
initial_there_are	"There are ." (no entity; rare)	"There are 12 candies on the table."
operation_buy_more	" buys more."	"He buys 3 more."
operation_get_more	" gets more ."	"She gets 4 more pencils."
operation_find_adds	" finds ."	"Sam finds 2 apples on the path."
operation_eat_loses	" eats ."	"Tom eats 4 candies."
operation_lose_loses	" loses ."	"Anna loses 3 marbles."
operation_sell_loses	" sells ."	"Lisa sells 2 books."
operation_donate_loses	" donates ."	"Lisa donates 3 books."
operation_use_loses	" uses ."	"He uses 2 sheets of paper."
operation_give_transfer	" gives to ."	"Anna gives 3 marbles to Ben."
operation_send_transfer	" sends to ."	"Tom sends 4 letters to Sara."
operation_double	" doubles ..."	"Sam doubles his savings."
operation_triple	" triples ..."	"Sam triples his stickers."
operation_split_divide	"splits/shares evenly"	"They split 12 candies evenly."
question_how_many_entity	"How many does have?"	"How many apples does Sam have?"
question_how_many_left	"How many ... left?"	"How many candies does Tom have left?"
question_how_many_total	"How many ... in total?" / "altogether"	"How many stickers do they have in total?"
question_how_many_now	"How many ... now?"	"How many marbles does Anna have now?"

How to author a new case (Codex contract)

For each case:

1. Draft the natural-language problem in the style of the seed cases. Use the patterns listed above. Stay within Phase 1.1 scope.
2. Solve it by hand to determine expected_answer and expected_unit.
3. Walk the problem sentence by sentence, emitting:
   • First introduction of an entity → add to entities.
   • "X has N " → initial_state entry.
   • Any state-mutating verb → operations entry. Choose the right kind from the registry. For transfer, set target.
   • The question sentence → unknown field.
4. Set patterns to the tags used.
5. Set notes to one sentence explaining the construction or any gotcha (anaphora resolution, sequence marker, etc.).
6. Verify: load the case via graph_from_dict. The constructor will raise MathGraphError on schema violations. Use:

import json
from generate.math_problem_graph import graph_from_dict
case = json.loads(line)
graph = graph_from_dict(case["ground_truth_graph"])
# canonicalize: parser output is compared against graph.canonical_bytes()

7. Re-solve the graph by hand using the operation semantics:

   • add/subtract on the actor's quantity of that unit
   • transfer = subtract from actor + add to target (same unit)
   • multiply/divide on the actor's quantity (scalar operand)
   • For Unknown.entity=null: sum across every entity holding unit
   • For Unknown.entity="X": look up X's final quantity of unit

   The result must equal expected_answer. If it doesn't, the graph is wrong.

Determinism check

python3 -c "
import json
from generate.math_problem_graph import graph_from_dict
with open('evals/gsm8k_parser_dev/cases.jsonl') as f:
    for line in f:
        c = json.loads(line)
        g = graph_from_dict(c['ground_truth_graph'])
        print(c['id'], 'OK', g.canonical_bytes().hex()[:16])
"

Every case should print OK plus a deterministic 16-hex-char prefix.

Authoring target

50 cases by case-id gpd-050. Distribution target:

• 30 single-entity cases (gpd-001 … gpd-030)
• 12 two-entity transfer cases (gpd-031 … gpd-042)
• 8 multi-entity sum/no-op cases (gpd-043 … gpd-050)

Within each tranche, vary which operation_* pattern is used so the parser is exercised across the registry.

The parser landing under ADR-0115 will be measured against this file. Exit criterion: parse correctness ≥ 0.90 (45 of 50 cases' ground-truth graphs reproduce byte-equal from the parser's output).