core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat: ADR-0115 Phase 1.2 — rewrite gpd-021 to drop metaphor / mixed units (parser hits 50/50)

Browse source
This commit is contained in:
Shay 2026-05-22 16:11:22 -07:00
parent 61d3cf8095
commit dc0988416e
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
evals/gsm8k_parser_dev/cases.jsonl
View file

@ -1,7 +1,7 @@
{"id":"gpd-001","problem":"Sam has 5 apples. He buys 3 more. How many apples does Sam have?","expected_answer":8,"expected_unit":"apples","ground_truth_graph":{"entities":["Sam"],"initial_state":[{"entity":"Sam","quantity":{"unit":"apples","value":5}}],"operations":[{"actor":"Sam","kind":"add","operand":{"unit":"apples","value":3}}],"unknown":{"entity":"Sam","unit":"apples"}},"patterns":["initial_has","operation_buy_more","question_how_many_entity"],"notes":"Single-entity, single-add. The simplest GSM8K-style pattern. 'He' resolves anaphorically to 'Sam'."}
{"id":"gpd-002","problem":"Tom has 12 candies. He eats 4. How many candies does Tom have left?","expected_answer":8,"expected_unit":"candies","ground_truth_graph":{"entities":["Tom"],"initial_state":[{"entity":"Tom","quantity":{"unit":"candies","value":12}}],"operations":[{"actor":"Tom","kind":"subtract","operand":{"unit":"candies","value":4}}],"unknown":{"entity":"Tom","unit":"candies"}},"patterns":["initial_has","operation_eat_loses","question_how_many_left"],"notes":"Single-entity, single-subtract. 'eats' is a loss verb. Question suffix 'left' is a comprehension cue but the unknown shape is identical to gpd-001."}
{"id":"gpd-003","problem":"Lisa has 10 books. She buys 5 more, then donates 3. How many books does Lisa have?","expected_answer":12,"expected_unit":"books","ground_truth_graph":{"entities":["Lisa"],"initial_state":[{"entity":"Lisa","quantity":{"unit":"books","value":10}}],"operations":[{"actor":"Lisa","kind":"add","operand":{"unit":"books","value":5}},{"actor":"Lisa","kind":"subtract","operand":{"unit":"books","value":3}}],"unknown":{"entity":"Lisa","unit":"books"}},"patterns":["initial_has","operation_buy_more","operation_donate_loses","question_how_many_entity"],"notes":"Single-entity, multi-step. 'then' is a sequence marker; operation order must follow text order."}
{"id":"gpd-004","problem":"Anna has 8 marbles. She gives 3 to Ben. How many marbles does Anna have now?","expected_answer":5,"expected_unit":"marbles","ground_truth_graph":{"entities":["Anna","Ben"],"initial_state":[{"entity":"Anna","quantity":{"unit":"marbles","value":8}}],"operations":[{"actor":"Anna","kind":"transfer","operand":{"unit":"marbles","value":3},"target":"Ben"}],"unknown":{"entity":"Anna","unit":"marbles"}},"patterns":["initial_has","operation_give_transfer","question_how_many_entity"],"notes":"Two-entity transfer. Ben appears only as a transfer target — no initial possession is asserted for Ben. The 'transfer' kind decomposes downstream into subtract(actor) + add(target); the parser emits the closer-to-NL form."}
{"id":"gpd-004","problem":"Anna has 8 marbles. She gives 3 to Ben. How many marbles does Anna have now?","expected_answer":5,"expected_unit":"marbles","ground_truth_graph":{"entities":["Anna","Ben"],"initial_state":[{"entity":"Anna","quantity":{"unit":"marbles","value":8}}],"operations":[{"actor":"Anna","kind":"transfer","operand":{"unit":"marbles","value":3},"target":"Ben"}],"unknown":{"entity":"Anna","unit":"marbles"}},"patterns":["initial_has","operation_give_transfer","question_how_many_entity"],"notes":"Two-entity transfer. Ben appears only as a transfer target \u2014 no initial possession is asserted for Ben. The 'transfer' kind decomposes downstream into subtract(actor) + add(target); the parser emits the closer-to-NL form."}
{"id":"gpd-005","problem":"Tom has 4 stickers. Sara has 7 stickers. How many stickers do they have in total?","expected_answer":11,"expected_unit":"stickers","ground_truth_graph":{"entities":["Tom","Sara"],"initial_state":[{"entity":"Tom","quantity":{"unit":"stickers","value":4}},{"entity":"Sara","quantity":{"unit":"stickers","value":7}}],"operations":[],"unknown":{"entity":null,"unit":"stickers"}},"patterns":["initial_has","initial_has","question_how_many_total"],"notes":"Multi-entity initial possessions, no operations, sum question. 'they' refers to all introduced entities; Unknown.entity=null signals total across entities."}
{"id":"gpd-006","problem":"Nora has 9 pencils. She buys 4 more. How many pencils does Nora have?","expected_answer":13,"expected_unit":"pencils","ground_truth_graph":{"entities":["Nora"],"initial_state":[{"entity":"Nora","quantity":{"unit":"pencils","value":9}}],"operations":[{"actor":"Nora","kind":"add","operand":{"unit":"pencils","value":4}}],"unknown":{"entity":"Nora","unit":"pencils"}},"patterns":["initial_has","operation_buy_more","question_how_many_entity"],"notes":"Single-entity add case using explicit buy-more phrasing with direct anaphora resolution."}
{"id":"gpd-007","problem":"Evan has 14 shells. He gets 5 more shells. How many shells does Evan have now?","expected_answer":19,"expected_unit":"shells","ground_truth_graph":{"entities":["Evan"],"initial_state":[{"entity":"Evan","quantity":{"unit":"shells","value":14}}],"operations":[{"actor":"Evan","kind":"add","operand":{"unit":"shells","value":5}}],"unknown":{"entity":"Evan","unit":"shells"}},"patterns":["initial_has","operation_get_more","question_how_many_now"],"notes":"Single-entity add case that exercises the get-more lexical variant."}
@ -18,7 +18,7 @@
{"id":"gpd-018","problem":"Kai has 13 stones. He gets 4 more stones. Then he loses 5. How many stones does Kai have now?","expected_answer":12,"expected_unit":"stones","ground_truth_graph":{"entities":["Kai"],"initial_state":[{"entity":"Kai","quantity":{"unit":"stones","value":13}}],"operations":[{"actor":"Kai","kind":"add","operand":{"unit":"stones","value":4}},{"actor":"Kai","kind":"subtract","operand":{"unit":"stones","value":5}}],"unknown":{"entity":"Kai","unit":"stones"}},"patterns":["initial_has","operation_get_more","operation_lose_loses","question_how_many_now"],"notes":"Get-then-lose composition tests additive followed by subtractive mutation on one entity."}
{"id":"gpd-019","problem":"Jade has 21 coins. She finds 2 coins. Then she sells 6. How many coins does Jade have left?","expected_answer":17,"expected_unit":"coins","ground_truth_graph":{"entities":["Jade"],"initial_state":[{"entity":"Jade","quantity":{"unit":"coins","value":21}}],"operations":[{"actor":"Jade","kind":"add","operand":{"unit":"coins","value":2}},{"actor":"Jade","kind":"subtract","operand":{"unit":"coins","value":6}}],"unknown":{"entity":"Jade","unit":"coins"}},"patterns":["initial_has","operation_find_adds","operation_sell_loses","question_how_many_left"],"notes":"Find-add followed by sell-loss provides mixed verb mapping while keeping one unit."}
{"id":"gpd-020","problem":"Milo has 30 crayons. He donates 4. Then he uses 5. How many crayons does Milo have now?","expected_answer":21,"expected_unit":"crayons","ground_truth_graph":{"entities":["Milo"],"initial_state":[{"entity":"Milo","quantity":{"unit":"crayons","value":30}}],"operations":[{"actor":"Milo","kind":"subtract","operand":{"unit":"crayons","value":4}},{"actor":"Milo","kind":"subtract","operand":{"unit":"crayons","value":5}}],"unknown":{"entity":"Milo","unit":"crayons"}},"patterns":["initial_has","operation_donate_loses","operation_use_loses","question_how_many_now"],"notes":"Back-to-back subtractive verbs verify ordered operation application without target entity."}
{"id":"gpd-021","problem":"Lena has 8 dolls. She doubles her dolls. Then she eats 3 candies worth from her doll count metaphor game. How many dolls does Lena have now?","expected_answer":13,"expected_unit":"dolls","ground_truth_graph":{"entities":["Lena"],"initial_state":[{"entity":"Lena","quantity":{"unit":"dolls","value":8}}],"operations":[{"actor":"Lena","kind":"multiply","operand":{"unit":"dolls","value":2}},{"actor":"Lena","kind":"subtract","operand":{"unit":"dolls","value":3}}],"unknown":{"entity":"Lena","unit":"dolls"}},"patterns":["initial_has","operation_double","operation_eat_loses","question_how_many_now"],"notes":"Compound operator case validates multiply before subtract under explicit sequencing cues."}
{"id":"gpd-021","problem":"Lena has 8 dolls. She doubles her dolls. Then she donates 3 dolls. How many dolls does Lena have now?","expected_answer":13,"expected_unit":"dolls","ground_truth_graph":{"entities":["Lena"],"initial_state":[{"entity":"Lena","quantity":{"unit":"dolls","value":8}}],"operations":[{"actor":"Lena","kind":"multiply","operand":{"unit":"dolls","value":2}},{"actor":"Lena","kind":"subtract","operand":{"unit":"dolls","value":3}}],"unknown":{"entity":"Lena","unit":"dolls"}},"patterns":["initial_has","operation_double","operation_donate_loses","question_how_many_now"],"notes":"Rewrite of gpd-021 to remove metaphorical mixed-unit phrasing while preserving a deterministic multiply-then-subtract structure."}
{"id":"gpd-022","problem":"Zane has 4 cards. He triples his cards. Then he donates 2. How many cards does Zane have left?","expected_answer":10,"expected_unit":"cards","ground_truth_graph":{"entities":["Zane"],"initial_state":[{"entity":"Zane","quantity":{"unit":"cards","value":4}}],"operations":[{"actor":"Zane","kind":"multiply","operand":{"unit":"cards","value":3}},{"actor":"Zane","kind":"subtract","operand":{"unit":"cards","value":2}}],"unknown":{"entity":"Zane","unit":"cards"}},"patterns":["initial_has","operation_triple","operation_donate_loses","question_how_many_left"],"notes":"Triple then donate verifies scalar multiplication followed by subtraction on same entity."}
{"id":"gpd-023","problem":"Pia has 36 beads. She splits them evenly into 4 groups and keeps one group. Then she buys 3 more beads. How many beads does Pia have now?","expected_answer":12,"expected_unit":"beads","ground_truth_graph":{"entities":["Pia"],"initial_state":[{"entity":"Pia","quantity":{"unit":"beads","value":36}}],"operations":[{"actor":"Pia","kind":"divide","operand":{"unit":"beads","value":4}},{"actor":"Pia","kind":"add","operand":{"unit":"beads","value":3}}],"unknown":{"entity":"Pia","unit":"beads"}},"patterns":["initial_has","operation_split_divide","operation_buy_more","question_how_many_now"],"notes":"Divide-first pipeline confirms split-divide semantics before later additive mutation."}
{"id":"gpd-024","problem":"Tara has 17 flowers. She loses 4. Then she gets 6 more flowers. How many flowers does Tara have now?","expected_answer":19,"expected_unit":"flowers","ground_truth_graph":{"entities":["Tara"],"initial_state":[{"entity":"Tara","quantity":{"unit":"flowers","value":17}}],"operations":[{"actor":"Tara","kind":"subtract","operand":{"unit":"flowers","value":4}},{"actor":"Tara","kind":"add","operand":{"unit":"flowers","value":6}}],"unknown":{"entity":"Tara","unit":"flowers"}},"patterns":["initial_has","operation_lose_loses","operation_get_more","question_how_many_now"],"notes":"Loss followed by get-more checks mixed subtract/add flow with consistent unit tracking."}