57b257ca1d
First Phase of ADR-0114's expert-capability roadmap. Decomposed into four sub-phases so each lands as its own auditable step: 1.1 schema + 5 seed cases + invariants ← this commit 1.2 45 more dev-set cases ← delegated (Codex) 1.3 the parser itself ← exit: ≥0.90 on dev set 1.4 runtime binding ← if non-trivial What landed - generate/math_problem_graph.py — typed dataclasses (Quantity, InitialPossession, Operation, Unknown, MathProblemGraph) + frozen validation + canonical_bytes() byte-deterministic serialization + graph_from_dict roundtrip. - evals/gsm8k_parser_dev/cases.jsonl — 5 seed cases (gpd-001..005) covering single-add, single-subtract, multi-step, two-entity transfer, and multi-entity sum constructions. Every case carries a ground_truth_graph and the documented patterns it exercises. - evals/gsm8k_parser_dev/README.md — authoring contract: schema, pattern registry, canonicalization rules, Phase 1.1 scope boundary, hand-solving rubric, distribution target for the remaining 45 cases. This is the spec Phase 1.2 authors work against. - tests/test_math_problem_graph.py — 26 cases pinning four invariants: round-trip byte equality, canonical_bytes() determinism, schema rejection of malformed graphs, and ground_truth_graph ↔ expected_answer agreement (a hand-solver inside the test module falsifies mis-authored cases). Why this is sticky The Phase 1.1 schema is load-bearing for Phase 1.2 (the 45 authored cases will be written against it) AND Phase 1.3 (the parser will be graded byte-equal against ground-truth graphs in this schema). Changing the schema after Phase 1.2 lands requires an amendment ADR + rewriting authored cases. The schema choices here are intentionally conservative. Tests: 26/26 new; 67/67 smoke green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
5 lines
3.2 KiB
JSON
5 lines
3.2 KiB
JSON
{"id":"gpd-001","problem":"Sam has 5 apples. He buys 3 more. How many apples does Sam have?","expected_answer":8,"expected_unit":"apples","ground_truth_graph":{"entities":["Sam"],"initial_state":[{"entity":"Sam","quantity":{"unit":"apples","value":5}}],"operations":[{"actor":"Sam","kind":"add","operand":{"unit":"apples","value":3}}],"unknown":{"entity":"Sam","unit":"apples"}},"patterns":["initial_has","operation_buy_more","question_how_many_entity"],"notes":"Single-entity, single-add. The simplest GSM8K-style pattern. 'He' resolves anaphorically to 'Sam'."}
|
|
{"id":"gpd-002","problem":"Tom has 12 candies. He eats 4. How many candies does Tom have left?","expected_answer":8,"expected_unit":"candies","ground_truth_graph":{"entities":["Tom"],"initial_state":[{"entity":"Tom","quantity":{"unit":"candies","value":12}}],"operations":[{"actor":"Tom","kind":"subtract","operand":{"unit":"candies","value":4}}],"unknown":{"entity":"Tom","unit":"candies"}},"patterns":["initial_has","operation_eat_loses","question_how_many_left"],"notes":"Single-entity, single-subtract. 'eats' is a loss verb. Question suffix 'left' is a comprehension cue but the unknown shape is identical to gpd-001."}
|
|
{"id":"gpd-003","problem":"Lisa has 10 books. She buys 5 more, then donates 3. How many books does Lisa have?","expected_answer":12,"expected_unit":"books","ground_truth_graph":{"entities":["Lisa"],"initial_state":[{"entity":"Lisa","quantity":{"unit":"books","value":10}}],"operations":[{"actor":"Lisa","kind":"add","operand":{"unit":"books","value":5}},{"actor":"Lisa","kind":"subtract","operand":{"unit":"books","value":3}}],"unknown":{"entity":"Lisa","unit":"books"}},"patterns":["initial_has","operation_buy_more","operation_donate_loses","question_how_many_entity"],"notes":"Single-entity, multi-step. 'then' is a sequence marker; operation order must follow text order."}
|
|
{"id":"gpd-004","problem":"Anna has 8 marbles. She gives 3 to Ben. How many marbles does Anna have now?","expected_answer":5,"expected_unit":"marbles","ground_truth_graph":{"entities":["Anna","Ben"],"initial_state":[{"entity":"Anna","quantity":{"unit":"marbles","value":8}}],"operations":[{"actor":"Anna","kind":"transfer","operand":{"unit":"marbles","value":3},"target":"Ben"}],"unknown":{"entity":"Anna","unit":"marbles"}},"patterns":["initial_has","operation_give_transfer","question_how_many_entity"],"notes":"Two-entity transfer. Ben appears only as a transfer target — no initial possession is asserted for Ben. The 'transfer' kind decomposes downstream into subtract(actor) + add(target); the parser emits the closer-to-NL form."}
|
|
{"id":"gpd-005","problem":"Tom has 4 stickers. Sara has 7 stickers. How many stickers do they have in total?","expected_answer":11,"expected_unit":"stickers","ground_truth_graph":{"entities":["Tom","Sara"],"initial_state":[{"entity":"Tom","quantity":{"unit":"stickers","value":4}},{"entity":"Sara","quantity":{"unit":"stickers","value":7}}],"operations":[],"unknown":{"entity":null,"unit":"stickers"}},"patterns":["initial_has","initial_has","question_how_many_total"],"notes":"Multi-entity initial possessions, no operations, sum question. 'they' refers to all introduced entities; Unknown.entity=null signals total across entities."}
|