core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 9 Pull requests Projects Releases Packages Wiki Activity Actions 3

test(gsm8k): add composition validation corpus

Browse source
This commit is contained in:
Shay 2026-06-03 16:30:29 -07:00
parent b2b3678deb
commit a8027ca34d
2 changed files with 82 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
evals/gsm8k_math/composition_validation/v1/cases.jsonl Normal file
View file

@ -0,0 +1,20 @@
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0001","composition":"R1","gate":"5b-R1","gold":64,"note":"Derived keyboard cost = 3 * mouse cost, then total accessories = keyboard + mouse.","question":"Fabian bought a brand new computer mouse and keyboard to be able to work from home. The cost of the keyboard was three times greater than the cost of the mouse. If the mouse cost $16, how much did Fabian spent on his new accessories?","source":"gsm8k_train_sample:0029","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0002","composition":"R1","gate":"5b-R1","gold":400,"note":"Derived second-floor count = 3 * first-floor count, then total floors.","question":"In a building, there are a hundred ladies on the first-floor studying. There are three times that many girls at a party being held on the second floor of the building. How many ladies are on the two floors in total?","source":"gsm8k_train_sample:0038","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0003","composition":"R1","gate":"5b-R1","gold":9,"note":"Derived total beads from two products, then divide by beads per bracelet.","question":"Marnie makes bead bracelets. She bought 5 bags of 50 beads and 2 bags of 100 beads. If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?","source":"gsm8k_train_sample:0008","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0004","composition":"R1","gate":"5b-R1","gold":3840,"note":"Chained intermediate symbols: combined IG+Facebook, Twitter, TikTok, YouTube, then total.","question":"Malcolm has 240 followers on Instagram and 500 followers on Facebook. The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined. Meanwhile, the number of followers he has on TikTok is 3 times the number of followers is has on Twitter, and he has 510 more followers on Youtube than he has on TikTok. How many followers does Malcolm have on all his social media?","source":"gsm8k_train_sample:0027","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss.","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0006","composition":"R5","gate":"5b-R5","gold":14,"note":"Multi-step duration: round-trip drive time, scalar beach time, total trip time.","question":"Jake decides to go to the beach for a fun day. It is a 2-hour drive each way. He then spends 2.5 times at long at the beach as his total driving time. How much time does the trip take?","source":"gsm8k_train_sample:0030","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0007","composition":"R6","gate":"5b-R6","gold":21,"note":"Fraction mutation: final temperature = 3/4 of current, decrease = current - final.","question":"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?","source":"gsm8k_train_sample:0005","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0008","composition":"R6","gate":"5b-R6","gold":15,"note":"Percent partition: girls/boys split, percent of each, then total owners.","question":"A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?","source":"gsm8k_train_sample:0046","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":1,"baseline_verdict":"refuse","case_id":"cv-0009","composition":"compare_mult","gate":"5b-compare-mult","gold":60,"note":"Referenced multiplicative comparison should eventually solve; today it is a completeness-guard refusal.","question":"Ivan has 20 dice. Jerry has twice as many dice as Ivan. How many dice do they have altogether?","source":"guard:real_gsm8k_605","target_verdict":"solve"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0010","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has 3 times as many apples. How many apples do they have together?","source":"guard:n_times_without_reference:3_as_many","target_verdict":"refuse"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0011","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has 5 times as many apples. How many apples do they have together?","source":"guard:n_times_without_reference:5_as_many","target_verdict":"refuse"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0012","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has three times more apples. How many apples do they have together?","source":"guard:n_times_without_reference:three_more","target_verdict":"refuse"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0013","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; any admission would be an incomplete-reading failure.","question":"Tom has 7 apples. Jerry has 3 times the number of apples. How many apples do they have together?","source":"guard:n_times_without_reference:3_the_number_of","target_verdict":"refuse"}
{"baseline_answer":null,"baseline_branches_enumerated":1,"baseline_verdict":"refuse","case_id":"cv-0014","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; completeness guard currently catches the unconsumed scalar.","question":"Tom has 7 apples. Jerry has twice as many apples. How many apples do they have together?","source":"guard:existing_multiplier_refusal:twice_no_reference","target_verdict":"refuse"}
{"baseline_answer":null,"baseline_branches_enumerated":1,"baseline_verdict":"refuse","case_id":"cv-0015","composition":"compare_mult","gate":"permanent","gold":null,"note":"No comparison reference; completeness guard currently catches the unconsumed scalar.","question":"Tom has 7 apples. Jerry has double the apples. How many apples do they have together?","source":"guard:existing_multiplier_refusal:double_no_reference","target_verdict":"refuse"}
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0016","composition":"world_knowledge","gate":"permanent","gold":72,"note":"Requires unstated legs-per-animal world knowledge; remains outside the decidable regime for this corpus.","question":"Over several years, Daniel has adopted any stray animals he sees on the side of the road. He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat. All of the animals are perfectly healthy. In total, how many legs do his animals have?","source":"gsm8k_train_sample:0040","target_verdict":"refuse"}
{"baseline_answer":438,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0017","composition":"compare_mult_additive","gate":"baseline","gold":438,"note":"Existing graph-path correct case; completeness guard must not over-refuse it.","question":"Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday. Brooke does three times as many jumping jacks as Sidney. How many jumping jacks did Brooke do?","source":"gsm8k_train_sample:0024","target_verdict":"solve"}
{"baseline_answer":28,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0018","composition":"compare_mult","gate":"baseline","gold":28,"note":"Referenced multiplicative comparison already solves; protects against over-broad no-reference refusal.","question":"Tom has 7 apples. Jerry has 3 times as many apples as Tom. How many apples do they have together?","source":"guard:n_times_with_reference:3_as_many_as_tom","target_verdict":"solve"}
{"baseline_answer":1200,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0019","composition":"additive","gate":"baseline","gold":1200,"note":"Stated-sum contrastive control for the derived-symbol Nicole/Cindy form.","question":"Nicole has 400 cards. Cindy has 800 cards. How many cards do they have together?","source":"analysis:composition-capability-scope:R1_contrastive_control","target_verdict":"solve"}
{"baseline_answer":450,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0020","composition":"product_additive","gate":"baseline","gold":450,"note":"Existing guarded product bridge correct case; protects current 6/44/0 lift.","question":"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. How much total weight does he move?","source":"gsm8k_train_sample:0021","target_verdict":"solve"}

62
evals/gsm8k_math/composition_validation/v1/contract.md Normal file
View file

@ -0,0 +1,62 @@
# GSM8K Composition Validation v1
This corpus is a Phase 5b measurement instrument, not a serving lane and not an
emission implementation. It pins a small set of real and guard-seeded GSM8K
composition shapes so each future slice is measured against two coupled facts:
coverage may rise only while `wrong == 0` remains true.
## Scope
- 20 cases total.
- Positive cases use `target_verdict = solve` and a non-baseline `gate`; they
refuse today and should flip only when that gate's capability slice lands.
- Hard-negative cases use `target_verdict = refuse` and `gate = permanent`; they
must continue refusing because the input is incomplete or outside the current
decidable regime.
- Baseline controls use `gate = baseline`; they already solve and must keep
solving to the exact gold answer.
## Schema
Each JSONL row has these fields:
- `case_id`: stable corpus-local id.
- `source`: exact source pointer (`gsm8k_train_sample:*`, `guard:*`, or
`analysis:*`).
- `question`: prompt passed to `generate.math_candidate_graph.parse_and_solve`.
- `gold`: numeric gold answer, or `null` for permanent hard-negatives where no
complete reading exists.
- `target_verdict`: `solve` or `refuse`.
- `composition`: coarse composition family.
- `gate`: `baseline`, `permanent`, or a future capability gate such as `5b-R1`.
- `baseline_verdict`: tree-verified current verdict at corpus creation.
- `baseline_answer`: current answer when `baseline_verdict = solve`, otherwise
`null`.
- `baseline_branches_enumerated`: current `CandidateGraphResult` branch count.
- `note`: short reason the case is included.
## Invariants
1. Wrong-zero firewall: every case must either refuse or answer exactly `gold`;
a non-gold admission is a failure.
2. Null-gold hard-negatives: when `gold` is `null`, any admission is a failure.
3. Baseline controls: every `gate = baseline` row must solve to `gold` at every
step.
4. Permanent refusals: every `gate = permanent` row must refuse until a future
reviewed ADR explicitly changes its decidable regime.
5. Progress metric: count rows where `target_verdict = solve` and the current
answer equals `gold`; this starts at the baseline controls and may increase
only when invariants 1-4 still pass.
## Baseline
At creation on `origin/main` lineage after PR #534 plus the docs-only runway
correction, the intended baseline is:
- 4 solve
- 16 refuse
- 0 wrong
The corpus deliberately includes both future positives and permanent
hard-negatives. A future Phase 5b slice should update only the measured current
result, not the row's baseline fields.