core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

audit(ADR-0136.S.3): refusal rescan v3 — exactly 1 barrier shift (gsm8k-0010) (#208)

Browse source 
Re-runs parse_and_solve on the 50-case GSM8K train sample on current
main (post-S.3) and compares to v2. Result: admitted=3/50 (unchanged),
wrong=0, exactly 1 barrier shifted v2→v3.

Shift: gsm8k-0010 (compound_statement → fraction_operand). S.3's
_INIT_MUTATION_RE resolves "Yun had 20 paperclips initially, but then
lost 12" to InitialPossession(Yun, 8, paperclips). First refusal moved
to sentence 2: "Marion has 1/4 more than what Yun currently has, plus
7" — needs fraction-operand + coreference-quantity + comparative-additive
arithmetic.

Top blockers (v3):
  compound_statement   5  (was 6)
  novel_initial_form   5  (unchanged)
  fraction_operand     4  (was 3 — gsm8k-0010 moved here)
  novel_initial_verb   4  (unchanged)

Artifacts:
- evals/gsm8k_math/train_sample/v1/rescan_v3.py
- evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json
- evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json
- docs/decisions/ADR-0136.S3-post-rescan.md
- tests/test_rescan_v3_invariants.py (7 tests; determinism + admission
  set unchanged + exactly-one-shift + 0010-specific shift assertions)
This commit is contained in:
Shay 2026-05-23 22:05:16 -07:00 committed by GitHub
parent b448657c15
commit a7feda3c19
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 1459 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

105
docs/decisions/ADR-0136.S3-post-rescan.md Normal file
View file

@ -0,0 +1,105 @@
# ADR-0136.S.3-post-rescan — Refusal Rescan v3: Barrier-Shift Ledger
**Status:** Accepted
**Parent:** [ADR-0136](./ADR-0136-statement-layer-corridor.md)
**Previous rescan:** [ADR-0136.S2-post-rescan](./ADR-0136.S2-post-rescan.md)
**Date:** 2026-05-23
## Context
The v2 refusal taxonomy was written before S.3 (compound initial-mutation)
landed. S.3 added one closed-shape extractor for `<Entity> had N <unit>,
but then <verb> M` and produced 0 direct admissions + 1 barrier shift
(gsm8k-0010). This rescan re-runs `parse_and_solve` on all 50 GSM8K
train-sample cases against current main (post-S.3) and refreshes the
ledger so any next phase inherits measured truth.
## Summary
| Metric | v2 | v3 | Δ |
|---|---|---|---|
| Admitted (correct) | 3 | 3 | 0 |
| Wrong | 0 | 0 | 0 |
| Refused | 47 | 47 | 0 |
| Cases shifted v2 → v3 | — | 1 | — |
## Current Admissions (unchanged from v2)
| Case | Answer | Path |
|---|---|---|
| gsm8k-0014 | 240.0 | S.1 capacity rate (short-circuit) |
| gsm8k-0018 | 16.0 | S.0 classifier + S.1 capacity (short-circuit) |
| gsm8k-0042 | 30.0 | S.2 conditional-op question (short-circuit) |
## Barrier-Shifted Cases (v2 → v3)
| Case | v2 barrier | v3 barrier | Reason |
|---|---|---|---|
| gsm8k-0010 | compound_statement | fraction_operand | S.3's `_INIT_MUTATION_RE` resolves sentence 1 ("Yun had 20 paperclips initially, but then lost 12") to `InitialPossession(Yun, 8, paperclips)`. Refuses on sentence 2: "Marion has 1/4 more than what Yun currently has, plus 7" — needs fraction-operand parsing + coreference to Yun's current quantity + comparative-additive arithmetic. |
S.3 produced exactly one barrier shift, as predicted by its brief
("0–1 new admissions; the real return is rescan v3 barrier-shifts").
The shape coverage worked as designed; the downstream barriers on the
other 5 `compound_statement` cases are structurally different and remain
deferred (see ADR-0136.S.3).
## Current Primary Barrier Distribution (v3)
| Barrier | v2 | v3 | Δ |
|---|---|---|---|
| compound_statement | 6 | 5 | −1 |
| novel_initial_form | 5 | 5 | 0 |
| fraction_operand | 3 | 4 | +1 |
| novel_initial_verb | 4 | 4 | 0 |
| admitted | 3 | 3 | 0 |
| compound_comparative | 3 | 3 | 0 |
| conditional_question | 3 | 3 | 0 |
| context_filler | 3 | 3 | 0 |
| distributive_multiply | 2 | 2 | 0 |
| multi_entity_initial | 2 | 2 | 0 |
| rate_price | 2 | 2 | 0 |
| temporal_frequency | 2 | 2 | 0 |
| capacity_rate | 1 | 1 | 0 |
| complex_question | 1 | 1 | 0 |
| compound_multi_event | 1 | 1 | 0 |
| conditional_branch | 1 | 1 | 0 |
| distributive_each_actor | 1 | 1 | 0 |
| goal_statement | 1 | 1 | 0 |
| multi_attribute_accumulation | 1 | 1 | 0 |
| multi_day_accumulation | 1 | 1 | 0 |
| partition_divide | 1 | 1 | 0 |
| percentage_rate | 1 | 1 | 0 |
| rate_earnings | 1 | 1 | 0 |
| temporal_age_anchor | 1 | 1 | 0 |
`compound_statement` (5) and `novel_initial_form` (5) are now tied for
the largest blocker bucket. `fraction_operand` (4) is now in the top 4
— a side-effect of S.3 exposing the sentence-2 barrier on gsm8k-0010.
## Next-Phase Candidates
The ledger does not dictate the next phase, but the largest tractable
clusters are:
- **novel_initial_form (5)** — sentence shapes the parser doesn't ground
(`A school has 100 students`, `In a building, there are a hundred
ladies`, etc.). Likely a widening of `_INITIAL_HAS_RE` /
`_INITIAL_THERE_ARE_RE` with the indefinite-article and impersonal-
existential cases. Possibly the lowest-cost statement-layer extension.
- **novel_initial_verb (4)** — `donated`, `collected`, `gained`, `bakes`
outside current closed sets. Pure vocabulary extension; probably
composes with the existing acquire-verb table.
- **compound_statement (5)** — five heterogeneous shapes (fraction +
coreference, nested temporal, comparative arithmetic, verb conjunction,
age-multiplier). Per ADR-0136.S.3 each is its own capability and
should not be bundled.
S.4 should target one of these clusters as a single closed shape, in
the same one-PR / one-shape / honest-delta discipline as S.1, S.2, S.3.
## Evidence
- Rescan ledger: `evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json`
- Taxonomy v3: `evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json`
- Script: `evals/gsm8k_math/train_sample/v1/rescan_v3.py`
- Tests: `tests/test_rescan_v3_invariants.py`

664
evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json Normal file
View file

@ -0,0 +1,664 @@
{
"adr": "0136.S.3-post-rescan",
"description": "Per-case barrier-shift ledger comparing v2 (post-S.2) to v3 (post-S.3) behavior of parse_and_solve.",
"per_case": [
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0001",
"current_first_refusal": "Tina makes $18.00 an hour.",
"current_outcome": "refused",
"current_primary_barrier": "rate_earnings",
"current_refusal_reason": "no admissible candidate for statement: 'Tina makes $18.00 an hour.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Tina makes $18.00 an hour.",
"previous_outcome": "refused",
"previous_primary_barrier": "rate_earnings"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0002",
"current_first_refusal": "She splits it up into 25-foot sections.",
"current_outcome": "refused",
"current_primary_barrier": "partition_divide",
"current_refusal_reason": "no admissible candidate for statement: 'She splits it up into 25-foot sections.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "She splits it up into 25-foot sections.",
"previous_outcome": "refused",
"previous_primary_barrier": "partition_divide"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0003",
"current_first_refusal": "The local bookstore donated 48 boxes of erasers.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_verb",
"current_refusal_reason": "no admissible candidate for statement: 'The local bookstore donated 48 boxes of erasers.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "The local bookstore donated 48 boxes of erasers.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_verb"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0004",
"current_first_refusal": "Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.",
"current_outcome": "refused",
"current_primary_barrier": "fraction_operand",
"current_refusal_reason": "no admissible candidate for statement: 'Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.",
"previous_outcome": "refused",
"previous_primary_barrier": "fraction_operand"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0005",
"current_first_refusal": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.",
"current_outcome": "refused",
"current_primary_barrier": "fraction_operand",
"current_refusal_reason": "no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\"",
"new_admission": false,
"notes": "",
"previous_first_refusal": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.",
"previous_outcome": "refused",
"previous_primary_barrier": "fraction_operand"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0006",
"current_first_refusal": "Mandy started reading books with only 8 pages when she was 6 years old.",
"current_outcome": "refused",
"current_primary_barrier": "temporal_age_anchor",
"current_refusal_reason": "no admissible candidate for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Mandy started reading books with only 8 pages when she was 6 years old.",
"previous_outcome": "refused",
"previous_primary_barrier": "temporal_age_anchor"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0007",
"current_first_refusal": "How many more boxes do they need if Francine has a total of 85 crayons?",
"current_outcome": "refused",
"current_primary_barrier": "conditional_question",
"current_refusal_reason": "no admissible candidate for question: 'How many more boxes do they need if Francine has a total of 85 crayons?'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "How many more boxes do they need if Francine has a total of 85 crayons?",
"previous_outcome": "refused",
"previous_primary_barrier": "conditional_question"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0008",
"current_first_refusal": "If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?",
"current_outcome": "refused",
"current_primary_barrier": "conditional_question",
"current_refusal_reason": "no admissible candidate for question: 'If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?",
"previous_outcome": "refused",
"previous_primary_barrier": "conditional_question"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0009",
"current_first_refusal": "If Jen has 150 ducks, how many total birds does she have?",
"current_outcome": "refused",
"current_primary_barrier": "conditional_question",
"current_refusal_reason": "no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "If Jen has 150 ducks, how many total birds does she have?",
"previous_outcome": "refused",
"previous_primary_barrier": "conditional_question"
},
{
"barrier_shifted": true,
"case_id": "gsm8k-train-sample-v1-0010",
"current_first_refusal": "Marion has 1/4 more than what Yun currently has, plus 7.",
"current_outcome": "refused",
"current_primary_barrier": "fraction_operand",
"current_refusal_reason": "no admissible candidate for statement: 'Marion has 1/4 more than what Yun currently has, plus 7.'",
"new_admission": false,
"notes": "Sentence 1 (compound_statement) now resolves via S.3's _INIT_MUTATION_RE to Yun=8 paperclips. Refuses on sentence 2: 'Marion has 1/4 more than what Yun currently has, plus 7' \u2014 needs fraction_operand parsing + coreference to Yun's current quantity + comparative-additive arithmetic.",
"previous_first_refusal": "Yun had 20 paperclips initially, but then lost 12.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0011",
"current_first_refusal": "Alexa has a lemonade stand where she sells lemonade for $2 for one cup.",
"current_outcome": "refused",
"current_primary_barrier": "rate_price",
"current_refusal_reason": "no admissible candidate for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Alexa has a lemonade stand where she sells lemonade for $2 for one cup.",
"previous_outcome": "refused",
"previous_primary_barrier": "rate_price"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0012",
"current_first_refusal": "He put all of them in his aquarium but his fish ate half of them.",
"current_outcome": "refused",
"current_primary_barrier": "compound_statement",
"current_refusal_reason": "no admissible candidate for statement: 'He put all of them in his aquarium but his fish ate half of them.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "He put all of them in his aquarium but his fish ate half of them.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0013",
"current_first_refusal": "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.",
"current_outcome": "refused",
"current_primary_barrier": "compound_statement",
"current_refusal_reason": "no admissible candidate for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0014",
"current_first_refusal": null,
"current_outcome": "admitted",
"current_primary_barrier": "admitted",
"current_refusal_reason": null,
"new_admission": false,
"notes": "",
"previous_first_refusal": null,
"previous_outcome": "admitted",
"previous_primary_barrier": "admitted"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0015",
"current_first_refusal": "Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.",
"current_outcome": "refused",
"current_primary_barrier": "compound_multi_event",
"current_refusal_reason": "no admissible candidate for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_multi_event"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0016",
"current_first_refusal": "On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.",
"current_outcome": "refused",
"current_primary_barrier": "compound_statement",
"current_refusal_reason": "no admissible candidate for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\"",
"new_admission": false,
"notes": "",
"previous_first_refusal": "On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0017",
"current_first_refusal": "He\u2019s charging $50.00 per day or $500.00 for 14 days.",
"current_outcome": "refused",
"current_primary_barrier": "conditional_branch",
"current_refusal_reason": "no admissible candidate for statement: 'He\u2019s charging $50.00 per day or $500.00 for 14 days.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "He\u2019s charging $50.00 per day or $500.00 for 14 days.",
"previous_outcome": "refused",
"previous_primary_barrier": "conditional_branch"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0018",
"current_first_refusal": null,
"current_outcome": "admitted",
"current_primary_barrier": "admitted",
"current_refusal_reason": null,
"new_admission": false,
"notes": "",
"previous_first_refusal": null,
"previous_outcome": "admitted",
"previous_primary_barrier": "admitted"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0019",
"current_first_refusal": "The dog ends up having health problems and this requires 3 vet appointments, which cost $400 each.",
"current_outcome": "refused",
"current_primary_barrier": "rate_price",
"current_refusal_reason": "no admissible candidate for statement: 'The dog ends up having health problems and this requires 3 vet appointments, which cost $400 each.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "The dog ends up having health problems and this requires 3 vet appointments, which cost $400 each.",
"previous_outcome": "refused",
"previous_primary_barrier": "rate_price"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0020",
"current_first_refusal": "Two puppies, two kittens, and three parakeets were for sale at the pet shop.",
"current_outcome": "refused",
"current_primary_barrier": "multi_entity_initial",
"current_refusal_reason": "no admissible candidate for statement: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Two puppies, two kittens, and three parakeets were for sale at the pet shop.",
"previous_outcome": "refused",
"previous_primary_barrier": "multi_entity_initial"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0021",
"current_first_refusal": "He bench presses 15 pounds for 10 reps and does 3 sets.",
"current_outcome": "refused",
"current_primary_barrier": "distributive_multiply",
"current_refusal_reason": "no admissible candidate for statement: 'He bench presses 15 pounds for 10 reps and does 3 sets.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "He bench presses 15 pounds for 10 reps and does 3 sets.",
"previous_outcome": "refused",
"previous_primary_barrier": "distributive_multiply"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0022",
"current_first_refusal": "Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.",
"current_outcome": "refused",
"current_primary_barrier": "context_filler",
"current_refusal_reason": "no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.",
"previous_outcome": "refused",
"previous_primary_barrier": "context_filler"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0023",
"current_first_refusal": "Nicole collected 400 Pokemon cards.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_verb",
"current_refusal_reason": "no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Nicole collected 400 Pokemon cards.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_verb"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0024",
"current_first_refusal": "Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.",
"current_outcome": "refused",
"current_primary_barrier": "multi_day_accumulation",
"current_refusal_reason": "no admissible candidate for statement: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.",
"previous_outcome": "refused",
"previous_primary_barrier": "multi_day_accumulation"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0025",
"current_first_refusal": "Lilibeth fills 6 baskets where each basket holds 50 strawberries.",
"current_outcome": "refused",
"current_primary_barrier": "distributive_multiply",
"current_refusal_reason": "no admissible candidate for statement: 'Lilibeth fills 6 baskets where each basket holds 50 strawberries.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Lilibeth fills 6 baskets where each basket holds 50 strawberries.",
"previous_outcome": "refused",
"previous_primary_barrier": "distributive_multiply"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0026",
"current_first_refusal": "Aaron and his brother Carson each saved up $40 to go to dinner.",
"current_outcome": "refused",
"current_primary_barrier": "distributive_each_actor",
"current_refusal_reason": "no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Aaron and his brother Carson each saved up $40 to go to dinner.",
"previous_outcome": "refused",
"previous_primary_barrier": "distributive_each_actor"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0027",
"current_first_refusal": "Malcolm has 240 followers on Instagram and 500 followers on Facebook.",
"current_outcome": "refused",
"current_primary_barrier": "multi_attribute_accumulation",
"current_refusal_reason": "no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Malcolm has 240 followers on Instagram and 500 followers on Facebook.",
"previous_outcome": "refused",
"previous_primary_barrier": "multi_attribute_accumulation"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0028",
"current_first_refusal": "It cost $100,000 to open initially.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_form",
"current_refusal_reason": "no admissible candidate for statement: 'It cost $100,000 to open initially.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "It cost $100,000 to open initially.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_form"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0029",
"current_first_refusal": "The cost of the keyboard was three times greater than the cost of the mouse.",
"current_outcome": "refused",
"current_primary_barrier": "compound_comparative",
"current_refusal_reason": "no admissible candidate for statement: 'The cost of the keyboard was three times greater than the cost of the mouse.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "The cost of the keyboard was three times greater than the cost of the mouse.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_comparative"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0030",
"current_first_refusal": "It is a 2-hour drive each way.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_form",
"current_refusal_reason": "no admissible candidate for statement: 'It is a 2-hour drive each way.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "It is a 2-hour drive each way.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_form"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0031",
"current_first_refusal": "Jeremie wants to go to an amusement park with 3 friends at the end of summer.",
"current_outcome": "refused",
"current_primary_barrier": "context_filler",
"current_refusal_reason": "no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Jeremie wants to go to an amusement park with 3 friends at the end of summer.",
"previous_outcome": "refused",
"previous_primary_barrier": "context_filler"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0032",
"current_first_refusal": "He draws and colors 10 pictures.",
"current_outcome": "refused",
"current_primary_barrier": "compound_statement",
"current_refusal_reason": "no admissible candidate for statement: 'He draws and colors 10 pictures.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "He draws and colors 10 pictures.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0033",
"current_first_refusal": "Rachel is 12 years old, and her grandfather is 7 times her age.",
"current_outcome": "refused",
"current_primary_barrier": "compound_statement",
"current_refusal_reason": "no admissible candidate for statement: 'Rachel is 12 years old, and her grandfather is 7 times her age.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Rachel is 12 years old, and her grandfather is 7 times her age.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0034",
"current_first_refusal": "He can run 40 yards within 5 seconds.",
"current_outcome": "refused",
"current_primary_barrier": "capacity_rate",
"current_refusal_reason": "no admissible candidate for statement: 'He can run 40 yards within 5 seconds.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "He can run 40 yards within 5 seconds.",
"previous_outcome": "refused",
"previous_primary_barrier": "capacity_rate"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0035",
"current_first_refusal": "How many more apples would Martha need to give away to be left with only 4 of them?",
"current_outcome": "refused",
"current_primary_barrier": "complex_question",
"current_refusal_reason": "no admissible candidate for question: 'How many more apples would Martha need to give away to be left with only 4 of them?'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "How many more apples would Martha need to give away to be left with only 4 of them?",
"previous_outcome": "refused",
"previous_primary_barrier": "complex_question"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0036",
"current_first_refusal": "She studied for 2 hours on Wednesday and three times as long on Thursday.",
"current_outcome": "refused",
"current_primary_barrier": "compound_comparative",
"current_refusal_reason": "no admissible candidate for statement: 'She studied for 2 hours on Wednesday and three times as long on Thursday.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "She studied for 2 hours on Wednesday and three times as long on Thursday.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_comparative"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0037",
"current_first_refusal": "Michael wants to lose 10 pounds by June.",
"current_outcome": "refused",
"current_primary_barrier": "goal_statement",
"current_refusal_reason": "no admissible candidate for statement: 'Michael wants to lose 10 pounds by June.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Michael wants to lose 10 pounds by June.",
"previous_outcome": "refused",
"previous_primary_barrier": "goal_statement"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0038",
"current_first_refusal": "In a building, there are a hundred ladies on the first-floor studying.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_form",
"current_refusal_reason": "no admissible candidate for statement: 'In a building, there are a hundred ladies on the first-floor studying.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "In a building, there are a hundred ladies on the first-floor studying.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_form"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0039",
"current_first_refusal": "Orlando gained 5 pounds.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_verb",
"current_refusal_reason": "no admissible candidate for statement: 'Orlando gained 5 pounds.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Orlando gained 5 pounds.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_verb"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0040",
"current_first_refusal": "He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat.",
"current_outcome": "refused",
"current_primary_barrier": "multi_entity_initial",
"current_refusal_reason": "no admissible candidate for statement: 'He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat.",
"previous_outcome": "refused",
"previous_primary_barrier": "multi_entity_initial"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0041",
"current_first_refusal": "The guests eat all of 1 pan, and 75% of the 2nd pan.",
"current_outcome": "refused",
"current_primary_barrier": "fraction_operand",
"current_refusal_reason": "no admissible candidate for statement: 'The guests eat all of 1 pan, and 75% of the 2nd pan.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "The guests eat all of 1 pan, and 75% of the 2nd pan.",
"previous_outcome": "refused",
"previous_primary_barrier": "fraction_operand"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0042",
"current_first_refusal": null,
"current_outcome": "admitted",
"current_primary_barrier": "admitted",
"current_refusal_reason": null,
"new_admission": false,
"notes": "",
"previous_first_refusal": null,
"previous_outcome": "admitted",
"previous_primary_barrier": "admitted"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0043",
"current_first_refusal": "Her mother gave her an additional $4, and her father twice as much as her mother.",
"current_outcome": "refused",
"current_primary_barrier": "compound_comparative",
"current_refusal_reason": "no admissible candidate for statement: 'Her mother gave her an additional $4, and her father twice as much as her mother.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Her mother gave her an additional $4, and her father twice as much as her mother.",
"previous_outcome": "refused",
"previous_primary_barrier": "compound_comparative"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0044",
"current_first_refusal": "John invests in a bank and gets 10% simple interest.",
"current_outcome": "refused",
"current_primary_barrier": "percentage_rate",
"current_refusal_reason": "no admissible candidate for statement: 'John invests in a bank and gets 10% simple interest.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "John invests in a bank and gets 10% simple interest.",
"previous_outcome": "refused",
"previous_primary_barrier": "percentage_rate"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0045",
"current_first_refusal": "Each survey has 10 questions.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_form",
"current_refusal_reason": "no admissible candidate for statement: 'Each survey has 10 questions.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Each survey has 10 questions.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_form"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0046",
"current_first_refusal": "A school has 100 students.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_form",
"current_refusal_reason": "no admissible candidate for statement: 'A school has 100 students.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "A school has 100 students.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_form"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0047",
"current_first_refusal": "John bakes 12 coconut macaroons, each weighing 5 ounces.",
"current_outcome": "refused",
"current_primary_barrier": "novel_initial_verb",
"current_refusal_reason": "no admissible candidate for statement: 'John bakes 12 coconut macaroons, each weighing 5 ounces.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "John bakes 12 coconut macaroons, each weighing 5 ounces.",
"previous_outcome": "refused",
"previous_primary_barrier": "novel_initial_verb"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0048",
"current_first_refusal": "Every week, he gets 6 cards.",
"current_outcome": "refused",
"current_primary_barrier": "temporal_frequency",
"current_refusal_reason": "no admissible candidate for statement: 'Every week, he gets 6 cards.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Every week, he gets 6 cards.",
"previous_outcome": "refused",
"previous_primary_barrier": "temporal_frequency"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0049",
"current_first_refusal": "Malcolm is trying to find the fastest walk to school and is currently comparing two routes.",
"current_outcome": "refused",
"current_primary_barrier": "context_filler",
"current_refusal_reason": "no admissible candidate for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Malcolm is trying to find the fastest walk to school and is currently comparing two routes.",
"previous_outcome": "refused",
"previous_primary_barrier": "context_filler"
},
{
"barrier_shifted": false,
"case_id": "gsm8k-train-sample-v1-0050",
"current_first_refusal": "Mark does a gig every other day for 2 weeks.",
"current_outcome": "refused",
"current_primary_barrier": "temporal_frequency",
"current_refusal_reason": "no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'",
"new_admission": false,
"notes": "",
"previous_first_refusal": "Mark does a gig every other day for 2 weeks.",
"previous_outcome": "refused",
"previous_primary_barrier": "temporal_frequency"
}
],
"schema_version": 1,
"summary": {
"admitted": 3,
"barrier_shifted_v2_to_v3": 1,
"refused": 47,
"total_cases": 50,
"wrong": 0
}
}

400
evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json Normal file
View file

@ -0,0 +1,400 @@
{
"adr": "0136.S.3-post-rescan",
"description": "Post-S.3 refusal taxonomy. Schema identical to v2; primary_barrier vocabulary unchanged.",
"per_case": [
{
"case_id": "gsm8k-train-sample-v1-0001",
"notes": "",
"primary_barrier": "rate_earnings",
"secondary_barriers": [
"conditional_branch",
"multi_statement"
]
},
{
"case_id": "gsm8k-train-sample-v1-0002",
"notes": "",
"primary_barrier": "partition_divide",
"secondary_barriers": [
"fraction_operand",
"coreference_pronoun"
]
},
{
"case_id": "gsm8k-train-sample-v1-0003",
"notes": "",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"distributive_multiply",
"rate_price"
]
},
{
"case_id": "gsm8k-train-sample-v1-0004",
"notes": "",
"primary_barrier": "fraction_operand",
"secondary_barriers": [
"conditional_question"
]
},
{
"case_id": "gsm8k-train-sample-v1-0005",
"notes": "",
"primary_barrier": "fraction_operand",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0006",
"notes": "",
"primary_barrier": "temporal_age_anchor",
"secondary_barriers": [
"multi_step_complex"
]
},
{
"case_id": "gsm8k-train-sample-v1-0007",
"notes": "",
"primary_barrier": "conditional_question",
"secondary_barriers": [
"multi_entity_initial"
]
},
{
"case_id": "gsm8k-train-sample-v1-0008",
"notes": "",
"primary_barrier": "conditional_question",
"secondary_barriers": [
"distributive_multiply"
]
},
{
"case_id": "gsm8k-train-sample-v1-0009",
"notes": "",
"primary_barrier": "conditional_question",
"secondary_barriers": [
"compound_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0010",
"notes": "Sentence 1 (compound_statement) now resolves via S.3's _INIT_MUTATION_RE to Yun=8 paperclips. Refuses on sentence 2: 'Marion has 1/4 more than what Yun currently has, plus 7' \u2014 needs fraction_operand parsing + coreference to Yun's current quantity + comparative-additive arithmetic.",
"primary_barrier": "fraction_operand",
"secondary_barriers": [
"coreference_quantity",
"comparative_arithmetic"
]
},
{
"case_id": "gsm8k-train-sample-v1-0011",
"notes": "",
"primary_barrier": "rate_price",
"secondary_barriers": [
"context_filler"
]
},
{
"case_id": "gsm8k-train-sample-v1-0012",
"notes": "",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"fraction_operand",
"coreference_pronoun"
]
},
{
"case_id": "gsm8k-train-sample-v1-0013",
"notes": "",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"temporal_frequency"
]
},
{
"case_id": "gsm8k-train-sample-v1-0014",
"notes": "",
"primary_barrier": "admitted",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0015",
"notes": "",
"primary_barrier": "compound_multi_event",
"secondary_barriers": [
"compound_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0016",
"notes": "",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"rate_question"
]
},
{
"case_id": "gsm8k-train-sample-v1-0017",
"notes": "",
"primary_barrier": "conditional_branch",
"secondary_barriers": [
"rate_price"
]
},
{
"case_id": "gsm8k-train-sample-v1-0018",
"notes": "",
"primary_barrier": "admitted",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0019",
"notes": "",
"primary_barrier": "rate_price",
"secondary_barriers": [
"percentage_of",
"conditional_branch"
]
},
{
"case_id": "gsm8k-train-sample-v1-0020",
"notes": "",
"primary_barrier": "multi_entity_initial",
"secondary_barriers": [
"rate_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0021",
"notes": "",
"primary_barrier": "distributive_multiply",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0022",
"notes": "",
"primary_barrier": "context_filler",
"secondary_barriers": [
"rate_earnings",
"compound_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0023",
"notes": "",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"fraction_operand",
"distributive_divide"
]
},
{
"case_id": "gsm8k-train-sample-v1-0024",
"notes": "",
"primary_barrier": "multi_day_accumulation",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0025",
"notes": "",
"primary_barrier": "distributive_multiply",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0026",
"notes": "",
"primary_barrier": "distributive_each_actor",
"secondary_barriers": [
"fraction_operand"
]
},
{
"case_id": "gsm8k-train-sample-v1-0027",
"notes": "",
"primary_barrier": "multi_attribute_accumulation",
"secondary_barriers": [
"multi_step_complex"
]
},
{
"case_id": "gsm8k-train-sample-v1-0028",
"notes": "",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"rate_price",
"temporal_frequency"
]
},
{
"case_id": "gsm8k-train-sample-v1-0029",
"notes": "",
"primary_barrier": "compound_comparative",
"secondary_barriers": [
"rate_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0030",
"notes": "",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"compound_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0031",
"notes": "",
"primary_barrier": "context_filler",
"secondary_barriers": [
"implicit_group_count"
]
},
{
"case_id": "gsm8k-train-sample-v1-0032",
"notes": "",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"percentage_of"
]
},
{
"case_id": "gsm8k-train-sample-v1-0033",
"notes": "",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"multi_step_complex"
]
},
{
"case_id": "gsm8k-train-sample-v1-0034",
"notes": "",
"primary_barrier": "capacity_rate",
"secondary_barriers": [
"percentage_of",
"capacity_rate"
]
},
{
"case_id": "gsm8k-train-sample-v1-0035",
"notes": "",
"primary_barrier": "complex_question",
"secondary_barriers": [
"coreference_pronoun"
]
},
{
"case_id": "gsm8k-train-sample-v1-0036",
"notes": "",
"primary_barrier": "compound_comparative",
"secondary_barriers": [
"compound_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0037",
"notes": "",
"primary_barrier": "goal_statement",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0038",
"notes": "",
"primary_barrier": "novel_initial_form",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0039",
"notes": "",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"multi_step_complex"
]
},
{
"case_id": "gsm8k-train-sample-v1-0040",
"notes": "",
"primary_barrier": "multi_entity_initial",
"secondary_barriers": [
"leg_count"
]
},
{
"case_id": "gsm8k-train-sample-v1-0041",
"notes": "",
"primary_barrier": "fraction_operand",
"secondary_barriers": [
"percentage_of"
]
},
{
"case_id": "gsm8k-train-sample-v1-0042",
"notes": "",
"primary_barrier": "admitted",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0043",
"notes": "",
"primary_barrier": "compound_comparative",
"secondary_barriers": [
"rate_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0044",
"notes": "",
"primary_barrier": "percentage_rate",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0045",
"notes": "",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"rate_price"
]
},
{
"case_id": "gsm8k-train-sample-v1-0046",
"notes": "",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"percentage_of"
]
},
{
"case_id": "gsm8k-train-sample-v1-0047",
"notes": "",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"embedded_per_unit"
]
},
{
"case_id": "gsm8k-train-sample-v1-0048",
"notes": "",
"primary_barrier": "temporal_frequency",
"secondary_barriers": []
},
{
"case_id": "gsm8k-train-sample-v1-0049",
"notes": "",
"primary_barrier": "context_filler",
"secondary_barriers": [
"compound_multi_event",
"rate_comparative"
]
},
{
"case_id": "gsm8k-train-sample-v1-0050",
"notes": "",
"primary_barrier": "temporal_frequency",
"secondary_barriers": []
}
],
"schema_version": 2,
"summary": {
"admitted": 3,
"refused": 47,
"total_cases": 50,
"wrong": 0
}
}

214
evals/gsm8k_math/train_sample/v1/rescan_v3.py Normal file
View file

@ -0,0 +1,214 @@
"""ADR-0136.S.3 post-rescan — barrier-shift ledger v3 over 50-case GSM8K train sample.
Measurement-only. Runs ``parse_and_solve`` on every case on current main
(post-S.3), compares to v2 taxonomy (ADR-0136.S.2-post-rescan), and writes:
- ``refusal_rescan_v3.json`` — per-case barrier-shift ledger (v2 → v3).
- ``refusal_taxonomy_v3.json`` — updated taxonomy reflecting post-S.3 parser.
Aborts on any ``wrong`` admission (answer != expected).
"""
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Any
from generate.math_candidate_graph import parse_and_solve
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_V2_TAXONOMY_PATH = _HERE / "refusal_taxonomy_v2.json"
_V2_RESCAN_PATH = _HERE / "refusal_rescan_v2.json"
_RESCAN_V3_PATH = _HERE / "refusal_rescan_v3.json"
_TAXONOMY_V3_PATH = _HERE / "refusal_taxonomy_v3.json"
_FIRST_REFUSAL_RE = re.compile(
r"no admissible candidate for (?:statement|question): ['\"](.+?)['\"]$"
)
# Barrier reclassification for cases whose first-refusal sentence changed since v2.
# Empty unless S.3 (or later) shifts a barrier. S.3 shifted exactly one case.
_V3_BARRIER_OVERRIDES: dict[str, dict[str, Any]] = {
"gsm8k-train-sample-v1-0010": {
"primary_barrier": "fraction_operand",
"secondary_barriers": ["coreference_quantity", "comparative_arithmetic"],
"notes": (
"Sentence 1 (compound_statement) now resolves via S.3's "
"_INIT_MUTATION_RE to Yun=8 paperclips. Refuses on sentence 2: "
"'Marion has 1/4 more than what Yun currently has, plus 7' — "
"needs fraction_operand parsing + coreference to Yun's current "
"quantity + comparative-additive arithmetic."
),
},
}
def _load_cases() -> list[dict[str, Any]]:
return [
json.loads(line)
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines()
if line.strip()
]
def _extract_first_refusal(reason: str | None) -> str | None:
if reason is None:
return None
body = reason.removeprefix("candidate_graph: ")
m = _FIRST_REFUSAL_RE.search(body)
return m.group(1) if m else None
def _classify_current_barrier(
case_id: str,
v2_barrier: str,
cur_outcome: str,
prev_first_ref: str | None,
cur_first_ref: str | None,
) -> str:
"""Carry over v2 barrier unless first-refusal shifted; then use override."""
if cur_outcome == "admitted":
return "admitted"
if cur_first_ref == prev_first_ref:
return v2_barrier
override = _V3_BARRIER_OVERRIDES.get(case_id)
if override is None:
raise RuntimeError(
f"{case_id}: first-refusal shifted but no v3 override registered. "
f"v2={prev_first_ref!r} v3={cur_first_ref!r}"
)
return override["primary_barrier"]
def build_rescan() -> tuple[dict[str, Any], dict[str, Any]]:
cases = _load_cases()
v2_data = json.loads(_V2_RESCAN_PATH.read_text(encoding="utf-8"))
v2_by_id = {c["case_id"]: c for c in v2_data["per_case"]}
v2_tax = json.loads(_V2_TAXONOMY_PATH.read_text(encoding="utf-8"))
v2_tax_by_id = {c["case_id"]: c for c in v2_tax["per_case"]}
rescan_per_case: list[dict[str, Any]] = []
tax_per_case: list[dict[str, Any]] = []
admitted: list[str] = []
wrong: list[tuple[str, float, float]] = []
barrier_shifted_count = 0
for c in cases:
cid = c["case_id"]
r = parse_and_solve(c["question"])
cur_outcome = "admitted" if r.answer is not None else "refused"
cur_first_ref = _extract_first_refusal(r.refusal_reason)
cur_reason = r.refusal_reason
if cur_outcome == "admitted":
if r.answer == c["answer_numeric"]:
admitted.append(cid)
else:
wrong.append((cid, r.answer, c["answer_numeric"]))
prev = v2_by_id[cid]
prev_outcome = prev["current_outcome"]
prev_first_ref = prev["current_first_refusal"]
prev_barrier = v2_tax_by_id[cid]["primary_barrier"]
cur_barrier = _classify_current_barrier(
cid, prev_barrier, cur_outcome, prev_first_ref, cur_first_ref
)
outcome_changed = cur_outcome != prev_outcome
first_ref_changed = cur_first_ref != prev_first_ref
barrier_changed = cur_barrier != prev_barrier
shifted = outcome_changed or first_ref_changed or barrier_changed
if shifted:
barrier_shifted_count += 1
rescan_per_case.append({
"case_id": cid,
"previous_outcome": prev_outcome,
"current_outcome": cur_outcome,
"previous_primary_barrier": prev_barrier,
"current_primary_barrier": cur_barrier,
"barrier_shifted": shifted,
"new_admission": cur_outcome == "admitted" and prev_outcome != "admitted",
"previous_first_refusal": prev_first_ref,
"current_first_refusal": cur_first_ref,
"current_refusal_reason": cur_reason,
"notes": _V3_BARRIER_OVERRIDES.get(cid, {}).get("notes", ""),
})
v2_tax_entry = v2_tax_by_id[cid]
if cid in _V3_BARRIER_OVERRIDES and cur_outcome == "refused":
ov = _V3_BARRIER_OVERRIDES[cid]
tax_per_case.append({
"case_id": cid,
"primary_barrier": ov["primary_barrier"],
"secondary_barriers": ov["secondary_barriers"],
"notes": ov["notes"],
})
else:
tax_per_case.append({
"case_id": cid,
"primary_barrier": cur_barrier,
"secondary_barriers": v2_tax_entry.get("secondary_barriers", []),
"notes": v2_tax_entry.get("notes", ""),
})
if wrong:
raise RuntimeError(f"wrong admissions detected: {wrong}")
rescan = {
"schema_version": 1,
"adr": "0136.S.3-post-rescan",
"description": (
"Per-case barrier-shift ledger comparing v2 (post-S.2) to v3 "
"(post-S.3) behavior of parse_and_solve."
),
"summary": {
"total_cases": len(cases),
"admitted": len(admitted),
"wrong": len(wrong),
"refused": len(cases) - len(admitted),
"barrier_shifted_v2_to_v3": barrier_shifted_count,
},
"per_case": rescan_per_case,
}
taxonomy = {
"schema_version": 2,
"adr": "0136.S.3-post-rescan",
"description": (
"Post-S.3 refusal taxonomy. Schema identical to v2; primary_barrier "
"vocabulary unchanged."
),
"summary": {
"total_cases": len(cases),
"admitted": len(admitted),
"wrong": len(wrong),
"refused": len(cases) - len(admitted),
},
"per_case": tax_per_case,
}
return rescan, taxonomy
def main() -> int:
rescan, taxonomy = build_rescan()
_RESCAN_V3_PATH.write_text(
json.dumps(rescan, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
_TAXONOMY_V3_PATH.write_text(
json.dumps(taxonomy, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
print(
f"v3 rescan: admitted={rescan['summary']['admitted']}/"
f"{rescan['summary']['total_cases']} wrong={rescan['summary']['wrong']} "
f"shifted_v2_to_v3={rescan['summary']['barrier_shifted_v2_to_v3']}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

76
tests/test_rescan_v3_invariants.py Normal file
View file

@ -0,0 +1,76 @@
"""ADR-0136.S.3-post-rescan — invariant tests for refusal rescan v3."""
from __future__ import annotations
import json
from pathlib import Path
from evals.gsm8k_math.train_sample.v1.rescan_v3 import build_rescan
_HERE = Path(__file__).resolve().parent.parent
_RESCAN_V3 = _HERE / "evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json"
_TAXONOMY_V3 = _HERE / "evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json"
def test_wrong_is_zero() -> None:
"""GSM8K probe under v3 must hold wrong == 0."""
rescan, _ = build_rescan()
assert rescan["summary"]["wrong"] == 0
def test_admission_set_unchanged() -> None:
"""S.3 did not change the admission set: {0014, 0018, 0042}."""
rescan, _ = build_rescan()
admitted = {
c["case_id"] for c in rescan["per_case"] if c["current_outcome"] == "admitted"
}
assert admitted == {
"gsm8k-train-sample-v1-0014",
"gsm8k-train-sample-v1-0018",
"gsm8k-train-sample-v1-0042",
}
def test_exactly_one_shift_v2_to_v3() -> None:
"""S.3 shifted exactly one case (gsm8k-0010)."""
rescan, _ = build_rescan()
assert rescan["summary"]["barrier_shifted_v2_to_v3"] == 1
shifted = [c["case_id"] for c in rescan["per_case"] if c["barrier_shifted"]]
assert shifted == ["gsm8k-train-sample-v1-0010"]
def test_gsm8k_0010_now_refuses_on_sentence_2() -> None:
"""gsm8k-0010 first-refusal moved from sentence 1 to sentence 2."""
rescan, _ = build_rescan()
case = next(
c for c in rescan["per_case"]
if c["case_id"] == "gsm8k-train-sample-v1-0010"
)
assert case["previous_first_refusal"].startswith("Yun had 20 paperclips")
assert case["current_first_refusal"].startswith("Marion has 1/4")
assert case["previous_primary_barrier"] == "compound_statement"
assert case["current_primary_barrier"] == "fraction_operand"
def test_artifacts_deterministic() -> None:
"""Two runs of build_rescan() produce byte-equal artifacts."""
r1, t1 = build_rescan()
r2, t2 = build_rescan()
assert json.dumps(r1, sort_keys=True) == json.dumps(r2, sort_keys=True)
assert json.dumps(t1, sort_keys=True) == json.dumps(t2, sort_keys=True)
def test_taxonomy_v3_has_50_entries() -> None:
_, taxonomy = build_rescan()
assert len(taxonomy["per_case"]) == 50
def test_disk_artifacts_match_freshly_computed() -> None:
"""If rescan v3 was committed, disk artifacts must match a fresh run."""
if not _RESCAN_V3.exists() or not _TAXONOMY_V3.exists():
return
rescan, taxonomy = build_rescan()
on_disk_rescan = json.loads(_RESCAN_V3.read_text(encoding="utf-8"))
on_disk_taxonomy = json.loads(_TAXONOMY_V3.read_text(encoding="utf-8"))
assert on_disk_rescan == rescan
assert on_disk_taxonomy == taxonomy