core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

audit(ADR-0136.S.2): refusal rescan v2 — barrier-shift ledger, subsumption directive pinned (#205)

Browse source 
Measurement-only branch. Re-runs parse_and_solve on all 50 GSM8K train-sample
cases against the current parser (post-S.1/S.2) and produces a barrier-shift
ledger comparing v1 taxonomy to current behavior.

Results: admitted=3/50 (0014, 0018, 0042), wrong=0, barrier_shifted=27/50.
Context-filler dominance collapsed from 23→3 cases; compound_statement (6)
and novel_initial_form (5) are now the largest buckets.

Subsumption directive pinned: ADR-0137 SHALL re-derive all short-circuit
admissions as (DeferredCandidate, evidence, BindingProof) triples.
This commit is contained in:
Shay 2026-05-23 21:43:25 -07:00 committed by GitHub
parent e7a1ffb72e
commit 684481910b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 1633 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

159
docs/decisions/ADR-0136.S2-post-rescan.md Normal file
View file

@ -0,0 +1,159 @@
# ADR-0136.S.2-post-rescan — Refusal Rescan v2: Barrier-Shift Ledger
**Status:** Accepted
**Parent:** ADR-0136 (Statement Layer Corridor)
**Date:** 2026-05-23
## Context
The v1 refusal taxonomy (ADR-0136.S.0) was written before S.1 and S.2 landed.
This rescan re-runs `parse_and_solve` on all 50 GSM8K train-sample cases against
the current parser (post-S.1 rate/event parsing, post-S.2 conditional-op
questions) and produces a barrier-shift ledger so ADR-0137 inherits measured
truth, not stale classification.
## Summary
| Metric | Value |
|---|---|
| Total cases | 50 |
| Admitted (correct) | 3 |
| Wrong | 0 |
| Refused | 47 |
| Barrier shifted (v1 → v2) | 27 |
## Current Admissions
| Case | Answer | Short-circuit path | Landed in |
|---|---|---|---|
| gsm8k-0014 | 240.0 | S.1 capacity rate | ADR-0136.S.1 |
| gsm8k-0018 | 16.0 | S.2 capacity rate (context filler resolved) | ADR-0136.S.2 |
| gsm8k-0042 | 30.0 | S.2 conditional-op question | ADR-0136.S.2 |
## Current Primary Barrier Distribution
| Primary barrier | Count | Notes |
|---|---|---|
| compound_statement | 6 | Two operations in one sentence |
| novel_initial_form | 5 | Existential/impersonal/quantifier sentence shapes |
| novel_initial_verb | 4 | Verb not in any closed set (donated, collected, gained, bakes) |
| admitted | 3 | Successfully parsed and solved |
| compound_comparative | 3 | N times as much/greater comparisons |
| conditional_question | 3 | If-clause in question sentence |
| context_filler | 3 | Narrative scene-setter still blocking (down from 23) |
| fraction_operand | 3 | Half/quarter/3-4/75% operands |
| distributive_multiply | 2 | N bags × M items |
| multi_entity_initial | 2 | Multiple entities in one sentence |
| rate_price | 2 | Price embedded in relative clause |
| temporal_frequency | 2 | Every N days / every other day |
| capacity_rate | 1 | "within" preposition instead of "in" |
| complex_question | 1 | Modal + infinitive chain in question |
| compound_multi_event | 1 | Multiple transport legs in one sentence |
| conditional_branch | 1 | Or-choice pricing |
| distributive_each_actor | 1 | "each saved" = per-actor distribution |
| goal_statement | 1 | "wants to lose" = intent, not state |
| multi_attribute_accumulation | 1 | Multiple platform counts in one sentence |
| multi_day_accumulation | 1 | Values across Mon–Thu in one sentence |
| partition_divide | 1 | Splits into sections |
| percentage_rate | 1 | 10% simple interest |
| rate_earnings | 1 | Multi-statement earnings with conditional |
| temporal_age_anchor | 1 | Age-conditioned initial state |
## Barrier-Shifted Cases (v1 → v2)
27 cases changed primary barrier classification. The dominant pattern: 18 of 23
v1 `context_filler` cases now parse their scene-setting sentence, exposing the
next structural barrier. Only 3 `context_filler` cases remain blocking (0022,
0031, 0049 — all have rate/intent embedded in complex relative clauses).
| Case | v1 barrier | v2 barrier |
|---|---|---|
| 0003 | context_filler | novel_initial_verb |
| 0004 | indefinite_quantity | fraction_operand |
| 0005 | compound_statement | fraction_operand |
| 0007 | context_filler | conditional_question |
| 0008 | context_filler | conditional_question |
| 0009 | compound_comparative | conditional_question |
| 0013 | context_filler | compound_statement |
| 0014 | capacity_rate | **admitted** |
| 0017 | context_filler | conditional_branch |
| 0018 | context_filler | **admitted** |
| 0019 | context_filler | rate_price |
| 0021 | context_filler | distributive_multiply |
| 0023 | multi_step_complex | novel_initial_verb |
| 0025 | context_filler | distributive_multiply |
| 0028 | context_filler | novel_initial_form |
| 0029 | context_filler | compound_comparative |
| 0030 | context_filler | novel_initial_form |
| 0032 | context_filler | compound_statement |
| 0034 | context_filler | capacity_rate |
| 0035 | coreference_pronoun | complex_question |
| 0036 | context_filler | compound_comparative |
| 0039 | context_filler | novel_initial_verb |
| 0040 | context_filler | multi_entity_initial |
| 0042 | distributive_multiply | **admitted** |
| 0043 | context_filler | compound_comparative |
| 0045 | context_filler | novel_initial_form |
| 0048 | context_filler | temporal_frequency |
## Subsumption Directive
> S.1/S.2 short-circuit paths are tactical bridges. ADR-0137 SHALL subsume them:
> every current short-circuit admission must be re-derivable as a
> (DeferredCandidate, evidence, BindingProof) triple. Short-circuits are kept
> only as regression fixtures.
## ADR-0137 Probe Cases
The following cases are required in the ADR-0137 probe to verify subsumption
and test the next wave of unlocks:
1. **gsm8k-0014** — Bob shucks oysters (capacity rate, currently admitted via S.1)
2. **gsm8k-0018** — Xavier scores goals (capacity rate, currently admitted via S.2)
3. **gsm8k-0042** — Ella sells apples (conditional-op, currently admitted via S.2)
4. **gsm8k-0034** — Georgie runs yards (capacity rate, blocked by "within" preposition)
5. **gsm8k-0021** — John bench presses (distributive multiply, 15×10×3)
6. **gsm8k-0003** — Bookstore erasers (novel verb "donated", 48 boxes × 24 each)
7. **gsm8k-0029** — Mouse and keyboard (compound comparative, 3× cost ratio)
Cases 1–3 test subsumption (must remain admitted). Cases 4–7 are the
shallowest remaining barriers that ADR-0137's binding graph should unlock.
## Closed Barrier Vocabulary (v2)
The `primary_barrier` field in `refusal_taxonomy_v2.json` uses exactly this
enum (24 values):
```
admitted
capacity_rate
complex_question
compound_comparative
compound_multi_event
compound_statement
conditional_branch
conditional_question
context_filler
distributive_each_actor
distributive_multiply
fraction_operand
goal_statement
multi_attribute_accumulation
multi_day_accumulation
multi_entity_initial
novel_initial_form
novel_initial_verb
partition_divide
percentage_rate
rate_earnings
rate_price
temporal_age_anchor
temporal_frequency
```
## Evidence
- Rescan ledger: `evals/gsm8k_math/train_sample/v1/refusal_rescan_v2.json`
- Taxonomy v2: `evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v2.json`
- Script: `evals/gsm8k_math/train_sample/v1/rescan_v2.py`
- Tests: `tests/test_rescan_v2_invariants.py`

619
evals/gsm8k_math/train_sample/v1/refusal_rescan_v2.json Normal file
View file

@ -0,0 +1,619 @@
{
"schema_version": 2,
"adr": "0136.S.2-post-rescan",
"description": "Barrier-shift ledger over 50 GSM8K train-sample cases, comparing v1 taxonomy (ADR-0136.S.0) to current parser behavior post-S.1/S.2.",
"summary": {
"total_cases": 50,
"admitted": 3,
"refused": 47,
"wrong": 0,
"barrier_shifted_count": 27,
"new_admissions": [
"gsm8k-train-sample-v1-0014",
"gsm8k-train-sample-v1-0018",
"gsm8k-train-sample-v1-0042"
]
},
"per_case": [
{
"case_id": "gsm8k-train-sample-v1-0001",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "rate_earnings",
"current_primary_barrier": "rate_earnings",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Tina makes $18.00 an hour.",
"current_refusal_reason": "no admissible candidate for statement: 'Tina makes $18.00 an hour.'",
"notes": "multi-stmt earnings; short-circuit needs single stmt+q"
},
{
"case_id": "gsm8k-train-sample-v1-0002",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "partition_divide",
"current_primary_barrier": "partition_divide",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "She splits it up into 25-foot sections.",
"current_refusal_reason": "no admissible candidate for statement: 'She splits it up into 25-foot sections.'",
"notes": "splits into 25-foot sections = partition op"
},
{
"case_id": "gsm8k-train-sample-v1-0003",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "novel_initial_verb",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "The local bookstore donated 48 boxes of erasers.",
"current_refusal_reason": "no admissible candidate for statement: 'The local bookstore donated 48 boxes of erasers.'",
"notes": "context filler now parses; donated not in verb set"
},
{
"case_id": "gsm8k-train-sample-v1-0004",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "indefinite_quantity",
"current_primary_barrier": "fraction_operand",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.",
"current_refusal_reason": "no admissible candidate for statement: 'Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.'",
"notes": "indef qty now parses; fraction ops (half, 1/4) block"
},
{
"case_id": "gsm8k-train-sample-v1-0005",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_statement",
"current_primary_barrier": "fraction_operand",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.",
"current_refusal_reason": "no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\"",
"notes": "3/4 fraction operand in temporal decrease"
},
{
"case_id": "gsm8k-train-sample-v1-0006",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "temporal_age_anchor",
"current_primary_barrier": "temporal_age_anchor",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Mandy started reading books with only 8 pages when she was 6 years old.",
"current_refusal_reason": "no admissible candidate for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.'",
"notes": "age anchor: 8 pages when 6 years old"
},
{
"case_id": "gsm8k-train-sample-v1-0007",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "conditional_question",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "How many more boxes do they need if Francine has a total of 85 crayons?",
"current_refusal_reason": "no admissible candidate for question: 'How many more boxes do they need if Francine has a total of 85 crayons?'",
"notes": "multi-entity stmt now parses; question has if-clause"
},
{
"case_id": "gsm8k-train-sample-v1-0008",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "conditional_question",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?",
"current_refusal_reason": "no admissible candidate for question: 'If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?'",
"notes": "context filler now parses; question has if-clause"
},
{
"case_id": "gsm8k-train-sample-v1-0009",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_comparative",
"current_primary_barrier": "conditional_question",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "If Jen has 150 ducks, how many total birds does she have?",
"current_refusal_reason": "no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'",
"notes": "if-clause question: If Jen has 150 ducks"
},
{
"case_id": "gsm8k-train-sample-v1-0010",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_statement",
"current_primary_barrier": "compound_statement",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Yun had 20 paperclips initially, but then lost 12.",
"current_refusal_reason": "no admissible candidate for statement: 'Yun had 20 paperclips initially, but then lost 12.'",
"notes": "two ops in one sentence: had X but lost Y"
},
{
"case_id": "gsm8k-train-sample-v1-0011",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "rate_price",
"current_primary_barrier": "rate_price",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Alexa has a lemonade stand where she sells lemonade for $2 for one cup.",
"current_refusal_reason": "no admissible candidate for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.'",
"notes": "$2 per cup in relative clause"
},
{
"case_id": "gsm8k-train-sample-v1-0012",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_statement",
"current_primary_barrier": "compound_statement",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "He put all of them in his aquarium but his fish ate half of them.",
"current_refusal_reason": "no admissible candidate for statement: 'He put all of them in his aquarium but his fish ate half of them.'",
"notes": "ate half = compound + fraction"
},
{
"case_id": "gsm8k-train-sample-v1-0013",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "compound_statement",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.",
"current_refusal_reason": "no admissible candidate for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.'",
"notes": "10 one-hour videos each day = nested quantities"
},
{
"case_id": "gsm8k-train-sample-v1-0014",
"previous_outcome": "refused",
"current_outcome": "admitted",
"previous_primary_barrier": "capacity_rate",
"current_primary_barrier": "admitted",
"barrier_shifted": true,
"new_admission": true,
"current_first_refusal": null,
"current_refusal_reason": null,
"notes": "ADMITTED via S.1 capacity short-circuit"
},
{
"case_id": "gsm8k-train-sample-v1-0015",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_multi_event",
"current_primary_barrier": "compound_multi_event",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.",
"current_refusal_reason": "no admissible candidate for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.'",
"notes": "subway+train+bike multi-event in one sentence"
},
{
"case_id": "gsm8k-train-sample-v1-0016",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_statement",
"current_primary_barrier": "compound_statement",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.",
"current_refusal_reason": "no admissible candidate for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\"",
"notes": "2 more than 5 miles AND 3 less than 17 signs"
},
{
"case_id": "gsm8k-train-sample-v1-0017",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "conditional_branch",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "He\u2019s charging $50.00 per day or $500.00 for 14 days.",
"current_refusal_reason": "no admissible candidate for statement: 'He\u2019s charging $50.00 per day or $500.00 for 14 days.'",
"notes": "context filler now parses; or-choice pricing blocks"
},
{
"case_id": "gsm8k-train-sample-v1-0018",
"previous_outcome": "refused",
"current_outcome": "admitted",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "admitted",
"barrier_shifted": true,
"new_admission": true,
"current_first_refusal": null,
"current_refusal_reason": null,
"notes": "ADMITTED via S.2 capacity short-circuit"
},
{
"case_id": "gsm8k-train-sample-v1-0019",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "rate_price",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "The dog ends up having health problems and this requires 3 vet appointments, which cost $400 each.",
"current_refusal_reason": "no admissible candidate for statement: 'The dog ends up having health problems and this requires 3 vet appointments, which cost $400 each.'",
"notes": "context filler now parses; 3 appts at $400 each"
},
{
"case_id": "gsm8k-train-sample-v1-0020",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "multi_entity_initial",
"current_primary_barrier": "multi_entity_initial",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Two puppies, two kittens, and three parakeets were for sale at the pet shop.",
"current_refusal_reason": "no admissible candidate for statement: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.'",
"notes": "multi-entity initial: 2+2+3 animals in one stmt"
},
{
"case_id": "gsm8k-train-sample-v1-0021",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "distributive_multiply",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "He bench presses 15 pounds for 10 reps and does 3 sets.",
"current_refusal_reason": "no admissible candidate for statement: 'He bench presses 15 pounds for 10 reps and does 3 sets.'",
"notes": "context filler now parses; 15lb x 10rep x 3set"
},
{
"case_id": "gsm8k-train-sample-v1-0022",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "context_filler",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.",
"current_refusal_reason": "no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.'",
"notes": "context filler still blocks: earning in rel clause"
},
{
"case_id": "gsm8k-train-sample-v1-0023",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "multi_step_complex",
"current_primary_barrier": "novel_initial_verb",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Nicole collected 400 Pokemon cards.",
"current_refusal_reason": "no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'",
"notes": "collected not in verb set"
},
{
"case_id": "gsm8k-train-sample-v1-0024",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "multi_day_accumulation",
"current_primary_barrier": "multi_day_accumulation",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.",
"current_refusal_reason": "no admissible candidate for statement: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.'",
"notes": "4 values across Mon-Thu in one sentence"
},
{
"case_id": "gsm8k-train-sample-v1-0025",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "distributive_multiply",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Lilibeth fills 6 baskets where each basket holds 50 strawberries.",
"current_refusal_reason": "no admissible candidate for statement: 'Lilibeth fills 6 baskets where each basket holds 50 strawberries.'",
"notes": "context filler now parses; 6 baskets x 50 each"
},
{
"case_id": "gsm8k-train-sample-v1-0026",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "distributive_each_actor",
"current_primary_barrier": "distributive_each_actor",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Aaron and his brother Carson each saved up $40 to go to dinner.",
"current_refusal_reason": "no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'",
"notes": "each actor: Aaron and Carson each saved $40"
},
{
"case_id": "gsm8k-train-sample-v1-0027",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "multi_attribute_accumulation",
"current_primary_barrier": "multi_attribute_accumulation",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Malcolm has 240 followers on Instagram and 500 followers on Facebook.",
"current_refusal_reason": "no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.'",
"notes": "240 Instagram + 500 Facebook in one stmt"
},
{
"case_id": "gsm8k-train-sample-v1-0028",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "novel_initial_form",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "It cost $100,000 to open initially.",
"current_refusal_reason": "no admissible candidate for statement: 'It cost $100,000 to open initially.'",
"notes": "context filler now parses; It cost $100k = novel form"
},
{
"case_id": "gsm8k-train-sample-v1-0029",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "compound_comparative",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "The cost of the keyboard was three times greater than the cost of the mouse.",
"current_refusal_reason": "no admissible candidate for statement: 'The cost of the keyboard was three times greater than the cost of the mouse.'",
"notes": "context filler now parses; 3x comparative blocks"
},
{
"case_id": "gsm8k-train-sample-v1-0030",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "novel_initial_form",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "It is a 2-hour drive each way.",
"current_refusal_reason": "no admissible candidate for statement: 'It is a 2-hour drive each way.'",
"notes": "context filler now parses; It is a 2-hour drive"
},
{
"case_id": "gsm8k-train-sample-v1-0031",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "context_filler",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Jeremie wants to go to an amusement park with 3 friends at the end of summer.",
"current_refusal_reason": "no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.'",
"notes": "intent sentence: wants to go = context filler"
},
{
"case_id": "gsm8k-train-sample-v1-0032",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "compound_statement",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "He draws and colors 10 pictures.",
"current_refusal_reason": "no admissible candidate for statement: 'He draws and colors 10 pictures.'",
"notes": "context filler now parses; draws AND colors"
},
{
"case_id": "gsm8k-train-sample-v1-0033",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "compound_statement",
"current_primary_barrier": "compound_statement",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Rachel is 12 years old, and her grandfather is 7 times her age.",
"current_refusal_reason": "no admissible candidate for statement: 'Rachel is 12 years old, and her grandfather is 7 times her age.'",
"notes": "12 years old AND 7 times her age in one stmt"
},
{
"case_id": "gsm8k-train-sample-v1-0034",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "capacity_rate",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "He can run 40 yards within 5 seconds.",
"current_refusal_reason": "no admissible candidate for statement: 'He can run 40 yards within 5 seconds.'",
"notes": "context filler now parses; within vs in preposition"
},
{
"case_id": "gsm8k-train-sample-v1-0035",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "coreference_pronoun",
"current_primary_barrier": "complex_question",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "How many more apples would Martha need to give away to be left with only 4 of them?",
"current_refusal_reason": "no admissible candidate for question: 'How many more apples would Martha need to give away to be left with only 4 of them?'",
"notes": "pronoun now resolves; question too complex (modal)"
},
{
"case_id": "gsm8k-train-sample-v1-0036",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "compound_comparative",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "She studied for 2 hours on Wednesday and three times as long on Thursday.",
"current_refusal_reason": "no admissible candidate for statement: 'She studied for 2 hours on Wednesday and three times as long on Thursday.'",
"notes": "context filler now parses; 3x comparative blocks"
},
{
"case_id": "gsm8k-train-sample-v1-0037",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "goal_statement",
"current_primary_barrier": "goal_statement",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Michael wants to lose 10 pounds by June.",
"current_refusal_reason": "no admissible candidate for statement: 'Michael wants to lose 10 pounds by June.'",
"notes": "goal statement: wants to lose 10 pounds"
},
{
"case_id": "gsm8k-train-sample-v1-0038",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "novel_initial_form",
"current_primary_barrier": "novel_initial_form",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "In a building, there are a hundred ladies on the first-floor studying.",
"current_refusal_reason": "no admissible candidate for statement: 'In a building, there are a hundred ladies on the first-floor studying.'",
"notes": "there are a hundred = existential + spelled number"
},
{
"case_id": "gsm8k-train-sample-v1-0039",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "novel_initial_verb",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Orlando gained 5 pounds.",
"current_refusal_reason": "no admissible candidate for statement: 'Orlando gained 5 pounds.'",
"notes": "context filler now parses; gained not in verb set"
},
{
"case_id": "gsm8k-train-sample-v1-0040",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "multi_entity_initial",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat.",
"current_refusal_reason": "no admissible candidate for statement: 'He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat.'",
"notes": "context filler now parses; 2+5+7+3+1 multi-entity"
},
{
"case_id": "gsm8k-train-sample-v1-0041",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "fraction_operand",
"current_primary_barrier": "fraction_operand",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "The guests eat all of 1 pan, and 75% of the 2nd pan.",
"current_refusal_reason": "no admissible candidate for statement: 'The guests eat all of 1 pan, and 75% of the 2nd pan.'",
"notes": "all of 1 pan + 75% of 2nd = fraction operand"
},
{
"case_id": "gsm8k-train-sample-v1-0042",
"previous_outcome": "refused",
"current_outcome": "admitted",
"previous_primary_barrier": "distributive_multiply",
"current_primary_barrier": "admitted",
"barrier_shifted": true,
"new_admission": true,
"current_first_refusal": null,
"current_refusal_reason": null,
"notes": "ADMITTED via S.2 conditional-op question"
},
{
"case_id": "gsm8k-train-sample-v1-0043",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "compound_comparative",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Her mother gave her an additional $4, and her father twice as much as her mother.",
"current_refusal_reason": "no admissible candidate for statement: 'Her mother gave her an additional $4, and her father twice as much as her mother.'",
"notes": "context filler now parses; twice as much comparative"
},
{
"case_id": "gsm8k-train-sample-v1-0044",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "percentage_rate",
"current_primary_barrier": "percentage_rate",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "John invests in a bank and gets 10% simple interest.",
"current_refusal_reason": "no admissible candidate for statement: 'John invests in a bank and gets 10% simple interest.'",
"notes": "10% simple interest = percentage rate"
},
{
"case_id": "gsm8k-train-sample-v1-0045",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "novel_initial_form",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Each survey has 10 questions.",
"current_refusal_reason": "no admissible candidate for statement: 'Each survey has 10 questions.'",
"notes": "context filler now parses; Each survey has = novel"
},
{
"case_id": "gsm8k-train-sample-v1-0046",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "novel_initial_form",
"current_primary_barrier": "novel_initial_form",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "A school has 100 students.",
"current_refusal_reason": "no admissible candidate for statement: 'A school has 100 students.'",
"notes": "A school has 100 = novel initial form (indef article)"
},
{
"case_id": "gsm8k-train-sample-v1-0047",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "novel_initial_verb",
"current_primary_barrier": "novel_initial_verb",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "John bakes 12 coconut macaroons, each weighing 5 ounces.",
"current_refusal_reason": "no admissible candidate for statement: 'John bakes 12 coconut macaroons, each weighing 5 ounces.'",
"notes": "bakes not in verb set + embedded per-unit"
},
{
"case_id": "gsm8k-train-sample-v1-0048",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "temporal_frequency",
"barrier_shifted": true,
"new_admission": false,
"current_first_refusal": "Every week, he gets 6 cards.",
"current_refusal_reason": "no admissible candidate for statement: 'Every week, he gets 6 cards.'",
"notes": "context filler now parses; every week frequency"
},
{
"case_id": "gsm8k-train-sample-v1-0049",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "context_filler",
"current_primary_barrier": "context_filler",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Malcolm is trying to find the fastest walk to school and is currently comparing two routes.",
"current_refusal_reason": "no admissible candidate for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.'",
"notes": "context filler: trying to find = intent"
},
{
"case_id": "gsm8k-train-sample-v1-0050",
"previous_outcome": "refused",
"current_outcome": "refused",
"previous_primary_barrier": "temporal_frequency",
"current_primary_barrier": "temporal_frequency",
"barrier_shifted": false,
"new_admission": false,
"current_first_refusal": "Mark does a gig every other day for 2 weeks.",
"current_refusal_reason": "no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'",
"notes": "every other day for 2 weeks = temporal frequency"
}
]
}

424
evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v2.json Normal file
View file

@ -0,0 +1,424 @@
{
"schema_version": 2,
"adr": "0136.S.2-post-rescan",
"description": "Updated refusal taxonomy reflecting post-S.1/S.2 parser. Supersedes v1 (ADR-0136.S.0).",
"summary": {
"total_cases": 50,
"admitted": 3,
"refused": 47,
"primary_barrier_counts": {
"compound_statement": 6,
"novel_initial_form": 5,
"novel_initial_verb": 4,
"admitted": 3,
"compound_comparative": 3,
"conditional_question": 3,
"context_filler": 3,
"fraction_operand": 3,
"distributive_multiply": 2,
"multi_entity_initial": 2,
"rate_price": 2,
"temporal_frequency": 2,
"capacity_rate": 1,
"complex_question": 1,
"compound_multi_event": 1,
"conditional_branch": 1,
"distributive_each_actor": 1,
"goal_statement": 1,
"multi_attribute_accumulation": 1,
"multi_day_accumulation": 1,
"partition_divide": 1,
"percentage_rate": 1,
"rate_earnings": 1,
"temporal_age_anchor": 1
}
},
"per_case": [
{
"case_id": "gsm8k-train-sample-v1-0001",
"primary_barrier": "rate_earnings",
"secondary_barriers": [
"conditional_branch",
"multi_statement"
],
"note": "multi-stmt earnings; short-circuit needs single stmt+q"
},
{
"case_id": "gsm8k-train-sample-v1-0002",
"primary_barrier": "partition_divide",
"secondary_barriers": [
"fraction_operand",
"coreference_pronoun"
],
"note": "splits into 25-foot sections = partition op"
},
{
"case_id": "gsm8k-train-sample-v1-0003",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"distributive_multiply",
"rate_price"
],
"note": "context filler now parses; donated not in verb set"
},
{
"case_id": "gsm8k-train-sample-v1-0004",
"primary_barrier": "fraction_operand",
"secondary_barriers": [
"conditional_question"
],
"note": "indef qty now parses; fraction ops (half, 1/4) block"
},
{
"case_id": "gsm8k-train-sample-v1-0005",
"primary_barrier": "fraction_operand",
"secondary_barriers": [],
"note": "3/4 fraction operand in temporal decrease"
},
{
"case_id": "gsm8k-train-sample-v1-0006",
"primary_barrier": "temporal_age_anchor",
"secondary_barriers": [
"multi_step_complex"
],
"note": "age anchor: 8 pages when 6 years old"
},
{
"case_id": "gsm8k-train-sample-v1-0007",
"primary_barrier": "conditional_question",
"secondary_barriers": [
"multi_entity_initial"
],
"note": "multi-entity stmt now parses; question has if-clause"
},
{
"case_id": "gsm8k-train-sample-v1-0008",
"primary_barrier": "conditional_question",
"secondary_barriers": [
"distributive_multiply"
],
"note": "context filler now parses; question has if-clause"
},
{
"case_id": "gsm8k-train-sample-v1-0009",
"primary_barrier": "conditional_question",
"secondary_barriers": [
"compound_comparative"
],
"note": "if-clause question: If Jen has 150 ducks"
},
{
"case_id": "gsm8k-train-sample-v1-0010",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"fraction_operand"
],
"note": "two ops in one sentence: had X but lost Y"
},
{
"case_id": "gsm8k-train-sample-v1-0011",
"primary_barrier": "rate_price",
"secondary_barriers": [
"context_filler"
],
"note": "$2 per cup in relative clause"
},
{
"case_id": "gsm8k-train-sample-v1-0012",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"fraction_operand",
"coreference_pronoun"
],
"note": "ate half = compound + fraction"
},
{
"case_id": "gsm8k-train-sample-v1-0013",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"temporal_frequency"
],
"note": "10 one-hour videos each day = nested quantities"
},
{
"case_id": "gsm8k-train-sample-v1-0014",
"primary_barrier": "admitted",
"secondary_barriers": [],
"note": "ADMITTED via S.1 capacity short-circuit"
},
{
"case_id": "gsm8k-train-sample-v1-0015",
"primary_barrier": "compound_multi_event",
"secondary_barriers": [
"compound_comparative"
],
"note": "subway+train+bike multi-event in one sentence"
},
{
"case_id": "gsm8k-train-sample-v1-0016",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"rate_question"
],
"note": "2 more than 5 miles AND 3 less than 17 signs"
},
{
"case_id": "gsm8k-train-sample-v1-0017",
"primary_barrier": "conditional_branch",
"secondary_barriers": [
"rate_price"
],
"note": "context filler now parses; or-choice pricing blocks"
},
{
"case_id": "gsm8k-train-sample-v1-0018",
"primary_barrier": "admitted",
"secondary_barriers": [],
"note": "ADMITTED via S.2 capacity short-circuit"
},
{
"case_id": "gsm8k-train-sample-v1-0019",
"primary_barrier": "rate_price",
"secondary_barriers": [
"percentage_of",
"conditional_branch"
],
"note": "context filler now parses; 3 appts at $400 each"
},
{
"case_id": "gsm8k-train-sample-v1-0020",
"primary_barrier": "multi_entity_initial",
"secondary_barriers": [
"rate_comparative"
],
"note": "multi-entity initial: 2+2+3 animals in one stmt"
},
{
"case_id": "gsm8k-train-sample-v1-0021",
"primary_barrier": "distributive_multiply",
"secondary_barriers": [],
"note": "context filler now parses; 15lb x 10rep x 3set"
},
{
"case_id": "gsm8k-train-sample-v1-0022",
"primary_barrier": "context_filler",
"secondary_barriers": [
"rate_earnings",
"compound_comparative"
],
"note": "context filler still blocks: earning in rel clause"
},
{
"case_id": "gsm8k-train-sample-v1-0023",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"fraction_operand",
"distributive_divide"
],
"note": "collected not in verb set"
},
{
"case_id": "gsm8k-train-sample-v1-0024",
"primary_barrier": "multi_day_accumulation",
"secondary_barriers": [],
"note": "4 values across Mon-Thu in one sentence"
},
{
"case_id": "gsm8k-train-sample-v1-0025",
"primary_barrier": "distributive_multiply",
"secondary_barriers": [],
"note": "context filler now parses; 6 baskets x 50 each"
},
{
"case_id": "gsm8k-train-sample-v1-0026",
"primary_barrier": "distributive_each_actor",
"secondary_barriers": [
"fraction_operand"
],
"note": "each actor: Aaron and Carson each saved $40"
},
{
"case_id": "gsm8k-train-sample-v1-0027",
"primary_barrier": "multi_attribute_accumulation",
"secondary_barriers": [
"multi_step_complex"
],
"note": "240 Instagram + 500 Facebook in one stmt"
},
{
"case_id": "gsm8k-train-sample-v1-0028",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"rate_price",
"temporal_frequency"
],
"note": "context filler now parses; It cost $100k = novel form"
},
{
"case_id": "gsm8k-train-sample-v1-0029",
"primary_barrier": "compound_comparative",
"secondary_barriers": [
"rate_comparative"
],
"note": "context filler now parses; 3x comparative blocks"
},
{
"case_id": "gsm8k-train-sample-v1-0030",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"compound_comparative"
],
"note": "context filler now parses; It is a 2-hour drive"
},
{
"case_id": "gsm8k-train-sample-v1-0031",
"primary_barrier": "context_filler",
"secondary_barriers": [
"implicit_group_count"
],
"note": "intent sentence: wants to go = context filler"
},
{
"case_id": "gsm8k-train-sample-v1-0032",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"percentage_of"
],
"note": "context filler now parses; draws AND colors"
},
{
"case_id": "gsm8k-train-sample-v1-0033",
"primary_barrier": "compound_statement",
"secondary_barriers": [
"multi_step_complex"
],
"note": "12 years old AND 7 times her age in one stmt"
},
{
"case_id": "gsm8k-train-sample-v1-0034",
"primary_barrier": "capacity_rate",
"secondary_barriers": [
"percentage_of",
"capacity_rate"
],
"note": "context filler now parses; within vs in preposition"
},
{
"case_id": "gsm8k-train-sample-v1-0035",
"primary_barrier": "complex_question",
"secondary_barriers": [
"coreference_pronoun"
],
"note": "pronoun now resolves; question too complex (modal)"
},
{
"case_id": "gsm8k-train-sample-v1-0036",
"primary_barrier": "compound_comparative",
"secondary_barriers": [
"compound_comparative"
],
"note": "context filler now parses; 3x comparative blocks"
},
{
"case_id": "gsm8k-train-sample-v1-0037",
"primary_barrier": "goal_statement",
"secondary_barriers": [],
"note": "goal statement: wants to lose 10 pounds"
},
{
"case_id": "gsm8k-train-sample-v1-0038",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [],
"note": "there are a hundred = existential + spelled number"
},
{
"case_id": "gsm8k-train-sample-v1-0039",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"multi_step_complex"
],
"note": "context filler now parses; gained not in verb set"
},
{
"case_id": "gsm8k-train-sample-v1-0040",
"primary_barrier": "multi_entity_initial",
"secondary_barriers": [
"leg_count"
],
"note": "context filler now parses; 2+5+7+3+1 multi-entity"
},
{
"case_id": "gsm8k-train-sample-v1-0041",
"primary_barrier": "fraction_operand",
"secondary_barriers": [
"percentage_of"
],
"note": "all of 1 pan + 75% of 2nd = fraction operand"
},
{
"case_id": "gsm8k-train-sample-v1-0042",
"primary_barrier": "admitted",
"secondary_barriers": [],
"note": "ADMITTED via S.2 conditional-op question"
},
{
"case_id": "gsm8k-train-sample-v1-0043",
"primary_barrier": "compound_comparative",
"secondary_barriers": [
"rate_comparative"
],
"note": "context filler now parses; twice as much comparative"
},
{
"case_id": "gsm8k-train-sample-v1-0044",
"primary_barrier": "percentage_rate",
"secondary_barriers": [],
"note": "10% simple interest = percentage rate"
},
{
"case_id": "gsm8k-train-sample-v1-0045",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"rate_price"
],
"note": "context filler now parses; Each survey has = novel"
},
{
"case_id": "gsm8k-train-sample-v1-0046",
"primary_barrier": "novel_initial_form",
"secondary_barriers": [
"percentage_of"
],
"note": "A school has 100 = novel initial form (indef article)"
},
{
"case_id": "gsm8k-train-sample-v1-0047",
"primary_barrier": "novel_initial_verb",
"secondary_barriers": [
"embedded_per_unit"
],
"note": "bakes not in verb set + embedded per-unit"
},
{
"case_id": "gsm8k-train-sample-v1-0048",
"primary_barrier": "temporal_frequency",
"secondary_barriers": [],
"note": "context filler now parses; every week frequency"
},
{
"case_id": "gsm8k-train-sample-v1-0049",
"primary_barrier": "context_filler",
"secondary_barriers": [
"compound_multi_event",
"rate_comparative"
],
"note": "context filler: trying to find = intent"
},
{
"case_id": "gsm8k-train-sample-v1-0050",
"primary_barrier": "temporal_frequency",
"secondary_barriers": [],
"note": "every other day for 2 weeks = temporal frequency"
}
]
}

329
evals/gsm8k_math/train_sample/v1/rescan_v2.py Normal file
View file

@ -0,0 +1,329 @@
"""ADR-0136.S.2 post-rescan — barrier-shift ledger over 50-case GSM8K train sample.
Measurement-only. Runs ``parse_and_solve`` on every case, compares current
refusal behavior to the v1 taxonomy (ADR-0136.S.0), and writes two artifacts:
- ``refusal_rescan_v2.json`` — per-case barrier-shift ledger.
- ``refusal_taxonomy_v2.json`` — updated taxonomy reflecting post-S.1/S.2 parser.
Aborts on any ``wrong`` admission (answer ≠ expected).
"""
from __future__ import annotations
import json
import re
from pathlib import Path
from typing import Any
from generate.math_candidate_graph import parse_and_solve
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_V1_TAXONOMY_PATH = _HERE / "refusal_taxonomy.json"
_RESCAN_PATH = _HERE / "refusal_rescan_v2.json"
_TAXONOMY_V2_PATH = _HERE / "refusal_taxonomy_v2.json"
_FIRST_REFUSAL_RE = re.compile(
r"no admissible candidate for (?:statement|question): ['\"](.+?)['\"]$"
)
_CURRENT_BARRIERS: dict[str, str] = {
"gsm8k-train-sample-v1-0001": "rate_earnings",
"gsm8k-train-sample-v1-0002": "partition_divide",
"gsm8k-train-sample-v1-0003": "novel_initial_verb",
"gsm8k-train-sample-v1-0004": "fraction_operand",
"gsm8k-train-sample-v1-0005": "fraction_operand",
"gsm8k-train-sample-v1-0006": "temporal_age_anchor",
"gsm8k-train-sample-v1-0007": "conditional_question",
"gsm8k-train-sample-v1-0008": "conditional_question",
"gsm8k-train-sample-v1-0009": "conditional_question",
"gsm8k-train-sample-v1-0010": "compound_statement",
"gsm8k-train-sample-v1-0011": "rate_price",
"gsm8k-train-sample-v1-0012": "compound_statement",
"gsm8k-train-sample-v1-0013": "compound_statement",
"gsm8k-train-sample-v1-0014": "admitted",
"gsm8k-train-sample-v1-0015": "compound_multi_event",
"gsm8k-train-sample-v1-0016": "compound_statement",
"gsm8k-train-sample-v1-0017": "conditional_branch",
"gsm8k-train-sample-v1-0018": "admitted",
"gsm8k-train-sample-v1-0019": "rate_price",
"gsm8k-train-sample-v1-0020": "multi_entity_initial",
"gsm8k-train-sample-v1-0021": "distributive_multiply",
"gsm8k-train-sample-v1-0022": "context_filler",
"gsm8k-train-sample-v1-0023": "novel_initial_verb",
"gsm8k-train-sample-v1-0024": "multi_day_accumulation",
"gsm8k-train-sample-v1-0025": "distributive_multiply",
"gsm8k-train-sample-v1-0026": "distributive_each_actor",
"gsm8k-train-sample-v1-0027": "multi_attribute_accumulation",
"gsm8k-train-sample-v1-0028": "novel_initial_form",
"gsm8k-train-sample-v1-0029": "compound_comparative",
"gsm8k-train-sample-v1-0030": "novel_initial_form",
"gsm8k-train-sample-v1-0031": "context_filler",
"gsm8k-train-sample-v1-0032": "compound_statement",
"gsm8k-train-sample-v1-0033": "compound_statement",
"gsm8k-train-sample-v1-0034": "capacity_rate",
"gsm8k-train-sample-v1-0035": "complex_question",
"gsm8k-train-sample-v1-0036": "compound_comparative",
"gsm8k-train-sample-v1-0037": "goal_statement",
"gsm8k-train-sample-v1-0038": "novel_initial_form",
"gsm8k-train-sample-v1-0039": "novel_initial_verb",
"gsm8k-train-sample-v1-0040": "multi_entity_initial",
"gsm8k-train-sample-v1-0041": "fraction_operand",
"gsm8k-train-sample-v1-0042": "admitted",
"gsm8k-train-sample-v1-0043": "compound_comparative",
"gsm8k-train-sample-v1-0044": "percentage_rate",
"gsm8k-train-sample-v1-0045": "novel_initial_form",
"gsm8k-train-sample-v1-0046": "novel_initial_form",
"gsm8k-train-sample-v1-0047": "novel_initial_verb",
"gsm8k-train-sample-v1-0048": "temporal_frequency",
"gsm8k-train-sample-v1-0049": "context_filler",
"gsm8k-train-sample-v1-0050": "temporal_frequency",
}
_CURRENT_NOTES: dict[str, str] = {
"gsm8k-train-sample-v1-0001": "multi-stmt earnings; short-circuit needs single stmt+q",
"gsm8k-train-sample-v1-0002": "splits into 25-foot sections = partition op",
"gsm8k-train-sample-v1-0003": "context filler now parses; donated not in verb set",
"gsm8k-train-sample-v1-0004": "indef qty now parses; fraction ops (half, 1/4) block",
"gsm8k-train-sample-v1-0005": "3/4 fraction operand in temporal decrease",
"gsm8k-train-sample-v1-0006": "age anchor: 8 pages when 6 years old",
"gsm8k-train-sample-v1-0007": "multi-entity stmt now parses; question has if-clause",
"gsm8k-train-sample-v1-0008": "context filler now parses; question has if-clause",
"gsm8k-train-sample-v1-0009": "if-clause question: If Jen has 150 ducks",
"gsm8k-train-sample-v1-0010": "two ops in one sentence: had X but lost Y",
"gsm8k-train-sample-v1-0011": "$2 per cup in relative clause",
"gsm8k-train-sample-v1-0012": "ate half = compound + fraction",
"gsm8k-train-sample-v1-0013": "10 one-hour videos each day = nested quantities",
"gsm8k-train-sample-v1-0014": "ADMITTED via S.1 capacity short-circuit",
"gsm8k-train-sample-v1-0015": "subway+train+bike multi-event in one sentence",
"gsm8k-train-sample-v1-0016": "2 more than 5 miles AND 3 less than 17 signs",
"gsm8k-train-sample-v1-0017": "context filler now parses; or-choice pricing blocks",
"gsm8k-train-sample-v1-0018": "ADMITTED via S.2 capacity short-circuit",
"gsm8k-train-sample-v1-0019": "context filler now parses; 3 appts at $400 each",
"gsm8k-train-sample-v1-0020": "multi-entity initial: 2+2+3 animals in one stmt",
"gsm8k-train-sample-v1-0021": "context filler now parses; 15lb x 10rep x 3set",
"gsm8k-train-sample-v1-0022": "context filler still blocks: earning in rel clause",
"gsm8k-train-sample-v1-0023": "collected not in verb set",
"gsm8k-train-sample-v1-0024": "4 values across Mon-Thu in one sentence",
"gsm8k-train-sample-v1-0025": "context filler now parses; 6 baskets x 50 each",
"gsm8k-train-sample-v1-0026": "each actor: Aaron and Carson each saved $40",
"gsm8k-train-sample-v1-0027": "240 Instagram + 500 Facebook in one stmt",
"gsm8k-train-sample-v1-0028": "context filler now parses; It cost $100k = novel form",
"gsm8k-train-sample-v1-0029": "context filler now parses; 3x comparative blocks",
"gsm8k-train-sample-v1-0030": "context filler now parses; It is a 2-hour drive",
"gsm8k-train-sample-v1-0031": "intent sentence: wants to go = context filler",
"gsm8k-train-sample-v1-0032": "context filler now parses; draws AND colors",
"gsm8k-train-sample-v1-0033": "12 years old AND 7 times her age in one stmt",
"gsm8k-train-sample-v1-0034": "context filler now parses; within vs in preposition",
"gsm8k-train-sample-v1-0035": "pronoun now resolves; question too complex (modal)",
"gsm8k-train-sample-v1-0036": "context filler now parses; 3x comparative blocks",
"gsm8k-train-sample-v1-0037": "goal statement: wants to lose 10 pounds",
"gsm8k-train-sample-v1-0038": "there are a hundred = existential + spelled number",
"gsm8k-train-sample-v1-0039": "context filler now parses; gained not in verb set",
"gsm8k-train-sample-v1-0040": "context filler now parses; 2+5+7+3+1 multi-entity",
"gsm8k-train-sample-v1-0041": "all of 1 pan + 75% of 2nd = fraction operand",
"gsm8k-train-sample-v1-0042": "ADMITTED via S.2 conditional-op question",
"gsm8k-train-sample-v1-0043": "context filler now parses; twice as much comparative",
"gsm8k-train-sample-v1-0044": "10% simple interest = percentage rate",
"gsm8k-train-sample-v1-0045": "context filler now parses; Each survey has = novel",
"gsm8k-train-sample-v1-0046": "A school has 100 = novel initial form (indef article)",
"gsm8k-train-sample-v1-0047": "bakes not in verb set + embedded per-unit",
"gsm8k-train-sample-v1-0048": "context filler now parses; every week frequency",
"gsm8k-train-sample-v1-0049": "context filler: trying to find = intent",
"gsm8k-train-sample-v1-0050": "every other day for 2 weeks = temporal frequency",
}
_CURRENT_SECONDARY: dict[str, list[str]] = {
"gsm8k-train-sample-v1-0001": ["conditional_branch", "multi_statement"],
"gsm8k-train-sample-v1-0002": ["fraction_operand", "coreference_pronoun"],
"gsm8k-train-sample-v1-0003": ["distributive_multiply", "rate_price"],
"gsm8k-train-sample-v1-0004": ["conditional_question"],
"gsm8k-train-sample-v1-0005": [],
"gsm8k-train-sample-v1-0006": ["multi_step_complex"],
"gsm8k-train-sample-v1-0007": ["multi_entity_initial"],
"gsm8k-train-sample-v1-0008": ["distributive_multiply"],
"gsm8k-train-sample-v1-0009": ["compound_comparative"],
"gsm8k-train-sample-v1-0010": ["fraction_operand"],
"gsm8k-train-sample-v1-0011": ["context_filler"],
"gsm8k-train-sample-v1-0012": ["fraction_operand", "coreference_pronoun"],
"gsm8k-train-sample-v1-0013": ["temporal_frequency"],
"gsm8k-train-sample-v1-0014": [],
"gsm8k-train-sample-v1-0015": ["compound_comparative"],
"gsm8k-train-sample-v1-0016": ["rate_question"],
"gsm8k-train-sample-v1-0017": ["rate_price"],
"gsm8k-train-sample-v1-0018": [],
"gsm8k-train-sample-v1-0019": ["percentage_of", "conditional_branch"],
"gsm8k-train-sample-v1-0020": ["rate_comparative"],
"gsm8k-train-sample-v1-0021": [],
"gsm8k-train-sample-v1-0022": ["rate_earnings", "compound_comparative"],
"gsm8k-train-sample-v1-0023": ["fraction_operand", "distributive_divide"],
"gsm8k-train-sample-v1-0024": [],
"gsm8k-train-sample-v1-0025": [],
"gsm8k-train-sample-v1-0026": ["fraction_operand"],
"gsm8k-train-sample-v1-0027": ["multi_step_complex"],
"gsm8k-train-sample-v1-0028": ["rate_price", "temporal_frequency"],
"gsm8k-train-sample-v1-0029": ["rate_comparative"],
"gsm8k-train-sample-v1-0030": ["compound_comparative"],
"gsm8k-train-sample-v1-0031": ["implicit_group_count"],
"gsm8k-train-sample-v1-0032": ["percentage_of"],
"gsm8k-train-sample-v1-0033": ["multi_step_complex"],
"gsm8k-train-sample-v1-0034": ["percentage_of", "capacity_rate"],
"gsm8k-train-sample-v1-0035": ["coreference_pronoun"],
"gsm8k-train-sample-v1-0036": ["compound_comparative"],
"gsm8k-train-sample-v1-0037": [],
"gsm8k-train-sample-v1-0038": [],
"gsm8k-train-sample-v1-0039": ["multi_step_complex"],
"gsm8k-train-sample-v1-0040": ["leg_count"],
"gsm8k-train-sample-v1-0041": ["percentage_of"],
"gsm8k-train-sample-v1-0042": [],
"gsm8k-train-sample-v1-0043": ["rate_comparative"],
"gsm8k-train-sample-v1-0044": [],
"gsm8k-train-sample-v1-0045": ["rate_price"],
"gsm8k-train-sample-v1-0046": ["percentage_of"],
"gsm8k-train-sample-v1-0047": ["embedded_per_unit"],
"gsm8k-train-sample-v1-0048": [],
"gsm8k-train-sample-v1-0049": ["compound_multi_event", "rate_comparative"],
"gsm8k-train-sample-v1-0050": [],
}
def _extract_first_refusal(reason: str | None) -> str | None:
if not reason:
return None
m = _FIRST_REFUSAL_RE.search(reason)
return m.group(1) if m else reason
def _load_v1_taxonomy() -> dict[str, dict[str, Any]]:
data = json.loads(_V1_TAXONOMY_PATH.read_text(encoding="utf-8"))
return {entry["case_id"]: entry for entry in data["per_case"]}
def build_rescan() -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
cases = [
json.loads(line)
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines()
if line.strip()
]
assert len(cases) == 50, f"Expected 50 cases, got {len(cases)}"
v1_tax = _load_v1_taxonomy()
rescan_records: list[dict[str, Any]] = []
taxonomy_records: list[dict[str, Any]] = []
for case in cases:
cid = case["case_id"]
r = parse_and_solve(case["question"])
current_outcome = "admitted" if r.answer is not None else "refused"
if r.answer is not None:
assert r.answer == case["answer_numeric"], (
f"WRONG: {cid} answer={r.answer} expected={case['answer_numeric']}"
)
v1_entry = v1_tax.get(cid, {})
previous_barrier = v1_entry.get("primary_barrier", "unknown")
previous_outcome = "refused"
current_barrier = _CURRENT_BARRIERS[cid]
barrier_shifted = previous_barrier != current_barrier
new_admission = current_outcome == "admitted" and previous_outcome == "refused"
rescan_records.append({
"case_id": cid,
"previous_outcome": previous_outcome,
"current_outcome": current_outcome,
"previous_primary_barrier": previous_barrier,
"current_primary_barrier": current_barrier,
"barrier_shifted": barrier_shifted,
"new_admission": new_admission,
"current_first_refusal": _extract_first_refusal(r.refusal_reason),
"current_refusal_reason": r.refusal_reason,
"notes": _CURRENT_NOTES.get(cid, ""),
})
taxonomy_records.append({
"case_id": cid,
"primary_barrier": current_barrier,
"secondary_barriers": _CURRENT_SECONDARY.get(cid, []),
"note": _CURRENT_NOTES.get(cid, ""),
})
return rescan_records, taxonomy_records
def _barrier_counts(records: list[dict[str, Any]]) -> dict[str, int]:
counts: dict[str, int] = {}
for r in records:
b = r["primary_barrier"]
counts[b] = counts.get(b, 0) + 1
return dict(sorted(counts.items(), key=lambda x: (-x[1], x[0])))
def write_rescan(
rescan_records: list[dict[str, Any]],
taxonomy_records: list[dict[str, Any]],
) -> None:
admitted = sum(1 for r in rescan_records if r["current_outcome"] == "admitted")
shifted = sum(1 for r in rescan_records if r["barrier_shifted"])
rescan_out = {
"schema_version": 2,
"adr": "0136.S.2-post-rescan",
"description": (
"Barrier-shift ledger over 50 GSM8K train-sample cases, "
"comparing v1 taxonomy (ADR-0136.S.0) to current parser "
"behavior post-S.1/S.2."
),
"summary": {
"total_cases": 50,
"admitted": admitted,
"refused": 50 - admitted,
"wrong": 0,
"barrier_shifted_count": shifted,
"new_admissions": [
r["case_id"] for r in rescan_records if r["new_admission"]
],
},
"per_case": rescan_records,
}
_RESCAN_PATH.write_text(
json.dumps(rescan_out, indent=2, sort_keys=False) + "\n",
encoding="utf-8",
)
taxonomy_out = {
"schema_version": 2,
"adr": "0136.S.2-post-rescan",
"description": (
"Updated refusal taxonomy reflecting post-S.1/S.2 parser. "
"Supersedes v1 (ADR-0136.S.0)."
),
"summary": {
"total_cases": 50,
"admitted": admitted,
"refused": 50 - admitted,
"primary_barrier_counts": _barrier_counts(taxonomy_records),
},
"per_case": taxonomy_records,
}
_TAXONOMY_V2_PATH.write_text(
json.dumps(taxonomy_out, indent=2, sort_keys=False) + "\n",
encoding="utf-8",
)
def main() -> int:
rescan, taxonomy = build_rescan()
write_rescan(rescan, taxonomy)
admitted = sum(1 for r in rescan if r["current_outcome"] == "admitted")
shifted = sum(1 for r in rescan if r["barrier_shifted"])
print(
f"ADR-0136.S.2 rescan: admitted={admitted}/50, wrong=0, "
f"barrier_shifted={shifted}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

102
tests/test_rescan_v2_invariants.py Normal file
View file

@ -0,0 +1,102 @@
"""ADR-0136.S.2-post-rescan — invariant tests for barrier-shift ledger.
Pins: wrong==0, admission regression set, determinism, taxonomy completeness,
and closed barrier vocabulary.
"""
from __future__ import annotations
import json
import pytest
from evals.gsm8k_math.train_sample.v1.rescan_v2 import build_rescan
_REQUIRED_ADMISSIONS = {
"gsm8k-train-sample-v1-0014",
"gsm8k-train-sample-v1-0018",
"gsm8k-train-sample-v1-0042",
}
_CLOSED_BARRIER_ENUM = {
"admitted",
"capacity_rate",
"complex_question",
"compound_comparative",
"compound_multi_event",
"compound_statement",
"conditional_branch",
"conditional_question",
"context_filler",
"distributive_each_actor",
"distributive_multiply",
"fraction_operand",
"goal_statement",
"multi_attribute_accumulation",
"multi_day_accumulation",
"multi_entity_initial",
"novel_initial_form",
"novel_initial_verb",
"partition_divide",
"percentage_rate",
"rate_earnings",
"rate_price",
"temporal_age_anchor",
"temporal_frequency",
}
@pytest.fixture(scope="module")
def rescan_result() -> tuple[list[dict], list[dict]]:
return build_rescan()
class TestWrongIsZero:
def test_no_wrong_admissions(self, rescan_result: tuple) -> None:
rescan, _ = rescan_result
for r in rescan:
if r["current_outcome"] == "admitted":
assert r["current_refusal_reason"] is None or r["current_refusal_reason"] == ""
class TestAdmissionRegression:
def test_required_admissions_present(self, rescan_result: tuple) -> None:
rescan, _ = rescan_result
admitted = {
r["case_id"] for r in rescan if r["current_outcome"] == "admitted"
}
assert admitted >= _REQUIRED_ADMISSIONS, (
f"Missing required admissions: {_REQUIRED_ADMISSIONS - admitted}"
)
class TestDeterminism:
def test_rescan_byte_equal_across_runs(self) -> None:
r1, t1 = build_rescan()
r2, t2 = build_rescan()
s1 = json.dumps(r1, indent=2, sort_keys=True)
s2 = json.dumps(r2, indent=2, sort_keys=True)
assert s1 == s2, "rescan records not deterministic"
s3 = json.dumps(t1, indent=2, sort_keys=True)
s4 = json.dumps(t2, indent=2, sort_keys=True)
assert s3 == s4, "taxonomy records not deterministic"
class TestTaxonomyCompleteness:
def test_taxonomy_has_50_entries(self, rescan_result: tuple) -> None:
_, taxonomy = rescan_result
assert len(taxonomy) == 50
def test_all_barriers_in_closed_enum(self, rescan_result: tuple) -> None:
_, taxonomy = rescan_result
for entry in taxonomy:
assert entry["primary_barrier"] in _CLOSED_BARRIER_ENUM, (
f"{entry['case_id']}: barrier {entry['primary_barrier']!r} "
f"not in closed enum"
)
def test_rescan_barriers_match_taxonomy(self, rescan_result: tuple) -> None:
rescan, taxonomy = rescan_result
for r, t in zip(rescan, taxonomy):
assert r["case_id"] == t["case_id"]
assert r["current_primary_barrier"] == t["primary_barrier"]