684481910b
Measurement-only branch. Re-runs parse_and_solve on all 50 GSM8K train-sample cases against the current parser (post-S.1/S.2) and produces a barrier-shift ledger comparing v1 taxonomy to current behavior. Results: admitted=3/50 (0014, 0018, 0042), wrong=0, barrier_shifted=27/50. Context-filler dominance collapsed from 23→3 cases; compound_statement (6) and novel_initial_form (5) are now the largest buckets. Subsumption directive pinned: ADR-0137 SHALL re-derive all short-circuit admissions as (DeferredCandidate, evidence, BindingProof) triples.
159 lines
6.1 KiB
Markdown
159 lines
6.1 KiB
Markdown
# ADR-0136.S.2-post-rescan — Refusal Rescan v2: Barrier-Shift Ledger
|
||
|
||
**Status:** Accepted
|
||
**Parent:** ADR-0136 (Statement Layer Corridor)
|
||
**Date:** 2026-05-23
|
||
|
||
## Context
|
||
|
||
The v1 refusal taxonomy (ADR-0136.S.0) was written before S.1 and S.2 landed.
|
||
This rescan re-runs `parse_and_solve` on all 50 GSM8K train-sample cases against
|
||
the current parser (post-S.1 rate/event parsing, post-S.2 conditional-op
|
||
questions) and produces a barrier-shift ledger so ADR-0137 inherits measured
|
||
truth, not stale classification.
|
||
|
||
## Summary
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Total cases | 50 |
|
||
| Admitted (correct) | 3 |
|
||
| Wrong | 0 |
|
||
| Refused | 47 |
|
||
| Barrier shifted (v1 → v2) | 27 |
|
||
|
||
## Current Admissions
|
||
|
||
| Case | Answer | Short-circuit path | Landed in |
|
||
|---|---|---|---|
|
||
| gsm8k-0014 | 240.0 | S.1 capacity rate | ADR-0136.S.1 |
|
||
| gsm8k-0018 | 16.0 | S.2 capacity rate (context filler resolved) | ADR-0136.S.2 |
|
||
| gsm8k-0042 | 30.0 | S.2 conditional-op question | ADR-0136.S.2 |
|
||
|
||
## Current Primary Barrier Distribution
|
||
|
||
| Primary barrier | Count | Notes |
|
||
|---|---|---|
|
||
| compound_statement | 6 | Two operations in one sentence |
|
||
| novel_initial_form | 5 | Existential/impersonal/quantifier sentence shapes |
|
||
| novel_initial_verb | 4 | Verb not in any closed set (donated, collected, gained, bakes) |
|
||
| admitted | 3 | Successfully parsed and solved |
|
||
| compound_comparative | 3 | N times as much/greater comparisons |
|
||
| conditional_question | 3 | If-clause in question sentence |
|
||
| context_filler | 3 | Narrative scene-setter still blocking (down from 23) |
|
||
| fraction_operand | 3 | Half/quarter/3-4/75% operands |
|
||
| distributive_multiply | 2 | N bags × M items |
|
||
| multi_entity_initial | 2 | Multiple entities in one sentence |
|
||
| rate_price | 2 | Price embedded in relative clause |
|
||
| temporal_frequency | 2 | Every N days / every other day |
|
||
| capacity_rate | 1 | "within" preposition instead of "in" |
|
||
| complex_question | 1 | Modal + infinitive chain in question |
|
||
| compound_multi_event | 1 | Multiple transport legs in one sentence |
|
||
| conditional_branch | 1 | Or-choice pricing |
|
||
| distributive_each_actor | 1 | "each saved" = per-actor distribution |
|
||
| goal_statement | 1 | "wants to lose" = intent, not state |
|
||
| multi_attribute_accumulation | 1 | Multiple platform counts in one sentence |
|
||
| multi_day_accumulation | 1 | Values across Mon–Thu in one sentence |
|
||
| partition_divide | 1 | Splits into sections |
|
||
| percentage_rate | 1 | 10% simple interest |
|
||
| rate_earnings | 1 | Multi-statement earnings with conditional |
|
||
| temporal_age_anchor | 1 | Age-conditioned initial state |
|
||
|
||
## Barrier-Shifted Cases (v1 → v2)
|
||
|
||
27 cases changed primary barrier classification. The dominant pattern: 18 of 23
|
||
v1 `context_filler` cases now parse their scene-setting sentence, exposing the
|
||
next structural barrier. Only 3 `context_filler` cases remain blocking (0022,
|
||
0031, 0049 — all have rate/intent embedded in complex relative clauses).
|
||
|
||
| Case | v1 barrier | v2 barrier |
|
||
|---|---|---|
|
||
| 0003 | context_filler | novel_initial_verb |
|
||
| 0004 | indefinite_quantity | fraction_operand |
|
||
| 0005 | compound_statement | fraction_operand |
|
||
| 0007 | context_filler | conditional_question |
|
||
| 0008 | context_filler | conditional_question |
|
||
| 0009 | compound_comparative | conditional_question |
|
||
| 0013 | context_filler | compound_statement |
|
||
| 0014 | capacity_rate | **admitted** |
|
||
| 0017 | context_filler | conditional_branch |
|
||
| 0018 | context_filler | **admitted** |
|
||
| 0019 | context_filler | rate_price |
|
||
| 0021 | context_filler | distributive_multiply |
|
||
| 0023 | multi_step_complex | novel_initial_verb |
|
||
| 0025 | context_filler | distributive_multiply |
|
||
| 0028 | context_filler | novel_initial_form |
|
||
| 0029 | context_filler | compound_comparative |
|
||
| 0030 | context_filler | novel_initial_form |
|
||
| 0032 | context_filler | compound_statement |
|
||
| 0034 | context_filler | capacity_rate |
|
||
| 0035 | coreference_pronoun | complex_question |
|
||
| 0036 | context_filler | compound_comparative |
|
||
| 0039 | context_filler | novel_initial_verb |
|
||
| 0040 | context_filler | multi_entity_initial |
|
||
| 0042 | distributive_multiply | **admitted** |
|
||
| 0043 | context_filler | compound_comparative |
|
||
| 0045 | context_filler | novel_initial_form |
|
||
| 0048 | context_filler | temporal_frequency |
|
||
|
||
## Subsumption Directive
|
||
|
||
> S.1/S.2 short-circuit paths are tactical bridges. ADR-0137 SHALL subsume them:
|
||
> every current short-circuit admission must be re-derivable as a
|
||
> (DeferredCandidate, evidence, BindingProof) triple. Short-circuits are kept
|
||
> only as regression fixtures.
|
||
|
||
## ADR-0137 Probe Cases
|
||
|
||
The following cases are required in the ADR-0137 probe to verify subsumption
|
||
and test the next wave of unlocks:
|
||
|
||
1. **gsm8k-0014** — Bob shucks oysters (capacity rate, currently admitted via S.1)
|
||
2. **gsm8k-0018** — Xavier scores goals (capacity rate, currently admitted via S.2)
|
||
3. **gsm8k-0042** — Ella sells apples (conditional-op, currently admitted via S.2)
|
||
4. **gsm8k-0034** — Georgie runs yards (capacity rate, blocked by "within" preposition)
|
||
5. **gsm8k-0021** — John bench presses (distributive multiply, 15×10×3)
|
||
6. **gsm8k-0003** — Bookstore erasers (novel verb "donated", 48 boxes × 24 each)
|
||
7. **gsm8k-0029** — Mouse and keyboard (compound comparative, 3× cost ratio)
|
||
|
||
Cases 1–3 test subsumption (must remain admitted). Cases 4–7 are the
|
||
shallowest remaining barriers that ADR-0137's binding graph should unlock.
|
||
|
||
## Closed Barrier Vocabulary (v2)
|
||
|
||
The `primary_barrier` field in `refusal_taxonomy_v2.json` uses exactly this
|
||
enum (24 values):
|
||
|
||
```
|
||
admitted
|
||
capacity_rate
|
||
complex_question
|
||
compound_comparative
|
||
compound_multi_event
|
||
compound_statement
|
||
conditional_branch
|
||
conditional_question
|
||
context_filler
|
||
distributive_each_actor
|
||
distributive_multiply
|
||
fraction_operand
|
||
goal_statement
|
||
multi_attribute_accumulation
|
||
multi_day_accumulation
|
||
multi_entity_initial
|
||
novel_initial_form
|
||
novel_initial_verb
|
||
partition_divide
|
||
percentage_rate
|
||
rate_earnings
|
||
rate_price
|
||
temporal_age_anchor
|
||
temporal_frequency
|
||
```
|
||
|
||
## Evidence
|
||
|
||
- Rescan ledger: `evals/gsm8k_math/train_sample/v1/refusal_rescan_v2.json`
|
||
- Taxonomy v2: `evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v2.json`
|
||
- Script: `evals/gsm8k_math/train_sample/v1/rescan_v2.py`
|
||
- Tests: `tests/test_rescan_v2_invariants.py`
|