a7feda3c19
Re-runs parse_and_solve on the 50-case GSM8K train sample on current main (post-S.3) and compares to v2. Result: admitted=3/50 (unchanged), wrong=0, exactly 1 barrier shifted v2→v3. Shift: gsm8k-0010 (compound_statement → fraction_operand). S.3's _INIT_MUTATION_RE resolves "Yun had 20 paperclips initially, but then lost 12" to InitialPossession(Yun, 8, paperclips). First refusal moved to sentence 2: "Marion has 1/4 more than what Yun currently has, plus 7" — needs fraction-operand + coreference-quantity + comparative-additive arithmetic. Top blockers (v3): compound_statement 5 (was 6) novel_initial_form 5 (unchanged) fraction_operand 4 (was 3 — gsm8k-0010 moved here) novel_initial_verb 4 (unchanged) Artifacts: - evals/gsm8k_math/train_sample/v1/rescan_v3.py - evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json - evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json - docs/decisions/ADR-0136.S3-post-rescan.md - tests/test_rescan_v3_invariants.py (7 tests; determinism + admission set unchanged + exactly-one-shift + 0010-specific shift assertions)
4.3 KiB
ADR-0136.S.3-post-rescan — Refusal Rescan v3: Barrier-Shift Ledger
Status: Accepted Parent: ADR-0136 Previous rescan: ADR-0136.S2-post-rescan Date: 2026-05-23
Context
The v2 refusal taxonomy was written before S.3 (compound initial-mutation)
landed. S.3 added one closed-shape extractor for <Entity> had N <unit>, but then <verb> M and produced 0 direct admissions + 1 barrier shift
(gsm8k-0010). This rescan re-runs parse_and_solve on all 50 GSM8K
train-sample cases against current main (post-S.3) and refreshes the
ledger so any next phase inherits measured truth.
Summary
| Metric | v2 | v3 | Δ |
|---|---|---|---|
| Admitted (correct) | 3 | 3 | 0 |
| Wrong | 0 | 0 | 0 |
| Refused | 47 | 47 | 0 |
| Cases shifted v2 → v3 | — | 1 | — |
Current Admissions (unchanged from v2)
| Case | Answer | Path |
|---|---|---|
| gsm8k-0014 | 240.0 | S.1 capacity rate (short-circuit) |
| gsm8k-0018 | 16.0 | S.0 classifier + S.1 capacity (short-circuit) |
| gsm8k-0042 | 30.0 | S.2 conditional-op question (short-circuit) |
Barrier-Shifted Cases (v2 → v3)
| Case | v2 barrier | v3 barrier | Reason |
|---|---|---|---|
| gsm8k-0010 | compound_statement | fraction_operand | S.3's _INIT_MUTATION_RE resolves sentence 1 ("Yun had 20 paperclips initially, but then lost 12") to InitialPossession(Yun, 8, paperclips). Refuses on sentence 2: "Marion has 1/4 more than what Yun currently has, plus 7" — needs fraction-operand parsing + coreference to Yun's current quantity + comparative-additive arithmetic. |
S.3 produced exactly one barrier shift, as predicted by its brief
("0–1 new admissions; the real return is rescan v3 barrier-shifts").
The shape coverage worked as designed; the downstream barriers on the
other 5 compound_statement cases are structurally different and remain
deferred (see ADR-0136.S.3).
Current Primary Barrier Distribution (v3)
| Barrier | v2 | v3 | Δ |
|---|---|---|---|
| compound_statement | 6 | 5 | −1 |
| novel_initial_form | 5 | 5 | 0 |
| fraction_operand | 3 | 4 | +1 |
| novel_initial_verb | 4 | 4 | 0 |
| admitted | 3 | 3 | 0 |
| compound_comparative | 3 | 3 | 0 |
| conditional_question | 3 | 3 | 0 |
| context_filler | 3 | 3 | 0 |
| distributive_multiply | 2 | 2 | 0 |
| multi_entity_initial | 2 | 2 | 0 |
| rate_price | 2 | 2 | 0 |
| temporal_frequency | 2 | 2 | 0 |
| capacity_rate | 1 | 1 | 0 |
| complex_question | 1 | 1 | 0 |
| compound_multi_event | 1 | 1 | 0 |
| conditional_branch | 1 | 1 | 0 |
| distributive_each_actor | 1 | 1 | 0 |
| goal_statement | 1 | 1 | 0 |
| multi_attribute_accumulation | 1 | 1 | 0 |
| multi_day_accumulation | 1 | 1 | 0 |
| partition_divide | 1 | 1 | 0 |
| percentage_rate | 1 | 1 | 0 |
| rate_earnings | 1 | 1 | 0 |
| temporal_age_anchor | 1 | 1 | 0 |
compound_statement (5) and novel_initial_form (5) are now tied for
the largest blocker bucket. fraction_operand (4) is now in the top 4
— a side-effect of S.3 exposing the sentence-2 barrier on gsm8k-0010.
Next-Phase Candidates
The ledger does not dictate the next phase, but the largest tractable clusters are:
- novel_initial_form (5) — sentence shapes the parser doesn't ground
(
A school has 100 students,In a building, there are a hundred ladies, etc.). Likely a widening of_INITIAL_HAS_RE/_INITIAL_THERE_ARE_REwith the indefinite-article and impersonal- existential cases. Possibly the lowest-cost statement-layer extension. - novel_initial_verb (4) —
donated,collected,gained,bakesoutside current closed sets. Pure vocabulary extension; probably composes with the existing acquire-verb table. - compound_statement (5) — five heterogeneous shapes (fraction + coreference, nested temporal, comparative arithmetic, verb conjunction, age-multiplier). Per ADR-0136.S.3 each is its own capability and should not be bundled.
S.4 should target one of these clusters as a single closed shape, in the same one-PR / one-shape / honest-delta discipline as S.1, S.2, S.3.
Evidence
- Rescan ledger:
evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json - Taxonomy v3:
evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json - Script:
evals/gsm8k_math/train_sample/v1/rescan_v3.py - Tests:
tests/test_rescan_v3_invariants.py