a7feda3c19
Re-runs parse_and_solve on the 50-case GSM8K train sample on current main (post-S.3) and compares to v2. Result: admitted=3/50 (unchanged), wrong=0, exactly 1 barrier shifted v2→v3. Shift: gsm8k-0010 (compound_statement → fraction_operand). S.3's _INIT_MUTATION_RE resolves "Yun had 20 paperclips initially, but then lost 12" to InitialPossession(Yun, 8, paperclips). First refusal moved to sentence 2: "Marion has 1/4 more than what Yun currently has, plus 7" — needs fraction-operand + coreference-quantity + comparative-additive arithmetic. Top blockers (v3): compound_statement 5 (was 6) novel_initial_form 5 (unchanged) fraction_operand 4 (was 3 — gsm8k-0010 moved here) novel_initial_verb 4 (unchanged) Artifacts: - evals/gsm8k_math/train_sample/v1/rescan_v3.py - evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json - evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json - docs/decisions/ADR-0136.S3-post-rescan.md - tests/test_rescan_v3_invariants.py (7 tests; determinism + admission set unchanged + exactly-one-shift + 0010-specific shift assertions)
105 lines
4.3 KiB
Markdown
105 lines
4.3 KiB
Markdown
# ADR-0136.S.3-post-rescan — Refusal Rescan v3: Barrier-Shift Ledger
|
||
|
||
**Status:** Accepted
|
||
**Parent:** [ADR-0136](./ADR-0136-statement-layer-corridor.md)
|
||
**Previous rescan:** [ADR-0136.S2-post-rescan](./ADR-0136.S2-post-rescan.md)
|
||
**Date:** 2026-05-23
|
||
|
||
## Context
|
||
|
||
The v2 refusal taxonomy was written before S.3 (compound initial-mutation)
|
||
landed. S.3 added one closed-shape extractor for `<Entity> had N <unit>,
|
||
but then <verb> M` and produced 0 direct admissions + 1 barrier shift
|
||
(gsm8k-0010). This rescan re-runs `parse_and_solve` on all 50 GSM8K
|
||
train-sample cases against current main (post-S.3) and refreshes the
|
||
ledger so any next phase inherits measured truth.
|
||
|
||
## Summary
|
||
|
||
| Metric | v2 | v3 | Δ |
|
||
|---|---|---|---|
|
||
| Admitted (correct) | 3 | 3 | 0 |
|
||
| Wrong | 0 | 0 | 0 |
|
||
| Refused | 47 | 47 | 0 |
|
||
| Cases shifted v2 → v3 | — | 1 | — |
|
||
|
||
## Current Admissions (unchanged from v2)
|
||
|
||
| Case | Answer | Path |
|
||
|---|---|---|
|
||
| gsm8k-0014 | 240.0 | S.1 capacity rate (short-circuit) |
|
||
| gsm8k-0018 | 16.0 | S.0 classifier + S.1 capacity (short-circuit) |
|
||
| gsm8k-0042 | 30.0 | S.2 conditional-op question (short-circuit) |
|
||
|
||
## Barrier-Shifted Cases (v2 → v3)
|
||
|
||
| Case | v2 barrier | v3 barrier | Reason |
|
||
|---|---|---|---|
|
||
| gsm8k-0010 | compound_statement | fraction_operand | S.3's `_INIT_MUTATION_RE` resolves sentence 1 ("Yun had 20 paperclips initially, but then lost 12") to `InitialPossession(Yun, 8, paperclips)`. Refuses on sentence 2: "Marion has 1/4 more than what Yun currently has, plus 7" — needs fraction-operand parsing + coreference to Yun's current quantity + comparative-additive arithmetic. |
|
||
|
||
S.3 produced exactly one barrier shift, as predicted by its brief
|
||
("0–1 new admissions; the real return is rescan v3 barrier-shifts").
|
||
The shape coverage worked as designed; the downstream barriers on the
|
||
other 5 `compound_statement` cases are structurally different and remain
|
||
deferred (see ADR-0136.S.3).
|
||
|
||
## Current Primary Barrier Distribution (v3)
|
||
|
||
| Barrier | v2 | v3 | Δ |
|
||
|---|---|---|---|
|
||
| compound_statement | 6 | 5 | −1 |
|
||
| novel_initial_form | 5 | 5 | 0 |
|
||
| fraction_operand | 3 | 4 | +1 |
|
||
| novel_initial_verb | 4 | 4 | 0 |
|
||
| admitted | 3 | 3 | 0 |
|
||
| compound_comparative | 3 | 3 | 0 |
|
||
| conditional_question | 3 | 3 | 0 |
|
||
| context_filler | 3 | 3 | 0 |
|
||
| distributive_multiply | 2 | 2 | 0 |
|
||
| multi_entity_initial | 2 | 2 | 0 |
|
||
| rate_price | 2 | 2 | 0 |
|
||
| temporal_frequency | 2 | 2 | 0 |
|
||
| capacity_rate | 1 | 1 | 0 |
|
||
| complex_question | 1 | 1 | 0 |
|
||
| compound_multi_event | 1 | 1 | 0 |
|
||
| conditional_branch | 1 | 1 | 0 |
|
||
| distributive_each_actor | 1 | 1 | 0 |
|
||
| goal_statement | 1 | 1 | 0 |
|
||
| multi_attribute_accumulation | 1 | 1 | 0 |
|
||
| multi_day_accumulation | 1 | 1 | 0 |
|
||
| partition_divide | 1 | 1 | 0 |
|
||
| percentage_rate | 1 | 1 | 0 |
|
||
| rate_earnings | 1 | 1 | 0 |
|
||
| temporal_age_anchor | 1 | 1 | 0 |
|
||
|
||
`compound_statement` (5) and `novel_initial_form` (5) are now tied for
|
||
the largest blocker bucket. `fraction_operand` (4) is now in the top 4
|
||
— a side-effect of S.3 exposing the sentence-2 barrier on gsm8k-0010.
|
||
|
||
## Next-Phase Candidates
|
||
|
||
The ledger does not dictate the next phase, but the largest tractable
|
||
clusters are:
|
||
|
||
- **novel_initial_form (5)** — sentence shapes the parser doesn't ground
|
||
(`A school has 100 students`, `In a building, there are a hundred
|
||
ladies`, etc.). Likely a widening of `_INITIAL_HAS_RE` /
|
||
`_INITIAL_THERE_ARE_RE` with the indefinite-article and impersonal-
|
||
existential cases. Possibly the lowest-cost statement-layer extension.
|
||
- **novel_initial_verb (4)** — `donated`, `collected`, `gained`, `bakes`
|
||
outside current closed sets. Pure vocabulary extension; probably
|
||
composes with the existing acquire-verb table.
|
||
- **compound_statement (5)** — five heterogeneous shapes (fraction +
|
||
coreference, nested temporal, comparative arithmetic, verb conjunction,
|
||
age-multiplier). Per ADR-0136.S.3 each is its own capability and
|
||
should not be bundled.
|
||
|
||
S.4 should target one of these clusters as a single closed shape, in
|
||
the same one-PR / one-shape / honest-delta discipline as S.1, S.2, S.3.
|
||
|
||
## Evidence
|
||
|
||
- Rescan ledger: `evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json`
|
||
- Taxonomy v3: `evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json`
|
||
- Script: `evals/gsm8k_math/train_sample/v1/rescan_v3.py`
|
||
- Tests: `tests/test_rescan_v3_invariants.py`
|