Commit graph

1 commit

Author SHA1 Message Date
Shay
a7feda3c19
audit(ADR-0136.S.3): refusal rescan v3 — exactly 1 barrier shift (gsm8k-0010) (#208)
Re-runs parse_and_solve on the 50-case GSM8K train sample on current
main (post-S.3) and compares to v2. Result: admitted=3/50 (unchanged),
wrong=0, exactly 1 barrier shifted v2→v3.

Shift: gsm8k-0010 (compound_statement → fraction_operand). S.3's
_INIT_MUTATION_RE resolves "Yun had 20 paperclips initially, but then
lost 12" to InitialPossession(Yun, 8, paperclips). First refusal moved
to sentence 2: "Marion has 1/4 more than what Yun currently has, plus
7" — needs fraction-operand + coreference-quantity + comparative-additive
arithmetic.

Top blockers (v3):
  compound_statement   5  (was 6)
  novel_initial_form   5  (unchanged)
  fraction_operand     4  (was 3 — gsm8k-0010 moved here)
  novel_initial_verb   4  (unchanged)

Artifacts:
- evals/gsm8k_math/train_sample/v1/rescan_v3.py
- evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json
- evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json
- docs/decisions/ADR-0136.S3-post-rescan.md
- tests/test_rescan_v3_invariants.py (7 tests; determinism + admission
  set unchanged + exactly-one-shift + 0010-specific shift assertions)
2026-05-23 22:05:16 -07:00