2342564883
S.4 extends initial-state parsing with two closed subject-slot widenings:
- Indefinite-article: `A <noun> has N <unit>` (gsm8k-0046 sentence 1)
- Prepositional-prefix existential: `In a <place>, there are N <unit>...`
(gsm8k-0038 sentence 1)
Design choice: sibling regexes (_INITIAL_HAS_INDEF_RE,
_INITIAL_THERE_ARE_PREFIX_RE) rather than widening the global _ENTITY
pattern — preserves existing behavior across all other initial-state
extractors (cascade-safety).
Per the S.x corridor discipline: no new short-circuit; new candidates
flow through extract_initial_candidates and the existing graph machinery.
No solver/graph/verifier changes.
Honest delta:
- Direct admissions: 0 (admission set unchanged at {0014, 0018, 0042})
- Barrier shifts: +2 (gsm8k-0038: novel_initial_form → compound_comparative;
gsm8k-0046: novel_initial_form → fraction_operand)
- wrong == 0 on every lane
Bundled with this PR for ledger currency:
1. tests/test_rescan_v3_invariants.py refactored to read frozen on-disk
v3 artifacts only (no more re-running build_rescan against live
parser). The previous design tied a historical snapshot to live code
and broke the moment any new phase landed.
2. rescan_v4.py + refusal_rescan_v4.json + refusal_taxonomy_v4.json +
tests/test_rescan_v4_invariants.py — the current live snapshot.
Shifts: exactly 2 (0038, 0046). Same pattern as v3.
Sonnet wrote: S.4 parser/axis-lane/tests/ADR.
Opus wrote: rescan_v4.py + v3 test refactor + bundling.
Files:
- generate/math_candidate_parser.py (+142 lines)
- evals/math_capability_axes/S4_novel_initial_form/v1/ (20-case lane)
- tests/test_adr_0136_S4_novel_initial_form.py (40 tests)
- docs/decisions/ADR-0136.S.4-novel-initial-form.md
- evals/gsm8k_math/train_sample/v1/{rescan_v4.py, *_v4.json}
- tests/test_rescan_v4_invariants.py (8 tests)
- tests/test_rescan_v3_invariants.py (refactored to artifact-only)
69 lines
2.3 KiB
Python
69 lines
2.3 KiB
Python
"""ADR-0136.S.3-post-rescan — invariant tests for refusal rescan v3.
|
|
|
|
v3 is a SNAPSHOT in time. These tests assert against the frozen on-disk
|
|
artifacts only — they do NOT re-run build_rescan(), because the live
|
|
parser drifts away from the v3 baseline as new phases (S.4, S.5, …)
|
|
land. The next snapshot (v4+) is asserted by its own invariant file.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
_HERE = Path(__file__).resolve().parent.parent
|
|
_RESCAN_V3 = _HERE / "evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json"
|
|
_TAXONOMY_V3 = _HERE / "evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v3.json"
|
|
|
|
|
|
def _load_rescan() -> dict:
|
|
return json.loads(_RESCAN_V3.read_text(encoding="utf-8"))
|
|
|
|
|
|
def _load_taxonomy() -> dict:
|
|
return json.loads(_TAXONOMY_V3.read_text(encoding="utf-8"))
|
|
|
|
|
|
def test_v3_snapshot_wrong_is_zero() -> None:
|
|
assert _load_rescan()["summary"]["wrong"] == 0
|
|
|
|
|
|
def test_v3_snapshot_admission_set() -> None:
|
|
rescan = _load_rescan()
|
|
admitted = {
|
|
c["case_id"] for c in rescan["per_case"] if c["current_outcome"] == "admitted"
|
|
}
|
|
assert admitted == {
|
|
"gsm8k-train-sample-v1-0014",
|
|
"gsm8k-train-sample-v1-0018",
|
|
"gsm8k-train-sample-v1-0042",
|
|
}
|
|
|
|
|
|
def test_v3_snapshot_exactly_one_shift_v2_to_v3() -> None:
|
|
rescan = _load_rescan()
|
|
assert rescan["summary"]["barrier_shifted_v2_to_v3"] == 1
|
|
shifted = [c["case_id"] for c in rescan["per_case"] if c["barrier_shifted"]]
|
|
assert shifted == ["gsm8k-train-sample-v1-0010"]
|
|
|
|
|
|
def test_v3_snapshot_gsm8k_0010_shifted_to_fraction_operand() -> None:
|
|
rescan = _load_rescan()
|
|
case = next(
|
|
c for c in rescan["per_case"]
|
|
if c["case_id"] == "gsm8k-train-sample-v1-0010"
|
|
)
|
|
assert case["previous_first_refusal"].startswith("Yun had 20 paperclips")
|
|
assert case["current_first_refusal"].startswith("Marion has 1/4")
|
|
assert case["previous_primary_barrier"] == "compound_statement"
|
|
assert case["current_primary_barrier"] == "fraction_operand"
|
|
|
|
|
|
def test_v3_snapshot_taxonomy_has_50_entries() -> None:
|
|
assert len(_load_taxonomy()["per_case"]) == 50
|
|
|
|
|
|
def test_v3_snapshot_adr_tag() -> None:
|
|
"""Identifies which ADR produced this snapshot."""
|
|
assert _load_rescan()["adr"] == "0136.S.3-post-rescan"
|
|
assert _load_taxonomy()["adr"] == "0136.S.3-post-rescan"
|