2342564883
S.4 extends initial-state parsing with two closed subject-slot widenings:
- Indefinite-article: `A <noun> has N <unit>` (gsm8k-0046 sentence 1)
- Prepositional-prefix existential: `In a <place>, there are N <unit>...`
(gsm8k-0038 sentence 1)
Design choice: sibling regexes (_INITIAL_HAS_INDEF_RE,
_INITIAL_THERE_ARE_PREFIX_RE) rather than widening the global _ENTITY
pattern — preserves existing behavior across all other initial-state
extractors (cascade-safety).
Per the S.x corridor discipline: no new short-circuit; new candidates
flow through extract_initial_candidates and the existing graph machinery.
No solver/graph/verifier changes.
Honest delta:
- Direct admissions: 0 (admission set unchanged at {0014, 0018, 0042})
- Barrier shifts: +2 (gsm8k-0038: novel_initial_form → compound_comparative;
gsm8k-0046: novel_initial_form → fraction_operand)
- wrong == 0 on every lane
Bundled with this PR for ledger currency:
1. tests/test_rescan_v3_invariants.py refactored to read frozen on-disk
v3 artifacts only (no more re-running build_rescan against live
parser). The previous design tied a historical snapshot to live code
and broke the moment any new phase landed.
2. rescan_v4.py + refusal_rescan_v4.json + refusal_taxonomy_v4.json +
tests/test_rescan_v4_invariants.py — the current live snapshot.
Shifts: exactly 2 (0038, 0046). Same pattern as v3.
Sonnet wrote: S.4 parser/axis-lane/tests/ADR.
Opus wrote: rescan_v4.py + v3 test refactor + bundling.
Files:
- generate/math_candidate_parser.py (+142 lines)
- evals/math_capability_axes/S4_novel_initial_form/v1/ (20-case lane)
- tests/test_adr_0136_S4_novel_initial_form.py (40 tests)
- docs/decisions/ADR-0136.S.4-novel-initial-form.md
- evals/gsm8k_math/train_sample/v1/{rescan_v4.py, *_v4.json}
- tests/test_rescan_v4_invariants.py (8 tests)
- tests/test_rescan_v3_invariants.py (refactored to artifact-only)
94 lines
3.3 KiB
Python
94 lines
3.3 KiB
Python
"""ADR-0136.S.4-post-rescan — invariant tests for refusal rescan v4.
|
|
|
|
v4 is the CURRENT snapshot (post-S.4). Unlike v3 (which is now frozen and
|
|
asserted against disk), v4 tests re-run build_rescan() to confirm the live
|
|
parser still produces the v4 ledger — until S.5 lands and shifts more.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
from evals.gsm8k_math.train_sample.v1.rescan_v4 import build_rescan
|
|
|
|
_HERE = Path(__file__).resolve().parent.parent
|
|
_RESCAN_V4 = _HERE / "evals/gsm8k_math/train_sample/v1/refusal_rescan_v4.json"
|
|
_TAXONOMY_V4 = _HERE / "evals/gsm8k_math/train_sample/v1/refusal_taxonomy_v4.json"
|
|
|
|
|
|
def test_wrong_is_zero() -> None:
|
|
rescan, _ = build_rescan()
|
|
assert rescan["summary"]["wrong"] == 0
|
|
|
|
|
|
def test_admission_set_unchanged() -> None:
|
|
"""S.4 did not change the admission set: {0014, 0018, 0042}."""
|
|
rescan, _ = build_rescan()
|
|
admitted = {
|
|
c["case_id"] for c in rescan["per_case"] if c["current_outcome"] == "admitted"
|
|
}
|
|
assert admitted == {
|
|
"gsm8k-train-sample-v1-0014",
|
|
"gsm8k-train-sample-v1-0018",
|
|
"gsm8k-train-sample-v1-0042",
|
|
}
|
|
|
|
|
|
def test_exactly_two_shifts_v3_to_v4() -> None:
|
|
"""S.4 shifted exactly two cases (gsm8k-0038, gsm8k-0046)."""
|
|
rescan, _ = build_rescan()
|
|
assert rescan["summary"]["barrier_shifted_v3_to_v4"] == 2
|
|
shifted = sorted(
|
|
c["case_id"] for c in rescan["per_case"] if c["barrier_shifted"]
|
|
)
|
|
assert shifted == [
|
|
"gsm8k-train-sample-v1-0038",
|
|
"gsm8k-train-sample-v1-0046",
|
|
]
|
|
|
|
|
|
def test_gsm8k_0038_now_refuses_on_sentence_2() -> None:
|
|
rescan, _ = build_rescan()
|
|
case = next(
|
|
c for c in rescan["per_case"]
|
|
if c["case_id"] == "gsm8k-train-sample-v1-0038"
|
|
)
|
|
assert case["previous_first_refusal"].startswith("In a building")
|
|
assert case["current_first_refusal"].startswith("There are three times")
|
|
assert case["previous_primary_barrier"] == "novel_initial_form"
|
|
assert case["current_primary_barrier"] == "compound_comparative"
|
|
|
|
|
|
def test_gsm8k_0046_now_refuses_on_sentence_2() -> None:
|
|
rescan, _ = build_rescan()
|
|
case = next(
|
|
c for c in rescan["per_case"]
|
|
if c["case_id"] == "gsm8k-train-sample-v1-0046"
|
|
)
|
|
assert case["previous_first_refusal"].startswith("A school has 100 students")
|
|
assert case["current_first_refusal"].startswith("Half of the students")
|
|
assert case["previous_primary_barrier"] == "novel_initial_form"
|
|
assert case["current_primary_barrier"] == "fraction_operand"
|
|
|
|
|
|
def test_artifacts_deterministic() -> None:
|
|
r1, t1 = build_rescan()
|
|
r2, t2 = build_rescan()
|
|
assert json.dumps(r1, sort_keys=True) == json.dumps(r2, sort_keys=True)
|
|
assert json.dumps(t1, sort_keys=True) == json.dumps(t2, sort_keys=True)
|
|
|
|
|
|
def test_taxonomy_v4_has_50_entries() -> None:
|
|
_, taxonomy = build_rescan()
|
|
assert len(taxonomy["per_case"]) == 50
|
|
|
|
|
|
def test_disk_artifacts_match_freshly_computed() -> None:
|
|
if not _RESCAN_V4.exists() or not _TAXONOMY_V4.exists():
|
|
return
|
|
rescan, taxonomy = build_rescan()
|
|
on_disk_rescan = json.loads(_RESCAN_V4.read_text(encoding="utf-8"))
|
|
on_disk_taxonomy = json.loads(_TAXONOMY_V4.read_text(encoding="utf-8"))
|
|
assert on_disk_rescan == rescan
|
|
assert on_disk_taxonomy == taxonomy
|