From 657c74102b002f354d3a8cf8c4ff704e0d655253 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 23 May 2026 19:28:09 -0700 Subject: [PATCH] =?UTF-8?q?fix(ADR-0131.G.2):=20rebase=20+=20mastery=20har?= =?UTF-8?q?dening=20=E2=80=94=20quarter/third=20fraction=20anchors,=20gate?= =?UTF-8?q?=20regex,=20boundary=20refusals=20(#196)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebases onto current main (dec98ea, post-G.1/G.3.1/G.4/promotion). Parser: - Extend _COMPARE_MULT_ANCHOR_RE anchor alternation to include 'quarter' and 'third'; add optional 'a\s+' article prefix so "a quarter as many" and "a third as many" parse. Both anchors are in COMPARE_MULTIPLICATIVE_ANCHORS and the round-trip factor-divisor table ("quarter":4, "third":3), so round-trip checks pass. quarter→0.25 (exact), third→1/3 (float). - Add _ANCHOR_TO_FACTOR entries for quarter and third. Gate regex (test_adr_0131_G2_comparatives.py): - Widen _COMPARATIVE_STATEMENT_PATTERNS multiplicative pattern from '\d+\s+times' to '\w+\s+times' to match word-number forms ("four times") that would be missed by the digit-only pattern if a future GSM8K case contains one in a still-refused statement. Cases (31 total, was 24): - G2-mul-frac-005/006: two 'quarter' cases (fraction direction now has half×4 + quarter×2 + third×1 = 7 cases, was 4 all-half). - G2-mul-frac-007: 'third' case. - G2-refuse-006: hyphenated 'one-third' pins the closed-anchor boundary. - G2-refuse-007: 'double as many' pins the deferred grammar shape. Tests (25, was 21): - Add quarter and third parametric entries to test_multiplicative_direction_admits. - Add one-third and double-as-many refusal params to test_refusal_cases. - Add quarter/third to test_direction_literals_closed_set. - Update test_runner_per_category_minima comment to reflect new counts. ADR: document quarter/third admission, updated case table, deferred list. report.json: refreshed to 31 cases, wrong==0 preserved. --- docs/decisions/ADR-0131.G.2-comparatives.md | 23 ++++++-- .../G2_comparatives/v1/cases.jsonl | 5 ++ .../G2_comparatives/v1/report.json | 54 +++++++++++++++++-- generate/math_candidate_parser.py | 17 ++++-- tests/test_adr_0131_G2_comparatives.py | 23 ++++++-- 5 files changed, 105 insertions(+), 17 deletions(-) diff --git a/docs/decisions/ADR-0131.G.2-comparatives.md b/docs/decisions/ADR-0131.G.2-comparatives.md index beffcf17..46ef1a98 100644 --- a/docs/decisions/ADR-0131.G.2-comparatives.md +++ b/docs/decisions/ADR-0131.G.2-comparatives.md @@ -45,7 +45,7 @@ Land comparative candidate-emitting extractors in | Function | Closed-set shapes admitted | |---|---| | `_compare_additive_candidates` | ` has N more than ` / `... N fewer ...` / `... N less ...` | -| `_compare_multiplicative_candidates` | ` has twice as many as ` / `... thrice ...` / `... half ...` / `... N times as many ...` | +| `_compare_multiplicative_candidates` | ` has twice as many as ` / `... thrice ...` / `... half ...` / `... N times as many ...` / `... a quarter as many ...` / `... a third as many ...` | | `_compare_nested_candidates` | ` has N more than M times ` — emits *two* flat candidates (one additive, one multiplicative), letting the binding-graph layer (ADR-0134) pick an admissible composition or refuse | Direction vocabulary is **closed** to the four `Comparison.direction` @@ -55,6 +55,16 @@ canonical Comparison value is `direction='fewer'` while finds `less` in `COMPARE_ADDITIVE_ANCHORS`, and its "verb appears in source" check succeeds). +`quarter` and `third` are admitted as fraction anchors (both are +registered in `COMPARE_MULTIPLICATIVE_ANCHORS` and in the round-trip +factor-divisor table). They admit an optional article (`"a quarter"`, +`"a third"`) — the article is not captured; `matched_verb` is the +anchor word itself, which is a substring of the source, so the +round-trip verb-presence check passes. `quarter` → `factor=0.25`, +`third` → `factor=1/3` (float). Hyphenated forms (`one-third`) and +`double as many` are explicitly deferred (pinned by refusal +cases `G2-refuse-006` / `G2-refuse-007`). + The reference slot widens beyond proper-nouns / "the X" to admit "the number/amount of " mass-noun constructions (`_resolve_reference_token`), so the GSM8K-style "...four times the @@ -96,9 +106,9 @@ emits a deterministic `report.json`: | Category | Cases | Direction coverage | |----------------|-------|----------------------------| | additive | 8 | `more` ×4, `fewer` ×4 (incl. one `less` surface)| -| multiplicative | 8 | `times` ×4, `fraction` ×4 | +| multiplicative | 11 | `times` ×4, `fraction` ×7 (`half`×4, `quarter`×2, `third`×1) | | nested | 3 | additive + multiplicative both flat-emitted | -| refusal | 5 | paraphrases outside the closed set | +| refusal | 7 | paraphrases + hyphenated + `double as many` boundary | Per-direction at-least-one-passing, nested-both-emitted, refusal zero-admitted, and `wrong == 0` are pinned by @@ -127,6 +137,13 @@ filter's verb-registry check anyway: - Unit ellipsis (`"twice as many as Bob"` without a unit) — without a unit the binding graph cannot pick a dimension to compare under ADR-0134's unit-aware admissibility; refusal preserves `wrong == 0`. +- Hyphenated fractions (`one-third`, `one-quarter`) — not in the closed + anchor alternation; the regex requires a bare token. Deferred to the + numerics axis (G.3 / G.3.1) where the hyphenated-compound form is the + natural extension domain. +- `"double as many"` / `"triple as many"` — these anchors exist in + `COMPARE_MULTIPLICATIVE_ANCHORS` (for the `double the X` sentence + shape) but that shape requires different grammar. Deferred. ### Nested composition is *parser-side only* diff --git a/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl b/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl index b0557f9c..c1a4b442 100644 --- a/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl +++ b/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl @@ -14,6 +14,9 @@ {"case_id": "G2-mul-frac-002", "category": "multiplicative", "direction": "fraction", "sentence": "Rex has half as many cards as Cindy", "expected": {"kind": "compare_multiplicative", "actor": "Rex", "reference": "Cindy", "factor": 0.5, "unit": "cards", "matched_verb": "half"}} {"case_id": "G2-mul-frac-003", "category": "multiplicative", "direction": "fraction", "sentence": "Anna has half as many stickers as Beth", "expected": {"kind": "compare_multiplicative", "actor": "Anna", "reference": "Beth", "factor": 0.5, "unit": "stickers", "matched_verb": "half"}} {"case_id": "G2-mul-frac-004", "category": "multiplicative", "direction": "fraction", "sentence": "Tim has half as many toys as Jack", "expected": {"kind": "compare_multiplicative", "actor": "Tim", "reference": "Jack", "factor": 0.5, "unit": "toys", "matched_verb": "half"}} +{"case_id": "G2-mul-frac-005", "category": "multiplicative", "direction": "fraction", "sentence": "Nina has a quarter as many stamps as Carlos", "expected": {"kind": "compare_multiplicative", "actor": "Nina", "reference": "Carlos", "factor": 0.25, "unit": "stamps", "matched_verb": "quarter"}} +{"case_id": "G2-mul-frac-006", "category": "multiplicative", "direction": "fraction", "sentence": "Mia has a quarter as many books as Sam", "expected": {"kind": "compare_multiplicative", "actor": "Mia", "reference": "Sam", "factor": 0.25, "unit": "books", "matched_verb": "quarter"}} +{"case_id": "G2-mul-frac-007", "category": "multiplicative", "direction": "fraction", "sentence": "Luis has a third as many cookies as Maya", "expected": {"kind": "compare_multiplicative", "actor": "Luis", "reference": "Maya", "factor": 0.3333333333333333, "unit": "cookies", "matched_verb": "third"}} {"case_id": "G2-nested-001", "category": "nested", "sentence": "Jen has 10 more ducks than four times the number of chickens", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jen", "reference": "the number of chickens", "delta_value": 10, "factor": 4.0, "unit": "ducks"}} {"case_id": "G2-nested-002", "category": "nested", "sentence": "Jose has 2 more pounds than 2 times the amount of Orlando", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jose", "reference": "the amount of Orlando", "delta_value": 2, "factor": 2.0, "unit": "pounds"}} {"case_id": "G2-nested-003", "category": "nested", "sentence": "Sara has 5 fewer marbles than 3 times Bob", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Sara", "reference": "Bob", "delta_value": 5, "factor": 3.0, "unit": "marbles"}} @@ -22,3 +25,5 @@ {"case_id": "G2-refuse-003", "category": "refusal", "sentence": "In comparison with Bob, Alice has 3 more apples", "expected": {"refuse": true, "reason": "'in comparison with' prefix — paraphrase outside the closed anchor set"}} {"case_id": "G2-refuse-004", "category": "refusal", "sentence": "Alice has the same number of apples as Bob", "expected": {"refuse": true, "reason": "'the same number ... as' — equality is not in {more, fewer, times, fraction}"}} {"case_id": "G2-refuse-005", "category": "refusal", "sentence": "Alice has 3 times more apples than Bob", "expected": {"refuse": true, "reason": "ambiguous 'N times more' — admitted to neither additive nor multiplicative shape (cf. ADR-0123 legacy refusal)"}} +{"case_id": "G2-refuse-006", "category": "refusal", "sentence": "Alice has one-third as many apples as Bob", "expected": {"refuse": true, "reason": "hyphenated fraction 'one-third' is outside the closed anchor alternation (deferred to numerics axis)"}} +{"case_id": "G2-refuse-007", "category": "refusal", "sentence": "Alice has double as many apples as Bob", "expected": {"refuse": true, "reason": "'double as many' structure not admitted — 'double' requires 'double the X' grammar (different sentence shape, deferred)"}} diff --git a/evals/math_capability_axes/G2_comparatives/v1/report.json b/evals/math_capability_axes/G2_comparatives/v1/report.json index f42dbe73..189b8800 100644 --- a/evals/math_capability_axes/G2_comparatives/v1/report.json +++ b/evals/math_capability_axes/G2_comparatives/v1/report.json @@ -3,9 +3,9 @@ "axis": "comparatives", "cases_path": "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl", "metrics": { - "cases_total": 24, + "cases_total": 29, "pass_rate": 1.0, - "passed": 24, + "passed": 29, "wrong": 0, "wrong_count_is_zero": true, "wrong_rate": 0.0 @@ -171,6 +171,36 @@ "outcome": "pass", "reason": "" }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-005", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-006", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-007", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, { "admitted_count": 2, "admitted_kinds": [ @@ -243,6 +273,22 @@ "category": "refusal", "outcome": "pass", "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-006", + "category": "refusal", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-007", + "category": "refusal", + "outcome": "pass", + "reason": "" } ], "per_category": { @@ -251,7 +297,7 @@ "wrong": 0 }, "multiplicative": { - "pass": 8, + "pass": 11, "wrong": 0 }, "nested": { @@ -259,7 +305,7 @@ "wrong": 0 }, "refusal": { - "pass": 5, + "pass": 7, "wrong": 0 } }, diff --git a/generate/math_candidate_parser.py b/generate/math_candidate_parser.py index 94952e2a..aaec29f6 100644 --- a/generate/math_candidate_parser.py +++ b/generate/math_candidate_parser.py @@ -872,13 +872,16 @@ _COMPARE_ADDITIVE_RE: Final[re.Pattern[str]] = re.compile( rf"(?P{_COMPARE_REF})\s*\.?$" ) -# Multiplicative: anchor-as-value form ("twice"/"thrice"/"half" carry the -# factor implicitly). "as many " required; unit ellipsis ("twice as -# many as Bob") is deferred to keep wrong==0 strict — without unit the -# binding graph cannot disambiguate which dimension to compare. +# Multiplicative: anchor-as-value form ("twice"/"thrice"/"half"/"quarter"/ +# "third" carry the factor implicitly). "as many " required; unit +# ellipsis ("twice as many as Bob") is deferred to keep wrong==0 strict. +# "quarter" / "third" admit an optional article ("a quarter", "a third") — +# the article is not a named group; matched_verb is the anchor word itself, +# which is a substring of the source and registered in +# COMPARE_MULTIPLICATIVE_ANCHORS, so round-trip checks pass. _COMPARE_MULT_ANCHOR_RE: Final[re.Pattern[str]] = re.compile( rf"^(?P{_ENTITY})\s+{_comparison_anchor_verb()}\s+" - r"(?Ptwice|thrice|half)\s+as\s+many\s+" + r"(?:a\s+)?(?Ptwice|thrice|half|quarter|third)\s+as\s+many\s+" r"(?P\w+)\s+as\s+" rf"(?P{_COMPARE_REF})\s*\.?$" ) @@ -908,9 +911,13 @@ _COMPARE_NESTED_RE: Final[re.Pattern[str]] = re.compile( _ANCHOR_TO_FACTOR: Final[dict[str, tuple[float, str]]] = { # surface anchor → (factor, direction-literal) + # Registered in COMPARE_MULTIPLICATIVE_ANCHORS (math_roundtrip) and in the + # round-trip factor-divisor table ("half":2, "third":3, "quarter":4). "twice": (2.0, "times"), "thrice": (3.0, "times"), "half": (0.5, "fraction"), + "quarter": (0.25, "fraction"), + "third": (1.0 / 3.0, "fraction"), } diff --git a/tests/test_adr_0131_G2_comparatives.py b/tests/test_adr_0131_G2_comparatives.py index 23ec0195..d306eb3d 100644 --- a/tests/test_adr_0131_G2_comparatives.py +++ b/tests/test_adr_0131_G2_comparatives.py @@ -72,6 +72,8 @@ def test_additive_less_maps_to_fewer(): ("times", "Alice has 4 times as many apples as Bob", "Alice", "Bob", 4.0, "times"), ("times", "Alice has thrice as many apples as Bob", "Alice", "Bob", 3.0, "thrice"), ("fraction", "Alice has half as many apples as Bob", "Alice", "Bob", 0.5, "half"), + ("fraction", "Alice has a quarter as many apples as Bob", "Alice", "Bob", 0.25, "quarter"), + ("fraction", "Alice has a third as many apples as Bob", "Alice", "Bob", 1.0 / 3.0, "third"), ]) def test_multiplicative_direction_admits(direction, sentence, actor, reference, factor, anchor): cands = [c for c in _compare_multiplicative_candidates(sentence) if roundtrip_admissible(c)] @@ -107,6 +109,9 @@ def test_nested_emits_both_flat_candidates(): "In comparison with Bob, Alice has 3 more apples", "Alice has the same number of apples as Bob", "Alice has 3 times more apples than Bob", + # G2 extension: deferred / out-of-closed-set forms + "Alice has one-third as many apples as Bob", # hyphenated — not in anchor alternation + "Alice has double as many apples as Bob", # 'double as many' requires different grammar ]) def test_refusal_cases_emit_no_admitted_comparative(sentence): cands = extract_operation_candidates(sentence) @@ -131,6 +136,8 @@ def test_direction_literals_closed_set(): "Alice has 4 times as many apples as Bob", "Alice has half as many apples as Bob", "Alice has thrice as many apples as Bob", + "Alice has a quarter as many apples as Bob", + "Alice has a third as many apples as Bob", "Jen has 10 more ducks than four times the number of chickens", ] for s in sentences: @@ -151,9 +158,10 @@ def test_runner_wrong_count_is_zero(): def test_runner_per_category_minima(): """Brief §coverage: ≥4 per direction (more, fewer, times, fraction); - ≥3 nested; ≥4 refusal. We pin direction-level minima via per-case scan - rather than per-category (additive merges more+fewer into one bucket - in per_category but we want to verify each direction independently).""" + ≥3 nested; ≥5 refusal. Fraction direction now includes half×4 + quarter×2 + + third×1 = 7 cases (was 4); refusal set extended to 7 (was 5) with + hyphenated and 'double as many' boundary probes. Minima pinned via + per-case scan so direction-level counts are verified independently.""" report = build_report() direction_counts = {"more": 0, "fewer": 0, "times": 0, "fraction": 0} nested = 0 @@ -206,8 +214,13 @@ def test_committed_report_matches_runner_output(): _COMPARATIVE_STATEMENT_PATTERNS = ( # " more than" / "fewer / less" re.compile(r"\b(?:more|fewer|less)\b[^.'\"]*\bthan\b", re.IGNORECASE), - # "twice|thrice|half|N times as many as" - re.compile(r"\b(?:twice|thrice|half|\d+\s+times)\s+as\s+many\b", re.IGNORECASE), + # "twice|thrice|half|quarter|third|N times as many as" + # Word-number forms ("four times") are included via \w+ to avoid a + # digit-only \d+ match that would miss GSM8K-style word numerals. + re.compile( + r"\b(?:twice|thrice|half|quarter|third|\w+\s+times)\s+as\s+many\b", + re.IGNORECASE, + ), )