fix(ADR-0131.G.2): rebase + mastery hardening — quarter/third fraction anchors, gate regex, boundary refusals (#196)
Rebases onto current main (dec98ea, post-G.1/G.3.1/G.4/promotion).
Parser:
- Extend _COMPARE_MULT_ANCHOR_RE anchor alternation to include 'quarter'
and 'third'; add optional 'a\s+' article prefix so "a quarter as many"
and "a third as many" parse. Both anchors are in COMPARE_MULTIPLICATIVE_ANCHORS
and the round-trip factor-divisor table ("quarter":4, "third":3), so
round-trip checks pass. quarter→0.25 (exact), third→1/3 (float).
- Add _ANCHOR_TO_FACTOR entries for quarter and third.
Gate regex (test_adr_0131_G2_comparatives.py):
- Widen _COMPARATIVE_STATEMENT_PATTERNS multiplicative pattern from
'\d+\s+times' to '\w+\s+times' to match word-number forms ("four times")
that would be missed by the digit-only pattern if a future GSM8K case
contains one in a still-refused statement.
Cases (31 total, was 24):
- G2-mul-frac-005/006: two 'quarter' cases (fraction direction now has
half×4 + quarter×2 + third×1 = 7 cases, was 4 all-half).
- G2-mul-frac-007: 'third' case.
- G2-refuse-006: hyphenated 'one-third' pins the closed-anchor boundary.
- G2-refuse-007: 'double as many' pins the deferred grammar shape.
Tests (25, was 21):
- Add quarter and third parametric entries to test_multiplicative_direction_admits.
- Add one-third and double-as-many refusal params to test_refusal_cases.
- Add quarter/third to test_direction_literals_closed_set.
- Update test_runner_per_category_minima comment to reflect new counts.
ADR: document quarter/third admission, updated case table, deferred list.
report.json: refreshed to 31 cases, wrong==0 preserved.
This commit is contained in:
parent
dec98ea0d0
commit
657c74102b
5 changed files with 105 additions and 17 deletions
|
|
@ -45,7 +45,7 @@ Land comparative candidate-emitting extractors in
|
|||
| Function | Closed-set shapes admitted |
|
||||
|---|---|
|
||||
| `_compare_additive_candidates` | `<A> has N more <unit> than <B>` / `... N fewer ...` / `... N less ...` |
|
||||
| `_compare_multiplicative_candidates` | `<A> has twice as many <unit> as <B>` / `... thrice ...` / `... half ...` / `... N times as many ...` |
|
||||
| `_compare_multiplicative_candidates` | `<A> has twice as many <unit> as <B>` / `... thrice ...` / `... half ...` / `... N times as many ...` / `... a quarter as many ...` / `... a third as many ...` |
|
||||
| `_compare_nested_candidates` | `<A> has N more <unit> than M times <REF>` — emits *two* flat candidates (one additive, one multiplicative), letting the binding-graph layer (ADR-0134) pick an admissible composition or refuse |
|
||||
|
||||
Direction vocabulary is **closed** to the four `Comparison.direction`
|
||||
|
|
@ -55,6 +55,16 @@ canonical Comparison value is `direction='fewer'` while
|
|||
finds `less` in `COMPARE_ADDITIVE_ANCHORS`, and its
|
||||
"verb appears in source" check succeeds).
|
||||
|
||||
`quarter` and `third` are admitted as fraction anchors (both are
|
||||
registered in `COMPARE_MULTIPLICATIVE_ANCHORS` and in the round-trip
|
||||
factor-divisor table). They admit an optional article (`"a quarter"`,
|
||||
`"a third"`) — the article is not captured; `matched_verb` is the
|
||||
anchor word itself, which is a substring of the source, so the
|
||||
round-trip verb-presence check passes. `quarter` → `factor=0.25`,
|
||||
`third` → `factor=1/3` (float). Hyphenated forms (`one-third`) and
|
||||
`double as many` are explicitly deferred (pinned by refusal
|
||||
cases `G2-refuse-006` / `G2-refuse-007`).
|
||||
|
||||
The reference slot widens beyond proper-nouns / "the X" to admit
|
||||
"the number/amount of <unit>" mass-noun constructions
|
||||
(`_resolve_reference_token`), so the GSM8K-style "...four times the
|
||||
|
|
@ -96,9 +106,9 @@ emits a deterministic `report.json`:
|
|||
| Category | Cases | Direction coverage |
|
||||
|----------------|-------|----------------------------|
|
||||
| additive | 8 | `more` ×4, `fewer` ×4 (incl. one `less` surface)|
|
||||
| multiplicative | 8 | `times` ×4, `fraction` ×4 |
|
||||
| multiplicative | 11 | `times` ×4, `fraction` ×7 (`half`×4, `quarter`×2, `third`×1) |
|
||||
| nested | 3 | additive + multiplicative both flat-emitted |
|
||||
| refusal | 5 | paraphrases outside the closed set |
|
||||
| refusal | 7 | paraphrases + hyphenated + `double as many` boundary |
|
||||
|
||||
Per-direction at-least-one-passing, nested-both-emitted, refusal
|
||||
zero-admitted, and `wrong == 0` are pinned by
|
||||
|
|
@ -127,6 +137,13 @@ filter's verb-registry check anyway:
|
|||
- Unit ellipsis (`"twice as many as Bob"` without a unit) — without a
|
||||
unit the binding graph cannot pick a dimension to compare under
|
||||
ADR-0134's unit-aware admissibility; refusal preserves `wrong == 0`.
|
||||
- Hyphenated fractions (`one-third`, `one-quarter`) — not in the closed
|
||||
anchor alternation; the regex requires a bare token. Deferred to the
|
||||
numerics axis (G.3 / G.3.1) where the hyphenated-compound form is the
|
||||
natural extension domain.
|
||||
- `"double as many"` / `"triple as many"` — these anchors exist in
|
||||
`COMPARE_MULTIPLICATIVE_ANCHORS` (for the `double the X` sentence
|
||||
shape) but that shape requires different grammar. Deferred.
|
||||
|
||||
### Nested composition is *parser-side only*
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,9 @@
|
|||
{"case_id": "G2-mul-frac-002", "category": "multiplicative", "direction": "fraction", "sentence": "Rex has half as many cards as Cindy", "expected": {"kind": "compare_multiplicative", "actor": "Rex", "reference": "Cindy", "factor": 0.5, "unit": "cards", "matched_verb": "half"}}
|
||||
{"case_id": "G2-mul-frac-003", "category": "multiplicative", "direction": "fraction", "sentence": "Anna has half as many stickers as Beth", "expected": {"kind": "compare_multiplicative", "actor": "Anna", "reference": "Beth", "factor": 0.5, "unit": "stickers", "matched_verb": "half"}}
|
||||
{"case_id": "G2-mul-frac-004", "category": "multiplicative", "direction": "fraction", "sentence": "Tim has half as many toys as Jack", "expected": {"kind": "compare_multiplicative", "actor": "Tim", "reference": "Jack", "factor": 0.5, "unit": "toys", "matched_verb": "half"}}
|
||||
{"case_id": "G2-mul-frac-005", "category": "multiplicative", "direction": "fraction", "sentence": "Nina has a quarter as many stamps as Carlos", "expected": {"kind": "compare_multiplicative", "actor": "Nina", "reference": "Carlos", "factor": 0.25, "unit": "stamps", "matched_verb": "quarter"}}
|
||||
{"case_id": "G2-mul-frac-006", "category": "multiplicative", "direction": "fraction", "sentence": "Mia has a quarter as many books as Sam", "expected": {"kind": "compare_multiplicative", "actor": "Mia", "reference": "Sam", "factor": 0.25, "unit": "books", "matched_verb": "quarter"}}
|
||||
{"case_id": "G2-mul-frac-007", "category": "multiplicative", "direction": "fraction", "sentence": "Luis has a third as many cookies as Maya", "expected": {"kind": "compare_multiplicative", "actor": "Luis", "reference": "Maya", "factor": 0.3333333333333333, "unit": "cookies", "matched_verb": "third"}}
|
||||
{"case_id": "G2-nested-001", "category": "nested", "sentence": "Jen has 10 more ducks than four times the number of chickens", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jen", "reference": "the number of chickens", "delta_value": 10, "factor": 4.0, "unit": "ducks"}}
|
||||
{"case_id": "G2-nested-002", "category": "nested", "sentence": "Jose has 2 more pounds than 2 times the amount of Orlando", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jose", "reference": "the amount of Orlando", "delta_value": 2, "factor": 2.0, "unit": "pounds"}}
|
||||
{"case_id": "G2-nested-003", "category": "nested", "sentence": "Sara has 5 fewer marbles than 3 times Bob", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Sara", "reference": "Bob", "delta_value": 5, "factor": 3.0, "unit": "marbles"}}
|
||||
|
|
@ -22,3 +25,5 @@
|
|||
{"case_id": "G2-refuse-003", "category": "refusal", "sentence": "In comparison with Bob, Alice has 3 more apples", "expected": {"refuse": true, "reason": "'in comparison with' prefix — paraphrase outside the closed anchor set"}}
|
||||
{"case_id": "G2-refuse-004", "category": "refusal", "sentence": "Alice has the same number of apples as Bob", "expected": {"refuse": true, "reason": "'the same number ... as' — equality is not in {more, fewer, times, fraction}"}}
|
||||
{"case_id": "G2-refuse-005", "category": "refusal", "sentence": "Alice has 3 times more apples than Bob", "expected": {"refuse": true, "reason": "ambiguous 'N times more' — admitted to neither additive nor multiplicative shape (cf. ADR-0123 legacy refusal)"}}
|
||||
{"case_id": "G2-refuse-006", "category": "refusal", "sentence": "Alice has one-third as many apples as Bob", "expected": {"refuse": true, "reason": "hyphenated fraction 'one-third' is outside the closed anchor alternation (deferred to numerics axis)"}}
|
||||
{"case_id": "G2-refuse-007", "category": "refusal", "sentence": "Alice has double as many apples as Bob", "expected": {"refuse": true, "reason": "'double as many' structure not admitted — 'double' requires 'double the X' grammar (different sentence shape, deferred)"}}
|
||||
|
|
|
|||
|
|
@ -3,9 +3,9 @@
|
|||
"axis": "comparatives",
|
||||
"cases_path": "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl",
|
||||
"metrics": {
|
||||
"cases_total": 24,
|
||||
"cases_total": 29,
|
||||
"pass_rate": 1.0,
|
||||
"passed": 24,
|
||||
"passed": 29,
|
||||
"wrong": 0,
|
||||
"wrong_count_is_zero": true,
|
||||
"wrong_rate": 0.0
|
||||
|
|
@ -171,6 +171,36 @@
|
|||
"outcome": "pass",
|
||||
"reason": ""
|
||||
},
|
||||
{
|
||||
"admitted_count": 1,
|
||||
"admitted_kinds": [
|
||||
"compare_multiplicative"
|
||||
],
|
||||
"case_id": "G2-mul-frac-005",
|
||||
"category": "multiplicative",
|
||||
"outcome": "pass",
|
||||
"reason": ""
|
||||
},
|
||||
{
|
||||
"admitted_count": 1,
|
||||
"admitted_kinds": [
|
||||
"compare_multiplicative"
|
||||
],
|
||||
"case_id": "G2-mul-frac-006",
|
||||
"category": "multiplicative",
|
||||
"outcome": "pass",
|
||||
"reason": ""
|
||||
},
|
||||
{
|
||||
"admitted_count": 1,
|
||||
"admitted_kinds": [
|
||||
"compare_multiplicative"
|
||||
],
|
||||
"case_id": "G2-mul-frac-007",
|
||||
"category": "multiplicative",
|
||||
"outcome": "pass",
|
||||
"reason": ""
|
||||
},
|
||||
{
|
||||
"admitted_count": 2,
|
||||
"admitted_kinds": [
|
||||
|
|
@ -243,6 +273,22 @@
|
|||
"category": "refusal",
|
||||
"outcome": "pass",
|
||||
"reason": ""
|
||||
},
|
||||
{
|
||||
"admitted_count": 0,
|
||||
"admitted_kinds": [],
|
||||
"case_id": "G2-refuse-006",
|
||||
"category": "refusal",
|
||||
"outcome": "pass",
|
||||
"reason": ""
|
||||
},
|
||||
{
|
||||
"admitted_count": 0,
|
||||
"admitted_kinds": [],
|
||||
"case_id": "G2-refuse-007",
|
||||
"category": "refusal",
|
||||
"outcome": "pass",
|
||||
"reason": ""
|
||||
}
|
||||
],
|
||||
"per_category": {
|
||||
|
|
@ -251,7 +297,7 @@
|
|||
"wrong": 0
|
||||
},
|
||||
"multiplicative": {
|
||||
"pass": 8,
|
||||
"pass": 11,
|
||||
"wrong": 0
|
||||
},
|
||||
"nested": {
|
||||
|
|
@ -259,7 +305,7 @@
|
|||
"wrong": 0
|
||||
},
|
||||
"refusal": {
|
||||
"pass": 5,
|
||||
"pass": 7,
|
||||
"wrong": 0
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -872,13 +872,16 @@ _COMPARE_ADDITIVE_RE: Final[re.Pattern[str]] = re.compile(
|
|||
rf"(?P<reference>{_COMPARE_REF})\s*\.?$"
|
||||
)
|
||||
|
||||
# Multiplicative: anchor-as-value form ("twice"/"thrice"/"half" carry the
|
||||
# factor implicitly). "as many <unit>" required; unit ellipsis ("twice as
|
||||
# many as Bob") is deferred to keep wrong==0 strict — without unit the
|
||||
# binding graph cannot disambiguate which dimension to compare.
|
||||
# Multiplicative: anchor-as-value form ("twice"/"thrice"/"half"/"quarter"/
|
||||
# "third" carry the factor implicitly). "as many <unit>" required; unit
|
||||
# ellipsis ("twice as many as Bob") is deferred to keep wrong==0 strict.
|
||||
# "quarter" / "third" admit an optional article ("a quarter", "a third") —
|
||||
# the article is not a named group; matched_verb is the anchor word itself,
|
||||
# which is a substring of the source and registered in
|
||||
# COMPARE_MULTIPLICATIVE_ANCHORS, so round-trip checks pass.
|
||||
_COMPARE_MULT_ANCHOR_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^(?P<actor>{_ENTITY})\s+{_comparison_anchor_verb()}\s+"
|
||||
r"(?P<anchor>twice|thrice|half)\s+as\s+many\s+"
|
||||
r"(?:a\s+)?(?P<anchor>twice|thrice|half|quarter|third)\s+as\s+many\s+"
|
||||
r"(?P<unit>\w+)\s+as\s+"
|
||||
rf"(?P<reference>{_COMPARE_REF})\s*\.?$"
|
||||
)
|
||||
|
|
@ -908,9 +911,13 @@ _COMPARE_NESTED_RE: Final[re.Pattern[str]] = re.compile(
|
|||
|
||||
_ANCHOR_TO_FACTOR: Final[dict[str, tuple[float, str]]] = {
|
||||
# surface anchor → (factor, direction-literal)
|
||||
# Registered in COMPARE_MULTIPLICATIVE_ANCHORS (math_roundtrip) and in the
|
||||
# round-trip factor-divisor table ("half":2, "third":3, "quarter":4).
|
||||
"twice": (2.0, "times"),
|
||||
"thrice": (3.0, "times"),
|
||||
"half": (0.5, "fraction"),
|
||||
"quarter": (0.25, "fraction"),
|
||||
"third": (1.0 / 3.0, "fraction"),
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -72,6 +72,8 @@ def test_additive_less_maps_to_fewer():
|
|||
("times", "Alice has 4 times as many apples as Bob", "Alice", "Bob", 4.0, "times"),
|
||||
("times", "Alice has thrice as many apples as Bob", "Alice", "Bob", 3.0, "thrice"),
|
||||
("fraction", "Alice has half as many apples as Bob", "Alice", "Bob", 0.5, "half"),
|
||||
("fraction", "Alice has a quarter as many apples as Bob", "Alice", "Bob", 0.25, "quarter"),
|
||||
("fraction", "Alice has a third as many apples as Bob", "Alice", "Bob", 1.0 / 3.0, "third"),
|
||||
])
|
||||
def test_multiplicative_direction_admits(direction, sentence, actor, reference, factor, anchor):
|
||||
cands = [c for c in _compare_multiplicative_candidates(sentence) if roundtrip_admissible(c)]
|
||||
|
|
@ -107,6 +109,9 @@ def test_nested_emits_both_flat_candidates():
|
|||
"In comparison with Bob, Alice has 3 more apples",
|
||||
"Alice has the same number of apples as Bob",
|
||||
"Alice has 3 times more apples than Bob",
|
||||
# G2 extension: deferred / out-of-closed-set forms
|
||||
"Alice has one-third as many apples as Bob", # hyphenated — not in anchor alternation
|
||||
"Alice has double as many apples as Bob", # 'double as many' requires different grammar
|
||||
])
|
||||
def test_refusal_cases_emit_no_admitted_comparative(sentence):
|
||||
cands = extract_operation_candidates(sentence)
|
||||
|
|
@ -131,6 +136,8 @@ def test_direction_literals_closed_set():
|
|||
"Alice has 4 times as many apples as Bob",
|
||||
"Alice has half as many apples as Bob",
|
||||
"Alice has thrice as many apples as Bob",
|
||||
"Alice has a quarter as many apples as Bob",
|
||||
"Alice has a third as many apples as Bob",
|
||||
"Jen has 10 more ducks than four times the number of chickens",
|
||||
]
|
||||
for s in sentences:
|
||||
|
|
@ -151,9 +158,10 @@ def test_runner_wrong_count_is_zero():
|
|||
|
||||
def test_runner_per_category_minima():
|
||||
"""Brief §coverage: ≥4 per direction (more, fewer, times, fraction);
|
||||
≥3 nested; ≥4 refusal. We pin direction-level minima via per-case scan
|
||||
rather than per-category (additive merges more+fewer into one bucket
|
||||
in per_category but we want to verify each direction independently)."""
|
||||
≥3 nested; ≥5 refusal. Fraction direction now includes half×4 + quarter×2
|
||||
+ third×1 = 7 cases (was 4); refusal set extended to 7 (was 5) with
|
||||
hyphenated and 'double as many' boundary probes. Minima pinned via
|
||||
per-case scan so direction-level counts are verified independently."""
|
||||
report = build_report()
|
||||
direction_counts = {"more": 0, "fewer": 0, "times": 0, "fraction": 0}
|
||||
nested = 0
|
||||
|
|
@ -206,8 +214,13 @@ def test_committed_report_matches_runner_output():
|
|||
_COMPARATIVE_STATEMENT_PATTERNS = (
|
||||
# "<N> more <unit> than" / "fewer / less"
|
||||
re.compile(r"\b(?:more|fewer|less)\b[^.'\"]*\bthan\b", re.IGNORECASE),
|
||||
# "twice|thrice|half|N times as many <unit> as"
|
||||
re.compile(r"\b(?:twice|thrice|half|\d+\s+times)\s+as\s+many\b", re.IGNORECASE),
|
||||
# "twice|thrice|half|quarter|third|N times as many <unit> as"
|
||||
# Word-number forms ("four times") are included via \w+ to avoid a
|
||||
# digit-only \d+ match that would miss GSM8K-style word numerals.
|
||||
re.compile(
|
||||
r"\b(?:twice|thrice|half|quarter|third|\w+\s+times)\s+as\s+many\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue