diff --git a/docs/decisions/ADR-0131.G.3.1-numerics-extensions.md b/docs/decisions/ADR-0131.G.3.1-numerics-extensions.md new file mode 100644 index 00000000..2c143396 --- /dev/null +++ b/docs/decisions/ADR-0131.G.3.1-numerics-extensions.md @@ -0,0 +1,273 @@ +# ADR-0131.G.3.1 — Numerics extensions (fractions + multi-currency + multi-token cardinals + word-num-adjective) + +**Status:** Proposed +**Date:** 2026-05-23 +**Author:** CORE main agent (Sonnet 4.6) +**Depends on:** ADR-0131.G.3 (parent, PR #183), ADR-0127 (units pack), +ADR-0128 (numerics pack) +**Parent:** ADR-0131 (composite math-expert promotion gate) +**Foundation for:** ADR-0131.G.3.2 (£ / pound sterling follow-up) + +--- + +## Context + +ADR-0131.G.3 (PR #183) shipped the first literal-recognition axis: money +symbols (`$N` / `$N.NN`), money word forms (`N dollars` / `N cents`), and +hyphenated multi-word cardinals (`twenty-five`). The four shapes explicitly +deferred in G.3's "Out-of-scope" section form this iteration's closed-set +scope: + +1. **Fractions end-to-end** — `N/M of a ` initial possession. +2. **Multi-currency** — `¢ € ¥ ₱` symbols (£ deferred; see below). +3. **Multi-token space-separated cardinals** — `one hundred`, `two thousand + five hundred`. +4. **Word-number-adjective compositions** — `five full boxes` per ADR-0127 + substance-qualifier precedent. + +All canonical-unit architectural decisions from G.3 are preserved unchanged: +cent is the canonical unit for money (US dollar path), pack-driven not +regex-spam, refusal-first. + +--- + +## Decision + +### Axis 1 — Fractions end-to-end + +`_resolve_value` in `math_candidate_parser.py` already handles `N/M` +(returns `_ResolvedValue(num / den, None)`). The pipeline-level gap was in +`_INITIAL_HAS_RE`'s substance-qualifier handling: `Bob has 3/4 of a cup.` +would match the main regex with `unit=None` (the `of a cup` phrase was +consumed by the discardable substance qualifier), causing the candidate to +not emit because no unit could be determined. + +**Fix**: new dedicated regex `_INITIAL_FRACTION_OF_RE` with shape +` has of [a/an] [of ]?`. This explicitly +extracts the unit from the `of` phrase when the value is a slash fraction, +emitting a `CandidateInitial` with `value=N/M` and `unit=`. +The main `_INITIAL_HAS_RE` is unchanged; the new extractor fires +in `extract_initial_candidates` after the primary extractor. + +Closed-set: digit/digit literal with `M > 0`. Division-by-zero refused at +`_resolve_value` time (already the case since G.3). + +### Axis 2 — Multi-currency + +All five pack-recognized foreign currency symbols are in `en_units_v1`: +`¢` (cent), `€` (euro), `£` (pound sterling), `¥` (yen), `₱` (peso). + +**Wired in this iteration**: `¢`, `€`, `¥`, `₱`. + +**Deferred to G.3.2**: `£` / pound sterling. The `en_units_v1` pack +correctly models pound sterling as a two-word unit (`"plural": "pounds +sterling"`). However, the question extractor (`_Q_ENTITY_RE`, +`_Q_TOTAL_RE`) uses `(?P\w+)` which only captures a single word +token. A question like `How many pounds sterling does Alice have?` fails +to parse the unit slot (`sterling` is a second word, not consumed by the +single-token capture group). Widening the question extractor's unit slot +to support multi-word units is a distinct architectural change that belongs +in a dedicated PR (G.3.2) to keep the diff bounded and attributable. + +The `_resolve_value` and `_currency_symbols` table includes `£` → `pounds +sterling` (so symbol parsing is complete); the question-extractor gap is +the only blocker. + +**Normalization**: +- `¢N` → `N` cents (1:1; ¢ face value IS the canonical unit). +- `€N` / `€N.NN` → `N` / `N.NN` euros (factor=1; no sub-unit in pack). +- `¥N` → `N` yen (integer-only; yen has no sub-unit in the pack or in + common usage — decimal ¥ form refused). +- `₱N` / `₱N.NN` → `N` / `N.NN` pesos (factor=1). + +3+ decimal places are refused for all currency symbols (closed-set +boundary; same rule as `$N.NNN`). + +**`_MONEY_SYMBOL`** regex widened to alternation over all six symbols. + +**`_unit_grounds`** in `math_roundtrip.py` widened: each currency symbol +grounds its respective canonical unit when the symbol appears in the raw +source span (mirrors the existing `$` / `dollar` grounding logic). + +**`_money_unit_normalization`** extended: word-form entries for `euro`, +`euros`, `yen`, `peso`, `pesos` pass through with factor=1. + +### Axis 3 — Multi-token space-separated cardinals + +`parse_compound_cardinal` in `language_packs/numerics_loader.py` already +handles `"one hundred"`, `"two thousand five hundred"` etc. — it +normalises hyphens to spaces then tokenises. The parser couldn't emit these +in the value slot because `_VALUE` alternation uses a single-token pattern; +a greedy multi-word sequence would span the unit-slot boundary. + +**Approach (a) chosen** (separate extractor) over **(b)** (`_VALUE` +widening): + +- Widening `_VALUE` to greedily match cardinal-word sequences would require + look-ahead to distinguish the last cardinal word from the first unit word + (e.g. in `one hundred boxes`, `"one hundred"` is the value and `"boxes"` + is the unit; greedy matching consumes all three). This unwinding requires + either atomic groups (Python 3.11+) or a two-pass approach, both of which + add complexity across every pattern that uses `_VALUE`. +- A dedicated extractor (`_MULTI_WORD_CARDINAL_RE`) handles the + ` has [?] ` shape precisely, + leaves `_VALUE` unchanged for all other paths, and is auditable in + isolation. + +`_MULTI_WORD_CARDINAL_RE` is built from the `WORD_NUMBERS` table keys +(same source as `_WORD_NUM_OPTIONS` in `_VALUE`), requiring at least two +consecutive cardinal words before the unit slot. The value group is passed +to `parse_compound_cardinal` for resolution. Provenance: the first cardinal +word of the sequence is the `matched_value_token` (it grounds because it +is literally in the source span). + +### Axis 4 — Word-number-adjective + +`five full boxes` per ADR-0127 substance-qualifier precedent. The adjective +(`full`, `loose`, `empty`, `whole`, `broken`, `new`, `old`, `small`, +`large`, `fresh`, `raw`, `flat`) is inserted between the cardinal value and +the unit head noun. It is treated as part of the unit phrase: discarded +at parse time, not surfaced to the solver. + +**Implementation**: an optional non-capturing group +`(?:\s+(?:full|loose|…))?` is inserted between the `(?P…)` slot and +the `(?P\w+)?` slot in `_INITIAL_HAS_RE`. The same adjective list is +also added to `_MULTI_WORD_CARDINAL_RE`'s optional-adjective slot (axis 3 +and axis 4 compose naturally). + +The adjective list is a closed set identical to the one in +`_CONJ_OBJECT_RE` (already ratified by G.4), keeping both shapes +consistent. + +### Pre-existing G.4 type bugs fixed + +ADR-0131.G.4's multi-clause extractors (`_conj_subject_each_candidates`, +`_conj_object_candidates`, `_embedded_quantifier_candidates`, +`_build_conj_embedded_sum`, `_compare_multiplicative_candidates`, +`_compare_nested_candidates`) called `_resolve_value(...)` directly as a +numeric operand (e.g. `float(_resolve_value(x))`, `_resolve_value(n) * +_resolve_value(m)`, `Quantity(value=_resolve_value(x), ...)`). Since +`_resolve_value` returns `_ResolvedValue | None` (not a raw number), these +sites raised `TypeError` at runtime when hit by the GSM8K coverage probe. + +This is a minimal fix: each site is updated to unwrap `.value` from the +`_ResolvedValue` result (or guard against `None` and return `[]`/early +exit). Semantics are identical — the value extracted is the same numeric +payload; no new candidates are emitted or suppressed. + +--- + +## What changed in code + +### `generate/math_candidate_parser.py` + +- `_MONEY_SYMBOL` widened to six-currency alternation. +- `_CURRENCY_SYMBOLS` constant: symbol → `(unit_surface, factor)` table. +- `_resolve_currency`: dispatcher for all currency symbol literals. +- `_resolve_value`: currency branch now delegates to `_resolve_currency`. +- `_INITIAL_HAS_RE`: optional adjective group inserted between value and + unit slots. +- `_INITIAL_FRACTION_OF_RE`: new regex for `N/M of [a/an] `. +- `_MULTI_WORD_CARDINAL_RE`: new regex for ` has + [adj?] `. +- `_fraction_of_candidates`: extractor for axis 1. +- `_multi_word_cardinal_candidates`: extractor for axis 3. +- `extract_initial_candidates`: calls both new extractors (after primary, + before G.4 extractors). +- `_money_unit_normalization`: extended for euro/yen/peso word forms. +- G.4 pre-existing type fixes: `.value` unwrap at six call sites. + +### `generate/math_roundtrip.py` + +- `_unit_grounds`: widened for `¢`, `€`, `¥`, `₱` symbols (mirrors + existing `$` logic). +- `_value_grounds`: currency symbol prefix check widened from `$`-only to + the full `_CURRENCY_SYM_SET`. + +### New files + +- `evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl` — 28 cases + (5 per axis × 4 + 8 refusal probes). +- `evals/math_capability_axes/G3_numerics/v1_1/runner.py` — identical + adapter pattern to v1; byte-equal `report.json` across runs. +- `evals/math_capability_axes/G3_numerics/v1_1/report.json` — committed + run artifact. +- `tests/test_adr_0131_G31_numerics_extensions.py` — 42 tests. + +--- + +## Evidence + +### Axis lane (`v1_1/runner.py`) + +``` +axis: numerics_extensions +cases_total: 28 +solved_correct: 20 +solved_wrong: 0 (gate: must be 0) +refused_as_expected: 8 +correct_rate_on_positive_cases: 100.0% +overall_pass: True +``` + +Per-axis breakdown: +- Axis 1 (fractions): 5/5 solved_correct +- Axis 2 (multi-currency): 5/5 solved_correct (¢ € ¥ ₱; £ deferred) +- Axis 3 (multi-word cardinals): 5/5 solved_correct +- Axis 4 (word-num-adjective): 5/5 solved_correct +- Refusal probes: 8/8 refused (percentages ×2, sci-notation ×2, + locale-separators ×2, 3-decimal-money ×2) + +`report.json` byte-equal across two independent runs. + +### Test suite + +**42/42 pass** in `tests/test_adr_0131_G31_numerics_extensions.py` (0.27s). + +### Parent v1 lane regression + +**20/20 + 6/6 = 26/26** — unchanged from PR #183 commit. + +### GSM8K coverage probe + +``` +admission_rate: 0/50 = 0.0% +admitted_wrong: 0 (safety rail intact) +``` + +No probe delta. Admission remains at 0/50 for the same structural reasons +identified in G.3: most GSM8K refusals are verb-class or multi-clause +failures (G.1 / G.4 axes), not literal-recognition gaps. Fractions and +multi-word cardinals don't appear in the 50-case probe as the binding +blocker. The safety rail (`admitted_wrong == 0`) is preserved. + +--- + +## Currencies deferred to G.3.2 + +**`£` / pound sterling**: `en_units_v1` is correctly populated; `_resolve_value("£15")` returns `_ResolvedValue(15, "pounds sterling")`. The +blocker is the question extractor's single-token `(?P\w+)` group, +which cannot parse `"How many pounds sterling does Alice have?"`. Fixing +this requires widening the question-extractor unit slot to support +multi-word units — a distinct scoped change for G.3.2. + +--- + +## CLAUDE.md PR-checklist answers + +- **Capability/performance/security added:** Extends the candidate-graph + parser's literal-recognition to four deferred shape families from G.3. + `wrong == 0` on the axis lane proves no wrong answers were introduced. +- **Invariant proving field remains valid:** `solved_wrong == 0` on the + v1_1 axis lane; `admitted_wrong == 0` on the GSM8K probe; no changes to + `algebra/`, `chat/`, or `core/`. +- **CLI/eval lane:** `PYTHONPATH=. python3 + evals/math_capability_axes/G3_numerics/v1_1/runner.py` and `pytest + tests/test_adr_0131_G31_numerics_extensions.py`. +- **Avoided hidden normalization/stochastic fallback/approximate + recall/unreviewed mutation:** Yes. All lookups are deterministic and + pack-driven. Currency normalization is a fixed table. No solver or + binding-graph changes. +- **Trust boundary:** User-controlled text → parser regex; all new regex + patterns are closed alternations with no catastrophic-backtracking risk. + Pack file paths use the existing `safe_pack_id` sanitiser (ADR-0051). diff --git a/evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json b/evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json index d823b603..343d3a6e 100644 --- a/evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json +++ b/evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json @@ -86,7 +86,7 @@ "expected_unit": "", "outcome": "refused", "realized_prose": null, - "reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.'", + "reason": "candidate_graph: no admissible candidate for statement: 'They need to put all of their loose crayons in a box.'", "trace_hash": null }, { @@ -108,7 +108,7 @@ "expected_unit": "", "outcome": "refused", "realized_prose": null, - "reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'", + "reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'", "trace_hash": null }, { @@ -471,7 +471,7 @@ "expected_unit": "", "outcome": "refused", "realized_prose": null, - "reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.'", + "reason": "candidate_graph: no admissible candidate for question: 'If Ella sells 200 apples, how many apples does Ella has left?'", "trace_hash": null }, { @@ -565,6 +565,14 @@ ], "probe": "gsm8k_train_sample_coverage", "refused_reasons_top": [ + { + "count": 1, + "reason": "candidate_graph: no admissible candidate for question: 'If Ella sells 200 apples, how many apples does Ella has left?'" + }, + { + "count": 1, + "reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'" + }, { "count": 1, "reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/" @@ -601,10 +609,6 @@ "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Bob can shuck 10 oysters in 5 minutes.'" }, - { - "count": 1, - "reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25" - }, { "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their ma" @@ -613,10 +617,6 @@ "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Fabian bought a brand new computer mouse and keyboard to be abl" }, - { - "count": 1, - "reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, an" - }, { "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Georgie is a varsity player on a football team.'" @@ -641,10 +641,6 @@ "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Jed collects stamp cards.'" }, - { - "count": 1, - "reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'" - }, { "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the " @@ -741,6 +737,10 @@ "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'There are some kids in camp.'" }, + { + "count": 1, + "reason": "candidate_graph: no admissible candidate for statement: 'They need to put all of their loose crayons in a box.'" + }, { "count": 1, "reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'" diff --git a/evals/math_capability_axes/G3_numerics/v1_1/__init__.py b/evals/math_capability_axes/G3_numerics/v1_1/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl b/evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl new file mode 100644 index 00000000..12ea9797 --- /dev/null +++ b/evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl @@ -0,0 +1,28 @@ +{"case_id":"g31-frac-01","class":"fraction_of_unit","problem":"Bob has 1/2 of a cup. How many cups does Bob have?","expected_answer":0.5,"expected_unit":"cups","expected_outcome":"solved_correct"} +{"case_id":"g31-frac-02","class":"fraction_of_unit","problem":"Sarah has 3/4 of a bag. How many bags does Sarah have?","expected_answer":0.75,"expected_unit":"bags","expected_outcome":"solved_correct"} +{"case_id":"g31-frac-03","class":"fraction_of_unit","problem":"Tom has 1/4 of a pie. How many pies does Tom have?","expected_answer":0.25,"expected_unit":"pies","expected_outcome":"solved_correct"} +{"case_id":"g31-frac-04","class":"fraction_of_unit","problem":"Sam has 3/2 of a liter. How many liters does Sam have?","expected_answer":1.5,"expected_unit":"liters","expected_outcome":"solved_correct"} +{"case_id":"g31-frac-05","class":"fraction_of_unit","problem":"Liam has 4/5 of a cup. How many cups does Liam have?","expected_answer":0.8,"expected_unit":"cups","expected_outcome":"solved_correct"} +{"case_id":"g31-cur-01","class":"multi_currency","problem":"Bob has ¢50. How many cents does Bob have?","expected_answer":50.0,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g31-cur-02","class":"multi_currency","problem":"Maria has €20. How many euros does Maria have?","expected_answer":20.0,"expected_unit":"euros","expected_outcome":"solved_correct"} +{"case_id":"g31-cur-03","class":"multi_currency","problem":"Kenji has ¥100. How many yen does Kenji have?","expected_answer":100.0,"expected_unit":"yen","expected_outcome":"solved_correct"} +{"case_id":"g31-cur-04","class":"multi_currency","problem":"Juan has ₱200. How many pesos does Juan have?","expected_answer":200.0,"expected_unit":"pesos","expected_outcome":"solved_correct"} +{"case_id":"g31-cur-05","class":"multi_currency","problem":"Maria has €30. Maria spends €10. How many euros does Maria have?","expected_answer":20.0,"expected_unit":"euros","expected_outcome":"solved_correct"} +{"case_id":"g31-mwc-01","class":"multi_word_cardinal","problem":"Bob has one hundred apples. How many apples does Bob have?","expected_answer":100.0,"expected_unit":"apples","expected_outcome":"solved_correct"} +{"case_id":"g31-mwc-02","class":"multi_word_cardinal","problem":"The store has one thousand books. How many books does the store have?","expected_answer":1000.0,"expected_unit":"books","expected_outcome":"solved_correct"} +{"case_id":"g31-mwc-03","class":"multi_word_cardinal","problem":"Anna has three hundred cookies. How many cookies does Anna have?","expected_answer":300.0,"expected_unit":"cookies","expected_outcome":"solved_correct"} +{"case_id":"g31-mwc-04","class":"multi_word_cardinal","problem":"Mike has two thousand five hundred marbles. How many marbles does Mike have?","expected_answer":2500.0,"expected_unit":"marbles","expected_outcome":"solved_correct"} +{"case_id":"g31-mwc-05","class":"multi_word_cardinal","problem":"Sam has five hundred dollars. How many cents does Sam have?","expected_answer":50000.0,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g31-adj-01","class":"word_num_adjective","problem":"Sam has five full boxes. How many boxes does Sam have?","expected_answer":5.0,"expected_unit":"boxes","expected_outcome":"solved_correct"} +{"case_id":"g31-adj-02","class":"word_num_adjective","problem":"Ella has three loose crayons. How many crayons does Ella have?","expected_answer":3.0,"expected_unit":"crayons","expected_outcome":"solved_correct"} +{"case_id":"g31-adj-03","class":"word_num_adjective","problem":"Bob has seven empty cans. How many cans does Bob have?","expected_answer":7.0,"expected_unit":"cans","expected_outcome":"solved_correct"} +{"case_id":"g31-adj-04","class":"word_num_adjective","problem":"Jane has twelve whole pies. How many pies does Jane have?","expected_answer":12.0,"expected_unit":"pies","expected_outcome":"solved_correct"} +{"case_id":"g31-adj-05","class":"word_num_adjective","problem":"Tom has eight new books. How many books does Tom have?","expected_answer":8.0,"expected_unit":"books","expected_outcome":"solved_correct"} +{"case_id":"g31-ref-pct-01","class":"refuse_percentage","problem":"Bob has 50% apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-pct-02","class":"refuse_percentage","problem":"Sam has 75% of a pie. How many pies does Sam have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-sci-01","class":"refuse_scientific_notation","problem":"Sam has 1e3 marbles. How many marbles does Sam have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-sci-02","class":"refuse_scientific_notation","problem":"Alice has 2.5e2 books. How many books does Alice have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-locale-01","class":"refuse_locale_separator","problem":"Alice has 1,000 pennies. How many pennies does Alice have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-locale-02","class":"refuse_locale_separator","problem":"Bob has 10,000 apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-3dec-01","class":"refuse_money_precision","problem":"Bob has $1.234. How many cents does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g31-ref-3dec-02","class":"refuse_money_precision","problem":"Sam has €5.678. How many euros does Sam have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} diff --git a/evals/math_capability_axes/G3_numerics/v1_1/report.json b/evals/math_capability_axes/G3_numerics/v1_1/report.json new file mode 100644 index 00000000..6c07dcef --- /dev/null +++ b/evals/math_capability_axes/G3_numerics/v1_1/report.json @@ -0,0 +1,367 @@ +{ + "adr": "0131.G.3.1", + "axis": "numerics_extensions", + "cases_path": "evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl", + "class_counts": { + "fraction_of_unit": 5, + "multi_currency": 5, + "multi_word_cardinal": 5, + "refuse_locale_separator": 2, + "refuse_money_precision": 2, + "refuse_percentage": 2, + "refuse_scientific_notation": 2, + "word_num_adjective": 5 + }, + "metrics": { + "cases_total": 28, + "correct_rate_on_positive_cases": 1.0, + "overall_pass": true, + "refused_as_expected": 8, + "solved_correct": 20, + "solved_wrong": 0, + "wrong_count_is_zero": true + }, + "per_case": [ + { + "actual_answer": 0.5, + "actual_outcome": "correct", + "actual_unit": "cups", + "case_id": "g31-frac-01", + "class": "fraction_of_unit", + "expected_answer": 0.5, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "b1b05ff2759cb6cf5d630d2396eb9f489d71043a3eb12a4c2cdd88062ad64848", + "verdict": "solved_correct" + }, + { + "actual_answer": 0.75, + "actual_outcome": "correct", + "actual_unit": "bags", + "case_id": "g31-frac-02", + "class": "fraction_of_unit", + "expected_answer": 0.75, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "f64e99dbc03193d143a151ac5c7e9e283ba9e4b7dfea676410cba2dcf92cba74", + "verdict": "solved_correct" + }, + { + "actual_answer": 0.25, + "actual_outcome": "correct", + "actual_unit": "pies", + "case_id": "g31-frac-03", + "class": "fraction_of_unit", + "expected_answer": 0.25, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "53c696733759f9c0c71d96fc004614d70494dabe0a63226ac73155b5942aabdb", + "verdict": "solved_correct" + }, + { + "actual_answer": 1.5, + "actual_outcome": "correct", + "actual_unit": "liters", + "case_id": "g31-frac-04", + "class": "fraction_of_unit", + "expected_answer": 1.5, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "9ff18f4ee0f7797e3972c9d5e088775b53acfaf28b622d65c8a88fa1309abc12", + "verdict": "solved_correct" + }, + { + "actual_answer": 0.8, + "actual_outcome": "correct", + "actual_unit": "cups", + "case_id": "g31-frac-05", + "class": "fraction_of_unit", + "expected_answer": 0.8, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "92e90758cc1ee3dab91dd8610dfed35b2c616e64bf95e2ed682467c9849eec07", + "verdict": "solved_correct" + }, + { + "actual_answer": 50.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g31-cur-01", + "class": "multi_currency", + "expected_answer": 50.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "490bbc2f0384acc8f94a2740f0e37572a6d2036692edd8b7ff77de80785cbbff", + "verdict": "solved_correct" + }, + { + "actual_answer": 20.0, + "actual_outcome": "correct", + "actual_unit": "euros", + "case_id": "g31-cur-02", + "class": "multi_currency", + "expected_answer": 20.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "d1da3f9a8af64633231a2157593e4d571de973f815801172ef8dac1c40e4bb1f", + "verdict": "solved_correct" + }, + { + "actual_answer": 100.0, + "actual_outcome": "correct", + "actual_unit": "yen", + "case_id": "g31-cur-03", + "class": "multi_currency", + "expected_answer": 100.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "c9a074741863a38a4d58fe4bfc17c18ff220b4172cf8541b3cc20d36be2dba9d", + "verdict": "solved_correct" + }, + { + "actual_answer": 200.0, + "actual_outcome": "correct", + "actual_unit": "pesos", + "case_id": "g31-cur-04", + "class": "multi_currency", + "expected_answer": 200.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "38597a8bef5b6e08134e3ca2448eb4199daec93a24e0358da932133a02938c36", + "verdict": "solved_correct" + }, + { + "actual_answer": 20.0, + "actual_outcome": "correct", + "actual_unit": "euros", + "case_id": "g31-cur-05", + "class": "multi_currency", + "expected_answer": 20.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "c313f6ad6fe05d09425942fb4e5022014cfd78fdd3d297d6a2349cc719181ac8", + "verdict": "solved_correct" + }, + { + "actual_answer": 100.0, + "actual_outcome": "correct", + "actual_unit": "apples", + "case_id": "g31-mwc-01", + "class": "multi_word_cardinal", + "expected_answer": 100.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "0023bf2b389b79528556e91cb984a0d6d22b3b9b95b61d3ae73414fa93ead560", + "verdict": "solved_correct" + }, + { + "actual_answer": 1000.0, + "actual_outcome": "correct", + "actual_unit": "books", + "case_id": "g31-mwc-02", + "class": "multi_word_cardinal", + "expected_answer": 1000.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "7c44c06806a79fd3d797d42b4311688bab66d8a62d7dd09606457faacaa11364", + "verdict": "solved_correct" + }, + { + "actual_answer": 300.0, + "actual_outcome": "correct", + "actual_unit": "cookies", + "case_id": "g31-mwc-03", + "class": "multi_word_cardinal", + "expected_answer": 300.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "e7a76500137f6f7936a904f6b916dc310defbbd8bdfc87727e98cdf7685187c0", + "verdict": "solved_correct" + }, + { + "actual_answer": 2500.0, + "actual_outcome": "correct", + "actual_unit": "marbles", + "case_id": "g31-mwc-04", + "class": "multi_word_cardinal", + "expected_answer": 2500.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "30b22bade97f050d89549af89c9d7512e69facab1346c584bb22c1e6c25e2ef7", + "verdict": "solved_correct" + }, + { + "actual_answer": 50000.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g31-mwc-05", + "class": "multi_word_cardinal", + "expected_answer": 50000.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "7ca5b0470f11ed2a806e95982211954202f99e48c707780431df2522f01b871f", + "verdict": "solved_correct" + }, + { + "actual_answer": 5.0, + "actual_outcome": "correct", + "actual_unit": "boxes", + "case_id": "g31-adj-01", + "class": "word_num_adjective", + "expected_answer": 5.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "80f43f99acfea39c9226f1a6bc46d56f950a799b698e4eb6bc4be086046828aa", + "verdict": "solved_correct" + }, + { + "actual_answer": 3.0, + "actual_outcome": "correct", + "actual_unit": "crayons", + "case_id": "g31-adj-02", + "class": "word_num_adjective", + "expected_answer": 3.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "46fcecab0fd1a1ab4db5b2bc3033a0d680f3919eef992714415cbdbcae5acc03", + "verdict": "solved_correct" + }, + { + "actual_answer": 7.0, + "actual_outcome": "correct", + "actual_unit": "cans", + "case_id": "g31-adj-03", + "class": "word_num_adjective", + "expected_answer": 7.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "0d10b1e8251b5298ed826c8c10526f0e28c54d443b6256c10c9be6f8190cab09", + "verdict": "solved_correct" + }, + { + "actual_answer": 12.0, + "actual_outcome": "correct", + "actual_unit": "pies", + "case_id": "g31-adj-04", + "class": "word_num_adjective", + "expected_answer": 12.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "74083e671f30421f0a968f1f8d4314f56b1a255854a757c6ab6f2eddafa74d0c", + "verdict": "solved_correct" + }, + { + "actual_answer": 8.0, + "actual_outcome": "correct", + "actual_unit": "books", + "case_id": "g31-adj-05", + "class": "word_num_adjective", + "expected_answer": 8.0, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "ba2b359492336fc2fcc71a0fde24f05a4fa446133c6530dc13bbce46ec577dc4", + "verdict": "solved_correct" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-pct-01", + "class": "refuse_percentage", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has 50% apples.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-pct-02", + "class": "refuse_percentage", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Sam has 75% of a pie.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-sci-01", + "class": "refuse_scientific_notation", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Sam has 1e3 marbles.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-sci-02", + "class": "refuse_scientific_notation", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Alice has 2.5e2 books.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-locale-01", + "class": "refuse_locale_separator", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Alice has 1,000 pennies.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-locale-02", + "class": "refuse_locale_separator", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has 10,000 apples.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-3dec-01", + "class": "refuse_money_precision", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has $1.234.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g31-ref-3dec-02", + "class": "refuse_money_precision", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Sam has \u20ac5.678.'", + "trace_hash": null, + "verdict": "refused" + } + ], + "schema_version": 1, + "verdict_counts": { + "refused": 8, + "solved_correct": 20 + } +} diff --git a/evals/math_capability_axes/G3_numerics/v1_1/runner.py b/evals/math_capability_axes/G3_numerics/v1_1/runner.py new file mode 100644 index 00000000..b829aa3e --- /dev/null +++ b/evals/math_capability_axes/G3_numerics/v1_1/runner.py @@ -0,0 +1,135 @@ +"""ADR-0131.G.3.1 — Numerics-extensions capability-axis runner (v1.1). + +Additive sibling to ``evals/math_capability_axes/G3_numerics/v1/``. +v1 is frozen as the audit-trail artifact for PR #183; v1.1 carries the +four axes deferred from G.3: + + 1. **Fractions end-to-end** — ``N/M of a `` initial possession. + 2. **Multi-currency** — ``¢``, ``€``, ``¥``, ``₱`` symbols. + (``£`` deferred to G.3.2: question extractor's single-token unit + slot cannot parse the two-word surface "pounds sterling".) + 3. **Multi-token space-separated cardinals** — ``one hundred``, + ``two thousand five hundred``. + 4. **Word-number-adjective** — ``five full boxes``. + +Runner interface is identical to v1 so the G3 axis lane CI check is +parameterisable over both versions. +""" + +from __future__ import annotations + +import json +from collections import Counter +from pathlib import Path +from typing import Any + +from evals.gsm8k_math.runner import _score_one_candidate_graph + +_HERE = Path(__file__).resolve().parent +_CASES_PATH = _HERE / "cases.jsonl" +_REPORT_PATH = _HERE / "report.json" + + +def _load_cases() -> list[dict[str, Any]]: + out: list[dict[str, Any]] = [] + for line in _CASES_PATH.read_text(encoding="utf-8").splitlines(): + if line.strip(): + out.append(json.loads(line)) + return out + + +def _adapt_case(raw: dict[str, Any]) -> dict[str, Any]: + return { + "id": raw["case_id"], + "problem": raw["problem"], + "expected_answer": float(raw["expected_answer"]), + "expected_unit": raw.get("expected_unit", ""), + } + + +def _classify(actual_outcome: str, expected_outcome: str) -> str: + if expected_outcome == "solved_correct" and actual_outcome == "correct": + return "solved_correct" + if expected_outcome == "refused" and actual_outcome == "refused": + return "refused" + return "solved_wrong" + + +def build_report() -> dict[str, Any]: + raw_cases = _load_cases() + case_results: list[dict[str, Any]] = [] + class_counts: Counter[str] = Counter() + verdict_counts: Counter[str] = Counter() + + for raw in raw_cases: + cls = raw["class"] + expected = raw["expected_outcome"] + class_counts[cls] += 1 + outcome = _score_one_candidate_graph(_adapt_case(raw)) + verdict = _classify(outcome.outcome, expected) + verdict_counts[verdict] += 1 + case_results.append({ + "case_id": raw["case_id"], + "class": cls, + "expected_outcome": expected, + "actual_outcome": outcome.outcome, + "verdict": verdict, + "expected_answer": raw["expected_answer"], + "actual_answer": outcome.actual_answer, + "actual_unit": outcome.actual_unit, + "reason": outcome.reason, + "trace_hash": outcome.trace_hash, + }) + + total = len(raw_cases) + correct = verdict_counts.get("solved_correct", 0) + wrong = verdict_counts.get("solved_wrong", 0) + refused_expected = verdict_counts.get("refused", 0) + positive_count = sum(1 for r in raw_cases if r["expected_outcome"] == "solved_correct") + correct_rate_on_positive = ( + correct / positive_count if positive_count else 0.0 + ) + + return { + "schema_version": 1, + "adr": "0131.G.3.1", + "axis": "numerics_extensions", + "cases_path": "evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl", + "metrics": { + "cases_total": total, + "solved_correct": correct, + "solved_wrong": wrong, + "refused_as_expected": refused_expected, + "wrong_count_is_zero": wrong == 0, + "correct_rate_on_positive_cases": correct_rate_on_positive, + "overall_pass": wrong == 0 and (correct + refused_expected == total), + }, + "class_counts": dict(sorted(class_counts.items())), + "verdict_counts": dict(sorted(verdict_counts.items())), + "per_case": case_results, + } + + +def write_report(report: dict[str, Any]) -> None: + _REPORT_PATH.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def main() -> int: + report = build_report() + write_report(report) + m = report["metrics"] + print(f"axis: {report['axis']}") + print(f"cases_total: {m['cases_total']}") + print(f"solved_correct: {m['solved_correct']}") + print(f"solved_wrong: {m['solved_wrong']} (gate: must be 0)") + print(f"refused_as_expected: {m['refused_as_expected']}") + print(f"correct_rate_on_positive_cases: {m['correct_rate_on_positive_cases']:.1%}") + print(f"overall_pass: {m['overall_pass']}") + return 0 if m["overall_pass"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/generate/math_candidate_parser.py b/generate/math_candidate_parser.py index fd593180..448090d3 100644 --- a/generate/math_candidate_parser.py +++ b/generate/math_candidate_parser.py @@ -114,8 +114,9 @@ _ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)" # Numeric value alternation. Listed longest-form-first so the regex # engine doesn't truncate on a shorter prefix: -# - Money symbol literal: ``$N`` or ``$N.NN`` (1-2 decimal places). -# ADR-0131.G.3. ``$N.NNN`` (3+ decimals) deliberately not matched +# - Money symbol literal: ``$N`` / ``$N.NN`` (1-2 decimal places) plus +# multi-currency symbols ``¢N`` ``€N`` ``£N`` ``¥N`` ``₱N``. +# ADR-0131.G.3.1. ``$N.NNN`` (3+ decimals) deliberately not matched # — refused as out-of-scope so wrong == 0 is preserved. # - Slash fraction literal: ``N/M``. Denominator-zero refused at # resolve time, not regex. @@ -123,7 +124,13 @@ _ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)" # Resolved via :func:`language_packs.numerics_loader.parse_compound_cardinal`. # - Digit run. # - Single-word cardinal (legacy ``WORD_NUMBERS`` set). -_MONEY_SYMBOL: Final[str] = r"\$\d+(?:\.\d{1,2})?" + +# ADR-0131.G.3.1: multi-currency symbol group. ¢ and $ are the only +# non-decimal currencies (sub-unit is the unit itself for ¢; $ converts +# to cents). €, £, ₱ admit 1-2 decimal places; ¥ is integer-only. +_MONEY_SYMBOL: Final[str] = ( + r"(?:\$\d+(?:\.\d{1,2})?|¢\d+|€\d+(?:\.\d{1,2})?|£\d+(?:\.\d{1,2})?|¥\d+|₱\d+(?:\.\d{1,2})?)" +) _SLASH_FRACTION: Final[str] = r"\d+/\d+" _HYPHENATED_CARDINAL: Final[str] = r"[A-Za-z]+-[A-Za-z]+" _WORD_NUM_OPTIONS: Final[str] = "|".join( @@ -160,6 +167,10 @@ _INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile( # (``$40``) carry their unit implicitly (``cent``); a missing unit # slot is admissible IFF the value resolves with a unit override. # Non-money values without a unit slot are refused at resolve time. + # ADR-0131.G.3.1 axis 4: optional adjective between value and unit + # ("five full boxes" — adjective 'full' is consumed and discarded; + # the unit head noun 'boxes' becomes the unit slot). + r"(?:\s+(?:full|loose|empty|whole|broken|new|old|small|large|fresh|raw|flat))?" r"(?:\s+(?P\w+))?" # ADR-0127 substance qualifier: "Sam has 5 feet of rope" — the # 'of ' tail is grammatically real but arithmetically inert. @@ -186,6 +197,45 @@ _INITIAL_THERE_ARE_RE: Final[re.Pattern[str]] = re.compile( flags=re.IGNORECASE, ) +# ADR-0131.G.3.1 — Axis 1: fraction-of-unit initial possession. +# "Bob has 3/4 of a cup." — the fraction is the value; "of a/an " +# carries the unit. The main _INITIAL_HAS_RE treats "of " as a +# discardable substance qualifier and emits no candidate (unit slot absent +# and no unit_override); this separate pattern extracts the unit from +# the "of" phrase explicitly. +_INITIAL_FRACTION_OF_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P{_ENTITY})\s+" + rf"(?Phas|have)\s+" + rf"(?P{_SLASH_FRACTION})\s+" + r"of\s+(?:a\s+|an\s+)?(?P\w+)" + r"(?:\s+of\s+.+)?" # optional further substance qualifier + r"\s*\.?$" +) + +# ADR-0131.G.3.1 — Axis 3: multi-token space-separated cardinal. +# "Bob has one hundred apples." — parse_compound_cardinal already handles +# the value; this pattern captures it before the unit-slot boundary. +# Approach (a) chosen over (b) (_VALUE widening) because greedy cardinal- +# word matching inside _VALUE would span the unit slot and require +# look-ahead unwinding; a separate dedicated extractor is narrower and +# leaves _VALUE unchanged for all other paths. +# Build cardinal-word alternation from the WORD_NUMBERS table. +_CARDINAL_WORD_OPTIONS: Final[str] = "|".join( + re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True) +) +# At least two cardinal words (single-word is handled by _VALUE/_resolve_value). +_MULTI_WORD_CARDINAL_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P{_ENTITY})\s+" + rf"(?Phas|have)\s+" + rf"(?P(?:{_CARDINAL_WORD_OPTIONS})(?:\s+(?:{_CARDINAL_WORD_OPTIONS}))+)" + # Optional adjective (axis 4 compound) between cardinal and unit. + r"(?:\s+(?:full|loose|empty|whole|broken|new|old|small|large|fresh|raw|flat))?" + r"\s+(?P\w+)" + r"(?:\s+(?:of|in|for|with)\s+.+)?" + r"\s*\.?$", + flags=re.IGNORECASE, +) + def _normalize_entity(raw: str) -> str: """Collapse whitespace + lowercase article. Mirrors math_parser @@ -216,6 +266,46 @@ class _ResolvedValue: # ``canonical_unit`` for the ``money`` dimension is ``cent``). _MONEY_UNIT: Final[str] = "cents" +# ADR-0131.G.3.1: multi-currency symbol → (unit_surface, factor_to_unit). +# ``factor_to_unit`` is the multiplier applied to the face value to +# produce the canonical unit. For USD ($): face is dollars → *100 cents. +# For ¢: face is already cents → *1. For all others the pack has no +# sub-unit defined, so face == canonical (factor=1) and the unit is the +# pack's plural surface form. +_CURRENCY_SYMBOLS: Final[dict[str, tuple[str, float]]] = { + "$": ("cents", 100.0), # dollar → 100 cents + "¢": ("cents", 1.0), # cent already canonical + "€": ("euros", 1.0), + "£": ("pounds sterling", 1.0), + "¥": ("yen", 1.0), + "₱": ("pesos", 1.0), +} + + +def _resolve_currency(t: str) -> _ResolvedValue | None: + """Resolve a currency-symbol value token (``$N``, ``¢N``, ``€N.NN``, …). + + Returns ``None`` when the format is out-of-scope (e.g. 3+ decimal places). + Yen (``¥``) is integer-only (no sub-unit in en_units_v1). + """ + for sym, (unit_surface, factor) in _CURRENCY_SYMBOLS.items(): + if not t.startswith(sym): + continue + body = t[len(sym):] + if re.fullmatch(r"\d+", body): + raw_val = int(body) + final = int(raw_val * factor) if factor == int(factor) else raw_val * factor + return _ResolvedValue(final, unit_surface) + # ¥ is integer-only. + if sym == "¥": + return None + if re.fullmatch(r"\d+\.\d{1,2}", body): + raw_val = float(body) + result = raw_val * factor + return _ResolvedValue(int(round(result)) if factor != 1.0 else raw_val, unit_surface) + return None # 3+ decimals refused for all currency symbols + return None + def _resolve_value(value_token: str) -> _ResolvedValue | None: """Resolve a value-slot token into a numeric value + optional unit @@ -230,15 +320,9 @@ def _resolve_value(value_token: str) -> _ResolvedValue | None: if not value_token: return None t = value_token.strip() - # Money symbol literal: $N or $N.NN. - if t.startswith("$"): - body = t[1:] - if re.fullmatch(r"\d+", body): - return _ResolvedValue(int(body) * 100, _MONEY_UNIT) - if re.fullmatch(r"\d+\.\d{1,2}", body): - # round() avoids float drift: $2.50 → 250, not 249 or 251. - return _ResolvedValue(int(round(float(body) * 100)), _MONEY_UNIT) - return None # $N.NNN (3+ decimals) refused — out-of-scope. + # Multi-currency symbols (ADR-0131.G.3.1): $, ¢, €, £, ¥, ₱. + if t and t[0] in _CURRENCY_SYMBOLS: + return _resolve_currency(t) # Slash fraction literal: N/M with M > 0. if "/" in t: m = re.fullmatch(r"(\d+)/(\d+)", t) @@ -292,19 +376,28 @@ def _is_indefinite_quantifier(token: str) -> bool: def _money_unit_normalization( value: int | float, unit: str | None ) -> tuple[int | float, str | None]: - """ADR-0131.G.3 — normalize ``dollar``/``dollars`` surface unit to the - canonical money unit (``cent``). + """ADR-0131.G.3 — normalize money word-form surface units to pack canonical. ``en_units_v1`` pins ``cent`` as ``canonical_unit`` for the ``money`` - dimension; ``dollar`` is convenience surface. A ``dollar`` value is - 100 ``cent``. Done at the candidate-build site so every money-bearing - path normalizes uniformly (Quantity equality is exact — mixing - ``cent`` and ``dollar`` units would silently break arithmetic). + dimension. ``dollar``/``dollars`` → 100 cents each. Other currencies + (ADR-0131.G.3.1) are already in canonical form when they arrive via + ``_resolve_currency``; this helper normalizes the word-form paths. """ if unit is None: return value, unit - if unit.lower() in ("dollar", "dollars"): + lower = unit.lower() + if lower in ("dollar", "dollars"): return value * 100, _MONEY_UNIT + # Euro/pound-sterling/yen/peso word forms: already canonical (factor=1). + # These enter via unit slot (word form) rather than symbol — pass through. + if lower in ("euro", "euros"): + return value, "euros" + if lower in ("pound sterling", "pounds sterling"): + return value, "pounds sterling" + if lower == "yen": + return value, "yen" + if lower in ("peso", "pesos"): + return value, "pesos" return value, unit @@ -362,6 +455,14 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]: ) ) + # ADR-0131.G.3.1 — Axis 1: fraction-of-unit shape. + # "Bob has 3/4 of a cup." — separate regex extracts unit from "of" phrase. + out.extend(_fraction_of_candidates(sentence)) + + # ADR-0131.G.3.1 — Axis 3: multi-token space-separated cardinals. + # "Bob has one hundred apples." — separate extractor; _VALUE is unchanged. + out.extend(_multi_word_cardinal_candidates(sentence)) + # ADR-0131.G.4 — multi-clause initial-state extractors. # Each may emit ≥1 candidates; deterministic order: conjoined-subject-each, # conjoined-object, embedded-quantifier, conjoined-embedded-quantifier. @@ -937,8 +1038,11 @@ def _compare_multiplicative_candidates(sentence: str) -> list[CandidateOperation if _is_indefinite_quantifier(value_raw): return out try: - factor = float(_resolve_value(value_raw)) - except KeyError: + _rv = _resolve_value(value_raw) + factor = float(_rv.value) if _rv is not None else None + except (KeyError, TypeError): + return out + if factor is None: return out cand = _build_compare_multiplicative( actor_raw=m.group("actor"), @@ -995,8 +1099,9 @@ def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]: factor_value_raw = m.group("factor_value") if not _is_indefinite_quantifier(factor_value_raw): try: - factor = float(_resolve_value(factor_value_raw)) - except KeyError: + _rv2 = _resolve_value(factor_value_raw) + factor = float(_rv2.value) if _rv2 is not None else None + except (KeyError, TypeError): factor = None if factor is not None: mult_cand = _build_compare_multiplicative( @@ -1015,6 +1120,88 @@ def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]: return out +# --------------------------------------------------------------------------- +# ADR-0131.G.3.1 — Axis 1 + Axis 3 extractor functions +# --------------------------------------------------------------------------- + +def _fraction_of_candidates(sentence: str) -> list[CandidateInitial]: + """Axis 1 (fractions): 'Bob has 3/4 of a cup.' → value=0.75, unit='cups'. + + The main _INITIAL_HAS_RE treats 'of ' as a discardable substance + qualifier and cannot fill the unit slot from it. This extractor uses + _INITIAL_FRACTION_OF_RE to explicitly capture the unit after 'of'. + """ + s = sentence.strip().rstrip(".") + m = _INITIAL_FRACTION_OF_RE.match(s) + if m is None: + return [] + value_raw = m.group("value") + rv = _resolve_value(value_raw) + if rv is None: + return [] + unit_raw = m.group("unit") + unit = _canonicalize_unit(unit_raw) + entity = _normalize_entity(m.group("entity")) + try: + return [ + CandidateInitial( + initial=InitialPossession( + entity=entity, + quantity=Quantity(value=rv.value, unit=unit), + ), + source_span=sentence, + matched_anchor=m.group("anchor"), + matched_value_token=value_raw, + matched_unit_token=unit_raw, + matched_entity_token=m.group("entity"), + ) + ] + except Exception: + return [] + + +def _multi_word_cardinal_candidates(sentence: str) -> list[CandidateInitial]: + """Axis 3 (multi-word cardinals): 'Bob has one hundred apples.' + + Approach (a): dedicated extractor leaving _VALUE unchanged. The value + group captures the full space-separated cardinal sequence; the unit + slot is the next word token after the cardinal sequence (and optional + adjective). + """ + s = sentence.strip().rstrip(".") + m = _MULTI_WORD_CARDINAL_RE.match(s) + if m is None: + return [] + value_raw = m.group("value") + from language_packs.numerics_loader import parse_compound_cardinal + parsed = parse_compound_cardinal(value_raw) + if parsed is None: + return [] + unit_raw = m.group("unit") + value_n, unit_n = _money_unit_normalization(parsed, _canonicalize_unit(unit_raw)) + if unit_n is None: + return [] + entity = _normalize_entity(m.group("entity")) + try: + return [ + CandidateInitial( + initial=InitialPossession( + entity=entity, + quantity=Quantity(value=value_n, unit=unit_n), + ), + source_span=sentence, + matched_anchor=m.group("anchor"), + # Provenance: use the first cardinal word as the value token + # for grounding (all cardinal words are in the source span). + matched_value_token=value_raw.split()[0], + matched_unit_token=unit_raw, + matched_entity_token=m.group("entity"), + ) + ] + except Exception: + return [] + + # --------------------------------------------------------------------------- # ADR-0131.G.4 — Multi-clause initial-state composition # --------------------------------------------------------------------------- @@ -1138,7 +1325,10 @@ def _conj_subject_each_candidates(sentence: str) -> list[CandidateInitial]: entity_b = _normalize_entity(m.group("b")) if entity_a == entity_b: return [] # 'Aaron and Aaron each ...' is degenerate - value = _resolve_value(value_raw) + _rv_conj = _resolve_value(value_raw) + if _rv_conj is None: + return [] + value = _rv_conj.value unit_raw = m.group("unit") unit = _canonicalize_unit(unit_raw) anchor = _canon_verb_to_anchor(m.group("verb")) @@ -1192,7 +1382,7 @@ def _conj_object_candidates(sentence: str) -> list[CandidateInitial]: CandidateInitial( initial=InitialPossession( entity=entity, - quantity=Quantity(value=_resolve_value(value_raw), unit=unit), + quantity=Quantity(value=_resolve_value(value_raw).value, unit=unit), # type: ignore[union-attr] ), source_span=sentence, matched_anchor=anchor, @@ -1233,9 +1423,11 @@ def _embedded_quantifier_candidates(sentence: str) -> list[CandidateInitial]: c2 = container2_raw.lower() if c2 not in (container, container.rstrip("s"), container + "s"): return [] - n = _resolve_value(n_raw) - per = _resolve_value(m_raw) - total = n * per + _rv_n = _resolve_value(n_raw) + _rv_per = _resolve_value(m_raw) + if _rv_n is None or _rv_per is None: + return [] + total = _rv_n.value * _rv_per.value entity = _normalize_entity(m.group("entity")) unit_raw = m.group("unit") unit = _canonicalize_unit(unit_raw) @@ -1314,9 +1506,13 @@ def _build_conj_embedded_sum( if u1 != u2: # Mixed-unit sum is meaningless; refuse. return [] - total = _resolve_value(n1_raw) * _resolve_value(m1_raw) + ( - _resolve_value(n2_raw) * _resolve_value(m2_raw) - ) + _n1 = _resolve_value(n1_raw) + _m1 = _resolve_value(m1_raw) + _n2 = _resolve_value(n2_raw) + _m2 = _resolve_value(m2_raw) + if _n1 is None or _m1 is None or _n2 is None or _m2 is None: + return [] + total = _n1.value * _m1.value + _n2.value * _m2.value entity = _normalize_entity(m.group("entity")) try: return [ diff --git a/generate/math_roundtrip.py b/generate/math_roundtrip.py index 0db7bda0..a03985f3 100644 --- a/generate/math_roundtrip.py +++ b/generate/math_roundtrip.py @@ -285,14 +285,35 @@ def _unit_grounds( on the raw source span rather than the token set. Similarly for ``dollar``: an author may write either ``$N`` or ``N dollars``; both ground a money unit. + + ADR-0131.G.3.1 widening: multi-currency symbols (¢ € £ ¥ ₱) each + ground their respective canonical unit when their symbol appears in + the raw source span. """ if _token_in(unit_token, haystack_tokens): return True - if unit_token.lower() in ("cent", "cents"): - if "$" in source_span: + lower = unit_token.lower() + if lower in ("cent", "cents"): + if "$" in source_span or "¢" in source_span: return True if "dollar" in haystack_tokens or "dollars" in haystack_tokens: return True + if lower in ("euro", "euros"): + if "€" in source_span: + return True + # "pounds sterling" is a two-word unit; check both the multi-word + # surface and the raw symbol. + if lower in ("pound sterling", "pounds sterling"): + if "£" in source_span: + return True + if "sterling" in haystack_tokens: + return True + if lower == "yen": + if "¥" in source_span: + return True + if lower in ("peso", "pesos"): + if "₱" in source_span: + return True return False @@ -319,9 +340,11 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool: every component lemma is a token (the tokenizer splits on hyphens), OR the compound's integer value's digit form appears. """ - # ADR-0131.G.3 widenings (handled first; the trailing existing path - # would never recognize these surface shapes). - if value_token.startswith("$"): + # ADR-0131.G.3 / G.3.1 widenings (handled first; the trailing existing + # path would never recognize these surface shapes). + # Currency symbol literals: extract digit parts, verify each in source. + _CURRENCY_SYM_SET = frozenset({"$", "¢", "€", "£", "¥", "₱"}) + if value_token and value_token[0] in _CURRENCY_SYM_SET: body = value_token[1:] parts = [p for p in body.split(".") if p] return bool(parts) and all(p in haystack_tokens for p in parts) diff --git a/tests/test_adr_0131_G31_numerics_extensions.py b/tests/test_adr_0131_G31_numerics_extensions.py new file mode 100644 index 00000000..91190731 --- /dev/null +++ b/tests/test_adr_0131_G31_numerics_extensions.py @@ -0,0 +1,251 @@ +"""ADR-0131.G.3.1 — Numerics extensions test suite. + +Per-axis at-least-one passing test, refusal probes, wrong==0 invariant, +replay byte-equality, and parent v1 lane regression gate. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from evals.gsm8k_math.runner import _score_one_candidate_graph +from generate.math_candidate_parser import ( + _resolve_value, + extract_initial_candidates, +) + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _score(problem: str, expected_answer: float, expected_unit: str) -> str: + r = _score_one_candidate_graph({ + "id": "test", + "problem": problem, + "expected_answer": expected_answer, + "expected_unit": expected_unit, + }) + return r.outcome + + +def _refused(problem: str) -> bool: + r = _score_one_candidate_graph({ + "id": "test", + "problem": problem, + "expected_answer": 0.0, + "expected_unit": "", + }) + return r.outcome == "refused" + + +# --------------------------------------------------------------------------- +# Axis 1: Fractions end-to-end +# --------------------------------------------------------------------------- + +class TestFractions: + def test_half_of_cup(self): + assert _score("Bob has 1/2 of a cup. How many cups does Bob have?", 0.5, "cups") == "correct" + + def test_three_quarters_of_bag(self): + assert _score("Sarah has 3/4 of a bag. How many bags does Sarah have?", 0.75, "bags") == "correct" + + def test_quarter_of_pie(self): + assert _score("Tom has 1/4 of a pie. How many pies does Tom have?", 0.25, "pies") == "correct" + + def test_improper_fraction(self): + assert _score("Sam has 3/2 of a liter. How many liters does Sam have?", 1.5, "liters") == "correct" + + def test_resolve_value_fraction(self): + rv = _resolve_value("3/4") + assert rv is not None + assert abs(rv.value - 0.75) < 1e-9 + assert rv.unit_override is None + + def test_fraction_zero_denominator_refused(self): + assert _refused("Bob has 5/0 apples. How many apples does Bob have?") + + def test_extract_initial_fraction_of(self): + cands = extract_initial_candidates("Bob has 1/2 of a cup.") + assert len(cands) > 0 + q = cands[0].initial.quantity + assert abs(q.value - 0.5) < 1e-9 + assert q.unit == "cups" + + +# --------------------------------------------------------------------------- +# Axis 2: Multi-currency +# --------------------------------------------------------------------------- + +class TestMultiCurrency: + def test_cent_symbol(self): + assert _score("Bob has ¢50. How many cents does Bob have?", 50.0, "cents") == "correct" + + def test_euro_symbol(self): + assert _score("Maria has €20. How many euros does Maria have?", 20.0, "euros") == "correct" + + def test_yen_symbol(self): + assert _score("Kenji has ¥100. How many yen does Kenji have?", 100.0, "yen") == "correct" + + def test_peso_symbol(self): + assert _score("Juan has ₱200. How many pesos does Juan have?", 200.0, "pesos") == "correct" + + def test_euro_with_operation(self): + assert _score("Maria has €30. Maria spends €10. How many euros does Maria have?", 20.0, "euros") == "correct" + + def test_resolve_cent_symbol(self): + rv = _resolve_value("¢50") + assert rv is not None and rv.value == 50 and rv.unit_override == "cents" + + def test_resolve_euro_symbol(self): + rv = _resolve_value("€20") + assert rv is not None and rv.value == 20 and rv.unit_override == "euros" + + def test_resolve_yen_integer_only(self): + # ¥ is integer-only; decimal form should be refused + rv = _resolve_value("¥100") + assert rv is not None and rv.value == 100 and rv.unit_override == "yen" + + def test_euro_three_decimal_refused(self): + assert _refused("Sam has €5.678. How many euros does Sam have?") + + def test_pound_sterling_deferred(self): + # £ symbol parses and resolves but question extractor cannot parse + # multi-word unit 'pounds sterling' — deferred to G.3.2. + rv = _resolve_value("£15") + assert rv is not None and rv.value == 15 and rv.unit_override == "pounds sterling" + + +# --------------------------------------------------------------------------- +# Axis 3: Multi-token space-separated cardinals +# --------------------------------------------------------------------------- + +class TestMultiWordCardinals: + def test_one_hundred(self): + assert _score("Bob has one hundred apples. How many apples does Bob have?", 100.0, "apples") == "correct" + + def test_one_thousand(self): + assert _score("The store has one thousand books. How many books does the store have?", 1000.0, "books") == "correct" + + def test_three_hundred(self): + assert _score("Anna has three hundred cookies. How many cookies does Anna have?", 300.0, "cookies") == "correct" + + def test_two_thousand_five_hundred(self): + assert _score("Mike has two thousand five hundred marbles. How many marbles does Mike have?", 2500.0, "marbles") == "correct" + + def test_five_hundred_dollars_to_cents(self): + assert _score("Sam has five hundred dollars. How many cents does Sam have?", 50000.0, "cents") == "correct" + + def test_extract_multi_word_cardinal(self): + cands = extract_initial_candidates("Bob has one hundred apples.") + assert len(cands) > 0 + q = cands[0].initial.quantity + assert q.value == 100 + assert q.unit == "apples" + + +# --------------------------------------------------------------------------- +# Axis 4: Word-number-adjective compositions +# --------------------------------------------------------------------------- + +class TestWordNumAdjective: + def test_five_full_boxes(self): + assert _score("Sam has five full boxes. How many boxes does Sam have?", 5.0, "boxes") == "correct" + + def test_three_loose_crayons(self): + assert _score("Ella has three loose crayons. How many crayons does Ella have?", 3.0, "crayons") == "correct" + + def test_seven_empty_cans(self): + assert _score("Bob has seven empty cans. How many cans does Bob have?", 7.0, "cans") == "correct" + + def test_twelve_whole_pies(self): + assert _score("Jane has twelve whole pies. How many pies does Jane have?", 12.0, "pies") == "correct" + + def test_eight_new_books(self): + assert _score("Tom has eight new books. How many books does Tom have?", 8.0, "books") == "correct" + + def test_extract_adjective_initial(self): + cands = extract_initial_candidates("Sam has five full boxes.") + assert len(cands) > 0 + q = cands[0].initial.quantity + assert q.value == 5 + assert q.unit == "boxes" + + +# --------------------------------------------------------------------------- +# Refusal probes — closed-set boundary enforcement +# --------------------------------------------------------------------------- + +class TestRefusals: + def test_percentage_refused(self): + assert _refused("Bob has 50% apples. How many apples does Bob have?") + + def test_percentage_of_refused(self): + assert _refused("Sam has 75% of a pie. How many pies does Sam have?") + + def test_scientific_notation_refused(self): + assert _refused("Sam has 1e3 marbles. How many marbles does Sam have?") + + def test_scientific_notation_float_refused(self): + assert _refused("Alice has 2.5e2 books. How many books does Alice have?") + + def test_locale_separator_refused(self): + assert _refused("Alice has 1,000 pennies. How many pennies does Alice have?") + + def test_locale_separator_large_refused(self): + assert _refused("Bob has 10,000 apples. How many apples does Bob have?") + + def test_three_decimal_dollar_refused(self): + assert _refused("Bob has $1.234. How many cents does Bob have?") + + def test_three_decimal_euro_refused(self): + assert _refused("Sam has €5.678. How many euros does Sam have?") + + +# --------------------------------------------------------------------------- +# Wrong == 0 invariant (load-bearing gate per ADR-0114a Obligation #4) +# --------------------------------------------------------------------------- + +class TestWrongEqualsZero: + def test_v1_1_wrong_is_zero(self): + from evals.math_capability_axes.G3_numerics.v1_1.runner import build_report + report = build_report() + assert report["metrics"]["solved_wrong"] == 0, ( + f"solved_wrong must be 0; got {report['metrics']['solved_wrong']}" + ) + + def test_v1_1_overall_pass(self): + from evals.math_capability_axes.G3_numerics.v1_1.runner import build_report + report = build_report() + assert report["metrics"]["overall_pass"] is True + + +# --------------------------------------------------------------------------- +# Replay byte-equality +# --------------------------------------------------------------------------- + +class TestReplayByteEquality: + def test_report_byte_equal_across_two_runs(self): + from evals.math_capability_axes.G3_numerics.v1_1.runner import build_report + r1 = json.dumps(build_report(), indent=2, sort_keys=True) + r2 = json.dumps(build_report(), indent=2, sort_keys=True) + assert r1 == r2, "v1.1 report must be byte-equal across runs" + + +# --------------------------------------------------------------------------- +# Parent v1 lane regression (no regression from G.3 changes) +# --------------------------------------------------------------------------- + +class TestParentV1Regression: + def test_v1_wrong_still_zero(self): + from evals.math_capability_axes.G3_numerics.v1.runner import build_report + report = build_report() + assert report["metrics"]["solved_wrong"] == 0 + + def test_v1_overall_pass(self): + from evals.math_capability_axes.G3_numerics.v1.runner import build_report + report = build_report() + assert report["metrics"]["overall_pass"] is True