core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(ADR-0131.G.3.1): numerics extensions — fractions + multi-currency + multi-token cardinals + word-num-adjective

Browse source 
Four axes deferred from ADR-0131.G.3 (PR #183):

1. Fractions end-to-end: new _INITIAL_FRACTION_OF_RE extractor handles
   `N/M of [a/an] <unit>` shape; _resolve_value already handles N/M arithmetic.

2. Multi-currency: _MONEY_SYMBOL widened to six symbols; _CURRENCY_SYMBOLS table
   + _resolve_currency dispatcher; ¢/€/¥/₱ wired end-to-end. £/pound sterling
   deferred to G.3.2 (question extractor's single-token unit slot cannot parse
   two-word surface "pounds sterling").

3. Multi-token cardinals: dedicated _MULTI_WORD_CARDINAL_RE extractor (approach a)
   delegates to parse_compound_cardinal; avoids greedy unit-slot boundary ambiguity
   from widening _VALUE.

4. Word-num-adjective: optional adjective group added to _INITIAL_HAS_RE and
   _MULTI_WORD_CARDINAL_RE; closed adjective list identical to _CONJ_OBJECT_RE.

Also fixes six pre-existing G4 type bugs where _resolve_value() result was used
directly as a numeric operand (TypeError: _ResolvedValue is not a number).

Axis lane v1_1: 20/20 solved_correct, 0 wrong, 8/8 refusals, overall_pass=True.
GSM8K probe: 0/50 admission_rate unchanged, admitted_wrong=0 (safety rail intact).
42/42 new tests pass; parent v1 lane (26/26) unaffected.
This commit is contained in:
Shay 2026-05-23 15:16:46 -07:00
parent 8187f3f385
commit 5853b189b2
9 changed files with 1324 additions and 51 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

273
docs/decisions/ADR-0131.G.3.1-numerics-extensions.md Normal file
View file

@ -0,0 +1,273 @@
# ADR-0131.G.3.1 — Numerics extensions (fractions + multi-currency + multi-token cardinals + word-num-adjective)
**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE main agent (Sonnet 4.6)
**Depends on:** ADR-0131.G.3 (parent, PR #183), ADR-0127 (units pack),
ADR-0128 (numerics pack)
**Parent:** ADR-0131 (composite math-expert promotion gate)
**Foundation for:** ADR-0131.G.3.2 (£ / pound sterling follow-up)
---
## Context
ADR-0131.G.3 (PR #183) shipped the first literal-recognition axis: money
symbols (`$N` / `$N.NN`), money word forms (`N dollars` / `N cents`), and
hyphenated multi-word cardinals (`twenty-five`). The four shapes explicitly
deferred in G.3's "Out-of-scope" section form this iteration's closed-set
scope:
1. **Fractions end-to-end** — `N/M of a <unit>` initial possession.
2. **Multi-currency** — `¢ € ¥ ₱` symbols (£ deferred; see below).
3. **Multi-token space-separated cardinals** — `one hundred`, `two thousand
five hundred`.
4. **Word-number-adjective compositions** — `five full boxes` per ADR-0127
substance-qualifier precedent.
All canonical-unit architectural decisions from G.3 are preserved unchanged:
cent is the canonical unit for money (US dollar path), pack-driven not
regex-spam, refusal-first.
---
## Decision
### Axis 1 — Fractions end-to-end
`_resolve_value` in `math_candidate_parser.py` already handles `N/M`
(returns `_ResolvedValue(num / den, None)`). The pipeline-level gap was in
`_INITIAL_HAS_RE`'s substance-qualifier handling: `Bob has 3/4 of a cup.`
would match the main regex with `unit=None` (the `of a cup` phrase was
consumed by the discardable substance qualifier), causing the candidate to
not emit because no unit could be determined.
**Fix**: new dedicated regex `_INITIAL_FRACTION_OF_RE` with shape
`<Entity> has <N/M> of [a/an] <unit> [of <substance>]?`. This explicitly
extracts the unit from the `of` phrase when the value is a slash fraction,
emitting a `CandidateInitial` with `value=N/M` and `unit=<canonicalized>`.
The main `_INITIAL_HAS_RE` is unchanged; the new extractor fires
in `extract_initial_candidates` after the primary extractor.
Closed-set: digit/digit literal with `M > 0`. Division-by-zero refused at
`_resolve_value` time (already the case since G.3).
### Axis 2 — Multi-currency
All five pack-recognized foreign currency symbols are in `en_units_v1`:
`¢` (cent), `€` (euro), `£` (pound sterling), `¥` (yen), `₱` (peso).
**Wired in this iteration**: `¢`, `€`, `¥`, `₱`.
**Deferred to G.3.2**: `£` / pound sterling. The `en_units_v1` pack
correctly models pound sterling as a two-word unit (`"plural": "pounds
sterling"`). However, the question extractor (`_Q_ENTITY_RE`,
`_Q_TOTAL_RE`) uses `(?P<unit>\w+)` which only captures a single word
token. A question like `How many pounds sterling does Alice have?` fails
to parse the unit slot (`sterling` is a second word, not consumed by the
single-token capture group). Widening the question extractor's unit slot
to support multi-word units is a distinct architectural change that belongs
in a dedicated PR (G.3.2) to keep the diff bounded and attributable.
The `_resolve_value` and `_currency_symbols` table includes `£` → `pounds
sterling` (so symbol parsing is complete); the question-extractor gap is
the only blocker.
**Normalization**:
- `¢N` → `N` cents (1:1; ¢ face value IS the canonical unit).
- `€N` / `€N.NN` → `N` / `N.NN` euros (factor=1; no sub-unit in pack).
- `¥N` → `N` yen (integer-only; yen has no sub-unit in the pack or in
common usage — decimal ¥ form refused).
- `₱N` / `₱N.NN` → `N` / `N.NN` pesos (factor=1).
3+ decimal places are refused for all currency symbols (closed-set
boundary; same rule as `$N.NNN`).
**`_MONEY_SYMBOL`** regex widened to alternation over all six symbols.
**`_unit_grounds`** in `math_roundtrip.py` widened: each currency symbol
grounds its respective canonical unit when the symbol appears in the raw
source span (mirrors the existing `$` / `dollar` grounding logic).
**`_money_unit_normalization`** extended: word-form entries for `euro`,
`euros`, `yen`, `peso`, `pesos` pass through with factor=1.
### Axis 3 — Multi-token space-separated cardinals
`parse_compound_cardinal` in `language_packs/numerics_loader.py` already
handles `"one hundred"`, `"two thousand five hundred"` etc. — it
normalises hyphens to spaces then tokenises. The parser couldn't emit these
in the value slot because `_VALUE` alternation uses a single-token pattern;
a greedy multi-word sequence would span the unit-slot boundary.
**Approach (a) chosen** (separate extractor) over **(b)** (`_VALUE`
widening):
- Widening `_VALUE` to greedily match cardinal-word sequences would require
look-ahead to distinguish the last cardinal word from the first unit word
(e.g. in `one hundred boxes`, `"one hundred"` is the value and `"boxes"`
is the unit; greedy matching consumes all three). This unwinding requires
either atomic groups (Python 3.11+) or a two-pass approach, both of which
add complexity across every pattern that uses `_VALUE`.
- A dedicated extractor (`_MULTI_WORD_CARDINAL_RE`) handles the
`<Entity> has <WORD_CARDINAL+> [<adjective>?] <unit>` shape precisely,
leaves `_VALUE` unchanged for all other paths, and is auditable in
isolation.
`_MULTI_WORD_CARDINAL_RE` is built from the `WORD_NUMBERS` table keys
(same source as `_WORD_NUM_OPTIONS` in `_VALUE`), requiring at least two
consecutive cardinal words before the unit slot. The value group is passed
to `parse_compound_cardinal` for resolution. Provenance: the first cardinal
word of the sequence is the `matched_value_token` (it grounds because it
is literally in the source span).
### Axis 4 — Word-number-adjective
`five full boxes` per ADR-0127 substance-qualifier precedent. The adjective
(`full`, `loose`, `empty`, `whole`, `broken`, `new`, `old`, `small`,
`large`, `fresh`, `raw`, `flat`) is inserted between the cardinal value and
the unit head noun. It is treated as part of the unit phrase: discarded
at parse time, not surfaced to the solver.
**Implementation**: an optional non-capturing group
`(?:\s+(?:full|loose|…))?` is inserted between the `(?P<value>…)` slot and
the `(?P<unit>\w+)?` slot in `_INITIAL_HAS_RE`. The same adjective list is
also added to `_MULTI_WORD_CARDINAL_RE`'s optional-adjective slot (axis 3
and axis 4 compose naturally).
The adjective list is a closed set identical to the one in
`_CONJ_OBJECT_RE` (already ratified by G.4), keeping both shapes
consistent.
### Pre-existing G.4 type bugs fixed
ADR-0131.G.4's multi-clause extractors (`_conj_subject_each_candidates`,
`_conj_object_candidates`, `_embedded_quantifier_candidates`,
`_build_conj_embedded_sum`, `_compare_multiplicative_candidates`,
`_compare_nested_candidates`) called `_resolve_value(...)` directly as a
numeric operand (e.g. `float(_resolve_value(x))`, `_resolve_value(n) *
_resolve_value(m)`, `Quantity(value=_resolve_value(x), ...)`). Since
`_resolve_value` returns `_ResolvedValue | None` (not a raw number), these
sites raised `TypeError` at runtime when hit by the GSM8K coverage probe.
This is a minimal fix: each site is updated to unwrap `.value` from the
`_ResolvedValue` result (or guard against `None` and return `[]`/early
exit). Semantics are identical — the value extracted is the same numeric
payload; no new candidates are emitted or suppressed.
---
## What changed in code
### `generate/math_candidate_parser.py`
- `_MONEY_SYMBOL` widened to six-currency alternation.
- `_CURRENCY_SYMBOLS` constant: symbol → `(unit_surface, factor)` table.
- `_resolve_currency`: dispatcher for all currency symbol literals.
- `_resolve_value`: currency branch now delegates to `_resolve_currency`.
- `_INITIAL_HAS_RE`: optional adjective group inserted between value and
unit slots.
- `_INITIAL_FRACTION_OF_RE`: new regex for `N/M of [a/an] <unit>`.
- `_MULTI_WORD_CARDINAL_RE`: new regex for `<Entity> has <WORD_CARDINAL+>
[adj?] <unit>`.
- `_fraction_of_candidates`: extractor for axis 1.
- `_multi_word_cardinal_candidates`: extractor for axis 3.
- `extract_initial_candidates`: calls both new extractors (after primary,
before G.4 extractors).
- `_money_unit_normalization`: extended for euro/yen/peso word forms.
- G.4 pre-existing type fixes: `.value` unwrap at six call sites.
### `generate/math_roundtrip.py`
- `_unit_grounds`: widened for `¢`, `€`, `¥`, `₱` symbols (mirrors
existing `$` logic).
- `_value_grounds`: currency symbol prefix check widened from `$`-only to
the full `_CURRENCY_SYM_SET`.
### New files
- `evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl` — 28 cases
(5 per axis × 4 + 8 refusal probes).
- `evals/math_capability_axes/G3_numerics/v1_1/runner.py` — identical
adapter pattern to v1; byte-equal `report.json` across runs.
- `evals/math_capability_axes/G3_numerics/v1_1/report.json` — committed
run artifact.
- `tests/test_adr_0131_G31_numerics_extensions.py` — 42 tests.
---
## Evidence
### Axis lane (`v1_1/runner.py`)
```
axis: numerics_extensions
cases_total: 28
solved_correct: 20
solved_wrong: 0 (gate: must be 0)
refused_as_expected: 8
correct_rate_on_positive_cases: 100.0%
overall_pass: True
```
Per-axis breakdown:
- Axis 1 (fractions): 5/5 solved_correct
- Axis 2 (multi-currency): 5/5 solved_correct (¢ € ¥ ₱; £ deferred)
- Axis 3 (multi-word cardinals): 5/5 solved_correct
- Axis 4 (word-num-adjective): 5/5 solved_correct
- Refusal probes: 8/8 refused (percentages ×2, sci-notation ×2,
locale-separators ×2, 3-decimal-money ×2)
`report.json` byte-equal across two independent runs.
### Test suite
**42/42 pass** in `tests/test_adr_0131_G31_numerics_extensions.py` (0.27s).
### Parent v1 lane regression
**20/20 + 6/6 = 26/26** — unchanged from PR #183 commit.
### GSM8K coverage probe
```
admission_rate: 0/50 = 0.0%
admitted_wrong: 0 (safety rail intact)
```
No probe delta. Admission remains at 0/50 for the same structural reasons
identified in G.3: most GSM8K refusals are verb-class or multi-clause
failures (G.1 / G.4 axes), not literal-recognition gaps. Fractions and
multi-word cardinals don't appear in the 50-case probe as the binding
blocker. The safety rail (`admitted_wrong == 0`) is preserved.
---
## Currencies deferred to G.3.2
**`£` / pound sterling**: `en_units_v1` is correctly populated; `_resolve_value("£15")` returns `_ResolvedValue(15, "pounds sterling")`. The
blocker is the question extractor's single-token `(?P<unit>\w+)` group,
which cannot parse `"How many pounds sterling does Alice have?"`. Fixing
this requires widening the question-extractor unit slot to support
multi-word units — a distinct scoped change for G.3.2.
---
## CLAUDE.md PR-checklist answers
- **Capability/performance/security added:** Extends the candidate-graph
parser's literal-recognition to four deferred shape families from G.3.
`wrong == 0` on the axis lane proves no wrong answers were introduced.
- **Invariant proving field remains valid:** `solved_wrong == 0` on the
v1_1 axis lane; `admitted_wrong == 0` on the GSM8K probe; no changes to
`algebra/`, `chat/`, or `core/`.
- **CLI/eval lane:** `PYTHONPATH=. python3
evals/math_capability_axes/G3_numerics/v1_1/runner.py` and `pytest
tests/test_adr_0131_G31_numerics_extensions.py`.
- **Avoided hidden normalization/stochastic fallback/approximate
recall/unreviewed mutation:** Yes. All lookups are deterministic and
pack-driven. Currency normalization is a fixed table. No solver or
binding-graph changes.
- **Trust boundary:** User-controlled text → parser regex; all new regex
patterns are closed alternations with no catastrophic-backtracking risk.
Pack file paths use the existing `safe_pack_id` sanitiser (ADR-0051).

30
evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
View file

@ -86,7 +86,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.'",
"reason": "candidate_graph: no admissible candidate for statement: 'They need to put all of their loose crayons in a box.'",
"trace_hash": null
},
{
@ -108,7 +108,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'",
"reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'",
"trace_hash": null
},
{
@ -471,7 +471,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.'",
"reason": "candidate_graph: no admissible candidate for question: 'If Ella sells 200 apples, how many apples does Ella has left?'",
"trace_hash": null
},
{
@ -565,6 +565,14 @@
],
"probe": "gsm8k_train_sample_coverage",
"refused_reasons_top": [
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for question: 'If Ella sells 200 apples, how many apples does Ella has left?'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/"
@ -601,10 +609,6 @@
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Bob can shuck 10 oysters in 5 minutes.'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their ma"
@ -613,10 +617,6 @@
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Fabian bought a brand new computer mouse and keyboard to be abl"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, an"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Georgie is a varsity player on a football team.'"
@ -641,10 +641,6 @@
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Jed collects stamp cards.'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the "
@ -741,6 +737,10 @@
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'There are some kids in camp.'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'They need to put all of their loose crayons in a box.'"
},
{
"count": 1,
"reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'"

0
evals/math_capability_axes/G3_numerics/v1_1/__init__.py Normal file
View file

28
evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl Normal file
View file

@ -0,0 +1,28 @@
{"case_id":"g31-frac-01","class":"fraction_of_unit","problem":"Bob has 1/2 of a cup. How many cups does Bob have?","expected_answer":0.5,"expected_unit":"cups","expected_outcome":"solved_correct"}
{"case_id":"g31-frac-02","class":"fraction_of_unit","problem":"Sarah has 3/4 of a bag. How many bags does Sarah have?","expected_answer":0.75,"expected_unit":"bags","expected_outcome":"solved_correct"}
{"case_id":"g31-frac-03","class":"fraction_of_unit","problem":"Tom has 1/4 of a pie. How many pies does Tom have?","expected_answer":0.25,"expected_unit":"pies","expected_outcome":"solved_correct"}
{"case_id":"g31-frac-04","class":"fraction_of_unit","problem":"Sam has 3/2 of a liter. How many liters does Sam have?","expected_answer":1.5,"expected_unit":"liters","expected_outcome":"solved_correct"}
{"case_id":"g31-frac-05","class":"fraction_of_unit","problem":"Liam has 4/5 of a cup. How many cups does Liam have?","expected_answer":0.8,"expected_unit":"cups","expected_outcome":"solved_correct"}
{"case_id":"g31-cur-01","class":"multi_currency","problem":"Bob has ¢50. How many cents does Bob have?","expected_answer":50.0,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g31-cur-02","class":"multi_currency","problem":"Maria has €20. How many euros does Maria have?","expected_answer":20.0,"expected_unit":"euros","expected_outcome":"solved_correct"}
{"case_id":"g31-cur-03","class":"multi_currency","problem":"Kenji has ¥100. How many yen does Kenji have?","expected_answer":100.0,"expected_unit":"yen","expected_outcome":"solved_correct"}
{"case_id":"g31-cur-04","class":"multi_currency","problem":"Juan has ₱200. How many pesos does Juan have?","expected_answer":200.0,"expected_unit":"pesos","expected_outcome":"solved_correct"}
{"case_id":"g31-cur-05","class":"multi_currency","problem":"Maria has €30. Maria spends €10. How many euros does Maria have?","expected_answer":20.0,"expected_unit":"euros","expected_outcome":"solved_correct"}
{"case_id":"g31-mwc-01","class":"multi_word_cardinal","problem":"Bob has one hundred apples. How many apples does Bob have?","expected_answer":100.0,"expected_unit":"apples","expected_outcome":"solved_correct"}
{"case_id":"g31-mwc-02","class":"multi_word_cardinal","problem":"The store has one thousand books. How many books does the store have?","expected_answer":1000.0,"expected_unit":"books","expected_outcome":"solved_correct"}
{"case_id":"g31-mwc-03","class":"multi_word_cardinal","problem":"Anna has three hundred cookies. How many cookies does Anna have?","expected_answer":300.0,"expected_unit":"cookies","expected_outcome":"solved_correct"}
{"case_id":"g31-mwc-04","class":"multi_word_cardinal","problem":"Mike has two thousand five hundred marbles. How many marbles does Mike have?","expected_answer":2500.0,"expected_unit":"marbles","expected_outcome":"solved_correct"}
{"case_id":"g31-mwc-05","class":"multi_word_cardinal","problem":"Sam has five hundred dollars. How many cents does Sam have?","expected_answer":50000.0,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g31-adj-01","class":"word_num_adjective","problem":"Sam has five full boxes. How many boxes does Sam have?","expected_answer":5.0,"expected_unit":"boxes","expected_outcome":"solved_correct"}
{"case_id":"g31-adj-02","class":"word_num_adjective","problem":"Ella has three loose crayons. How many crayons does Ella have?","expected_answer":3.0,"expected_unit":"crayons","expected_outcome":"solved_correct"}
{"case_id":"g31-adj-03","class":"word_num_adjective","problem":"Bob has seven empty cans. How many cans does Bob have?","expected_answer":7.0,"expected_unit":"cans","expected_outcome":"solved_correct"}
{"case_id":"g31-adj-04","class":"word_num_adjective","problem":"Jane has twelve whole pies. How many pies does Jane have?","expected_answer":12.0,"expected_unit":"pies","expected_outcome":"solved_correct"}
{"case_id":"g31-adj-05","class":"word_num_adjective","problem":"Tom has eight new books. How many books does Tom have?","expected_answer":8.0,"expected_unit":"books","expected_outcome":"solved_correct"}
{"case_id":"g31-ref-pct-01","class":"refuse_percentage","problem":"Bob has 50% apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-pct-02","class":"refuse_percentage","problem":"Sam has 75% of a pie. How many pies does Sam have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-sci-01","class":"refuse_scientific_notation","problem":"Sam has 1e3 marbles. How many marbles does Sam have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-sci-02","class":"refuse_scientific_notation","problem":"Alice has 2.5e2 books. How many books does Alice have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-locale-01","class":"refuse_locale_separator","problem":"Alice has 1,000 pennies. How many pennies does Alice have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-locale-02","class":"refuse_locale_separator","problem":"Bob has 10,000 apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-3dec-01","class":"refuse_money_precision","problem":"Bob has $1.234. How many cents does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g31-ref-3dec-02","class":"refuse_money_precision","problem":"Sam has €5.678. How many euros does Sam have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}

367
evals/math_capability_axes/G3_numerics/v1_1/report.json Normal file
View file

@ -0,0 +1,367 @@
{
"adr": "0131.G.3.1",
"axis": "numerics_extensions",
"cases_path": "evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl",
"class_counts": {
"fraction_of_unit": 5,
"multi_currency": 5,
"multi_word_cardinal": 5,
"refuse_locale_separator": 2,
"refuse_money_precision": 2,
"refuse_percentage": 2,
"refuse_scientific_notation": 2,
"word_num_adjective": 5
},
"metrics": {
"cases_total": 28,
"correct_rate_on_positive_cases": 1.0,
"overall_pass": true,
"refused_as_expected": 8,
"solved_correct": 20,
"solved_wrong": 0,
"wrong_count_is_zero": true
},
"per_case": [
{
"actual_answer": 0.5,
"actual_outcome": "correct",
"actual_unit": "cups",
"case_id": "g31-frac-01",
"class": "fraction_of_unit",
"expected_answer": 0.5,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "b1b05ff2759cb6cf5d630d2396eb9f489d71043a3eb12a4c2cdd88062ad64848",
"verdict": "solved_correct"
},
{
"actual_answer": 0.75,
"actual_outcome": "correct",
"actual_unit": "bags",
"case_id": "g31-frac-02",
"class": "fraction_of_unit",
"expected_answer": 0.75,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "f64e99dbc03193d143a151ac5c7e9e283ba9e4b7dfea676410cba2dcf92cba74",
"verdict": "solved_correct"
},
{
"actual_answer": 0.25,
"actual_outcome": "correct",
"actual_unit": "pies",
"case_id": "g31-frac-03",
"class": "fraction_of_unit",
"expected_answer": 0.25,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "53c696733759f9c0c71d96fc004614d70494dabe0a63226ac73155b5942aabdb",
"verdict": "solved_correct"
},
{
"actual_answer": 1.5,
"actual_outcome": "correct",
"actual_unit": "liters",
"case_id": "g31-frac-04",
"class": "fraction_of_unit",
"expected_answer": 1.5,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "9ff18f4ee0f7797e3972c9d5e088775b53acfaf28b622d65c8a88fa1309abc12",
"verdict": "solved_correct"
},
{
"actual_answer": 0.8,
"actual_outcome": "correct",
"actual_unit": "cups",
"case_id": "g31-frac-05",
"class": "fraction_of_unit",
"expected_answer": 0.8,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "92e90758cc1ee3dab91dd8610dfed35b2c616e64bf95e2ed682467c9849eec07",
"verdict": "solved_correct"
},
{
"actual_answer": 50.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g31-cur-01",
"class": "multi_currency",
"expected_answer": 50.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "490bbc2f0384acc8f94a2740f0e37572a6d2036692edd8b7ff77de80785cbbff",
"verdict": "solved_correct"
},
{
"actual_answer": 20.0,
"actual_outcome": "correct",
"actual_unit": "euros",
"case_id": "g31-cur-02",
"class": "multi_currency",
"expected_answer": 20.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "d1da3f9a8af64633231a2157593e4d571de973f815801172ef8dac1c40e4bb1f",
"verdict": "solved_correct"
},
{
"actual_answer": 100.0,
"actual_outcome": "correct",
"actual_unit": "yen",
"case_id": "g31-cur-03",
"class": "multi_currency",
"expected_answer": 100.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "c9a074741863a38a4d58fe4bfc17c18ff220b4172cf8541b3cc20d36be2dba9d",
"verdict": "solved_correct"
},
{
"actual_answer": 200.0,
"actual_outcome": "correct",
"actual_unit": "pesos",
"case_id": "g31-cur-04",
"class": "multi_currency",
"expected_answer": 200.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "38597a8bef5b6e08134e3ca2448eb4199daec93a24e0358da932133a02938c36",
"verdict": "solved_correct"
},
{
"actual_answer": 20.0,
"actual_outcome": "correct",
"actual_unit": "euros",
"case_id": "g31-cur-05",
"class": "multi_currency",
"expected_answer": 20.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "c313f6ad6fe05d09425942fb4e5022014cfd78fdd3d297d6a2349cc719181ac8",
"verdict": "solved_correct"
},
{
"actual_answer": 100.0,
"actual_outcome": "correct",
"actual_unit": "apples",
"case_id": "g31-mwc-01",
"class": "multi_word_cardinal",
"expected_answer": 100.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "0023bf2b389b79528556e91cb984a0d6d22b3b9b95b61d3ae73414fa93ead560",
"verdict": "solved_correct"
},
{
"actual_answer": 1000.0,
"actual_outcome": "correct",
"actual_unit": "books",
"case_id": "g31-mwc-02",
"class": "multi_word_cardinal",
"expected_answer": 1000.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "7c44c06806a79fd3d797d42b4311688bab66d8a62d7dd09606457faacaa11364",
"verdict": "solved_correct"
},
{
"actual_answer": 300.0,
"actual_outcome": "correct",
"actual_unit": "cookies",
"case_id": "g31-mwc-03",
"class": "multi_word_cardinal",
"expected_answer": 300.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "e7a76500137f6f7936a904f6b916dc310defbbd8bdfc87727e98cdf7685187c0",
"verdict": "solved_correct"
},
{
"actual_answer": 2500.0,
"actual_outcome": "correct",
"actual_unit": "marbles",
"case_id": "g31-mwc-04",
"class": "multi_word_cardinal",
"expected_answer": 2500.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "30b22bade97f050d89549af89c9d7512e69facab1346c584bb22c1e6c25e2ef7",
"verdict": "solved_correct"
},
{
"actual_answer": 50000.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g31-mwc-05",
"class": "multi_word_cardinal",
"expected_answer": 50000.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "7ca5b0470f11ed2a806e95982211954202f99e48c707780431df2522f01b871f",
"verdict": "solved_correct"
},
{
"actual_answer": 5.0,
"actual_outcome": "correct",
"actual_unit": "boxes",
"case_id": "g31-adj-01",
"class": "word_num_adjective",
"expected_answer": 5.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "80f43f99acfea39c9226f1a6bc46d56f950a799b698e4eb6bc4be086046828aa",
"verdict": "solved_correct"
},
{
"actual_answer": 3.0,
"actual_outcome": "correct",
"actual_unit": "crayons",
"case_id": "g31-adj-02",
"class": "word_num_adjective",
"expected_answer": 3.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "46fcecab0fd1a1ab4db5b2bc3033a0d680f3919eef992714415cbdbcae5acc03",
"verdict": "solved_correct"
},
{
"actual_answer": 7.0,
"actual_outcome": "correct",
"actual_unit": "cans",
"case_id": "g31-adj-03",
"class": "word_num_adjective",
"expected_answer": 7.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "0d10b1e8251b5298ed826c8c10526f0e28c54d443b6256c10c9be6f8190cab09",
"verdict": "solved_correct"
},
{
"actual_answer": 12.0,
"actual_outcome": "correct",
"actual_unit": "pies",
"case_id": "g31-adj-04",
"class": "word_num_adjective",
"expected_answer": 12.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "74083e671f30421f0a968f1f8d4314f56b1a255854a757c6ab6f2eddafa74d0c",
"verdict": "solved_correct"
},
{
"actual_answer": 8.0,
"actual_outcome": "correct",
"actual_unit": "books",
"case_id": "g31-adj-05",
"class": "word_num_adjective",
"expected_answer": 8.0,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "ba2b359492336fc2fcc71a0fde24f05a4fa446133c6530dc13bbce46ec577dc4",
"verdict": "solved_correct"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-pct-01",
"class": "refuse_percentage",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has 50% apples.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-pct-02",
"class": "refuse_percentage",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Sam has 75% of a pie.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-sci-01",
"class": "refuse_scientific_notation",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Sam has 1e3 marbles.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-sci-02",
"class": "refuse_scientific_notation",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Alice has 2.5e2 books.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-locale-01",
"class": "refuse_locale_separator",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Alice has 1,000 pennies.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-locale-02",
"class": "refuse_locale_separator",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has 10,000 apples.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-3dec-01",
"class": "refuse_money_precision",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has $1.234.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g31-ref-3dec-02",
"class": "refuse_money_precision",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Sam has \u20ac5.678.'",
"trace_hash": null,
"verdict": "refused"
}
],
"schema_version": 1,
"verdict_counts": {
"refused": 8,
"solved_correct": 20
}
}

135
evals/math_capability_axes/G3_numerics/v1_1/runner.py Normal file
View file

@ -0,0 +1,135 @@
"""ADR-0131.G.3.1 — Numerics-extensions capability-axis runner (v1.1).
Additive sibling to ``evals/math_capability_axes/G3_numerics/v1/``.
v1 is frozen as the audit-trail artifact for PR #183; v1.1 carries the
four axes deferred from G.3:
1. **Fractions end-to-end** — ``N/M of a <unit>`` initial possession.
2. **Multi-currency** — ``¢``, ``€``, ``¥``, ``₱`` symbols.
(``£`` deferred to G.3.2: question extractor's single-token unit
slot cannot parse the two-word surface "pounds sterling".)
3. **Multi-token space-separated cardinals** — ``one hundred``,
``two thousand five hundred``.
4. **Word-number-adjective** — ``five full boxes``.
Runner interface is identical to v1 so the G3 axis lane CI check is
parameterisable over both versions.
"""
from __future__ import annotations
import json
from collections import Counter
from pathlib import Path
from typing import Any
from evals.gsm8k_math.runner import _score_one_candidate_graph
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_REPORT_PATH = _HERE / "report.json"
def _load_cases() -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines():
if line.strip():
out.append(json.loads(line))
return out
def _adapt_case(raw: dict[str, Any]) -> dict[str, Any]:
return {
"id": raw["case_id"],
"problem": raw["problem"],
"expected_answer": float(raw["expected_answer"]),
"expected_unit": raw.get("expected_unit", ""),
}
def _classify(actual_outcome: str, expected_outcome: str) -> str:
if expected_outcome == "solved_correct" and actual_outcome == "correct":
return "solved_correct"
if expected_outcome == "refused" and actual_outcome == "refused":
return "refused"
return "solved_wrong"
def build_report() -> dict[str, Any]:
raw_cases = _load_cases()
case_results: list[dict[str, Any]] = []
class_counts: Counter[str] = Counter()
verdict_counts: Counter[str] = Counter()
for raw in raw_cases:
cls = raw["class"]
expected = raw["expected_outcome"]
class_counts[cls] += 1
outcome = _score_one_candidate_graph(_adapt_case(raw))
verdict = _classify(outcome.outcome, expected)
verdict_counts[verdict] += 1
case_results.append({
"case_id": raw["case_id"],
"class": cls,
"expected_outcome": expected,
"actual_outcome": outcome.outcome,
"verdict": verdict,
"expected_answer": raw["expected_answer"],
"actual_answer": outcome.actual_answer,
"actual_unit": outcome.actual_unit,
"reason": outcome.reason,
"trace_hash": outcome.trace_hash,
})
total = len(raw_cases)
correct = verdict_counts.get("solved_correct", 0)
wrong = verdict_counts.get("solved_wrong", 0)
refused_expected = verdict_counts.get("refused", 0)
positive_count = sum(1 for r in raw_cases if r["expected_outcome"] == "solved_correct")
correct_rate_on_positive = (
correct / positive_count if positive_count else 0.0
)
return {
"schema_version": 1,
"adr": "0131.G.3.1",
"axis": "numerics_extensions",
"cases_path": "evals/math_capability_axes/G3_numerics/v1_1/cases.jsonl",
"metrics": {
"cases_total": total,
"solved_correct": correct,
"solved_wrong": wrong,
"refused_as_expected": refused_expected,
"wrong_count_is_zero": wrong == 0,
"correct_rate_on_positive_cases": correct_rate_on_positive,
"overall_pass": wrong == 0 and (correct + refused_expected == total),
},
"class_counts": dict(sorted(class_counts.items())),
"verdict_counts": dict(sorted(verdict_counts.items())),
"per_case": case_results,
}
def write_report(report: dict[str, Any]) -> None:
_REPORT_PATH.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
report = build_report()
write_report(report)
m = report["metrics"]
print(f"axis: {report['axis']}")
print(f"cases_total: {m['cases_total']}")
print(f"solved_correct: {m['solved_correct']}")
print(f"solved_wrong: {m['solved_wrong']} (gate: must be 0)")
print(f"refused_as_expected: {m['refused_as_expected']}")
print(f"correct_rate_on_positive_cases: {m['correct_rate_on_positive_cases']:.1%}")
print(f"overall_pass: {m['overall_pass']}")
return 0 if m["overall_pass"] else 1
if __name__ == "__main__":
raise SystemExit(main())

258
generate/math_candidate_parser.py
View file

@ -114,8 +114,9 @@ _ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)"
# Numeric value alternation. Listed longest-form-first so the regex
# engine doesn't truncate on a shorter prefix:
# - Money symbol literal: ``$N`` or ``$N.NN`` (1-2 decimal places).
# ADR-0131.G.3. ``$N.NNN`` (3+ decimals) deliberately not matched
# - Money symbol literal: ``$N`` / ``$N.NN`` (1-2 decimal places) plus
# multi-currency symbols ``¢N`` ``€N`` ``£N`` ``¥N`` ``₱N``.
# ADR-0131.G.3.1. ``$N.NNN`` (3+ decimals) deliberately not matched
# — refused as out-of-scope so wrong == 0 is preserved.
# - Slash fraction literal: ``N/M``. Denominator-zero refused at
# resolve time, not regex.
@ -123,7 +124,13 @@ _ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)"
# Resolved via :func:`language_packs.numerics_loader.parse_compound_cardinal`.
# - Digit run.
# - Single-word cardinal (legacy ``WORD_NUMBERS`` set).
_MONEY_SYMBOL: Final[str] = r"\$\d+(?:\.\d{1,2})?"
# ADR-0131.G.3.1: multi-currency symbol group. ¢ and $ are the only
# non-decimal currencies (sub-unit is the unit itself for ¢; $ converts
# to cents). €, £, ₱ admit 1-2 decimal places; ¥ is integer-only.
_MONEY_SYMBOL: Final[str] = (
r"(?:\$\d+(?:\.\d{1,2})?|¢\d+|€\d+(?:\.\d{1,2})?|£\d+(?:\.\d{1,2})?|¥\d+|₱\d+(?:\.\d{1,2})?)"
)
_SLASH_FRACTION: Final[str] = r"\d+/\d+"
_HYPHENATED_CARDINAL: Final[str] = r"[A-Za-z]+-[A-Za-z]+"
_WORD_NUM_OPTIONS: Final[str] = "|".join(
@ -160,6 +167,10 @@ _INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile(
# (``$40``) carry their unit implicitly (``cent``); a missing unit
# slot is admissible IFF the value resolves with a unit override.
# Non-money values without a unit slot are refused at resolve time.
# ADR-0131.G.3.1 axis 4: optional adjective between value and unit
# ("five full boxes" — adjective 'full' is consumed and discarded;
# the unit head noun 'boxes' becomes the unit slot).
r"(?:\s+(?:full|loose|empty|whole|broken|new|old|small|large|fresh|raw|flat))?"
r"(?:\s+(?P<unit>\w+))?"
# ADR-0127 substance qualifier: "Sam has 5 feet of rope" — the
# 'of <NP>' tail is grammatically real but arithmetically inert.
@ -186,6 +197,45 @@ _INITIAL_THERE_ARE_RE: Final[re.Pattern[str]] = re.compile(
flags=re.IGNORECASE,
)
# ADR-0131.G.3.1 — Axis 1: fraction-of-unit initial possession.
# "Bob has 3/4 of a cup." — the fraction is the value; "of a/an <unit>"
# carries the unit. The main _INITIAL_HAS_RE treats "of <NP>" as a
# discardable substance qualifier and emits no candidate (unit slot absent
# and no unit_override); this separate pattern extracts the unit from
# the "of" phrase explicitly.
_INITIAL_FRACTION_OF_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<entity>{_ENTITY})\s+"
rf"(?P<anchor>has|have)\s+"
rf"(?P<value>{_SLASH_FRACTION})\s+"
r"of\s+(?:a\s+|an\s+)?(?P<unit>\w+)"
r"(?:\s+of\s+.+)?" # optional further substance qualifier
r"\s*\.?$"
)
# ADR-0131.G.3.1 — Axis 3: multi-token space-separated cardinal.
# "Bob has one hundred apples." — parse_compound_cardinal already handles
# the value; this pattern captures it before the unit-slot boundary.
# Approach (a) chosen over (b) (_VALUE widening) because greedy cardinal-
# word matching inside _VALUE would span the unit slot and require
# look-ahead unwinding; a separate dedicated extractor is narrower and
# leaves _VALUE unchanged for all other paths.
# Build cardinal-word alternation from the WORD_NUMBERS table.
_CARDINAL_WORD_OPTIONS: Final[str] = "|".join(
re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True)
)
# At least two cardinal words (single-word is handled by _VALUE/_resolve_value).
_MULTI_WORD_CARDINAL_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<entity>{_ENTITY})\s+"
rf"(?P<anchor>has|have)\s+"
rf"(?P<value>(?:{_CARDINAL_WORD_OPTIONS})(?:\s+(?:{_CARDINAL_WORD_OPTIONS}))+)"
# Optional adjective (axis 4 compound) between cardinal and unit.
r"(?:\s+(?:full|loose|empty|whole|broken|new|old|small|large|fresh|raw|flat))?"
r"\s+(?P<unit>\w+)"
r"(?:\s+(?:of|in|for|with)\s+.+)?"
r"\s*\.?$",
flags=re.IGNORECASE,
)
def _normalize_entity(raw: str) -> str:
"""Collapse whitespace + lowercase article. Mirrors math_parser
@ -216,6 +266,46 @@ class _ResolvedValue:
# ``canonical_unit`` for the ``money`` dimension is ``cent``).
_MONEY_UNIT: Final[str] = "cents"
# ADR-0131.G.3.1: multi-currency symbol → (unit_surface, factor_to_unit).
# ``factor_to_unit`` is the multiplier applied to the face value to
# produce the canonical unit. For USD ($): face is dollars → *100 cents.
# For ¢: face is already cents → *1. For all others the pack has no
# sub-unit defined, so face == canonical (factor=1) and the unit is the
# pack's plural surface form.
_CURRENCY_SYMBOLS: Final[dict[str, tuple[str, float]]] = {
"$": ("cents", 100.0), # dollar → 100 cents
"¢": ("cents", 1.0), # cent already canonical
"€": ("euros", 1.0),
"£": ("pounds sterling", 1.0),
"¥": ("yen", 1.0),
"₱": ("pesos", 1.0),
}
def _resolve_currency(t: str) -> _ResolvedValue | None:
"""Resolve a currency-symbol value token (``$N``, ``¢N``, ``€N.NN``, …).
Returns ``None`` when the format is out-of-scope (e.g. 3+ decimal places).
Yen (``¥``) is integer-only (no sub-unit in en_units_v1).
"""
for sym, (unit_surface, factor) in _CURRENCY_SYMBOLS.items():
if not t.startswith(sym):
continue
body = t[len(sym):]
if re.fullmatch(r"\d+", body):
raw_val = int(body)
final = int(raw_val * factor) if factor == int(factor) else raw_val * factor
return _ResolvedValue(final, unit_surface)
# ¥ is integer-only.
if sym == "¥":
return None
if re.fullmatch(r"\d+\.\d{1,2}", body):
raw_val = float(body)
result = raw_val * factor
return _ResolvedValue(int(round(result)) if factor != 1.0 else raw_val, unit_surface)
return None # 3+ decimals refused for all currency symbols
return None
def _resolve_value(value_token: str) -> _ResolvedValue | None:
"""Resolve a value-slot token into a numeric value + optional unit
@ -230,15 +320,9 @@ def _resolve_value(value_token: str) -> _ResolvedValue | None:
if not value_token:
return None
t = value_token.strip()
# Money symbol literal: $N or $N.NN.
if t.startswith("$"):
body = t[1:]
if re.fullmatch(r"\d+", body):
return _ResolvedValue(int(body) * 100, _MONEY_UNIT)
if re.fullmatch(r"\d+\.\d{1,2}", body):
# round() avoids float drift: $2.50 → 250, not 249 or 251.
return _ResolvedValue(int(round(float(body) * 100)), _MONEY_UNIT)
return None # $N.NNN (3+ decimals) refused — out-of-scope.
# Multi-currency symbols (ADR-0131.G.3.1): $, ¢, €, £, ¥, ₱.
if t and t[0] in _CURRENCY_SYMBOLS:
return _resolve_currency(t)
# Slash fraction literal: N/M with M > 0.
if "/" in t:
m = re.fullmatch(r"(\d+)/(\d+)", t)
@ -292,19 +376,28 @@ def _is_indefinite_quantifier(token: str) -> bool:
def _money_unit_normalization(
value: int | float, unit: str | None
) -> tuple[int | float, str | None]:
"""ADR-0131.G.3 — normalize ``dollar``/``dollars`` surface unit to the
canonical money unit (``cent``).
"""ADR-0131.G.3 — normalize money word-form surface units to pack canonical.
``en_units_v1`` pins ``cent`` as ``canonical_unit`` for the ``money``
dimension; ``dollar`` is convenience surface. A ``dollar`` value is
100 ``cent``. Done at the candidate-build site so every money-bearing
path normalizes uniformly (Quantity equality is exact — mixing
``cent`` and ``dollar`` units would silently break arithmetic).
dimension. ``dollar``/``dollars`` → 100 cents each. Other currencies
(ADR-0131.G.3.1) are already in canonical form when they arrive via
``_resolve_currency``; this helper normalizes the word-form paths.
"""
if unit is None:
return value, unit
if unit.lower() in ("dollar", "dollars"):
lower = unit.lower()
if lower in ("dollar", "dollars"):
return value * 100, _MONEY_UNIT
# Euro/pound-sterling/yen/peso word forms: already canonical (factor=1).
# These enter via unit slot (word form) rather than symbol — pass through.
if lower in ("euro", "euros"):
return value, "euros"
if lower in ("pound sterling", "pounds sterling"):
return value, "pounds sterling"
if lower == "yen":
return value, "yen"
if lower in ("peso", "pesos"):
return value, "pesos"
return value, unit
@ -362,6 +455,14 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
)
)
# ADR-0131.G.3.1 — Axis 1: fraction-of-unit shape.
# "Bob has 3/4 of a cup." — separate regex extracts unit from "of" phrase.
out.extend(_fraction_of_candidates(sentence))
# ADR-0131.G.3.1 — Axis 3: multi-token space-separated cardinals.
# "Bob has one hundred apples." — separate extractor; _VALUE is unchanged.
out.extend(_multi_word_cardinal_candidates(sentence))
# ADR-0131.G.4 — multi-clause initial-state extractors.
# Each may emit ≥1 candidates; deterministic order: conjoined-subject-each,
# conjoined-object, embedded-quantifier, conjoined-embedded-quantifier.
@ -937,8 +1038,11 @@ def _compare_multiplicative_candidates(sentence: str) -> list[CandidateOperation
if _is_indefinite_quantifier(value_raw):
return out
try:
factor = float(_resolve_value(value_raw))
except KeyError:
_rv = _resolve_value(value_raw)
factor = float(_rv.value) if _rv is not None else None
except (KeyError, TypeError):
return out
if factor is None:
return out
cand = _build_compare_multiplicative(
actor_raw=m.group("actor"),
@ -995,8 +1099,9 @@ def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]:
factor_value_raw = m.group("factor_value")
if not _is_indefinite_quantifier(factor_value_raw):
try:
factor = float(_resolve_value(factor_value_raw))
except KeyError:
_rv2 = _resolve_value(factor_value_raw)
factor = float(_rv2.value) if _rv2 is not None else None
except (KeyError, TypeError):
factor = None
if factor is not None:
mult_cand = _build_compare_multiplicative(
@ -1015,6 +1120,88 @@ def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]:
return out
# ---------------------------------------------------------------------------
# ADR-0131.G.3.1 — Axis 1 + Axis 3 extractor functions
# ---------------------------------------------------------------------------
def _fraction_of_candidates(sentence: str) -> list[CandidateInitial]:
"""Axis 1 (fractions): 'Bob has 3/4 of a cup.' → value=0.75, unit='cups'.
The main _INITIAL_HAS_RE treats 'of <NP>' as a discardable substance
qualifier and cannot fill the unit slot from it. This extractor uses
_INITIAL_FRACTION_OF_RE to explicitly capture the unit after 'of'.
"""
s = sentence.strip().rstrip(".")
m = _INITIAL_FRACTION_OF_RE.match(s)
if m is None:
return []
value_raw = m.group("value")
rv = _resolve_value(value_raw)
if rv is None:
return []
unit_raw = m.group("unit")
unit = _canonicalize_unit(unit_raw)
entity = _normalize_entity(m.group("entity"))
try:
return [
CandidateInitial(
initial=InitialPossession(
entity=entity,
quantity=Quantity(value=rv.value, unit=unit),
),
source_span=sentence,
matched_anchor=m.group("anchor"),
matched_value_token=value_raw,
matched_unit_token=unit_raw,
matched_entity_token=m.group("entity"),
)
]
except Exception:
return []
def _multi_word_cardinal_candidates(sentence: str) -> list[CandidateInitial]:
"""Axis 3 (multi-word cardinals): 'Bob has one hundred apples.'
Approach (a): dedicated extractor leaving _VALUE unchanged. The value
group captures the full space-separated cardinal sequence; the unit
slot is the next word token after the cardinal sequence (and optional
adjective).
"""
s = sentence.strip().rstrip(".")
m = _MULTI_WORD_CARDINAL_RE.match(s)
if m is None:
return []
value_raw = m.group("value")
from language_packs.numerics_loader import parse_compound_cardinal
parsed = parse_compound_cardinal(value_raw)
if parsed is None:
return []
unit_raw = m.group("unit")
value_n, unit_n = _money_unit_normalization(parsed, _canonicalize_unit(unit_raw))
if unit_n is None:
return []
entity = _normalize_entity(m.group("entity"))
try:
return [
CandidateInitial(
initial=InitialPossession(
entity=entity,
quantity=Quantity(value=value_n, unit=unit_n),
),
source_span=sentence,
matched_anchor=m.group("anchor"),
# Provenance: use the first cardinal word as the value token
# for grounding (all cardinal words are in the source span).
matched_value_token=value_raw.split()[0],
matched_unit_token=unit_raw,
matched_entity_token=m.group("entity"),
)
]
except Exception:
return []
# ---------------------------------------------------------------------------
# ADR-0131.G.4 — Multi-clause initial-state composition
# ---------------------------------------------------------------------------
@ -1138,7 +1325,10 @@ def _conj_subject_each_candidates(sentence: str) -> list[CandidateInitial]:
entity_b = _normalize_entity(m.group("b"))
if entity_a == entity_b:
return [] # 'Aaron and Aaron each ...' is degenerate
value = _resolve_value(value_raw)
_rv_conj = _resolve_value(value_raw)
if _rv_conj is None:
return []
value = _rv_conj.value
unit_raw = m.group("unit")
unit = _canonicalize_unit(unit_raw)
anchor = _canon_verb_to_anchor(m.group("verb"))
@ -1192,7 +1382,7 @@ def _conj_object_candidates(sentence: str) -> list[CandidateInitial]:
CandidateInitial(
initial=InitialPossession(
entity=entity,
quantity=Quantity(value=_resolve_value(value_raw), unit=unit),
quantity=Quantity(value=_resolve_value(value_raw).value, unit=unit), # type: ignore[union-attr]
),
source_span=sentence,
matched_anchor=anchor,
@ -1233,9 +1423,11 @@ def _embedded_quantifier_candidates(sentence: str) -> list[CandidateInitial]:
c2 = container2_raw.lower()
if c2 not in (container, container.rstrip("s"), container + "s"):
return []
n = _resolve_value(n_raw)
per = _resolve_value(m_raw)
total = n * per
_rv_n = _resolve_value(n_raw)
_rv_per = _resolve_value(m_raw)
if _rv_n is None or _rv_per is None:
return []
total = _rv_n.value * _rv_per.value
entity = _normalize_entity(m.group("entity"))
unit_raw = m.group("unit")
unit = _canonicalize_unit(unit_raw)
@ -1314,9 +1506,13 @@ def _build_conj_embedded_sum(
if u1 != u2:
# Mixed-unit sum is meaningless; refuse.
return []
total = _resolve_value(n1_raw) * _resolve_value(m1_raw) + (
_resolve_value(n2_raw) * _resolve_value(m2_raw)
)
_n1 = _resolve_value(n1_raw)
_m1 = _resolve_value(m1_raw)
_n2 = _resolve_value(n2_raw)
_m2 = _resolve_value(m2_raw)
if _n1 is None or _m1 is None or _n2 is None or _m2 is None:
return []
total = _n1.value * _m1.value + _n2.value * _m2.value
entity = _normalize_entity(m.group("entity"))
try:
return [

33
generate/math_roundtrip.py
View file

@ -285,14 +285,35 @@ def _unit_grounds(
on the raw source span rather than the token set. Similarly for
``dollar``: an author may write either ``$N`` or ``N dollars``;
both ground a money unit.
ADR-0131.G.3.1 widening: multi-currency symbols (¢ € £ ¥ ₱) each
ground their respective canonical unit when their symbol appears in
the raw source span.
"""
if _token_in(unit_token, haystack_tokens):
return True
if unit_token.lower() in ("cent", "cents"):
if "$" in source_span:
lower = unit_token.lower()
if lower in ("cent", "cents"):
if "$" in source_span or "¢" in source_span:
return True
if "dollar" in haystack_tokens or "dollars" in haystack_tokens:
return True
if lower in ("euro", "euros"):
if "€" in source_span:
return True
# "pounds sterling" is a two-word unit; check both the multi-word
# surface and the raw symbol.
if lower in ("pound sterling", "pounds sterling"):
if "£" in source_span:
return True
if "sterling" in haystack_tokens:
return True
if lower == "yen":
if "¥" in source_span:
return True
if lower in ("peso", "pesos"):
if "₱" in source_span:
return True
return False
@ -319,9 +340,11 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
every component lemma is a token (the tokenizer splits on
hyphens), OR the compound's integer value's digit form appears.
"""
# ADR-0131.G.3 widenings (handled first; the trailing existing path
# would never recognize these surface shapes).
if value_token.startswith("$"):
# ADR-0131.G.3 / G.3.1 widenings (handled first; the trailing existing
# path would never recognize these surface shapes).
# Currency symbol literals: extract digit parts, verify each in source.
_CURRENCY_SYM_SET = frozenset({"$", "¢", "€", "£", "¥", "₱"})
if value_token and value_token[0] in _CURRENCY_SYM_SET:
body = value_token[1:]
parts = [p for p in body.split(".") if p]
return bool(parts) and all(p in haystack_tokens for p in parts)

251
tests/test_adr_0131_G31_numerics_extensions.py Normal file
View file

@ -0,0 +1,251 @@
"""ADR-0131.G.3.1 — Numerics extensions test suite.
Per-axis at-least-one passing test, refusal probes, wrong==0 invariant,
replay byte-equality, and parent v1 lane regression gate.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.gsm8k_math.runner import _score_one_candidate_graph
from generate.math_candidate_parser import (
_resolve_value,
extract_initial_candidates,
)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _score(problem: str, expected_answer: float, expected_unit: str) -> str:
r = _score_one_candidate_graph({
"id": "test",
"problem": problem,
"expected_answer": expected_answer,
"expected_unit": expected_unit,
})
return r.outcome
def _refused(problem: str) -> bool:
r = _score_one_candidate_graph({
"id": "test",
"problem": problem,
"expected_answer": 0.0,
"expected_unit": "",
})
return r.outcome == "refused"
# ---------------------------------------------------------------------------
# Axis 1: Fractions end-to-end
# ---------------------------------------------------------------------------
class TestFractions:
def test_half_of_cup(self):
assert _score("Bob has 1/2 of a cup. How many cups does Bob have?", 0.5, "cups") == "correct"
def test_three_quarters_of_bag(self):
assert _score("Sarah has 3/4 of a bag. How many bags does Sarah have?", 0.75, "bags") == "correct"
def test_quarter_of_pie(self):
assert _score("Tom has 1/4 of a pie. How many pies does Tom have?", 0.25, "pies") == "correct"
def test_improper_fraction(self):
assert _score("Sam has 3/2 of a liter. How many liters does Sam have?", 1.5, "liters") == "correct"
def test_resolve_value_fraction(self):
rv = _resolve_value("3/4")
assert rv is not None
assert abs(rv.value - 0.75) < 1e-9
assert rv.unit_override is None
def test_fraction_zero_denominator_refused(self):
assert _refused("Bob has 5/0 apples. How many apples does Bob have?")
def test_extract_initial_fraction_of(self):
cands = extract_initial_candidates("Bob has 1/2 of a cup.")
assert len(cands) > 0
q = cands[0].initial.quantity
assert abs(q.value - 0.5) < 1e-9
assert q.unit == "cups"
# ---------------------------------------------------------------------------
# Axis 2: Multi-currency
# ---------------------------------------------------------------------------
class TestMultiCurrency:
def test_cent_symbol(self):
assert _score("Bob has ¢50. How many cents does Bob have?", 50.0, "cents") == "correct"
def test_euro_symbol(self):
assert _score("Maria has €20. How many euros does Maria have?", 20.0, "euros") == "correct"
def test_yen_symbol(self):
assert _score("Kenji has ¥100. How many yen does Kenji have?", 100.0, "yen") == "correct"
def test_peso_symbol(self):
assert _score("Juan has ₱200. How many pesos does Juan have?", 200.0, "pesos") == "correct"
def test_euro_with_operation(self):
assert _score("Maria has €30. Maria spends €10. How many euros does Maria have?", 20.0, "euros") == "correct"
def test_resolve_cent_symbol(self):
rv = _resolve_value("¢50")
assert rv is not None and rv.value == 50 and rv.unit_override == "cents"
def test_resolve_euro_symbol(self):
rv = _resolve_value("€20")
assert rv is not None and rv.value == 20 and rv.unit_override == "euros"
def test_resolve_yen_integer_only(self):
# ¥ is integer-only; decimal form should be refused
rv = _resolve_value("¥100")
assert rv is not None and rv.value == 100 and rv.unit_override == "yen"
def test_euro_three_decimal_refused(self):
assert _refused("Sam has €5.678. How many euros does Sam have?")
def test_pound_sterling_deferred(self):
# £ symbol parses and resolves but question extractor cannot parse
# multi-word unit 'pounds sterling' — deferred to G.3.2.
rv = _resolve_value("£15")
assert rv is not None and rv.value == 15 and rv.unit_override == "pounds sterling"
# ---------------------------------------------------------------------------
# Axis 3: Multi-token space-separated cardinals
# ---------------------------------------------------------------------------
class TestMultiWordCardinals:
def test_one_hundred(self):
assert _score("Bob has one hundred apples. How many apples does Bob have?", 100.0, "apples") == "correct"
def test_one_thousand(self):
assert _score("The store has one thousand books. How many books does the store have?", 1000.0, "books") == "correct"
def test_three_hundred(self):
assert _score("Anna has three hundred cookies. How many cookies does Anna have?", 300.0, "cookies") == "correct"
def test_two_thousand_five_hundred(self):
assert _score("Mike has two thousand five hundred marbles. How many marbles does Mike have?", 2500.0, "marbles") == "correct"
def test_five_hundred_dollars_to_cents(self):
assert _score("Sam has five hundred dollars. How many cents does Sam have?", 50000.0, "cents") == "correct"
def test_extract_multi_word_cardinal(self):
cands = extract_initial_candidates("Bob has one hundred apples.")
assert len(cands) > 0
q = cands[0].initial.quantity
assert q.value == 100
assert q.unit == "apples"
# ---------------------------------------------------------------------------
# Axis 4: Word-number-adjective compositions
# ---------------------------------------------------------------------------
class TestWordNumAdjective:
def test_five_full_boxes(self):
assert _score("Sam has five full boxes. How many boxes does Sam have?", 5.0, "boxes") == "correct"
def test_three_loose_crayons(self):
assert _score("Ella has three loose crayons. How many crayons does Ella have?", 3.0, "crayons") == "correct"
def test_seven_empty_cans(self):
assert _score("Bob has seven empty cans. How many cans does Bob have?", 7.0, "cans") == "correct"
def test_twelve_whole_pies(self):
assert _score("Jane has twelve whole pies. How many pies does Jane have?", 12.0, "pies") == "correct"
def test_eight_new_books(self):
assert _score("Tom has eight new books. How many books does Tom have?", 8.0, "books") == "correct"
def test_extract_adjective_initial(self):
cands = extract_initial_candidates("Sam has five full boxes.")
assert len(cands) > 0
q = cands[0].initial.quantity
assert q.value == 5
assert q.unit == "boxes"
# ---------------------------------------------------------------------------
# Refusal probes — closed-set boundary enforcement
# ---------------------------------------------------------------------------
class TestRefusals:
def test_percentage_refused(self):
assert _refused("Bob has 50% apples. How many apples does Bob have?")
def test_percentage_of_refused(self):
assert _refused("Sam has 75% of a pie. How many pies does Sam have?")
def test_scientific_notation_refused(self):
assert _refused("Sam has 1e3 marbles. How many marbles does Sam have?")
def test_scientific_notation_float_refused(self):
assert _refused("Alice has 2.5e2 books. How many books does Alice have?")
def test_locale_separator_refused(self):
assert _refused("Alice has 1,000 pennies. How many pennies does Alice have?")
def test_locale_separator_large_refused(self):
assert _refused("Bob has 10,000 apples. How many apples does Bob have?")
def test_three_decimal_dollar_refused(self):
assert _refused("Bob has $1.234. How many cents does Bob have?")
def test_three_decimal_euro_refused(self):
assert _refused("Sam has €5.678. How many euros does Sam have?")
# ---------------------------------------------------------------------------
# Wrong == 0 invariant (load-bearing gate per ADR-0114a Obligation #4)
# ---------------------------------------------------------------------------
class TestWrongEqualsZero:
def test_v1_1_wrong_is_zero(self):
from evals.math_capability_axes.G3_numerics.v1_1.runner import build_report
report = build_report()
assert report["metrics"]["solved_wrong"] == 0, (
f"solved_wrong must be 0; got {report['metrics']['solved_wrong']}"
)
def test_v1_1_overall_pass(self):
from evals.math_capability_axes.G3_numerics.v1_1.runner import build_report
report = build_report()
assert report["metrics"]["overall_pass"] is True
# ---------------------------------------------------------------------------
# Replay byte-equality
# ---------------------------------------------------------------------------
class TestReplayByteEquality:
def test_report_byte_equal_across_two_runs(self):
from evals.math_capability_axes.G3_numerics.v1_1.runner import build_report
r1 = json.dumps(build_report(), indent=2, sort_keys=True)
r2 = json.dumps(build_report(), indent=2, sort_keys=True)
assert r1 == r2, "v1.1 report must be byte-equal across runs"
# ---------------------------------------------------------------------------
# Parent v1 lane regression (no regression from G.3 changes)
# ---------------------------------------------------------------------------
class TestParentV1Regression:
def test_v1_wrong_still_zero(self):
from evals.math_capability_axes.G3_numerics.v1.runner import build_report
report = build_report()
assert report["metrics"]["solved_wrong"] == 0
def test_v1_overall_pass(self):
from evals.math_capability_axes.G3_numerics.v1.runner import build_report
report = build_report()
assert report["metrics"]["overall_pass"] is True