core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Merge pull request #183 from AssetOverflow/feat/adr-0131-g3-numerics

Browse source 
feat(ADR-0131.G.3): numeric literals (money + hyphenated cardinals) — axis lane 20/20, wrong==0
This commit is contained in:
Shay 2026-05-23 14:49:42 -07:00 committed by GitHub
commit 34e9546e16
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 1211 additions and 46 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

226
docs/decisions/ADR-0131.G.3-numerics.md Normal file
View file

@ -0,0 +1,226 @@
# ADR-0131.G.3 — Numeric Literals (money + hyphenated cardinals)
**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE main agent (Opus 4.7)
**Depends on:** ADR-0131.G (capability-axis iteration discipline),
ADR-0126 (candidate-graph parser), ADR-0127 (units pack),
ADR-0128 (numerics pack)
**Parent:** ADR-0131 (composite math-expert promotion gate)
**Foundation for:** ADR-0131.G.3.1 (fractions + multi-currency follow-up)
---
## Context
ADR-0131.G pinned the GSM8K coverage probe as a diff-able admission
number with the iteration discipline that *each subsequent
ADR-0131.G.<n> extends a single capability axis with its own
curated coverage cases, independent of GSM8K*. G.3 is the first
literal-recognition axis: extending the candidate-graph parser's
`<value>` slot to recognize money literals and hyphenated multi-word
cardinals by consuming the already-ratified `en_units_v1`
(ADR-0127) and `en_numerics_v1` (ADR-0128) packs.
The baseline `train_sample_coverage_report.json` (#181) shows the
target shapes appear in real refusals: `Tina makes $18.00 an hour`,
`Aaron and his brother Carson each saved up $40`, hyphenated forms
like `10 one-hour videos`. Most refusals fail on *structural*
clause shape first (verb, multi-clause), so admission lift on the
probe requires sibling iterations (G.1 verbs, G.4 multi-clause) to
land alongside. G.3 alone is **necessary but not sufficient** for
probe lift; the load-bearing measurement for this iteration is the
**axis lane**.
## Decision
### Scope (v1 — closed set)
1. **Money symbol literal** — `$N` and `$N.NN` (1–2 decimal places).
`$N.NNN` (3+ decimals) refused.
2. **Money word form** — `<N> dollars` and `<N> cents`. Recognized
via existing unit-slot path; normalized at candidate build.
3. **Hyphenated multi-word cardinal** — `twenty-five`, `ninety-nine`,
etc. Resolved via
`language_packs.numerics_loader.parse_compound_cardinal`.
### Canonical money normalization (load-bearing)
All money values normalize to **integer cents, unit `'cents'`**.
This is pinned by `en_units_v1`:
```jsonl
{"surface": "money", "canonical_unit": "cent",
"is_canonical_for_dimension": true, "dimension": "money"}
```
`cent` is the canonical surface; `lookup_unit("cent").plural` is
`"cents"`. The parser canonicalizes both money symbols and the
word forms `N dollars` / `N cents` to plural `cents` so the
question-side canonicalization (`How many cents does X have?` →
`Unknown.unit='cents'`) matches by exact equality.
`dollar` deliberately does **not** become the surface unit, even
though it's more user-natural. Diverging from a ratified pack's
`canonical_unit` because the alternative sounds nicer is the small
precedent that erodes the "consult the pack" architecture. The
realizer/formatter is the right layer for user-facing display of
"$40.00" — that's a separate concern.
### Refusal probes (closed set)
- `$N.NNN+` (3+ decimal places) — out-of-scope precision.
- `N/0` — division by zero.
- Unrecognized hyphenated compositions (`five-and-a-half`,
`gobbledy-gook`) — not resolvable via
`parse_compound_cardinal`.
- Percent literals (`50%`) — out-of-scope.
### Out-of-scope, explicitly deferred to G.3.1+
- **Fractions in initial-possession / operations** (`N/M`). The
`_resolve_value` resolver supports `N/M` token-level (returns
`Fraction(N,M)` → float), but no axis cases exercise it
end-to-end because the initial-possession regex's "of <NP>"
substance-qualifier handling needs widening to admit
`"Bob has 3/4 of a cup."` cleanly. Token-level recognition is
ratified now; pipeline-level use is G.3.1.
- **Multi-currency** (`¢ € £ ¥ ₱`). US-only in v1.
- **Word-number compositions with adjective insertion** (`five
full boxes` per ADR-0127 substance-qualifier precedent).
- **Multi-token space-separated cardinals** (`one hundred`, `two
thousand`). `parse_compound_cardinal` supports these; the parser
doesn't yet match them in the value slot because they'd span
the unit slot boundary. Defer.
- **Scientific notation, locale separators, percentages.**
## What changed in code
### `generate/math_candidate_parser.py`
- New constants for the three widened value shapes (`_MONEY_SYMBOL`,
`_SLASH_FRACTION`, `_HYPHENATED_CARDINAL`); `_VALUE` widened to
include them in alternation.
- `_resolve_value(token)` refactored to return `_ResolvedValue |
None` (was `int`). Returns `unit_override='cents'` for money
symbols; returns `None` (refusal) for out-of-scope shapes.
- `_INITIAL_HAS_RE` unit slot made optional (money symbols carry
their unit implicitly); trailing `(?:in|of|for|with)` preposition
phrases discardable.
- `extract_initial_candidates` + `_build_op_candidate` updated to
honor `unit_override`, apply dollar/dollars → cents normalization
via `_money_unit_normalization`, and emit no candidate when neither
the resolver nor the regex provides a unit.
### `generate/math_roundtrip.py`
- New `_unit_grounds(unit_token, source_span, haystack_tokens)`
helper: word-token containment plus money-aware grounding (when
unit is `cent`/`cents`, accept `$` in source or `dollar`/`dollars`
in tokens).
- `_value_grounds` widened to handle money symbols (digit-parts of
`$N.NN` must each be in source tokens), slash fractions
(numerator + denominator both ground), hyphenated cardinals
(every component grounds OR the compound's digit form grounds).
- `roundtrip_admissible` unit check upgraded from `_token_in` to
`_unit_grounds`.
### `generate/math_candidate_graph.py`
- `_initial_admissible` and `_question_admissible` unit checks
upgraded to `_unit_grounds`.
### New files
- `evals/math_capability_axes/G3_numerics/v1/cases.jsonl` — 26
curated cases (5 per positive class + 6 refusal probes).
- `evals/math_capability_axes/G3_numerics/v1/runner.py` — pure
adapter over `evals.gsm8k_math.runner._score_one_candidate_graph`;
byte-equal `report.json` across runs.
- `tests/test_adr_0131_G3_numerics.py` — 27 tests.
## Evidence
- **Axis lane** (`python3 -m
evals.math_capability_axes.G3_numerics.v1.runner`):
- 20/20 positive cases solved correct
- 6/6 refusal probes refused with typed reason
- `solved_wrong == 0` (safety rail intact)
- `correct_rate_on_positive_cases == 1.0`
- report.json byte-equal across two runs
- **Test suite**: 27/27 pass in 0.19s.
- **Regression**: 420 existing candidate-parser + math-parser +
pack tests pass with the widening.
- **GSM8K probe** (`evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`):
unchanged at 0/50 admission (probe uses the legacy parser path,
not the candidate-graph pipeline this iteration extends).
`admitted_wrong == 0` (safety rail) preserved.
## Honest scope-limit disclosure
ADR-0131.G's discipline says each iteration's GSM8K-probe gate is
"`admission_rate` strictly increases OR a refused-reason family is
deliberately reduced." Neither holds for G.3 in isolation, because:
1. The `run_coverage_probe.py` probe goes through
`evals.gsm8k_math.runner._score_one` → `parse_problem` (the
legacy first-match-wins parser, ADR pre-0126), not the
candidate-graph pipeline G.3 extends.
2. Even if the probe consulted the candidate graph, most
money-bearing GSM8K cases (`Tina makes $18.00 an hour`) fail
first on the *verb* (`makes`, rate-introducing) or *multi-clause*
shape (`Aaron and his brother Carson each saved up $40`); the
money literal is a *downstream* refusal cause.
G.3's load-bearing gate is therefore the **axis lane** (full
end-to-end correctness on 20 curated cases that *do* exercise the
new capability through the candidate-graph pipeline). Probe lift
will accumulate as sibling axes (G.1 verb classes, G.4 multi-
clause) land alongside.
**Reserved follow-up**: a small probe-infrastructure ADR should
switch `run_coverage_probe.py` to call
`_score_one_candidate_graph` instead of `_score_one`, so future
G.<n> iterations show probe lift. That's a one-line change but
needs its own scoped PR (it shifts the probe's measurement
substrate; the resulting admission_rate delta should be
explicitly attributed, not buried inside a capability-axis PR).
## Composition with other in-flight work
- **L9 / ADR-0131.G.1 (verb classes)**: lands the verbs that unlock
`Tina makes $18.00 an hour` shape on the probe; G.3 already
handles the `$18.00` literal so admission flips when G.1 lands.
- **L10 / ADR-0131.G.2 (comparatives)**: independent; no
literal-class interaction.
- **L12 / ADR-0131.G.4 (multi-clause)**: unlocks `Aaron and his
brother Carson each saved up $40` shape; G.3 already handles
the `$40` literal.
- **ADR-0131.G.3.1 (deferred follow-up)**: fractions end-to-end,
multi-currency (`¢ € £ ¥ ₱`), space-separated multi-word
cardinals (`one hundred`), word-number-adjective compositions
(`five full boxes`).
## CLAUDE.md PR-checklist answers
- **Capability/performance/security added or protected:** Adds
pack-consuming literal recognition for money symbols, money
word forms, and hyphenated multi-word cardinals at the
candidate-graph parser layer.
- **Invariant proving the field remains valid:** `solved_wrong ==
0` on the axis lane (27 tests); `admitted_wrong == 0` on the
GSM8K probe (preserved).
- **CLI/eval lane proving correctness:** `python3 -m
evals.math_capability_axes.G3_numerics.v1.runner` and `pytest
tests/test_adr_0131_G3_numerics.py`.
- **Avoided hidden normalization / stochastic fallback /
approximate recall / unreviewed mutation:** Yes. All lookups
are deterministic, pack-driven (`en_units_v1`,
`en_numerics_v1`). Money normalization is a single fixed rule
(cents). Refusal-first preserved at every layer.
- **Trust boundary:** Inputs are user-controlled text → parser
regex; widened regex stays bounded (closed alternation set,
no backreferences, no catastrophic-backtracking-risk
patterns). Pack loader paths consult
`language_packs/data/<pack_id>/` only; no dynamic imports, no
filesystem traversal beyond pack root.

26
evals/math_capability_axes/G3_numerics/v1/cases.jsonl Normal file
View file

@ -0,0 +1,26 @@
{"case_id":"g3-money-sym-int-01","class":"money_symbol_integer","problem":"Bob has $40. How many cents does Bob have?","expected_answer":4000,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-int-02","class":"money_symbol_integer","problem":"Sarah has $25 in her wallet. How many cents does Sarah have?","expected_answer":2500,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-int-03","class":"money_symbol_integer","problem":"Tina has $100. Tina spends $40. How many cents does Tina have?","expected_answer":6000,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-int-04","class":"money_symbol_integer","problem":"Sam has $50. Sam earns $20. How many cents does Sam have?","expected_answer":7000,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-int-05","class":"money_symbol_integer","problem":"Pat has $60. Pat gives $25 to Jordan. How many cents does Pat have?","expected_answer":3500,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-dec-01","class":"money_symbol_decimal","problem":"Tina has $18.00 in savings. How many cents does Tina have?","expected_answer":1800,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-dec-02","class":"money_symbol_decimal","problem":"Bob has $2.50 in change. How many cents does Bob have?","expected_answer":250,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-dec-03","class":"money_symbol_decimal","problem":"Maya has $7.5 in her piggy bank. How many cents does Maya have?","expected_answer":750,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-dec-04","class":"money_symbol_decimal","problem":"Liam has $12.25. Liam spends $5.50. How many cents does Liam have?","expected_answer":675,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-sym-dec-05","class":"money_symbol_decimal","problem":"Noah has $9.99 in his pocket. How many cents does Noah have?","expected_answer":999,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-word-01","class":"money_word","problem":"Bob has 40 dollars. How many cents does Bob have?","expected_answer":4000,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-word-02","class":"money_word","problem":"Anna has 75 dollars. Anna spends 30 dollars. How many cents does Anna have?","expected_answer":4500,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-word-03","class":"money_word","problem":"Charlie has 250 cents. How many cents does Charlie have?","expected_answer":250,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-word-04","class":"money_word","problem":"Diana has 100 dollars. Diana earns 50 dollars. How many cents does Diana have?","expected_answer":15000,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-money-word-05","class":"money_word","problem":"Ethan has 20 dollars. Ethan gives 5 dollars to Felix. How many cents does Ethan have?","expected_answer":1500,"expected_unit":"cents","expected_outcome":"solved_correct"}
{"case_id":"g3-hyphen-card-01","class":"hyphenated_cardinal","problem":"Bob has twenty-five apples. How many apples does Bob have?","expected_answer":25,"expected_unit":"apples","expected_outcome":"solved_correct"}
{"case_id":"g3-hyphen-card-02","class":"hyphenated_cardinal","problem":"The store has fifty-five toys. How many toys does the store have?","expected_answer":55,"expected_unit":"toys","expected_outcome":"solved_correct"}
{"case_id":"g3-hyphen-card-03","class":"hyphenated_cardinal","problem":"Jane has ninety-nine pencils. Jane gives twelve pencils to Tom. How many pencils does Jane have?","expected_answer":87,"expected_unit":"pencils","expected_outcome":"solved_correct"}
{"case_id":"g3-hyphen-card-04","class":"hyphenated_cardinal","problem":"Bob has thirty-three apples. Bob buys seven apples. How many apples does Bob have?","expected_answer":40,"expected_unit":"apples","expected_outcome":"solved_correct"}
{"case_id":"g3-hyphen-card-05","class":"hyphenated_cardinal","problem":"Liam has twenty-one books. Liam loses three books. How many books does Liam have?","expected_answer":18,"expected_unit":"books","expected_outcome":"solved_correct"}
{"case_id":"g3-refuse-money-prec-01","class":"refuse_money_precision","problem":"Bob has $40.000 in savings. How many cents does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g3-refuse-money-prec-02","class":"refuse_money_precision","problem":"Sarah has $1.2345 in change. How many cents does Sarah have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g3-refuse-div-zero-01","class":"refuse_division_by_zero","problem":"Bob has 5/0 apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g3-refuse-hyphen-01","class":"refuse_unknown_compound","problem":"Bob has five-and-a-half apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g3-refuse-hyphen-02","class":"refuse_unknown_compound","problem":"Bob has gobbledy-gook apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}
{"case_id":"g3-refuse-percent-01","class":"refuse_percentage","problem":"Bob has 50% apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"}

343
evals/math_capability_axes/G3_numerics/v1/report.json Normal file
View file

@ -0,0 +1,343 @@
{
"adr": "0131.G.3",
"axis": "numeric_literals",
"cases_path": "evals/math_capability_axes/G3_numerics/v1/cases.jsonl",
"class_counts": {
"hyphenated_cardinal": 5,
"money_symbol_decimal": 5,
"money_symbol_integer": 5,
"money_word": 5,
"refuse_division_by_zero": 1,
"refuse_money_precision": 2,
"refuse_percentage": 1,
"refuse_unknown_compound": 2
},
"metrics": {
"cases_total": 26,
"correct_rate_on_positive_cases": 1.0,
"overall_pass": true,
"refused_as_expected": 6,
"solved_correct": 20,
"solved_wrong": 0,
"wrong_count_is_zero": true
},
"per_case": [
{
"actual_answer": 4000.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-int-01",
"class": "money_symbol_integer",
"expected_answer": 4000,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "742b18b40f28213ce2361e208b8deb01cedf2ab16fe40b001e570070b5c48fdd",
"verdict": "solved_correct"
},
{
"actual_answer": 2500.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-int-02",
"class": "money_symbol_integer",
"expected_answer": 2500,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "1e2ba5472e84656edabc744bfd036b001f342ac6b76760e411d5b51cf524753b",
"verdict": "solved_correct"
},
{
"actual_answer": 6000.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-int-03",
"class": "money_symbol_integer",
"expected_answer": 6000,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "0db7b91caeb1721c99982938e5bafd45df5d3ba849982494e82984dbefaf8c76",
"verdict": "solved_correct"
},
{
"actual_answer": 7000.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-int-04",
"class": "money_symbol_integer",
"expected_answer": 7000,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "9efdb9599d909c9501fae743b6b07e496febc474991f27090fc02b2c0ce24aef",
"verdict": "solved_correct"
},
{
"actual_answer": 3500.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-int-05",
"class": "money_symbol_integer",
"expected_answer": 3500,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "0113e1b71499c4ad66008115e9b1e611790c432eb8c430f42ad64312a4b01484",
"verdict": "solved_correct"
},
{
"actual_answer": 1800.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-dec-01",
"class": "money_symbol_decimal",
"expected_answer": 1800,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "cbe7694b94ec417069feb636114da8b3dfbbd6df5ba4226c0a17cf896a564a96",
"verdict": "solved_correct"
},
{
"actual_answer": 250.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-dec-02",
"class": "money_symbol_decimal",
"expected_answer": 250,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "2a2b8050917b97d120fd90cd5a5246b6626a4e0a5be48b42a87aab39fba2112d",
"verdict": "solved_correct"
},
{
"actual_answer": 750.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-dec-03",
"class": "money_symbol_decimal",
"expected_answer": 750,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "32543d70d2caf8befb616082361d7c59107e1f9b8785180da6f44bd4bf22ce72",
"verdict": "solved_correct"
},
{
"actual_answer": 675.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-dec-04",
"class": "money_symbol_decimal",
"expected_answer": 675,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "dd6c3b8cc0c83a0fe9d65f993759a5f14243aae9a6086e7ce4ca92c26897350b",
"verdict": "solved_correct"
},
{
"actual_answer": 999.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-sym-dec-05",
"class": "money_symbol_decimal",
"expected_answer": 999,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "c4bda23a263700abd5e03c7d94d3393ae9a38a5587afcb2c374688db5b1f56df",
"verdict": "solved_correct"
},
{
"actual_answer": 4000.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-word-01",
"class": "money_word",
"expected_answer": 4000,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "742b18b40f28213ce2361e208b8deb01cedf2ab16fe40b001e570070b5c48fdd",
"verdict": "solved_correct"
},
{
"actual_answer": 4500.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-word-02",
"class": "money_word",
"expected_answer": 4500,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "0e222771bf42bd808f4d9d5794ac044d515e94a3b47377bdee695b6f8173b93c",
"verdict": "solved_correct"
},
{
"actual_answer": 250.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-word-03",
"class": "money_word",
"expected_answer": 250,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "a48d47cf0f4c41907df10058a246940352e5cae719c4a32778f80a6e88f46799",
"verdict": "solved_correct"
},
{
"actual_answer": 15000.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-word-04",
"class": "money_word",
"expected_answer": 15000,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "855733e843f7caa9d924aed87841ea21a67e74325e9445aae3f2477de3805dc2",
"verdict": "solved_correct"
},
{
"actual_answer": 1500.0,
"actual_outcome": "correct",
"actual_unit": "cents",
"case_id": "g3-money-word-05",
"class": "money_word",
"expected_answer": 1500,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "10ed9467df9c42de95ba8104b0f1ff72fc50579839a6d85ce95aed184add37c0",
"verdict": "solved_correct"
},
{
"actual_answer": 25.0,
"actual_outcome": "correct",
"actual_unit": "apples",
"case_id": "g3-hyphen-card-01",
"class": "hyphenated_cardinal",
"expected_answer": 25,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "5d486cf0eadaab2f52039c28b239c78ab58eca80bd154ab912be4acd03f06182",
"verdict": "solved_correct"
},
{
"actual_answer": 55.0,
"actual_outcome": "correct",
"actual_unit": "toys",
"case_id": "g3-hyphen-card-02",
"class": "hyphenated_cardinal",
"expected_answer": 55,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "a094f20be6f6664e499a6f6fa9e68dbef6a27c5fcdd29973053919dfa20a322a",
"verdict": "solved_correct"
},
{
"actual_answer": 87.0,
"actual_outcome": "correct",
"actual_unit": "pencils",
"case_id": "g3-hyphen-card-03",
"class": "hyphenated_cardinal",
"expected_answer": 87,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "bc0696023e69f665e1551344319e3eb40e0b7e97108b4f896b12b2eb9e85d6e5",
"verdict": "solved_correct"
},
{
"actual_answer": 40.0,
"actual_outcome": "correct",
"actual_unit": "apples",
"case_id": "g3-hyphen-card-04",
"class": "hyphenated_cardinal",
"expected_answer": 40,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "69a3edca720fa45b42ce4b6c0c068ef51d0655ef4cd7098012898260dac1c4da",
"verdict": "solved_correct"
},
{
"actual_answer": 18.0,
"actual_outcome": "correct",
"actual_unit": "books",
"case_id": "g3-hyphen-card-05",
"class": "hyphenated_cardinal",
"expected_answer": 18,
"expected_outcome": "solved_correct",
"reason": "",
"trace_hash": "fa82ed15b02dcbf1931623ab8a9cd1fcdeb4ae5e2ae15a0e50dcc6078fde5a2f",
"verdict": "solved_correct"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g3-refuse-money-prec-01",
"class": "refuse_money_precision",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has $40.000 in savings.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g3-refuse-money-prec-02",
"class": "refuse_money_precision",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Sarah has $1.2345 in change.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g3-refuse-div-zero-01",
"class": "refuse_division_by_zero",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has 5/0 apples.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g3-refuse-hyphen-01",
"class": "refuse_unknown_compound",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has five-and-a-half apples.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g3-refuse-hyphen-02",
"class": "refuse_unknown_compound",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has gobbledy-gook apples.'",
"trace_hash": null,
"verdict": "refused"
},
{
"actual_answer": null,
"actual_outcome": "refused",
"actual_unit": null,
"case_id": "g3-refuse-percent-01",
"class": "refuse_percentage",
"expected_answer": 0,
"expected_outcome": "refused",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob has 50% apples.'",
"trace_hash": null,
"verdict": "refused"
}
],
"schema_version": 1,
"verdict_counts": {
"refused": 6,
"solved_correct": 20
}
}

164
evals/math_capability_axes/G3_numerics/v1/runner.py Normal file
View file

@ -0,0 +1,164 @@
"""ADR-0131.G.3 — Numeric-literals capability-axis runner.
First sibling under ``evals/math_capability_axes/`` — the iteration
pattern from ADR-0131.G ("each ADR-0131.G.<n> extends a single
capability axis with its own curated coverage cases, independent of
GSM8K"). G.3's axis is numeric-literal recognition: money symbols
(``$N`` / ``$N.NN``), the word forms ``N dollars`` / ``N cents``, and
hyphenated multi-word cardinals (``twenty-five``).
The runner wraps :func:`evals.gsm8k_math.runner._score_one_candidate_graph`
(the candidate-graph pipeline ADR-0126 introduced) so that any future
G.<n> axis extending the same parser layer shows up on the same lane
without parallel infrastructure.
Outcome classification mirrors the GSM8K runner:
| Pipeline result | Outcome |
|------------------------|-----------|
| parser+solver+verifier OK and answer/unit match | ``solved_correct`` |
| parser+solver OK but verifier fails or answer mismatches | ``solved_wrong`` (gate: must be 0) |
| parser/solver refuses with typed reason | ``refused`` |
Cases ship with an ``expected_outcome`` so the runner can score
positive-coverage cases (``solved_correct``) AND adversarial-refusal
probes (``refused``) on the same axis. ``wrong == 0`` is preserved as
the load-bearing invariant per ADR-0114a Obligation #4.
The runner is pure / deterministic: same case set → byte-equal
``report.json`` across runs.
"""
from __future__ import annotations
import json
from collections import Counter
from pathlib import Path
from typing import Any
from evals.gsm8k_math.runner import _score_one_candidate_graph
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_REPORT_PATH = _HERE / "report.json"
def _load_cases() -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines():
if line.strip():
out.append(json.loads(line))
return out
def _adapt_case(raw: dict[str, Any]) -> dict[str, Any]:
"""Translate axis-case shape to the candidate-graph runner's expected
shape. Refusal cases pass ``expected_unit=""`` so the unit-mismatch
branch doesn't fire on cases that never reach the answer comparison.
"""
return {
"id": raw["case_id"],
"problem": raw["problem"],
"expected_answer": float(raw["expected_answer"]),
"expected_unit": raw.get("expected_unit", ""),
}
def _classify(actual_outcome: str, expected_outcome: str) -> str:
"""Map the candidate-graph runner's outcome + the case's expected
outcome into a unified axis-lane verdict.
- ``solved_correct``: pipeline returned ``correct`` AND expected was
``solved_correct``.
- ``refused``: pipeline returned ``refused`` AND expected was
``refused``.
- ``solved_wrong``: any disagreement — either ``correct`` for a
``refused`` case, ``wrong`` ever, or ``refused`` for a
``solved_correct`` case. All map to ``solved_wrong``, which the
lane gate requires to be zero.
"""
if expected_outcome == "solved_correct" and actual_outcome == "correct":
return "solved_correct"
if expected_outcome == "refused" and actual_outcome == "refused":
return "refused"
return "solved_wrong"
def build_report() -> dict[str, Any]:
raw_cases = _load_cases()
case_results: list[dict[str, Any]] = []
class_counts: Counter[str] = Counter()
verdict_counts: Counter[str] = Counter()
for raw in raw_cases:
cls = raw["class"]
expected = raw["expected_outcome"]
class_counts[cls] += 1
outcome = _score_one_candidate_graph(_adapt_case(raw))
verdict = _classify(outcome.outcome, expected)
verdict_counts[verdict] += 1
case_results.append({
"case_id": raw["case_id"],
"class": cls,
"expected_outcome": expected,
"actual_outcome": outcome.outcome,
"verdict": verdict,
"expected_answer": raw["expected_answer"],
"actual_answer": outcome.actual_answer,
"actual_unit": outcome.actual_unit,
"reason": outcome.reason,
"trace_hash": outcome.trace_hash,
})
total = len(raw_cases)
correct = verdict_counts.get("solved_correct", 0)
wrong = verdict_counts.get("solved_wrong", 0)
refused_expected = verdict_counts.get("refused", 0)
positive_count = sum(1 for r in raw_cases if r["expected_outcome"] == "solved_correct")
correct_rate_on_positive = (
correct / positive_count if positive_count else 0.0
)
return {
"schema_version": 1,
"adr": "0131.G.3",
"axis": "numeric_literals",
"cases_path": "evals/math_capability_axes/G3_numerics/v1/cases.jsonl",
"metrics": {
"cases_total": total,
"solved_correct": correct,
"solved_wrong": wrong,
"refused_as_expected": refused_expected,
"wrong_count_is_zero": wrong == 0,
"correct_rate_on_positive_cases": correct_rate_on_positive,
"overall_pass": wrong == 0 and (correct + refused_expected == total),
},
"class_counts": dict(sorted(class_counts.items())),
"verdict_counts": dict(sorted(verdict_counts.items())),
"per_case": case_results,
}
def write_report(report: dict[str, Any]) -> None:
_REPORT_PATH.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
report = build_report()
write_report(report)
m = report["metrics"]
print(f"axis: {report['axis']}")
print(f"cases_total: {m['cases_total']}")
print(f"solved_correct: {m['solved_correct']}")
print(f"solved_wrong: {m['solved_wrong']} (gate: must be 0)")
print(f"refused_as_expected: {m['refused_as_expected']}")
print(f"correct_rate_on_positive_cases: {m['correct_rate_on_positive_cases']:.1%}")
print(f"overall_pass: {m['overall_pass']}")
return 0 if m["overall_pass"] else 1
if __name__ == "__main__":
raise SystemExit(main())

8
generate/math_candidate_graph.py
View file

@ -154,13 +154,13 @@ def _initial_admissible(ic: CandidateInitial) -> bool:
Same shape as roundtrip_admissible but for the initial-possession
slot set (entity, anchor, value, unit)."""
from generate.math_roundtrip import _tokens, _value_grounds, _token_in
from generate.math_roundtrip import _tokens, _value_grounds, _token_in, _unit_grounds
haystack = _tokens(ic.source_span)
if not _token_in(ic.matched_anchor, haystack):
return False
if not _value_grounds(ic.matched_value_token, haystack):
return False
if not _token_in(ic.matched_unit_token, haystack):
if not _unit_grounds(ic.matched_unit_token, ic.source_span, haystack):
return False
# Entity token: for multi-word entities ("the boys"), all words
# must ground. Split + check each.
@ -172,9 +172,9 @@ def _initial_admissible(ic: CandidateInitial) -> bool:
def _question_admissible(qc: CandidateUnknown) -> bool:
"""Light structural ground-check for question candidates."""
from generate.math_roundtrip import _tokens, _token_in
from generate.math_roundtrip import _tokens, _token_in, _unit_grounds
haystack = _tokens(qc.source_span)
if not _token_in(qc.matched_unit_token, haystack):
if not _unit_grounds(qc.matched_unit_token, qc.source_span, haystack):
return False
if qc.matched_entity_token is not None:
for tok in qc.matched_entity_token.split():

229
generate/math_candidate_parser.py
View file

@ -98,14 +98,28 @@ class CandidateInitial:
# math_parser._INITIAL_HAS_RE's ADR-0123a entity slot.
_ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)"
# Numeric value: digit run OR word-form integer (one..twelve initially;
# WORD_NUMBERS table is wider but we cap the regex at the common range
# for syntactic parsing and let the filter handle ground-truth value
# equivalence).
# Numeric value alternation. Listed longest-form-first so the regex
# engine doesn't truncate on a shorter prefix:
# - Money symbol literal: ``$N`` or ``$N.NN`` (1-2 decimal places).
# ADR-0131.G.3. ``$N.NNN`` (3+ decimals) deliberately not matched
# — refused as out-of-scope so wrong == 0 is preserved.
# - Slash fraction literal: ``N/M``. Denominator-zero refused at
# resolve time, not regex.
# - Hyphenated multi-word cardinal: ``twenty-five``, ``ninety-nine``.
# Resolved via :func:`language_packs.numerics_loader.parse_compound_cardinal`.
# - Digit run.
# - Single-word cardinal (legacy ``WORD_NUMBERS`` set).
_MONEY_SYMBOL: Final[str] = r"\$\d+(?:\.\d{1,2})?"
_SLASH_FRACTION: Final[str] = r"\d+/\d+"
_HYPHENATED_CARDINAL: Final[str] = r"[A-Za-z]+-[A-Za-z]+"
_WORD_NUM_OPTIONS: Final[str] = "|".join(
re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True)
)
_VALUE: Final[str] = rf"(?:\d+|{_WORD_NUM_OPTIONS})"
_VALUE: Final[str] = (
rf"(?:{_MONEY_SYMBOL}|{_SLASH_FRACTION}|"
rf"{_HYPHENATED_CARDINAL}|"
rf"\d+|{_WORD_NUM_OPTIONS})"
)
# Verb alternation built from the permissive registry. Pre-compute one
# pattern per kind so we can attribute matched verbs to candidates.
@ -127,11 +141,17 @@ _TRANSFER_VERBS_PATTERN: Final[str] = _verbs_pattern(TRANSFER_VERBS)
_INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<entity>{_ENTITY})\s+"
rf"(?P<anchor>has|have)\s+"
rf"(?P<value>{_VALUE})\s+"
r"(?P<unit>\w+)"
rf"(?P<value>{_VALUE})"
# ADR-0131.G.3: unit slot is optional. Money-symbol value literals
# (``$40``) carry their unit implicitly (``cent``); a missing unit
# slot is admissible IFF the value resolves with a unit override.
# Non-money values without a unit slot are refused at resolve time.
r"(?:\s+(?P<unit>\w+))?"
# ADR-0127 substance qualifier: "Sam has 5 feet of rope" — the
# 'of <NP>' tail is grammatically real but arithmetically inert.
r"(?:\s+of\s+.+)?"
# ADR-0131.G.3: 'in <NP>' is also discardable
# ("Bob has $40 in savings"; "Bob has $40 in his wallet").
r"(?:\s+(?:of|in|for|with)\s+.+)?"
r"\s*\.?$"
)
@ -162,10 +182,79 @@ def _normalize_entity(raw: str) -> str:
return e
def _resolve_value(value_token: str) -> int:
if value_token.isdigit():
return int(value_token)
return WORD_NUMBERS[value_token.lower()]
@dataclass(frozen=True, slots=True)
class _ResolvedValue:
"""Resolved value-slot reading.
ADR-0131.G.3 widens the value slot beyond integer + single-word
cardinal to include money literals (``$N`` / ``$N.NN``), slash
fractions (``N/M``), and hyphenated multi-word cardinals
(``twenty-five``). Money literals carry an implicit canonical unit
(``cent``); when set, ``unit_override`` replaces the unit slot the
regex captured (or fills it when the unit slot is absent).
"""
value: int | float
unit_override: str | None
# Money: canonical normalization to integer cents (en_units_v1
# ``canonical_unit`` for the ``money`` dimension is ``cent``).
_MONEY_UNIT: Final[str] = "cents"
def _resolve_value(value_token: str) -> _ResolvedValue | None:
"""Resolve a value-slot token into a numeric value + optional unit
override. Returns ``None`` on refusal (indefinite quantifier,
division-by-zero in slash fraction, unrecognized hyphenated form,
unparseable money).
Refusal at this layer is first-class: a ``None`` upstream means the
candidate is not emitted, which preserves ``wrong == 0`` per
ADR-0114a Obligation #4.
"""
if not value_token:
return None
t = value_token.strip()
# Money symbol literal: $N or $N.NN.
if t.startswith("$"):
body = t[1:]
if re.fullmatch(r"\d+", body):
return _ResolvedValue(int(body) * 100, _MONEY_UNIT)
if re.fullmatch(r"\d+\.\d{1,2}", body):
# round() avoids float drift: $2.50 → 250, not 249 or 251.
return _ResolvedValue(int(round(float(body) * 100)), _MONEY_UNIT)
return None # $N.NNN (3+ decimals) refused — out-of-scope.
# Slash fraction literal: N/M with M > 0.
if "/" in t:
m = re.fullmatch(r"(\d+)/(\d+)", t)
if m is None:
return None
num, den = int(m.group(1)), int(m.group(2))
if den == 0:
return None # division-by-zero refused.
if num % den == 0:
return _ResolvedValue(num // den, None)
return _ResolvedValue(num / den, None)
# Digit run.
if t.isdigit():
return _ResolvedValue(int(t), None)
# Indefinite quantifier (ADR-0128.4) — refuse, never guess.
if _is_indefinite_quantifier(t):
return None
# Hyphenated multi-word cardinal: twenty-five, ninety-nine, etc.
if "-" in t:
from language_packs.numerics_loader import parse_compound_cardinal
parsed = parse_compound_cardinal(t)
if parsed is None:
return None # Unrecognized hyphenated form refused.
return _ResolvedValue(parsed, None)
# Single-word cardinal (legacy WORD_NUMBERS table).
lower = t.lower()
if lower in WORD_NUMBERS:
return _ResolvedValue(WORD_NUMBERS[lower], None)
return None
def _is_indefinite_quantifier(token: str) -> bool:
@ -186,6 +275,25 @@ def _is_indefinite_quantifier(token: str) -> bool:
return False
def _money_unit_normalization(
value: int | float, unit: str | None
) -> tuple[int | float, str | None]:
"""ADR-0131.G.3 — normalize ``dollar``/``dollars`` surface unit to the
canonical money unit (``cent``).
``en_units_v1`` pins ``cent`` as ``canonical_unit`` for the ``money``
dimension; ``dollar`` is convenience surface. A ``dollar`` value is
100 ``cent``. Done at the candidate-build site so every money-bearing
path normalizes uniformly (Quantity equality is exact — mixing
``cent`` and ``dollar`` units would silently break arithmetic).
"""
if unit is None:
return value, unit
if unit.lower() in ("dollar", "dollars"):
return value * 100, _MONEY_UNIT
return value, unit
def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
"""Return all admissible initial-possession candidates for ``sentence``.
@ -193,9 +301,14 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
1. "<Entity> has <N> <unit> [of <substance>]" — canonical.
2. "There are <N> <unit> [in <place>]" — implicit-subject shape.
ADR-0128.4: if the value slot resolves to an indefinite quantifier
(`some kids`, `many things`), no candidate is emitted (refusal
preserves wrong == 0).
Value-slot widenings (ADR-0131.G.3) apply to both shapes via
:func:`_resolve_value`: money literals (``$N`` / ``$N.NN``), slash
fractions (``N/M``), hyphenated multi-word cardinals (``twenty-five``).
Refusal-first: indefinite quantifiers, division-by-zero fractions,
unrecognized compound forms, and money literals with >2 decimals
all return ``None`` from :func:`_resolve_value` and emit no
candidate (preserves ``wrong == 0`` per ADR-0114a Obligation #4).
"""
s = sentence.strip().rstrip(".")
out: list[CandidateInitial] = []
@ -203,32 +316,53 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
m = _INITIAL_HAS_RE.match(s)
if m is not None:
value_raw = m.group("value")
if not _is_indefinite_quantifier(value_raw):
rv = _resolve_value(value_raw)
if rv is not None:
entity = _normalize_entity(m.group("entity"))
value = _resolve_value(value_raw)
unit_raw = m.group("unit")
unit = _canonicalize_unit(unit_raw)
out.append(
CandidateInitial(
initial=InitialPossession(
entity=entity,
quantity=Quantity(value=value, unit=unit),
),
source_span=sentence,
matched_anchor=m.group("anchor"),
matched_value_token=value_raw,
matched_unit_token=unit_raw,
matched_entity_token=m.group("entity"),
unit_raw = m.group("unit") # may be None when value is money symbol
# Unit precedence: explicit override from value (money symbol)
# wins over the regex's unit slot. The unit slot is required
# for non-money values; if both are absent the candidate
# cannot be constructed.
resolved_unit: str | None
if rv.unit_override is not None:
resolved_unit = rv.unit_override
elif unit_raw is not None:
resolved_unit = _canonicalize_unit(unit_raw)
else:
resolved_unit = None
if resolved_unit is not None:
value, final_unit = _money_unit_normalization(rv.value, resolved_unit)
assert final_unit is not None
out.append(
CandidateInitial(
initial=InitialPossession(
entity=entity,
quantity=Quantity(value=value, unit=final_unit),
),
source_span=sentence,
matched_anchor=m.group("anchor"),
matched_value_token=value_raw,
matched_unit_token=unit_raw if unit_raw is not None else final_unit,
matched_entity_token=m.group("entity"),
)
)
)
m2 = _INITIAL_THERE_ARE_RE.match(s)
if m2 is not None:
value_raw = m2.group("value")
if not _is_indefinite_quantifier(value_raw):
rv = _resolve_value(value_raw)
if rv is not None:
unit_raw = m2.group("unit")
unit = _canonicalize_unit(unit_raw)
value = _resolve_value(value_raw)
assert unit_raw is not None # there-are regex always captures unit slot
if rv.unit_override is not None:
unit_str: str = rv.unit_override
else:
unit_str = _canonicalize_unit(unit_raw)
v_norm, u_norm = _money_unit_normalization(rv.value, unit_str)
assert u_norm is not None
value: int | float = v_norm
unit: str = u_norm
place = m2.group("place")
# When a 'in <place>' phrase is present, treat the place as
# the implicit entity. Otherwise use the unit's plural as
@ -342,15 +476,28 @@ def _build_op_candidate(
m: re.Match[str], kind: str, source: str
) -> CandidateOperation | None:
"""Build a CandidateOperation from a regex match. Returns None if
the match lacks a required slot (e.g. unit token absent — P2 does
not emit unit-inherited candidates)."""
unit_raw = m.group("unit")
if unit_raw is None:
the value cannot be resolved or if no unit can be determined
(unit slot absent AND value carries no implicit unit override).
"""
value_raw = m.group("value")
rv = _resolve_value(value_raw)
if rv is None:
return None
unit = _canonicalize_unit(unit_raw)
unit_raw = m.group("unit")
# ADR-0131.G.3: a money-symbol value carries its unit implicitly
# (override 'cent'); for plain-numeric values, the unit slot must
# be present.
if rv.unit_override is not None:
unit: str = rv.unit_override
elif unit_raw is not None:
unit = _canonicalize_unit(unit_raw)
else:
return None # P2 does not emit unit-inherited candidates.
subject = _normalize_entity(m.group("subject"))
verb = m.group("verb").lower()
value = _resolve_value(m.group("value"))
value, unit_normalized = _money_unit_normalization(rv.value, unit)
assert unit_normalized is not None
unit = unit_normalized
target_raw = m.group("target") if "target" in m.groupdict() else None
target = target_raw if target_raw is not None else None
@ -372,7 +519,7 @@ def _build_op_candidate(
source_span=source,
matched_verb=verb,
matched_value_token=m.group("value"),
matched_unit_token=unit_raw,
matched_unit_token=unit_raw if unit_raw is not None else unit,
matched_actor_token=m.group("subject"),
matched_target_token=target,
)

61
generate/math_roundtrip.py
View file

@ -272,6 +272,30 @@ def _token_in(needle: str, haystack_tokens: frozenset[str]) -> bool:
return needle.lower() in haystack_tokens
def _unit_grounds(
unit_token: str,
source_span: str,
haystack_tokens: frozenset[str],
) -> bool:
"""A unit token grounds if it appears as a word token in source.
ADR-0131.G.3 widening: when the canonical money unit ``cent`` is
claimed, the source's ``$`` symbol counts as grounding evidence —
the word-boundary tokenizer strips ``$`` so it must be inspected
on the raw source span rather than the token set. Similarly for
``dollar``: an author may write either ``$N`` or ``N dollars``;
both ground a money unit.
"""
if _token_in(unit_token, haystack_tokens):
return True
if unit_token.lower() in ("cent", "cents"):
if "$" in source_span:
return True
if "dollar" in haystack_tokens or "dollars" in haystack_tokens:
return True
return False
def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
"""A numeric value grounds if its surface token appears, OR if the token
is a digit-string and any equivalent word-form appears, OR if it's a
@ -282,7 +306,42 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
1-12 to the full pack cardinal range (0-1000+ plus compound rule). The
hard-coded WORD_NUMBERS remains as a fast path and as a fallback if
the pack is unavailable; the pack adds, never replaces.
ADR-0131.G.3 widens the literal-class grounding:
- Money symbol ``$N`` / ``$N.NN`` grounds when every digit run on
either side of the optional decimal appears as a token in the
source. The ``$`` itself is dropped by the word-boundary
tokenizer; what survives is exactly the digit form an author
would write.
- Slash fraction ``N/M`` grounds when both numerator and
denominator digit tokens appear.
- Hyphenated multi-word cardinal (``twenty-five``) grounds when
every component lemma is a token (the tokenizer splits on
hyphens), OR the compound's integer value's digit form appears.
"""
# ADR-0131.G.3 widenings (handled first; the trailing existing path
# would never recognize these surface shapes).
if value_token.startswith("$"):
body = value_token[1:]
parts = [p for p in body.split(".") if p]
return bool(parts) and all(p in haystack_tokens for p in parts)
if "/" in value_token:
m = re.fullmatch(r"(\d+)/(\d+)", value_token)
if m is not None:
return m.group(1) in haystack_tokens and m.group(2) in haystack_tokens
if "-" in value_token and not value_token[0].isdigit():
try:
from language_packs.numerics_loader import parse_compound_cardinal
parsed = parse_compound_cardinal(value_token)
if parsed is not None:
components = [c for c in value_token.lower().split("-") if c]
if all(c in haystack_tokens for c in components):
return True
if str(parsed) in haystack_tokens:
return True
except Exception:
pass
if _token_in(value_token, haystack_tokens):
return True
lowered = value_token.lower()
@ -366,7 +425,7 @@ def roundtrip_admissible(c: CandidateOperation) -> bool:
# for comparison operands without explicit unit phrasing
# ("Sam has twice as many as Tom").
if c.matched_unit_token:
if not _token_in(c.matched_unit_token, haystack):
if not _unit_grounds(c.matched_unit_token, c.source_span, haystack):
return False
else:
if not isinstance(c.op.operand, Comparison):

200
tests/test_adr_0131_G3_numerics.py Normal file
View file

@ -0,0 +1,200 @@
"""ADR-0131.G.3 — Numeric-literals capability-axis lane tests.
Gate (must all pass for the lane to be considered green):
- safety rail: ``solved_wrong == 0`` on the axis lane
- safety rail: ``admitted_wrong == 0`` on the GSM8K probe (unchanged
by this iteration — G.3 widens the candidate-graph parser, which
the probe currently does not consult, so admission is not expected
to move; the wrong-count invariant is what's gated)
- axis-lane correctness: every ``solved_correct`` case in
``cases.jsonl`` passes end-to-end; every ``refused`` probe refuses
with a typed reason at parser-or-solver layer
- per-class diversity: at least one case per non-refusal class
- replay determinism: ``report.json`` byte-equal across two runs
- resolver-level invariants for the new literal shapes
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.math_capability_axes.G3_numerics.v1 import runner as axis_runner
from generate.math_candidate_parser import _resolve_value
_AXIS_DIR = Path(__file__).resolve().parent.parent / "evals" / "math_capability_axes" / "G3_numerics" / "v1"
_CASES_PATH = _AXIS_DIR / "cases.jsonl"
# Closed set of classes this iteration is responsible for. The lane
# refuses to silently grow the class taxonomy — adding a new class is
# an ADR-level scope change.
_KNOWN_POSITIVE_CLASSES = frozenset({
"money_symbol_integer",
"money_symbol_decimal",
"money_word",
"hyphenated_cardinal",
})
_KNOWN_REFUSAL_CLASSES = frozenset({
"refuse_money_precision",
"refuse_division_by_zero",
"refuse_unknown_compound",
"refuse_percentage",
})
def _load_cases() -> list[dict]:
out = []
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines():
if line.strip():
out.append(json.loads(line))
return out
# ---------------------------------------------------------------------------
# Resolver-level invariants (cheap, independent of the lane runner).
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
"token, expected_value, expected_unit_override",
[
("$40", 4000, "cents"),
("$2.50", 250, "cents"),
("$18.00", 1800, "cents"),
("$0.99", 99, "cents"),
("twenty-five", 25, None),
("ninety-nine", 99, None),
("3/4", 0.75, None),
("5", 5, None),
("twelve", 12, None),
],
)
def test_resolve_value_admits_new_literals(token, expected_value, expected_unit_override):
rv = _resolve_value(token)
assert rv is not None, f"{token!r} should resolve"
assert rv.value == expected_value
assert rv.unit_override == expected_unit_override
@pytest.mark.parametrize(
"token",
[
"$40.000", # >2 decimals — money precision out of scope
"$1.2345",
"5/0", # division-by-zero
"five-and-a-half", # unrecognized compound
"gobbledy-gook", # unrecognized compound
"50%", # percentage out of scope
"some", # indefinite quantifier
],
)
def test_resolve_value_refuses_out_of_scope(token):
assert _resolve_value(token) is None, f"{token!r} should refuse"
# ---------------------------------------------------------------------------
# Dataset integrity invariants.
# ---------------------------------------------------------------------------
def test_dataset_case_ids_unique():
cases = _load_cases()
ids = [c["case_id"] for c in cases]
assert len(ids) == len(set(ids)), "duplicate case_id"
def test_dataset_class_taxonomy_is_closed():
cases = _load_cases()
seen = {c["class"] for c in cases}
allowed = _KNOWN_POSITIVE_CLASSES | _KNOWN_REFUSAL_CLASSES
extra = seen - allowed
assert not extra, f"unknown class(es) — extend ADR scope before adding: {extra}"
def test_dataset_every_positive_class_has_at_least_one_case():
cases = _load_cases()
by_class = {c["class"] for c in cases if c["expected_outcome"] == "solved_correct"}
missing = _KNOWN_POSITIVE_CLASSES - by_class
assert not missing, f"missing coverage for positive classes: {missing}"
def test_dataset_every_refusal_class_has_at_least_one_case():
cases = _load_cases()
by_class = {c["class"] for c in cases if c["expected_outcome"] == "refused"}
missing = _KNOWN_REFUSAL_CLASSES - by_class
assert not missing, f"missing coverage for refusal classes: {missing}"
# ---------------------------------------------------------------------------
# Lane-level invariants (run the full runner).
# ---------------------------------------------------------------------------
def test_axis_lane_safety_rail_no_wrong_answers():
"""ADR-0114a Obligation #4 — refusal-first; wrong-count must be 0."""
report = axis_runner.build_report()
assert report["metrics"]["solved_wrong"] == 0, (
f"wrong != 0: {report['verdict_counts']}"
)
assert report["metrics"]["wrong_count_is_zero"] is True
def test_axis_lane_all_positive_cases_solved_correct():
report = axis_runner.build_report()
assert report["metrics"]["correct_rate_on_positive_cases"] == 1.0
def test_axis_lane_all_refusal_probes_refused_typed():
report = axis_runner.build_report()
refusal_cases = [c for c in report["per_case"] if c["expected_outcome"] == "refused"]
for c in refusal_cases:
assert c["actual_outcome"] == "refused", (
f"{c['case_id']}: expected refused, got {c['actual_outcome']}"
)
assert c["reason"], f"{c['case_id']}: refusal must carry typed reason"
def test_axis_lane_overall_pass():
report = axis_runner.build_report()
assert report["metrics"]["overall_pass"] is True
# ---------------------------------------------------------------------------
# Replay determinism — load-bearing per ADR-0114a.
# ---------------------------------------------------------------------------
def test_axis_lane_report_replay_byte_equal():
r1 = json.dumps(axis_runner.build_report(), indent=2, sort_keys=True)
r2 = json.dumps(axis_runner.build_report(), indent=2, sort_keys=True)
assert r1 == r2, "axis lane must be deterministic"
def test_committed_report_matches_fresh_run():
"""The committed ``report.json`` must equal a fresh run — keeps the
artifact diff-able as load-bearing evidence per ADR-0131.G."""
fresh = json.dumps(axis_runner.build_report(), indent=2, sort_keys=True) + "\n"
committed = (_AXIS_DIR / "report.json").read_text(encoding="utf-8")
assert fresh == committed, (
"committed report.json is stale; run "
"`python3 -m evals.math_capability_axes.G3_numerics.v1.runner` to refresh"
)
# ---------------------------------------------------------------------------
# GSM8K-probe safety rail — must still hold (the non-negotiable gate).
# ---------------------------------------------------------------------------
def test_gsm8k_probe_safety_rail_unchanged():
"""ADR-0131.G's safety rail: ``admitted_wrong == 0`` on the GSM8K
coverage probe is the load-bearing invariant every G.<n> iteration
must preserve. G.3 widens the candidate-graph parser; this asserts
that the probe (which runs through the legacy parser path) still
refuses cleanly without confabulating.
"""
from evals.gsm8k_math.train_sample.v1.run_coverage_probe import build_report
probe_report = build_report()
m = probe_report["metrics"]
assert m["admitted_wrong"] == 0, (
"GSM8K probe safety rail breached — admitted_wrong > 0"
)
assert m["safety_rail_intact"] is True