diff --git a/docs/decisions/ADR-0131.G.3-numerics.md b/docs/decisions/ADR-0131.G.3-numerics.md new file mode 100644 index 00000000..6e95e684 --- /dev/null +++ b/docs/decisions/ADR-0131.G.3-numerics.md @@ -0,0 +1,226 @@ +# ADR-0131.G.3 — Numeric Literals (money + hyphenated cardinals) + +**Status:** Proposed +**Date:** 2026-05-23 +**Author:** CORE main agent (Opus 4.7) +**Depends on:** ADR-0131.G (capability-axis iteration discipline), +ADR-0126 (candidate-graph parser), ADR-0127 (units pack), +ADR-0128 (numerics pack) +**Parent:** ADR-0131 (composite math-expert promotion gate) +**Foundation for:** ADR-0131.G.3.1 (fractions + multi-currency follow-up) + +--- + +## Context + +ADR-0131.G pinned the GSM8K coverage probe as a diff-able admission +number with the iteration discipline that *each subsequent +ADR-0131.G. extends a single capability axis with its own +curated coverage cases, independent of GSM8K*. G.3 is the first +literal-recognition axis: extending the candidate-graph parser's +`` slot to recognize money literals and hyphenated multi-word +cardinals by consuming the already-ratified `en_units_v1` +(ADR-0127) and `en_numerics_v1` (ADR-0128) packs. + +The baseline `train_sample_coverage_report.json` (#181) shows the +target shapes appear in real refusals: `Tina makes $18.00 an hour`, +`Aaron and his brother Carson each saved up $40`, hyphenated forms +like `10 one-hour videos`. Most refusals fail on *structural* +clause shape first (verb, multi-clause), so admission lift on the +probe requires sibling iterations (G.1 verbs, G.4 multi-clause) to +land alongside. G.3 alone is **necessary but not sufficient** for +probe lift; the load-bearing measurement for this iteration is the +**axis lane**. + +## Decision + +### Scope (v1 — closed set) + +1. **Money symbol literal** — `$N` and `$N.NN` (1–2 decimal places). + `$N.NNN` (3+ decimals) refused. +2. **Money word form** — ` dollars` and ` cents`. Recognized + via existing unit-slot path; normalized at candidate build. +3. **Hyphenated multi-word cardinal** — `twenty-five`, `ninety-nine`, + etc. Resolved via + `language_packs.numerics_loader.parse_compound_cardinal`. + +### Canonical money normalization (load-bearing) + +All money values normalize to **integer cents, unit `'cents'`**. +This is pinned by `en_units_v1`: + +```jsonl +{"surface": "money", "canonical_unit": "cent", + "is_canonical_for_dimension": true, "dimension": "money"} +``` + +`cent` is the canonical surface; `lookup_unit("cent").plural` is +`"cents"`. The parser canonicalizes both money symbols and the +word forms `N dollars` / `N cents` to plural `cents` so the +question-side canonicalization (`How many cents does X have?` → +`Unknown.unit='cents'`) matches by exact equality. + +`dollar` deliberately does **not** become the surface unit, even +though it's more user-natural. Diverging from a ratified pack's +`canonical_unit` because the alternative sounds nicer is the small +precedent that erodes the "consult the pack" architecture. The +realizer/formatter is the right layer for user-facing display of +"$40.00" — that's a separate concern. + +### Refusal probes (closed set) + +- `$N.NNN+` (3+ decimal places) — out-of-scope precision. +- `N/0` — division by zero. +- Unrecognized hyphenated compositions (`five-and-a-half`, + `gobbledy-gook`) — not resolvable via + `parse_compound_cardinal`. +- Percent literals (`50%`) — out-of-scope. + +### Out-of-scope, explicitly deferred to G.3.1+ + +- **Fractions in initial-possession / operations** (`N/M`). The + `_resolve_value` resolver supports `N/M` token-level (returns + `Fraction(N,M)` → float), but no axis cases exercise it + end-to-end because the initial-possession regex's "of " + substance-qualifier handling needs widening to admit + `"Bob has 3/4 of a cup."` cleanly. Token-level recognition is + ratified now; pipeline-level use is G.3.1. +- **Multi-currency** (`¢ € £ ¥ ₱`). US-only in v1. +- **Word-number compositions with adjective insertion** (`five + full boxes` per ADR-0127 substance-qualifier precedent). +- **Multi-token space-separated cardinals** (`one hundred`, `two + thousand`). `parse_compound_cardinal` supports these; the parser + doesn't yet match them in the value slot because they'd span + the unit slot boundary. Defer. +- **Scientific notation, locale separators, percentages.** + +## What changed in code + +### `generate/math_candidate_parser.py` + +- New constants for the three widened value shapes (`_MONEY_SYMBOL`, + `_SLASH_FRACTION`, `_HYPHENATED_CARDINAL`); `_VALUE` widened to + include them in alternation. +- `_resolve_value(token)` refactored to return `_ResolvedValue | + None` (was `int`). Returns `unit_override='cents'` for money + symbols; returns `None` (refusal) for out-of-scope shapes. +- `_INITIAL_HAS_RE` unit slot made optional (money symbols carry + their unit implicitly); trailing `(?:in|of|for|with)` preposition + phrases discardable. +- `extract_initial_candidates` + `_build_op_candidate` updated to + honor `unit_override`, apply dollar/dollars → cents normalization + via `_money_unit_normalization`, and emit no candidate when neither + the resolver nor the regex provides a unit. + +### `generate/math_roundtrip.py` + +- New `_unit_grounds(unit_token, source_span, haystack_tokens)` + helper: word-token containment plus money-aware grounding (when + unit is `cent`/`cents`, accept `$` in source or `dollar`/`dollars` + in tokens). +- `_value_grounds` widened to handle money symbols (digit-parts of + `$N.NN` must each be in source tokens), slash fractions + (numerator + denominator both ground), hyphenated cardinals + (every component grounds OR the compound's digit form grounds). +- `roundtrip_admissible` unit check upgraded from `_token_in` to + `_unit_grounds`. + +### `generate/math_candidate_graph.py` + +- `_initial_admissible` and `_question_admissible` unit checks + upgraded to `_unit_grounds`. + +### New files + +- `evals/math_capability_axes/G3_numerics/v1/cases.jsonl` — 26 + curated cases (5 per positive class + 6 refusal probes). +- `evals/math_capability_axes/G3_numerics/v1/runner.py` — pure + adapter over `evals.gsm8k_math.runner._score_one_candidate_graph`; + byte-equal `report.json` across runs. +- `tests/test_adr_0131_G3_numerics.py` — 27 tests. + +## Evidence + +- **Axis lane** (`python3 -m + evals.math_capability_axes.G3_numerics.v1.runner`): + - 20/20 positive cases solved correct + - 6/6 refusal probes refused with typed reason + - `solved_wrong == 0` (safety rail intact) + - `correct_rate_on_positive_cases == 1.0` + - report.json byte-equal across two runs +- **Test suite**: 27/27 pass in 0.19s. +- **Regression**: 420 existing candidate-parser + math-parser + + pack tests pass with the widening. +- **GSM8K probe** (`evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`): + unchanged at 0/50 admission (probe uses the legacy parser path, + not the candidate-graph pipeline this iteration extends). + `admitted_wrong == 0` (safety rail) preserved. + +## Honest scope-limit disclosure + +ADR-0131.G's discipline says each iteration's GSM8K-probe gate is +"`admission_rate` strictly increases OR a refused-reason family is +deliberately reduced." Neither holds for G.3 in isolation, because: + +1. The `run_coverage_probe.py` probe goes through + `evals.gsm8k_math.runner._score_one` → `parse_problem` (the + legacy first-match-wins parser, ADR pre-0126), not the + candidate-graph pipeline G.3 extends. +2. Even if the probe consulted the candidate graph, most + money-bearing GSM8K cases (`Tina makes $18.00 an hour`) fail + first on the *verb* (`makes`, rate-introducing) or *multi-clause* + shape (`Aaron and his brother Carson each saved up $40`); the + money literal is a *downstream* refusal cause. + +G.3's load-bearing gate is therefore the **axis lane** (full +end-to-end correctness on 20 curated cases that *do* exercise the +new capability through the candidate-graph pipeline). Probe lift +will accumulate as sibling axes (G.1 verb classes, G.4 multi- +clause) land alongside. + +**Reserved follow-up**: a small probe-infrastructure ADR should +switch `run_coverage_probe.py` to call +`_score_one_candidate_graph` instead of `_score_one`, so future +G. iterations show probe lift. That's a one-line change but +needs its own scoped PR (it shifts the probe's measurement +substrate; the resulting admission_rate delta should be +explicitly attributed, not buried inside a capability-axis PR). + +## Composition with other in-flight work + +- **L9 / ADR-0131.G.1 (verb classes)**: lands the verbs that unlock + `Tina makes $18.00 an hour` shape on the probe; G.3 already + handles the `$18.00` literal so admission flips when G.1 lands. +- **L10 / ADR-0131.G.2 (comparatives)**: independent; no + literal-class interaction. +- **L12 / ADR-0131.G.4 (multi-clause)**: unlocks `Aaron and his + brother Carson each saved up $40` shape; G.3 already handles + the `$40` literal. +- **ADR-0131.G.3.1 (deferred follow-up)**: fractions end-to-end, + multi-currency (`¢ € £ ¥ ₱`), space-separated multi-word + cardinals (`one hundred`), word-number-adjective compositions + (`five full boxes`). + +## CLAUDE.md PR-checklist answers + +- **Capability/performance/security added or protected:** Adds + pack-consuming literal recognition for money symbols, money + word forms, and hyphenated multi-word cardinals at the + candidate-graph parser layer. +- **Invariant proving the field remains valid:** `solved_wrong == + 0` on the axis lane (27 tests); `admitted_wrong == 0` on the + GSM8K probe (preserved). +- **CLI/eval lane proving correctness:** `python3 -m + evals.math_capability_axes.G3_numerics.v1.runner` and `pytest + tests/test_adr_0131_G3_numerics.py`. +- **Avoided hidden normalization / stochastic fallback / + approximate recall / unreviewed mutation:** Yes. All lookups + are deterministic, pack-driven (`en_units_v1`, + `en_numerics_v1`). Money normalization is a single fixed rule + (cents). Refusal-first preserved at every layer. +- **Trust boundary:** Inputs are user-controlled text → parser + regex; widened regex stays bounded (closed alternation set, + no backreferences, no catastrophic-backtracking-risk + patterns). Pack loader paths consult + `language_packs/data//` only; no dynamic imports, no + filesystem traversal beyond pack root. diff --git a/evals/math_capability_axes/G3_numerics/v1/cases.jsonl b/evals/math_capability_axes/G3_numerics/v1/cases.jsonl new file mode 100644 index 00000000..21b60c9d --- /dev/null +++ b/evals/math_capability_axes/G3_numerics/v1/cases.jsonl @@ -0,0 +1,26 @@ +{"case_id":"g3-money-sym-int-01","class":"money_symbol_integer","problem":"Bob has $40. How many cents does Bob have?","expected_answer":4000,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-int-02","class":"money_symbol_integer","problem":"Sarah has $25 in her wallet. How many cents does Sarah have?","expected_answer":2500,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-int-03","class":"money_symbol_integer","problem":"Tina has $100. Tina spends $40. How many cents does Tina have?","expected_answer":6000,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-int-04","class":"money_symbol_integer","problem":"Sam has $50. Sam earns $20. How many cents does Sam have?","expected_answer":7000,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-int-05","class":"money_symbol_integer","problem":"Pat has $60. Pat gives $25 to Jordan. How many cents does Pat have?","expected_answer":3500,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-dec-01","class":"money_symbol_decimal","problem":"Tina has $18.00 in savings. How many cents does Tina have?","expected_answer":1800,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-dec-02","class":"money_symbol_decimal","problem":"Bob has $2.50 in change. How many cents does Bob have?","expected_answer":250,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-dec-03","class":"money_symbol_decimal","problem":"Maya has $7.5 in her piggy bank. How many cents does Maya have?","expected_answer":750,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-dec-04","class":"money_symbol_decimal","problem":"Liam has $12.25. Liam spends $5.50. How many cents does Liam have?","expected_answer":675,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-sym-dec-05","class":"money_symbol_decimal","problem":"Noah has $9.99 in his pocket. How many cents does Noah have?","expected_answer":999,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-word-01","class":"money_word","problem":"Bob has 40 dollars. How many cents does Bob have?","expected_answer":4000,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-word-02","class":"money_word","problem":"Anna has 75 dollars. Anna spends 30 dollars. How many cents does Anna have?","expected_answer":4500,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-word-03","class":"money_word","problem":"Charlie has 250 cents. How many cents does Charlie have?","expected_answer":250,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-word-04","class":"money_word","problem":"Diana has 100 dollars. Diana earns 50 dollars. How many cents does Diana have?","expected_answer":15000,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-money-word-05","class":"money_word","problem":"Ethan has 20 dollars. Ethan gives 5 dollars to Felix. How many cents does Ethan have?","expected_answer":1500,"expected_unit":"cents","expected_outcome":"solved_correct"} +{"case_id":"g3-hyphen-card-01","class":"hyphenated_cardinal","problem":"Bob has twenty-five apples. How many apples does Bob have?","expected_answer":25,"expected_unit":"apples","expected_outcome":"solved_correct"} +{"case_id":"g3-hyphen-card-02","class":"hyphenated_cardinal","problem":"The store has fifty-five toys. How many toys does the store have?","expected_answer":55,"expected_unit":"toys","expected_outcome":"solved_correct"} +{"case_id":"g3-hyphen-card-03","class":"hyphenated_cardinal","problem":"Jane has ninety-nine pencils. Jane gives twelve pencils to Tom. How many pencils does Jane have?","expected_answer":87,"expected_unit":"pencils","expected_outcome":"solved_correct"} +{"case_id":"g3-hyphen-card-04","class":"hyphenated_cardinal","problem":"Bob has thirty-three apples. Bob buys seven apples. How many apples does Bob have?","expected_answer":40,"expected_unit":"apples","expected_outcome":"solved_correct"} +{"case_id":"g3-hyphen-card-05","class":"hyphenated_cardinal","problem":"Liam has twenty-one books. Liam loses three books. How many books does Liam have?","expected_answer":18,"expected_unit":"books","expected_outcome":"solved_correct"} +{"case_id":"g3-refuse-money-prec-01","class":"refuse_money_precision","problem":"Bob has $40.000 in savings. How many cents does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g3-refuse-money-prec-02","class":"refuse_money_precision","problem":"Sarah has $1.2345 in change. How many cents does Sarah have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g3-refuse-div-zero-01","class":"refuse_division_by_zero","problem":"Bob has 5/0 apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g3-refuse-hyphen-01","class":"refuse_unknown_compound","problem":"Bob has five-and-a-half apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g3-refuse-hyphen-02","class":"refuse_unknown_compound","problem":"Bob has gobbledy-gook apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} +{"case_id":"g3-refuse-percent-01","class":"refuse_percentage","problem":"Bob has 50% apples. How many apples does Bob have?","expected_answer":0,"expected_unit":"","expected_outcome":"refused"} diff --git a/evals/math_capability_axes/G3_numerics/v1/report.json b/evals/math_capability_axes/G3_numerics/v1/report.json new file mode 100644 index 00000000..15add95e --- /dev/null +++ b/evals/math_capability_axes/G3_numerics/v1/report.json @@ -0,0 +1,343 @@ +{ + "adr": "0131.G.3", + "axis": "numeric_literals", + "cases_path": "evals/math_capability_axes/G3_numerics/v1/cases.jsonl", + "class_counts": { + "hyphenated_cardinal": 5, + "money_symbol_decimal": 5, + "money_symbol_integer": 5, + "money_word": 5, + "refuse_division_by_zero": 1, + "refuse_money_precision": 2, + "refuse_percentage": 1, + "refuse_unknown_compound": 2 + }, + "metrics": { + "cases_total": 26, + "correct_rate_on_positive_cases": 1.0, + "overall_pass": true, + "refused_as_expected": 6, + "solved_correct": 20, + "solved_wrong": 0, + "wrong_count_is_zero": true + }, + "per_case": [ + { + "actual_answer": 4000.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-int-01", + "class": "money_symbol_integer", + "expected_answer": 4000, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "742b18b40f28213ce2361e208b8deb01cedf2ab16fe40b001e570070b5c48fdd", + "verdict": "solved_correct" + }, + { + "actual_answer": 2500.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-int-02", + "class": "money_symbol_integer", + "expected_answer": 2500, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "1e2ba5472e84656edabc744bfd036b001f342ac6b76760e411d5b51cf524753b", + "verdict": "solved_correct" + }, + { + "actual_answer": 6000.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-int-03", + "class": "money_symbol_integer", + "expected_answer": 6000, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "0db7b91caeb1721c99982938e5bafd45df5d3ba849982494e82984dbefaf8c76", + "verdict": "solved_correct" + }, + { + "actual_answer": 7000.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-int-04", + "class": "money_symbol_integer", + "expected_answer": 7000, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "9efdb9599d909c9501fae743b6b07e496febc474991f27090fc02b2c0ce24aef", + "verdict": "solved_correct" + }, + { + "actual_answer": 3500.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-int-05", + "class": "money_symbol_integer", + "expected_answer": 3500, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "0113e1b71499c4ad66008115e9b1e611790c432eb8c430f42ad64312a4b01484", + "verdict": "solved_correct" + }, + { + "actual_answer": 1800.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-dec-01", + "class": "money_symbol_decimal", + "expected_answer": 1800, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "cbe7694b94ec417069feb636114da8b3dfbbd6df5ba4226c0a17cf896a564a96", + "verdict": "solved_correct" + }, + { + "actual_answer": 250.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-dec-02", + "class": "money_symbol_decimal", + "expected_answer": 250, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "2a2b8050917b97d120fd90cd5a5246b6626a4e0a5be48b42a87aab39fba2112d", + "verdict": "solved_correct" + }, + { + "actual_answer": 750.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-dec-03", + "class": "money_symbol_decimal", + "expected_answer": 750, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "32543d70d2caf8befb616082361d7c59107e1f9b8785180da6f44bd4bf22ce72", + "verdict": "solved_correct" + }, + { + "actual_answer": 675.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-dec-04", + "class": "money_symbol_decimal", + "expected_answer": 675, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "dd6c3b8cc0c83a0fe9d65f993759a5f14243aae9a6086e7ce4ca92c26897350b", + "verdict": "solved_correct" + }, + { + "actual_answer": 999.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-sym-dec-05", + "class": "money_symbol_decimal", + "expected_answer": 999, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "c4bda23a263700abd5e03c7d94d3393ae9a38a5587afcb2c374688db5b1f56df", + "verdict": "solved_correct" + }, + { + "actual_answer": 4000.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-word-01", + "class": "money_word", + "expected_answer": 4000, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "742b18b40f28213ce2361e208b8deb01cedf2ab16fe40b001e570070b5c48fdd", + "verdict": "solved_correct" + }, + { + "actual_answer": 4500.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-word-02", + "class": "money_word", + "expected_answer": 4500, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "0e222771bf42bd808f4d9d5794ac044d515e94a3b47377bdee695b6f8173b93c", + "verdict": "solved_correct" + }, + { + "actual_answer": 250.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-word-03", + "class": "money_word", + "expected_answer": 250, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "a48d47cf0f4c41907df10058a246940352e5cae719c4a32778f80a6e88f46799", + "verdict": "solved_correct" + }, + { + "actual_answer": 15000.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-word-04", + "class": "money_word", + "expected_answer": 15000, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "855733e843f7caa9d924aed87841ea21a67e74325e9445aae3f2477de3805dc2", + "verdict": "solved_correct" + }, + { + "actual_answer": 1500.0, + "actual_outcome": "correct", + "actual_unit": "cents", + "case_id": "g3-money-word-05", + "class": "money_word", + "expected_answer": 1500, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "10ed9467df9c42de95ba8104b0f1ff72fc50579839a6d85ce95aed184add37c0", + "verdict": "solved_correct" + }, + { + "actual_answer": 25.0, + "actual_outcome": "correct", + "actual_unit": "apples", + "case_id": "g3-hyphen-card-01", + "class": "hyphenated_cardinal", + "expected_answer": 25, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "5d486cf0eadaab2f52039c28b239c78ab58eca80bd154ab912be4acd03f06182", + "verdict": "solved_correct" + }, + { + "actual_answer": 55.0, + "actual_outcome": "correct", + "actual_unit": "toys", + "case_id": "g3-hyphen-card-02", + "class": "hyphenated_cardinal", + "expected_answer": 55, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "a094f20be6f6664e499a6f6fa9e68dbef6a27c5fcdd29973053919dfa20a322a", + "verdict": "solved_correct" + }, + { + "actual_answer": 87.0, + "actual_outcome": "correct", + "actual_unit": "pencils", + "case_id": "g3-hyphen-card-03", + "class": "hyphenated_cardinal", + "expected_answer": 87, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "bc0696023e69f665e1551344319e3eb40e0b7e97108b4f896b12b2eb9e85d6e5", + "verdict": "solved_correct" + }, + { + "actual_answer": 40.0, + "actual_outcome": "correct", + "actual_unit": "apples", + "case_id": "g3-hyphen-card-04", + "class": "hyphenated_cardinal", + "expected_answer": 40, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "69a3edca720fa45b42ce4b6c0c068ef51d0655ef4cd7098012898260dac1c4da", + "verdict": "solved_correct" + }, + { + "actual_answer": 18.0, + "actual_outcome": "correct", + "actual_unit": "books", + "case_id": "g3-hyphen-card-05", + "class": "hyphenated_cardinal", + "expected_answer": 18, + "expected_outcome": "solved_correct", + "reason": "", + "trace_hash": "fa82ed15b02dcbf1931623ab8a9cd1fcdeb4ae5e2ae15a0e50dcc6078fde5a2f", + "verdict": "solved_correct" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g3-refuse-money-prec-01", + "class": "refuse_money_precision", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has $40.000 in savings.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g3-refuse-money-prec-02", + "class": "refuse_money_precision", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Sarah has $1.2345 in change.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g3-refuse-div-zero-01", + "class": "refuse_division_by_zero", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has 5/0 apples.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g3-refuse-hyphen-01", + "class": "refuse_unknown_compound", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has five-and-a-half apples.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g3-refuse-hyphen-02", + "class": "refuse_unknown_compound", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has gobbledy-gook apples.'", + "trace_hash": null, + "verdict": "refused" + }, + { + "actual_answer": null, + "actual_outcome": "refused", + "actual_unit": null, + "case_id": "g3-refuse-percent-01", + "class": "refuse_percentage", + "expected_answer": 0, + "expected_outcome": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Bob has 50% apples.'", + "trace_hash": null, + "verdict": "refused" + } + ], + "schema_version": 1, + "verdict_counts": { + "refused": 6, + "solved_correct": 20 + } +} diff --git a/evals/math_capability_axes/G3_numerics/v1/runner.py b/evals/math_capability_axes/G3_numerics/v1/runner.py new file mode 100644 index 00000000..e12c6b82 --- /dev/null +++ b/evals/math_capability_axes/G3_numerics/v1/runner.py @@ -0,0 +1,164 @@ +"""ADR-0131.G.3 — Numeric-literals capability-axis runner. + +First sibling under ``evals/math_capability_axes/`` — the iteration +pattern from ADR-0131.G ("each ADR-0131.G. extends a single +capability axis with its own curated coverage cases, independent of +GSM8K"). G.3's axis is numeric-literal recognition: money symbols +(``$N`` / ``$N.NN``), the word forms ``N dollars`` / ``N cents``, and +hyphenated multi-word cardinals (``twenty-five``). + +The runner wraps :func:`evals.gsm8k_math.runner._score_one_candidate_graph` +(the candidate-graph pipeline ADR-0126 introduced) so that any future +G. axis extending the same parser layer shows up on the same lane +without parallel infrastructure. + +Outcome classification mirrors the GSM8K runner: + + | Pipeline result | Outcome | + |------------------------|-----------| + | parser+solver+verifier OK and answer/unit match | ``solved_correct`` | + | parser+solver OK but verifier fails or answer mismatches | ``solved_wrong`` (gate: must be 0) | + | parser/solver refuses with typed reason | ``refused`` | + +Cases ship with an ``expected_outcome`` so the runner can score +positive-coverage cases (``solved_correct``) AND adversarial-refusal +probes (``refused``) on the same axis. ``wrong == 0`` is preserved as +the load-bearing invariant per ADR-0114a Obligation #4. + +The runner is pure / deterministic: same case set → byte-equal +``report.json`` across runs. +""" + +from __future__ import annotations + +import json +from collections import Counter +from pathlib import Path +from typing import Any + +from evals.gsm8k_math.runner import _score_one_candidate_graph + +_HERE = Path(__file__).resolve().parent +_CASES_PATH = _HERE / "cases.jsonl" +_REPORT_PATH = _HERE / "report.json" + + +def _load_cases() -> list[dict[str, Any]]: + out: list[dict[str, Any]] = [] + for line in _CASES_PATH.read_text(encoding="utf-8").splitlines(): + if line.strip(): + out.append(json.loads(line)) + return out + + +def _adapt_case(raw: dict[str, Any]) -> dict[str, Any]: + """Translate axis-case shape to the candidate-graph runner's expected + shape. Refusal cases pass ``expected_unit=""`` so the unit-mismatch + branch doesn't fire on cases that never reach the answer comparison. + """ + return { + "id": raw["case_id"], + "problem": raw["problem"], + "expected_answer": float(raw["expected_answer"]), + "expected_unit": raw.get("expected_unit", ""), + } + + +def _classify(actual_outcome: str, expected_outcome: str) -> str: + """Map the candidate-graph runner's outcome + the case's expected + outcome into a unified axis-lane verdict. + + - ``solved_correct``: pipeline returned ``correct`` AND expected was + ``solved_correct``. + - ``refused``: pipeline returned ``refused`` AND expected was + ``refused``. + - ``solved_wrong``: any disagreement — either ``correct`` for a + ``refused`` case, ``wrong`` ever, or ``refused`` for a + ``solved_correct`` case. All map to ``solved_wrong``, which the + lane gate requires to be zero. + """ + if expected_outcome == "solved_correct" and actual_outcome == "correct": + return "solved_correct" + if expected_outcome == "refused" and actual_outcome == "refused": + return "refused" + return "solved_wrong" + + +def build_report() -> dict[str, Any]: + raw_cases = _load_cases() + case_results: list[dict[str, Any]] = [] + class_counts: Counter[str] = Counter() + verdict_counts: Counter[str] = Counter() + + for raw in raw_cases: + cls = raw["class"] + expected = raw["expected_outcome"] + class_counts[cls] += 1 + outcome = _score_one_candidate_graph(_adapt_case(raw)) + verdict = _classify(outcome.outcome, expected) + verdict_counts[verdict] += 1 + case_results.append({ + "case_id": raw["case_id"], + "class": cls, + "expected_outcome": expected, + "actual_outcome": outcome.outcome, + "verdict": verdict, + "expected_answer": raw["expected_answer"], + "actual_answer": outcome.actual_answer, + "actual_unit": outcome.actual_unit, + "reason": outcome.reason, + "trace_hash": outcome.trace_hash, + }) + + total = len(raw_cases) + correct = verdict_counts.get("solved_correct", 0) + wrong = verdict_counts.get("solved_wrong", 0) + refused_expected = verdict_counts.get("refused", 0) + positive_count = sum(1 for r in raw_cases if r["expected_outcome"] == "solved_correct") + correct_rate_on_positive = ( + correct / positive_count if positive_count else 0.0 + ) + + return { + "schema_version": 1, + "adr": "0131.G.3", + "axis": "numeric_literals", + "cases_path": "evals/math_capability_axes/G3_numerics/v1/cases.jsonl", + "metrics": { + "cases_total": total, + "solved_correct": correct, + "solved_wrong": wrong, + "refused_as_expected": refused_expected, + "wrong_count_is_zero": wrong == 0, + "correct_rate_on_positive_cases": correct_rate_on_positive, + "overall_pass": wrong == 0 and (correct + refused_expected == total), + }, + "class_counts": dict(sorted(class_counts.items())), + "verdict_counts": dict(sorted(verdict_counts.items())), + "per_case": case_results, + } + + +def write_report(report: dict[str, Any]) -> None: + _REPORT_PATH.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def main() -> int: + report = build_report() + write_report(report) + m = report["metrics"] + print(f"axis: {report['axis']}") + print(f"cases_total: {m['cases_total']}") + print(f"solved_correct: {m['solved_correct']}") + print(f"solved_wrong: {m['solved_wrong']} (gate: must be 0)") + print(f"refused_as_expected: {m['refused_as_expected']}") + print(f"correct_rate_on_positive_cases: {m['correct_rate_on_positive_cases']:.1%}") + print(f"overall_pass: {m['overall_pass']}") + return 0 if m["overall_pass"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index 9c739162..703ca0c2 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -154,13 +154,13 @@ def _initial_admissible(ic: CandidateInitial) -> bool: Same shape as roundtrip_admissible but for the initial-possession slot set (entity, anchor, value, unit).""" - from generate.math_roundtrip import _tokens, _value_grounds, _token_in + from generate.math_roundtrip import _tokens, _value_grounds, _token_in, _unit_grounds haystack = _tokens(ic.source_span) if not _token_in(ic.matched_anchor, haystack): return False if not _value_grounds(ic.matched_value_token, haystack): return False - if not _token_in(ic.matched_unit_token, haystack): + if not _unit_grounds(ic.matched_unit_token, ic.source_span, haystack): return False # Entity token: for multi-word entities ("the boys"), all words # must ground. Split + check each. @@ -172,9 +172,9 @@ def _initial_admissible(ic: CandidateInitial) -> bool: def _question_admissible(qc: CandidateUnknown) -> bool: """Light structural ground-check for question candidates.""" - from generate.math_roundtrip import _tokens, _token_in + from generate.math_roundtrip import _tokens, _token_in, _unit_grounds haystack = _tokens(qc.source_span) - if not _token_in(qc.matched_unit_token, haystack): + if not _unit_grounds(qc.matched_unit_token, qc.source_span, haystack): return False if qc.matched_entity_token is not None: for tok in qc.matched_entity_token.split(): diff --git a/generate/math_candidate_parser.py b/generate/math_candidate_parser.py index 342562e3..0973852b 100644 --- a/generate/math_candidate_parser.py +++ b/generate/math_candidate_parser.py @@ -98,14 +98,28 @@ class CandidateInitial: # math_parser._INITIAL_HAS_RE's ADR-0123a entity slot. _ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)" -# Numeric value: digit run OR word-form integer (one..twelve initially; -# WORD_NUMBERS table is wider but we cap the regex at the common range -# for syntactic parsing and let the filter handle ground-truth value -# equivalence). +# Numeric value alternation. Listed longest-form-first so the regex +# engine doesn't truncate on a shorter prefix: +# - Money symbol literal: ``$N`` or ``$N.NN`` (1-2 decimal places). +# ADR-0131.G.3. ``$N.NNN`` (3+ decimals) deliberately not matched +# — refused as out-of-scope so wrong == 0 is preserved. +# - Slash fraction literal: ``N/M``. Denominator-zero refused at +# resolve time, not regex. +# - Hyphenated multi-word cardinal: ``twenty-five``, ``ninety-nine``. +# Resolved via :func:`language_packs.numerics_loader.parse_compound_cardinal`. +# - Digit run. +# - Single-word cardinal (legacy ``WORD_NUMBERS`` set). +_MONEY_SYMBOL: Final[str] = r"\$\d+(?:\.\d{1,2})?" +_SLASH_FRACTION: Final[str] = r"\d+/\d+" +_HYPHENATED_CARDINAL: Final[str] = r"[A-Za-z]+-[A-Za-z]+" _WORD_NUM_OPTIONS: Final[str] = "|".join( re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True) ) -_VALUE: Final[str] = rf"(?:\d+|{_WORD_NUM_OPTIONS})" +_VALUE: Final[str] = ( + rf"(?:{_MONEY_SYMBOL}|{_SLASH_FRACTION}|" + rf"{_HYPHENATED_CARDINAL}|" + rf"\d+|{_WORD_NUM_OPTIONS})" +) # Verb alternation built from the permissive registry. Pre-compute one # pattern per kind so we can attribute matched verbs to candidates. @@ -127,11 +141,17 @@ _TRANSFER_VERBS_PATTERN: Final[str] = _verbs_pattern(TRANSFER_VERBS) _INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile( rf"^(?P{_ENTITY})\s+" rf"(?Phas|have)\s+" - rf"(?P{_VALUE})\s+" - r"(?P\w+)" + rf"(?P{_VALUE})" + # ADR-0131.G.3: unit slot is optional. Money-symbol value literals + # (``$40``) carry their unit implicitly (``cent``); a missing unit + # slot is admissible IFF the value resolves with a unit override. + # Non-money values without a unit slot are refused at resolve time. + r"(?:\s+(?P\w+))?" # ADR-0127 substance qualifier: "Sam has 5 feet of rope" — the # 'of ' tail is grammatically real but arithmetically inert. - r"(?:\s+of\s+.+)?" + # ADR-0131.G.3: 'in ' is also discardable + # ("Bob has $40 in savings"; "Bob has $40 in his wallet"). + r"(?:\s+(?:of|in|for|with)\s+.+)?" r"\s*\.?$" ) @@ -162,10 +182,79 @@ def _normalize_entity(raw: str) -> str: return e -def _resolve_value(value_token: str) -> int: - if value_token.isdigit(): - return int(value_token) - return WORD_NUMBERS[value_token.lower()] +@dataclass(frozen=True, slots=True) +class _ResolvedValue: + """Resolved value-slot reading. + + ADR-0131.G.3 widens the value slot beyond integer + single-word + cardinal to include money literals (``$N`` / ``$N.NN``), slash + fractions (``N/M``), and hyphenated multi-word cardinals + (``twenty-five``). Money literals carry an implicit canonical unit + (``cent``); when set, ``unit_override`` replaces the unit slot the + regex captured (or fills it when the unit slot is absent). + """ + + value: int | float + unit_override: str | None + + +# Money: canonical normalization to integer cents (en_units_v1 +# ``canonical_unit`` for the ``money`` dimension is ``cent``). +_MONEY_UNIT: Final[str] = "cents" + + +def _resolve_value(value_token: str) -> _ResolvedValue | None: + """Resolve a value-slot token into a numeric value + optional unit + override. Returns ``None`` on refusal (indefinite quantifier, + division-by-zero in slash fraction, unrecognized hyphenated form, + unparseable money). + + Refusal at this layer is first-class: a ``None`` upstream means the + candidate is not emitted, which preserves ``wrong == 0`` per + ADR-0114a Obligation #4. + """ + if not value_token: + return None + t = value_token.strip() + # Money symbol literal: $N or $N.NN. + if t.startswith("$"): + body = t[1:] + if re.fullmatch(r"\d+", body): + return _ResolvedValue(int(body) * 100, _MONEY_UNIT) + if re.fullmatch(r"\d+\.\d{1,2}", body): + # round() avoids float drift: $2.50 → 250, not 249 or 251. + return _ResolvedValue(int(round(float(body) * 100)), _MONEY_UNIT) + return None # $N.NNN (3+ decimals) refused — out-of-scope. + # Slash fraction literal: N/M with M > 0. + if "/" in t: + m = re.fullmatch(r"(\d+)/(\d+)", t) + if m is None: + return None + num, den = int(m.group(1)), int(m.group(2)) + if den == 0: + return None # division-by-zero refused. + if num % den == 0: + return _ResolvedValue(num // den, None) + return _ResolvedValue(num / den, None) + # Digit run. + if t.isdigit(): + return _ResolvedValue(int(t), None) + # Indefinite quantifier (ADR-0128.4) — refuse, never guess. + if _is_indefinite_quantifier(t): + return None + # Hyphenated multi-word cardinal: twenty-five, ninety-nine, etc. + if "-" in t: + from language_packs.numerics_loader import parse_compound_cardinal + + parsed = parse_compound_cardinal(t) + if parsed is None: + return None # Unrecognized hyphenated form refused. + return _ResolvedValue(parsed, None) + # Single-word cardinal (legacy WORD_NUMBERS table). + lower = t.lower() + if lower in WORD_NUMBERS: + return _ResolvedValue(WORD_NUMBERS[lower], None) + return None def _is_indefinite_quantifier(token: str) -> bool: @@ -186,6 +275,25 @@ def _is_indefinite_quantifier(token: str) -> bool: return False +def _money_unit_normalization( + value: int | float, unit: str | None +) -> tuple[int | float, str | None]: + """ADR-0131.G.3 — normalize ``dollar``/``dollars`` surface unit to the + canonical money unit (``cent``). + + ``en_units_v1`` pins ``cent`` as ``canonical_unit`` for the ``money`` + dimension; ``dollar`` is convenience surface. A ``dollar`` value is + 100 ``cent``. Done at the candidate-build site so every money-bearing + path normalizes uniformly (Quantity equality is exact — mixing + ``cent`` and ``dollar`` units would silently break arithmetic). + """ + if unit is None: + return value, unit + if unit.lower() in ("dollar", "dollars"): + return value * 100, _MONEY_UNIT + return value, unit + + def extract_initial_candidates(sentence: str) -> list[CandidateInitial]: """Return all admissible initial-possession candidates for ``sentence``. @@ -193,9 +301,14 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]: 1. " has [of ]" — canonical. 2. "There are [in ]" — implicit-subject shape. - ADR-0128.4: if the value slot resolves to an indefinite quantifier - (`some kids`, `many things`), no candidate is emitted (refusal - preserves wrong == 0). + Value-slot widenings (ADR-0131.G.3) apply to both shapes via + :func:`_resolve_value`: money literals (``$N`` / ``$N.NN``), slash + fractions (``N/M``), hyphenated multi-word cardinals (``twenty-five``). + + Refusal-first: indefinite quantifiers, division-by-zero fractions, + unrecognized compound forms, and money literals with >2 decimals + all return ``None`` from :func:`_resolve_value` and emit no + candidate (preserves ``wrong == 0`` per ADR-0114a Obligation #4). """ s = sentence.strip().rstrip(".") out: list[CandidateInitial] = [] @@ -203,32 +316,53 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]: m = _INITIAL_HAS_RE.match(s) if m is not None: value_raw = m.group("value") - if not _is_indefinite_quantifier(value_raw): + rv = _resolve_value(value_raw) + if rv is not None: entity = _normalize_entity(m.group("entity")) - value = _resolve_value(value_raw) - unit_raw = m.group("unit") - unit = _canonicalize_unit(unit_raw) - out.append( - CandidateInitial( - initial=InitialPossession( - entity=entity, - quantity=Quantity(value=value, unit=unit), - ), - source_span=sentence, - matched_anchor=m.group("anchor"), - matched_value_token=value_raw, - matched_unit_token=unit_raw, - matched_entity_token=m.group("entity"), + unit_raw = m.group("unit") # may be None when value is money symbol + # Unit precedence: explicit override from value (money symbol) + # wins over the regex's unit slot. The unit slot is required + # for non-money values; if both are absent the candidate + # cannot be constructed. + resolved_unit: str | None + if rv.unit_override is not None: + resolved_unit = rv.unit_override + elif unit_raw is not None: + resolved_unit = _canonicalize_unit(unit_raw) + else: + resolved_unit = None + if resolved_unit is not None: + value, final_unit = _money_unit_normalization(rv.value, resolved_unit) + assert final_unit is not None + out.append( + CandidateInitial( + initial=InitialPossession( + entity=entity, + quantity=Quantity(value=value, unit=final_unit), + ), + source_span=sentence, + matched_anchor=m.group("anchor"), + matched_value_token=value_raw, + matched_unit_token=unit_raw if unit_raw is not None else final_unit, + matched_entity_token=m.group("entity"), + ) ) - ) m2 = _INITIAL_THERE_ARE_RE.match(s) if m2 is not None: value_raw = m2.group("value") - if not _is_indefinite_quantifier(value_raw): + rv = _resolve_value(value_raw) + if rv is not None: unit_raw = m2.group("unit") - unit = _canonicalize_unit(unit_raw) - value = _resolve_value(value_raw) + assert unit_raw is not None # there-are regex always captures unit slot + if rv.unit_override is not None: + unit_str: str = rv.unit_override + else: + unit_str = _canonicalize_unit(unit_raw) + v_norm, u_norm = _money_unit_normalization(rv.value, unit_str) + assert u_norm is not None + value: int | float = v_norm + unit: str = u_norm place = m2.group("place") # When a 'in ' phrase is present, treat the place as # the implicit entity. Otherwise use the unit's plural as @@ -342,15 +476,28 @@ def _build_op_candidate( m: re.Match[str], kind: str, source: str ) -> CandidateOperation | None: """Build a CandidateOperation from a regex match. Returns None if - the match lacks a required slot (e.g. unit token absent — P2 does - not emit unit-inherited candidates).""" - unit_raw = m.group("unit") - if unit_raw is None: + the value cannot be resolved or if no unit can be determined + (unit slot absent AND value carries no implicit unit override). + """ + value_raw = m.group("value") + rv = _resolve_value(value_raw) + if rv is None: return None - unit = _canonicalize_unit(unit_raw) + unit_raw = m.group("unit") + # ADR-0131.G.3: a money-symbol value carries its unit implicitly + # (override 'cent'); for plain-numeric values, the unit slot must + # be present. + if rv.unit_override is not None: + unit: str = rv.unit_override + elif unit_raw is not None: + unit = _canonicalize_unit(unit_raw) + else: + return None # P2 does not emit unit-inherited candidates. subject = _normalize_entity(m.group("subject")) verb = m.group("verb").lower() - value = _resolve_value(m.group("value")) + value, unit_normalized = _money_unit_normalization(rv.value, unit) + assert unit_normalized is not None + unit = unit_normalized target_raw = m.group("target") if "target" in m.groupdict() else None target = target_raw if target_raw is not None else None @@ -372,7 +519,7 @@ def _build_op_candidate( source_span=source, matched_verb=verb, matched_value_token=m.group("value"), - matched_unit_token=unit_raw, + matched_unit_token=unit_raw if unit_raw is not None else unit, matched_actor_token=m.group("subject"), matched_target_token=target, ) diff --git a/generate/math_roundtrip.py b/generate/math_roundtrip.py index 4a2076d3..0db7bda0 100644 --- a/generate/math_roundtrip.py +++ b/generate/math_roundtrip.py @@ -272,6 +272,30 @@ def _token_in(needle: str, haystack_tokens: frozenset[str]) -> bool: return needle.lower() in haystack_tokens +def _unit_grounds( + unit_token: str, + source_span: str, + haystack_tokens: frozenset[str], +) -> bool: + """A unit token grounds if it appears as a word token in source. + + ADR-0131.G.3 widening: when the canonical money unit ``cent`` is + claimed, the source's ``$`` symbol counts as grounding evidence — + the word-boundary tokenizer strips ``$`` so it must be inspected + on the raw source span rather than the token set. Similarly for + ``dollar``: an author may write either ``$N`` or ``N dollars``; + both ground a money unit. + """ + if _token_in(unit_token, haystack_tokens): + return True + if unit_token.lower() in ("cent", "cents"): + if "$" in source_span: + return True + if "dollar" in haystack_tokens or "dollars" in haystack_tokens: + return True + return False + + def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool: """A numeric value grounds if its surface token appears, OR if the token is a digit-string and any equivalent word-form appears, OR if it's a @@ -282,7 +306,42 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool: 1-12 to the full pack cardinal range (0-1000+ plus compound rule). The hard-coded WORD_NUMBERS remains as a fast path and as a fallback if the pack is unavailable; the pack adds, never replaces. + + ADR-0131.G.3 widens the literal-class grounding: + - Money symbol ``$N`` / ``$N.NN`` grounds when every digit run on + either side of the optional decimal appears as a token in the + source. The ``$`` itself is dropped by the word-boundary + tokenizer; what survives is exactly the digit form an author + would write. + - Slash fraction ``N/M`` grounds when both numerator and + denominator digit tokens appear. + - Hyphenated multi-word cardinal (``twenty-five``) grounds when + every component lemma is a token (the tokenizer splits on + hyphens), OR the compound's integer value's digit form appears. """ + # ADR-0131.G.3 widenings (handled first; the trailing existing path + # would never recognize these surface shapes). + if value_token.startswith("$"): + body = value_token[1:] + parts = [p for p in body.split(".") if p] + return bool(parts) and all(p in haystack_tokens for p in parts) + if "/" in value_token: + m = re.fullmatch(r"(\d+)/(\d+)", value_token) + if m is not None: + return m.group(1) in haystack_tokens and m.group(2) in haystack_tokens + if "-" in value_token and not value_token[0].isdigit(): + try: + from language_packs.numerics_loader import parse_compound_cardinal + parsed = parse_compound_cardinal(value_token) + if parsed is not None: + components = [c for c in value_token.lower().split("-") if c] + if all(c in haystack_tokens for c in components): + return True + if str(parsed) in haystack_tokens: + return True + except Exception: + pass + if _token_in(value_token, haystack_tokens): return True lowered = value_token.lower() @@ -366,7 +425,7 @@ def roundtrip_admissible(c: CandidateOperation) -> bool: # for comparison operands without explicit unit phrasing # ("Sam has twice as many as Tom"). if c.matched_unit_token: - if not _token_in(c.matched_unit_token, haystack): + if not _unit_grounds(c.matched_unit_token, c.source_span, haystack): return False else: if not isinstance(c.op.operand, Comparison): diff --git a/tests/test_adr_0131_G3_numerics.py b/tests/test_adr_0131_G3_numerics.py new file mode 100644 index 00000000..60319ce4 --- /dev/null +++ b/tests/test_adr_0131_G3_numerics.py @@ -0,0 +1,200 @@ +"""ADR-0131.G.3 — Numeric-literals capability-axis lane tests. + +Gate (must all pass for the lane to be considered green): + - safety rail: ``solved_wrong == 0`` on the axis lane + - safety rail: ``admitted_wrong == 0`` on the GSM8K probe (unchanged + by this iteration — G.3 widens the candidate-graph parser, which + the probe currently does not consult, so admission is not expected + to move; the wrong-count invariant is what's gated) + - axis-lane correctness: every ``solved_correct`` case in + ``cases.jsonl`` passes end-to-end; every ``refused`` probe refuses + with a typed reason at parser-or-solver layer + - per-class diversity: at least one case per non-refusal class + - replay determinism: ``report.json`` byte-equal across two runs + - resolver-level invariants for the new literal shapes +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from evals.math_capability_axes.G3_numerics.v1 import runner as axis_runner +from generate.math_candidate_parser import _resolve_value + + +_AXIS_DIR = Path(__file__).resolve().parent.parent / "evals" / "math_capability_axes" / "G3_numerics" / "v1" +_CASES_PATH = _AXIS_DIR / "cases.jsonl" + +# Closed set of classes this iteration is responsible for. The lane +# refuses to silently grow the class taxonomy — adding a new class is +# an ADR-level scope change. +_KNOWN_POSITIVE_CLASSES = frozenset({ + "money_symbol_integer", + "money_symbol_decimal", + "money_word", + "hyphenated_cardinal", +}) +_KNOWN_REFUSAL_CLASSES = frozenset({ + "refuse_money_precision", + "refuse_division_by_zero", + "refuse_unknown_compound", + "refuse_percentage", +}) + + +def _load_cases() -> list[dict]: + out = [] + for line in _CASES_PATH.read_text(encoding="utf-8").splitlines(): + if line.strip(): + out.append(json.loads(line)) + return out + + +# --------------------------------------------------------------------------- +# Resolver-level invariants (cheap, independent of the lane runner). +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize( + "token, expected_value, expected_unit_override", + [ + ("$40", 4000, "cents"), + ("$2.50", 250, "cents"), + ("$18.00", 1800, "cents"), + ("$0.99", 99, "cents"), + ("twenty-five", 25, None), + ("ninety-nine", 99, None), + ("3/4", 0.75, None), + ("5", 5, None), + ("twelve", 12, None), + ], +) +def test_resolve_value_admits_new_literals(token, expected_value, expected_unit_override): + rv = _resolve_value(token) + assert rv is not None, f"{token!r} should resolve" + assert rv.value == expected_value + assert rv.unit_override == expected_unit_override + + +@pytest.mark.parametrize( + "token", + [ + "$40.000", # >2 decimals — money precision out of scope + "$1.2345", + "5/0", # division-by-zero + "five-and-a-half", # unrecognized compound + "gobbledy-gook", # unrecognized compound + "50%", # percentage out of scope + "some", # indefinite quantifier + ], +) +def test_resolve_value_refuses_out_of_scope(token): + assert _resolve_value(token) is None, f"{token!r} should refuse" + + +# --------------------------------------------------------------------------- +# Dataset integrity invariants. +# --------------------------------------------------------------------------- + +def test_dataset_case_ids_unique(): + cases = _load_cases() + ids = [c["case_id"] for c in cases] + assert len(ids) == len(set(ids)), "duplicate case_id" + + +def test_dataset_class_taxonomy_is_closed(): + cases = _load_cases() + seen = {c["class"] for c in cases} + allowed = _KNOWN_POSITIVE_CLASSES | _KNOWN_REFUSAL_CLASSES + extra = seen - allowed + assert not extra, f"unknown class(es) — extend ADR scope before adding: {extra}" + + +def test_dataset_every_positive_class_has_at_least_one_case(): + cases = _load_cases() + by_class = {c["class"] for c in cases if c["expected_outcome"] == "solved_correct"} + missing = _KNOWN_POSITIVE_CLASSES - by_class + assert not missing, f"missing coverage for positive classes: {missing}" + + +def test_dataset_every_refusal_class_has_at_least_one_case(): + cases = _load_cases() + by_class = {c["class"] for c in cases if c["expected_outcome"] == "refused"} + missing = _KNOWN_REFUSAL_CLASSES - by_class + assert not missing, f"missing coverage for refusal classes: {missing}" + + +# --------------------------------------------------------------------------- +# Lane-level invariants (run the full runner). +# --------------------------------------------------------------------------- + +def test_axis_lane_safety_rail_no_wrong_answers(): + """ADR-0114a Obligation #4 — refusal-first; wrong-count must be 0.""" + report = axis_runner.build_report() + assert report["metrics"]["solved_wrong"] == 0, ( + f"wrong != 0: {report['verdict_counts']}" + ) + assert report["metrics"]["wrong_count_is_zero"] is True + + +def test_axis_lane_all_positive_cases_solved_correct(): + report = axis_runner.build_report() + assert report["metrics"]["correct_rate_on_positive_cases"] == 1.0 + + +def test_axis_lane_all_refusal_probes_refused_typed(): + report = axis_runner.build_report() + refusal_cases = [c for c in report["per_case"] if c["expected_outcome"] == "refused"] + for c in refusal_cases: + assert c["actual_outcome"] == "refused", ( + f"{c['case_id']}: expected refused, got {c['actual_outcome']}" + ) + assert c["reason"], f"{c['case_id']}: refusal must carry typed reason" + + +def test_axis_lane_overall_pass(): + report = axis_runner.build_report() + assert report["metrics"]["overall_pass"] is True + + +# --------------------------------------------------------------------------- +# Replay determinism — load-bearing per ADR-0114a. +# --------------------------------------------------------------------------- + +def test_axis_lane_report_replay_byte_equal(): + r1 = json.dumps(axis_runner.build_report(), indent=2, sort_keys=True) + r2 = json.dumps(axis_runner.build_report(), indent=2, sort_keys=True) + assert r1 == r2, "axis lane must be deterministic" + + +def test_committed_report_matches_fresh_run(): + """The committed ``report.json`` must equal a fresh run — keeps the + artifact diff-able as load-bearing evidence per ADR-0131.G.""" + fresh = json.dumps(axis_runner.build_report(), indent=2, sort_keys=True) + "\n" + committed = (_AXIS_DIR / "report.json").read_text(encoding="utf-8") + assert fresh == committed, ( + "committed report.json is stale; run " + "`python3 -m evals.math_capability_axes.G3_numerics.v1.runner` to refresh" + ) + + +# --------------------------------------------------------------------------- +# GSM8K-probe safety rail — must still hold (the non-negotiable gate). +# --------------------------------------------------------------------------- + +def test_gsm8k_probe_safety_rail_unchanged(): + """ADR-0131.G's safety rail: ``admitted_wrong == 0`` on the GSM8K + coverage probe is the load-bearing invariant every G. iteration + must preserve. G.3 widens the candidate-graph parser; this asserts + that the probe (which runs through the legacy parser path) still + refuses cleanly without confabulating. + """ + from evals.gsm8k_math.train_sample.v1.run_coverage_probe import build_report + probe_report = build_report() + m = probe_report["metrics"] + assert m["admitted_wrong"] == 0, ( + "GSM8K probe safety rail breached — admitted_wrong > 0" + ) + assert m["safety_rail_intact"] is True