feat(ADR-0131.G.3): numeric literals (money + fractions + compounds) — admission 0/50 (Δ+0)
This commit is contained in:
parent
481e0c3b9c
commit
5213233836
9 changed files with 798 additions and 127 deletions
57
docs/decisions/ADR-0131.G.3-numerics.md
Normal file
57
docs/decisions/ADR-0131.G.3-numerics.md
Normal file
|
|
@ -0,0 +1,57 @@
|
||||||
|
# ADR-0131.G.3 — Capability Axis: Numeric Literals (Money, Fractions, Compound Numbers)
|
||||||
|
|
||||||
|
## Status
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Date
|
||||||
|
2026-05-23
|
||||||
|
|
||||||
|
## Context
|
||||||
|
As part of the ADR-0131 re-benchmarking roadmap, the GSM8K coverage probe identifies capability axes along which the NL-to-typed-graph parser must be extended. This decision specifies the capability axis G.3: support for money, fractions, and compound numeric literals.
|
||||||
|
|
||||||
|
Historically, the parser was restricted to simple integer digits and single-word numbers (1 to 12). Real-world math problems introduce currency notation, fractional quantities (both digit slash forms and spelled equivalents), adjectives qualifying units, and hyphenated numeric nouns.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
Extend the candidate-graph parser layer to consume the existing semantic packs `en_units_v1` (ADR-0127) and `en_numerics_v1` (ADR-0128) to recognize these literal shapes without hardcoding inline regex alternations.
|
||||||
|
|
||||||
|
### 1. Closed Literal Classes
|
||||||
|
We implement recognition for four distinct literal classes:
|
||||||
|
* **Money**: Recognizes currency symbols and currency word forms. Leading symbols map to canonical plural units via `en_units_v1`.
|
||||||
|
* **Fractions**: Recognizes digit slash forms (e.g., `3/4`) and spelled equivalents in the pack (e.g., `one-half`, `three-quarters`). Fractions are converted to floats in `Quantity` structures. Prepositional `of <substance>` tails (e.g. `3/4 of a cake`) are parsed, using the substance as the unit where explicit unit tokens are absent.
|
||||||
|
* **Word-number compositions**: Recognizes combinations of spelling numbers, adjectives, and head nouns (e.g., `five full boxes of crayons`). The adjective is correctly retained as part of the unit (`full boxes of crayons` ≢ `boxes`).
|
||||||
|
* **Hyphenated compound numerics**: Recognizes compound adjectives modifying a unit head noun (e.g., `10 one-hour videos`).
|
||||||
|
|
||||||
|
### 2. Currency-Symbol → Unit Mapping
|
||||||
|
Leading currency symbols resolve deterministically to plural units in the unit pack:
|
||||||
|
* `$` → `"dollars"`
|
||||||
|
* `¢` → `"cents"`
|
||||||
|
* `€` → `"euros"`
|
||||||
|
* `£` → `"pounds"`
|
||||||
|
* `¥` → `"yens"`
|
||||||
|
* `₱` → `"pesos"`
|
||||||
|
|
||||||
|
Currency symbols compose with rate denominators (e.g., `$` and `hour` → `"dollars per hour"`; `$0.75 each` → `"dollars per item"`).
|
||||||
|
|
||||||
|
### 3. Scope Pinning and Refusal Probes
|
||||||
|
To preserve the `wrong == 0` firewall, inputs that exceed well-defined boundaries are cleanly refused:
|
||||||
|
* **Decimals > 2 in currency**: `$18.0000` is refused at the parser layer.
|
||||||
|
* **Division by zero**: Fractions like `1/0` fail solver/graph construction and refuse.
|
||||||
|
* **Ambiguous hyphenations**: Expressions like `one-hour-old` refuse.
|
||||||
|
* **Out of scope notation**: Percentages (`10%`) and scientific notation (`1e3`) are excluded and refuse.
|
||||||
|
|
||||||
|
### 4. Deferred Shapes
|
||||||
|
The following shapes remain out of scope for G.3 and will be addressed in future capability iterations:
|
||||||
|
* **Distributive operators**: Distributive scopes (like "each saved up $40" or "each weighing 5 ounces") are deferred to G.4.
|
||||||
|
* **Unit inheritance**: Re-using the last referenced unit (e.g., "gives 1/4 of that") requires state threading and is deferred to P3 grammar integration.
|
||||||
|
* **Multi-clause sentence syntax**: Compound sentences linked by conjunctions (like `but then lost 12` or `and her friend has`) are refused.
|
||||||
|
|
||||||
|
## Verification Plan
|
||||||
|
* **Curated Benchmarks**: Author 25 isolated capability cases under `evals/math_capability_axes/G3_numerics/v1/cases.jsonl` (5 per literal class, 5 refusal probes).
|
||||||
|
* **Runner Verification**: The runner at `evals/math_capability_axes/G3_numerics/v1/runner.py` must score `correct_rate == 100%` against expectations and `wrong == 0`.
|
||||||
|
* **Determinism**: The runner output `report.json` must be byte-equal across consecutive runs.
|
||||||
|
* **Regression Testing**: All test suites must remain green (`pytest tests/test_adr_0131_G3_numerics.py` and `tests/test_adr_0131_*.py`).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
* The G3 capability axis is fully verified and landed under the safety rail.
|
||||||
|
* Parsing of money, fractions, and compound noun phrases is now fully pack-driven and cached at start-up.
|
||||||
|
* The baseline GSM8K train-sample coverage probe shows that targeted first-sentence refusals (like `Tina makes $18.00 an hour`) now successfully parse, moving the refusal frontier further down the problem structure.
|
||||||
|
|
@ -13,7 +13,7 @@
|
||||||
"per_case": [
|
"per_case": [
|
||||||
{
|
{
|
||||||
"case_id": "gsm8k-train-sample-v1-0001",
|
"case_id": "gsm8k-train-sample-v1-0001",
|
||||||
"reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'",
|
"reason": "candidate_graph: no admissible candidate for statement: 'If she works more than 8 hours per shift, she is eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage.'",
|
||||||
"verdict": "refused"
|
"verdict": "refused"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
@ -123,7 +123,7 @@
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"case_id": "gsm8k-train-sample-v1-0023",
|
"case_id": "gsm8k-train-sample-v1-0023",
|
||||||
"reason": "candidate_graph: no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'",
|
"reason": "candidate_graph: no admissible candidate for statement: \"Cindy collected twice as many, and Rex collected half of Nicole and Cindy's combined total.\"",
|
||||||
"verdict": "refused"
|
"verdict": "refused"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
@ -143,7 +143,7 @@
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"case_id": "gsm8k-train-sample-v1-0027",
|
"case_id": "gsm8k-train-sample-v1-0027",
|
||||||
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.'",
|
"reason": "candidate_graph: no admissible candidate for statement: 'The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined.'",
|
||||||
"verdict": "refused"
|
"verdict": "refused"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
@ -218,7 +218,7 @@
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"case_id": "gsm8k-train-sample-v1-0042",
|
"case_id": "gsm8k-train-sample-v1-0042",
|
||||||
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.'",
|
"reason": "candidate_graph: no admissible candidate for question: 'If Ella sells 200 apples, how many apples does Ella has left?'",
|
||||||
"verdict": "refused"
|
"verdict": "refused"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
|
||||||
25
evals/math_capability_axes/G3_numerics/v1/cases.jsonl
Normal file
25
evals/math_capability_axes/G3_numerics/v1/cases.jsonl
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
{"case_id": "g3-001", "problem": "Tina has $18.00. How many dollars does Tina have?", "expected": "solved_correct", "expected_answer": 18.0, "expected_unit": "dollars", "shape_category": "money"}
|
||||||
|
{"case_id": "g3-002", "problem": "Sam has 50 cents. How many cents does Sam have?", "expected": "solved_correct", "expected_answer": 50.0, "expected_unit": "cents", "shape_category": "money"}
|
||||||
|
{"case_id": "g3-003", "problem": "Jan has $40. How many dollars does Jan have?", "expected": "solved_correct", "expected_answer": 40.0, "expected_unit": "dollars", "shape_category": "money"}
|
||||||
|
{"case_id": "g3-004", "problem": "Marc has 5 dollars. How many dollars does Marc have?", "expected": "solved_correct", "expected_answer": 5.0, "expected_unit": "dollars", "shape_category": "money"}
|
||||||
|
{"case_id": "g3-005", "problem": "Tina has $18.00. Tina buys $2.00. How many dollars does Tina have?", "expected": "solved_correct", "expected_answer": 20.0, "expected_unit": "dollars", "shape_category": "money"}
|
||||||
|
{"case_id": "g3-006", "problem": "Jan has 3/4 of a cake. How many cake does Jan have?", "expected": "solved_correct", "expected_answer": 0.75, "expected_unit": "cakes", "shape_category": "fractions"}
|
||||||
|
{"case_id": "g3-007", "problem": "Sam has one-half of an apple. How many apples does Sam have?", "expected": "solved_correct", "expected_answer": 0.5, "expected_unit": "apples", "shape_category": "fractions"}
|
||||||
|
{"case_id": "g3-008", "problem": "Tom has three-quarters of a pie. How many pies does Tom have?", "expected": "solved_correct", "expected_answer": 0.75, "expected_unit": "pies", "shape_category": "fractions"}
|
||||||
|
{"case_id": "g3-009", "problem": "Amy has 1/2 of a cookie. How many cookies does Amy have?", "expected": "solved_correct", "expected_answer": 0.5, "expected_unit": "cookies", "shape_category": "fractions"}
|
||||||
|
{"case_id": "g3-010", "problem": "Dan has 2/5 of a pizza. How many pizzas does Dan have?", "expected": "solved_correct", "expected_answer": 0.4, "expected_unit": "pizzas", "shape_category": "fractions"}
|
||||||
|
{"case_id": "g3-011", "problem": "Francine has five full boxes of crayons. How many full boxes of crayons does Francine have?", "expected": "solved_correct", "expected_answer": 5.0, "expected_unit": "full boxes of crayons", "shape_category": "word_number_compositions"}
|
||||||
|
{"case_id": "g3-012", "problem": "Sam has three large apples. How many large apples does Sam have?", "expected": "solved_correct", "expected_answer": 3.0, "expected_unit": "large apples", "shape_category": "word_number_compositions"}
|
||||||
|
{"case_id": "g3-013", "problem": "Jan has two empty bags. How many empty bags does Jan have?", "expected": "solved_correct", "expected_answer": 2.0, "expected_unit": "empty bags", "shape_category": "word_number_compositions"}
|
||||||
|
{"case_id": "g3-014", "problem": "Tim has ten clean shirts. How many clean shirts does Tim have?", "expected": "solved_correct", "expected_answer": 10.0, "expected_unit": "clean shirts", "shape_category": "word_number_compositions"}
|
||||||
|
{"case_id": "g3-015", "problem": "Dan has six blue pens. How many blue pens does Dan have?", "expected": "solved_correct", "expected_answer": 6.0, "expected_unit": "blue pens", "shape_category": "word_number_compositions"}
|
||||||
|
{"case_id": "g3-016", "problem": "Allison has 10 one-hour videos. How many one-hour videos does Allison have?", "expected": "solved_correct", "expected_answer": 10.0, "expected_unit": "one-hour videos", "shape_category": "hyphenated_compound_numerics"}
|
||||||
|
{"case_id": "g3-017", "problem": "Sam has 5 ten-page books. How many ten-page books does Sam have?", "expected": "solved_correct", "expected_answer": 5.0, "expected_unit": "ten-page books", "shape_category": "hyphenated_compound_numerics"}
|
||||||
|
{"case_id": "g3-018", "problem": "Jan has two five-mile runs. How many five-mile runs does Jan have?", "expected": "solved_correct", "expected_answer": 2.0, "expected_unit": "five-mile runs", "shape_category": "hyphenated_compound_numerics"}
|
||||||
|
{"case_id": "g3-019", "problem": "Marc has three two-liter bottles. How many two-liter bottles does Marc have?", "expected": "solved_correct", "expected_answer": 3.0, "expected_unit": "two-liter bottles", "shape_category": "hyphenated_compound_numerics"}
|
||||||
|
{"case_id": "g3-020", "problem": "Tina has 12 five-minute talks. How many five-minute talks does Tina have?", "expected": "solved_correct", "expected_answer": 12.0, "expected_unit": "five-minute talks", "shape_category": "hyphenated_compound_numerics"}
|
||||||
|
{"case_id": "g3-021", "problem": "Tina has $18.0000. How many dollars does Tina have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_probe"}
|
||||||
|
{"case_id": "g3-022", "problem": "Jan has 1/0 of a cake. How many cake does Jan have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_probe"}
|
||||||
|
{"case_id": "g3-023", "problem": "Sam has one-hour-old baby. How many babies does Sam have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_probe"}
|
||||||
|
{"case_id": "g3-024", "problem": "Marc has 10% of a pizza. How many pizzas does Marc have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_probe"}
|
||||||
|
{"case_id": "g3-025", "problem": "Dan has 1e3 pizzas. How many pizzas does Dan have?", "expected": "refused", "expected_answer": null, "expected_unit": null, "shape_category": "refused_probe"}
|
||||||
172
evals/math_capability_axes/G3_numerics/v1/report.json
Normal file
172
evals/math_capability_axes/G3_numerics/v1/report.json
Normal file
|
|
@ -0,0 +1,172 @@
|
||||||
|
{
|
||||||
|
"adr": "0131.G.3",
|
||||||
|
"exit_criterion": {
|
||||||
|
"correct_min_rate": 1.0,
|
||||||
|
"passed": true,
|
||||||
|
"wrong_max": 0
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"cases_total": 25,
|
||||||
|
"correct": 25,
|
||||||
|
"correct_rate": 1.0,
|
||||||
|
"overall_pass": true,
|
||||||
|
"refused": 0,
|
||||||
|
"wrong": 0,
|
||||||
|
"wrong_count_is_zero": true
|
||||||
|
},
|
||||||
|
"per_case": [
|
||||||
|
{
|
||||||
|
"case_id": "g3-001",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-002",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-003",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-004",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-005",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-006",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-007",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-008",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-009",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-010",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-011",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-012",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-013",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-014",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-015",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-016",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-017",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-018",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-019",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-020",
|
||||||
|
"outcome": "correct",
|
||||||
|
"reason": "",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-021",
|
||||||
|
"outcome": "refused",
|
||||||
|
"reason": "candidate_graph: no admissible candidate for statement: 'Tina has $18.0000.'",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-022",
|
||||||
|
"outcome": "refused",
|
||||||
|
"reason": "candidate_graph: no admissible candidate for statement: 'Jan has 1/0 of a cake.'",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-023",
|
||||||
|
"outcome": "refused",
|
||||||
|
"reason": "candidate_graph: no admissible candidate for statement: 'Sam has one-hour-old baby.'",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-024",
|
||||||
|
"outcome": "refused",
|
||||||
|
"reason": "candidate_graph: no admissible candidate for statement: 'Marc has 10% of a pizza.'",
|
||||||
|
"verdict": "correct"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"case_id": "g3-025",
|
||||||
|
"outcome": "refused",
|
||||||
|
"reason": "candidate_graph: no admissible candidate for statement: 'Dan has 1e3 pizzas.'",
|
||||||
|
"verdict": "correct"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"sample_count": 25,
|
||||||
|
"sample_path": "evals/math_capability_axes/G3_numerics/v1/cases.jsonl",
|
||||||
|
"schema_version": 1
|
||||||
|
}
|
||||||
120
evals/math_capability_axes/G3_numerics/v1/runner.py
Normal file
120
evals/math_capability_axes/G3_numerics/v1/runner.py
Normal file
|
|
@ -0,0 +1,120 @@
|
||||||
|
"""ADR-0131.G.3 — G3 Numerics capability runner.
|
||||||
|
|
||||||
|
Feeds curated cases from cases.jsonl through the candidate-graph pipeline,
|
||||||
|
ensuring wrong == 0 is preserved and verifying the correct outcomes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from evals.gsm8k_math.runner import _score_one_candidate_graph
|
||||||
|
|
||||||
|
_HERE = Path(__file__).resolve().parent
|
||||||
|
_CASES_PATH = _HERE / "cases.jsonl"
|
||||||
|
_REPORT_PATH = _HERE / "report.json"
|
||||||
|
|
||||||
|
|
||||||
|
def load_cases(path: Path = _CASES_PATH) -> list[dict[str, Any]]:
|
||||||
|
records: list[dict[str, Any]] = []
|
||||||
|
with path.open("r", encoding="utf-8") as fh:
|
||||||
|
for line in fh:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
records.append(json.loads(line))
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def _adapt(case: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": case["case_id"],
|
||||||
|
"problem": case["problem"],
|
||||||
|
"expected_answer": case["expected_answer"] if case["expected_answer"] is not None else 0.0,
|
||||||
|
"expected_unit": case["expected_unit"] if case["expected_unit"] is not None else "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def build_report(cases: list[dict[str, Any]]) -> dict[str, Any]:
|
||||||
|
per_case: list[dict[str, Any]] = []
|
||||||
|
counts = {"correct": 0, "wrong": 0, "refused": 0}
|
||||||
|
|
||||||
|
for raw in cases:
|
||||||
|
expected_outcome = raw["expected"]
|
||||||
|
outcome = _score_one_candidate_graph(_adapt(raw))
|
||||||
|
|
||||||
|
# Decide if the outcome matches expectation
|
||||||
|
if expected_outcome == "solved_correct":
|
||||||
|
if outcome.outcome == "correct":
|
||||||
|
verdict = "correct"
|
||||||
|
else:
|
||||||
|
verdict = outcome.outcome
|
||||||
|
elif expected_outcome == "refused":
|
||||||
|
if outcome.outcome == "refused":
|
||||||
|
verdict = "correct"
|
||||||
|
else:
|
||||||
|
verdict = "wrong"
|
||||||
|
else:
|
||||||
|
verdict = "wrong"
|
||||||
|
|
||||||
|
counts[verdict] += 1
|
||||||
|
per_case.append(
|
||||||
|
{
|
||||||
|
"case_id": raw["case_id"],
|
||||||
|
"verdict": verdict,
|
||||||
|
"outcome": outcome.outcome,
|
||||||
|
"reason": outcome.reason,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
total = len(cases)
|
||||||
|
correct_rate = counts["correct"] / total if total else 0.0
|
||||||
|
wrong_count_is_zero = counts["wrong"] == 0
|
||||||
|
passed = wrong_count_is_zero and (correct_rate >= 1.0)
|
||||||
|
|
||||||
|
metrics = {
|
||||||
|
"cases_total": total,
|
||||||
|
"correct": counts["correct"],
|
||||||
|
"wrong": counts["wrong"],
|
||||||
|
"refused": counts["refused"],
|
||||||
|
"correct_rate": correct_rate,
|
||||||
|
"wrong_count_is_zero": wrong_count_is_zero,
|
||||||
|
"overall_pass": passed,
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"adr": "0131.G.3",
|
||||||
|
"sample_path": "evals/math_capability_axes/G3_numerics/v1/cases.jsonl",
|
||||||
|
"sample_count": total,
|
||||||
|
"metrics": metrics,
|
||||||
|
"exit_criterion": {
|
||||||
|
"correct_min_rate": 1.0,
|
||||||
|
"wrong_max": 0,
|
||||||
|
"passed": passed,
|
||||||
|
},
|
||||||
|
"per_case": per_case,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def write_report(report: dict[str, Any], path: Path = _REPORT_PATH) -> None:
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(report, indent=2, sort_keys=True) + "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
cases = load_cases()
|
||||||
|
report = build_report(cases)
|
||||||
|
write_report(report)
|
||||||
|
print(f"Metrics: {report['metrics']}")
|
||||||
|
return 0 if report["exit_criterion"]["passed"] else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
|
|
@ -158,9 +158,17 @@ def _initial_admissible(ic: CandidateInitial) -> bool:
|
||||||
haystack = _tokens(ic.source_span)
|
haystack = _tokens(ic.source_span)
|
||||||
if not _token_in(ic.matched_anchor, haystack):
|
if not _token_in(ic.matched_anchor, haystack):
|
||||||
return False
|
return False
|
||||||
if not _value_grounds(ic.matched_value_token, haystack):
|
if not _value_grounds(ic.matched_value_token, haystack, ic.source_span):
|
||||||
return False
|
return False
|
||||||
if not _token_in(ic.matched_unit_token, haystack):
|
if ic.matched_unit_token:
|
||||||
|
parts = re.split(r'[- ]', ic.matched_unit_token)
|
||||||
|
for part in parts:
|
||||||
|
part = part.strip()
|
||||||
|
if part and not _token_in(part, haystack):
|
||||||
|
if part in ("dollar", "dollars") and "$" in ic.source_span:
|
||||||
|
continue
|
||||||
|
if part in ("cent", "cents") and "¢" in ic.source_span:
|
||||||
|
continue
|
||||||
return False
|
return False
|
||||||
# Entity token: for multi-word entities ("the boys"), all words
|
# Entity token: for multi-word entities ("the boys"), all words
|
||||||
# must ground. Split + check each.
|
# must ground. Split + check each.
|
||||||
|
|
@ -174,7 +182,13 @@ def _question_admissible(qc: CandidateUnknown) -> bool:
|
||||||
"""Light structural ground-check for question candidates."""
|
"""Light structural ground-check for question candidates."""
|
||||||
from generate.math_roundtrip import _tokens, _token_in
|
from generate.math_roundtrip import _tokens, _token_in
|
||||||
haystack = _tokens(qc.source_span)
|
haystack = _tokens(qc.source_span)
|
||||||
if not _token_in(qc.matched_unit_token, haystack):
|
if qc.matched_unit_token:
|
||||||
|
parts = re.split(r'[- ]', qc.matched_unit_token)
|
||||||
|
for part in parts:
|
||||||
|
part = part.strip()
|
||||||
|
if part and not _token_in(part, haystack):
|
||||||
|
return False
|
||||||
|
else:
|
||||||
return False
|
return False
|
||||||
if qc.matched_entity_token is not None:
|
if qc.matched_entity_token is not None:
|
||||||
for tok in qc.matched_entity_token.split():
|
for tok in qc.matched_entity_token.split():
|
||||||
|
|
|
||||||
|
|
@ -92,14 +92,54 @@ class CandidateInitial:
|
||||||
# math_parser._INITIAL_HAS_RE's ADR-0123a entity slot.
|
# math_parser._INITIAL_HAS_RE's ADR-0123a entity slot.
|
||||||
_ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)"
|
_ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)"
|
||||||
|
|
||||||
# Numeric value: digit run OR word-form integer (one..twelve initially;
|
# Dynamic value-slot regex builder
|
||||||
# WORD_NUMBERS table is wider but we cap the regex at the common range
|
def _build_value_regex() -> str:
|
||||||
# for syntactic parsing and let the filter handle ground-truth value
|
fallback_words = "|".join(
|
||||||
# equivalence).
|
|
||||||
_WORD_NUM_OPTIONS: Final[str] = "|".join(
|
|
||||||
re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True)
|
re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True)
|
||||||
)
|
)
|
||||||
_VALUE: Final[str] = rf"(?:\d+|{_WORD_NUM_OPTIONS})"
|
fallback = rf"(?:\d+|{fallback_words})"
|
||||||
|
try:
|
||||||
|
from language_packs.numerics_loader import _index
|
||||||
|
idx = _index()
|
||||||
|
cardinal_words = sorted(idx.cardinals.keys(), key=len, reverse=True)
|
||||||
|
ordinal_words = sorted(idx.ordinals.keys(), key=len, reverse=True)
|
||||||
|
fraction_words = sorted(idx.fractions.keys(), key=len, reverse=True)
|
||||||
|
multiplier_words = sorted(idx.multipliers.keys(), key=len, reverse=True)
|
||||||
|
quantifier_words = sorted(idx.quantifiers.keys(), key=len, reverse=True)
|
||||||
|
|
||||||
|
denom_plurals = ["halves", "thirds", "quarters", "fourths", "fifths", "sixths", "sevenths", "eighths", "ninths", "tenths", "sixteenths"]
|
||||||
|
|
||||||
|
all_singles = set(cardinal_words + ordinal_words + fraction_words + multiplier_words + quantifier_words)
|
||||||
|
|
||||||
|
cards_pat = "|".join(re.escape(w) for w in cardinal_words)
|
||||||
|
ords_pat = "|".join(re.escape(w) for w in ordinal_words)
|
||||||
|
fracs_pat = "|".join(re.escape(w) for w in fraction_words)
|
||||||
|
denoms_pat = "|".join(re.escape(w) for w in (ordinal_words + fraction_words + denom_plurals))
|
||||||
|
|
||||||
|
comp_card_pat = rf"(?:{cards_pat})(?:[- ](?:and[- ])?(?:{cards_pat})){{0,4}}"
|
||||||
|
comp_frac_pat = rf"(?:{comp_card_pat})[- ](?:{denoms_pat})"
|
||||||
|
|
||||||
|
patterns = [
|
||||||
|
r"\d+\s+\d+/\d+",
|
||||||
|
r"\d+/\d+",
|
||||||
|
r"[\$\u20ac\u00a3\u00a5\u20b1\u00a2]?\d+(?:\.\d+)?",
|
||||||
|
comp_frac_pat,
|
||||||
|
comp_card_pat,
|
||||||
|
]
|
||||||
|
for w in all_singles:
|
||||||
|
patterns.append(re.escape(w))
|
||||||
|
return "|".join(patterns)
|
||||||
|
except Exception:
|
||||||
|
return fallback
|
||||||
|
|
||||||
|
_VALUE: Final[str] = _build_value_regex()
|
||||||
|
|
||||||
|
_UNIT: Final[str] = (
|
||||||
|
r"(?:(?!to\b)(?!more\b)(?!on\b)(?!from\b)(?!at\b)(?!in\b)"
|
||||||
|
r"(?!onto\b)(?!into\b)(?!under\b)(?!over\b)(?!of\b)(?!for\b)(?!with\b)"
|
||||||
|
r"(?!today\b)(?!now\b)(?!yesterday\b)(?!initially\b)\w+)+"
|
||||||
|
r"(?:[- ]\w+)*"
|
||||||
|
)
|
||||||
|
|
||||||
# Verb alternation built from the permissive registry. Pre-compute one
|
# Verb alternation built from the permissive registry. Pre-compute one
|
||||||
# pattern per kind so we can attribute matched verbs to candidates.
|
# pattern per kind so we can attribute matched verbs to candidates.
|
||||||
|
|
@ -118,30 +158,13 @@ _TRANSFER_VERBS_PATTERN: Final[str] = _verbs_pattern(TRANSFER_VERBS)
|
||||||
# Initial-possession extractor
|
# Initial-possession extractor
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
_INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile(
|
|
||||||
rf"^(?P<entity>{_ENTITY})\s+"
|
|
||||||
rf"(?P<anchor>has|have)\s+"
|
|
||||||
rf"(?P<value>{_VALUE})\s+"
|
|
||||||
r"(?P<unit>\w+)"
|
|
||||||
# ADR-0127 substance qualifier: "Sam has 5 feet of rope" — the
|
|
||||||
# 'of <NP>' tail is grammatically real but arithmetically inert.
|
|
||||||
r"(?:\s+of\s+.+)?"
|
|
||||||
r"\s*\.?$"
|
|
||||||
)
|
|
||||||
|
|
||||||
# ADR-0127 "There are/were N <unit> [in <place>]" initial-possession shape.
|
|
||||||
# The implicit-subject anchor 'there are' is the only initial-possession
|
|
||||||
# shape that doesn't name an entity in the source; we treat the
|
|
||||||
# place phrase (when present) as the entity and treat the unit as the
|
|
||||||
# count noun. When no place is named, the entity is the unit itself
|
|
||||||
# (collective). Indefinite quantifiers ('some', 'few', 'many') in the
|
|
||||||
# value slot are refused upstream by extract_initial_candidates via
|
|
||||||
# the quantifier-driven refusal helper (ADR-0128.4).
|
|
||||||
_INITIAL_THERE_ARE_RE: Final[re.Pattern[str]] = re.compile(
|
_INITIAL_THERE_ARE_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
r"^There\s+(?P<anchor>are|were|is|was)\s+"
|
r"^(?:.*\b)?there\s+(?P<anchor>are|were|is|was)\s+"
|
||||||
rf"(?P<value>{_VALUE})\s+"
|
rf"(?P<value>{_VALUE})\s+"
|
||||||
r"(?P<unit>\w+)"
|
rf"(?P<unit>{_UNIT})"
|
||||||
r"(?:\s+in\s+(?P<place>[A-Za-z]\w*(?:\s+\w+)?))?"
|
r"(?:\s+of\s+(?P<substance>[a-zA-Z]\w*(?:\s+\w+)*))?"
|
||||||
|
r"(?:\s+(?:in|on|at|inside|outside)\s+(?P<place>[A-Za-z]\w*(?:\s+\w+)?))?"
|
||||||
|
r"(?:\s+[a-zA-Z]+)*"
|
||||||
r"\s*\.?$",
|
r"\s*\.?$",
|
||||||
flags=re.IGNORECASE,
|
flags=re.IGNORECASE,
|
||||||
)
|
)
|
||||||
|
|
@ -156,10 +179,115 @@ def _normalize_entity(raw: str) -> str:
|
||||||
return e
|
return e
|
||||||
|
|
||||||
|
|
||||||
def _resolve_value(value_token: str) -> int:
|
def _resolve_currency_and_value(value_token: str) -> tuple[float | int, str | None]:
|
||||||
if value_token.isdigit():
|
token = value_token.strip()
|
||||||
return int(value_token)
|
currency_unit = None
|
||||||
return WORD_NUMBERS[value_token.lower()]
|
|
||||||
|
# Check for leading currency symbols ($, €, £, ¥, ₱, ¢)
|
||||||
|
if token and not token[0].isalnum() and token[0] != '-':
|
||||||
|
symbol = token[0]
|
||||||
|
try:
|
||||||
|
from language_packs.loader import lookup_unit
|
||||||
|
entry = lookup_unit(symbol)
|
||||||
|
if entry is not None:
|
||||||
|
currency_unit = entry.plural.lower()
|
||||||
|
token = token[1:].strip()
|
||||||
|
except Exception:
|
||||||
|
if symbol == '$':
|
||||||
|
currency_unit = "dollars"
|
||||||
|
token = token[1:].strip()
|
||||||
|
|
||||||
|
if currency_unit is not None:
|
||||||
|
if '.' in token:
|
||||||
|
decimals = token.split('.')[-1].strip('%')
|
||||||
|
if len(decimals) > 2:
|
||||||
|
raise ValueError("Too many decimal places for currency")
|
||||||
|
|
||||||
|
# Parse numeric value
|
||||||
|
try:
|
||||||
|
from language_packs.loader import match_number_format
|
||||||
|
parsed = match_number_format(token)
|
||||||
|
if parsed is not None:
|
||||||
|
val = parsed.value
|
||||||
|
from fractions import Fraction
|
||||||
|
if isinstance(val, Fraction):
|
||||||
|
val = float(val)
|
||||||
|
return val, currency_unit
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
from language_packs.loader import lookup_fraction
|
||||||
|
frac_entry = lookup_fraction(token)
|
||||||
|
if frac_entry is not None:
|
||||||
|
return float(frac_entry.decimal_value), currency_unit
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
from language_packs.loader import parse_compound_cardinal
|
||||||
|
comp_val = parse_compound_cardinal(token)
|
||||||
|
if comp_val is not None:
|
||||||
|
return comp_val, currency_unit
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if token.isdigit():
|
||||||
|
return int(token), currency_unit
|
||||||
|
try:
|
||||||
|
return float(token), currency_unit
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
lowered = token.lower()
|
||||||
|
if lowered in WORD_NUMBERS:
|
||||||
|
return WORD_NUMBERS[lowered], currency_unit
|
||||||
|
|
||||||
|
raise ValueError(f"Could not resolve numeric value from token: {value_token!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_value(value_token: str) -> float | int:
|
||||||
|
val, _ = _resolve_currency_and_value(value_token)
|
||||||
|
return val
|
||||||
|
|
||||||
|
|
||||||
|
def _compose_unit(currency_unit: str | None, matched_unit: str | None) -> str | None:
|
||||||
|
if not currency_unit:
|
||||||
|
if not matched_unit:
|
||||||
|
return None
|
||||||
|
return _canonicalize_unit(matched_unit)
|
||||||
|
if not matched_unit:
|
||||||
|
return currency_unit
|
||||||
|
|
||||||
|
mu_low = matched_unit.lower().strip()
|
||||||
|
for conn in ("an ", "a ", "per ", "each "):
|
||||||
|
if mu_low.startswith(conn):
|
||||||
|
mu_low = mu_low[len(conn):].strip()
|
||||||
|
|
||||||
|
if mu_low == "each" or mu_low == "":
|
||||||
|
rate_denom = "item"
|
||||||
|
else:
|
||||||
|
rate_denom = _canonicalize_unit(mu_low)
|
||||||
|
try:
|
||||||
|
from language_packs.loader import lookup_unit
|
||||||
|
entry = lookup_unit(rate_denom)
|
||||||
|
if entry is not None:
|
||||||
|
rate_denom = entry.singular
|
||||||
|
elif rate_denom.endswith("s"):
|
||||||
|
rate_denom = rate_denom[:-1]
|
||||||
|
except Exception:
|
||||||
|
if rate_denom.endswith("s"):
|
||||||
|
rate_denom = rate_denom[:-1]
|
||||||
|
|
||||||
|
composed_raw = f"{currency_unit} per {rate_denom}"
|
||||||
|
try:
|
||||||
|
from language_packs.loader import lookup_unit
|
||||||
|
entry = lookup_unit(composed_raw)
|
||||||
|
if entry is not None:
|
||||||
|
return entry.plural.lower()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return _canonicalize_unit(composed_raw)
|
||||||
|
|
||||||
|
|
||||||
def _is_indefinite_quantifier(token: str) -> bool:
|
def _is_indefinite_quantifier(token: str) -> bool:
|
||||||
|
|
@ -184,50 +312,102 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
|
||||||
"""Return all admissible initial-possession candidates for ``sentence``.
|
"""Return all admissible initial-possession candidates for ``sentence``.
|
||||||
|
|
||||||
Recognized shapes:
|
Recognized shapes:
|
||||||
1. "<Entity> has <N> <unit> [of <substance>]" — canonical.
|
1. "<Entity> has <N> <unit> [of <substance>]" — canonical, supporting compound possessions.
|
||||||
2. "There are <N> <unit> [in <place>]" — implicit-subject shape.
|
2. "There are <N> <unit> [in <place>]" — implicit-subject shape.
|
||||||
|
|
||||||
ADR-0128.4: if the value slot resolves to an indefinite quantifier
|
|
||||||
(`some kids`, `many things`), no candidate is emitted (refusal
|
|
||||||
preserves wrong == 0).
|
|
||||||
"""
|
"""
|
||||||
s = sentence.strip().rstrip(".")
|
s = sentence.strip().rstrip(".")
|
||||||
out: list[CandidateInitial] = []
|
out: list[CandidateInitial] = []
|
||||||
|
|
||||||
m = _INITIAL_HAS_RE.match(s)
|
m_has = re.match(
|
||||||
if m is not None:
|
rf"^(?P<entity>{_ENTITY})\s+(?P<anchor>has|have)\s+(?P<quantities>.+)$",
|
||||||
value_raw = m.group("value")
|
s,
|
||||||
|
flags=re.IGNORECASE
|
||||||
|
)
|
||||||
|
if m_has is not None:
|
||||||
|
entity_raw = m_has.group("entity")
|
||||||
|
entity = _normalize_entity(entity_raw)
|
||||||
|
anchor = m_has.group("anchor")
|
||||||
|
quantities_str = m_has.group("quantities")
|
||||||
|
|
||||||
|
parts = re.split(r",?\s+and\s+", quantities_str, flags=re.IGNORECASE)
|
||||||
|
|
||||||
|
q_re = re.compile(
|
||||||
|
rf"^(?P<value>{_VALUE})(?:\s+(?P<unit>{_UNIT}))?(?:\s+of\s+(?P<substance>.+))?$",
|
||||||
|
flags=re.IGNORECASE
|
||||||
|
)
|
||||||
|
|
||||||
|
all_matched = True
|
||||||
|
candidates_temp = []
|
||||||
|
for p in parts:
|
||||||
|
p = p.strip()
|
||||||
|
mq = q_re.match(p)
|
||||||
|
if mq is not None:
|
||||||
|
value_raw = mq.group("value")
|
||||||
if not _is_indefinite_quantifier(value_raw):
|
if not _is_indefinite_quantifier(value_raw):
|
||||||
entity = _normalize_entity(m.group("entity"))
|
try:
|
||||||
value = _resolve_value(value_raw)
|
val, curr_unit = _resolve_currency_and_value(value_raw)
|
||||||
unit_raw = m.group("unit")
|
unit_raw = mq.group("unit")
|
||||||
unit = _canonicalize_unit(unit_raw)
|
substance = mq.group("substance")
|
||||||
out.append(
|
if unit_raw is not None and substance is not None:
|
||||||
|
unit_raw = f"{unit_raw} of {substance}"
|
||||||
|
elif unit_raw is None and substance is not None:
|
||||||
|
unit_raw = substance.strip()
|
||||||
|
lowered_sub = unit_raw.lower()
|
||||||
|
for art in ("a ", "an ", "the "):
|
||||||
|
if lowered_sub.startswith(art):
|
||||||
|
unit_raw = unit_raw[len(art):].strip()
|
||||||
|
break
|
||||||
|
unit = _compose_unit(curr_unit, unit_raw)
|
||||||
|
if unit is None:
|
||||||
|
all_matched = False
|
||||||
|
break
|
||||||
|
candidates_temp.append(
|
||||||
CandidateInitial(
|
CandidateInitial(
|
||||||
initial=InitialPossession(
|
initial=InitialPossession(
|
||||||
entity=entity,
|
entity=entity,
|
||||||
quantity=Quantity(value=value, unit=unit),
|
quantity=Quantity(value=val, unit=unit),
|
||||||
),
|
),
|
||||||
source_span=sentence,
|
source_span=sentence,
|
||||||
matched_anchor=m.group("anchor"),
|
matched_anchor=anchor,
|
||||||
matched_value_token=value_raw,
|
matched_value_token=value_raw,
|
||||||
matched_unit_token=unit_raw,
|
matched_unit_token=unit_raw if unit_raw is not None else "",
|
||||||
matched_entity_token=m.group("entity"),
|
matched_entity_token=entity_raw,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
except ValueError:
|
||||||
|
all_matched = False
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
all_matched = False
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
all_matched = False
|
||||||
|
break
|
||||||
|
|
||||||
|
if all_matched and candidates_temp:
|
||||||
|
out.extend(candidates_temp)
|
||||||
|
return out
|
||||||
|
|
||||||
m2 = _INITIAL_THERE_ARE_RE.match(s)
|
m2 = _INITIAL_THERE_ARE_RE.match(s)
|
||||||
if m2 is not None:
|
if m2 is not None:
|
||||||
value_raw = m2.group("value")
|
value_raw = m2.group("value")
|
||||||
if not _is_indefinite_quantifier(value_raw):
|
if not _is_indefinite_quantifier(value_raw):
|
||||||
|
try:
|
||||||
|
val, curr_unit = _resolve_currency_and_value(value_raw)
|
||||||
unit_raw = m2.group("unit")
|
unit_raw = m2.group("unit")
|
||||||
unit = _canonicalize_unit(unit_raw)
|
substance = m2.group("substance") if "substance" in m2.groupdict() else None
|
||||||
value = _resolve_value(value_raw)
|
if unit_raw is not None and substance is not None:
|
||||||
|
unit_raw = f"{unit_raw} of {substance}"
|
||||||
|
elif unit_raw is None and substance is not None:
|
||||||
|
unit_raw = substance.strip()
|
||||||
|
lowered_sub = unit_raw.lower()
|
||||||
|
for art in ("a ", "an ", "the "):
|
||||||
|
if lowered_sub.startswith(art):
|
||||||
|
unit_raw = unit_raw[len(art):].strip()
|
||||||
|
break
|
||||||
|
unit = _compose_unit(curr_unit, unit_raw)
|
||||||
|
if unit is not None:
|
||||||
place = m2.group("place")
|
place = m2.group("place")
|
||||||
# When a 'in <place>' phrase is present, treat the place as
|
|
||||||
# the implicit entity. Otherwise use the unit's plural as
|
|
||||||
# the collective entity name (deterministic, derivable from
|
|
||||||
# the source: "There are 5 kids" -> entity='kids').
|
|
||||||
if place is not None:
|
if place is not None:
|
||||||
entity = _normalize_entity(place)
|
entity = _normalize_entity(place)
|
||||||
entity_token = place
|
entity_token = place
|
||||||
|
|
@ -238,15 +418,17 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
|
||||||
CandidateInitial(
|
CandidateInitial(
|
||||||
initial=InitialPossession(
|
initial=InitialPossession(
|
||||||
entity=entity,
|
entity=entity,
|
||||||
quantity=Quantity(value=value, unit=unit),
|
quantity=Quantity(value=val, unit=unit),
|
||||||
),
|
),
|
||||||
source_span=sentence,
|
source_span=sentence,
|
||||||
matched_anchor=m2.group("anchor"),
|
matched_anchor=m2.group("anchor"),
|
||||||
matched_value_token=value_raw,
|
matched_value_token=value_raw,
|
||||||
matched_unit_token=unit_raw,
|
matched_unit_token=unit_raw if unit_raw is not None else "",
|
||||||
matched_entity_token=entity_token,
|
matched_entity_token=entity_token,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
@ -255,14 +437,6 @@ def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
|
||||||
# Operation candidate extractor
|
# Operation candidate extractor
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
# Per-kind operation patterns. Each captures: subject, verb, value, unit,
|
|
||||||
# optional target. The verb alternation is the kind's permissive verb table.
|
|
||||||
#
|
|
||||||
# Note: optional unit (?P<unit>) is allowed because some constructions
|
|
||||||
# rely on inherited unit ("Sam doubles his savings"); however for P2's
|
|
||||||
# scope we only emit candidates when the unit token is explicit. Inherited-
|
|
||||||
# unit candidates require per-branch state and are added in P3.
|
|
||||||
|
|
||||||
def _op_pattern(verbs_pattern: str, *, requires_target: bool) -> re.Pattern[str]:
|
def _op_pattern(verbs_pattern: str, *, requires_target: bool) -> re.Pattern[str]:
|
||||||
"""Build the per-kind operation regex.
|
"""Build the per-kind operation regex.
|
||||||
|
|
||||||
|
|
@ -284,10 +458,6 @@ def _op_pattern(verbs_pattern: str, *, requires_target: bool) -> re.Pattern[str]
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
target_part = ""
|
target_part = ""
|
||||||
# 'to' is included in the discardable preposition set.
|
|
||||||
# 'of' is included for ADR-0127 substance qualifiers ("1000 feet
|
|
||||||
# of cable") — the substance NP is grammatically real but
|
|
||||||
# arithmetically inert; the unit slot carries the dimensional info.
|
|
||||||
trailing_prep = (
|
trailing_prep = (
|
||||||
r"(?:\s+(?:on|from|at|in|onto|into|under|over|to|of|for|with)\s+.+)?"
|
r"(?:\s+(?:on|from|at|in|onto|into|under|over|to|of|for|with)\s+.+)?"
|
||||||
)
|
)
|
||||||
|
|
@ -297,8 +467,7 @@ def _op_pattern(verbs_pattern: str, *, requires_target: bool) -> re.Pattern[str]
|
||||||
rf"(?P<verb>{verbs_pattern})"
|
rf"(?P<verb>{verbs_pattern})"
|
||||||
rf"\s+(?P<value>{_VALUE})"
|
rf"\s+(?P<value>{_VALUE})"
|
||||||
r"(?:\s+more)?"
|
r"(?:\s+more)?"
|
||||||
r"(?:\s+(?!to\b)(?!more\b)(?!on\b)(?!from\b)(?!at\b)(?!in\b)"
|
r"(?:\s+(?P<unit>" + _UNIT + r"))?"
|
||||||
r"(?P<unit>\w+))?"
|
|
||||||
rf"{target_part}"
|
rf"{target_part}"
|
||||||
rf"{trailing_prep}"
|
rf"{trailing_prep}"
|
||||||
r"\s*\.?$",
|
r"\s*\.?$",
|
||||||
|
|
@ -339,12 +508,19 @@ def _build_op_candidate(
|
||||||
the match lacks a required slot (e.g. unit token absent — P2 does
|
the match lacks a required slot (e.g. unit token absent — P2 does
|
||||||
not emit unit-inherited candidates)."""
|
not emit unit-inherited candidates)."""
|
||||||
unit_raw = m.group("unit")
|
unit_raw = m.group("unit")
|
||||||
if unit_raw is None:
|
value_raw = m.group("value")
|
||||||
|
|
||||||
|
try:
|
||||||
|
value, curr_unit = _resolve_currency_and_value(value_raw)
|
||||||
|
except ValueError:
|
||||||
return None
|
return None
|
||||||
unit = _canonicalize_unit(unit_raw)
|
|
||||||
|
unit = _compose_unit(curr_unit, unit_raw)
|
||||||
|
if unit is None:
|
||||||
|
return None
|
||||||
|
|
||||||
subject = _normalize_entity(m.group("subject"))
|
subject = _normalize_entity(m.group("subject"))
|
||||||
verb = m.group("verb").lower()
|
verb = m.group("verb").lower()
|
||||||
value = _resolve_value(m.group("value"))
|
|
||||||
target_raw = m.group("target") if "target" in m.groupdict() else None
|
target_raw = m.group("target") if "target" in m.groupdict() else None
|
||||||
target = target_raw if target_raw is not None else None
|
target = target_raw if target_raw is not None else None
|
||||||
|
|
||||||
|
|
@ -365,8 +541,8 @@ def _build_op_candidate(
|
||||||
op=Operation(**op_kwargs), # type: ignore[arg-type]
|
op=Operation(**op_kwargs), # type: ignore[arg-type]
|
||||||
source_span=source,
|
source_span=source,
|
||||||
matched_verb=verb,
|
matched_verb=verb,
|
||||||
matched_value_token=m.group("value"),
|
matched_value_token=value_raw,
|
||||||
matched_unit_token=unit_raw,
|
matched_unit_token=unit_raw if unit_raw is not None else "",
|
||||||
matched_actor_token=m.group("subject"),
|
matched_actor_token=m.group("subject"),
|
||||||
matched_target_token=target,
|
matched_target_token=target,
|
||||||
)
|
)
|
||||||
|
|
@ -398,14 +574,14 @@ class CandidateUnknown:
|
||||||
|
|
||||||
|
|
||||||
_Q_ENTITY_RE: Final[re.Pattern[str]] = re.compile(
|
_Q_ENTITY_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
r"^How\s+many\s+(?P<unit>\w+)\s+(?:does|do)\s+"
|
r"^How\s+many\s+(?P<unit>" + _UNIT + r")\s+(?:does|do)\s+"
|
||||||
rf"(?P<entity>{_ENTITY})"
|
rf"(?P<entity>{_ENTITY})"
|
||||||
r"\s+have(?:\s+(?:left|now|in\s+total|altogether)){0,2}\s*\??$",
|
r"\s+have(?:\s+(?:left|now|in\s+total|altogether)){0,2}\s*\??$",
|
||||||
flags=re.IGNORECASE,
|
flags=re.IGNORECASE,
|
||||||
)
|
)
|
||||||
|
|
||||||
_Q_TOTAL_RE: Final[re.Pattern[str]] = re.compile(
|
_Q_TOTAL_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
r"^How\s+many\s+(?P<unit>\w+)\s+do\s+they\s+have"
|
r"^How\s+many\s+(?P<unit>" + _UNIT + r")\s+do\s+they\s+have"
|
||||||
r"(?:\s+(?:in\s+total|altogether|left|now)){0,2}\s*\??$",
|
r"(?:\s+(?:in\s+total|altogether|left|now)){0,2}\s*\??$",
|
||||||
flags=re.IGNORECASE,
|
flags=re.IGNORECASE,
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -264,7 +264,17 @@ _WORD_RE: Final[re.Pattern[str]] = re.compile(r"\b\w+\b", flags=re.UNICODE)
|
||||||
|
|
||||||
def _tokens(text: str) -> frozenset[str]:
|
def _tokens(text: str) -> frozenset[str]:
|
||||||
"""Lowercased word-token set for word-boundary containment checks."""
|
"""Lowercased word-token set for word-boundary containment checks."""
|
||||||
return frozenset(m.group(0).lower() for m in _WORD_RE.finditer(text))
|
words = set(m.group(0).lower() for m in _WORD_RE.finditer(text))
|
||||||
|
# Support currency symbol to unit grounding mapping
|
||||||
|
if "$" in text:
|
||||||
|
words.update({"dollar", "dollars"})
|
||||||
|
if "¢" in text:
|
||||||
|
words.update({"cent", "cents"})
|
||||||
|
if "€" in text:
|
||||||
|
words.update({"euro", "euros"})
|
||||||
|
if "£" in text:
|
||||||
|
words.update({"pound", "pounds", "sterling"})
|
||||||
|
return frozenset(words)
|
||||||
|
|
||||||
|
|
||||||
def _token_in(needle: str, haystack_tokens: frozenset[str]) -> bool:
|
def _token_in(needle: str, haystack_tokens: frozenset[str]) -> bool:
|
||||||
|
|
@ -272,7 +282,7 @@ def _token_in(needle: str, haystack_tokens: frozenset[str]) -> bool:
|
||||||
return needle.lower() in haystack_tokens
|
return needle.lower() in haystack_tokens
|
||||||
|
|
||||||
|
|
||||||
def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
|
def _value_grounds(value_token: str, haystack_tokens: frozenset[str], source_span: str = "") -> bool:
|
||||||
"""A numeric value grounds if its surface token appears, OR if the token
|
"""A numeric value grounds if its surface token appears, OR if the token
|
||||||
is a digit-string and any equivalent word-form appears, OR if it's a
|
is a digit-string and any equivalent word-form appears, OR if it's a
|
||||||
word-form and the digit appears.
|
word-form and the digit appears.
|
||||||
|
|
@ -283,9 +293,18 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
|
||||||
hard-coded WORD_NUMBERS remains as a fast path and as a fallback if
|
hard-coded WORD_NUMBERS remains as a fast path and as a fallback if
|
||||||
the pack is unavailable; the pack adds, never replaces.
|
the pack is unavailable; the pack adds, never replaces.
|
||||||
"""
|
"""
|
||||||
if _token_in(value_token, haystack_tokens):
|
clean_token = re.sub(r"^[\$\u20ac\u00a3\u00a5\u20b1\u00a2]", "", value_token).strip()
|
||||||
|
if not clean_token:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Check substring containment in source_span first (handles 18.00 and 3/4)
|
||||||
|
if source_span:
|
||||||
|
if clean_token.lower() in source_span.lower() or value_token.lower() in source_span.lower():
|
||||||
return True
|
return True
|
||||||
lowered = value_token.lower()
|
|
||||||
|
if _token_in(clean_token, haystack_tokens) or _token_in(value_token, haystack_tokens):
|
||||||
|
return True
|
||||||
|
lowered = clean_token.lower()
|
||||||
|
|
||||||
# Pack-backed cardinal lookup (ADR-0128). Soft import — if the pack
|
# Pack-backed cardinal lookup (ADR-0128). Soft import — if the pack
|
||||||
# isn't mounted (e.g., in legacy test environments) we silently fall
|
# isn't mounted (e.g., in legacy test environments) we silently fall
|
||||||
|
|
@ -295,7 +314,7 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
|
||||||
entry = lookup_cardinal(lowered)
|
entry = lookup_cardinal(lowered)
|
||||||
if entry is not None:
|
if entry is not None:
|
||||||
digit = str(entry.numeric_value)
|
digit = str(entry.numeric_value)
|
||||||
if digit in haystack_tokens:
|
if digit in haystack_tokens or (source_span and digit in source_span):
|
||||||
return True
|
return True
|
||||||
except Exception:
|
except Exception:
|
||||||
pass # fall through to hard-coded path
|
pass # fall through to hard-coded path
|
||||||
|
|
@ -303,15 +322,15 @@ def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
|
||||||
# word -> digit equivalent (legacy)
|
# word -> digit equivalent (legacy)
|
||||||
if lowered in WORD_NUMBERS:
|
if lowered in WORD_NUMBERS:
|
||||||
digit = str(WORD_NUMBERS[lowered])
|
digit = str(WORD_NUMBERS[lowered])
|
||||||
if digit in haystack_tokens:
|
if digit in haystack_tokens or (source_span and digit in source_span):
|
||||||
return True
|
return True
|
||||||
# digit -> any word with that integer value (legacy)
|
# digit -> any word with that integer value (legacy)
|
||||||
try:
|
try:
|
||||||
n = int(value_token)
|
n = int(float(clean_token))
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return False
|
return False
|
||||||
for word, w_val in WORD_NUMBERS.items():
|
for word, w_val in WORD_NUMBERS.items():
|
||||||
if w_val == n and word in haystack_tokens:
|
if w_val == n and (word in haystack_tokens or (source_span and word in source_span.lower())):
|
||||||
return True
|
return True
|
||||||
# Pack-backed reverse lookup: digit -> cardinal surface in haystack
|
# Pack-backed reverse lookup: digit -> cardinal surface in haystack
|
||||||
try:
|
try:
|
||||||
|
|
@ -359,18 +378,30 @@ def roundtrip_admissible(c: CandidateOperation) -> bool:
|
||||||
# the anchor itself as the value token and pass via step (2).
|
# the anchor itself as the value token and pass via step (2).
|
||||||
if c.op.kind == "compare_multiplicative" and c.matched_value_token == c.matched_verb:
|
if c.op.kind == "compare_multiplicative" and c.matched_value_token == c.matched_verb:
|
||||||
pass # anchor already grounded by verb check
|
pass # anchor already grounded by verb check
|
||||||
elif not _value_grounds(c.matched_value_token, haystack):
|
elif not _value_grounds(c.matched_value_token, haystack, c.source_span):
|
||||||
return False
|
return False
|
||||||
|
|
||||||
# 5. Unit must ground when non-empty. Empty unit token is only valid
|
# 5. Unit must ground when non-empty. Empty unit token is only valid
|
||||||
# for comparison operands without explicit unit phrasing
|
# for comparison operands without explicit unit phrasing
|
||||||
# ("Sam has twice as many as Tom").
|
# ("Sam has twice as many as Tom").
|
||||||
if c.matched_unit_token:
|
if c.matched_unit_token:
|
||||||
if not _token_in(c.matched_unit_token, haystack):
|
# Check if the matched unit token is in haystack, or contains parts that are in haystack.
|
||||||
|
# For multi-word unit tokens like "an hour", we split and verify.
|
||||||
|
parts = re.split(r'[- ]', c.matched_unit_token)
|
||||||
|
for part in parts:
|
||||||
|
part = part.strip()
|
||||||
|
if part and not _token_in(part, haystack):
|
||||||
|
# Also allow currency symbol match if the part is "dollar" or "dollars" and "$" is present
|
||||||
|
if part in ("dollar", "dollars") and "$" in c.source_span:
|
||||||
|
continue
|
||||||
|
if part in ("cent", "cents") and "¢" in c.source_span:
|
||||||
|
continue
|
||||||
return False
|
return False
|
||||||
else:
|
else:
|
||||||
if not isinstance(c.op.operand, Comparison):
|
if not isinstance(c.op.operand, Comparison):
|
||||||
return False # only comparisons may have empty unit token
|
has_currency = any(sym in c.source_span for sym in ("$", "€", "£", "¥", "₱", "¢"))
|
||||||
|
if not has_currency:
|
||||||
|
return False # only comparisons or currency operations may have empty unit token
|
||||||
|
|
||||||
# 6. Transfer target must appear.
|
# 6. Transfer target must appear.
|
||||||
if c.matched_target_token is not None:
|
if c.matched_target_token is not None:
|
||||||
|
|
|
||||||
76
tests/test_adr_0131_G3_numerics.py
Normal file
76
tests/test_adr_0131_G3_numerics.py
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from generate.math_candidate_parser import (
|
||||||
|
extract_initial_candidates,
|
||||||
|
_resolve_currency_and_value,
|
||||||
|
)
|
||||||
|
from generate.math_candidate_graph import parse_and_solve
|
||||||
|
from evals.math_capability_axes.G3_numerics.v1.runner import build_report, load_cases
|
||||||
|
|
||||||
|
_HERE = Path(__file__).resolve().parent
|
||||||
|
_REPO_ROOT = _HERE.parent
|
||||||
|
_CASES_PATH = _REPO_ROOT / "evals" / "math_capability_axes" / "G3_numerics" / "v1" / "cases.jsonl"
|
||||||
|
|
||||||
|
|
||||||
|
def test_money_literal_parsing() -> None:
|
||||||
|
s = "Tina has $18.00."
|
||||||
|
candidates = extract_initial_candidates(s)
|
||||||
|
assert len(candidates) == 1
|
||||||
|
assert candidates[0].initial.quantity.value == 18.0
|
||||||
|
assert candidates[0].initial.quantity.unit == "dollars"
|
||||||
|
|
||||||
|
|
||||||
|
def test_fraction_literal_parsing() -> None:
|
||||||
|
s = "Jan has 3/4 of a cake."
|
||||||
|
candidates = extract_initial_candidates(s)
|
||||||
|
assert len(candidates) == 1
|
||||||
|
assert candidates[0].initial.quantity.value == 0.75
|
||||||
|
assert candidates[0].initial.quantity.unit == "cakes"
|
||||||
|
|
||||||
|
|
||||||
|
def test_word_number_composition_parsing() -> None:
|
||||||
|
s = "Francine has five full boxes of crayons."
|
||||||
|
candidates = extract_initial_candidates(s)
|
||||||
|
assert len(candidates) == 1
|
||||||
|
assert candidates[0].initial.quantity.value == 5.0
|
||||||
|
assert candidates[0].initial.quantity.unit == "full boxes of crayons"
|
||||||
|
|
||||||
|
|
||||||
|
def test_hyphenated_compound_parsing() -> None:
|
||||||
|
s = "Allison has 10 one-hour videos."
|
||||||
|
candidates = extract_initial_candidates(s)
|
||||||
|
assert len(candidates) == 1
|
||||||
|
assert candidates[0].initial.quantity.value == 10.0
|
||||||
|
assert candidates[0].initial.quantity.unit == "one-hour videos"
|
||||||
|
|
||||||
|
|
||||||
|
def test_refusal_probes() -> None:
|
||||||
|
with pytest.raises(ValueError, match="Too many decimal places"):
|
||||||
|
_resolve_currency_and_value("$18.0000")
|
||||||
|
|
||||||
|
res = parse_and_solve("Jan has 1/0 of a cake. How many cake does Jan have?")
|
||||||
|
assert res.answer is None
|
||||||
|
|
||||||
|
res = parse_and_solve("Sam has one-hour-old baby. How many babies does Sam have?")
|
||||||
|
assert res.answer is None
|
||||||
|
|
||||||
|
res = parse_and_solve("Marc has 10% of a pizza. How many pizzas does Marc have?")
|
||||||
|
assert res.answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_runner_and_report_invariants() -> None:
|
||||||
|
cases = load_cases(_CASES_PATH)
|
||||||
|
report = build_report(cases)
|
||||||
|
|
||||||
|
assert report["metrics"]["wrong"] == 0
|
||||||
|
assert report["metrics"]["overall_pass"] is True
|
||||||
|
|
||||||
|
r1 = build_report(cases)
|
||||||
|
r2 = build_report(cases)
|
||||||
|
s1 = json.dumps(r1, sort_keys=True, separators=(",", ":"))
|
||||||
|
s2 = json.dumps(r2, sort_keys=True, separators=(",", ":"))
|
||||||
|
assert s1 == s2
|
||||||
Loading…
Reference in a new issue