From b891eb243c20fb4905a438ae787374d3844cbc35 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 23 May 2026 14:15:25 -0700 Subject: [PATCH] =?UTF-8?q?feat(ADR-0131.G.2):=20comparative=20operations?= =?UTF-8?q?=20(additive=20+=20multiplicative)=20=E2=80=94=20admission=20un?= =?UTF-8?q?changed,=20comparative-clause=20refusals=202=E2=86=921?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire compare_additive / compare_multiplicative extractors into the candidate-emitting sentence parser, closing the deferred phase flagged at generate/math_candidate_parser.py:30. Capability axis: comparatives (additive + multiplicative) - generate/math_candidate_parser.py: new _compare_additive_candidates, _compare_multiplicative_candidates, _compare_nested_candidates emitting CandidateOperation records keyed to the four Comparison.direction literals registered in ADR-0123. - Closed-set anchor alternation; 'less' admitted as surface synonym of 'fewer'; reference slot widened to admit "the number/amount of " for nested forms. - Nested 'A has N more than M times ' emits two flat candidates (additive + multiplicative); binding-graph picks the admissible composition or refuses (no solver stub). Curated axis lane (24 cases) - evals/math_capability_axes/G2_comparatives/v1/cases.jsonl: 8 additive / 8 multiplicative / 3 nested / 5 refusal - evals/math_capability_axes/G2_comparatives/v1/runner.py + report.json: deterministic, wrong==0 gate, byte-equal across runs. Tests (21 new) - tests/test_adr_0131_G2_comparatives.py: per-direction at-least-one passing, nested-both-emitted, closed-set refusal, runner byte-equality, GSM8K-probe gate (comparative-clause refusals strictly decrease). GSM8K-probe gate (chosen: comparative-clause refusals ↓) - evals/gsm8k_math/train_sample/v1/report.json (candidate-graph probe): comparative-clause refusal count 2 → 1 (case 0009 'Jen has 10 more ducks than four times the number of chickens' moves from statement-clause refusal to question-layer refusal). admitted_wrong remains 0; admission_rate unchanged (downstream composition is a follow-up ADR). - evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json (legacy probe): refreshed, byte-identical (legacy parser untouched). B3 + candidate-graph + GSM8K probe lanes all pass (90/90). Direction vocab stays closed to {more, fewer, times, fraction}; wrong==0 preserved everywhere. --- docs/decisions/ADR-0131.G.2-comparatives.md | 265 +++++++++++++ evals/gsm8k_math/train_sample/v1/report.json | 2 +- .../G2_comparatives/__init__.py | 0 .../G2_comparatives/v1/__init__.py | 0 .../G2_comparatives/v1/cases.jsonl | 24 ++ .../G2_comparatives/v1/report.json | 267 +++++++++++++ .../G2_comparatives/v1/runner.py | 205 ++++++++++ evals/math_capability_axes/__init__.py | 0 generate/math_candidate_parser.py | 362 +++++++++++++++++- tests/test_adr_0131_G2_comparatives.py | 275 +++++++++++++ 10 files changed, 1398 insertions(+), 2 deletions(-) create mode 100644 docs/decisions/ADR-0131.G.2-comparatives.md create mode 100644 evals/math_capability_axes/G2_comparatives/__init__.py create mode 100644 evals/math_capability_axes/G2_comparatives/v1/__init__.py create mode 100644 evals/math_capability_axes/G2_comparatives/v1/cases.jsonl create mode 100644 evals/math_capability_axes/G2_comparatives/v1/report.json create mode 100644 evals/math_capability_axes/G2_comparatives/v1/runner.py create mode 100644 evals/math_capability_axes/__init__.py create mode 100644 tests/test_adr_0131_G2_comparatives.py diff --git a/docs/decisions/ADR-0131.G.2-comparatives.md b/docs/decisions/ADR-0131.G.2-comparatives.md new file mode 100644 index 00000000..beffcf17 --- /dev/null +++ b/docs/decisions/ADR-0131.G.2-comparatives.md @@ -0,0 +1,265 @@ +# ADR-0131.G.2 — Capability axis: comparative operations (additive + multiplicative) + +**Status:** Proposed +**Date:** 2026-05-23 +**Author:** CORE agents + reviewers +**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md) +**Depends on:** +[ADR-0123](./ADR-0123-comparison-substrate.md) (Comparison op shape), +[ADR-0126](./ADR-0126-candidate-graph-parser.md) (candidate-graph topology), +[ADR-0131.3](./ADR-0131.3-bounded-grammar.md), +[ADR-0132](./ADR-0132-binding-graph-data-model.md)..[ADR-0134](./ADR-0134-unit-aware-admissibility.md) + +--- + +## Context + +The `Comparison` operand shape, the four `direction` literals (`more`, +`fewer`, `times`, `fraction`), and the round-trip anchor tables +(`COMPARE_ADDITIVE_ANCHORS`, `COMPARE_MULTIPLICATIVE_ANCHORS`) all +landed with ADR-0123. The legacy first-match-wins parser +(`generate/math_parser.py`) wires those for the bounded-grammar lane. + +The candidate-emitting sentence parser +(`generate/math_candidate_parser.py:30`) explicitly deferred +comparative candidates to a "later phase of ADR-0126". Without that +phase, multi-sentence GSM8K problems that contain a comparative +clause are refused by the candidate-graph topology even when the +comparative is well-formed — the comparative sentence emits zero +candidates, the per-sentence choice space is empty, and the whole +problem refuses with `no admissible candidate for statement: 'X'`. + +This ADR is the deferred comparative phase, framed under ADR-0131.G's +capability-first iteration discipline: comparatives are a single +capability axis (not a list of GSM8K templates), exercised by curated +sentences independent of GSM8K, with the probe's admission / +refused-reasons numbers moving only as a side effect. + +--- + +## Decision + +Land comparative candidate-emitting extractors in +`generate/math_candidate_parser.py`: + +| Function | Closed-set shapes admitted | +|---|---| +| `_compare_additive_candidates` | ` has N more than ` / `... N fewer ...` / `... N less ...` | +| `_compare_multiplicative_candidates` | ` has twice as many as ` / `... thrice ...` / `... half ...` / `... N times as many ...` | +| `_compare_nested_candidates` | ` has N more than M times ` — emits *two* flat candidates (one additive, one multiplicative), letting the binding-graph layer (ADR-0134) pick an admissible composition or refuse | + +Direction vocabulary is **closed** to the four `Comparison.direction` +literals. `less` is admitted as a surface synonym of `fewer`; its +canonical Comparison value is `direction='fewer'` while +`matched_verb='less'` (so the round-trip filter's verb-registry check +finds `less` in `COMPARE_ADDITIVE_ANCHORS`, and its +"verb appears in source" check succeeds). + +The reference slot widens beyond proper-nouns / "the X" to admit +"the number/amount of " mass-noun constructions +(`_resolve_reference_token`), so the GSM8K-style "...four times the +number of chickens" composes. The canonical entity name is the full +noun phrase; binding-graph referential integrity is preserved +(distinct noun phrases stay distinct entities). + +### Specificity / precedence + +Extractor order in `extract_operation_candidates`: + +1. add / subtract / transfer (existing — ADR-0126) +2. **`_compare_nested_candidates`** (most specific — requires both + `(more|fewer|less) ... than` *and* `N times ` tail) +3. **`_compare_multiplicative_candidates`** (anchor-as-value forms + `twice/thrice/half`, then `N times as many`) +4. **`_compare_additive_candidates`** (`N more/fewer/less than + `) + +Multiplicative anchors that overlap lexically with additive are +disjoint at the lexical level: `WORD_NUMBERS` registers `two`/`three` +but **not** `twice`/`thrice`, so the additive VALUE alternation cannot +swallow a multiplicative anchor. Nested matching consumes its +trailing `than N times ` greedily, so a sentence the nested +regex matches cannot also be claimed by the bare additive pattern +(the bare additive regex requires the `than ` tail with `` +matching `_COMPARE_REF`, but the nested regex's `` is preceded +by `N times`, which the bare additive regex's `` slot does not +admit). Documented + tested in +`tests/test_adr_0131_G2_comparatives.py`. + +### Curated coverage cases (G.2 axis lane) + +`evals/math_capability_axes/G2_comparatives/v1/cases.jsonl` — +**24 cases**, exercised by an axis-specific runner +(`evals/math_capability_axes/G2_comparatives/v1/runner.py`) that +emits a deterministic `report.json`: + +| Category | Cases | Direction coverage | +|----------------|-------|----------------------------| +| additive | 8 | `more` ×4, `fewer` ×4 (incl. one `less` surface)| +| multiplicative | 8 | `times` ×4, `fraction` ×4 | +| nested | 3 | additive + multiplicative both flat-emitted | +| refusal | 5 | paraphrases outside the closed set | + +Per-direction at-least-one-passing, nested-both-emitted, refusal +zero-admitted, and `wrong == 0` are pinned by +`tests/test_adr_0131_G2_comparatives.py`. Report is byte-equal across +runs. + +### Deferred (out of scope for G.2) + +Documented refusal classes — these have no entry in +`COMPARE_*_ANCHORS`, so admitting them would breach the round-trip +filter's verb-registry check anyway: + +- `"as many ... as"` without a direction word ("Alice has as many + apples as Bob") — equality is not in + `{more, fewer, times, fraction}`. +- `"compared to , ..."` and `"in comparison with , ..."` + — prefix paraphrases; no anchor verb is consumed. +- `"the same number of ... as"` — equality, see above. +- `"N times more"` — legacy parser already refuses as ambiguous + (ADR-0123 line 797); G.2 preserves that refusal. +- Pronoun resolution in actor / reference slots — needs per-branch + state (ADR-0126 P3 future work). +- Past-tense / lemma widening (`had`, `gets`, `took`, `buys`) — adds + surface variants without growing the capability axis; deferred to a + separate axis to keep precedence narrow. +- Unit ellipsis (`"twice as many as Bob"` without a unit) — without a + unit the binding graph cannot pick a dimension to compare under + ADR-0134's unit-aware admissibility; refusal preserves `wrong == 0`. + +### Nested composition is *parser-side only* + +The parser emits two flat candidates for nested forms; it does **not** +introduce a `Comparison ∋ Comparison` operand shape (which the data +model doesn't support today). The binding-graph layer (ADR-0134) +picks the admissible composition or refuses. **Refusal of nested +forms is the safe outcome and is what we observe today on +gsm8k-train-sample-v1-0009**: the comparative clause now parses (it +no longer refuses at the statement layer), but the downstream +composition has not yet been wired, so the case continues to refuse +— at the question layer instead. That downstream gap is a follow-up +ADR (`compare_additive ∘ compare_multiplicative` graph composition); +this ADR explicitly does **not** stub the solver to bypass it. + +### GSM8K-probe gate (chosen) + +G.2 gates on: + +> **Comparative-clause refusals in the candidate-graph probe report +> (`evals/gsm8k_math/train_sample/v1/report.json`) strictly +> decrease.** + +Test: `test_gsm8k_candidate_graph_comparative_clause_refusals_decreased` +(measures count of refused cases whose reason cites a statement +clause containing either additive `more/fewer/less ... than` or +multiplicative `twice/N times/half as many` anchors). + +| Probe report | Baseline (origin/main 481e0c3) | After G.2 | Δ | +|---|---|---|---| +| Comparative-clause refusals (`report.json`) | 2 | 1 | −1 | +| `admitted_wrong` (`report.json`) | 0 | 0 | 0 | +| `admission_rate` (`report.json`) | 0/50 | 0/50 | 0 | + +The baseline-2 cases are `gsm8k-train-sample-v1-0009` ("Jen has 10 +more ducks than four times the number of chickens.") and +`-0016` ("Rudolph traveled 2 more than 5 miles"). G.2's extractor +parses the comparative clause in 0009; 0016's "2 more than 5" is an +arithmetic expression between numbers, not a between-entity +comparison, and is deliberately out of G.2 scope (the additive shape +requires an entity reference, not a numeric reference). + +`admission_rate` does **not** rise because the downstream +binding-graph composition for nested comparatives + the question +layer's compatibility with comparison-derived state are out of G.2 +scope. The honest claim is that the parser admits one more shape, +not that the engine solves more GSM8K problems. + +### Legacy probe report (refreshed, byte-identical) + +`evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json` +runs through `generate.math_parser.parse_problem` (legacy +first-match-wins), which already supported ADR-0123 comparative +patterns. G.2 does not touch the legacy parser, so the legacy +probe report is byte-identical to the baseline. Refreshed and +pinned via `test_gsm8k_legacy_probe_safety_rail_intact`. + +--- + +## Invariants + +- **`compare_direction_closed_set`** — every emitted comparative + candidate has `op.operand.direction ∈ {more, fewer, times, + fraction}`. Pinned by + `test_direction_literals_closed_set`. +- **`compare_additive_wrong_zero`** / **`compare_multiplicative_wrong_zero`** + — every additive / multiplicative coverage case passes through + `roundtrip_admissible`. Pinned by the per-direction parametric + tests. +- **`nested_emits_both_flat`** — the nested-composition extractor + emits exactly the two flat candidates (additive + multiplicative); + whether either composes is the binding-graph layer's call. Pinned + by `test_nested_emits_both_flat_candidates`. +- **`closed_set_refusal_holds`** — none of the five documented + refusal paraphrases admit any comparative candidate. Pinned by + `test_refusal_cases_emit_no_admitted_comparative`. +- **`gsm8k_safety_rail_intact`** — `admitted_wrong == 0` on both the + legacy and candidate-graph GSM8K probe reports. +- **`gsm8k_comparative_refusal_strictly_decreased`** — chosen G.2 + gate (see above). +- **`report_deterministic`** — `report.json` is byte-equal across two + back-to-back runs. + +--- + +## Acceptance evidence + +- `evals/math_capability_axes/G2_comparatives/v1/runner.py` exits 0 + with `wrong == 0` on all 24 curated cases (8 additive / 8 + multiplicative / 3 nested / 5 refusal). +- `tests/test_adr_0131_G2_comparatives.py` (21 tests): per-direction + at-least-one-passing, nested-both-emitted, refusal-set zero-admit, + closed-set discipline, runner byte-equality, GSM8K-probe gate. +- `evals/gsm8k_math/train_sample/v1/report.json` shows + comparative-clause refusal count drop from 2 → 1. +- `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json` + refreshed and byte-identical (legacy parser untouched). +- B3 lane (`tests/test_adr_0131_3_bounded_grammar_lane.py`) + + candidate-graph runner tests + GSM8K probe tests all pass. + +--- + +## Consequences + +- The candidate-graph topology can finally see comparative + statements as first-class operations. The downstream composition + layer (ADR-0134 unit-aware admissibility + a future "graph composes + nested comparison" ADR) becomes the next honest unblock. +- `gsm8k-train-sample-v1-0009` moves from "refuse at comparative + clause" to "refuse at question layer" — a refusal-class shift, + not an admission. That is the correct, honest classification: the + case is still refused; the reason it's refused has changed. +- Future axes (G.3 rate-verbs, G.4 multi-clause distributive + subjects) inherit the same axis-lane harness shape under + `evals/math_capability_axes/`. +- Capability-first discipline is preserved: no GSM8K-specific + templates were added; the new shapes are exercised by 24 + hand-authored sentences plus whatever GSM8K has incidentally. + +--- + +## Out of scope + +- **Solver changes.** If a comparative parses but doesn't solve, + that's a binding-graph composition gap, deferred to a follow-up + ADR (no solver stubs). +- **Implicit-comparison shapes** ("Alice has the same number of + apples as Bob") — equality is not in + `{more, fewer, times, fraction}`; admitting it requires extending + the closed direction set, which is its own ADR. +- **Probe runner contract.** ADR-0131.G pinned `run_lane` as the + legacy probe's contract; G.2 does not change that. The + candidate-graph probe (`report.json` via `runner.py`) is the + measurement surface that actually moves under G.2. +- **Frontier head-to-head on comparative accuracy.** Deferred until + the composition layer admits at least one nested form end-to-end. diff --git a/evals/gsm8k_math/train_sample/v1/report.json b/evals/gsm8k_math/train_sample/v1/report.json index 04a8c5b2..4d6ea85d 100644 --- a/evals/gsm8k_math/train_sample/v1/report.json +++ b/evals/gsm8k_math/train_sample/v1/report.json @@ -53,7 +53,7 @@ }, { "case_id": "gsm8k-train-sample-v1-0009", - "reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'", + "reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'", "verdict": "refused" }, { diff --git a/evals/math_capability_axes/G2_comparatives/__init__.py b/evals/math_capability_axes/G2_comparatives/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/evals/math_capability_axes/G2_comparatives/v1/__init__.py b/evals/math_capability_axes/G2_comparatives/v1/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl b/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl new file mode 100644 index 00000000..b0557f9c --- /dev/null +++ b/evals/math_capability_axes/G2_comparatives/v1/cases.jsonl @@ -0,0 +1,24 @@ +{"case_id": "G2-add-more-001", "category": "additive", "direction": "more", "sentence": "Alice has 3 more apples than Bob", "expected": {"kind": "compare_additive", "actor": "Alice", "reference": "Bob", "delta_value": 3, "unit": "apples", "matched_verb": "more"}} +{"case_id": "G2-add-more-002", "category": "additive", "direction": "more", "sentence": "James has 7 more marbles than Carter", "expected": {"kind": "compare_additive", "actor": "James", "reference": "Carter", "delta_value": 7, "unit": "marbles", "matched_verb": "more"}} +{"case_id": "G2-add-more-003", "category": "additive", "direction": "more", "sentence": "Tom has four more cookies than Sara", "expected": {"kind": "compare_additive", "actor": "Tom", "reference": "Sara", "delta_value": 4, "unit": "cookies", "matched_verb": "more"}} +{"case_id": "G2-add-more-004", "category": "additive", "direction": "more", "sentence": "Marion has 12 more paperclips than Yun", "expected": {"kind": "compare_additive", "actor": "Marion", "reference": "Yun", "delta_value": 12, "unit": "paperclips", "matched_verb": "more"}} +{"case_id": "G2-add-fewer-001", "category": "additive", "direction": "fewer", "sentence": "Alice has 5 fewer pencils than Bob", "expected": {"kind": "compare_additive", "actor": "Alice", "reference": "Bob", "delta_value": 5, "unit": "pencils", "matched_verb": "fewer"}} +{"case_id": "G2-add-fewer-002", "category": "additive", "direction": "fewer", "sentence": "Carter has two fewer pets than James", "expected": {"kind": "compare_additive", "actor": "Carter", "reference": "James", "delta_value": 2, "unit": "pets", "matched_verb": "fewer"}} +{"case_id": "G2-add-fewer-003", "category": "additive", "direction": "fewer", "sentence": "Dana has 8 less coins than Eli", "expected": {"kind": "compare_additive", "actor": "Dana", "reference": "Eli", "delta_value": 8, "unit": "coins", "matched_verb": "less"}} +{"case_id": "G2-add-fewer-004", "category": "additive", "direction": "fewer", "sentence": "Beth has 3 fewer books than Sam", "expected": {"kind": "compare_additive", "actor": "Beth", "reference": "Sam", "delta_value": 3, "unit": "books", "matched_verb": "fewer"}} +{"case_id": "G2-mul-times-001", "category": "multiplicative", "direction": "times", "sentence": "Alice has twice as many apples as Bob", "expected": {"kind": "compare_multiplicative", "actor": "Alice", "reference": "Bob", "factor": 2.0, "unit": "apples", "matched_verb": "twice"}} +{"case_id": "G2-mul-times-002", "category": "multiplicative", "direction": "times", "sentence": "Brooke has 3 times as many jacks as Sidney", "expected": {"kind": "compare_multiplicative", "actor": "Brooke", "reference": "Sidney", "factor": 3.0, "unit": "jacks", "matched_verb": "times"}} +{"case_id": "G2-mul-times-003", "category": "multiplicative", "direction": "times", "sentence": "Cindy has thrice as many cards as Nicole", "expected": {"kind": "compare_multiplicative", "actor": "Cindy", "reference": "Nicole", "factor": 3.0, "unit": "cards", "matched_verb": "thrice"}} +{"case_id": "G2-mul-times-004", "category": "multiplicative", "direction": "times", "sentence": "Erica has 5 times as many fish as Mark", "expected": {"kind": "compare_multiplicative", "actor": "Erica", "reference": "Mark", "factor": 5.0, "unit": "fish", "matched_verb": "times"}} +{"case_id": "G2-mul-frac-001", "category": "multiplicative", "direction": "fraction", "sentence": "Monica has half as many hours as Sara", "expected": {"kind": "compare_multiplicative", "actor": "Monica", "reference": "Sara", "factor": 0.5, "unit": "hours", "matched_verb": "half"}} +{"case_id": "G2-mul-frac-002", "category": "multiplicative", "direction": "fraction", "sentence": "Rex has half as many cards as Cindy", "expected": {"kind": "compare_multiplicative", "actor": "Rex", "reference": "Cindy", "factor": 0.5, "unit": "cards", "matched_verb": "half"}} +{"case_id": "G2-mul-frac-003", "category": "multiplicative", "direction": "fraction", "sentence": "Anna has half as many stickers as Beth", "expected": {"kind": "compare_multiplicative", "actor": "Anna", "reference": "Beth", "factor": 0.5, "unit": "stickers", "matched_verb": "half"}} +{"case_id": "G2-mul-frac-004", "category": "multiplicative", "direction": "fraction", "sentence": "Tim has half as many toys as Jack", "expected": {"kind": "compare_multiplicative", "actor": "Tim", "reference": "Jack", "factor": 0.5, "unit": "toys", "matched_verb": "half"}} +{"case_id": "G2-nested-001", "category": "nested", "sentence": "Jen has 10 more ducks than four times the number of chickens", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jen", "reference": "the number of chickens", "delta_value": 10, "factor": 4.0, "unit": "ducks"}} +{"case_id": "G2-nested-002", "category": "nested", "sentence": "Jose has 2 more pounds than 2 times the amount of Orlando", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jose", "reference": "the amount of Orlando", "delta_value": 2, "factor": 2.0, "unit": "pounds"}} +{"case_id": "G2-nested-003", "category": "nested", "sentence": "Sara has 5 fewer marbles than 3 times Bob", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Sara", "reference": "Bob", "delta_value": 5, "factor": 3.0, "unit": "marbles"}} +{"case_id": "G2-refuse-001", "category": "refusal", "sentence": "Alice has as many apples as Bob", "expected": {"refuse": true, "reason": "implicit-comparison 'as many ... as' without a direction anchor — out of closed set"}} +{"case_id": "G2-refuse-002", "category": "refusal", "sentence": "Compared to Bob, Alice has 3 more apples", "expected": {"refuse": true, "reason": "'compared to' prefix — paraphrase outside the closed anchor set"}} +{"case_id": "G2-refuse-003", "category": "refusal", "sentence": "In comparison with Bob, Alice has 3 more apples", "expected": {"refuse": true, "reason": "'in comparison with' prefix — paraphrase outside the closed anchor set"}} +{"case_id": "G2-refuse-004", "category": "refusal", "sentence": "Alice has the same number of apples as Bob", "expected": {"refuse": true, "reason": "'the same number ... as' — equality is not in {more, fewer, times, fraction}"}} +{"case_id": "G2-refuse-005", "category": "refusal", "sentence": "Alice has 3 times more apples than Bob", "expected": {"refuse": true, "reason": "ambiguous 'N times more' — admitted to neither additive nor multiplicative shape (cf. ADR-0123 legacy refusal)"}} diff --git a/evals/math_capability_axes/G2_comparatives/v1/report.json b/evals/math_capability_axes/G2_comparatives/v1/report.json new file mode 100644 index 00000000..f42dbe73 --- /dev/null +++ b/evals/math_capability_axes/G2_comparatives/v1/report.json @@ -0,0 +1,267 @@ +{ + "adr": "0131.G.2", + "axis": "comparatives", + "cases_path": "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl", + "metrics": { + "cases_total": 24, + "pass_rate": 1.0, + "passed": 24, + "wrong": 0, + "wrong_count_is_zero": true, + "wrong_rate": 0.0 + }, + "per_case": [ + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-more-001", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-more-002", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-more-003", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-more-004", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-fewer-001", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-fewer-002", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-fewer-003", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_additive" + ], + "case_id": "G2-add-fewer-004", + "category": "additive", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-times-001", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-times-002", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-times-003", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-times-004", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-001", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-002", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-003", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 1, + "admitted_kinds": [ + "compare_multiplicative" + ], + "case_id": "G2-mul-frac-004", + "category": "multiplicative", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 2, + "admitted_kinds": [ + "compare_additive", + "compare_multiplicative" + ], + "case_id": "G2-nested-001", + "category": "nested", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 2, + "admitted_kinds": [ + "compare_additive", + "compare_multiplicative" + ], + "case_id": "G2-nested-002", + "category": "nested", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 2, + "admitted_kinds": [ + "compare_additive", + "compare_multiplicative" + ], + "case_id": "G2-nested-003", + "category": "nested", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-001", + "category": "refusal", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-002", + "category": "refusal", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-003", + "category": "refusal", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-004", + "category": "refusal", + "outcome": "pass", + "reason": "" + }, + { + "admitted_count": 0, + "admitted_kinds": [], + "case_id": "G2-refuse-005", + "category": "refusal", + "outcome": "pass", + "reason": "" + } + ], + "per_category": { + "additive": { + "pass": 8, + "wrong": 0 + }, + "multiplicative": { + "pass": 8, + "wrong": 0 + }, + "nested": { + "pass": 3, + "wrong": 0 + }, + "refusal": { + "pass": 5, + "wrong": 0 + } + }, + "schema_version": 1 +} diff --git a/evals/math_capability_axes/G2_comparatives/v1/runner.py b/evals/math_capability_axes/G2_comparatives/v1/runner.py new file mode 100644 index 00000000..f831d781 --- /dev/null +++ b/evals/math_capability_axes/G2_comparatives/v1/runner.py @@ -0,0 +1,205 @@ +"""ADR-0131.G.2 — Capability axis runner for comparative operations. + +Exercises the ``compare_additive`` / ``compare_multiplicative`` extractors +in :mod:`generate.math_candidate_parser` against curated coverage cases +that are independent of GSM8K. + +Per-case classification: + +| Case category | pass criterion | +|-----------------|---------------------------------------------------------| +| additive | ≥1 admitted candidate with kind=compare_additive whose | +| | direction, actor, reference, delta_value, unit match | +| multiplicative | ≥1 admitted candidate with kind=compare_multiplicative | +| | whose direction, actor, reference, factor, unit match | +| nested | both compare_additive AND compare_multiplicative flat | +| | candidates emitted + admitted (binding-graph picks) | +| refusal | zero admitted comparative candidates | + +``wrong`` is non-zero only if a refusal-case emits an admitted comparative +or a positive-case emits a comparative with the wrong shape. ``wrong == 0`` +is the load-bearing gate (ADR-0114a Obligation #4). + +Determinism: case order in ``cases.jsonl`` is the report order; same +input file → byte-equal report. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from generate.math_candidate_parser import extract_operation_candidates +from generate.math_roundtrip import roundtrip_admissible + +_HERE = Path(__file__).resolve().parent +_CASES_PATH = _HERE / "cases.jsonl" +_REPORT_PATH = _HERE / "report.json" + + +def _load_cases() -> list[dict[str, Any]]: + return [ + json.loads(line) + for line in _CASES_PATH.read_text(encoding="utf-8").splitlines() + if line.strip() + ] + + +def _admitted(sentence: str) -> list[Any]: + cands = extract_operation_candidates(sentence) + return [ + c for c in cands + if c.op.kind in ("compare_additive", "compare_multiplicative") + and roundtrip_admissible(c) + ] + + +def _score_positive_additive(case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]: + exp = case["expected"] + matches = [ + c for c in admitted + if c.op.kind == "compare_additive" + and c.op.operand.direction == case["direction"] + and c.op.actor == exp["actor"] + and c.op.operand.reference_actor == exp["reference"] + and c.op.operand.delta is not None + and c.op.operand.delta.value == exp["delta_value"] + and c.op.operand.delta.unit == exp["unit"] + and c.matched_verb == exp["matched_verb"] + ] + if matches: + return "pass", "" + if not admitted: + return "wrong", "no admissible comparative candidate emitted" + return ( + "wrong", + f"admitted comparative did not match expected additive shape; got {[(c.op.kind, c.op.operand.as_json()) for c in admitted]}", + ) + + +def _score_positive_multiplicative(case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]: + exp = case["expected"] + matches = [ + c for c in admitted + if c.op.kind == "compare_multiplicative" + and c.op.operand.direction == case["direction"] + and c.op.actor == exp["actor"] + and c.op.operand.reference_actor == exp["reference"] + and c.op.operand.factor is not None + and float(c.op.operand.factor) == float(exp["factor"]) + and c.matched_verb == exp["matched_verb"] + ] + if matches: + return "pass", "" + if not admitted: + return "wrong", "no admissible comparative candidate emitted" + return ( + "wrong", + f"admitted comparative did not match expected multiplicative shape; got {[(c.op.kind, c.op.operand.as_json()) for c in admitted]}", + ) + + +def _score_nested(_case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]: + kinds = {c.op.kind for c in admitted} + if kinds == {"compare_additive", "compare_multiplicative"}: + return "pass", "" + return ( + "wrong", + f"nested case must emit both flat candidate kinds; got {sorted(kinds)}", + ) + + +def _score_refusal(_case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]: + if not admitted: + return "pass", "" + return ( + "wrong", + f"refusal case admitted a comparative candidate ({[c.op.kind for c in admitted]}); " + f"closed-set boundary breached", + ) + + +def _score_case(case: dict[str, Any]) -> dict[str, Any]: + sentence = case["sentence"] + admitted = _admitted(sentence) + category = case["category"] + if category == "additive": + outcome, reason = _score_positive_additive(case, admitted) + elif category == "multiplicative": + outcome, reason = _score_positive_multiplicative(case, admitted) + elif category == "nested": + outcome, reason = _score_nested(case, admitted) + elif category == "refusal": + outcome, reason = _score_refusal(case, admitted) + else: + outcome, reason = "wrong", f"unknown case category {category!r}" + return { + "case_id": case["case_id"], + "category": category, + "outcome": outcome, + "reason": reason, + "admitted_count": len(admitted), + "admitted_kinds": sorted({c.op.kind for c in admitted}), + } + + +def build_report() -> dict[str, Any]: + cases = _load_cases() + per_case = [_score_case(c) for c in cases] + total = len(per_case) + passed = sum(1 for d in per_case if d["outcome"] == "pass") + wrong = sum(1 for d in per_case if d["outcome"] == "wrong") + by_category: dict[str, dict[str, int]] = {} + for d in per_case: + slot = by_category.setdefault(d["category"], {"passed": 0, "wrong": 0}) + slot[d["outcome"] if d["outcome"] == "passed" else d["outcome"]] = ( + slot.get(d["outcome"], 0) + 1 + ) + # Rebuild by_category with normalized keys. + by_category = {} + for d in per_case: + slot = by_category.setdefault(d["category"], {"pass": 0, "wrong": 0}) + slot[d["outcome"]] = slot.get(d["outcome"], 0) + 1 + return { + "schema_version": 1, + "adr": "0131.G.2", + "axis": "comparatives", + "cases_path": "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl", + "metrics": { + "cases_total": total, + "passed": passed, + "wrong": wrong, + "pass_rate": (passed / total) if total else 0.0, + "wrong_rate": (wrong / total) if total else 0.0, + "wrong_count_is_zero": wrong == 0, + }, + "per_category": { + k: dict(sorted(v.items())) for k, v in sorted(by_category.items()) + }, + "per_case": per_case, + } + + +def write_report(report: dict[str, Any]) -> None: + _REPORT_PATH.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def main() -> int: + report = build_report() + write_report(report) + m = report["metrics"] + print( + f"ADR-0131.G.2 comparatives: passed {m['passed']}/{m['cases_total']} " + f"({m['pass_rate']:.1%}); wrong={m['wrong']} (gate: must be 0)" + ) + for cat, counts in report["per_category"].items(): + print(f" {cat:14s} {counts}") + return 0 if m["wrong_count_is_zero"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/math_capability_axes/__init__.py b/evals/math_capability_axes/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/generate/math_candidate_parser.py b/generate/math_candidate_parser.py index f2835ca0..342562e3 100644 --- a/generate/math_candidate_parser.py +++ b/generate/math_candidate_parser.py @@ -36,9 +36,10 @@ from __future__ import annotations import re from dataclasses import dataclass -from typing import Final +from typing import Final, Literal, cast from generate.math_problem_graph import ( + Comparison, InitialPossession, Operation, Quantity, @@ -53,6 +54,11 @@ from generate.math_roundtrip import ( ) +# Locally re-typed alias mirroring Comparison.direction's literal slot — +# used only to satisfy pyright when narrowing surface-direction strings. +_CompDirection = Literal["more", "fewer", "times", "fraction"] + + # --------------------------------------------------------------------------- # Initial-possession candidate # --------------------------------------------------------------------------- @@ -483,4 +489,358 @@ def extract_operation_candidates(sentence: str) -> list[CandidateOperation]: if candidate is not None: out.append(candidate) + # ADR-0131.G.2 — comparative operations. + # Specificity order: nested > multiplicative > additive. Multiplicative + # anchors that overlap with additive ("twice" vs "two more") are disjoint + # at the lexical level (WORD_NUMBERS has 'two' not 'twice'); nesting + # consumes a *trailing* "than N times " tail so it cannot be confused + # with the bare additive pattern. See ADR-0131.G.2 for precedence + # rationale. + out.extend(_compare_nested_candidates(sentence)) + out.extend(_compare_multiplicative_candidates(sentence)) + out.extend(_compare_additive_candidates(sentence)) + + return out + + +# --------------------------------------------------------------------------- +# ADR-0131.G.2 — Comparative operation extractors +# --------------------------------------------------------------------------- +# +# Closed-set anchor alternation, aligned 1:1 with the four +# ``Comparison.direction`` literals registered in +# :data:`generate.math_roundtrip.COMPARE_ADDITIVE_ANCHORS` / +# :data:`COMPARE_MULTIPLICATIVE_ANCHORS`: +# +# additive — direction ∈ {more, fewer}; "less" admitted as a +# surface synonym mapped to direction='fewer'. +# ``matched_verb`` = the lowercased direction word +# ('more' / 'fewer' / 'less'); these are members of +# COMPARE_ADDITIVE_ANCHORS so the round-trip filter's +# verb-registry check (math_roundtrip step 1) passes. +# multiplicative — direction ∈ {times, fraction}; surface anchors are +# 'twice' / 'thrice' / 'N times' / 'half'. The +# anchor-as-value-token convention from math_roundtrip +# step 4 lets word-form factor anchors skip +# value-grounding (the anchor's own appearance in +# the source already grounds the factor). +# +# Out of scope (refused by deliberate non-match): "as many … as" without +# a direction anchor, "compared to …", "in comparison with …", +# "the same … as". These have no entry in COMPARE_*_ANCHORS — admitting +# them would breach the round-trip filter's verb-registry check anyway. + +# Comparative entity slot: proper-noun, "the X" collective, "the number/amount +# of " mass-noun construction. Possessives ("Bob's"/"his") are deferred. +_COMPARE_REF: Final[str] = ( + r"(?:" + r"the\s+(?:number|amount)\s+of\s+\w+" + r"|[Tt]he\s+\w+" + r"|[A-Z]\w+" + r")" +) + + +def _resolve_reference_token(raw: str) -> tuple[str, str]: + """Return ``(canonical_entity, head_token_for_grounding)``. + + For "the number of chickens" the head token is "chickens"; the + canonical entity uses the full noun-phrase so binding-graph + referential-integrity isn't subverted by collapsing different + references to the same noun. + """ + collapsed = re.sub(r"\s+", " ", raw.strip()) + lowered = collapsed.lower() + if lowered.startswith("the number of ") or lowered.startswith("the amount of "): + head = collapsed.split()[-1] + return collapsed, head + if lowered.startswith("the "): + head = collapsed[4:].split()[0] + return "the " + collapsed[4:], head + return collapsed, collapsed + + +def _comparison_anchor_verb() -> str: + # 'has' / 'have' carry the comparator phrase. We don't include 'had/gets' + # etc. in P2 — past-tense + lemma-widening are deferred to a later axis + # to keep the precedence story narrow. + return r"(?:has|have)" + + +_COMPARE_ADDITIVE_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P{_ENTITY})\s+{_comparison_anchor_verb()}\s+" + rf"(?P{_VALUE})\s+" + r"(?Pmore|fewer|less)\s+" + r"(?P\w+)\s+than\s+" + rf"(?P{_COMPARE_REF})\s*\.?$" +) + +# Multiplicative: anchor-as-value form ("twice"/"thrice"/"half" carry the +# factor implicitly). "as many " required; unit ellipsis ("twice as +# many as Bob") is deferred to keep wrong==0 strict — without unit the +# binding graph cannot disambiguate which dimension to compare. +_COMPARE_MULT_ANCHOR_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P{_ENTITY})\s+{_comparison_anchor_verb()}\s+" + r"(?Ptwice|thrice|half)\s+as\s+many\s+" + r"(?P\w+)\s+as\s+" + rf"(?P{_COMPARE_REF})\s*\.?$" +) + +# Multiplicative: explicit "N times as many as ". +_COMPARE_MULT_NTIMES_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P{_ENTITY})\s+{_comparison_anchor_verb()}\s+" + rf"(?P{_VALUE})\s+times\s+as\s+many\s+" + r"(?P\w+)\s+as\s+" + rf"(?P{_COMPARE_REF})\s*\.?$" +) + +# Nested: additive over multiplicative — "A has N more than M times ". +# Emits *two* flat candidates so the binding graph (ADR-0134) can decide which +# admissible composition (if any) survives. The parser does not commit to a +# nested operand shape (Comparison ∋ Comparison is not a supported operand +# type today); composition admissibility is the round-trip layer's call. +_COMPARE_NESTED_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P{_ENTITY})\s+{_comparison_anchor_verb()}\s+" + rf"(?P{_VALUE})\s+" + r"(?Pmore|fewer|less)\s+" + r"(?P\w+)\s+than\s+" + rf"(?P{_VALUE})\s+times\s+" + rf"(?P{_COMPARE_REF})\s*\.?$" +) + + +_ANCHOR_TO_FACTOR: Final[dict[str, tuple[float, str]]] = { + # surface anchor → (factor, direction-literal) + "twice": (2.0, "times"), + "thrice": (3.0, "times"), + "half": (0.5, "fraction"), +} + + +def _direction_to_anchor(direction_raw: str) -> tuple[str, str]: + """Map surface direction word → (canonical Comparison.direction, + matched_verb registered in COMPARE_ADDITIVE_ANCHORS). + + 'less' is a surface synonym of 'fewer'; the Comparison value uses + direction='fewer', but matched_verb retains the source surface + ('less') so the round-trip filter's "verb appears in source" check + succeeds. 'less' is registered in COMPARE_ADDITIVE_ANCHORS, so the + verb-registry check also succeeds. + """ + lowered = direction_raw.lower() + if lowered in ("more", "fewer"): + return lowered, lowered + if lowered == "less": + return "fewer", "less" + raise ValueError(f"unknown comparative direction surface {direction_raw!r}") + + +def _build_compare_additive( + *, + actor_raw: str, + delta_value_raw: str, + direction_raw: str, + unit_raw: str, + reference_raw: str, + source: str, +) -> CandidateOperation | None: + if _is_indefinite_quantifier(delta_value_raw): + return None + direction, matched_verb = _direction_to_anchor(direction_raw) + actor = _normalize_entity(actor_raw) + reference_canon, reference_head = _resolve_reference_token(reference_raw) + if reference_canon == actor: + return None # self-reference; constructor would refuse anyway + delta_value = _resolve_value(delta_value_raw) + unit = _canonicalize_unit(unit_raw) + try: + op = Operation( + actor=actor, + kind="compare_additive", + operand=Comparison( + reference_actor=reference_canon, + delta=Quantity(value=delta_value, unit=unit), + factor=None, + direction=cast(_CompDirection, direction), + ), + ) + except Exception: + return None + try: + return CandidateOperation( + op=op, + source_span=source, + matched_verb=matched_verb, + matched_value_token=delta_value_raw, + matched_unit_token=unit_raw, + matched_actor_token=actor_raw, + matched_reference_actor_token=reference_head, + ) + except Exception: + return None + + +def _build_compare_multiplicative( + *, + actor_raw: str, + factor: float, + matched_verb: str, + matched_value_token: str, + unit_raw: str, + reference_raw: str, + source: str, + direction: str, +) -> CandidateOperation | None: + actor = _normalize_entity(actor_raw) + reference_canon, reference_head = _resolve_reference_token(reference_raw) + if reference_canon == actor: + return None + _ = _canonicalize_unit(unit_raw) # validation only; multiplicative compares + # carry the unit on the source-span side, not the operand + try: + op = Operation( + actor=actor, + kind="compare_multiplicative", + operand=Comparison( + reference_actor=reference_canon, + delta=None, + factor=factor, + direction=cast(_CompDirection, direction), + ), + ) + except Exception: + return None + try: + return CandidateOperation( + op=op, + source_span=source, + matched_verb=matched_verb, + matched_value_token=matched_value_token, + matched_unit_token=unit_raw, + matched_actor_token=actor_raw, + matched_reference_actor_token=reference_head, + ) + except Exception: + return None + + +def _compare_additive_candidates(sentence: str) -> list[CandidateOperation]: + s = sentence.strip() + m = _COMPARE_ADDITIVE_RE.match(s) + if m is None: + return [] + cand = _build_compare_additive( + actor_raw=m.group("actor"), + delta_value_raw=m.group("value"), + direction_raw=m.group("direction"), + unit_raw=m.group("unit"), + reference_raw=m.group("reference"), + source=sentence, + ) + return [cand] if cand is not None else [] + + +def _compare_multiplicative_candidates(sentence: str) -> list[CandidateOperation]: + s = sentence.strip() + out: list[CandidateOperation] = [] + + m = _COMPARE_MULT_ANCHOR_RE.match(s) + if m is not None: + anchor = m.group("anchor").lower() + factor, direction = _ANCHOR_TO_FACTOR[anchor] + cand = _build_compare_multiplicative( + actor_raw=m.group("actor"), + factor=factor, + matched_verb=anchor, + matched_value_token=anchor, # anchor-as-value (math_roundtrip step 4) + unit_raw=m.group("unit"), + reference_raw=m.group("reference"), + source=sentence, + direction=direction, + ) + if cand is not None: + out.append(cand) + return out # specificity — don't also try N-times pattern + + m = _COMPARE_MULT_NTIMES_RE.match(s) + if m is not None: + value_raw = m.group("value") + if _is_indefinite_quantifier(value_raw): + return out + try: + factor = float(_resolve_value(value_raw)) + except KeyError: + return out + cand = _build_compare_multiplicative( + actor_raw=m.group("actor"), + factor=factor, + matched_verb="times", + matched_value_token=value_raw, + unit_raw=m.group("unit"), + reference_raw=m.group("reference"), + source=sentence, + direction="times", + ) + if cand is not None: + out.append(cand) + return out + + +def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]: + """Emit two flat candidates for nested 'N more than M times '. + + The parser does not commit to a composed Comparison-of-Comparison + operand (operand type Comparison ∋ Comparison is not modelled + today). Both flat candidates are forwarded; the binding-graph / + round-trip layer (ADR-0134) picks an admissible composition or + refuses. Refusal is the safe outcome — never a wrong answer. + """ + s = sentence.strip() + m = _COMPARE_NESTED_RE.match(s) + if m is None: + return [] + out: list[CandidateOperation] = [] + + actor_raw = m.group("actor") + unit_raw = m.group("unit") + reference_raw = m.group("reference") + + # Candidate 1: additive — "A has N more than " treating + # as the comparison reference directly. The "M times" multiplier + # is dropped on this candidate (the binding-graph composition is + # what would re-introduce it). + add_cand = _build_compare_additive( + actor_raw=actor_raw, + delta_value_raw=m.group("delta_value"), + direction_raw=m.group("direction"), + unit_raw=unit_raw, + reference_raw=reference_raw, + source=sentence, + ) + if add_cand is not None: + out.append(add_cand) + + # Candidate 2: multiplicative — "A has M times as many as " + # treating the multiplier M and the same as the multiplicative + # comparison. The additive offset N is dropped on this candidate. + factor_value_raw = m.group("factor_value") + if not _is_indefinite_quantifier(factor_value_raw): + try: + factor = float(_resolve_value(factor_value_raw)) + except KeyError: + factor = None + if factor is not None: + mult_cand = _build_compare_multiplicative( + actor_raw=actor_raw, + factor=factor, + matched_verb="times", + matched_value_token=factor_value_raw, + unit_raw=unit_raw, + reference_raw=reference_raw, + source=sentence, + direction="times", + ) + if mult_cand is not None: + out.append(mult_cand) + return out diff --git a/tests/test_adr_0131_G2_comparatives.py b/tests/test_adr_0131_G2_comparatives.py new file mode 100644 index 00000000..23ec0195 --- /dev/null +++ b/tests/test_adr_0131_G2_comparatives.py @@ -0,0 +1,275 @@ +"""ADR-0131.G.2 — comparative operations (additive + multiplicative). + +Pins the curated coverage axis under +``evals/math_capability_axes/G2_comparatives/v1/`` plus the +``math_candidate_parser`` comparative extractors. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + +import pytest + +from evals.math_capability_axes.G2_comparatives.v1.runner import ( + build_report, + write_report, + _REPORT_PATH, +) +from generate.math_candidate_parser import ( + _compare_additive_candidates, + _compare_multiplicative_candidates, + _compare_nested_candidates, + extract_operation_candidates, +) +from generate.math_roundtrip import ( + COMPARE_ADDITIVE_ANCHORS, + COMPARE_MULTIPLICATIVE_ANCHORS, + roundtrip_admissible, +) + + +_REPO = Path(__file__).resolve().parents[1] +_GSM8K_LEGACY_REPORT = ( + _REPO / "evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json" +) +_GSM8K_CG_REPORT = _REPO / "evals/gsm8k_math/train_sample/v1/report.json" + + +# --------------------------------------------------------------------------- +# Per-direction at-least-one-passing. +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("direction,sentence,actor,reference,delta", [ + ("more", "Alice has 3 more apples than Bob", "Alice", "Bob", 3), + ("fewer", "Alice has 5 fewer pencils than Bob", "Alice", "Bob", 5), +]) +def test_additive_direction_admits(direction, sentence, actor, reference, delta): + cands = [c for c in _compare_additive_candidates(sentence) if roundtrip_admissible(c)] + assert len(cands) == 1 + c = cands[0] + assert c.op.kind == "compare_additive" + assert c.op.operand.direction == direction + assert c.op.actor == actor + assert c.op.operand.reference_actor == reference + assert c.op.operand.delta.value == delta + + +def test_additive_less_maps_to_fewer(): + cands = [c for c in _compare_additive_candidates("Dana has 8 less coins than Eli") if roundtrip_admissible(c)] + assert len(cands) == 1 + c = cands[0] + assert c.op.operand.direction == "fewer" + assert c.matched_verb == "less" + # 'less' is a registered additive anchor (COMPARE_ADDITIVE_ANCHORS). + assert "less" in COMPARE_ADDITIVE_ANCHORS + + +@pytest.mark.parametrize("direction,sentence,actor,reference,factor,anchor", [ + ("times", "Alice has twice as many apples as Bob", "Alice", "Bob", 2.0, "twice"), + ("times", "Alice has 4 times as many apples as Bob", "Alice", "Bob", 4.0, "times"), + ("times", "Alice has thrice as many apples as Bob", "Alice", "Bob", 3.0, "thrice"), + ("fraction", "Alice has half as many apples as Bob", "Alice", "Bob", 0.5, "half"), +]) +def test_multiplicative_direction_admits(direction, sentence, actor, reference, factor, anchor): + cands = [c for c in _compare_multiplicative_candidates(sentence) if roundtrip_admissible(c)] + assert len(cands) == 1 + c = cands[0] + assert c.op.kind == "compare_multiplicative" + assert c.op.operand.direction == direction + assert c.op.actor == actor + assert c.op.operand.reference_actor == reference + assert c.op.operand.factor == factor + assert c.matched_verb == anchor + assert anchor in COMPARE_MULTIPLICATIVE_ANCHORS + + +# --------------------------------------------------------------------------- +# Nested composition emits BOTH flat candidates. +# --------------------------------------------------------------------------- + +def test_nested_emits_both_flat_candidates(): + s = "Jen has 10 more ducks than four times the number of chickens" + cands = [c for c in _compare_nested_candidates(s) if roundtrip_admissible(c)] + kinds = {c.op.kind for c in cands} + assert kinds == {"compare_additive", "compare_multiplicative"} + + +# --------------------------------------------------------------------------- +# Refusal cases — closed-set boundary holds. +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("sentence", [ + "Alice has as many apples as Bob", + "Compared to Bob, Alice has 3 more apples", + "In comparison with Bob, Alice has 3 more apples", + "Alice has the same number of apples as Bob", + "Alice has 3 times more apples than Bob", +]) +def test_refusal_cases_emit_no_admitted_comparative(sentence): + cands = extract_operation_candidates(sentence) + admitted_comparatives = [ + c for c in cands + if c.op.kind in ("compare_additive", "compare_multiplicative") + and roundtrip_admissible(c) + ] + assert admitted_comparatives == [] + + +# --------------------------------------------------------------------------- +# Closed-set anchor discipline — emitter never invents a direction literal. +# --------------------------------------------------------------------------- + +def test_direction_literals_closed_set(): + sentences = [ + "Alice has 3 more apples than Bob", + "Alice has 5 fewer pencils than Bob", + "Alice has 2 less books than Bob", + "Alice has twice as many apples as Bob", + "Alice has 4 times as many apples as Bob", + "Alice has half as many apples as Bob", + "Alice has thrice as many apples as Bob", + "Jen has 10 more ducks than four times the number of chickens", + ] + for s in sentences: + for c in extract_operation_candidates(s): + if c.op.kind in ("compare_additive", "compare_multiplicative"): + assert c.op.operand.direction in ("more", "fewer", "times", "fraction") + + +# --------------------------------------------------------------------------- +# Runner / report contract. +# --------------------------------------------------------------------------- + +def test_runner_wrong_count_is_zero(): + report = build_report() + assert report["metrics"]["wrong"] == 0 + assert report["metrics"]["wrong_count_is_zero"] is True + + +def test_runner_per_category_minima(): + """Brief §coverage: ≥4 per direction (more, fewer, times, fraction); + ≥3 nested; ≥4 refusal. We pin direction-level minima via per-case scan + rather than per-category (additive merges more+fewer into one bucket + in per_category but we want to verify each direction independently).""" + report = build_report() + direction_counts = {"more": 0, "fewer": 0, "times": 0, "fraction": 0} + nested = 0 + refusal = 0 + cases_path = _REPO / "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl" + for line in cases_path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + c = json.loads(line) + if c["category"] in ("additive", "multiplicative"): + direction_counts[c["direction"]] += 1 + elif c["category"] == "nested": + nested += 1 + elif c["category"] == "refusal": + refusal += 1 + for d, n in direction_counts.items(): + assert n >= 4, f"direction {d!r} has only {n} cases (need ≥4)" + assert nested >= 3 + assert refusal >= 4 + # And every one of those cases passes (wrong==0 already asserts no + # wrong-shaped pass, but verify the positive direction too). + assert report["metrics"]["passed"] == report["metrics"]["cases_total"] + + +def test_report_byte_equal_across_runs(): + a = json.dumps(build_report(), indent=2, sort_keys=True) + b = json.dumps(build_report(), indent=2, sort_keys=True) + assert a == b + + +def test_committed_report_matches_runner_output(): + report = build_report() + written = json.dumps(report, indent=2, sort_keys=True) + "\n" + on_disk = _REPORT_PATH.read_text(encoding="utf-8") + if written != on_disk: + # Re-write to be helpful in local dev (so CI failures point at the + # actual delta, not a stale file). + write_report(report) + assert written == on_disk, "G2 report.json is stale — re-run runner.py" + + +# --------------------------------------------------------------------------- +# GSM8K probe gate — chosen gate (per ADR-0131.G.2): +# comparative-clause refusals in the candidate-graph probe report.json +# strictly decrease. +# Legacy train_sample_coverage_report.json is byte-identical (no legacy +# parser change); we pin both invariants below. +# --------------------------------------------------------------------------- + +_COMPARATIVE_STATEMENT_PATTERNS = ( + # " more than" / "fewer / less" + re.compile(r"\b(?:more|fewer|less)\b[^.'\"]*\bthan\b", re.IGNORECASE), + # "twice|thrice|half|N times as many as" + re.compile(r"\b(?:twice|thrice|half|\d+\s+times)\s+as\s+many\b", re.IGNORECASE), +) + + +def _comparative_clause_refusal_count(probe_report_path: Path) -> int: + """Count refused cases whose refusal reason names a statement clause + that itself contains a comparative anchor (additive ``more/fewer/less + … than`` or multiplicative ``twice/N times/half as many``). We anchor + on the embedded statement clause inside the ``candidate_graph`` / + ``parser`` refusal reason, not on isolated keywords, to avoid + false-positives on incidental mentions ('he is more careful than', + discourse glue, etc.).""" + data = json.loads(probe_report_path.read_text(encoding="utf-8")) + per_case = data["per_case"] + count = 0 + for d in per_case: + if d.get("verdict", d.get("outcome")) != "refused": + continue + reason = d["reason"] + if "statement" not in reason and "statement clause" not in reason: + continue + for pat in _COMPARATIVE_STATEMENT_PATTERNS: + if pat.search(reason): + count += 1 + break + return count + + +def test_gsm8k_legacy_probe_safety_rail_intact(): + """ADR-0131.G invariant: legacy probe still shows admitted_wrong == 0.""" + data = json.loads(_GSM8K_LEGACY_REPORT.read_text(encoding="utf-8")) + assert data["metrics"]["admitted_wrong"] == 0 + assert data["metrics"]["safety_rail_intact"] is True + + +def test_gsm8k_candidate_graph_probe_wrong_zero(): + """Candidate-graph probe also preserves wrong == 0.""" + data = json.loads(_GSM8K_CG_REPORT.read_text(encoding="utf-8")) + assert data["counts"]["wrong"] == 0 + + +def test_gsm8k_candidate_graph_comparative_clause_refusals_decreased(): + """G2 gate: at least one previously-comparative-clause-refused case + no longer cites a comparative-clause refusal in the candidate-graph + probe. Baseline (pre-G.2) for the train sample had case 0009 ('Jen + has 10 more ducks than four times the number of chickens') refused + at the comparative statement; with G.2 wired into + math_candidate_parser, that comparative clause now parses (the case + may still refuse downstream — at the question or solver — but the + refusal reason no longer cites the comparative clause).""" + current_count = _comparative_clause_refusal_count(_GSM8K_CG_REPORT) + # Pre-G.2 baseline (`origin/main` @ 481e0c3): 2 cases refused at a + # statement clause whose text contains additive ('more/fewer/less ... + # than') or multiplicative ('twice/N times/half as many') anchors — + # cases gsm8k-train-sample-v1-0009 ('Jen has 10 more ducks than four + # times the number of chickens.') and -0016 (arithmetic 'more than' + # in 'Rudolph traveled 2 more than 5 miles'). G.2 wires comparatives + # into math_candidate_parser; case 0009 now parses the comparative + # clause (refusal moves to the question form), so the count drops to + # 1 (case 0016's arithmetic 'N more than M' is *not* a between-entity + # comparative and is deliberately out of G.2 scope). + baseline_count = 2 + assert current_count < baseline_count, ( + f"expected comparative-clause refusal count to drop below " + f"baseline {baseline_count}; got {current_count}" + )