core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(ADR-0131.G.2): comparative operations (additive + multiplicative) — admission unchanged, comparative-clause refusals 2→1

Browse source 
Wire compare_additive / compare_multiplicative extractors into the
candidate-emitting sentence parser, closing the deferred phase flagged
at generate/math_candidate_parser.py:30.

Capability axis: comparatives (additive + multiplicative)
- generate/math_candidate_parser.py: new _compare_additive_candidates,
  _compare_multiplicative_candidates, _compare_nested_candidates
  emitting CandidateOperation records keyed to the four
  Comparison.direction literals registered in ADR-0123.
- Closed-set anchor alternation; 'less' admitted as surface synonym of
  'fewer'; reference slot widened to admit "the number/amount of <unit>"
  for nested forms.
- Nested 'A has N more <unit> than M times <REF>' emits two flat
  candidates (additive + multiplicative); binding-graph picks the
  admissible composition or refuses (no solver stub).

Curated axis lane (24 cases)
- evals/math_capability_axes/G2_comparatives/v1/cases.jsonl:
  8 additive / 8 multiplicative / 3 nested / 5 refusal
- evals/math_capability_axes/G2_comparatives/v1/runner.py +
  report.json: deterministic, wrong==0 gate, byte-equal across runs.

Tests (21 new)
- tests/test_adr_0131_G2_comparatives.py: per-direction at-least-one
  passing, nested-both-emitted, closed-set refusal, runner
  byte-equality, GSM8K-probe gate (comparative-clause refusals
  strictly decrease).

GSM8K-probe gate (chosen: comparative-clause refusals ↓)
- evals/gsm8k_math/train_sample/v1/report.json (candidate-graph
  probe): comparative-clause refusal count 2 → 1 (case 0009 'Jen has
  10 more ducks than four times the number of chickens' moves from
  statement-clause refusal to question-layer refusal). admitted_wrong
  remains 0; admission_rate unchanged (downstream composition is a
  follow-up ADR).
- evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
  (legacy probe): refreshed, byte-identical (legacy parser untouched).

B3 + candidate-graph + GSM8K probe lanes all pass (90/90). Direction
vocab stays closed to {more, fewer, times, fraction}; wrong==0
preserved everywhere.
This commit is contained in:
Shay 2026-05-23 14:15:25 -07:00
parent 481e0c3b9c
commit b891eb243c
10 changed files with 1398 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

265
docs/decisions/ADR-0131.G.2-comparatives.md Normal file
View file

@ -0,0 +1,265 @@
# ADR-0131.G.2 — Capability axis: comparative operations (additive + multiplicative)
**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Parent:** [ADR-0131.G](./ADR-0131.G-gsm8k-coverage-probe.md)
**Depends on:**
[ADR-0123](./ADR-0123-comparison-substrate.md) (Comparison op shape),
[ADR-0126](./ADR-0126-candidate-graph-parser.md) (candidate-graph topology),
[ADR-0131.3](./ADR-0131.3-bounded-grammar.md),
[ADR-0132](./ADR-0132-binding-graph-data-model.md)..[ADR-0134](./ADR-0134-unit-aware-admissibility.md)
---
## Context
The `Comparison` operand shape, the four `direction` literals (`more`,
`fewer`, `times`, `fraction`), and the round-trip anchor tables
(`COMPARE_ADDITIVE_ANCHORS`, `COMPARE_MULTIPLICATIVE_ANCHORS`) all
landed with ADR-0123. The legacy first-match-wins parser
(`generate/math_parser.py`) wires those for the bounded-grammar lane.
The candidate-emitting sentence parser
(`generate/math_candidate_parser.py:30`) explicitly deferred
comparative candidates to a "later phase of ADR-0126". Without that
phase, multi-sentence GSM8K problems that contain a comparative
clause are refused by the candidate-graph topology even when the
comparative is well-formed — the comparative sentence emits zero
candidates, the per-sentence choice space is empty, and the whole
problem refuses with `no admissible candidate for statement: 'X'`.
This ADR is the deferred comparative phase, framed under ADR-0131.G's
capability-first iteration discipline: comparatives are a single
capability axis (not a list of GSM8K templates), exercised by curated
sentences independent of GSM8K, with the probe's admission /
refused-reasons numbers moving only as a side effect.
---
## Decision
Land comparative candidate-emitting extractors in
`generate/math_candidate_parser.py`:
| Function | Closed-set shapes admitted |
|---|---|
| `_compare_additive_candidates` | `<A> has N more <unit> than <B>` / `... N fewer ...` / `... N less ...` |
| `_compare_multiplicative_candidates` | `<A> has twice as many <unit> as <B>` / `... thrice ...` / `... half ...` / `... N times as many ...` |
| `_compare_nested_candidates` | `<A> has N more <unit> than M times <REF>` — emits *two* flat candidates (one additive, one multiplicative), letting the binding-graph layer (ADR-0134) pick an admissible composition or refuse |
Direction vocabulary is **closed** to the four `Comparison.direction`
literals. `less` is admitted as a surface synonym of `fewer`; its
canonical Comparison value is `direction='fewer'` while
`matched_verb='less'` (so the round-trip filter's verb-registry check
finds `less` in `COMPARE_ADDITIVE_ANCHORS`, and its
"verb appears in source" check succeeds).
The reference slot widens beyond proper-nouns / "the X" to admit
"the number/amount of <unit>" mass-noun constructions
(`_resolve_reference_token`), so the GSM8K-style "...four times the
number of chickens" composes. The canonical entity name is the full
noun phrase; binding-graph referential integrity is preserved
(distinct noun phrases stay distinct entities).
### Specificity / precedence
Extractor order in `extract_operation_candidates`:
1. add / subtract / transfer (existing — ADR-0126)
2. **`_compare_nested_candidates`** (most specific — requires both
`(more|fewer|less) ... than` *and* `N times <REF>` tail)
3. **`_compare_multiplicative_candidates`** (anchor-as-value forms
`twice/thrice/half`, then `N times as many`)
4. **`_compare_additive_candidates`** (`N more/fewer/less <unit> than
<REF>`)
Multiplicative anchors that overlap lexically with additive are
disjoint at the lexical level: `WORD_NUMBERS` registers `two`/`three`
but **not** `twice`/`thrice`, so the additive VALUE alternation cannot
swallow a multiplicative anchor. Nested matching consumes its
trailing `than N times <REF>` greedily, so a sentence the nested
regex matches cannot also be claimed by the bare additive pattern
(the bare additive regex requires the `than <REF>` tail with `<REF>`
matching `_COMPARE_REF`, but the nested regex's `<REF>` is preceded
by `N times`, which the bare additive regex's `<REF>` slot does not
admit). Documented + tested in
`tests/test_adr_0131_G2_comparatives.py`.
### Curated coverage cases (G.2 axis lane)
`evals/math_capability_axes/G2_comparatives/v1/cases.jsonl` —
**24 cases**, exercised by an axis-specific runner
(`evals/math_capability_axes/G2_comparatives/v1/runner.py`) that
emits a deterministic `report.json`:
| Category | Cases | Direction coverage |
|----------------|-------|----------------------------|
| additive | 8 | `more` ×4, `fewer` ×4 (incl. one `less` surface)|
| multiplicative | 8 | `times` ×4, `fraction` ×4 |
| nested | 3 | additive + multiplicative both flat-emitted |
| refusal | 5 | paraphrases outside the closed set |
Per-direction at-least-one-passing, nested-both-emitted, refusal
zero-admitted, and `wrong == 0` are pinned by
`tests/test_adr_0131_G2_comparatives.py`. Report is byte-equal across
runs.
### Deferred (out of scope for G.2)
Documented refusal classes — these have no entry in
`COMPARE_*_ANCHORS`, so admitting them would breach the round-trip
filter's verb-registry check anyway:
- `"as many ... as"` without a direction word ("Alice has as many
apples as Bob") — equality is not in
`{more, fewer, times, fraction}`.
- `"compared to <B>, <A> ..."` and `"in comparison with <B>, <A> ..."`
— prefix paraphrases; no anchor verb is consumed.
- `"the same number of ... as"` — equality, see above.
- `"N times more"` — legacy parser already refuses as ambiguous
(ADR-0123 line 797); G.2 preserves that refusal.
- Pronoun resolution in actor / reference slots — needs per-branch
state (ADR-0126 P3 future work).
- Past-tense / lemma widening (`had`, `gets`, `took`, `buys`) — adds
surface variants without growing the capability axis; deferred to a
separate axis to keep precedence narrow.
- Unit ellipsis (`"twice as many as Bob"` without a unit) — without a
unit the binding graph cannot pick a dimension to compare under
ADR-0134's unit-aware admissibility; refusal preserves `wrong == 0`.
### Nested composition is *parser-side only*
The parser emits two flat candidates for nested forms; it does **not**
introduce a `Comparison ∋ Comparison` operand shape (which the data
model doesn't support today). The binding-graph layer (ADR-0134)
picks the admissible composition or refuses. **Refusal of nested
forms is the safe outcome and is what we observe today on
gsm8k-train-sample-v1-0009**: the comparative clause now parses (it
no longer refuses at the statement layer), but the downstream
composition has not yet been wired, so the case continues to refuse
— at the question layer instead. That downstream gap is a follow-up
ADR (`compare_additive ∘ compare_multiplicative` graph composition);
this ADR explicitly does **not** stub the solver to bypass it.
### GSM8K-probe gate (chosen)
G.2 gates on:
> **Comparative-clause refusals in the candidate-graph probe report
> (`evals/gsm8k_math/train_sample/v1/report.json`) strictly
> decrease.**
Test: `test_gsm8k_candidate_graph_comparative_clause_refusals_decreased`
(measures count of refused cases whose reason cites a statement
clause containing either additive `more/fewer/less ... than` or
multiplicative `twice/N times/half as many` anchors).
| Probe report | Baseline (origin/main 481e0c3) | After G.2 | Δ |
|---|---|---|---|
| Comparative-clause refusals (`report.json`) | 2 | 1 | −1 |
| `admitted_wrong` (`report.json`) | 0 | 0 | 0 |
| `admission_rate` (`report.json`) | 0/50 | 0/50 | 0 |
The baseline-2 cases are `gsm8k-train-sample-v1-0009` ("Jen has 10
more ducks than four times the number of chickens.") and
`-0016` ("Rudolph traveled 2 more than 5 miles"). G.2's extractor
parses the comparative clause in 0009; 0016's "2 more than 5" is an
arithmetic expression between numbers, not a between-entity
comparison, and is deliberately out of G.2 scope (the additive shape
requires an entity reference, not a numeric reference).
`admission_rate` does **not** rise because the downstream
binding-graph composition for nested comparatives + the question
layer's compatibility with comparison-derived state are out of G.2
scope. The honest claim is that the parser admits one more shape,
not that the engine solves more GSM8K problems.
### Legacy probe report (refreshed, byte-identical)
`evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
runs through `generate.math_parser.parse_problem` (legacy
first-match-wins), which already supported ADR-0123 comparative
patterns. G.2 does not touch the legacy parser, so the legacy
probe report is byte-identical to the baseline. Refreshed and
pinned via `test_gsm8k_legacy_probe_safety_rail_intact`.
---
## Invariants
- **`compare_direction_closed_set`** — every emitted comparative
candidate has `op.operand.direction ∈ {more, fewer, times,
fraction}`. Pinned by
`test_direction_literals_closed_set`.
- **`compare_additive_wrong_zero`** / **`compare_multiplicative_wrong_zero`**
— every additive / multiplicative coverage case passes through
`roundtrip_admissible`. Pinned by the per-direction parametric
tests.
- **`nested_emits_both_flat`** — the nested-composition extractor
emits exactly the two flat candidates (additive + multiplicative);
whether either composes is the binding-graph layer's call. Pinned
by `test_nested_emits_both_flat_candidates`.
- **`closed_set_refusal_holds`** — none of the five documented
refusal paraphrases admit any comparative candidate. Pinned by
`test_refusal_cases_emit_no_admitted_comparative`.
- **`gsm8k_safety_rail_intact`** — `admitted_wrong == 0` on both the
legacy and candidate-graph GSM8K probe reports.
- **`gsm8k_comparative_refusal_strictly_decreased`** — chosen G.2
gate (see above).
- **`report_deterministic`** — `report.json` is byte-equal across two
back-to-back runs.
---
## Acceptance evidence
- `evals/math_capability_axes/G2_comparatives/v1/runner.py` exits 0
with `wrong == 0` on all 24 curated cases (8 additive / 8
multiplicative / 3 nested / 5 refusal).
- `tests/test_adr_0131_G2_comparatives.py` (21 tests): per-direction
at-least-one-passing, nested-both-emitted, refusal-set zero-admit,
closed-set discipline, runner byte-equality, GSM8K-probe gate.
- `evals/gsm8k_math/train_sample/v1/report.json` shows
comparative-clause refusal count drop from 2 → 1.
- `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
refreshed and byte-identical (legacy parser untouched).
- B3 lane (`tests/test_adr_0131_3_bounded_grammar_lane.py`) +
candidate-graph runner tests + GSM8K probe tests all pass.
---
## Consequences
- The candidate-graph topology can finally see comparative
statements as first-class operations. The downstream composition
layer (ADR-0134 unit-aware admissibility + a future "graph composes
nested comparison" ADR) becomes the next honest unblock.
- `gsm8k-train-sample-v1-0009` moves from "refuse at comparative
clause" to "refuse at question layer" — a refusal-class shift,
not an admission. That is the correct, honest classification: the
case is still refused; the reason it's refused has changed.
- Future axes (G.3 rate-verbs, G.4 multi-clause distributive
subjects) inherit the same axis-lane harness shape under
`evals/math_capability_axes/`.
- Capability-first discipline is preserved: no GSM8K-specific
templates were added; the new shapes are exercised by 24
hand-authored sentences plus whatever GSM8K has incidentally.
---
## Out of scope
- **Solver changes.** If a comparative parses but doesn't solve,
that's a binding-graph composition gap, deferred to a follow-up
ADR (no solver stubs).
- **Implicit-comparison shapes** ("Alice has the same number of
apples as Bob") — equality is not in
`{more, fewer, times, fraction}`; admitting it requires extending
the closed direction set, which is its own ADR.
- **Probe runner contract.** ADR-0131.G pinned `run_lane` as the
legacy probe's contract; G.2 does not change that. The
candidate-graph probe (`report.json` via `runner.py`) is the
measurement surface that actually moves under G.2.
- **Frontier head-to-head on comparative accuracy.** Deferred until
the composition layer admits at least one nested form end-to-end.

2
evals/gsm8k_math/train_sample/v1/report.json
View file

@ -53,7 +53,7 @@
},
{
"case_id": "gsm8k-train-sample-v1-0009",
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'",
"reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'",
"verdict": "refused"
},
{

0
evals/math_capability_axes/G2_comparatives/__init__.py Normal file
View file

0
evals/math_capability_axes/G2_comparatives/v1/__init__.py Normal file
View file

24
evals/math_capability_axes/G2_comparatives/v1/cases.jsonl Normal file
View file

@ -0,0 +1,24 @@
{"case_id": "G2-add-more-001", "category": "additive", "direction": "more", "sentence": "Alice has 3 more apples than Bob", "expected": {"kind": "compare_additive", "actor": "Alice", "reference": "Bob", "delta_value": 3, "unit": "apples", "matched_verb": "more"}}
{"case_id": "G2-add-more-002", "category": "additive", "direction": "more", "sentence": "James has 7 more marbles than Carter", "expected": {"kind": "compare_additive", "actor": "James", "reference": "Carter", "delta_value": 7, "unit": "marbles", "matched_verb": "more"}}
{"case_id": "G2-add-more-003", "category": "additive", "direction": "more", "sentence": "Tom has four more cookies than Sara", "expected": {"kind": "compare_additive", "actor": "Tom", "reference": "Sara", "delta_value": 4, "unit": "cookies", "matched_verb": "more"}}
{"case_id": "G2-add-more-004", "category": "additive", "direction": "more", "sentence": "Marion has 12 more paperclips than Yun", "expected": {"kind": "compare_additive", "actor": "Marion", "reference": "Yun", "delta_value": 12, "unit": "paperclips", "matched_verb": "more"}}
{"case_id": "G2-add-fewer-001", "category": "additive", "direction": "fewer", "sentence": "Alice has 5 fewer pencils than Bob", "expected": {"kind": "compare_additive", "actor": "Alice", "reference": "Bob", "delta_value": 5, "unit": "pencils", "matched_verb": "fewer"}}
{"case_id": "G2-add-fewer-002", "category": "additive", "direction": "fewer", "sentence": "Carter has two fewer pets than James", "expected": {"kind": "compare_additive", "actor": "Carter", "reference": "James", "delta_value": 2, "unit": "pets", "matched_verb": "fewer"}}
{"case_id": "G2-add-fewer-003", "category": "additive", "direction": "fewer", "sentence": "Dana has 8 less coins than Eli", "expected": {"kind": "compare_additive", "actor": "Dana", "reference": "Eli", "delta_value": 8, "unit": "coins", "matched_verb": "less"}}
{"case_id": "G2-add-fewer-004", "category": "additive", "direction": "fewer", "sentence": "Beth has 3 fewer books than Sam", "expected": {"kind": "compare_additive", "actor": "Beth", "reference": "Sam", "delta_value": 3, "unit": "books", "matched_verb": "fewer"}}
{"case_id": "G2-mul-times-001", "category": "multiplicative", "direction": "times", "sentence": "Alice has twice as many apples as Bob", "expected": {"kind": "compare_multiplicative", "actor": "Alice", "reference": "Bob", "factor": 2.0, "unit": "apples", "matched_verb": "twice"}}
{"case_id": "G2-mul-times-002", "category": "multiplicative", "direction": "times", "sentence": "Brooke has 3 times as many jacks as Sidney", "expected": {"kind": "compare_multiplicative", "actor": "Brooke", "reference": "Sidney", "factor": 3.0, "unit": "jacks", "matched_verb": "times"}}
{"case_id": "G2-mul-times-003", "category": "multiplicative", "direction": "times", "sentence": "Cindy has thrice as many cards as Nicole", "expected": {"kind": "compare_multiplicative", "actor": "Cindy", "reference": "Nicole", "factor": 3.0, "unit": "cards", "matched_verb": "thrice"}}
{"case_id": "G2-mul-times-004", "category": "multiplicative", "direction": "times", "sentence": "Erica has 5 times as many fish as Mark", "expected": {"kind": "compare_multiplicative", "actor": "Erica", "reference": "Mark", "factor": 5.0, "unit": "fish", "matched_verb": "times"}}
{"case_id": "G2-mul-frac-001", "category": "multiplicative", "direction": "fraction", "sentence": "Monica has half as many hours as Sara", "expected": {"kind": "compare_multiplicative", "actor": "Monica", "reference": "Sara", "factor": 0.5, "unit": "hours", "matched_verb": "half"}}
{"case_id": "G2-mul-frac-002", "category": "multiplicative", "direction": "fraction", "sentence": "Rex has half as many cards as Cindy", "expected": {"kind": "compare_multiplicative", "actor": "Rex", "reference": "Cindy", "factor": 0.5, "unit": "cards", "matched_verb": "half"}}
{"case_id": "G2-mul-frac-003", "category": "multiplicative", "direction": "fraction", "sentence": "Anna has half as many stickers as Beth", "expected": {"kind": "compare_multiplicative", "actor": "Anna", "reference": "Beth", "factor": 0.5, "unit": "stickers", "matched_verb": "half"}}
{"case_id": "G2-mul-frac-004", "category": "multiplicative", "direction": "fraction", "sentence": "Tim has half as many toys as Jack", "expected": {"kind": "compare_multiplicative", "actor": "Tim", "reference": "Jack", "factor": 0.5, "unit": "toys", "matched_verb": "half"}}
{"case_id": "G2-nested-001", "category": "nested", "sentence": "Jen has 10 more ducks than four times the number of chickens", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jen", "reference": "the number of chickens", "delta_value": 10, "factor": 4.0, "unit": "ducks"}}
{"case_id": "G2-nested-002", "category": "nested", "sentence": "Jose has 2 more pounds than 2 times the amount of Orlando", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Jose", "reference": "the amount of Orlando", "delta_value": 2, "factor": 2.0, "unit": "pounds"}}
{"case_id": "G2-nested-003", "category": "nested", "sentence": "Sara has 5 fewer marbles than 3 times Bob", "expected": {"emits_flat_candidates": ["compare_additive", "compare_multiplicative"], "actor": "Sara", "reference": "Bob", "delta_value": 5, "factor": 3.0, "unit": "marbles"}}
{"case_id": "G2-refuse-001", "category": "refusal", "sentence": "Alice has as many apples as Bob", "expected": {"refuse": true, "reason": "implicit-comparison 'as many ... as' without a direction anchor — out of closed set"}}
{"case_id": "G2-refuse-002", "category": "refusal", "sentence": "Compared to Bob, Alice has 3 more apples", "expected": {"refuse": true, "reason": "'compared to' prefix — paraphrase outside the closed anchor set"}}
{"case_id": "G2-refuse-003", "category": "refusal", "sentence": "In comparison with Bob, Alice has 3 more apples", "expected": {"refuse": true, "reason": "'in comparison with' prefix — paraphrase outside the closed anchor set"}}
{"case_id": "G2-refuse-004", "category": "refusal", "sentence": "Alice has the same number of apples as Bob", "expected": {"refuse": true, "reason": "'the same number ... as' — equality is not in {more, fewer, times, fraction}"}}
{"case_id": "G2-refuse-005", "category": "refusal", "sentence": "Alice has 3 times more apples than Bob", "expected": {"refuse": true, "reason": "ambiguous 'N times more' — admitted to neither additive nor multiplicative shape (cf. ADR-0123 legacy refusal)"}}

267
evals/math_capability_axes/G2_comparatives/v1/report.json Normal file
View file

@ -0,0 +1,267 @@
{
"adr": "0131.G.2",
"axis": "comparatives",
"cases_path": "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl",
"metrics": {
"cases_total": 24,
"pass_rate": 1.0,
"passed": 24,
"wrong": 0,
"wrong_count_is_zero": true,
"wrong_rate": 0.0
},
"per_case": [
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-more-001",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-more-002",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-more-003",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-more-004",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-fewer-001",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-fewer-002",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-fewer-003",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_additive"
],
"case_id": "G2-add-fewer-004",
"category": "additive",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-times-001",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-times-002",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-times-003",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-times-004",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-frac-001",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-frac-002",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-frac-003",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 1,
"admitted_kinds": [
"compare_multiplicative"
],
"case_id": "G2-mul-frac-004",
"category": "multiplicative",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 2,
"admitted_kinds": [
"compare_additive",
"compare_multiplicative"
],
"case_id": "G2-nested-001",
"category": "nested",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 2,
"admitted_kinds": [
"compare_additive",
"compare_multiplicative"
],
"case_id": "G2-nested-002",
"category": "nested",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 2,
"admitted_kinds": [
"compare_additive",
"compare_multiplicative"
],
"case_id": "G2-nested-003",
"category": "nested",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 0,
"admitted_kinds": [],
"case_id": "G2-refuse-001",
"category": "refusal",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 0,
"admitted_kinds": [],
"case_id": "G2-refuse-002",
"category": "refusal",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 0,
"admitted_kinds": [],
"case_id": "G2-refuse-003",
"category": "refusal",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 0,
"admitted_kinds": [],
"case_id": "G2-refuse-004",
"category": "refusal",
"outcome": "pass",
"reason": ""
},
{
"admitted_count": 0,
"admitted_kinds": [],
"case_id": "G2-refuse-005",
"category": "refusal",
"outcome": "pass",
"reason": ""
}
],
"per_category": {
"additive": {
"pass": 8,
"wrong": 0
},
"multiplicative": {
"pass": 8,
"wrong": 0
},
"nested": {
"pass": 3,
"wrong": 0
},
"refusal": {
"pass": 5,
"wrong": 0
}
},
"schema_version": 1
}

205
evals/math_capability_axes/G2_comparatives/v1/runner.py Normal file
View file

@ -0,0 +1,205 @@
"""ADR-0131.G.2 — Capability axis runner for comparative operations.
Exercises the ``compare_additive`` / ``compare_multiplicative`` extractors
in :mod:`generate.math_candidate_parser` against curated coverage cases
that are independent of GSM8K.
Per-case classification:
| Case category | pass criterion |
|-----------------|---------------------------------------------------------|
| additive | ≥1 admitted candidate with kind=compare_additive whose |
| | direction, actor, reference, delta_value, unit match |
| multiplicative | ≥1 admitted candidate with kind=compare_multiplicative |
| | whose direction, actor, reference, factor, unit match |
| nested | both compare_additive AND compare_multiplicative flat |
| | candidates emitted + admitted (binding-graph picks) |
| refusal | zero admitted comparative candidates |
``wrong`` is non-zero only if a refusal-case emits an admitted comparative
or a positive-case emits a comparative with the wrong shape. ``wrong == 0``
is the load-bearing gate (ADR-0114a Obligation #4).
Determinism: case order in ``cases.jsonl`` is the report order; same
input file → byte-equal report.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
from generate.math_candidate_parser import extract_operation_candidates
from generate.math_roundtrip import roundtrip_admissible
_HERE = Path(__file__).resolve().parent
_CASES_PATH = _HERE / "cases.jsonl"
_REPORT_PATH = _HERE / "report.json"
def _load_cases() -> list[dict[str, Any]]:
return [
json.loads(line)
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines()
if line.strip()
]
def _admitted(sentence: str) -> list[Any]:
cands = extract_operation_candidates(sentence)
return [
c for c in cands
if c.op.kind in ("compare_additive", "compare_multiplicative")
and roundtrip_admissible(c)
]
def _score_positive_additive(case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]:
exp = case["expected"]
matches = [
c for c in admitted
if c.op.kind == "compare_additive"
and c.op.operand.direction == case["direction"]
and c.op.actor == exp["actor"]
and c.op.operand.reference_actor == exp["reference"]
and c.op.operand.delta is not None
and c.op.operand.delta.value == exp["delta_value"]
and c.op.operand.delta.unit == exp["unit"]
and c.matched_verb == exp["matched_verb"]
]
if matches:
return "pass", ""
if not admitted:
return "wrong", "no admissible comparative candidate emitted"
return (
"wrong",
f"admitted comparative did not match expected additive shape; got {[(c.op.kind, c.op.operand.as_json()) for c in admitted]}",
)
def _score_positive_multiplicative(case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]:
exp = case["expected"]
matches = [
c for c in admitted
if c.op.kind == "compare_multiplicative"
and c.op.operand.direction == case["direction"]
and c.op.actor == exp["actor"]
and c.op.operand.reference_actor == exp["reference"]
and c.op.operand.factor is not None
and float(c.op.operand.factor) == float(exp["factor"])
and c.matched_verb == exp["matched_verb"]
]
if matches:
return "pass", ""
if not admitted:
return "wrong", "no admissible comparative candidate emitted"
return (
"wrong",
f"admitted comparative did not match expected multiplicative shape; got {[(c.op.kind, c.op.operand.as_json()) for c in admitted]}",
)
def _score_nested(_case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]:
kinds = {c.op.kind for c in admitted}
if kinds == {"compare_additive", "compare_multiplicative"}:
return "pass", ""
return (
"wrong",
f"nested case must emit both flat candidate kinds; got {sorted(kinds)}",
)
def _score_refusal(_case: dict[str, Any], admitted: list[Any]) -> tuple[str, str]:
if not admitted:
return "pass", ""
return (
"wrong",
f"refusal case admitted a comparative candidate ({[c.op.kind for c in admitted]}); "
f"closed-set boundary breached",
)
def _score_case(case: dict[str, Any]) -> dict[str, Any]:
sentence = case["sentence"]
admitted = _admitted(sentence)
category = case["category"]
if category == "additive":
outcome, reason = _score_positive_additive(case, admitted)
elif category == "multiplicative":
outcome, reason = _score_positive_multiplicative(case, admitted)
elif category == "nested":
outcome, reason = _score_nested(case, admitted)
elif category == "refusal":
outcome, reason = _score_refusal(case, admitted)
else:
outcome, reason = "wrong", f"unknown case category {category!r}"
return {
"case_id": case["case_id"],
"category": category,
"outcome": outcome,
"reason": reason,
"admitted_count": len(admitted),
"admitted_kinds": sorted({c.op.kind for c in admitted}),
}
def build_report() -> dict[str, Any]:
cases = _load_cases()
per_case = [_score_case(c) for c in cases]
total = len(per_case)
passed = sum(1 for d in per_case if d["outcome"] == "pass")
wrong = sum(1 for d in per_case if d["outcome"] == "wrong")
by_category: dict[str, dict[str, int]] = {}
for d in per_case:
slot = by_category.setdefault(d["category"], {"passed": 0, "wrong": 0})
slot[d["outcome"] if d["outcome"] == "passed" else d["outcome"]] = (
slot.get(d["outcome"], 0) + 1
)
# Rebuild by_category with normalized keys.
by_category = {}
for d in per_case:
slot = by_category.setdefault(d["category"], {"pass": 0, "wrong": 0})
slot[d["outcome"]] = slot.get(d["outcome"], 0) + 1
return {
"schema_version": 1,
"adr": "0131.G.2",
"axis": "comparatives",
"cases_path": "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl",
"metrics": {
"cases_total": total,
"passed": passed,
"wrong": wrong,
"pass_rate": (passed / total) if total else 0.0,
"wrong_rate": (wrong / total) if total else 0.0,
"wrong_count_is_zero": wrong == 0,
},
"per_category": {
k: dict(sorted(v.items())) for k, v in sorted(by_category.items())
},
"per_case": per_case,
}
def write_report(report: dict[str, Any]) -> None:
_REPORT_PATH.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
report = build_report()
write_report(report)
m = report["metrics"]
print(
f"ADR-0131.G.2 comparatives: passed {m['passed']}/{m['cases_total']} "
f"({m['pass_rate']:.1%}); wrong={m['wrong']} (gate: must be 0)"
)
for cat, counts in report["per_category"].items():
print(f" {cat:14s} {counts}")
return 0 if m["wrong_count_is_zero"] else 1
if __name__ == "__main__":
raise SystemExit(main())

0
evals/math_capability_axes/__init__.py Normal file
View file

362
generate/math_candidate_parser.py
View file

@ -36,9 +36,10 @@ from __future__ import annotations
import re
from dataclasses import dataclass
from typing import Final
from typing import Final, Literal, cast
from generate.math_problem_graph import (
Comparison,
InitialPossession,
Operation,
Quantity,
@ -53,6 +54,11 @@ from generate.math_roundtrip import (
)
# Locally re-typed alias mirroring Comparison.direction's literal slot —
# used only to satisfy pyright when narrowing surface-direction strings.
_CompDirection = Literal["more", "fewer", "times", "fraction"]
# ---------------------------------------------------------------------------
# Initial-possession candidate
# ---------------------------------------------------------------------------
@ -483,4 +489,358 @@ def extract_operation_candidates(sentence: str) -> list[CandidateOperation]:
if candidate is not None:
out.append(candidate)
# ADR-0131.G.2 — comparative operations.
# Specificity order: nested > multiplicative > additive. Multiplicative
# anchors that overlap with additive ("twice" vs "two more") are disjoint
# at the lexical level (WORD_NUMBERS has 'two' not 'twice'); nesting
# consumes a *trailing* "than N times <REF>" tail so it cannot be confused
# with the bare additive pattern. See ADR-0131.G.2 for precedence
# rationale.
out.extend(_compare_nested_candidates(sentence))
out.extend(_compare_multiplicative_candidates(sentence))
out.extend(_compare_additive_candidates(sentence))
return out
# ---------------------------------------------------------------------------
# ADR-0131.G.2 — Comparative operation extractors
# ---------------------------------------------------------------------------
#
# Closed-set anchor alternation, aligned 1:1 with the four
# ``Comparison.direction`` literals registered in
# :data:`generate.math_roundtrip.COMPARE_ADDITIVE_ANCHORS` /
# :data:`COMPARE_MULTIPLICATIVE_ANCHORS`:
#
# additive — direction ∈ {more, fewer}; "less" admitted as a
# surface synonym mapped to direction='fewer'.
# ``matched_verb`` = the lowercased direction word
# ('more' / 'fewer' / 'less'); these are members of
# COMPARE_ADDITIVE_ANCHORS so the round-trip filter's
# verb-registry check (math_roundtrip step 1) passes.
# multiplicative — direction ∈ {times, fraction}; surface anchors are
# 'twice' / 'thrice' / 'N times' / 'half'. The
# anchor-as-value-token convention from math_roundtrip
# step 4 lets word-form factor anchors skip
# value-grounding (the anchor's own appearance in
# the source already grounds the factor).
#
# Out of scope (refused by deliberate non-match): "as many … as" without
# a direction anchor, "compared to …", "in comparison with …",
# "the same … as". These have no entry in COMPARE_*_ANCHORS — admitting
# them would breach the round-trip filter's verb-registry check anyway.
# Comparative entity slot: proper-noun, "the X" collective, "the number/amount
# of <noun>" mass-noun construction. Possessives ("Bob's"/"his") are deferred.
_COMPARE_REF: Final[str] = (
r"(?:"
r"the\s+(?:number|amount)\s+of\s+\w+"
r"|[Tt]he\s+\w+"
r"|[A-Z]\w+"
r")"
)
def _resolve_reference_token(raw: str) -> tuple[str, str]:
"""Return ``(canonical_entity, head_token_for_grounding)``.
For "the number of chickens" the head token is "chickens"; the
canonical entity uses the full noun-phrase so binding-graph
referential-integrity isn't subverted by collapsing different
references to the same noun.
"""
collapsed = re.sub(r"\s+", " ", raw.strip())
lowered = collapsed.lower()
if lowered.startswith("the number of ") or lowered.startswith("the amount of "):
head = collapsed.split()[-1]
return collapsed, head
if lowered.startswith("the "):
head = collapsed[4:].split()[0]
return "the " + collapsed[4:], head
return collapsed, collapsed
def _comparison_anchor_verb() -> str:
# 'has' / 'have' carry the comparator phrase. We don't include 'had/gets'
# etc. in P2 — past-tense + lemma-widening are deferred to a later axis
# to keep the precedence story narrow.
return r"(?:has|have)"
_COMPARE_ADDITIVE_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<actor>{_ENTITY})\s+{_comparison_anchor_verb()}\s+"
rf"(?P<value>{_VALUE})\s+"
r"(?P<direction>more|fewer|less)\s+"
r"(?P<unit>\w+)\s+than\s+"
rf"(?P<reference>{_COMPARE_REF})\s*\.?$"
)
# Multiplicative: anchor-as-value form ("twice"/"thrice"/"half" carry the
# factor implicitly). "as many <unit>" required; unit ellipsis ("twice as
# many as Bob") is deferred to keep wrong==0 strict — without unit the
# binding graph cannot disambiguate which dimension to compare.
_COMPARE_MULT_ANCHOR_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<actor>{_ENTITY})\s+{_comparison_anchor_verb()}\s+"
r"(?P<anchor>twice|thrice|half)\s+as\s+many\s+"
r"(?P<unit>\w+)\s+as\s+"
rf"(?P<reference>{_COMPARE_REF})\s*\.?$"
)
# Multiplicative: explicit "N times as many <unit> as <REF>".
_COMPARE_MULT_NTIMES_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<actor>{_ENTITY})\s+{_comparison_anchor_verb()}\s+"
rf"(?P<value>{_VALUE})\s+times\s+as\s+many\s+"
r"(?P<unit>\w+)\s+as\s+"
rf"(?P<reference>{_COMPARE_REF})\s*\.?$"
)
# Nested: additive over multiplicative — "A has N more <unit> than M times <REF>".
# Emits *two* flat candidates so the binding graph (ADR-0134) can decide which
# admissible composition (if any) survives. The parser does not commit to a
# nested operand shape (Comparison ∋ Comparison is not a supported operand
# type today); composition admissibility is the round-trip layer's call.
_COMPARE_NESTED_RE: Final[re.Pattern[str]] = re.compile(
rf"^(?P<actor>{_ENTITY})\s+{_comparison_anchor_verb()}\s+"
rf"(?P<delta_value>{_VALUE})\s+"
r"(?P<direction>more|fewer|less)\s+"
r"(?P<unit>\w+)\s+than\s+"
rf"(?P<factor_value>{_VALUE})\s+times\s+"
rf"(?P<reference>{_COMPARE_REF})\s*\.?$"
)
_ANCHOR_TO_FACTOR: Final[dict[str, tuple[float, str]]] = {
# surface anchor → (factor, direction-literal)
"twice": (2.0, "times"),
"thrice": (3.0, "times"),
"half": (0.5, "fraction"),
}
def _direction_to_anchor(direction_raw: str) -> tuple[str, str]:
"""Map surface direction word → (canonical Comparison.direction,
matched_verb registered in COMPARE_ADDITIVE_ANCHORS).
'less' is a surface synonym of 'fewer'; the Comparison value uses
direction='fewer', but matched_verb retains the source surface
('less') so the round-trip filter's "verb appears in source" check
succeeds. 'less' is registered in COMPARE_ADDITIVE_ANCHORS, so the
verb-registry check also succeeds.
"""
lowered = direction_raw.lower()
if lowered in ("more", "fewer"):
return lowered, lowered
if lowered == "less":
return "fewer", "less"
raise ValueError(f"unknown comparative direction surface {direction_raw!r}")
def _build_compare_additive(
*,
actor_raw: str,
delta_value_raw: str,
direction_raw: str,
unit_raw: str,
reference_raw: str,
source: str,
) -> CandidateOperation | None:
if _is_indefinite_quantifier(delta_value_raw):
return None
direction, matched_verb = _direction_to_anchor(direction_raw)
actor = _normalize_entity(actor_raw)
reference_canon, reference_head = _resolve_reference_token(reference_raw)
if reference_canon == actor:
return None # self-reference; constructor would refuse anyway
delta_value = _resolve_value(delta_value_raw)
unit = _canonicalize_unit(unit_raw)
try:
op = Operation(
actor=actor,
kind="compare_additive",
operand=Comparison(
reference_actor=reference_canon,
delta=Quantity(value=delta_value, unit=unit),
factor=None,
direction=cast(_CompDirection, direction),
),
)
except Exception:
return None
try:
return CandidateOperation(
op=op,
source_span=source,
matched_verb=matched_verb,
matched_value_token=delta_value_raw,
matched_unit_token=unit_raw,
matched_actor_token=actor_raw,
matched_reference_actor_token=reference_head,
)
except Exception:
return None
def _build_compare_multiplicative(
*,
actor_raw: str,
factor: float,
matched_verb: str,
matched_value_token: str,
unit_raw: str,
reference_raw: str,
source: str,
direction: str,
) -> CandidateOperation | None:
actor = _normalize_entity(actor_raw)
reference_canon, reference_head = _resolve_reference_token(reference_raw)
if reference_canon == actor:
return None
_ = _canonicalize_unit(unit_raw) # validation only; multiplicative compares
# carry the unit on the source-span side, not the operand
try:
op = Operation(
actor=actor,
kind="compare_multiplicative",
operand=Comparison(
reference_actor=reference_canon,
delta=None,
factor=factor,
direction=cast(_CompDirection, direction),
),
)
except Exception:
return None
try:
return CandidateOperation(
op=op,
source_span=source,
matched_verb=matched_verb,
matched_value_token=matched_value_token,
matched_unit_token=unit_raw,
matched_actor_token=actor_raw,
matched_reference_actor_token=reference_head,
)
except Exception:
return None
def _compare_additive_candidates(sentence: str) -> list[CandidateOperation]:
s = sentence.strip()
m = _COMPARE_ADDITIVE_RE.match(s)
if m is None:
return []
cand = _build_compare_additive(
actor_raw=m.group("actor"),
delta_value_raw=m.group("value"),
direction_raw=m.group("direction"),
unit_raw=m.group("unit"),
reference_raw=m.group("reference"),
source=sentence,
)
return [cand] if cand is not None else []
def _compare_multiplicative_candidates(sentence: str) -> list[CandidateOperation]:
s = sentence.strip()
out: list[CandidateOperation] = []
m = _COMPARE_MULT_ANCHOR_RE.match(s)
if m is not None:
anchor = m.group("anchor").lower()
factor, direction = _ANCHOR_TO_FACTOR[anchor]
cand = _build_compare_multiplicative(
actor_raw=m.group("actor"),
factor=factor,
matched_verb=anchor,
matched_value_token=anchor, # anchor-as-value (math_roundtrip step 4)
unit_raw=m.group("unit"),
reference_raw=m.group("reference"),
source=sentence,
direction=direction,
)
if cand is not None:
out.append(cand)
return out # specificity — don't also try N-times pattern
m = _COMPARE_MULT_NTIMES_RE.match(s)
if m is not None:
value_raw = m.group("value")
if _is_indefinite_quantifier(value_raw):
return out
try:
factor = float(_resolve_value(value_raw))
except KeyError:
return out
cand = _build_compare_multiplicative(
actor_raw=m.group("actor"),
factor=factor,
matched_verb="times",
matched_value_token=value_raw,
unit_raw=m.group("unit"),
reference_raw=m.group("reference"),
source=sentence,
direction="times",
)
if cand is not None:
out.append(cand)
return out
def _compare_nested_candidates(sentence: str) -> list[CandidateOperation]:
"""Emit two flat candidates for nested 'N more <unit> than M times <REF>'.
The parser does not commit to a composed Comparison-of-Comparison
operand (operand type Comparison ∋ Comparison is not modelled
today). Both flat candidates are forwarded; the binding-graph /
round-trip layer (ADR-0134) picks an admissible composition or
refuses. Refusal is the safe outcome — never a wrong answer.
"""
s = sentence.strip()
m = _COMPARE_NESTED_RE.match(s)
if m is None:
return []
out: list[CandidateOperation] = []
actor_raw = m.group("actor")
unit_raw = m.group("unit")
reference_raw = m.group("reference")
# Candidate 1: additive — "A has N more <unit> than <REF>" treating
# <REF> as the comparison reference directly. The "M times" multiplier
# is dropped on this candidate (the binding-graph composition is
# what would re-introduce it).
add_cand = _build_compare_additive(
actor_raw=actor_raw,
delta_value_raw=m.group("delta_value"),
direction_raw=m.group("direction"),
unit_raw=unit_raw,
reference_raw=reference_raw,
source=sentence,
)
if add_cand is not None:
out.append(add_cand)
# Candidate 2: multiplicative — "A has M times as many <unit> as <REF>"
# treating the multiplier M and the same <REF> as the multiplicative
# comparison. The additive offset N is dropped on this candidate.
factor_value_raw = m.group("factor_value")
if not _is_indefinite_quantifier(factor_value_raw):
try:
factor = float(_resolve_value(factor_value_raw))
except KeyError:
factor = None
if factor is not None:
mult_cand = _build_compare_multiplicative(
actor_raw=actor_raw,
factor=factor,
matched_verb="times",
matched_value_token=factor_value_raw,
unit_raw=unit_raw,
reference_raw=reference_raw,
source=sentence,
direction="times",
)
if mult_cand is not None:
out.append(mult_cand)
return out

275
tests/test_adr_0131_G2_comparatives.py Normal file
View file

@ -0,0 +1,275 @@
"""ADR-0131.G.2 — comparative operations (additive + multiplicative).
Pins the curated coverage axis under
``evals/math_capability_axes/G2_comparatives/v1/`` plus the
``math_candidate_parser`` comparative extractors.
"""
from __future__ import annotations
import json
import re
from pathlib import Path
import pytest
from evals.math_capability_axes.G2_comparatives.v1.runner import (
build_report,
write_report,
_REPORT_PATH,
)
from generate.math_candidate_parser import (
_compare_additive_candidates,
_compare_multiplicative_candidates,
_compare_nested_candidates,
extract_operation_candidates,
)
from generate.math_roundtrip import (
COMPARE_ADDITIVE_ANCHORS,
COMPARE_MULTIPLICATIVE_ANCHORS,
roundtrip_admissible,
)
_REPO = Path(__file__).resolve().parents[1]
_GSM8K_LEGACY_REPORT = (
_REPO / "evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json"
)
_GSM8K_CG_REPORT = _REPO / "evals/gsm8k_math/train_sample/v1/report.json"
# ---------------------------------------------------------------------------
# Per-direction at-least-one-passing.
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("direction,sentence,actor,reference,delta", [
("more", "Alice has 3 more apples than Bob", "Alice", "Bob", 3),
("fewer", "Alice has 5 fewer pencils than Bob", "Alice", "Bob", 5),
])
def test_additive_direction_admits(direction, sentence, actor, reference, delta):
cands = [c for c in _compare_additive_candidates(sentence) if roundtrip_admissible(c)]
assert len(cands) == 1
c = cands[0]
assert c.op.kind == "compare_additive"
assert c.op.operand.direction == direction
assert c.op.actor == actor
assert c.op.operand.reference_actor == reference
assert c.op.operand.delta.value == delta
def test_additive_less_maps_to_fewer():
cands = [c for c in _compare_additive_candidates("Dana has 8 less coins than Eli") if roundtrip_admissible(c)]
assert len(cands) == 1
c = cands[0]
assert c.op.operand.direction == "fewer"
assert c.matched_verb == "less"
# 'less' is a registered additive anchor (COMPARE_ADDITIVE_ANCHORS).
assert "less" in COMPARE_ADDITIVE_ANCHORS
@pytest.mark.parametrize("direction,sentence,actor,reference,factor,anchor", [
("times", "Alice has twice as many apples as Bob", "Alice", "Bob", 2.0, "twice"),
("times", "Alice has 4 times as many apples as Bob", "Alice", "Bob", 4.0, "times"),
("times", "Alice has thrice as many apples as Bob", "Alice", "Bob", 3.0, "thrice"),
("fraction", "Alice has half as many apples as Bob", "Alice", "Bob", 0.5, "half"),
])
def test_multiplicative_direction_admits(direction, sentence, actor, reference, factor, anchor):
cands = [c for c in _compare_multiplicative_candidates(sentence) if roundtrip_admissible(c)]
assert len(cands) == 1
c = cands[0]
assert c.op.kind == "compare_multiplicative"
assert c.op.operand.direction == direction
assert c.op.actor == actor
assert c.op.operand.reference_actor == reference
assert c.op.operand.factor == factor
assert c.matched_verb == anchor
assert anchor in COMPARE_MULTIPLICATIVE_ANCHORS
# ---------------------------------------------------------------------------
# Nested composition emits BOTH flat candidates.
# ---------------------------------------------------------------------------
def test_nested_emits_both_flat_candidates():
s = "Jen has 10 more ducks than four times the number of chickens"
cands = [c for c in _compare_nested_candidates(s) if roundtrip_admissible(c)]
kinds = {c.op.kind for c in cands}
assert kinds == {"compare_additive", "compare_multiplicative"}
# ---------------------------------------------------------------------------
# Refusal cases — closed-set boundary holds.
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("sentence", [
"Alice has as many apples as Bob",
"Compared to Bob, Alice has 3 more apples",
"In comparison with Bob, Alice has 3 more apples",
"Alice has the same number of apples as Bob",
"Alice has 3 times more apples than Bob",
])
def test_refusal_cases_emit_no_admitted_comparative(sentence):
cands = extract_operation_candidates(sentence)
admitted_comparatives = [
c for c in cands
if c.op.kind in ("compare_additive", "compare_multiplicative")
and roundtrip_admissible(c)
]
assert admitted_comparatives == []
# ---------------------------------------------------------------------------
# Closed-set anchor discipline — emitter never invents a direction literal.
# ---------------------------------------------------------------------------
def test_direction_literals_closed_set():
sentences = [
"Alice has 3 more apples than Bob",
"Alice has 5 fewer pencils than Bob",
"Alice has 2 less books than Bob",
"Alice has twice as many apples as Bob",
"Alice has 4 times as many apples as Bob",
"Alice has half as many apples as Bob",
"Alice has thrice as many apples as Bob",
"Jen has 10 more ducks than four times the number of chickens",
]
for s in sentences:
for c in extract_operation_candidates(s):
if c.op.kind in ("compare_additive", "compare_multiplicative"):
assert c.op.operand.direction in ("more", "fewer", "times", "fraction")
# ---------------------------------------------------------------------------
# Runner / report contract.
# ---------------------------------------------------------------------------
def test_runner_wrong_count_is_zero():
report = build_report()
assert report["metrics"]["wrong"] == 0
assert report["metrics"]["wrong_count_is_zero"] is True
def test_runner_per_category_minima():
"""Brief §coverage: ≥4 per direction (more, fewer, times, fraction);
≥3 nested; ≥4 refusal. We pin direction-level minima via per-case scan
rather than per-category (additive merges more+fewer into one bucket
in per_category but we want to verify each direction independently)."""
report = build_report()
direction_counts = {"more": 0, "fewer": 0, "times": 0, "fraction": 0}
nested = 0
refusal = 0
cases_path = _REPO / "evals/math_capability_axes/G2_comparatives/v1/cases.jsonl"
for line in cases_path.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
c = json.loads(line)
if c["category"] in ("additive", "multiplicative"):
direction_counts[c["direction"]] += 1
elif c["category"] == "nested":
nested += 1
elif c["category"] == "refusal":
refusal += 1
for d, n in direction_counts.items():
assert n >= 4, f"direction {d!r} has only {n} cases (need ≥4)"
assert nested >= 3
assert refusal >= 4
# And every one of those cases passes (wrong==0 already asserts no
# wrong-shaped pass, but verify the positive direction too).
assert report["metrics"]["passed"] == report["metrics"]["cases_total"]
def test_report_byte_equal_across_runs():
a = json.dumps(build_report(), indent=2, sort_keys=True)
b = json.dumps(build_report(), indent=2, sort_keys=True)
assert a == b
def test_committed_report_matches_runner_output():
report = build_report()
written = json.dumps(report, indent=2, sort_keys=True) + "\n"
on_disk = _REPORT_PATH.read_text(encoding="utf-8")
if written != on_disk:
# Re-write to be helpful in local dev (so CI failures point at the
# actual delta, not a stale file).
write_report(report)
assert written == on_disk, "G2 report.json is stale — re-run runner.py"
# ---------------------------------------------------------------------------
# GSM8K probe gate — chosen gate (per ADR-0131.G.2):
# comparative-clause refusals in the candidate-graph probe report.json
# strictly decrease.
# Legacy train_sample_coverage_report.json is byte-identical (no legacy
# parser change); we pin both invariants below.
# ---------------------------------------------------------------------------
_COMPARATIVE_STATEMENT_PATTERNS = (
# "<N> more <unit> than" / "fewer / less"
re.compile(r"\b(?:more|fewer|less)\b[^.'\"]*\bthan\b", re.IGNORECASE),
# "twice|thrice|half|N times as many <unit> as"
re.compile(r"\b(?:twice|thrice|half|\d+\s+times)\s+as\s+many\b", re.IGNORECASE),
)
def _comparative_clause_refusal_count(probe_report_path: Path) -> int:
"""Count refused cases whose refusal reason names a statement clause
that itself contains a comparative anchor (additive ``more/fewer/less
… than`` or multiplicative ``twice/N times/half as many``). We anchor
on the embedded statement clause inside the ``candidate_graph`` /
``parser`` refusal reason, not on isolated keywords, to avoid
false-positives on incidental mentions ('he is more careful than',
discourse glue, etc.)."""
data = json.loads(probe_report_path.read_text(encoding="utf-8"))
per_case = data["per_case"]
count = 0
for d in per_case:
if d.get("verdict", d.get("outcome")) != "refused":
continue
reason = d["reason"]
if "statement" not in reason and "statement clause" not in reason:
continue
for pat in _COMPARATIVE_STATEMENT_PATTERNS:
if pat.search(reason):
count += 1
break
return count
def test_gsm8k_legacy_probe_safety_rail_intact():
"""ADR-0131.G invariant: legacy probe still shows admitted_wrong == 0."""
data = json.loads(_GSM8K_LEGACY_REPORT.read_text(encoding="utf-8"))
assert data["metrics"]["admitted_wrong"] == 0
assert data["metrics"]["safety_rail_intact"] is True
def test_gsm8k_candidate_graph_probe_wrong_zero():
"""Candidate-graph probe also preserves wrong == 0."""
data = json.loads(_GSM8K_CG_REPORT.read_text(encoding="utf-8"))
assert data["counts"]["wrong"] == 0
def test_gsm8k_candidate_graph_comparative_clause_refusals_decreased():
"""G2 gate: at least one previously-comparative-clause-refused case
no longer cites a comparative-clause refusal in the candidate-graph
probe. Baseline (pre-G.2) for the train sample had case 0009 ('Jen
has 10 more ducks than four times the number of chickens') refused
at the comparative statement; with G.2 wired into
math_candidate_parser, that comparative clause now parses (the case
may still refuse downstream — at the question or solver — but the
refusal reason no longer cites the comparative clause)."""
current_count = _comparative_clause_refusal_count(_GSM8K_CG_REPORT)
# Pre-G.2 baseline (`origin/main` @ 481e0c3): 2 cases refused at a
# statement clause whose text contains additive ('more/fewer/less ...
# than') or multiplicative ('twice/N times/half as many') anchors —
# cases gsm8k-train-sample-v1-0009 ('Jen has 10 more ducks than four
# times the number of chickens.') and -0016 (arithmetic 'more than'
# in 'Rudolph traveled 2 more than 5 miles'). G.2 wires comparatives
# into math_candidate_parser; case 0009 now parses the comparative
# clause (refusal moves to the question form), so the count drops to
# 1 (case 0016's arithmetic 'N more than M' is *not* a between-entity
# comparative and is deliberately out of G.2 scope).
baseline_count = 2
assert current_count < baseline_count, (
f"expected comparative-clause refusal count to drop below "
f"baseline {baseline_count}; got {current_count}"
)