core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/decisions/ADR-0189-comparative-verb-unit-widening.md
Shay 0770648257
feat(GSM8K): comprehension reading → first metric move 3/47/0 → 4/46/0 (#488) 
* feat(adr-0189): comparative reading — anchor-verb widening + multi-word units

The candidate-graph comparative extractor (ADR-0131.G.2) read only has/have +
single-word units, so real-GSM8K comparatives ('Brooke does three times as many
jumping jacks as Sidney') didn't parse — a dark statement in 17 places blocking
15 of the 47 refused train_sample cases, despite the ADR-0123 solver already
supporting compare_additive/compare_multiplicative.

Widens the anchor-verb set (reusing legacy vetted lemmas + does/collected/
gained/studied…), EXCLUDING polarity-inverting verbs (lose/spend/give/sell/win)
to preserve wrong=0; admits 1-2 word units via the existing multi-word
_unit_grounds branch. Feeds the existing solver unchanged.

wrong=0 proven: G2_comparatives 29/29, G3 20/0, G4 32/32, train_sample 3/47/0
byte-identical; polarity-inverting verbs proven refused (failing-under-violation).
Chain composes correctly in isolation (146 -> 438). Flips 0 cases ALONE — every
comparative case needs a composing partner (aggregation / multi-word-noun
injection); this ships the component, not yet a flip.

- generate/math_candidate_parser.py: _comparison_anchor_verb widening + 1-2 word
  unit slots in the two multiplicative comparative regexes.
- tests/test_adr_0131_G2a_*: 5 tests incl. polarity-inversion wrong=0 guards.
- docs/decisions/ADR-0189: gap, change, wrong=0 evidence, honest scope.

* feat(adr-0189a): first metric move 3/47/0 -> 4/46/0 (case 0024, comprehension-composed)

Case 0024 now SOLVES (answer 438) by composing three general comprehension
capabilities feeding the unchanged ADR-0123 solver:
  1. day-of-week count enumeration: Sidney = 20+36+40+50 = 146
     (_day_enumeration_candidates; derived sum grounds via first count token,
      mirroring _embedded_quantifier; closed to the 7 day names)
  2. comparative reading (ADR-0189): Brooke = 3 x Sidney
  3. activity question 'How many <unit> did <Entity> <verb>?' (_Q_DID_RE)
Plus do/does/did added to the CandidateInitial anchor whitelist (production-
possession), admitted only via the closed day-enumeration shape.

wrong=0 PROVEN across every lane: all 8 capability axes wrong=0 (G2_comparatives
29/29, G3 20/0, G4 32/32, G5/S1/S3/S4 all pass), train_sample 4/46/0 wrong=0,
verify_lane_shas exit 0 (no pinned lane changed), generate_claims --check OK.
872 tests pass; new tests are failing-under-violation incl. wrong=0 guards
(non-day comma list not summed; polarity-inverting comparative verbs refused).

Re-baselined report.json + train_sample_coverage_report.json (latter also clears
pre-existing reason drift) + CLAIMS.md to the new 4/46/0 metric. Decode-not-guess:
0024 solved by READING its structure, not storing an answer. Remaining pre-existing
failures (G3 committed-report, telemetry) unrelated, fail on pristine main.

- generate/math_candidate_parser.py: day-enum extractor + _Q_DID_RE + does-anchor.
- tests/test_adr_0189a_day_enum_activity.py: 5 tests (incl. end-to-end 0024=438).
- docs/decisions/ADR-0189a + report.json/coverage/CLAIMS re-baseline.
2026-05-30 09:21:48 -07:00

4 KiB
Raw Blame History

ADR-0189 — Comparative reading: anchor-verb widening + multi-word units

Status: Proposed (implemented in this PR). Extends ADR-0131.G.2 (candidate-graph comparative extractors). Serving-path capability; landed wrong=0-proven.

One line. The candidate-graph comparative extractor read only has/have

  • single-word units, so real-GSM8K comparatives (Brooke **does** three times as many **jumping jacks** as Sidney) didn't parse. This widens the anchor-verb set (excluding polarity-inverting verbs) and admits 1–2-word units. It feeds the existing ADR-0123 compare_multiplicative/compare_additive solver; wrong=0 is preserved (G2_comparatives 29/29; train_sample 3/47/0 byte-identical).

1. The gap (microscope finding, 2026-05-29)

The candidate-graph parser emits no comparative operation for the most common real-GSM8K comparative surfaces, even though the solver already supports them (ADR-0123 compare_additive/compare_multiplicative, fully wired in math_solver). Measured: comparatives are a dark statement in 17 places, blocking 15 of the 47 refused train_sample cases.

Two extractor limits caused the misses (_comparison_anchor_verb() was (?:has|have) and units were \w+):

  • Verb coverage. does/collected/gained/studied/makes/… were not recognized. (ADR-0131.G.2 deferred this explicitly: "past-tense + lemma-widening are deferred … to keep the precedence story narrow" — a scope limit, not a wrong=0 constraint.)
  • Multi-word units. jumping jacks (case 0024) could not match.

2. The change

  • generate/math_candidate_parser._comparison_anchor_verb() — widen to the already-vetted legacy math_parser._COMPARE_VERB lemmas plus the production/activity verbs seen in real GSM8K comparatives. Deliberately EXCLUDES polarity-inverting verbs (lose/lost, win/won, spend/spent, use/used, give/gave, sell/sold) — admitting them could read a comparison backwards and breach wrong=0.
  • The two multiplicative regexes admit an optional second unit word (\w+(?:\s+\w+)?), grounded via the existing multi-word branch of _unit_grounds.

3. wrong=0 evidence

  • G2_comparatives lane: 29/29, wrong=0 (the dedicated comparative lane).
  • G3_numerics 20 correct / 0 wrong; G4_multi_clause 32/32; train_sample 3/47/0 byte-identical (no flip, no wrong).
  • Polarity-inverting verbs proven refused (test_…_polarity_inverting_verb_not_admitted, test_spend_verb_not_admitted) — failing-under-violation.
  • The round-trip filter is unchanged: the comparator anchor (twice/N times/ more/fewer) and the reference actor still must ground.

4. Honest scope — a necessary component, not a standalone solve

This widening flips zero train_sample cases by itself, because every comparative-blocked case also needs a composing partner the reader still lacks (e.g. case 0024 needs Sidney's 20+36+40+50=146 day-of-week aggregation and multi-word counted-noun injection before Brooke = 3 × 146 = 438 can solve). The comparative + question chain is proven to compose correctly in isolation (Sidney has 146 apples. Brooke has three times as many apples as Sidney. → 438). This ADR ships the comparative component; the first metric flip (3/47/0 → 4/46/0) lands when its companion capability (aggregation / multi-word-noun injection) lands for a shared target case.

5. Why this obeys the standing principles

  • Decode, don't guess. The engine could already reason about comparison (ADR-0123 solver); this teaches it to read the comparison — closing a comprehension gap, not adding a stored answer.
  • General, not overfit. A widened closed verb set + multi-word units is a fundamental construction across 15 cases, not a per-case surface regex.
  • wrong=0 > coverage. Polarity-inverting verbs excluded; G2 lane 29/29; serving 3/47/0 byte-identical; round-trip filter untouched.
  • No contradiction with in-use ADRs: extends ADR-0131.G.2's extractor and feeds the ADR-0123 solver unchanged.