Merge pull request #161 from AssetOverflow/feat/adr-0126-candidate-graph
feat(ADR-0126): candidate-graph parser + round-trip verifier-filter
This commit is contained in:
Shay
GitHub
commit
8851067982
13 changed files with 3312 additions and 52 deletions
259
docs/decisions/ADR-0126-candidate-graph-parser.md
Normal file
259
docs/decisions/ADR-0126-candidate-graph-parser.md
Normal file
|
|
@ -0,0 +1,259 @@
|
|||
# ADR-0126 — Candidate-Graph Parser with Round-Trip Verifier-Filter
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-23
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:**
|
||||
- ADR-0115 (parser substrate),
|
||||
- ADR-0116 (deterministic solver),
|
||||
- ADR-0117 (`SolutionTrace` verifier),
|
||||
- ADR-0118 (stepped realizer),
|
||||
- ADR-0119 (+ all 8 sub-phases — GSM8K-math lane substrate),
|
||||
- ADR-0119.7 (sealed GSM8K test),
|
||||
- ADR-0120 (expert promotion contract),
|
||||
- ADR-0121 (math expert promotion DEFERRED with named blocker
|
||||
`correct_rate = 0/1319` on sealed holdout).
|
||||
**Supersedes:** ADR-0123b (`feat/adr-0123b-upstream-shape-gaps`,
|
||||
commit `8c4070e`, never opened as PR). Substrate of ADR-0122 /
|
||||
ADR-0123 / ADR-0123a remains in main and is consumed by this ADR.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0121 deferred the `mathematics_logic` → `expert` promotion with
|
||||
the named blocker `correct_rate = 0/1319 on sealed GSM8K holdout`
|
||||
and proposed a parser-expansion arc of 4–8 construction classes.
|
||||
Three ADRs of that arc landed (ADR-0122 rate / per-unit, ADR-0123
|
||||
comparison-phrasing, ADR-0123a comparison-shape-expansion) and a
|
||||
fourth was prepared (ADR-0123b upstream-shape-gaps). All four:
|
||||
|
||||
- preserved every invariant (`wrong == 0`, byte-equal trace,
|
||||
pack-binding, replay determinism),
|
||||
- passed their author-drafted dev-set exit criteria,
|
||||
- **measured 0/1319 sealed-holdout lift** (Gemini Tasks 6 and 8).
|
||||
|
||||
Two independent external assessments (Kimi K2.6, Grok 4.2) and an
|
||||
internal review converged on the same diagnosis:
|
||||
|
||||
1. **The dev set is hermetically sealed from the real GSM8K
|
||||
distribution.** It was authored by the people who authored the
|
||||
parser, against the parser's grammar. Dev-set success is
|
||||
uncorrelated (in expectation, anti-correlated) with sealed-holdout
|
||||
lift.
|
||||
|
||||
2. **Per-axis vocabulary expansion is structurally optimistic.**
|
||||
Each sealed case lives in the *intersection* of grammar gaps
|
||||
(verb × number-word × question-shape × ellipsis × multi-step),
|
||||
not the union. If each gap has independent coverage `p`, joint
|
||||
pass rate is `p^k`. Adding one verb raises `p` infinitesimally;
|
||||
joint pass rate barely moves.
|
||||
|
||||
3. **The current parser fails hard at the first unmatched
|
||||
construction.** A single brittle regex (e.g., an ambiguous verb
|
||||
match that steals the slot from a correct one) refuses an entire
|
||||
problem with `ParseError`. This makes vocabulary expansion
|
||||
adversarial against itself — adding new patterns can *lose*
|
||||
coverage on already-passing problems.
|
||||
|
||||
The architectural error is the *topology* of failure, not the
|
||||
*size* of vocabulary. Four zero-lift ADRs are the empirical
|
||||
evidence that more grammar in the same shape will not move the
|
||||
sealed-holdout score.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Replace the **first-match-wins, single-graph, fail-hard** parser
|
||||
topology with a **candidate-graph + verifier-filter** topology
|
||||
that preserves every existing invariant while converting
|
||||
compound-gap failure from multiplicative (`p^k`) to parallel
|
||||
(`1 - (1-p)^k`).
|
||||
|
||||
### Pipeline change
|
||||
|
||||
```
|
||||
OLD: surface → ONE graph → solve → verify(trace_replay)
|
||||
NEW: surface → [graph_1, graph_2, … graph_n] → solve each
|
||||
→ round-trip filter → admissibility filter → decision rule
|
||||
```
|
||||
|
||||
### Per-sentence change
|
||||
|
||||
Each sentence emits `list[CandidateOperation]` (possibly empty)
|
||||
instead of one `Operation` or `ParseError`. Multiple matching
|
||||
patterns no longer race for a single slot; they all emit
|
||||
candidates and the verifier filters wrong ones downstream.
|
||||
|
||||
Hard bound: `MAX_CANDIDATES_PER_SENTENCE = 4`. Exceeding raises
|
||||
`ParseError` (preserves determinism cost ceiling).
|
||||
|
||||
### Graph assembly
|
||||
|
||||
Cartesian product of per-sentence candidate lists, enumerated
|
||||
depth-first in canonical pattern-priority order
|
||||
(most-specific-first). Hard bound: `MAX_TOTAL_BRANCHES = 64`.
|
||||
Exceeding refuses the entire problem (preserves runtime ceiling).
|
||||
|
||||
### Round-trip filter (load-bearing new invariant)
|
||||
|
||||
A candidate operation is *round-trip admissible* iff
|
||||
reconstructing the source sentence from its parsed slots produces
|
||||
a string byte-equal to the original sentence under normalization
|
||||
N (lowercase + collapse whitespace + strip terminal punctuation).
|
||||
|
||||
Formally, for source sentence `s` and candidate operation `op`:
|
||||
|
||||
```
|
||||
admissible(op, s) ⟺ N(reconstruct(op)) == N(s)
|
||||
```
|
||||
|
||||
This is the **wrong-answer firewall**. A regex that misreads
|
||||
"gives" as `subtract` instead of `transfer` will reconstruct a
|
||||
sentence that doesn't byte-equal the source; the candidate is
|
||||
rejected before it can produce a wrong number.
|
||||
|
||||
### Decision rule (final emit)
|
||||
|
||||
For the set `A` of admissible candidate graphs:
|
||||
|
||||
| `|A|` | answer distribution | decision |
|
||||
|-------|--------------------|----------|
|
||||
| 0 | — | **refuse** (preserves `wrong == 0`) |
|
||||
| 1 | — | **emit** the single answer |
|
||||
| ≥2 | all identical | **emit** the common answer |
|
||||
| ≥2 | non-identical | **refuse** (genuine ambiguity) |
|
||||
|
||||
### Invariant preservation
|
||||
|
||||
| Invariant | How preserved |
|
||||
|-----------|---------------|
|
||||
| `wrong == 0` | Round-trip filter rejects mis-parses; ambiguity refuses. |
|
||||
| Trace determinism | Candidate enumeration canonical (lexical pattern key); branch order canonical (depth-first by sentence). |
|
||||
| `trace_hash` byte-equality | Selected graph alone enters `SolutionTrace`; non-selected branches are not hashed. |
|
||||
| Pack-binding | Candidate generators consume the same pack lemmas as today. |
|
||||
| Replay equivalence | Replay re-runs candidate enumeration + filter; same input → same selected graph. |
|
||||
| Adversarial `wrong == 0` | Adversarial suite gate runs unchanged; round-trip filter is *stricter* than today's gates. |
|
||||
|
||||
### New invariant added
|
||||
|
||||
**Round-trip admissibility:** any operation that emits in a final
|
||||
`SolutionTrace` must satisfy `N(reconstruct(op)) == N(source_span)`
|
||||
for the source span it claims to parse. This is a stricter
|
||||
contract than `trace_hash` byte-equality across runs (which only
|
||||
guards determinism) — it guards against semantic mis-parse.
|
||||
|
||||
---
|
||||
|
||||
## Exit criterion
|
||||
|
||||
**Inner-loop signal (new):** draw 50 cases from GSM8K *train*
|
||||
split (deterministic seed, committed unsealed at
|
||||
`evals/gsm8k_math/train_sample/v1/cases.jsonl`). Run candidate-graph
|
||||
pipeline. The architecture is validated iff:
|
||||
|
||||
```
|
||||
correct ≥ 10 / 50 (20% absolute)
|
||||
wrong == 0
|
||||
```
|
||||
|
||||
This is the first non-synthetic gradient signal in the GSM8K
|
||||
lane. If the architecture is structurally sound, ≥20% lift on a
|
||||
random sample is achievable without any new vocabulary work — the
|
||||
gain comes purely from converting fail-hard to filter-and-pick.
|
||||
|
||||
**Path-B trigger:** if 50-case train sample shows `correct < 10`
|
||||
after the candidate-graph pipeline is integrated, the parser-by-rule
|
||||
architecture (in any topology) is the wrong abstraction for GSM8K
|
||||
coverage. ADR-0126 itself is deferred and the work pivots to
|
||||
benchmark re-selection (see "Alternatives considered → Path B"
|
||||
below).
|
||||
|
||||
**Sealed-holdout protocol unchanged:** if the train-sample gate
|
||||
passes, the sealed holdout runs exactly once and the number is
|
||||
frozen in ADR-0126-results. The sealed `.age` ciphertext is never
|
||||
modified.
|
||||
|
||||
---
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### Path A.1 — More vocabulary (status-quo continuation)
|
||||
|
||||
Continue the ADR-0122/0123/0123a/0123b cadence: catalog refusal
|
||||
modes, add verbs/numbers/patterns, ship per-axis ADRs.
|
||||
|
||||
**Rejected** because four consecutive ADRs in this shape produced
|
||||
0/1319 lift. The compound-gap math (`p^k`) explains why; no
|
||||
amount of `p` improvement at the rate we can produce ADRs moves
|
||||
joint pass rate meaningfully.
|
||||
|
||||
### Path B — Benchmark re-selection
|
||||
|
||||
Demote GSM8K to a stress test we honestly refuse on; re-target
|
||||
the math-expert promotion to a benchmark where exact-recall,
|
||||
determinism, and provenance are the discriminators (formal-math
|
||||
symbolic equivalence, CORE-native teaching-corpus eval, MATH
|
||||
symbolic subset).
|
||||
|
||||
**Held as fallback** triggered by the Path-B condition above. Not
|
||||
chosen first because it forecloses the GSM8K target without
|
||||
having tried the architectural fix that the compound-gap diagnosis
|
||||
actually implies.
|
||||
|
||||
### Path C — Learned (LLM-aided) parser
|
||||
|
||||
Replace the rule-based parser with an LLM-assisted candidate
|
||||
generator.
|
||||
|
||||
**Rejected** as a contract violation. ADR-0114a and the
|
||||
project-wide stance (CLAUDE.md "no opaque LLM fallbacks, no
|
||||
stochastic sampling, no hidden normalization") forbid this. The
|
||||
candidate-graph approach in this ADR is the deterministic
|
||||
analogue: parallel hypotheses with a verifier filter, no
|
||||
sampling, no learned scoring.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan
|
||||
|
||||
| Phase | Module | Description |
|
||||
|-------|--------|-------------|
|
||||
| P1 | `generate/math_roundtrip.py` (new) | Standalone `roundtrip_matches(op, source) -> bool`. Unit tests over every existing op kind. This is the load-bearing primitive; nothing else matters if it doesn't work. |
|
||||
| P2 | `generate/math_parser.py` (refactor) | `_process_statement` / `_process_question` return `list[CandidateOperation]`. Verb tables widened permissively. |
|
||||
| P3 | `generate/math_candidate_graph.py` (new) | Branch enumerator + filter + decision rule. |
|
||||
| P4 | `evals/gsm8k_math/runner.py` (wire) | Replace single-graph call with candidate-graph call. Preserve all current passing cases. |
|
||||
| P5 | `evals/gsm8k_math/train_sample/v1/cases.jsonl` (new) | 50-case deterministic train-split sample (unsealed). |
|
||||
| P6 | `evals/gsm8k_math/train_sample_runner.py` (new) | Run candidate-graph on train sample; emit `correct_rate`, `wrong_count`. |
|
||||
|
||||
Regression gates (must remain green at every phase):
|
||||
|
||||
- `core test --suite smoke`
|
||||
- `core test --suite math` (existing 507/507)
|
||||
- `core test --suite algebra` (82/82)
|
||||
- 150/150 public split
|
||||
- 200/200 dev set
|
||||
- adversarial suite (`wrong == 0`)
|
||||
|
||||
---
|
||||
|
||||
## PR checklist (when proposing for merge)
|
||||
|
||||
```
|
||||
What capability did this add?
|
||||
→ Candidate-graph parser topology + round-trip verifier-filter.
|
||||
What invariant proves the field remains valid?
|
||||
→ Round-trip admissibility (new); wrong == 0 (preserved);
|
||||
trace_hash byte-equality (preserved).
|
||||
Which CLI suite/eval proves the lane?
|
||||
→ smoke + math + algebra + 200/200 dev + train_sample_runner.
|
||||
Did this avoid hidden normalization, stochastic fallback,
|
||||
approximate recall, unreviewed mutation?
|
||||
→ Yes. Candidate enumeration is deterministic + bounded.
|
||||
Round-trip filter is deterministic byte comparison.
|
||||
No learned scoring, no sampling.
|
||||
If it touches user input, what trust boundary was enforced?
|
||||
→ No new user-input surfaces. Round-trip reconstruction does
|
||||
not echo unvalidated user content into logs.
|
||||
```
|
||||
409
docs/decisions/ADR-0127-units-pack-and-units-aware-parser.md
Normal file
409
docs/decisions/ADR-0127-units-pack-and-units-aware-parser.md
Normal file
|
|
@ -0,0 +1,409 @@
|
|||
# ADR-0127 — `en_units_v1` Pack + Units-Aware Candidate Extractors
|
||||
|
||||
**Status:** Proposed (scope-only; implementation follow-up to ADR-0126)
|
||||
**Date:** 2026-05-23
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:**
|
||||
- ADR-0115 / 0116 / 0117 / 0118 (parser / solver / verifier / realizer)
|
||||
- ADR-0122 (Rate operand)
|
||||
- ADR-0126 (candidate-graph parser + round-trip filter)
|
||||
**Supersedes:** none
|
||||
**Blocks:** the Path-B decision for the GSM8K-math lane (the real
|
||||
empirical answer can't be obtained until the parser consumes a
|
||||
units pack).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0126's first empirical run produced **0 / 50 correct, 0 wrong,
|
||||
50 refused** on the GSM8K train-sample inner-loop gate. Inspection
|
||||
of the refusal reasons revealed a pattern: every refusal happens at
|
||||
the *first statement* of each problem, and every refused first
|
||||
statement shares the same shape — **the parser fails on the
|
||||
unit-of-measurement construction, not on the operation grammar**.
|
||||
|
||||
Representative train-sample first statements (all refused):
|
||||
|
||||
| Case | First statement | What's structurally not recognized |
|
||||
|------|----------------|-----------------------------------|
|
||||
| 1 | `Tina makes $18.00 an hour.` | Money + time = rate dimension |
|
||||
| 2 | `Jan buys 1000 feet of cable.` | Length unit + substance qualifier |
|
||||
| 3 | `... bookstore donated 48 boxes of erasers.` | Container + count-content |
|
||||
| 5 | `In one hour, Addison mountain's temperature ...` | Temperature dimension + fractional ratio |
|
||||
|
||||
The shared failure mode is not "unknown verb" or "unknown sentence
|
||||
shape." It is "the parser has no ontology of units." The current
|
||||
parser asks "is `feet` a valid noun?" via `_canonical_unit` which
|
||||
just pluralizes. The right question is "what *dimension* is `feet`,
|
||||
and does `of cable` modify it as a substance qualifier?"
|
||||
|
||||
This is structurally different from the per-axis grammar treadmill
|
||||
that produced four zero-lift ADRs (0122 / 0123 / 0123a / 0123b).
|
||||
Vocabulary expansion is unbounded and adversarial; **units of
|
||||
measurement form a finite, externally well-defined ontology**
|
||||
(NIST SI tables for physical units; closed sets for currency, time,
|
||||
English container nouns). A units pack is *semantic substrate the
|
||||
parser consults*, not more grammar regex.
|
||||
|
||||
ADR-0126's candidate-graph topology is built to consume exactly
|
||||
this kind of substrate — the round-trip filter already does
|
||||
token-level grounding; adding a "matched_unit_token must resolve
|
||||
to a known dimension in the pack" check is a natural extension.
|
||||
|
||||
Without ADR-0127 the ADR-0126 empirical result is uninformative:
|
||||
0/50 reflects P2's deliberate minimum-viable scope, not the
|
||||
architecture's capacity. The architecture's real verdict lives
|
||||
behind the units pack.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add a `language_packs/data/en_units_v1/` ratified semantic pack and
|
||||
extend the ADR-0126 candidate parser to consult it during candidate
|
||||
emission. **No new operation kinds.** **No new ADR-0114a
|
||||
obligations.** **No new exit gates beyond ADR-0126's** (`correct ≥
|
||||
10/50, wrong == 0` on the train sample, this time with units pack
|
||||
in scope).
|
||||
|
||||
### Pack content (`en_units_v1`) — EXHAUSTIVE SCOPE
|
||||
|
||||
Structure mirrors `en_arithmetic_v1`: `lexicon.jsonl` +
|
||||
`manifest.json` + `glosses.jsonl` + `conversions.jsonl` + a
|
||||
self-sealing `.mastery_report.json` ratification artifact.
|
||||
|
||||
**Dimension classes** (entry_id prefix `en-units-dim-`):
|
||||
|
||||
| Dimension | Canonical unit | Notes |
|
||||
|-----------|---------------|-------|
|
||||
| `count` | (dimensionless) | The no-unit dimension |
|
||||
| `length` | foot | imperial + metric families |
|
||||
| `time` | minute | seconds through millennia |
|
||||
| `mass` | pound | imperial + metric |
|
||||
| `money` | cent | with currency-symbol attachments |
|
||||
| `temperature` | °F | affine scales (F/C/K/R) |
|
||||
| `volume` | cup | cooking + liquid + dry |
|
||||
| `area` | sq ft | **derived** = length × length |
|
||||
| `speed` | mph | **derived** = length / time |
|
||||
| `density` | lb/cu ft | **derived** = mass / volume |
|
||||
| `frequency` | Hz | **derived** = 1 / time |
|
||||
| `wage` | $/hour | **derived** = money / time |
|
||||
| `unit_price` | $/item | **derived** = money / count |
|
||||
|
||||
Derived dimensions are encoded *compositionally* via a
|
||||
`dimension_algebra.jsonl` table. The parser recognizes
|
||||
"miles per hour" structurally as `length-unit + per +
|
||||
time-unit`; it does NOT need a `mph` primitive. Reduces unit
|
||||
combinatorics from O(unit²) to O(unit + connector).
|
||||
|
||||
**Unit lemmas** (entry_id prefix `en-units-unit-`) — exhaustive
|
||||
for grade-school math word problems (~150 entries):
|
||||
|
||||
| Dimension | Lemmas |
|
||||
|-----------|--------|
|
||||
| length | inch/in, foot/ft, yard/yd, mile/mi, millimeter/mm, centimeter/cm, meter/m, kilometer/km |
|
||||
| time | second/sec/s, minute/min, hour/hr/h, day, week, month, year/yr, decade, century, millennium |
|
||||
| mass | ounce/oz, pound/lb, ton, gram/g, kilogram/kg, milligram/mg, metric_ton/tonne |
|
||||
| volume | teaspoon/tsp, tablespoon/tbsp, cup, pint/pt, quart/qt, gallon/gal, milliliter/mL, liter/L |
|
||||
| money | cent/¢, dollar/$, plus optional foreign: euro/€, pound_sterling/£, yen/¥, peso |
|
||||
| temperature | degree/°, Fahrenheit/°F, Celsius/°C, Kelvin/K |
|
||||
| area | sq_inch, sq_foot, sq_yard, sq_mile, sq_meter, sq_kilometer, acre, hectare |
|
||||
| count | (open class — head-noun grammar; pack does not enumerate, but provides count-dimension stamp for any unmatched noun in countable position) |
|
||||
|
||||
Each unit carries `singular`, `plural`, `symbol`, `dimension`,
|
||||
`is_canonical_for_dimension` (bool). Plural forms include the
|
||||
irregulars: foot/feet, child/children (count), person/people
|
||||
(count), mouse/mice (count), goose/geese (count),
|
||||
fish/fish (count), sheep/sheep (count), deer/deer (count),
|
||||
woman/women (count), man/men (count), leaf/leaves,
|
||||
knife/knives, life/lives.
|
||||
|
||||
**Multi-word units** (entry_id prefix `en-units-multiword-`):
|
||||
|
||||
Structural composition rules (not lexicon enumeration):
|
||||
|
||||
- `square <length-unit>` → area (sq_ft, sq_meter, sq_mile, …)
|
||||
- `cubic <length-unit>` → volume_derived
|
||||
- `<length-unit> per <time-unit>` → speed
|
||||
- `<money-unit> per <count-noun-or-unit>` → unit_price or wage
|
||||
- `<mass-unit> per <volume-unit>` → density
|
||||
|
||||
The pack ships the *patterns*, not every concrete combination —
|
||||
the parser composes them at recognition time.
|
||||
|
||||
**Container nouns** (entry_id prefix `en-units-container-`) —
|
||||
closed set (~25):
|
||||
|
||||
`box`, `bag`, `pack`, `basket`, `batch`, `dozen`, `group`,
|
||||
`set`, `case`, `pair`, `pile`, `stack`, `bunch`, `bundle`,
|
||||
`carton`, `crate`, `jar`, `can`, `bottle`, `cup` (when
|
||||
container, not volume-unit — context-disambiguated), `plate`,
|
||||
`bowl`, `tray`, `sack`, `container`, `gross` (=144).
|
||||
|
||||
Each declares syntax `<count> <container> of <content>` and
|
||||
optional `default_size` (only `dozen=12`, `pair=2`, `gross=144`,
|
||||
others null).
|
||||
|
||||
**Rate connectors** (entry_id prefix `en-units-rate-`) — closed
|
||||
set (~6):
|
||||
|
||||
`per`, `an`, `every`, `each`, `by`, `in` (as in "every 3 days").
|
||||
Each declares the dimension-algebra template it participates in.
|
||||
|
||||
**Symbol/affix table** (entry_id prefix `en-units-symbol-`):
|
||||
|
||||
`$`, `¢`, `°`, `°F`, `°C`, `%`, `½`, `¼`, `¾`, `⅓`, `⅔`, `⅛`,
|
||||
`⅜`, `⅝`, `⅞`. Each maps to its lexical lemma. Cross-linked to
|
||||
`en_numerics_v1` (sibling pack, ADR-0128) for fraction symbols.
|
||||
|
||||
**Substance qualifiers** (entry_id prefix `en-units-substance-`):
|
||||
|
||||
Pattern `N <unit> of <substance>` (e.g., "1000 feet of cable").
|
||||
The pack does NOT enumerate substances — that's open class. It
|
||||
encodes the *structural rule* that any measure-dimension unit
|
||||
admits an `of <NP>` substance tail, which the round-trip filter
|
||||
treats as a discarded grounded modifier.
|
||||
|
||||
**Unit-conversion graph** (`conversions.jsonl`, separate file):
|
||||
|
||||
Each line declares a directed edge `(from_unit, to_unit, ratio)`
|
||||
where `to_unit = ratio × from_unit`. The pack ratifies the
|
||||
*connected* weighted graph per dimension. Examples:
|
||||
|
||||
```jsonl
|
||||
{"edge_id":"en-units-conv-001","from":"inches","to":"feet","ratio":0.0833333333,"dimension":"length"}
|
||||
{"edge_id":"en-units-conv-002","from":"feet","to":"inches","ratio":12,"dimension":"length"}
|
||||
{"edge_id":"en-units-conv-003","from":"cents","to":"dollars","ratio":0.01,"dimension":"money"}
|
||||
{"edge_id":"en-units-conv-004","from":"dollars","to":"cents","ratio":100,"dimension":"money"}
|
||||
{"edge_id":"en-units-conv-005","from":"minutes","to":"hours","ratio":0.01666666667,"dimension":"time"}
|
||||
{"edge_id":"en-units-conv-006","from":"hours","to":"days","ratio":0.04166666667,"dimension":"time"}
|
||||
```
|
||||
|
||||
**Temperature edges carry an affine `offset` field** because
|
||||
temperature conversion is not pure multiplication:
|
||||
`°F = 1.8 × °C + 32`. Schema: `{"from","to","ratio","offset",
|
||||
"dimension"}`. Multiplicative-only edges have `offset=0`.
|
||||
|
||||
The graph is **the conversion table as data**, not as code. The
|
||||
solver consults it; the parser doesn't need to. Coverage target
|
||||
is EXHAUSTIVE within each dimension's lemma set (~80 edges
|
||||
total: every pair of in-pack units in the same dimension must be
|
||||
reachable). Bounded by NIST SI tables (closed set); ratification
|
||||
proves completeness — see invariants below.
|
||||
|
||||
### Why conversions matter
|
||||
|
||||
Without conversions, the pack solves only single-unit arithmetic
|
||||
(`5 apples + 3 apples`). With conversions, it solves
|
||||
within-dimension mixed-unit arithmetic — which is the majority
|
||||
of real word problems:
|
||||
|
||||
| Without conversions | With conversions |
|
||||
|--------------------|------------------|
|
||||
| `5 feet + 8 inches = ?` (refused) | canonicalize to feet (or inches), add, emit |
|
||||
| `$2 + 75¢ = ?` (refused) | canonicalize to cents (or dollars), add, emit |
|
||||
| `2 hours 30 minutes` (refused) | canonicalize to minutes (or hours), normalize |
|
||||
|
||||
Solver-side responsibility (delegated to ADR-0127.5 below):
|
||||
when an operation's operand has a unit in the same dimension as
|
||||
the actor's last quantity but a different unit, the solver
|
||||
canonicalizes via shortest path in the conversion graph before
|
||||
performing arithmetic. The canonical_bytes of `SolutionTrace`
|
||||
records which edges fired, preserving determinism and
|
||||
replay-equality.
|
||||
|
||||
### Graph ratification invariants (`en_units_v1`-specific)
|
||||
|
||||
The pack ratification process validates *graph correctness*, not
|
||||
just lexicon well-formedness:
|
||||
|
||||
1. **Round-trip identity** — for every edge `(A, B, r)`, there
|
||||
must exist an edge `(B, A, 1/r)` with `|round_trip_error| <
|
||||
1e-9`. Asymmetric tables are rejected at ratification.
|
||||
2. **Per-dimension connectivity** — within each dimension, the
|
||||
subgraph induced by that dimension's units must be connected
|
||||
(every unit reachable from every other). Isolated unit
|
||||
lemmas are rejected.
|
||||
3. **Path consistency** — for any two units A and C in the same
|
||||
dimension, all shortest paths from A to C must yield the
|
||||
same product of ratios within `1e-9`. Inconsistent paths
|
||||
(e.g., 12 in/ft × 3 ft/yd ≠ 1 yd/36 in) are rejected at
|
||||
ratification, not at runtime.
|
||||
4. **Canonical unit per dimension** — each dimension declares
|
||||
one canonical unit (`feet` for length, `seconds` or
|
||||
`minutes` for time, `dollars` for money, etc.). All
|
||||
canonicalization routes through the canonical unit; this
|
||||
bounds the shortest-path computation to O(1) lookups.
|
||||
5. **Exhaustive coverage gate** — every dimension's lemma set
|
||||
must produce a fully connected conversion subgraph. Adding
|
||||
a unit lemma without conversion edges (or with edges that
|
||||
don't connect to the canonical) is rejected at ratification.
|
||||
This prevents the inventory from silently fragmenting.
|
||||
6. **NIST/ISO provenance** — every conversion ratio cites a
|
||||
source via `provenance_id` (NIST SP 811 for SI, ISO 4217 for
|
||||
currency, etc.). Unsourced ratios are rejected.
|
||||
7. **Dimension algebra closure** — every derived-dimension entry
|
||||
(speed, density, wage, area) has both decomposed forms
|
||||
present (e.g., for `speed`: at least one length-unit and one
|
||||
time-unit are present in their respective dimensions).
|
||||
|
||||
These invariants live in `tests/test_adr_0127_pack_ratification.py`
|
||||
and run at every pack-change PR — the conversion graph cannot
|
||||
ship broken or incomplete.
|
||||
|
||||
### Parser integration
|
||||
|
||||
Three load-bearing changes to `generate/math_candidate_parser.py`
|
||||
(no changes to legacy `math_parser.py`):
|
||||
|
||||
1. **`extract_initial_candidates` widens** to recognize three
|
||||
additional shapes when pack consultation confirms dimensional
|
||||
typing:
|
||||
- `<Entity> has N <pack-unit>` (with `of <substance>` tail
|
||||
discarded)
|
||||
- `<Entity> has N <pack-container> of <content>`
|
||||
- `There are N <pack-unit> [of <substance>]`
|
||||
|
||||
2. **New `extract_rate_declaration_candidates`** recognizes
|
||||
`<Entity> makes $N <rate-connector> <pack-time-unit>` as an
|
||||
`apply_rate` candidate (ADR-0122 shape; the pack supplies the
|
||||
dimensional check that today's narrow regex misses).
|
||||
|
||||
3. **Round-trip filter gains a dimensional check.** The existing
|
||||
`roundtrip_admissible` adds an optional `require_pack_typed_unit`
|
||||
parameter (default False to preserve P1 behavior). When True,
|
||||
`matched_unit_token` must resolve to a known unit lemma in
|
||||
`en_units_v1`. This is the wrong-answer firewall for unit
|
||||
hallucination: a parser that fires on `Sam buys 3 contemplations`
|
||||
would now fail because `contemplations` is not pack-typed.
|
||||
|
||||
4. **Solver gains a dimensional-canonicalization helper.** New
|
||||
module `generate/math_unit_conversion.py` exports
|
||||
`canonicalize_to_dimension_canonical(quantity, conversion_graph)
|
||||
-> (quantity_in_canonical_unit, [edges_fired])`. Operations whose
|
||||
operand unit differs from the actor's tracked unit (but shares
|
||||
dimension) get canonicalized before arithmetic. The fired-edges
|
||||
list joins `SolutionTrace.steps` so replay reproduces the
|
||||
conversion path byte-equal. Mixed-dimension operands
|
||||
(`apples + dollars`) remain a SolveError — units type the
|
||||
arithmetic.
|
||||
|
||||
### What ADR-0127 explicitly does NOT do
|
||||
|
||||
- Does NOT model fractional-of-prior-quantity phrasing
|
||||
(`3/4 of its temperature`). That's a separate compositional
|
||||
pattern, not a unit-ontology question.
|
||||
- Does NOT model multi-step rate compositions
|
||||
(`overtime = base + 1/2 base`). That's solver-level.
|
||||
- Does NOT add new operation kinds. ADR-0126 + 0122 cover the
|
||||
needed shapes (add, subtract, transfer, multiply, divide,
|
||||
apply_rate, compare_*). The pack just helps the parser
|
||||
*recognize* the operand inputs.
|
||||
- Does NOT replace `_canonical_unit`. The pack lookup is
|
||||
*additive* — when the pack confirms a token, dimensional
|
||||
routing kicks in; when not, the legacy plural-canonicalization
|
||||
fallback remains.
|
||||
- Does NOT touch the sealed holdout. Re-runs against the
|
||||
unsealed train sample only.
|
||||
|
||||
---
|
||||
|
||||
## Invariants preserved / added
|
||||
|
||||
| Invariant | Preserved or added | How |
|
||||
|-----------|--------------------|-----|
|
||||
| `wrong == 0` | Preserved | Pack-typed unit check is a *stricter* gate, never a looser one |
|
||||
| `trace_hash` byte-equality | Preserved | Pack consultation is deterministic; same pack version → same lookup result |
|
||||
| Pack-binding (ADR-0114a #10) | Reinforced | The parser now explicitly cites pack entry_ids in `SolutionTrace.provenance` for unit-typed candidates |
|
||||
| Round-trip admissibility | Strengthened | When `require_pack_typed_unit=True`, hallucinated units fail to ground |
|
||||
| `versor_condition(F) < 1e-6` | Untouched | No runtime field changes |
|
||||
| Manifest checksum SHA-256 of bytes-on-disk | Required | Mastery report self-seals like `en_arithmetic_v1` |
|
||||
|
||||
## Exit criterion
|
||||
|
||||
**Same gate as ADR-0126 but with `en_units_v1` mounted:**
|
||||
|
||||
```
|
||||
correct >= 10 / 50 on evals/gsm8k_math/train_sample/v1/cases.jsonl
|
||||
wrong == 0
|
||||
```
|
||||
|
||||
**If passed:** run sealed holdout *once* and freeze the number in
|
||||
ADR-0127-results. Architecture + units substrate jointly validated.
|
||||
|
||||
**If missed (Path-B trigger, now real):** the deterministic
|
||||
parser-by-rule approach + units ontology + candidate-graph
|
||||
topology is the *full* design we believed was right, and it still
|
||||
doesn't move GSM8K. That is the empirical signal to demote GSM8K
|
||||
and re-target the math expert promotion to a benchmark where
|
||||
exact-recall and determinism are the discriminators.
|
||||
|
||||
---
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### A. Skip the units pack, just expand parser regex.
|
||||
Rejected — this IS the per-axis grammar treadmill (4 zero-lift
|
||||
ADRs). Adding patterns like `<Entity> has N \w+ of \w+` without
|
||||
dimensional typing produces wrong candidates the round-trip
|
||||
filter can't catch (any noun would ground; nothing rejects "5
|
||||
contemplations of cable").
|
||||
|
||||
### B. Pull units from a third-party library (Pint, etc.).
|
||||
Rejected — violates "no opaque dependencies in the runtime path."
|
||||
A curated pack of ≤ 60 lemmas is auditable, version-pinnable,
|
||||
ratifiable, and inspectable; an external library is none of those.
|
||||
|
||||
### C. Defer units, ship ADR-0126 with the honest 0/50 + open
|
||||
question. Considered. Discarded because 0126's empirical result
|
||||
is genuinely uninformative without the substrate it was designed
|
||||
to consume. Shipping the architecture with a misleading "0 lift"
|
||||
headline invites the wrong conclusion (Path B prematurely).
|
||||
|
||||
### D. Build the units pack but DON'T integrate it; ship as
|
||||
inventory. Rejected — the pack must be load-bearing per CLAUDE.md
|
||||
"prefer compact, curated packs" plus the project's general stance
|
||||
against decoration without integration.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan (proposed sub-phases)
|
||||
|
||||
| Phase | Module | Description |
|
||||
|-------|--------|-------------|
|
||||
| 0127.1 | `language_packs/data/en_units_v1/` | Pack content: lexicon (dimensions + units + containers + rate connectors) + `conversions.jsonl` + manifest + glosses + mastery report |
|
||||
| 0127.2 | `language_packs/loader.py` (or sibling) | Pack loader API: `lookup_unit(token) -> UnitEntry \| None`; `get_conversion_graph(dimension) -> ConversionGraph` |
|
||||
| 0127.3 | `generate/math_roundtrip.py` | Optional `require_pack_typed_unit` parameter on `roundtrip_admissible` |
|
||||
| 0127.4 | `generate/math_candidate_parser.py` | Three new initial-possession shapes + rate-declaration extractor, consulting the pack loader |
|
||||
| 0127.5 | `generate/math_unit_conversion.py` (new) | `canonicalize_to_dimension_canonical(quantity, graph)`; shortest-path lookup + edges-fired provenance for `SolutionTrace.steps` |
|
||||
| 0127.6 | `generate/math_solver.py` | Wire dimensional canonicalization into add/subtract/transfer/compare arithmetic; mixed-dimension operands → SolveError |
|
||||
| 0127.7 | `evals/gsm8k_math/train_sample/v1/runner.py` | Re-run with units pack engaged; new `report.json` |
|
||||
| 0127.8 | `tests/test_adr_0127_*.py` | Pack ratification (round-trip identity + connectivity + path consistency + canonical-unit-per-dimension) + parser integration + solver canonicalization + train-sample lift gate |
|
||||
|
||||
Regression gates (must remain green at every phase):
|
||||
- `core test --suite smoke -q`
|
||||
- `core test --suite math -q` (existing 714/714 + ADR-0126 74/74)
|
||||
- `core test --suite packs -q` (new `en_units_v1` ratification entries)
|
||||
- ADR-0126 P3+P4 tests (the candidate-graph machinery is unchanged)
|
||||
|
||||
## PR checklist (when proposing for merge)
|
||||
|
||||
```
|
||||
What capability did this add?
|
||||
→ Pack-typed unit recognition for the candidate-graph parser; the
|
||||
semantic substrate ADR-0126 was designed to consume.
|
||||
What invariant proves the field remains valid?
|
||||
→ Pack-typed unit check (new); wrong == 0 (preserved); manifest
|
||||
checksum SHA-256 of bytes-on-disk (required).
|
||||
Which CLI suite/eval proves the lane?
|
||||
→ smoke + math + packs + train_sample_runner (this is the
|
||||
re-measurement that decides Path B or not).
|
||||
Did this avoid hidden normalization, stochastic fallback,
|
||||
approximate recall, unreviewed mutation?
|
||||
→ Yes. Pack lookup is deterministic; unit ontology is bounded
|
||||
and curated; no learned scoring; ratified pack only.
|
||||
If it touches user input, what trust boundary was enforced?
|
||||
→ Pack file paths are validated via the existing safe_pack_id
|
||||
sanitiser (ADR-0051). Pack content is mastery-report-sealed.
|
||||
```
|
||||
338
docs/decisions/ADR-0128-numerics-pack.md
Normal file
338
docs/decisions/ADR-0128-numerics-pack.md
Normal file
|
|
@ -0,0 +1,338 @@
|
|||
# ADR-0128 — `en_numerics_v1` Pack
|
||||
|
||||
**Status:** Proposed (scope-only; sibling to ADR-0127)
|
||||
**Date:** 2026-05-23
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:**
|
||||
- ADR-0126 (candidate-graph parser + round-trip filter)
|
||||
- ADR-0127 (`en_units_v1` — units pack, cross-referenced for
|
||||
shared symbol/affix table)
|
||||
**Supersedes:** none
|
||||
**Sibling:** ADR-0127. Both packs are jointly required for the
|
||||
units-aware candidate-graph parser to produce a fair empirical
|
||||
result on the GSM8K-math lane. Either pack failing ratification
|
||||
independently blocks the joint exit gate.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0127 introduces `en_units_v1` (dimensions, units, conversions,
|
||||
containers, rate connectors). That pack handles the *physical /
|
||||
economic* substrate. It does NOT handle the *linguistic forms* a
|
||||
quantity can take in English math word problems:
|
||||
|
||||
- Cardinal number words ("seventeen", "two hundred and fifty")
|
||||
- Ordinal words ("third", "twentieth")
|
||||
- Fractions as words ("two-thirds", "a quarter") and as symbols
|
||||
(`½`, `¾`)
|
||||
- Multiplier words ("twice", "triple", "half")
|
||||
- Quantifiers ("all", "some", "each", "every", "many", "few")
|
||||
- Comparison anchors ("more", "fewer", "less", "additional")
|
||||
- Numeric format strings ("1,000", "1.5", "1/2", "75%")
|
||||
|
||||
Today these are scattered:
|
||||
- `WORD_NUMBERS` table hard-coded in `generate/math_roundtrip.py`
|
||||
(one through twelve only)
|
||||
- `_COMPARE_VERB` / comparison anchors hard-coded in
|
||||
`generate/math_parser.py`
|
||||
- Fraction handling absent
|
||||
- Percentage handling absent
|
||||
- Multi-digit number-word parsing ("two hundred") absent
|
||||
|
||||
The GSM8K train sample shows the cost: e.g., "Half of the kids
|
||||
are going to soccer camp" (refused — `half` not handled),
|
||||
"3/4 of its temperature" (refused — `3/4` not recognized as
|
||||
fractional form). These are unrecognized *quantity forms*, not
|
||||
unrecognized *units*; they sit on a different lexical axis from
|
||||
ADR-0127's substrate.
|
||||
|
||||
A units pack without a numerics pack solves "5 feet + 8 inches"
|
||||
but still refuses "half a foot." Both packs are needed for the
|
||||
architecture to get a fair empirical reading on the train sample.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add a `language_packs/data/en_numerics_v1/` ratified semantic pack
|
||||
that exhaustively encodes the English linguistic forms of
|
||||
quantities. Parser changes are minimal — most pack content
|
||||
replaces today's scattered hard-coded tables with ratified
|
||||
lookups.
|
||||
|
||||
### Pack content (`en_numerics_v1`) — EXHAUSTIVE SCOPE
|
||||
|
||||
Structure mirrors `en_arithmetic_v1` / `en_units_v1`:
|
||||
`lexicon.jsonl` + `manifest.json` + `glosses.jsonl` +
|
||||
`.mastery_report.json`.
|
||||
|
||||
**Cardinal number words** (entry_id prefix `en-num-card-`):
|
||||
|
||||
EXHAUSTIVE for grade-school range:
|
||||
|
||||
- `zero` through `twenty` (21 entries)
|
||||
- Tens: `thirty`, `forty`, `fifty`, `sixty`, `seventy`, `eighty`,
|
||||
`ninety` (7 entries)
|
||||
- Compound rule: `<tens-word>-<unit-word>` (e.g., "twenty-one",
|
||||
"ninety-nine") — structural composition, not enumerated
|
||||
- `hundred`, `thousand`, `million`, `billion` (4 entries)
|
||||
- Compound rule: `<N> hundred [and <M>]`, `<N> thousand
|
||||
[<conjunction> <M>]` — structural composition
|
||||
|
||||
Each entry carries `surface`, `numeric_value` (int), `morphology`
|
||||
(`cardinal`).
|
||||
|
||||
**Ordinal number words** (entry_id prefix `en-num-ord-`):
|
||||
|
||||
EXHAUSTIVE for grade-school range (1st–31st covers most
|
||||
calendar / position references in word problems):
|
||||
|
||||
- `first` through `tenth` (10 entries — irregular morphology)
|
||||
- Suffix rule: `<cardinal>th` for 11–31 (eleventh, twelfth,
|
||||
thirteenth, …, thirty-first) — structural composition with
|
||||
spelling-irregularity table (`fifth` not `fiveth`, `eighth` not
|
||||
`eightth`, `twelfth` not `twelveth`, etc.)
|
||||
- `twentieth`, `thirtieth`, `hundredth`, `thousandth` (4 entries)
|
||||
|
||||
Each entry carries `surface`, `position` (int), `morphology`
|
||||
(`ordinal`).
|
||||
|
||||
**Fraction words** (entry_id prefix `en-num-frac-`):
|
||||
|
||||
- Named fractions: `half` (½), `third` (⅓), `quarter` (¼),
|
||||
`fifth` (⅕), `sixth` (⅙), `seventh` (⅐), `eighth` (⅛),
|
||||
`ninth` (⅑), `tenth` (⅒), `sixteenth` (¹/₁₆) — ~10 entries
|
||||
- Compound rule: `<cardinal>/<ordinal-as-denominator>` (e.g.,
|
||||
"two-thirds", "three-quarters") — structural composition
|
||||
- Article-bound: `a half`, `a quarter`, `a third` resolve to
|
||||
the same numeric value as the bare form
|
||||
- **Symbol cross-link** to `en_units_v1` symbol table for `½`,
|
||||
`¼`, `¾`, `⅓`, `⅔`, `⅛`, `⅜`, `⅝`, `⅞`
|
||||
|
||||
Each entry carries `surface`, `numerator` (int), `denominator`
|
||||
(int), `decimal_value` (float — `1/3 = 0.333…` etc.),
|
||||
`morphology` (`fraction`).
|
||||
|
||||
**Multiplier words** (entry_id prefix `en-num-mult-`) — closed
|
||||
set:
|
||||
|
||||
`double` (×2), `triple` (×3), `quadruple` (×4), `quintuple` (×5),
|
||||
`twice` (×2), `thrice` (×3), `half` (×0.5 — also a fraction
|
||||
word; both entries cross-reference). Plus structural rule
|
||||
`N times` for arbitrary integer multipliers.
|
||||
|
||||
**Quantifiers** (entry_id prefix `en-num-quant-`) — closed set:
|
||||
|
||||
`all`, `none`, `some`, `both`, `each`, `every`, `many`, `few`,
|
||||
`several`, `most`, `any`, `no`, `single`.
|
||||
|
||||
Each declares its `semantic_type`: `total`, `empty`, `partial`,
|
||||
`paired`, `distributive`, `indefinite`. The parser uses this to
|
||||
decide whether the quantifier yields a determinate value
|
||||
(`both = 2`, `single = 1`) or is undeterminate and triggers
|
||||
refusal (`some`, `many`, `few` — refuse rather than guess; this
|
||||
preserves `wrong == 0`).
|
||||
|
||||
**Comparison anchors** (entry_id prefix `en-num-compare-`):
|
||||
|
||||
Migrated from `generate/math_roundtrip.py`'s hard-coded
|
||||
`COMPARE_ADDITIVE_ANCHORS` / `COMPARE_MULTIPLICATIVE_ANCHORS`.
|
||||
|
||||
- Additive: `more`, `fewer`, `less`, `additional`, `extra`,
|
||||
`missing`, `remaining`
|
||||
- Multiplicative: `twice`, `thrice`, `times`, `half`, `double`,
|
||||
`triple`, `quadruple`, `third`, `quarter`
|
||||
|
||||
Each cross-references its multiplier or fraction entry where
|
||||
applicable (avoiding duplicate truth).
|
||||
|
||||
**Number formats** (entry_id prefix `en-num-format-`) —
|
||||
structural rules, not enumeration:
|
||||
|
||||
- Digit groups: `1,000` (US thousand separator), `10,000`,
|
||||
`1,000,000` — `(\d{1,3})(?:,\d{3})+` → strip commas, parse as int
|
||||
- Decimals: `1.5`, `3.14`, `0.25` — `\d+\.\d+` → parse as float
|
||||
- Slash-fractions: `1/2`, `3/4`, `7/8` — `(\d+)/(\d+)` →
|
||||
parse as `Fraction`
|
||||
- Mixed numbers: `1 1/2`, `2 3/4` — `(\d+) (\d+)/(\d+)` →
|
||||
`whole + numerator/denominator`
|
||||
- Percentages: `75%`, `1.5%` — `\d+(?:\.\d+)?%` → divide by 100
|
||||
- Negative numbers: `-3`, `-0.5` — leading minus (rarely in
|
||||
grade-school but cheap to include)
|
||||
|
||||
Each format rule declares its regex pattern, `parser_function`
|
||||
(by name), and `output_type` (`int`, `float`, `Fraction`).
|
||||
|
||||
### Cross-references with `en_units_v1`
|
||||
|
||||
- Fraction symbols `½`, `¼`, `¾`, etc. appear in both packs.
|
||||
Single source of truth lives in `en_numerics_v1`;
|
||||
`en_units_v1` symbol-affix table contains a *reference* entry
|
||||
pointing to the numerics pack via `cross_pack_id`. Ratification
|
||||
cross-pack consistency check (see ADR-0127 ratification
|
||||
invariants and below) verifies the references resolve.
|
||||
- Percentage (`%`) cross-references between packs because it's
|
||||
both a numeric format and a dimensionless modifier.
|
||||
- Multipliers (`double`, `twice`) cross-reference because they
|
||||
also appear as comparison anchors in `en_units_v1`'s parser-
|
||||
consumed register.
|
||||
|
||||
### Ratification invariants (`en_numerics_v1`-specific)
|
||||
|
||||
1. **Cardinal exhaustiveness** — every English cardinal 0..20,
|
||||
every "tens" form, every magnitude word
|
||||
(hundred/thousand/million) present.
|
||||
2. **Ordinal exhaustiveness** — every English ordinal 1st..31st
|
||||
present (covers month days + most grade-school position
|
||||
references).
|
||||
3. **Fraction exhaustiveness** — every named fraction 1/2 through
|
||||
1/10 present + the irregular set (sixteenth, thirty-second).
|
||||
4. **Cross-pack consistency** — every fraction-symbol entry in
|
||||
`en_units_v1` resolves to a fraction entry in
|
||||
`en_numerics_v1`. Verified by joint pack-mount ratification.
|
||||
5. **Quantifier semantic-type completeness** — every quantifier
|
||||
lemma carries a `semantic_type` from the closed set
|
||||
`{total, empty, partial, paired, distributive, indefinite}`.
|
||||
6. **Format regex test corpus** — each format rule has a
|
||||
minimum of 10 positive + 10 negative test strings in
|
||||
`tests/test_adr_0128_numeric_formats.py`. Format ambiguity
|
||||
(e.g., `1.000` could be `1` or `1000`) is *refused* per
|
||||
`wrong == 0`, not guessed.
|
||||
|
||||
### Parser integration
|
||||
|
||||
1. **`generate/math_roundtrip.py`** — replace hard-coded
|
||||
`WORD_NUMBERS` and `COMPARE_*_ANCHORS` tables with calls into
|
||||
`en_numerics_v1` loader. Behavior preserved (current entries
|
||||
are a subset); future extensions land in the pack, not the
|
||||
source.
|
||||
|
||||
2. **`generate/math_candidate_parser.py`** — new value-token
|
||||
normalization helper `normalize_value_token(token) -> Quantity
|
||||
| Fraction | None` that consults `en_numerics_v1` format
|
||||
rules. Handles "two-thirds" → `Fraction(2,3)`, `75%` → `0.75`,
|
||||
`1,500` → `1500`, etc.
|
||||
|
||||
3. **Round-trip filter (`roundtrip_admissible`)** — the existing
|
||||
`_value_grounds` helper is extended to ground word-forms via
|
||||
the numerics pack loader (current hard-coded `WORD_NUMBERS`
|
||||
capped at "twelve" widens to the full cardinal table).
|
||||
|
||||
4. **Quantifier-driven refusal** — when a candidate's value-token
|
||||
resolves to an `indefinite` quantifier (`some`, `many`, `few`),
|
||||
the parser emits NO candidate for that sentence. Empty list →
|
||||
refusal at decision rule. Preserves `wrong == 0`.
|
||||
|
||||
### What ADR-0128 explicitly does NOT do
|
||||
|
||||
- Does NOT replace `en_units_v1`. The packs are siblings.
|
||||
- Does NOT introduce new operation kinds. ADR-0126 + 0122 cover
|
||||
the operation grammar; this pack supplies value-token forms.
|
||||
- Does NOT model number-system alternates (Roman numerals,
|
||||
Chinese numerals, etc.) — out of scope for English math
|
||||
problems.
|
||||
- Does NOT model implicit numbers ("a dozen" / "a few") as
|
||||
determinate values — `a dozen = 12` is encoded via the
|
||||
container's `default_size` (ADR-0127's container entry, not
|
||||
here); `a few` is `indefinite` and triggers refusal.
|
||||
- Does NOT add new exit gates. Joint exit criterion with
|
||||
ADR-0127 (re-run train sample with both packs mounted).
|
||||
|
||||
---
|
||||
|
||||
## Invariants preserved / added
|
||||
|
||||
| Invariant | Preserved or added | How |
|
||||
|-----------|--------------------|-----|
|
||||
| `wrong == 0` | Preserved | Indefinite quantifiers trigger refusal; format ambiguity triggers refusal |
|
||||
| `trace_hash` byte-equality | Preserved | Pack lookup is deterministic |
|
||||
| Pack-binding (ADR-0114a #10) | Reinforced | Value-token resolution cites `en-num-*` entry_ids in `SolutionTrace.provenance` |
|
||||
| Round-trip admissibility | Strengthened | Word-form numbers ground via pack lookup, not regex enumeration |
|
||||
| Replay equivalence | Preserved | Same pack version → same lookup result |
|
||||
| `versor_condition(F) < 1e-6` | Untouched | No runtime field changes |
|
||||
| Manifest checksum SHA-256 of bytes-on-disk | Required | Mastery report self-seals |
|
||||
|
||||
## Exit criterion
|
||||
|
||||
**Joint with ADR-0127** — both packs ratified, both mounted,
|
||||
re-run train sample:
|
||||
|
||||
```
|
||||
correct >= 10 / 50 on evals/gsm8k_math/train_sample/v1/cases.jsonl
|
||||
wrong == 0
|
||||
```
|
||||
|
||||
**If passed:** run sealed holdout once, freeze in ADR-0127/0128
|
||||
joint results.
|
||||
|
||||
**If missed:** **Path-B trigger** — full deterministic design
|
||||
(candidate-graph + units + numerics) failed to move GSM8K.
|
||||
Demote benchmark, re-target math expert promotion.
|
||||
|
||||
---
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### A. Fold numerics into `en_units_v1` as one big pack.
|
||||
Rejected per ADR-0127 discussion: domain-distinct lexicons,
|
||||
risk-isolation, future i18n composability.
|
||||
|
||||
### B. Keep hard-coded tables in `math_roundtrip.py` / `math_parser.py`.
|
||||
Rejected — violates pack-binding (ADR-0114a #10); future
|
||||
language packs (Spanish, German) would have to duplicate the
|
||||
hard-coded tables; ratification is impossible.
|
||||
|
||||
### C. Use a third-party number-parsing library (`word2number`,
|
||||
`num2words`, etc.). Rejected — same opacity / non-auditability
|
||||
critique as ADR-0127 alt B.
|
||||
|
||||
### D. Ship only cardinals + fractions; defer quantifiers /
|
||||
ordinals / formats. Rejected — quantifiers (`half`, `some`) are
|
||||
*specifically* what the GSM8K train sample refuses on. Partial
|
||||
pack delivers partial empirical signal — same trap ADR-0127
|
||||
addresses re: scope mismatch.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan (proposed sub-phases)
|
||||
|
||||
| Phase | Module | Description |
|
||||
|-------|--------|-------------|
|
||||
| 0128.1 | `language_packs/data/en_numerics_v1/` | Pack content (lexicon + manifest + glosses + mastery report) |
|
||||
| 0128.2 | `language_packs/loader.py` | `lookup_cardinal`, `lookup_ordinal`, `lookup_fraction`, `lookup_quantifier`, `match_number_format` |
|
||||
| 0128.3 | `generate/math_roundtrip.py` | Replace hard-coded `WORD_NUMBERS` + `COMPARE_*_ANCHORS` with pack-backed lookups |
|
||||
| 0128.4 | `generate/math_candidate_parser.py` | `normalize_value_token` helper; quantifier-driven refusal |
|
||||
| 0128.5 | `tests/test_adr_0128_*.py` | Pack ratification (exhaustiveness gates) + parser integration + format regex corpus |
|
||||
| 0128.6 | Joint with ADR-0127.7 | Re-run train sample with both packs mounted |
|
||||
|
||||
Regression gates:
|
||||
- `core test --suite smoke -q`
|
||||
- `core test --suite math -q`
|
||||
- `core test --suite packs -q`
|
||||
- ADR-0126 P3+P4 tests
|
||||
- ADR-0127 pack ratification
|
||||
|
||||
## PR checklist (when proposing for merge)
|
||||
|
||||
```
|
||||
What capability did this add?
|
||||
→ Exhaustive English linguistic-form ontology for quantities;
|
||||
sibling substrate to en_units_v1 (ADR-0127).
|
||||
What invariant proves the field remains valid?
|
||||
→ Wrong==0 preserved via indefinite-quantifier refusal +
|
||||
format-ambiguity refusal; cardinal/ordinal/fraction
|
||||
exhaustiveness ratification gates.
|
||||
Which CLI suite/eval proves the lane?
|
||||
→ smoke + math + packs + joint train_sample re-run with
|
||||
ADR-0127.
|
||||
Did this avoid hidden normalization, stochastic fallback,
|
||||
approximate recall, unreviewed mutation?
|
||||
→ Yes. Pack lookup is deterministic; lexicon is bounded and
|
||||
ratified; no learned tokenization; no fuzzy matching.
|
||||
If it touches user input, what trust boundary was enforced?
|
||||
→ Pack file paths validated via safe_pack_id (ADR-0051).
|
||||
Format regexes carry test corpora documenting accepted vs
|
||||
rejected inputs; user input is matched against ratified
|
||||
patterns only.
|
||||
```
|
||||
|
|
@ -29,6 +29,7 @@ import json
|
|||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.math_parser import ParseError, parse_problem
|
||||
from generate.math_problem_graph import MathProblemGraph
|
||||
from generate.math_realizer import RealizerError, realize
|
||||
|
|
@ -204,6 +205,141 @@ def _score_one(case: dict[str, Any]) -> CaseOutcome:
|
|||
)
|
||||
|
||||
|
||||
def _score_one_candidate_graph(case: dict[str, Any]) -> CaseOutcome:
|
||||
"""ADR-0126 P4 — score one case via the candidate-graph pipeline.
|
||||
|
||||
Mirrors :func:`_score_one` end-to-end (parser → solver → verifier →
|
||||
realizer → expected-answer check) but the parse stage uses
|
||||
:func:`generate.math_candidate_graph.parse_and_solve` instead of
|
||||
the first-match-wins :func:`generate.math_parser.parse_problem`.
|
||||
|
||||
Preserves wrong == 0: any deviation in the new pipeline still
|
||||
routes through the same verifier-replay + answer/unit equality
|
||||
checks. Refusals are first-class — branches with no admissible
|
||||
parse, branches that disagree on the answer, and branches that
|
||||
exceed MAX_TOTAL_BRANCHES all classify as ``refused``.
|
||||
|
||||
Callers that want to evaluate the candidate-graph topology
|
||||
(e.g. ``evals/gsm8k_math/train_sample/v1/runner.py`` from PR
|
||||
#160) substitute this function for ``_score_one``; the
|
||||
``CaseOutcome`` shape is identical.
|
||||
"""
|
||||
case_id = case["id"]
|
||||
expected_answer = case["expected_answer"]
|
||||
expected_unit = case["expected_unit"]
|
||||
|
||||
# Stage 1 — candidate-graph parse + internal solve + decision rule.
|
||||
cg_result = parse_and_solve(case["problem"])
|
||||
if not cg_result.is_admitted:
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="refused",
|
||||
reason=f"candidate_graph: {cg_result.refusal_reason}",
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=None,
|
||||
actual_unit=None,
|
||||
trace_hash=None,
|
||||
realized_prose=None,
|
||||
)
|
||||
graph = cg_result.selected_graph
|
||||
assert graph is not None # is_admitted implies non-None graph
|
||||
|
||||
# Stage 2 — canonical solve for the full SolutionTrace (verifier
|
||||
# needs the trace; parse_and_solve only kept the numeric answer).
|
||||
try:
|
||||
trace = solve(graph)
|
||||
except SolveError as exc:
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="refused",
|
||||
reason=f"solver: {exc}",
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=None,
|
||||
actual_unit=None,
|
||||
trace_hash=None,
|
||||
realized_prose=None,
|
||||
)
|
||||
|
||||
# Stage 3 — verify (independent re-derivation, ADR-0117).
|
||||
verdict = verify(graph, trace)
|
||||
trace_hash = hashlib.sha256(trace.canonical_bytes()).hexdigest()
|
||||
if not verdict.passed:
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="wrong",
|
||||
reason=f"verifier: {verdict.reason}",
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=trace.answer_value,
|
||||
actual_unit=trace.answer_unit,
|
||||
trace_hash=trace_hash,
|
||||
realized_prose=None,
|
||||
)
|
||||
|
||||
# Stage 4 — realize.
|
||||
try:
|
||||
realized = realize(graph.initial_state, trace)
|
||||
prose = realized.as_prose()
|
||||
except RealizerError as exc:
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="wrong",
|
||||
reason=f"realizer: {exc}",
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=trace.answer_value,
|
||||
actual_unit=trace.answer_unit,
|
||||
trace_hash=trace_hash,
|
||||
realized_prose=None,
|
||||
)
|
||||
|
||||
# Stage 5 — expected-answer comparison (same logic as _score_one).
|
||||
if expected_unit != "" and trace.answer_unit != expected_unit:
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="wrong",
|
||||
reason=(
|
||||
f"unit mismatch: got {trace.answer_unit!r}, "
|
||||
f"expected {expected_unit!r}"
|
||||
),
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=trace.answer_value,
|
||||
actual_unit=trace.answer_unit,
|
||||
trace_hash=trace_hash,
|
||||
realized_prose=prose,
|
||||
)
|
||||
if trace.answer_value != expected_answer:
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="wrong",
|
||||
reason=(
|
||||
f"answer mismatch: got {trace.answer_value!r}, "
|
||||
f"expected {expected_answer!r}"
|
||||
),
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=trace.answer_value,
|
||||
actual_unit=trace.answer_unit,
|
||||
trace_hash=trace_hash,
|
||||
realized_prose=prose,
|
||||
)
|
||||
|
||||
return CaseOutcome(
|
||||
case_id=case_id,
|
||||
outcome="correct",
|
||||
reason="",
|
||||
expected_answer=expected_answer,
|
||||
expected_unit=expected_unit,
|
||||
actual_answer=trace.answer_value,
|
||||
actual_unit=trace.answer_unit,
|
||||
trace_hash=trace_hash,
|
||||
realized_prose=prose,
|
||||
)
|
||||
|
||||
|
||||
def run_lane(
|
||||
cases: list[dict[str, Any]],
|
||||
*,
|
||||
|
|
|
|||
|
|
@ -13,252 +13,252 @@
|
|||
"per_case": [
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0001",
|
||||
"reason": "parser: could not parse statement clause: 'Tina makes $18.00 an hour' (in sentence: 'Tina makes $18.00 an hour.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0002",
|
||||
"reason": "parser: could not parse statement clause: 'Jan buys 1000 feet of cable' (in sentence: 'Jan buys 1000 feet of cable.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jan buys 1000 feet of cable.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0003",
|
||||
"reason": "parser: could not parse statement clause: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances' (in sentence: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0004",
|
||||
"reason": "parser: could not parse statement clause: 'There are some kids in camp' (in sentence: 'There are some kids in camp.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'There are some kids in camp.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0005",
|
||||
"reason": "parser: could not parse statement clause: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature\" (in sentence: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\")",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\"",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0006",
|
||||
"reason": "parser: could not parse statement clause: 'Mandy started reading books with only 8 pages when she was 6 years old' (in sentence: 'Mandy started reading books with only 8 pages when she was 6 years old.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0007",
|
||||
"reason": "parser: could not parse statement clause: 'Francine has five full boxes of crayons and 5 loose crayons' (in sentence: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0008",
|
||||
"reason": "parser: could not parse statement clause: 'Marnie makes bead bracelets' (in sentence: 'Marnie makes bead bracelets.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Marnie makes bead bracelets.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0009",
|
||||
"reason": "parser: could not parse statement clause: 'Jen has 10 more ducks than four times the number of chickens' (in sentence: 'Jen has 10 more ducks than four times the number of chickens.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0010",
|
||||
"reason": "parser: could not parse statement clause: 'Yun had 20 paperclips initially, but then lost 12' (in sentence: 'Yun had 20 paperclips initially, but then lost 12.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Yun had 20 paperclips initially, but then lost 12.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0011",
|
||||
"reason": "parser: could not parse statement clause: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup' (in sentence: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0012",
|
||||
"reason": "parser: could not parse statement clause: 'Dennis collected 10 rocks' (in sentence: 'Dennis collected 10 rocks.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'He put all of them in his aquarium but his fish ate half of them.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0013",
|
||||
"reason": "parser: could not parse statement clause: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel' (in sentence: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0014",
|
||||
"reason": "parser: could not parse statement clause: 'Bob can shuck 10 oysters in 5 minutes' (in sentence: 'Bob can shuck 10 oysters in 5 minutes.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Bob can shuck 10 oysters in 5 minutes.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0015",
|
||||
"reason": "parser: could not parse statement clause: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride' (in sentence: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0016",
|
||||
"reason": "parser: could not parse statement clause: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs\" (in sentence: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\")",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\"",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0017",
|
||||
"reason": "parser: could not parse statement clause: 'Jason has a carriage house that he rents out' (in sentence: 'Jason has a carriage house that he rents out.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jason has a carriage house that he rents out.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0018",
|
||||
"reason": "parser: could not parse statement clause: 'Xavier plays football with his friends' (in sentence: 'Xavier plays football with his friends.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Xavier plays football with his friends.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0019",
|
||||
"reason": "parser: could not parse statement clause: 'John adopts a dog from a shelter' (in sentence: 'John adopts a dog from a shelter.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John adopts a dog from a shelter.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0020",
|
||||
"reason": "parser: could not parse statement clause: 'Two puppies, two kittens' (in sentence: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0021",
|
||||
"reason": "parser: could not parse statement clause: 'John is lifting weights' (in sentence: 'John is lifting weights.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John is lifting weights.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0022",
|
||||
"reason": "parser: could not parse statement clause: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish' (in sentence: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0023",
|
||||
"reason": "parser: could not parse statement clause: 'Nicole collected 400 Pokemon cards' (in sentence: 'Nicole collected 400 Pokemon cards.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0024",
|
||||
"reason": "parser: could not parse statement clause: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday' (in sentence: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0025",
|
||||
"reason": "parser: could not parse statement clause: 'Lilibeth and her friends go strawberry picking' (in sentence: 'Lilibeth and her friends go strawberry picking.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Lilibeth and her friends go strawberry picking.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0026",
|
||||
"reason": "parser: could not parse statement clause: 'Aaron and his brother Carson each saved up $40 to go to dinner' (in sentence: 'Aaron and his brother Carson each saved up $40 to go to dinner.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0027",
|
||||
"reason": "parser: could not parse statement clause: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook' (in sentence: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0028",
|
||||
"reason": "parser: could not parse statement clause: 'Tom opens an amusement park' (in sentence: 'Tom opens an amusement park.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Tom opens an amusement park.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0029",
|
||||
"reason": "parser: could not parse statement clause: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home' (in sentence: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0030",
|
||||
"reason": "parser: could not parse statement clause: 'Jake decides to go to the beach for a fun day' (in sentence: 'Jake decides to go to the beach for a fun day.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jake decides to go to the beach for a fun day.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0031",
|
||||
"reason": "parser: could not parse statement clause: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer' (in sentence: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0032",
|
||||
"reason": "parser: could not parse statement clause: 'John decides to take up illustration' (in sentence: 'John decides to take up illustration.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John decides to take up illustration.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0033",
|
||||
"reason": "parser: could not parse statement clause: 'Rachel is 12 years old' (in sentence: 'Rachel is 12 years old, and her grandfather is 7 times her age.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Rachel is 12 years old, and her grandfather is 7 times her age.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0034",
|
||||
"reason": "parser: could not parse statement clause: 'Georgie is a varsity player on a football team' (in sentence: 'Georgie is a varsity player on a football team.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Georgie is a varsity player on a football team.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0035",
|
||||
"reason": "parser: could not parse statement clause: 'She decided to split them among her friends' (in sentence: 'She decided to split them among her friends.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'She decided to split them among her friends.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0036",
|
||||
"reason": "parser: could not parse statement clause: 'Monica way studying for an exam' (in sentence: 'Monica way studying for an exam.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Monica way studying for an exam.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0037",
|
||||
"reason": "parser: could not parse statement clause: 'Michael wants to lose 10 pounds by June' (in sentence: 'Michael wants to lose 10 pounds by June.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Michael wants to lose 10 pounds by June.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0038",
|
||||
"reason": "parser: could not parse statement clause: 'In a building, there are a hundred ladies on the first-floor studying' (in sentence: 'In a building, there are a hundred ladies on the first-floor studying.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'In a building, there are a hundred ladies on the first-floor studying.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0039",
|
||||
"reason": "parser: could not parse statement clause: 'At the family reunion, everyone ate too much food and gained weight' (in sentence: 'At the family reunion, everyone ate too much food and gained weight.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'At the family reunion, everyone ate too much food and gained weight.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0040",
|
||||
"reason": "parser: could not parse statement clause: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road' (in sentence: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0041",
|
||||
"reason": "parser: could not parse statement clause: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan' (in sentence: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0042",
|
||||
"reason": "parser: could not parse statement clause: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag' (in sentence: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0043",
|
||||
"reason": "parser: could not parse statement clause: 'Sandra wants to buy some sweets' (in sentence: 'Sandra wants to buy some sweets.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Sandra wants to buy some sweets.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0044",
|
||||
"reason": "parser: could not parse statement clause: 'John invests in a bank and gets 10% simple interest' (in sentence: 'John invests in a bank and gets 10% simple interest.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John invests in a bank and gets 10% simple interest.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0045",
|
||||
"reason": "parser: could not parse statement clause: 'Bart fills out surveys to earn money' (in sentence: 'Bart fills out surveys to earn money.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Bart fills out surveys to earn money.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0046",
|
||||
"reason": "parser: could not parse statement clause: 'A school has 100 students' (in sentence: 'A school has 100 students.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'A school has 100 students.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0047",
|
||||
"reason": "parser: could not parse statement clause: 'John bakes 12 coconut macaroons, each weighing 5 ounces' (in sentence: 'John bakes 12 coconut macaroons, each weighing 5 ounces.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John bakes 12 coconut macaroons, each weighing 5 ounces.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0048",
|
||||
"reason": "parser: could not parse statement clause: 'Jed collects stamp cards' (in sentence: 'Jed collects stamp cards.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jed collects stamp cards.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0049",
|
||||
"reason": "parser: could not parse statement clause: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes' (in sentence: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0050",
|
||||
"reason": "parser: could not parse statement clause: 'Mark does a gig every other day for 2 weeks' (in sentence: 'Mark does a gig every other day for 2 weeks.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'",
|
||||
"verdict": "refused"
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ import sys
|
|||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from evals.gsm8k_math.runner import _score_one
|
||||
from evals.gsm8k_math.runner import _score_one_candidate_graph
|
||||
|
||||
_HERE = Path(__file__).resolve().parent
|
||||
_CASES_PATH = _HERE / "cases.jsonl"
|
||||
|
|
@ -71,7 +71,7 @@ def build_report(cases: list[dict[str, Any]]) -> dict[str, Any]:
|
|||
per_case: list[dict[str, str]] = []
|
||||
counts = {"correct": 0, "wrong": 0, "refused": 0}
|
||||
for raw in cases:
|
||||
outcome = _score_one(_adapt(raw))
|
||||
outcome = _score_one_candidate_graph(_adapt(raw))
|
||||
counts[outcome.outcome] += 1
|
||||
per_case.append(
|
||||
{
|
||||
|
|
|
|||
406
generate/math_candidate_graph.py
Normal file
406
generate/math_candidate_graph.py
Normal file
|
|
@ -0,0 +1,406 @@
|
|||
"""ADR-0126 P3 — Candidate-graph assembly + decision rule.
|
||||
|
||||
End-to-end orchestration:
|
||||
|
||||
text
|
||||
→ sentence split
|
||||
→ per-sentence candidate extraction (P2)
|
||||
→ per-candidate round-trip admissibility filter (P1)
|
||||
→ bounded branch enumeration (Cartesian product, cap=64)
|
||||
→ per-branch graph construction + solve
|
||||
→ decision rule
|
||||
|
||||
Decision rule (preserves wrong == 0):
|
||||
|
||||
|admissible answers| == 0 → refuse
|
||||
|admissible answers| == 1 → emit
|
||||
|admissible answers| >= 2,
|
||||
all answers identical → emit common answer
|
||||
|admissible answers| >= 2,
|
||||
answers differ → refuse (genuine ambiguity)
|
||||
|
||||
Per-sentence ambiguity tiebreaker (P3-local; orthogonal to the
|
||||
decision rule above):
|
||||
|
||||
When a single sentence has multiple admissible candidates AND the
|
||||
resulting graphs all solve to the same numeric answer, we collapse
|
||||
to one candidate via the "most-grounded-slots-wins" heuristic.
|
||||
This handles cases like "Sam gives 3 apples to Tom" where both
|
||||
subtract and transfer pass round-trip — transfer has a target slot
|
||||
(more grounded content), so it wins on the tiebreaker. If the
|
||||
graphs differ in answer, we let the decision rule above refuse.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from itertools import product
|
||||
from typing import Final, Union
|
||||
|
||||
from generate.math_candidate_parser import (
|
||||
CandidateInitial,
|
||||
CandidateUnknown,
|
||||
extract_initial_candidates,
|
||||
extract_operation_candidates,
|
||||
extract_question_candidates,
|
||||
)
|
||||
from generate.math_problem_graph import (
|
||||
MathGraphError,
|
||||
MathProblemGraph,
|
||||
)
|
||||
from generate.math_roundtrip import CandidateOperation, roundtrip_admissible
|
||||
from generate.math_solver import SolveError, solve
|
||||
|
||||
|
||||
MAX_TOTAL_BRANCHES: Final[int] = 64
|
||||
"""Hard cap on Cartesian-product branch enumeration; exceeding refuses."""
|
||||
|
||||
MAX_CANDIDATES_PER_SENTENCE: Final[int] = 4
|
||||
"""Hard cap on per-sentence candidate emission; exceeding refuses."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Result types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CandidateGraphAnswer:
|
||||
"""A successfully solved candidate graph.
|
||||
|
||||
``answer`` is the numeric answer the solver produced for this
|
||||
branch. Multiple branches may produce the same answer; the
|
||||
decision rule collapses on equality.
|
||||
"""
|
||||
|
||||
graph: MathProblemGraph
|
||||
answer: int | float
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CandidateGraphResult:
|
||||
"""Outcome of candidate-graph parsing + filtering + deciding.
|
||||
|
||||
Exactly one of ``answer`` / ``refusal_reason`` is non-None.
|
||||
"""
|
||||
|
||||
answer: int | float | None
|
||||
selected_graph: MathProblemGraph | None
|
||||
refusal_reason: str | None
|
||||
# Diagnostics for inner-loop signal in P6 runner.
|
||||
branches_enumerated: int
|
||||
branches_admissible: int
|
||||
|
||||
@property
|
||||
def is_admitted(self) -> bool:
|
||||
return self.answer is not None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sentence splitting + classification (mirrors math_parser._split_sentences)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SENTENCE_SPLIT_RE: Final[re.Pattern[str]] = re.compile(r"(?<=[.?!])\s+")
|
||||
|
||||
|
||||
def _split_sentences(text: str) -> list[str]:
|
||||
text = text.strip()
|
||||
return [p.strip() for p in _SENTENCE_SPLIT_RE.split(text) if p.strip()]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-sentence choice typing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# A statement sentence's choice space: a list of (initial-or-operation)
|
||||
# candidates that all passed the round-trip filter. A question sentence's
|
||||
# choice space: a list of CandidateUnknown.
|
||||
SentenceChoice = Union[CandidateInitial, CandidateOperation]
|
||||
|
||||
|
||||
def _filtered_statement_choices(sentence: str) -> list[SentenceChoice]:
|
||||
"""Return all admissible (initial | operation) candidates for a
|
||||
statement sentence, after applying the round-trip filter."""
|
||||
out: list[SentenceChoice] = []
|
||||
|
||||
# Initial-possession candidates are checked structurally — we use
|
||||
# the operation round-trip filter shape only for CandidateOperation.
|
||||
# For CandidateInitial we apply a light structural check inline:
|
||||
# entity, value, unit, anchor must all ground in source. (P1's
|
||||
# roundtrip_admissible signature is operation-specific.)
|
||||
for ic in extract_initial_candidates(sentence):
|
||||
if _initial_admissible(ic):
|
||||
out.append(ic)
|
||||
|
||||
for oc in extract_operation_candidates(sentence):
|
||||
if roundtrip_admissible(oc):
|
||||
out.append(oc)
|
||||
|
||||
return out[:MAX_CANDIDATES_PER_SENTENCE]
|
||||
|
||||
|
||||
def _filtered_question_choices(sentence: str) -> list[CandidateUnknown]:
|
||||
"""Return all admissible question candidates after the question-
|
||||
specific structural check."""
|
||||
out: list[CandidateUnknown] = []
|
||||
for qc in extract_question_candidates(sentence):
|
||||
if _question_admissible(qc):
|
||||
out.append(qc)
|
||||
return out[:MAX_CANDIDATES_PER_SENTENCE]
|
||||
|
||||
|
||||
def _initial_admissible(ic: CandidateInitial) -> bool:
|
||||
"""Light structural ground-check for initial-possession candidates.
|
||||
|
||||
Same shape as roundtrip_admissible but for the initial-possession
|
||||
slot set (entity, anchor, value, unit)."""
|
||||
from generate.math_roundtrip import _tokens, _value_grounds, _token_in
|
||||
haystack = _tokens(ic.source_span)
|
||||
if not _token_in(ic.matched_anchor, haystack):
|
||||
return False
|
||||
if not _value_grounds(ic.matched_value_token, haystack):
|
||||
return False
|
||||
if not _token_in(ic.matched_unit_token, haystack):
|
||||
return False
|
||||
# Entity token: for multi-word entities ("the boys"), all words
|
||||
# must ground. Split + check each.
|
||||
for tok in ic.matched_entity_token.split():
|
||||
if not _token_in(tok, haystack):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _question_admissible(qc: CandidateUnknown) -> bool:
|
||||
"""Light structural ground-check for question candidates."""
|
||||
from generate.math_roundtrip import _tokens, _token_in
|
||||
haystack = _tokens(qc.source_span)
|
||||
if not _token_in(qc.matched_unit_token, haystack):
|
||||
return False
|
||||
if qc.matched_entity_token is not None:
|
||||
for tok in qc.matched_entity_token.split():
|
||||
if not _token_in(tok, haystack):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-sentence ambiguity tiebreaker (most-grounded-slots-wins)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _slot_count(choice: SentenceChoice) -> int:
|
||||
"""Count the number of distinct grounded content slots.
|
||||
|
||||
More grounded slots → 'tighter' parse → preferred when answers
|
||||
agree. Implements the give-with-target case: transfer (4 slots:
|
||||
actor, verb, value, unit, target = 5) wins over subtract (4 slots)
|
||||
on the same sentence.
|
||||
"""
|
||||
if isinstance(choice, CandidateInitial):
|
||||
return 4 # entity, anchor, value, unit
|
||||
n = 4 # actor, verb, value, unit
|
||||
if choice.matched_target_token is not None:
|
||||
n += 1
|
||||
if choice.matched_reference_actor_token is not None:
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def _collapse_per_sentence_ties(
|
||||
choices: list[SentenceChoice],
|
||||
) -> list[SentenceChoice]:
|
||||
"""If multiple choices exist for one sentence, prefer the one with
|
||||
the most grounded slots (deterministic tiebreaker). Ties at the
|
||||
max slot-count return all tied choices; cross-sentence ambiguity
|
||||
still gets enumerated."""
|
||||
if len(choices) <= 1:
|
||||
return choices
|
||||
max_slots = max(_slot_count(c) for c in choices)
|
||||
return [c for c in choices if _slot_count(c) == max_slots]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Graph construction from one branch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _build_graph(
|
||||
statement_choices: list[SentenceChoice],
|
||||
question_choice: CandidateUnknown,
|
||||
) -> MathProblemGraph | None:
|
||||
"""Build a MathProblemGraph from one consistent branch of sentence
|
||||
choices, or return None if the branch cannot form a valid graph
|
||||
(entity universe violations, referential integrity, etc.).
|
||||
|
||||
State threading is minimal in P3 scope (no pronoun resolution, no
|
||||
unit inheritance — those need richer per-branch state and land in
|
||||
a later sub-phase). The dataclass constructors catch every
|
||||
referential-integrity violation deterministically.
|
||||
"""
|
||||
entities: list[str] = []
|
||||
seen_entities: set[str] = set()
|
||||
|
||||
def add_entity(e: str) -> None:
|
||||
if e not in seen_entities:
|
||||
entities.append(e)
|
||||
seen_entities.add(e)
|
||||
|
||||
initials_list = []
|
||||
operations_list = []
|
||||
for choice in statement_choices:
|
||||
if isinstance(choice, CandidateInitial):
|
||||
add_entity(choice.initial.entity)
|
||||
initials_list.append(choice.initial)
|
||||
else:
|
||||
add_entity(choice.op.actor)
|
||||
if choice.op.target is not None:
|
||||
add_entity(choice.op.target)
|
||||
operations_list.append(choice.op)
|
||||
|
||||
if question_choice.unknown.entity is not None:
|
||||
if question_choice.unknown.entity not in seen_entities:
|
||||
return None # question references unknown entity
|
||||
|
||||
try:
|
||||
return MathProblemGraph(
|
||||
entities=tuple(entities),
|
||||
initial_state=tuple(initials_list),
|
||||
operations=tuple(operations_list),
|
||||
unknown=question_choice.unknown,
|
||||
)
|
||||
except MathGraphError:
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orchestrator
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_and_solve(text: str) -> CandidateGraphResult:
|
||||
"""End-to-end: parse text via candidate-graph topology, solve each
|
||||
admissible branch, apply decision rule.
|
||||
|
||||
Returns :class:`CandidateGraphResult` with either an admitted
|
||||
``answer`` + ``selected_graph`` or a ``refusal_reason`` string
|
||||
naming why the problem was refused.
|
||||
|
||||
Preserves wrong == 0 by construction:
|
||||
|
||||
- A sentence the parser cannot match contributes [] to its choice
|
||||
list → Cartesian product is empty → refusal.
|
||||
- Every branch's graph must round-trip through the round-trip
|
||||
filter at the per-sentence level (already applied during
|
||||
filtering).
|
||||
- Branches that disagree on the final answer trigger refusal.
|
||||
"""
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason="empty or non-string problem",
|
||||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
sentences = _split_sentences(text)
|
||||
if not sentences:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason="no sentences found",
|
||||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
question_sentences = [s for s in sentences if s.rstrip().endswith("?")]
|
||||
statement_sentences = [s for s in sentences if not s.rstrip().endswith("?")]
|
||||
|
||||
if len(question_sentences) != 1:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason=(
|
||||
f"expected exactly one question sentence; "
|
||||
f"got {len(question_sentences)}"
|
||||
),
|
||||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
# Per-sentence choice spaces (after round-trip filter + tiebreaker).
|
||||
per_sentence_choices: list[list[SentenceChoice]] = []
|
||||
for s in statement_sentences:
|
||||
choices = _filtered_statement_choices(s)
|
||||
if not choices:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason=f"no admissible candidate for statement: {s!r}",
|
||||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
per_sentence_choices.append(_collapse_per_sentence_ties(choices))
|
||||
|
||||
question_choices = _filtered_question_choices(question_sentences[0])
|
||||
if not question_choices:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason=(
|
||||
f"no admissible candidate for question: "
|
||||
f"{question_sentences[0]!r}"
|
||||
),
|
||||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
# Cartesian product across statement choices × question choices.
|
||||
total = 1
|
||||
for choices in per_sentence_choices:
|
||||
total *= len(choices)
|
||||
total *= len(question_choices)
|
||||
if total > MAX_TOTAL_BRANCHES:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason=(
|
||||
f"branch count {total} exceeds MAX_TOTAL_BRANCHES="
|
||||
f"{MAX_TOTAL_BRANCHES} (refusing rather than truncating)"
|
||||
),
|
||||
branches_enumerated=total, branches_admissible=0,
|
||||
)
|
||||
|
||||
admissible: list[CandidateGraphAnswer] = []
|
||||
branches_enumerated = 0
|
||||
for combo in product(*per_sentence_choices, question_choices):
|
||||
branches_enumerated += 1
|
||||
*stmt_choices, q_choice = combo # type: ignore[misc]
|
||||
graph = _build_graph(list(stmt_choices), q_choice) # type: ignore[arg-type]
|
||||
if graph is None:
|
||||
continue
|
||||
try:
|
||||
trace = solve(graph)
|
||||
except SolveError:
|
||||
continue
|
||||
admissible.append(
|
||||
CandidateGraphAnswer(graph=graph, answer=trace.answer_value)
|
||||
)
|
||||
|
||||
if not admissible:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason="no branch produced a solvable graph",
|
||||
branches_enumerated=branches_enumerated,
|
||||
branches_admissible=0,
|
||||
)
|
||||
|
||||
# Decision rule: all answers identical → emit; otherwise → refuse.
|
||||
distinct_answers = {a.answer for a in admissible}
|
||||
if len(distinct_answers) > 1:
|
||||
return CandidateGraphResult(
|
||||
answer=None, selected_graph=None,
|
||||
refusal_reason=(
|
||||
f"branches disagree on answer "
|
||||
f"(distinct values: {sorted(distinct_answers)})"
|
||||
),
|
||||
branches_enumerated=branches_enumerated,
|
||||
branches_admissible=len(admissible),
|
||||
)
|
||||
|
||||
# Single agreed answer. Pick the first admissible graph as the
|
||||
# canonical representative (deterministic since product() is ordered).
|
||||
chosen = admissible[0]
|
||||
return CandidateGraphResult(
|
||||
answer=chosen.answer,
|
||||
selected_graph=chosen.graph,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=branches_enumerated,
|
||||
branches_admissible=len(admissible),
|
||||
)
|
||||
388
generate/math_candidate_parser.py
Normal file
388
generate/math_candidate_parser.py
Normal file
|
|
@ -0,0 +1,388 @@
|
|||
"""ADR-0126 — Candidate-emitting sentence parser.
|
||||
|
||||
Sibling to ``generate/math_parser.py``. Same regex spirit, different
|
||||
topology: instead of first-match-wins with a single mutable state and
|
||||
``ParseError`` on miss, each per-sentence extractor returns a *list of
|
||||
candidates* (possibly empty) carrying full source-span provenance.
|
||||
|
||||
The wrong-answer firewall is :func:`generate.math_roundtrip.roundtrip_admissible`,
|
||||
applied downstream in P3 (graph assembly). This module's job is purely
|
||||
to *enumerate* the parses the grammar admits — telling truth from
|
||||
falsehood is not its concern.
|
||||
|
||||
Determinism: candidate lists are returned in deterministic order
|
||||
(canonical pattern key); the same input always produces the same
|
||||
ordered output.
|
||||
|
||||
Scope of P2 (this module):
|
||||
- Initial-possession candidate extraction.
|
||||
- Operation candidate extraction for add / subtract / transfer
|
||||
via the canonical "<Subject> <verb> <value> <unit> [to <target>]"
|
||||
shape.
|
||||
- Permissive verb tables imported from
|
||||
:data:`generate.math_roundtrip.KIND_TO_VERBS` — much wider than
|
||||
``math_parser._ADD_VERBS`` / ``_SUBTRACT_VERBS`` / ``_TRANSFER_VERBS``
|
||||
because the round-trip filter rejects wrong candidates downstream.
|
||||
|
||||
Out of scope for P2 (added in later phases):
|
||||
- Pronoun resolution (needs per-branch state — P3).
|
||||
- Unit inheritance from ``last_unit`` (needs per-branch state — P3).
|
||||
- Multiply / divide / rate / comparison candidates (later phases of
|
||||
ADR-0126; the candidate-emission machinery is identical, just more
|
||||
pattern matchers).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Final
|
||||
|
||||
from generate.math_problem_graph import (
|
||||
InitialPossession,
|
||||
Operation,
|
||||
Quantity,
|
||||
Unknown,
|
||||
)
|
||||
from generate.math_roundtrip import (
|
||||
ADD_VERBS,
|
||||
SUBTRACT_VERBS,
|
||||
TRANSFER_VERBS,
|
||||
WORD_NUMBERS,
|
||||
CandidateOperation,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Initial-possession candidate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CandidateInitial:
|
||||
"""Initial-possession candidate with source-span provenance.
|
||||
|
||||
Mirrors :class:`CandidateOperation` but for ``InitialPossession``.
|
||||
The round-trip filter for initials is the same shape: every claimed
|
||||
content slot (entity, value, unit, anchor verb 'has'/'have') must
|
||||
ground in the source sentence.
|
||||
"""
|
||||
|
||||
initial: InitialPossession
|
||||
source_span: str
|
||||
matched_anchor: str # 'has' or 'have'
|
||||
matched_value_token: str # '3' or 'three'
|
||||
matched_unit_token: str
|
||||
matched_entity_token: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.matched_anchor.lower() not in ("has", "have"):
|
||||
raise ValueError(
|
||||
f"CandidateInitial.matched_anchor must be has/have; "
|
||||
f"got {self.matched_anchor!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared regex building blocks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Title-cased proper noun OR "the <noun>" collective. Same widening as
|
||||
# math_parser._INITIAL_HAS_RE's ADR-0123a entity slot.
|
||||
_ENTITY: Final[str] = r"(?:[A-Z]\w+|[Tt]he\s+\w+)"
|
||||
|
||||
# Numeric value: digit run OR word-form integer (one..twelve initially;
|
||||
# WORD_NUMBERS table is wider but we cap the regex at the common range
|
||||
# for syntactic parsing and let the filter handle ground-truth value
|
||||
# equivalence).
|
||||
_WORD_NUM_OPTIONS: Final[str] = "|".join(
|
||||
re.escape(w) for w in sorted(WORD_NUMBERS.keys(), key=len, reverse=True)
|
||||
)
|
||||
_VALUE: Final[str] = rf"(?:\d+|{_WORD_NUM_OPTIONS})"
|
||||
|
||||
# Verb alternation built from the permissive registry. Pre-compute one
|
||||
# pattern per kind so we can attribute matched verbs to candidates.
|
||||
def _verbs_pattern(verbs: frozenset[str]) -> str:
|
||||
# Longest-first so "passes" matches before "pass" inside the alternation.
|
||||
options = sorted(verbs, key=len, reverse=True)
|
||||
return r"(?:" + "|".join(re.escape(v) for v in options) + r")"
|
||||
|
||||
|
||||
_ADD_VERBS_PATTERN: Final[str] = _verbs_pattern(ADD_VERBS)
|
||||
_SUBTRACT_VERBS_PATTERN: Final[str] = _verbs_pattern(SUBTRACT_VERBS)
|
||||
_TRANSFER_VERBS_PATTERN: Final[str] = _verbs_pattern(TRANSFER_VERBS)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Initial-possession extractor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_INITIAL_HAS_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^(?P<entity>{_ENTITY})\s+"
|
||||
rf"(?P<anchor>has|have)\s+"
|
||||
rf"(?P<value>{_VALUE})\s+"
|
||||
r"(?P<unit>\w+)\s*\.?$"
|
||||
)
|
||||
|
||||
|
||||
def _normalize_entity(raw: str) -> str:
|
||||
"""Collapse whitespace + lowercase article. Mirrors math_parser
|
||||
canonicalization so candidate entity names hash-equal to legacy."""
|
||||
e = re.sub(r"\s+", " ", raw.strip())
|
||||
if e.lower().startswith("the "):
|
||||
return "the " + e[4:]
|
||||
return e
|
||||
|
||||
|
||||
def _resolve_value(value_token: str) -> int:
|
||||
if value_token.isdigit():
|
||||
return int(value_token)
|
||||
return WORD_NUMBERS[value_token.lower()]
|
||||
|
||||
|
||||
def extract_initial_candidates(sentence: str) -> list[CandidateInitial]:
|
||||
"""Return all admissible initial-possession candidates for ``sentence``.
|
||||
|
||||
Currently emits at most one candidate (the single canonical shape
|
||||
"<Entity> has <N> <unit>"). Returns an empty list if no shape matches.
|
||||
"""
|
||||
s = sentence.strip().rstrip(".")
|
||||
m = _INITIAL_HAS_RE.match(s)
|
||||
if not m:
|
||||
return []
|
||||
entity = _normalize_entity(m.group("entity"))
|
||||
value = _resolve_value(m.group("value"))
|
||||
unit_raw = m.group("unit")
|
||||
# Canonicalize: lowercase + ensure plural (matching math_parser._canonical_unit).
|
||||
unit = unit_raw.lower()
|
||||
if not unit.endswith("s"):
|
||||
unit = unit + "s"
|
||||
return [
|
||||
CandidateInitial(
|
||||
initial=InitialPossession(
|
||||
entity=entity,
|
||||
quantity=Quantity(value=value, unit=unit),
|
||||
),
|
||||
source_span=sentence,
|
||||
matched_anchor=m.group("anchor"),
|
||||
matched_value_token=m.group("value"),
|
||||
matched_unit_token=unit_raw,
|
||||
matched_entity_token=m.group("entity"),
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Operation candidate extractor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Per-kind operation patterns. Each captures: subject, verb, value, unit,
|
||||
# optional target. The verb alternation is the kind's permissive verb table.
|
||||
#
|
||||
# Note: optional unit (?P<unit>) is allowed because some constructions
|
||||
# rely on inherited unit ("Sam doubles his savings"); however for P2's
|
||||
# scope we only emit candidates when the unit token is explicit. Inherited-
|
||||
# unit candidates require per-branch state and are added in P3.
|
||||
|
||||
def _op_pattern(verbs_pattern: str, *, requires_target: bool) -> re.Pattern[str]:
|
||||
"""Build the per-kind operation regex.
|
||||
|
||||
For ``requires_target=True`` (transfer): the trailing ``to <Target>``
|
||||
clause is a captured slot.
|
||||
|
||||
For ``requires_target=False`` (add/subtract): there is no target
|
||||
slot. A trailing ``to <noun>`` phrase, if present, is consumed as
|
||||
part of the discardable preposition tail so the regex still matches
|
||||
ambiguous sentences like "Sam gives 3 apples to Tom" (which we
|
||||
*do* want to match as a subtract candidate; the transfer-vs-subtract
|
||||
disambiguation happens at the candidate / filter / decision-rule
|
||||
layer, not by regex specificity).
|
||||
"""
|
||||
if requires_target:
|
||||
target_part = r"\s+to\s+(?P<target>[A-Z]\w+)"
|
||||
trailing_prep = (
|
||||
r"(?:\s+(?:on|from|at|in|onto|into|under|over)\s+.+)?"
|
||||
)
|
||||
else:
|
||||
target_part = ""
|
||||
# Note: 'to' is included in the discardable preposition set.
|
||||
trailing_prep = (
|
||||
r"(?:\s+(?:on|from|at|in|onto|into|under|over|to)\s+.+)?"
|
||||
)
|
||||
return re.compile(
|
||||
r"^"
|
||||
rf"(?P<subject>{_ENTITY})\s+"
|
||||
rf"(?P<verb>{verbs_pattern})"
|
||||
rf"\s+(?P<value>{_VALUE})"
|
||||
r"(?:\s+more)?"
|
||||
r"(?:\s+(?!to\b)(?!more\b)(?!on\b)(?!from\b)(?!at\b)(?!in\b)"
|
||||
r"(?P<unit>\w+))?"
|
||||
rf"{target_part}"
|
||||
rf"{trailing_prep}"
|
||||
r"\s*\.?$",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
_ADD_OP_RE: Final[re.Pattern[str]] = _op_pattern(_ADD_VERBS_PATTERN, requires_target=False)
|
||||
_SUBTRACT_OP_RE: Final[re.Pattern[str]] = _op_pattern(_SUBTRACT_VERBS_PATTERN, requires_target=False)
|
||||
_TRANSFER_OP_RE: Final[re.Pattern[str]] = _op_pattern(_TRANSFER_VERBS_PATTERN, requires_target=True)
|
||||
|
||||
|
||||
def _build_op_candidate(
|
||||
m: re.Match[str], kind: str, source: str
|
||||
) -> CandidateOperation | None:
|
||||
"""Build a CandidateOperation from a regex match. Returns None if
|
||||
the match lacks a required slot (e.g. unit token absent — P2 does
|
||||
not emit unit-inherited candidates)."""
|
||||
unit_raw = m.group("unit")
|
||||
if unit_raw is None:
|
||||
return None
|
||||
unit = unit_raw.lower()
|
||||
if not unit.endswith("s"):
|
||||
unit = unit + "s"
|
||||
subject = _normalize_entity(m.group("subject"))
|
||||
verb = m.group("verb").lower()
|
||||
value = _resolve_value(m.group("value"))
|
||||
target_raw = m.group("target") if "target" in m.groupdict() else None
|
||||
target = target_raw if target_raw is not None else None
|
||||
|
||||
op_kwargs: dict[str, object] = {
|
||||
"actor": subject,
|
||||
"kind": kind,
|
||||
"operand": Quantity(value=value, unit=unit),
|
||||
}
|
||||
if kind == "transfer":
|
||||
if target is None:
|
||||
return None # transfer requires target
|
||||
op_kwargs["target"] = target
|
||||
else:
|
||||
if target is not None:
|
||||
return None # add/subtract don't take targets
|
||||
|
||||
return CandidateOperation(
|
||||
op=Operation(**op_kwargs), # type: ignore[arg-type]
|
||||
source_span=source,
|
||||
matched_verb=verb,
|
||||
matched_value_token=m.group("value"),
|
||||
matched_unit_token=unit_raw,
|
||||
matched_actor_token=m.group("subject"),
|
||||
matched_target_token=target,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Question candidate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CandidateUnknown:
|
||||
"""Question-candidate with source-span provenance.
|
||||
|
||||
Two question shapes in P3 scope:
|
||||
|
||||
- ``How many <unit> does <Entity> have [left|now|in total|altogether]?``
|
||||
→ ``Unknown(entity=<Entity>, unit=<unit>)``
|
||||
- ``How many <unit> do they have [left|now|in total|altogether]?``
|
||||
→ ``Unknown(entity=None, unit=<unit>)`` (total-across)
|
||||
|
||||
The round-trip filter for questions checks the unit token and (when
|
||||
present) the entity token both appear in the source span.
|
||||
"""
|
||||
|
||||
unknown: Unknown
|
||||
source_span: str
|
||||
matched_unit_token: str
|
||||
matched_entity_token: str | None # None for total-across questions
|
||||
|
||||
|
||||
_Q_ENTITY_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"^How\s+many\s+(?P<unit>\w+)\s+(?:does|do)\s+"
|
||||
rf"(?P<entity>{_ENTITY})"
|
||||
r"\s+have(?:\s+(?:left|now|in\s+total|altogether)){0,2}\s*\??$",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
_Q_TOTAL_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"^How\s+many\s+(?P<unit>\w+)\s+do\s+they\s+have"
|
||||
r"(?:\s+(?:in\s+total|altogether|left|now)){0,2}\s*\??$",
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def extract_question_candidates(sentence: str) -> list[CandidateUnknown]:
|
||||
"""Return all admissible question candidates for ``sentence``.
|
||||
|
||||
Tries the total-across pattern FIRST (same specificity order as
|
||||
legacy math_parser). The entity-pattern's widened regex would
|
||||
otherwise capture "they" as an entity name.
|
||||
|
||||
Empty list if no shape matches.
|
||||
"""
|
||||
s = sentence.strip()
|
||||
out: list[CandidateUnknown] = []
|
||||
|
||||
m = _Q_TOTAL_RE.match(s)
|
||||
if m is not None:
|
||||
unit_raw = m.group("unit")
|
||||
unit = unit_raw.lower()
|
||||
if not unit.endswith("s"):
|
||||
unit = unit + "s"
|
||||
out.append(
|
||||
CandidateUnknown(
|
||||
unknown=Unknown(entity=None, unit=unit),
|
||||
source_span=sentence,
|
||||
matched_unit_token=unit_raw,
|
||||
matched_entity_token=None,
|
||||
)
|
||||
)
|
||||
return out # specificity order: don't also try entity pattern
|
||||
|
||||
m = _Q_ENTITY_RE.match(s)
|
||||
if m is not None:
|
||||
unit_raw = m.group("unit")
|
||||
unit = unit_raw.lower()
|
||||
if not unit.endswith("s"):
|
||||
unit = unit + "s"
|
||||
entity = _normalize_entity(m.group("entity"))
|
||||
out.append(
|
||||
CandidateUnknown(
|
||||
unknown=Unknown(entity=entity, unit=unit),
|
||||
source_span=sentence,
|
||||
matched_unit_token=unit_raw,
|
||||
matched_entity_token=m.group("entity"),
|
||||
)
|
||||
)
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def extract_operation_candidates(sentence: str) -> list[CandidateOperation]:
|
||||
"""Return all operation candidates for ``sentence``.
|
||||
|
||||
Tries every verb-kind pattern independently. A sentence with an
|
||||
ambiguous verb (e.g. "Sam gives 3 apples to Tom" — "gives" appears
|
||||
in both SUBTRACT_VERBS and TRANSFER_VERBS) may emit multiple
|
||||
candidates. The round-trip filter
|
||||
(:func:`generate.math_roundtrip.roundtrip_admissible`) and the
|
||||
decision rule (P3) resolve which one becomes the chosen graph.
|
||||
|
||||
Candidate emission order is canonical: add, subtract, transfer.
|
||||
Within each kind, the regex emits at most one candidate per
|
||||
sentence.
|
||||
"""
|
||||
s = sentence.strip()
|
||||
out: list[CandidateOperation] = []
|
||||
|
||||
for pattern, kind in (
|
||||
(_ADD_OP_RE, "add"),
|
||||
(_SUBTRACT_OP_RE, "subtract"),
|
||||
(_TRANSFER_OP_RE, "transfer"),
|
||||
):
|
||||
m = pattern.match(s)
|
||||
if m is None:
|
||||
continue
|
||||
candidate = _build_op_candidate(m, kind, source=sentence)
|
||||
if candidate is not None:
|
||||
out.append(candidate)
|
||||
|
||||
return out
|
||||
372
generate/math_roundtrip.py
Normal file
372
generate/math_roundtrip.py
Normal file
|
|
@ -0,0 +1,372 @@
|
|||
"""ADR-0126 — Round-trip admissibility filter.
|
||||
|
||||
The wrong-answer firewall for the candidate-graph parser topology.
|
||||
|
||||
A :class:`CandidateOperation` carries an :class:`Operation` plus the
|
||||
source-span provenance for every content slot the parser claimed: the
|
||||
verb token, the numeric value token, the unit token, the actor name as
|
||||
it appeared, and (for transfers) the target name. Admissibility is a
|
||||
deterministic byte-level check that each claimed slot's surface token
|
||||
actually appears in the source span, AND that the verb the parser
|
||||
consumed is registered for the operation kind it produced.
|
||||
|
||||
This is the load-bearing invariant of ADR-0126:
|
||||
|
||||
admissible(c) iff every content slot in c.op grounds in c.source_span
|
||||
AND c.matched_verb is registered for c.op.kind
|
||||
|
||||
Two consequences:
|
||||
|
||||
1. A regex that mis-reads "loses" as add fails because "loses" is not
|
||||
in the add-verb registry — even if the resulting Operation looks
|
||||
numerically plausible.
|
||||
|
||||
2. A regex that hallucinates a number or unit not present in the
|
||||
source fails because the matched token won't ground.
|
||||
|
||||
Normalization is deliberately conservative: lowercase + word-boundary
|
||||
containment. We do not strip morphology (plural "apples" must equal
|
||||
the matched unit token "apples", not "apple"); we do not stem
|
||||
("eats" != "ate"); we do not handle synonyms. The parser is expected
|
||||
to canonicalize units before constructing the Quantity, so the
|
||||
matched_unit_token carries the surface form.
|
||||
|
||||
Determinism: every check is pure byte / regex containment. No
|
||||
randomness, no learning, no approximation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Final, Mapping
|
||||
|
||||
from generate.math_problem_graph import Comparison, Operation, Quantity, Rate
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Verb registry — single source of truth for {operation kind -> valid verbs}.
|
||||
#
|
||||
# This is intentionally permissive (much wider than today's math_parser tables)
|
||||
# because the candidate-graph topology relies on the round-trip filter to
|
||||
# reject wrong candidates, not on the parser's regex narrowness.
|
||||
#
|
||||
# P2 will refactor math_parser.py to import these constants instead of
|
||||
# maintaining its own _ADD_VERBS / _SUBTRACT_VERBS / etc. tables.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Surface verbs that grammatically place the actor as the *gainer* of the
|
||||
# operand quantity. Past tense and present tense both registered.
|
||||
ADD_VERBS: Final[frozenset[str]] = frozenset({
|
||||
# acquisition
|
||||
"buy", "buys", "bought",
|
||||
"get", "gets", "got",
|
||||
"find", "finds", "found",
|
||||
"receive", "receives", "received",
|
||||
"earn", "earns", "earned",
|
||||
"add", "adds", "added",
|
||||
"pick", "picks", "picked", # "picks up N"
|
||||
"collect", "collects", "collected",
|
||||
"gather", "gathers", "gathered",
|
||||
"catch", "catches", "caught",
|
||||
"save", "saves", "saved",
|
||||
# production (actor creates instances of the unit)
|
||||
"bake", "bakes", "baked",
|
||||
"make", "makes", "made",
|
||||
"cook", "cooks", "cooked",
|
||||
"slice", "slices", "sliced",
|
||||
"pack", "packs", "packed",
|
||||
"build", "builds", "built",
|
||||
"grow", "grows", "grew",
|
||||
})
|
||||
|
||||
# Surface verbs that grammatically place the actor as the *loser* of the
|
||||
# operand quantity.
|
||||
SUBTRACT_VERBS: Final[frozenset[str]] = frozenset({
|
||||
"eat", "eats", "ate",
|
||||
"lose", "loses", "lost",
|
||||
"sell", "sells", "sold",
|
||||
"donate", "donates", "donated",
|
||||
"use", "uses", "used",
|
||||
"spend", "spends", "spent",
|
||||
"drop", "drops", "dropped",
|
||||
"remove", "removes", "removed",
|
||||
"break", "breaks", "broke",
|
||||
"destroy", "destroys", "destroyed",
|
||||
"throw", "throws", "threw", # "throws out N"
|
||||
"discard", "discards", "discarded",
|
||||
"return", "returns", "returned", # ambiguous — see TRANSFER_VERBS
|
||||
"consume", "consumes", "consumed",
|
||||
"give", "gives", "gave", # ambiguous — see TRANSFER_VERBS
|
||||
"send", "sends", "sent", # ambiguous — see TRANSFER_VERBS
|
||||
})
|
||||
|
||||
# Surface verbs that grammatically place the actor as the *sender* and a
|
||||
# named target as the *receiver*. These verbs ALSO appear in SUBTRACT_VERBS
|
||||
# because the same surface token can take a transfer reading (with target)
|
||||
# or a subtract reading (without target) — both candidates fire and the
|
||||
# decision rule picks based on whether a target slot was grounded.
|
||||
TRANSFER_VERBS: Final[frozenset[str]] = frozenset({
|
||||
"give", "gives", "gave",
|
||||
"send", "sends", "sent",
|
||||
"hand", "hands", "handed",
|
||||
"pass", "passes", "passed",
|
||||
"mail", "mails", "mailed",
|
||||
"deliver", "delivers", "delivered",
|
||||
"return", "returns", "returned",
|
||||
})
|
||||
|
||||
MULTIPLY_VERBS: Final[frozenset[str]] = frozenset({
|
||||
"double", "doubles", "doubled",
|
||||
"triple", "triples", "tripled",
|
||||
"quadruple", "quadruples", "quadrupled",
|
||||
"multiply", "multiplies", "multiplied",
|
||||
})
|
||||
|
||||
DIVIDE_VERBS: Final[frozenset[str]] = frozenset({
|
||||
"halve", "halves", "halved",
|
||||
"split", "splits", "split",
|
||||
"divide", "divides", "divided",
|
||||
"share", "shares", "shared",
|
||||
})
|
||||
|
||||
# Comparison "verbs" — the surface anchor for compare_additive /
|
||||
# compare_multiplicative is usually 'has'/'have' + comparator phrase
|
||||
# ('N more than', 'twice as many as', etc.). The matched_verb slot for
|
||||
# comparison candidates carries the comparator phrase head ('more',
|
||||
# 'fewer', 'twice', 'times', 'half').
|
||||
COMPARE_ADDITIVE_ANCHORS: Final[frozenset[str]] = frozenset({
|
||||
"more", "fewer", "less", "additional", "extra",
|
||||
})
|
||||
COMPARE_MULTIPLICATIVE_ANCHORS: Final[frozenset[str]] = frozenset({
|
||||
"twice", "thrice", "times", "half", "double", "triple",
|
||||
"quadruple", "third", "quarter",
|
||||
})
|
||||
|
||||
# Rate anchors (ADR-0122): "per", "each", "every", "a/an" (when followed
|
||||
# by unit and price).
|
||||
RATE_ANCHORS: Final[frozenset[str]] = frozenset({
|
||||
"per", "each", "every",
|
||||
})
|
||||
|
||||
|
||||
KIND_TO_VERBS: Final[Mapping[str, frozenset[str]]] = {
|
||||
"add": ADD_VERBS,
|
||||
"subtract": SUBTRACT_VERBS,
|
||||
"transfer": TRANSFER_VERBS,
|
||||
"multiply": MULTIPLY_VERBS,
|
||||
"divide": DIVIDE_VERBS,
|
||||
"apply_rate": RATE_ANCHORS,
|
||||
"compare_additive": COMPARE_ADDITIVE_ANCHORS,
|
||||
"compare_multiplicative": COMPARE_MULTIPLICATIVE_ANCHORS,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Number-word table — for grounding numeric value tokens that appear as
|
||||
# words ("three apples") rather than digits ("3 apples").
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
WORD_NUMBERS: Final[Mapping[str, int]] = {
|
||||
"zero": 0, "one": 1, "two": 2, "three": 3, "four": 4,
|
||||
"five": 5, "six": 6, "seven": 7, "eight": 8, "nine": 9,
|
||||
"ten": 10, "eleven": 11, "twelve": 12, "thirteen": 13,
|
||||
"fourteen": 14, "fifteen": 15, "sixteen": 16, "seventeen": 17,
|
||||
"eighteen": 18, "nineteen": 19, "twenty": 20, "thirty": 30,
|
||||
"forty": 40, "fifty": 50, "sixty": 60, "seventy": 70,
|
||||
"eighty": 80, "ninety": 90, "hundred": 100, "thousand": 1000,
|
||||
# ordinals as factor-bearing forms ("a third", "a quarter")
|
||||
"half": 2, "third": 3, "quarter": 4,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public dataclass — what the candidate-graph parser will emit per match.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CandidateOperation:
|
||||
"""An Operation candidate plus the source-span provenance proving it.
|
||||
|
||||
Every content slot the parser claims must trace back to a surface
|
||||
token in :attr:`source_span`. The round-trip filter
|
||||
(:func:`roundtrip_admissible`) verifies this; candidates that fail
|
||||
are rejected before they can produce a wrong answer.
|
||||
|
||||
Slot conventions:
|
||||
|
||||
- ``matched_verb``: the surface verb (or comparator phrase head) the
|
||||
parser consumed. MUST be a member of
|
||||
``KIND_TO_VERBS[op.kind]``.
|
||||
- ``matched_value_token``: the surface form of the numeric value, as
|
||||
it appeared in the source ("3" or "three"). Required for all
|
||||
kinds except ``compare_multiplicative`` with factor anchors
|
||||
like "twice"/"half" where the anchor itself carries the factor —
|
||||
in that case set to the anchor word.
|
||||
- ``matched_unit_token``: the surface noun for the operand's unit.
|
||||
For Rate operands, this is the numerator_unit surface form. For
|
||||
Comparison operands, this can be empty when the comparison uses
|
||||
an implicit unit ("Sam has twice as many as Tom" — no unit token).
|
||||
- ``matched_actor_token``: the actor's name as it appeared. Case-
|
||||
preserving; the filter does case-insensitive matching.
|
||||
- ``matched_target_token``: required iff ``op.kind == 'transfer'``;
|
||||
otherwise must be None.
|
||||
- ``matched_reference_actor_token``: required iff ``op.operand`` is
|
||||
a Comparison; otherwise must be None.
|
||||
"""
|
||||
|
||||
op: Operation
|
||||
source_span: str
|
||||
matched_verb: str
|
||||
matched_value_token: str
|
||||
matched_unit_token: str
|
||||
matched_actor_token: str
|
||||
matched_target_token: str | None = None
|
||||
matched_reference_actor_token: str | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not isinstance(self.source_span, str) or not self.source_span:
|
||||
raise ValueError("CandidateOperation.source_span must be non-empty")
|
||||
if not isinstance(self.matched_verb, str) or not self.matched_verb:
|
||||
raise ValueError("CandidateOperation.matched_verb must be non-empty")
|
||||
if not isinstance(self.matched_actor_token, str) or not self.matched_actor_token:
|
||||
raise ValueError(
|
||||
"CandidateOperation.matched_actor_token must be non-empty"
|
||||
)
|
||||
if self.op.kind == "transfer":
|
||||
if not self.matched_target_token:
|
||||
raise ValueError(
|
||||
"matched_target_token required when op.kind='transfer'"
|
||||
)
|
||||
elif self.matched_target_token is not None:
|
||||
raise ValueError(
|
||||
"matched_target_token only valid when op.kind='transfer'"
|
||||
)
|
||||
if isinstance(self.op.operand, Comparison):
|
||||
if not self.matched_reference_actor_token:
|
||||
raise ValueError(
|
||||
"matched_reference_actor_token required when operand is "
|
||||
"Comparison"
|
||||
)
|
||||
elif self.matched_reference_actor_token is not None:
|
||||
raise ValueError(
|
||||
"matched_reference_actor_token only valid when operand is "
|
||||
"Comparison"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Normalization + containment primitives.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_WORD_RE: Final[re.Pattern[str]] = re.compile(r"\b\w+\b", flags=re.UNICODE)
|
||||
|
||||
|
||||
def _tokens(text: str) -> frozenset[str]:
|
||||
"""Lowercased word-token set for word-boundary containment checks."""
|
||||
return frozenset(m.group(0).lower() for m in _WORD_RE.finditer(text))
|
||||
|
||||
|
||||
def _token_in(needle: str, haystack_tokens: frozenset[str]) -> bool:
|
||||
"""Word-boundary containment: 'ate' must not match 'states'."""
|
||||
return needle.lower() in haystack_tokens
|
||||
|
||||
|
||||
def _value_grounds(value_token: str, haystack_tokens: frozenset[str]) -> bool:
|
||||
"""A numeric value grounds if its surface token appears, OR if the token
|
||||
is a digit-string and any equivalent word-form appears, OR if it's a
|
||||
word-form and the digit appears."""
|
||||
if _token_in(value_token, haystack_tokens):
|
||||
return True
|
||||
# word -> digit equivalent
|
||||
lowered = value_token.lower()
|
||||
if lowered in WORD_NUMBERS:
|
||||
digit = str(WORD_NUMBERS[lowered])
|
||||
if digit in haystack_tokens:
|
||||
return True
|
||||
# digit -> any word with that integer value
|
||||
try:
|
||||
n = int(value_token)
|
||||
except ValueError:
|
||||
return False
|
||||
for word, w_val in WORD_NUMBERS.items():
|
||||
if w_val == n and word in haystack_tokens:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The load-bearing primitive.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def roundtrip_admissible(c: CandidateOperation) -> bool:
|
||||
"""True iff every content slot in ``c`` grounds in ``c.source_span``
|
||||
AND the matched verb is registered for the operation kind.
|
||||
|
||||
This is the deterministic wrong-answer firewall. A candidate that
|
||||
fails is silently dropped from the candidate set — it never reaches
|
||||
the solver, never produces a number, and never appears in any
|
||||
``SolutionTrace``.
|
||||
"""
|
||||
# 1. Verb must be registered for the claimed kind.
|
||||
valid_verbs = KIND_TO_VERBS.get(c.op.kind)
|
||||
if valid_verbs is None or c.matched_verb.lower() not in valid_verbs:
|
||||
return False
|
||||
|
||||
haystack = _tokens(c.source_span)
|
||||
|
||||
# 2. Matched verb must appear in source.
|
||||
if not _token_in(c.matched_verb, haystack):
|
||||
return False
|
||||
|
||||
# 3. Actor name must appear in source.
|
||||
if not _token_in(c.matched_actor_token, haystack):
|
||||
return False
|
||||
|
||||
# 4. Numeric value must ground.
|
||||
# Skipped only for multiplicative comparison anchors that carry
|
||||
# the factor implicitly ("twice", "half", "thrice") — those use
|
||||
# the anchor itself as the value token and pass via step (2).
|
||||
if c.op.kind == "compare_multiplicative" and c.matched_value_token == c.matched_verb:
|
||||
pass # anchor already grounded by verb check
|
||||
elif not _value_grounds(c.matched_value_token, haystack):
|
||||
return False
|
||||
|
||||
# 5. Unit must ground when non-empty. Empty unit token is only valid
|
||||
# for comparison operands without explicit unit phrasing
|
||||
# ("Sam has twice as many as Tom").
|
||||
if c.matched_unit_token:
|
||||
if not _token_in(c.matched_unit_token, haystack):
|
||||
return False
|
||||
else:
|
||||
if not isinstance(c.op.operand, Comparison):
|
||||
return False # only comparisons may have empty unit token
|
||||
|
||||
# 6. Transfer target must appear.
|
||||
if c.matched_target_token is not None:
|
||||
if not _token_in(c.matched_target_token, haystack):
|
||||
return False
|
||||
|
||||
# 7. Comparison reference_actor must appear.
|
||||
if c.matched_reference_actor_token is not None:
|
||||
if not _token_in(c.matched_reference_actor_token, haystack):
|
||||
return False
|
||||
|
||||
# 8. Operand kind/shape sanity (defense-in-depth — Operation
|
||||
# constructor already enforces shape, but we re-check kind ↔
|
||||
# operand-type consistency here so an upstream bug can't slip a
|
||||
# Quantity-as-Comparison candidate past the filter).
|
||||
if c.op.kind == "apply_rate":
|
||||
if not isinstance(c.op.operand, Rate):
|
||||
return False
|
||||
# Rate denominator unit must also ground.
|
||||
if not _token_in(c.op.operand.denominator_unit, haystack):
|
||||
return False
|
||||
elif c.op.kind in ("compare_additive", "compare_multiplicative"):
|
||||
if not isinstance(c.op.operand, Comparison):
|
||||
return False
|
||||
else:
|
||||
if not isinstance(c.op.operand, Quantity):
|
||||
return False
|
||||
|
||||
return True
|
||||
131
tests/test_adr_0126_runner_wiring.py
Normal file
131
tests/test_adr_0126_runner_wiring.py
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
"""ADR-0126 P4 — tests for the candidate-graph scorer wiring.
|
||||
|
||||
Proves :func:`evals.gsm8k_math.runner._score_one_candidate_graph`:
|
||||
|
||||
- Produces ``correct`` on simple cases that the legacy ``_score_one``
|
||||
also handles (no regression on solvable cases).
|
||||
- Produces ``correct`` on cases that the legacy ``_score_one`` would
|
||||
``refuse`` because of restrictive verb tables (the whole point of
|
||||
the architecture pivot).
|
||||
- Produces ``refused`` (never ``wrong``) on out-of-grammar cases —
|
||||
the ``wrong == 0`` invariant is preserved.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from evals.gsm8k_math.runner import _score_one, _score_one_candidate_graph
|
||||
|
||||
|
||||
def _case(problem: str, *, answer: float, unit: str = "") -> dict[str, object]:
|
||||
return {
|
||||
"id": "test-case",
|
||||
"problem": problem,
|
||||
"expected_answer": answer,
|
||||
"expected_unit": unit,
|
||||
}
|
||||
|
||||
|
||||
class TestNoRegressionOnLegacySolvable:
|
||||
"""Cases the legacy parser handles must still be correct."""
|
||||
|
||||
def test_simple_add(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 5 apples. Sam buys 3 apples. "
|
||||
"How many apples does Sam have?",
|
||||
answer=8.0, unit="apples",
|
||||
)
|
||||
# Both pipelines should produce correct.
|
||||
assert _score_one(case).outcome == "correct"
|
||||
assert _score_one_candidate_graph(case).outcome == "correct"
|
||||
|
||||
def test_simple_subtract(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 10 apples. Sam eats 3 apples. "
|
||||
"How many apples does Sam have?",
|
||||
answer=7.0, unit="apples",
|
||||
)
|
||||
assert _score_one(case).outcome == "correct"
|
||||
assert _score_one_candidate_graph(case).outcome == "correct"
|
||||
|
||||
def test_transfer(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 8 apples. Tom has 2 apples. "
|
||||
"Sam gives 3 apples to Tom. "
|
||||
"How many apples does Tom have?",
|
||||
answer=5.0, unit="apples",
|
||||
)
|
||||
assert _score_one(case).outcome == "correct"
|
||||
assert _score_one_candidate_graph(case).outcome == "correct"
|
||||
|
||||
|
||||
class TestLiftOnPermissiveVerbs:
|
||||
"""Cases the legacy parser refuses must now solve."""
|
||||
|
||||
def test_bought_past_tense(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 5 apples. Sam bought 3 apples. "
|
||||
"How many apples does Sam have?",
|
||||
answer=8.0, unit="apples",
|
||||
)
|
||||
legacy = _score_one(case)
|
||||
new = _score_one_candidate_graph(case)
|
||||
# Legacy refuses ('bought' not in _ADD_VERBS); new solves.
|
||||
assert legacy.outcome == "refused"
|
||||
assert new.outcome == "correct"
|
||||
|
||||
def test_ate_past_tense(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 10 apples. Sam ate 3 apples. "
|
||||
"How many apples does Sam have?",
|
||||
answer=7.0, unit="apples",
|
||||
)
|
||||
legacy = _score_one(case)
|
||||
new = _score_one_candidate_graph(case)
|
||||
assert legacy.outcome == "refused"
|
||||
assert new.outcome == "correct"
|
||||
|
||||
def test_bakes_production_verb(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 2 pies. Sam bakes 4 pies. "
|
||||
"How many pies does Sam have?",
|
||||
answer=6.0, unit="pies",
|
||||
)
|
||||
legacy = _score_one(case)
|
||||
new = _score_one_candidate_graph(case)
|
||||
assert legacy.outcome == "refused"
|
||||
assert new.outcome == "correct"
|
||||
|
||||
|
||||
class TestWrongZeroPreserved:
|
||||
"""Out-of-grammar cases must REFUSE, never wrong."""
|
||||
|
||||
def test_unparseable_refuses(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 5 apples. Sam contemplates 3 apples. "
|
||||
"How many apples does Sam have?",
|
||||
answer=8.0, unit="apples",
|
||||
)
|
||||
outcome = _score_one_candidate_graph(case)
|
||||
assert outcome.outcome == "refused"
|
||||
assert "no admissible candidate" in outcome.reason
|
||||
|
||||
def test_question_with_unknown_entity_refuses(self) -> None:
|
||||
case = _case(
|
||||
"Sam has 5 apples. "
|
||||
"How many apples does Alice have?",
|
||||
answer=0.0, unit="apples",
|
||||
)
|
||||
outcome = _score_one_candidate_graph(case)
|
||||
# Either refused (graph rejects unknown entity) or refused via
|
||||
# solve failure — both preserve wrong == 0.
|
||||
assert outcome.outcome == "refused"
|
||||
|
||||
def test_value_only_grading_for_train_sample_shape(self) -> None:
|
||||
# When expected_unit == "" (the train-sample shape), the runner
|
||||
# grades on numeric value alone.
|
||||
case = _case(
|
||||
"Sam has 5 apples. Sam buys 3 apples. "
|
||||
"How many apples does Sam have?",
|
||||
answer=8.0, unit="", # empty
|
||||
)
|
||||
assert _score_one_candidate_graph(case).outcome == "correct"
|
||||
226
tests/test_math_candidate_graph.py
Normal file
226
tests/test_math_candidate_graph.py
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
"""ADR-0126 P3 — tests for candidate-graph assembly + decision rule.
|
||||
|
||||
Proves the end-to-end candidate-graph pipeline:
|
||||
|
||||
text → per-sentence candidates → filter → branch enumeration
|
||||
→ per-branch solve → decision rule → answer | refusal
|
||||
|
||||
Critical assertions:
|
||||
|
||||
- Unambiguous problems produce a single answer.
|
||||
- Ambiguous-verb problems ('gives') resolve via the slot-count
|
||||
tiebreaker; both readings agree on the answer, so emission proceeds.
|
||||
- Out-of-grammar sentences refuse (no exception, deterministic
|
||||
refusal_reason string).
|
||||
- Branches that disagree on the answer refuse (wrong == 0 preserved).
|
||||
- Permissive verbs that the legacy parser refused now produce answers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from generate.math_candidate_graph import (
|
||||
MAX_TOTAL_BRANCHES,
|
||||
parse_and_solve,
|
||||
)
|
||||
from generate.math_candidate_parser import (
|
||||
extract_question_candidates,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Question extractor (P2 addition tested here for cohesion)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestQuestionExtraction:
|
||||
def test_entity_question(self) -> None:
|
||||
qcs = extract_question_candidates("How many apples does Sam have?")
|
||||
assert len(qcs) == 1
|
||||
assert qcs[0].unknown.entity == "Sam"
|
||||
assert qcs[0].unknown.unit == "apples"
|
||||
|
||||
def test_total_question(self) -> None:
|
||||
qcs = extract_question_candidates("How many apples do they have?")
|
||||
assert len(qcs) == 1
|
||||
assert qcs[0].unknown.entity is None
|
||||
assert qcs[0].unknown.unit == "apples"
|
||||
|
||||
def test_collective_entity_question(self) -> None:
|
||||
qcs = extract_question_candidates("How many cards do the girls have?")
|
||||
assert len(qcs) == 1
|
||||
assert qcs[0].unknown.entity == "the girls"
|
||||
|
||||
def test_with_trailing_modifier(self) -> None:
|
||||
qcs = extract_question_candidates(
|
||||
"How many apples does Sam have left?"
|
||||
)
|
||||
assert len(qcs) == 1
|
||||
assert qcs[0].unknown.entity == "Sam"
|
||||
|
||||
def test_no_match(self) -> None:
|
||||
assert extract_question_candidates("What is the answer?") == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# End-to-end happy path
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestHappyPath:
|
||||
def test_simple_add(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 5 apples. Sam buys 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 8
|
||||
|
||||
def test_simple_subtract(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 10 apples. Sam eats 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 7
|
||||
|
||||
def test_transfer(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 8 apples. Tom has 2 apples. "
|
||||
"Sam gives 3 apples to Tom. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 5
|
||||
|
||||
def test_transfer_other_side(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 8 apples. Tom has 2 apples. "
|
||||
"Sam gives 3 apples to Tom. "
|
||||
"How many apples does Tom have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 5
|
||||
|
||||
def test_total_across_entities(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 5 apples. Tom has 3 apples. "
|
||||
"How many apples do they have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 8
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Permissive verbs the legacy parser would have refused
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestPermissiveVerbsNowSolve:
|
||||
def test_past_tense_add(self) -> None:
|
||||
# 'bought' is permissive-only; the round-trip filter is what
|
||||
# makes it safe.
|
||||
result = parse_and_solve(
|
||||
"Sam has 5 apples. Sam bought 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 8
|
||||
|
||||
def test_past_tense_subtract(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 10 apples. Sam ate 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 7
|
||||
|
||||
def test_production_verb_bakes(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 2 pies. Sam bakes 4 pies. "
|
||||
"How many pies does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 6
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ambiguity that the slot-count tiebreaker resolves
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestAmbiguityResolution:
|
||||
def test_gives_with_target_resolves_to_transfer(self) -> None:
|
||||
# "Sam gives 3 apples to Tom" emits BOTH subtract and transfer
|
||||
# candidates per P2 tests. Both pass round-trip. The slot-count
|
||||
# tiebreaker collapses to transfer (more grounded slots), so
|
||||
# the graph is the transfer reading and Tom gets the apples.
|
||||
result = parse_and_solve(
|
||||
"Sam has 8 apples. Tom has 2 apples. "
|
||||
"Sam gives 3 apples to Tom. "
|
||||
"How many apples does Tom have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 5 # transfer reading: 2 + 3 = 5
|
||||
|
||||
def test_gives_without_target_resolves_to_subtract(self) -> None:
|
||||
# "Sam gives 3 apples" → only subtract candidate is admissible.
|
||||
result = parse_and_solve(
|
||||
"Sam has 8 apples. Sam gives 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.is_admitted
|
||||
assert result.answer == 5
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Refusals (preserve wrong == 0)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestRefusals:
|
||||
def test_empty_input(self) -> None:
|
||||
result = parse_and_solve("")
|
||||
assert not result.is_admitted
|
||||
assert "empty" in (result.refusal_reason or "").lower()
|
||||
|
||||
def test_no_question(self) -> None:
|
||||
result = parse_and_solve("Sam has 5 apples.")
|
||||
assert not result.is_admitted
|
||||
assert "question" in (result.refusal_reason or "").lower()
|
||||
|
||||
def test_unparseable_statement(self) -> None:
|
||||
# Verb not in any permissive table.
|
||||
result = parse_and_solve(
|
||||
"Sam has 5 apples. Sam contemplates 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert not result.is_admitted
|
||||
assert "no admissible candidate" in (result.refusal_reason or "")
|
||||
|
||||
def test_question_references_unknown_entity(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 5 apples. "
|
||||
"How many apples does Alice have?"
|
||||
)
|
||||
assert not result.is_admitted
|
||||
|
||||
def test_branch_count_cap_refuses(self) -> None:
|
||||
# Hard to construct without writing a multiplicatively-ambiguous
|
||||
# corpus; for now just assert the cap constant is sensible.
|
||||
assert MAX_TOTAL_BRANCHES == 64
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Diagnostics surfaced for P6 inner-loop signal
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestDiagnostics:
|
||||
def test_diagnostics_on_admission(self) -> None:
|
||||
result = parse_and_solve(
|
||||
"Sam has 5 apples. Sam buys 3 apples. "
|
||||
"How many apples does Sam have?"
|
||||
)
|
||||
assert result.branches_enumerated >= 1
|
||||
assert result.branches_admissible >= 1
|
||||
|
||||
def test_diagnostics_on_refusal(self) -> None:
|
||||
result = parse_and_solve("foobar baz quux?")
|
||||
# Refusal occurs before enumeration when no statement candidates
|
||||
# exist; diagnostics still report 0/0 cleanly.
|
||||
assert result.branches_enumerated == 0
|
||||
assert result.branches_admissible == 0
|
||||
191
tests/test_math_candidate_parser.py
Normal file
191
tests/test_math_candidate_parser.py
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
"""ADR-0126 — tests for the candidate-emitting parser (P2).
|
||||
|
||||
Proves the candidate-emission topology end-to-end against the round-trip
|
||||
filter from P1:
|
||||
|
||||
- Unambiguous sentences emit exactly one candidate, which the filter
|
||||
admits.
|
||||
- Ambiguous sentences (e.g. verb in both SUBTRACT_VERBS and
|
||||
TRANSFER_VERBS) emit multiple candidates; the filter admits the
|
||||
correct one based on grounded slots.
|
||||
- Out-of-grammar sentences emit zero candidates (no ParseError raised).
|
||||
- Permissive verbs not in the legacy math_parser tables (e.g. "bought",
|
||||
"lost", "gave") now produce admissible candidates — the whole point
|
||||
of P2 + filter.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from generate.math_candidate_parser import (
|
||||
extract_initial_candidates,
|
||||
extract_operation_candidates,
|
||||
)
|
||||
from generate.math_roundtrip import roundtrip_admissible
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Initial-possession extraction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestInitialExtraction:
|
||||
def test_single_entity_digit(self) -> None:
|
||||
cands = extract_initial_candidates("Sam has 5 apples.")
|
||||
assert len(cands) == 1
|
||||
c = cands[0]
|
||||
assert c.initial.entity == "Sam"
|
||||
assert c.initial.quantity.value == 5
|
||||
assert c.initial.quantity.unit == "apples"
|
||||
|
||||
def test_single_entity_word_number(self) -> None:
|
||||
cands = extract_initial_candidates("Sam has three apples.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].initial.quantity.value == 3
|
||||
|
||||
def test_collective_entity(self) -> None:
|
||||
cands = extract_initial_candidates("The boys have 10 marbles.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].initial.entity == "the boys"
|
||||
|
||||
def test_singular_unit_pluralized(self) -> None:
|
||||
cands = extract_initial_candidates("Sam has 1 apple.")
|
||||
assert len(cands) == 1
|
||||
# math_parser canonicalization rule: always pluralize
|
||||
assert cands[0].initial.quantity.unit == "apples"
|
||||
|
||||
def test_no_match_returns_empty(self) -> None:
|
||||
# Out-of-grammar shape — empty list, NOT an exception.
|
||||
assert extract_initial_candidates("Sam went to the store.") == []
|
||||
assert extract_initial_candidates("How many apples?") == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Operation extraction — unambiguous verbs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestUnambiguousOperations:
|
||||
def test_add_present_tense(self) -> None:
|
||||
cands = extract_operation_candidates("Sam buys 3 apples.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].op.kind == "add"
|
||||
assert cands[0].op.operand.value == 3
|
||||
assert roundtrip_admissible(cands[0])
|
||||
|
||||
def test_add_past_tense_permissive(self) -> None:
|
||||
# "bought" is in the new permissive ADD_VERBS but NOT in the
|
||||
# legacy math_parser._ADD_VERBS. The whole point of P2 is to
|
||||
# admit these via the round-trip filter.
|
||||
cands = extract_operation_candidates("Sam bought 3 apples.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].op.kind == "add"
|
||||
assert cands[0].matched_verb == "bought"
|
||||
assert roundtrip_admissible(cands[0])
|
||||
|
||||
def test_subtract_present_tense(self) -> None:
|
||||
cands = extract_operation_candidates("Sam eats 2 apples.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].op.kind == "subtract"
|
||||
assert roundtrip_admissible(cands[0])
|
||||
|
||||
def test_subtract_past_tense_permissive(self) -> None:
|
||||
# "ate" is in the new permissive SUBTRACT_VERBS but not legacy.
|
||||
cands = extract_operation_candidates("Sam ate 2 apples.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].op.kind == "subtract"
|
||||
assert cands[0].matched_verb == "ate"
|
||||
assert roundtrip_admissible(cands[0])
|
||||
|
||||
def test_production_verb_permissive(self) -> None:
|
||||
# "bakes" is a production verb — actor creates instances. Not
|
||||
# in legacy ADD_VERBS, accepted now via the permissive table.
|
||||
cands = extract_operation_candidates("Sam bakes 4 pies.")
|
||||
assert len(cands) == 1
|
||||
assert cands[0].op.kind == "add"
|
||||
assert cands[0].matched_verb == "bakes"
|
||||
assert roundtrip_admissible(cands[0])
|
||||
|
||||
def test_no_match_returns_empty(self) -> None:
|
||||
# Out-of-grammar: a verb we don't recognize at all.
|
||||
assert extract_operation_candidates("Sam contemplates 3 apples.") == []
|
||||
# Sentence missing required slots (no value).
|
||||
assert extract_operation_candidates("Sam buys apples.") == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Operation extraction — ambiguous verbs (THE key test for P2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestAmbiguousOperations:
|
||||
def test_gives_with_target_emits_subtract_and_transfer(self) -> None:
|
||||
# "gives" appears in both SUBTRACT_VERBS (intransitive-like
|
||||
# reading "Sam gives 3 apples") and TRANSFER_VERBS (transitive
|
||||
# "Sam gives 3 apples to Tom"). When a target IS present, both
|
||||
# candidates fire by design — the filter and decision rule
|
||||
# resolve the ambiguity downstream.
|
||||
cands = extract_operation_candidates("Sam gives 3 apples to Tom.")
|
||||
kinds = sorted(c.op.kind for c in cands)
|
||||
assert kinds == ["subtract", "transfer"]
|
||||
|
||||
def test_filter_admits_both_for_gives_to_target(self) -> None:
|
||||
# Both candidates pass round-trip — neither claims a slot that
|
||||
# isn't in the source. The P3 decision rule will need a
|
||||
# tiebreaker (most-grounded-slots-wins is one option). This
|
||||
# test pins the current filter behavior; the tiebreaker is
|
||||
# P3's responsibility.
|
||||
cands = extract_operation_candidates("Sam gives 3 apples to Tom.")
|
||||
admitted = [c for c in cands if roundtrip_admissible(c)]
|
||||
assert len(admitted) == 2
|
||||
# Transfer candidate has a target slot (4 grounded entities),
|
||||
# subtract candidate does not (3 grounded entities). Slot count
|
||||
# is the structural signal P3 will use.
|
||||
|
||||
def test_gives_without_target_only_subtract_admits(self) -> None:
|
||||
# "Sam gives 3 apples." — no target slot in source. The
|
||||
# transfer pattern requires a "to <Target>" clause and won't
|
||||
# match; subtract pattern matches and is admissible.
|
||||
cands = extract_operation_candidates("Sam gives 3 apples.")
|
||||
admitted = [c for c in cands if roundtrip_admissible(c)]
|
||||
assert len(admitted) == 1
|
||||
assert admitted[0].op.kind == "subtract"
|
||||
|
||||
def test_returns_emits_both_subtract_and_transfer(self) -> None:
|
||||
# "returns" is also overloaded.
|
||||
cands = extract_operation_candidates("Sam returns 2 books to Tom.")
|
||||
kinds = sorted(c.op.kind for c in cands)
|
||||
assert kinds == ["subtract", "transfer"]
|
||||
admitted = [c for c in cands if roundtrip_admissible(c)]
|
||||
assert len(admitted) == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Wrong-answer firewall demonstrated end-to-end
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestFirewallEndToEnd:
|
||||
def test_filter_rejects_when_legacy_parser_would_have_misparsed(self) -> None:
|
||||
# Imagine the old parser had a bug where "loses" was registered
|
||||
# as ADD. Under candidate-graph, even if such a buggy candidate
|
||||
# were emitted, the round-trip filter would catch it because
|
||||
# "loses" is not in ADD_VERBS.
|
||||
#
|
||||
# We simulate by constructing the buggy candidate by hand and
|
||||
# showing the filter rejects it.
|
||||
from generate.math_problem_graph import Operation, Quantity
|
||||
from generate.math_roundtrip import CandidateOperation
|
||||
buggy = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=2, unit="apples")),
|
||||
source_span="Sam loses 2 apples.",
|
||||
matched_verb="loses", # the bug
|
||||
matched_value_token="2",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert not roundtrip_admissible(buggy)
|
||||
|
||||
def test_correct_subtract_candidate_for_loses_is_admissible(self) -> None:
|
||||
# And the correct subtract reading IS emitted by the extractor.
|
||||
cands = extract_operation_candidates("Sam loses 2 apples.")
|
||||
admitted = [c for c in cands if roundtrip_admissible(c)]
|
||||
assert len(admitted) == 1
|
||||
assert admitted[0].op.kind == "subtract"
|
||||
assert admitted[0].matched_verb == "loses"
|
||||
404
tests/test_math_roundtrip.py
Normal file
404
tests/test_math_roundtrip.py
Normal file
|
|
@ -0,0 +1,404 @@
|
|||
"""ADR-0126 — tests for the round-trip admissibility filter.
|
||||
|
||||
Exercises every rejection criterion. If a single test here fails, the
|
||||
wrong-answer firewall has a hole and the whole candidate-graph
|
||||
architecture is unsound — these are the load-bearing assertions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.math_problem_graph import (
|
||||
Comparison,
|
||||
Operation,
|
||||
Quantity,
|
||||
Rate,
|
||||
)
|
||||
from generate.math_roundtrip import (
|
||||
ADD_VERBS,
|
||||
KIND_TO_VERBS,
|
||||
SUBTRACT_VERBS,
|
||||
CandidateOperation,
|
||||
_tokens,
|
||||
roundtrip_admissible,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sanity: verb registry contracts.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestVerbRegistry:
|
||||
def test_kind_to_verbs_covers_all_operation_kinds(self) -> None:
|
||||
from generate.math_problem_graph import VALID_OPERATION_KINDS
|
||||
for kind in VALID_OPERATION_KINDS:
|
||||
assert kind in KIND_TO_VERBS, f"{kind} missing from KIND_TO_VERBS"
|
||||
|
||||
def test_add_subtract_disjoint_modulo_known_overlaps(self) -> None:
|
||||
# The two ambiguous overlap groups (give/send/return) are
|
||||
# intentional — they fire as both subtract candidates and
|
||||
# transfer candidates; the filter rejects whichever does not
|
||||
# match the grounded slots.
|
||||
intentional_overlap = {"give", "gives", "gave", "send", "sends", "sent",
|
||||
"return", "returns", "returned"}
|
||||
unintentional = (ADD_VERBS & SUBTRACT_VERBS) - intentional_overlap
|
||||
assert unintentional == frozenset()
|
||||
|
||||
def test_tokenization_word_boundary(self) -> None:
|
||||
toks = _tokens("Sam ate three apples.")
|
||||
assert "ate" in toks
|
||||
assert "states" not in toks
|
||||
assert "apples" in toks # plural NOT stemmed
|
||||
assert "apple" not in toks
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Positive cases — every operation kind admits a clean candidate.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestPositiveAdmission:
|
||||
def test_add_with_digit(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_add_with_word_number(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys three apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="three",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_subtract_past_tense(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="subtract",
|
||||
operand=Quantity(value=2, unit="apples")),
|
||||
source_span="Sam ate 2 apples.",
|
||||
matched_verb="ate",
|
||||
matched_value_token="2",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_transfer(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="transfer",
|
||||
operand=Quantity(value=4, unit="apples"),
|
||||
target="Tom"),
|
||||
source_span="Sam gives 4 apples to Tom.",
|
||||
matched_verb="gives",
|
||||
matched_value_token="4",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_target_token="Tom",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_apply_rate(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="apply_rate",
|
||||
operand=Rate(value=2.0, numerator_unit="dollars",
|
||||
denominator_unit="apple")),
|
||||
source_span="Apples cost 2 dollars per apple.",
|
||||
matched_verb="per",
|
||||
matched_value_token="2",
|
||||
matched_unit_token="dollars",
|
||||
matched_actor_token="Apples",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_compare_additive(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(
|
||||
actor="Sam", kind="compare_additive",
|
||||
operand=Comparison(
|
||||
reference_actor="Tom",
|
||||
delta=Quantity(value=3, unit="apples"),
|
||||
factor=None, direction="more",
|
||||
),
|
||||
),
|
||||
source_span="Sam has 3 more apples than Tom.",
|
||||
matched_verb="more",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_reference_actor_token="Tom",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_compare_multiplicative_anchor(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(
|
||||
actor="Sam", kind="compare_multiplicative",
|
||||
operand=Comparison(
|
||||
reference_actor="Tom", delta=None, factor=2.0,
|
||||
direction="times",
|
||||
),
|
||||
),
|
||||
source_span="Sam has twice as many apples as Tom.",
|
||||
matched_verb="twice",
|
||||
matched_value_token="twice", # anchor carries the factor
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_reference_actor_token="Tom",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_compare_multiplicative_implicit_unit(self) -> None:
|
||||
# "Sam has twice as many as Tom" — no unit token in source
|
||||
c = CandidateOperation(
|
||||
op=Operation(
|
||||
actor="Sam", kind="compare_multiplicative",
|
||||
operand=Comparison(
|
||||
reference_actor="Tom", delta=None, factor=2.0,
|
||||
direction="times",
|
||||
),
|
||||
),
|
||||
source_span="Sam has twice as many as Tom.",
|
||||
matched_verb="twice",
|
||||
matched_value_token="twice",
|
||||
matched_unit_token="", # empty allowed for comparisons
|
||||
matched_actor_token="Sam",
|
||||
matched_reference_actor_token="Tom",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Negative cases — the wrong-answer firewall.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestRejection:
|
||||
def test_rejects_verb_not_registered_for_kind(self) -> None:
|
||||
# Parser hallucinates: "loses" claimed as add. "loses" is in
|
||||
# SUBTRACT_VERBS, not ADD_VERBS, so this must reject.
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=2, unit="apples")),
|
||||
source_span="Sam loses 2 apples.",
|
||||
matched_verb="loses", # wrong kind!
|
||||
matched_value_token="2",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_verb_absent_from_source(self) -> None:
|
||||
# Parser hallucinates: "buys" claimed but source doesn't contain it.
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=2, unit="apples")),
|
||||
source_span="Sam has 2 apples.", # no "buys"
|
||||
matched_verb="buys",
|
||||
matched_value_token="2",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_value_absent_from_source(self) -> None:
|
||||
# Parser hallucinates: claims value 7 but source has 3.
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=7, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="7", # not in source
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_unit_absent_from_source(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="oranges")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="oranges", # not in source
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_actor_absent_from_source(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Tom", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Tom", # not in source
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_transfer_target_absent_from_source(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="transfer",
|
||||
operand=Quantity(value=4, unit="apples"),
|
||||
target="Alice"),
|
||||
source_span="Sam gives 4 apples to Tom.",
|
||||
matched_verb="gives",
|
||||
matched_value_token="4",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_target_token="Alice", # not in source
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_comparison_reference_absent_from_source(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(
|
||||
actor="Sam", kind="compare_additive",
|
||||
operand=Comparison(
|
||||
reference_actor="Alice",
|
||||
delta=Quantity(value=3, unit="apples"),
|
||||
factor=None, direction="more",
|
||||
),
|
||||
),
|
||||
source_span="Sam has 3 more apples than Tom.",
|
||||
matched_verb="more",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_reference_actor_token="Alice", # not in source
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_rate_denominator_unit_absent(self) -> None:
|
||||
# Rate claims "per banana" but source says "per apple".
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Apples", kind="apply_rate",
|
||||
operand=Rate(value=2.0, numerator_unit="dollars",
|
||||
denominator_unit="banana")),
|
||||
source_span="Apples cost 2 dollars per apple.",
|
||||
matched_verb="per",
|
||||
matched_value_token="2",
|
||||
matched_unit_token="dollars",
|
||||
matched_actor_token="Apples",
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
def test_rejects_empty_unit_for_non_comparison(self) -> None:
|
||||
# Empty unit token is only legal for comparison operands.
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="", # empty not allowed for add
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert not roundtrip_admissible(c)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Number-form grounding cross-equivalence.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestNumberGrounding:
|
||||
def test_digit_grounds_when_source_uses_word(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys three apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3", # source has "three"
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
def test_word_grounds_when_source_uses_digit(self) -> None:
|
||||
c = CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="three", # source has "3"
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
)
|
||||
assert roundtrip_admissible(c)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constructor validation — illegal CandidateOperation states are
|
||||
# refused at construction (not at filter time).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestConstructorInvariants:
|
||||
def test_transfer_requires_target_token(self) -> None:
|
||||
with pytest.raises(ValueError, match="matched_target_token required"):
|
||||
CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="transfer",
|
||||
operand=Quantity(value=4, unit="apples"),
|
||||
target="Tom"),
|
||||
source_span="Sam gives 4 apples to Tom.",
|
||||
matched_verb="gives",
|
||||
matched_value_token="4",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_target_token=None, # missing
|
||||
)
|
||||
|
||||
def test_non_transfer_must_not_carry_target_token(self) -> None:
|
||||
with pytest.raises(ValueError, match="matched_target_token only valid"):
|
||||
CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_target_token="Tom", # not allowed for add
|
||||
)
|
||||
|
||||
def test_comparison_requires_reference_actor_token(self) -> None:
|
||||
with pytest.raises(ValueError, match="matched_reference_actor_token required"):
|
||||
CandidateOperation(
|
||||
op=Operation(
|
||||
actor="Sam", kind="compare_additive",
|
||||
operand=Comparison(
|
||||
reference_actor="Tom",
|
||||
delta=Quantity(value=3, unit="apples"),
|
||||
factor=None, direction="more",
|
||||
),
|
||||
),
|
||||
source_span="Sam has 3 more apples than Tom.",
|
||||
matched_verb="more",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_reference_actor_token=None, # missing
|
||||
)
|
||||
|
||||
def test_non_comparison_must_not_carry_reference_actor_token(self) -> None:
|
||||
with pytest.raises(ValueError, match="matched_reference_actor_token only valid"):
|
||||
CandidateOperation(
|
||||
op=Operation(actor="Sam", kind="add",
|
||||
operand=Quantity(value=3, unit="apples")),
|
||||
source_span="Sam buys 3 apples.",
|
||||
matched_verb="buys",
|
||||
matched_value_token="3",
|
||||
matched_unit_token="apples",
|
||||
matched_actor_token="Sam",
|
||||
matched_reference_actor_token="Tom", # not allowed
|
||||
)
|
||||
Loading…
Reference in a new issue