docs(ADR-0127): units pack + units-aware parser + conversion graph (proposed)
Diagnostic from ADR-0126's first train-sample run (0/0/50): every refusal happens at the first statement of each problem, and every refused first statement fails on the unit-of-measurement construction, not on the operation grammar. Adding more verb regexes is the per-axis treadmill that produced 4 zero-lift ADRs. Units form a finite, externally well-defined ontology (NIST SI tables, currency, English container nouns) that is semantic substrate the candidate-graph parser was designed to consume. Scope: - en_units_v1 pack: dimensions, units (<=60), containers, rate connectors - conversions.jsonl: directed weighted graph of within-dimension unit pairs - 3 new initial-possession shapes + rate-declaration extractor in the candidate parser - Round-trip filter gains optional pack-typed-unit check - Solver gains dimensional canonicalization helper (shortest path through conversion graph); fired edges join SolutionTrace.steps for replay - Pack ratification invariants: round-trip identity, per-dimension connectivity, path consistency, canonical unit per dimension Wire the same train-sample exit criterion as ADR-0126 (correct >=10/50, wrong==0). If passed -> sealed holdout. If still missed -> Path B trigger is REAL (full deterministic design with units substrate failed), demote GSM8K, re-target math expert promotion. Also commits the empirical evidence: train_sample/v1/runner.py swapped _score_one -> _score_one_candidate_graph; report.json baseline 0/0/50 confirming the candidate-graph topology refuses cleanly without units substrate.
This commit is contained in:
Shay
parent
feeb64818c
commit
fde62d713d
3 changed files with 391 additions and 52 deletions
339
docs/decisions/ADR-0127-units-pack-and-units-aware-parser.md
Normal file
339
docs/decisions/ADR-0127-units-pack-and-units-aware-parser.md
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
# ADR-0127 — `en_units_v1` Pack + Units-Aware Candidate Extractors
|
||||
|
||||
**Status:** Proposed (scope-only; implementation follow-up to ADR-0126)
|
||||
**Date:** 2026-05-23
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:**
|
||||
- ADR-0115 / 0116 / 0117 / 0118 (parser / solver / verifier / realizer)
|
||||
- ADR-0122 (Rate operand)
|
||||
- ADR-0126 (candidate-graph parser + round-trip filter)
|
||||
**Supersedes:** none
|
||||
**Blocks:** the Path-B decision for the GSM8K-math lane (the real
|
||||
empirical answer can't be obtained until the parser consumes a
|
||||
units pack).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0126's first empirical run produced **0 / 50 correct, 0 wrong,
|
||||
50 refused** on the GSM8K train-sample inner-loop gate. Inspection
|
||||
of the refusal reasons revealed a pattern: every refusal happens at
|
||||
the *first statement* of each problem, and every refused first
|
||||
statement shares the same shape — **the parser fails on the
|
||||
unit-of-measurement construction, not on the operation grammar**.
|
||||
|
||||
Representative train-sample first statements (all refused):
|
||||
|
||||
| Case | First statement | What's structurally not recognized |
|
||||
|------|----------------|-----------------------------------|
|
||||
| 1 | `Tina makes $18.00 an hour.` | Money + time = rate dimension |
|
||||
| 2 | `Jan buys 1000 feet of cable.` | Length unit + substance qualifier |
|
||||
| 3 | `... bookstore donated 48 boxes of erasers.` | Container + count-content |
|
||||
| 5 | `In one hour, Addison mountain's temperature ...` | Temperature dimension + fractional ratio |
|
||||
|
||||
The shared failure mode is not "unknown verb" or "unknown sentence
|
||||
shape." It is "the parser has no ontology of units." The current
|
||||
parser asks "is `feet` a valid noun?" via `_canonical_unit` which
|
||||
just pluralizes. The right question is "what *dimension* is `feet`,
|
||||
and does `of cable` modify it as a substance qualifier?"
|
||||
|
||||
This is structurally different from the per-axis grammar treadmill
|
||||
that produced four zero-lift ADRs (0122 / 0123 / 0123a / 0123b).
|
||||
Vocabulary expansion is unbounded and adversarial; **units of
|
||||
measurement form a finite, externally well-defined ontology**
|
||||
(NIST SI tables for physical units; closed sets for currency, time,
|
||||
English container nouns). A units pack is *semantic substrate the
|
||||
parser consults*, not more grammar regex.
|
||||
|
||||
ADR-0126's candidate-graph topology is built to consume exactly
|
||||
this kind of substrate — the round-trip filter already does
|
||||
token-level grounding; adding a "matched_unit_token must resolve
|
||||
to a known dimension in the pack" check is a natural extension.
|
||||
|
||||
Without ADR-0127 the ADR-0126 empirical result is uninformative:
|
||||
0/50 reflects P2's deliberate minimum-viable scope, not the
|
||||
architecture's capacity. The architecture's real verdict lives
|
||||
behind the units pack.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
Add a `language_packs/data/en_units_v1/` ratified semantic pack and
|
||||
extend the ADR-0126 candidate parser to consult it during candidate
|
||||
emission. **No new operation kinds.** **No new ADR-0114a
|
||||
obligations.** **No new exit gates beyond ADR-0126's** (`correct ≥
|
||||
10/50, wrong == 0` on the train sample, this time with units pack
|
||||
in scope).
|
||||
|
||||
### Pack content (`en_units_v1`)
|
||||
|
||||
Structure mirrors `en_arithmetic_v1`: `lexicon.jsonl` +
|
||||
`manifest.json` + `glosses.jsonl` + a self-sealing
|
||||
`.mastery_report.json` ratification artifact.
|
||||
|
||||
**Dimension classes** (entry_id prefix `en-units-dim-`):
|
||||
|
||||
| Dimension | Notes |
|
||||
|-----------|-------|
|
||||
| `count` | Discrete countable nouns (apples, kids, boxes, books) |
|
||||
| `length` | feet, inches, miles, yards, meters, centimeters |
|
||||
| `time` | hours, minutes, days, weeks, months, years |
|
||||
| `mass` | pounds, ounces, kilograms, grams |
|
||||
| `money` | dollars, cents (anchored by `$` / `¢` symbols) |
|
||||
| `temperature` | degrees, Fahrenheit, Celsius |
|
||||
| `volume` | gallons, quarts, cups, liters |
|
||||
|
||||
**Unit lemmas** (entry_id prefix `en-units-unit-`):
|
||||
|
||||
Each unit binds to exactly one dimension. Carries `singular`,
|
||||
`plural`, and `symbol` (where applicable: `$`, `°`). Initial pack
|
||||
targets ≤ 60 lemmas — bounded by GSM8K-frequent vocabulary, not
|
||||
NIST-complete coverage.
|
||||
|
||||
**Container nouns** (entry_id prefix `en-units-container-`):
|
||||
|
||||
Closed set: `box`, `bag`, `pack`, `basket`, `batch`, `dozen`,
|
||||
`group`, `set`, `case`, `pair`, `pile`, `stack`. Each declares
|
||||
its canonical syntax (`<count> <container> of <content>`) and an
|
||||
optional `default_size` (only `dozen=12`, `pair=2`, others null).
|
||||
|
||||
**Rate connectors** (entry_id prefix `en-units-rate-`):
|
||||
|
||||
Closed set: `per`, `an`, `every`, `each`. Each declares the
|
||||
template `<money|count> <rate_connector> <time|count>` it
|
||||
participates in.
|
||||
|
||||
**Substance qualifiers** (entry_id prefix `en-units-substance-`):
|
||||
|
||||
Pattern `N <unit> of <substance>` (e.g., "1000 feet of cable").
|
||||
The pack does NOT enumerate substances — that's open class. It
|
||||
encodes the *structural rule* that any measure-dimension unit
|
||||
admits an `of <NP>` substance tail, which the round-trip filter
|
||||
treats as a discarded grounded modifier.
|
||||
|
||||
**Unit-conversion graph** (`conversions.jsonl`, separate file):
|
||||
|
||||
Each line declares a directed edge `(from_unit, to_unit, ratio)`
|
||||
where `to_unit = ratio × from_unit`. The pack ratifies the
|
||||
*connected* weighted graph per dimension. Examples:
|
||||
|
||||
```jsonl
|
||||
{"edge_id":"en-units-conv-001","from":"inches","to":"feet","ratio":0.0833333333,"dimension":"length"}
|
||||
{"edge_id":"en-units-conv-002","from":"feet","to":"inches","ratio":12,"dimension":"length"}
|
||||
{"edge_id":"en-units-conv-003","from":"cents","to":"dollars","ratio":0.01,"dimension":"money"}
|
||||
{"edge_id":"en-units-conv-004","from":"dollars","to":"cents","ratio":100,"dimension":"money"}
|
||||
{"edge_id":"en-units-conv-005","from":"minutes","to":"hours","ratio":0.01666666667,"dimension":"time"}
|
||||
{"edge_id":"en-units-conv-006","from":"hours","to":"days","ratio":0.04166666667,"dimension":"time"}
|
||||
```
|
||||
|
||||
The graph is **the conversion table as data**, not as code. The
|
||||
solver consults it; the parser doesn't need to. Initial coverage
|
||||
target: complete edges for `length`, `time`, `money`, `mass`,
|
||||
`volume` between GSM8K-frequent unit pairs — bounded by NIST SI
|
||||
tables (closed set), not free expansion.
|
||||
|
||||
### Why conversions matter
|
||||
|
||||
Without conversions, the pack solves only single-unit arithmetic
|
||||
(`5 apples + 3 apples`). With conversions, it solves
|
||||
within-dimension mixed-unit arithmetic — which is the majority
|
||||
of real word problems:
|
||||
|
||||
| Without conversions | With conversions |
|
||||
|--------------------|------------------|
|
||||
| `5 feet + 8 inches = ?` (refused) | canonicalize to feet (or inches), add, emit |
|
||||
| `$2 + 75¢ = ?` (refused) | canonicalize to cents (or dollars), add, emit |
|
||||
| `2 hours 30 minutes` (refused) | canonicalize to minutes (or hours), normalize |
|
||||
|
||||
Solver-side responsibility (delegated to ADR-0127.5 below):
|
||||
when an operation's operand has a unit in the same dimension as
|
||||
the actor's last quantity but a different unit, the solver
|
||||
canonicalizes via shortest path in the conversion graph before
|
||||
performing arithmetic. The canonical_bytes of `SolutionTrace`
|
||||
records which edges fired, preserving determinism and
|
||||
replay-equality.
|
||||
|
||||
### Graph ratification invariants (`en_units_v1`-specific)
|
||||
|
||||
The pack ratification process validates *graph correctness*, not
|
||||
just lexicon well-formedness:
|
||||
|
||||
1. **Round-trip identity** — for every edge `(A, B, r)`, there
|
||||
must exist an edge `(B, A, 1/r)` with `|round_trip_error| <
|
||||
1e-9`. Asymmetric tables are rejected at ratification.
|
||||
2. **Per-dimension connectivity** — within each dimension, the
|
||||
subgraph induced by that dimension's units must be connected
|
||||
(every unit reachable from every other). Isolated unit
|
||||
lemmas are rejected.
|
||||
3. **Path consistency** — for any two units A and C in the same
|
||||
dimension, all shortest paths from A to C must yield the
|
||||
same product of ratios within `1e-9`. Inconsistent paths
|
||||
(e.g., 12 in/ft × 3 ft/yd ≠ 1 yd/36 in) are rejected at
|
||||
ratification, not at runtime.
|
||||
4. **Canonical unit per dimension** — each dimension declares
|
||||
one canonical unit (`feet` for length, `seconds` or
|
||||
`minutes` for time, `dollars` for money, etc.). All
|
||||
canonicalization routes through the canonical unit; this
|
||||
bounds the shortest-path computation to O(1) lookups.
|
||||
|
||||
These invariants live in `tests/test_adr_0127_pack_ratification.py`
|
||||
and run at every pack-change PR — the conversion graph cannot
|
||||
ship broken.
|
||||
|
||||
### Parser integration
|
||||
|
||||
Three load-bearing changes to `generate/math_candidate_parser.py`
|
||||
(no changes to legacy `math_parser.py`):
|
||||
|
||||
1. **`extract_initial_candidates` widens** to recognize three
|
||||
additional shapes when pack consultation confirms dimensional
|
||||
typing:
|
||||
- `<Entity> has N <pack-unit>` (with `of <substance>` tail
|
||||
discarded)
|
||||
- `<Entity> has N <pack-container> of <content>`
|
||||
- `There are N <pack-unit> [of <substance>]`
|
||||
|
||||
2. **New `extract_rate_declaration_candidates`** recognizes
|
||||
`<Entity> makes $N <rate-connector> <pack-time-unit>` as an
|
||||
`apply_rate` candidate (ADR-0122 shape; the pack supplies the
|
||||
dimensional check that today's narrow regex misses).
|
||||
|
||||
3. **Round-trip filter gains a dimensional check.** The existing
|
||||
`roundtrip_admissible` adds an optional `require_pack_typed_unit`
|
||||
parameter (default False to preserve P1 behavior). When True,
|
||||
`matched_unit_token` must resolve to a known unit lemma in
|
||||
`en_units_v1`. This is the wrong-answer firewall for unit
|
||||
hallucination: a parser that fires on `Sam buys 3 contemplations`
|
||||
would now fail because `contemplations` is not pack-typed.
|
||||
|
||||
4. **Solver gains a dimensional-canonicalization helper.** New
|
||||
module `generate/math_unit_conversion.py` exports
|
||||
`canonicalize_to_dimension_canonical(quantity, conversion_graph)
|
||||
-> (quantity_in_canonical_unit, [edges_fired])`. Operations whose
|
||||
operand unit differs from the actor's tracked unit (but shares
|
||||
dimension) get canonicalized before arithmetic. The fired-edges
|
||||
list joins `SolutionTrace.steps` so replay reproduces the
|
||||
conversion path byte-equal. Mixed-dimension operands
|
||||
(`apples + dollars`) remain a SolveError — units type the
|
||||
arithmetic.
|
||||
|
||||
### What ADR-0127 explicitly does NOT do
|
||||
|
||||
- Does NOT model fractional-of-prior-quantity phrasing
|
||||
(`3/4 of its temperature`). That's a separate compositional
|
||||
pattern, not a unit-ontology question.
|
||||
- Does NOT model multi-step rate compositions
|
||||
(`overtime = base + 1/2 base`). That's solver-level.
|
||||
- Does NOT add new operation kinds. ADR-0126 + 0122 cover the
|
||||
needed shapes (add, subtract, transfer, multiply, divide,
|
||||
apply_rate, compare_*). The pack just helps the parser
|
||||
*recognize* the operand inputs.
|
||||
- Does NOT replace `_canonical_unit`. The pack lookup is
|
||||
*additive* — when the pack confirms a token, dimensional
|
||||
routing kicks in; when not, the legacy plural-canonicalization
|
||||
fallback remains.
|
||||
- Does NOT touch the sealed holdout. Re-runs against the
|
||||
unsealed train sample only.
|
||||
|
||||
---
|
||||
|
||||
## Invariants preserved / added
|
||||
|
||||
| Invariant | Preserved or added | How |
|
||||
|-----------|--------------------|-----|
|
||||
| `wrong == 0` | Preserved | Pack-typed unit check is a *stricter* gate, never a looser one |
|
||||
| `trace_hash` byte-equality | Preserved | Pack consultation is deterministic; same pack version → same lookup result |
|
||||
| Pack-binding (ADR-0114a #10) | Reinforced | The parser now explicitly cites pack entry_ids in `SolutionTrace.provenance` for unit-typed candidates |
|
||||
| Round-trip admissibility | Strengthened | When `require_pack_typed_unit=True`, hallucinated units fail to ground |
|
||||
| `versor_condition(F) < 1e-6` | Untouched | No runtime field changes |
|
||||
| Manifest checksum SHA-256 of bytes-on-disk | Required | Mastery report self-seals like `en_arithmetic_v1` |
|
||||
|
||||
## Exit criterion
|
||||
|
||||
**Same gate as ADR-0126 but with `en_units_v1` mounted:**
|
||||
|
||||
```
|
||||
correct >= 10 / 50 on evals/gsm8k_math/train_sample/v1/cases.jsonl
|
||||
wrong == 0
|
||||
```
|
||||
|
||||
**If passed:** run sealed holdout *once* and freeze the number in
|
||||
ADR-0127-results. Architecture + units substrate jointly validated.
|
||||
|
||||
**If missed (Path-B trigger, now real):** the deterministic
|
||||
parser-by-rule approach + units ontology + candidate-graph
|
||||
topology is the *full* design we believed was right, and it still
|
||||
doesn't move GSM8K. That is the empirical signal to demote GSM8K
|
||||
and re-target the math expert promotion to a benchmark where
|
||||
exact-recall and determinism are the discriminators.
|
||||
|
||||
---
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### A. Skip the units pack, just expand parser regex.
|
||||
Rejected — this IS the per-axis grammar treadmill (4 zero-lift
|
||||
ADRs). Adding patterns like `<Entity> has N \w+ of \w+` without
|
||||
dimensional typing produces wrong candidates the round-trip
|
||||
filter can't catch (any noun would ground; nothing rejects "5
|
||||
contemplations of cable").
|
||||
|
||||
### B. Pull units from a third-party library (Pint, etc.).
|
||||
Rejected — violates "no opaque dependencies in the runtime path."
|
||||
A curated pack of ≤ 60 lemmas is auditable, version-pinnable,
|
||||
ratifiable, and inspectable; an external library is none of those.
|
||||
|
||||
### C. Defer units, ship ADR-0126 with the honest 0/50 + open
|
||||
question. Considered. Discarded because 0126's empirical result
|
||||
is genuinely uninformative without the substrate it was designed
|
||||
to consume. Shipping the architecture with a misleading "0 lift"
|
||||
headline invites the wrong conclusion (Path B prematurely).
|
||||
|
||||
### D. Build the units pack but DON'T integrate it; ship as
|
||||
inventory. Rejected — the pack must be load-bearing per CLAUDE.md
|
||||
"prefer compact, curated packs" plus the project's general stance
|
||||
against decoration without integration.
|
||||
|
||||
---
|
||||
|
||||
## Implementation plan (proposed sub-phases)
|
||||
|
||||
| Phase | Module | Description |
|
||||
|-------|--------|-------------|
|
||||
| 0127.1 | `language_packs/data/en_units_v1/` | Pack content: lexicon (dimensions + units + containers + rate connectors) + `conversions.jsonl` + manifest + glosses + mastery report |
|
||||
| 0127.2 | `language_packs/loader.py` (or sibling) | Pack loader API: `lookup_unit(token) -> UnitEntry \| None`; `get_conversion_graph(dimension) -> ConversionGraph` |
|
||||
| 0127.3 | `generate/math_roundtrip.py` | Optional `require_pack_typed_unit` parameter on `roundtrip_admissible` |
|
||||
| 0127.4 | `generate/math_candidate_parser.py` | Three new initial-possession shapes + rate-declaration extractor, consulting the pack loader |
|
||||
| 0127.5 | `generate/math_unit_conversion.py` (new) | `canonicalize_to_dimension_canonical(quantity, graph)`; shortest-path lookup + edges-fired provenance for `SolutionTrace.steps` |
|
||||
| 0127.6 | `generate/math_solver.py` | Wire dimensional canonicalization into add/subtract/transfer/compare arithmetic; mixed-dimension operands → SolveError |
|
||||
| 0127.7 | `evals/gsm8k_math/train_sample/v1/runner.py` | Re-run with units pack engaged; new `report.json` |
|
||||
| 0127.8 | `tests/test_adr_0127_*.py` | Pack ratification (round-trip identity + connectivity + path consistency + canonical-unit-per-dimension) + parser integration + solver canonicalization + train-sample lift gate |
|
||||
|
||||
Regression gates (must remain green at every phase):
|
||||
- `core test --suite smoke -q`
|
||||
- `core test --suite math -q` (existing 714/714 + ADR-0126 74/74)
|
||||
- `core test --suite packs -q` (new `en_units_v1` ratification entries)
|
||||
- ADR-0126 P3+P4 tests (the candidate-graph machinery is unchanged)
|
||||
|
||||
## PR checklist (when proposing for merge)
|
||||
|
||||
```
|
||||
What capability did this add?
|
||||
→ Pack-typed unit recognition for the candidate-graph parser; the
|
||||
semantic substrate ADR-0126 was designed to consume.
|
||||
What invariant proves the field remains valid?
|
||||
→ Pack-typed unit check (new); wrong == 0 (preserved); manifest
|
||||
checksum SHA-256 of bytes-on-disk (required).
|
||||
Which CLI suite/eval proves the lane?
|
||||
→ smoke + math + packs + train_sample_runner (this is the
|
||||
re-measurement that decides Path B or not).
|
||||
Did this avoid hidden normalization, stochastic fallback,
|
||||
approximate recall, unreviewed mutation?
|
||||
→ Yes. Pack lookup is deterministic; unit ontology is bounded
|
||||
and curated; no learned scoring; ratified pack only.
|
||||
If it touches user input, what trust boundary was enforced?
|
||||
→ Pack file paths are validated via the existing safe_pack_id
|
||||
sanitiser (ADR-0051). Pack content is mastery-report-sealed.
|
||||
```
|
||||
|
|
@ -13,252 +13,252 @@
|
|||
"per_case": [
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0001",
|
||||
"reason": "parser: could not parse statement clause: 'Tina makes $18.00 an hour' (in sentence: 'Tina makes $18.00 an hour.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0002",
|
||||
"reason": "parser: could not parse statement clause: 'Jan buys 1000 feet of cable' (in sentence: 'Jan buys 1000 feet of cable.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jan buys 1000 feet of cable.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0003",
|
||||
"reason": "parser: could not parse statement clause: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances' (in sentence: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0004",
|
||||
"reason": "parser: could not parse statement clause: 'There are some kids in camp' (in sentence: 'There are some kids in camp.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'There are some kids in camp.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0005",
|
||||
"reason": "parser: could not parse statement clause: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature\" (in sentence: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\")",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\"",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0006",
|
||||
"reason": "parser: could not parse statement clause: 'Mandy started reading books with only 8 pages when she was 6 years old' (in sentence: 'Mandy started reading books with only 8 pages when she was 6 years old.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0007",
|
||||
"reason": "parser: could not parse statement clause: 'Francine has five full boxes of crayons and 5 loose crayons' (in sentence: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0008",
|
||||
"reason": "parser: could not parse statement clause: 'Marnie makes bead bracelets' (in sentence: 'Marnie makes bead bracelets.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Marnie makes bead bracelets.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0009",
|
||||
"reason": "parser: could not parse statement clause: 'Jen has 10 more ducks than four times the number of chickens' (in sentence: 'Jen has 10 more ducks than four times the number of chickens.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0010",
|
||||
"reason": "parser: could not parse statement clause: 'Yun had 20 paperclips initially, but then lost 12' (in sentence: 'Yun had 20 paperclips initially, but then lost 12.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Yun had 20 paperclips initially, but then lost 12.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0011",
|
||||
"reason": "parser: could not parse statement clause: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup' (in sentence: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0012",
|
||||
"reason": "parser: could not parse statement clause: 'Dennis collected 10 rocks' (in sentence: 'Dennis collected 10 rocks.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'He put all of them in his aquarium but his fish ate half of them.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0013",
|
||||
"reason": "parser: could not parse statement clause: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel' (in sentence: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0014",
|
||||
"reason": "parser: could not parse statement clause: 'Bob can shuck 10 oysters in 5 minutes' (in sentence: 'Bob can shuck 10 oysters in 5 minutes.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Bob can shuck 10 oysters in 5 minutes.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0015",
|
||||
"reason": "parser: could not parse statement clause: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride' (in sentence: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0016",
|
||||
"reason": "parser: could not parse statement clause: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs\" (in sentence: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\")",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\"",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0017",
|
||||
"reason": "parser: could not parse statement clause: 'Jason has a carriage house that he rents out' (in sentence: 'Jason has a carriage house that he rents out.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jason has a carriage house that he rents out.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0018",
|
||||
"reason": "parser: could not parse statement clause: 'Xavier plays football with his friends' (in sentence: 'Xavier plays football with his friends.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Xavier plays football with his friends.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0019",
|
||||
"reason": "parser: could not parse statement clause: 'John adopts a dog from a shelter' (in sentence: 'John adopts a dog from a shelter.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John adopts a dog from a shelter.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0020",
|
||||
"reason": "parser: could not parse statement clause: 'Two puppies, two kittens' (in sentence: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0021",
|
||||
"reason": "parser: could not parse statement clause: 'John is lifting weights' (in sentence: 'John is lifting weights.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John is lifting weights.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0022",
|
||||
"reason": "parser: could not parse statement clause: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish' (in sentence: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0023",
|
||||
"reason": "parser: could not parse statement clause: 'Nicole collected 400 Pokemon cards' (in sentence: 'Nicole collected 400 Pokemon cards.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0024",
|
||||
"reason": "parser: could not parse statement clause: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday' (in sentence: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0025",
|
||||
"reason": "parser: could not parse statement clause: 'Lilibeth and her friends go strawberry picking' (in sentence: 'Lilibeth and her friends go strawberry picking.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Lilibeth and her friends go strawberry picking.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0026",
|
||||
"reason": "parser: could not parse statement clause: 'Aaron and his brother Carson each saved up $40 to go to dinner' (in sentence: 'Aaron and his brother Carson each saved up $40 to go to dinner.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0027",
|
||||
"reason": "parser: could not parse statement clause: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook' (in sentence: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0028",
|
||||
"reason": "parser: could not parse statement clause: 'Tom opens an amusement park' (in sentence: 'Tom opens an amusement park.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Tom opens an amusement park.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0029",
|
||||
"reason": "parser: could not parse statement clause: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home' (in sentence: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0030",
|
||||
"reason": "parser: could not parse statement clause: 'Jake decides to go to the beach for a fun day' (in sentence: 'Jake decides to go to the beach for a fun day.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jake decides to go to the beach for a fun day.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0031",
|
||||
"reason": "parser: could not parse statement clause: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer' (in sentence: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0032",
|
||||
"reason": "parser: could not parse statement clause: 'John decides to take up illustration' (in sentence: 'John decides to take up illustration.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John decides to take up illustration.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0033",
|
||||
"reason": "parser: could not parse statement clause: 'Rachel is 12 years old' (in sentence: 'Rachel is 12 years old, and her grandfather is 7 times her age.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Rachel is 12 years old, and her grandfather is 7 times her age.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0034",
|
||||
"reason": "parser: could not parse statement clause: 'Georgie is a varsity player on a football team' (in sentence: 'Georgie is a varsity player on a football team.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Georgie is a varsity player on a football team.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0035",
|
||||
"reason": "parser: could not parse statement clause: 'She decided to split them among her friends' (in sentence: 'She decided to split them among her friends.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'She decided to split them among her friends.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0036",
|
||||
"reason": "parser: could not parse statement clause: 'Monica way studying for an exam' (in sentence: 'Monica way studying for an exam.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Monica way studying for an exam.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0037",
|
||||
"reason": "parser: could not parse statement clause: 'Michael wants to lose 10 pounds by June' (in sentence: 'Michael wants to lose 10 pounds by June.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Michael wants to lose 10 pounds by June.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0038",
|
||||
"reason": "parser: could not parse statement clause: 'In a building, there are a hundred ladies on the first-floor studying' (in sentence: 'In a building, there are a hundred ladies on the first-floor studying.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'In a building, there are a hundred ladies on the first-floor studying.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0039",
|
||||
"reason": "parser: could not parse statement clause: 'At the family reunion, everyone ate too much food and gained weight' (in sentence: 'At the family reunion, everyone ate too much food and gained weight.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'At the family reunion, everyone ate too much food and gained weight.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0040",
|
||||
"reason": "parser: could not parse statement clause: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road' (in sentence: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0041",
|
||||
"reason": "parser: could not parse statement clause: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan' (in sentence: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0042",
|
||||
"reason": "parser: could not parse statement clause: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag' (in sentence: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0043",
|
||||
"reason": "parser: could not parse statement clause: 'Sandra wants to buy some sweets' (in sentence: 'Sandra wants to buy some sweets.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Sandra wants to buy some sweets.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0044",
|
||||
"reason": "parser: could not parse statement clause: 'John invests in a bank and gets 10% simple interest' (in sentence: 'John invests in a bank and gets 10% simple interest.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John invests in a bank and gets 10% simple interest.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0045",
|
||||
"reason": "parser: could not parse statement clause: 'Bart fills out surveys to earn money' (in sentence: 'Bart fills out surveys to earn money.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Bart fills out surveys to earn money.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0046",
|
||||
"reason": "parser: could not parse statement clause: 'A school has 100 students' (in sentence: 'A school has 100 students.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'A school has 100 students.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0047",
|
||||
"reason": "parser: could not parse statement clause: 'John bakes 12 coconut macaroons, each weighing 5 ounces' (in sentence: 'John bakes 12 coconut macaroons, each weighing 5 ounces.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'John bakes 12 coconut macaroons, each weighing 5 ounces.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0048",
|
||||
"reason": "parser: could not parse statement clause: 'Jed collects stamp cards' (in sentence: 'Jed collects stamp cards.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Jed collects stamp cards.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0049",
|
||||
"reason": "parser: could not parse statement clause: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes' (in sentence: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.'",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0050",
|
||||
"reason": "parser: could not parse statement clause: 'Mark does a gig every other day for 2 weeks' (in sentence: 'Mark does a gig every other day for 2 weeks.')",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'",
|
||||
"verdict": "refused"
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ import sys
|
|||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from evals.gsm8k_math.runner import _score_one
|
||||
from evals.gsm8k_math.runner import _score_one_candidate_graph
|
||||
|
||||
_HERE = Path(__file__).resolve().parent
|
||||
_CASES_PATH = _HERE / "cases.jsonl"
|
||||
|
|
@ -71,7 +71,7 @@ def build_report(cases: list[dict[str, Any]]) -> dict[str, Any]:
|
|||
per_case: list[dict[str, str]] = []
|
||||
counts = {"correct": 0, "wrong": 0, "refused": 0}
|
||||
for raw in cases:
|
||||
outcome = _score_one(_adapt(raw))
|
||||
outcome = _score_one_candidate_graph(_adapt(raw))
|
||||
counts[outcome.outcome] += 1
|
||||
per_case.append(
|
||||
{
|
||||
|
|
|
|||
Loading…
Reference in a new issue