core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

feat(ADR-0131.G.0): switch GSM8K coverage probe to candidate-graph pipeline

Browse source 
Zero behavior delta on the main baseline (both substrates produce
0/50 admission today) — but every subsequent ADR-0131.G.<n> iteration
now produces attributable admission deltas on the probe, instead of
silently extending a parser layer the probe wasn't measuring.

Background: ADR-0131.G's probe consulted run_lane → _score_one →
parse_problem (legacy first-match-wins parser, pre-ADR-0126). Every
G.<n> iteration extends the candidate-graph parser via
_score_one_candidate_graph → parse_and_solve. The mismatch was
discovered during G.3 development and explicitly reserved as this
follow-up.

Changes:
  - run_coverage_probe.py: switch import to _score_one_candidate_graph;
    new private _score_lane aggregator mirrors run_lane's output shape
    via per-case scoring; report root adds "substrate": "candidate_graph"
    for audit trail.
  - train_sample_coverage_report.json: regenerated. All metrics
    byte-identical to prior baseline (0/50 admission, wrong=0).
    refused_reasons_top text differs (candidate_graph: prefix instead
    of parser:) — expected and part of the substrate audit-trail shift.

Discipline: separate small PR per ADR-0131.G's "expansion that only
moves admission must be a standalone PR" principle. Substrate swap
attributable; future G.<n> deltas attributable.

Evidence:
  - python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe
    → admission 0/50, wrong=0, safety_rail_intact=True, exit 0
  - pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py
    → 8/8 pass in 0.18s (no test edits needed; tests pin invariants
    not numbers)
  - No changes to runner.py, no changes to any G.<n> work in flight.

Effect on in-flight iterations: each G.<n> PR (G.1 Gemini / G.2 #182 /
G.3 #183 / G.4 Opus#2) rebases after this lands and refreshes its
committed train_sample_coverage_report.json with the new substrate's
numbers. Rebase is mechanical.
This commit is contained in:
Shay 2026-05-23 14:43:05 -07:00
parent 481e0c3b9c
commit 801287bba6
3 changed files with 311 additions and 110 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

167
docs/decisions/ADR-0131.G.0-probe-substrate.md Normal file
View file

@ -0,0 +1,167 @@
# ADR-0131.G.0 — Probe Substrate: Candidate-Graph Pipeline
**Status:** Proposed
**Date:** 2026-05-23
**Author:** CORE main agent (Opus 4.7)
**Depends on:** ADR-0126 (candidate-graph parser),
ADR-0131.G (GSM8K coverage probe + iteration discipline)
**Parent:** ADR-0131.G
**Sibling iterations (in flight):** ADR-0131.G.1 (verbs),
ADR-0131.G.2 (comparatives), ADR-0131.G.3 (numerics),
ADR-0131.G.4 (multi-clause)
---
## Context
ADR-0131.G (PR #181) pinned a GSM8K coverage probe as a diff-able
admission number with capability-first iteration discipline. Each
subsequent ADR-0131.G.<n> extends a single capability axis of the
parser layer and re-runs the probe, with the gate that
`admission_rate` strictly increases (or a refused-reason family
deliberately shrinks).
But the probe currently consults
`evals.gsm8k_math.runner.run_lane`, which scores each case via
`_score_one` → `parse_problem` — the **legacy first-match-wins
parser** that pre-dates ADR-0126. Every ADR-0131.G.<n> iteration
extends the **candidate-graph parser** (ADR-0126 topology), via
`_score_one_candidate_graph` → `parse_and_solve`. **The probe and
the iteration work measure two different parsers.**
The mismatch was discovered during ADR-0131.G.3 development: the
G.3 axis lane (which routes through the candidate-graph pipeline
via `_score_one_candidate_graph`) showed full end-to-end
correctness on 20 curated money + hyphenated-cardinal cases, while
the GSM8K probe (legacy substrate) showed zero admission delta.
The G.3 ADR documented the limit honestly and reserved this
follow-up.
## Decision
Switch `evals/gsm8k_math/train_sample/v1/run_coverage_probe.py` to
call `_score_one_candidate_graph` instead of `run_lane`. The probe
now measures the parser layer the iteration discipline extends.
Mechanically, the probe builds its own thin `_score_lane` aggregator
that mirrors `run_lane`'s shape (cases_total / correct / wrong /
refused / rates / wrong_count_is_zero) but invokes
`_score_one_candidate_graph` per case. No changes to
`runner.py` itself.
Report schema gains one field — `"substrate": "candidate_graph"` —
so the audit trail records which pipeline produced the numbers.
Every other field (metrics, refused_reasons_top, per_case) keeps
its shape; the 8 contract tests in
`tests/test_adr_0131_G_gsm8k_coverage_probe.py` pass without
modification.
## Why this is a separate (small) PR
ADR-0131.G's discipline explicitly forbids bundling probe-infra
changes into capability-axis PRs ("Reject any expansion that only
moves GSM8K admission and not B3 / curated coverage — that's the
smell test for a template in disguise"). The mirror image applies
here: a substrate swap that moves admission numbers must be a
standalone PR so the delta is attributable to the substrate
change, not silently absorbed into a capability iteration.
**Today the substrate swap is zero-delta** — both
`_score_one` (legacy) and `_score_one_candidate_graph` produce
0/50 admission on the current main baseline. That makes this the
cleanest-possible substrate PR: behavior unchanged, audit trail
shifted, future iterations attributable.
## What changed in code
### `evals/gsm8k_math/train_sample/v1/run_coverage_probe.py`
- Import switched: `run_lane` → `_score_one_candidate_graph` (and
the existing `LaneReport` no longer needed at the probe layer).
- New private `_score_lane` aggregator mirrors `run_lane`'s
output shape via `_score_one_candidate_graph` per case.
- `build_report` adds `"substrate": "candidate_graph"` to the
report root.
- Docstring updated to document the substrate choice.
### `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
Regenerated. Top-level `"substrate"` field added. All metrics
byte-identical to the prior baseline (both substrates produce the
same 0/50 numbers today). `refused_reasons_top` text changes
because the candidate-graph pipeline emits slightly different
refusal strings (prefix `candidate_graph:` instead of
`parser:`); this is expected and is part of the substrate
audit-trail shift.
### No changes to
- `evals/gsm8k_math/runner.py` — `run_lane`,
`_score_one`, `_score_one_candidate_graph` all unchanged.
- `tests/test_adr_0131_G_gsm8k_coverage_probe.py` — the 8
contract tests pin invariants, not specific numbers, and all
pass after the substrate swap.
- Any G.<n> iteration's lane code or axis cases.
## Evidence
- **Probe**: `python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe`
→ `admission_rate: 0/50 = 0.0%`, `wrong: 0`,
`safety_rail_intact: True`, exit code 0.
- **Contract tests**: 8/8 pass in 0.18s.
- **Substrate equivalence on main baseline**: both pipelines
produce 0/50 admission. The substrate swap is zero behavior
delta today.
## Effect on in-flight iterations
| Iteration | Effect of this PR |
|---|---|
| ADR-0131.G.1 (verbs, Gemini) | After landing, probe admission will rise on cases the new verbs unlock (e.g., `Nicole collected 400 Pokemon cards`). |
| ADR-0131.G.2 (comparatives, PR #182) | Refused-reason family for comparatives moves at the probe layer for the first time. |
| ADR-0131.G.3 (numerics, PR #183) | Money + hyphenated cases the candidate-graph pipeline already admits start showing as probe admission (combined with G.1 unlocks `Tina makes $18.00 an hour` etc.). |
| ADR-0131.G.4 (multi-clause, Opus#2) | Conjoined-subject cases like `Aaron and his brother Carson each saved up $40` become probe-admittable (combined with G.3 for the money literal). |
Reconciliation order after merge: each G.<n> PR rebases onto
`main` after this PR lands, runs the probe, refreshes its
committed `train_sample_coverage_report.json`. The refreshed
report becomes the iteration's load-bearing evidence on the
probe, complementing the axis lane.
## What this does NOT do
- Does NOT change the legacy parser path. `parse_problem` and
`_score_one` remain available for other lanes that depend on
them.
- Does NOT change the GSM8K case set, the sample, or the answer
vocabulary.
- Does NOT change the GSM8K probe's role: it remains a *probe*
(honest disclosure of what the architecture can/can't admit),
not a *gate* for the math-expert promotion (that's still
B1+B2+B3 per ADR-0131).
- Does NOT modify any G.<n> capability work in flight. Each
rebase is mechanical (one merged report file refresh).
- Does NOT introduce stochastic behavior, opaque fallbacks, or
approximate recall anywhere in the probe.
## CLAUDE.md PR-checklist answers
- **Capability / performance / security added or protected:**
Aligns the probe's measurement substrate with the parser layer
the iteration discipline extends. Audit-trail correctness for
every future capability iteration.
- **Invariant proving the field remains valid:**
`admitted_wrong == 0` preserved on the probe;
`safety_rail_intact: True`; 8/8 contract tests pass.
- **CLI/eval lane proving the lane:**
`python3 -m evals.gsm8k_math.train_sample.v1.run_coverage_probe`
and `pytest tests/test_adr_0131_G_gsm8k_coverage_probe.py`.
- **Avoided hidden normalization / stochastic / approximate /
unreviewed mutation:** Yes. Pure deterministic substrate swap,
same case set, same scoring rules, different parser entry
point.
- **Trust boundary:** No new user-input surface, no new
filesystem paths, no new dynamic imports. The probe still
reads only `cases.jsonl` and writes only
`train_sample_coverage_report.json` under
`evals/gsm8k_math/train_sample/v1/`.

51
evals/gsm8k_math/train_sample/v1/run_coverage_probe.py
View file

@ -1,9 +1,8 @@
"""ADR-0131.G — GSM8K coverage probe.
Wraps the existing :mod:`evals.gsm8k_math.runner` to score the
real-GSM8K train sample (case_id / question / answer_numeric format)
against the bounded grammar + binding graph pipeline. Emits a
``train_sample_coverage_report.json`` that pins:
Scores the real-GSM8K train sample (case_id / question /
answer_numeric format) against the bounded grammar + binding graph
pipeline. Emits a ``train_sample_coverage_report.json`` that pins:
- ``admission_rate`` — fraction parsed + solved + verifier-passed
(i.e. ``correct``)
@ -17,6 +16,15 @@ The report is committed alongside this script so admission progress
across iterations (G.1, G.2, ...) is a diff-able number, not a
narrative claim.
ADR-0131.G.0 — probe-substrate fix: the probe consults the
**candidate-graph pipeline** (:func:`_score_one_candidate_graph`),
not the legacy first-match-wins parser (:func:`_score_one`). Every
ADR-0131.G.<n> iteration extends the candidate-graph parser layer;
the probe must measure that layer for the iteration discipline to
produce attributable admission deltas. The substrate swap is
zero-delta on the current main baseline (both paths produce 0/50)
but load-bearing for every subsequent iteration.
The probe is deterministic: same case set → byte-equal report.
"""
@ -27,7 +35,7 @@ from collections import Counter
from pathlib import Path
from typing import Any
from evals.gsm8k_math.runner import run_lane
from evals.gsm8k_math.runner import _score_one_candidate_graph
_HERE = Path(__file__).resolve().parent
@ -70,6 +78,31 @@ def _summarize_refusal_reasons(case_details: list[dict[str, Any]]) -> list[dict[
]
def _score_lane(adapted_cases: list[dict[str, Any]]) -> tuple[dict[str, Any], list[dict[str, Any]]]:
"""Run every case through the candidate-graph pipeline and aggregate.
Mirrors :func:`evals.gsm8k_math.runner.run_lane` but uses
:func:`_score_one_candidate_graph` so the probe measures the
parser layer that every ADR-0131.G.<n> iteration extends.
"""
outcomes = [_score_one_candidate_graph(c) for c in adapted_cases]
total = len(outcomes)
correct = sum(1 for o in outcomes if o.outcome == "correct")
wrong = sum(1 for o in outcomes if o.outcome == "wrong")
refused = sum(1 for o in outcomes if o.outcome == "refused")
metrics = {
"cases_total": total,
"correct": correct,
"wrong": wrong,
"refused": refused,
"correct_rate": (correct / total) if total else 0.0,
"wrong_rate": (wrong / total) if total else 0.0,
"refused_rate": (refused / total) if total else 0.0,
"wrong_count_is_zero": wrong == 0,
}
return metrics, [o.as_json() for o in outcomes]
def build_report() -> dict[str, Any]:
raw_cases = [
json.loads(line)
@ -77,13 +110,13 @@ def build_report() -> dict[str, Any]:
if line.strip()
]
adapted = [_adapt_case(c) for c in raw_cases]
lane_report = run_lane(adapted)
metrics = dict(lane_report.metrics)
metrics, case_details = _score_lane(adapted)
total = metrics["cases_total"]
return {
"schema_version": 1,
"adr": "0131.G",
"probe": "gsm8k_train_sample_coverage",
"substrate": "candidate_graph",
"sample_path": str(_CASES_PATH.relative_to(_HERE.parent.parent.parent.parent)),
"metrics": {
"cases_total": total,
@ -96,8 +129,8 @@ def build_report() -> dict[str, Any]:
"wrong_count_is_zero": metrics["wrong_count_is_zero"],
"safety_rail_intact": metrics["wrong_count_is_zero"],
},
"refused_reasons_top": _summarize_refusal_reasons(lane_report.case_details),
"per_case": lane_report.case_details,
"refused_reasons_top": _summarize_refusal_reasons(case_details),
"per_case": case_details,
}

203
evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json
View file

@ -20,7 +20,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Tina makes $18.00 an hour' (in sentence: 'Tina makes $18.00 an hour.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'",
"trace_hash": null
},
{
@ -31,7 +31,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jan buys 1000 feet of cable' (in sentence: 'Jan buys 1000 feet of cable.')",
"reason": "candidate_graph: no admissible candidate for statement: 'She splits it up into 25-foot sections.'",
"trace_hash": null
},
{
@ -42,7 +42,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances' (in sentence: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.')",
"reason": "candidate_graph: no admissible candidate for statement: 'The student council sells scented erasers in the morning before school starts to help raise money for school dances.'",
"trace_hash": null
},
{
@ -53,7 +53,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'There are some kids in camp' (in sentence: 'There are some kids in camp.')",
"reason": "candidate_graph: no admissible candidate for statement: 'There are some kids in camp.'",
"trace_hash": null
},
{
@ -64,7 +64,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature\" (in sentence: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\")",
"reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\"",
"trace_hash": null
},
{
@ -75,7 +75,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Mandy started reading books with only 8 pages when she was 6 years old' (in sentence: 'Mandy started reading books with only 8 pages when she was 6 years old.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.'",
"trace_hash": null
},
{
@ -86,7 +86,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Francine has five full boxes of crayons and 5 loose crayons' (in sentence: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, and her friend has 27 loose crayons.'",
"trace_hash": null
},
{
@ -97,7 +97,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Marnie makes bead bracelets' (in sentence: 'Marnie makes bead bracelets.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Marnie makes bead bracelets.'",
"trace_hash": null
},
{
@ -108,7 +108,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jen has 10 more ducks than four times the number of chickens' (in sentence: 'Jen has 10 more ducks than four times the number of chickens.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'",
"trace_hash": null
},
{
@ -119,7 +119,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Yun had 20 paperclips initially, but then lost 12' (in sentence: 'Yun had 20 paperclips initially, but then lost 12.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Yun had 20 paperclips initially, but then lost 12.'",
"trace_hash": null
},
{
@ -130,7 +130,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup' (in sentence: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.'",
"trace_hash": null
},
{
@ -141,7 +141,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: unit 'rocks' not in en_units_v1 and not an allowed count noun",
"reason": "candidate_graph: no admissible candidate for statement: 'He put all of them in his aquarium but his fish ate half of them.'",
"trace_hash": null
},
{
@ -152,7 +152,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel' (in sentence: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.'",
"trace_hash": null
},
{
@ -163,7 +163,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Bob can shuck 10 oysters in 5 minutes' (in sentence: 'Bob can shuck 10 oysters in 5 minutes.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Bob can shuck 10 oysters in 5 minutes.'",
"trace_hash": null
},
{
@ -174,7 +174,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride' (in sentence: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.'",
"trace_hash": null
},
{
@ -185,7 +185,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs\" (in sentence: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\")",
"reason": "candidate_graph: no admissible candidate for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\"",
"trace_hash": null
},
{
@ -196,7 +196,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jason has a carriage house that he rents out' (in sentence: 'Jason has a carriage house that he rents out.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Jason has a carriage house that he rents out.'",
"trace_hash": null
},
{
@ -207,7 +207,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Xavier plays football with his friends' (in sentence: 'Xavier plays football with his friends.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Xavier plays football with his friends.'",
"trace_hash": null
},
{
@ -218,7 +218,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John adopts a dog from a shelter' (in sentence: 'John adopts a dog from a shelter.')",
"reason": "candidate_graph: no admissible candidate for statement: 'John adopts a dog from a shelter.'",
"trace_hash": null
},
{
@ -229,7 +229,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Two puppies, two kittens' (in sentence: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Two puppies, two kittens, and three parakeets were for sale at the pet shop.'",
"trace_hash": null
},
{
@ -240,7 +240,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John is lifting weights' (in sentence: 'John is lifting weights.')",
"reason": "candidate_graph: no admissible candidate for statement: 'John is lifting weights.'",
"trace_hash": null
},
{
@ -251,7 +251,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish' (in sentence: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.'",
"trace_hash": null
},
{
@ -262,7 +262,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Nicole collected 400 Pokemon cards' (in sentence: 'Nicole collected 400 Pokemon cards.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'",
"trace_hash": null
},
{
@ -273,7 +273,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday' (in sentence: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday.'",
"trace_hash": null
},
{
@ -284,7 +284,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Lilibeth and her friends go strawberry picking' (in sentence: 'Lilibeth and her friends go strawberry picking.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Lilibeth and her friends go strawberry picking.'",
"trace_hash": null
},
{
@ -295,7 +295,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Aaron and his brother Carson each saved up $40 to go to dinner' (in sentence: 'Aaron and his brother Carson each saved up $40 to go to dinner.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner.'",
"trace_hash": null
},
{
@ -306,7 +306,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: unit 'followers' not in en_units_v1 and not an allowed count noun",
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Facebook.'",
"trace_hash": null
},
{
@ -317,7 +317,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Tom opens an amusement park' (in sentence: 'Tom opens an amusement park.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Tom opens an amusement park.'",
"trace_hash": null
},
{
@ -328,7 +328,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home' (in sentence: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Fabian bought a brand new computer mouse and keyboard to be able to work from home.'",
"trace_hash": null
},
{
@ -339,7 +339,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jake decides to go to the beach for a fun day' (in sentence: 'Jake decides to go to the beach for a fun day.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Jake decides to go to the beach for a fun day.'",
"trace_hash": null
},
{
@ -350,7 +350,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer' (in sentence: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer.'",
"trace_hash": null
},
{
@ -361,7 +361,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John decides to take up illustration' (in sentence: 'John decides to take up illustration.')",
"reason": "candidate_graph: no admissible candidate for statement: 'John decides to take up illustration.'",
"trace_hash": null
},
{
@ -372,7 +372,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Rachel is 12 years old' (in sentence: 'Rachel is 12 years old, and her grandfather is 7 times her age.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Rachel is 12 years old, and her grandfather is 7 times her age.'",
"trace_hash": null
},
{
@ -383,7 +383,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Georgie is a varsity player on a football team' (in sentence: 'Georgie is a varsity player on a football team.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Georgie is a varsity player on a football team.'",
"trace_hash": null
},
{
@ -394,7 +394,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'She decided to split them among her friends' (in sentence: 'She decided to split them among her friends.')",
"reason": "candidate_graph: no admissible candidate for statement: 'She decided to split them among her friends.'",
"trace_hash": null
},
{
@ -405,7 +405,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Monica way studying for an exam' (in sentence: 'Monica way studying for an exam.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Monica way studying for an exam.'",
"trace_hash": null
},
{
@ -416,7 +416,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Michael wants to lose 10 pounds by June' (in sentence: 'Michael wants to lose 10 pounds by June.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Michael wants to lose 10 pounds by June.'",
"trace_hash": null
},
{
@ -427,7 +427,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'In a building, there are a hundred ladies on the first-floor studying' (in sentence: 'In a building, there are a hundred ladies on the first-floor studying.')",
"reason": "candidate_graph: no admissible candidate for statement: 'In a building, there are a hundred ladies on the first-floor studying.'",
"trace_hash": null
},
{
@ -438,7 +438,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'At the family reunion, everyone ate too much food and gained weight' (in sentence: 'At the family reunion, everyone ate too much food and gained weight.')",
"reason": "candidate_graph: no admissible candidate for statement: 'At the family reunion, everyone ate too much food and gained weight.'",
"trace_hash": null
},
{
@ -449,7 +449,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road' (in sentence: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Over several years, Daniel has adopted any stray animals he sees on the side of the road.'",
"trace_hash": null
},
{
@ -460,7 +460,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan' (in sentence: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan.')",
"reason": "candidate_graph: no admissible candidate for statement: 'The guests eat all of 1 pan, and 75% of the 2nd pan.'",
"trace_hash": null
},
{
@ -471,7 +471,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag' (in sentence: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in each bag.'",
"trace_hash": null
},
{
@ -482,7 +482,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Sandra wants to buy some sweets' (in sentence: 'Sandra wants to buy some sweets.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Sandra wants to buy some sweets.'",
"trace_hash": null
},
{
@ -493,7 +493,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John invests in a bank and gets 10% simple interest' (in sentence: 'John invests in a bank and gets 10% simple interest.')",
"reason": "candidate_graph: no admissible candidate for statement: 'John invests in a bank and gets 10% simple interest.'",
"trace_hash": null
},
{
@ -504,7 +504,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Bart fills out surveys to earn money' (in sentence: 'Bart fills out surveys to earn money.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Bart fills out surveys to earn money.'",
"trace_hash": null
},
{
@ -515,7 +515,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'A school has 100 students' (in sentence: 'A school has 100 students.')",
"reason": "candidate_graph: no admissible candidate for statement: 'A school has 100 students.'",
"trace_hash": null
},
{
@ -526,7 +526,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'John bakes 12 coconut macaroons, each weighing 5 ounces' (in sentence: 'John bakes 12 coconut macaroons, each weighing 5 ounces.')",
"reason": "candidate_graph: no admissible candidate for statement: 'John bakes 12 coconut macaroons, each weighing 5 ounces.'",
"trace_hash": null
},
{
@ -537,7 +537,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Jed collects stamp cards' (in sentence: 'Jed collects stamp cards.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Jed collects stamp cards.'",
"trace_hash": null
},
{
@ -548,7 +548,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes' (in sentence: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm is trying to find the fastest walk to school and is currently comparing two routes.'",
"trace_hash": null
},
{
@ -559,7 +559,7 @@
"expected_unit": "",
"outcome": "refused",
"realized_prose": null,
"reason": "parser: could not parse statement clause: 'Mark does a gig every other day for 2 weeks' (in sentence: 'Mark does a gig every other day for 2 weeks.')",
"reason": "candidate_graph: no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'",
"trace_hash": null
}
],
@ -567,205 +567,206 @@
"refused_reasons_top": [
{
"count": 1,
"reason": "parser: could not parse statement clause: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temp"
"reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encoun"
"reason": "candidate_graph: no admissible candidate for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 mi"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'A school has 100 students' (in sentence: 'A school has 100 students.')"
"reason": "candidate_graph: no admissible candidate for statement: 'A school has 100 students.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Aaron and his brother Carson each saved up $40 to go to dinner' (in sentence:"
"reason": "candidate_graph: no admissible candidate for statement: 'Aaron and his brother Carson each saved up $40 to go to dinner."
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup' (in s"
"reason": "candidate_graph: no admissible candidate for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to h"
"reason": "candidate_graph: no admissible candidate for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'At the family reunion, everyone ate too much food and gained weight' (in sent"
"reason": "candidate_graph: no admissible candidate for statement: 'At the family reunion, everyone ate too much food and gained we"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Bart fills out surveys to earn money' (in sentence: 'Bart fills out surveys t"
"reason": "candidate_graph: no admissible candidate for statement: 'Bart fills out surveys to earn money.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Bob can shuck 10 oysters in 5 minutes' (in sentence: 'Bob can shuck 10 oyster"
"reason": "candidate_graph: no admissible candidate for statement: 'Bob can shuck 10 oysters in 5 minutes.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Ella has 4 bags with 20 apples in each bag and six bags with 25 apples in eac"
"reason": "candidate_graph: no admissible candidate for statement: 'Ella has 4 bags with 20 apples in each bag and six bags with 25"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Erica lives near a lake where most locals sell fish as their main source of i"
"reason": "candidate_graph: no admissible candidate for statement: 'Erica lives near a lake where most locals sell fish as their ma"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Fabian bought a brand new computer mouse and keyboard to be able to work from"
"reason": "candidate_graph: no admissible candidate for statement: 'Fabian bought a brand new computer mouse and keyboard to be abl"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Francine has five full boxes of crayons and 5 loose crayons' (in sentence: 'F"
"reason": "candidate_graph: no admissible candidate for statement: 'Francine has five full boxes of crayons and 5 loose crayons, an"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Georgie is a varsity player on a football team' (in sentence: 'Georgie is a v"
"reason": "candidate_graph: no admissible candidate for statement: 'Georgie is a varsity player on a football team.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'In a building, there are a hundred ladies on the first-floor studying' (in se"
"reason": "candidate_graph: no admissible candidate for statement: 'He put all of them in his aquarium but his fish ate half of the"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jake decides to go to the beach for a fun day' (in sentence: 'Jake decides to"
"reason": "candidate_graph: no admissible candidate for statement: 'In a building, there are a hundred ladies on the first-floor st"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jan buys 1000 feet of cable' (in sentence: 'Jan buys 1000 feet of cable.')"
"reason": "candidate_graph: no admissible candidate for statement: 'Jake decides to go to the beach for a fun day.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jason has a carriage house that he rents out' (in sentence: 'Jason has a carr"
"reason": "candidate_graph: no admissible candidate for statement: 'Jason has a carriage house that he rents out.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jed collects stamp cards' (in sentence: 'Jed collects stamp cards.')"
"reason": "candidate_graph: no admissible candidate for statement: 'Jed collects stamp cards.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jen has 10 more ducks than four times the number of chickens' (in sentence: '"
"reason": "candidate_graph: no admissible candidate for statement: 'Jen has 10 more ducks than four times the number of chickens.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Jeremie wants to go to an amusement park with 3 friends at the end of summer'"
"reason": "candidate_graph: no admissible candidate for statement: 'Jeremie wants to go to an amusement park with 3 friends at the "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John adopts a dog from a shelter' (in sentence: 'John adopts a dog from a she"
"reason": "candidate_graph: no admissible candidate for statement: 'John adopts a dog from a shelter.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John bakes 12 coconut macaroons, each weighing 5 ounces' (in sentence: 'John "
"reason": "candidate_graph: no admissible candidate for statement: 'John bakes 12 coconut macaroons, each weighing 5 ounces.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John decides to take up illustration' (in sentence: 'John decides to take up "
"reason": "candidate_graph: no admissible candidate for statement: 'John decides to take up illustration.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John invests in a bank and gets 10% simple interest' (in sentence: 'John inve"
"reason": "candidate_graph: no admissible candidate for statement: 'John invests in a bank and gets 10% simple interest.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'John is lifting weights' (in sentence: 'John is lifting weights.')"
"reason": "candidate_graph: no admissible candidate for statement: 'John is lifting weights.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Lilibeth and her friends go strawberry picking' (in sentence: 'Lilibeth and h"
"reason": "candidate_graph: no admissible candidate for statement: 'Lilibeth and her friends go strawberry picking.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Malcolm is trying to find the fastest walk to school and is currently compari"
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm has 240 followers on Instagram and 500 followers on Fac"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Mandy started reading books with only 8 pages when she was 6 years old' (in s"
"reason": "candidate_graph: no admissible candidate for statement: 'Malcolm is trying to find the fastest walk to school and is cur"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Mark does a gig every other day for 2 weeks' (in sentence: 'Mark does a gig e"
"reason": "candidate_graph: no admissible candidate for statement: 'Mandy started reading books with only 8 pages when she was 6 ye"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Marnie makes bead bracelets' (in sentence: 'Marnie makes bead bracelets.')"
"reason": "candidate_graph: no admissible candidate for statement: 'Mark does a gig every other day for 2 weeks.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Michael wants to lose 10 pounds by June' (in sentence: 'Michael wants to lose"
"reason": "candidate_graph: no admissible candidate for statement: 'Marnie makes bead bracelets.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Monica way studying for an exam' (in sentence: 'Monica way studying for an ex"
"reason": "candidate_graph: no admissible candidate for statement: 'Michael wants to lose 10 pounds by June.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Nicole collected 400 Pokemon cards' (in sentence: 'Nicole collected 400 Pokem"
"reason": "candidate_graph: no admissible candidate for statement: 'Monica way studying for an exam.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Over several years, Daniel has adopted any stray animals he sees on the side "
"reason": "candidate_graph: no admissible candidate for statement: 'Nicole collected 400 Pokemon cards.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Rachel is 12 years old' (in sentence: 'Rachel is 12 years old, and her grandf"
"reason": "candidate_graph: no admissible candidate for statement: 'Over several years, Daniel has adopted any stray animals he see"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Sandra wants to buy some sweets' (in sentence: 'Sandra wants to buy some swee"
"reason": "candidate_graph: no admissible candidate for statement: 'Rachel is 12 years old, and her grandfather is 7 times her age."
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'She decided to split them among her friends' (in sentence: 'She decided to sp"
"reason": "candidate_graph: no admissible candidate for statement: 'Sandra wants to buy some sweets.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday' (in s"
"reason": "candidate_graph: no admissible candidate for statement: 'She decided to split them among her friends.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'The student council sells scented erasers in the morning before school starts"
"reason": "candidate_graph: no admissible candidate for statement: 'She splits it up into 25-foot sections.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'There are some kids in camp' (in sentence: 'There are some kids in camp.')"
"reason": "candidate_graph: no admissible candidate for statement: 'Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on We"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Tina makes $18.00 an hour' (in sentence: 'Tina makes $18.00 an hour.')"
"reason": "candidate_graph: no admissible candidate for statement: 'The guests eat all of 1 pan, and 75% of the 2nd pan.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Tom opens an amusement park' (in sentence: 'Tom opens an amusement park.')"
"reason": "candidate_graph: no admissible candidate for statement: 'The student council sells scented erasers in the morning before"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, "
"reason": "candidate_graph: no admissible candidate for statement: 'There are some kids in camp.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Troy bakes 2 pans of brownies, cut into 16 pieces per pan' (in sentence: 'Tro"
"reason": "candidate_graph: no admissible candidate for statement: 'Tina makes $18.00 an hour.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Two puppies, two kittens' (in sentence: 'Two puppies, two kittens, and three "
"reason": "candidate_graph: no admissible candidate for statement: 'Tom opens an amusement park.'"
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Xavier plays football with his friends' (in sentence: 'Xavier plays football "
"reason": "candidate_graph: no admissible candidate for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway "
},
{
"count": 1,
"reason": "parser: could not parse statement clause: 'Yun had 20 paperclips initially, but then lost 12' (in sentence: 'Yun had 20 "
"reason": "candidate_graph: no admissible candidate for statement: 'Two puppies, two kittens, and three parakeets were for sale at "
},
{
"count": 1,
"reason": "parser: unit 'followers' not in en_units_v1 and not an allowed count noun"
"reason": "candidate_graph: no admissible candidate for statement: 'Xavier plays football with his friends.'"
},
{
"count": 1,
"reason": "parser: unit 'rocks' not in en_units_v1 and not an allowed count noun"
"reason": "candidate_graph: no admissible candidate for statement: 'Yun had 20 paperclips initially, but then lost 12.'"
}
],
"sample_path": "evals/gsm8k_math/train_sample/v1/cases.jsonl",
"schema_version": 1
"schema_version": 1,
"substrate": "candidate_graph"
}