chore(evals): ratify GSM8K train proxy rebaseline 6/44/0 → 30/20/0
- evals/gsm8k_math/train_sample/v1/report.json: rebaselined from correct=6/refused=44/wrong=0 to correct=30/refused=20/wrong=0. wrong_ids == []. exit_criterion.passed: false → true. Written deterministically by the canonical runner (write_report / json.dumps sort_keys=True). - docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md: new audit note documenting why #846's report update was unauthorized, why #847 restored the pin, and why this PR is the explicit authorization boundary for the committed rebaseline. - HANDOFF-gpt55-2026-06-20.md: small factual note appended to the 'Latest continuity pointer' section recording this rebaseline status. Authorization chain: ADR-0126 (exit criterion) → #846 kernel changes (produce 30/20/0) → #847 reconciliation (restores pin, delegates to this PR) → this PR (explicit rebaseline authorization) Non-goals: no runtime, serving, teaching, capability, or kernel change. No quantity-entity work. No new eval artifacts. No teaching proposals.
This commit is contained in:
Shay
parent
b6d422834b
commit
4e1a9bbea5
3 changed files with 305 additions and 54 deletions
|
|
@ -754,3 +754,15 @@ The latest session on 2026-06-20 is the
|
|||
Merge that documentation PR before starting the bounded diagnostic-only
|
||||
implementation brief. No `state_change.transition`, serving, general entity
|
||||
extraction, or broad parsing work is authorized.
|
||||
|
||||
### GSM8K train proxy rebaseline status (post-#847)
|
||||
|
||||
PR #847 restored `evals/gsm8k_math/train_sample/v1/report.json` to the governed
|
||||
6/44/0 pin after #846 committed a 30/20/0 rebaseline without explicit authorization.
|
||||
A dedicated rebaseline PR (`chore/evals-ratify-gsm8k-train-proxy-rebaseline`) was
|
||||
opened against the #847 merge commit (`b6d42283`) to ratify the committed 30/20/0
|
||||
counts. That PR is the explicit authorization boundary for this rebaseline.
|
||||
|
||||
**Ratified counts:** `correct: 30, refused: 20, wrong: 0`, `wrong_ids: []`.
|
||||
**Controlling artifact:** `docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md`.
|
||||
**No runtime, serving, teaching, or capability change in this rebaseline PR.**
|
||||
|
|
|
|||
239
docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md
Normal file
239
docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
# GSM8K Train Proxy Rebaseline — 2026-06-20
|
||||
|
||||
**PR:** `chore/evals-ratify-gsm8k-train-proxy-rebaseline`
|
||||
**Title:** `chore(evals): ratify GSM8K train proxy rebaseline`
|
||||
**Date:** 2026-06-20
|
||||
**Author:** Sonnet 4.6 (governance/rebaseline role)
|
||||
|
||||
---
|
||||
|
||||
## 1. Background: What Happened in #846 and #847
|
||||
|
||||
### PR #846 — Unauthorized Rebaseline
|
||||
|
||||
PR #846 (`feat(kernel): implement ProblemFrame proportional-change closure`) merged at
|
||||
`a145b7c3d62ab97726f87c4a711b4b4dafe9f2b6` on 2026-06-21T02:19:07Z.
|
||||
|
||||
The PR was a legitimate capability PR delivering proportional-change closure work in
|
||||
`generate/problem_frame.py`, `generate/problem_frame_builder.py`, and
|
||||
`generate/problem_frame_contracts.py`. Its kernel and test changes were valid and
|
||||
authorized.
|
||||
|
||||
However, the merge commit also included **two unauthorized artifact-class changes**:
|
||||
|
||||
1. **`evals/gsm8k_math/train_sample/v1/report.json`** — rebaselined from `6/44/0` to
|
||||
`30/20/0`. The live runner output was correct, but committing an updated `report.json`
|
||||
inside a capability PR violates governance doctrine: the committed proxy baseline is a
|
||||
sealed artifact that requires a dedicated, explicitly ratified rebaseline PR. No ADR,
|
||||
session note, or explicit PR section in #846 authorized committing the new counts.
|
||||
|
||||
2. **`teaching/proposals/comprehension_failures/34ce9254…json`** — a new teaching proposal
|
||||
file added without authorization. Committed via a `chore(kernel)` message; no review-gate
|
||||
passage; contradicted the HANDOFF claim that "teaching/review/proposal paths [were]
|
||||
untouched".
|
||||
|
||||
### PR #847 — Reconciliation
|
||||
|
||||
PR #847 (`reconcile: restore eval proxy and remove unauthorized teaching proposal from #846`)
|
||||
merged at `b6d422834b243073fc748754ad78b1a956774065`.
|
||||
|
||||
#847 performed the following narrow reversions:
|
||||
- Restored `evals/gsm8k_math/train_sample/v1/report.json` to the governed 6/44/0 pin.
|
||||
- Deleted the unauthorized teaching proposal artifact.
|
||||
- **Preserved** the valid kernel and test changes from #846.
|
||||
|
||||
The rationale: the live runner output (30/20/0) was already correct; only the committed
|
||||
artifact was wrong. Reverting the artifact does not revert the underlying capability.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why This PR Is the Explicit Authorization Boundary
|
||||
|
||||
The governance doctrine for `report.json` has been enforced consistently across prior
|
||||
sessions:
|
||||
|
||||
- `gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md`:
|
||||
"No report.json rebaseline" / "report.json rebaseline only via separate ratified PR"
|
||||
- `gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md`:
|
||||
"No write of updated report.json in this increment"
|
||||
- `gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md`: "report.json untouched"
|
||||
- `gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md`: "report.json untouched"
|
||||
|
||||
This PR is the **explicit authorization** for the single `report.json` update. It:
|
||||
|
||||
- Is opened **after** #847 is merged to main (required base integrity).
|
||||
- Is based on a **fresh worktree from `origin/main`** at the #847 merge commit
|
||||
(`b6d422834b243073fc748754ad78b1a956774065`).
|
||||
- Changes **only** `evals/gsm8k_math/train_sample/v1/report.json` and this session note.
|
||||
- Does **not** touch any kernel code, tests, packs, policy, identity, recall, vault, field,
|
||||
algebra, teaching proposals, or serving runtime paths.
|
||||
|
||||
---
|
||||
|
||||
## 3. Before/After Counts
|
||||
|
||||
### Before (governed 6/44/0 pin, restored by #847)
|
||||
|
||||
```json
|
||||
"counts": {
|
||||
"correct": 6,
|
||||
"refused": 44,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
"correct_min": 10,
|
||||
"passed": false,
|
||||
"wrong_max": 0
|
||||
}
|
||||
```
|
||||
|
||||
### After (ratified 30/20/0 rebaseline, this PR)
|
||||
|
||||
```json
|
||||
"counts": {
|
||||
"correct": 30,
|
||||
"refused": 20,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
"correct_min": 10,
|
||||
"passed": true,
|
||||
"wrong_max": 0
|
||||
}
|
||||
```
|
||||
|
||||
**`wrong_ids == []` (sacred invariant: no wrong answers committed)**
|
||||
|
||||
---
|
||||
|
||||
## 4. Live Runner Verification
|
||||
|
||||
The report.json was generated by running the canonical train-sample runner from the fresh
|
||||
worktree at HEAD (`b6d42283`) immediately before this PR:
|
||||
|
||||
```bash
|
||||
# Verification command (run in core-gsm8k-train-rebaseline worktree):
|
||||
uv run python - <<'PY'
|
||||
from evals.gsm8k_math.train_sample.v1.runner import _CASES_PATH, _load_cases, build_report
|
||||
r = build_report(_load_cases(_CASES_PATH))
|
||||
print("counts:", r["counts"])
|
||||
print("wrong_ids:", [x["case_id"] for x in r["per_case"] if x["verdict"] == "wrong"])
|
||||
print("exit_criterion:", r["exit_criterion"])
|
||||
PY
|
||||
|
||||
# Output:
|
||||
# counts: {'correct': 30, 'wrong': 0, 'refused': 20}
|
||||
# wrong_ids: []
|
||||
# exit_criterion: {'correct_min': 10, 'wrong_max': 0, 'passed': True}
|
||||
```
|
||||
|
||||
The report was then written deterministically using the runner's own `write_report()`:
|
||||
|
||||
```bash
|
||||
uv run python - <<'PY'
|
||||
from evals.gsm8k_math.train_sample.v1.runner import _CASES_PATH, _load_cases, build_report, write_report, _REPORT_PATH
|
||||
r = build_report(_load_cases(_CASES_PATH))
|
||||
write_report(r)
|
||||
print("Written to:", _REPORT_PATH)
|
||||
PY
|
||||
```
|
||||
|
||||
The runner uses `json.dumps(report, indent=2, sort_keys=True)`, so the output is
|
||||
deterministic and key-order is alphabetical throughout.
|
||||
|
||||
---
|
||||
|
||||
## 5. Scope
|
||||
|
||||
**Changed files in this PR:**
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `evals/gsm8k_math/train_sample/v1/report.json` | Rebaselined from 6/44/0 to 30/20/0 |
|
||||
| `docs/sessions/gsm8k-train-proxy-rebaseline-2026-06-20.md` | This audit note (new) |
|
||||
|
||||
**No other files changed.**
|
||||
|
||||
---
|
||||
|
||||
## 6. Non-Goals
|
||||
|
||||
This PR explicitly does **not**:
|
||||
|
||||
- Change any runtime code, serving path, or derivation organ.
|
||||
- Add or modify any test.
|
||||
- Touch packs, policy, identity, recall, vault, field, or algebra.
|
||||
- Create any teaching proposal or teaching artifact.
|
||||
- Start or advance quantity-entity implementation work.
|
||||
- Rebaseline any other eval artifact (holdout, composition validation, etc.).
|
||||
- Add any new eval artifact.
|
||||
|
||||
The quantity-entity foundational slice implementation is authorized separately by
|
||||
`docs/sessions/quantity-entity-foundational-slice-authorization-2026-06-20.md` and must
|
||||
not begin until that document's own gating conditions are met.
|
||||
|
||||
---
|
||||
|
||||
## 7. Remaining Risk
|
||||
|
||||
The primary systemic risk exposed by this sequence is:
|
||||
|
||||
> **Report pins must not be casually updated inside implementation or capability PRs.**
|
||||
|
||||
The `report.json` proxy baseline is a sealed artifact. Even when the live runner output
|
||||
is correct and the capability work is authorized, committing the updated report inside
|
||||
a capability PR:
|
||||
- Conflates two authorization domains (capability work ≠ rebaseline authorization).
|
||||
- Makes the committed artifact hard to audit (which PR actually authorized it?).
|
||||
- Undermines the governed rebaseline PR discipline that has protected wrong-zero invariance
|
||||
across all prior sessions.
|
||||
|
||||
**All future PRs must leave `evals/gsm8k_math/train_sample/v1/report.json` (and all other
|
||||
`report.json` proxy baselines) untouched unless the PR is explicitly and solely dedicated
|
||||
to a governed rebaseline.**
|
||||
|
||||
---
|
||||
|
||||
## 8. Authorization Chain
|
||||
|
||||
This rebaseline is authorized by the following chain:
|
||||
|
||||
1. **ADR-0126** — establishes the train-sample exit criterion (`correct >= 10`, `wrong == 0`).
|
||||
2. **#846 kernel changes** — the capability work that produced the 30/20/0 live runner
|
||||
output (proportional-change closure in ProblemFrame). These changes are valid and merged.
|
||||
3. **#847 reconciliation** — restored the committed artifact to 6/44/0, explicitly delegating
|
||||
authorization to a dedicated rebaseline PR (this PR).
|
||||
4. **This PR** — the explicit, governed authorization to commit the 30/20/0 counts.
|
||||
|
||||
The `wrong_ids == []` invariant is preserved: no wrong answer is committed, and the
|
||||
`wrong_max: 0` gate remains active.
|
||||
|
||||
---
|
||||
|
||||
## 9. Fresh-Base Verification (Pre-Edit)
|
||||
|
||||
```bash
|
||||
# In /Users/kaizenpro/Projects/core:
|
||||
git fetch origin --prune
|
||||
# -> b6d42283 reconcile: restore eval proxy and remove unauthorized teaching proposal from #846 (#847)
|
||||
|
||||
git worktree add ../core-gsm8k-train-rebaseline origin/main \
|
||||
-b chore/evals-ratify-gsm8k-train-proxy-rebaseline
|
||||
|
||||
# In new worktree:
|
||||
echo "HEAD=$(git rev-parse HEAD)"
|
||||
# HEAD=b6d422834b243073fc748754ad78b1a956774065
|
||||
|
||||
echo "ORIGIN_MAIN=$(git rev-parse origin/main)"
|
||||
# ORIGIN_MAIN=b6d422834b243073fc748754ad78b1a956774065
|
||||
|
||||
git status --porcelain
|
||||
# (empty — clean)
|
||||
|
||||
git log --oneline -n 3
|
||||
# b6d42283 reconcile: restore eval proxy and remove unauthorized teaching proposal from #846 (#847)
|
||||
# a145b7c3 feat(kernel): implement ProblemFrame proportional-change closure (#846)
|
||||
# ad0bae29 docs(kernel): authorize quantity-entity foundational implementation slice (#845)
|
||||
```
|
||||
|
||||
HEAD == ORIGIN_MAIN == the #847 merge commit. Pre-edit state verified clean.
|
||||
|
|
@ -1,69 +1,69 @@
|
|||
{
|
||||
"adr": "0126",
|
||||
"counts": {
|
||||
"correct": 6,
|
||||
"refused": 44,
|
||||
"correct": 30,
|
||||
"refused": 20,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
"correct_min": 10,
|
||||
"passed": false,
|
||||
"passed": true,
|
||||
"wrong_max": 0
|
||||
},
|
||||
"per_case": [
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0001",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Tina makes $18.00 an hour.' (category=rate_with_currency)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0002",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'She splits it up into 25-foot sections.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0003",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The local bookstore donated 48 boxes of erasers.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0004",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going to soccer camp in the morning.'",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0005",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: \"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature.\"",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0006",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Mandy started reading books with only 8 pages when she was 6 years old.' (category=multiplicative_aggregation)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0007",
|
||||
"reason": "candidate_graph: no admissible candidate for question: 'How many more boxes do they need if Francine has a total of 85 crayons?'",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0008",
|
||||
"reason": "candidate_graph: no admissible candidate for question: 'If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?'",
|
||||
"verdict": "refused"
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0009",
|
||||
"reason": "candidate_graph: no admissible candidate for question: 'If Jen has 150 ducks, how many total birds does she have?'",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0010",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'Marion has 1/4 more than what Yun currently has, plus 7.'",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0011",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.' (category=rate_with_currency)",
|
||||
"reason": "candidate_graph: no branch produced a solvable graph",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
|
|
@ -73,8 +73,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0013",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel.' (category=multiplicative_aggregation)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0014",
|
||||
|
|
@ -83,18 +83,18 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0015",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0016",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: \"On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less than 17 stop signs.\" (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0017",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He\u2019s charging $50.00 per day or $500.00 for 14 days.' (category=temporal_aggregation)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0018",
|
||||
|
|
@ -113,12 +113,12 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0021",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He bench presses 15 pounds for 10 reps and does 3 sets.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0022",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.' (category=rate_with_currency)",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'She goes out fishing today and catches twice as many fish as she caught in total in the past four months.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
|
|
@ -133,8 +133,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0025",
|
||||
"reason": "candidate_graph: no admissible candidate for question: \"If three of Lilibeth's friends pick the same amount as her, how many strawberries do Lilibeth and her friends pick in all?\"",
|
||||
"verdict": "refused"
|
||||
"reason": "",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0026",
|
||||
|
|
@ -143,8 +143,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0027",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined.' (category=descriptive_setup_no_quantity)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0028",
|
||||
|
|
@ -158,8 +158,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0030",
|
||||
"reason": "candidate_graph: no admissible candidate for statement: 'It is a 2-hour drive each way.'",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0031",
|
||||
|
|
@ -178,13 +178,13 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0034",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He can run 40 yards within 5 seconds.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0035",
|
||||
"reason": "candidate_graph: no admissible candidate for question: 'How many more apples would Martha need to give away to be left with only 4 of them?'",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0036",
|
||||
|
|
@ -193,8 +193,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0037",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Michael wants to lose 10 pounds by June.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0038",
|
||||
|
|
@ -203,8 +203,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0039",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Orlando gained 5 pounds.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0040",
|
||||
|
|
@ -223,7 +223,7 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0043",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Her mother gave her an additional $4, and her father twice as much as her mother.' (category=currency_amount)",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'She saved $10 for this purpose.' (category=currency_amount)",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
|
|
@ -233,13 +233,13 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0045",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Each survey has 10 questions.' (category=multiplicative_aggregation)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0046",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Half of the students are girls, the other half are boys.' (category=descriptive_setup_no_quantity)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0047",
|
||||
|
|
|
|||
Loading…
Reference in a new issue