core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5→0 (#496)

Browse source 
* fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5->0

The candidate-graph reader (serving) checked grounding + round-trip but had
no completeness obligation, so problems whose later clauses failed to parse
emitted a partial reading. Over the full 7,473-question real GSM8K train
split this confabulated 5 answers (wrong!=0) the 47-case train_sample hid;
2 were regressions from #488.

Add the missing admissibility leg (mirrors the derivation reader's verify.py):
every source quantity (all statements + question) must be consumed by the
chosen reading, else refuse. Refusal-only -> cannot create a wrong answer.
Number-sense is pack-authoritative (en_numerics_v1 parse_compound_cardinal +
lookup_multiplier + all 6 currency symbols) so it never disagrees with the
engine; aggregating initials expose consumed_value_tokens provenance.

Evidence: real-corpus wrong 5->0, correct held at 4; train_sample byte-
identical 4/46/0; G1-G5+S1+G3.1 green; smoke 67 passed; math_teaching_corpus
lane byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(adr-0191): committed full-corpus GSM8K microscope (standing wrong=0 + coverage instrument)

Promotes the throwaway tmp/ microscope that found the 5 confabulations into a
committed tool. Runs the canonical serving reader over any GSM8K corpus and
reports failures-first: correct/wrong/refused, every wrong answer by name,
refusal families, and the no-injection per-category coverage map that ranks
which injector to build next by real frequency.

Default corpus is the committed 47-case train_sample (always available);
--corpus path runs the full real split. This is the ADR-0191 follow-up: re-run
after every capability PR, not just train_sample — a flip is only real if it
does not widen the confabulation surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Shay 2026-05-30 15:45:07 -07:00 committed by GitHub
parent 0770648257
commit 25580e18b0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 668 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

157
docs/decisions/ADR-0191-candidate-graph-completeness-guard.md Normal file
View file

@ -0,0 +1,157 @@
# ADR-0191 — Candidate-graph completeness guard (the missing wrong=0 leg)
**Status:** Proposed (implemented in this PR). Hardens the
[ADR-0123](./ADR-0123-candidate-graph-reader.md) candidate-graph reader's
admissibility gate. Serving-path firewall fix; landed wrong=0-proven on
the **full real GSM8K train split**, not just the 47-case sample.
> **One line.** The candidate-graph reader checked *grounding* and
> *round-trip* but had **no completeness obligation**, so a problem whose
> later clauses failed to parse still emitted whatever partial graph
> remained. Over the full 7,473-question real GSM8K train split this
> confabulated **5 answers** (wrong≠0). This adds the completeness leg the
> derivation reader's `verify.py` already has: every source quantity must be
> consumed by the chosen reading, else refuse. Result: real-corpus
> **wrong 5 → 0**; `train_sample` byte-identical **4/46/0**.
---
## 1. The gap (full-corpus microscope finding, 2026-05-30)
The 47-case `train_sample` reports wrong=0. Running the **canonical serving
reader** (`generate.math_candidate_graph.parse_and_solve`) over the *entire*
real GSM8K train split (7,473 questions) revealed the sample was hiding a
firewall breach:
```
correct 4 · wrong 5 · refused 7,464 (origin/main @ #488)
```
The 5 confabulations (deterministic, 3× reproduced):
| idx | problem (abridged) | reader | gold |
|----:|--------------------|-------:|-----:|
| 553 | Emma buys 2/school-day … in 3 weeks? | 2 | 30 |
| 605 | Ivan 20 dice; Jerry twice as many; altogether? | 20 | 60 |
| 693 | Ian 20 roses; gave 6/9/4; kept rest? | 20 | 1 |
| 6172| Jimmy 18 cards; gives 3; Mary twice that; left? | 15 | 9 |
| 7369| Wilfred 4 Tue, 6 Wed; total 15 Tue–Thu; Thu? | -4 | 5 |
Two of these (693, 7369) were **regressions introduced by #488** (ADR-0189/0189a):
they refused correctly before that PR and confabulated after. The 47-case gate
could not see it — exactly the lookback hazard CLAUDE.md §Lookback Review warns
about.
**Root cause — one structural hole, not five anecdotes.** A graph is admitted
when its *present* elements ground and round-trip. Nothing checks that every
question-relevant source quantity is *represented*. When later clauses fail to
parse into operations, the residual partial graph still solves and is emitted:
- `605` builds `initial=(Ivan:20), operations=()` — a zero-operation graph
answering "altogether"; the "twice as many" and the aggregate vanished.
- `693` the "He gave …" subtractions don't bind to "Ian" (live ADR-0174
pronoun hazard) and 2 of 3 are dropped → bare initial survives.
The derivation reader already refuses this (`verify.py`: grounding ∧ cue ∧
unit ∧ **completeness** ∧ uniqueness). The candidate-graph reader was missing
the completeness leg.
## 2. Decision
Add a **completeness guard** as the final admissibility check in
`parse_and_solve`, in a dedicated module `generate/math_completeness.py`:
> Collect every numeric / multiplier quantity in the source (all statement
> sentences **before** the numeric-only filter, plus the question). Collect
> every quantity the chosen reading actually **consumed** (candidate
> provenance). If a source quantity is not consumed, the reading is
> incomplete → **refuse**.
```text
uncovered = quantity_values(all_statements + question) − consumed_values(chosen_branch)
if uncovered: refuse("incomplete reading: …")
```
### Why this preserves wrong=0 and cannot regress
- **Refusal-only.** The guard only ever flips an *emitted answer* to a
*refusal*. It can never invent an answer, so it can only remove wrong
answers — never create one. Its entire regression surface is the
graph-path *correct* set, which is exactly `{train_sample 0024}` /
`{real-train 3343}` (the same Sidney/Brooke day-enum + comparative shape).
Both still solve (438).
- **Set semantics, not multiset.** `required − consumed` over value SETS
tolerates a quantity echoed in the question (no false refusal) while still
catching a clause whose distinct quantity was dropped — which is what every
observed confabulation does.
- **Short-circuits are immune.** Capacity / earnings / conditional / embedded
short-circuits return before the graph decision rule, so the guard never
touches them.
### Pack-authoritative number-sense (no hand-rolled lexicon)
Both the *required* scan and the *consumed* normalization resolve quantities
through the `en_numerics_v1` pack and the parser's own `_resolve_value`, so
identical surface forms cancel exactly and the guard never disagrees with the
engine about what a number is:
- Compound cardinals via `parse_compound_cardinal` (`one hundred` → 100,
`two thousand five hundred` → 2500, `twenty-five` → 25).
- Multiplier anchors via `lookup_multiplier` — **read from the pack, not
hardcoded**, so the guard automatically covers `twice, thrice, half,
double, triple, quadruple, quintuple` and excludes ordinal-ambiguous
`third` / `quarter` (which are not multipliers in the pack).
- All six currency symbols (`$ ¢ € £ ¥ ₱`) tokenize as whole spans.
### Provenance for aggregating extractors
Aggregating initials collapse several source tokens into one derived value, so
they now expose every consumed token via a new
`CandidateInitial.consumed_value_tokens` field (default `()` → falls back to
`matched_value_token`, preserving all existing behavior):
- day-enumeration → every per-day count;
- embedded-quantifier → both `N` and `M` of `N×M`;
- conjoined-embedded → all four factors;
- multi-word cardinal → the full phrase.
## 3. Evidence
- **Real GSM8K train (7,473):** wrong **5 → 0**; correct held at 4. Firewall
HOLDS. The 5 confabulations now refuse with `incomplete reading: …`.
- **`train_sample` (official metric):** byte-identical **4/46/0**, set
`{0014, 0018, 0024, 0042}`.
- **Capability axes:** G1–G5 + S1 + numerics extensions (G3.1) all green; the
first guard draft over-refused 20 G3 numerics cases (currency/decimal/
hyphenated/compound-cardinal mis-parse) — fixed by making the number-sense
pack-authoritative. Two pre-existing failures
(`test_committed_report_matches_fresh_run`, `test_full_session_round_trips`)
fail identically on `origin/main` and are unrelated.
- **Smoke suite:** 67 passed.
- **Pinned serving lanes:** `math_teaching_corpus_v1` report byte-identical;
no pinned lane exercises `parse_and_solve`.
- **New tests:** `tests/test_candidate_graph_completeness_guard.py` (5
confabulations refuse; Sidney/Brooke still solves; refusal-only invariant).
## 4. Consequences
- The candidate-graph reader now refuses partial readings instead of
confabulating — the wrong=0 firewall holds on real data, not just the
sample. This is the prerequisite for any further capability work: a flip is
only real if it does not also widen the confabulation surface.
- The guard makes some genuinely-incomplete shapes refuse that previously
emitted a (wrong) answer. That is the point. It never blocks a *complete*
reading; once a future capability consumes the dropped quantity, coverage is
satisfied and the case admits.
- **Limitation (documented, not load-bearing):** fraction *words* beyond the
multiplier set (e.g. `two-thirds`) are not yet recognized as required
quantities; this is conservative (can only under-refuse, never confabulate)
and is future work when a fraction capability lands.
## 5. Follow-ups
- Re-run the full-corpus microscope after each future capability PR as a
standing wrong=0 regression check (not just `train_sample`).
- The 693 confabulation also exposed the live ADR-0174 multi-actor pronoun
hazard; the completeness guard now refuses it, but the pronoun-binding fix
remains the proper long-term resolution.

43
generate/math_candidate_graph.py
View file

@ -60,6 +60,7 @@ from generate.math_problem_graph import (
MathGraphError,
MathProblemGraph,
)
from generate.math_completeness import uncovered_quantities
from generate.math_roundtrip import CandidateOperation, roundtrip_admissible
from generate.math_solver import SolveError, solve
@ -103,6 +104,10 @@ class CandidateGraphAnswer:
graph: MathProblemGraph
answer: int | float
# ADR-0191 — the originating branch (statement choices + question
# choice). Carries per-candidate consumed-token provenance the
# completeness guard needs; the MathProblemGraph itself discards it.
branch: tuple["SentenceChoice | CandidateUnknown", ...] = ()
@dataclass(frozen=True, slots=True)
@ -463,6 +468,12 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
question_sentences = [s for s in sentences if s.rstrip().endswith("?")]
statement_sentences = [s for s in sentences if not s.rstrip().endswith("?")]
# ADR-0191 — preserve EVERY statement sentence before the numeric-only
# filter below drops non-numeric ones. The completeness guard must see
# quantity signals carried in dropped sentences (e.g. "Jerry has twice
# as many … as Ivan" has no digit but a multiplier the reading must
# account for) to catch confabulations that emit a partial reading.
all_statement_sentences = list(statement_sentences)
# ADR-0136.S.0 — Strip context-filler sentences before any extraction.
# A sentence with no digit and no word-number cannot introduce parseable
@ -1026,7 +1037,11 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
except SolveError:
continue
admissible.append(
CandidateGraphAnswer(graph=graph, answer=trace.answer_value)
CandidateGraphAnswer(
graph=graph,
answer=trace.answer_value,
branch=(*stmt_choices, q_choice),
)
)
if not admissible:
@ -1055,6 +1070,32 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
# Single agreed answer. Pick the first admissible graph as the
# canonical representative (deterministic since product() is ordered).
chosen = admissible[0]
# ADR-0191 — completeness guard (the missing admissibility leg).
# The branch grounded + round-tripped, but that only proves the
# quantities it DID read are real — not that it read ALL of them.
# If any source quantity (across every statement sentence + the
# question) is absent from the chosen reading, emitting its answer
# would confabulate a partial reading. Refuse instead (wrong==0).
# Refusal-only: this can never turn a refusal into an answer, so it
# cannot create a wrong answer — only remove confabulations.
uncovered = uncovered_quantities(
statement_sentences=all_statement_sentences,
question_text=question_sentences[0],
branch=chosen.branch,
)
if uncovered:
return CandidateGraphResult(
answer=None, selected_graph=None,
refusal_reason=(
"incomplete reading: source quantities "
f"{sorted(uncovered)} not consumed by the solved graph"
),
branches_enumerated=branches_enumerated,
branches_admissible=len(admissible),
reader_trace=tuple(reader_trace),
)
return CandidateGraphResult(
answer=chosen.answer,
selected_graph=chosen.graph,

21
generate/math_candidate_parser.py
View file

@ -87,6 +87,14 @@ class CandidateInitial:
# count_token, amount_token, currency_symbol, composition_shape,
# entity_source.
composition_evidence: Mapping[str, str] | None = None
# ADR-0191 — completeness provenance. Aggregating extractors that
# collapse several source tokens into one derived value (day-enum sum,
# embedded-quantifier product, multi-word cardinal) list EVERY source
# quantity token they consumed here, so the candidate-graph reader's
# completeness guard (generate/math_completeness.py) can confirm no
# source quantity was silently dropped. Empty () means "single token"
# and the guard falls back to ``matched_value_token``.
consumed_value_tokens: tuple[str, ...] = ()
def __post_init__(self) -> None:
# ADR-0127 widens the anchor set to include 'there are/were/is/was'
@ -1358,6 +1366,9 @@ def _multi_word_cardinal_candidates(sentence: str) -> list[CandidateInitial]:
matched_value_token=value_raw.split()[0],
matched_unit_token=unit_raw,
matched_entity_token=m.group("entity"),
# ADR-0191 — the compound cardinal collapses every word into
# one value; the guard sees them via the joined surface form.
consumed_value_tokens=(value_raw,),
)
]
except Exception:
@ -1595,7 +1606,8 @@ def _day_enumeration_candidates(sentence: str) -> list[CandidateInitial]:
if m is None:
return []
n1 = int(m.group("n1"))
rest_nums = [int(x) for x in _DAY_ENUM_REST_RE.findall(m.group("rest"))]
rest_raw = _DAY_ENUM_REST_RE.findall(m.group("rest"))
rest_nums = [int(x) for x in rest_raw]
if not rest_nums:
return []
total = float(n1 + sum(rest_nums))
@ -1614,6 +1626,9 @@ def _day_enumeration_candidates(sentence: str) -> list[CandidateInitial]:
matched_value_token=m.group("n1"),
matched_unit_token=noun_raw,
matched_entity_token=m.group("entity"),
# ADR-0191 — the sum collapses every per-day count; record
# them all so the completeness guard sees full coverage.
consumed_value_tokens=(m.group("n1"), *rest_raw),
)
]
except Exception:
@ -1670,6 +1685,8 @@ def _embedded_quantifier_candidates(sentence: str) -> list[CandidateInitial]:
matched_value_token=m_raw,
matched_unit_token=unit_raw,
matched_entity_token=m.group("entity"),
# ADR-0191 — the product N*M consumes both source tokens.
consumed_value_tokens=(n_raw, m_raw),
)
]
except Exception:
@ -1750,6 +1767,8 @@ def _build_conj_embedded_sum(
matched_value_token=m1_raw, # provenance: first per-container M
matched_unit_token=m.group("u1"),
matched_entity_token=m.group("entity"),
# ADR-0191 — the sum of two products consumes all four tokens.
consumed_value_tokens=(n1_raw, m1_raw, n2_raw, m2_raw),
)
]
except Exception:

210
generate/math_completeness.py Normal file
View file

@ -0,0 +1,210 @@
"""Completeness leg of the candidate-graph reader's admissibility gate.
ADR-0191 — the candidate-graph reader checked *grounding* (every claimed
slot traces to a source token) and *round-trip* (the parsed candidate
re-realizes), but had no *completeness* obligation. A problem whose
later clauses failed to parse into operations still emitted whatever
partial graph remained — the classic confabulation the derivation
reader's ``verify.py`` already refuses (grounding ∧ cue ∧ unit ∧
**completeness** ∧ uniqueness).
This module supplies the missing leg as a pure, side-effect-free check:
Collect every numeric / multiplier quantity present in the source
(all statement sentences + the question). Collect every quantity the
chosen reading actually CONSUMED (candidate provenance). If a source
quantity is not consumed, the reading is incomplete → the reader must
refuse.
Design properties (why this preserves wrong==0 and cannot regress):
- **Refusal-only.** The check only ever flips an emitted answer to a
refusal; it never invents an answer. So it can only *remove* wrong
answers, never create one.
- **Set semantics, not multiset.** ``uncovered = required - consumed``
over value SETS. This deliberately tolerates a source quantity echoed
in the question (avoids false refusals) while still catching a clause
whose distinct quantity was dropped — which is what every observed
confabulation does.
- **Pack-authoritative number-sense.** Quantities are resolved through
the ``en_numerics_v1`` pack (``parse_compound_cardinal``) and the
parser's own ``_resolve_value`` — the same machinery the extractors
use. Identical surface forms (``$40``, ``twenty-five``, ``one
hundred``, ``3/4``) therefore resolve to identical values on both the
required and the consumed side and cancel exactly; the guard never
disagrees with the engine about what a number is.
- **Conservative multiplier set.** Only the unambiguous multiplier
anchors ``twice / thrice / half`` count as standalone quantity signals
(these are not cardinals). Ordinal-ambiguous words (``third`` /
``quarter`` — usually "the third day") are excluded to avoid spurious
refusals.
"""
from __future__ import annotations
import re
from typing import TYPE_CHECKING
from generate.math_candidate_parser import _CURRENCY_SYMBOLS, _resolve_value
from language_packs.numerics_loader import (
lookup_cardinal,
lookup_multiplier,
parse_compound_cardinal,
)
if TYPE_CHECKING: # pragma: no cover - typing only
from generate.math_candidate_parser import CandidateInitial
from generate.math_roundtrip import CandidateOperation
# Multiplier-anchor quantity signals (``twice``/``double``/``half`` ...) are
# read from the en_numerics_v1 pack via ``lookup_multiplier`` — NOT hardcoded
# — so the guard never drifts from the pack lexicon (it carries twice,
# thrice, half, double, triple, quadruple, quintuple). Ordinal-ambiguous
# words (``third`` / ``quarter``) are not multipliers in the pack, so they are
# excluded automatically rather than by a hand-maintained denylist.
def _multiplier_value(token: str) -> float | None:
entry = lookup_multiplier(token)
return float(entry.factor) if entry is not None else None
# Currency-symbol character class, taken from the parser's pinned symbol set
# (``$ ¢ € £ ¥ ₱``) so symbol-prefixed amounts tokenize as one span and
# resolve identically to the consumed candidate token.
_CURRENCY_CLASS = "".join(re.escape(c) for c in _CURRENCY_SYMBOLS)
# One pass that yields, in order: currency/digit/decimal/slash-fraction
# literals, and word tokens (incl. hyphenated cardinals like "twenty-five").
# Word runs are re-joined below so multi-word cardinals ("one hundred",
# "two thousand five hundred") resolve as a single quantity.
_TOKEN_RE = re.compile(
rf"[{_CURRENCY_CLASS}]?\d[\d,]*(?:\.\d+)?(?:/\d+)?" # $40 / 18.00 / 3/4
r"|[A-Za-z]+(?:-[A-Za-z]+)*" # words incl. hyphenated
)
def _numeric_token_value(token: str) -> float | None:
"""Value of a single non-cardinal token (digit/currency/fraction)."""
resolved = _resolve_value(token)
return float(resolved.value) if resolved is not None else None
def _token_value(token: str) -> float | None:
"""Canonical numeric value of a single quantity token, or None.
Multiplier anchors first, then compound cardinals (pack), then the
parser's value resolver for digit / currency / fraction surface
forms. Used to normalize CONSUMED candidate tokens identically to
the required scan.
"""
if not token:
return None
t = token.strip()
mult = _multiplier_value(t)
if mult is not None:
return mult
cardinal = parse_compound_cardinal(t)
if cardinal is not None:
return float(cardinal)
return _numeric_token_value(t)
def quantity_values_in_text(text: str) -> set[float]:
"""Every numeric / multiplier quantity value present in ``text``.
Greedily merges runs of cardinal words (joined by hyphens or "and")
so "two thousand five hundred" is one quantity, not five. Digit /
currency / fraction literals and multiplier anchors are resolved per
token. Pack-authoritative throughout.
"""
if not text:
return set()
values: set[float] = set()
tokens = _TOKEN_RE.findall(text)
i = 0
n = len(tokens)
while i < n:
tok = tokens[i]
low = tok.lower()
# Multiplier anchor (standalone quantity signal), per the pack.
mult = _multiplier_value(low)
if mult is not None:
values.add(mult)
i += 1
continue
# Cardinal-word run: extend across adjacent cardinal words and
# interior "and" connectors ("three hundred and fifty").
if lookup_cardinal(low) is not None:
run = [tok]
j = i + 1
while j < n:
nxt = tokens[j].lower()
if lookup_cardinal(nxt) is not None:
run.append(tokens[j])
j += 1
elif nxt == "and" and j + 1 < n and lookup_cardinal(
tokens[j + 1].lower()
) is not None:
run.append(tokens[j])
j += 1
else:
break
v = parse_compound_cardinal(" ".join(run))
if v is not None:
values.add(float(v))
i = j
continue
# Digit / currency / fraction literal.
v = _numeric_token_value(tok)
if v is not None:
values.add(v)
i += 1
return values
def _candidate_consumed_tokens(
choice: "CandidateInitial | CandidateOperation",
) -> tuple[str, ...]:
"""Source quantity tokens a single candidate consumed.
Aggregating initials (day-enumeration, embedded-quantifier,
multi-word-cardinal) collapse several source tokens into one derived
value; they expose every consumed token via ``consumed_value_tokens``.
Every other candidate consumes exactly its ``matched_value_token``.
"""
consumed = getattr(choice, "consumed_value_tokens", ())
if consumed:
return tuple(consumed)
tok = getattr(choice, "matched_value_token", "")
return (tok,) if tok else ()
def consumed_values(branch: tuple[object, ...]) -> set[float]:
"""Canonical quantity values consumed by a chosen reading (branch)."""
values: set[float] = set()
for choice in branch:
for tok in _candidate_consumed_tokens(choice): # type: ignore[arg-type]
v = _token_value(tok)
if v is not None:
values.add(v)
return values
def uncovered_quantities(
*,
statement_sentences: list[str],
question_text: str,
branch: tuple[object, ...],
) -> set[float]:
"""Source quantities the chosen reading failed to consume.
A non-empty result means the reading is incomplete: the source
carries a quantity the solved graph never accounts for, so emitting
its answer would confabulate. The reader must refuse.
"""
required: set[float] = set()
for s in statement_sentences:
required |= quantity_values_in_text(s)
required |= quantity_values_in_text(question_text)
return required - consumed_values(branch)

148
scripts/gsm8k_microscope.py Normal file
View file

@ -0,0 +1,148 @@
"""Full-corpus GSM8K microscope — the standing wrong=0 + coverage instrument.
ADR-0191 follow-up. The 47-case ``train_sample`` cannot see confabulations
that only fire on rarer real-corpus shapes: ADR-0191 found 5 wrong answers
on the full 7,473-question real GSM8K train split that ``train_sample``
reported as wrong=0. This tool runs the canonical serving reader
(``generate.math_candidate_graph.parse_and_solve``) over an arbitrary GSM8K
corpus and reports, failures-first:
- correct / wrong / refused counts (wrong MUST be 0 — the firewall);
- every wrong answer (so a regression is named, not hidden in a count);
- refusal families, and for the dominant "recognizer matched but produced
no injection" family, the per-category breakdown — the coverage map
that ranks which injector to build next by real frequency.
Run after EVERY capability PR, not just the sample: a flip is only real if
it does not also widen the confabulation surface.
Usage:
# Default: the committed 47-case train_sample (always available).
uv run python scripts/gsm8k_microscope.py
# Full real corpus (download train.jsonl from openai/grade-school-math):
uv run python scripts/gsm8k_microscope.py --corpus path/to/train.jsonl
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections import Counter
from pathlib import Path
_REPO_ROOT = Path(__file__).resolve().parents[1]
if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))
from generate.math_candidate_graph import parse_and_solve # noqa: E402
_TRAIN_SAMPLE = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
_CATEGORY_RE = re.compile(r"category=([a-z_]+)")
def _gold(record: dict) -> float | None:
"""Numeric gold answer from either GSM8K-raw or train_sample schema."""
if "answer_numeric" in record:
raw = str(record["answer_numeric"])
elif "answer" in record:
raw = record["answer"].split("####")[-1]
else:
return None
raw = raw.strip().replace(",", "")
try:
return float(raw)
except ValueError:
return None
def _question(record: dict) -> str:
return record.get("question") or record.get("problem") or record.get("text") or ""
def _refusal_family(reason: str | None) -> str:
if not reason:
return "(no reason)"
if "no injection" in reason:
m = _CATEGORY_RE.search(reason)
return f"no_injection:{m.group(1) if m else '?'}"
if "no admissible candidate for statement" in reason:
return "statement_unparsed"
if "no admissible candidate for question" in reason:
return "question_unparsed"
if "no branch produced" in reason:
return "no_solvable_branch"
if "disagree" in reason:
return "branch_disagreement"
if "incomplete reading" in reason:
return "incomplete_reading"
if "round-trip" in reason or "round trip" in reason:
return "roundtrip_reject"
return reason[:48]
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument(
"--corpus", type=Path, default=_TRAIN_SAMPLE,
help="JSONL of GSM8K records (default: committed train_sample).",
)
ap.add_argument("--json", action="store_true", help="Emit machine-readable JSON.")
args = ap.parse_args()
rows = [json.loads(line) for line in args.corpus.read_text().splitlines() if line.strip()]
outcome: Counter[str] = Counter()
families: Counter[str] = Counter()
no_injection_categories: Counter[str] = Counter()
wrongs: list[dict] = []
for rec in rows:
q = _question(rec)
gold = _gold(rec)
res = parse_and_solve(q)
if res.answer is None:
outcome["refused"] += 1
fam = _refusal_family(res.refusal_reason)
families[fam] += 1
if fam.startswith("no_injection:"):
no_injection_categories[fam.split(":", 1)[1]] += 1
elif gold is not None and abs(float(res.answer) - gold) < 1e-6:
outcome["correct"] += 1
else:
outcome["wrong"] += 1
wrongs.append({"q": q[:160], "reader": float(res.answer), "gold": gold})
total = len(rows)
report = {
"corpus": str(args.corpus),
"total": total,
"outcome": dict(outcome),
"wrong_is_zero": outcome["wrong"] == 0,
"wrongs": wrongs,
"refusal_families": dict(families.most_common()),
"no_injection_categories": dict(no_injection_categories.most_common()),
}
if args.json:
print(json.dumps(report, indent=2))
return 0 if report["wrong_is_zero"] else 1
print(f"=== GSM8K microscope: {args.corpus.name} ({total} questions) ===")
for k in ("correct", "wrong", "refused"):
print(f" {k:9s}: {outcome[k]:6d} ({100 * outcome[k] / total:.2f}%)")
print(f"\nwrong==0 firewall: {'HOLDS' if report['wrong_is_zero'] else '*** BREACHED ***'}")
for w in wrongs:
print(f" reader={w['reader']} gold={w['gold']} {w['q']}")
print("\n=== refusal families (failures-first) ===")
for fam, n in families.most_common():
print(f" {n:6d} {fam}")
if no_injection_categories:
print("\n=== coverage map: recognizer categories with no injector ===")
for cat, n in no_injection_categories.most_common():
print(f" {n:6d} {cat}")
return 0 if report["wrong_is_zero"] else 1
if __name__ == "__main__":
raise SystemExit(main())

91
tests/test_candidate_graph_completeness_guard.py Normal file
View file

@ -0,0 +1,91 @@
"""Completeness guard for the candidate-graph reader (wrong==0 firewall).
The microscope over the full real GSM8K train split (7,473 questions)
found 5 confabulations the 47-case ``train_sample`` could not see: the
reader emitted a *partial* reading (the first grounded quantity) instead
of refusing, because admissibility checked grounding + round-trip but had
no COMPLETENESS leg.
This guard adds the missing leg, mirroring the derivation reader's
``verify.py`` (grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness):
Every numeric / multiplier quantity present in the source (all
statement sentences + the question) must be consumed by the chosen
reading. An uncovered source quantity => refuse.
The guard is REFUSAL-ONLY: it can never turn a refusal into an answer,
so it cannot create a wrong answer — it can only remove confabulations.
Its entire regression surface is the graph-path correct set, which on
train_sample is exactly {0024} and on real-train is {3343} (the same
Sidney/Brooke shape). Both MUST still solve.
"""
from __future__ import annotations
import pytest
from generate.math_candidate_graph import parse_and_solve
# The 5 real-GSM8K confabulations (exact corpus strings). Each MUST now
# refuse (answer is None) instead of emitting a partial reading.
CONFABULATIONS = {
553: (
"Emma buys 2 containers of milk every school day for lunch. She does "
"not go to school on the weekends. How many containers of milk does "
"she buy in 3 weeks?"
),
605: (
"Ivan has 20 dice. Jerry has twice as many dice as Ivan. How many "
"dice do they have altogether?"
),
693: (
"Ian had twenty roses. He gave six roses to his mother, nine roses "
"to his grandmother, four roses to his sister, and he kept the rest. "
"How many roses did Ian keep?"
),
6172: (
"Jimmy has 18 cards. Jimmy gives three cards to Bob. If Jimmy gives "
"Mary twice as many cards as he gave to Bob, how many cards does "
"Jimmy have left?"
),
7369: (
"Wilfred eats 4 carrots on Tuesday and 6 carrots on Wednesday. If "
"Wilfred wants to eat a total of 15 carrots from Tuesday to Thursday, "
"how many carrots does Wilfred need to eat on Thursday?"
),
}
# The graph-path correct case the guard MUST NOT break (train_sample 0024
# == real-train 3343).
SIDNEY_BROOKE = (
"Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, "
"and 50 on Thursday. Brooke does three times as many jumping jacks as "
"Sidney. How many jumping jacks did Brooke do?"
)
@pytest.mark.parametrize("idx", sorted(CONFABULATIONS))
def test_confabulation_now_refuses(idx: int) -> None:
"""Each previously-confabulated case must refuse (wrong==0 restored)."""
res = parse_and_solve(CONFABULATIONS[idx])
assert res.answer is None, (
f"[{idx}] expected refusal, got answer={res.answer!r} "
f"(refusal_reason={res.refusal_reason!r})"
)
def test_sidney_brooke_still_solves() -> None:
"""The day-enum + comparative graph case must still solve to 438."""
res = parse_and_solve(SIDNEY_BROOKE)
assert res.answer == 438.0, (
f"completeness guard over-refused the correct graph-path case: "
f"answer={res.answer!r} refusal_reason={res.refusal_reason!r}"
)
def test_guard_is_refusal_only_not_answer_changing() -> None:
"""A case that already solves correctly keeps its exact answer; the
guard never rewrites an answer value (refusal-only invariant)."""
res = parse_and_solve(SIDNEY_BROOKE)
# Same value, same unit-bearing graph — guard does not mutate solving.
assert res.answer == 438.0
assert res.selected_graph is not None