From 25580e18b0d1b4e735e218b047407f0b0ca4dcc6 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 30 May 2026 15:45:07 -0700 Subject: [PATCH] =?UTF-8?q?fix(adr-0191):=20candidate-graph=20completeness?= =?UTF-8?q?=20guard=20=E2=80=94=20real-corpus=20wrong=205=E2=86=920=20(#49?= =?UTF-8?q?6)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5->0 The candidate-graph reader (serving) checked grounding + round-trip but had no completeness obligation, so problems whose later clauses failed to parse emitted a partial reading. Over the full 7,473-question real GSM8K train split this confabulated 5 answers (wrong!=0) the 47-case train_sample hid; 2 were regressions from #488. Add the missing admissibility leg (mirrors the derivation reader's verify.py): every source quantity (all statements + question) must be consumed by the chosen reading, else refuse. Refusal-only -> cannot create a wrong answer. Number-sense is pack-authoritative (en_numerics_v1 parse_compound_cardinal + lookup_multiplier + all 6 currency symbols) so it never disagrees with the engine; aggregating initials expose consumed_value_tokens provenance. Evidence: real-corpus wrong 5->0, correct held at 4; train_sample byte- identical 4/46/0; G1-G5+S1+G3.1 green; smoke 67 passed; math_teaching_corpus lane byte-identical. Co-Authored-By: Claude Opus 4.8 * feat(adr-0191): committed full-corpus GSM8K microscope (standing wrong=0 + coverage instrument) Promotes the throwaway tmp/ microscope that found the 5 confabulations into a committed tool. Runs the canonical serving reader over any GSM8K corpus and reports failures-first: correct/wrong/refused, every wrong answer by name, refusal families, and the no-injection per-category coverage map that ranks which injector to build next by real frequency. Default corpus is the committed 47-case train_sample (always available); --corpus path runs the full real split. This is the ADR-0191 follow-up: re-run after every capability PR, not just train_sample — a flip is only real if it does not widen the confabulation surface. Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- ...0191-candidate-graph-completeness-guard.md | 157 +++++++++++++ generate/math_candidate_graph.py | 43 +++- generate/math_candidate_parser.py | 21 +- generate/math_completeness.py | 210 ++++++++++++++++++ scripts/gsm8k_microscope.py | 148 ++++++++++++ ...test_candidate_graph_completeness_guard.py | 91 ++++++++ 6 files changed, 668 insertions(+), 2 deletions(-) create mode 100644 docs/decisions/ADR-0191-candidate-graph-completeness-guard.md create mode 100644 generate/math_completeness.py create mode 100644 scripts/gsm8k_microscope.py create mode 100644 tests/test_candidate_graph_completeness_guard.py diff --git a/docs/decisions/ADR-0191-candidate-graph-completeness-guard.md b/docs/decisions/ADR-0191-candidate-graph-completeness-guard.md new file mode 100644 index 00000000..63b73573 --- /dev/null +++ b/docs/decisions/ADR-0191-candidate-graph-completeness-guard.md @@ -0,0 +1,157 @@ +# ADR-0191 — Candidate-graph completeness guard (the missing wrong=0 leg) + +**Status:** Proposed (implemented in this PR). Hardens the +[ADR-0123](./ADR-0123-candidate-graph-reader.md) candidate-graph reader's +admissibility gate. Serving-path firewall fix; landed wrong=0-proven on +the **full real GSM8K train split**, not just the 47-case sample. + +> **One line.** The candidate-graph reader checked *grounding* and +> *round-trip* but had **no completeness obligation**, so a problem whose +> later clauses failed to parse still emitted whatever partial graph +> remained. Over the full 7,473-question real GSM8K train split this +> confabulated **5 answers** (wrong≠0). This adds the completeness leg the +> derivation reader's `verify.py` already has: every source quantity must be +> consumed by the chosen reading, else refuse. Result: real-corpus +> **wrong 5 → 0**; `train_sample` byte-identical **4/46/0**. + +--- + +## 1. The gap (full-corpus microscope finding, 2026-05-30) + +The 47-case `train_sample` reports wrong=0. Running the **canonical serving +reader** (`generate.math_candidate_graph.parse_and_solve`) over the *entire* +real GSM8K train split (7,473 questions) revealed the sample was hiding a +firewall breach: + +``` +correct 4 · wrong 5 · refused 7,464 (origin/main @ #488) +``` + +The 5 confabulations (deterministic, 3× reproduced): + +| idx | problem (abridged) | reader | gold | +|----:|--------------------|-------:|-----:| +| 553 | Emma buys 2/school-day … in 3 weeks? | 2 | 30 | +| 605 | Ivan 20 dice; Jerry twice as many; altogether? | 20 | 60 | +| 693 | Ian 20 roses; gave 6/9/4; kept rest? | 20 | 1 | +| 6172| Jimmy 18 cards; gives 3; Mary twice that; left? | 15 | 9 | +| 7369| Wilfred 4 Tue, 6 Wed; total 15 Tue–Thu; Thu? | -4 | 5 | + +Two of these (693, 7369) were **regressions introduced by #488** (ADR-0189/0189a): +they refused correctly before that PR and confabulated after. The 47-case gate +could not see it — exactly the lookback hazard CLAUDE.md §Lookback Review warns +about. + +**Root cause — one structural hole, not five anecdotes.** A graph is admitted +when its *present* elements ground and round-trip. Nothing checks that every +question-relevant source quantity is *represented*. When later clauses fail to +parse into operations, the residual partial graph still solves and is emitted: + +- `605` builds `initial=(Ivan:20), operations=()` — a zero-operation graph + answering "altogether"; the "twice as many" and the aggregate vanished. +- `693` the "He gave …" subtractions don't bind to "Ian" (live ADR-0174 + pronoun hazard) and 2 of 3 are dropped → bare initial survives. + +The derivation reader already refuses this (`verify.py`: grounding ∧ cue ∧ +unit ∧ **completeness** ∧ uniqueness). The candidate-graph reader was missing +the completeness leg. + +## 2. Decision + +Add a **completeness guard** as the final admissibility check in +`parse_and_solve`, in a dedicated module `generate/math_completeness.py`: + +> Collect every numeric / multiplier quantity in the source (all statement +> sentences **before** the numeric-only filter, plus the question). Collect +> every quantity the chosen reading actually **consumed** (candidate +> provenance). If a source quantity is not consumed, the reading is +> incomplete → **refuse**. + +```text +uncovered = quantity_values(all_statements + question) − consumed_values(chosen_branch) +if uncovered: refuse("incomplete reading: …") +``` + +### Why this preserves wrong=0 and cannot regress + +- **Refusal-only.** The guard only ever flips an *emitted answer* to a + *refusal*. It can never invent an answer, so it can only remove wrong + answers — never create one. Its entire regression surface is the + graph-path *correct* set, which is exactly `{train_sample 0024}` / + `{real-train 3343}` (the same Sidney/Brooke day-enum + comparative shape). + Both still solve (438). +- **Set semantics, not multiset.** `required − consumed` over value SETS + tolerates a quantity echoed in the question (no false refusal) while still + catching a clause whose distinct quantity was dropped — which is what every + observed confabulation does. +- **Short-circuits are immune.** Capacity / earnings / conditional / embedded + short-circuits return before the graph decision rule, so the guard never + touches them. + +### Pack-authoritative number-sense (no hand-rolled lexicon) + +Both the *required* scan and the *consumed* normalization resolve quantities +through the `en_numerics_v1` pack and the parser's own `_resolve_value`, so +identical surface forms cancel exactly and the guard never disagrees with the +engine about what a number is: + +- Compound cardinals via `parse_compound_cardinal` (`one hundred` → 100, + `two thousand five hundred` → 2500, `twenty-five` → 25). +- Multiplier anchors via `lookup_multiplier` — **read from the pack, not + hardcoded**, so the guard automatically covers `twice, thrice, half, + double, triple, quadruple, quintuple` and excludes ordinal-ambiguous + `third` / `quarter` (which are not multipliers in the pack). +- All six currency symbols (`$ ¢ € £ ¥ ₱`) tokenize as whole spans. + +### Provenance for aggregating extractors + +Aggregating initials collapse several source tokens into one derived value, so +they now expose every consumed token via a new +`CandidateInitial.consumed_value_tokens` field (default `()` → falls back to +`matched_value_token`, preserving all existing behavior): + +- day-enumeration → every per-day count; +- embedded-quantifier → both `N` and `M` of `N×M`; +- conjoined-embedded → all four factors; +- multi-word cardinal → the full phrase. + +## 3. Evidence + +- **Real GSM8K train (7,473):** wrong **5 → 0**; correct held at 4. Firewall + HOLDS. The 5 confabulations now refuse with `incomplete reading: …`. +- **`train_sample` (official metric):** byte-identical **4/46/0**, set + `{0014, 0018, 0024, 0042}`. +- **Capability axes:** G1–G5 + S1 + numerics extensions (G3.1) all green; the + first guard draft over-refused 20 G3 numerics cases (currency/decimal/ + hyphenated/compound-cardinal mis-parse) — fixed by making the number-sense + pack-authoritative. Two pre-existing failures + (`test_committed_report_matches_fresh_run`, `test_full_session_round_trips`) + fail identically on `origin/main` and are unrelated. +- **Smoke suite:** 67 passed. +- **Pinned serving lanes:** `math_teaching_corpus_v1` report byte-identical; + no pinned lane exercises `parse_and_solve`. +- **New tests:** `tests/test_candidate_graph_completeness_guard.py` (5 + confabulations refuse; Sidney/Brooke still solves; refusal-only invariant). + +## 4. Consequences + +- The candidate-graph reader now refuses partial readings instead of + confabulating — the wrong=0 firewall holds on real data, not just the + sample. This is the prerequisite for any further capability work: a flip is + only real if it does not also widen the confabulation surface. +- The guard makes some genuinely-incomplete shapes refuse that previously + emitted a (wrong) answer. That is the point. It never blocks a *complete* + reading; once a future capability consumes the dropped quantity, coverage is + satisfied and the case admits. +- **Limitation (documented, not load-bearing):** fraction *words* beyond the + multiplier set (e.g. `two-thirds`) are not yet recognized as required + quantities; this is conservative (can only under-refuse, never confabulate) + and is future work when a fraction capability lands. + +## 5. Follow-ups + +- Re-run the full-corpus microscope after each future capability PR as a + standing wrong=0 regression check (not just `train_sample`). +- The 693 confabulation also exposed the live ADR-0174 multi-actor pronoun + hazard; the completeness guard now refuses it, but the pronoun-binding fix + remains the proper long-term resolution. diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index 067e777b..e67884ab 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -60,6 +60,7 @@ from generate.math_problem_graph import ( MathGraphError, MathProblemGraph, ) +from generate.math_completeness import uncovered_quantities from generate.math_roundtrip import CandidateOperation, roundtrip_admissible from generate.math_solver import SolveError, solve @@ -103,6 +104,10 @@ class CandidateGraphAnswer: graph: MathProblemGraph answer: int | float + # ADR-0191 — the originating branch (statement choices + question + # choice). Carries per-candidate consumed-token provenance the + # completeness guard needs; the MathProblemGraph itself discards it. + branch: tuple["SentenceChoice | CandidateUnknown", ...] = () @dataclass(frozen=True, slots=True) @@ -463,6 +468,12 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: question_sentences = [s for s in sentences if s.rstrip().endswith("?")] statement_sentences = [s for s in sentences if not s.rstrip().endswith("?")] + # ADR-0191 — preserve EVERY statement sentence before the numeric-only + # filter below drops non-numeric ones. The completeness guard must see + # quantity signals carried in dropped sentences (e.g. "Jerry has twice + # as many … as Ivan" has no digit but a multiplier the reading must + # account for) to catch confabulations that emit a partial reading. + all_statement_sentences = list(statement_sentences) # ADR-0136.S.0 — Strip context-filler sentences before any extraction. # A sentence with no digit and no word-number cannot introduce parseable @@ -1026,7 +1037,11 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: except SolveError: continue admissible.append( - CandidateGraphAnswer(graph=graph, answer=trace.answer_value) + CandidateGraphAnswer( + graph=graph, + answer=trace.answer_value, + branch=(*stmt_choices, q_choice), + ) ) if not admissible: @@ -1055,6 +1070,32 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: # Single agreed answer. Pick the first admissible graph as the # canonical representative (deterministic since product() is ordered). chosen = admissible[0] + + # ADR-0191 — completeness guard (the missing admissibility leg). + # The branch grounded + round-tripped, but that only proves the + # quantities it DID read are real — not that it read ALL of them. + # If any source quantity (across every statement sentence + the + # question) is absent from the chosen reading, emitting its answer + # would confabulate a partial reading. Refuse instead (wrong==0). + # Refusal-only: this can never turn a refusal into an answer, so it + # cannot create a wrong answer — only remove confabulations. + uncovered = uncovered_quantities( + statement_sentences=all_statement_sentences, + question_text=question_sentences[0], + branch=chosen.branch, + ) + if uncovered: + return CandidateGraphResult( + answer=None, selected_graph=None, + refusal_reason=( + "incomplete reading: source quantities " + f"{sorted(uncovered)} not consumed by the solved graph" + ), + branches_enumerated=branches_enumerated, + branches_admissible=len(admissible), + reader_trace=tuple(reader_trace), + ) + return CandidateGraphResult( answer=chosen.answer, selected_graph=chosen.graph, diff --git a/generate/math_candidate_parser.py b/generate/math_candidate_parser.py index d65b07c5..8bf45c9b 100644 --- a/generate/math_candidate_parser.py +++ b/generate/math_candidate_parser.py @@ -87,6 +87,14 @@ class CandidateInitial: # count_token, amount_token, currency_symbol, composition_shape, # entity_source. composition_evidence: Mapping[str, str] | None = None + # ADR-0191 — completeness provenance. Aggregating extractors that + # collapse several source tokens into one derived value (day-enum sum, + # embedded-quantifier product, multi-word cardinal) list EVERY source + # quantity token they consumed here, so the candidate-graph reader's + # completeness guard (generate/math_completeness.py) can confirm no + # source quantity was silently dropped. Empty () means "single token" + # and the guard falls back to ``matched_value_token``. + consumed_value_tokens: tuple[str, ...] = () def __post_init__(self) -> None: # ADR-0127 widens the anchor set to include 'there are/were/is/was' @@ -1358,6 +1366,9 @@ def _multi_word_cardinal_candidates(sentence: str) -> list[CandidateInitial]: matched_value_token=value_raw.split()[0], matched_unit_token=unit_raw, matched_entity_token=m.group("entity"), + # ADR-0191 — the compound cardinal collapses every word into + # one value; the guard sees them via the joined surface form. + consumed_value_tokens=(value_raw,), ) ] except Exception: @@ -1595,7 +1606,8 @@ def _day_enumeration_candidates(sentence: str) -> list[CandidateInitial]: if m is None: return [] n1 = int(m.group("n1")) - rest_nums = [int(x) for x in _DAY_ENUM_REST_RE.findall(m.group("rest"))] + rest_raw = _DAY_ENUM_REST_RE.findall(m.group("rest")) + rest_nums = [int(x) for x in rest_raw] if not rest_nums: return [] total = float(n1 + sum(rest_nums)) @@ -1614,6 +1626,9 @@ def _day_enumeration_candidates(sentence: str) -> list[CandidateInitial]: matched_value_token=m.group("n1"), matched_unit_token=noun_raw, matched_entity_token=m.group("entity"), + # ADR-0191 — the sum collapses every per-day count; record + # them all so the completeness guard sees full coverage. + consumed_value_tokens=(m.group("n1"), *rest_raw), ) ] except Exception: @@ -1670,6 +1685,8 @@ def _embedded_quantifier_candidates(sentence: str) -> list[CandidateInitial]: matched_value_token=m_raw, matched_unit_token=unit_raw, matched_entity_token=m.group("entity"), + # ADR-0191 — the product N*M consumes both source tokens. + consumed_value_tokens=(n_raw, m_raw), ) ] except Exception: @@ -1750,6 +1767,8 @@ def _build_conj_embedded_sum( matched_value_token=m1_raw, # provenance: first per-container M matched_unit_token=m.group("u1"), matched_entity_token=m.group("entity"), + # ADR-0191 — the sum of two products consumes all four tokens. + consumed_value_tokens=(n1_raw, m1_raw, n2_raw, m2_raw), ) ] except Exception: diff --git a/generate/math_completeness.py b/generate/math_completeness.py new file mode 100644 index 00000000..34c1cbd4 --- /dev/null +++ b/generate/math_completeness.py @@ -0,0 +1,210 @@ +"""Completeness leg of the candidate-graph reader's admissibility gate. + +ADR-0191 — the candidate-graph reader checked *grounding* (every claimed +slot traces to a source token) and *round-trip* (the parsed candidate +re-realizes), but had no *completeness* obligation. A problem whose +later clauses failed to parse into operations still emitted whatever +partial graph remained — the classic confabulation the derivation +reader's ``verify.py`` already refuses (grounding ∧ cue ∧ unit ∧ +**completeness** ∧ uniqueness). + +This module supplies the missing leg as a pure, side-effect-free check: + + Collect every numeric / multiplier quantity present in the source + (all statement sentences + the question). Collect every quantity the + chosen reading actually CONSUMED (candidate provenance). If a source + quantity is not consumed, the reading is incomplete → the reader must + refuse. + +Design properties (why this preserves wrong==0 and cannot regress): + +- **Refusal-only.** The check only ever flips an emitted answer to a + refusal; it never invents an answer. So it can only *remove* wrong + answers, never create one. +- **Set semantics, not multiset.** ``uncovered = required - consumed`` + over value SETS. This deliberately tolerates a source quantity echoed + in the question (avoids false refusals) while still catching a clause + whose distinct quantity was dropped — which is what every observed + confabulation does. +- **Pack-authoritative number-sense.** Quantities are resolved through + the ``en_numerics_v1`` pack (``parse_compound_cardinal``) and the + parser's own ``_resolve_value`` — the same machinery the extractors + use. Identical surface forms (``$40``, ``twenty-five``, ``one + hundred``, ``3/4``) therefore resolve to identical values on both the + required and the consumed side and cancel exactly; the guard never + disagrees with the engine about what a number is. +- **Conservative multiplier set.** Only the unambiguous multiplier + anchors ``twice / thrice / half`` count as standalone quantity signals + (these are not cardinals). Ordinal-ambiguous words (``third`` / + ``quarter`` — usually "the third day") are excluded to avoid spurious + refusals. +""" +from __future__ import annotations + +import re +from typing import TYPE_CHECKING + +from generate.math_candidate_parser import _CURRENCY_SYMBOLS, _resolve_value +from language_packs.numerics_loader import ( + lookup_cardinal, + lookup_multiplier, + parse_compound_cardinal, +) + +if TYPE_CHECKING: # pragma: no cover - typing only + from generate.math_candidate_parser import CandidateInitial + from generate.math_roundtrip import CandidateOperation + +# Multiplier-anchor quantity signals (``twice``/``double``/``half`` ...) are +# read from the en_numerics_v1 pack via ``lookup_multiplier`` — NOT hardcoded +# — so the guard never drifts from the pack lexicon (it carries twice, +# thrice, half, double, triple, quadruple, quintuple). Ordinal-ambiguous +# words (``third`` / ``quarter``) are not multipliers in the pack, so they are +# excluded automatically rather than by a hand-maintained denylist. + + +def _multiplier_value(token: str) -> float | None: + entry = lookup_multiplier(token) + return float(entry.factor) if entry is not None else None + + +# Currency-symbol character class, taken from the parser's pinned symbol set +# (``$ ¢ € £ ¥ ₱``) so symbol-prefixed amounts tokenize as one span and +# resolve identically to the consumed candidate token. +_CURRENCY_CLASS = "".join(re.escape(c) for c in _CURRENCY_SYMBOLS) + +# One pass that yields, in order: currency/digit/decimal/slash-fraction +# literals, and word tokens (incl. hyphenated cardinals like "twenty-five"). +# Word runs are re-joined below so multi-word cardinals ("one hundred", +# "two thousand five hundred") resolve as a single quantity. +_TOKEN_RE = re.compile( + rf"[{_CURRENCY_CLASS}]?\d[\d,]*(?:\.\d+)?(?:/\d+)?" # $40 / 18.00 / 3/4 + r"|[A-Za-z]+(?:-[A-Za-z]+)*" # words incl. hyphenated +) + + +def _numeric_token_value(token: str) -> float | None: + """Value of a single non-cardinal token (digit/currency/fraction).""" + resolved = _resolve_value(token) + return float(resolved.value) if resolved is not None else None + + +def _token_value(token: str) -> float | None: + """Canonical numeric value of a single quantity token, or None. + + Multiplier anchors first, then compound cardinals (pack), then the + parser's value resolver for digit / currency / fraction surface + forms. Used to normalize CONSUMED candidate tokens identically to + the required scan. + """ + if not token: + return None + t = token.strip() + mult = _multiplier_value(t) + if mult is not None: + return mult + cardinal = parse_compound_cardinal(t) + if cardinal is not None: + return float(cardinal) + return _numeric_token_value(t) + + +def quantity_values_in_text(text: str) -> set[float]: + """Every numeric / multiplier quantity value present in ``text``. + + Greedily merges runs of cardinal words (joined by hyphens or "and") + so "two thousand five hundred" is one quantity, not five. Digit / + currency / fraction literals and multiplier anchors are resolved per + token. Pack-authoritative throughout. + """ + if not text: + return set() + values: set[float] = set() + tokens = _TOKEN_RE.findall(text) + + i = 0 + n = len(tokens) + while i < n: + tok = tokens[i] + low = tok.lower() + # Multiplier anchor (standalone quantity signal), per the pack. + mult = _multiplier_value(low) + if mult is not None: + values.add(mult) + i += 1 + continue + # Cardinal-word run: extend across adjacent cardinal words and + # interior "and" connectors ("three hundred and fifty"). + if lookup_cardinal(low) is not None: + run = [tok] + j = i + 1 + while j < n: + nxt = tokens[j].lower() + if lookup_cardinal(nxt) is not None: + run.append(tokens[j]) + j += 1 + elif nxt == "and" and j + 1 < n and lookup_cardinal( + tokens[j + 1].lower() + ) is not None: + run.append(tokens[j]) + j += 1 + else: + break + v = parse_compound_cardinal(" ".join(run)) + if v is not None: + values.add(float(v)) + i = j + continue + # Digit / currency / fraction literal. + v = _numeric_token_value(tok) + if v is not None: + values.add(v) + i += 1 + return values + + +def _candidate_consumed_tokens( + choice: "CandidateInitial | CandidateOperation", +) -> tuple[str, ...]: + """Source quantity tokens a single candidate consumed. + + Aggregating initials (day-enumeration, embedded-quantifier, + multi-word-cardinal) collapse several source tokens into one derived + value; they expose every consumed token via ``consumed_value_tokens``. + Every other candidate consumes exactly its ``matched_value_token``. + """ + consumed = getattr(choice, "consumed_value_tokens", ()) + if consumed: + return tuple(consumed) + tok = getattr(choice, "matched_value_token", "") + return (tok,) if tok else () + + +def consumed_values(branch: tuple[object, ...]) -> set[float]: + """Canonical quantity values consumed by a chosen reading (branch).""" + values: set[float] = set() + for choice in branch: + for tok in _candidate_consumed_tokens(choice): # type: ignore[arg-type] + v = _token_value(tok) + if v is not None: + values.add(v) + return values + + +def uncovered_quantities( + *, + statement_sentences: list[str], + question_text: str, + branch: tuple[object, ...], +) -> set[float]: + """Source quantities the chosen reading failed to consume. + + A non-empty result means the reading is incomplete: the source + carries a quantity the solved graph never accounts for, so emitting + its answer would confabulate. The reader must refuse. + """ + required: set[float] = set() + for s in statement_sentences: + required |= quantity_values_in_text(s) + required |= quantity_values_in_text(question_text) + return required - consumed_values(branch) diff --git a/scripts/gsm8k_microscope.py b/scripts/gsm8k_microscope.py new file mode 100644 index 00000000..cc61daf0 --- /dev/null +++ b/scripts/gsm8k_microscope.py @@ -0,0 +1,148 @@ +"""Full-corpus GSM8K microscope — the standing wrong=0 + coverage instrument. + +ADR-0191 follow-up. The 47-case ``train_sample`` cannot see confabulations +that only fire on rarer real-corpus shapes: ADR-0191 found 5 wrong answers +on the full 7,473-question real GSM8K train split that ``train_sample`` +reported as wrong=0. This tool runs the canonical serving reader +(``generate.math_candidate_graph.parse_and_solve``) over an arbitrary GSM8K +corpus and reports, failures-first: + + - correct / wrong / refused counts (wrong MUST be 0 — the firewall); + - every wrong answer (so a regression is named, not hidden in a count); + - refusal families, and for the dominant "recognizer matched but produced + no injection" family, the per-category breakdown — the coverage map + that ranks which injector to build next by real frequency. + +Run after EVERY capability PR, not just the sample: a flip is only real if +it does not also widen the confabulation surface. + +Usage: + # Default: the committed 47-case train_sample (always available). + uv run python scripts/gsm8k_microscope.py + + # Full real corpus (download train.jsonl from openai/grade-school-math): + uv run python scripts/gsm8k_microscope.py --corpus path/to/train.jsonl +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +from collections import Counter +from pathlib import Path + +_REPO_ROOT = Path(__file__).resolve().parents[1] +if str(_REPO_ROOT) not in sys.path: + sys.path.insert(0, str(_REPO_ROOT)) + +from generate.math_candidate_graph import parse_and_solve # noqa: E402 + +_TRAIN_SAMPLE = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl" +_CATEGORY_RE = re.compile(r"category=([a-z_]+)") + + +def _gold(record: dict) -> float | None: + """Numeric gold answer from either GSM8K-raw or train_sample schema.""" + if "answer_numeric" in record: + raw = str(record["answer_numeric"]) + elif "answer" in record: + raw = record["answer"].split("####")[-1] + else: + return None + raw = raw.strip().replace(",", "") + try: + return float(raw) + except ValueError: + return None + + +def _question(record: dict) -> str: + return record.get("question") or record.get("problem") or record.get("text") or "" + + +def _refusal_family(reason: str | None) -> str: + if not reason: + return "(no reason)" + if "no injection" in reason: + m = _CATEGORY_RE.search(reason) + return f"no_injection:{m.group(1) if m else '?'}" + if "no admissible candidate for statement" in reason: + return "statement_unparsed" + if "no admissible candidate for question" in reason: + return "question_unparsed" + if "no branch produced" in reason: + return "no_solvable_branch" + if "disagree" in reason: + return "branch_disagreement" + if "incomplete reading" in reason: + return "incomplete_reading" + if "round-trip" in reason or "round trip" in reason: + return "roundtrip_reject" + return reason[:48] + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument( + "--corpus", type=Path, default=_TRAIN_SAMPLE, + help="JSONL of GSM8K records (default: committed train_sample).", + ) + ap.add_argument("--json", action="store_true", help="Emit machine-readable JSON.") + args = ap.parse_args() + + rows = [json.loads(line) for line in args.corpus.read_text().splitlines() if line.strip()] + outcome: Counter[str] = Counter() + families: Counter[str] = Counter() + no_injection_categories: Counter[str] = Counter() + wrongs: list[dict] = [] + + for rec in rows: + q = _question(rec) + gold = _gold(rec) + res = parse_and_solve(q) + if res.answer is None: + outcome["refused"] += 1 + fam = _refusal_family(res.refusal_reason) + families[fam] += 1 + if fam.startswith("no_injection:"): + no_injection_categories[fam.split(":", 1)[1]] += 1 + elif gold is not None and abs(float(res.answer) - gold) < 1e-6: + outcome["correct"] += 1 + else: + outcome["wrong"] += 1 + wrongs.append({"q": q[:160], "reader": float(res.answer), "gold": gold}) + + total = len(rows) + report = { + "corpus": str(args.corpus), + "total": total, + "outcome": dict(outcome), + "wrong_is_zero": outcome["wrong"] == 0, + "wrongs": wrongs, + "refusal_families": dict(families.most_common()), + "no_injection_categories": dict(no_injection_categories.most_common()), + } + + if args.json: + print(json.dumps(report, indent=2)) + return 0 if report["wrong_is_zero"] else 1 + + print(f"=== GSM8K microscope: {args.corpus.name} ({total} questions) ===") + for k in ("correct", "wrong", "refused"): + print(f" {k:9s}: {outcome[k]:6d} ({100 * outcome[k] / total:.2f}%)") + print(f"\nwrong==0 firewall: {'HOLDS' if report['wrong_is_zero'] else '*** BREACHED ***'}") + for w in wrongs: + print(f" reader={w['reader']} gold={w['gold']} {w['q']}") + print("\n=== refusal families (failures-first) ===") + for fam, n in families.most_common(): + print(f" {n:6d} {fam}") + if no_injection_categories: + print("\n=== coverage map: recognizer categories with no injector ===") + for cat, n in no_injection_categories.most_common(): + print(f" {n:6d} {cat}") + return 0 if report["wrong_is_zero"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_candidate_graph_completeness_guard.py b/tests/test_candidate_graph_completeness_guard.py new file mode 100644 index 00000000..8207f973 --- /dev/null +++ b/tests/test_candidate_graph_completeness_guard.py @@ -0,0 +1,91 @@ +"""Completeness guard for the candidate-graph reader (wrong==0 firewall). + +The microscope over the full real GSM8K train split (7,473 questions) +found 5 confabulations the 47-case ``train_sample`` could not see: the +reader emitted a *partial* reading (the first grounded quantity) instead +of refusing, because admissibility checked grounding + round-trip but had +no COMPLETENESS leg. + +This guard adds the missing leg, mirroring the derivation reader's +``verify.py`` (grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness): + + Every numeric / multiplier quantity present in the source (all + statement sentences + the question) must be consumed by the chosen + reading. An uncovered source quantity => refuse. + +The guard is REFUSAL-ONLY: it can never turn a refusal into an answer, +so it cannot create a wrong answer — it can only remove confabulations. +Its entire regression surface is the graph-path correct set, which on +train_sample is exactly {0024} and on real-train is {3343} (the same +Sidney/Brooke shape). Both MUST still solve. +""" +from __future__ import annotations + +import pytest + +from generate.math_candidate_graph import parse_and_solve + +# The 5 real-GSM8K confabulations (exact corpus strings). Each MUST now +# refuse (answer is None) instead of emitting a partial reading. +CONFABULATIONS = { + 553: ( + "Emma buys 2 containers of milk every school day for lunch. She does " + "not go to school on the weekends. How many containers of milk does " + "she buy in 3 weeks?" + ), + 605: ( + "Ivan has 20 dice. Jerry has twice as many dice as Ivan. How many " + "dice do they have altogether?" + ), + 693: ( + "Ian had twenty roses. He gave six roses to his mother, nine roses " + "to his grandmother, four roses to his sister, and he kept the rest. " + "How many roses did Ian keep?" + ), + 6172: ( + "Jimmy has 18 cards. Jimmy gives three cards to Bob. If Jimmy gives " + "Mary twice as many cards as he gave to Bob, how many cards does " + "Jimmy have left?" + ), + 7369: ( + "Wilfred eats 4 carrots on Tuesday and 6 carrots on Wednesday. If " + "Wilfred wants to eat a total of 15 carrots from Tuesday to Thursday, " + "how many carrots does Wilfred need to eat on Thursday?" + ), +} + +# The graph-path correct case the guard MUST NOT break (train_sample 0024 +# == real-train 3343). +SIDNEY_BROOKE = ( + "Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, " + "and 50 on Thursday. Brooke does three times as many jumping jacks as " + "Sidney. How many jumping jacks did Brooke do?" +) + + +@pytest.mark.parametrize("idx", sorted(CONFABULATIONS)) +def test_confabulation_now_refuses(idx: int) -> None: + """Each previously-confabulated case must refuse (wrong==0 restored).""" + res = parse_and_solve(CONFABULATIONS[idx]) + assert res.answer is None, ( + f"[{idx}] expected refusal, got answer={res.answer!r} " + f"(refusal_reason={res.refusal_reason!r})" + ) + + +def test_sidney_brooke_still_solves() -> None: + """The day-enum + comparative graph case must still solve to 438.""" + res = parse_and_solve(SIDNEY_BROOKE) + assert res.answer == 438.0, ( + f"completeness guard over-refused the correct graph-path case: " + f"answer={res.answer!r} refusal_reason={res.refusal_reason!r}" + ) + + +def test_guard_is_refusal_only_not_answer_changing() -> None: + """A case that already solves correctly keeps its exact answer; the + guard never rewrites an answer value (refusal-only invariant).""" + res = parse_and_solve(SIDNEY_BROOKE) + # Same value, same unit-bearing graph — guard does not mutate solving. + assert res.answer == 438.0 + assert res.selected_graph is not None