fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5→0 (#496)

* fix(adr-0191): candidate-graph completeness guard — real-corpus wrong 5->0

The candidate-graph reader (serving) checked grounding + round-trip but had
no completeness obligation, so problems whose later clauses failed to parse
emitted a partial reading. Over the full 7,473-question real GSM8K train
split this confabulated 5 answers (wrong!=0) the 47-case train_sample hid;
2 were regressions from #488.

Add the missing admissibility leg (mirrors the derivation reader's verify.py):
every source quantity (all statements + question) must be consumed by the
chosen reading, else refuse. Refusal-only -> cannot create a wrong answer.
Number-sense is pack-authoritative (en_numerics_v1 parse_compound_cardinal +
lookup_multiplier + all 6 currency symbols) so it never disagrees with the
engine; aggregating initials expose consumed_value_tokens provenance.

Evidence: real-corpus wrong 5->0, correct held at 4; train_sample byte-
identical 4/46/0; G1-G5+S1+G3.1 green; smoke 67 passed; math_teaching_corpus
lane byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(adr-0191): committed full-corpus GSM8K microscope (standing wrong=0 + coverage instrument)

Promotes the throwaway tmp/ microscope that found the 5 confabulations into a
committed tool. Runs the canonical serving reader over any GSM8K corpus and
reports failures-first: correct/wrong/refused, every wrong answer by name,
refusal families, and the no-injection per-category coverage map that ranks
which injector to build next by real frequency.

Default corpus is the committed 47-case train_sample (always available);
--corpus path runs the full real split. This is the ADR-0191 follow-up: re-run
after every capability PR, not just train_sample — a flip is only real if it
does not widen the confabulation surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

docs/decisions/ADR-0191-candidate-graph-completeness-guard.md

@@ -0,0 +1,157 @@
+# ADR-0191 — Candidate-graph completeness guard (the missing wrong=0 leg)
+
+**Status:** Proposed (implemented in this PR). Hardens the
+[ADR-0123](./ADR-0123-candidate-graph-reader.md) candidate-graph reader's
+admissibility gate. Serving-path firewall fix; landed wrong=0-proven on
+the **full real GSM8K train split**, not just the 47-case sample.
+
+> **One line.** The candidate-graph reader checked *grounding* and
+> *round-trip* but had **no completeness obligation**, so a problem whose
+> later clauses failed to parse still emitted whatever partial graph
+> remained. Over the full 7,473-question real GSM8K train split this
+> confabulated **5 answers** (wrong≠0). This adds the completeness leg the
+> derivation reader's `verify.py` already has: every source quantity must be
+> consumed by the chosen reading, else refuse. Result: real-corpus
+> **wrong 5 → 0**; `train_sample` byte-identical **4/46/0**.
+
+---
+
+## 1. The gap (full-corpus microscope finding, 2026-05-30)
+
+The 47-case `train_sample` reports wrong=0. Running the **canonical serving
+reader** (`generate.math_candidate_graph.parse_and_solve`) over the *entire*
+real GSM8K train split (7,473 questions) revealed the sample was hiding a
+firewall breach:
+
+```
+correct 4 · wrong 5 · refused 7,464      (origin/main @ #488)
+```
+
+The 5 confabulations (deterministic, 3× reproduced):
+
+| idx | problem (abridged) | reader | gold |
+|----:|--------------------|-------:|-----:|
+| 553 | Emma buys 2/school-day … in 3 weeks? | 2 | 30 |
+| 605 | Ivan 20 dice; Jerry twice as many; altogether? | 20 | 60 |
+| 693 | Ian 20 roses; gave 6/9/4; kept rest? | 20 | 1 |
+| 6172| Jimmy 18 cards; gives 3; Mary twice that; left? | 15 | 9 |
+| 7369| Wilfred 4 Tue, 6 Wed; total 15 Tue–Thu; Thu? | -4 | 5 |
+
+Two of these (693, 7369) were **regressions introduced by #488** (ADR-0189/0189a):
+they refused correctly before that PR and confabulated after. The 47-case gate
+could not see it — exactly the lookback hazard CLAUDE.md §Lookback Review warns
+about.
+
+**Root cause — one structural hole, not five anecdotes.** A graph is admitted
+when its *present* elements ground and round-trip. Nothing checks that every
+question-relevant source quantity is *represented*. When later clauses fail to
+parse into operations, the residual partial graph still solves and is emitted:
+
+- `605` builds `initial=(Ivan:20), operations=()` — a zero-operation graph
+  answering "altogether"; the "twice as many" and the aggregate vanished.
+- `693` the "He gave …" subtractions don't bind to "Ian" (live ADR-0174
+  pronoun hazard) and 2 of 3 are dropped → bare initial survives.
+
+The derivation reader already refuses this (`verify.py`: grounding ∧ cue ∧
+unit ∧ **completeness** ∧ uniqueness). The candidate-graph reader was missing
+the completeness leg.
+
+## 2. Decision
+
+Add a **completeness guard** as the final admissibility check in
+`parse_and_solve`, in a dedicated module `generate/math_completeness.py`:
+
+> Collect every numeric / multiplier quantity in the source (all statement
+> sentences **before** the numeric-only filter, plus the question). Collect
+> every quantity the chosen reading actually **consumed** (candidate
+> provenance). If a source quantity is not consumed, the reading is
+> incomplete → **refuse**.
+
+```text
+uncovered = quantity_values(all_statements + question) − consumed_values(chosen_branch)
+if uncovered:  refuse("incomplete reading: …")
+```
+
+### Why this preserves wrong=0 and cannot regress
+
+- **Refusal-only.** The guard only ever flips an *emitted answer* to a
+  *refusal*. It can never invent an answer, so it can only remove wrong
+  answers — never create one. Its entire regression surface is the
+  graph-path *correct* set, which is exactly `{train_sample 0024}` /
+  `{real-train 3343}` (the same Sidney/Brooke day-enum + comparative shape).
+  Both still solve (438).
+- **Set semantics, not multiset.** `required − consumed` over value SETS
+  tolerates a quantity echoed in the question (no false refusal) while still
+  catching a clause whose distinct quantity was dropped — which is what every
+  observed confabulation does.
+- **Short-circuits are immune.** Capacity / earnings / conditional / embedded
+  short-circuits return before the graph decision rule, so the guard never
+  touches them.
+
+### Pack-authoritative number-sense (no hand-rolled lexicon)
+
+Both the *required* scan and the *consumed* normalization resolve quantities
+through the `en_numerics_v1` pack and the parser's own `_resolve_value`, so
+identical surface forms cancel exactly and the guard never disagrees with the
+engine about what a number is:
+
+- Compound cardinals via `parse_compound_cardinal` (`one hundred` → 100,
+  `two thousand five hundred` → 2500, `twenty-five` → 25).
+- Multiplier anchors via `lookup_multiplier` — **read from the pack, not
+  hardcoded**, so the guard automatically covers `twice, thrice, half,
+  double, triple, quadruple, quintuple` and excludes ordinal-ambiguous
+  `third` / `quarter` (which are not multipliers in the pack).
+- All six currency symbols (`$ ¢ € £ ¥ ₱`) tokenize as whole spans.
+
+### Provenance for aggregating extractors
+
+Aggregating initials collapse several source tokens into one derived value, so
+they now expose every consumed token via a new
+`CandidateInitial.consumed_value_tokens` field (default `()` → falls back to
+`matched_value_token`, preserving all existing behavior):
+
+- day-enumeration → every per-day count;
+- embedded-quantifier → both `N` and `M` of `N×M`;
+- conjoined-embedded → all four factors;
+- multi-word cardinal → the full phrase.
+
+## 3. Evidence
+
+- **Real GSM8K train (7,473):** wrong **5 → 0**; correct held at 4. Firewall
+  HOLDS. The 5 confabulations now refuse with `incomplete reading: …`.
+- **`train_sample` (official metric):** byte-identical **4/46/0**, set
+  `{0014, 0018, 0024, 0042}`.
+- **Capability axes:** G1–G5 + S1 + numerics extensions (G3.1) all green; the
+  first guard draft over-refused 20 G3 numerics cases (currency/decimal/
+  hyphenated/compound-cardinal mis-parse) — fixed by making the number-sense
+  pack-authoritative. Two pre-existing failures
+  (`test_committed_report_matches_fresh_run`, `test_full_session_round_trips`)
+  fail identically on `origin/main` and are unrelated.
+- **Smoke suite:** 67 passed.
+- **Pinned serving lanes:** `math_teaching_corpus_v1` report byte-identical;
+  no pinned lane exercises `parse_and_solve`.
+- **New tests:** `tests/test_candidate_graph_completeness_guard.py` (5
+  confabulations refuse; Sidney/Brooke still solves; refusal-only invariant).
+
+## 4. Consequences
+
+- The candidate-graph reader now refuses partial readings instead of
+  confabulating — the wrong=0 firewall holds on real data, not just the
+  sample. This is the prerequisite for any further capability work: a flip is
+  only real if it does not also widen the confabulation surface.
+- The guard makes some genuinely-incomplete shapes refuse that previously
+  emitted a (wrong) answer. That is the point. It never blocks a *complete*
+  reading; once a future capability consumes the dropped quantity, coverage is
+  satisfied and the case admits.
+- **Limitation (documented, not load-bearing):** fraction *words* beyond the
+  multiplier set (e.g. `two-thirds`) are not yet recognized as required
+  quantities; this is conservative (can only under-refuse, never confabulate)
+  and is future work when a fraction capability lands.
+
+## 5. Follow-ups
+
+- Re-run the full-corpus microscope after each future capability PR as a
+  standing wrong=0 regression check (not just `train_sample`).
+- The 693 confabulation also exposed the live ADR-0174 multi-actor pronoun
+  hazard; the completeness guard now refuses it, but the pronoun-binding fix
+  remains the proper long-term resolution.

generate/math_candidate_graph.py

@@ -60,6 +60,7 @@ from generate.math_problem_graph import (
     MathGraphError,
     MathProblemGraph,
 )
+from generate.math_completeness import uncovered_quantities
 from generate.math_roundtrip import CandidateOperation, roundtrip_admissible
 from generate.math_solver import SolveError, solve
 
@@ -103,6 +104,10 @@ class CandidateGraphAnswer:
 
     graph: MathProblemGraph
     answer: int | float
+    # ADR-0191 — the originating branch (statement choices + question
+    # choice).  Carries per-candidate consumed-token provenance the
+    # completeness guard needs; the MathProblemGraph itself discards it.
+    branch: tuple["SentenceChoice | CandidateUnknown", ...] = ()
 
 
 @dataclass(frozen=True, slots=True)
@@ -463,6 +468,12 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
 
     question_sentences = [s for s in sentences if s.rstrip().endswith("?")]
     statement_sentences = [s for s in sentences if not s.rstrip().endswith("?")]
+    # ADR-0191 — preserve EVERY statement sentence before the numeric-only
+    # filter below drops non-numeric ones.  The completeness guard must see
+    # quantity signals carried in dropped sentences (e.g. "Jerry has twice
+    # as many … as Ivan" has no digit but a multiplier the reading must
+    # account for) to catch confabulations that emit a partial reading.
+    all_statement_sentences = list(statement_sentences)
 
     # ADR-0136.S.0 — Strip context-filler sentences before any extraction.
     # A sentence with no digit and no word-number cannot introduce parseable
@@ -1026,7 +1037,11 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
         except SolveError:
             continue
         admissible.append(
-            CandidateGraphAnswer(graph=graph, answer=trace.answer_value)
+            CandidateGraphAnswer(
+                graph=graph,
+                answer=trace.answer_value,
+                branch=(*stmt_choices, q_choice),
+            )
         )
 
     if not admissible:
@@ -1055,6 +1070,32 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
     # Single agreed answer. Pick the first admissible graph as the
     # canonical representative (deterministic since product() is ordered).
     chosen = admissible[0]
+
+    # ADR-0191 — completeness guard (the missing admissibility leg).
+    # The branch grounded + round-tripped, but that only proves the
+    # quantities it DID read are real — not that it read ALL of them.
+    # If any source quantity (across every statement sentence + the
+    # question) is absent from the chosen reading, emitting its answer
+    # would confabulate a partial reading.  Refuse instead (wrong==0).
+    # Refusal-only: this can never turn a refusal into an answer, so it
+    # cannot create a wrong answer — only remove confabulations.
+    uncovered = uncovered_quantities(
+        statement_sentences=all_statement_sentences,
+        question_text=question_sentences[0],
+        branch=chosen.branch,
+    )
+    if uncovered:
+        return CandidateGraphResult(
+            answer=None, selected_graph=None,
+            refusal_reason=(
+                "incomplete reading: source quantities "
+                f"{sorted(uncovered)} not consumed by the solved graph"
+            ),
+            branches_enumerated=branches_enumerated,
+            branches_admissible=len(admissible),
+            reader_trace=tuple(reader_trace),
+        )
+
     return CandidateGraphResult(
         answer=chosen.answer,
         selected_graph=chosen.graph,

generate/math_candidate_parser.py

@@ -87,6 +87,14 @@ class CandidateInitial:
     #   count_token, amount_token, currency_symbol, composition_shape,
     #   entity_source.
     composition_evidence: Mapping[str, str] | None = None
+    # ADR-0191 — completeness provenance. Aggregating extractors that
+    # collapse several source tokens into one derived value (day-enum sum,
+    # embedded-quantifier product, multi-word cardinal) list EVERY source
+    # quantity token they consumed here, so the candidate-graph reader's
+    # completeness guard (generate/math_completeness.py) can confirm no
+    # source quantity was silently dropped. Empty () means "single token"
+    # and the guard falls back to ``matched_value_token``.
+    consumed_value_tokens: tuple[str, ...] = ()
 
     def __post_init__(self) -> None:
         # ADR-0127 widens the anchor set to include 'there are/were/is/was'
@@ -1358,6 +1366,9 @@ def _multi_word_cardinal_candidates(sentence: str) -> list[CandidateInitial]:
                 matched_value_token=value_raw.split()[0],
                 matched_unit_token=unit_raw,
                 matched_entity_token=m.group("entity"),
+                # ADR-0191 — the compound cardinal collapses every word into
+                # one value; the guard sees them via the joined surface form.
+                consumed_value_tokens=(value_raw,),
             )
         ]
     except Exception:
@@ -1595,7 +1606,8 @@ def _day_enumeration_candidates(sentence: str) -> list[CandidateInitial]:
     if m is None:
         return []
     n1 = int(m.group("n1"))
-    rest_nums = [int(x) for x in _DAY_ENUM_REST_RE.findall(m.group("rest"))]
+    rest_raw = _DAY_ENUM_REST_RE.findall(m.group("rest"))
+    rest_nums = [int(x) for x in rest_raw]
     if not rest_nums:
         return []
     total = float(n1 + sum(rest_nums))
@@ -1614,6 +1626,9 @@ def _day_enumeration_candidates(sentence: str) -> list[CandidateInitial]:
                 matched_value_token=m.group("n1"),
                 matched_unit_token=noun_raw,
                 matched_entity_token=m.group("entity"),
+                # ADR-0191 — the sum collapses every per-day count; record
+                # them all so the completeness guard sees full coverage.
+                consumed_value_tokens=(m.group("n1"), *rest_raw),
             )
         ]
     except Exception:
@@ -1670,6 +1685,8 @@ def _embedded_quantifier_candidates(sentence: str) -> list[CandidateInitial]:
                 matched_value_token=m_raw,
                 matched_unit_token=unit_raw,
                 matched_entity_token=m.group("entity"),
+                # ADR-0191 — the product N*M consumes both source tokens.
+                consumed_value_tokens=(n_raw, m_raw),
             )
         ]
     except Exception:
@@ -1750,6 +1767,8 @@ def _build_conj_embedded_sum(
                 matched_value_token=m1_raw,  # provenance: first per-container M
                 matched_unit_token=m.group("u1"),
                 matched_entity_token=m.group("entity"),
+                # ADR-0191 — the sum of two products consumes all four tokens.
+                consumed_value_tokens=(n1_raw, m1_raw, n2_raw, m2_raw),
             )
         ]
     except Exception:

generate/math_completeness.py

@@ -0,0 +1,210 @@
+"""Completeness leg of the candidate-graph reader's admissibility gate.
+
+ADR-0191 — the candidate-graph reader checked *grounding* (every claimed
+slot traces to a source token) and *round-trip* (the parsed candidate
+re-realizes), but had no *completeness* obligation.  A problem whose
+later clauses failed to parse into operations still emitted whatever
+partial graph remained — the classic confabulation the derivation
+reader's ``verify.py`` already refuses (grounding ∧ cue ∧ unit ∧
+**completeness** ∧ uniqueness).
+
+This module supplies the missing leg as a pure, side-effect-free check:
+
+    Collect every numeric / multiplier quantity present in the source
+    (all statement sentences + the question).  Collect every quantity the
+    chosen reading actually CONSUMED (candidate provenance).  If a source
+    quantity is not consumed, the reading is incomplete → the reader must
+    refuse.
+
+Design properties (why this preserves wrong==0 and cannot regress):
+
+- **Refusal-only.**  The check only ever flips an emitted answer to a
+  refusal; it never invents an answer.  So it can only *remove* wrong
+  answers, never create one.
+- **Set semantics, not multiset.**  ``uncovered = required - consumed``
+  over value SETS.  This deliberately tolerates a source quantity echoed
+  in the question (avoids false refusals) while still catching a clause
+  whose distinct quantity was dropped — which is what every observed
+  confabulation does.
+- **Pack-authoritative number-sense.**  Quantities are resolved through
+  the ``en_numerics_v1`` pack (``parse_compound_cardinal``) and the
+  parser's own ``_resolve_value`` — the same machinery the extractors
+  use.  Identical surface forms (``$40``, ``twenty-five``, ``one
+  hundred``, ``3/4``) therefore resolve to identical values on both the
+  required and the consumed side and cancel exactly; the guard never
+  disagrees with the engine about what a number is.
+- **Conservative multiplier set.**  Only the unambiguous multiplier
+  anchors ``twice / thrice / half`` count as standalone quantity signals
+  (these are not cardinals).  Ordinal-ambiguous words (``third`` /
+  ``quarter`` — usually "the third day") are excluded to avoid spurious
+  refusals.
+"""
+from __future__ import annotations
+
+import re
+from typing import TYPE_CHECKING
+
+from generate.math_candidate_parser import _CURRENCY_SYMBOLS, _resolve_value
+from language_packs.numerics_loader import (
+    lookup_cardinal,
+    lookup_multiplier,
+    parse_compound_cardinal,
+)
+
+if TYPE_CHECKING:  # pragma: no cover - typing only
+    from generate.math_candidate_parser import CandidateInitial
+    from generate.math_roundtrip import CandidateOperation
+
+# Multiplier-anchor quantity signals (``twice``/``double``/``half`` ...) are
+# read from the en_numerics_v1 pack via ``lookup_multiplier`` — NOT hardcoded
+# — so the guard never drifts from the pack lexicon (it carries twice,
+# thrice, half, double, triple, quadruple, quintuple).  Ordinal-ambiguous
+# words (``third`` / ``quarter``) are not multipliers in the pack, so they are
+# excluded automatically rather than by a hand-maintained denylist.
+
+
+def _multiplier_value(token: str) -> float | None:
+    entry = lookup_multiplier(token)
+    return float(entry.factor) if entry is not None else None
+
+
+# Currency-symbol character class, taken from the parser's pinned symbol set
+# (``$ ¢ € £ ¥ ₱``) so symbol-prefixed amounts tokenize as one span and
+# resolve identically to the consumed candidate token.
+_CURRENCY_CLASS = "".join(re.escape(c) for c in _CURRENCY_SYMBOLS)
+
+# One pass that yields, in order: currency/digit/decimal/slash-fraction
+# literals, and word tokens (incl. hyphenated cardinals like "twenty-five").
+# Word runs are re-joined below so multi-word cardinals ("one hundred",
+# "two thousand five hundred") resolve as a single quantity.
+_TOKEN_RE = re.compile(
+    rf"[{_CURRENCY_CLASS}]?\d[\d,]*(?:\.\d+)?(?:/\d+)?"  # $40 / 18.00 / 3/4
+    r"|[A-Za-z]+(?:-[A-Za-z]+)*"                          # words incl. hyphenated
+)
+
+
+def _numeric_token_value(token: str) -> float | None:
+    """Value of a single non-cardinal token (digit/currency/fraction)."""
+    resolved = _resolve_value(token)
+    return float(resolved.value) if resolved is not None else None
+
+
+def _token_value(token: str) -> float | None:
+    """Canonical numeric value of a single quantity token, or None.
+
+    Multiplier anchors first, then compound cardinals (pack), then the
+    parser's value resolver for digit / currency / fraction surface
+    forms.  Used to normalize CONSUMED candidate tokens identically to
+    the required scan.
+    """
+    if not token:
+        return None
+    t = token.strip()
+    mult = _multiplier_value(t)
+    if mult is not None:
+        return mult
+    cardinal = parse_compound_cardinal(t)
+    if cardinal is not None:
+        return float(cardinal)
+    return _numeric_token_value(t)
+
+
+def quantity_values_in_text(text: str) -> set[float]:
+    """Every numeric / multiplier quantity value present in ``text``.
+
+    Greedily merges runs of cardinal words (joined by hyphens or "and")
+    so "two thousand five hundred" is one quantity, not five.  Digit /
+    currency / fraction literals and multiplier anchors are resolved per
+    token.  Pack-authoritative throughout.
+    """
+    if not text:
+        return set()
+    values: set[float] = set()
+    tokens = _TOKEN_RE.findall(text)
+
+    i = 0
+    n = len(tokens)
+    while i < n:
+        tok = tokens[i]
+        low = tok.lower()
+        # Multiplier anchor (standalone quantity signal), per the pack.
+        mult = _multiplier_value(low)
+        if mult is not None:
+            values.add(mult)
+            i += 1
+            continue
+        # Cardinal-word run: extend across adjacent cardinal words and
+        # interior "and" connectors ("three hundred and fifty").
+        if lookup_cardinal(low) is not None:
+            run = [tok]
+            j = i + 1
+            while j < n:
+                nxt = tokens[j].lower()
+                if lookup_cardinal(nxt) is not None:
+                    run.append(tokens[j])
+                    j += 1
+                elif nxt == "and" and j + 1 < n and lookup_cardinal(
+                    tokens[j + 1].lower()
+                ) is not None:
+                    run.append(tokens[j])
+                    j += 1
+                else:
+                    break
+            v = parse_compound_cardinal(" ".join(run))
+            if v is not None:
+                values.add(float(v))
+            i = j
+            continue
+        # Digit / currency / fraction literal.
+        v = _numeric_token_value(tok)
+        if v is not None:
+            values.add(v)
+        i += 1
+    return values
+
+
+def _candidate_consumed_tokens(
+    choice: "CandidateInitial | CandidateOperation",
+) -> tuple[str, ...]:
+    """Source quantity tokens a single candidate consumed.
+
+    Aggregating initials (day-enumeration, embedded-quantifier,
+    multi-word-cardinal) collapse several source tokens into one derived
+    value; they expose every consumed token via ``consumed_value_tokens``.
+    Every other candidate consumes exactly its ``matched_value_token``.
+    """
+    consumed = getattr(choice, "consumed_value_tokens", ())
+    if consumed:
+        return tuple(consumed)
+    tok = getattr(choice, "matched_value_token", "")
+    return (tok,) if tok else ()
+
+
+def consumed_values(branch: tuple[object, ...]) -> set[float]:
+    """Canonical quantity values consumed by a chosen reading (branch)."""
+    values: set[float] = set()
+    for choice in branch:
+        for tok in _candidate_consumed_tokens(choice):  # type: ignore[arg-type]
+            v = _token_value(tok)
+            if v is not None:
+                values.add(v)
+    return values
+
+
+def uncovered_quantities(
+    *,
+    statement_sentences: list[str],
+    question_text: str,
+    branch: tuple[object, ...],
+) -> set[float]:
+    """Source quantities the chosen reading failed to consume.
+
+    A non-empty result means the reading is incomplete: the source
+    carries a quantity the solved graph never accounts for, so emitting
+    its answer would confabulate.  The reader must refuse.
+    """
+    required: set[float] = set()
+    for s in statement_sentences:
+        required |= quantity_values_in_text(s)
+    required |= quantity_values_in_text(question_text)
+    return required - consumed_values(branch)

scripts/gsm8k_microscope.py

@@ -0,0 +1,148 @@
+"""Full-corpus GSM8K microscope — the standing wrong=0 + coverage instrument.
+
+ADR-0191 follow-up. The 47-case ``train_sample`` cannot see confabulations
+that only fire on rarer real-corpus shapes: ADR-0191 found 5 wrong answers
+on the full 7,473-question real GSM8K train split that ``train_sample``
+reported as wrong=0. This tool runs the canonical serving reader
+(``generate.math_candidate_graph.parse_and_solve``) over an arbitrary GSM8K
+corpus and reports, failures-first:
+
+  - correct / wrong / refused counts (wrong MUST be 0 — the firewall);
+  - every wrong answer (so a regression is named, not hidden in a count);
+  - refusal families, and for the dominant "recognizer matched but produced
+    no injection" family, the per-category breakdown — the coverage map
+    that ranks which injector to build next by real frequency.
+
+Run after EVERY capability PR, not just the sample: a flip is only real if
+it does not also widen the confabulation surface.
+
+Usage:
+    # Default: the committed 47-case train_sample (always available).
+    uv run python scripts/gsm8k_microscope.py
+
+    # Full real corpus (download train.jsonl from openai/grade-school-math):
+    uv run python scripts/gsm8k_microscope.py --corpus path/to/train.jsonl
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from collections import Counter
+from pathlib import Path
+
+_REPO_ROOT = Path(__file__).resolve().parents[1]
+if str(_REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(_REPO_ROOT))
+
+from generate.math_candidate_graph import parse_and_solve  # noqa: E402
+
+_TRAIN_SAMPLE = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
+_CATEGORY_RE = re.compile(r"category=([a-z_]+)")
+
+
+def _gold(record: dict) -> float | None:
+    """Numeric gold answer from either GSM8K-raw or train_sample schema."""
+    if "answer_numeric" in record:
+        raw = str(record["answer_numeric"])
+    elif "answer" in record:
+        raw = record["answer"].split("####")[-1]
+    else:
+        return None
+    raw = raw.strip().replace(",", "")
+    try:
+        return float(raw)
+    except ValueError:
+        return None
+
+
+def _question(record: dict) -> str:
+    return record.get("question") or record.get("problem") or record.get("text") or ""
+
+
+def _refusal_family(reason: str | None) -> str:
+    if not reason:
+        return "(no reason)"
+    if "no injection" in reason:
+        m = _CATEGORY_RE.search(reason)
+        return f"no_injection:{m.group(1) if m else '?'}"
+    if "no admissible candidate for statement" in reason:
+        return "statement_unparsed"
+    if "no admissible candidate for question" in reason:
+        return "question_unparsed"
+    if "no branch produced" in reason:
+        return "no_solvable_branch"
+    if "disagree" in reason:
+        return "branch_disagreement"
+    if "incomplete reading" in reason:
+        return "incomplete_reading"
+    if "round-trip" in reason or "round trip" in reason:
+        return "roundtrip_reject"
+    return reason[:48]
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument(
+        "--corpus", type=Path, default=_TRAIN_SAMPLE,
+        help="JSONL of GSM8K records (default: committed train_sample).",
+    )
+    ap.add_argument("--json", action="store_true", help="Emit machine-readable JSON.")
+    args = ap.parse_args()
+
+    rows = [json.loads(line) for line in args.corpus.read_text().splitlines() if line.strip()]
+    outcome: Counter[str] = Counter()
+    families: Counter[str] = Counter()
+    no_injection_categories: Counter[str] = Counter()
+    wrongs: list[dict] = []
+
+    for rec in rows:
+        q = _question(rec)
+        gold = _gold(rec)
+        res = parse_and_solve(q)
+        if res.answer is None:
+            outcome["refused"] += 1
+            fam = _refusal_family(res.refusal_reason)
+            families[fam] += 1
+            if fam.startswith("no_injection:"):
+                no_injection_categories[fam.split(":", 1)[1]] += 1
+        elif gold is not None and abs(float(res.answer) - gold) < 1e-6:
+            outcome["correct"] += 1
+        else:
+            outcome["wrong"] += 1
+            wrongs.append({"q": q[:160], "reader": float(res.answer), "gold": gold})
+
+    total = len(rows)
+    report = {
+        "corpus": str(args.corpus),
+        "total": total,
+        "outcome": dict(outcome),
+        "wrong_is_zero": outcome["wrong"] == 0,
+        "wrongs": wrongs,
+        "refusal_families": dict(families.most_common()),
+        "no_injection_categories": dict(no_injection_categories.most_common()),
+    }
+
+    if args.json:
+        print(json.dumps(report, indent=2))
+        return 0 if report["wrong_is_zero"] else 1
+
+    print(f"=== GSM8K microscope: {args.corpus.name} ({total} questions) ===")
+    for k in ("correct", "wrong", "refused"):
+        print(f"  {k:9s}: {outcome[k]:6d}  ({100 * outcome[k] / total:.2f}%)")
+    print(f"\nwrong==0 firewall: {'HOLDS' if report['wrong_is_zero'] else '*** BREACHED ***'}")
+    for w in wrongs:
+        print(f"  reader={w['reader']} gold={w['gold']}  {w['q']}")
+    print("\n=== refusal families (failures-first) ===")
+    for fam, n in families.most_common():
+        print(f"  {n:6d}  {fam}")
+    if no_injection_categories:
+        print("\n=== coverage map: recognizer categories with no injector ===")
+        for cat, n in no_injection_categories.most_common():
+            print(f"  {n:6d}  {cat}")
+    return 0 if report["wrong_is_zero"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/test_candidate_graph_completeness_guard.py

@@ -0,0 +1,91 @@
+"""Completeness guard for the candidate-graph reader (wrong==0 firewall).
+
+The microscope over the full real GSM8K train split (7,473 questions)
+found 5 confabulations the 47-case ``train_sample`` could not see: the
+reader emitted a *partial* reading (the first grounded quantity) instead
+of refusing, because admissibility checked grounding + round-trip but had
+no COMPLETENESS leg.
+
+This guard adds the missing leg, mirroring the derivation reader's
+``verify.py`` (grounding ∧ cue ∧ unit ∧ completeness ∧ uniqueness):
+
+    Every numeric / multiplier quantity present in the source (all
+    statement sentences + the question) must be consumed by the chosen
+    reading.  An uncovered source quantity => refuse.
+
+The guard is REFUSAL-ONLY: it can never turn a refusal into an answer,
+so it cannot create a wrong answer — it can only remove confabulations.
+Its entire regression surface is the graph-path correct set, which on
+train_sample is exactly {0024} and on real-train is {3343} (the same
+Sidney/Brooke shape).  Both MUST still solve.
+"""
+from __future__ import annotations
+
+import pytest
+
+from generate.math_candidate_graph import parse_and_solve
+
+# The 5 real-GSM8K confabulations (exact corpus strings).  Each MUST now
+# refuse (answer is None) instead of emitting a partial reading.
+CONFABULATIONS = {
+    553: (
+        "Emma buys 2 containers of milk every school day for lunch. She does "
+        "not go to school on the weekends. How many containers of milk does "
+        "she buy in 3 weeks?"
+    ),
+    605: (
+        "Ivan has 20 dice. Jerry has twice as many dice as Ivan. How many "
+        "dice do they have altogether?"
+    ),
+    693: (
+        "Ian had twenty roses. He gave six roses to his mother,  nine roses "
+        "to his grandmother, four roses to his sister, and he kept the rest. "
+        "How many roses did Ian keep?"
+    ),
+    6172: (
+        "Jimmy has 18 cards. Jimmy gives three cards to Bob. If Jimmy gives "
+        "Mary twice as many cards as he gave to Bob, how many cards does "
+        "Jimmy have left?"
+    ),
+    7369: (
+        "Wilfred eats 4 carrots on Tuesday and 6 carrots on Wednesday. If "
+        "Wilfred wants to eat a total of 15 carrots from Tuesday to Thursday, "
+        "how many carrots does Wilfred need to eat on Thursday?"
+    ),
+}
+
+# The graph-path correct case the guard MUST NOT break (train_sample 0024
+# == real-train 3343).
+SIDNEY_BROOKE = (
+    "Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, "
+    "and 50 on Thursday. Brooke does three times as many jumping jacks as "
+    "Sidney. How many jumping jacks did Brooke do?"
+)
+
+
+@pytest.mark.parametrize("idx", sorted(CONFABULATIONS))
+def test_confabulation_now_refuses(idx: int) -> None:
+    """Each previously-confabulated case must refuse (wrong==0 restored)."""
+    res = parse_and_solve(CONFABULATIONS[idx])
+    assert res.answer is None, (
+        f"[{idx}] expected refusal, got answer={res.answer!r} "
+        f"(refusal_reason={res.refusal_reason!r})"
+    )
+
+
+def test_sidney_brooke_still_solves() -> None:
+    """The day-enum + comparative graph case must still solve to 438."""
+    res = parse_and_solve(SIDNEY_BROOKE)
+    assert res.answer == 438.0, (
+        f"completeness guard over-refused the correct graph-path case: "
+        f"answer={res.answer!r} refusal_reason={res.refusal_reason!r}"
+    )
+
+
+def test_guard_is_refusal_only_not_answer_changing() -> None:
+    """A case that already solves correctly keeps its exact answer; the
+    guard never rewrites an answer value (refusal-only invariant)."""
+    res = parse_and_solve(SIDNEY_BROOKE)
+    # Same value, same unit-bearing graph — guard does not mutate solving.
+    assert res.answer == 438.0
+    assert res.selected_graph is not None