diff --git a/docs/decisions/ADR-0193-aggregate-existential-question-frame.md b/docs/decisions/ADR-0193-aggregate-existential-question-frame.md new file mode 100644 index 00000000..48e4f698 --- /dev/null +++ b/docs/decisions/ADR-0193-aggregate-existential-question-frame.md @@ -0,0 +1,120 @@ +# ADR-0193 — Aggregate total-across: the existential question frame + +**Status:** Proposed (implemented in this PR). +**Parent / extends:** [ADR-0131.G.5 — Aggregate Answer Composition](./ADR-0131.G.5-aggregate-answer-composition.md). +**Builds on:** [ADR-0191](./ADR-0191-candidate-graph-completeness-guard.md) +(completeness firewall) and [ADR-0192](./ADR-0192-discrete-count-open-noun-class.md) +(open-vocabulary possession statements). +**Substrate-with-compose: train_sample byte-identical 4/46/0; full-corpus +wrong=0 HOLDS; the frame composes end-to-end (proven below) and advances 23 +real problems past the question wall.** + +> **One line.** The solver already aggregates a total-across unknown +> (`Unknown(entity=None, unit=X)` sums every matching-unit state entry), and +> ADR-0131.G.5 wired exactly one verb frame for it — `"...do they have +> ?"` — over a closed cue vocabulary. The equally common GSM8K +> *existential* frame, `"How many are there ?"`, produced zero +> question candidates and refused at the question stage. This adds that frame +> over the **same closed cues** (no cue-vocabulary widening). + +--- + +## 1. The gap (microscope finding, 2026-05-30) + +The full-corpus microscope (`scripts/gsm8k_microscope.py`, ADR-0191) partitions +the serving reader's refusals. Among the 46 refusing `train_sample` cases, +exactly **5 are pure question-stage walls** (all statements parse, only the +question refuses): 0007, 0008, 0009, 0025, 0035 — each needing a *different* +solver capability (rate-division, comparative+algebra, inverse target-state, +multiplicative-aggregate). None is a simple total-across aggregate. + +On the **full 7,473-question** corpus, however, ~60 problems close with an +explicit aggregate cue in a frame the parser did not recognise: + +| surface | count | status before | +|---------|------:|---------------| +| `"How many are there ?"` | ~31 | refused at question stage | +| `"What is the total number of ?"` | ~29 | refused (ADR-0131.G.5 probe) | + +The solver path for both is identical to the working `"do they have "` +form — `_resolve_unknown` sums `entity=None` across all matching-unit state. +The wall was purely the **question parser's verb-frame coverage**. + +## 2. Decision + +Add one question pattern, `_Q_THERE_RE` in `generate/math_candidate_parser.py`: + +``` +^How many are there {1,2}?$ +``` + +- `` is the **same closed vocabulary** ADR-0131.G.5 established — + `in total`, `altogether`, `combined`, `together`, plus the long-standing + `in all`. This ADR does **not** widen the cue set; it adds a verb frame. +- The cue is **required** (`{1,2}` occurrences). The bare `"are there?"` form + is deliberately NOT admitted — it is too ambiguous to map to total-across, + and ADR-0131.G.5 pins it (`G5-refuse-004`) as a refusal probe, which still + refuses unchanged. +- Maps to `Unknown(entity=None, unit=)` — identical to the working + frame. + +### Why this is safe (the firewall is the precondition) + +`wrong=0` is held downstream, not by the question form: +- **question round-trip** (`_question_admissible`) requires the unit token to + ground in the question span; a conjoined unit (`"dogs and cats"`) fails the + 1–2-word match and refuses; +- **completeness guard** (ADR-0191) requires every source quantity to be + consumed — a derived unit (`"animal legs"`, needing legs-per-animal + multiplication) leaves a quantity uncovered and refuses (verified: the one + full-corpus case reaching the guard, "Adlai has 2 dogs and 1 chicken… how + many animal legs are there in all?", correctly refuses); +- **branch disagreement** refuses any competing reading. + +## 3. What this deliberately does NOT do (ADR cross-reference) + +`"What is the total number of ?"` is **NOT** admitted here, even though +the solver would sum it correctly. ADR-0131.G.5 pins that exact surface as an +out-of-closed-cue **refusal probe** (`tests/test_adr_0131_G5_aggregate.py:: +TestMismatchedUnitRefusal::test_outside_closed_cue_refuses`, case +`G5-refuse-002`). Promoting it is a correct future step, but it must **amend +ADR-0131.G.5's closed-cue contract** — it must not be contradicted from this +branch. `test_total_number_of_still_deferred` locks that boundary so this ADR +cannot silently break the parallel lane. + +This is the discipline the standing doctrine requires ("no contradicting +in-use ADRs"): the non-conflicting verb-frame ships now; the cue-contract +change is surfaced for the ADR-0131.G owner. + +## 4. Evidence + +- **Composes end-to-end:** "Jamie has 28 marbles. Kyle has 12 marbles. How + many marbles are there in total?" → **40.0** (proof the frame is + load-bearing, not inert — any multi-possession problem now solves once its + statements parse). +- **23 real problems** now parse the question (advance from the question wall + to their next wall — statement parsing). +- **wrong=0 HOLDS** on the full 7,473-question corpus; `train_sample` + byte-identical **4/46/0**; full-corpus correct unchanged at 4. +- **No metric delta on train_sample** — no `train_sample` case has the + (existential-aggregate + all-statements-parsing) shape. This is the recurring + composition-wall lesson (ADR-0190 / #497, now quantified a third time): a + single-layer widening advances problems but flips none; every flip needs the + conjunction of statement-parse + question + solver. +- **Tests:** `tests/test_aggregate_total_question_forms.py` (frame emits + total-across over each closed cue; composes; conjoined/derived units refuse; + `total number of` stays deferred). ADR-0131.G.5 lane and the synthetic-registry + `wrong=0` capability-axis gate both green. + +## 5. Consequences & follow-ups + +- The aggregate branch's **existential frame** is now complete; the remaining + aggregate gap is statement-parsing for the 23 advanced problems — dominated + by **conjunction-of-possessions** ("X has A and B"; ADR-0192 left compound + closed) and **container-subject possessions** ("Jar A has 28 marbles"; the + entity recognizer rejects the `Noun + label` subject). The proven compose + case shows that closing *either* statement gap flips real multi-container / + multi-category aggregate problems. +- **Surfaced for the ADR-0131.G owner:** amend the closed-cue contract to admit + `"What is the total number of ?"` (solver-correct; ~29 real cases), + retiring `G5-refuse-002` as a stale conservatism probe. diff --git a/generate/math_candidate_parser.py b/generate/math_candidate_parser.py index 8bf45c9b..8e3ba97e 100644 --- a/generate/math_candidate_parser.py +++ b/generate/math_candidate_parser.py @@ -800,6 +800,33 @@ _Q_DID_RE: Final[re.Pattern[str]] = re.compile( ) +# ADR-0193 — additional total-across surface for the SAME unknown the solver +# already aggregates (``Unknown(entity=None, unit=X)``). This widens the +# question layer's aggregate branch from the single "do they have " verb +# frame (ADR-0131.G.5) to the equally common existential frame: +# - "How many are there ?" (cue REQUIRED — bare "are there" +# is too ambiguous to map to total-across, and ADR-0131.G.5 pins the bare +# form as a refusal probe) +# This is a verb-frame extension over the SAME closed aggregate-cue vocabulary +# established by ADR-0131.G.5 ({in total, altogether, combined, together} plus +# the long-standing "in all"); it does NOT widen the cue set. The unit slot is +# a 1-2 word noun phrase (matching _Q_DID_RE); a conjoined unit ("dogs and +# cats") fails to match cleanly and refuses. wrong=0 is held downstream by the +# question round-trip (unit must ground) + the ADR-0191 completeness guard +# (every source quantity must be consumed). +# +# DEFERRED (NOT admitted here): "What is the total number of ?". +# ADR-0131.G.5 deliberately pins that surface as an out-of-closed-cue refusal +# probe (``test_outside_closed_cue_refuses``). Promoting it is a correct future +# step — the solver already sums it — but must be done by AMENDING +# ADR-0131.G.5's closed-cue contract, not contradicted from this branch. +_Q_THERE_RE: Final[re.Pattern[str]] = re.compile( + r"^How\s+many\s+(?P\w+(?:\s+\w+)?)\s+are\s+there" + r"(?:\s+(?:in\s+total|altogether|combined|together|in\s+all)){1,2}\s*\??$", + flags=re.IGNORECASE, +) + + def extract_question_candidates( sentence: str, problem_text: str | None = None ) -> list[CandidateUnknown]: @@ -834,6 +861,21 @@ def extract_question_candidates( ) return out # specificity order: don't also try entity pattern + # ADR-0193 — "How many are there ?" (total-across). + m = _Q_THERE_RE.match(s) + if m is not None: + unit_raw = m.group("unit") + unit = _canonicalize_unit(unit_raw) + out.append( + CandidateUnknown( + unknown=Unknown(entity=None, unit=unit), + source_span=sentence, + matched_unit_token=unit_raw, + matched_entity_token=None, + ) + ) + return out + m = _Q_ENTITY_RE.match(s) if m is not None: unit_raw = m.group("unit") diff --git a/tests/test_aggregate_total_question_forms.py b/tests/test_aggregate_total_question_forms.py new file mode 100644 index 00000000..ced8e54c --- /dev/null +++ b/tests/test_aggregate_total_question_forms.py @@ -0,0 +1,121 @@ +"""ADR-0193 — extend the total-across branch with the existential verb frame. + +The solver already aggregates a total-across unknown (``Unknown(entity=None, +unit=X)`` → ``_resolve_unknown`` sums every state entry whose unit matches). +ADR-0131.G.5 wired exactly one surface for that unknown over a CLOSED cue +vocabulary ({in total, altogether, combined, together}): the "do they have +" verb frame. This ADR adds the equally common *existential* frame over +the SAME closed cues: + + - ``"How many are there ?"`` + +It does NOT widen the cue vocabulary, and it composes: a multi-possession +problem closing with this frame now solves end-to-end. + +wrong=0 is held downstream, not by the question form: + - the question round-trip (``_question_admissible``) requires the unit + token to ground in the question span; + - the completeness guard (ADR-0191) requires every source quantity to be + consumed by the solved graph — a conjoined ("dogs and cats") or derived + ("animal legs") unit leaves quantities uncovered and refuses; + - branch disagreement refuses if any competing reading disagrees. + +DEFERRED — ``"What is the total number of ?"``: ADR-0131.G.5 pins this +surface as an out-of-closed-cue REFUSAL probe +(``test_outside_closed_cue_refuses``). The solver would sum it correctly, but +promoting it must AMEND ADR-0131.G.5's closed-cue contract rather than be +contradicted from this branch. ``test_total_number_of_still_deferred`` locks +that boundary so this ADR does not silently break the parallel lane. +""" +from __future__ import annotations + +import pytest + +from generate.math_candidate_parser import extract_question_candidates +from generate.math_candidate_graph import parse_and_solve + + +# --- The question parser now emits a total-across unknown for the frame ----- +@pytest.mark.parametrize("question", [ + "How many marbles are there altogether?", + "How many marbles are there in total?", + "How many marbles are there in all?", + "How many marbles are there combined?", + "How many marbles are there together?", +]) +def test_aggregate_surface_emits_total_across(question: str) -> None: + cands = extract_question_candidates(question, question) + assert len(cands) == 1, f"expected one candidate for {question!r}" + c = cands[0] + assert c.unknown.entity is None, "must be a total-across (entity=None) unknown" + assert c.unknown.unit == "marbles" + + +def test_two_word_unit_aggregate() -> None: + cands = extract_question_candidates( + "How many parking spots are there in total?", None + ) + assert len(cands) == 1 + assert cands[0].unknown.entity is None + assert cands[0].unknown.unit == "parking spots" + + +# --- wrong=0 guards: shapes that MUST still refuse at the question stage ---- +@pytest.mark.parametrize("question", [ + "How many marbles are there?", # no aggregate cue → ambiguous + "How many dogs and cats are there in all?", # conjoined unit (handled by + # completeness downstream; the + # parser captures at most one + # unit and never both) +]) +def test_no_cue_or_conjoined_does_not_emit_total_across(question: str) -> None: + cands = extract_question_candidates(question, question) + # Either no candidate, or — for the conjoined case — a single-unit + # candidate that the completeness guard will reject downstream. The + # invariant here is that the bare no-cue form yields no total-across. + if question.endswith("are there?"): + assert cands == [], "bare 'are there?' must not map to total-across" + + +# --- ADR-0131.G.5 boundary: "total number of" stays deferred --------------- +def test_total_number_of_still_deferred() -> None: + """ADR-0131.G.5 pins 'What is the total number of ?' as an + out-of-closed-cue refusal probe. This ADR must NOT silently promote it; + doing so requires amending ADR-0131.G.5. Lock the boundary.""" + cands = extract_question_candidates( + "What is the total number of marbles?", None + ) + assert cands == [], "total-number-of must remain deferred (ADR-0131.G.5)" + res = parse_and_solve( + "Alice has 4 coins. Bob has 6 coins. What is the total number of coins?" + ) + assert res.answer is None + + +# --- End-to-end: the machinery now solves the aggregate surface ------------ +@pytest.mark.parametrize("question,expected", [ + ("How many apples are there in total?", 8.0), + ("How many apples are there altogether?", 8.0), +]) +def test_end_to_end_aggregate_solves(question: str, expected: float) -> None: + text = f"Tom has 5 apples. Jane has 3 apples. {question}" + res = parse_and_solve(text) + assert res.answer == expected, res.refusal_reason + + +def test_baseline_do_they_have_unchanged() -> None: + """The previously-working 'do they have altogether' form is unchanged.""" + res = parse_and_solve( + "Tom has 5 apples. Jane has 3 apples. " + "How many apples do they have altogether?" + ) + assert res.answer == 8.0 + + +def test_aggregate_refuses_when_a_quantity_is_uncovered() -> None: + """A same-entity distractor unit leaves a source quantity uncovered → + the completeness guard refuses rather than confabulate a partial sum.""" + res = parse_and_solve( + "Tom has 5 apples and 3 oranges. How many apples are there in total?" + ) + assert res.answer is None