From a7024bb1f8891f11b40499dd6ffd92f05a4416d2 Mon Sep 17 00:00:00 2001 From: Shay Date: Sat, 30 May 2026 16:06:25 -0700 Subject: [PATCH] =?UTF-8?q?feat(adr-0192):=20open=20discrete=5Fcount=20nou?= =?UTF-8?q?n=20class=20=E2=80=94=208x=20statements=20parse,=20wrong=3D0-pr?= =?UTF-8?q?oven=20(substrate)=20(#497)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The discrete_count matcher gated the counted noun on a CLOSED ratified set (observed_counted_nouns): 'Betty has 24 marbles' matched, 'Randy has 60 mango trees' / 'Sam has 12 red apples' did not — purely because the noun was unseen. Open the single-anchor possession/acquisition path to an open noun phrase (adjective* + 1-3 word head, bounded by a stop-word lookahead so it never swallows a trailing PP), keeping every other narrowness layer (proper-noun subject, verb whitelist, single numeric token, no clause-split). Closed observed nouns still match (capitalized compounds preserved); compound enumeration stays closed. Safe because ADR-0191 moved the wrong=0 guarantee downstream: an open-vocab mis-parse hits the completeness guard + round-trip + branch-disagreement. Proof: full real corpus 61->494 discrete_count anchors (8x), wrong=0 HOLDS, zero confabulations. Substrate PR — 0 metric delta by design (train_sample byte-identical 4/46/0; the problems still need composition downstream). Value: the foundation every discrete_count flip consumes, and empirical proof open-vocab is firewall-safe. Co-authored-by: Claude Opus 4.8 --- ...ADR-0192-discrete-count-open-noun-class.md | 114 ++++++++++++++++++ generate/recognizer_match.py | 55 ++++++++- ...st_adr_0163_d2_discrete_count_injection.py | 14 ++- tests/test_discrete_count_open_noun_class.py | 62 ++++++++++ 4 files changed, 240 insertions(+), 5 deletions(-) create mode 100644 docs/decisions/ADR-0192-discrete-count-open-noun-class.md create mode 100644 tests/test_discrete_count_open_noun_class.py diff --git a/docs/decisions/ADR-0192-discrete-count-open-noun-class.md b/docs/decisions/ADR-0192-discrete-count-open-noun-class.md new file mode 100644 index 00000000..462207ce --- /dev/null +++ b/docs/decisions/ADR-0192-discrete-count-open-noun-class.md @@ -0,0 +1,114 @@ +# ADR-0192 — Open the discrete_count counted-noun class (firewall-backed) + +**Status:** Proposed (implemented in this PR). Widens the +[ADR-0163.D.2](./ADR-0163-recognizer-storage.md) discrete_count matcher. +Builds directly on [ADR-0191](./ADR-0191-candidate-graph-completeness-guard.md) +— the completeness firewall is the precondition that makes this safe. +**Substrate PR: 0 metric delta by design; the value is 8× more statements +parsing into solver state, wrong=0-proven on the full real corpus.** + +> **One line.** The discrete_count matcher gated the counted noun against a +> CLOSED ratified set (`observed_counted_nouns`): "Betty has 24 marbles" +> matched only because "marbles" was ratified, while "Randy has 60 mango +> trees" / "Sam has 12 red apples" produced no anchor purely because the noun +> was unseen. This opens the single-anchor possession/acquisition path to an +> open noun phrase, keeping every other narrowness layer. Wrong=0 is held +> downstream by the ADR-0191 completeness guard + round-trip + branch +> disagreement — not by the curated noun list. + +--- + +## 1. The gap (microscope finding, 2026-05-30) + +The full-corpus microscope (`scripts/gsm8k_microscope.py`) ranked the serving +reader's refusals across all 7,473 real GSM8K train questions. +**`discrete_count_statement` is the dominant wall: 3,850 first-wall refusals** +("recognizer matched but produced no injection"). Dissecting *why* the matcher +emits no anchor: + +| sub-shape | count | extractable? | +|-----------|------:|--------------| +| `subj verb N ` ("Randy has 60 **mango trees**") | ~1,004 | **yes — matcher too narrow** | +| count on a prepositional object ("sold clips **to 48** friends") | ~550 | no — correctly conservative | +| attributive number ("a **120-page** book") | ~120 | no — verb not possession/acquisition | +| number is a unit (rate/currency/time) | ~380 | no — different category | +| relational / "other" | ~1,400 | no — needs composition | + +Pinned blocker: the matcher only extracts when the counted noun is in +`spec.observed_counted_nouns` (a closed ratified set). `"Betty has 24 +marbles"` matched (ratified); `"Randy has 60 mango trees"` / `"Sam has 12 red +apples"` / `"Randy has 60 trees on his farm"` all emitted **anchors=0** solely +because the noun (or noun phrase) was unseen — not because of the trailing PP +(the regex already allowed trailing content) and not because the shape was +ambiguous. + +## 2. Decision + +Open the counted-noun slot of the **single-anchor** discrete_count extractor +(`_extract_discrete_count_re_open` in `generate/recognizer_match.py`): + +- The noun slot matches either a ratified `observed_counted_nouns` entry + (closed branch — preserves casing canonicalization and capitalized + compounds like "Pokemon cards") **OR** an OPEN lowercase noun phrase: + 1–3 consecutive lowercase word tokens, none a boundary/stop word + (prepositions, conjunctions, determiners, comparatives). +- `(?-i:...)` makes the open branch lowercase-only so it never captures a + following proper noun; the stop-word lookahead bounds the phrase so it + never swallows a trailing prepositional phrase ("mango trees on his farm" + → "mango trees"). +- **Every other narrowness layer is unchanged**: proper-noun subject, + possession/acquisition verb whitelist, single numeric token, no + clause-split. The compound-enumeration path stays closed. + +### Why this is safe (the firewall is the precondition) + +The closed noun set existed to prevent open-vocabulary mis-parses from +reaching the solver. ADR-0191 moved that guarantee downstream: an open-vocab +mis-parse now hits the **completeness guard** (every source quantity must be +consumed), the **round-trip filter** (every slot must ground in source), and +**branch-disagreement** refusal. So wrong=0 is held by the firewall, not by +the noun list. The dangerous shapes are still refused *before* the open noun +even applies — `"is reading a 120-page book"` refuses because "is" is not a +possession/acquisition verb; `"has many apples"` refuses on the count token; +`"has 60 apples and 30 oranges"` refuses on the single-count / clause-split +layers. + +## 3. Evidence + +- **Substrate gain: 61 → 494** discrete_count anchors extracted+injected over + the full real corpus (8×), all clean. +- **wrong=0 holds** on the full 7,473-question corpus — 494 statements parse, + **zero confabulations**. This is the direct proof that open-vocabulary + recognition is safe under the ADR-0191 firewall. +- **0 metric delta** (`train_sample` byte-identical **4/46/0**; full-corpus + correct unchanged at 4). The widening makes *statements* parse; the + *problems* still refuse downstream at the composition wall (multi-statement + chaining + question-target). This is expected: statement parsing is + necessary, not sufficient. Refusal families shift accordingly — problems + advance from the discrete_count first-wall to later walls. +- **Tests:** new `tests/test_discrete_count_open_noun_class.py` (open-vocab + now extracts; noun phrase stops before prepositions; dangerous shapes still + refuse). The one closed-contract assertion + (`test_unobserved_counted_noun_refused`) is updated to the new open + contract. All other discrete_count narrowness tests unchanged and passing. + +## 4. Consequences + +- This is **substrate**, deliberately landed with no metric movement. Its + value is (a) the foundation every discrete_count composition will consume — + a statement cannot be composed before it parses — and (b) the empirical + proof that the firewall makes open-vocabulary recognition wrong=0-safe, + retiring the closed-set constraint for the simple possession/acquisition + shape. +- The remaining discrete_count walls (prepositional-object counts, + attributive numbers, rate/currency) are correctly still refused — they are + *not* simple possession and must not be admitted by this path. +- The next layer is composition (multi-statement same-unit aggregate + + question-target parsing) which now has parsing statements to consume. + +## 5. Follow-ups + +- Re-run `scripts/gsm8k_microscope.py --corpus ` after the + composition layer lands to confirm wrong=0 holds *and* the metric moves. +- Compound-enumeration ("N1 noun1 and N2 noun2") noun class remains closed; + open it only after the single-anchor open path is proven in serving. diff --git a/generate/recognizer_match.py b/generate/recognizer_match.py index d623975e..37eaa32b 100644 --- a/generate/recognizer_match.py +++ b/generate/recognizer_match.py @@ -895,6 +895,53 @@ def _extract_discrete_count_re_for(counted_nouns: list[str]) -> re.Pattern[str]: ) +# ADR-0192 — words that terminate (cannot be part of) an open counted-noun +# phrase: prepositions, conjunctions, determiners, and comparative markers. +# Bounding the phrase against these is what stops the open noun from +# swallowing a trailing prepositional phrase ("mango trees on his farm" → +# "mango trees", not "mango trees on his farm"). +_OPEN_NOUN_STOP: Final[str] = ( + "on|in|at|to|for|with|of|from|by|per|into|onto|over|under|" + "and|or|but|than|as|that|which|who|whose|whom|while|when|because|" + "the|a|an|his|her|its|their|our|your|my|each|every|" + "more|fewer|less|most|fewest|other|another" +) + + +def _extract_discrete_count_re_open(counted_nouns: list[str]) -> re.Pattern[str]: + """ADR-0192 — open-vocabulary variant of the single-anchor extractor. + + Strictly additive: the counted-noun slot matches either a ratified + ``observed_counted_nouns`` entry (closed branch — preserves casing + canonicalization and capitalized compounds like ``Pokemon cards``) OR + an OPEN lowercase noun phrase: 1–3 consecutive lowercase word tokens, + none a boundary/stop word. The ``(?-i:...)`` makes the open branch + lowercase-only so it never captures a following proper noun, and the + stop-word lookahead bounds the phrase so it never swallows a trailing + prepositional phrase. Every other narrowness layer (proper-noun + subject, verb whitelist, single numeric token, no clause-split) is + unchanged; wrong=0 is held downstream by the ADR-0191 completeness + guard + round-trip + branch-disagreement. + """ + options = sorted({n for n in counted_nouns if n}, key=len, reverse=True) + closed_alt = "|".join(re.escape(n) for n in options) + open_tok = rf"(?-i:(?!(?:{_OPEN_NOUN_STOP})\b)[a-z]+)" + open_noun = rf"{open_tok}(?:\s+{open_tok}){{0,2}}" + noun_group = ( + rf"(?P{closed_alt}|{open_noun})" if closed_alt + else rf"(?P{open_noun})" + ) + return re.compile( + r"^\s*" + r"(?P(?-i:[A-Z][a-z]+))" + r"\s+(?P[A-Za-z]+)" + r"\s+(?P\d+|[A-Za-z\-]+)" + r"\s+" + noun_group + + r"(?:\b.*)?$", + flags=re.IGNORECASE, + ) + + _DIGIT_RUN_RE: Final[re.Pattern[str]] = re.compile(r"\d+(?:\.\d+)?") @@ -948,8 +995,12 @@ def _try_extract_discrete_count_anchor( if _count_quantity_tokens(statement, padded_lower) != 1: return None - # Narrowness #1 + #5 — shape + counted-noun lemma. - extract_re = _extract_discrete_count_re_for(observed_nouns) + # Narrowness #1 — shape. ADR-0192: the counted-noun slot is open + # (adjective* + multi-word head) rather than gated on the closed + # observed_counted_nouns set; the other narrowness layers above plus + # the downstream ADR-0191 completeness guard / round-trip / branch + # disagreement hold wrong=0 without the curated noun list. + extract_re = _extract_discrete_count_re_open(observed_nouns) m = extract_re.match(statement.strip()) if m is None: return None diff --git a/tests/test_adr_0163_d2_discrete_count_injection.py b/tests/test_adr_0163_d2_discrete_count_injection.py index 2f9402af..1a81cdba 100644 --- a/tests/test_adr_0163_d2_discrete_count_injection.py +++ b/tests/test_adr_0163_d2_discrete_count_injection.py @@ -162,9 +162,17 @@ class TestExtractionRefusal: # Two digit runs — v1 admits exactly one count. assert _try_extract("He has 2 horses, 5 dogs.") is None - def test_unobserved_counted_noun_refused(self) -> None: - # 'widgets' is not in the spec's observed_counted_nouns. - assert _try_extract("Sam has 5 widgets.") is None + def test_unobserved_counted_noun_now_admits(self) -> None: + # ADR-0192 — the counted-noun slot is OPEN: an unobserved noun + # ('widgets', not in the spec's observed_counted_nouns) admits + # under the simple possession shape. The other narrowness layers + # (subject/verb/count/clause) and the downstream ADR-0191 + # completeness guard + round-trip hold wrong=0, not the noun list. + result = _try_extract("Sam has 5 widgets.") + assert result is not None + assert result["counted_noun"].lower() == "widgets" + assert result["count_token"] == "5" + assert result["anchor_kind"] == "possession" def test_non_possession_non_acquisition_verb_refused(self) -> None: # Post-W2 (ADR-0170): possession verbs (has/have/had) AND diff --git a/tests/test_discrete_count_open_noun_class.py b/tests/test_discrete_count_open_noun_class.py new file mode 100644 index 00000000..9859fdb4 --- /dev/null +++ b/tests/test_discrete_count_open_noun_class.py @@ -0,0 +1,62 @@ +"""ADR-0192 — open the discrete_count counted-noun class. + +The discrete_count matcher gated the counted noun against a CLOSED ratified +set (``observed_counted_nouns``): "Betty has 24 marbles" matched only +because "marbles" was ratified, while "Randy has 60 mango trees" / "Sam has +12 red apples" emitted no anchor purely because the noun was unseen. + +This opens the single-anchor possession/acquisition path to an open +noun-phrase (adjective* + multi-word head), keeping every other narrowness +layer (proper-noun subject, possession/acquisition verb whitelist, single +numeric token, no clause-split). Wrong=0 is held downstream by the ADR-0191 +completeness guard + round-trip + branch-disagreement — not by the curated +noun list. +""" +from __future__ import annotations + +import pytest + +from generate.recognizer_match import match as rmatch +from generate.math_candidate_graph import _load_ratified_registry_or_empty +from generate.recognizer_anchor_inject import inject_from_match + +_REG = _load_ratified_registry_or_empty() + + +def _anchors(sentence: str) -> int: + m = rmatch(sentence, _REG, prior_subject=None) if _REG else None + return len(m.parsed_anchors) if m is not None else -1 + + +# --- Now-extractable open-vocabulary possession/acquisition statements ---- +@pytest.mark.parametrize("sentence", [ + "Randy has 60 mango trees.", # multi-word head + "Randy has 60 trees on his farm.", # single head + trailing PP + "Randy has 60 mango trees on his farm.",# both + "Sam has 12 red apples.", # adjective + head + "Tom bought 5 green bottles.", # acquisition + adjective +]) +def test_open_noun_now_extracts(sentence: str) -> None: + assert _anchors(sentence) == 1, f"expected one anchor for {sentence!r}" + + +def test_baseline_single_word_still_works() -> None: + """The previously-working closed-set case is unchanged.""" + assert _anchors("Betty has 24 marbles.") == 1 + + +# --- Noun phrase must NOT swallow the trailing prepositional phrase ------- +def test_noun_phrase_stops_before_preposition() -> None: + m = rmatch("Randy has 60 mango trees on his farm.", _REG, prior_subject=None) + assert m is not None and m.parsed_anchors + assert m.parsed_anchors[0]["counted_noun"].lower() == "mango trees" + + +# --- wrong=0 guards: shapes that MUST still refuse (no anchor) ------------- +@pytest.mark.parametrize("sentence", [ + "Julie is reading a 120-page book.", # verb not possession/acquisition + "Randy has many apples.", # indefinite quantifier, no count + "Randy has 60 apples and 30 oranges.", # clause/enumeration split +]) +def test_dangerous_shapes_still_refuse(sentence: str) -> None: + assert _anchors(sentence) == 0, f"expected no anchor for {sentence!r}"