diff --git a/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md b/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md new file mode 100644 index 00000000..e2fc41b5 --- /dev/null +++ b/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md @@ -0,0 +1,56 @@ +# CORE Problem-Solving Capability Roadmap v2 — 2026-06-17 + +**Status:** Living document (docs-only update) +**Date:** 2026-06-17 +**Context:** Post PR #797 (rate injection) + #798; preparing Inc3 rate follow-up before Gate A1 comparative injection. + +## Overview + +This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate injection delivered by PR #797. + +As of 2026-06-17, PR #797 is merged and #798 is merged. + +## GSM8K Workstream A + +- Inc 1: reader/recognizer baseline lift (discrete etc.) +- Inc 2: frontier measurement + stale doctrine repair + narrow rate injection (PR #797) +- **Inc 3 (current seam):** Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection. + +### Recommended Inc3 target (narrow) + +Make the rate frontier evidence actionable by resolving the next narrow blocker exposed by #797. + +Scope candidates (in preference order for this increment): +1. Denominator-state support for rate application (if failures surface as "actor has rate but no denom-unit quantity reachable"). +2. Safe connector expansion only if frontier proves "for one cup" is a dominant blocker. +3. Measurement-only frontier report refresh if artifacts stale. + +Inc3 selected #2 (connector for "for one cup"/"one" token) because live debug on the pinned report + cases showed the exact remaining rate injector deferral from Inc2 (matcher left rate_anchor_token=None for "one"; spec unresolved_notes explicitly called it out for the Alexa surface). This was the minimal change that reclassifies the rate_with_currency no-injection bucket (making evidence actionable) while preserving all guards. Denom production is larger future work (see ratification for rationale and out-of-scope). + +"Complete and harden PR #797" is revised as: Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection. + +As of 2026-06-17, PR #797 is merged and #798 is merged. + +Explicitly: do not broaden to full rate language family, comparative injection, or non-rate categories in this increment. + +## Gate A1 / Comparative Injection + +Deferred until after the post-#797 rate follow-up loop is closed with Inc3 measurement. + +## Success Criteria for This Phase + +- Frontier report run on current main (train-sample proxy). +- One narrow ratified Inc3 change. +- Wrong=0 preserved on train_sample, practice, and relevant confusers. +- Rate-family "recognized_no_injection" bucket reduced or its refusal mode made actionable (e.g. surfaces the true next blocker like denom reachability). +- No rebaseline of sealed lanes or SHA movement without separate ratification. +- Documentation (this roadmap + Inc3 ratification) committed as docs-first. + +## Out of Scope (for Inc3) + +- Full comparative (Gate A1) implementation. +- Broad recognizer anchor work or other shape categories. +- Changes to serving sealed paths. +- Any mutation of identity, policy, or algebra invariants. + +Follow the ratified Inc3 doc for the exact bounded change. diff --git a/docs/analysis/gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md b/docs/analysis/gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md new file mode 100644 index 00000000..2b89a843 --- /dev/null +++ b/docs/analysis/gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md @@ -0,0 +1,85 @@ +# GSM8K Workstream A Increment 3 — rate followup (post-#797) ratification + +**Date:** 2026-06-17 +**Workstream:** A +**Increment:** 3 — post-#797 rate frontier evidence loop closure (narrow) +**Status:** Ratified for implementation (BEFORE code changes) +**Scope lock:** Bounded to making the rate "recognized_no_injection" bucket produce actionable evidence by resolving the explicit remaining connector blocker left open in #797. One smallest change only. + +## 1. Which exact refusal bucket is being attacked? + +From frontier report run on the (stale but authoritative post-#797) committed proxy `evals/gsm8k_math/train_sample/v1/report.json`: + +- Overall: 6 correct / 44 refused / 0 wrong (passed=false) +- recognized_no_injection: 32 +- recognized_no_injection_by_category (rate relevant): rate_with_currency: 3 + +The three rate_with_currency cases still emitting the "recognizer matched but produced no injection" (category=rate_with_currency) are exactly the ones referencing the surfaces left partially unhandled after Inc2: +- 'Tina makes $18.00 an hour.' (category=rate_with_currency) +- 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.' (category=rate_with_currency) +- 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.' (category=rate_with_currency) + +Post-#797, the matcher fires for all three and "an"/"per" surfaces now reach the injector and emit a CandidateOperation (verified by live debug on current main). The "for one cup" explicitly sets `rate_anchor_token: None` (see matcher comment and spec unresolved_notes: "Non-canonical 'for one X' framing"). + +The injector returns () for the "one" case (and any elimination downstream for the others surfaces as the same top-level refusal reason because the statement-level inject did not contribute an admitted choice). + +This is the narrow remaining rate-family blocker visible in the rate bucket of the frontier analyzer. + +## 2. Which cases are expected to lift? + +- On the train_sample proxy: expected 0 net lift in correct count (the Alexa "for one cup" case uses inverse semantics — target cups from known revenue, not forward apply_rate on a held cup count; Tina/Erica denom qty statements use verbs/shapes that do not yet emit the required Initial for "hour"/"kg" unit). The change makes injection succeed for the "one" framing; the case will surface a downstream refusal reason ("no branch produced a solvable graph", "no admissible...", or "requires ... state") instead of the "no injection" one. +- The rate_with_currency slice of recognized_no_injection is expected to drop from 3 (at least the connector case will no longer refuse at the injector boundary). +- No change to non-rate buckets. Wrong remains 0. + +The primary deliverable is **actionable evidence**: after the change the frontier report will show the rate category either empty or reclassified to the true next blocker (denom state reachability), closing the post-#797 measurement loop without claiming a correct-count jump. + +## 3. Which confusers must still refuse? + +All existing confusers from the Inc2 ratification and test suite: +- No denom state for the actor (e.g. isolated rate sentence). +- Wrong actor (rate stated for A, quantity held by B). +- Multiple rates in one sentence. +- Time-unit without conversion (days vs hours). +- Any surface that would produce ambiguous or ungrounded Rate / actor / verb. + +The change adds "one" only in the exact "for " rate framing already present in the ratified rate_with_currency exemplars; no broadening of actor binding, no pronoun support, no new verbs outside the rate anchor list. + +## 4. What is the wrong=0 guard? + +- All paths still go through the existing five-layer net (matcher narrowness, source grounding in anchors, injector returns () on any construction failure, roundtrip_admissible + constraint propagation elimination, candidate-graph multi-branch disagreement + completeness). +- New surfaces exercise the same `CandidateOperation` + `roundtrip_admissible` + `KIND_TO_VERBS["apply_rate"]` checks. +- "one" treated as a surface alias only for the already-ratified "for one X" exemplar in the rate proposal; added to RATE_ANCHORS and injector allow-list with no other semantic change. +- No sealed path touched (train_sample runner + serving use sealed=False). +- Pre/post change: run the frontier script + `parse_and_solve` on the three rate surfaces + full proxy cases; assert wrong==0 on all. +- The `tests/test_math_candidate_graph_rate_injection.py` and `test_gsm8k_frontier_report.py` continue to pass (the existing test already tolerates non-"no injection" refusals for the Alexa stmt). +- If after change any train_sample case flips from refused to wrong, revert. + +## 5. Does this touch serving, sealed lanes, report.json, or solver semantics? + +- No changes to sealed injector lane (`_SEALED_INJECTORS` remains empty for this). +- No write of updated report.json in this increment (proxy remains at 6/44/0 unless a later runner run is separately committed; the ratification does not require rebaseline). +- No solver changes (`_apply_rate` unchanged; still requires denom state). +- No graph construction or cartesian changes. +- Touches only: the rate anchor token allow-list (matcher + roundtrip set + injector guard) + comments. This is the minimal patch to retire the explicit "narrow for Inc 2" deferral. +- The frontier report script, ratification, and roadmap update are docs/evidence only. + +## 6. What is explicitly out of scope? + +- Denominator-state production (seeding Initials for "hour", "kg", "cup" from "works N hours", "trawled 80 kg", etc.). That is future work once the connector surface is closed and the frontier reclassifies the bucket. +- Any change that would allow apply_rate without prior denom state. +- "for one cup" solving (would require inverse/division op or goal-residual style for price-per). +- Expansion to other temporal_aggregation, currency_amount, or non-rate categories. +- Comparative injection / Gate A1. +- Any movement of sealed SHAs, practice lane, or CLAIMS. +- Broad verb or subject binding relaxations. +- Re-running the train_sample runner and committing a new report.json as part of this PR (measurement-only refresh is out; the script run on the committed report is the evidence). + +## Implementation Notes (for the PR) + +- Smallest diff: 3 locations (RATE_ANCHORS, matcher "one" case, injector allow-list) + doc updates. +- Update comments that say "narrow for Inc 2" or "deferred". +- Run `uv run python scripts/gsm8k_frontier_report.py evals/gsm8k_math/train_sample/v1/report.json` before/after for the artifact. +- `core test --suite gsm8k` or equivalent lane (pytest on the rate graph test + frontier test) + full `core test --suite full -q` before merge when practical. +- Preserve `passed=false` on the proxy. + +This Inc3 closes the rate follow-up loop narrowly so that Ladder A has a clean evidence boundary before any comparative work. diff --git a/generate/math_roundtrip.py b/generate/math_roundtrip.py index 10c97968..b50a7c28 100644 --- a/generate/math_roundtrip.py +++ b/generate/math_roundtrip.py @@ -150,7 +150,7 @@ COMPARE_MULTIPLICATIVE_ANCHORS: Final[frozenset[str]] = frozenset({ # succeeds. "a"/"an" were documented in the comment but missing from the # set; added here (Inc 2) with corresponding injector tests. RATE_ANCHORS: Final[frozenset[str]] = frozenset({ - "per", "each", "every", "a", "an", + "per", "each", "every", "a", "an", "one", }) diff --git a/generate/recognizer_anchor_inject.py b/generate/recognizer_anchor_inject.py index 7f8aa154..d25a165a 100644 --- a/generate/recognizer_anchor_inject.py +++ b/generate/recognizer_anchor_inject.py @@ -591,7 +591,7 @@ def _locate_rate_verb(sentence: str) -> str | None: apply_rate. The literal form is required so CandidateOperation post-init + roundtrip_admissible grounding checks pass. """ - rate_verbs = ("per", "each", "every", "a", "an") + rate_verbs = ("per", "each", "every", "a", "an", "one") for raw in sentence.split(): tok = raw.strip(".,;:!?\"'()[]{}").lower() if tok in rate_verbs: @@ -670,11 +670,12 @@ def inject_rate_with_currency( # No whole-sentence fallback is allowed, because _locate_rate_verb # can still pick an unrelated earlier "a". rate_anchor_token = anchor.get("rate_anchor_token") - if not rate_anchor_token or rate_anchor_token not in ("per", "each", "every", "a", "an"): - # Missing or invalid connector for this rate surface (e.g. "one" - # from "for one cup", or absent token). Refuse — do not emit - # a CandidateOperation with a verb that does not belong to the - # matched rate expression. + if not rate_anchor_token or rate_anchor_token not in ( + "per", "each", "every", "a", "an", "one", + ): + # Missing or invalid connector for this rate surface (e.g. absent + # token). "one" (from "for one cup") is now supported (Inc 3). + # Refuse on anything else. return () verb_token = rate_anchor_token diff --git a/generate/recognizer_match.py b/generate/recognizer_match.py index 904aec0f..02775350 100644 --- a/generate/recognizer_match.py +++ b/generate/recognizer_match.py @@ -340,11 +340,9 @@ def _match_rate_with_currency( elif m.group(8): q = m.group(8).lower() per_unit = m.group(9) - if q in ("each", "every", "a"): + if q in ("each", "every", "a", "one"): connector = q else: - # "one" in "for one X" is not a direct RATE_ANCHORS token; - # leave None so injector will refuse (narrow for Inc 2). connector = None if not per_unit: diff --git a/tests/test_candidate_graph_recognizer_wiring.py b/tests/test_candidate_graph_recognizer_wiring.py index 43a2297b..534e21e8 100644 --- a/tests/test_candidate_graph_recognizer_wiring.py +++ b/tests/test_candidate_graph_recognizer_wiring.py @@ -29,8 +29,12 @@ from tests._phase_d_fixture import build_synthetic_registry _REPO_ROOT = Path(__file__).resolve().parent.parent -_GSM8K_CASES = _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "cases.jsonl" -_GSM8K_REPORT = _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "report.json" +_GSM8K_CASES = ( + _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "cases.jsonl" +) +_GSM8K_REPORT = ( + _REPO_ROOT / "evals" / "gsm8k_math" / "train_sample" / "v1" / "report.json" +) @pytest.fixture(scope="module") @@ -46,7 +50,9 @@ def with_synthetic_registry( """Patch ``math_candidate_graph._load_ratified_registry_or_empty`` to return the synthetic registry for the duration of the test.""" monkeypatch.setattr( - cg, "_load_ratified_registry_or_empty", lambda: synthetic_registry, + cg, + "_load_ratified_registry_or_empty", + lambda: synthetic_registry, ) return synthetic_registry @@ -89,31 +95,38 @@ def test_empty_registry_preserves_existing_refusal_reason() -> None: def test_recognized_rate_statement_refuses_explicitly_post_wrong_zero_fix( with_synthetic_registry: tuple[RatifiedRecognizer, ...], ) -> None: - """With the rate_with_currency recognizer loaded, "Tina makes $18.00 - an hour" is recognized but the v1 injector returns () (the - SentenceChoice union does not yet model rates — see ADR follow-up). + """With the rate_with_currency recognizer loaded (synthetic), rate surfaces + that now have v1 injector support ("an" from Inc2, "one" from Inc3) are + injected (CandidateOperation). The early "recognizer matched but produced + no injection" refusal no longer triggers for these supported surfaces. - Pre-#359 behavior: silently drop the recognized-but-uninjectable - statement and admit a partial graph from the rest — a wrong>0 - hazard analogous to case 0050. + The full sentence provides no denom-unit Initial for the actor, so the + candidate graph produces no admissible branch. Refusal is at question or + "no admissible candidate" level (downstream of injection). - Post-#359 (this test's contract): refuse explicitly with reason - "recognizer matched but produced no injection" naming the - statement and category. This pinned behavior is the wrong=0 - safety net for the recognizer path. + Pre-#359: silent drop (wrong>0 hazard). + Post-#359 + Inc2/Inc3: explicit diagnostic for unsupported; supported + rates proceed to state/admissibility checks (wrong=0 preserved). + This test pins the wiring for the synthetic registry path; the + explicit no-injection guard remains for categories without injector. """ result = cg.parse_and_solve( "Tina makes $18.00 an hour. How much does Tina earn after 8 hours?" ) assert result.refusal_reason is not None + # For this supported rate surface the statement is injected; refusal + # is now "no admissible candidate for question" (or similar) because + # no full admissible graph (missing denom state). The no-injection + # reason is the guard only for injector-return-() cases. assert ( - "recognizer matched but produced no injection" in result.refusal_reason - ), f"expected explicit recognizer-refusal, got: {result.refusal_reason!r}" - # The statement IS named in the reason — that's the diagnostic shape - # the post-#359 refusal carries. Update the prior assertion which - # forbade naming, since that assertion encoded the silent-drop - # premise that #359 retired. - assert "Tina makes $18.00 an hour" in result.refusal_reason + "no admissible candidate" in result.refusal_reason + or "recognizer matched but produced no injection" in result.refusal_reason + ), f"expected downstream or explicit refusal, got: {result.refusal_reason!r}" + # Keep diagnostic: the problematic rate statement context is involved. + assert ( + "Tina makes $18.00 an hour" in result.refusal_reason + or "question" in result.refusal_reason + ) def test_recognized_descriptive_statement_refuses_explicitly_post_wrong_zero_fix( @@ -152,12 +165,13 @@ def _run_gsm8k_train_sample_with_patch( """Re-run the gsm8k train_sample under the patched registry and return the {correct, wrong, refused} counts.""" monkeypatch.setattr( - cg, "_load_ratified_registry_or_empty", lambda: registry, + cg, + "_load_ratified_registry_or_empty", + lambda: registry, ) import importlib - runner_mod = importlib.import_module( - "evals.gsm8k_math.train_sample.v1.runner" - ) + + runner_mod = importlib.import_module("evals.gsm8k_math.train_sample.v1.runner") cases = runner_mod._load_cases(runner_mod._CASES_PATH) report = runner_mod.build_report(cases) return { @@ -178,7 +192,8 @@ def test_wrong_count_stays_zero_under_synthetic_registry( baseline_report = json.loads(_GSM8K_REPORT.read_text(encoding="utf-8")) baseline_counts = baseline_report["counts"] candidate_counts = _run_gsm8k_train_sample_with_patch( - monkeypatch, synthetic_registry, + monkeypatch, + synthetic_registry, ) assert candidate_counts["wrong"] == 0, ( f"Phase D wiring regressed wrong=0: {candidate_counts}" @@ -196,9 +211,12 @@ def test_capability_axis_wrong_unchanged_under_synthetic_registry( guarded by a narrow recognizer; it cannot mis-admit a well-parsed capability-axis statement.""" monkeypatch.setattr( - cg, "_load_ratified_registry_or_empty", lambda: synthetic_registry, + cg, + "_load_ratified_registry_or_empty", + lambda: synthetic_registry, ) import importlib + lanes = [ ("G1_verb_classes", "evals.math_capability_axes.G1_verb_classes.v1.runner"), ("G2_comparatives", "evals.math_capability_axes.G2_comparatives.v1.runner"), @@ -238,9 +256,15 @@ def test_per_category_admission_counts_on_gsm8k_train_sample( pin them to specific numbers, so the test stays robust to Phase B corpus updates that narrow or widen specific axes. """ - cases = [json.loads(l) for l in _GSM8K_CASES.read_text(encoding="utf-8").splitlines() if l.strip()] + cases = [ + json.loads(line) + for line in _GSM8K_CASES.read_text(encoding="utf-8").splitlines() + if line.strip() + ] report = json.loads(_GSM8K_REPORT.read_text(encoding="utf-8")) - refused_ids = {e["case_id"] for e in report["per_case"] if e["verdict"] == "refused"} + refused_ids = { + e["case_id"] for e in report["per_case"] if e["verdict"] == "refused" + } counts: dict[str, int] = { ShapeCategory.DESCRIPTIVE_SETUP_NO_QUANTITY.value: 0, @@ -262,7 +286,9 @@ def test_per_category_admission_counts_on_gsm8k_train_sample( assert counts[ShapeCategory.RATE_WITH_CURRENCY.value] >= 1 assert counts[ShapeCategory.TEMPORAL_AGGREGATION.value] >= 1 # Surface the counts to stdout for the PR body. - print(f"\nPhase D admission counts (synthetic registry vs GSM8K train_sample refused-set):") + print( + "\nPhase D admission counts (synthetic registry vs GSM8K train_sample refused-set):" + ) for k, v in counts.items(): print(f" {k}: {v}") diff --git a/tests/test_gsm8k_frontier_report.py b/tests/test_gsm8k_frontier_report.py index 1e248102..16241151 100644 --- a/tests/test_gsm8k_frontier_report.py +++ b/tests/test_gsm8k_frontier_report.py @@ -6,12 +6,12 @@ These tests pin: - rate_with_currency appears as a prominent recognized_no_injection category on the committed train-sample report (the measurement target of Inc 2). - Fully deterministic output (sorted keys, no timestamps, repeatable across runs). """ + from __future__ import annotations import json from pathlib import Path -import pytest from scripts.gsm8k_frontier_report import ( analyze_report, @@ -54,9 +54,21 @@ def test_classify_and_extract_category_logic(): # We exercise via the public analyze path with a tiny synthetic report fake = { "per_case": [ - {"case_id": "c1", "verdict": "refused", "reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Tina makes $18.00 an hour.' (category=rate_with_currency)"}, - {"case_id": "c2", "verdict": "refused", "reason": "candidate_graph: no admissible candidate for statement: 'foo'"}, - {"case_id": "c3", "verdict": "refused", "reason": "candidate_graph: no admissible candidate for question: 'bar?'"}, + { + "case_id": "c1", + "verdict": "refused", + "reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Tina makes $18.00 an hour.' (category=rate_with_currency)", + }, + { + "case_id": "c2", + "verdict": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'foo'", + }, + { + "case_id": "c3", + "verdict": "refused", + "reason": "candidate_graph: no admissible candidate for question: 'bar?'", + }, {"case_id": "c4", "verdict": "correct", "reason": "fast-path"}, {"case_id": "c5", "verdict": "refused", "reason": "some other refusal"}, ], @@ -64,6 +76,7 @@ def test_classify_and_extract_category_logic(): } # Write temp and analyze (or monkey the path; for simplicity use temp file) import tempfile + with tempfile.TemporaryDirectory() as td: rp = Path(td) / "fake_report.json" rp.write_text(json.dumps(fake), encoding="utf-8") @@ -88,13 +101,18 @@ def test_markdown_render_is_stable_and_mentions_rate(): """Markdown output is deterministic and surfaces the rate frontier for humans.""" fake = { "per_case": [ - {"case_id": "r1", "verdict": "refused", "reason": "candidate_graph: recognizer matched but produced no injection for statement: 'X' (category=rate_with_currency)"}, + { + "case_id": "r1", + "verdict": "refused", + "reason": "candidate_graph: recognizer matched but produced no injection for statement: 'X' (category=rate_with_currency)", + }, {"case_id": "c1", "verdict": "correct", "reason": ""}, ], "sample_count": 2, "exit_criterion": {"correct_min": 10, "passed": False, "wrong_max": 0}, } import tempfile + with tempfile.TemporaryDirectory() as td: rp = Path(td) / "r.json" rp.write_text(json.dumps(fake), encoding="utf-8") @@ -107,4 +125,42 @@ def test_markdown_render_is_stable_and_mentions_rate(): # No timestamps or nondet text assert "202" not in md and "T" not in md.split("\n", 5)[-1] # rough # Re-render identical - assert render_markdown(summary) == md \ No newline at end of file + assert render_markdown(summary) == md + + +def test_inc3_connector_makes_rate_no_injection_actionable(): + """Inc3 effect: supporting 'one' (and prior 'an'/'per') means rate_with_currency + surfaces no longer contribute to recognized_no_injection bucket when injector + succeeds. Use synthetic report to show the reclassification without mutating + the pinned 6/44/0 artifact. rate bucket for no_inj goes to 0 for covered cases; + refusal becomes generic (no_admissible etc).""" + # Synthetic report where the rate stmt now injects (Inc3), so no "no injection" + # for rate; instead a later generic refusal for the case. + fake = { + "per_case": [ + { + "case_id": "r1", + "verdict": "refused", + "reason": "candidate_graph: no admissible candidate for statement: 'Alexa ... for one cup'", + }, + { + "case_id": "r2", + "verdict": "refused", + "reason": "candidate_graph: recognizer matched but produced no injection for statement: 'unsupported' (category=temporal_aggregation)", + }, + ], + "sample_count": 2, + } + import tempfile + from pathlib import Path + + with tempfile.TemporaryDirectory() as td: + rp = Path(td) / "post_inc3_fake.json" + rp.write_text(json.dumps(fake), encoding="utf-8") + s = analyze_report(rp) + no_inj = s["recognized_no_injection_by_category"] + assert ( + "rate_with_currency" not in no_inj or no_inj.get("rate_with_currency", 0) == 0 + ) + assert s["counts"]["recognized_no_injection"] == 1 # only the unsupported temporal + assert s["counts"].get("no_admissible_statement", 0) == 1 diff --git a/tests/test_math_candidate_graph_rate_injection.py b/tests/test_math_candidate_graph_rate_injection.py index 93ecab9d..3204f1a7 100644 --- a/tests/test_math_candidate_graph_rate_injection.py +++ b/tests/test_math_candidate_graph_rate_injection.py @@ -8,9 +8,9 @@ If the exact "hours" denom state is not yet produced by discrete injection for t the test records the gap (per brief) and still proves the wiring when a covered denom unit is used, plus that the solver-level refusal for missing denom still works. """ + from __future__ import annotations -import pytest from generate.math_candidate_graph import parse_and_solve from generate.recognizer_registry import load_ratified_registry @@ -60,9 +60,7 @@ def test_confuser_no_denom_state_refuses(): def test_confuser_wrong_actor_refuses(): """Sam has the hours; Tina states the rate. Must not apply Sam's rate to Tina or vice-versa.""" text = ( - "Sam works 3 hours. " - "Tina makes $18.00 an hour. " - "How many dollars does Tina make?" + "Sam works 3 hours. Tina makes $18.00 an hour. How many dollars does Tina make?" ) res = _run(text) assert res.answer is None @@ -86,9 +84,7 @@ def test_confuser_multiple_rates_refuses(): def test_confuser_time_unit_without_conversion_refuses(): """3 days + per-hour rate has no conversion path in scope. Must refuse.""" text = ( - "Tina works 3 days. " - "Tina makes $18.00 an hour. " - "How many dollars does Tina make?" + "Tina works 3 days. Tina makes $18.00 an hour. How many dollars does Tina make?" ) res = _run(text) assert res.answer is None @@ -110,4 +106,32 @@ def test_injected_apply_rate_does_not_create_wrong_on_known_refused_cases(): res = parse_and_solve(stmt, sealed=False) assert res.answer is None assert res.refusal_reason is not None - assert "no injection" in (res.refusal_reason or "") or "requires" in (res.refusal_reason or "").lower() or "question" in (res.refusal_reason or "").lower() + # "one" (Inc3) now injects; refusal for isolated rate is downstream + # ("no admissible", "question", "requires state"). Loose or keeps + # coverage of both pre/post connector cases while wrong=0. + assert ( + "no injection" in (res.refusal_reason or "") + or "requires" in (res.refusal_reason or "").lower() + or "question" in (res.refusal_reason or "").lower() + or "no admissible" in (res.refusal_reason or "").lower() + ) + + # Positive unit coverage for "one" surface injection (Inc3): direct + # from matcher+injector before any graph solve. Unconditional asserts for + # the canonical Alexa "for one cup" case (no silent if-skip). + from generate.recognizer_match import match as _match + from generate.recognizer_anchor_inject import inject_from_match + + m = _match( + "Alexa has a lemonade stand where she sells lemonade for $2 for one cup.", + load_ratified_registry(), + ) + assert m is not None + assert m.category.name == "RATE_WITH_CURRENCY" + inj = inject_from_match( + m, + "Alexa has a lemonade stand where she sells lemonade for $2 for one cup.", + sealed=False, + ) + assert len(inj) == 1 + assert getattr(inj[0], "matched_verb", None) == "one" diff --git a/tests/test_recognizer_anchor_inject.py b/tests/test_recognizer_anchor_inject.py index 168e528b..168e1555 100644 --- a/tests/test_recognizer_anchor_inject.py +++ b/tests/test_recognizer_anchor_inject.py @@ -10,6 +10,7 @@ Covers the exact acceptance cases from the Workstream A Inc 2 brief: - zero amount refuses - matched_*_token values are literal substrings from the source sentence """ + from __future__ import annotations import types @@ -32,7 +33,9 @@ def _stub_recognizer(category: ShapeCategory) -> types.SimpleNamespace: return types.SimpleNamespace(shape_category=category, canonical_pattern={}) -def _make_match(anchor: dict, category: ShapeCategory = ShapeCategory.RATE_WITH_CURRENCY) -> RecognizerMatch: +def _make_match( + anchor: dict, category: ShapeCategory = ShapeCategory.RATE_WITH_CURRENCY +) -> RecognizerMatch: """Minimal RecognizerMatch for direct injector testing of the rate path.""" return RecognizerMatch( recognizer=_stub_recognizer(category), @@ -43,7 +46,13 @@ def _make_match(anchor: dict, category: ShapeCategory = ShapeCategory.RATE_WITH_ ) -def _rate_anchor(symbol: str = "$", amount: str = "2", per_unit: str = "cup", amount_kind: str = "integer", rate_anchor_token: str = "per") -> dict: +def _rate_anchor( + symbol: str = "$", + amount: str = "2", + per_unit: str = "cup", + amount_kind: str = "integer", + rate_anchor_token: str = "per", +) -> dict: return { "kind": "currency_per_unit_rate", "currency_symbol": symbol, @@ -68,14 +77,22 @@ def test_rate_per_cup_emits_apply_rate_with_grounded_tokens(): assert cand.matched_actor_token == "Tina" assert cand.matched_value_token == "2" assert cand.matched_unit_token == "dollars" - assert cand.matched_verb in {"per", "a", "an", "each", "every"} # literal surface in sentence + assert cand.matched_verb in { + "per", + "a", + "an", + "each", + "every", + } # literal surface in sentence assert roundtrip_admissible(cand) is True def test_rate_an_hour_emits_when_an_in_rate_anchors(): """$18.00 an hour is a major proxy case. With 'an' in RATE_ANCHORS the literal verb token must ground.""" - m = _make_match(_rate_anchor("$", "18.00", "hour", "decimal", rate_anchor_token="an")) + m = _make_match( + _rate_anchor("$", "18.00", "hour", "decimal", rate_anchor_token="an") + ) emitted = inject_rate_with_currency(m, "Tina makes $18.00 an hour.") assert len(emitted) == 1 cand = emitted[0] @@ -91,12 +108,13 @@ def test_unknown_actor_refuses_narrow_binding(): m = _make_match(_rate_anchor("$", "20", "kg")) # No clear ProperName subject (use lowercase common noun at head so the # ratified extract_proper_noun_subject does not bind; "fish" is not a name). - emitted = inject_rate_with_currency(m, "fish are sold for $20 per kg at the market.") + emitted = inject_rate_with_currency( + m, "fish are sold for $20 per kg at the market." + ) assert emitted == () def test_multiple_rates_in_one_sentence_refuses(): - m = _make_match(_rate_anchor("$", "18", "hour", rate_anchor_token="an")) # the anchor list would have >1 in real, but we simulate # Force two by calling the multi logic path (injector sees >1 after loop) # Simpler: construct a match with two anchors a1 = _rate_anchor("$", "18", "hour") @@ -159,6 +177,7 @@ def test_dispatch_table_routes_rate_with_currency(): emitted = inject_from_match(m, stmt, sealed=False) assert len(emitted) == 1 from generate.math_roundtrip import roundtrip_admissible + assert roundtrip_admissible(emitted[0]) is True @@ -217,6 +236,7 @@ def test_rate_anchor_token_from_matcher_not_whole_sentence_scan(): emitted = inject_from_match(m, stmt, sealed=False) assert len(emitted) == 1 from generate.math_roundtrip import roundtrip_admissible + cand = emitted[0] assert isinstance(cand, CandidateOperation) assert cand.op.kind == "apply_rate" @@ -224,15 +244,16 @@ def test_rate_anchor_token_from_matcher_not_whole_sentence_scan(): assert roundtrip_admissible(cand) is True -def test_for_one_cup_hard_confuser_emits_nothing_no_fallback_to_earlier_a(): - """Hard confuser for whole-sentence fallback removal. +def test_rate_for_one_cup_emits_apply_rate_with_matched_verb_one(): + """Positive coverage for Inc3 "for one cup" connector support (rate_with_currency). "Alexa has a lemonade stand where she sells lemonade for $2 for one cup." - The live registry will match it as RATE_WITH_CURRENCY (from exemplars). - But rate_anchor_token will be None (from "one" in "for one"), which is - not in the allowed set. With no fallback to _locate_rate_verb, the - injector MUST return (). - This proves we do not bind the unrelated "a" from "a lemonade stand". + The live registry matches as RATE_WITH_CURRENCY. + rate_anchor_token == "one" (from the "for one X" group) is now allowed. + Injector must emit exactly one CandidateOperation with matched_verb="one", + using the rate surface (not falling back to earlier "a" from "a lemonade stand"). + roundtrip_admissible must hold. This makes the rate no-injection bucket + actionable (downstream refusal for missing denom state, not injector ()). """ registry = load_ratified_registry() stmt = "Alexa has a lemonade stand where she sells lemonade for $2 for one cup." @@ -240,4 +261,14 @@ def test_for_one_cup_hard_confuser_emits_nothing_no_fallback_to_earlier_a(): assert m is not None assert m.category is ShapeCategory.RATE_WITH_CURRENCY emitted = inject_from_match(m, stmt, sealed=False) - assert emitted == () + assert len(emitted) == 1 + from generate.math_roundtrip import roundtrip_admissible + + cand = emitted[0] + assert isinstance(cand, CandidateOperation) + assert cand.op.kind == "apply_rate" + assert cand.matched_verb == "one" + assert cand.matched_actor_token == "Alexa" + assert roundtrip_admissible(cand) is True + # Explicitly no fallback to the distracting earlier "a" + assert "one" in stmt.lower() # the token came from the rate span