diff --git a/docs/analysis/gsm8k-capability-paradigm-sprint8-lookback-2026-06-17.md b/docs/analysis/gsm8k-capability-paradigm-sprint8-lookback-2026-06-17.md new file mode 100644 index 00000000..5d58d851 --- /dev/null +++ b/docs/analysis/gsm8k-capability-paradigm-sprint8-lookback-2026-06-17.md @@ -0,0 +1,177 @@ +# GSM8K Capability Paradigm Sprint 8 — Lookback (2026-06-17) + +## 1. Starting baseline + +| Metric | Value | +|--------|------:| +| correct | **16** | +| refused | **34** | +| wrong | **0** | + +After #818 (Sprint 7: round_trip_trip_duration + giveaway_target_residual). Preserved solved: 0002, 0003, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045. + +## 2. Experience Flywheel output summary + +`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint8.json` + +| Field | Value | +|-------|------:| +| retained_record_count | 50 | +| lift_refused_to_correct | **0** | +| promotion_candidates | 4 families, **all** `blocked_by_wrong_risk` | + +Top missing primitives: `relation_hypothesis` (DCS), `multiplicative_aggregate`, `temporal_tariff`, `diagnostic_hold`. + +**Key signal:** No scout refused→correct delta (prior sprints already promoted clean lifts). Composition-validation pins **cv-0007** (0005 fraction-decrease) and **cv-0008** (0046 percent-partition) provided typed R6 chains with low wrong-risk — aligned with Sprint 7 handoff. + +## 3. Scout/frontier output summary + +| Regime | correct | refused | wrong | +|--------|--------:|--------:|------:| +| Serving | 16 | 34 | 0 | +| Sealed (resolve_pooled) | 3 | 39 | 8 | + +- `lift_refused_to_correct`: **0** +- Dominant bottleneck: joint refusal (28/50) +- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047 + +Microscope slices: `affine_equation_fraction_target` (0005), `affine_equation` / percent partition (0046). + +## 4. Candidate families ranked + +| Rank | Family | Cases | Lift signal | Wrong-risk | Sprint 8 verdict | +|------|--------|-------|-------------|------------|------------------| +| 1 | R6 fraction-decrease (`cv-0007`) | **0005** | typed `base × (1−N/M)` | low | **Selected (A2k)** | +| 2 | R6 percent-partition (`cv-0008`) | **0046** | typed half-split + dual % | low | **Selected (A2l)** | +| 3 | `temporal_tariff:temporal_aggregation` | 0001, 0017 | joint only | medium | Deferred — confuser design | +| 4 | `affine_equation_fraction_delta` | 0010 | joint only | medium | Deferred — 1/4 more than affine | +| 5 | `relation_hypothesis:DCS` | 14 joint + 2 wrong | none | **blocked** (0032, 0047) | Deferred | +| 6 | `diagnostic_hold:currency_amount` | 0019, 0028 | sealed-wrong | **blocked** | Deferred | + +## 5. Blocked families and why + +| Family | Blocked because | +|--------|-----------------| +| `relation_hypothesis:DCS` | 0032, 0047 sealed-wrong share discrete-count / partition surface | +| `conservative_boundary` | 0018, 0025 sealed-wrong on served-correct cases | +| `sealed_elimination` | 0011, 0026 hard elimination | +| `diagnostic_hold:currency_amount` | 0019, 0028 sealed-wrong | + +## 6. Selected paradigm + +**Dual narrow R6 organs** (composition-validation decomposition, not generic affine parser): + +1. **Gate A2k `fraction_decrease`** — forecast ``decrease to N/M of`` + explicit current base + question asks **decrease by** delta (`cv-0007` / 0005). +2. **Gate A2l `percent_partition`** — total population + equal ``half`` / ``other half`` split + two ``N% of the `` clauses + aggregate ownership question (`cv-0008` / 0046). + +Rejected: broad product_bridge, DCS injector, generic affine equation parser, temporal_tariff wholesale promotion. + +## 7. Typed chains for target cases + +**0005 (gold 21):** `84 × (1 − 3/4) = 84 × 1/4 = 21` degrees decrease. + +**0046 (gold 15):** `100 × 0.5 = 50` per group → `50 × 0.20 = 10` girl dog owners → `+ 50 × 0.10 = 5` boy dog owners → `15` total. + +## 8. Implemented organs + +| Module | Promotion entry | +|--------|-----------------| +| `generate/derivation/fraction_decrease.py` | `resolve_promotable_fraction_decrease` | +| `generate/derivation/percent_partition.py` | `resolve_promotable_percent_partition` | + +Wired in `generate/math_candidate_graph.py` after Gate A2j, before rate short-circuits. + +## 9. Before/after score + +| Metric | Before | After | Δ | +|--------|-------:|------:|--:| +| correct | 16 | **18** | +2 | +| refused | 34 | **32** | −2 | +| wrong | 0 | **0** | 0 | + +**State A met:** 18+/32-/0. + +## 10. Newly solved IDs + +**0005**, **0046** + +## 11. Preserved solved IDs + +0002, 0003, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045 + +## 12. Wrong=0 proof + +- Full train_sample replay: 18 correct / 32 refused / 0 wrong +- `tests/test_math_candidate_graph_sprint8_r6_affine_lift.py::TestTrainSampleScore` +- Composition-validation firewall: 14 solve / 8 refuse / 0 wrong +- Holdout_dev: 0 new organ admissions +- Smoke suite: 108 passed + +## 13. Confuser matrix + +| Confuser | Organ | Result | +|----------|-------|--------| +| 0010 affine `1/4 more than` | A2k | refused | +| Final-value question (not decrease-by) | A2k | refused | +| Unequal 60/40 split (not half/half) | A2l | refused | +| 0032 percent time-decrease | A2l | refused | +| 0047 DCS equal-pack | A2l | refused | +| Goal-residual (cv-0005) | A2k | refused (goal_residual serves) | + +## 14. Sibling generality proof + +| Sibling | Gold | Organ | +|---------|-----:|-------| +| Cedar peak 2/3 decrease, 60° base | 20 | A2k | +| Club 80 members, 25%/15% half-split | 16 | A2l | + +## 15. Sealed-wrong negative evidence used + +- 0032 percent-decrease shape → A2l hazard refusal (not sequential % chain) +- 0047 equal-pack DCS → A2l hazard refusal (no half-split / no % of group) +- DCS flywheel `blocked_by_wrong_risk` → no broad partition injector + +## 16. Composition-validation pin changes + +| Pin | Before | After | +|-----|--------|-------| +| cv-0007 (0005) | refuse | **solve** (21) | +| cv-0008 (0046) | refuse | **solve** (15) | + +Aggregate snapshot: 12/10/0 → **14/8/0** + +## 17. report.json untouched + +`evals/gsm8k_math/train_sample/v1/report.json` not modified. + +## 18. Sealed artifacts untouched + +No changes to sealed practice lanes, holdout, or `resolve_pooled` wiring. + +## 19. No case-id logic + +Promotion is family-typed only; no `case_id` conditionals in serving path. + +## 20. No hardcoded answer + +Answers derived from grounded arithmetic chains; no answer literals. + +## 21. No direct-answer shortcut + +No `answer_numeric` extraction or final-answer parsing. + +## 22. Sprint 9 recommendation + +**Track A — Capability Sprint 9** (continue capability paradigm) + +| Evidence | Detail | +|----------|--------| +| Flywheel | `lift_refused_to_correct == 0`; dense `joint_sealed_no_resolution` cluster remains | +| Next targets | `temporal_tariff` (0001, 0017) after confuser matrix; `affine_equation_fraction_delta` (0010) | +| Secondary | `multiplicative_aggregate` joint cluster (0006, 0013) — only with hard verifier | +| Blocked | DCS (0032/0047), currency_amount (0019/0028), sealed_elimination (0011/0026) | +| Expected lift | +1–2 correct if temporal_tariff organ passes adversarial review | +| Modules | `generate/derivation/temporal_tariff.py` (proposed) | +| Non-goals | Flywheel PR-2, broad product_bridge, DCS injector, report.json rebaseline | + +Track B (Experience Flywheel PR-2) and Track C (Diagnostic Hardening) deferred — flywheel still shows promotable typed chains in joint-refusal cluster, not exclusively wrong-risk blockage. \ No newline at end of file diff --git a/generate/derivation/fraction_decrease.py b/generate/derivation/fraction_decrease.py new file mode 100644 index 00000000..fbc21893 --- /dev/null +++ b/generate/derivation/fraction_decrease.py @@ -0,0 +1,187 @@ +"""Gate A2k — fraction-decrease delta (decrease-to fraction of base). + +Experience Flywheel Sprint 8 / composition-validation **cv-0007** (train_sample +**0005**): forecast clause states a quantity will ``decrease to N/M of`` its +current value; a follow-on clause states the current amount; the question asks +the **decrease by** delta (not the final value). + + decrease_by = base × (1 − N/M) + +Narrow organ — not a generic affine equation parser, not ``N/M more than`` affine +surfaces (0010-class), not percent-decrease. Promotion requires: + +- exactly one ``decrease to N/M of`` forecast (no ``more than`` fraction affine); +- question binds ``decrease`` + ``by`` (delta target, not final-value question); +- exactly one explicit current/base quantity with a unit; +- hazard refusal (percent, goal language, multiple decrease-to fractions). + +Deterministic; sealed module (no ``chat/`` import). +""" + +from __future__ import annotations + +import re +from typing import Final + +from generate.derivation.clauses import segment_clauses +from generate.derivation.extract import extract_quantities +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.target import _question_clause +from generate.derivation.verify import Resolution, SelfVerification, select_self_verified +from generate.math_roundtrip import _tokens + +_DECREASE_TO_FRACTION_RE: Final[re.Pattern[str]] = re.compile( + r"decrease\s+to\s+(\d+)\s*/\s*(\d+)\s+of", + re.IGNORECASE, +) +_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+") +_GOAL_INTENT: Final[frozenset[str]] = frozenset( + {"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"} +) +_FINAL_VALUE_CUES: Final[frozenset[str]] = frozenset({"be", "will", "what"}) + + +def _asks_decrease_delta(question_clause: str) -> bool: + tokens = _tokens(question_clause) + return "decrease" in tokens and "by" in tokens + + +def _asks_final_value(question_clause: str) -> bool: + tokens = _tokens(question_clause) + if "by" in tokens: + return False + if "decrease" not in tokens: + return False + return bool(_FINAL_VALUE_CUES & tokens) or "temperature" in tokens + + +def _target_fraction(problem_text: str) -> tuple[int, int, str] | None: + matches = list(_DECREASE_TO_FRACTION_RE.finditer(problem_text)) + if len(matches) != 1: + return None + num_s, den_s = matches[0].group(1), matches[0].group(2) + try: + num = int(num_s) + den = int(den_s) + except ValueError: + return None + if num <= 0 or den <= 0 or num >= den: + return None + return num, den, f"{num_s}/{den_s}" + + +def _has_hazard_surface(problem_text: str, question_clause: str) -> bool: + text_tokens = _tokens(problem_text) + question_tokens = _tokens(question_clause) + if "more" in text_tokens and "than" in text_tokens: + return True + if "%" in problem_text or "percent" in text_tokens or "percentage" in text_tokens: + return True + if text_tokens & _GOAL_INTENT: + return True + if _asks_final_value(question_clause): + return True + fractions = _EXTRA_FRACTION_RE.findall(problem_text) + target = _target_fraction(problem_text) + if target is None: + return True + _, _, token = target + extra = [f for f in fractions if f.replace(" ", "") != token.replace(" ", "")] + return len(extra) > 0 + + +def _current_base_quantity(problem_text: str, question_clause: str) -> Quantity | None: + for clause in segment_clauses(problem_text): + if _DECREASE_TO_FRACTION_RE.search(clause): + continue + tokens = _tokens(clause) + if "current" not in tokens and "now" not in tokens: + continue + quantities = [q for q in extract_quantities(clause) if q.unit] + if len(quantities) == 1: + return quantities[0] + for clause in segment_clauses(problem_text): + if clause == question_clause: + continue + if _DECREASE_TO_FRACTION_RE.search(clause): + continue + quantities = [q for q in extract_quantities(clause) if q.unit] + if len(quantities) == 1: + return quantities[0] + return None + + +def build_fraction_decrease(problem_text: str) -> GroundedDerivation | None: + """Construct ``base × (1 − N/M)`` decrease delta, or ``None``.""" + question_clause = _question_clause(problem_text) + if not _asks_decrease_delta(question_clause): + return None + if _has_hazard_surface(problem_text, question_clause): + return None + + fraction = _target_fraction(problem_text) + base = _current_base_quantity(problem_text, question_clause) + if fraction is None or base is None: + return None + + num, den, source_token = fraction + delta_factor = 1.0 - (num / den) + if delta_factor <= 0: + return None + + return GroundedDerivation( + start=base, + steps=( + Step( + op="multiply", + operand=Quantity( + value=delta_factor, + unit="", + source_token=source_token, + ), + cue="decrease", + ), + ), + ) + + +def _self_verifies_fraction_decrease( + derivation: GroundedDerivation, problem_text: str +) -> SelfVerification: + from generate.derivation.verify import _base_reasons + + tokens = _tokens(problem_text) + reasons = list(_base_reasons(derivation, tokens)) + + fraction = _target_fraction(problem_text) + if fraction is None: + reasons.append("missing decrease-to fraction forecast") + else: + _, _, token = fraction + if token not in problem_text.replace(" ", ""): + reasons.append(f"fraction token {token!r} not grounded in text") + + base = _current_base_quantity(problem_text, _question_clause(problem_text)) + if base is None: + reasons.append("missing explicit current/base quantity") + + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def compose_fraction_decrease(problem_text: str) -> Resolution | None: + """Gate the typed fraction-decrease chain through self-verification.""" + derivation = build_fraction_decrease(problem_text) + if derivation is None: + return None + if not _self_verifies_fraction_decrease(derivation, problem_text).verified: + return None + return Resolution( + answer=derivation.answer, + answer_unit=derivation.answer_unit, + derivation=derivation, + ) + + +def resolve_promotable_fraction_decrease(problem_text: str) -> Resolution | None: + """Serving promotion bridge (Gate A2k).""" + return compose_fraction_decrease(problem_text) diff --git a/generate/derivation/percent_partition.py b/generate/derivation/percent_partition.py new file mode 100644 index 00000000..766e7186 --- /dev/null +++ b/generate/derivation/percent_partition.py @@ -0,0 +1,180 @@ +"""Gate A2l — equal half-split percent partition aggregate. + +Experience Flywheel Sprint 8 / composition-validation **cv-0008** (train_sample +**0046**): total population, equal ``half`` girls / ``other half`` boys split, +two subgroup ``N% of the `` clauses, question asks how many total own the +attribute. + + subgroup = total × half + answer = subgroup × pct_girls + subgroup × pct_boys + +Narrow organ — not a generic percent equation parser, not unequal partitions, not +multi-container DCS. Promotion requires: + +- question asks ``how many`` + aggregate ownership cue; +- total population quantity with unit; +- equal half / other-half split language; +- exactly two percent-of-subgroup clauses with distinct group referents; +- hazard refusal (fraction surfaces, money, profit, goal language). + +Deterministic; sealed module (no ``chat/`` import). +""" + +from __future__ import annotations + +import re +from typing import Final + +from generate.derivation.clauses import segment_clauses +from generate.derivation.comparatives import comparative_step, extract_comparative_scalars +from generate.derivation.extract import extract_quantities +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.target import _question_clause +from generate.derivation.verify import Resolution, SelfVerification +from generate.math_roundtrip import _tokens + +_PERCENT_OF_GROUP_RE: Final[re.Pattern[str]] = re.compile( + r"(\d+(?:\.\d+)?)\s*%\s+of\s+the\s+(\w+)", + re.IGNORECASE, +) +_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+") +_GOAL_INTENT: Final[frozenset[str]] = frozenset( + {"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"} +) +_OWNERSHIP_CUES: Final[frozenset[str]] = frozenset( + {"own", "owns", "have", "has", "had"} +) + + +def _asks_partition_total(question_clause: str) -> bool: + tokens = _tokens(question_clause) + return "how" in tokens and "many" in tokens and bool(_OWNERSHIP_CUES & tokens) + + +def _has_equal_half_split(problem_text: str) -> bool: + lowered = problem_text.lower() + return "half" in lowered and "other half" in lowered + + +def _percent_clauses(problem_text: str) -> tuple[tuple[float, str, str], tuple[float, str, str]] | None: + question_clause = _question_clause(problem_text) + matches: list[tuple[float, str, str]] = [] + for clause in segment_clauses(problem_text): + if clause == question_clause: + continue + for match in _PERCENT_OF_GROUP_RE.finditer(clause): + pct = float(match.group(1)) / 100.0 + group = match.group(2).lower() + source = match.group(1) + matches.append((pct, group, source)) + if len(matches) != 2: + return None + groups = {m[1] for m in matches} + if len(groups) != 2: + return None + return matches[0], matches[1] + + +def _total_population(problem_text: str, question_clause: str) -> Quantity | None: + for clause in segment_clauses(problem_text): + if clause == question_clause: + continue + lowered = clause.lower() + if "half" in lowered or _PERCENT_OF_GROUP_RE.search(clause): + continue + quantities = [q for q in extract_quantities(clause) if q.unit] + if len(quantities) == 1: + return quantities[0] + return None + + +def _has_hazard_surface(problem_text: str, question_clause: str) -> bool: + text_tokens = _tokens(problem_text) + if _FRACTION_RE.search(problem_text): + return True + if text_tokens & _GOAL_INTENT: + return True + if "profit" in text_tokens or "money" in text_tokens or "$" in problem_text: + return True + if not _has_equal_half_split(problem_text): + return True + comparatives = extract_comparative_scalars(problem_text) + if not any(cs.cue == "half" for cs in comparatives): + return True + return False + + +def build_percent_partition(problem_text: str) -> GroundedDerivation | None: + """Construct equal half-split percent partition total, or ``None``.""" + question_clause = _question_clause(problem_text) + if not _asks_partition_total(question_clause): + return None + if _has_hazard_surface(problem_text, question_clause): + return None + + total = _total_population(problem_text, question_clause) + percent_pair = _percent_clauses(problem_text) + if total is None or percent_pair is None: + return None + + (pct_a, _, source_a), (pct_b, _, source_b) = percent_pair + comparatives = extract_comparative_scalars(problem_text) + half = next((cs for cs in comparatives if cs.cue == "half"), None) + if half is None: + return None + + subgroup = total.value * half.scalar + contrib_a = subgroup * pct_a + contrib_b = subgroup * pct_b + + return GroundedDerivation( + start=total, + steps=( + comparative_step(half), + Step( + op="multiply", + operand=Quantity(value=pct_a, unit="", source_token=source_a), + cue="of", + ), + Step( + op="add", + operand=Quantity(value=contrib_b, unit=total.unit, source_token=source_b), + cue="and", + ), + ), + ) + + +def _self_verifies_percent_partition( + derivation: GroundedDerivation, problem_text: str +) -> SelfVerification: + from generate.derivation.verify import _base_reasons + + tokens = _tokens(problem_text) + reasons = list(_base_reasons(derivation, tokens)) + + if _percent_clauses(problem_text) is None: + reasons.append("expected two distinct percent-of-subgroup clauses") + if _total_population(problem_text, _question_clause(problem_text)) is None: + reasons.append("missing total population quantity") + + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def compose_percent_partition(problem_text: str) -> Resolution | None: + """Gate the typed percent-partition chain through self-verification.""" + derivation = build_percent_partition(problem_text) + if derivation is None: + return None + if not _self_verifies_percent_partition(derivation, problem_text).verified: + return None + return Resolution( + answer=derivation.answer, + answer_unit=derivation.answer_unit, + derivation=derivation, + ) + + +def resolve_promotable_percent_partition(problem_text: str) -> Resolution | None: + """Serving promotion bridge (Gate A2l).""" + return compose_percent_partition(problem_text) diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index 4d31bdfd..cb70c506 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -789,6 +789,36 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: branches_admissible=1, ) + # Gate A2k — fraction decrease delta (Sprint 8 R6 lift). + from generate.derivation.fraction_decrease import ( + resolve_promotable_fraction_decrease, + ) + + fraction_decrease_resolution = resolve_promotable_fraction_decrease(text) + if fraction_decrease_resolution is not None: + return CandidateGraphResult( + answer=fraction_decrease_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + + # Gate A2l — equal half-split percent partition (Sprint 8 R6 lift). + from generate.derivation.percent_partition import ( + resolve_promotable_percent_partition, + ) + + percent_partition_resolution = resolve_promotable_percent_partition(text) + if percent_partition_resolution is not None: + return CandidateGraphResult( + answer=percent_partition_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + # ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product). # Capacity path: single statement with one CandidateCapacity + matching question. if len(statement_sentences) == 1: diff --git a/tests/test_composition_validation_corpus.py b/tests/test_composition_validation_corpus.py index 96fc2a50..e9a4d2a1 100644 --- a/tests/test_composition_validation_corpus.py +++ b/tests/test_composition_validation_corpus.py @@ -168,6 +168,8 @@ def test_current_baseline_snapshot() -> None: Sprint 6 (2026-06-17): Gate A2g duration_segment_total admits cv-0022 (0015). Sprint 7 (2026-06-17): Gate A2i round_trip_trip_duration admits cv-0006 (0030); Gate A2j giveaway_target_residual admits cv-0021 (0035). + Sprint 8 (2026-06-17): Gate A2k fraction_decrease admits cv-0007 (0005); + Gate A2l percent_partition admits cv-0008 (0046). """ solve = refuse = wrong = 0 for case in _CASES: @@ -179,7 +181,7 @@ def test_current_baseline_snapshot() -> None: else: refuse += 1 assert wrong == 0 - assert (solve, refuse) == (12, 10), ( + assert (solve, refuse) == (14, 8), ( f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b " f"slice landed, update this expectation and the affected rows' " f"baseline fields in lockstep" diff --git a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py index e31a1ca3..f9f287d4 100644 --- a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py +++ b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py @@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate(): summary = build_microscope_report(_load_cases()) md = render_markdown(summary) assert "partition_chunking" in md - assert "correct: 16" in md + assert "correct: 18" in md assert "Gate A2a unit_partition" in md @@ -122,5 +122,7 @@ def test_gate_a2_lifts_are_not_in_refusal_table(): assert "gsm8k-train-sample-v1-0035" not in refused_ids assert "gsm8k-train-sample-v1-0037" not in refused_ids assert "gsm8k-train-sample-v1-0045" not in refused_ids - assert summary["counts"]["correct"] >= 16 + assert "gsm8k-train-sample-v1-0005" not in refused_ids + assert "gsm8k-train-sample-v1-0046" not in refused_ids + assert summary["counts"]["correct"] >= 18 assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0 diff --git a/tests/test_math_candidate_graph_question_bound_product_lift.py b/tests/test_math_candidate_graph_question_bound_product_lift.py index a8d46c38..cb70a827 100644 --- a/tests/test_math_candidate_graph_question_bound_product_lift.py +++ b/tests/test_math_candidate_graph_question_bound_product_lift.py @@ -163,8 +163,8 @@ def test_full_train_sample_wrong_zero_and_chunk_lift(): report = build_report(_load_cases(_CASES_PATH)) counts = report["counts"] assert counts["wrong"] == 0 - assert counts["correct"] == 16 - assert counts["refused"] == 34 + assert counts["correct"] == 18 + assert counts["refused"] == 32 by_case = {row["case_id"]: row for row in report["per_case"]} assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct" assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct" diff --git a/tests/test_math_candidate_graph_sprint6_experience_guided_lift.py b/tests/test_math_candidate_graph_sprint6_experience_guided_lift.py index 31d33b41..ff7dd273 100644 --- a/tests/test_math_candidate_graph_sprint6_experience_guided_lift.py +++ b/tests/test_math_candidate_graph_sprint6_experience_guided_lift.py @@ -173,7 +173,7 @@ class TestTrainSampleScore: wrong += 1 assert wrong == 0 - def test_aggregate_16_34_0(self): + def test_aggregate_18_32_0(self): correct = refused = wrong = 0 for case in _load_train_cases(): res = _run(case["question"]) @@ -184,8 +184,8 @@ class TestTrainSampleScore: correct += 1 else: wrong += 1 - assert correct == 16 - assert refused == 34 + assert correct == 18 + assert refused == 32 assert wrong == 0 diff --git a/tests/test_math_candidate_graph_sprint7_experience_guided_lift.py b/tests/test_math_candidate_graph_sprint7_experience_guided_lift.py index 310d0e41..89ea9462 100644 --- a/tests/test_math_candidate_graph_sprint7_experience_guided_lift.py +++ b/tests/test_math_candidate_graph_sprint7_experience_guided_lift.py @@ -185,7 +185,7 @@ class TestTrainSampleScore: wrong += 1 assert wrong == 0 - def test_aggregate_16_34_0(self): + def test_aggregate_18_32_0(self): correct = refused = wrong = 0 for case in _load_train_cases(): res = _run(case["question"]) @@ -196,8 +196,8 @@ class TestTrainSampleScore: correct += 1 else: wrong += 1 - assert correct == 16 - assert refused == 34 + assert correct == 18 + assert refused == 32 assert wrong == 0 diff --git a/tests/test_math_candidate_graph_sprint8_r6_affine_lift.py b/tests/test_math_candidate_graph_sprint8_r6_affine_lift.py new file mode 100644 index 00000000..9f724937 --- /dev/null +++ b/tests/test_math_candidate_graph_sprint8_r6_affine_lift.py @@ -0,0 +1,245 @@ +"""Gate A2k/A2l — Capability Paradigm Sprint 8 R6 affine/fraction/partition lift.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from generate.math_candidate_graph import parse_and_solve +from generate.derivation.fraction_decrease import ( + compose_fraction_decrease, + resolve_promotable_fraction_decrease, +) +from generate.derivation.percent_partition import ( + compose_percent_partition, + resolve_promotable_percent_partition, +) + +_CASES_PATH = ( + Path(__file__).resolve().parents[1] + / "evals" + / "gsm8k_math" + / "train_sample" + / "v1" + / "cases.jsonl" +) +_HOLDOUT_PATH = ( + Path(__file__).resolve().parents[1] + / "evals" + / "gsm8k_math" + / "holdout_dev" + / "v1" + / "cases.jsonl" +) + +CASE_0005 = ( + "In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. " + "If the current temperature of the mountain is 84 degrees, what will the temperature " + "decrease by?" +) + +CASE_0046 = ( + "A school has 100 students. Half of the students are girls, the other half are boys. " + "20% of the girls have dogs at home and 10% of the boys have dogs at home. " + "How many students own dogs?" +) + +SIBLING_FRACTION_DECREASE = ( + "In two hours, Cedar peak's temperature will decrease to 2/3 of its temperature. " + "If the current temperature of the peak is 60 degrees, what will the temperature " + "decrease by?" +) + +SIBLING_PERCENT_PARTITION = ( + "A club has 80 members. Half of the members are adults, the other half are teens. " + "25% of the adults have snacks and 15% of the teens have snacks. " + "How many members have snacks?" +) + +AFFINE_CONFUSER_0010 = ( + "Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what " + "Yun currently has, plus 7. How many paperclips does Marion have?" +) + +FINAL_VALUE_CONFUSER = ( + "In one hour, the lake's temperature will decrease to 3/4 of its temperature. " + "If the current temperature of the lake is 80 degrees, what will the temperature be?" +) + +UNEQUAL_SPLIT_CONFUSER = ( + "A school has 100 students. 60 of the students are girls and 40 are boys. " + "20% of the girls have dogs and 10% of the boys have dogs. " + "How many students own dogs?" +) + +SEALED_WRONG_0032_SHAPE = ( + "He draws and colors 10 pictures. He draws and colors 7 pictures, but each " + "picture takes 30% less time than the previous one." +) + +SEALED_WRONG_0047_SHAPE = ( + "John bakes 12 coconut macaroons, each weighing 5 ounces. He then packs an equal " + "number of the macaroons in 4 different brown bags, ready for delivery. When he " + "briefly leaves the kitchen to pick the phone, his little brother Steve eats the " + "entire contents of one of the brown bags. What is the total weight, in ounces, " + "of the remaining coconut macaroons?" +) + +PRESERVED_SOLVED = ( + "0002", + "0003", + "0008", + "0014", + "0015", + "0018", + "0021", + "0024", + "0025", + "0029", + "0030", + "0035", + "0037", + "0038", + "0042", + "0045", +) + + +def _run(text: str): + return parse_and_solve(text, sealed=False) + + +def _load_train_cases() -> list[dict]: + return [json.loads(line) for line in _CASES_PATH.read_text().splitlines() if line.strip()] + + +class TestTargetCases: + def test_train_sample_0005_end_to_end(self): + res = _run(CASE_0005) + assert res.answer == 21.0 + assert res.refusal_reason is None + + def test_train_sample_0046_end_to_end(self): + res = _run(CASE_0046) + assert res.answer == 15.0 + assert res.refusal_reason is None + + +class TestSiblingGeneralization: + def test_fraction_decrease_sibling(self): + res = _run(SIBLING_FRACTION_DECREASE) + assert res.answer is not None + assert abs(res.answer - 20.0) < 1e-6 + assert res.refusal_reason is None + + def test_percent_partition_sibling(self): + res = _run(SIBLING_PERCENT_PARTITION) + assert res.answer == 16.0 + assert res.refusal_reason is None + + +class TestConfuserRefusals: + def test_affine_more_than_fraction_refuses(self): + assert resolve_promotable_fraction_decrease(AFFINE_CONFUSER_0010) is None + assert _run(AFFINE_CONFUSER_0010).answer is None + + def test_final_value_question_refuses_fraction_decrease(self): + assert resolve_promotable_fraction_decrease(FINAL_VALUE_CONFUSER) is None + assert _run(FINAL_VALUE_CONFUSER).answer is None + + def test_unequal_split_refuses_percent_partition(self): + assert resolve_promotable_percent_partition(UNEQUAL_SPLIT_CONFUSER) is None + assert _run(UNEQUAL_SPLIT_CONFUSER).answer is None + + def test_goal_residual_not_fraction_decrease(self): + goal = ( + "Michael wants to lose 10 pounds by June. He lost 3 pounds in March and " + "4 pounds in April. How much weight does he have to lose in May to meet " + "his goal?" + ) + assert resolve_promotable_fraction_decrease(goal) is None + assert _run(goal).answer == 3.0 + + +class TestSealedWrongPatternRefusal: + def test_percent_time_decrease_not_partition(self): + assert resolve_promotable_percent_partition(SEALED_WRONG_0032_SHAPE) is None + assert _run(SEALED_WRONG_0032_SHAPE).answer is None + + def test_dcs_partition_not_percent_partition(self): + assert resolve_promotable_percent_partition(SEALED_WRONG_0047_SHAPE) is None + assert _run(SEALED_WRONG_0047_SHAPE).answer is None + + +class TestCompositionValidationPins: + def test_cv_0007_fraction_decrease(self): + res = _run(CASE_0005) + assert res.answer == 21.0 + + def test_cv_0008_percent_partition(self): + res = _run(CASE_0046) + assert res.answer == 15.0 + + +class TestTrainSampleScore: + def test_full_train_sample_wrong_zero(self): + wrong = 0 + for case in _load_train_cases(): + res = _run(case["question"]) + if res.answer is not None and res.answer != float(case["answer_numeric"]): + wrong += 1 + assert wrong == 0 + + def test_aggregate_18_32_0(self): + correct = refused = wrong = 0 + for case in _load_train_cases(): + res = _run(case["question"]) + gold = float(case["answer_numeric"]) + if res.answer is None: + refused += 1 + elif res.answer == gold: + correct += 1 + else: + wrong += 1 + assert correct == 18 + assert refused == 32 + assert wrong == 0 + + +class TestPriorSolvedRegression: + @pytest.mark.parametrize("case_suffix", PRESERVED_SOLVED) + def test_prior_solved_still_correct(self, case_suffix: str): + cases = {c["case_id"].split("-")[-1]: c for c in _load_train_cases()} + case = cases[case_suffix] + res = _run(case["question"]) + assert res.answer == float(case["answer_numeric"]) + + +class TestHoldoutDevSafety: + def test_holdout_dev_no_new_admissions(self): + if not _HOLDOUT_PATH.exists(): + pytest.skip("holdout_dev corpus unavailable") + admitted = 0 + for line in _HOLDOUT_PATH.read_text().splitlines(): + if not line.strip(): + continue + case = json.loads(line) + text = case["problem"] + if ( + compose_fraction_decrease(text) is not None + or compose_percent_partition(text) is not None + ): + admitted += 1 + assert admitted == 0 + + +class TestComposeAPI: + def test_fraction_decrease_compose_matches_promote(self): + assert compose_fraction_decrease(CASE_0005) is not None + assert resolve_promotable_fraction_decrease(CASE_0005) is not None + + def test_percent_partition_compose_matches_promote(self): + assert compose_percent_partition(CASE_0046) is not None + assert resolve_promotable_percent_partition(CASE_0046) is not None \ No newline at end of file diff --git a/tests/test_math_candidate_graph_sprint8_review_patch.py b/tests/test_math_candidate_graph_sprint8_review_patch.py new file mode 100644 index 00000000..cacc7593 --- /dev/null +++ b/tests/test_math_candidate_graph_sprint8_review_patch.py @@ -0,0 +1,50 @@ +"""Review-hardening tests for Sprint 8 A2k/A2l gates.""" + +from __future__ import annotations + +from generate.derivation.fraction_decrease import resolve_promotable_fraction_decrease +from generate.derivation.percent_partition import resolve_promotable_percent_partition +from generate.math_candidate_graph import parse_and_solve + + +def _run(text: str): + return parse_and_solve(text, sealed=False) + + +def test_fraction_decrease_refuses_spaced_extra_fraction_hazard(): + text = ( + "In one hour, Addison mountain's temperature will decrease to 3/4 of its " + "temperature. Last week it decreased to 1 / 2 of its temperature. If the " + "current temperature of the mountain is 84 degrees, what will the temperature " + "decrease by?" + ) + assert resolve_promotable_fraction_decrease(text) is None + assert _run(text).answer is None + + +def test_fraction_decrease_does_not_use_forecast_duration_as_base(): + text = ( + "In one hour, Addison mountain's temperature will decrease to 3/4 of its " + "temperature. What will the temperature decrease by?" + ) + assert resolve_promotable_fraction_decrease(text) is None + assert _run(text).answer is None + + +def test_percent_partition_refuses_spaced_fraction_surface(): + text = ( + "A school has 100 students. Half of the students are girls, the other half are boys. " + "20% of the girls have dogs, 10% of the boys have dogs, and 1 / 2 of the dogs are black. " + "How many students own dogs?" + ) + assert resolve_promotable_percent_partition(text) is None + assert _run(text).answer is None + + +def test_percent_partition_does_not_use_subgroup_quantity_as_total(): + text = ( + "Half of the students are girls, the other half are boys. The girls group has 50 students. " + "20% of the girls have dogs and 10% of the boys have dogs. How many students own dogs?" + ) + assert resolve_promotable_percent_partition(text) is None + assert _run(text).answer is None