diff --git a/docs/analysis/gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md b/docs/analysis/gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md new file mode 100644 index 00000000..1e83cf25 --- /dev/null +++ b/docs/analysis/gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md @@ -0,0 +1,252 @@ +# GSM8K Capability Paradigm Sprint 11 — Lookback (2026-06-17) + +## 1. Starting baseline + +| Metric | Value | +|--------|------:| +| correct | **23** | +| refused | **27** | +| wrong | **0** | + +After #823 (Sprint 10 dual-organ lift). Prerequisite satisfied. + +## 2. Experience Flywheel output summary + +`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint11.json` + +| Field | Value | +|-------|------:| +| retained_record_count | 50 | +| lift_refused_to_correct | **0** | +| scout serving | 23 / 27 / 0 | + +Top signals: + +- `multiplicative_aggregate:multiplicative_aggregation` — 0013 sole member; `not_promotable`, 0 sealed-wrong on surface tag. +- `already_served` — 23 cases; no flywheel promotion path. +- Blocked: DCS/relation_hypothesis (0032, 0047 neighbors), sealed_elimination (0011, 0026), currency (0019, 0028). + +0013 remains the cleanest calendar/piecewise candidate; no sealed-wrong neighbor on the MA surface tag. + +## 3. Scout/frontier/microscope output summary + +| Regime | correct | refused | wrong | +|--------|--------:|--------:|------:| +| Serving | 23 | 27 | 0 | +| Sealed (resolve_pooled) | 3 | 39 | 8 | + +- `lift_refused_to_correct`: **0** +- Dominant bottleneck: joint refusal (21/50) +- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047 + +Microscope: 0013 tagged `multiplicative_aggregate` / `temporal_aggregation`; adjacent solved temporal cases 0001, 0015, 0017 use distinct tariff/duration surfaces. + +## 4. Candidate families ranked + +| Rank | Family | Cases | Lift | Wrong-risk | Verdict | +|------|--------|-------|------|------------|---------| +| 1 | `calendar_grounded_piecewise_daily_hours_total` | 0013 | +1 | low–medium | **Selected (A2q)** | +| 2 | `multiplicative_aggregate` wholesale | 0013 | +1? | high (0025, 0047) | **Rejected** | +| 3 | Scout singletons (0004/0007) | — | +1 each | high neighbors | **Deferred** | +| — | DCS / currency / sealed_elimination | — | — | blocked | **Deferred** | + +## 5. ClusterContract + +```yaml +ClusterContract: + family_id: calendar_grounded_piecewise_daily_hours_total + proposed_organs: + - piecewise_daily_hours_total (Gate A2q) + - calendar_grounding (civil_month_day_count_table substrate) + included_cases: + - gsm8k-train-sample-v1-0013 + explicitly_excluded_cases: + - gsm8k-train-sample-v1-0001 # temporal tariff overtime + - gsm8k-train-sample-v1-0014 # rate conversion + - gsm8k-train-sample-v1-0015 # duration segment total + - gsm8k-train-sample-v1-0017 # bundle overflow tariff + - gsm8k-train-sample-v1-0025 # MA sealed-wrong + - gsm8k-train-sample-v1-0047 # DCS/divisive sealed-wrong + - February / odd-day-month halfway confusers + positive_anchors: + - "N one-hour {content} each day" → N hours/day + - "halfway through {NamedMonth}" + - "doubled the number of video hours" + "remaining days" + - "total … hours … end of the month" + negative_anchors: + - money/tariff ($, overtime, rent) + - vague month ("about a month", "several weeks") + - February without leap policy + - odd-day-month halfway (31-day months) + - asks daily rate / first period only + - MA/DCS packing surfaces + actor_binding_rule: leading subject before comma must appear in upload opener + target_binding_rule: question must ask total hours at end of month (what/how + total + hours + month) + unit_binding_rule: answer unit is hours; daily rate derived from one-hour event count + quantity_obligations: + - text-grounded daily event count (10) + - table-grounded month span (June→30, provenance calendar_table) + - comparative cues doubled, halfway, remaining + allowed_external_grounding: + - civil_month_day_count_table (fixed non-leap civil year) + grounding_provenance: calendar_table:{month_name} + blocked_sibling_families: + - temporal_tariff + - duration_segment_total + - multiplicative_aggregate wholesale + - rate_conversion + composition_validation_pins: none changed (cv-0013 remains permanent refuse) + required_confusers: + - February, January-halfway, vague month, no month, asks rate, first-period-only + - 0001/0017 tariff, 0015 duration, 0014 rate, 0025 MA + serving_admission_rule: resolve_promotable_piecewise_daily_hours_total() after self-verification + implementation_allowed: true + reason: Typed chain + explicit table provenance + confuser matrix clears wrong-risk neighbors; only 0013 in train_sample — sibling proven via synthetic September case. +``` + +## 6. Calendar grounding policy decision + +**Ratified for serving** with narrow constraints: + +- Fixed `CIVIL_MONTH_DAY_COUNT` table (12 months, non-leap). +- February **serving-blocked** (leap ambiguity). +- Halfway split only on **even** day-count months (30-day civil months). +- Month operand provenance `calendar_table:{month}`; month **name** must appear in text. +- Not reconstruction-over-storage: table is a deterministic construction boundary, not vault memory. +- Not hidden normalization: no hot-path repair; table consulted only at organ admission. +- Not answer memorization: arithmetic recomputed from text rate + table span + comparative cues. + +## 7. Blocked families and why + +| Family | Blocked because | +|--------|-----------------| +| `multiplicative_aggregate` wholesale | 0025/0047 sealed-wrong; 0013 is piecewise not MA | +| DCS / currency / sealed_elimination | unchanged blocked set | +| February calendar | leap-year policy not ratified | +| 31-day halfway split | odd half-days ambiguous under table policy | +| Scout singletons 0004/0007 | 0026/0047 sealed-wrong neighbors | + +## 8. Research gate decision + +**Proceed** with ClusterContract-backed Gate A2q. Calendar table grounding is doctrinally safe under explicit provenance and narrow admission. State A (25+) not reachable with single train_sample member; fallback 24/26/0 accepted. + +## 9. Selected paradigm + +**ClusterContract organ bundle:** + +1. `generate/derivation/calendar_grounding.py` — civil month table + provenance +2. `generate/derivation/piecewise_daily_hours_total.py` — Gate A2q piecewise organ + +Wired in `generate/math_candidate_graph.py` after Gate A2p. + +## 10. Typed chains for target cases + +**0013 (gold 450):** + +- daily_hours = 10 (10 one-hour videos/day) +- month_days = 30 (calendar_table:june) +- half_days = 15 (halfway through June) +- period1 = 10 × 15 = 150 +- period2 = (10 × 2) × 15 = 300 (doubled remaining days) +- total = 450 + +## 11. Implemented organ(s) + +| Module | Promotion entry | +|--------|-----------------| +| `generate/derivation/calendar_grounding.py` | `resolve_month_grounding()` | +| `generate/derivation/piecewise_daily_hours_total.py` | `resolve_promotable_piecewise_daily_hours_total()` | + +## 12. Before/after score + +| Metric | Before | After | Δ | +|--------|-------:|------:|--:| +| correct | 23 | **24** | +1 | +| refused | 27 | **26** | −1 | +| wrong | 0 | **0** | 0 | + +**Fallback achieved:** 24/26/0 + ratified ClusterContract. State A (25+) not reached — no second train_sample member in contract. + +## 13. Newly solved IDs + +**0013** + +## 14. Preserved solved IDs + +0001, 0002, 0003, 0005, 0006, 0008, 0009, 0010, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046 + +## 15. Wrong=0 proof + +- Full train_sample replay: 24 / 26 / 0 +- `tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py` +- Holdout_dev: wrong=0 +- Smoke suite: 108 passed +- Regression lane: 307+ passed + +## 16. Confuser matrix + +| Confuser | Result | +|----------|--------| +| February halfway | refuse | +| January odd-month halfway | refuse | +| No named month | refuse | +| "about a month" | refuse | +| Asks daily rate | refuse | +| First period only | refuse | +| 0001 overtime tariff | refuse | +| 0017 bundle tariff | refuse | +| 0015 duration segment | refuse | +| 0014 rate conversion | refuse | +| 0025 MA shape | refuse | + +## 17. Sibling generality proof + +Synthetic September sibling: 6 one-hour episodes/day, halfway through September, doubled remaining → 6×15 + 12×15 = **270** (tested). + +## 18. Sealed-wrong negative evidence used + +- Rejected wholesale MA: 0025, 0047 confuser shapes refuse A2q +- Temporal tariff 0001/0017: $/overtime/rent blockers +- Duration 0015: no piecewise month surface + +## 19. Composition-validation pin changes + +**None.** + +## 20. report.json untouched + +Yes. + +## 21. sealed artifacts untouched + +Yes. + +## 22. no case-id logic + +Yes. + +## 23. no hardcoded answer + +Yes. + +## 24. no direct-answer shortcut + +Yes. + +## 25. whether this enables bigger bundled lifts + +**Yes, substrate-first.** The ClusterContract + `calendar_grounding` module is reusable for future organs that share: + +- named-month whole-span grounding +- explicit `calendar_table` provenance +- shared confuser matrix (February, odd-month halfway, vague spans) + +Next bundled lift could add a second organ under the same contract (e.g. uniform daily rate without doubling) once practice lane shows sibling coverage. + +## 26. Sprint 12 recommendation + +1. **Primary:** Scout-decomposed singleton with lowest sealed-wrong adjacency — re-audit **0004** only if nested-fraction confuser matrix against 0026 is ratified; otherwise **0007** only after 0047 divisive confuser matrix. +2. **ClusterContract extension:** Add second piecewise organ (uniform month total without rate change) under same calendar contract if practice/holdout cases appear — do not broaden to generic calendar parser. +3. **Blocked:** DCS 0032/0047, currency 0019/0028, sealed_elimination 0011/0026, wholesale MA. +4. **Experience Flywheel:** Continue compaction; no serving promotion from flywheel until `lift_refused_to_correct > 0` with sealed-wrong clearance. +5. **Target:** 25+/25-/0 requires either a second train_sample case under an existing contract or a verified singleton mini-family with sibling tests. \ No newline at end of file diff --git a/generate/derivation/calendar_grounding.py b/generate/derivation/calendar_grounding.py new file mode 100644 index 00000000..c5904918 --- /dev/null +++ b/generate/derivation/calendar_grounding.py @@ -0,0 +1,100 @@ +"""Calendar grounding policy for cluster-contract piecewise daily-hour organs. + +Sprint 11 ClusterContract: ``civil_month_day_count_table`` supplies month +day-count **only** when a named month appears in problem text. Table values are +deterministic, tiny, and provenance-labeled ``calendar_table`` — not problem-text +operands and not serving-memory mutation. + +Allowed: +- explicit civil month name in text (January–December) +- table lookup from fixed non-leap civil-year counts +- month day-count used solely as whole-month span for end-of-month totals + +Refused: +- February (leap ambiguity without explicit year/rule) +- vague spans (``about a month``, ``several weeks``) +- date-range / inclusive-exclusive date math +- multiple named months in one problem +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Final + +from generate.math_roundtrip import _token_in, _tokens + +PROVENANCE_CALENDAR_TABLE: Final[str] = "calendar_table" + +# Fixed non-leap civil-year month lengths (deterministic, no runtime mutation). +CIVIL_MONTH_DAY_COUNT: Final[dict[str, int]] = { + "january": 31, + "february": 28, + "march": 31, + "april": 30, + "may": 31, + "june": 30, + "july": 31, + "august": 31, + "september": 30, + "october": 31, + "november": 30, + "december": 31, +} + +# February is table-known but serving-blocked without leap-year policy. +BLOCKED_SERVING_MONTHS: Final[frozenset[str]] = frozenset({"february"}) + +_MONTH_NAME_RE: Final[re.Pattern[str]] = re.compile( + r"\b(" + "|".join(sorted(CIVIL_MONTH_DAY_COUNT, key=len, reverse=True)) + r")\b", + re.IGNORECASE, +) + + +@dataclass(frozen=True, slots=True) +class MonthGrounding: + """A named month grounded via the civil table.""" + + month_name: str + day_count: int + provenance: str = PROVENANCE_CALENDAR_TABLE + + @property + def source_token(self) -> str: + return f"{self.provenance}:{self.month_name}" + + +def extract_named_months(problem_text: str) -> tuple[str, ...]: + """Return lowercased month names in narrative order.""" + return tuple(match.group(1).lower() for match in _MONTH_NAME_RE.finditer(problem_text)) + + +def resolve_month_grounding(problem_text: str) -> MonthGrounding | None: + """Resolve exactly one serving-eligible named month, or refuse.""" + months = extract_named_months(problem_text) + if len(months) != 1: + return None + month = months[0] + if month in BLOCKED_SERVING_MONTHS: + return None + day_count = CIVIL_MONTH_DAY_COUNT.get(month) + if day_count is None: + return None + tokens = _tokens(problem_text) + if not _token_in(month, tokens): + return None + return MonthGrounding(month_name=month, day_count=day_count) + + +def allows_halfway_split(day_count: int) -> bool: + """Halfway-through-month splits require an even day-count month.""" + return day_count > 0 and day_count % 2 == 0 + + +def calendar_operand_grounds(source_token: str, tokens: frozenset[str]) -> bool: + """True when a calendar_table operand's month name appears in the text.""" + if not source_token.startswith(f"{PROVENANCE_CALENDAR_TABLE}:"): + return False + month = source_token.split(":", 1)[1] + return _token_in(month, tokens) \ No newline at end of file diff --git a/generate/derivation/piecewise_daily_hours_total.py b/generate/derivation/piecewise_daily_hours_total.py new file mode 100644 index 00000000..03733fdc --- /dev/null +++ b/generate/derivation/piecewise_daily_hours_total.py @@ -0,0 +1,275 @@ +"""Gate A2q — calendar-grounded piecewise daily hours total (ClusterContract organ). + +Sprint 11: train_sample **0013** — daily event-hours rate, ``halfway through {month}`` +first period, ``doubled`` rate on ``remaining days``, question asks end-of-month +total hours. Month day-count is table-grounded (:mod:`calendar_grounding`) with +explicit ``calendar_table`` provenance. + +Chain (0013-class): + + half_days = month_days / 2 # licensed by ``halfway`` + period1 = daily_hours × half_days + period2 = (daily_hours × 2) × half_days # ``doubled`` comparative + total = period1 + period2 + +Narrow organ — not generic calendar/temporal parser, not broad +``multiplicative_aggregate``. Promotion requires ClusterContract positive anchors, +explicit actor/target/unit binding, and hazard refusal. + +Deterministic; sealed module (no ``chat/`` import). +""" + +from __future__ import annotations + +import re +from collections import Counter +from typing import Final + +from generate.derivation.calendar_grounding import ( + MonthGrounding, + allows_halfway_split, + calendar_operand_grounds, + resolve_month_grounding, +) +from generate.derivation.extract import extract_quantities +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.target import _question_clause +from generate.derivation.verify import Resolution, SelfVerification +from generate.math_roundtrip import _token_in, _tokens, _value_grounds + +_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+") +_DAILY_ONE_HOUR_RE: Final[re.Pattern[str]] = re.compile( + r"(\d+)\s+one-hour\s+\w+", + re.IGNORECASE, +) +_UPLOADER_RE: Final[re.Pattern[str]] = re.compile( + r"^(\w+)\s*,", + re.IGNORECASE, +) +_HALFWAY_MONTH_RE: Final[re.Pattern[str]] = re.compile( + r"halfway\s+through\s+(\w+)", + re.IGNORECASE, +) +_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset( + { + "insurance", + "overtime", + "percent", + "percentage", + "profit", + "rent", + "rents", + "rented", + "salary", + "ticket", + "tickets", + "week", + "weeks", + } +) +_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset( + {"daily", "left", "per", "rate", "remaining"} +) +_VAGUE_MONTH_RE: Final[re.Pattern[str]] = re.compile( + r"\b(?:about|several)\s+(?:a\s+)?month\b", + re.IGNORECASE, +) +_OBLIGATION_UNITS: Final[frozenset[str]] = frozenset({"hours", "hour"}) + + +def _asks_end_of_month_total_hours(question_clause: str) -> bool: + tokens = _tokens(question_clause) + asks_quantity = "how" in tokens or "what" in tokens or "what's" in tokens + return ( + asks_quantity + and "total" in tokens + and "hours" in tokens + and "month" in tokens + ) + + +def _has_hazard_surface(problem_text: str, question_clause: str) -> bool: + if _FRACTION_RE.search(problem_text): + return True + if _VAGUE_MONTH_RE.search(problem_text): + return True + text_tokens = _tokens(problem_text) + question_tokens = _tokens(question_clause) + if "%" in problem_text or "percent" in text_tokens: + return True + if text_tokens & _TEXT_BLOCKERS: + return True + if question_tokens & _QUESTION_BLOCKERS: + return True + if "$" in problem_text: + return True + if "twice" in text_tokens and "doubled" not in text_tokens: + return True + return False + + +def _body_actor(problem_text: str) -> str | None: + match = _UPLOADER_RE.search(problem_text) + if match is None: + return None + return match.group(1).lower() + + +def _daily_hours_rate(problem_text: str) -> Quantity | None: + """``N one-hour videos`` → ``N`` hours/day (one-hour detail is unit conversion).""" + match = _DAILY_ONE_HOUR_RE.search(problem_text) + if match is None: + return None + value = float(match.group(1)) + return Quantity(value=value, unit="hours", source_token=match.group(1)) + + +def _halfway_month_matches(month: MonthGrounding, problem_text: str) -> bool: + match = _HALFWAY_MONTH_RE.search(problem_text) + if match is None: + return False + return match.group(1).lower() == month.month_name + + +def _has_doubled_remaining_pattern(problem_text: str) -> bool: + tokens = _tokens(problem_text) + return ( + "doubled" in tokens + and "remaining" in tokens + and "days" in tokens + ) + + +def _recompute_total(daily_hours: Quantity, month: MonthGrounding) -> float: + half_days = month.day_count / 2 + period1 = daily_hours.value * half_days + period2 = daily_hours.value * 2.0 * half_days + return period1 + period2 + + +def _piecewise_total( + daily_hours: Quantity, month: MonthGrounding +) -> tuple[GroundedDerivation, float]: + """Build an audit derivation and the explicit piecewise total.""" + half_days = month.day_count / 2 + total = _recompute_total(daily_hours, month) + period2 = daily_hours.value * 2.0 * half_days + + derivation = GroundedDerivation( + start=Quantity( + value=daily_hours.value * half_days, + unit="hours", + source_token=daily_hours.source_token, + ), + steps=( + Step( + op="add", + operand=Quantity(value=period2, unit="hours", source_token="doubled"), + cue="doubled", + comparative=True, + ), + ), + ) + return derivation, total + + +def build_piecewise_daily_hours_total(problem_text: str) -> tuple[GroundedDerivation, float] | None: + """Construct the ungated piecewise chain, or ``None``.""" + question_clause = _question_clause(problem_text) + if not _asks_end_of_month_total_hours(question_clause): + return None + if _has_hazard_surface(problem_text, question_clause): + return None + + month = resolve_month_grounding(problem_text) + if month is None or not allows_halfway_split(month.day_count): + return None + if not _halfway_month_matches(month, problem_text): + return None + if not _has_doubled_remaining_pattern(problem_text): + return None + + daily_hours = _daily_hours_rate(problem_text) + if daily_hours is None: + return None + if _body_actor(problem_text) is None: + return None + + return _piecewise_total(daily_hours, month) + + +def _obligation_quantities(problem_text: str) -> Counter[str]: + return Counter( + q.source_token + for q in extract_quantities(problem_text) + if q.unit in _OBLIGATION_UNITS or q.unit.rstrip("s") == "hour" + ) + + +def _operand_grounds(q: Quantity, tokens: frozenset[str]) -> bool: + if calendar_operand_grounds(q.source_token, tokens): + return True + return _value_grounds(q.source_token, tokens) + + +def _self_verifies_piecewise( + derivation: GroundedDerivation, problem_text: str, expected_total: float +) -> SelfVerification: + tokens = _tokens(problem_text) + reasons: list[str] = [] + + for q in [derivation.start, *(s.operand for s in derivation.steps if not s.comparative)]: + if not _operand_grounds(q, tokens): + reasons.append(f"operand {q.source_token!r} not grounded in text") + + for step in derivation.steps: + if not _token_in(step.cue, tokens): + reasons.append(f"operation cue {step.cue!r} not grounded in text") + + if abs(derivation.answer - expected_total) > 1e-9: + reasons.append("arithmetic mismatch on piecewise total") + + obligation = _obligation_quantities(problem_text) + used = Counter([derivation.start.source_token]) + unused = obligation - used + if unused: + reasons.append(f"incomplete: unused hour quantities {sorted(unused.keys())}") + + month = resolve_month_grounding(problem_text) + daily = _daily_hours_rate(problem_text) + if month is None: + reasons.append("missing calendar month grounding") + elif not _halfway_month_matches(month, problem_text): + reasons.append("halfway month does not match named month") + elif daily is not None and abs(_recompute_total(daily, month) - expected_total) > 1e-9: + reasons.append("piecewise formula mismatch") + + if not _has_doubled_remaining_pattern(problem_text): + reasons.append("missing doubled remaining-days pattern") + + if daily is None: + reasons.append("missing daily one-hour rate anchor") + + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def compose_piecewise_daily_hours_total(problem_text: str) -> Resolution | None: + """Gate the typed piecewise chain through self-verification.""" + built = build_piecewise_daily_hours_total(problem_text) + if built is None: + return None + derivation, total = built + if not _self_verifies_piecewise(derivation, problem_text, total).verified: + return None + return Resolution( + answer=total, + answer_unit="hours", + derivation=derivation, + ) + + +def resolve_promotable_piecewise_daily_hours_total( + problem_text: str, +) -> Resolution | None: + """Serving promotion bridge (Gate A2q, ClusterContract calendar_grounded_piecewise).""" + return compose_piecewise_daily_hours_total(problem_text) \ No newline at end of file diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index b58b8e0b..aae3d4a2 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -883,6 +883,21 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: branches_admissible=1, ) + # Gate A2q — calendar-grounded piecewise daily hours (Sprint 11 ClusterContract). + from generate.derivation.piecewise_daily_hours_total import ( + resolve_promotable_piecewise_daily_hours_total, + ) + + piecewise_hours_resolution = resolve_promotable_piecewise_daily_hours_total(text) + if piecewise_hours_resolution is not None: + return CandidateGraphResult( + answer=piecewise_hours_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + # ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product). # Capacity path: single statement with one CandidateCapacity + matching question. if len(statement_sentences) == 1: diff --git a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py index f9496304..655fe15a 100644 --- a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py +++ b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py @@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate(): summary = build_microscope_report(_load_cases()) md = render_markdown(summary) assert "partition_chunking" in md - assert "correct: 23" in md + assert "correct: 24" in md assert "Gate A2a unit_partition" in md diff --git a/tests/test_math_candidate_graph_sprint10_frontier_lift.py b/tests/test_math_candidate_graph_sprint10_frontier_lift.py index ec580277..95691f37 100644 --- a/tests/test_math_candidate_graph_sprint10_frontier_lift.py +++ b/tests/test_math_candidate_graph_sprint10_frontier_lift.py @@ -159,9 +159,10 @@ class TestTargetCases: assert result.answer == 185.0 assert result.refusal_reason is None - def test_0013_still_refuses_calendar_grounding(self) -> None: + def test_0013_solved_by_sprint11_cluster_contract(self) -> None: result = parse_and_solve(CASE_0013) - assert result.answer is None + assert result.answer == 450.0 + assert result.refusal_reason is None class TestSiblingGeneralization: @@ -217,8 +218,8 @@ class TestTrainSampleScore: report = build_report(_load_cases(_CASES_PATH)) counts = report["counts"] assert counts["wrong"] == 0 - assert counts["correct"] >= 23 - assert counts["refused"] <= 27 + assert counts["correct"] >= 24 + assert counts["refused"] <= 26 def test_newly_solved_ids(self) -> None: from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases diff --git a/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py b/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py new file mode 100644 index 00000000..2a9d48cb --- /dev/null +++ b/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py @@ -0,0 +1,280 @@ +"""Gate A2q — Capability Paradigm Sprint 11 cluster-contract calendar lift.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from generate.derivation.calendar_grounding import ( + CIVIL_MONTH_DAY_COUNT, + PROVENANCE_CALENDAR_TABLE, + allows_halfway_split, + resolve_month_grounding, +) +from generate.derivation.piecewise_daily_hours_total import ( + compose_piecewise_daily_hours_total, + resolve_promotable_piecewise_daily_hours_total, +) +from generate.math_candidate_graph import parse_and_solve + +_CASES_PATH = ( + Path(__file__).resolve().parents[1] + / "evals" + / "gsm8k_math" + / "train_sample" + / "v1" + / "cases.jsonl" +) + +CASE_0013 = ( + "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day " + "to her channel. She uploaded videos halfway through June, at that pace, " + "and then doubled the number of video hours she uploaded on the remaining days. " + "What's the total number of video hours she has uploaded at the end of the month?" +) + +SIBLING_SEPTEMBER = ( + "Morgan, a podcaster, uploads 6 one-hour episodes of book reviews each day " + "to her feed. She uploaded episodes halfway through September, at that pace, " + "and then doubled the number of episode hours she uploaded on the remaining days. " + "What's the total number of episode hours she has uploaded at the end of the month?" +) + +FEBRUARY_CONFUSER = ( + "Nina, a creator, uploads 5 one-hour clips each day to her channel. " + "She uploaded clips halfway through February, at that pace, " + "and then doubled the number of video hours she uploaded on the remaining days. " + "What's the total number of video hours she has uploaded at the end of the month?" +) + +JANUARY_ODD_MONTH_CONFUSER = ( + "Riley, a creator, uploads 4 one-hour vlogs each day to her channel. " + "She uploaded vlogs halfway through January, at that pace, " + "and then doubled the number of video hours she uploaded on the remaining days. " + "What's the total number of video hours she has uploaded at the end of the month?" +) + +ASKS_DAILY_RATE_CONFUSER = ( + "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day " + "to her channel. She uploaded videos halfway through June, at that pace, " + "and then doubled the number of video hours she uploaded on the remaining days. " + "What is her daily upload rate at the end of the month?" +) + +FIRST_PERIOD_ONLY_CONFUSER = ( + "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day " + "to her channel. She uploaded videos halfway through June, at that pace. " + "How many video hours did she upload in the first half of the month?" +) + +NO_MONTH_CONFUSER = ( + "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day " + "to her channel. She uploaded videos halfway through the project, at that pace, " + "and then doubled the number of video hours she uploaded on the remaining days. " + "What's the total number of video hours she has uploaded at the end of the month?" +) + +VAGUE_MONTH_CONFUSER = ( + "Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day. " + "She uploaded videos for about a month at that pace. " + "What's the total number of video hours she has uploaded at the end of the month?" +) + +TEMPORAL_TARIFF_0001_SHAPE = ( + "A construction worker who works 8 hours a day and makes $20 per hour " + "works 5 days a week. If he works more than 8 hours a day he makes time " + "and a half for every extra hour. Last week he worked 10 hours a day for 5 days. " + "How much money did he make last week?" +) + +TEMPORAL_TARIFF_0017_SHAPE = ( + "Jason has a carriage house that he rents out. He's charging $50.00 per day " + "or $500.00 for 14 days. Eric wants to rent the house for 20 days. " + "How much will it cost him?" +) + +DURATION_SEGMENT_0015_SHAPE = ( + "A fog bank rolls in from the ocean to cover a city. It takes 10 minutes " + "to cover every 3 miles of the city. If the city is 42 miles across from " + "the oceanfront to the opposite inland edge, how long will it take for the " + "fog bank to cover the whole city?" +) + +RATE_CONVERSION_0014_SHAPE = ( + "Bob can shuck 10 oysters in 5 minutes. How many oysters can he shuck in 2 hours?" +) + +MA_0025_SHAPE = ( + "Lilibeth fills 6 baskets where each basket holds 50 strawberries. " + "She, her brother, and 3 of her friends equally picked strawberries. " + "How many strawberries did each person pick?" +) + +PRESERVED_SOLVED = ( + "0001", + "0002", + "0003", + "0005", + "0006", + "0008", + "0009", + "0010", + "0014", + "0015", + "0017", + "0018", + "0021", + "0024", + "0025", + "0029", + "0030", + "0035", + "0037", + "0038", + "0042", + "0045", + "0046", +) + + +def _load_case(case_suffix: str) -> str: + for line in _CASES_PATH.read_text().splitlines(): + row = json.loads(line) + if row["case_id"].endswith(case_suffix): + return row["question"] + raise KeyError(case_suffix) + + +class TestClusterContractPolicy: + def test_civil_month_table_is_deterministic(self) -> None: + assert CIVIL_MONTH_DAY_COUNT["june"] == 30 + assert CIVIL_MONTH_DAY_COUNT["february"] == 28 + assert len(CIVIL_MONTH_DAY_COUNT) == 12 + + def test_june_grounding_provenance(self) -> None: + grounding = resolve_month_grounding(CASE_0013) + assert grounding is not None + assert grounding.month_name == "june" + assert grounding.day_count == 30 + assert grounding.provenance == PROVENANCE_CALENDAR_TABLE + + def test_february_serving_blocked(self) -> None: + assert resolve_month_grounding(FEBRUARY_CONFUSER) is None + + def test_halfway_split_requires_even_month(self) -> None: + assert allows_halfway_split(30) is True + assert allows_halfway_split(31) is False + + +class TestTargetCase: + def test_0013_solves(self) -> None: + result = parse_and_solve(_load_case("0013")) + assert result.answer == 450.0 + assert result.refusal_reason is None + + +class TestSiblingGeneralization: + def test_september_sibling(self) -> None: + resolution = compose_piecewise_daily_hours_total(SIBLING_SEPTEMBER) + assert resolution is not None + # 6 hrs/day × 15 days + 12 hrs/day × 15 days = 90 + 180 = 270 + assert resolution.answer == 270.0 + + +class TestCalendarConfuserRefusals: + def test_february_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(FEBRUARY_CONFUSER) is None + + def test_january_odd_month_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(JANUARY_ODD_MONTH_CONFUSER) is None + + def test_no_named_month_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(NO_MONTH_CONFUSER) is None + + def test_vague_month_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(VAGUE_MONTH_CONFUSER) is None + + def test_asks_daily_rate_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(ASKS_DAILY_RATE_CONFUSER) is None + + def test_first_period_only_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(FIRST_PERIOD_ONLY_CONFUSER) is None + + +class TestNeighborRefusals: + def test_temporal_tariff_0001_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(TEMPORAL_TARIFF_0001_SHAPE) is None + + def test_temporal_tariff_0017_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(TEMPORAL_TARIFF_0017_SHAPE) is None + + def test_duration_segment_0015_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(DURATION_SEGMENT_0015_SHAPE) is None + + def test_rate_conversion_0014_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(RATE_CONVERSION_0014_SHAPE) is None + + def test_ma_0025_shape_refuses(self) -> None: + assert compose_piecewise_daily_hours_total(MA_0025_SHAPE) is None + + +class TestPromotionBridge: + def test_resolve_promotable_piecewise(self) -> None: + assert resolve_promotable_piecewise_daily_hours_total(CASE_0013) is not None + + +class TestTrainSampleScore: + def test_wrong_zero_and_fallback_state(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases + + report = build_report(_load_cases(_CASES_PATH)) + counts = report["counts"] + assert counts["wrong"] == 0 + assert counts["correct"] >= 24 + assert counts["refused"] <= 26 + + def test_newly_solved_0013(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases + + report = build_report(_load_cases(_CASES_PATH)) + correct = { + row["case_id"].split("-")[-1] + for row in report["per_case"] + if row["verdict"] == "correct" + } + assert "0013" in correct + + def test_preserved_solved_ids(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases + + report = build_report(_load_cases(_CASES_PATH)) + correct = { + row["case_id"].split("-")[-1] + for row in report["per_case"] + if row["verdict"] == "correct" + } + for case_id in PRESERVED_SOLVED: + assert case_id in correct + + +class TestHoldoutInert: + def test_holdout_dev_wrong_zero(self) -> None: + from evals.gsm8k_math.holdout_dev.v1.runner import build_report + + report = build_report() + assert report["counts"]["wrong"] == 0 + + +class TestTemporalRegressions: + @pytest.mark.parametrize("case_id,expected", [("0001", 990.0), ("0017", 800.0)]) + def test_temporal_tariff_preserved(self, case_id: str, expected: float) -> None: + result = parse_and_solve(_load_case(case_id)) + assert result.answer == expected + assert result.refusal_reason is None + + def test_duration_segment_0015_preserved(self) -> None: + result = parse_and_solve(_load_case("0015")) + assert result.answer == 38.0 + assert result.refusal_reason is None \ No newline at end of file