diff --git a/docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md b/docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md new file mode 100644 index 00000000..2ae5aff3 --- /dev/null +++ b/docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md @@ -0,0 +1,265 @@ +# GSM8K Capability Paradigm Sprint 12 — Lookback (2026-06-17) + +## 1. Starting baseline + +| Metric | Value | +|--------|------:| +| correct | **24** | +| refused | **26** | +| wrong | **0** | + +After #824 (Sprint 11 calendar ClusterContract). Prerequisite satisfied. + +## 2. Experience Flywheel output summary + +`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint12.json` + +| Field | Value | +|-------|------:| +| retained_record_count | 50 | +| lift_refused_to_correct | **0** | +| scout serving | 24 / 26 / 0 | + +Top singleton signals: + +- **0004** — `joint_refusal`, `fraction_surface`, `not_promotable`; no sealed-wrong on surface. +- **0007** — `joint_refusal`, `unbound_target`, `not_promotable`; no sealed-wrong on surface. +- Blocked families unchanged: `sealed_elimination` (0011, 0026), `relation_hypothesis` (0032, 0047), currency. +- No second calendar sibling under Sprint 11 contract. + +## 3. Scout/frontier/microscope output summary + +| Regime | correct | refused | wrong | +|--------|--------:|--------:|------:| +| Serving | 24 | 26 | 0 | +| Sealed (resolve_pooled) | 3 | 39 | 8 | + +- `lift_refused_to_correct`: **0** +- Dominant bottleneck: joint refusal (20/50) +- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047 + +Microscope: 0004 tagged `fraction_operand`; 0007 tagged `conditional_question`. Adjacent sealed-wrong neighbors 0026 and 0047 on distinct surfaces. + +## 4. Candidate families ranked + +| Rank | Candidate | Lift | Wrong-risk neighbor | Verdict | +|------|-----------|------|---------------------|---------| +| 1 | `nested_fraction_remainder_total` (0004) | +1 | 0026 sealed_elimination | **Selected (A2r)** | +| 2 | `loose_crayon_box_capacity` (0007) | +1 | 0047 DCS/divisive | **Selected (A2s)** | +| 3 | calendar contract extension | +0 | — | **Rejected** (no second sibling) | +| — | DCS / currency / sealed_elimination wholesale | — | blocked | **Deferred** | + +0004 and 0007 rank equally on flywheel; both admitted with hard neighbor confuser matrices. + +## 5. ClusterContracts + +### nested_fraction_remainder_contract (Gate A2r) + +```yaml +ClusterContract: + family_id: nested_fraction_remainder_total + proposed_organs: + - nested_fraction_remainder_total + included_cases: + - gsm8k-train-sample-v1-0004 + explicitly_excluded_cases: + - gsm8k-train-sample-v1-0026 # sealed_elimination / currency / scoops + - gsm8k-train-sample-v1-0011 # sealed_elimination + - asks subgroup only (not root altogether) + - multiple fraction relations + - 3/4 bill-of-money surfaces + positive_anchors: + - "Half of the {population}" + - "1/4 of the {population} going to {camp} camp" + - "{camp} camp in the morning" + afternoon complement count + - question asks root {population} altogether + negative_anchors: + - $ currency / each saved / scoops / bill / ice cream + - 3/4 of total money + - asks soccer-camp subgroup only + - multiple fractions + actor_binding_rule: shared population token across half/quarter/afternoon clauses + target_binding_rule: question must ask root population altogether + unit_binding_rule: afternoon count unit matches bound population + quantity_obligations: + - text-grounded afternoon complement count + - exactly one 1/4 inner partition + allowed_external_grounding: none + blocked_sibling_families: + - sealed_elimination + - currency_amount + - percent_partition + sealed_wrong_neighbors: + - 0026 (blocked by currency/each/scoops/3/4 bill hazards) + serving_admission_rule: resolve_promotable_nested_fraction_remainder_total() + implementation_allowed: true + reason: Typed morning/afternoon complement chain + 0026 confuser matrix clears wrong-risk. +``` + +### loose_crayon_box_capacity_contract (Gate A2s) + +```yaml +ClusterContract: + family_id: loose_crayon_box_capacity + proposed_organs: + - loose_crayon_box_capacity + included_cases: + - gsm8k-train-sample-v1-0007 + explicitly_excluded_cases: + - gsm8k-train-sample-v1-0047 # equal-distribution weight / eaten bag + - asks per-box capacity (not more boxes) + - equal number in bags + - weight/ounce surfaces + positive_anchors: + - "{actor} has N full boxes of {item}" + - "M loose {item}" before friend clause + - "friend has K loose" + - conditional "total of T" in question + - "how many more boxes" + negative_anchors: + - ounces / macaroons / equal number / eats / Steve + - asks per-box not more-boxes + - DCS divisive packing + actor_binding_rule: primary actor owns full boxes + loose; friend contributes loose only + target_binding_rule: question asks additional boxes for all loose items + unit_binding_rule: answer unit is boxes + quantity_obligations: + - total, loose_primary, loose_friend, num_full_boxes all text-grounded + allowed_external_grounding: none + blocked_sibling_families: + - multiplicative_aggregate / DCS + - currency_amount + sealed_wrong_neighbors: + - 0047 (blocked by weight/equal-number/eaten-bag hazards) + serving_admission_rule: resolve_promotable_loose_crayon_box_capacity() + implementation_allowed: true + reason: Capacity derivation from boxed portion + loose aggregation; 0047 confuser matrix holds. +``` + +## 6. Blocked families and why + +| Family | Blocked because | +|--------|-----------------| +| calendar extension | No second train_sample sibling under #824 policy | +| DCS 0032/0047 | sealed-wrong; relation_hypothesis blocked | +| currency 0019/0028 | sealed-wrong | +| sealed_elimination 0011/0026 | sealed-wrong | +| wholesale multiplicative_aggregate | 0025/0047 sealed-wrong | + +## 7. Research gate decision + +**Proceed** with two independent singleton ClusterContracts (A2r + A2s). Calendar extension rejected for lack of sibling evidence. + +## 8. Selected paradigm + +Dual singleton organs under explicit contracts: + +1. `generate/derivation/nested_fraction_remainder_total.py` — Gate A2r +2. `generate/derivation/loose_crayon_box_capacity.py` — Gate A2s + +Wired in `generate/math_candidate_graph.py` after Gate A2q. + +## 9. Typed chains for target cases + +**0004 (gold 2000):** + +- afternoon = 750 kids (3/4 of soccer-camp subgroup) +- subgroup = 750 / 0.75 = 1000 +- total = 1000 × 2 = 2000 (half go to soccer camp) + +**0007 (gold 2):** + +- boxed = 85 − 5 = 80 +- per_box = 80 / 5 = 16 +- loose_total = 5 + 27 = 32 +- boxes_needed = 32 / 16 = 2 + +## 10. Implemented organ(s) + +| Module | Gate | Promotion entry | +|--------|------|-----------------| +| `nested_fraction_remainder_total.py` | A2r | `resolve_promotable_nested_fraction_remainder_total()` | +| `loose_crayon_box_capacity.py` | A2s | `resolve_promotable_loose_crayon_box_capacity()` | + +## 11. Before/after score + +| Metric | Before | After | Δ | +|--------|-------:|------:|--:| +| correct | 24 | **26** | +2 | +| refused | 26 | **24** | −2 | +| wrong | 0 | **0** | 0 | + +**State A achieved:** 26/24/0. + +## 12. Newly solved IDs + +**0004**, **0007** + +## 13. Preserved solved IDs + +0001, 0002, 0003, 0005, 0006, 0008, 0009, 0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046 + +## 14. Wrong=0 proof + +- Full train_sample replay: 26 / 24 / 0 +- `tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py` +- Holdout_dev: wrong=0 +- Regression lane: sprint6–12 + flywheel + scout + microscope + +## 15. Confuser matrix + +| Confuser | Organ | Result | +|----------|-------|--------| +| 0026 sealed_elimination | A2r | refuse | +| asks subgroup only | A2r | refuse | +| multiple fractions | A2r | refuse | +| 0047 equal-distribution weight | A2s | refuse | +| asks per-box capacity | A2s | refuse | +| equal number in bags | A2s | refuse | + +## 16. Sibling generality proof + +| Sibling | Expected | Result | +|---------|----------|--------| +| music camp / 600 afternoon | 1600 | pass | +| four marker boxes / 67 total | 6 boxes | pass | + +## 17. Sealed-wrong negative evidence used + +- 0026: currency + each + scoops + 3/4 bill → A2r hazard refusal +- 0047: ounces + equal number + eaten bag → A2s hazard refusal + +## 18. Composition-validation pin changes + +None. + +## 19. report.json untouched + +Confirmed — no changes to committed `report.json`. + +## 20. sealed artifacts untouched + +Confirmed — no sealed lane artifact movement. + +## 21. no case-id logic + +Confirmed — admission via structural anchors and bindings only. + +## 22. no hardcoded answer + +Confirmed — arithmetic recomputed from grounded quantities. + +## 23. no direct-answer shortcut + +Confirmed — self-verified derivation chains required. + +## 24. ClusterContract method enabled bigger lift + +Yes — bundling two narrow singleton contracts under explicit neighbor exclusions delivered +2 with wrong=0, exceeding single-organ sprint cadence. + +## 25. Sprint 13 recommendation + +1. **Practice-lane mini-family clustering** — flywheel joint_refusal clusters (20 cases) may yield a 2-case mini-family safer than next singleton. +2. **Deferred blocked surfaces** — DCS 0032/0047 and currency 0019/0028 remain blocked until dedicated confuser ADRs land. +3. **Calendar contract** — revisit only if practice lane surfaces a genuine second piecewise sibling without policy expansion. +4. Target: 27+/23-/0 via one mini-family or two verified singletons with lowest sealed-wrong adjacency (candidates from flywheel: 0004/0007 neighbors exhausted; next scout frontier TBD). \ No newline at end of file diff --git a/generate/derivation/loose_crayon_box_capacity.py b/generate/derivation/loose_crayon_box_capacity.py new file mode 100644 index 00000000..59478770 --- /dev/null +++ b/generate/derivation/loose_crayon_box_capacity.py @@ -0,0 +1,285 @@ +"""Gate A2s — loose crayon box capacity remainder (ClusterContract organ). + +Sprint 12: train_sample **0007** — actor has ``N full boxes`` plus loose items, +friend contributes loose items, conditional total count, derive per-box capacity +from boxed portion, question asks ``how many more boxes`` for all loose items. + +Chain (0007-class): + + boxed = total − loose_primary + per_box = boxed / num_full_boxes + loose_total = loose_primary + loose_friend + boxes_needed = loose_total / per_box + +Algebraically (single fold): + + boxes_needed = (loose_primary + loose_friend) × num_full_boxes / (total − loose_primary) + +Narrow organ — not equal-distribution/DCS (0047), not currency, not weight surfaces. +Promotion requires ClusterContract positive anchors and 0047-class hazard refusal. + +Deterministic; sealed module (no ``chat/`` import). +""" + +from __future__ import annotations + +import re +from collections import Counter +from typing import Final + +from generate.derivation.extract import extract_quantities +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.target import _question_clause +from generate.derivation.verify import Resolution, SelfVerification +from generate.math_roundtrip import WORD_NUMBERS, _token_in, _tokens, _value_grounds + +_FULL_BOXES_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)(\w+)\s+has\s+(\w+)\s+full\s+boxes\s+of\s+(\w+)" +) +_LOOSE_PRIMARY_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)(\d+)\s+loose\s+(\w+)" +) +_FRIEND_LOOSE_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)(?:\w+\s+)?friend\s+has\s+(\d+)\s+loose" +) +_TOTAL_CONDITIONAL_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)total\s+of\s+(\d+)" +) +_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset( + { + "bags", + "bakes", + "brother", + "eats", + "equal", + "macaroons", + "ounces", + "packs", + "steve", + "weighing", + "weight", + } +) +_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset( + {"ounces", "weight", "per", "macaroons"} +) + + +def _box_count_token(problem_text: str, num_boxes: float) -> str: + match = _FULL_BOXES_RE.search(problem_text) + if match is not None: + return match.group(2).lower() + return str(int(num_boxes)) + + +def _resolve_count(token: str) -> float | None: + lowered = token.lower() + if lowered in WORD_NUMBERS: + return float(WORD_NUMBERS[lowered]) + try: + return float(token) + except ValueError: + return None + + +def _asks_more_boxes(question_clause: str) -> bool: + tokens = _tokens(question_clause) + return ( + "how" in tokens + and "many" in tokens + and "more" in tokens + and "boxes" in tokens + ) + + +def _asks_per_box(question_clause: str) -> bool: + tokens = _tokens(question_clause) + return "each" in tokens or "per" in tokens + + +def _has_hazard_surface(problem_text: str, question_clause: str) -> bool: + if "$" in problem_text: + return True + text_tokens = _tokens(problem_text) + question_tokens = _tokens(question_clause) + if text_tokens & _TEXT_BLOCKERS: + return True + if question_tokens & _QUESTION_BLOCKERS: + return True + if _asks_per_box(question_clause): + return True + if "equal number" in problem_text.lower(): + return True + if not _FULL_BOXES_RE.search(problem_text): + return True + if not _FRIEND_LOOSE_RE.search(problem_text): + return True + if not _TOTAL_CONDITIONAL_RE.search(question_clause): + return True + return False + + +def _parse_bindings(problem_text: str) -> dict[str, float] | None: + boxes_match = _FULL_BOXES_RE.search(problem_text) + if boxes_match is None: + return None + num_boxes = _resolve_count(boxes_match.group(2)) + if num_boxes is None or num_boxes <= 0: + return None + + friend_match = _FRIEND_LOOSE_RE.search(problem_text) + if friend_match is None: + return None + friend_pos = friend_match.start() + + loose_primary_matches = [ + match + for match in _LOOSE_PRIMARY_RE.finditer(problem_text) + if match.start() < friend_pos + ] + if len(loose_primary_matches) != 1: + return None + loose_primary = float(loose_primary_matches[0].group(1)) + loose_friend = float(friend_match.group(1)) + + total_match = _TOTAL_CONDITIONAL_RE.search(_question_clause(problem_text)) + if total_match is None: + return None + total = float(total_match.group(1)) + + item_unit = loose_primary_matches[0].group(2).lower() + return { + "num_boxes": num_boxes, + "loose_primary": loose_primary, + "loose_friend": loose_friend, + "total": total, + "item_unit": item_unit, + } + + +def _recompute_boxes_needed(bindings: dict[str, float]) -> float: + boxed = bindings["total"] - bindings["loose_primary"] + per_box = boxed / bindings["num_boxes"] + loose_total = bindings["loose_primary"] + bindings["loose_friend"] + return loose_total / per_box + + +def build_loose_crayon_box_capacity( + problem_text: str, +) -> tuple[GroundedDerivation, float] | None: + """Construct the ungated box-capacity chain, or ``None``.""" + question_clause = _question_clause(problem_text) + if not _asks_more_boxes(question_clause): + return None + if _has_hazard_surface(problem_text, question_clause): + return None + + bindings = _parse_bindings(problem_text) + if bindings is None: + return None + + boxed_total = bindings["total"] - bindings["loose_primary"] + if boxed_total <= 0: + return None + + loose_total = bindings["loose_primary"] + bindings["loose_friend"] + answer = _recompute_boxes_needed(bindings) + + item_unit = bindings["item_unit"] + derivation = GroundedDerivation( + start=Quantity( + value=bindings["loose_primary"], + unit=item_unit, + source_token=str(int(bindings["loose_primary"])), + ), + steps=( + Step( + op="add", + operand=Quantity( + value=bindings["loose_friend"], + unit=item_unit, + source_token=str(int(bindings["loose_friend"])), + ), + cue="friend", + ), + Step( + op="multiply", + operand=Quantity( + value=bindings["num_boxes"], + unit="boxes", + source_token=_box_count_token(problem_text, bindings["num_boxes"]), + ), + cue=_box_count_token(problem_text, bindings["num_boxes"]), + ), + Step( + op="divide", + operand=Quantity( + value=boxed_total, + unit=item_unit, + source_token=str(int(bindings["total"])), + ), + cue="total", + comparative=True, + ), + ), + ) + return derivation, answer + + +def _self_verifies_box_capacity( + derivation: GroundedDerivation, problem_text: str, expected: float +) -> SelfVerification: + tokens = _tokens(problem_text) + reasons: list[str] = [] + + for q in [derivation.start, *(s.operand for s in derivation.steps if not s.comparative)]: + if not _value_grounds(q.source_token, tokens): + reasons.append(f"operand {q.source_token!r} not grounded in text") + + for step in derivation.steps: + if not _token_in(step.cue, tokens): + reasons.append(f"operation cue {step.cue!r} not grounded in text") + + bindings = _parse_bindings(problem_text) + if bindings is None: + reasons.append("missing box-capacity bindings") + elif abs(_recompute_boxes_needed(bindings) - expected) > 1e-9: + reasons.append("arithmetic mismatch on boxes needed") + + if abs(derivation.answer - expected) > 1e-9: + reasons.append("derivation fold mismatch") + + obligation = Counter(q.source_token for q in extract_quantities(problem_text)) + used = Counter( + [ + derivation.start.source_token, + *(step.operand.source_token for step in derivation.steps), + ] + ) + unused = obligation - used + if unused: + reasons.append(f"incomplete: unused quantities {sorted(unused.keys())}") + + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def compose_loose_crayon_box_capacity(problem_text: str) -> Resolution | None: + """Gate the typed box-capacity chain through self-verification.""" + built = build_loose_crayon_box_capacity(problem_text) + if built is None: + return None + derivation, answer = built + if not _self_verifies_box_capacity(derivation, problem_text, answer).verified: + return None + return Resolution( + answer=answer, + answer_unit="boxes", + derivation=derivation, + ) + + +def resolve_promotable_loose_crayon_box_capacity( + problem_text: str, +) -> Resolution | None: + """Serving promotion bridge (Gate A2s, ClusterContract loose_crayon_box).""" + return compose_loose_crayon_box_capacity(problem_text) \ No newline at end of file diff --git a/generate/derivation/nested_fraction_remainder_total.py b/generate/derivation/nested_fraction_remainder_total.py new file mode 100644 index 00000000..37f2fa08 --- /dev/null +++ b/generate/derivation/nested_fraction_remainder_total.py @@ -0,0 +1,260 @@ +"""Gate A2r — nested fraction complement remainder total (ClusterContract organ). + +Sprint 12: train_sample **0004** — outer ``Half of`` partition, inner ``1/4 of`` +subgroup split (morning vs afternoon), known afternoon complement count, question +asks root population ``altogether``. + +Chain (0004-class): + + subgroup = afternoon / (1 − inner_fraction) + total = subgroup / outer_half_scalar = subgroup × 2 + +Narrow organ — not generic fraction parser, not sealed_elimination, not currency. +Promotion requires ClusterContract positive anchors, explicit target binding, and +0026-class hazard refusal. + +Deterministic; sealed module (no ``chat/`` import). +""" + +from __future__ import annotations + +import re +from collections import Counter +from dataclasses import dataclass +from typing import Final + +from generate.derivation.extract import extract_quantities +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.target import _question_clause +from generate.derivation.verify import Resolution, SelfVerification +from generate.math_roundtrip import _token_in, _tokens, _value_grounds + +_OUTER_HALF_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)\bhalf\s+of\s+the\s+(\w+)\b" +) +_INNER_QUARTER_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)\b1/4\s+of\s+the\s+(\w+)\s+going\s+to\s+(\w+)\s+camp\b" +) +_MORNING_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)\bgoing\s+to\s+(\w+)\s+camp\s+in\s+the\s+morning\b" +) +_AFTERNOON_COUNT_RE: Final[re.Pattern[str]] = re.compile( + r"(?i)(\d+)\s+(\w+)\s+are\s+going\s+to\s+(\w+)\s+camp\s+in\s+the\s+afternoon" +) +_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+") +_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset( + { + "bill", + "dinner", + "dollars", + "each", + "ice", + "money", + "profit", + "saved", + "scoops", + } +) +_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset( + {"afternoon", "morning", "scoops"} +) + + +@dataclass(frozen=True, slots=True) +class _CampBinding: + population: str + camp: str + afternoon_count: float + afternoon_token: str + + +def _asks_root_altogether(question_clause: str, population: str) -> bool: + tokens = _tokens(question_clause) + return ( + "how" in tokens + and "many" in tokens + and population in tokens + and ("altogether" in tokens or "there" in tokens) + ) + + +def _asks_subgroup_only(question_clause: str) -> bool: + tokens = _tokens(question_clause) + return "camp" in tokens and "altogether" not in tokens + + +def _parse_camp_binding(problem_text: str) -> _CampBinding | None: + outer = _OUTER_HALF_RE.search(problem_text) + inner = _INNER_QUARTER_RE.search(problem_text) + morning = _MORNING_RE.search(problem_text) + afternoon = _AFTERNOON_COUNT_RE.search(problem_text) + if outer is None or inner is None or morning is None or afternoon is None: + return None + + population = outer.group(1).lower() + inner_pop = inner.group(1).lower() + inner_camp = inner.group(2).lower() + morning_camp = morning.group(1).lower() + afternoon_count = float(afternoon.group(1)) + afternoon_pop = afternoon.group(2).lower() + afternoon_camp = afternoon.group(3).lower() + + if not {population, inner_pop, afternoon_pop} == {population}: + return None + if not {inner_camp, morning_camp, afternoon_camp} == {inner_camp}: + return None + return _CampBinding( + population=population, + camp=inner_camp, + afternoon_count=afternoon_count, + afternoon_token=afternoon.group(1), + ) + + +def _has_hazard_surface(problem_text: str, question_clause: str) -> bool: + if "$" in problem_text: + return True + text_tokens = _tokens(problem_text) + question_tokens = _tokens(question_clause) + if text_tokens & _TEXT_BLOCKERS: + return True + if question_tokens & _QUESTION_BLOCKERS: + return True + if _asks_subgroup_only(question_clause): + return True + if "3/4" in problem_text.replace(" ", ""): + return True + if "saved" in text_tokens and "each" in text_tokens: + return True + fractions = _EXTRA_FRACTION_RE.findall(problem_text) + if len(fractions) != 1 or fractions[0].replace(" ", "") != "1/4": + return True + return False + + +def _inner_complement_scalar() -> float: + return 1.0 - (1.0 / 4.0) + + +def _outer_inverse_half_scalar() -> float: + return 2.0 + + +def _recompute_total(afternoon_count: float) -> float: + subgroup = afternoon_count / _inner_complement_scalar() + return subgroup * _outer_inverse_half_scalar() + + +def build_nested_fraction_remainder_total( + problem_text: str, +) -> tuple[GroundedDerivation, float, _CampBinding] | None: + """Construct the ungated nested-fraction chain, or ``None``.""" + binding = _parse_camp_binding(problem_text) + if binding is None: + return None + + question_clause = _question_clause(problem_text) + if not _asks_root_altogether(question_clause, binding.population): + return None + if _has_hazard_surface(problem_text, question_clause): + return None + + afternoon = Quantity( + value=binding.afternoon_count, + unit=binding.population, + source_token=binding.afternoon_token, + ) + total = _recompute_total(binding.afternoon_count) + derivation = GroundedDerivation( + start=afternoon, + steps=( + Step( + op="divide", + operand=Quantity( + value=_inner_complement_scalar(), + unit="", + source_token="1/4", + ), + cue="4", + comparative=True, + ), + Step( + op="multiply", + operand=Quantity( + value=_outer_inverse_half_scalar(), + unit="", + source_token="half", + ), + cue="half", + comparative=True, + ), + ), + ) + return derivation, total, binding + + +def _self_verifies_nested_fraction( + derivation: GroundedDerivation, + problem_text: str, + expected_total: float, + binding: _CampBinding, +) -> SelfVerification: + tokens = _tokens(problem_text) + reasons: list[str] = [] + + if not _operand_grounds(derivation.start, tokens): + reasons.append(f"operand {derivation.start.source_token!r} not grounded in text") + + for step in derivation.steps: + if not step.comparative and not _operand_grounds(step.operand, tokens): + reasons.append(f"operand {step.operand.source_token!r} not grounded in text") + if not _token_in(step.cue, tokens): + reasons.append(f"operation cue {step.cue!r} not grounded in text") + + if _parse_camp_binding(problem_text) is None: + reasons.append("missing camp partition binding") + elif abs(_recompute_total(binding.afternoon_count) - expected_total) > 1e-9: + reasons.append("arithmetic mismatch on nested fraction total") + + if abs(derivation.answer - expected_total) > 1e-9: + reasons.append("derivation fold mismatch") + + obligation = Counter( + q.source_token + for q in extract_quantities(problem_text) + if q.unit == binding.population and q.source_token.isdigit() + ) + used = Counter([derivation.start.source_token]) + unused = obligation - used + if unused: + reasons.append( + f"incomplete: unused {binding.population} quantities {sorted(unused.keys())}" + ) + + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def _operand_grounds(q: Quantity, tokens: frozenset[str]) -> bool: + return _value_grounds(q.source_token, tokens) + + +def compose_nested_fraction_remainder_total(problem_text: str) -> Resolution | None: + """Gate the typed nested-fraction chain through self-verification.""" + built = build_nested_fraction_remainder_total(problem_text) + if built is None: + return None + derivation, total, binding = built + if not _self_verifies_nested_fraction(derivation, problem_text, total, binding).verified: + return None + return Resolution( + answer=total, + answer_unit=binding.population, + derivation=derivation, + ) + + +def resolve_promotable_nested_fraction_remainder_total( + problem_text: str, +) -> Resolution | None: + """Serving promotion bridge (Gate A2r, ClusterContract nested_fraction).""" + return compose_nested_fraction_remainder_total(problem_text) \ No newline at end of file diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index aae3d4a2..6f832b43 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -898,6 +898,38 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: branches_admissible=1, ) + # Gate A2r — nested fraction complement remainder total (Sprint 12 singleton). + from generate.derivation.nested_fraction_remainder_total import ( + resolve_promotable_nested_fraction_remainder_total, + ) + + nested_fraction_resolution = resolve_promotable_nested_fraction_remainder_total( + text + ) + if nested_fraction_resolution is not None: + return CandidateGraphResult( + answer=nested_fraction_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + + # Gate A2s — loose crayon box capacity (Sprint 12 singleton). + from generate.derivation.loose_crayon_box_capacity import ( + resolve_promotable_loose_crayon_box_capacity, + ) + + box_capacity_resolution = resolve_promotable_loose_crayon_box_capacity(text) + if box_capacity_resolution is not None: + return CandidateGraphResult( + answer=box_capacity_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + # ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product). # Capacity path: single statement with one CandidateCapacity + matching question. if len(statement_sentences) == 1: diff --git a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py index 655fe15a..60eb0d10 100644 --- a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py +++ b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py @@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate(): summary = build_microscope_report(_load_cases()) md = render_markdown(summary) assert "partition_chunking" in md - assert "correct: 24" in md + assert "correct: 26" in md assert "Gate A2a unit_partition" in md diff --git a/tests/test_math_candidate_graph_sprint10_frontier_lift.py b/tests/test_math_candidate_graph_sprint10_frontier_lift.py index 95691f37..051818ad 100644 --- a/tests/test_math_candidate_graph_sprint10_frontier_lift.py +++ b/tests/test_math_candidate_graph_sprint10_frontier_lift.py @@ -218,8 +218,8 @@ class TestTrainSampleScore: report = build_report(_load_cases(_CASES_PATH)) counts = report["counts"] assert counts["wrong"] == 0 - assert counts["correct"] >= 24 - assert counts["refused"] <= 26 + assert counts["correct"] >= 26 + assert counts["refused"] <= 24 def test_newly_solved_ids(self) -> None: from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases diff --git a/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py b/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py index 2a9d48cb..323b4826 100644 --- a/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py +++ b/tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py @@ -232,8 +232,8 @@ class TestTrainSampleScore: report = build_report(_load_cases(_CASES_PATH)) counts = report["counts"] assert counts["wrong"] == 0 - assert counts["correct"] >= 24 - assert counts["refused"] <= 26 + assert counts["correct"] >= 26 + assert counts["refused"] <= 24 def test_newly_solved_0013(self) -> None: from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases diff --git a/tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py b/tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py new file mode 100644 index 00000000..c4bfe06b --- /dev/null +++ b/tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py @@ -0,0 +1,231 @@ +"""Gate A2r/A2s — Capability Paradigm Sprint 12 singleton ClusterContract lift.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from generate.math_candidate_graph import parse_and_solve +from generate.derivation.nested_fraction_remainder_total import ( + compose_nested_fraction_remainder_total, + resolve_promotable_nested_fraction_remainder_total, +) +from generate.derivation.loose_crayon_box_capacity import ( + compose_loose_crayon_box_capacity, + resolve_promotable_loose_crayon_box_capacity, +) + +_CASES_PATH = ( + Path(__file__).resolve().parents[1] + / "evals" + / "gsm8k_math" + / "train_sample" + / "v1" + / "cases.jsonl" +) + +CASE_0004 = ( + "There are some kids in camp. Half of the kids are going to soccer camp, " + "and 1/4 of the kids going to soccer camp are going to soccer camp in the morning. " + "750 kids are going to soccer camp in the afternoon. " + "How many kids there in camp altogether?" +) + +CASE_0007 = ( + "Francine has five full boxes of crayons and 5 loose crayons, and her friend " + "has 27 loose crayons. They need to put all of their loose crayons in a box. " + "How many more boxes do they need if Francine has a total of 85 crayons?" +) + +SIBLING_CAMP = ( + "There are some students in camp. Half of the students are going to music camp, " + "and 1/4 of the students going to music camp are going to music camp in the morning. " + "600 students are going to music camp in the afternoon. " + "How many students there in camp altogether?" +) + +SIBLING_BOXES = ( + "Jordan has four full boxes of markers and 3 loose markers, and her friend " + "has 21 loose markers. They need to put all of their loose markers in a box. " + "How many more boxes do they need if Jordan has a total of 19 markers?" +) + +SEALED_ELIMINATION_0026 = ( + "Aaron and his brother Carson each saved up $40 to go to dinner. " + "The bill is 3/4 of their total money. After, they go out for ice cream. " + "Each scoop costs $1.5 and they get the same amount as each other. " + "If they leave with $1 in change each, how many scoops did they each buy?" +) + +SEALED_WRONG_0047_SHAPE = ( + "John bakes 12 coconut macaroons, each weighing 5 ounces. " + "He then packs an equal number of the macaroons in 4 different brown bags, " + "ready for delivery. When he briefly leaves the kitchen to pick the phone, " + "his little brother Steve eats the entire contents of one of the brown bags. " + "What is the total weight, in ounces, of the remaining coconut macaroons?" +) + +ASKS_SUBGROUP_ONLY_CONFUSER = ( + "There are some kids in camp. Half of the kids are going to soccer camp, " + "and 1/4 of the kids going to soccer camp are going to soccer camp in the morning. " + "750 kids are going to soccer camp in the afternoon. " + "How many kids are going to soccer camp?" +) + +ASKS_PER_BOX_CONFUSER = ( + "Francine has five full boxes of crayons and 5 loose crayons, and her friend " + "has 27 loose crayons. They need to put all of their loose crayons in a box. " + "How many crayons are in each full box if Francine has a total of 85 crayons?" +) + +MULTIPLE_FRACTION_CONFUSER = ( + "Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp " + "are going in the morning. 1/2 of the afternoon group brought lunch. " + "750 kids are going to soccer camp in the afternoon. " + "How many kids there in camp altogether?" +) + +PRESERVED_SOLVED = ( + "0001", + "0002", + "0003", + "0005", + "0006", + "0008", + "0009", + "0010", + "0013", + "0014", + "0015", + "0017", + "0018", + "0021", + "0024", + "0025", + "0029", + "0030", + "0035", + "0037", + "0038", + "0042", + "0045", + "0046", +) + + +def _load_case(case_suffix: str) -> str: + for line in _CASES_PATH.read_text().splitlines(): + row = json.loads(line) + if row["case_id"].endswith(case_suffix): + return row["question"] + raise KeyError(case_suffix) + + +class TestClusterContractPolicy: + def test_nested_fraction_contract_admits_0004_shape(self) -> None: + assert compose_nested_fraction_remainder_total(CASE_0004) is not None + + def test_box_capacity_contract_admits_0007_shape(self) -> None: + assert compose_loose_crayon_box_capacity(CASE_0007) is not None + + +class TestTargetCases: + def test_0004_solves(self) -> None: + result = parse_and_solve(CASE_0004) + assert result.answer == 2000.0 + assert result.refusal_reason is None + + def test_0007_solves(self) -> None: + result = parse_and_solve(CASE_0007) + assert result.answer == 2.0 + assert result.refusal_reason is None + + +class TestSiblingGeneralization: + def test_music_camp_sibling(self) -> None: + resolution = compose_nested_fraction_remainder_total(SIBLING_CAMP) + assert resolution is not None + assert resolution.answer == 1600.0 + + def test_marker_box_sibling(self) -> None: + resolution = compose_loose_crayon_box_capacity(SIBLING_BOXES) + assert resolution is not None + # (3+21)*4/(67-3) = 96/16 = 6 + assert resolution.answer == 6.0 + + +class TestNeighborConfuserRefusals: + def test_sealed_elimination_0026_refuses(self) -> None: + assert compose_nested_fraction_remainder_total(SEALED_ELIMINATION_0026) is None + + def test_dcs_0047_refuses(self) -> None: + assert compose_loose_crayon_box_capacity(SEALED_WRONG_0047_SHAPE) is None + + def test_asks_subgroup_only_refuses(self) -> None: + assert compose_nested_fraction_remainder_total(ASKS_SUBGROUP_ONLY_CONFUSER) is None + + def test_asks_per_box_refuses(self) -> None: + assert compose_loose_crayon_box_capacity(ASKS_PER_BOX_CONFUSER) is None + + def test_multiple_fraction_refuses(self) -> None: + assert compose_nested_fraction_remainder_total(MULTIPLE_FRACTION_CONFUSER) is None + + +class TestPromotionBridges: + def test_resolve_promotable_nested_fraction(self) -> None: + assert resolve_promotable_nested_fraction_remainder_total(CASE_0004) is not None + + def test_resolve_promotable_box_capacity(self) -> None: + assert resolve_promotable_loose_crayon_box_capacity(CASE_0007) is not None + + +class TestTrainSampleScore: + def test_wrong_zero_and_state_a(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases + + report = build_report(_load_cases(_CASES_PATH)) + counts = report["counts"] + assert counts["wrong"] == 0 + assert counts["correct"] >= 26 + assert counts["refused"] <= 24 + + def test_newly_solved_ids(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases + + report = build_report(_load_cases(_CASES_PATH)) + correct = { + row["case_id"].split("-")[-1] + for row in report["per_case"] + if row["verdict"] == "correct" + } + assert "0004" in correct + assert "0007" in correct + + def test_preserved_solved_ids(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases + + report = build_report(_load_cases(_CASES_PATH)) + correct = { + row["case_id"].split("-")[-1] + for row in report["per_case"] + if row["verdict"] == "correct" + } + for case_id in PRESERVED_SOLVED: + assert case_id in correct + + +class TestHoldoutInert: + def test_holdout_dev_wrong_zero(self) -> None: + from evals.gsm8k_math.holdout_dev.v1.runner import build_report + + report = build_report() + assert report["counts"]["wrong"] == 0 + + +class TestPriorOrganRegressions: + def test_piecewise_0013_preserved(self) -> None: + result = parse_and_solve(_load_case("0013")) + assert result.answer == 450.0 + assert result.refusal_reason is None \ No newline at end of file