feat(gsm8k): capability paradigm sprint 12 singleton contract lift
Add Gate A2r nested_fraction_remainder_total (0004) and Gate A2s loose_crayon_box_capacity (0007) under explicit ClusterContracts with hard 0026/0047 neighbor confuser refusal. Serving lifts 24/26/0 to 26/24/0 with wrong=0 preserved; report.json and sealed artifacts untouched.
This commit is contained in:
parent
50e8de03fe
commit
ae974ce48d
8 changed files with 1078 additions and 5 deletions
|
|
@ -0,0 +1,265 @@
|
||||||
|
# GSM8K Capability Paradigm Sprint 12 — Lookback (2026-06-17)
|
||||||
|
|
||||||
|
## 1. Starting baseline
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|------:|
|
||||||
|
| correct | **24** |
|
||||||
|
| refused | **26** |
|
||||||
|
| wrong | **0** |
|
||||||
|
|
||||||
|
After #824 (Sprint 11 calendar ClusterContract). Prerequisite satisfied.
|
||||||
|
|
||||||
|
## 2. Experience Flywheel output summary
|
||||||
|
|
||||||
|
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint12.json`
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|------:|
|
||||||
|
| retained_record_count | 50 |
|
||||||
|
| lift_refused_to_correct | **0** |
|
||||||
|
| scout serving | 24 / 26 / 0 |
|
||||||
|
|
||||||
|
Top singleton signals:
|
||||||
|
|
||||||
|
- **0004** — `joint_refusal`, `fraction_surface`, `not_promotable`; no sealed-wrong on surface.
|
||||||
|
- **0007** — `joint_refusal`, `unbound_target`, `not_promotable`; no sealed-wrong on surface.
|
||||||
|
- Blocked families unchanged: `sealed_elimination` (0011, 0026), `relation_hypothesis` (0032, 0047), currency.
|
||||||
|
- No second calendar sibling under Sprint 11 contract.
|
||||||
|
|
||||||
|
## 3. Scout/frontier/microscope output summary
|
||||||
|
|
||||||
|
| Regime | correct | refused | wrong |
|
||||||
|
|--------|--------:|--------:|------:|
|
||||||
|
| Serving | 24 | 26 | 0 |
|
||||||
|
| Sealed (resolve_pooled) | 3 | 39 | 8 |
|
||||||
|
|
||||||
|
- `lift_refused_to_correct`: **0**
|
||||||
|
- Dominant bottleneck: joint refusal (20/50)
|
||||||
|
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
|
||||||
|
|
||||||
|
Microscope: 0004 tagged `fraction_operand`; 0007 tagged `conditional_question`. Adjacent sealed-wrong neighbors 0026 and 0047 on distinct surfaces.
|
||||||
|
|
||||||
|
## 4. Candidate families ranked
|
||||||
|
|
||||||
|
| Rank | Candidate | Lift | Wrong-risk neighbor | Verdict |
|
||||||
|
|------|-----------|------|---------------------|---------|
|
||||||
|
| 1 | `nested_fraction_remainder_total` (0004) | +1 | 0026 sealed_elimination | **Selected (A2r)** |
|
||||||
|
| 2 | `loose_crayon_box_capacity` (0007) | +1 | 0047 DCS/divisive | **Selected (A2s)** |
|
||||||
|
| 3 | calendar contract extension | +0 | — | **Rejected** (no second sibling) |
|
||||||
|
| — | DCS / currency / sealed_elimination wholesale | — | blocked | **Deferred** |
|
||||||
|
|
||||||
|
0004 and 0007 rank equally on flywheel; both admitted with hard neighbor confuser matrices.
|
||||||
|
|
||||||
|
## 5. ClusterContracts
|
||||||
|
|
||||||
|
### nested_fraction_remainder_contract (Gate A2r)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
ClusterContract:
|
||||||
|
family_id: nested_fraction_remainder_total
|
||||||
|
proposed_organs:
|
||||||
|
- nested_fraction_remainder_total
|
||||||
|
included_cases:
|
||||||
|
- gsm8k-train-sample-v1-0004
|
||||||
|
explicitly_excluded_cases:
|
||||||
|
- gsm8k-train-sample-v1-0026 # sealed_elimination / currency / scoops
|
||||||
|
- gsm8k-train-sample-v1-0011 # sealed_elimination
|
||||||
|
- asks subgroup only (not root altogether)
|
||||||
|
- multiple fraction relations
|
||||||
|
- 3/4 bill-of-money surfaces
|
||||||
|
positive_anchors:
|
||||||
|
- "Half of the {population}"
|
||||||
|
- "1/4 of the {population} going to {camp} camp"
|
||||||
|
- "{camp} camp in the morning" + afternoon complement count
|
||||||
|
- question asks root {population} altogether
|
||||||
|
negative_anchors:
|
||||||
|
- $ currency / each saved / scoops / bill / ice cream
|
||||||
|
- 3/4 of total money
|
||||||
|
- asks soccer-camp subgroup only
|
||||||
|
- multiple fractions
|
||||||
|
actor_binding_rule: shared population token across half/quarter/afternoon clauses
|
||||||
|
target_binding_rule: question must ask root population altogether
|
||||||
|
unit_binding_rule: afternoon count unit matches bound population
|
||||||
|
quantity_obligations:
|
||||||
|
- text-grounded afternoon complement count
|
||||||
|
- exactly one 1/4 inner partition
|
||||||
|
allowed_external_grounding: none
|
||||||
|
blocked_sibling_families:
|
||||||
|
- sealed_elimination
|
||||||
|
- currency_amount
|
||||||
|
- percent_partition
|
||||||
|
sealed_wrong_neighbors:
|
||||||
|
- 0026 (blocked by currency/each/scoops/3/4 bill hazards)
|
||||||
|
serving_admission_rule: resolve_promotable_nested_fraction_remainder_total()
|
||||||
|
implementation_allowed: true
|
||||||
|
reason: Typed morning/afternoon complement chain + 0026 confuser matrix clears wrong-risk.
|
||||||
|
```
|
||||||
|
|
||||||
|
### loose_crayon_box_capacity_contract (Gate A2s)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
ClusterContract:
|
||||||
|
family_id: loose_crayon_box_capacity
|
||||||
|
proposed_organs:
|
||||||
|
- loose_crayon_box_capacity
|
||||||
|
included_cases:
|
||||||
|
- gsm8k-train-sample-v1-0007
|
||||||
|
explicitly_excluded_cases:
|
||||||
|
- gsm8k-train-sample-v1-0047 # equal-distribution weight / eaten bag
|
||||||
|
- asks per-box capacity (not more boxes)
|
||||||
|
- equal number in bags
|
||||||
|
- weight/ounce surfaces
|
||||||
|
positive_anchors:
|
||||||
|
- "{actor} has N full boxes of {item}"
|
||||||
|
- "M loose {item}" before friend clause
|
||||||
|
- "friend has K loose"
|
||||||
|
- conditional "total of T" in question
|
||||||
|
- "how many more boxes"
|
||||||
|
negative_anchors:
|
||||||
|
- ounces / macaroons / equal number / eats / Steve
|
||||||
|
- asks per-box not more-boxes
|
||||||
|
- DCS divisive packing
|
||||||
|
actor_binding_rule: primary actor owns full boxes + loose; friend contributes loose only
|
||||||
|
target_binding_rule: question asks additional boxes for all loose items
|
||||||
|
unit_binding_rule: answer unit is boxes
|
||||||
|
quantity_obligations:
|
||||||
|
- total, loose_primary, loose_friend, num_full_boxes all text-grounded
|
||||||
|
allowed_external_grounding: none
|
||||||
|
blocked_sibling_families:
|
||||||
|
- multiplicative_aggregate / DCS
|
||||||
|
- currency_amount
|
||||||
|
sealed_wrong_neighbors:
|
||||||
|
- 0047 (blocked by weight/equal-number/eaten-bag hazards)
|
||||||
|
serving_admission_rule: resolve_promotable_loose_crayon_box_capacity()
|
||||||
|
implementation_allowed: true
|
||||||
|
reason: Capacity derivation from boxed portion + loose aggregation; 0047 confuser matrix holds.
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Blocked families and why
|
||||||
|
|
||||||
|
| Family | Blocked because |
|
||||||
|
|--------|-----------------|
|
||||||
|
| calendar extension | No second train_sample sibling under #824 policy |
|
||||||
|
| DCS 0032/0047 | sealed-wrong; relation_hypothesis blocked |
|
||||||
|
| currency 0019/0028 | sealed-wrong |
|
||||||
|
| sealed_elimination 0011/0026 | sealed-wrong |
|
||||||
|
| wholesale multiplicative_aggregate | 0025/0047 sealed-wrong |
|
||||||
|
|
||||||
|
## 7. Research gate decision
|
||||||
|
|
||||||
|
**Proceed** with two independent singleton ClusterContracts (A2r + A2s). Calendar extension rejected for lack of sibling evidence.
|
||||||
|
|
||||||
|
## 8. Selected paradigm
|
||||||
|
|
||||||
|
Dual singleton organs under explicit contracts:
|
||||||
|
|
||||||
|
1. `generate/derivation/nested_fraction_remainder_total.py` — Gate A2r
|
||||||
|
2. `generate/derivation/loose_crayon_box_capacity.py` — Gate A2s
|
||||||
|
|
||||||
|
Wired in `generate/math_candidate_graph.py` after Gate A2q.
|
||||||
|
|
||||||
|
## 9. Typed chains for target cases
|
||||||
|
|
||||||
|
**0004 (gold 2000):**
|
||||||
|
|
||||||
|
- afternoon = 750 kids (3/4 of soccer-camp subgroup)
|
||||||
|
- subgroup = 750 / 0.75 = 1000
|
||||||
|
- total = 1000 × 2 = 2000 (half go to soccer camp)
|
||||||
|
|
||||||
|
**0007 (gold 2):**
|
||||||
|
|
||||||
|
- boxed = 85 − 5 = 80
|
||||||
|
- per_box = 80 / 5 = 16
|
||||||
|
- loose_total = 5 + 27 = 32
|
||||||
|
- boxes_needed = 32 / 16 = 2
|
||||||
|
|
||||||
|
## 10. Implemented organ(s)
|
||||||
|
|
||||||
|
| Module | Gate | Promotion entry |
|
||||||
|
|--------|------|-----------------|
|
||||||
|
| `nested_fraction_remainder_total.py` | A2r | `resolve_promotable_nested_fraction_remainder_total()` |
|
||||||
|
| `loose_crayon_box_capacity.py` | A2s | `resolve_promotable_loose_crayon_box_capacity()` |
|
||||||
|
|
||||||
|
## 11. Before/after score
|
||||||
|
|
||||||
|
| Metric | Before | After | Δ |
|
||||||
|
|--------|-------:|------:|--:|
|
||||||
|
| correct | 24 | **26** | +2 |
|
||||||
|
| refused | 26 | **24** | −2 |
|
||||||
|
| wrong | 0 | **0** | 0 |
|
||||||
|
|
||||||
|
**State A achieved:** 26/24/0.
|
||||||
|
|
||||||
|
## 12. Newly solved IDs
|
||||||
|
|
||||||
|
**0004**, **0007**
|
||||||
|
|
||||||
|
## 13. Preserved solved IDs
|
||||||
|
|
||||||
|
0001, 0002, 0003, 0005, 0006, 0008, 0009, 0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046
|
||||||
|
|
||||||
|
## 14. Wrong=0 proof
|
||||||
|
|
||||||
|
- Full train_sample replay: 26 / 24 / 0
|
||||||
|
- `tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py`
|
||||||
|
- Holdout_dev: wrong=0
|
||||||
|
- Regression lane: sprint6–12 + flywheel + scout + microscope
|
||||||
|
|
||||||
|
## 15. Confuser matrix
|
||||||
|
|
||||||
|
| Confuser | Organ | Result |
|
||||||
|
|----------|-------|--------|
|
||||||
|
| 0026 sealed_elimination | A2r | refuse |
|
||||||
|
| asks subgroup only | A2r | refuse |
|
||||||
|
| multiple fractions | A2r | refuse |
|
||||||
|
| 0047 equal-distribution weight | A2s | refuse |
|
||||||
|
| asks per-box capacity | A2s | refuse |
|
||||||
|
| equal number in bags | A2s | refuse |
|
||||||
|
|
||||||
|
## 16. Sibling generality proof
|
||||||
|
|
||||||
|
| Sibling | Expected | Result |
|
||||||
|
|---------|----------|--------|
|
||||||
|
| music camp / 600 afternoon | 1600 | pass |
|
||||||
|
| four marker boxes / 67 total | 6 boxes | pass |
|
||||||
|
|
||||||
|
## 17. Sealed-wrong negative evidence used
|
||||||
|
|
||||||
|
- 0026: currency + each + scoops + 3/4 bill → A2r hazard refusal
|
||||||
|
- 0047: ounces + equal number + eaten bag → A2s hazard refusal
|
||||||
|
|
||||||
|
## 18. Composition-validation pin changes
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## 19. report.json untouched
|
||||||
|
|
||||||
|
Confirmed — no changes to committed `report.json`.
|
||||||
|
|
||||||
|
## 20. sealed artifacts untouched
|
||||||
|
|
||||||
|
Confirmed — no sealed lane artifact movement.
|
||||||
|
|
||||||
|
## 21. no case-id logic
|
||||||
|
|
||||||
|
Confirmed — admission via structural anchors and bindings only.
|
||||||
|
|
||||||
|
## 22. no hardcoded answer
|
||||||
|
|
||||||
|
Confirmed — arithmetic recomputed from grounded quantities.
|
||||||
|
|
||||||
|
## 23. no direct-answer shortcut
|
||||||
|
|
||||||
|
Confirmed — self-verified derivation chains required.
|
||||||
|
|
||||||
|
## 24. ClusterContract method enabled bigger lift
|
||||||
|
|
||||||
|
Yes — bundling two narrow singleton contracts under explicit neighbor exclusions delivered +2 with wrong=0, exceeding single-organ sprint cadence.
|
||||||
|
|
||||||
|
## 25. Sprint 13 recommendation
|
||||||
|
|
||||||
|
1. **Practice-lane mini-family clustering** — flywheel joint_refusal clusters (20 cases) may yield a 2-case mini-family safer than next singleton.
|
||||||
|
2. **Deferred blocked surfaces** — DCS 0032/0047 and currency 0019/0028 remain blocked until dedicated confuser ADRs land.
|
||||||
|
3. **Calendar contract** — revisit only if practice lane surfaces a genuine second piecewise sibling without policy expansion.
|
||||||
|
4. Target: 27+/23-/0 via one mini-family or two verified singletons with lowest sealed-wrong adjacency (candidates from flywheel: 0004/0007 neighbors exhausted; next scout frontier TBD).
|
||||||
285
generate/derivation/loose_crayon_box_capacity.py
Normal file
285
generate/derivation/loose_crayon_box_capacity.py
Normal file
|
|
@ -0,0 +1,285 @@
|
||||||
|
"""Gate A2s — loose crayon box capacity remainder (ClusterContract organ).
|
||||||
|
|
||||||
|
Sprint 12: train_sample **0007** — actor has ``N full boxes`` plus loose items,
|
||||||
|
friend contributes loose items, conditional total count, derive per-box capacity
|
||||||
|
from boxed portion, question asks ``how many more boxes`` for all loose items.
|
||||||
|
|
||||||
|
Chain (0007-class):
|
||||||
|
|
||||||
|
boxed = total − loose_primary
|
||||||
|
per_box = boxed / num_full_boxes
|
||||||
|
loose_total = loose_primary + loose_friend
|
||||||
|
boxes_needed = loose_total / per_box
|
||||||
|
|
||||||
|
Algebraically (single fold):
|
||||||
|
|
||||||
|
boxes_needed = (loose_primary + loose_friend) × num_full_boxes / (total − loose_primary)
|
||||||
|
|
||||||
|
Narrow organ — not equal-distribution/DCS (0047), not currency, not weight surfaces.
|
||||||
|
Promotion requires ClusterContract positive anchors and 0047-class hazard refusal.
|
||||||
|
|
||||||
|
Deterministic; sealed module (no ``chat/`` import).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections import Counter
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
from generate.derivation.extract import extract_quantities
|
||||||
|
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||||
|
from generate.derivation.target import _question_clause
|
||||||
|
from generate.derivation.verify import Resolution, SelfVerification
|
||||||
|
from generate.math_roundtrip import WORD_NUMBERS, _token_in, _tokens, _value_grounds
|
||||||
|
|
||||||
|
_FULL_BOXES_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)(\w+)\s+has\s+(\w+)\s+full\s+boxes\s+of\s+(\w+)"
|
||||||
|
)
|
||||||
|
_LOOSE_PRIMARY_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)(\d+)\s+loose\s+(\w+)"
|
||||||
|
)
|
||||||
|
_FRIEND_LOOSE_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)(?:\w+\s+)?friend\s+has\s+(\d+)\s+loose"
|
||||||
|
)
|
||||||
|
_TOTAL_CONDITIONAL_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)total\s+of\s+(\d+)"
|
||||||
|
)
|
||||||
|
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||||
|
{
|
||||||
|
"bags",
|
||||||
|
"bakes",
|
||||||
|
"brother",
|
||||||
|
"eats",
|
||||||
|
"equal",
|
||||||
|
"macaroons",
|
||||||
|
"ounces",
|
||||||
|
"packs",
|
||||||
|
"steve",
|
||||||
|
"weighing",
|
||||||
|
"weight",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||||
|
{"ounces", "weight", "per", "macaroons"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _box_count_token(problem_text: str, num_boxes: float) -> str:
|
||||||
|
match = _FULL_BOXES_RE.search(problem_text)
|
||||||
|
if match is not None:
|
||||||
|
return match.group(2).lower()
|
||||||
|
return str(int(num_boxes))
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_count(token: str) -> float | None:
|
||||||
|
lowered = token.lower()
|
||||||
|
if lowered in WORD_NUMBERS:
|
||||||
|
return float(WORD_NUMBERS[lowered])
|
||||||
|
try:
|
||||||
|
return float(token)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_more_boxes(question_clause: str) -> bool:
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
return (
|
||||||
|
"how" in tokens
|
||||||
|
and "many" in tokens
|
||||||
|
and "more" in tokens
|
||||||
|
and "boxes" in tokens
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_per_box(question_clause: str) -> bool:
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
return "each" in tokens or "per" in tokens
|
||||||
|
|
||||||
|
|
||||||
|
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||||
|
if "$" in problem_text:
|
||||||
|
return True
|
||||||
|
text_tokens = _tokens(problem_text)
|
||||||
|
question_tokens = _tokens(question_clause)
|
||||||
|
if text_tokens & _TEXT_BLOCKERS:
|
||||||
|
return True
|
||||||
|
if question_tokens & _QUESTION_BLOCKERS:
|
||||||
|
return True
|
||||||
|
if _asks_per_box(question_clause):
|
||||||
|
return True
|
||||||
|
if "equal number" in problem_text.lower():
|
||||||
|
return True
|
||||||
|
if not _FULL_BOXES_RE.search(problem_text):
|
||||||
|
return True
|
||||||
|
if not _FRIEND_LOOSE_RE.search(problem_text):
|
||||||
|
return True
|
||||||
|
if not _TOTAL_CONDITIONAL_RE.search(question_clause):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_bindings(problem_text: str) -> dict[str, float] | None:
|
||||||
|
boxes_match = _FULL_BOXES_RE.search(problem_text)
|
||||||
|
if boxes_match is None:
|
||||||
|
return None
|
||||||
|
num_boxes = _resolve_count(boxes_match.group(2))
|
||||||
|
if num_boxes is None or num_boxes <= 0:
|
||||||
|
return None
|
||||||
|
|
||||||
|
friend_match = _FRIEND_LOOSE_RE.search(problem_text)
|
||||||
|
if friend_match is None:
|
||||||
|
return None
|
||||||
|
friend_pos = friend_match.start()
|
||||||
|
|
||||||
|
loose_primary_matches = [
|
||||||
|
match
|
||||||
|
for match in _LOOSE_PRIMARY_RE.finditer(problem_text)
|
||||||
|
if match.start() < friend_pos
|
||||||
|
]
|
||||||
|
if len(loose_primary_matches) != 1:
|
||||||
|
return None
|
||||||
|
loose_primary = float(loose_primary_matches[0].group(1))
|
||||||
|
loose_friend = float(friend_match.group(1))
|
||||||
|
|
||||||
|
total_match = _TOTAL_CONDITIONAL_RE.search(_question_clause(problem_text))
|
||||||
|
if total_match is None:
|
||||||
|
return None
|
||||||
|
total = float(total_match.group(1))
|
||||||
|
|
||||||
|
item_unit = loose_primary_matches[0].group(2).lower()
|
||||||
|
return {
|
||||||
|
"num_boxes": num_boxes,
|
||||||
|
"loose_primary": loose_primary,
|
||||||
|
"loose_friend": loose_friend,
|
||||||
|
"total": total,
|
||||||
|
"item_unit": item_unit,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _recompute_boxes_needed(bindings: dict[str, float]) -> float:
|
||||||
|
boxed = bindings["total"] - bindings["loose_primary"]
|
||||||
|
per_box = boxed / bindings["num_boxes"]
|
||||||
|
loose_total = bindings["loose_primary"] + bindings["loose_friend"]
|
||||||
|
return loose_total / per_box
|
||||||
|
|
||||||
|
|
||||||
|
def build_loose_crayon_box_capacity(
|
||||||
|
problem_text: str,
|
||||||
|
) -> tuple[GroundedDerivation, float] | None:
|
||||||
|
"""Construct the ungated box-capacity chain, or ``None``."""
|
||||||
|
question_clause = _question_clause(problem_text)
|
||||||
|
if not _asks_more_boxes(question_clause):
|
||||||
|
return None
|
||||||
|
if _has_hazard_surface(problem_text, question_clause):
|
||||||
|
return None
|
||||||
|
|
||||||
|
bindings = _parse_bindings(problem_text)
|
||||||
|
if bindings is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
boxed_total = bindings["total"] - bindings["loose_primary"]
|
||||||
|
if boxed_total <= 0:
|
||||||
|
return None
|
||||||
|
|
||||||
|
loose_total = bindings["loose_primary"] + bindings["loose_friend"]
|
||||||
|
answer = _recompute_boxes_needed(bindings)
|
||||||
|
|
||||||
|
item_unit = bindings["item_unit"]
|
||||||
|
derivation = GroundedDerivation(
|
||||||
|
start=Quantity(
|
||||||
|
value=bindings["loose_primary"],
|
||||||
|
unit=item_unit,
|
||||||
|
source_token=str(int(bindings["loose_primary"])),
|
||||||
|
),
|
||||||
|
steps=(
|
||||||
|
Step(
|
||||||
|
op="add",
|
||||||
|
operand=Quantity(
|
||||||
|
value=bindings["loose_friend"],
|
||||||
|
unit=item_unit,
|
||||||
|
source_token=str(int(bindings["loose_friend"])),
|
||||||
|
),
|
||||||
|
cue="friend",
|
||||||
|
),
|
||||||
|
Step(
|
||||||
|
op="multiply",
|
||||||
|
operand=Quantity(
|
||||||
|
value=bindings["num_boxes"],
|
||||||
|
unit="boxes",
|
||||||
|
source_token=_box_count_token(problem_text, bindings["num_boxes"]),
|
||||||
|
),
|
||||||
|
cue=_box_count_token(problem_text, bindings["num_boxes"]),
|
||||||
|
),
|
||||||
|
Step(
|
||||||
|
op="divide",
|
||||||
|
operand=Quantity(
|
||||||
|
value=boxed_total,
|
||||||
|
unit=item_unit,
|
||||||
|
source_token=str(int(bindings["total"])),
|
||||||
|
),
|
||||||
|
cue="total",
|
||||||
|
comparative=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return derivation, answer
|
||||||
|
|
||||||
|
|
||||||
|
def _self_verifies_box_capacity(
|
||||||
|
derivation: GroundedDerivation, problem_text: str, expected: float
|
||||||
|
) -> SelfVerification:
|
||||||
|
tokens = _tokens(problem_text)
|
||||||
|
reasons: list[str] = []
|
||||||
|
|
||||||
|
for q in [derivation.start, *(s.operand for s in derivation.steps if not s.comparative)]:
|
||||||
|
if not _value_grounds(q.source_token, tokens):
|
||||||
|
reasons.append(f"operand {q.source_token!r} not grounded in text")
|
||||||
|
|
||||||
|
for step in derivation.steps:
|
||||||
|
if not _token_in(step.cue, tokens):
|
||||||
|
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||||
|
|
||||||
|
bindings = _parse_bindings(problem_text)
|
||||||
|
if bindings is None:
|
||||||
|
reasons.append("missing box-capacity bindings")
|
||||||
|
elif abs(_recompute_boxes_needed(bindings) - expected) > 1e-9:
|
||||||
|
reasons.append("arithmetic mismatch on boxes needed")
|
||||||
|
|
||||||
|
if abs(derivation.answer - expected) > 1e-9:
|
||||||
|
reasons.append("derivation fold mismatch")
|
||||||
|
|
||||||
|
obligation = Counter(q.source_token for q in extract_quantities(problem_text))
|
||||||
|
used = Counter(
|
||||||
|
[
|
||||||
|
derivation.start.source_token,
|
||||||
|
*(step.operand.source_token for step in derivation.steps),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
unused = obligation - used
|
||||||
|
if unused:
|
||||||
|
reasons.append(f"incomplete: unused quantities {sorted(unused.keys())}")
|
||||||
|
|
||||||
|
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||||
|
|
||||||
|
|
||||||
|
def compose_loose_crayon_box_capacity(problem_text: str) -> Resolution | None:
|
||||||
|
"""Gate the typed box-capacity chain through self-verification."""
|
||||||
|
built = build_loose_crayon_box_capacity(problem_text)
|
||||||
|
if built is None:
|
||||||
|
return None
|
||||||
|
derivation, answer = built
|
||||||
|
if not _self_verifies_box_capacity(derivation, problem_text, answer).verified:
|
||||||
|
return None
|
||||||
|
return Resolution(
|
||||||
|
answer=answer,
|
||||||
|
answer_unit="boxes",
|
||||||
|
derivation=derivation,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_promotable_loose_crayon_box_capacity(
|
||||||
|
problem_text: str,
|
||||||
|
) -> Resolution | None:
|
||||||
|
"""Serving promotion bridge (Gate A2s, ClusterContract loose_crayon_box)."""
|
||||||
|
return compose_loose_crayon_box_capacity(problem_text)
|
||||||
260
generate/derivation/nested_fraction_remainder_total.py
Normal file
260
generate/derivation/nested_fraction_remainder_total.py
Normal file
|
|
@ -0,0 +1,260 @@
|
||||||
|
"""Gate A2r — nested fraction complement remainder total (ClusterContract organ).
|
||||||
|
|
||||||
|
Sprint 12: train_sample **0004** — outer ``Half of`` partition, inner ``1/4 of``
|
||||||
|
subgroup split (morning vs afternoon), known afternoon complement count, question
|
||||||
|
asks root population ``altogether``.
|
||||||
|
|
||||||
|
Chain (0004-class):
|
||||||
|
|
||||||
|
subgroup = afternoon / (1 − inner_fraction)
|
||||||
|
total = subgroup / outer_half_scalar = subgroup × 2
|
||||||
|
|
||||||
|
Narrow organ — not generic fraction parser, not sealed_elimination, not currency.
|
||||||
|
Promotion requires ClusterContract positive anchors, explicit target binding, and
|
||||||
|
0026-class hazard refusal.
|
||||||
|
|
||||||
|
Deterministic; sealed module (no ``chat/`` import).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections import Counter
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
from generate.derivation.extract import extract_quantities
|
||||||
|
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||||
|
from generate.derivation.target import _question_clause
|
||||||
|
from generate.derivation.verify import Resolution, SelfVerification
|
||||||
|
from generate.math_roundtrip import _token_in, _tokens, _value_grounds
|
||||||
|
|
||||||
|
_OUTER_HALF_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)\bhalf\s+of\s+the\s+(\w+)\b"
|
||||||
|
)
|
||||||
|
_INNER_QUARTER_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)\b1/4\s+of\s+the\s+(\w+)\s+going\s+to\s+(\w+)\s+camp\b"
|
||||||
|
)
|
||||||
|
_MORNING_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)\bgoing\s+to\s+(\w+)\s+camp\s+in\s+the\s+morning\b"
|
||||||
|
)
|
||||||
|
_AFTERNOON_COUNT_RE: Final[re.Pattern[str]] = re.compile(
|
||||||
|
r"(?i)(\d+)\s+(\w+)\s+are\s+going\s+to\s+(\w+)\s+camp\s+in\s+the\s+afternoon"
|
||||||
|
)
|
||||||
|
_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||||
|
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||||
|
{
|
||||||
|
"bill",
|
||||||
|
"dinner",
|
||||||
|
"dollars",
|
||||||
|
"each",
|
||||||
|
"ice",
|
||||||
|
"money",
|
||||||
|
"profit",
|
||||||
|
"saved",
|
||||||
|
"scoops",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||||
|
{"afternoon", "morning", "scoops"}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class _CampBinding:
|
||||||
|
population: str
|
||||||
|
camp: str
|
||||||
|
afternoon_count: float
|
||||||
|
afternoon_token: str
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_root_altogether(question_clause: str, population: str) -> bool:
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
return (
|
||||||
|
"how" in tokens
|
||||||
|
and "many" in tokens
|
||||||
|
and population in tokens
|
||||||
|
and ("altogether" in tokens or "there" in tokens)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_subgroup_only(question_clause: str) -> bool:
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
return "camp" in tokens and "altogether" not in tokens
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_camp_binding(problem_text: str) -> _CampBinding | None:
|
||||||
|
outer = _OUTER_HALF_RE.search(problem_text)
|
||||||
|
inner = _INNER_QUARTER_RE.search(problem_text)
|
||||||
|
morning = _MORNING_RE.search(problem_text)
|
||||||
|
afternoon = _AFTERNOON_COUNT_RE.search(problem_text)
|
||||||
|
if outer is None or inner is None or morning is None or afternoon is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
population = outer.group(1).lower()
|
||||||
|
inner_pop = inner.group(1).lower()
|
||||||
|
inner_camp = inner.group(2).lower()
|
||||||
|
morning_camp = morning.group(1).lower()
|
||||||
|
afternoon_count = float(afternoon.group(1))
|
||||||
|
afternoon_pop = afternoon.group(2).lower()
|
||||||
|
afternoon_camp = afternoon.group(3).lower()
|
||||||
|
|
||||||
|
if not {population, inner_pop, afternoon_pop} == {population}:
|
||||||
|
return None
|
||||||
|
if not {inner_camp, morning_camp, afternoon_camp} == {inner_camp}:
|
||||||
|
return None
|
||||||
|
return _CampBinding(
|
||||||
|
population=population,
|
||||||
|
camp=inner_camp,
|
||||||
|
afternoon_count=afternoon_count,
|
||||||
|
afternoon_token=afternoon.group(1),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||||
|
if "$" in problem_text:
|
||||||
|
return True
|
||||||
|
text_tokens = _tokens(problem_text)
|
||||||
|
question_tokens = _tokens(question_clause)
|
||||||
|
if text_tokens & _TEXT_BLOCKERS:
|
||||||
|
return True
|
||||||
|
if question_tokens & _QUESTION_BLOCKERS:
|
||||||
|
return True
|
||||||
|
if _asks_subgroup_only(question_clause):
|
||||||
|
return True
|
||||||
|
if "3/4" in problem_text.replace(" ", ""):
|
||||||
|
return True
|
||||||
|
if "saved" in text_tokens and "each" in text_tokens:
|
||||||
|
return True
|
||||||
|
fractions = _EXTRA_FRACTION_RE.findall(problem_text)
|
||||||
|
if len(fractions) != 1 or fractions[0].replace(" ", "") != "1/4":
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _inner_complement_scalar() -> float:
|
||||||
|
return 1.0 - (1.0 / 4.0)
|
||||||
|
|
||||||
|
|
||||||
|
def _outer_inverse_half_scalar() -> float:
|
||||||
|
return 2.0
|
||||||
|
|
||||||
|
|
||||||
|
def _recompute_total(afternoon_count: float) -> float:
|
||||||
|
subgroup = afternoon_count / _inner_complement_scalar()
|
||||||
|
return subgroup * _outer_inverse_half_scalar()
|
||||||
|
|
||||||
|
|
||||||
|
def build_nested_fraction_remainder_total(
|
||||||
|
problem_text: str,
|
||||||
|
) -> tuple[GroundedDerivation, float, _CampBinding] | None:
|
||||||
|
"""Construct the ungated nested-fraction chain, or ``None``."""
|
||||||
|
binding = _parse_camp_binding(problem_text)
|
||||||
|
if binding is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
question_clause = _question_clause(problem_text)
|
||||||
|
if not _asks_root_altogether(question_clause, binding.population):
|
||||||
|
return None
|
||||||
|
if _has_hazard_surface(problem_text, question_clause):
|
||||||
|
return None
|
||||||
|
|
||||||
|
afternoon = Quantity(
|
||||||
|
value=binding.afternoon_count,
|
||||||
|
unit=binding.population,
|
||||||
|
source_token=binding.afternoon_token,
|
||||||
|
)
|
||||||
|
total = _recompute_total(binding.afternoon_count)
|
||||||
|
derivation = GroundedDerivation(
|
||||||
|
start=afternoon,
|
||||||
|
steps=(
|
||||||
|
Step(
|
||||||
|
op="divide",
|
||||||
|
operand=Quantity(
|
||||||
|
value=_inner_complement_scalar(),
|
||||||
|
unit="",
|
||||||
|
source_token="1/4",
|
||||||
|
),
|
||||||
|
cue="4",
|
||||||
|
comparative=True,
|
||||||
|
),
|
||||||
|
Step(
|
||||||
|
op="multiply",
|
||||||
|
operand=Quantity(
|
||||||
|
value=_outer_inverse_half_scalar(),
|
||||||
|
unit="",
|
||||||
|
source_token="half",
|
||||||
|
),
|
||||||
|
cue="half",
|
||||||
|
comparative=True,
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return derivation, total, binding
|
||||||
|
|
||||||
|
|
||||||
|
def _self_verifies_nested_fraction(
|
||||||
|
derivation: GroundedDerivation,
|
||||||
|
problem_text: str,
|
||||||
|
expected_total: float,
|
||||||
|
binding: _CampBinding,
|
||||||
|
) -> SelfVerification:
|
||||||
|
tokens = _tokens(problem_text)
|
||||||
|
reasons: list[str] = []
|
||||||
|
|
||||||
|
if not _operand_grounds(derivation.start, tokens):
|
||||||
|
reasons.append(f"operand {derivation.start.source_token!r} not grounded in text")
|
||||||
|
|
||||||
|
for step in derivation.steps:
|
||||||
|
if not step.comparative and not _operand_grounds(step.operand, tokens):
|
||||||
|
reasons.append(f"operand {step.operand.source_token!r} not grounded in text")
|
||||||
|
if not _token_in(step.cue, tokens):
|
||||||
|
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||||
|
|
||||||
|
if _parse_camp_binding(problem_text) is None:
|
||||||
|
reasons.append("missing camp partition binding")
|
||||||
|
elif abs(_recompute_total(binding.afternoon_count) - expected_total) > 1e-9:
|
||||||
|
reasons.append("arithmetic mismatch on nested fraction total")
|
||||||
|
|
||||||
|
if abs(derivation.answer - expected_total) > 1e-9:
|
||||||
|
reasons.append("derivation fold mismatch")
|
||||||
|
|
||||||
|
obligation = Counter(
|
||||||
|
q.source_token
|
||||||
|
for q in extract_quantities(problem_text)
|
||||||
|
if q.unit == binding.population and q.source_token.isdigit()
|
||||||
|
)
|
||||||
|
used = Counter([derivation.start.source_token])
|
||||||
|
unused = obligation - used
|
||||||
|
if unused:
|
||||||
|
reasons.append(
|
||||||
|
f"incomplete: unused {binding.population} quantities {sorted(unused.keys())}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||||
|
|
||||||
|
|
||||||
|
def _operand_grounds(q: Quantity, tokens: frozenset[str]) -> bool:
|
||||||
|
return _value_grounds(q.source_token, tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def compose_nested_fraction_remainder_total(problem_text: str) -> Resolution | None:
|
||||||
|
"""Gate the typed nested-fraction chain through self-verification."""
|
||||||
|
built = build_nested_fraction_remainder_total(problem_text)
|
||||||
|
if built is None:
|
||||||
|
return None
|
||||||
|
derivation, total, binding = built
|
||||||
|
if not _self_verifies_nested_fraction(derivation, problem_text, total, binding).verified:
|
||||||
|
return None
|
||||||
|
return Resolution(
|
||||||
|
answer=total,
|
||||||
|
answer_unit=binding.population,
|
||||||
|
derivation=derivation,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_promotable_nested_fraction_remainder_total(
|
||||||
|
problem_text: str,
|
||||||
|
) -> Resolution | None:
|
||||||
|
"""Serving promotion bridge (Gate A2r, ClusterContract nested_fraction)."""
|
||||||
|
return compose_nested_fraction_remainder_total(problem_text)
|
||||||
|
|
@ -898,6 +898,38 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
||||||
branches_admissible=1,
|
branches_admissible=1,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Gate A2r — nested fraction complement remainder total (Sprint 12 singleton).
|
||||||
|
from generate.derivation.nested_fraction_remainder_total import (
|
||||||
|
resolve_promotable_nested_fraction_remainder_total,
|
||||||
|
)
|
||||||
|
|
||||||
|
nested_fraction_resolution = resolve_promotable_nested_fraction_remainder_total(
|
||||||
|
text
|
||||||
|
)
|
||||||
|
if nested_fraction_resolution is not None:
|
||||||
|
return CandidateGraphResult(
|
||||||
|
answer=nested_fraction_resolution.answer,
|
||||||
|
selected_graph=None,
|
||||||
|
refusal_reason=None,
|
||||||
|
branches_enumerated=1,
|
||||||
|
branches_admissible=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Gate A2s — loose crayon box capacity (Sprint 12 singleton).
|
||||||
|
from generate.derivation.loose_crayon_box_capacity import (
|
||||||
|
resolve_promotable_loose_crayon_box_capacity,
|
||||||
|
)
|
||||||
|
|
||||||
|
box_capacity_resolution = resolve_promotable_loose_crayon_box_capacity(text)
|
||||||
|
if box_capacity_resolution is not None:
|
||||||
|
return CandidateGraphResult(
|
||||||
|
answer=box_capacity_resolution.answer,
|
||||||
|
selected_graph=None,
|
||||||
|
refusal_reason=None,
|
||||||
|
branches_enumerated=1,
|
||||||
|
branches_admissible=1,
|
||||||
|
)
|
||||||
|
|
||||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||||
if len(statement_sentences) == 1:
|
if len(statement_sentences) == 1:
|
||||||
|
|
|
||||||
|
|
@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
|
||||||
summary = build_microscope_report(_load_cases())
|
summary = build_microscope_report(_load_cases())
|
||||||
md = render_markdown(summary)
|
md = render_markdown(summary)
|
||||||
assert "partition_chunking" in md
|
assert "partition_chunking" in md
|
||||||
assert "correct: 24" in md
|
assert "correct: 26" in md
|
||||||
assert "Gate A2a unit_partition" in md
|
assert "Gate A2a unit_partition" in md
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -218,8 +218,8 @@ class TestTrainSampleScore:
|
||||||
report = build_report(_load_cases(_CASES_PATH))
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
counts = report["counts"]
|
counts = report["counts"]
|
||||||
assert counts["wrong"] == 0
|
assert counts["wrong"] == 0
|
||||||
assert counts["correct"] >= 24
|
assert counts["correct"] >= 26
|
||||||
assert counts["refused"] <= 26
|
assert counts["refused"] <= 24
|
||||||
|
|
||||||
def test_newly_solved_ids(self) -> None:
|
def test_newly_solved_ids(self) -> None:
|
||||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||||
|
|
|
||||||
|
|
@ -232,8 +232,8 @@ class TestTrainSampleScore:
|
||||||
report = build_report(_load_cases(_CASES_PATH))
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
counts = report["counts"]
|
counts = report["counts"]
|
||||||
assert counts["wrong"] == 0
|
assert counts["wrong"] == 0
|
||||||
assert counts["correct"] >= 24
|
assert counts["correct"] >= 26
|
||||||
assert counts["refused"] <= 26
|
assert counts["refused"] <= 24
|
||||||
|
|
||||||
def test_newly_solved_0013(self) -> None:
|
def test_newly_solved_0013(self) -> None:
|
||||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,231 @@
|
||||||
|
"""Gate A2r/A2s — Capability Paradigm Sprint 12 singleton ClusterContract lift."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from generate.math_candidate_graph import parse_and_solve
|
||||||
|
from generate.derivation.nested_fraction_remainder_total import (
|
||||||
|
compose_nested_fraction_remainder_total,
|
||||||
|
resolve_promotable_nested_fraction_remainder_total,
|
||||||
|
)
|
||||||
|
from generate.derivation.loose_crayon_box_capacity import (
|
||||||
|
compose_loose_crayon_box_capacity,
|
||||||
|
resolve_promotable_loose_crayon_box_capacity,
|
||||||
|
)
|
||||||
|
|
||||||
|
_CASES_PATH = (
|
||||||
|
Path(__file__).resolve().parents[1]
|
||||||
|
/ "evals"
|
||||||
|
/ "gsm8k_math"
|
||||||
|
/ "train_sample"
|
||||||
|
/ "v1"
|
||||||
|
/ "cases.jsonl"
|
||||||
|
)
|
||||||
|
|
||||||
|
CASE_0004 = (
|
||||||
|
"There are some kids in camp. Half of the kids are going to soccer camp, "
|
||||||
|
"and 1/4 of the kids going to soccer camp are going to soccer camp in the morning. "
|
||||||
|
"750 kids are going to soccer camp in the afternoon. "
|
||||||
|
"How many kids there in camp altogether?"
|
||||||
|
)
|
||||||
|
|
||||||
|
CASE_0007 = (
|
||||||
|
"Francine has five full boxes of crayons and 5 loose crayons, and her friend "
|
||||||
|
"has 27 loose crayons. They need to put all of their loose crayons in a box. "
|
||||||
|
"How many more boxes do they need if Francine has a total of 85 crayons?"
|
||||||
|
)
|
||||||
|
|
||||||
|
SIBLING_CAMP = (
|
||||||
|
"There are some students in camp. Half of the students are going to music camp, "
|
||||||
|
"and 1/4 of the students going to music camp are going to music camp in the morning. "
|
||||||
|
"600 students are going to music camp in the afternoon. "
|
||||||
|
"How many students there in camp altogether?"
|
||||||
|
)
|
||||||
|
|
||||||
|
SIBLING_BOXES = (
|
||||||
|
"Jordan has four full boxes of markers and 3 loose markers, and her friend "
|
||||||
|
"has 21 loose markers. They need to put all of their loose markers in a box. "
|
||||||
|
"How many more boxes do they need if Jordan has a total of 19 markers?"
|
||||||
|
)
|
||||||
|
|
||||||
|
SEALED_ELIMINATION_0026 = (
|
||||||
|
"Aaron and his brother Carson each saved up $40 to go to dinner. "
|
||||||
|
"The bill is 3/4 of their total money. After, they go out for ice cream. "
|
||||||
|
"Each scoop costs $1.5 and they get the same amount as each other. "
|
||||||
|
"If they leave with $1 in change each, how many scoops did they each buy?"
|
||||||
|
)
|
||||||
|
|
||||||
|
SEALED_WRONG_0047_SHAPE = (
|
||||||
|
"John bakes 12 coconut macaroons, each weighing 5 ounces. "
|
||||||
|
"He then packs an equal number of the macaroons in 4 different brown bags, "
|
||||||
|
"ready for delivery. When he briefly leaves the kitchen to pick the phone, "
|
||||||
|
"his little brother Steve eats the entire contents of one of the brown bags. "
|
||||||
|
"What is the total weight, in ounces, of the remaining coconut macaroons?"
|
||||||
|
)
|
||||||
|
|
||||||
|
ASKS_SUBGROUP_ONLY_CONFUSER = (
|
||||||
|
"There are some kids in camp. Half of the kids are going to soccer camp, "
|
||||||
|
"and 1/4 of the kids going to soccer camp are going to soccer camp in the morning. "
|
||||||
|
"750 kids are going to soccer camp in the afternoon. "
|
||||||
|
"How many kids are going to soccer camp?"
|
||||||
|
)
|
||||||
|
|
||||||
|
ASKS_PER_BOX_CONFUSER = (
|
||||||
|
"Francine has five full boxes of crayons and 5 loose crayons, and her friend "
|
||||||
|
"has 27 loose crayons. They need to put all of their loose crayons in a box. "
|
||||||
|
"How many crayons are in each full box if Francine has a total of 85 crayons?"
|
||||||
|
)
|
||||||
|
|
||||||
|
MULTIPLE_FRACTION_CONFUSER = (
|
||||||
|
"Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp "
|
||||||
|
"are going in the morning. 1/2 of the afternoon group brought lunch. "
|
||||||
|
"750 kids are going to soccer camp in the afternoon. "
|
||||||
|
"How many kids there in camp altogether?"
|
||||||
|
)
|
||||||
|
|
||||||
|
PRESERVED_SOLVED = (
|
||||||
|
"0001",
|
||||||
|
"0002",
|
||||||
|
"0003",
|
||||||
|
"0005",
|
||||||
|
"0006",
|
||||||
|
"0008",
|
||||||
|
"0009",
|
||||||
|
"0010",
|
||||||
|
"0013",
|
||||||
|
"0014",
|
||||||
|
"0015",
|
||||||
|
"0017",
|
||||||
|
"0018",
|
||||||
|
"0021",
|
||||||
|
"0024",
|
||||||
|
"0025",
|
||||||
|
"0029",
|
||||||
|
"0030",
|
||||||
|
"0035",
|
||||||
|
"0037",
|
||||||
|
"0038",
|
||||||
|
"0042",
|
||||||
|
"0045",
|
||||||
|
"0046",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_case(case_suffix: str) -> str:
|
||||||
|
for line in _CASES_PATH.read_text().splitlines():
|
||||||
|
row = json.loads(line)
|
||||||
|
if row["case_id"].endswith(case_suffix):
|
||||||
|
return row["question"]
|
||||||
|
raise KeyError(case_suffix)
|
||||||
|
|
||||||
|
|
||||||
|
class TestClusterContractPolicy:
|
||||||
|
def test_nested_fraction_contract_admits_0004_shape(self) -> None:
|
||||||
|
assert compose_nested_fraction_remainder_total(CASE_0004) is not None
|
||||||
|
|
||||||
|
def test_box_capacity_contract_admits_0007_shape(self) -> None:
|
||||||
|
assert compose_loose_crayon_box_capacity(CASE_0007) is not None
|
||||||
|
|
||||||
|
|
||||||
|
class TestTargetCases:
|
||||||
|
def test_0004_solves(self) -> None:
|
||||||
|
result = parse_and_solve(CASE_0004)
|
||||||
|
assert result.answer == 2000.0
|
||||||
|
assert result.refusal_reason is None
|
||||||
|
|
||||||
|
def test_0007_solves(self) -> None:
|
||||||
|
result = parse_and_solve(CASE_0007)
|
||||||
|
assert result.answer == 2.0
|
||||||
|
assert result.refusal_reason is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestSiblingGeneralization:
|
||||||
|
def test_music_camp_sibling(self) -> None:
|
||||||
|
resolution = compose_nested_fraction_remainder_total(SIBLING_CAMP)
|
||||||
|
assert resolution is not None
|
||||||
|
assert resolution.answer == 1600.0
|
||||||
|
|
||||||
|
def test_marker_box_sibling(self) -> None:
|
||||||
|
resolution = compose_loose_crayon_box_capacity(SIBLING_BOXES)
|
||||||
|
assert resolution is not None
|
||||||
|
# (3+21)*4/(67-3) = 96/16 = 6
|
||||||
|
assert resolution.answer == 6.0
|
||||||
|
|
||||||
|
|
||||||
|
class TestNeighborConfuserRefusals:
|
||||||
|
def test_sealed_elimination_0026_refuses(self) -> None:
|
||||||
|
assert compose_nested_fraction_remainder_total(SEALED_ELIMINATION_0026) is None
|
||||||
|
|
||||||
|
def test_dcs_0047_refuses(self) -> None:
|
||||||
|
assert compose_loose_crayon_box_capacity(SEALED_WRONG_0047_SHAPE) is None
|
||||||
|
|
||||||
|
def test_asks_subgroup_only_refuses(self) -> None:
|
||||||
|
assert compose_nested_fraction_remainder_total(ASKS_SUBGROUP_ONLY_CONFUSER) is None
|
||||||
|
|
||||||
|
def test_asks_per_box_refuses(self) -> None:
|
||||||
|
assert compose_loose_crayon_box_capacity(ASKS_PER_BOX_CONFUSER) is None
|
||||||
|
|
||||||
|
def test_multiple_fraction_refuses(self) -> None:
|
||||||
|
assert compose_nested_fraction_remainder_total(MULTIPLE_FRACTION_CONFUSER) is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestPromotionBridges:
|
||||||
|
def test_resolve_promotable_nested_fraction(self) -> None:
|
||||||
|
assert resolve_promotable_nested_fraction_remainder_total(CASE_0004) is not None
|
||||||
|
|
||||||
|
def test_resolve_promotable_box_capacity(self) -> None:
|
||||||
|
assert resolve_promotable_loose_crayon_box_capacity(CASE_0007) is not None
|
||||||
|
|
||||||
|
|
||||||
|
class TestTrainSampleScore:
|
||||||
|
def test_wrong_zero_and_state_a(self) -> None:
|
||||||
|
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||||
|
|
||||||
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
|
counts = report["counts"]
|
||||||
|
assert counts["wrong"] == 0
|
||||||
|
assert counts["correct"] >= 26
|
||||||
|
assert counts["refused"] <= 24
|
||||||
|
|
||||||
|
def test_newly_solved_ids(self) -> None:
|
||||||
|
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||||
|
|
||||||
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
|
correct = {
|
||||||
|
row["case_id"].split("-")[-1]
|
||||||
|
for row in report["per_case"]
|
||||||
|
if row["verdict"] == "correct"
|
||||||
|
}
|
||||||
|
assert "0004" in correct
|
||||||
|
assert "0007" in correct
|
||||||
|
|
||||||
|
def test_preserved_solved_ids(self) -> None:
|
||||||
|
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||||
|
|
||||||
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
|
correct = {
|
||||||
|
row["case_id"].split("-")[-1]
|
||||||
|
for row in report["per_case"]
|
||||||
|
if row["verdict"] == "correct"
|
||||||
|
}
|
||||||
|
for case_id in PRESERVED_SOLVED:
|
||||||
|
assert case_id in correct
|
||||||
|
|
||||||
|
|
||||||
|
class TestHoldoutInert:
|
||||||
|
def test_holdout_dev_wrong_zero(self) -> None:
|
||||||
|
from evals.gsm8k_math.holdout_dev.v1.runner import build_report
|
||||||
|
|
||||||
|
report = build_report()
|
||||||
|
assert report["counts"]["wrong"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
class TestPriorOrganRegressions:
|
||||||
|
def test_piecewise_0013_preserved(self) -> None:
|
||||||
|
result = parse_and_solve(_load_case("0013"))
|
||||||
|
assert result.answer == 450.0
|
||||||
|
assert result.refusal_reason is None
|
||||||
Loading…
Reference in a new issue