feat(gsm8k): capability paradigm sprint 12 singleton contract lift
Add Gate A2r nested_fraction_remainder_total (0004) and Gate A2s loose_crayon_box_capacity (0007) under explicit ClusterContracts with hard 0026/0047 neighbor confuser refusal. Serving lifts 24/26/0 to 26/24/0 with wrong=0 preserved; report.json and sealed artifacts untouched.
This commit is contained in:
Shay
parent
50e8de03fe
commit
ae974ce48d
8 changed files with 1078 additions and 5 deletions
|
|
@ -0,0 +1,265 @@
|
|||
# GSM8K Capability Paradigm Sprint 12 — Lookback (2026-06-17)
|
||||
|
||||
## 1. Starting baseline
|
||||
|
||||
| Metric | Value |
|
||||
|--------|------:|
|
||||
| correct | **24** |
|
||||
| refused | **26** |
|
||||
| wrong | **0** |
|
||||
|
||||
After #824 (Sprint 11 calendar ClusterContract). Prerequisite satisfied.
|
||||
|
||||
## 2. Experience Flywheel output summary
|
||||
|
||||
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint12.json`
|
||||
|
||||
| Field | Value |
|
||||
|-------|------:|
|
||||
| retained_record_count | 50 |
|
||||
| lift_refused_to_correct | **0** |
|
||||
| scout serving | 24 / 26 / 0 |
|
||||
|
||||
Top singleton signals:
|
||||
|
||||
- **0004** — `joint_refusal`, `fraction_surface`, `not_promotable`; no sealed-wrong on surface.
|
||||
- **0007** — `joint_refusal`, `unbound_target`, `not_promotable`; no sealed-wrong on surface.
|
||||
- Blocked families unchanged: `sealed_elimination` (0011, 0026), `relation_hypothesis` (0032, 0047), currency.
|
||||
- No second calendar sibling under Sprint 11 contract.
|
||||
|
||||
## 3. Scout/frontier/microscope output summary
|
||||
|
||||
| Regime | correct | refused | wrong |
|
||||
|--------|--------:|--------:|------:|
|
||||
| Serving | 24 | 26 | 0 |
|
||||
| Sealed (resolve_pooled) | 3 | 39 | 8 |
|
||||
|
||||
- `lift_refused_to_correct`: **0**
|
||||
- Dominant bottleneck: joint refusal (20/50)
|
||||
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
|
||||
|
||||
Microscope: 0004 tagged `fraction_operand`; 0007 tagged `conditional_question`. Adjacent sealed-wrong neighbors 0026 and 0047 on distinct surfaces.
|
||||
|
||||
## 4. Candidate families ranked
|
||||
|
||||
| Rank | Candidate | Lift | Wrong-risk neighbor | Verdict |
|
||||
|------|-----------|------|---------------------|---------|
|
||||
| 1 | `nested_fraction_remainder_total` (0004) | +1 | 0026 sealed_elimination | **Selected (A2r)** |
|
||||
| 2 | `loose_crayon_box_capacity` (0007) | +1 | 0047 DCS/divisive | **Selected (A2s)** |
|
||||
| 3 | calendar contract extension | +0 | — | **Rejected** (no second sibling) |
|
||||
| — | DCS / currency / sealed_elimination wholesale | — | blocked | **Deferred** |
|
||||
|
||||
0004 and 0007 rank equally on flywheel; both admitted with hard neighbor confuser matrices.
|
||||
|
||||
## 5. ClusterContracts
|
||||
|
||||
### nested_fraction_remainder_contract (Gate A2r)
|
||||
|
||||
```yaml
|
||||
ClusterContract:
|
||||
family_id: nested_fraction_remainder_total
|
||||
proposed_organs:
|
||||
- nested_fraction_remainder_total
|
||||
included_cases:
|
||||
- gsm8k-train-sample-v1-0004
|
||||
explicitly_excluded_cases:
|
||||
- gsm8k-train-sample-v1-0026 # sealed_elimination / currency / scoops
|
||||
- gsm8k-train-sample-v1-0011 # sealed_elimination
|
||||
- asks subgroup only (not root altogether)
|
||||
- multiple fraction relations
|
||||
- 3/4 bill-of-money surfaces
|
||||
positive_anchors:
|
||||
- "Half of the {population}"
|
||||
- "1/4 of the {population} going to {camp} camp"
|
||||
- "{camp} camp in the morning" + afternoon complement count
|
||||
- question asks root {population} altogether
|
||||
negative_anchors:
|
||||
- $ currency / each saved / scoops / bill / ice cream
|
||||
- 3/4 of total money
|
||||
- asks soccer-camp subgroup only
|
||||
- multiple fractions
|
||||
actor_binding_rule: shared population token across half/quarter/afternoon clauses
|
||||
target_binding_rule: question must ask root population altogether
|
||||
unit_binding_rule: afternoon count unit matches bound population
|
||||
quantity_obligations:
|
||||
- text-grounded afternoon complement count
|
||||
- exactly one 1/4 inner partition
|
||||
allowed_external_grounding: none
|
||||
blocked_sibling_families:
|
||||
- sealed_elimination
|
||||
- currency_amount
|
||||
- percent_partition
|
||||
sealed_wrong_neighbors:
|
||||
- 0026 (blocked by currency/each/scoops/3/4 bill hazards)
|
||||
serving_admission_rule: resolve_promotable_nested_fraction_remainder_total()
|
||||
implementation_allowed: true
|
||||
reason: Typed morning/afternoon complement chain + 0026 confuser matrix clears wrong-risk.
|
||||
```
|
||||
|
||||
### loose_crayon_box_capacity_contract (Gate A2s)
|
||||
|
||||
```yaml
|
||||
ClusterContract:
|
||||
family_id: loose_crayon_box_capacity
|
||||
proposed_organs:
|
||||
- loose_crayon_box_capacity
|
||||
included_cases:
|
||||
- gsm8k-train-sample-v1-0007
|
||||
explicitly_excluded_cases:
|
||||
- gsm8k-train-sample-v1-0047 # equal-distribution weight / eaten bag
|
||||
- asks per-box capacity (not more boxes)
|
||||
- equal number in bags
|
||||
- weight/ounce surfaces
|
||||
positive_anchors:
|
||||
- "{actor} has N full boxes of {item}"
|
||||
- "M loose {item}" before friend clause
|
||||
- "friend has K loose"
|
||||
- conditional "total of T" in question
|
||||
- "how many more boxes"
|
||||
negative_anchors:
|
||||
- ounces / macaroons / equal number / eats / Steve
|
||||
- asks per-box not more-boxes
|
||||
- DCS divisive packing
|
||||
actor_binding_rule: primary actor owns full boxes + loose; friend contributes loose only
|
||||
target_binding_rule: question asks additional boxes for all loose items
|
||||
unit_binding_rule: answer unit is boxes
|
||||
quantity_obligations:
|
||||
- total, loose_primary, loose_friend, num_full_boxes all text-grounded
|
||||
allowed_external_grounding: none
|
||||
blocked_sibling_families:
|
||||
- multiplicative_aggregate / DCS
|
||||
- currency_amount
|
||||
sealed_wrong_neighbors:
|
||||
- 0047 (blocked by weight/equal-number/eaten-bag hazards)
|
||||
serving_admission_rule: resolve_promotable_loose_crayon_box_capacity()
|
||||
implementation_allowed: true
|
||||
reason: Capacity derivation from boxed portion + loose aggregation; 0047 confuser matrix holds.
|
||||
```
|
||||
|
||||
## 6. Blocked families and why
|
||||
|
||||
| Family | Blocked because |
|
||||
|--------|-----------------|
|
||||
| calendar extension | No second train_sample sibling under #824 policy |
|
||||
| DCS 0032/0047 | sealed-wrong; relation_hypothesis blocked |
|
||||
| currency 0019/0028 | sealed-wrong |
|
||||
| sealed_elimination 0011/0026 | sealed-wrong |
|
||||
| wholesale multiplicative_aggregate | 0025/0047 sealed-wrong |
|
||||
|
||||
## 7. Research gate decision
|
||||
|
||||
**Proceed** with two independent singleton ClusterContracts (A2r + A2s). Calendar extension rejected for lack of sibling evidence.
|
||||
|
||||
## 8. Selected paradigm
|
||||
|
||||
Dual singleton organs under explicit contracts:
|
||||
|
||||
1. `generate/derivation/nested_fraction_remainder_total.py` — Gate A2r
|
||||
2. `generate/derivation/loose_crayon_box_capacity.py` — Gate A2s
|
||||
|
||||
Wired in `generate/math_candidate_graph.py` after Gate A2q.
|
||||
|
||||
## 9. Typed chains for target cases
|
||||
|
||||
**0004 (gold 2000):**
|
||||
|
||||
- afternoon = 750 kids (3/4 of soccer-camp subgroup)
|
||||
- subgroup = 750 / 0.75 = 1000
|
||||
- total = 1000 × 2 = 2000 (half go to soccer camp)
|
||||
|
||||
**0007 (gold 2):**
|
||||
|
||||
- boxed = 85 − 5 = 80
|
||||
- per_box = 80 / 5 = 16
|
||||
- loose_total = 5 + 27 = 32
|
||||
- boxes_needed = 32 / 16 = 2
|
||||
|
||||
## 10. Implemented organ(s)
|
||||
|
||||
| Module | Gate | Promotion entry |
|
||||
|--------|------|-----------------|
|
||||
| `nested_fraction_remainder_total.py` | A2r | `resolve_promotable_nested_fraction_remainder_total()` |
|
||||
| `loose_crayon_box_capacity.py` | A2s | `resolve_promotable_loose_crayon_box_capacity()` |
|
||||
|
||||
## 11. Before/after score
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|--------|-------:|------:|--:|
|
||||
| correct | 24 | **26** | +2 |
|
||||
| refused | 26 | **24** | −2 |
|
||||
| wrong | 0 | **0** | 0 |
|
||||
|
||||
**State A achieved:** 26/24/0.
|
||||
|
||||
## 12. Newly solved IDs
|
||||
|
||||
**0004**, **0007**
|
||||
|
||||
## 13. Preserved solved IDs
|
||||
|
||||
0001, 0002, 0003, 0005, 0006, 0008, 0009, 0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046
|
||||
|
||||
## 14. Wrong=0 proof
|
||||
|
||||
- Full train_sample replay: 26 / 24 / 0
|
||||
- `tests/test_math_candidate_graph_sprint12_singleton_contract_lift.py`
|
||||
- Holdout_dev: wrong=0
|
||||
- Regression lane: sprint6–12 + flywheel + scout + microscope
|
||||
|
||||
## 15. Confuser matrix
|
||||
|
||||
| Confuser | Organ | Result |
|
||||
|----------|-------|--------|
|
||||
| 0026 sealed_elimination | A2r | refuse |
|
||||
| asks subgroup only | A2r | refuse |
|
||||
| multiple fractions | A2r | refuse |
|
||||
| 0047 equal-distribution weight | A2s | refuse |
|
||||
| asks per-box capacity | A2s | refuse |
|
||||
| equal number in bags | A2s | refuse |
|
||||
|
||||
## 16. Sibling generality proof
|
||||
|
||||
| Sibling | Expected | Result |
|
||||
|---------|----------|--------|
|
||||
| music camp / 600 afternoon | 1600 | pass |
|
||||
| four marker boxes / 67 total | 6 boxes | pass |
|
||||
|
||||
## 17. Sealed-wrong negative evidence used
|
||||
|
||||
- 0026: currency + each + scoops + 3/4 bill → A2r hazard refusal
|
||||
- 0047: ounces + equal number + eaten bag → A2s hazard refusal
|
||||
|
||||
## 18. Composition-validation pin changes
|
||||
|
||||
None.
|
||||
|
||||
## 19. report.json untouched
|
||||
|
||||
Confirmed — no changes to committed `report.json`.
|
||||
|
||||
## 20. sealed artifacts untouched
|
||||
|
||||
Confirmed — no sealed lane artifact movement.
|
||||
|
||||
## 21. no case-id logic
|
||||
|
||||
Confirmed — admission via structural anchors and bindings only.
|
||||
|
||||
## 22. no hardcoded answer
|
||||
|
||||
Confirmed — arithmetic recomputed from grounded quantities.
|
||||
|
||||
## 23. no direct-answer shortcut
|
||||
|
||||
Confirmed — self-verified derivation chains required.
|
||||
|
||||
## 24. ClusterContract method enabled bigger lift
|
||||
|
||||
Yes — bundling two narrow singleton contracts under explicit neighbor exclusions delivered +2 with wrong=0, exceeding single-organ sprint cadence.
|
||||
|
||||
## 25. Sprint 13 recommendation
|
||||
|
||||
1. **Practice-lane mini-family clustering** — flywheel joint_refusal clusters (20 cases) may yield a 2-case mini-family safer than next singleton.
|
||||
2. **Deferred blocked surfaces** — DCS 0032/0047 and currency 0019/0028 remain blocked until dedicated confuser ADRs land.
|
||||
3. **Calendar contract** — revisit only if practice lane surfaces a genuine second piecewise sibling without policy expansion.
|
||||
4. Target: 27+/23-/0 via one mini-family or two verified singletons with lowest sealed-wrong adjacency (candidates from flywheel: 0004/0007 neighbors exhausted; next scout frontier TBD).
|
||||
285
generate/derivation/loose_crayon_box_capacity.py
Normal file
285
generate/derivation/loose_crayon_box_capacity.py
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
"""Gate A2s — loose crayon box capacity remainder (ClusterContract organ).
|
||||
|
||||
Sprint 12: train_sample **0007** — actor has ``N full boxes`` plus loose items,
|
||||
friend contributes loose items, conditional total count, derive per-box capacity
|
||||
from boxed portion, question asks ``how many more boxes`` for all loose items.
|
||||
|
||||
Chain (0007-class):
|
||||
|
||||
boxed = total − loose_primary
|
||||
per_box = boxed / num_full_boxes
|
||||
loose_total = loose_primary + loose_friend
|
||||
boxes_needed = loose_total / per_box
|
||||
|
||||
Algebraically (single fold):
|
||||
|
||||
boxes_needed = (loose_primary + loose_friend) × num_full_boxes / (total − loose_primary)
|
||||
|
||||
Narrow organ — not equal-distribution/DCS (0047), not currency, not weight surfaces.
|
||||
Promotion requires ClusterContract positive anchors and 0047-class hazard refusal.
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from typing import Final
|
||||
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification
|
||||
from generate.math_roundtrip import WORD_NUMBERS, _token_in, _tokens, _value_grounds
|
||||
|
||||
_FULL_BOXES_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)(\w+)\s+has\s+(\w+)\s+full\s+boxes\s+of\s+(\w+)"
|
||||
)
|
||||
_LOOSE_PRIMARY_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)(\d+)\s+loose\s+(\w+)"
|
||||
)
|
||||
_FRIEND_LOOSE_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)(?:\w+\s+)?friend\s+has\s+(\d+)\s+loose"
|
||||
)
|
||||
_TOTAL_CONDITIONAL_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)total\s+of\s+(\d+)"
|
||||
)
|
||||
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
"bags",
|
||||
"bakes",
|
||||
"brother",
|
||||
"eats",
|
||||
"equal",
|
||||
"macaroons",
|
||||
"ounces",
|
||||
"packs",
|
||||
"steve",
|
||||
"weighing",
|
||||
"weight",
|
||||
}
|
||||
)
|
||||
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{"ounces", "weight", "per", "macaroons"}
|
||||
)
|
||||
|
||||
|
||||
def _box_count_token(problem_text: str, num_boxes: float) -> str:
|
||||
match = _FULL_BOXES_RE.search(problem_text)
|
||||
if match is not None:
|
||||
return match.group(2).lower()
|
||||
return str(int(num_boxes))
|
||||
|
||||
|
||||
def _resolve_count(token: str) -> float | None:
|
||||
lowered = token.lower()
|
||||
if lowered in WORD_NUMBERS:
|
||||
return float(WORD_NUMBERS[lowered])
|
||||
try:
|
||||
return float(token)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _asks_more_boxes(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return (
|
||||
"how" in tokens
|
||||
and "many" in tokens
|
||||
and "more" in tokens
|
||||
and "boxes" in tokens
|
||||
)
|
||||
|
||||
|
||||
def _asks_per_box(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return "each" in tokens or "per" in tokens
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
if "$" in problem_text:
|
||||
return True
|
||||
text_tokens = _tokens(problem_text)
|
||||
question_tokens = _tokens(question_clause)
|
||||
if text_tokens & _TEXT_BLOCKERS:
|
||||
return True
|
||||
if question_tokens & _QUESTION_BLOCKERS:
|
||||
return True
|
||||
if _asks_per_box(question_clause):
|
||||
return True
|
||||
if "equal number" in problem_text.lower():
|
||||
return True
|
||||
if not _FULL_BOXES_RE.search(problem_text):
|
||||
return True
|
||||
if not _FRIEND_LOOSE_RE.search(problem_text):
|
||||
return True
|
||||
if not _TOTAL_CONDITIONAL_RE.search(question_clause):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _parse_bindings(problem_text: str) -> dict[str, float] | None:
|
||||
boxes_match = _FULL_BOXES_RE.search(problem_text)
|
||||
if boxes_match is None:
|
||||
return None
|
||||
num_boxes = _resolve_count(boxes_match.group(2))
|
||||
if num_boxes is None or num_boxes <= 0:
|
||||
return None
|
||||
|
||||
friend_match = _FRIEND_LOOSE_RE.search(problem_text)
|
||||
if friend_match is None:
|
||||
return None
|
||||
friend_pos = friend_match.start()
|
||||
|
||||
loose_primary_matches = [
|
||||
match
|
||||
for match in _LOOSE_PRIMARY_RE.finditer(problem_text)
|
||||
if match.start() < friend_pos
|
||||
]
|
||||
if len(loose_primary_matches) != 1:
|
||||
return None
|
||||
loose_primary = float(loose_primary_matches[0].group(1))
|
||||
loose_friend = float(friend_match.group(1))
|
||||
|
||||
total_match = _TOTAL_CONDITIONAL_RE.search(_question_clause(problem_text))
|
||||
if total_match is None:
|
||||
return None
|
||||
total = float(total_match.group(1))
|
||||
|
||||
item_unit = loose_primary_matches[0].group(2).lower()
|
||||
return {
|
||||
"num_boxes": num_boxes,
|
||||
"loose_primary": loose_primary,
|
||||
"loose_friend": loose_friend,
|
||||
"total": total,
|
||||
"item_unit": item_unit,
|
||||
}
|
||||
|
||||
|
||||
def _recompute_boxes_needed(bindings: dict[str, float]) -> float:
|
||||
boxed = bindings["total"] - bindings["loose_primary"]
|
||||
per_box = boxed / bindings["num_boxes"]
|
||||
loose_total = bindings["loose_primary"] + bindings["loose_friend"]
|
||||
return loose_total / per_box
|
||||
|
||||
|
||||
def build_loose_crayon_box_capacity(
|
||||
problem_text: str,
|
||||
) -> tuple[GroundedDerivation, float] | None:
|
||||
"""Construct the ungated box-capacity chain, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_more_boxes(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
|
||||
bindings = _parse_bindings(problem_text)
|
||||
if bindings is None:
|
||||
return None
|
||||
|
||||
boxed_total = bindings["total"] - bindings["loose_primary"]
|
||||
if boxed_total <= 0:
|
||||
return None
|
||||
|
||||
loose_total = bindings["loose_primary"] + bindings["loose_friend"]
|
||||
answer = _recompute_boxes_needed(bindings)
|
||||
|
||||
item_unit = bindings["item_unit"]
|
||||
derivation = GroundedDerivation(
|
||||
start=Quantity(
|
||||
value=bindings["loose_primary"],
|
||||
unit=item_unit,
|
||||
source_token=str(int(bindings["loose_primary"])),
|
||||
),
|
||||
steps=(
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(
|
||||
value=bindings["loose_friend"],
|
||||
unit=item_unit,
|
||||
source_token=str(int(bindings["loose_friend"])),
|
||||
),
|
||||
cue="friend",
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(
|
||||
value=bindings["num_boxes"],
|
||||
unit="boxes",
|
||||
source_token=_box_count_token(problem_text, bindings["num_boxes"]),
|
||||
),
|
||||
cue=_box_count_token(problem_text, bindings["num_boxes"]),
|
||||
),
|
||||
Step(
|
||||
op="divide",
|
||||
operand=Quantity(
|
||||
value=boxed_total,
|
||||
unit=item_unit,
|
||||
source_token=str(int(bindings["total"])),
|
||||
),
|
||||
cue="total",
|
||||
comparative=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
return derivation, answer
|
||||
|
||||
|
||||
def _self_verifies_box_capacity(
|
||||
derivation: GroundedDerivation, problem_text: str, expected: float
|
||||
) -> SelfVerification:
|
||||
tokens = _tokens(problem_text)
|
||||
reasons: list[str] = []
|
||||
|
||||
for q in [derivation.start, *(s.operand for s in derivation.steps if not s.comparative)]:
|
||||
if not _value_grounds(q.source_token, tokens):
|
||||
reasons.append(f"operand {q.source_token!r} not grounded in text")
|
||||
|
||||
for step in derivation.steps:
|
||||
if not _token_in(step.cue, tokens):
|
||||
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||
|
||||
bindings = _parse_bindings(problem_text)
|
||||
if bindings is None:
|
||||
reasons.append("missing box-capacity bindings")
|
||||
elif abs(_recompute_boxes_needed(bindings) - expected) > 1e-9:
|
||||
reasons.append("arithmetic mismatch on boxes needed")
|
||||
|
||||
if abs(derivation.answer - expected) > 1e-9:
|
||||
reasons.append("derivation fold mismatch")
|
||||
|
||||
obligation = Counter(q.source_token for q in extract_quantities(problem_text))
|
||||
used = Counter(
|
||||
[
|
||||
derivation.start.source_token,
|
||||
*(step.operand.source_token for step in derivation.steps),
|
||||
]
|
||||
)
|
||||
unused = obligation - used
|
||||
if unused:
|
||||
reasons.append(f"incomplete: unused quantities {sorted(unused.keys())}")
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_loose_crayon_box_capacity(problem_text: str) -> Resolution | None:
|
||||
"""Gate the typed box-capacity chain through self-verification."""
|
||||
built = build_loose_crayon_box_capacity(problem_text)
|
||||
if built is None:
|
||||
return None
|
||||
derivation, answer = built
|
||||
if not _self_verifies_box_capacity(derivation, problem_text, answer).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=answer,
|
||||
answer_unit="boxes",
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def resolve_promotable_loose_crayon_box_capacity(
|
||||
problem_text: str,
|
||||
) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2s, ClusterContract loose_crayon_box)."""
|
||||
return compose_loose_crayon_box_capacity(problem_text)
|
||||
260
generate/derivation/nested_fraction_remainder_total.py
Normal file
260
generate/derivation/nested_fraction_remainder_total.py
Normal file
|
|
@ -0,0 +1,260 @@
|
|||
"""Gate A2r — nested fraction complement remainder total (ClusterContract organ).
|
||||
|
||||
Sprint 12: train_sample **0004** — outer ``Half of`` partition, inner ``1/4 of``
|
||||
subgroup split (morning vs afternoon), known afternoon complement count, question
|
||||
asks root population ``altogether``.
|
||||
|
||||
Chain (0004-class):
|
||||
|
||||
subgroup = afternoon / (1 − inner_fraction)
|
||||
total = subgroup / outer_half_scalar = subgroup × 2
|
||||
|
||||
Narrow organ — not generic fraction parser, not sealed_elimination, not currency.
|
||||
Promotion requires ClusterContract positive anchors, explicit target binding, and
|
||||
0026-class hazard refusal.
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from typing import Final
|
||||
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification
|
||||
from generate.math_roundtrip import _token_in, _tokens, _value_grounds
|
||||
|
||||
_OUTER_HALF_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)\bhalf\s+of\s+the\s+(\w+)\b"
|
||||
)
|
||||
_INNER_QUARTER_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)\b1/4\s+of\s+the\s+(\w+)\s+going\s+to\s+(\w+)\s+camp\b"
|
||||
)
|
||||
_MORNING_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)\bgoing\s+to\s+(\w+)\s+camp\s+in\s+the\s+morning\b"
|
||||
)
|
||||
_AFTERNOON_COUNT_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(?i)(\d+)\s+(\w+)\s+are\s+going\s+to\s+(\w+)\s+camp\s+in\s+the\s+afternoon"
|
||||
)
|
||||
_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
"bill",
|
||||
"dinner",
|
||||
"dollars",
|
||||
"each",
|
||||
"ice",
|
||||
"money",
|
||||
"profit",
|
||||
"saved",
|
||||
"scoops",
|
||||
}
|
||||
)
|
||||
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{"afternoon", "morning", "scoops"}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _CampBinding:
|
||||
population: str
|
||||
camp: str
|
||||
afternoon_count: float
|
||||
afternoon_token: str
|
||||
|
||||
|
||||
def _asks_root_altogether(question_clause: str, population: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return (
|
||||
"how" in tokens
|
||||
and "many" in tokens
|
||||
and population in tokens
|
||||
and ("altogether" in tokens or "there" in tokens)
|
||||
)
|
||||
|
||||
|
||||
def _asks_subgroup_only(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return "camp" in tokens and "altogether" not in tokens
|
||||
|
||||
|
||||
def _parse_camp_binding(problem_text: str) -> _CampBinding | None:
|
||||
outer = _OUTER_HALF_RE.search(problem_text)
|
||||
inner = _INNER_QUARTER_RE.search(problem_text)
|
||||
morning = _MORNING_RE.search(problem_text)
|
||||
afternoon = _AFTERNOON_COUNT_RE.search(problem_text)
|
||||
if outer is None or inner is None or morning is None or afternoon is None:
|
||||
return None
|
||||
|
||||
population = outer.group(1).lower()
|
||||
inner_pop = inner.group(1).lower()
|
||||
inner_camp = inner.group(2).lower()
|
||||
morning_camp = morning.group(1).lower()
|
||||
afternoon_count = float(afternoon.group(1))
|
||||
afternoon_pop = afternoon.group(2).lower()
|
||||
afternoon_camp = afternoon.group(3).lower()
|
||||
|
||||
if not {population, inner_pop, afternoon_pop} == {population}:
|
||||
return None
|
||||
if not {inner_camp, morning_camp, afternoon_camp} == {inner_camp}:
|
||||
return None
|
||||
return _CampBinding(
|
||||
population=population,
|
||||
camp=inner_camp,
|
||||
afternoon_count=afternoon_count,
|
||||
afternoon_token=afternoon.group(1),
|
||||
)
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
if "$" in problem_text:
|
||||
return True
|
||||
text_tokens = _tokens(problem_text)
|
||||
question_tokens = _tokens(question_clause)
|
||||
if text_tokens & _TEXT_BLOCKERS:
|
||||
return True
|
||||
if question_tokens & _QUESTION_BLOCKERS:
|
||||
return True
|
||||
if _asks_subgroup_only(question_clause):
|
||||
return True
|
||||
if "3/4" in problem_text.replace(" ", ""):
|
||||
return True
|
||||
if "saved" in text_tokens and "each" in text_tokens:
|
||||
return True
|
||||
fractions = _EXTRA_FRACTION_RE.findall(problem_text)
|
||||
if len(fractions) != 1 or fractions[0].replace(" ", "") != "1/4":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _inner_complement_scalar() -> float:
|
||||
return 1.0 - (1.0 / 4.0)
|
||||
|
||||
|
||||
def _outer_inverse_half_scalar() -> float:
|
||||
return 2.0
|
||||
|
||||
|
||||
def _recompute_total(afternoon_count: float) -> float:
|
||||
subgroup = afternoon_count / _inner_complement_scalar()
|
||||
return subgroup * _outer_inverse_half_scalar()
|
||||
|
||||
|
||||
def build_nested_fraction_remainder_total(
|
||||
problem_text: str,
|
||||
) -> tuple[GroundedDerivation, float, _CampBinding] | None:
|
||||
"""Construct the ungated nested-fraction chain, or ``None``."""
|
||||
binding = _parse_camp_binding(problem_text)
|
||||
if binding is None:
|
||||
return None
|
||||
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_root_altogether(question_clause, binding.population):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
|
||||
afternoon = Quantity(
|
||||
value=binding.afternoon_count,
|
||||
unit=binding.population,
|
||||
source_token=binding.afternoon_token,
|
||||
)
|
||||
total = _recompute_total(binding.afternoon_count)
|
||||
derivation = GroundedDerivation(
|
||||
start=afternoon,
|
||||
steps=(
|
||||
Step(
|
||||
op="divide",
|
||||
operand=Quantity(
|
||||
value=_inner_complement_scalar(),
|
||||
unit="",
|
||||
source_token="1/4",
|
||||
),
|
||||
cue="4",
|
||||
comparative=True,
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(
|
||||
value=_outer_inverse_half_scalar(),
|
||||
unit="",
|
||||
source_token="half",
|
||||
),
|
||||
cue="half",
|
||||
comparative=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
return derivation, total, binding
|
||||
|
||||
|
||||
def _self_verifies_nested_fraction(
|
||||
derivation: GroundedDerivation,
|
||||
problem_text: str,
|
||||
expected_total: float,
|
||||
binding: _CampBinding,
|
||||
) -> SelfVerification:
|
||||
tokens = _tokens(problem_text)
|
||||
reasons: list[str] = []
|
||||
|
||||
if not _operand_grounds(derivation.start, tokens):
|
||||
reasons.append(f"operand {derivation.start.source_token!r} not grounded in text")
|
||||
|
||||
for step in derivation.steps:
|
||||
if not step.comparative and not _operand_grounds(step.operand, tokens):
|
||||
reasons.append(f"operand {step.operand.source_token!r} not grounded in text")
|
||||
if not _token_in(step.cue, tokens):
|
||||
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||
|
||||
if _parse_camp_binding(problem_text) is None:
|
||||
reasons.append("missing camp partition binding")
|
||||
elif abs(_recompute_total(binding.afternoon_count) - expected_total) > 1e-9:
|
||||
reasons.append("arithmetic mismatch on nested fraction total")
|
||||
|
||||
if abs(derivation.answer - expected_total) > 1e-9:
|
||||
reasons.append("derivation fold mismatch")
|
||||
|
||||
obligation = Counter(
|
||||
q.source_token
|
||||
for q in extract_quantities(problem_text)
|
||||
if q.unit == binding.population and q.source_token.isdigit()
|
||||
)
|
||||
used = Counter([derivation.start.source_token])
|
||||
unused = obligation - used
|
||||
if unused:
|
||||
reasons.append(
|
||||
f"incomplete: unused {binding.population} quantities {sorted(unused.keys())}"
|
||||
)
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def _operand_grounds(q: Quantity, tokens: frozenset[str]) -> bool:
|
||||
return _value_grounds(q.source_token, tokens)
|
||||
|
||||
|
||||
def compose_nested_fraction_remainder_total(problem_text: str) -> Resolution | None:
|
||||
"""Gate the typed nested-fraction chain through self-verification."""
|
||||
built = build_nested_fraction_remainder_total(problem_text)
|
||||
if built is None:
|
||||
return None
|
||||
derivation, total, binding = built
|
||||
if not _self_verifies_nested_fraction(derivation, problem_text, total, binding).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=total,
|
||||
answer_unit=binding.population,
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def resolve_promotable_nested_fraction_remainder_total(
|
||||
problem_text: str,
|
||||
) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2r, ClusterContract nested_fraction)."""
|
||||
return compose_nested_fraction_remainder_total(problem_text)
|
||||
|
|
@ -898,6 +898,38 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2r — nested fraction complement remainder total (Sprint 12 singleton).
|
||||
from generate.derivation.nested_fraction_remainder_total import (
|
||||
resolve_promotable_nested_fraction_remainder_total,
|
||||
)
|
||||
|
||||
nested_fraction_resolution = resolve_promotable_nested_fraction_remainder_total(
|
||||
text
|
||||
)
|
||||
if nested_fraction_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=nested_fraction_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2s — loose crayon box capacity (Sprint 12 singleton).
|
||||
from generate.derivation.loose_crayon_box_capacity import (
|
||||
resolve_promotable_loose_crayon_box_capacity,
|
||||
)
|
||||
|
||||
box_capacity_resolution = resolve_promotable_loose_crayon_box_capacity(text)
|
||||
if box_capacity_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=box_capacity_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
|
|||
summary = build_microscope_report(_load_cases())
|
||||
md = render_markdown(summary)
|
||||
assert "partition_chunking" in md
|
||||
assert "correct: 24" in md
|
||||
assert "correct: 26" in md
|
||||
assert "Gate A2a unit_partition" in md
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -218,8 +218,8 @@ class TestTrainSampleScore:
|
|||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 24
|
||||
assert counts["refused"] <= 26
|
||||
assert counts["correct"] >= 26
|
||||
assert counts["refused"] <= 24
|
||||
|
||||
def test_newly_solved_ids(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
|
|
|||
|
|
@ -232,8 +232,8 @@ class TestTrainSampleScore:
|
|||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 24
|
||||
assert counts["refused"] <= 26
|
||||
assert counts["correct"] >= 26
|
||||
assert counts["refused"] <= 24
|
||||
|
||||
def test_newly_solved_0013(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
|
|
|||
|
|
@ -0,0 +1,231 @@
|
|||
"""Gate A2r/A2s — Capability Paradigm Sprint 12 singleton ClusterContract lift."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.derivation.nested_fraction_remainder_total import (
|
||||
compose_nested_fraction_remainder_total,
|
||||
resolve_promotable_nested_fraction_remainder_total,
|
||||
)
|
||||
from generate.derivation.loose_crayon_box_capacity import (
|
||||
compose_loose_crayon_box_capacity,
|
||||
resolve_promotable_loose_crayon_box_capacity,
|
||||
)
|
||||
|
||||
_CASES_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "evals"
|
||||
/ "gsm8k_math"
|
||||
/ "train_sample"
|
||||
/ "v1"
|
||||
/ "cases.jsonl"
|
||||
)
|
||||
|
||||
CASE_0004 = (
|
||||
"There are some kids in camp. Half of the kids are going to soccer camp, "
|
||||
"and 1/4 of the kids going to soccer camp are going to soccer camp in the morning. "
|
||||
"750 kids are going to soccer camp in the afternoon. "
|
||||
"How many kids there in camp altogether?"
|
||||
)
|
||||
|
||||
CASE_0007 = (
|
||||
"Francine has five full boxes of crayons and 5 loose crayons, and her friend "
|
||||
"has 27 loose crayons. They need to put all of their loose crayons in a box. "
|
||||
"How many more boxes do they need if Francine has a total of 85 crayons?"
|
||||
)
|
||||
|
||||
SIBLING_CAMP = (
|
||||
"There are some students in camp. Half of the students are going to music camp, "
|
||||
"and 1/4 of the students going to music camp are going to music camp in the morning. "
|
||||
"600 students are going to music camp in the afternoon. "
|
||||
"How many students there in camp altogether?"
|
||||
)
|
||||
|
||||
SIBLING_BOXES = (
|
||||
"Jordan has four full boxes of markers and 3 loose markers, and her friend "
|
||||
"has 21 loose markers. They need to put all of their loose markers in a box. "
|
||||
"How many more boxes do they need if Jordan has a total of 19 markers?"
|
||||
)
|
||||
|
||||
SEALED_ELIMINATION_0026 = (
|
||||
"Aaron and his brother Carson each saved up $40 to go to dinner. "
|
||||
"The bill is 3/4 of their total money. After, they go out for ice cream. "
|
||||
"Each scoop costs $1.5 and they get the same amount as each other. "
|
||||
"If they leave with $1 in change each, how many scoops did they each buy?"
|
||||
)
|
||||
|
||||
SEALED_WRONG_0047_SHAPE = (
|
||||
"John bakes 12 coconut macaroons, each weighing 5 ounces. "
|
||||
"He then packs an equal number of the macaroons in 4 different brown bags, "
|
||||
"ready for delivery. When he briefly leaves the kitchen to pick the phone, "
|
||||
"his little brother Steve eats the entire contents of one of the brown bags. "
|
||||
"What is the total weight, in ounces, of the remaining coconut macaroons?"
|
||||
)
|
||||
|
||||
ASKS_SUBGROUP_ONLY_CONFUSER = (
|
||||
"There are some kids in camp. Half of the kids are going to soccer camp, "
|
||||
"and 1/4 of the kids going to soccer camp are going to soccer camp in the morning. "
|
||||
"750 kids are going to soccer camp in the afternoon. "
|
||||
"How many kids are going to soccer camp?"
|
||||
)
|
||||
|
||||
ASKS_PER_BOX_CONFUSER = (
|
||||
"Francine has five full boxes of crayons and 5 loose crayons, and her friend "
|
||||
"has 27 loose crayons. They need to put all of their loose crayons in a box. "
|
||||
"How many crayons are in each full box if Francine has a total of 85 crayons?"
|
||||
)
|
||||
|
||||
MULTIPLE_FRACTION_CONFUSER = (
|
||||
"Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp "
|
||||
"are going in the morning. 1/2 of the afternoon group brought lunch. "
|
||||
"750 kids are going to soccer camp in the afternoon. "
|
||||
"How many kids there in camp altogether?"
|
||||
)
|
||||
|
||||
PRESERVED_SOLVED = (
|
||||
"0001",
|
||||
"0002",
|
||||
"0003",
|
||||
"0005",
|
||||
"0006",
|
||||
"0008",
|
||||
"0009",
|
||||
"0010",
|
||||
"0013",
|
||||
"0014",
|
||||
"0015",
|
||||
"0017",
|
||||
"0018",
|
||||
"0021",
|
||||
"0024",
|
||||
"0025",
|
||||
"0029",
|
||||
"0030",
|
||||
"0035",
|
||||
"0037",
|
||||
"0038",
|
||||
"0042",
|
||||
"0045",
|
||||
"0046",
|
||||
)
|
||||
|
||||
|
||||
def _load_case(case_suffix: str) -> str:
|
||||
for line in _CASES_PATH.read_text().splitlines():
|
||||
row = json.loads(line)
|
||||
if row["case_id"].endswith(case_suffix):
|
||||
return row["question"]
|
||||
raise KeyError(case_suffix)
|
||||
|
||||
|
||||
class TestClusterContractPolicy:
|
||||
def test_nested_fraction_contract_admits_0004_shape(self) -> None:
|
||||
assert compose_nested_fraction_remainder_total(CASE_0004) is not None
|
||||
|
||||
def test_box_capacity_contract_admits_0007_shape(self) -> None:
|
||||
assert compose_loose_crayon_box_capacity(CASE_0007) is not None
|
||||
|
||||
|
||||
class TestTargetCases:
|
||||
def test_0004_solves(self) -> None:
|
||||
result = parse_and_solve(CASE_0004)
|
||||
assert result.answer == 2000.0
|
||||
assert result.refusal_reason is None
|
||||
|
||||
def test_0007_solves(self) -> None:
|
||||
result = parse_and_solve(CASE_0007)
|
||||
assert result.answer == 2.0
|
||||
assert result.refusal_reason is None
|
||||
|
||||
|
||||
class TestSiblingGeneralization:
|
||||
def test_music_camp_sibling(self) -> None:
|
||||
resolution = compose_nested_fraction_remainder_total(SIBLING_CAMP)
|
||||
assert resolution is not None
|
||||
assert resolution.answer == 1600.0
|
||||
|
||||
def test_marker_box_sibling(self) -> None:
|
||||
resolution = compose_loose_crayon_box_capacity(SIBLING_BOXES)
|
||||
assert resolution is not None
|
||||
# (3+21)*4/(67-3) = 96/16 = 6
|
||||
assert resolution.answer == 6.0
|
||||
|
||||
|
||||
class TestNeighborConfuserRefusals:
|
||||
def test_sealed_elimination_0026_refuses(self) -> None:
|
||||
assert compose_nested_fraction_remainder_total(SEALED_ELIMINATION_0026) is None
|
||||
|
||||
def test_dcs_0047_refuses(self) -> None:
|
||||
assert compose_loose_crayon_box_capacity(SEALED_WRONG_0047_SHAPE) is None
|
||||
|
||||
def test_asks_subgroup_only_refuses(self) -> None:
|
||||
assert compose_nested_fraction_remainder_total(ASKS_SUBGROUP_ONLY_CONFUSER) is None
|
||||
|
||||
def test_asks_per_box_refuses(self) -> None:
|
||||
assert compose_loose_crayon_box_capacity(ASKS_PER_BOX_CONFUSER) is None
|
||||
|
||||
def test_multiple_fraction_refuses(self) -> None:
|
||||
assert compose_nested_fraction_remainder_total(MULTIPLE_FRACTION_CONFUSER) is None
|
||||
|
||||
|
||||
class TestPromotionBridges:
|
||||
def test_resolve_promotable_nested_fraction(self) -> None:
|
||||
assert resolve_promotable_nested_fraction_remainder_total(CASE_0004) is not None
|
||||
|
||||
def test_resolve_promotable_box_capacity(self) -> None:
|
||||
assert resolve_promotable_loose_crayon_box_capacity(CASE_0007) is not None
|
||||
|
||||
|
||||
class TestTrainSampleScore:
|
||||
def test_wrong_zero_and_state_a(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 26
|
||||
assert counts["refused"] <= 24
|
||||
|
||||
def test_newly_solved_ids(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
correct = {
|
||||
row["case_id"].split("-")[-1]
|
||||
for row in report["per_case"]
|
||||
if row["verdict"] == "correct"
|
||||
}
|
||||
assert "0004" in correct
|
||||
assert "0007" in correct
|
||||
|
||||
def test_preserved_solved_ids(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
correct = {
|
||||
row["case_id"].split("-")[-1]
|
||||
for row in report["per_case"]
|
||||
if row["verdict"] == "correct"
|
||||
}
|
||||
for case_id in PRESERVED_SOLVED:
|
||||
assert case_id in correct
|
||||
|
||||
|
||||
class TestHoldoutInert:
|
||||
def test_holdout_dev_wrong_zero(self) -> None:
|
||||
from evals.gsm8k_math.holdout_dev.v1.runner import build_report
|
||||
|
||||
report = build_report()
|
||||
assert report["counts"]["wrong"] == 0
|
||||
|
||||
|
||||
class TestPriorOrganRegressions:
|
||||
def test_piecewise_0013_preserved(self) -> None:
|
||||
result = parse_and_solve(_load_case("0013"))
|
||||
assert result.answer == 450.0
|
||||
assert result.refusal_reason is None
|
||||
Loading…
Reference in a new issue