feat(derivation): capability paradigm sprint 9 temporal tariff + affine lift (#820)
Gate A2m promotes narrow temporal_tariff organs for overtime shift earnings (0001) and bundle-overflow rental tariffs (0017). Gate A2n promotes affine fraction-delta chains for 0010-class references (reference × N/M + K). Serving moves 18/32/0 → 21/29/0 with wrong=0 preserved. Hardens percent_partition to refuse subgroup-as-total confusers surfaced in review.
This commit is contained in:
Shay
GitHub
parent
282c5126de
commit
63a9bea823
11 changed files with 1107 additions and 12 deletions
|
|
@ -0,0 +1,183 @@
|
|||
# GSM8K Capability Paradigm Sprint 9 — Lookback (2026-06-17)
|
||||
|
||||
## 1. Starting baseline
|
||||
|
||||
| Metric | Value |
|
||||
|--------|------:|
|
||||
| correct | **18** |
|
||||
| refused | **32** |
|
||||
| wrong | **0** |
|
||||
|
||||
After #819 (Sprint 8: fraction_decrease + percent_partition). Preserved solved: 0002, 0003, 0005, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046.
|
||||
|
||||
## 2. Experience Flywheel output summary
|
||||
|
||||
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint9.json`
|
||||
|
||||
| Field | Value |
|
||||
|-------|------:|
|
||||
| retained_record_count | 50 |
|
||||
| lift_refused_to_correct | **0** |
|
||||
| promotion_candidates | 4 families, **all** `blocked_by_wrong_risk` |
|
||||
|
||||
Top missing primitives: `temporal_tariff` (0001, 0017), `relation_hypothesis` (DCS), `diagnostic_hold` (currency).
|
||||
|
||||
**Key signal:** `temporal_tariff:temporal_aggregation` cluster (0001, 0017) has **zero sealed-wrong** neighbors — joint refusal only. `affine_equation_fraction_delta` (0010) sits in `joint_sealed_no_resolution` with zero sealed-wrong. Both families outrank blocked DCS/currency clusters for Sprint 9.
|
||||
|
||||
## 3. Scout/frontier/microscope output summary
|
||||
|
||||
| Regime | correct | refused | wrong |
|
||||
|--------|--------:|--------:|------:|
|
||||
| Serving | 18 | 32 | 0 |
|
||||
| Sealed (resolve_pooled) | 3 | 39 | 8 |
|
||||
|
||||
- `lift_refused_to_correct`: **0**
|
||||
- Dominant bottleneck: joint refusal (26/50)
|
||||
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
|
||||
|
||||
Microscope slices: `no_injection_temporal_aggregation` (0001, 0017), `affine_equation_fraction_delta` (0010).
|
||||
|
||||
## 4. Candidate families ranked
|
||||
|
||||
| Rank | Family | Cases | Lift signal | Wrong-risk | Sprint 9 verdict |
|
||||
|------|--------|-------|-------------|------------|------------------|
|
||||
| 1 | `temporal_tariff` overtime | **0001** | typed threshold + ½ OT + days | low | **Selected (A2m-A)** |
|
||||
| 2 | `temporal_tariff` bundle overflow | **0017** | typed per-day-or-bundle + rental days | low | **Selected (A2m-B)** |
|
||||
| 3 | `affine_fraction_delta` | **0010** | typed N/M-more-than-reference + plus K | low | **Selected (A2n)** |
|
||||
| 4 | `multiplicative_aggregate` | 0006, 0013 | joint only | medium | Deferred |
|
||||
| 5 | `relation_hypothesis:DCS` | 14 joint + 2 wrong | none | **blocked** (0032, 0047) | Deferred |
|
||||
| 6 | `diagnostic_hold:currency_amount` | 0019, 0028 | sealed-wrong | **blocked** | Deferred |
|
||||
|
||||
## 5. Blocked families and why
|
||||
|
||||
| Family | Blocked because |
|
||||
|--------|-----------------|
|
||||
| `relation_hypothesis:DCS` | 0032, 0047 sealed-wrong share discrete-count / partition surface |
|
||||
| `conservative_boundary` | 0018, 0025 sealed-wrong on served-correct cases |
|
||||
| `sealed_elimination` | 0011, 0026 hard elimination |
|
||||
| `diagnostic_hold:currency_amount` | 0019, 0028 sealed-wrong; vet-insurance / profit-recovery shapes |
|
||||
|
||||
## 6. Selected paradigm
|
||||
|
||||
**Dual-organ Sprint 9** (Sprint 8 handoff targets; no cleaner family outranked on wrong-risk):
|
||||
|
||||
1. **Gate A2m `temporal_tariff`** — two typed sub-patterns in one module:
|
||||
- **Overtime shift earnings:** hourly rate + `more than N hours` threshold + `1/2` OT premium + hours/day × days.
|
||||
- **Bundle overflow tariff:** `$A per day or $B for C days` + rental days `> C` → `B + (rental−C)×A`.
|
||||
2. **Gate A2n `affine_fraction_delta`** — prior initial-loss mutation establishes reference; `N/M more than what <entity> currently has, plus K` → `reference × (N/M) + K` (GSM8K gold semantics for 0010).
|
||||
|
||||
Rejected: broad temporal parser, generic affine equation parser, broad product_bridge, DCS injector.
|
||||
|
||||
**Research note:** No alternative family offered higher lift with lower wrong-risk than the Sprint 8 handoff targets. `multiplicative_aggregate` remains joint-only without end-to-end typed chains.
|
||||
|
||||
## 7. Typed chains for target cases
|
||||
|
||||
**0001 (gold 990):** Regular `8×18×5=720` + OT `(10−8)×(18+9)×5=270` → **990**.
|
||||
|
||||
**0017 (gold 800):** Bundle `500` + overflow `(20−14)×50=300` → **800**.
|
||||
|
||||
**0010 (gold 9):** Yun current `20−12=8` → Marion `8×(1/4)+7=9`.
|
||||
|
||||
## 8. Implemented organs
|
||||
|
||||
| Module | Promotion entry |
|
||||
|--------|-----------------|
|
||||
| `generate/derivation/temporal_tariff.py` | `resolve_promotable_temporal_tariff` |
|
||||
| `generate/derivation/affine_fraction_delta.py` | `resolve_promotable_affine_fraction_delta` |
|
||||
|
||||
Wired in `generate/math_candidate_graph.py` after Gate A2l, before rate short-circuits.
|
||||
|
||||
**Review hardening:** `percent_partition` now refuses subgroup-quantity-as-total confusers (`"girls group has 50"`).
|
||||
|
||||
## 9. Before/after score
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|--------|-------:|------:|--:|
|
||||
| correct | 18 | **21** | +3 |
|
||||
| refused | 32 | **29** | −3 |
|
||||
| wrong | 0 | **0** | 0 |
|
||||
|
||||
**State A exceeded:** 21/29/0 (target was 20+/30-/0).
|
||||
|
||||
## 10. Newly solved IDs
|
||||
|
||||
**0001**, **0010**, **0017**
|
||||
|
||||
## 11. Preserved solved IDs
|
||||
|
||||
0002, 0003, 0005, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046
|
||||
|
||||
## 12. Wrong=0 proof
|
||||
|
||||
- Full train_sample replay: 21 correct / 29 refused / 0 wrong
|
||||
- `tests/test_math_candidate_graph_sprint9_temporal_affine_lift.py::TestTrainSampleScore`
|
||||
- Composition-validation firewall: 14 solve / 8 refuse / 0 wrong
|
||||
- Holdout_dev: 0 new organ admissions
|
||||
- Smoke suite: 108 passed
|
||||
- Full regression lane: 313 passed
|
||||
|
||||
## 13. Confuser matrix
|
||||
|
||||
| Confuser | Organ | Result |
|
||||
|----------|-------|--------|
|
||||
| 0019 vet + insurance | A2m | refused |
|
||||
| 0028 amusement profit recovery | A2m | refused |
|
||||
| OT asks hours not money | A2m | refused |
|
||||
| Bundle rental ≤ bundle period | A2m | refused |
|
||||
| 0005 fraction-decrease | A2n | refused (A2k serves) |
|
||||
| Twice-as-many comparative | A2n | refused |
|
||||
| 0032 percent time-decrease | A2m | refused |
|
||||
| Subgroup-as-total percent partition | A2l | refused (hardened) |
|
||||
|
||||
## 14. Sibling generality proof
|
||||
|
||||
| Sibling | Gold | Organ |
|
||||
|---------|-----:|-------|
|
||||
| Rosa $20/hr, 6hr threshold, 9hr×4 days | 840 | A2m overtime |
|
||||
| $40/day or $300/10d, 15-day rental | 500 | A2m bundle |
|
||||
| Sam 20−4 marbles, 1/4 more + 3 | 7 | A2n affine |
|
||||
|
||||
## 15. Sealed-wrong negative evidence used
|
||||
|
||||
- 0019/0028 currency_amount flywheel `blocked_by_wrong_risk` → A2m hazard blockers (insurance, profit, percent, amusement, vet)
|
||||
- 0032 percent-decrease → A2m hazard refusal
|
||||
- DCS 0032/0047 → no partition/tariff broadening
|
||||
|
||||
## 16. Composition-validation pin changes
|
||||
|
||||
No new cv pins flipped this sprint (0001/0010/0017 lack dedicated cv rows). Aggregate snapshot unchanged: **14/8/0**.
|
||||
|
||||
## 17. report.json untouched
|
||||
|
||||
`evals/gsm8k_math/train_sample/v1/report.json` not modified.
|
||||
|
||||
## 18. Sealed artifacts untouched
|
||||
|
||||
No changes to sealed practice lanes, holdout, or `resolve_pooled` wiring.
|
||||
|
||||
## 19. No case-id logic
|
||||
|
||||
Promotion is family-typed only; no `case_id` conditionals in serving path.
|
||||
|
||||
## 20. No hardcoded answer
|
||||
|
||||
Answers derived from grounded arithmetic chains; no answer literals.
|
||||
|
||||
## 21. No direct-answer shortcut
|
||||
|
||||
No `answer_numeric` extraction or final-answer parsing.
|
||||
|
||||
## 22. Sprint 10 recommendation
|
||||
|
||||
**Track A — Capability Sprint 10** (continue capability paradigm)
|
||||
|
||||
| Evidence | Detail |
|
||||
|----------|--------|
|
||||
| Flywheel | `lift_refused_to_correct == 0`; 29 refusals remain, mostly `joint_sealed_no_resolution` |
|
||||
| Next targets | `multiplicative_aggregate` cluster (0006, 0013) — only with typed end-to-end chain + confuser matrix |
|
||||
| Secondary | `joint_sealed_no_resolution` singletons with clean decomposition (0004 fraction, 0007 divisive, 0009 additive) — case-by-case after scout decomposition |
|
||||
| Blocked | DCS (0032/0047), currency_amount (0019/0028), sealed_elimination (0011/0026) |
|
||||
| Expected lift | +1–2 correct if MA organ passes adversarial review; else diagnostic hardening |
|
||||
| Non-goals | Flywheel PR-2, broad product_bridge, DCS injector, report.json rebaseline |
|
||||
|
||||
Track B (Experience Flywheel PR-2) deferred until joint-refusal cluster lacks typed promotable chains.
|
||||
236
generate/derivation/affine_fraction_delta.py
Normal file
236
generate/derivation/affine_fraction_delta.py
Normal file
|
|
@ -0,0 +1,236 @@
|
|||
"""Gate A2n — affine fraction-delta (fraction-of-reference plus offset).
|
||||
|
||||
Experience Flywheel Sprint 9 / microscope ``affine_equation_fraction_delta``
|
||||
(train_sample **0010**): a prior clause establishes a reference entity's current
|
||||
amount (including initial-loss mutation); a follow-on clause states
|
||||
``N/M more than what <entity> currently has, plus K``; the question asks the
|
||||
subject's total count.
|
||||
|
||||
GSM8K gold for this family computes:
|
||||
|
||||
answer = reference × (N/M) + K
|
||||
|
||||
not ``reference × (1 + N/M)``. Narrow organ — not ``decrease to N/M of``
|
||||
(fraction_decrease), not comparative ``twice as many``, not a generic affine
|
||||
equation parser. Promotion requires:
|
||||
|
||||
- exactly one ``N/M more than what <entity> currently has`` clause;
|
||||
- explicit ``plus K`` offset in the same clause;
|
||||
- resolvable reference quantity from a single prior initial-mutation clause;
|
||||
- question binds ``how many`` + possession cue for the affine subject;
|
||||
- hazard refusal (decrease-to fraction, percent, goal language, multiple fractions).
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Final
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from generate.derivation.clauses import segment_clauses
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification
|
||||
from generate.math_candidate_parser import _init_mutation_admitted
|
||||
from generate.math_roundtrip import _token_in, _tokens, _value_grounds
|
||||
|
||||
_AFFINE_FRACTION_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(\d+)\s*/\s*(\d+)\s+more\s+than\s+what\s+(\w+)\s+currently\s+has,\s+plus\s+(\d+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_DECREASE_TO_FRACTION_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"decrease\s+to\s+\d+\s*/\s*\d+\s+of",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||
_GOAL_INTENT: Final[frozenset[str]] = frozenset(
|
||||
{"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"}
|
||||
)
|
||||
_POSSESSION_CUES: Final[frozenset[str]] = frozenset({"has", "have", "had"})
|
||||
|
||||
|
||||
def _asks_subject_total(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return "how" in tokens and "many" in tokens and bool(_POSSESSION_CUES & tokens)
|
||||
|
||||
|
||||
def _affine_match(problem_text: str) -> re.Match[str] | None:
|
||||
matches = list(_AFFINE_FRACTION_RE.finditer(problem_text))
|
||||
if len(matches) != 1:
|
||||
return None
|
||||
return matches[0]
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
if _DECREASE_TO_FRACTION_RE.search(problem_text):
|
||||
return True
|
||||
text_tokens = _tokens(problem_text)
|
||||
question_tokens = _tokens(question_clause)
|
||||
if "%" in problem_text or "percent" in text_tokens or "percentage" in text_tokens:
|
||||
return True
|
||||
if text_tokens & _GOAL_INTENT:
|
||||
return True
|
||||
if "twice" in text_tokens or "thrice" in text_tokens or "double" in text_tokens:
|
||||
return True
|
||||
match = _affine_match(problem_text)
|
||||
if match is None:
|
||||
return True
|
||||
fractions = _EXTRA_FRACTION_RE.findall(problem_text)
|
||||
token = f"{match.group(1)}/{match.group(2)}"
|
||||
extra = [f for f in fractions if f.replace(" ", "") != token.replace(" ", "")]
|
||||
return len(extra) > 0
|
||||
|
||||
|
||||
def _reference_candidate(problem_text: str, entity: str):
|
||||
question_clause = _question_clause(problem_text)
|
||||
match = _affine_match(problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
affine_clause = match.group(0)
|
||||
for clause in segment_clauses(problem_text):
|
||||
if clause == question_clause or affine_clause in clause:
|
||||
continue
|
||||
for candidate in _init_mutation_admitted(clause):
|
||||
if candidate.initial.entity.lower() != entity.lower():
|
||||
continue
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _reference_quantity(problem_text: str, entity: str) -> Quantity | None:
|
||||
candidate = _reference_candidate(problem_text, entity)
|
||||
if candidate is None:
|
||||
return None
|
||||
qty = candidate.initial.quantity
|
||||
return Quantity(
|
||||
value=qty.value,
|
||||
unit=qty.unit,
|
||||
source_token=candidate.matched_value_token,
|
||||
)
|
||||
|
||||
|
||||
def build_affine_fraction_delta(problem_text: str) -> GroundedDerivation | None:
|
||||
"""Construct ``reference × (N/M) + K``, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_subject_total(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
|
||||
match = _affine_match(problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
|
||||
num_s, den_s, entity, offset_s = (
|
||||
match.group(1),
|
||||
match.group(2),
|
||||
match.group(3),
|
||||
match.group(4),
|
||||
)
|
||||
try:
|
||||
num = int(num_s)
|
||||
den = int(den_s)
|
||||
offset = float(offset_s)
|
||||
except ValueError:
|
||||
return None
|
||||
if num <= 0 or den <= 0:
|
||||
return None
|
||||
|
||||
reference = _reference_quantity(problem_text, entity)
|
||||
if reference is None:
|
||||
return None
|
||||
|
||||
factor = num / den
|
||||
fraction_token = f"{num_s}/{den_s}"
|
||||
|
||||
return GroundedDerivation(
|
||||
start=reference,
|
||||
steps=(
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(value=factor, unit="", source_token=fraction_token),
|
||||
cue="more",
|
||||
),
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(value=offset, unit=reference.unit, source_token=offset_s),
|
||||
cue="plus",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _self_verifies_affine_fraction_delta(
|
||||
derivation: GroundedDerivation, problem_text: str
|
||||
) -> SelfVerification:
|
||||
reasons: list[str] = []
|
||||
tokens = _tokens(problem_text)
|
||||
question_clause = _question_clause(problem_text)
|
||||
|
||||
operands = [derivation.start, *(s.operand for s in derivation.steps if not s.comparative)]
|
||||
for q in operands:
|
||||
if not _value_grounds(q.source_token, tokens):
|
||||
reasons.append(f"operand {q.source_token!r} not grounded in text")
|
||||
|
||||
for step in derivation.steps:
|
||||
if not _token_in(step.cue, tokens):
|
||||
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||
|
||||
primary_unit = derivation.start.unit
|
||||
for step in derivation.steps:
|
||||
if step.op in {"add", "subtract"} and step.operand.unit != primary_unit:
|
||||
reasons.append(
|
||||
f"unit mismatch: {step.op} of {step.operand.unit!r} into {primary_unit!r}"
|
||||
)
|
||||
|
||||
match = _affine_match(problem_text)
|
||||
if match is None:
|
||||
reasons.append("missing affine fraction clause")
|
||||
else:
|
||||
reference = _reference_quantity(problem_text, match.group(3))
|
||||
if reference is None:
|
||||
reasons.append("missing reference initial-mutation quantity")
|
||||
|
||||
body = problem_text.replace(question_clause, "").strip()
|
||||
body_quantities = Counter(q.source_token for q in extract_quantities(body))
|
||||
used = Counter(
|
||||
[derivation.start.source_token]
|
||||
+ [step.operand.source_token for step in derivation.steps]
|
||||
)
|
||||
if match is not None:
|
||||
used[match.group(4)] += 1
|
||||
used[f"{match.group(1)}/{match.group(2)}"] += 1
|
||||
used[match.group(1)] += 1
|
||||
used[match.group(2)] += 1
|
||||
reference_candidate = _reference_candidate(problem_text, match.group(3))
|
||||
if reference_candidate is not None:
|
||||
for token in reference_candidate.consumed_value_tokens:
|
||||
used[token] += 1
|
||||
unused = body_quantities - used
|
||||
if unused:
|
||||
reasons.append(f"incomplete: unused body quantities {sorted(unused.elements())}")
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_affine_fraction_delta(problem_text: str) -> Resolution | None:
|
||||
"""Gate the typed affine fraction-delta chain through self-verification."""
|
||||
derivation = build_affine_fraction_delta(problem_text)
|
||||
if derivation is None:
|
||||
return None
|
||||
if not _self_verifies_affine_fraction_delta(derivation, problem_text).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=derivation.answer,
|
||||
answer_unit=derivation.answer_unit,
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def resolve_promotable_affine_fraction_delta(problem_text: str) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2n)."""
|
||||
return compose_affine_fraction_delta(problem_text)
|
||||
|
|
@ -82,6 +82,8 @@ def _total_population(problem_text: str, question_clause: str) -> Quantity | Non
|
|||
lowered = clause.lower()
|
||||
if "half" in lowered or _PERCENT_OF_GROUP_RE.search(clause):
|
||||
continue
|
||||
if "group" in lowered:
|
||||
continue
|
||||
quantities = [q for q in extract_quantities(clause) if q.unit]
|
||||
if len(quantities) == 1:
|
||||
return quantities[0]
|
||||
|
|
|
|||
370
generate/derivation/temporal_tariff.py
Normal file
370
generate/derivation/temporal_tariff.py
Normal file
|
|
@ -0,0 +1,370 @@
|
|||
"""Gate A2m — temporal tariff (overtime shift earnings + bundle overflow tariff).
|
||||
|
||||
Experience Flywheel Sprint 9 / microscope ``temporal_tariff:temporal_aggregation``
|
||||
(train_sample **0001**, **0017**): two narrow tariff shapes, not a generic time
|
||||
parser.
|
||||
|
||||
**Pattern A — overtime shift earnings (0001)**
|
||||
|
||||
regular = threshold_hours × hourly_rate × days
|
||||
overtime_total = (hours_per_day − threshold_hours) × (rate + rate × ½) × days
|
||||
total = regular + overtime_total
|
||||
|
||||
**Pattern B — bundle overflow tariff (0017)**
|
||||
|
||||
overflow_cost = (rental_days − bundle_days) × daily_rate
|
||||
total = bundle_price + overflow_cost
|
||||
|
||||
Promotion requires explicit positive anchors per pattern, question-target binding
|
||||
(money/cost), and hazard refusal (profit, insurance, percent, multiple tariffs).
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Final
|
||||
|
||||
from collections import Counter
|
||||
|
||||
from generate.derivation.clauses import segment_clauses
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification
|
||||
from generate.math_roundtrip import _token_in, _tokens, _value_grounds
|
||||
|
||||
_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||
_HOURLY_RATE_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"\$\s*(\d+(?:\.\d+)?)\s*(?:an\s+hour|/hour|per\s+hour)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_THRESHOLD_HOURS_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"more\s+than\s+(\d+)\s+hours?",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_BUNDLE_TARIFF_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"\$\s*(\d+(?:\.\d+)?)\s+per\s+day\s+or\s+\$\s*(\d+(?:\.\d+)?)\s+for\s+(\d+)\s+days?",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_EARN_VERBS: Final[frozenset[str]] = frozenset(
|
||||
{"earn", "earned", "make", "made", "receive", "received", "get", "got"}
|
||||
)
|
||||
_COST_VERBS: Final[frozenset[str]] = frozenset({"cost", "costs", "charge", "charges", "pay", "paid"})
|
||||
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
"insurance",
|
||||
"percent",
|
||||
"percentage",
|
||||
"profit",
|
||||
"ticket",
|
||||
"tickets",
|
||||
"vet",
|
||||
"amusement",
|
||||
}
|
||||
)
|
||||
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset({"long", "take"})
|
||||
|
||||
|
||||
def _asks_money_total(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
if "money" in tokens and bool(_EARN_VERBS & tokens):
|
||||
return True
|
||||
if "how" in tokens and "much" in tokens and bool(_COST_VERBS & tokens):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _body_text(problem_text: str) -> str:
|
||||
question_clause = _question_clause(problem_text)
|
||||
return problem_text.replace(question_clause, "").strip()
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
if _FRACTION_RE.search(problem_text) and "overtime" not in _tokens(problem_text):
|
||||
return True
|
||||
text_tokens = _tokens(problem_text)
|
||||
question_tokens = _tokens(question_clause)
|
||||
if "%" in problem_text or "percent" in text_tokens:
|
||||
return True
|
||||
if text_tokens & _TEXT_BLOCKERS:
|
||||
return True
|
||||
if question_tokens & _QUESTION_BLOCKERS:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _half_fraction_operand(problem_text: str) -> Quantity | None:
|
||||
match = re.search(r"1\s*/\s*2", problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
return Quantity(value=0.5, unit="", source_token=match.group(0).replace(" ", ""))
|
||||
|
||||
|
||||
def _hourly_rate(problem_text: str) -> Quantity | None:
|
||||
match = _HOURLY_RATE_RE.search(problem_text)
|
||||
if match is None:
|
||||
match = re.search(r"makes\s+\$\s*(\d+(?:\.\d+)?)", problem_text, re.IGNORECASE)
|
||||
if match is None:
|
||||
return None
|
||||
token = match.group(1)
|
||||
return Quantity(value=float(token), unit="dollars", source_token=token)
|
||||
|
||||
|
||||
def _threshold_hours(problem_text: str) -> Quantity | None:
|
||||
match = _THRESHOLD_HOURS_RE.search(problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
token = match.group(1)
|
||||
return Quantity(value=float(token), unit="hours", source_token=token)
|
||||
|
||||
|
||||
def _hours_per_day(problem_text: str, question_clause: str) -> Quantity | None:
|
||||
threshold = _threshold_hours(problem_text)
|
||||
for clause in segment_clauses(problem_text):
|
||||
tokens = _tokens(clause)
|
||||
if "hours" not in tokens and "hour" not in tokens:
|
||||
continue
|
||||
if "more" in tokens and "than" in tokens:
|
||||
continue
|
||||
if "every" not in tokens and "per" not in tokens and "each" not in tokens:
|
||||
continue
|
||||
quantities = [
|
||||
q
|
||||
for q in extract_quantities(clause)
|
||||
if q.unit in {"hours", "hour"} or q.unit.rstrip("s") == "hour"
|
||||
]
|
||||
if len(quantities) != 1:
|
||||
continue
|
||||
q = quantities[0]
|
||||
if threshold is not None and q.value <= threshold.value:
|
||||
continue
|
||||
return Quantity(value=q.value, unit="hours", source_token=q.source_token)
|
||||
return None
|
||||
|
||||
|
||||
def _day_count(problem_text: str, question_clause: str) -> Quantity | None:
|
||||
for clause in segment_clauses(problem_text):
|
||||
tokens = _tokens(clause)
|
||||
if "days" not in tokens and "day" not in tokens:
|
||||
continue
|
||||
if clause == question_clause and "hours" not in tokens and "hour" not in tokens:
|
||||
continue
|
||||
quantities = [
|
||||
q
|
||||
for q in extract_quantities(clause)
|
||||
if q.unit in {"days", "day"} or q.unit.rstrip("s") == "day"
|
||||
]
|
||||
if len(quantities) == 1:
|
||||
q = quantities[0]
|
||||
return Quantity(value=q.value, unit="days", source_token=q.source_token)
|
||||
return None
|
||||
|
||||
|
||||
def _rental_days(problem_text: str, question_clause: str) -> Quantity | None:
|
||||
for clause in segment_clauses(problem_text):
|
||||
if clause == question_clause:
|
||||
continue
|
||||
tokens = _tokens(clause)
|
||||
if "rent" not in tokens and "rents" not in tokens:
|
||||
continue
|
||||
quantities = [
|
||||
q
|
||||
for q in extract_quantities(clause)
|
||||
if q.unit in {"days", "day"} or q.unit.rstrip("s") == "day"
|
||||
]
|
||||
if len(quantities) == 1:
|
||||
q = quantities[0]
|
||||
return Quantity(value=q.value, unit="days", source_token=q.source_token)
|
||||
return None
|
||||
|
||||
|
||||
def _bundle_tariff(problem_text: str) -> tuple[Quantity, Quantity, Quantity] | None:
|
||||
match = _BUNDLE_TARIFF_RE.search(problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
daily_token, bundle_token, bundle_days_token = (
|
||||
match.group(1),
|
||||
match.group(2),
|
||||
match.group(3),
|
||||
)
|
||||
return (
|
||||
Quantity(value=float(daily_token), unit="dollars", source_token=daily_token),
|
||||
Quantity(value=float(bundle_token), unit="dollars", source_token=bundle_token),
|
||||
Quantity(value=float(bundle_days_token), unit="days", source_token=bundle_days_token),
|
||||
)
|
||||
|
||||
|
||||
def build_overtime_shift_earnings(problem_text: str) -> GroundedDerivation | None:
|
||||
"""Construct overtime shift earnings total, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_money_total(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
if "overtime" not in _tokens(problem_text):
|
||||
return None
|
||||
|
||||
rate = _hourly_rate(problem_text)
|
||||
threshold = _threshold_hours(problem_text)
|
||||
hours_per_day = _hours_per_day(problem_text, question_clause)
|
||||
days = _day_count(problem_text, question_clause)
|
||||
half = _half_fraction_operand(problem_text)
|
||||
if None in (rate, threshold, hours_per_day, days, half):
|
||||
return None
|
||||
if hours_per_day.value <= threshold.value:
|
||||
return None
|
||||
|
||||
ot_hours_per_day = hours_per_day.value - threshold.value
|
||||
ot_rate = rate.value + (rate.value * half.value)
|
||||
ot_total = ot_hours_per_day * ot_rate * days.value
|
||||
|
||||
return GroundedDerivation(
|
||||
start=threshold,
|
||||
steps=(
|
||||
Step(op="multiply", operand=rate, cue="hour"),
|
||||
Step(op="multiply", operand=days, cue="days"),
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(
|
||||
value=ot_total,
|
||||
unit="hours",
|
||||
source_token=hours_per_day.source_token,
|
||||
),
|
||||
cue="overtime",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def build_bundle_overflow_tariff(problem_text: str) -> GroundedDerivation | None:
|
||||
"""Construct bundle + overflow-day tariff total, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_money_total(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
if "overtime" in _tokens(problem_text):
|
||||
return None
|
||||
|
||||
tariff = _bundle_tariff(problem_text)
|
||||
rental_days = _rental_days(problem_text, question_clause)
|
||||
if tariff is None or rental_days is None:
|
||||
return None
|
||||
|
||||
daily_rate, bundle_price, bundle_days = tariff
|
||||
if rental_days.value <= bundle_days.value:
|
||||
return None
|
||||
|
||||
overflow_cost = (rental_days.value - bundle_days.value) * daily_rate.value
|
||||
|
||||
return GroundedDerivation(
|
||||
start=bundle_price,
|
||||
steps=(
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(
|
||||
value=overflow_cost,
|
||||
unit="dollars",
|
||||
source_token=rental_days.source_token,
|
||||
),
|
||||
cue="per",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _self_verifies_temporal_tariff(
|
||||
derivation: GroundedDerivation, problem_text: str, *, pattern: str
|
||||
) -> SelfVerification:
|
||||
reasons: list[str] = []
|
||||
tokens = _tokens(problem_text)
|
||||
question_clause = _question_clause(problem_text)
|
||||
|
||||
operands = [derivation.start, *(s.operand for s in derivation.steps)]
|
||||
for q in operands:
|
||||
if not _value_grounds(q.source_token, tokens):
|
||||
reasons.append(f"operand {q.source_token!r} not grounded in text")
|
||||
|
||||
for step in derivation.steps:
|
||||
if not _token_in(step.cue, tokens):
|
||||
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||
|
||||
body = _body_text(problem_text)
|
||||
body_quantities = Counter(q.source_token for q in extract_quantities(body))
|
||||
used = Counter(
|
||||
[derivation.start.source_token]
|
||||
+ [step.operand.source_token for step in derivation.steps]
|
||||
)
|
||||
if pattern == "overtime":
|
||||
half = _half_fraction_operand(problem_text)
|
||||
if half is not None:
|
||||
used[half.source_token] += 1
|
||||
if half.source_token == "1/2":
|
||||
used["2"] += 1
|
||||
elif pattern == "bundle":
|
||||
tariff = _bundle_tariff(problem_text)
|
||||
if tariff is not None:
|
||||
daily_rate, _, bundle_days = tariff
|
||||
used[daily_rate.source_token] += 1
|
||||
used[bundle_days.source_token] += 1
|
||||
unused = body_quantities - used
|
||||
if unused:
|
||||
reasons.append(f"incomplete: unused body quantities {sorted(unused.elements())}")
|
||||
|
||||
if pattern == "overtime":
|
||||
if _hourly_rate(problem_text) is None:
|
||||
reasons.append("missing hourly rate anchor")
|
||||
if _threshold_hours(problem_text) is None:
|
||||
reasons.append("missing overtime threshold")
|
||||
if _hours_per_day(problem_text, question_clause) is None:
|
||||
reasons.append("missing hours-per-day anchor")
|
||||
if _day_count(problem_text, question_clause) is None:
|
||||
reasons.append("missing day-count anchor")
|
||||
if _half_fraction_operand(problem_text) is None:
|
||||
reasons.append("missing overtime half-fraction anchor")
|
||||
elif pattern == "bundle":
|
||||
if _bundle_tariff(problem_text) is None:
|
||||
reasons.append("missing bundle tariff anchor")
|
||||
if _rental_days(problem_text, question_clause) is None:
|
||||
reasons.append("missing rental-day question anchor")
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def _compose_pattern(
|
||||
builder, problem_text: str, *, pattern: str
|
||||
) -> Resolution | None:
|
||||
derivation = builder(problem_text)
|
||||
if derivation is None:
|
||||
return None
|
||||
if not _self_verifies_temporal_tariff(derivation, problem_text, pattern=pattern).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=derivation.answer,
|
||||
answer_unit=derivation.answer_unit,
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def compose_overtime_shift_earnings(problem_text: str) -> Resolution | None:
|
||||
return _compose_pattern(build_overtime_shift_earnings, problem_text, pattern="overtime")
|
||||
|
||||
|
||||
def compose_bundle_overflow_tariff(problem_text: str) -> Resolution | None:
|
||||
return _compose_pattern(build_bundle_overflow_tariff, problem_text, pattern="bundle")
|
||||
|
||||
|
||||
def compose_temporal_tariff(problem_text: str) -> Resolution | None:
|
||||
"""Try overtime then bundle overflow; refuse on ambiguity."""
|
||||
overtime = compose_overtime_shift_earnings(problem_text)
|
||||
bundle = compose_bundle_overflow_tariff(problem_text)
|
||||
if overtime is not None and bundle is not None:
|
||||
return None
|
||||
return overtime or bundle
|
||||
|
||||
|
||||
def resolve_promotable_temporal_tariff(problem_text: str) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2m)."""
|
||||
return compose_temporal_tariff(problem_text)
|
||||
|
|
@ -819,6 +819,36 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2m — temporal tariff overtime + bundle overflow (Sprint 9 lift).
|
||||
from generate.derivation.temporal_tariff import (
|
||||
resolve_promotable_temporal_tariff,
|
||||
)
|
||||
|
||||
temporal_tariff_resolution = resolve_promotable_temporal_tariff(text)
|
||||
if temporal_tariff_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=temporal_tariff_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2n — affine fraction-delta (Sprint 9 lift).
|
||||
from generate.derivation.affine_fraction_delta import (
|
||||
resolve_promotable_affine_fraction_delta,
|
||||
)
|
||||
|
||||
affine_fraction_resolution = resolve_promotable_affine_fraction_delta(text)
|
||||
if affine_fraction_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=affine_fraction_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
|
|||
summary = build_microscope_report(_load_cases())
|
||||
md = render_markdown(summary)
|
||||
assert "partition_chunking" in md
|
||||
assert "correct: 18" in md
|
||||
assert "correct: 21" in md
|
||||
assert "Gate A2a unit_partition" in md
|
||||
|
||||
|
||||
|
|
@ -124,5 +124,8 @@ def test_gate_a2_lifts_are_not_in_refusal_table():
|
|||
assert "gsm8k-train-sample-v1-0045" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0005" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0046" not in refused_ids
|
||||
assert summary["counts"]["correct"] >= 18
|
||||
assert "gsm8k-train-sample-v1-0001" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0010" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0017" not in refused_ids
|
||||
assert summary["counts"]["correct"] >= 21
|
||||
assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0
|
||||
|
|
|
|||
|
|
@ -163,8 +163,8 @@ def test_full_train_sample_wrong_zero_and_chunk_lift():
|
|||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] == 18
|
||||
assert counts["refused"] == 32
|
||||
assert counts["correct"] == 21
|
||||
assert counts["refused"] == 29
|
||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||
|
|
|
|||
|
|
@ -184,8 +184,8 @@ class TestTrainSampleScore:
|
|||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 18
|
||||
assert refused == 32
|
||||
assert correct == 21
|
||||
assert refused == 29
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -196,8 +196,8 @@ class TestTrainSampleScore:
|
|||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 18
|
||||
assert refused == 32
|
||||
assert correct == 21
|
||||
assert refused == 29
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -141,9 +141,8 @@ class TestSiblingGeneralization:
|
|||
|
||||
|
||||
class TestConfuserRefusals:
|
||||
def test_affine_more_than_fraction_refuses(self):
|
||||
def test_affine_more_than_fraction_refuses_fraction_decrease(self):
|
||||
assert resolve_promotable_fraction_decrease(AFFINE_CONFUSER_0010) is None
|
||||
assert _run(AFFINE_CONFUSER_0010).answer is None
|
||||
|
||||
def test_final_value_question_refuses_fraction_decrease(self):
|
||||
assert resolve_promotable_fraction_decrease(FINAL_VALUE_CONFUSER) is None
|
||||
|
|
@ -203,8 +202,8 @@ class TestTrainSampleScore:
|
|||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 18
|
||||
assert refused == 32
|
||||
assert correct == 21
|
||||
assert refused == 29
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
|
|
|
|||
272
tests/test_math_candidate_graph_sprint9_temporal_affine_lift.py
Normal file
272
tests/test_math_candidate_graph_sprint9_temporal_affine_lift.py
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
"""Gate A2m/A2n — Capability Paradigm Sprint 9 temporal tariff + affine lift."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.derivation.temporal_tariff import (
|
||||
compose_bundle_overflow_tariff,
|
||||
compose_overtime_shift_earnings,
|
||||
compose_temporal_tariff,
|
||||
resolve_promotable_temporal_tariff,
|
||||
)
|
||||
from generate.derivation.affine_fraction_delta import (
|
||||
compose_affine_fraction_delta,
|
||||
resolve_promotable_affine_fraction_delta,
|
||||
)
|
||||
|
||||
_CASES_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "evals"
|
||||
/ "gsm8k_math"
|
||||
/ "train_sample"
|
||||
/ "v1"
|
||||
/ "cases.jsonl"
|
||||
)
|
||||
_HOLDOUT_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "evals"
|
||||
/ "gsm8k_math"
|
||||
/ "holdout_dev"
|
||||
/ "v1"
|
||||
/ "cases.jsonl"
|
||||
)
|
||||
|
||||
CASE_0001 = (
|
||||
"Tina makes $18.00 an hour. If she works more than 8 hours per shift, she is "
|
||||
"eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage. "
|
||||
"If she works 10 hours every day for 5 days, how much money does she make?"
|
||||
)
|
||||
|
||||
CASE_0017 = (
|
||||
"Jason has a carriage house that he rents out. He's charging $50.00 per day or "
|
||||
"$500.00 for 14 days. Eric wants to rent the house for 20 days. "
|
||||
"How much will it cost him?"
|
||||
)
|
||||
|
||||
CASE_0010 = (
|
||||
"Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what "
|
||||
"Yun currently has, plus 7. How many paperclips does Marion have?"
|
||||
)
|
||||
|
||||
SIBLING_OVERTIME = (
|
||||
"Rosa makes $20.00 an hour. If she works more than 6 hours per shift, she is "
|
||||
"eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage. "
|
||||
"If she works 9 hours every day for 4 days, how much money does she make?"
|
||||
)
|
||||
|
||||
SIBLING_BUNDLE = (
|
||||
"Mia owns a guest cottage. She's charging $40.00 per day or $300.00 for 10 days. "
|
||||
"Leo wants to rent the cottage for 15 days. How much will it cost him?"
|
||||
)
|
||||
|
||||
SIBLING_AFFINE = (
|
||||
"Sam had 20 marbles initially, but then lost 4. Riley has 1/4 more than what Sam "
|
||||
"currently has, plus 3. How many marbles does Riley have?"
|
||||
)
|
||||
|
||||
FRACTION_DECREASE_CONFUSER = (
|
||||
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. "
|
||||
"If the current temperature of the mountain is 84 degrees, what will the temperature "
|
||||
"decrease by?"
|
||||
)
|
||||
|
||||
CURRENCY_AMOUNT_0019_SHAPE = (
|
||||
"John adopts a dog from a shelter. The dog ends up having health problems and this "
|
||||
"requires 3 vet appointments, which cost $400 each. After the first appointment, "
|
||||
"John paid $100 for pet insurance that covers 80% of the subsequent visits. "
|
||||
"How much did he pay in total?"
|
||||
)
|
||||
|
||||
CURRENCY_AMOUNT_0028_SHAPE = (
|
||||
"Tom opens an amusement park. It cost $100,000 to open initially. It also cost 1% "
|
||||
"of that to run per day. He sells 150 tickets a day for $10 each. "
|
||||
"How long will it take to make back his money?"
|
||||
)
|
||||
|
||||
SEALED_WRONG_0032_SHAPE = (
|
||||
"He draws and colors 10 pictures. He draws and colors 7 pictures, but each "
|
||||
"picture takes 30% less time than the previous one."
|
||||
)
|
||||
|
||||
OT_ASKS_HOURS_CONFUSER = (
|
||||
"Tina makes $18.00 an hour. If she works more than 8 hours per shift, she is "
|
||||
"eligible for overtime, which is paid by your hourly wage + 1/2 your hourly wage. "
|
||||
"If she works 10 hours every day for 5 days, how many hours does she work in total?"
|
||||
)
|
||||
|
||||
BUNDLE_UNDER_PERIOD_CONFUSER = (
|
||||
"Mia owns a guest cottage. She's charging $40.00 per day or $300.00 for 10 days. "
|
||||
"Leo wants to rent the cottage for 8 days. How much will it cost him?"
|
||||
)
|
||||
|
||||
AFFINE_TWICE_CONFUSER = (
|
||||
"Yun had 20 paperclips initially, but then lost 12. Marion has twice as many "
|
||||
"paperclips as Yun currently has. How many paperclips does Marion have?"
|
||||
)
|
||||
|
||||
PRESERVED_SOLVED = (
|
||||
"0002",
|
||||
"0003",
|
||||
"0005",
|
||||
"0008",
|
||||
"0014",
|
||||
"0015",
|
||||
"0018",
|
||||
"0021",
|
||||
"0024",
|
||||
"0025",
|
||||
"0029",
|
||||
"0030",
|
||||
"0035",
|
||||
"0037",
|
||||
"0038",
|
||||
"0042",
|
||||
"0045",
|
||||
"0046",
|
||||
)
|
||||
|
||||
|
||||
def _run(text: str):
|
||||
return parse_and_solve(text, sealed=False)
|
||||
|
||||
|
||||
def _load_train_cases() -> list[dict]:
|
||||
return [json.loads(line) for line in _CASES_PATH.read_text().splitlines() if line.strip()]
|
||||
|
||||
|
||||
class TestTargetCases:
|
||||
def test_train_sample_0001_overtime(self):
|
||||
res = _run(CASE_0001)
|
||||
assert res.answer == 990.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
def test_train_sample_0017_bundle_tariff(self):
|
||||
res = _run(CASE_0017)
|
||||
assert res.answer == 800.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
def test_train_sample_0010_affine_fraction_delta(self):
|
||||
res = _run(CASE_0010)
|
||||
assert res.answer == 9.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
|
||||
class TestSiblingGeneralization:
|
||||
def test_overtime_sibling(self):
|
||||
res = _run(SIBLING_OVERTIME)
|
||||
assert res.answer is not None
|
||||
assert abs(res.answer - 840.0) < 1e-6
|
||||
assert res.refusal_reason is None
|
||||
|
||||
def test_bundle_sibling(self):
|
||||
res = _run(SIBLING_BUNDLE)
|
||||
assert res.answer == 500.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
def test_affine_sibling(self):
|
||||
res = _run(SIBLING_AFFINE)
|
||||
assert res.answer == 7.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
|
||||
class TestConfuserRefusals:
|
||||
def test_fraction_decrease_not_affine(self):
|
||||
assert resolve_promotable_affine_fraction_delta(FRACTION_DECREASE_CONFUSER) is None
|
||||
assert _run(FRACTION_DECREASE_CONFUSER).answer == 21.0
|
||||
|
||||
def test_currency_amount_vet_not_tariff(self):
|
||||
assert resolve_promotable_temporal_tariff(CURRENCY_AMOUNT_0019_SHAPE) is None
|
||||
assert _run(CURRENCY_AMOUNT_0019_SHAPE).answer is None
|
||||
|
||||
def test_currency_amount_amusement_not_tariff(self):
|
||||
assert resolve_promotable_temporal_tariff(CURRENCY_AMOUNT_0028_SHAPE) is None
|
||||
assert _run(CURRENCY_AMOUNT_0028_SHAPE).answer is None
|
||||
|
||||
def test_overtime_asks_hours_refuses(self):
|
||||
assert resolve_promotable_temporal_tariff(OT_ASKS_HOURS_CONFUSER) is None
|
||||
assert _run(OT_ASKS_HOURS_CONFUSER).answer is None
|
||||
|
||||
def test_bundle_under_period_refuses(self):
|
||||
assert resolve_promotable_temporal_tariff(BUNDLE_UNDER_PERIOD_CONFUSER) is None
|
||||
assert _run(BUNDLE_UNDER_PERIOD_CONFUSER).answer is None
|
||||
|
||||
def test_twice_as_many_not_affine_fraction(self):
|
||||
assert resolve_promotable_affine_fraction_delta(AFFINE_TWICE_CONFUSER) is None
|
||||
assert _run(AFFINE_TWICE_CONFUSER).answer is None
|
||||
|
||||
|
||||
class TestSealedWrongPatternRefusal:
|
||||
def test_percent_decrease_not_temporal_tariff(self):
|
||||
assert resolve_promotable_temporal_tariff(SEALED_WRONG_0032_SHAPE) is None
|
||||
assert _run(SEALED_WRONG_0032_SHAPE).answer is None
|
||||
|
||||
|
||||
class TestTrainSampleScore:
|
||||
def test_full_train_sample_wrong_zero(self):
|
||||
wrong = 0
|
||||
for case in _load_train_cases():
|
||||
res = _run(case["question"])
|
||||
if res.answer is not None and res.answer != float(case["answer_numeric"]):
|
||||
wrong += 1
|
||||
assert wrong == 0
|
||||
|
||||
def test_aggregate_21_29_0(self):
|
||||
correct = refused = wrong = 0
|
||||
for case in _load_train_cases():
|
||||
res = _run(case["question"])
|
||||
gold = float(case["answer_numeric"])
|
||||
if res.answer is None:
|
||||
refused += 1
|
||||
elif res.answer == gold:
|
||||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 21
|
||||
assert refused == 29
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
class TestPriorSolvedRegression:
|
||||
@pytest.mark.parametrize("case_suffix", PRESERVED_SOLVED)
|
||||
def test_prior_solved_still_correct(self, case_suffix: str):
|
||||
cases = {c["case_id"].split("-")[-1]: c for c in _load_train_cases()}
|
||||
case = cases[case_suffix]
|
||||
res = _run(case["question"])
|
||||
assert res.answer == float(case["answer_numeric"])
|
||||
|
||||
|
||||
class TestHoldoutDevSafety:
|
||||
def test_holdout_dev_no_new_admissions(self):
|
||||
if not _HOLDOUT_PATH.exists():
|
||||
pytest.skip("holdout_dev corpus unavailable")
|
||||
admitted = 0
|
||||
for line in _HOLDOUT_PATH.read_text().splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
case = json.loads(line)
|
||||
text = case["problem"]
|
||||
if (
|
||||
compose_temporal_tariff(text) is not None
|
||||
or compose_affine_fraction_delta(text) is not None
|
||||
):
|
||||
admitted += 1
|
||||
assert admitted == 0
|
||||
|
||||
|
||||
class TestComposeAPI:
|
||||
def test_overtime_compose_matches_promote(self):
|
||||
assert compose_overtime_shift_earnings(CASE_0001) is not None
|
||||
assert resolve_promotable_temporal_tariff(CASE_0001) is not None
|
||||
|
||||
def test_bundle_compose_matches_promote(self):
|
||||
assert compose_bundle_overflow_tariff(CASE_0017) is not None
|
||||
assert resolve_promotable_temporal_tariff(CASE_0017) is not None
|
||||
|
||||
def test_affine_compose_matches_promote(self):
|
||||
assert compose_affine_fraction_delta(CASE_0010) is not None
|
||||
assert resolve_promotable_affine_fraction_delta(CASE_0010) is not None
|
||||
Loading…
Reference in a new issue