feat(derivation): capability paradigm sprint 8 R6 affine/fraction lift (#819)
This commit is contained in:
parent
5d529b425f
commit
282c5126de
11 changed files with 884 additions and 11 deletions
|
|
@ -0,0 +1,177 @@
|
|||
# GSM8K Capability Paradigm Sprint 8 — Lookback (2026-06-17)
|
||||
|
||||
## 1. Starting baseline
|
||||
|
||||
| Metric | Value |
|
||||
|--------|------:|
|
||||
| correct | **16** |
|
||||
| refused | **34** |
|
||||
| wrong | **0** |
|
||||
|
||||
After #818 (Sprint 7: round_trip_trip_duration + giveaway_target_residual). Preserved solved: 0002, 0003, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045.
|
||||
|
||||
## 2. Experience Flywheel output summary
|
||||
|
||||
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint8.json`
|
||||
|
||||
| Field | Value |
|
||||
|-------|------:|
|
||||
| retained_record_count | 50 |
|
||||
| lift_refused_to_correct | **0** |
|
||||
| promotion_candidates | 4 families, **all** `blocked_by_wrong_risk` |
|
||||
|
||||
Top missing primitives: `relation_hypothesis` (DCS), `multiplicative_aggregate`, `temporal_tariff`, `diagnostic_hold`.
|
||||
|
||||
**Key signal:** No scout refused→correct delta (prior sprints already promoted clean lifts). Composition-validation pins **cv-0007** (0005 fraction-decrease) and **cv-0008** (0046 percent-partition) provided typed R6 chains with low wrong-risk — aligned with Sprint 7 handoff.
|
||||
|
||||
## 3. Scout/frontier output summary
|
||||
|
||||
| Regime | correct | refused | wrong |
|
||||
|--------|--------:|--------:|------:|
|
||||
| Serving | 16 | 34 | 0 |
|
||||
| Sealed (resolve_pooled) | 3 | 39 | 8 |
|
||||
|
||||
- `lift_refused_to_correct`: **0**
|
||||
- Dominant bottleneck: joint refusal (28/50)
|
||||
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
|
||||
|
||||
Microscope slices: `affine_equation_fraction_target` (0005), `affine_equation` / percent partition (0046).
|
||||
|
||||
## 4. Candidate families ranked
|
||||
|
||||
| Rank | Family | Cases | Lift signal | Wrong-risk | Sprint 8 verdict |
|
||||
|------|--------|-------|-------------|------------|------------------|
|
||||
| 1 | R6 fraction-decrease (`cv-0007`) | **0005** | typed `base × (1−N/M)` | low | **Selected (A2k)** |
|
||||
| 2 | R6 percent-partition (`cv-0008`) | **0046** | typed half-split + dual % | low | **Selected (A2l)** |
|
||||
| 3 | `temporal_tariff:temporal_aggregation` | 0001, 0017 | joint only | medium | Deferred — confuser design |
|
||||
| 4 | `affine_equation_fraction_delta` | 0010 | joint only | medium | Deferred — 1/4 more than affine |
|
||||
| 5 | `relation_hypothesis:DCS` | 14 joint + 2 wrong | none | **blocked** (0032, 0047) | Deferred |
|
||||
| 6 | `diagnostic_hold:currency_amount` | 0019, 0028 | sealed-wrong | **blocked** | Deferred |
|
||||
|
||||
## 5. Blocked families and why
|
||||
|
||||
| Family | Blocked because |
|
||||
|--------|-----------------|
|
||||
| `relation_hypothesis:DCS` | 0032, 0047 sealed-wrong share discrete-count / partition surface |
|
||||
| `conservative_boundary` | 0018, 0025 sealed-wrong on served-correct cases |
|
||||
| `sealed_elimination` | 0011, 0026 hard elimination |
|
||||
| `diagnostic_hold:currency_amount` | 0019, 0028 sealed-wrong |
|
||||
|
||||
## 6. Selected paradigm
|
||||
|
||||
**Dual narrow R6 organs** (composition-validation decomposition, not generic affine parser):
|
||||
|
||||
1. **Gate A2k `fraction_decrease`** — forecast ``decrease to N/M of`` + explicit current base + question asks **decrease by** delta (`cv-0007` / 0005).
|
||||
2. **Gate A2l `percent_partition`** — total population + equal ``half`` / ``other half`` split + two ``N% of the <group>`` clauses + aggregate ownership question (`cv-0008` / 0046).
|
||||
|
||||
Rejected: broad product_bridge, DCS injector, generic affine equation parser, temporal_tariff wholesale promotion.
|
||||
|
||||
## 7. Typed chains for target cases
|
||||
|
||||
**0005 (gold 21):** `84 × (1 − 3/4) = 84 × 1/4 = 21` degrees decrease.
|
||||
|
||||
**0046 (gold 15):** `100 × 0.5 = 50` per group → `50 × 0.20 = 10` girl dog owners → `+ 50 × 0.10 = 5` boy dog owners → `15` total.
|
||||
|
||||
## 8. Implemented organs
|
||||
|
||||
| Module | Promotion entry |
|
||||
|--------|-----------------|
|
||||
| `generate/derivation/fraction_decrease.py` | `resolve_promotable_fraction_decrease` |
|
||||
| `generate/derivation/percent_partition.py` | `resolve_promotable_percent_partition` |
|
||||
|
||||
Wired in `generate/math_candidate_graph.py` after Gate A2j, before rate short-circuits.
|
||||
|
||||
## 9. Before/after score
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|--------|-------:|------:|--:|
|
||||
| correct | 16 | **18** | +2 |
|
||||
| refused | 34 | **32** | −2 |
|
||||
| wrong | 0 | **0** | 0 |
|
||||
|
||||
**State A met:** 18+/32-/0.
|
||||
|
||||
## 10. Newly solved IDs
|
||||
|
||||
**0005**, **0046**
|
||||
|
||||
## 11. Preserved solved IDs
|
||||
|
||||
0002, 0003, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045
|
||||
|
||||
## 12. Wrong=0 proof
|
||||
|
||||
- Full train_sample replay: 18 correct / 32 refused / 0 wrong
|
||||
- `tests/test_math_candidate_graph_sprint8_r6_affine_lift.py::TestTrainSampleScore`
|
||||
- Composition-validation firewall: 14 solve / 8 refuse / 0 wrong
|
||||
- Holdout_dev: 0 new organ admissions
|
||||
- Smoke suite: 108 passed
|
||||
|
||||
## 13. Confuser matrix
|
||||
|
||||
| Confuser | Organ | Result |
|
||||
|----------|-------|--------|
|
||||
| 0010 affine `1/4 more than` | A2k | refused |
|
||||
| Final-value question (not decrease-by) | A2k | refused |
|
||||
| Unequal 60/40 split (not half/half) | A2l | refused |
|
||||
| 0032 percent time-decrease | A2l | refused |
|
||||
| 0047 DCS equal-pack | A2l | refused |
|
||||
| Goal-residual (cv-0005) | A2k | refused (goal_residual serves) |
|
||||
|
||||
## 14. Sibling generality proof
|
||||
|
||||
| Sibling | Gold | Organ |
|
||||
|---------|-----:|-------|
|
||||
| Cedar peak 2/3 decrease, 60° base | 20 | A2k |
|
||||
| Club 80 members, 25%/15% half-split | 16 | A2l |
|
||||
|
||||
## 15. Sealed-wrong negative evidence used
|
||||
|
||||
- 0032 percent-decrease shape → A2l hazard refusal (not sequential % chain)
|
||||
- 0047 equal-pack DCS → A2l hazard refusal (no half-split / no % of group)
|
||||
- DCS flywheel `blocked_by_wrong_risk` → no broad partition injector
|
||||
|
||||
## 16. Composition-validation pin changes
|
||||
|
||||
| Pin | Before | After |
|
||||
|-----|--------|-------|
|
||||
| cv-0007 (0005) | refuse | **solve** (21) |
|
||||
| cv-0008 (0046) | refuse | **solve** (15) |
|
||||
|
||||
Aggregate snapshot: 12/10/0 → **14/8/0**
|
||||
|
||||
## 17. report.json untouched
|
||||
|
||||
`evals/gsm8k_math/train_sample/v1/report.json` not modified.
|
||||
|
||||
## 18. Sealed artifacts untouched
|
||||
|
||||
No changes to sealed practice lanes, holdout, or `resolve_pooled` wiring.
|
||||
|
||||
## 19. No case-id logic
|
||||
|
||||
Promotion is family-typed only; no `case_id` conditionals in serving path.
|
||||
|
||||
## 20. No hardcoded answer
|
||||
|
||||
Answers derived from grounded arithmetic chains; no answer literals.
|
||||
|
||||
## 21. No direct-answer shortcut
|
||||
|
||||
No `answer_numeric` extraction or final-answer parsing.
|
||||
|
||||
## 22. Sprint 9 recommendation
|
||||
|
||||
**Track A — Capability Sprint 9** (continue capability paradigm)
|
||||
|
||||
| Evidence | Detail |
|
||||
|----------|--------|
|
||||
| Flywheel | `lift_refused_to_correct == 0`; dense `joint_sealed_no_resolution` cluster remains |
|
||||
| Next targets | `temporal_tariff` (0001, 0017) after confuser matrix; `affine_equation_fraction_delta` (0010) |
|
||||
| Secondary | `multiplicative_aggregate` joint cluster (0006, 0013) — only with hard verifier |
|
||||
| Blocked | DCS (0032/0047), currency_amount (0019/0028), sealed_elimination (0011/0026) |
|
||||
| Expected lift | +1–2 correct if temporal_tariff organ passes adversarial review |
|
||||
| Modules | `generate/derivation/temporal_tariff.py` (proposed) |
|
||||
| Non-goals | Flywheel PR-2, broad product_bridge, DCS injector, report.json rebaseline |
|
||||
|
||||
Track B (Experience Flywheel PR-2) and Track C (Diagnostic Hardening) deferred — flywheel still shows promotable typed chains in joint-refusal cluster, not exclusively wrong-risk blockage.
|
||||
187
generate/derivation/fraction_decrease.py
Normal file
187
generate/derivation/fraction_decrease.py
Normal file
|
|
@ -0,0 +1,187 @@
|
|||
"""Gate A2k — fraction-decrease delta (decrease-to fraction of base).
|
||||
|
||||
Experience Flywheel Sprint 8 / composition-validation **cv-0007** (train_sample
|
||||
**0005**): forecast clause states a quantity will ``decrease to N/M of`` its
|
||||
current value; a follow-on clause states the current amount; the question asks
|
||||
the **decrease by** delta (not the final value).
|
||||
|
||||
decrease_by = base × (1 − N/M)
|
||||
|
||||
Narrow organ — not a generic affine equation parser, not ``N/M more than`` affine
|
||||
surfaces (0010-class), not percent-decrease. Promotion requires:
|
||||
|
||||
- exactly one ``decrease to N/M of`` forecast (no ``more than`` fraction affine);
|
||||
- question binds ``decrease`` + ``by`` (delta target, not final-value question);
|
||||
- exactly one explicit current/base quantity with a unit;
|
||||
- hazard refusal (percent, goal language, multiple decrease-to fractions).
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Final
|
||||
|
||||
from generate.derivation.clauses import segment_clauses
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification, select_self_verified
|
||||
from generate.math_roundtrip import _tokens
|
||||
|
||||
_DECREASE_TO_FRACTION_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"decrease\s+to\s+(\d+)\s*/\s*(\d+)\s+of",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||
_GOAL_INTENT: Final[frozenset[str]] = frozenset(
|
||||
{"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"}
|
||||
)
|
||||
_FINAL_VALUE_CUES: Final[frozenset[str]] = frozenset({"be", "will", "what"})
|
||||
|
||||
|
||||
def _asks_decrease_delta(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return "decrease" in tokens and "by" in tokens
|
||||
|
||||
|
||||
def _asks_final_value(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
if "by" in tokens:
|
||||
return False
|
||||
if "decrease" not in tokens:
|
||||
return False
|
||||
return bool(_FINAL_VALUE_CUES & tokens) or "temperature" in tokens
|
||||
|
||||
|
||||
def _target_fraction(problem_text: str) -> tuple[int, int, str] | None:
|
||||
matches = list(_DECREASE_TO_FRACTION_RE.finditer(problem_text))
|
||||
if len(matches) != 1:
|
||||
return None
|
||||
num_s, den_s = matches[0].group(1), matches[0].group(2)
|
||||
try:
|
||||
num = int(num_s)
|
||||
den = int(den_s)
|
||||
except ValueError:
|
||||
return None
|
||||
if num <= 0 or den <= 0 or num >= den:
|
||||
return None
|
||||
return num, den, f"{num_s}/{den_s}"
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
text_tokens = _tokens(problem_text)
|
||||
question_tokens = _tokens(question_clause)
|
||||
if "more" in text_tokens and "than" in text_tokens:
|
||||
return True
|
||||
if "%" in problem_text or "percent" in text_tokens or "percentage" in text_tokens:
|
||||
return True
|
||||
if text_tokens & _GOAL_INTENT:
|
||||
return True
|
||||
if _asks_final_value(question_clause):
|
||||
return True
|
||||
fractions = _EXTRA_FRACTION_RE.findall(problem_text)
|
||||
target = _target_fraction(problem_text)
|
||||
if target is None:
|
||||
return True
|
||||
_, _, token = target
|
||||
extra = [f for f in fractions if f.replace(" ", "") != token.replace(" ", "")]
|
||||
return len(extra) > 0
|
||||
|
||||
|
||||
def _current_base_quantity(problem_text: str, question_clause: str) -> Quantity | None:
|
||||
for clause in segment_clauses(problem_text):
|
||||
if _DECREASE_TO_FRACTION_RE.search(clause):
|
||||
continue
|
||||
tokens = _tokens(clause)
|
||||
if "current" not in tokens and "now" not in tokens:
|
||||
continue
|
||||
quantities = [q for q in extract_quantities(clause) if q.unit]
|
||||
if len(quantities) == 1:
|
||||
return quantities[0]
|
||||
for clause in segment_clauses(problem_text):
|
||||
if clause == question_clause:
|
||||
continue
|
||||
if _DECREASE_TO_FRACTION_RE.search(clause):
|
||||
continue
|
||||
quantities = [q for q in extract_quantities(clause) if q.unit]
|
||||
if len(quantities) == 1:
|
||||
return quantities[0]
|
||||
return None
|
||||
|
||||
|
||||
def build_fraction_decrease(problem_text: str) -> GroundedDerivation | None:
|
||||
"""Construct ``base × (1 − N/M)`` decrease delta, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_decrease_delta(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
|
||||
fraction = _target_fraction(problem_text)
|
||||
base = _current_base_quantity(problem_text, question_clause)
|
||||
if fraction is None or base is None:
|
||||
return None
|
||||
|
||||
num, den, source_token = fraction
|
||||
delta_factor = 1.0 - (num / den)
|
||||
if delta_factor <= 0:
|
||||
return None
|
||||
|
||||
return GroundedDerivation(
|
||||
start=base,
|
||||
steps=(
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(
|
||||
value=delta_factor,
|
||||
unit="",
|
||||
source_token=source_token,
|
||||
),
|
||||
cue="decrease",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _self_verifies_fraction_decrease(
|
||||
derivation: GroundedDerivation, problem_text: str
|
||||
) -> SelfVerification:
|
||||
from generate.derivation.verify import _base_reasons
|
||||
|
||||
tokens = _tokens(problem_text)
|
||||
reasons = list(_base_reasons(derivation, tokens))
|
||||
|
||||
fraction = _target_fraction(problem_text)
|
||||
if fraction is None:
|
||||
reasons.append("missing decrease-to fraction forecast")
|
||||
else:
|
||||
_, _, token = fraction
|
||||
if token not in problem_text.replace(" ", ""):
|
||||
reasons.append(f"fraction token {token!r} not grounded in text")
|
||||
|
||||
base = _current_base_quantity(problem_text, _question_clause(problem_text))
|
||||
if base is None:
|
||||
reasons.append("missing explicit current/base quantity")
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_fraction_decrease(problem_text: str) -> Resolution | None:
|
||||
"""Gate the typed fraction-decrease chain through self-verification."""
|
||||
derivation = build_fraction_decrease(problem_text)
|
||||
if derivation is None:
|
||||
return None
|
||||
if not _self_verifies_fraction_decrease(derivation, problem_text).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=derivation.answer,
|
||||
answer_unit=derivation.answer_unit,
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def resolve_promotable_fraction_decrease(problem_text: str) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2k)."""
|
||||
return compose_fraction_decrease(problem_text)
|
||||
180
generate/derivation/percent_partition.py
Normal file
180
generate/derivation/percent_partition.py
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
"""Gate A2l — equal half-split percent partition aggregate.
|
||||
|
||||
Experience Flywheel Sprint 8 / composition-validation **cv-0008** (train_sample
|
||||
**0046**): total population, equal ``half`` girls / ``other half`` boys split,
|
||||
two subgroup ``N% of the <group>`` clauses, question asks how many total own the
|
||||
attribute.
|
||||
|
||||
subgroup = total × half
|
||||
answer = subgroup × pct_girls + subgroup × pct_boys
|
||||
|
||||
Narrow organ — not a generic percent equation parser, not unequal partitions, not
|
||||
multi-container DCS. Promotion requires:
|
||||
|
||||
- question asks ``how many`` + aggregate ownership cue;
|
||||
- total population quantity with unit;
|
||||
- equal half / other-half split language;
|
||||
- exactly two percent-of-subgroup clauses with distinct group referents;
|
||||
- hazard refusal (fraction surfaces, money, profit, goal language).
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Final
|
||||
|
||||
from generate.derivation.clauses import segment_clauses
|
||||
from generate.derivation.comparatives import comparative_step, extract_comparative_scalars
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification
|
||||
from generate.math_roundtrip import _tokens
|
||||
|
||||
_PERCENT_OF_GROUP_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(\d+(?:\.\d+)?)\s*%\s+of\s+the\s+(\w+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||
_GOAL_INTENT: Final[frozenset[str]] = frozenset(
|
||||
{"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"}
|
||||
)
|
||||
_OWNERSHIP_CUES: Final[frozenset[str]] = frozenset(
|
||||
{"own", "owns", "have", "has", "had"}
|
||||
)
|
||||
|
||||
|
||||
def _asks_partition_total(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
return "how" in tokens and "many" in tokens and bool(_OWNERSHIP_CUES & tokens)
|
||||
|
||||
|
||||
def _has_equal_half_split(problem_text: str) -> bool:
|
||||
lowered = problem_text.lower()
|
||||
return "half" in lowered and "other half" in lowered
|
||||
|
||||
|
||||
def _percent_clauses(problem_text: str) -> tuple[tuple[float, str, str], tuple[float, str, str]] | None:
|
||||
question_clause = _question_clause(problem_text)
|
||||
matches: list[tuple[float, str, str]] = []
|
||||
for clause in segment_clauses(problem_text):
|
||||
if clause == question_clause:
|
||||
continue
|
||||
for match in _PERCENT_OF_GROUP_RE.finditer(clause):
|
||||
pct = float(match.group(1)) / 100.0
|
||||
group = match.group(2).lower()
|
||||
source = match.group(1)
|
||||
matches.append((pct, group, source))
|
||||
if len(matches) != 2:
|
||||
return None
|
||||
groups = {m[1] for m in matches}
|
||||
if len(groups) != 2:
|
||||
return None
|
||||
return matches[0], matches[1]
|
||||
|
||||
|
||||
def _total_population(problem_text: str, question_clause: str) -> Quantity | None:
|
||||
for clause in segment_clauses(problem_text):
|
||||
if clause == question_clause:
|
||||
continue
|
||||
lowered = clause.lower()
|
||||
if "half" in lowered or _PERCENT_OF_GROUP_RE.search(clause):
|
||||
continue
|
||||
quantities = [q for q in extract_quantities(clause) if q.unit]
|
||||
if len(quantities) == 1:
|
||||
return quantities[0]
|
||||
return None
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
text_tokens = _tokens(problem_text)
|
||||
if _FRACTION_RE.search(problem_text):
|
||||
return True
|
||||
if text_tokens & _GOAL_INTENT:
|
||||
return True
|
||||
if "profit" in text_tokens or "money" in text_tokens or "$" in problem_text:
|
||||
return True
|
||||
if not _has_equal_half_split(problem_text):
|
||||
return True
|
||||
comparatives = extract_comparative_scalars(problem_text)
|
||||
if not any(cs.cue == "half" for cs in comparatives):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def build_percent_partition(problem_text: str) -> GroundedDerivation | None:
|
||||
"""Construct equal half-split percent partition total, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_partition_total(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
|
||||
total = _total_population(problem_text, question_clause)
|
||||
percent_pair = _percent_clauses(problem_text)
|
||||
if total is None or percent_pair is None:
|
||||
return None
|
||||
|
||||
(pct_a, _, source_a), (pct_b, _, source_b) = percent_pair
|
||||
comparatives = extract_comparative_scalars(problem_text)
|
||||
half = next((cs for cs in comparatives if cs.cue == "half"), None)
|
||||
if half is None:
|
||||
return None
|
||||
|
||||
subgroup = total.value * half.scalar
|
||||
contrib_a = subgroup * pct_a
|
||||
contrib_b = subgroup * pct_b
|
||||
|
||||
return GroundedDerivation(
|
||||
start=total,
|
||||
steps=(
|
||||
comparative_step(half),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(value=pct_a, unit="", source_token=source_a),
|
||||
cue="of",
|
||||
),
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(value=contrib_b, unit=total.unit, source_token=source_b),
|
||||
cue="and",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _self_verifies_percent_partition(
|
||||
derivation: GroundedDerivation, problem_text: str
|
||||
) -> SelfVerification:
|
||||
from generate.derivation.verify import _base_reasons
|
||||
|
||||
tokens = _tokens(problem_text)
|
||||
reasons = list(_base_reasons(derivation, tokens))
|
||||
|
||||
if _percent_clauses(problem_text) is None:
|
||||
reasons.append("expected two distinct percent-of-subgroup clauses")
|
||||
if _total_population(problem_text, _question_clause(problem_text)) is None:
|
||||
reasons.append("missing total population quantity")
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_percent_partition(problem_text: str) -> Resolution | None:
|
||||
"""Gate the typed percent-partition chain through self-verification."""
|
||||
derivation = build_percent_partition(problem_text)
|
||||
if derivation is None:
|
||||
return None
|
||||
if not _self_verifies_percent_partition(derivation, problem_text).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=derivation.answer,
|
||||
answer_unit=derivation.answer_unit,
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def resolve_promotable_percent_partition(problem_text: str) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2l)."""
|
||||
return compose_percent_partition(problem_text)
|
||||
|
|
@ -789,6 +789,36 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2k — fraction decrease delta (Sprint 8 R6 lift).
|
||||
from generate.derivation.fraction_decrease import (
|
||||
resolve_promotable_fraction_decrease,
|
||||
)
|
||||
|
||||
fraction_decrease_resolution = resolve_promotable_fraction_decrease(text)
|
||||
if fraction_decrease_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=fraction_decrease_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2l — equal half-split percent partition (Sprint 8 R6 lift).
|
||||
from generate.derivation.percent_partition import (
|
||||
resolve_promotable_percent_partition,
|
||||
)
|
||||
|
||||
percent_partition_resolution = resolve_promotable_percent_partition(text)
|
||||
if percent_partition_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=percent_partition_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -168,6 +168,8 @@ def test_current_baseline_snapshot() -> None:
|
|||
Sprint 6 (2026-06-17): Gate A2g duration_segment_total admits cv-0022 (0015).
|
||||
Sprint 7 (2026-06-17): Gate A2i round_trip_trip_duration admits cv-0006 (0030);
|
||||
Gate A2j giveaway_target_residual admits cv-0021 (0035).
|
||||
Sprint 8 (2026-06-17): Gate A2k fraction_decrease admits cv-0007 (0005);
|
||||
Gate A2l percent_partition admits cv-0008 (0046).
|
||||
"""
|
||||
solve = refuse = wrong = 0
|
||||
for case in _CASES:
|
||||
|
|
@ -179,7 +181,7 @@ def test_current_baseline_snapshot() -> None:
|
|||
else:
|
||||
refuse += 1
|
||||
assert wrong == 0
|
||||
assert (solve, refuse) == (12, 10), (
|
||||
assert (solve, refuse) == (14, 8), (
|
||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||
f"slice landed, update this expectation and the affected rows' "
|
||||
f"baseline fields in lockstep"
|
||||
|
|
|
|||
|
|
@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
|
|||
summary = build_microscope_report(_load_cases())
|
||||
md = render_markdown(summary)
|
||||
assert "partition_chunking" in md
|
||||
assert "correct: 16" in md
|
||||
assert "correct: 18" in md
|
||||
assert "Gate A2a unit_partition" in md
|
||||
|
||||
|
||||
|
|
@ -122,5 +122,7 @@ def test_gate_a2_lifts_are_not_in_refusal_table():
|
|||
assert "gsm8k-train-sample-v1-0035" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0037" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0045" not in refused_ids
|
||||
assert summary["counts"]["correct"] >= 16
|
||||
assert "gsm8k-train-sample-v1-0005" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0046" not in refused_ids
|
||||
assert summary["counts"]["correct"] >= 18
|
||||
assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0
|
||||
|
|
|
|||
|
|
@ -163,8 +163,8 @@ def test_full_train_sample_wrong_zero_and_chunk_lift():
|
|||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] == 16
|
||||
assert counts["refused"] == 34
|
||||
assert counts["correct"] == 18
|
||||
assert counts["refused"] == 32
|
||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||
|
|
|
|||
|
|
@ -173,7 +173,7 @@ class TestTrainSampleScore:
|
|||
wrong += 1
|
||||
assert wrong == 0
|
||||
|
||||
def test_aggregate_16_34_0(self):
|
||||
def test_aggregate_18_32_0(self):
|
||||
correct = refused = wrong = 0
|
||||
for case in _load_train_cases():
|
||||
res = _run(case["question"])
|
||||
|
|
@ -184,8 +184,8 @@ class TestTrainSampleScore:
|
|||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 16
|
||||
assert refused == 34
|
||||
assert correct == 18
|
||||
assert refused == 32
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -185,7 +185,7 @@ class TestTrainSampleScore:
|
|||
wrong += 1
|
||||
assert wrong == 0
|
||||
|
||||
def test_aggregate_16_34_0(self):
|
||||
def test_aggregate_18_32_0(self):
|
||||
correct = refused = wrong = 0
|
||||
for case in _load_train_cases():
|
||||
res = _run(case["question"])
|
||||
|
|
@ -196,8 +196,8 @@ class TestTrainSampleScore:
|
|||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 16
|
||||
assert refused == 34
|
||||
assert correct == 18
|
||||
assert refused == 32
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
|
|
|
|||
245
tests/test_math_candidate_graph_sprint8_r6_affine_lift.py
Normal file
245
tests/test_math_candidate_graph_sprint8_r6_affine_lift.py
Normal file
|
|
@ -0,0 +1,245 @@
|
|||
"""Gate A2k/A2l — Capability Paradigm Sprint 8 R6 affine/fraction/partition lift."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.derivation.fraction_decrease import (
|
||||
compose_fraction_decrease,
|
||||
resolve_promotable_fraction_decrease,
|
||||
)
|
||||
from generate.derivation.percent_partition import (
|
||||
compose_percent_partition,
|
||||
resolve_promotable_percent_partition,
|
||||
)
|
||||
|
||||
_CASES_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "evals"
|
||||
/ "gsm8k_math"
|
||||
/ "train_sample"
|
||||
/ "v1"
|
||||
/ "cases.jsonl"
|
||||
)
|
||||
_HOLDOUT_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "evals"
|
||||
/ "gsm8k_math"
|
||||
/ "holdout_dev"
|
||||
/ "v1"
|
||||
/ "cases.jsonl"
|
||||
)
|
||||
|
||||
CASE_0005 = (
|
||||
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. "
|
||||
"If the current temperature of the mountain is 84 degrees, what will the temperature "
|
||||
"decrease by?"
|
||||
)
|
||||
|
||||
CASE_0046 = (
|
||||
"A school has 100 students. Half of the students are girls, the other half are boys. "
|
||||
"20% of the girls have dogs at home and 10% of the boys have dogs at home. "
|
||||
"How many students own dogs?"
|
||||
)
|
||||
|
||||
SIBLING_FRACTION_DECREASE = (
|
||||
"In two hours, Cedar peak's temperature will decrease to 2/3 of its temperature. "
|
||||
"If the current temperature of the peak is 60 degrees, what will the temperature "
|
||||
"decrease by?"
|
||||
)
|
||||
|
||||
SIBLING_PERCENT_PARTITION = (
|
||||
"A club has 80 members. Half of the members are adults, the other half are teens. "
|
||||
"25% of the adults have snacks and 15% of the teens have snacks. "
|
||||
"How many members have snacks?"
|
||||
)
|
||||
|
||||
AFFINE_CONFUSER_0010 = (
|
||||
"Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what "
|
||||
"Yun currently has, plus 7. How many paperclips does Marion have?"
|
||||
)
|
||||
|
||||
FINAL_VALUE_CONFUSER = (
|
||||
"In one hour, the lake's temperature will decrease to 3/4 of its temperature. "
|
||||
"If the current temperature of the lake is 80 degrees, what will the temperature be?"
|
||||
)
|
||||
|
||||
UNEQUAL_SPLIT_CONFUSER = (
|
||||
"A school has 100 students. 60 of the students are girls and 40 are boys. "
|
||||
"20% of the girls have dogs and 10% of the boys have dogs. "
|
||||
"How many students own dogs?"
|
||||
)
|
||||
|
||||
SEALED_WRONG_0032_SHAPE = (
|
||||
"He draws and colors 10 pictures. He draws and colors 7 pictures, but each "
|
||||
"picture takes 30% less time than the previous one."
|
||||
)
|
||||
|
||||
SEALED_WRONG_0047_SHAPE = (
|
||||
"John bakes 12 coconut macaroons, each weighing 5 ounces. He then packs an equal "
|
||||
"number of the macaroons in 4 different brown bags, ready for delivery. When he "
|
||||
"briefly leaves the kitchen to pick the phone, his little brother Steve eats the "
|
||||
"entire contents of one of the brown bags. What is the total weight, in ounces, "
|
||||
"of the remaining coconut macaroons?"
|
||||
)
|
||||
|
||||
PRESERVED_SOLVED = (
|
||||
"0002",
|
||||
"0003",
|
||||
"0008",
|
||||
"0014",
|
||||
"0015",
|
||||
"0018",
|
||||
"0021",
|
||||
"0024",
|
||||
"0025",
|
||||
"0029",
|
||||
"0030",
|
||||
"0035",
|
||||
"0037",
|
||||
"0038",
|
||||
"0042",
|
||||
"0045",
|
||||
)
|
||||
|
||||
|
||||
def _run(text: str):
|
||||
return parse_and_solve(text, sealed=False)
|
||||
|
||||
|
||||
def _load_train_cases() -> list[dict]:
|
||||
return [json.loads(line) for line in _CASES_PATH.read_text().splitlines() if line.strip()]
|
||||
|
||||
|
||||
class TestTargetCases:
|
||||
def test_train_sample_0005_end_to_end(self):
|
||||
res = _run(CASE_0005)
|
||||
assert res.answer == 21.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
def test_train_sample_0046_end_to_end(self):
|
||||
res = _run(CASE_0046)
|
||||
assert res.answer == 15.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
|
||||
class TestSiblingGeneralization:
|
||||
def test_fraction_decrease_sibling(self):
|
||||
res = _run(SIBLING_FRACTION_DECREASE)
|
||||
assert res.answer is not None
|
||||
assert abs(res.answer - 20.0) < 1e-6
|
||||
assert res.refusal_reason is None
|
||||
|
||||
def test_percent_partition_sibling(self):
|
||||
res = _run(SIBLING_PERCENT_PARTITION)
|
||||
assert res.answer == 16.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
|
||||
class TestConfuserRefusals:
|
||||
def test_affine_more_than_fraction_refuses(self):
|
||||
assert resolve_promotable_fraction_decrease(AFFINE_CONFUSER_0010) is None
|
||||
assert _run(AFFINE_CONFUSER_0010).answer is None
|
||||
|
||||
def test_final_value_question_refuses_fraction_decrease(self):
|
||||
assert resolve_promotable_fraction_decrease(FINAL_VALUE_CONFUSER) is None
|
||||
assert _run(FINAL_VALUE_CONFUSER).answer is None
|
||||
|
||||
def test_unequal_split_refuses_percent_partition(self):
|
||||
assert resolve_promotable_percent_partition(UNEQUAL_SPLIT_CONFUSER) is None
|
||||
assert _run(UNEQUAL_SPLIT_CONFUSER).answer is None
|
||||
|
||||
def test_goal_residual_not_fraction_decrease(self):
|
||||
goal = (
|
||||
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and "
|
||||
"4 pounds in April. How much weight does he have to lose in May to meet "
|
||||
"his goal?"
|
||||
)
|
||||
assert resolve_promotable_fraction_decrease(goal) is None
|
||||
assert _run(goal).answer == 3.0
|
||||
|
||||
|
||||
class TestSealedWrongPatternRefusal:
|
||||
def test_percent_time_decrease_not_partition(self):
|
||||
assert resolve_promotable_percent_partition(SEALED_WRONG_0032_SHAPE) is None
|
||||
assert _run(SEALED_WRONG_0032_SHAPE).answer is None
|
||||
|
||||
def test_dcs_partition_not_percent_partition(self):
|
||||
assert resolve_promotable_percent_partition(SEALED_WRONG_0047_SHAPE) is None
|
||||
assert _run(SEALED_WRONG_0047_SHAPE).answer is None
|
||||
|
||||
|
||||
class TestCompositionValidationPins:
|
||||
def test_cv_0007_fraction_decrease(self):
|
||||
res = _run(CASE_0005)
|
||||
assert res.answer == 21.0
|
||||
|
||||
def test_cv_0008_percent_partition(self):
|
||||
res = _run(CASE_0046)
|
||||
assert res.answer == 15.0
|
||||
|
||||
|
||||
class TestTrainSampleScore:
|
||||
def test_full_train_sample_wrong_zero(self):
|
||||
wrong = 0
|
||||
for case in _load_train_cases():
|
||||
res = _run(case["question"])
|
||||
if res.answer is not None and res.answer != float(case["answer_numeric"]):
|
||||
wrong += 1
|
||||
assert wrong == 0
|
||||
|
||||
def test_aggregate_18_32_0(self):
|
||||
correct = refused = wrong = 0
|
||||
for case in _load_train_cases():
|
||||
res = _run(case["question"])
|
||||
gold = float(case["answer_numeric"])
|
||||
if res.answer is None:
|
||||
refused += 1
|
||||
elif res.answer == gold:
|
||||
correct += 1
|
||||
else:
|
||||
wrong += 1
|
||||
assert correct == 18
|
||||
assert refused == 32
|
||||
assert wrong == 0
|
||||
|
||||
|
||||
class TestPriorSolvedRegression:
|
||||
@pytest.mark.parametrize("case_suffix", PRESERVED_SOLVED)
|
||||
def test_prior_solved_still_correct(self, case_suffix: str):
|
||||
cases = {c["case_id"].split("-")[-1]: c for c in _load_train_cases()}
|
||||
case = cases[case_suffix]
|
||||
res = _run(case["question"])
|
||||
assert res.answer == float(case["answer_numeric"])
|
||||
|
||||
|
||||
class TestHoldoutDevSafety:
|
||||
def test_holdout_dev_no_new_admissions(self):
|
||||
if not _HOLDOUT_PATH.exists():
|
||||
pytest.skip("holdout_dev corpus unavailable")
|
||||
admitted = 0
|
||||
for line in _HOLDOUT_PATH.read_text().splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
case = json.loads(line)
|
||||
text = case["problem"]
|
||||
if (
|
||||
compose_fraction_decrease(text) is not None
|
||||
or compose_percent_partition(text) is not None
|
||||
):
|
||||
admitted += 1
|
||||
assert admitted == 0
|
||||
|
||||
|
||||
class TestComposeAPI:
|
||||
def test_fraction_decrease_compose_matches_promote(self):
|
||||
assert compose_fraction_decrease(CASE_0005) is not None
|
||||
assert resolve_promotable_fraction_decrease(CASE_0005) is not None
|
||||
|
||||
def test_percent_partition_compose_matches_promote(self):
|
||||
assert compose_percent_partition(CASE_0046) is not None
|
||||
assert resolve_promotable_percent_partition(CASE_0046) is not None
|
||||
50
tests/test_math_candidate_graph_sprint8_review_patch.py
Normal file
50
tests/test_math_candidate_graph_sprint8_review_patch.py
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
"""Review-hardening tests for Sprint 8 A2k/A2l gates."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from generate.derivation.fraction_decrease import resolve_promotable_fraction_decrease
|
||||
from generate.derivation.percent_partition import resolve_promotable_percent_partition
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
|
||||
|
||||
def _run(text: str):
|
||||
return parse_and_solve(text, sealed=False)
|
||||
|
||||
|
||||
def test_fraction_decrease_refuses_spaced_extra_fraction_hazard():
|
||||
text = (
|
||||
"In one hour, Addison mountain's temperature will decrease to 3/4 of its "
|
||||
"temperature. Last week it decreased to 1 / 2 of its temperature. If the "
|
||||
"current temperature of the mountain is 84 degrees, what will the temperature "
|
||||
"decrease by?"
|
||||
)
|
||||
assert resolve_promotable_fraction_decrease(text) is None
|
||||
assert _run(text).answer is None
|
||||
|
||||
|
||||
def test_fraction_decrease_does_not_use_forecast_duration_as_base():
|
||||
text = (
|
||||
"In one hour, Addison mountain's temperature will decrease to 3/4 of its "
|
||||
"temperature. What will the temperature decrease by?"
|
||||
)
|
||||
assert resolve_promotable_fraction_decrease(text) is None
|
||||
assert _run(text).answer is None
|
||||
|
||||
|
||||
def test_percent_partition_refuses_spaced_fraction_surface():
|
||||
text = (
|
||||
"A school has 100 students. Half of the students are girls, the other half are boys. "
|
||||
"20% of the girls have dogs, 10% of the boys have dogs, and 1 / 2 of the dogs are black. "
|
||||
"How many students own dogs?"
|
||||
)
|
||||
assert resolve_promotable_percent_partition(text) is None
|
||||
assert _run(text).answer is None
|
||||
|
||||
|
||||
def test_percent_partition_does_not_use_subgroup_quantity_as_total():
|
||||
text = (
|
||||
"Half of the students are girls, the other half are boys. The girls group has 50 students. "
|
||||
"20% of the girls have dogs and 10% of the boys have dogs. How many students own dogs?"
|
||||
)
|
||||
assert resolve_promotable_percent_partition(text) is None
|
||||
assert _run(text).answer is None
|
||||
Loading…
Reference in a new issue