core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

feat(derivation): capability paradigm sprint 8 R6 affine/fraction lift

Browse source 
Add Gate A2k fraction_decrease and Gate A2l percent_partition serving
organs from Experience Flywheel + composition-validation decomposition.
Lifts train_sample 0005 and 0046 (cv-0007/cv-0008) to 18/32/0 with wrong=0.
This commit is contained in:
Shay 2026-06-17 23:00:49 -07:00
parent 5d529b425f
commit 1936a4a5cf
10 changed files with 827 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

177
docs/analysis/gsm8k-capability-paradigm-sprint8-lookback-2026-06-17.md Normal file
View file

@ -0,0 +1,177 @@
# GSM8K Capability Paradigm Sprint 8 — Lookback (2026-06-17)
## 1. Starting baseline
| Metric | Value |
|--------|------:|
| correct | **16** |
| refused | **34** |
| wrong | **0** |
After #818 (Sprint 7: round_trip_trip_duration + giveaway_target_residual). Preserved solved: 0002, 0003, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045.
## 2. Experience Flywheel output summary
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint8.json`
| Field | Value |
|-------|------:|
| retained_record_count | 50 |
| lift_refused_to_correct | **0** |
| promotion_candidates | 4 families, **all** `blocked_by_wrong_risk` |
Top missing primitives: `relation_hypothesis` (DCS), `multiplicative_aggregate`, `temporal_tariff`, `diagnostic_hold`.
**Key signal:** No scout refused→correct delta (prior sprints already promoted clean lifts). Composition-validation pins **cv-0007** (0005 fraction-decrease) and **cv-0008** (0046 percent-partition) provided typed R6 chains with low wrong-risk — aligned with Sprint 7 handoff.
## 3. Scout/frontier output summary
| Regime | correct | refused | wrong |
|--------|--------:|--------:|------:|
| Serving | 16 | 34 | 0 |
| Sealed (resolve_pooled) | 3 | 39 | 8 |
- `lift_refused_to_correct`: **0**
- Dominant bottleneck: joint refusal (28/50)
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
Microscope slices: `affine_equation_fraction_target` (0005), `affine_equation` / percent partition (0046).
## 4. Candidate families ranked
| Rank | Family | Cases | Lift signal | Wrong-risk | Sprint 8 verdict |
|------|--------|-------|-------------|------------|------------------|
| 1 | R6 fraction-decrease (`cv-0007`) | **0005** | typed `base × (1−N/M)` | low | **Selected (A2k)** |
| 2 | R6 percent-partition (`cv-0008`) | **0046** | typed half-split + dual % | low | **Selected (A2l)** |
| 3 | `temporal_tariff:temporal_aggregation` | 0001, 0017 | joint only | medium | Deferred — confuser design |
| 4 | `affine_equation_fraction_delta` | 0010 | joint only | medium | Deferred — 1/4 more than affine |
| 5 | `relation_hypothesis:DCS` | 14 joint + 2 wrong | none | **blocked** (0032, 0047) | Deferred |
| 6 | `diagnostic_hold:currency_amount` | 0019, 0028 | sealed-wrong | **blocked** | Deferred |
## 5. Blocked families and why
| Family | Blocked because |
|--------|-----------------|
| `relation_hypothesis:DCS` | 0032, 0047 sealed-wrong share discrete-count / partition surface |
| `conservative_boundary` | 0018, 0025 sealed-wrong on served-correct cases |
| `sealed_elimination` | 0011, 0026 hard elimination |
| `diagnostic_hold:currency_amount` | 0019, 0028 sealed-wrong |
## 6. Selected paradigm
**Dual narrow R6 organs** (composition-validation decomposition, not generic affine parser):
1. **Gate A2k `fraction_decrease`** — forecast ``decrease to N/M of`` + explicit current base + question asks **decrease by** delta (`cv-0007` / 0005).
2. **Gate A2l `percent_partition`** — total population + equal ``half`` / ``other half`` split + two ``N% of the <group>`` clauses + aggregate ownership question (`cv-0008` / 0046).
Rejected: broad product_bridge, DCS injector, generic affine equation parser, temporal_tariff wholesale promotion.
## 7. Typed chains for target cases
**0005 (gold 21):** `84 × (1 − 3/4) = 84 × 1/4 = 21` degrees decrease.
**0046 (gold 15):** `100 × 0.5 = 50` per group → `50 × 0.20 = 10` girl dog owners → `+ 50 × 0.10 = 5` boy dog owners → `15` total.
## 8. Implemented organs
| Module | Promotion entry |
|--------|-----------------|
| `generate/derivation/fraction_decrease.py` | `resolve_promotable_fraction_decrease` |
| `generate/derivation/percent_partition.py` | `resolve_promotable_percent_partition` |
Wired in `generate/math_candidate_graph.py` after Gate A2j, before rate short-circuits.
## 9. Before/after score
| Metric | Before | After | Δ |
|--------|-------:|------:|--:|
| correct | 16 | **18** | +2 |
| refused | 34 | **32** | −2 |
| wrong | 0 | **0** | 0 |
**State A met:** 18+/32-/0.
## 10. Newly solved IDs
**0005**, **0046**
## 11. Preserved solved IDs
0002, 0003, 0008, 0014, 0015, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045
## 12. Wrong=0 proof
- Full train_sample replay: 18 correct / 32 refused / 0 wrong
- `tests/test_math_candidate_graph_sprint8_r6_affine_lift.py::TestTrainSampleScore`
- Composition-validation firewall: 14 solve / 8 refuse / 0 wrong
- Holdout_dev: 0 new organ admissions
- Smoke suite: 108 passed
## 13. Confuser matrix
| Confuser | Organ | Result |
|----------|-------|--------|
| 0010 affine `1/4 more than` | A2k | refused |
| Final-value question (not decrease-by) | A2k | refused |
| Unequal 60/40 split (not half/half) | A2l | refused |
| 0032 percent time-decrease | A2l | refused |
| 0047 DCS equal-pack | A2l | refused |
| Goal-residual (cv-0005) | A2k | refused (goal_residual serves) |
## 14. Sibling generality proof
| Sibling | Gold | Organ |
|---------|-----:|-------|
| Cedar peak 2/3 decrease, 60° base | 20 | A2k |
| Club 80 members, 25%/15% half-split | 16 | A2l |
## 15. Sealed-wrong negative evidence used
- 0032 percent-decrease shape → A2l hazard refusal (not sequential % chain)
- 0047 equal-pack DCS → A2l hazard refusal (no half-split / no % of group)
- DCS flywheel `blocked_by_wrong_risk` → no broad partition injector
## 16. Composition-validation pin changes
| Pin | Before | After |
|-----|--------|-------|
| cv-0007 (0005) | refuse | **solve** (21) |
| cv-0008 (0046) | refuse | **solve** (15) |
Aggregate snapshot: 12/10/0 → **14/8/0**
## 17. report.json untouched
`evals/gsm8k_math/train_sample/v1/report.json` not modified.
## 18. Sealed artifacts untouched
No changes to sealed practice lanes, holdout, or `resolve_pooled` wiring.
## 19. No case-id logic
Promotion is family-typed only; no `case_id` conditionals in serving path.
## 20. No hardcoded answer
Answers derived from grounded arithmetic chains; no answer literals.
## 21. No direct-answer shortcut
No `answer_numeric` extraction or final-answer parsing.
## 22. Sprint 9 recommendation
**Track A — Capability Sprint 9** (continue capability paradigm)
| Evidence | Detail |
|----------|--------|
| Flywheel | `lift_refused_to_correct == 0`; dense `joint_sealed_no_resolution` cluster remains |
| Next targets | `temporal_tariff` (0001, 0017) after confuser matrix; `affine_equation_fraction_delta` (0010) |
| Secondary | `multiplicative_aggregate` joint cluster (0006, 0013) — only with hard verifier |
| Blocked | DCS (0032/0047), currency_amount (0019/0028), sealed_elimination (0011/0026) |
| Expected lift | +1–2 correct if temporal_tariff organ passes adversarial review |
| Modules | `generate/derivation/temporal_tariff.py` (proposed) |
| Non-goals | Flywheel PR-2, broad product_bridge, DCS injector, report.json rebaseline |
Track B (Experience Flywheel PR-2) and Track C (Diagnostic Hardening) deferred — flywheel still shows promotable typed chains in joint-refusal cluster, not exclusively wrong-risk blockage.

183
generate/derivation/fraction_decrease.py Normal file
View file

@ -0,0 +1,183 @@
"""Gate A2k — fraction-decrease delta (decrease-to fraction of base).
Experience Flywheel Sprint 8 / composition-validation **cv-0007** (train_sample
**0005**): forecast clause states a quantity will ``decrease to N/M of`` its
current value; a follow-on clause states the current amount; the question asks
the **decrease by** delta (not the final value).
decrease_by = base × (1 − N/M)
Narrow organ — not a generic affine equation parser, not ``N/M more than`` affine
surfaces (0010-class), not percent-decrease. Promotion requires:
- exactly one ``decrease to N/M of`` forecast (no ``more than`` fraction affine);
- question binds ``decrease`` + ``by`` (delta target, not final-value question);
- exactly one explicit current/base quantity with a unit;
- hazard refusal (percent, goal language, multiple decrease-to fractions).
Deterministic; sealed module (no ``chat/`` import).
"""
from __future__ import annotations
import re
from typing import Final
from generate.derivation.clauses import segment_clauses
from generate.derivation.extract import extract_quantities
from generate.derivation.model import GroundedDerivation, Quantity, Step
from generate.derivation.target import _question_clause
from generate.derivation.verify import Resolution, SelfVerification, select_self_verified
from generate.math_roundtrip import _tokens
_DECREASE_TO_FRACTION_RE: Final[re.Pattern[str]] = re.compile(
r"decrease\s+to\s+(\d+)\s*/\s*(\d+)\s+of",
re.IGNORECASE,
)
_EXTRA_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+/\d+")
_GOAL_INTENT: Final[frozenset[str]] = frozenset(
{"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"}
)
_FINAL_VALUE_CUES: Final[frozenset[str]] = frozenset({"be", "will", "what"})
def _asks_decrease_delta(question_clause: str) -> bool:
tokens = _tokens(question_clause)
return "decrease" in tokens and "by" in tokens
def _asks_final_value(question_clause: str) -> bool:
tokens = _tokens(question_clause)
if "by" in tokens:
return False
if "decrease" not in tokens:
return False
return bool(_FINAL_VALUE_CUES & tokens) or "temperature" in tokens
def _target_fraction(problem_text: str) -> tuple[int, int, str] | None:
matches = list(_DECREASE_TO_FRACTION_RE.finditer(problem_text))
if len(matches) != 1:
return None
num_s, den_s = matches[0].group(1), matches[0].group(2)
try:
num = int(num_s)
den = int(den_s)
except ValueError:
return None
if num <= 0 or den <= 0 or num >= den:
return None
return num, den, f"{num_s}/{den_s}"
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
text_tokens = _tokens(problem_text)
question_tokens = _tokens(question_clause)
if "more" in text_tokens and "than" in text_tokens:
return True
if "%" in problem_text or "percent" in text_tokens or "percentage" in text_tokens:
return True
if text_tokens & _GOAL_INTENT:
return True
if _asks_final_value(question_clause):
return True
fractions = _EXTRA_FRACTION_RE.findall(problem_text)
target = _target_fraction(problem_text)
if target is None:
return True
_, _, token = target
extra = [f for f in fractions if f.replace(" ", "") != token.replace(" ", "")]
return len(extra) > 0
def _current_base_quantity(problem_text: str, question_clause: str) -> Quantity | None:
for clause in segment_clauses(problem_text):
tokens = _tokens(clause)
if "current" not in tokens and "now" not in tokens:
continue
quantities = [q for q in extract_quantities(clause) if q.unit]
if len(quantities) == 1:
return quantities[0]
for clause in segment_clauses(problem_text):
if clause == question_clause:
continue
quantities = [q for q in extract_quantities(clause) if q.unit]
if len(quantities) == 1:
return quantities[0]
return None
def build_fraction_decrease(problem_text: str) -> GroundedDerivation | None:
"""Construct ``base × (1 − N/M)`` decrease delta, or ``None``."""
question_clause = _question_clause(problem_text)
if not _asks_decrease_delta(question_clause):
return None
if _has_hazard_surface(problem_text, question_clause):
return None
fraction = _target_fraction(problem_text)
base = _current_base_quantity(problem_text, question_clause)
if fraction is None or base is None:
return None
num, den, source_token = fraction
delta_factor = 1.0 - (num / den)
if delta_factor <= 0:
return None
return GroundedDerivation(
start=base,
steps=(
Step(
op="multiply",
operand=Quantity(
value=delta_factor,
unit="",
source_token=source_token,
),
cue="decrease",
),
),
)
def _self_verifies_fraction_decrease(
derivation: GroundedDerivation, problem_text: str
) -> SelfVerification:
from generate.derivation.verify import _base_reasons
tokens = _tokens(problem_text)
reasons = list(_base_reasons(derivation, tokens))
fraction = _target_fraction(problem_text)
if fraction is None:
reasons.append("missing decrease-to fraction forecast")
else:
_, _, token = fraction
if token not in problem_text.replace(" ", ""):
reasons.append(f"fraction token {token!r} not grounded in text")
base = _current_base_quantity(problem_text, _question_clause(problem_text))
if base is None:
reasons.append("missing explicit current/base quantity")
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
def compose_fraction_decrease(problem_text: str) -> Resolution | None:
"""Gate the typed fraction-decrease chain through self-verification."""
derivation = build_fraction_decrease(problem_text)
if derivation is None:
return None
if not _self_verifies_fraction_decrease(derivation, problem_text).verified:
return None
return Resolution(
answer=derivation.answer,
answer_unit=derivation.answer_unit,
derivation=derivation,
)
def resolve_promotable_fraction_decrease(problem_text: str) -> Resolution | None:
"""Serving promotion bridge (Gate A2k)."""
return compose_fraction_decrease(problem_text)

177
generate/derivation/percent_partition.py Normal file
View file

@ -0,0 +1,177 @@
"""Gate A2l — equal half-split percent partition aggregate.
Experience Flywheel Sprint 8 / composition-validation **cv-0008** (train_sample
**0046**): total population, equal ``half`` girls / ``other half`` boys split,
two subgroup ``N% of the <group>`` clauses, question asks how many total own the
attribute.
subgroup = total × half
answer = subgroup × pct_girls + subgroup × pct_boys
Narrow organ — not a generic percent equation parser, not unequal partitions, not
multi-container DCS. Promotion requires:
- question asks ``how many`` + aggregate ownership cue;
- total population quantity with unit;
- equal half / other-half split language;
- exactly two percent-of-subgroup clauses with distinct group referents;
- hazard refusal (fraction surfaces, money, profit, goal language).
Deterministic; sealed module (no ``chat/`` import).
"""
from __future__ import annotations
import re
from typing import Final
from generate.derivation.clauses import segment_clauses
from generate.derivation.comparatives import comparative_step, extract_comparative_scalars
from generate.derivation.extract import extract_quantities
from generate.derivation.model import GroundedDerivation, Quantity, Step
from generate.derivation.target import _question_clause
from generate.derivation.verify import Resolution, SelfVerification
from generate.math_roundtrip import _tokens
_PERCENT_OF_GROUP_RE: Final[re.Pattern[str]] = re.compile(
r"(\d+(?:\.\d+)?)\s*%\s+of\s+the\s+(\w+)",
re.IGNORECASE,
)
_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+/\d+")
_GOAL_INTENT: Final[frozenset[str]] = frozenset(
{"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"}
)
_OWNERSHIP_CUES: Final[frozenset[str]] = frozenset(
{"own", "owns", "have", "has", "had"}
)
def _asks_partition_total(question_clause: str) -> bool:
tokens = _tokens(question_clause)
return "how" in tokens and "many" in tokens and bool(_OWNERSHIP_CUES & tokens)
def _has_equal_half_split(problem_text: str) -> bool:
lowered = problem_text.lower()
return "half" in lowered and "other half" in lowered
def _percent_clauses(problem_text: str) -> tuple[tuple[float, str, str], tuple[float, str, str]] | None:
question_clause = _question_clause(problem_text)
matches: list[tuple[float, str, str]] = []
for clause in segment_clauses(problem_text):
if clause == question_clause:
continue
for match in _PERCENT_OF_GROUP_RE.finditer(clause):
pct = float(match.group(1)) / 100.0
group = match.group(2).lower()
source = match.group(1)
matches.append((pct, group, source))
if len(matches) != 2:
return None
groups = {m[1] for m in matches}
if len(groups) != 2:
return None
return matches[0], matches[1]
def _total_population(problem_text: str, question_clause: str) -> Quantity | None:
for clause in segment_clauses(problem_text):
if clause == question_clause:
continue
quantities = [q for q in extract_quantities(clause) if q.unit]
if len(quantities) == 1:
return quantities[0]
return None
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
text_tokens = _tokens(problem_text)
if _FRACTION_RE.search(problem_text):
return True
if text_tokens & _GOAL_INTENT:
return True
if "profit" in text_tokens or "money" in text_tokens or "$" in problem_text:
return True
if not _has_equal_half_split(problem_text):
return True
comparatives = extract_comparative_scalars(problem_text)
if not any(cs.cue == "half" for cs in comparatives):
return True
return False
def build_percent_partition(problem_text: str) -> GroundedDerivation | None:
"""Construct equal half-split percent partition total, or ``None``."""
question_clause = _question_clause(problem_text)
if not _asks_partition_total(question_clause):
return None
if _has_hazard_surface(problem_text, question_clause):
return None
total = _total_population(problem_text, question_clause)
percent_pair = _percent_clauses(problem_text)
if total is None or percent_pair is None:
return None
(pct_a, _, source_a), (pct_b, _, source_b) = percent_pair
comparatives = extract_comparative_scalars(problem_text)
half = next((cs for cs in comparatives if cs.cue == "half"), None)
if half is None:
return None
subgroup = total.value * half.scalar
contrib_a = subgroup * pct_a
contrib_b = subgroup * pct_b
return GroundedDerivation(
start=total,
steps=(
comparative_step(half),
Step(
op="multiply",
operand=Quantity(value=pct_a, unit="", source_token=source_a),
cue="of",
),
Step(
op="add",
operand=Quantity(value=contrib_b, unit=total.unit, source_token=source_b),
cue="and",
),
),
)
def _self_verifies_percent_partition(
derivation: GroundedDerivation, problem_text: str
) -> SelfVerification:
from generate.derivation.verify import _base_reasons
tokens = _tokens(problem_text)
reasons = list(_base_reasons(derivation, tokens))
if _percent_clauses(problem_text) is None:
reasons.append("expected two distinct percent-of-subgroup clauses")
if _total_population(problem_text, _question_clause(problem_text)) is None:
reasons.append("missing total population quantity")
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
def compose_percent_partition(problem_text: str) -> Resolution | None:
"""Gate the typed percent-partition chain through self-verification."""
derivation = build_percent_partition(problem_text)
if derivation is None:
return None
if not _self_verifies_percent_partition(derivation, problem_text).verified:
return None
return Resolution(
answer=derivation.answer,
answer_unit=derivation.answer_unit,
derivation=derivation,
)
def resolve_promotable_percent_partition(problem_text: str) -> Resolution | None:
"""Serving promotion bridge (Gate A2l)."""
return compose_percent_partition(problem_text)

30
generate/math_candidate_graph.py
View file

@ -789,6 +789,36 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
branches_admissible=1, branches_admissible=1,
) )
# Gate A2k — fraction decrease delta (Sprint 8 R6 lift).
from generate.derivation.fraction_decrease import (
resolve_promotable_fraction_decrease,
)
fraction_decrease_resolution = resolve_promotable_fraction_decrease(text)
if fraction_decrease_resolution is not None:
return CandidateGraphResult(
answer=fraction_decrease_resolution.answer,
selected_graph=None,
refusal_reason=None,
branches_enumerated=1,
branches_admissible=1,
)
# Gate A2l — equal half-split percent partition (Sprint 8 R6 lift).
from generate.derivation.percent_partition import (
resolve_promotable_percent_partition,
)
percent_partition_resolution = resolve_promotable_percent_partition(text)
if percent_partition_resolution is not None:
return CandidateGraphResult(
answer=percent_partition_resolution.answer,
selected_graph=None,
refusal_reason=None,
branches_enumerated=1,
branches_admissible=1,
)
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product). # ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
# Capacity path: single statement with one CandidateCapacity + matching question. # Capacity path: single statement with one CandidateCapacity + matching question.
if len(statement_sentences) == 1: if len(statement_sentences) == 1:

4
tests/test_composition_validation_corpus.py
View file

@ -168,6 +168,8 @@ def test_current_baseline_snapshot() -> None:
Sprint 6 (2026-06-17): Gate A2g duration_segment_total admits cv-0022 (0015). Sprint 6 (2026-06-17): Gate A2g duration_segment_total admits cv-0022 (0015).
Sprint 7 (2026-06-17): Gate A2i round_trip_trip_duration admits cv-0006 (0030); Sprint 7 (2026-06-17): Gate A2i round_trip_trip_duration admits cv-0006 (0030);
Gate A2j giveaway_target_residual admits cv-0021 (0035). Gate A2j giveaway_target_residual admits cv-0021 (0035).
Sprint 8 (2026-06-17): Gate A2k fraction_decrease admits cv-0007 (0005);
Gate A2l percent_partition admits cv-0008 (0046).
""" """
solve = refuse = wrong = 0 solve = refuse = wrong = 0
for case in _CASES: for case in _CASES:
@ -179,7 +181,7 @@ def test_current_baseline_snapshot() -> None:
else: else:
refuse += 1 refuse += 1
assert wrong == 0 assert wrong == 0
assert (solve, refuse) == (12, 10), ( assert (solve, refuse) == (14, 8), (
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b " f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
f"slice landed, update this expectation and the affected rows' " f"slice landed, update this expectation and the affected rows' "
f"baseline fields in lockstep" f"baseline fields in lockstep"

6
tests/test_gsm8k_post_gate_a1_frontier_microscope.py
View file

@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
summary = build_microscope_report(_load_cases()) summary = build_microscope_report(_load_cases())
md = render_markdown(summary) md = render_markdown(summary)
assert "partition_chunking" in md assert "partition_chunking" in md
assert "correct: 16" in md assert "correct: 18" in md
assert "Gate A2a unit_partition" in md assert "Gate A2a unit_partition" in md
@ -122,5 +122,7 @@ def test_gate_a2_lifts_are_not_in_refusal_table():
assert "gsm8k-train-sample-v1-0035" not in refused_ids assert "gsm8k-train-sample-v1-0035" not in refused_ids
assert "gsm8k-train-sample-v1-0037" not in refused_ids assert "gsm8k-train-sample-v1-0037" not in refused_ids
assert "gsm8k-train-sample-v1-0045" not in refused_ids assert "gsm8k-train-sample-v1-0045" not in refused_ids
assert summary["counts"]["correct"] >= 16 assert "gsm8k-train-sample-v1-0005" not in refused_ids
assert "gsm8k-train-sample-v1-0046" not in refused_ids
assert summary["counts"]["correct"] >= 18
assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0 assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0

4
tests/test_math_candidate_graph_question_bound_product_lift.py
View file

@ -163,8 +163,8 @@ def test_full_train_sample_wrong_zero_and_chunk_lift():
report = build_report(_load_cases(_CASES_PATH)) report = build_report(_load_cases(_CASES_PATH))
counts = report["counts"] counts = report["counts"]
assert counts["wrong"] == 0 assert counts["wrong"] == 0
assert counts["correct"] == 16 assert counts["correct"] == 18
assert counts["refused"] == 34 assert counts["refused"] == 32
by_case = {row["case_id"]: row for row in report["per_case"]} by_case = {row["case_id"]: row for row in report["per_case"]}
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct" assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct" assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"

6
tests/test_math_candidate_graph_sprint6_experience_guided_lift.py
View file

@ -173,7 +173,7 @@ class TestTrainSampleScore:
wrong += 1 wrong += 1
assert wrong == 0 assert wrong == 0
def test_aggregate_16_34_0(self): def test_aggregate_18_32_0(self):
correct = refused = wrong = 0 correct = refused = wrong = 0
for case in _load_train_cases(): for case in _load_train_cases():
res = _run(case["question"]) res = _run(case["question"])
@ -184,8 +184,8 @@ class TestTrainSampleScore:
correct += 1 correct += 1
else: else:
wrong += 1 wrong += 1
assert correct == 16 assert correct == 18
assert refused == 34 assert refused == 32
assert wrong == 0 assert wrong == 0

6
tests/test_math_candidate_graph_sprint7_experience_guided_lift.py
View file

@ -185,7 +185,7 @@ class TestTrainSampleScore:
wrong += 1 wrong += 1
assert wrong == 0 assert wrong == 0
def test_aggregate_16_34_0(self): def test_aggregate_18_32_0(self):
correct = refused = wrong = 0 correct = refused = wrong = 0
for case in _load_train_cases(): for case in _load_train_cases():
res = _run(case["question"]) res = _run(case["question"])
@ -196,8 +196,8 @@ class TestTrainSampleScore:
correct += 1 correct += 1
else: else:
wrong += 1 wrong += 1
assert correct == 16 assert correct == 18
assert refused == 34 assert refused == 32
assert wrong == 0 assert wrong == 0

245
tests/test_math_candidate_graph_sprint8_r6_affine_lift.py Normal file
View file

@ -0,0 +1,245 @@
"""Gate A2k/A2l — Capability Paradigm Sprint 8 R6 affine/fraction/partition lift."""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from generate.math_candidate_graph import parse_and_solve
from generate.derivation.fraction_decrease import (
compose_fraction_decrease,
resolve_promotable_fraction_decrease,
)
from generate.derivation.percent_partition import (
compose_percent_partition,
resolve_promotable_percent_partition,
)
_CASES_PATH = (
Path(__file__).resolve().parents[1]
/ "evals"
/ "gsm8k_math"
/ "train_sample"
/ "v1"
/ "cases.jsonl"
)
_HOLDOUT_PATH = (
Path(__file__).resolve().parents[1]
/ "evals"
/ "gsm8k_math"
/ "holdout_dev"
/ "v1"
/ "cases.jsonl"
)
CASE_0005 = (
"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. "
"If the current temperature of the mountain is 84 degrees, what will the temperature "
"decrease by?"
)
CASE_0046 = (
"A school has 100 students. Half of the students are girls, the other half are boys. "
"20% of the girls have dogs at home and 10% of the boys have dogs at home. "
"How many students own dogs?"
)
SIBLING_FRACTION_DECREASE = (
"In two hours, Cedar peak's temperature will decrease to 2/3 of its temperature. "
"If the current temperature of the peak is 60 degrees, what will the temperature "
"decrease by?"
)
SIBLING_PERCENT_PARTITION = (
"A club has 80 members. Half of the members are adults, the other half are teens. "
"25% of the adults have snacks and 15% of the teens have snacks. "
"How many members have snacks?"
)
AFFINE_CONFUSER_0010 = (
"Yun had 20 paperclips initially, but then lost 12. Marion has 1/4 more than what "
"Yun currently has, plus 7. How many paperclips does Marion have?"
)
FINAL_VALUE_CONFUSER = (
"In one hour, the lake's temperature will decrease to 3/4 of its temperature. "
"If the current temperature of the lake is 80 degrees, what will the temperature be?"
)
UNEQUAL_SPLIT_CONFUSER = (
"A school has 100 students. 60 of the students are girls and 40 are boys. "
"20% of the girls have dogs and 10% of the boys have dogs. "
"How many students own dogs?"
)
SEALED_WRONG_0032_SHAPE = (
"He draws and colors 10 pictures. He draws and colors 7 pictures, but each "
"picture takes 30% less time than the previous one."
)
SEALED_WRONG_0047_SHAPE = (
"John bakes 12 coconut macaroons, each weighing 5 ounces. He then packs an equal "
"number of the macaroons in 4 different brown bags, ready for delivery. When he "
"briefly leaves the kitchen to pick the phone, his little brother Steve eats the "
"entire contents of one of the brown bags. What is the total weight, in ounces, "
"of the remaining coconut macaroons?"
)
PRESERVED_SOLVED = (
"0002",
"0003",
"0008",
"0014",
"0015",
"0018",
"0021",
"0024",
"0025",
"0029",
"0030",
"0035",
"0037",
"0038",
"0042",
"0045",
)
def _run(text: str):
return parse_and_solve(text, sealed=False)
def _load_train_cases() -> list[dict]:
return [json.loads(line) for line in _CASES_PATH.read_text().splitlines() if line.strip()]
class TestTargetCases:
def test_train_sample_0005_end_to_end(self):
res = _run(CASE_0005)
assert res.answer == 21.0
assert res.refusal_reason is None
def test_train_sample_0046_end_to_end(self):
res = _run(CASE_0046)
assert res.answer == 15.0
assert res.refusal_reason is None
class TestSiblingGeneralization:
def test_fraction_decrease_sibling(self):
res = _run(SIBLING_FRACTION_DECREASE)
assert res.answer is not None
assert abs(res.answer - 20.0) < 1e-6
assert res.refusal_reason is None
def test_percent_partition_sibling(self):
res = _run(SIBLING_PERCENT_PARTITION)
assert res.answer == 16.0
assert res.refusal_reason is None
class TestConfuserRefusals:
def test_affine_more_than_fraction_refuses(self):
assert resolve_promotable_fraction_decrease(AFFINE_CONFUSER_0010) is None
assert _run(AFFINE_CONFUSER_0010).answer is None
def test_final_value_question_refuses_fraction_decrease(self):
assert resolve_promotable_fraction_decrease(FINAL_VALUE_CONFUSER) is None
assert _run(FINAL_VALUE_CONFUSER).answer is None
def test_unequal_split_refuses_percent_partition(self):
assert resolve_promotable_percent_partition(UNEQUAL_SPLIT_CONFUSER) is None
assert _run(UNEQUAL_SPLIT_CONFUSER).answer is None
def test_goal_residual_not_fraction_decrease(self):
goal = (
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and "
"4 pounds in April. How much weight does he have to lose in May to meet "
"his goal?"
)
assert resolve_promotable_fraction_decrease(goal) is None
assert _run(goal).answer == 3.0
class TestSealedWrongPatternRefusal:
def test_percent_time_decrease_not_partition(self):
assert resolve_promotable_percent_partition(SEALED_WRONG_0032_SHAPE) is None
assert _run(SEALED_WRONG_0032_SHAPE).answer is None
def test_dcs_partition_not_percent_partition(self):
assert resolve_promotable_percent_partition(SEALED_WRONG_0047_SHAPE) is None
assert _run(SEALED_WRONG_0047_SHAPE).answer is None
class TestCompositionValidationPins:
def test_cv_0007_fraction_decrease(self):
res = _run(CASE_0005)
assert res.answer == 21.0
def test_cv_0008_percent_partition(self):
res = _run(CASE_0046)
assert res.answer == 15.0
class TestTrainSampleScore:
def test_full_train_sample_wrong_zero(self):
wrong = 0
for case in _load_train_cases():
res = _run(case["question"])
if res.answer is not None and res.answer != float(case["answer_numeric"]):
wrong += 1
assert wrong == 0
def test_aggregate_18_32_0(self):
correct = refused = wrong = 0
for case in _load_train_cases():
res = _run(case["question"])
gold = float(case["answer_numeric"])
if res.answer is None:
refused += 1
elif res.answer == gold:
correct += 1
else:
wrong += 1
assert correct == 18
assert refused == 32
assert wrong == 0
class TestPriorSolvedRegression:
@pytest.mark.parametrize("case_suffix", PRESERVED_SOLVED)
def test_prior_solved_still_correct(self, case_suffix: str):
cases = {c["case_id"].split("-")[-1]: c for c in _load_train_cases()}
case = cases[case_suffix]
res = _run(case["question"])
assert res.answer == float(case["answer_numeric"])
class TestHoldoutDevSafety:
def test_holdout_dev_no_new_admissions(self):
if not _HOLDOUT_PATH.exists():
pytest.skip("holdout_dev corpus unavailable")
admitted = 0
for line in _HOLDOUT_PATH.read_text().splitlines():
if not line.strip():
continue
case = json.loads(line)
text = case["problem"]
if (
compose_fraction_decrease(text) is not None
or compose_percent_partition(text) is not None
):
admitted += 1
assert admitted == 0
class TestComposeAPI:
def test_fraction_decrease_compose_matches_promote(self):
assert compose_fraction_decrease(CASE_0005) is not None
assert resolve_promotable_fraction_decrease(CASE_0005) is not None
def test_percent_partition_compose_matches_promote(self):
assert compose_percent_partition(CASE_0046) is not None
assert resolve_promotable_percent_partition(CASE_0046) is not None