feat(gsm8k): capability paradigm sprint 11 cluster-contract lift
Ratify calendar_grounded_piecewise_daily_hours_total ClusterContract and Gate A2q organ with civil month table provenance. Solves 0013 (24/26/0, wrong=0) while refusing February, odd-month halfway, tariff/duration/MA neighbors. report.json and sealed artifacts untouched.
This commit is contained in:
Shay
parent
a37f390009
commit
b8443fdd58
7 changed files with 928 additions and 5 deletions
|
|
@ -0,0 +1,252 @@
|
|||
# GSM8K Capability Paradigm Sprint 11 — Lookback (2026-06-17)
|
||||
|
||||
## 1. Starting baseline
|
||||
|
||||
| Metric | Value |
|
||||
|--------|------:|
|
||||
| correct | **23** |
|
||||
| refused | **27** |
|
||||
| wrong | **0** |
|
||||
|
||||
After #823 (Sprint 10 dual-organ lift). Prerequisite satisfied.
|
||||
|
||||
## 2. Experience Flywheel output summary
|
||||
|
||||
`scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-sprint11.json`
|
||||
|
||||
| Field | Value |
|
||||
|-------|------:|
|
||||
| retained_record_count | 50 |
|
||||
| lift_refused_to_correct | **0** |
|
||||
| scout serving | 23 / 27 / 0 |
|
||||
|
||||
Top signals:
|
||||
|
||||
- `multiplicative_aggregate:multiplicative_aggregation` — 0013 sole member; `not_promotable`, 0 sealed-wrong on surface tag.
|
||||
- `already_served` — 23 cases; no flywheel promotion path.
|
||||
- Blocked: DCS/relation_hypothesis (0032, 0047 neighbors), sealed_elimination (0011, 0026), currency (0019, 0028).
|
||||
|
||||
0013 remains the cleanest calendar/piecewise candidate; no sealed-wrong neighbor on the MA surface tag.
|
||||
|
||||
## 3. Scout/frontier/microscope output summary
|
||||
|
||||
| Regime | correct | refused | wrong |
|
||||
|--------|--------:|--------:|------:|
|
||||
| Serving | 23 | 27 | 0 |
|
||||
| Sealed (resolve_pooled) | 3 | 39 | 8 |
|
||||
|
||||
- `lift_refused_to_correct`: **0**
|
||||
- Dominant bottleneck: joint refusal (21/50)
|
||||
- Sealed-wrong unchanged: 0011, 0018, 0019, 0025, 0026, 0028, 0032, 0047
|
||||
|
||||
Microscope: 0013 tagged `multiplicative_aggregate` / `temporal_aggregation`; adjacent solved temporal cases 0001, 0015, 0017 use distinct tariff/duration surfaces.
|
||||
|
||||
## 4. Candidate families ranked
|
||||
|
||||
| Rank | Family | Cases | Lift | Wrong-risk | Verdict |
|
||||
|------|--------|-------|------|------------|---------|
|
||||
| 1 | `calendar_grounded_piecewise_daily_hours_total` | 0013 | +1 | low–medium | **Selected (A2q)** |
|
||||
| 2 | `multiplicative_aggregate` wholesale | 0013 | +1? | high (0025, 0047) | **Rejected** |
|
||||
| 3 | Scout singletons (0004/0007) | — | +1 each | high neighbors | **Deferred** |
|
||||
| — | DCS / currency / sealed_elimination | — | — | blocked | **Deferred** |
|
||||
|
||||
## 5. ClusterContract
|
||||
|
||||
```yaml
|
||||
ClusterContract:
|
||||
family_id: calendar_grounded_piecewise_daily_hours_total
|
||||
proposed_organs:
|
||||
- piecewise_daily_hours_total (Gate A2q)
|
||||
- calendar_grounding (civil_month_day_count_table substrate)
|
||||
included_cases:
|
||||
- gsm8k-train-sample-v1-0013
|
||||
explicitly_excluded_cases:
|
||||
- gsm8k-train-sample-v1-0001 # temporal tariff overtime
|
||||
- gsm8k-train-sample-v1-0014 # rate conversion
|
||||
- gsm8k-train-sample-v1-0015 # duration segment total
|
||||
- gsm8k-train-sample-v1-0017 # bundle overflow tariff
|
||||
- gsm8k-train-sample-v1-0025 # MA sealed-wrong
|
||||
- gsm8k-train-sample-v1-0047 # DCS/divisive sealed-wrong
|
||||
- February / odd-day-month halfway confusers
|
||||
positive_anchors:
|
||||
- "N one-hour {content} each day" → N hours/day
|
||||
- "halfway through {NamedMonth}"
|
||||
- "doubled the number of video hours" + "remaining days"
|
||||
- "total … hours … end of the month"
|
||||
negative_anchors:
|
||||
- money/tariff ($, overtime, rent)
|
||||
- vague month ("about a month", "several weeks")
|
||||
- February without leap policy
|
||||
- odd-day-month halfway (31-day months)
|
||||
- asks daily rate / first period only
|
||||
- MA/DCS packing surfaces
|
||||
actor_binding_rule: leading subject before comma must appear in upload opener
|
||||
target_binding_rule: question must ask total hours at end of month (what/how + total + hours + month)
|
||||
unit_binding_rule: answer unit is hours; daily rate derived from one-hour event count
|
||||
quantity_obligations:
|
||||
- text-grounded daily event count (10)
|
||||
- table-grounded month span (June→30, provenance calendar_table)
|
||||
- comparative cues doubled, halfway, remaining
|
||||
allowed_external_grounding:
|
||||
- civil_month_day_count_table (fixed non-leap civil year)
|
||||
grounding_provenance: calendar_table:{month_name}
|
||||
blocked_sibling_families:
|
||||
- temporal_tariff
|
||||
- duration_segment_total
|
||||
- multiplicative_aggregate wholesale
|
||||
- rate_conversion
|
||||
composition_validation_pins: none changed (cv-0013 remains permanent refuse)
|
||||
required_confusers:
|
||||
- February, January-halfway, vague month, no month, asks rate, first-period-only
|
||||
- 0001/0017 tariff, 0015 duration, 0014 rate, 0025 MA
|
||||
serving_admission_rule: resolve_promotable_piecewise_daily_hours_total() after self-verification
|
||||
implementation_allowed: true
|
||||
reason: Typed chain + explicit table provenance + confuser matrix clears wrong-risk neighbors; only 0013 in train_sample — sibling proven via synthetic September case.
|
||||
```
|
||||
|
||||
## 6. Calendar grounding policy decision
|
||||
|
||||
**Ratified for serving** with narrow constraints:
|
||||
|
||||
- Fixed `CIVIL_MONTH_DAY_COUNT` table (12 months, non-leap).
|
||||
- February **serving-blocked** (leap ambiguity).
|
||||
- Halfway split only on **even** day-count months (30-day civil months).
|
||||
- Month operand provenance `calendar_table:{month}`; month **name** must appear in text.
|
||||
- Not reconstruction-over-storage: table is a deterministic construction boundary, not vault memory.
|
||||
- Not hidden normalization: no hot-path repair; table consulted only at organ admission.
|
||||
- Not answer memorization: arithmetic recomputed from text rate + table span + comparative cues.
|
||||
|
||||
## 7. Blocked families and why
|
||||
|
||||
| Family | Blocked because |
|
||||
|--------|-----------------|
|
||||
| `multiplicative_aggregate` wholesale | 0025/0047 sealed-wrong; 0013 is piecewise not MA |
|
||||
| DCS / currency / sealed_elimination | unchanged blocked set |
|
||||
| February calendar | leap-year policy not ratified |
|
||||
| 31-day halfway split | odd half-days ambiguous under table policy |
|
||||
| Scout singletons 0004/0007 | 0026/0047 sealed-wrong neighbors |
|
||||
|
||||
## 8. Research gate decision
|
||||
|
||||
**Proceed** with ClusterContract-backed Gate A2q. Calendar table grounding is doctrinally safe under explicit provenance and narrow admission. State A (25+) not reachable with single train_sample member; fallback 24/26/0 accepted.
|
||||
|
||||
## 9. Selected paradigm
|
||||
|
||||
**ClusterContract organ bundle:**
|
||||
|
||||
1. `generate/derivation/calendar_grounding.py` — civil month table + provenance
|
||||
2. `generate/derivation/piecewise_daily_hours_total.py` — Gate A2q piecewise organ
|
||||
|
||||
Wired in `generate/math_candidate_graph.py` after Gate A2p.
|
||||
|
||||
## 10. Typed chains for target cases
|
||||
|
||||
**0013 (gold 450):**
|
||||
|
||||
- daily_hours = 10 (10 one-hour videos/day)
|
||||
- month_days = 30 (calendar_table:june)
|
||||
- half_days = 15 (halfway through June)
|
||||
- period1 = 10 × 15 = 150
|
||||
- period2 = (10 × 2) × 15 = 300 (doubled remaining days)
|
||||
- total = 450
|
||||
|
||||
## 11. Implemented organ(s)
|
||||
|
||||
| Module | Promotion entry |
|
||||
|--------|-----------------|
|
||||
| `generate/derivation/calendar_grounding.py` | `resolve_month_grounding()` |
|
||||
| `generate/derivation/piecewise_daily_hours_total.py` | `resolve_promotable_piecewise_daily_hours_total()` |
|
||||
|
||||
## 12. Before/after score
|
||||
|
||||
| Metric | Before | After | Δ |
|
||||
|--------|-------:|------:|--:|
|
||||
| correct | 23 | **24** | +1 |
|
||||
| refused | 27 | **26** | −1 |
|
||||
| wrong | 0 | **0** | 0 |
|
||||
|
||||
**Fallback achieved:** 24/26/0 + ratified ClusterContract. State A (25+) not reached — no second train_sample member in contract.
|
||||
|
||||
## 13. Newly solved IDs
|
||||
|
||||
**0013**
|
||||
|
||||
## 14. Preserved solved IDs
|
||||
|
||||
0001, 0002, 0003, 0005, 0006, 0008, 0009, 0010, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, 0038, 0042, 0045, 0046
|
||||
|
||||
## 15. Wrong=0 proof
|
||||
|
||||
- Full train_sample replay: 24 / 26 / 0
|
||||
- `tests/test_math_candidate_graph_sprint11_cluster_contract_lift.py`
|
||||
- Holdout_dev: wrong=0
|
||||
- Smoke suite: 108 passed
|
||||
- Regression lane: 307+ passed
|
||||
|
||||
## 16. Confuser matrix
|
||||
|
||||
| Confuser | Result |
|
||||
|----------|--------|
|
||||
| February halfway | refuse |
|
||||
| January odd-month halfway | refuse |
|
||||
| No named month | refuse |
|
||||
| "about a month" | refuse |
|
||||
| Asks daily rate | refuse |
|
||||
| First period only | refuse |
|
||||
| 0001 overtime tariff | refuse |
|
||||
| 0017 bundle tariff | refuse |
|
||||
| 0015 duration segment | refuse |
|
||||
| 0014 rate conversion | refuse |
|
||||
| 0025 MA shape | refuse |
|
||||
|
||||
## 17. Sibling generality proof
|
||||
|
||||
Synthetic September sibling: 6 one-hour episodes/day, halfway through September, doubled remaining → 6×15 + 12×15 = **270** (tested).
|
||||
|
||||
## 18. Sealed-wrong negative evidence used
|
||||
|
||||
- Rejected wholesale MA: 0025, 0047 confuser shapes refuse A2q
|
||||
- Temporal tariff 0001/0017: $/overtime/rent blockers
|
||||
- Duration 0015: no piecewise month surface
|
||||
|
||||
## 19. Composition-validation pin changes
|
||||
|
||||
**None.**
|
||||
|
||||
## 20. report.json untouched
|
||||
|
||||
Yes.
|
||||
|
||||
## 21. sealed artifacts untouched
|
||||
|
||||
Yes.
|
||||
|
||||
## 22. no case-id logic
|
||||
|
||||
Yes.
|
||||
|
||||
## 23. no hardcoded answer
|
||||
|
||||
Yes.
|
||||
|
||||
## 24. no direct-answer shortcut
|
||||
|
||||
Yes.
|
||||
|
||||
## 25. whether this enables bigger bundled lifts
|
||||
|
||||
**Yes, substrate-first.** The ClusterContract + `calendar_grounding` module is reusable for future organs that share:
|
||||
|
||||
- named-month whole-span grounding
|
||||
- explicit `calendar_table` provenance
|
||||
- shared confuser matrix (February, odd-month halfway, vague spans)
|
||||
|
||||
Next bundled lift could add a second organ under the same contract (e.g. uniform daily rate without doubling) once practice lane shows sibling coverage.
|
||||
|
||||
## 26. Sprint 12 recommendation
|
||||
|
||||
1. **Primary:** Scout-decomposed singleton with lowest sealed-wrong adjacency — re-audit **0004** only if nested-fraction confuser matrix against 0026 is ratified; otherwise **0007** only after 0047 divisive confuser matrix.
|
||||
2. **ClusterContract extension:** Add second piecewise organ (uniform month total without rate change) under same calendar contract if practice/holdout cases appear — do not broaden to generic calendar parser.
|
||||
3. **Blocked:** DCS 0032/0047, currency 0019/0028, sealed_elimination 0011/0026, wholesale MA.
|
||||
4. **Experience Flywheel:** Continue compaction; no serving promotion from flywheel until `lift_refused_to_correct > 0` with sealed-wrong clearance.
|
||||
5. **Target:** 25+/25-/0 requires either a second train_sample case under an existing contract or a verified singleton mini-family with sibling tests.
|
||||
100
generate/derivation/calendar_grounding.py
Normal file
100
generate/derivation/calendar_grounding.py
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
"""Calendar grounding policy for cluster-contract piecewise daily-hour organs.
|
||||
|
||||
Sprint 11 ClusterContract: ``civil_month_day_count_table`` supplies month
|
||||
day-count **only** when a named month appears in problem text. Table values are
|
||||
deterministic, tiny, and provenance-labeled ``calendar_table`` — not problem-text
|
||||
operands and not serving-memory mutation.
|
||||
|
||||
Allowed:
|
||||
- explicit civil month name in text (January–December)
|
||||
- table lookup from fixed non-leap civil-year counts
|
||||
- month day-count used solely as whole-month span for end-of-month totals
|
||||
|
||||
Refused:
|
||||
- February (leap ambiguity without explicit year/rule)
|
||||
- vague spans (``about a month``, ``several weeks``)
|
||||
- date-range / inclusive-exclusive date math
|
||||
- multiple named months in one problem
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Final
|
||||
|
||||
from generate.math_roundtrip import _token_in, _tokens
|
||||
|
||||
PROVENANCE_CALENDAR_TABLE: Final[str] = "calendar_table"
|
||||
|
||||
# Fixed non-leap civil-year month lengths (deterministic, no runtime mutation).
|
||||
CIVIL_MONTH_DAY_COUNT: Final[dict[str, int]] = {
|
||||
"january": 31,
|
||||
"february": 28,
|
||||
"march": 31,
|
||||
"april": 30,
|
||||
"may": 31,
|
||||
"june": 30,
|
||||
"july": 31,
|
||||
"august": 31,
|
||||
"september": 30,
|
||||
"october": 31,
|
||||
"november": 30,
|
||||
"december": 31,
|
||||
}
|
||||
|
||||
# February is table-known but serving-blocked without leap-year policy.
|
||||
BLOCKED_SERVING_MONTHS: Final[frozenset[str]] = frozenset({"february"})
|
||||
|
||||
_MONTH_NAME_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"\b(" + "|".join(sorted(CIVIL_MONTH_DAY_COUNT, key=len, reverse=True)) + r")\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class MonthGrounding:
|
||||
"""A named month grounded via the civil table."""
|
||||
|
||||
month_name: str
|
||||
day_count: int
|
||||
provenance: str = PROVENANCE_CALENDAR_TABLE
|
||||
|
||||
@property
|
||||
def source_token(self) -> str:
|
||||
return f"{self.provenance}:{self.month_name}"
|
||||
|
||||
|
||||
def extract_named_months(problem_text: str) -> tuple[str, ...]:
|
||||
"""Return lowercased month names in narrative order."""
|
||||
return tuple(match.group(1).lower() for match in _MONTH_NAME_RE.finditer(problem_text))
|
||||
|
||||
|
||||
def resolve_month_grounding(problem_text: str) -> MonthGrounding | None:
|
||||
"""Resolve exactly one serving-eligible named month, or refuse."""
|
||||
months = extract_named_months(problem_text)
|
||||
if len(months) != 1:
|
||||
return None
|
||||
month = months[0]
|
||||
if month in BLOCKED_SERVING_MONTHS:
|
||||
return None
|
||||
day_count = CIVIL_MONTH_DAY_COUNT.get(month)
|
||||
if day_count is None:
|
||||
return None
|
||||
tokens = _tokens(problem_text)
|
||||
if not _token_in(month, tokens):
|
||||
return None
|
||||
return MonthGrounding(month_name=month, day_count=day_count)
|
||||
|
||||
|
||||
def allows_halfway_split(day_count: int) -> bool:
|
||||
"""Halfway-through-month splits require an even day-count month."""
|
||||
return day_count > 0 and day_count % 2 == 0
|
||||
|
||||
|
||||
def calendar_operand_grounds(source_token: str, tokens: frozenset[str]) -> bool:
|
||||
"""True when a calendar_table operand's month name appears in the text."""
|
||||
if not source_token.startswith(f"{PROVENANCE_CALENDAR_TABLE}:"):
|
||||
return False
|
||||
month = source_token.split(":", 1)[1]
|
||||
return _token_in(month, tokens)
|
||||
275
generate/derivation/piecewise_daily_hours_total.py
Normal file
275
generate/derivation/piecewise_daily_hours_total.py
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
"""Gate A2q — calendar-grounded piecewise daily hours total (ClusterContract organ).
|
||||
|
||||
Sprint 11: train_sample **0013** — daily event-hours rate, ``halfway through {month}``
|
||||
first period, ``doubled`` rate on ``remaining days``, question asks end-of-month
|
||||
total hours. Month day-count is table-grounded (:mod:`calendar_grounding`) with
|
||||
explicit ``calendar_table`` provenance.
|
||||
|
||||
Chain (0013-class):
|
||||
|
||||
half_days = month_days / 2 # licensed by ``halfway``
|
||||
period1 = daily_hours × half_days
|
||||
period2 = (daily_hours × 2) × half_days # ``doubled`` comparative
|
||||
total = period1 + period2
|
||||
|
||||
Narrow organ — not generic calendar/temporal parser, not broad
|
||||
``multiplicative_aggregate``. Promotion requires ClusterContract positive anchors,
|
||||
explicit actor/target/unit binding, and hazard refusal.
|
||||
|
||||
Deterministic; sealed module (no ``chat/`` import).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from typing import Final
|
||||
|
||||
from generate.derivation.calendar_grounding import (
|
||||
MonthGrounding,
|
||||
allows_halfway_split,
|
||||
calendar_operand_grounds,
|
||||
resolve_month_grounding,
|
||||
)
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, SelfVerification
|
||||
from generate.math_roundtrip import _token_in, _tokens, _value_grounds
|
||||
|
||||
_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+\s*/\s*\d+")
|
||||
_DAILY_ONE_HOUR_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"(\d+)\s+one-hour\s+\w+",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_UPLOADER_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"^(\w+)\s*,",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_HALFWAY_MONTH_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"halfway\s+through\s+(\w+)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
"insurance",
|
||||
"overtime",
|
||||
"percent",
|
||||
"percentage",
|
||||
"profit",
|
||||
"rent",
|
||||
"rents",
|
||||
"rented",
|
||||
"salary",
|
||||
"ticket",
|
||||
"tickets",
|
||||
"week",
|
||||
"weeks",
|
||||
}
|
||||
)
|
||||
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{"daily", "left", "per", "rate", "remaining"}
|
||||
)
|
||||
_VAGUE_MONTH_RE: Final[re.Pattern[str]] = re.compile(
|
||||
r"\b(?:about|several)\s+(?:a\s+)?month\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_OBLIGATION_UNITS: Final[frozenset[str]] = frozenset({"hours", "hour"})
|
||||
|
||||
|
||||
def _asks_end_of_month_total_hours(question_clause: str) -> bool:
|
||||
tokens = _tokens(question_clause)
|
||||
asks_quantity = "how" in tokens or "what" in tokens or "what's" in tokens
|
||||
return (
|
||||
asks_quantity
|
||||
and "total" in tokens
|
||||
and "hours" in tokens
|
||||
and "month" in tokens
|
||||
)
|
||||
|
||||
|
||||
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||
if _FRACTION_RE.search(problem_text):
|
||||
return True
|
||||
if _VAGUE_MONTH_RE.search(problem_text):
|
||||
return True
|
||||
text_tokens = _tokens(problem_text)
|
||||
question_tokens = _tokens(question_clause)
|
||||
if "%" in problem_text or "percent" in text_tokens:
|
||||
return True
|
||||
if text_tokens & _TEXT_BLOCKERS:
|
||||
return True
|
||||
if question_tokens & _QUESTION_BLOCKERS:
|
||||
return True
|
||||
if "$" in problem_text:
|
||||
return True
|
||||
if "twice" in text_tokens and "doubled" not in text_tokens:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _body_actor(problem_text: str) -> str | None:
|
||||
match = _UPLOADER_RE.search(problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
return match.group(1).lower()
|
||||
|
||||
|
||||
def _daily_hours_rate(problem_text: str) -> Quantity | None:
|
||||
"""``N one-hour videos`` → ``N`` hours/day (one-hour detail is unit conversion)."""
|
||||
match = _DAILY_ONE_HOUR_RE.search(problem_text)
|
||||
if match is None:
|
||||
return None
|
||||
value = float(match.group(1))
|
||||
return Quantity(value=value, unit="hours", source_token=match.group(1))
|
||||
|
||||
|
||||
def _halfway_month_matches(month: MonthGrounding, problem_text: str) -> bool:
|
||||
match = _HALFWAY_MONTH_RE.search(problem_text)
|
||||
if match is None:
|
||||
return False
|
||||
return match.group(1).lower() == month.month_name
|
||||
|
||||
|
||||
def _has_doubled_remaining_pattern(problem_text: str) -> bool:
|
||||
tokens = _tokens(problem_text)
|
||||
return (
|
||||
"doubled" in tokens
|
||||
and "remaining" in tokens
|
||||
and "days" in tokens
|
||||
)
|
||||
|
||||
|
||||
def _recompute_total(daily_hours: Quantity, month: MonthGrounding) -> float:
|
||||
half_days = month.day_count / 2
|
||||
period1 = daily_hours.value * half_days
|
||||
period2 = daily_hours.value * 2.0 * half_days
|
||||
return period1 + period2
|
||||
|
||||
|
||||
def _piecewise_total(
|
||||
daily_hours: Quantity, month: MonthGrounding
|
||||
) -> tuple[GroundedDerivation, float]:
|
||||
"""Build an audit derivation and the explicit piecewise total."""
|
||||
half_days = month.day_count / 2
|
||||
total = _recompute_total(daily_hours, month)
|
||||
period2 = daily_hours.value * 2.0 * half_days
|
||||
|
||||
derivation = GroundedDerivation(
|
||||
start=Quantity(
|
||||
value=daily_hours.value * half_days,
|
||||
unit="hours",
|
||||
source_token=daily_hours.source_token,
|
||||
),
|
||||
steps=(
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(value=period2, unit="hours", source_token="doubled"),
|
||||
cue="doubled",
|
||||
comparative=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
return derivation, total
|
||||
|
||||
|
||||
def build_piecewise_daily_hours_total(problem_text: str) -> tuple[GroundedDerivation, float] | None:
|
||||
"""Construct the ungated piecewise chain, or ``None``."""
|
||||
question_clause = _question_clause(problem_text)
|
||||
if not _asks_end_of_month_total_hours(question_clause):
|
||||
return None
|
||||
if _has_hazard_surface(problem_text, question_clause):
|
||||
return None
|
||||
|
||||
month = resolve_month_grounding(problem_text)
|
||||
if month is None or not allows_halfway_split(month.day_count):
|
||||
return None
|
||||
if not _halfway_month_matches(month, problem_text):
|
||||
return None
|
||||
if not _has_doubled_remaining_pattern(problem_text):
|
||||
return None
|
||||
|
||||
daily_hours = _daily_hours_rate(problem_text)
|
||||
if daily_hours is None:
|
||||
return None
|
||||
if _body_actor(problem_text) is None:
|
||||
return None
|
||||
|
||||
return _piecewise_total(daily_hours, month)
|
||||
|
||||
|
||||
def _obligation_quantities(problem_text: str) -> Counter[str]:
|
||||
return Counter(
|
||||
q.source_token
|
||||
for q in extract_quantities(problem_text)
|
||||
if q.unit in _OBLIGATION_UNITS or q.unit.rstrip("s") == "hour"
|
||||
)
|
||||
|
||||
|
||||
def _operand_grounds(q: Quantity, tokens: frozenset[str]) -> bool:
|
||||
if calendar_operand_grounds(q.source_token, tokens):
|
||||
return True
|
||||
return _value_grounds(q.source_token, tokens)
|
||||
|
||||
|
||||
def _self_verifies_piecewise(
|
||||
derivation: GroundedDerivation, problem_text: str, expected_total: float
|
||||
) -> SelfVerification:
|
||||
tokens = _tokens(problem_text)
|
||||
reasons: list[str] = []
|
||||
|
||||
for q in [derivation.start, *(s.operand for s in derivation.steps if not s.comparative)]:
|
||||
if not _operand_grounds(q, tokens):
|
||||
reasons.append(f"operand {q.source_token!r} not grounded in text")
|
||||
|
||||
for step in derivation.steps:
|
||||
if not _token_in(step.cue, tokens):
|
||||
reasons.append(f"operation cue {step.cue!r} not grounded in text")
|
||||
|
||||
if abs(derivation.answer - expected_total) > 1e-9:
|
||||
reasons.append("arithmetic mismatch on piecewise total")
|
||||
|
||||
obligation = _obligation_quantities(problem_text)
|
||||
used = Counter([derivation.start.source_token])
|
||||
unused = obligation - used
|
||||
if unused:
|
||||
reasons.append(f"incomplete: unused hour quantities {sorted(unused.keys())}")
|
||||
|
||||
month = resolve_month_grounding(problem_text)
|
||||
daily = _daily_hours_rate(problem_text)
|
||||
if month is None:
|
||||
reasons.append("missing calendar month grounding")
|
||||
elif not _halfway_month_matches(month, problem_text):
|
||||
reasons.append("halfway month does not match named month")
|
||||
elif daily is not None and abs(_recompute_total(daily, month) - expected_total) > 1e-9:
|
||||
reasons.append("piecewise formula mismatch")
|
||||
|
||||
if not _has_doubled_remaining_pattern(problem_text):
|
||||
reasons.append("missing doubled remaining-days pattern")
|
||||
|
||||
if daily is None:
|
||||
reasons.append("missing daily one-hour rate anchor")
|
||||
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_piecewise_daily_hours_total(problem_text: str) -> Resolution | None:
|
||||
"""Gate the typed piecewise chain through self-verification."""
|
||||
built = build_piecewise_daily_hours_total(problem_text)
|
||||
if built is None:
|
||||
return None
|
||||
derivation, total = built
|
||||
if not _self_verifies_piecewise(derivation, problem_text, total).verified:
|
||||
return None
|
||||
return Resolution(
|
||||
answer=total,
|
||||
answer_unit="hours",
|
||||
derivation=derivation,
|
||||
)
|
||||
|
||||
|
||||
def resolve_promotable_piecewise_daily_hours_total(
|
||||
problem_text: str,
|
||||
) -> Resolution | None:
|
||||
"""Serving promotion bridge (Gate A2q, ClusterContract calendar_grounded_piecewise)."""
|
||||
return compose_piecewise_daily_hours_total(problem_text)
|
||||
|
|
@ -883,6 +883,21 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2q — calendar-grounded piecewise daily hours (Sprint 11 ClusterContract).
|
||||
from generate.derivation.piecewise_daily_hours_total import (
|
||||
resolve_promotable_piecewise_daily_hours_total,
|
||||
)
|
||||
|
||||
piecewise_hours_resolution = resolve_promotable_piecewise_daily_hours_total(text)
|
||||
if piecewise_hours_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=piecewise_hours_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
|
|||
summary = build_microscope_report(_load_cases())
|
||||
md = render_markdown(summary)
|
||||
assert "partition_chunking" in md
|
||||
assert "correct: 23" in md
|
||||
assert "correct: 24" in md
|
||||
assert "Gate A2a unit_partition" in md
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -159,9 +159,10 @@ class TestTargetCases:
|
|||
assert result.answer == 185.0
|
||||
assert result.refusal_reason is None
|
||||
|
||||
def test_0013_still_refuses_calendar_grounding(self) -> None:
|
||||
def test_0013_solved_by_sprint11_cluster_contract(self) -> None:
|
||||
result = parse_and_solve(CASE_0013)
|
||||
assert result.answer is None
|
||||
assert result.answer == 450.0
|
||||
assert result.refusal_reason is None
|
||||
|
||||
|
||||
class TestSiblingGeneralization:
|
||||
|
|
@ -217,8 +218,8 @@ class TestTrainSampleScore:
|
|||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 23
|
||||
assert counts["refused"] <= 27
|
||||
assert counts["correct"] >= 24
|
||||
assert counts["refused"] <= 26
|
||||
|
||||
def test_newly_solved_ids(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
|
|
|||
|
|
@ -0,0 +1,280 @@
|
|||
"""Gate A2q — Capability Paradigm Sprint 11 cluster-contract calendar lift."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.derivation.calendar_grounding import (
|
||||
CIVIL_MONTH_DAY_COUNT,
|
||||
PROVENANCE_CALENDAR_TABLE,
|
||||
allows_halfway_split,
|
||||
resolve_month_grounding,
|
||||
)
|
||||
from generate.derivation.piecewise_daily_hours_total import (
|
||||
compose_piecewise_daily_hours_total,
|
||||
resolve_promotable_piecewise_daily_hours_total,
|
||||
)
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
|
||||
_CASES_PATH = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "evals"
|
||||
/ "gsm8k_math"
|
||||
/ "train_sample"
|
||||
/ "v1"
|
||||
/ "cases.jsonl"
|
||||
)
|
||||
|
||||
CASE_0013 = (
|
||||
"Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day "
|
||||
"to her channel. She uploaded videos halfway through June, at that pace, "
|
||||
"and then doubled the number of video hours she uploaded on the remaining days. "
|
||||
"What's the total number of video hours she has uploaded at the end of the month?"
|
||||
)
|
||||
|
||||
SIBLING_SEPTEMBER = (
|
||||
"Morgan, a podcaster, uploads 6 one-hour episodes of book reviews each day "
|
||||
"to her feed. She uploaded episodes halfway through September, at that pace, "
|
||||
"and then doubled the number of episode hours she uploaded on the remaining days. "
|
||||
"What's the total number of episode hours she has uploaded at the end of the month?"
|
||||
)
|
||||
|
||||
FEBRUARY_CONFUSER = (
|
||||
"Nina, a creator, uploads 5 one-hour clips each day to her channel. "
|
||||
"She uploaded clips halfway through February, at that pace, "
|
||||
"and then doubled the number of video hours she uploaded on the remaining days. "
|
||||
"What's the total number of video hours she has uploaded at the end of the month?"
|
||||
)
|
||||
|
||||
JANUARY_ODD_MONTH_CONFUSER = (
|
||||
"Riley, a creator, uploads 4 one-hour vlogs each day to her channel. "
|
||||
"She uploaded vlogs halfway through January, at that pace, "
|
||||
"and then doubled the number of video hours she uploaded on the remaining days. "
|
||||
"What's the total number of video hours she has uploaded at the end of the month?"
|
||||
)
|
||||
|
||||
ASKS_DAILY_RATE_CONFUSER = (
|
||||
"Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day "
|
||||
"to her channel. She uploaded videos halfway through June, at that pace, "
|
||||
"and then doubled the number of video hours she uploaded on the remaining days. "
|
||||
"What is her daily upload rate at the end of the month?"
|
||||
)
|
||||
|
||||
FIRST_PERIOD_ONLY_CONFUSER = (
|
||||
"Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day "
|
||||
"to her channel. She uploaded videos halfway through June, at that pace. "
|
||||
"How many video hours did she upload in the first half of the month?"
|
||||
)
|
||||
|
||||
NO_MONTH_CONFUSER = (
|
||||
"Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day "
|
||||
"to her channel. She uploaded videos halfway through the project, at that pace, "
|
||||
"and then doubled the number of video hours she uploaded on the remaining days. "
|
||||
"What's the total number of video hours she has uploaded at the end of the month?"
|
||||
)
|
||||
|
||||
VAGUE_MONTH_CONFUSER = (
|
||||
"Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day. "
|
||||
"She uploaded videos for about a month at that pace. "
|
||||
"What's the total number of video hours she has uploaded at the end of the month?"
|
||||
)
|
||||
|
||||
TEMPORAL_TARIFF_0001_SHAPE = (
|
||||
"A construction worker who works 8 hours a day and makes $20 per hour "
|
||||
"works 5 days a week. If he works more than 8 hours a day he makes time "
|
||||
"and a half for every extra hour. Last week he worked 10 hours a day for 5 days. "
|
||||
"How much money did he make last week?"
|
||||
)
|
||||
|
||||
TEMPORAL_TARIFF_0017_SHAPE = (
|
||||
"Jason has a carriage house that he rents out. He's charging $50.00 per day "
|
||||
"or $500.00 for 14 days. Eric wants to rent the house for 20 days. "
|
||||
"How much will it cost him?"
|
||||
)
|
||||
|
||||
DURATION_SEGMENT_0015_SHAPE = (
|
||||
"A fog bank rolls in from the ocean to cover a city. It takes 10 minutes "
|
||||
"to cover every 3 miles of the city. If the city is 42 miles across from "
|
||||
"the oceanfront to the opposite inland edge, how long will it take for the "
|
||||
"fog bank to cover the whole city?"
|
||||
)
|
||||
|
||||
RATE_CONVERSION_0014_SHAPE = (
|
||||
"Bob can shuck 10 oysters in 5 minutes. How many oysters can he shuck in 2 hours?"
|
||||
)
|
||||
|
||||
MA_0025_SHAPE = (
|
||||
"Lilibeth fills 6 baskets where each basket holds 50 strawberries. "
|
||||
"She, her brother, and 3 of her friends equally picked strawberries. "
|
||||
"How many strawberries did each person pick?"
|
||||
)
|
||||
|
||||
PRESERVED_SOLVED = (
|
||||
"0001",
|
||||
"0002",
|
||||
"0003",
|
||||
"0005",
|
||||
"0006",
|
||||
"0008",
|
||||
"0009",
|
||||
"0010",
|
||||
"0014",
|
||||
"0015",
|
||||
"0017",
|
||||
"0018",
|
||||
"0021",
|
||||
"0024",
|
||||
"0025",
|
||||
"0029",
|
||||
"0030",
|
||||
"0035",
|
||||
"0037",
|
||||
"0038",
|
||||
"0042",
|
||||
"0045",
|
||||
"0046",
|
||||
)
|
||||
|
||||
|
||||
def _load_case(case_suffix: str) -> str:
|
||||
for line in _CASES_PATH.read_text().splitlines():
|
||||
row = json.loads(line)
|
||||
if row["case_id"].endswith(case_suffix):
|
||||
return row["question"]
|
||||
raise KeyError(case_suffix)
|
||||
|
||||
|
||||
class TestClusterContractPolicy:
|
||||
def test_civil_month_table_is_deterministic(self) -> None:
|
||||
assert CIVIL_MONTH_DAY_COUNT["june"] == 30
|
||||
assert CIVIL_MONTH_DAY_COUNT["february"] == 28
|
||||
assert len(CIVIL_MONTH_DAY_COUNT) == 12
|
||||
|
||||
def test_june_grounding_provenance(self) -> None:
|
||||
grounding = resolve_month_grounding(CASE_0013)
|
||||
assert grounding is not None
|
||||
assert grounding.month_name == "june"
|
||||
assert grounding.day_count == 30
|
||||
assert grounding.provenance == PROVENANCE_CALENDAR_TABLE
|
||||
|
||||
def test_february_serving_blocked(self) -> None:
|
||||
assert resolve_month_grounding(FEBRUARY_CONFUSER) is None
|
||||
|
||||
def test_halfway_split_requires_even_month(self) -> None:
|
||||
assert allows_halfway_split(30) is True
|
||||
assert allows_halfway_split(31) is False
|
||||
|
||||
|
||||
class TestTargetCase:
|
||||
def test_0013_solves(self) -> None:
|
||||
result = parse_and_solve(_load_case("0013"))
|
||||
assert result.answer == 450.0
|
||||
assert result.refusal_reason is None
|
||||
|
||||
|
||||
class TestSiblingGeneralization:
|
||||
def test_september_sibling(self) -> None:
|
||||
resolution = compose_piecewise_daily_hours_total(SIBLING_SEPTEMBER)
|
||||
assert resolution is not None
|
||||
# 6 hrs/day × 15 days + 12 hrs/day × 15 days = 90 + 180 = 270
|
||||
assert resolution.answer == 270.0
|
||||
|
||||
|
||||
class TestCalendarConfuserRefusals:
|
||||
def test_february_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(FEBRUARY_CONFUSER) is None
|
||||
|
||||
def test_january_odd_month_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(JANUARY_ODD_MONTH_CONFUSER) is None
|
||||
|
||||
def test_no_named_month_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(NO_MONTH_CONFUSER) is None
|
||||
|
||||
def test_vague_month_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(VAGUE_MONTH_CONFUSER) is None
|
||||
|
||||
def test_asks_daily_rate_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(ASKS_DAILY_RATE_CONFUSER) is None
|
||||
|
||||
def test_first_period_only_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(FIRST_PERIOD_ONLY_CONFUSER) is None
|
||||
|
||||
|
||||
class TestNeighborRefusals:
|
||||
def test_temporal_tariff_0001_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(TEMPORAL_TARIFF_0001_SHAPE) is None
|
||||
|
||||
def test_temporal_tariff_0017_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(TEMPORAL_TARIFF_0017_SHAPE) is None
|
||||
|
||||
def test_duration_segment_0015_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(DURATION_SEGMENT_0015_SHAPE) is None
|
||||
|
||||
def test_rate_conversion_0014_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(RATE_CONVERSION_0014_SHAPE) is None
|
||||
|
||||
def test_ma_0025_shape_refuses(self) -> None:
|
||||
assert compose_piecewise_daily_hours_total(MA_0025_SHAPE) is None
|
||||
|
||||
|
||||
class TestPromotionBridge:
|
||||
def test_resolve_promotable_piecewise(self) -> None:
|
||||
assert resolve_promotable_piecewise_daily_hours_total(CASE_0013) is not None
|
||||
|
||||
|
||||
class TestTrainSampleScore:
|
||||
def test_wrong_zero_and_fallback_state(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 24
|
||||
assert counts["refused"] <= 26
|
||||
|
||||
def test_newly_solved_0013(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
correct = {
|
||||
row["case_id"].split("-")[-1]
|
||||
for row in report["per_case"]
|
||||
if row["verdict"] == "correct"
|
||||
}
|
||||
assert "0013" in correct
|
||||
|
||||
def test_preserved_solved_ids(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
correct = {
|
||||
row["case_id"].split("-")[-1]
|
||||
for row in report["per_case"]
|
||||
if row["verdict"] == "correct"
|
||||
}
|
||||
for case_id in PRESERVED_SOLVED:
|
||||
assert case_id in correct
|
||||
|
||||
|
||||
class TestHoldoutInert:
|
||||
def test_holdout_dev_wrong_zero(self) -> None:
|
||||
from evals.gsm8k_math.holdout_dev.v1.runner import build_report
|
||||
|
||||
report = build_report()
|
||||
assert report["counts"]["wrong"] == 0
|
||||
|
||||
|
||||
class TestTemporalRegressions:
|
||||
@pytest.mark.parametrize("case_id,expected", [("0001", 990.0), ("0017", 800.0)])
|
||||
def test_temporal_tariff_preserved(self, case_id: str, expected: float) -> None:
|
||||
result = parse_and_solve(_load_case(case_id))
|
||||
assert result.answer == expected
|
||||
assert result.refusal_reason is None
|
||||
|
||||
def test_duration_segment_0015_preserved(self) -> None:
|
||||
result = parse_and_solve(_load_case("0015"))
|
||||
assert result.answer == 38.0
|
||||
assert result.refusal_reason is None
|
||||
Loading…
Reference in a new issue