feat(gsm8k): xhigh capability sprint 13 robust lift (#827)
* test(gsm8k): pin xhigh sprint13 confusers and completeness Add the Sprint 13 contract bundle with target-chain pins, sealed-wrong neighbors, fraction-word refusals, and non-vacuous completeness cases for extra mile and extra-actor obligations. * feat(gsm8k): add xhigh sprint13 verified organs Gate A2t bounded_rate_projection admits 0016 and 0034 with honest affine/percent derivations. Gate A2u closed_reference_affine_aggregate admits 0027 and 0039 with statement-scoped numeric obligations and repaired comparative provenance. Serving lift: 26/24/0 -> 30/20/0 with wrong=0 preserved. * docs(gsm8k): add xhigh sprint13 lookback Record XHIGH draft recovery, unsafe evidence findings, repairs, Path A salvage decision, validation outputs, and Sprint 14 recommendation.
This commit is contained in:
Shay
GitHub
parent
69febc6c0e
commit
57a62043d1
7 changed files with 1247 additions and 2 deletions
|
|
@ -0,0 +1,387 @@
|
|||
# GSM8K XHIGH Capability Sprint 13 — Lookback (2026-06-18)
|
||||
|
||||
> Status: research gate complete; implementation and final validation pending.
|
||||
|
||||
## 1. Starting baseline
|
||||
|
||||
The canonical current API is `evals.gsm8k_math.train_sample.v1.runner.build_report`.
|
||||
The brief's `train_sample.v1.report` import is stale and does not exist on `main`.
|
||||
Using the canonical runner on the committed 50-case sample:
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| correct | **26** |
|
||||
| refused | **24** |
|
||||
| wrong | **0** |
|
||||
|
||||
Correct IDs: 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008, 0009,
|
||||
0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030,
|
||||
0035, 0037, 0038, 0042, 0045, 0046. Wrong IDs: none.
|
||||
|
||||
## 2. Intrinsic space and architectural directions
|
||||
|
||||
The refused frontier is not one arithmetic space. It is a disjoint union of
|
||||
small relational manifolds. Surface categories such as
|
||||
`discrete_count_statement` and `descriptive_setup_no_quantity` distort this
|
||||
geometry by grouping different typed chains under one recognizer label. The
|
||||
serving invariant is therefore preserved by admitting only a complete typed
|
||||
relation field: named actors/objects, licensed operators, a bound question
|
||||
target, source-quantity obligations, and a reconstructible final value.
|
||||
|
||||
Directions mapped before implementation:
|
||||
|
||||
1. **Two contract-backed mini-families (Path A candidate).** Pair 0016/0034 as
|
||||
bounded rate projections with explicit numerator, denominator, and target
|
||||
dimensions. Pair 0027/0039 as closed explicit-reference affine aggregates,
|
||||
implemented as two surface-specific builders rather than a generic relation
|
||||
graph. This can yield +4 without a generic equation, fraction, DCS, temporal,
|
||||
or multiplicative parser.
|
||||
2. **Four unrelated strict singletons.** 0012, 0016, 0034, and 0049 each have a
|
||||
clear arithmetic graph, but this repeats binding logic and offers less
|
||||
reusable structure. It is a score bundle, not a coherent capability bundle.
|
||||
3. **One mini-family plus defensive substrate (Path B/C).** Ship only the
|
||||
mini-family that clears confusers and holdout, plus a tiny quantity-obligation
|
||||
or actor-binding helper only if it is adopted without widening behavior.
|
||||
4. **No serving lift (Path D).** If either selected family creates a train or
|
||||
holdout wrong answer, or cannot prove complete binding, preserve 26/24/0 and
|
||||
land only this diagnostic evidence.
|
||||
|
||||
## 3. Scout and Experience Flywheel summary
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
uv run python scripts/gsm8k_sealed_attempt_scout.py --limit 50
|
||||
uv run python scripts/gsm8k_experience_flywheel.py --limit 50
|
||||
uv run python scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-xhigh-sprint13.json
|
||||
```
|
||||
|
||||
| Regime | correct | refused | wrong |
|
||||
|---|---:|---:|---:|
|
||||
| serving | 26 | 24 | 0 |
|
||||
| sealed scout | 3 | 39 | 8 |
|
||||
|
||||
Scout deltas: 3 already served, 6 elimination-refused-to-wrong, 18 joint
|
||||
refusals, 0 refused-to-correct lifts, 23 conservative wins. The flywheel has no
|
||||
promotable family. Its broad family summaries remain diagnostic only.
|
||||
|
||||
| family | case IDs | refused-to-correct | sealed-wrong | blocked | top missing primitive | action |
|
||||
|---|---|---:|---:|---:|---|---|
|
||||
| already_served | 0003, 0021, 0037 | 0 | 0 | 0 | — | preserve |
|
||||
| conservative_boundary | 23 served cases | 0 | 2 | 2 | — | preserve serving gates |
|
||||
| diagnostic_hold:currency_amount | 0019, 0028, 0043 | 0 | 2 | 2 | diagnostic_hold | block wholesale currency |
|
||||
| diagnostic_hold:descriptive_setup_no_quantity | 0012, 0023, 0027 | 0 | 0 | 0 | diagnostic_hold | decompose by typed relation |
|
||||
| joint_sealed_no_resolution | 0048, 0050 | 0 | 0 | 0 | — | diagnostic hold |
|
||||
| relation_hypothesis:discrete_count_statement | 0016, 0020, 0022, 0031, 0032, 0033, 0034, 0036, 0039, 0040, 0041, 0044, 0047, 0049 | 0 | 2 | 2 | relation_hypothesis | reject DCS; decompose exact chains |
|
||||
| sealed_elimination | 0011, 0026 | 0 | 2 | 2 | — | block |
|
||||
|
||||
Sealed-wrong IDs are 0011, 0018, 0019, 0025, 0026, 0028, 0032, and
|
||||
0047. Only 0011, 0019, 0026, 0028, 0032, and 0047 are currently refused;
|
||||
0018 and 0025 are conservative serving wins and regression anchors.
|
||||
|
||||
## 4. Full refused-case inventory
|
||||
|
||||
| ID | Gold | Question text | Typed arithmetic graph | First missing / current refusal | Nearest evidence and candidate |
|
||||
|---|---:|---|---|---|---|
|
||||
| 0011 | 50 | Alexa sells lemonade for $2/cup, spent $20, and asks cups needed for $80 profit. | `(80 + 20) / 2` | graph solve; sealed elimination | 0019/0028 currency wrongs; blocked profit-cost recovery |
|
||||
| 0012 | 7 | Dennis has 10 rocks; fish ate half; two were recovered; asks rocks left. | `10 × (1-1/2) + 2` | descriptive no-quantity injection | clean fraction-loss/recovery singleton; neighbor 0005 is a different fraction-decrease organ |
|
||||
| 0016 | 2 | Rudolph travels 2 more than 5 miles and sees 3 less than 17 signs; asks signs per mile. | `(17-3)/(5+2)` | relation hypothesis | clean dual-offset ratio singleton; changed-target hazard |
|
||||
| 0019 | 660 | Three $400 vet visits, $100 insurance after first, 80% coverage on later visits; asks total paid. | `400 + 100 + 2×400×0.2` | currency injection | sealed wrong 120000; currency family blocked |
|
||||
| 0020 | 130 | Two puppies, two kittens, three parakeets; comparative prices from a $10 parakeet; asks all-pet cost. | `3×10 + 2×20 + 2×30` | relation hypothesis | multi-species price graph; currency and ambiguous “three times more” hazards |
|
||||
| 0022 | 4800 | $20/kg, past four-month catch 80 kg, today twice past total; asks total earnings including today. | `(80 + 2×80)×20` | relation hypothesis | bounded historical aggregate; currency overlap |
|
||||
| 0023 | 150 | Nicole has 400 cards, Cindy twice that, Rex half their combined total, then divides among himself and three siblings. | `(400 + 2×400)×1/2/(3+1)` | descriptive no-quantity injection | **blocked**: Cindy's multiplier has no explicit reference and is adjacent to permanent cv-0014/0015 guards; 0027 does not retroactively license it |
|
||||
| 0026 | 6 | Two people each save $40, bill is 3/4, equal $1.5 scoops, $1 change each; asks scoops each. | `(((2×40)×1/4)-2)/2/1.5` | statement parse | sealed wrong 180; sealed-elimination/currency blocked |
|
||||
| 0027 | 3840 | Instagram 240 + Facebook 500; Twitter half combined; TikTok 3× Twitter; YouTube TikTok+510; asks all followers. | `S=240+500; T=S/2; K=3T; Y=K+510; total=240+500+T+K+Y` | descriptive no-quantity injection | positive anchor for selected combined-half family; 0023 sibling |
|
||||
| 0028 | 200 | $100,000 opening cost, daily cost 1%, 150 tickets/day at $10; asks days to recover investment. | `100000 / (150×10 - 100000×0.01)` | currency injection | sealed wrong 0; currency/profit recovery blocked |
|
||||
| 0031 | 92 | Jeremie plus three friends buy $18 tickets and $5 snacks each; asks total. | `(3+1)×(18+5)` | relation hypothesis | bounded group-cost aggregate but currency family adjacency |
|
||||
| 0032 | 34 | Ten pictures, 2 hours drawing each, coloring takes 30% less; asks total hours. | `10×(2 + 2×0.7)` | relation hypothesis | sealed wrong 20; DCS/percent-time blocked |
|
||||
| 0033 | 60 | Rachel is 12; grandfather 7×; mother half grandfather; father 5 older; asks father age when Rachel is 25. | `father_now=(12×7)/2+5; answer=father_now+(25-12)` | relation hypothesis | selected named-actor affine chain; 0039 sibling; 0032 negative |
|
||||
| 0034 | 112 | 40 yards in 5 seconds, speed improves 40%; asks yards in 10 seconds. | `(40/5)×1.4×10` | relation hypothesis | clean percent-rate projection singleton; 0032 percent-time negative |
|
||||
| 0036 | 22 | Study 2 hours Wednesday, 3× Thursday, half Thursday Friday, weekend equals weekday total; asks five-day total. | `W=2; T=3W; F=T/2; total=2×(W+T+F)` | relation hypothesis | bounded schedule aggregate; temporal ambiguity on “during weekend” |
|
||||
| 0039 | 20 | Orlando gains 5; Jose gains two more than twice Orlando; Fernando gains 3 less than half Jose; asks all three total. | `J=2×5+2; F=J/2-3; total=5+J+F` | relation hypothesis | selected named-actor affine chain; 0033 sibling; 0032 negative |
|
||||
| 0040 | 72 | Counts of horses, dogs, cats, turtles, goat; asks total legs. | `(2+5+7+3+1)×4` | relation hypothesis | requires animal-leg external grounding; implementation blocked |
|
||||
| 0041 | 6 | Two 16-piece brownie pans; guests eat one plus 75% of second; all but four use two scoops; eight scoops/tub. | `guests=16+0.75×16; tubs=(guests-4)×2/8` | relation hypothesis | multi-stage implicit guest binding; fraction/container hazards |
|
||||
| 0043 | 11 | Sandra has $10+$4+twice $4; buys 14×$0.5 and 20×$0.2; asks money left. | `10+4+2×4-14×0.5-20×0.2` | currency injection | no sealed wrong itself, but 0019/0028 family blockers |
|
||||
| 0044 | 1300 | $1000 earns 10% simple interest for three years; asks final money. | `1000×(1+0.1×3)` | relation hypothesis | explicit simple-interest singleton; currency adjacency |
|
||||
| 0047 | 45 | 12 five-ounce macaroons split over four bags; one bag eaten; asks remaining weight. | `(12-12/4)×5` | relation hypothesis | sealed wrong 240; DCS/divisive blocked |
|
||||
| 0048 | 4 | Start 20 cards; +6 weekly; -2 every two weeks; asks weeks to reach 40. | `20 + 6w - 2×(w/2)=40`, with even-period obligation | statement parse | bounded periodic recurrence; generic temporal parser prohibited |
|
||||
| 0049 | 18 | Two walking routes with twice/third segment relations; asks how much longer route two is. | `R1=6+12+(6+12)/3; R2=14+28; R2-R1` | relation hypothesis | clean route-duration comparator; nearest duration organs are narrower |
|
||||
| 0050 | 280 (artifact inconsistent) | Gig every other day for two weeks; two five-minute songs and one twice as long; asks minutes. | Source rationale computes `7×2=14` gigs and `14×20=140`, while stored gold is 280. | statement parse | **implementation forbidden**: dataset's rationale and gold contradict; do not hardcode the final answer |
|
||||
|
||||
## 5. Candidate ranking and research gate
|
||||
|
||||
| Rank | Candidate / cases | Expected lift | Typed-chain clarity | Sealed-wrong / blocked overlap | Decision |
|
||||
|---:|---|---:|---|---|---|
|
||||
| 1 | `bounded_rate_projection` / 0016, 0034 | +2 | high; two explicit affine/rate graphs with dimension-bound questions | 0018/0032 nearest rate/percent sealed wrong; 0019/0028 currency and 0047 packing confusers | implement as two narrow modes under one ClusterContract |
|
||||
| 2 | `closed_explicit_reference_affine_aggregate` / 0027, 0039 | +2 | high; every comparative names its source and target aggregate is closed | DCS surface includes 0032/0047 and aggregate 0025; permanent no-reference guards must remain refused | implement as two surface-specific modes under one ClusterContract |
|
||||
| 3 | `fraction_loss_recovery` / 0012 | +1 | high | fraction surface; 0026 is structurally different but dangerous | fallback only |
|
||||
| 4 | `route_duration_difference` / 0049 | +1 | high | temporal organs and unrelated-clause completeness | fallback only |
|
||||
| 5 | `periodic_net_target_horizon` / 0048 | +1 | high only at integral two-week boundaries | 0050 calendar ambiguity and 0028 recovery surface | fallback only |
|
||||
| 6 | `intergenerational_age_projection` / 0033 | +1 | high but timeline-specific | broad DCS family and changed-time target hazards | defer behind selected contracts |
|
||||
| 7 | `bounded_schedule_aggregate` / 0036 | +1 | medium | “during weekend” binding ambiguity | defer |
|
||||
| 8 | `half_combined_pair` / 0023, 0027 | +2? | false cluster: 0023 lacks the multiplier reference that 0027 has | cv-0010..0015 permanently refuse no-reference multiplication | reject |
|
||||
| 9 | currency mini-family / 0031, 0043, 0044 | +3 | mixed | 0019/0028 sealed wrong | reject |
|
||||
| 10 | external taxonomy aggregate / 0040 | +1 | arithmetic clear, grounding absent | cv-0016 permanently refuses world knowledge | reject |
|
||||
| 11 | 0050 | unsafe | rationale/gold contradiction | direct-answer temptation | reject |
|
||||
|
||||
Selected path: **Path A, conditional on tests and holdout**. Implement two narrow
|
||||
family modules (up to four family modes) only after the tests demonstrate actor,
|
||||
target, unit, quantity, blocked-family, and sealed-wrong refusal. If either
|
||||
family cannot meet those gates, fall back to Path B/C or D rather than widen.
|
||||
|
||||
## 6. Negative-evidence audit
|
||||
|
||||
| Proposed family | Accidental admission risk | Positive license | Negative anchors / proof |
|
||||
|---|---|---|---|
|
||||
| bounded rate projection | a generic rate organ admits 0018/0032, or currency/packing rates; reciprocal questions invert the graph | mode 0016 requires one distance offset, one stop-count offset, same trip actor, and stop-signs-per-distance target; mode 0034 requires one distance/time pair, positive speed-improvement percentage, same actor, and projected-distance target | exact 0018, 0019, 0028, 0032, 0047; cross-mode refusal; changed actor/target/unit; reciprocal target; same numbers unrelated; extra relevant quantity |
|
||||
| closed explicit-reference affine aggregate | a generic relation graph admits no-reference multipliers, partial aggregates, 0032, 0047, or 0025 | mode 0027 names every platform reference and closes exactly five nodes; mode 0039 names each prior actor and closes exactly three family members | cv-0010..0016 all remain refused; exact 0023, 0025, 0032, 0033, 0040, 0047; renamed source, omitted node, changed target/unit, extra relevant quantity |
|
||||
| fraction loss recovery | confuses a subgroup/expense fraction with remaining root state | possession start + `ate half` loss + explicit recovery + root remainder question | 0026, currency, camp, multiple fractions, subgroup target |
|
||||
| dual-offset ratio | generic nearby-number arithmetic | named miles and stop signs, one affine offset per unit, per-mile question | 0032/0047, target/unit swaps, extra relevant quantity |
|
||||
|
||||
## 7. ClusterContracts
|
||||
|
||||
### Selected: bounded_rate_projection
|
||||
|
||||
```yaml
|
||||
ClusterContract:
|
||||
family_id: bounded_rate_projection
|
||||
proposed_organs:
|
||||
- affine_event_per_distance
|
||||
- percent_improved_distance_projection
|
||||
included_cases: [0016, 0034]
|
||||
explicitly_excluded_cases: [0018, 0019, 0028, 0032, 0047]
|
||||
positive_anchors:
|
||||
- one explicitly bound numerator quantity and denominator quantity
|
||||
- mode-specific affine offsets or a positive speed-improvement percentage
|
||||
- same named actor and a dimension-exact question target
|
||||
negative_anchors:
|
||||
- currency/insurance/profit/tickets
|
||||
- draw/color/each-picture/less-time
|
||||
- bags/ounces/equal-number/eaten-weight
|
||||
- reciprocal or changed target and additional relevant rates
|
||||
actor_binding_rule: body and question resolve to the same unique named actor; pronouns must agree with the bound surface
|
||||
target_binding_rule: affine-event mode asks events per distance; projection mode asks distance in an explicit target time
|
||||
unit_binding_rule: numerator and denominator dimensions match their exact question order; distance/time units never cross modes
|
||||
quantity_obligations: all four mode quantities are consumed once by the independent reconstruction
|
||||
allowed_external_grounding: none
|
||||
grounding_provenance: text_only
|
||||
blocked_sibling_families: [DCS, currency_amount, percent_time, divisive_packing]
|
||||
sealed_wrong_neighbors: [0018, 0019, 0028, 0032, 0047]
|
||||
composition_validation_pins: no existing pin may flip
|
||||
required_confusers: [cross-mode, changed actor, reciprocal target, unit mismatch, same numbers unrelated, 0018, 0019, 0028, 0032, 0047, unused relevant numeral]
|
||||
serving_admission_rule: unique mode + complete dimensional binding + independent arithmetic reconstruction + GroundedDerivation fold agreement
|
||||
implementation_allowed: true
|
||||
reason: two rate projections share a dimensionally explicit contract while retaining distinct surface grammars
|
||||
```
|
||||
|
||||
### Selected: closed_explicit_reference_affine_aggregate
|
||||
|
||||
```yaml
|
||||
ClusterContract:
|
||||
family_id: closed_explicit_reference_affine_aggregate
|
||||
proposed_organs:
|
||||
- five_node_social_media_total
|
||||
- three_actor_family_weight_total
|
||||
included_cases: [0027, 0039]
|
||||
explicitly_excluded_cases: [0023, 0025, 0032, 0033, 0040, 0047]
|
||||
positive_anchors:
|
||||
- every comparative relation explicitly names its source node
|
||||
- one owner/event and one unit throughout the closed graph
|
||||
- terminal question closes exactly the licensed node set
|
||||
negative_anchors:
|
||||
- missing explicit comparison reference
|
||||
- percent/pictures/coloring/hours and age/future timelines
|
||||
- packs/bags/ounces/equal-number and external world knowledge
|
||||
- omitted, added, cyclic, or unit-mismatched nodes
|
||||
actor_binding_rule: each comparative subject binds to exactly one named prior node; owner/event stays constant
|
||||
target_binding_rule: social mode asks all five platforms; weight mode asks exactly the three named family members
|
||||
unit_binding_rule: followers remain followers; pounds remain pounds; cross-unit chains refuse
|
||||
quantity_obligations: every source numeral, comparative scalar, and named derived node participates in the closed total
|
||||
allowed_external_grounding: none
|
||||
grounding_provenance: text_only
|
||||
blocked_sibling_families: [DCS, multiplicative_aggregate, no_reference_multiplier, world_knowledge, generic_equation]
|
||||
sealed_wrong_neighbors: [0025, 0032, 0047]
|
||||
composition_validation_pins: cv-0004 may flip to solve; cv-0010..cv-0016 must remain refused
|
||||
required_confusers: [0023, 0025, 0032, 0033, 0040, 0047, changed reference, changed target, unit mismatch, same numbers unrelated, unused relevant numeral]
|
||||
serving_admission_rule: unique surface-specific mode + acyclic explicit-reference graph + closed target set + independent reconstruction + GroundedDerivation fold agreement
|
||||
implementation_allowed: true
|
||||
reason: both cases are closed affine DAGs with explicit references; builders remain surface-specific to avoid a generic equation parser
|
||||
```
|
||||
|
||||
### Rejected contracts
|
||||
|
||||
```yaml
|
||||
- family_id: currency_bundle
|
||||
included_cases: [0011, 0019, 0028, 0031, 0043, 0044]
|
||||
sealed_wrong_neighbors: [0011, 0019, 0028]
|
||||
implementation_allowed: false
|
||||
reason: surface currency anchors do not identify one typed chain; three refused cases are sealed wrong
|
||||
- family_id: relation_hypothesis_DCS
|
||||
included_cases: [0016, 0020, 0022, 0031, 0032, 0033, 0034, 0036, 0039, 0040, 0041, 0044, 0047, 0049]
|
||||
sealed_wrong_neighbors: [0032, 0047]
|
||||
implementation_allowed: false
|
||||
reason: recognizer category is a failure label, not an arithmetic family
|
||||
- family_id: generic_fraction_partition
|
||||
included_cases: [0012, 0023, 0026, 0027, 0041]
|
||||
sealed_wrong_neighbors: [0026]
|
||||
implementation_allowed: false
|
||||
reason: fraction surface spans loss, relation, expenditure, and guest-count graphs
|
||||
- family_id: half_combined_pair
|
||||
included_cases: [0023, 0027]
|
||||
sealed_wrong_neighbors: [0026, 0047]
|
||||
implementation_allowed: false
|
||||
reason: 0023 contains an unreferenced multiplier and conflicts with permanent cv-0010..0015 guards
|
||||
- family_id: generic_temporal
|
||||
included_cases: [0033, 0036, 0048, 0049, 0050]
|
||||
implementation_allowed: false
|
||||
reason: timeline, schedule, recurrence, route comparison, and inconsistent calendar artifacts are distinct geometries
|
||||
```
|
||||
|
||||
## 8. Architecture opportunity scan
|
||||
|
||||
Recent organs repeat three defensive ideas: exact question-target binding,
|
||||
source-token quantity obligations, and actor/unit equality checks. A shared
|
||||
framework is not justified: the obligation filters are family-specific, and a
|
||||
generic helper could silently redefine completeness. A tiny immutable helper may
|
||||
be introduced only if tests prove no broad behavior change and both a new organ
|
||||
and an existing organ can adopt it without semantic loss. Otherwise local,
|
||||
explicit checks are safer and more inspectable.
|
||||
|
||||
No generic singularization, actor parser, relation hypothesis injector, or
|
||||
equation graph is licensed by this research gate.
|
||||
|
||||
## 9. XHIGH draft recovery status
|
||||
|
||||
Grok Build recovered the uncommitted GPT-5.5 XHIGH draft from worktree
|
||||
`/Users/kaizenpro/.codex/worktrees/b73c/core-xhigh-capability-sprint13`.
|
||||
|
||||
| Item | XHIGH draft at recovery | After repair |
|
||||
|---|---|---|
|
||||
| train_sample | 30 / 20 / 0 (tentative) | **30 / 20 / 0** |
|
||||
| holdout_dev wrong | 0 | **0** |
|
||||
| staged / committed | none | repaired and validated |
|
||||
| shippable | rejected | **Path A — full salvage** |
|
||||
|
||||
## 10. What was unsafe in the XHIGH draft
|
||||
|
||||
1. **Affine rate divide operand (0016).** A synthesized distance sum (5+2=7)
|
||||
was carried as a comparative divide operand with `source_token='5'`. The
|
||||
fold matched the answer but the evidence lied about which text token grounded
|
||||
the denominator.
|
||||
2. **Social aggregate multiply (0027).** A composite scalar 4.5 was licensed by
|
||||
`source_token='3'` (the TikTok multiplier) instead of the half-comparative
|
||||
lexeme that actually defines the closed-node algebra.
|
||||
3. **Weight aggregate add (0039).** Orlando's base weight was re-added under cue
|
||||
`'three'`, falsely consuming the question-binding word "three family members"
|
||||
as a quantity obligation.
|
||||
4. **Completeness obligation scope.** Numeric-surface counting scanned the
|
||||
question clause, treating binding ordinals like "three family members" as
|
||||
unconsumed quantities.
|
||||
5. **Fraction-word hazard surface.** Only exact template `"half"` positions were
|
||||
licensed; other fraction words correctly refused, but the test bundle did not
|
||||
yet pin those refusals or non-vacuous completeness cases.
|
||||
|
||||
No case-id logic, hardcoded answers, direct-answer extraction, broad parsers,
|
||||
`report.json` rebaseline, or sealed artifact movement were present in the draft.
|
||||
|
||||
## 11. Repairs applied
|
||||
|
||||
| Organ | Repair |
|
||||
|---|---|
|
||||
| `bounded_rate_projection` / affine mode | Rebuilt fold as `(event_base - event_delta) / distance_base` then a comparative correction licensed by the distance-delta cue and token; divide uses text-grounded `distance_base` only. |
|
||||
| `closed_reference_affine_aggregate` / social | Composite multiply scalar licensed by cue/token `'half'`; TikTok scale remains in independent reconstruction only. |
|
||||
| `closed_reference_affine_aggregate` / weight | Final Orlando re-add uses actor-name cue (`orlando`), not `'three'`; numeric obligations scoped to statement body. |
|
||||
| Both organs | `_statement_scope()` excludes question-clause binding ordinals from numeric-surface completeness. |
|
||||
| Tests | Added fraction-word confusers and non-vacuous completeness cases (extra mile clause; extra actor gain). |
|
||||
|
||||
## 12. Selected implementation path
|
||||
|
||||
**Path A — full salvage.** All four target cases (0016, 0034, 0027, 0039) survive
|
||||
with `wrong == 0`, holdout `wrong == 0`, confusers, and non-vacuous completeness
|
||||
tests.
|
||||
|
||||
## 13. Implemented organs
|
||||
|
||||
| Gate | Module | Modes | Resolver |
|
||||
|---|---|---|---|
|
||||
| A2t | `generate/derivation/bounded_rate_projection.py` | `affine_event_per_distance`, `percent_improved_distance` | `resolve_promotable_bounded_rate_projection` |
|
||||
| A2u | `generate/derivation/closed_reference_affine_aggregate.py` | `five_platform_followers`, `three_actor_weight` | `resolve_promotable_closed_reference_affine_aggregate` |
|
||||
|
||||
Both wired in `generate/math_candidate_graph.py` before ADR-0136 rate short-circuits.
|
||||
|
||||
## 14. Score proof
|
||||
|
||||
```text
|
||||
train_sample: {'correct': 30, 'wrong': 0, 'refused': 20}
|
||||
holdout_dev: {'correct': 5, 'wrong': 0, 'refused': 495}
|
||||
wrong_ids: []
|
||||
```
|
||||
|
||||
**Newly solved IDs:** 0016, 0034, 0027, 0039.
|
||||
|
||||
**Preserved solved IDs:** 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008, 0009,
|
||||
0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037,
|
||||
0038, 0042, 0045, 0046.
|
||||
|
||||
## 15. Validation outputs
|
||||
|
||||
```text
|
||||
pytest tests/test_math_candidate_graph_xhigh_sprint13_lift.py -q
|
||||
55 passed
|
||||
|
||||
pytest tests/test_math_candidate_graph_sprint{6..12}_*.py (available lanes) -q
|
||||
225 passed
|
||||
|
||||
pytest tests/test_gsm8k_experience_flywheel*.py
|
||||
tests/test_gsm8k_sealed_attempt_scout.py
|
||||
tests/test_gsm8k_frontier_report.py
|
||||
tests/test_gsm8k_post_gate_a1_frontier_microscope.py
|
||||
tests/test_adr_0195_product_bridge.py
|
||||
tests/test_composition_validation_corpus.py -q
|
||||
111 passed
|
||||
|
||||
uv run python -m core.cli test --suite smoke -q
|
||||
108 passed
|
||||
```
|
||||
|
||||
`git diff --check origin/main...HEAD` — clean at commit time.
|
||||
|
||||
## 16. Confuser matrix (highlights)
|
||||
|
||||
| Family | Confuser | Result |
|
||||
|---|---|---|
|
||||
| rate | sealed-wrong neighbors 0018, 0019, 0028, 0032, 0047 | refuse |
|
||||
| rate | changed actor / reciprocal target / unit mismatch | refuse |
|
||||
| rate | extra mile obligation in affine clause | refuse |
|
||||
| rate | half-hour / quarter-coin / nonlicensed fraction-percent | refuse |
|
||||
| affine | 0023, 0025, 0032, 0033, 0040, 0047 | refuse |
|
||||
| affine | third / quarter / three-quarters / three-times word fractions | refuse |
|
||||
| affine | extra actor gain (Maria +7) in weight family | refuse |
|
||||
| composition_validation | permanent rows | unchanged refuse |
|
||||
| composition_validation | cv-0004 (0027) | solve (15 solve / 7 refuse snapshot) |
|
||||
|
||||
## 17. Artifact integrity
|
||||
|
||||
- `evals/gsm8k_math/train_sample/v1/report.json` — **untouched**
|
||||
- Sealed practice / confusers / holdout case files — **untouched**
|
||||
- No case-id serving logic, hardcoded answers, or direct-answer shortcuts
|
||||
|
||||
## 18. Rejected organs (unchanged from research gate)
|
||||
|
||||
currency bundle, relation_hypothesis DCS, generic fraction partition,
|
||||
half_combined_pair (0023+0027), generic temporal — all remain refused.
|
||||
|
||||
## 19. Sprint 14 recommendation
|
||||
|
||||
The next lowest-risk frontier is **not** a broader fraction or equation parser.
|
||||
Prioritize:
|
||||
|
||||
1. **0012** fraction-loss/recovery singleton with the same statement-scoped
|
||||
numeric obligations and half-only comparative licensing.
|
||||
2. **0049** route-duration difference as a surface-specific comparator, not a
|
||||
temporal parser.
|
||||
3. Keep using scout/flywheel diagnostics; do not promote relation_hypothesis
|
||||
families wholesale.
|
||||
|
||||
Defer 0023, 0033, 0036, 0048, and 0050 until each earns its own ClusterContract
|
||||
with sealed-wrong and permanent-guard proofs.
|
||||
276
generate/derivation/bounded_rate_projection.py
Normal file
276
generate/derivation/bounded_rate_projection.py
Normal file
|
|
@ -0,0 +1,276 @@
|
|||
"""Gate A2t — bounded rate projection ClusterContract.
|
||||
|
||||
Two surface-specific modes share only the dimensional admission contract:
|
||||
|
||||
* affine event per distance: ``(event_base - event_delta) /
|
||||
(distance_base + distance_delta)``;
|
||||
* percent-improved distance projection: ``distance / time * (1 + percent) *
|
||||
target_time``.
|
||||
|
||||
This is not a generic rate, percent, or relation parser. Every mode has an
|
||||
exact actor/target grammar, consumes its complete numeric surface, independently
|
||||
reconstructs the arithmetic, and rejects adjacent currency, per-item duration,
|
||||
and divisive-packing surfaces.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from typing import Final, Literal
|
||||
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.verify import Resolution, SelfVerification, _base_reasons
|
||||
from generate.math_roundtrip import WORD_NUMBERS, _tokens
|
||||
|
||||
|
||||
_NUMBER: Final[str] = r"(?:\d+(?:\.\d+)?|[A-Za-z]+)"
|
||||
_AFFINE_RATE_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^On\s+(?P<actor>[A-Z][A-Za-z'-]+)'s\s+"
|
||||
rf"(?P<trip_kind>[A-Za-z]+)\s+trip\s+across\s+town,\s+"
|
||||
rf"(?P<pronoun>he|she|they)\s+traveled\s+"
|
||||
rf"(?P<distance_delta>{_NUMBER})\s+more\s+than\s+"
|
||||
rf"(?P<distance_base>{_NUMBER})\s+"
|
||||
rf"(?P<distance_unit>miles|kilometers)\s+and\s+encountered\s+"
|
||||
rf"(?P<event_delta>{_NUMBER})\s+less\s+than\s+"
|
||||
rf"(?P<event_base>{_NUMBER})\s+"
|
||||
rf"(?P<event_unit>stop\s+signs|traffic\s+lights)\.\s*"
|
||||
rf"How\s+many\s+(?P<question_event>stop\s+signs|traffic\s+lights)\s+per\s+"
|
||||
rf"(?P<question_distance>mile|kilometer)\s+did\s+(?P<question_actor>[A-Z][A-Za-z'-]+)\s+"
|
||||
rf"encounter\s+on\s+(?P<question_possessive>his|her|their)\s+trip\s+across\s+town\?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_PERCENT_PROJECTION_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^(?P<actor>[A-Z][A-Za-z'-]+)\s+is\s+a\s+varsity\s+player\s+on\s+a\s+"
|
||||
rf"football\s+team\.\s+(?P<pronoun>He|She|They)\s+can\s+run\s+"
|
||||
rf"(?P<distance>{_NUMBER})\s+(?P<distance_unit>yards|meters)\s+within\s+"
|
||||
rf"(?P<base_time>{_NUMBER})\s+seconds\.\s+If\s+(?P=pronoun)\s+can\s+improve\s+"
|
||||
rf"(?P<possessive>his|her|their)\s+speed\s+by\s+(?P<percent>{_NUMBER})\s+percent,\s+"
|
||||
rf"how\s+many\s+(?P<question_unit>yards|meters)\s+will\s+(?P=pronoun)\s+be\s+able\s+"
|
||||
rf"to\s+run\s+within\s+(?P<target_time>{_NUMBER})\s+seconds\?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
"appointment", "bags", "bill", "color", "cost", "dollars", "draw",
|
||||
"each", "eats", "insurance", "macaroons", "ounces", "packs",
|
||||
"pictures", "profit", "subsequent", "tickets", "weight",
|
||||
}
|
||||
)
|
||||
_DISTANCE_SINGULAR: Final[dict[str, str]] = {
|
||||
"miles": "mile",
|
||||
"kilometers": "kilometer",
|
||||
}
|
||||
_EVENT_SINGULAR: Final[dict[str, str]] = {
|
||||
"stop signs": "stop sign",
|
||||
"traffic lights": "traffic light",
|
||||
}
|
||||
_PRONOUN_POSSESSIVE: Final[dict[str, str]] = {
|
||||
"he": "his",
|
||||
"she": "her",
|
||||
"they": "their",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BoundedRateProjectionBuild:
|
||||
mode: Literal["affine_event_per_distance", "percent_improved_distance"]
|
||||
derivation: GroundedDerivation
|
||||
answer: float
|
||||
answer_unit: str
|
||||
numeric_tokens: tuple[str, ...]
|
||||
|
||||
|
||||
def _number(token: str) -> float | None:
|
||||
lowered = token.lower()
|
||||
if lowered in WORD_NUMBERS:
|
||||
return float(WORD_NUMBERS[lowered])
|
||||
try:
|
||||
return float(token)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _all_numeric_tokens(text: str) -> Counter[str]:
|
||||
result: Counter[str] = Counter()
|
||||
for token in re.findall(r"\b\d+(?:\.\d+)?\b|\b[A-Za-z]+\b", text):
|
||||
lowered = token.lower()
|
||||
if re.fullmatch(r"\d+(?:\.\d+)?", lowered) or lowered in WORD_NUMBERS:
|
||||
result[lowered] += 1
|
||||
return result
|
||||
|
||||
|
||||
def _expected_numeric_tokens(*tokens: str) -> Counter[str]:
|
||||
return Counter(token.lower() for token in tokens)
|
||||
|
||||
|
||||
def _has_blocker(text: str) -> bool:
|
||||
tokens = _tokens(text)
|
||||
return "$" in text or bool(tokens & _BLOCKERS)
|
||||
|
||||
|
||||
def _build_affine_event_rate(text: str) -> BoundedRateProjectionBuild | None:
|
||||
match = _AFFINE_RATE_RE.fullmatch(text.strip())
|
||||
if match is None:
|
||||
return None
|
||||
groups = {key: value.lower() for key, value in match.groupdict().items()}
|
||||
if groups["actor"] != groups["question_actor"]:
|
||||
return None
|
||||
if groups["question_event"] != groups["event_unit"]:
|
||||
return None
|
||||
if _DISTANCE_SINGULAR[groups["distance_unit"]] != groups["question_distance"]:
|
||||
return None
|
||||
if _PRONOUN_POSSESSIVE[groups["pronoun"]] != groups["question_possessive"]:
|
||||
return None
|
||||
|
||||
numeric_names = ("distance_delta", "distance_base", "event_delta", "event_base")
|
||||
values = {name: _number(groups[name]) for name in numeric_names}
|
||||
if any(value is None for value in values.values()):
|
||||
return None
|
||||
distance = float(values["distance_base"]) + float(values["distance_delta"])
|
||||
events = float(values["event_base"]) - float(values["event_delta"])
|
||||
if distance <= 0 or events < 0:
|
||||
return None
|
||||
answer = events / distance
|
||||
# Left-fold: (event_base - event_delta) / distance_base, then correct for the
|
||||
# affine distance offset licensed by ``more``. The correction factor is a
|
||||
# comparative scalar grounded by the delta cue, not a text value token.
|
||||
distance_correction = float(values["distance_base"]) / distance
|
||||
derivation = GroundedDerivation(
|
||||
start=Quantity(
|
||||
value=float(values["event_base"]),
|
||||
unit=groups["event_unit"],
|
||||
source_token=groups["event_base"],
|
||||
),
|
||||
steps=(
|
||||
Step(
|
||||
op="subtract",
|
||||
operand=Quantity(
|
||||
value=float(values["event_delta"]),
|
||||
unit=groups["event_unit"],
|
||||
source_token=groups["event_delta"],
|
||||
),
|
||||
cue="less",
|
||||
),
|
||||
Step(
|
||||
op="divide",
|
||||
operand=Quantity(
|
||||
value=float(values["distance_base"]),
|
||||
unit=groups["distance_unit"],
|
||||
source_token=groups["distance_base"],
|
||||
),
|
||||
cue="per",
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(
|
||||
value=distance_correction,
|
||||
unit="distance_correction",
|
||||
source_token=groups["distance_delta"],
|
||||
),
|
||||
cue="more",
|
||||
comparative=True,
|
||||
),
|
||||
),
|
||||
)
|
||||
return BoundedRateProjectionBuild(
|
||||
mode="affine_event_per_distance",
|
||||
derivation=derivation,
|
||||
answer=answer,
|
||||
answer_unit=f"{_EVENT_SINGULAR[groups['event_unit']]}_per_{groups['question_distance']}",
|
||||
numeric_tokens=tuple(groups[name] for name in numeric_names),
|
||||
)
|
||||
|
||||
|
||||
def _build_percent_projection(text: str) -> BoundedRateProjectionBuild | None:
|
||||
match = _PERCENT_PROJECTION_RE.fullmatch(text.strip())
|
||||
if match is None:
|
||||
return None
|
||||
groups = {key: value.lower() for key, value in match.groupdict().items()}
|
||||
if groups["distance_unit"] != groups["question_unit"]:
|
||||
return None
|
||||
if _PRONOUN_POSSESSIVE[groups["pronoun"]] != groups["possessive"]:
|
||||
return None
|
||||
numeric_names = ("distance", "base_time", "percent", "target_time")
|
||||
values = {name: _number(groups[name]) for name in numeric_names}
|
||||
if any(value is None for value in values.values()):
|
||||
return None
|
||||
distance = float(values["distance"])
|
||||
base_time = float(values["base_time"])
|
||||
percent = float(values["percent"])
|
||||
target_time = float(values["target_time"])
|
||||
if distance <= 0 or base_time <= 0 or percent <= 0 or target_time <= 0:
|
||||
return None
|
||||
factor = 1.0 + percent / 100.0
|
||||
answer = distance / base_time * factor * target_time
|
||||
derivation = GroundedDerivation(
|
||||
start=Quantity(distance, groups["distance_unit"], groups["distance"]),
|
||||
steps=(
|
||||
Step(
|
||||
op="divide",
|
||||
operand=Quantity(base_time, "seconds", groups["base_time"]),
|
||||
cue="within",
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(factor, "percent_factor", groups["percent"]),
|
||||
cue="improve",
|
||||
comparative=True,
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(target_time, "seconds", groups["target_time"]),
|
||||
cue="within",
|
||||
),
|
||||
),
|
||||
)
|
||||
return BoundedRateProjectionBuild(
|
||||
mode="percent_improved_distance",
|
||||
derivation=derivation,
|
||||
answer=answer,
|
||||
answer_unit=groups["distance_unit"],
|
||||
numeric_tokens=tuple(groups[name] for name in numeric_names),
|
||||
)
|
||||
|
||||
|
||||
def build_bounded_rate_projection(text: str) -> BoundedRateProjectionBuild | None:
|
||||
"""Build exactly one licensed rate mode, otherwise refuse."""
|
||||
if not isinstance(text, str) or not text.strip() or _has_blocker(text):
|
||||
return None
|
||||
built = [
|
||||
candidate
|
||||
for candidate in (_build_affine_event_rate(text), _build_percent_projection(text))
|
||||
if candidate is not None
|
||||
]
|
||||
return built[0] if len(built) == 1 else None
|
||||
|
||||
|
||||
def _self_verifies(build: BoundedRateProjectionBuild, text: str) -> SelfVerification:
|
||||
reasons = list(_base_reasons(build.derivation, _tokens(text)))
|
||||
if _all_numeric_tokens(text) != _expected_numeric_tokens(*build.numeric_tokens):
|
||||
reasons.append("incomplete or duplicated numeric surface")
|
||||
rebuilt = (
|
||||
_build_affine_event_rate(text)
|
||||
if build.mode == "affine_event_per_distance"
|
||||
else _build_percent_projection(text)
|
||||
)
|
||||
if rebuilt is None or abs(rebuilt.answer - build.answer) > 1e-9:
|
||||
reasons.append("independent reconstruction failed")
|
||||
if abs(build.derivation.answer - build.answer) > 1e-9:
|
||||
reasons.append("derivation fold mismatch")
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_bounded_rate_projection(text: str) -> Resolution | None:
|
||||
"""Self-verify a unique bounded-rate mode."""
|
||||
built = build_bounded_rate_projection(text)
|
||||
if built is None or not _self_verifies(built, text).verified:
|
||||
return None
|
||||
return Resolution(built.answer, built.answer_unit, built.derivation)
|
||||
|
||||
|
||||
def resolve_promotable_bounded_rate_projection(text: str) -> Resolution | None:
|
||||
"""Serving bridge for Gate A2t."""
|
||||
return compose_bounded_rate_projection(text)
|
||||
268
generate/derivation/closed_reference_affine_aggregate.py
Normal file
268
generate/derivation/closed_reference_affine_aggregate.py
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
"""Gate A2u — closed explicit-reference affine aggregate ClusterContract.
|
||||
|
||||
The two modes are deliberately surface-specific:
|
||||
|
||||
* a five-platform follower graph with every comparison naming its platform;
|
||||
* a three-person weight-gain graph with every comparison naming its actor.
|
||||
|
||||
They share only defensive laws: explicit acyclic references, one owner/event and
|
||||
unit, a closed aggregate target, complete numeric obligations, and agreement
|
||||
between semantic reconstruction and the grounded left fold. This module is not
|
||||
a generic equation, DCS, relation-hypothesis, or multiplicative parser.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from typing import Final, Literal
|
||||
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.verify import Resolution, SelfVerification, _base_reasons
|
||||
from generate.math_roundtrip import WORD_NUMBERS, _tokens
|
||||
|
||||
|
||||
_NUMBER: Final[str] = r"(?:\d+(?:\.\d+)?|[A-Za-z]+)"
|
||||
_SOCIAL_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^(?P<owner>[A-Z][A-Za-z'-]+)\s+has\s+(?P<instagram>{_NUMBER})\s+followers\s+on\s+"
|
||||
rf"Instagram\s+and\s+(?P<facebook>{_NUMBER})\s+followers\s+on\s+Facebook\.\s+"
|
||||
rf"The\s+number\s+of\s+followers\s+(?P<pronoun>he|she)\s+has\s+on\s+Twitter\s+is\s+"
|
||||
rf"half\s+the\s+number\s+of\s+followers\s+(?P=pronoun)\s+has\s+on\s+Instagram\s+and\s+"
|
||||
rf"Facebook\s+combined\.\s+Meanwhile,\s+the\s+number\s+of\s+followers\s+(?P=pronoun)\s+"
|
||||
rf"has\s+on\s+TikTok\s+is\s+(?P<scale>{_NUMBER})\s+times\s+the\s+number\s+of\s+"
|
||||
rf"followers\s+(?:(?P=pronoun)|is)\s+has\s+on\s+Twitter,\s+and\s+(?P=pronoun)\s+has\s+"
|
||||
rf"(?P<delta>{_NUMBER})\s+more\s+followers\s+on\s+Youtube\s+than\s+(?P=pronoun)\s+has\s+"
|
||||
rf"on\s+TikTok\.\s+How\s+many\s+followers\s+does\s+(?P<question_owner>[A-Z][A-Za-z'-]+)\s+"
|
||||
rf"have\s+on\s+all\s+(?P<possessive>his|her)\s+social\s+media\?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_WEIGHT_RE: Final[re.Pattern[str]] = re.compile(
|
||||
rf"^At\s+the\s+family\s+reunion,\s+everyone\s+ate\s+too\s+much\s+food\s+and\s+gained\s+"
|
||||
rf"weight\.\s+(?P<first>[A-Z][A-Za-z'-]+)\s+gained\s+(?P<base>{_NUMBER})\s+pounds\.\s+"
|
||||
rf"(?P<second>[A-Z][A-Za-z'-]+)\s+gained\s+(?P<more>{_NUMBER})\s+pounds\s+more\s+than\s+"
|
||||
rf"twice\s+what\s+(?P<second_ref>[A-Z][A-Za-z'-]+)\s+gained\.\s+"
|
||||
rf"(?P<third>[A-Z][A-Za-z'-]+)\s+gained\s+(?P<less>{_NUMBER})\s+pounds\s+less\s+than\s+"
|
||||
rf"half\s+of\s+what\s+(?P<third_ref>[A-Z][A-Za-z'-]+)\s+gained\.\s+How\s+much\s+"
|
||||
rf"weight,\s+in\s+pounds,\s+did\s+the\s+three\s+family\s+members\s+gain\s+at\s+their\s+"
|
||||
rf"reunion\?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||
{
|
||||
"bags", "bill", "color", "dollars", "draw", "each", "eats",
|
||||
"hours", "insurance", "macaroons", "ounces", "packs", "percent",
|
||||
"pictures", "scoops", "years",
|
||||
}
|
||||
)
|
||||
_PRONOUN_POSSESSIVE: Final[dict[str, str]] = {"he": "his", "she": "her"}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ClosedReferenceAffineAggregateBuild:
|
||||
mode: Literal["five_platform_followers", "three_actor_weight"]
|
||||
derivation: GroundedDerivation
|
||||
answer: float
|
||||
answer_unit: str
|
||||
numeric_tokens: tuple[str, ...]
|
||||
|
||||
|
||||
def _number(token: str) -> float | None:
|
||||
lowered = token.lower()
|
||||
if lowered in WORD_NUMBERS:
|
||||
return float(WORD_NUMBERS[lowered])
|
||||
try:
|
||||
return float(token)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _statement_scope(text: str) -> str:
|
||||
"""Count numeric obligations only in the statement body, not question targets."""
|
||||
for marker in ("how much", "how many"):
|
||||
idx = text.lower().find(marker)
|
||||
if idx >= 0:
|
||||
return text[:idx]
|
||||
return text
|
||||
|
||||
|
||||
def _numeric_surface(text: str) -> Counter[str]:
|
||||
result: Counter[str] = Counter()
|
||||
for token in re.findall(r"\b\d+(?:\.\d+)?\b|\b[A-Za-z]+\b", _statement_scope(text)):
|
||||
lowered = token.lower()
|
||||
if re.fullmatch(r"\d+(?:\.\d+)?", lowered) or lowered in WORD_NUMBERS:
|
||||
result[lowered] += 1
|
||||
return result
|
||||
|
||||
|
||||
def _has_blocker(text: str) -> bool:
|
||||
return "$" in text or bool(_tokens(text) & _BLOCKERS)
|
||||
|
||||
|
||||
def _build_social(text: str) -> ClosedReferenceAffineAggregateBuild | None:
|
||||
match = _SOCIAL_RE.fullmatch(text.strip())
|
||||
if match is None:
|
||||
return None
|
||||
groups = {key: value.lower() for key, value in match.groupdict().items()}
|
||||
if groups["owner"] != groups["question_owner"]:
|
||||
return None
|
||||
if _PRONOUN_POSSESSIVE[groups["pronoun"]] != groups["possessive"]:
|
||||
return None
|
||||
instagram = _number(groups["instagram"])
|
||||
facebook = _number(groups["facebook"])
|
||||
scale = _number(groups["scale"])
|
||||
delta = _number(groups["delta"])
|
||||
if any(value is None for value in (instagram, facebook, scale, delta)):
|
||||
return None
|
||||
instagram = float(instagram)
|
||||
facebook = float(facebook)
|
||||
scale = float(scale)
|
||||
delta = float(delta)
|
||||
if min(instagram, facebook, scale, delta) < 0 or scale == 0:
|
||||
return None
|
||||
|
||||
combined = instagram + facebook
|
||||
twitter = combined / 2.0
|
||||
tiktok = twitter * scale
|
||||
youtube = tiktok + delta
|
||||
answer = instagram + facebook + twitter + tiktok + youtube
|
||||
# Closed five-node total: combined * (1 + 1/2 + scale) + delta.
|
||||
aggregate_factor = 1.5 + scale
|
||||
derivation = GroundedDerivation(
|
||||
start=Quantity(instagram, "followers", groups["instagram"]),
|
||||
steps=(
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(facebook, "followers", groups["facebook"]),
|
||||
cue="combined",
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(aggregate_factor, "closed_nodes", "half"),
|
||||
cue="half",
|
||||
comparative=True,
|
||||
),
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(delta, "followers", groups["delta"]),
|
||||
cue="more",
|
||||
),
|
||||
),
|
||||
)
|
||||
return ClosedReferenceAffineAggregateBuild(
|
||||
mode="five_platform_followers",
|
||||
derivation=derivation,
|
||||
answer=answer,
|
||||
answer_unit="followers",
|
||||
numeric_tokens=(
|
||||
groups["instagram"],
|
||||
groups["facebook"],
|
||||
"half",
|
||||
groups["scale"],
|
||||
groups["delta"],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _build_weight(text: str) -> ClosedReferenceAffineAggregateBuild | None:
|
||||
match = _WEIGHT_RE.fullmatch(text.strip())
|
||||
if match is None:
|
||||
return None
|
||||
groups = {key: value.lower() for key, value in match.groupdict().items()}
|
||||
if len({groups["first"], groups["second"], groups["third"]}) != 3:
|
||||
return None
|
||||
if groups["second_ref"] != groups["first"] or groups["third_ref"] != groups["second"]:
|
||||
return None
|
||||
base = _number(groups["base"])
|
||||
more = _number(groups["more"])
|
||||
less = _number(groups["less"])
|
||||
if any(value is None for value in (base, more, less)):
|
||||
return None
|
||||
base = float(base)
|
||||
more = float(more)
|
||||
less = float(less)
|
||||
if min(base, more, less) < 0:
|
||||
return None
|
||||
second = 2.0 * base + more
|
||||
third = second / 2.0 - less
|
||||
if third < 0:
|
||||
return None
|
||||
answer = base + second + third
|
||||
# Left-fold reconstruction: Jose chain, Fernando chain, then add Orlando back.
|
||||
derivation = GroundedDerivation(
|
||||
start=Quantity(base, "pounds", groups["base"]),
|
||||
steps=(
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(2.0, "comparative", "twice"),
|
||||
cue="twice",
|
||||
comparative=True,
|
||||
),
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(more, "pounds", groups["more"]),
|
||||
cue="more",
|
||||
),
|
||||
Step(
|
||||
op="multiply",
|
||||
operand=Quantity(1.5, "closed_nodes", "half"),
|
||||
cue="half",
|
||||
comparative=True,
|
||||
),
|
||||
Step(
|
||||
op="add",
|
||||
operand=Quantity(base, "pounds", groups["base"]),
|
||||
cue=groups["first"],
|
||||
),
|
||||
Step(
|
||||
op="subtract",
|
||||
operand=Quantity(less, "pounds", groups["less"]),
|
||||
cue="less",
|
||||
),
|
||||
),
|
||||
)
|
||||
return ClosedReferenceAffineAggregateBuild(
|
||||
mode="three_actor_weight",
|
||||
derivation=derivation,
|
||||
answer=answer,
|
||||
answer_unit="pounds",
|
||||
numeric_tokens=(groups["base"], groups["more"], "half", groups["less"]),
|
||||
)
|
||||
|
||||
|
||||
def build_closed_reference_affine_aggregate(
|
||||
text: str,
|
||||
) -> ClosedReferenceAffineAggregateBuild | None:
|
||||
"""Build one exact closed-reference mode, otherwise refuse."""
|
||||
if not isinstance(text, str) or not text.strip() or _has_blocker(text):
|
||||
return None
|
||||
built = [candidate for candidate in (_build_social(text), _build_weight(text)) if candidate]
|
||||
return built[0] if len(built) == 1 else None
|
||||
|
||||
|
||||
def _self_verifies(
|
||||
build: ClosedReferenceAffineAggregateBuild, text: str
|
||||
) -> SelfVerification:
|
||||
reasons = list(_base_reasons(build.derivation, _tokens(text)))
|
||||
if _numeric_surface(text) != Counter(token.lower() for token in build.numeric_tokens):
|
||||
reasons.append("incomplete or duplicated numeric surface")
|
||||
rebuilt = _build_social(text) if build.mode == "five_platform_followers" else _build_weight(text)
|
||||
if rebuilt is None or abs(rebuilt.answer - build.answer) > 1e-9:
|
||||
reasons.append("independent reconstruction failed")
|
||||
if abs(build.derivation.answer - build.answer) > 1e-9:
|
||||
reasons.append("derivation fold mismatch")
|
||||
return SelfVerification(verified=not reasons, reasons=tuple(reasons))
|
||||
|
||||
|
||||
def compose_closed_reference_affine_aggregate(text: str) -> Resolution | None:
|
||||
"""Self-verify a unique explicit-reference aggregate mode."""
|
||||
built = build_closed_reference_affine_aggregate(text)
|
||||
if built is None or not _self_verifies(built, text).verified:
|
||||
return None
|
||||
return Resolution(built.answer, built.answer_unit, built.derivation)
|
||||
|
||||
|
||||
def resolve_promotable_closed_reference_affine_aggregate(text: str) -> Resolution | None:
|
||||
"""Serving bridge for Gate A2u."""
|
||||
return compose_closed_reference_affine_aggregate(text)
|
||||
|
|
@ -930,6 +930,38 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2t — bounded affine/percent rate projections (Sprint 13 contract).
|
||||
from generate.derivation.bounded_rate_projection import (
|
||||
resolve_promotable_bounded_rate_projection,
|
||||
)
|
||||
|
||||
bounded_rate_resolution = resolve_promotable_bounded_rate_projection(text)
|
||||
if bounded_rate_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=bounded_rate_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# Gate A2u — closed explicit-reference affine aggregates (Sprint 13).
|
||||
from generate.derivation.closed_reference_affine_aggregate import (
|
||||
resolve_promotable_closed_reference_affine_aggregate,
|
||||
)
|
||||
|
||||
closed_affine_resolution = resolve_promotable_closed_reference_affine_aggregate(
|
||||
text
|
||||
)
|
||||
if closed_affine_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=closed_affine_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -170,6 +170,8 @@ def test_current_baseline_snapshot() -> None:
|
|||
Gate A2j giveaway_target_residual admits cv-0021 (0035).
|
||||
Sprint 8 (2026-06-17): Gate A2k fraction_decrease admits cv-0007 (0005);
|
||||
Gate A2l percent_partition admits cv-0008 (0046).
|
||||
Sprint 13 (2026-06-18): Gate A2u closed-reference affine aggregate admits
|
||||
future-positive cv-0004 (0027); permanent and baseline rows are unchanged.
|
||||
"""
|
||||
solve = refuse = wrong = 0
|
||||
for case in _CASES:
|
||||
|
|
@ -181,7 +183,7 @@ def test_current_baseline_snapshot() -> None:
|
|||
else:
|
||||
refuse += 1
|
||||
assert wrong == 0
|
||||
assert (solve, refuse) == (14, 8), (
|
||||
assert (solve, refuse) == (15, 7), (
|
||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||
f"slice landed, update this expectation and the affected rows' "
|
||||
f"baseline fields in lockstep"
|
||||
|
|
|
|||
|
|
@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate():
|
|||
summary = build_microscope_report(_load_cases())
|
||||
md = render_markdown(summary)
|
||||
assert "partition_chunking" in md
|
||||
assert "correct: 26" in md
|
||||
assert "correct: 30" in md
|
||||
assert "Gate A2a unit_partition" in md
|
||||
|
||||
|
||||
|
|
|
|||
280
tests/test_math_candidate_graph_xhigh_sprint13_lift.py
Normal file
280
tests/test_math_candidate_graph_xhigh_sprint13_lift.py
Normal file
|
|
@ -0,0 +1,280 @@
|
|||
"""Sprint 13 contract-backed capability bundle.
|
||||
|
||||
The tests are the serving license. They pin four target chains while proving
|
||||
that the selected organs remain inert on sealed-wrong, blocked-family, changed
|
||||
binding, and permanent composition-validation surfaces.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from generate.derivation.bounded_rate_projection import (
|
||||
build_bounded_rate_projection,
|
||||
compose_bounded_rate_projection,
|
||||
resolve_promotable_bounded_rate_projection,
|
||||
)
|
||||
from generate.derivation.closed_reference_affine_aggregate import (
|
||||
build_closed_reference_affine_aggregate,
|
||||
compose_closed_reference_affine_aggregate,
|
||||
resolve_promotable_closed_reference_affine_aggregate,
|
||||
)
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
|
||||
|
||||
_ROOT = Path(__file__).resolve().parents[1]
|
||||
_TRAIN_PATH = _ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
|
||||
_CV_PATH = _ROOT / "evals/gsm8k_math/composition_validation/v1/cases.jsonl"
|
||||
|
||||
|
||||
def _load_train() -> dict[str, str]:
|
||||
result: dict[str, str] = {}
|
||||
for line in _TRAIN_PATH.read_text(encoding="utf-8").splitlines():
|
||||
row = json.loads(line)
|
||||
result[row["case_id"].split("-")[-1]] = row["question"]
|
||||
return result
|
||||
|
||||
|
||||
CASES = _load_train()
|
||||
PRESERVED_SOLVED = {
|
||||
"0001", "0002", "0003", "0004", "0005", "0006", "0007", "0008",
|
||||
"0009", "0010", "0013", "0014", "0015", "0017", "0018", "0021",
|
||||
"0024", "0025", "0029", "0030", "0035", "0037", "0038", "0042",
|
||||
"0045", "0046",
|
||||
}
|
||||
|
||||
|
||||
class TestBoundedRateProjectionTargets:
|
||||
@pytest.mark.parametrize(("case_id", "answer"), [("0016", 2.0), ("0034", 112.0)])
|
||||
def test_target_build_compose_and_promote(self, case_id: str, answer: float) -> None:
|
||||
text = CASES[case_id]
|
||||
built = build_bounded_rate_projection(text)
|
||||
assert built is not None
|
||||
assert built.derivation.answer == pytest.approx(answer)
|
||||
assert compose_bounded_rate_projection(text).answer == pytest.approx(answer)
|
||||
assert resolve_promotable_bounded_rate_projection(text).answer == pytest.approx(answer)
|
||||
assert parse_and_solve(text).answer == pytest.approx(answer)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("text", "answer"),
|
||||
[
|
||||
(
|
||||
"On Mira's bicycle trip across town, she traveled 4 more than 8 "
|
||||
"kilometers and encountered 6 less than 30 traffic lights. How many "
|
||||
"traffic lights per kilometer did Mira encounter on her trip across town?",
|
||||
2.0,
|
||||
),
|
||||
(
|
||||
"Amira is a varsity player on a football team. She can run 60 meters "
|
||||
"within 6 seconds. If she can improve her speed by twenty percent, how "
|
||||
"many meters will she be able to run within 12 seconds?",
|
||||
144.0,
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_sibling_generality(self, text: str, answer: float) -> None:
|
||||
assert resolve_promotable_bounded_rate_projection(text).answer == pytest.approx(answer)
|
||||
|
||||
@pytest.mark.parametrize("case_id", ["0018", "0019", "0028", "0032", "0047"])
|
||||
def test_sealed_wrong_neighbors_refuse(self, case_id: str) -> None:
|
||||
assert resolve_promotable_bounded_rate_projection(CASES[case_id]) is None
|
||||
|
||||
@pytest.mark.parametrize("case_id", ["0027", "0039"])
|
||||
def test_cross_family_refuses(self, case_id: str) -> None:
|
||||
assert resolve_promotable_bounded_rate_projection(CASES[case_id]) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
CASES["0016"].replace("did Rudolph encounter", "did Marco encounter"),
|
||||
CASES["0016"].replace("stop signs per mile", "miles per stop sign"),
|
||||
CASES["0016"].replace("per mile", "per hour"),
|
||||
CASES["0016"].replace(
|
||||
"How many stop signs", "He passed 9 billboards. How many stop signs"
|
||||
),
|
||||
CASES["0034"].replace("will he be able", "will Taylor be able"),
|
||||
CASES["0034"].replace("how many yards", "what speed"),
|
||||
CASES["0034"].replace("how many yards", "how many meters"),
|
||||
CASES["0034"].replace(
|
||||
"how many yards", "he rested for 7 seconds. how many yards"
|
||||
),
|
||||
(
|
||||
"Rudolph counted 2 blue signs, 5 miles, 3 red signs, and 17 cones. "
|
||||
"How many stop signs per mile did Rudolph encounter?"
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_binding_target_unit_and_completeness_confusers_refuse(self, text: str) -> None:
|
||||
assert resolve_promotable_bounded_rate_projection(text) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
CASES["0016"].replace(
|
||||
"traveled 2 more than 5 miles",
|
||||
"traveled for half an hour",
|
||||
),
|
||||
CASES["0016"].replace(
|
||||
"encountered 3 less than 17 stop signs",
|
||||
"spent a quarter on tolls and encountered 3 less than 17 stop signs",
|
||||
),
|
||||
CASES["0034"].replace(
|
||||
"forty percent",
|
||||
"a third percent",
|
||||
),
|
||||
CASES["0034"].replace(
|
||||
"forty percent",
|
||||
"one quarter percent",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_fraction_word_and_nonlicensed_percent_confusers_refuse(self, text: str) -> None:
|
||||
assert resolve_promotable_bounded_rate_projection(text) is None
|
||||
|
||||
def test_affine_rate_completeness_refuses_extra_distance_obligation(self) -> None:
|
||||
"""Non-vacuous: an extra mile quantity the regex binds must refuse."""
|
||||
text = (
|
||||
"On Rudolph's car trip across town, he traveled 2 more than 5 miles "
|
||||
"and also 9 more than 1 miles and encountered 3 less than 17 stop signs. "
|
||||
"How many stop signs per mile did Rudolph encounter on his trip across town?"
|
||||
)
|
||||
assert resolve_promotable_bounded_rate_projection(text) is None
|
||||
|
||||
|
||||
class TestClosedReferenceAffineAggregateTargets:
|
||||
@pytest.mark.parametrize(("case_id", "answer"), [("0027", 3840.0), ("0039", 20.0)])
|
||||
def test_target_build_compose_and_promote(self, case_id: str, answer: float) -> None:
|
||||
text = CASES[case_id]
|
||||
built = build_closed_reference_affine_aggregate(text)
|
||||
assert built is not None
|
||||
assert built.derivation.answer == pytest.approx(answer)
|
||||
assert compose_closed_reference_affine_aggregate(text).answer == pytest.approx(answer)
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(text).answer == pytest.approx(answer)
|
||||
assert parse_and_solve(text).answer == pytest.approx(answer)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("text", "answer"),
|
||||
[
|
||||
(
|
||||
"Lena has 100 followers on Instagram and 300 followers on Facebook. "
|
||||
"The number of followers she has on Twitter is half the number of "
|
||||
"followers she has on Instagram and Facebook combined. Meanwhile, the "
|
||||
"number of followers she has on TikTok is 2 times the number of followers "
|
||||
"she has on Twitter, and she has 50 more followers on Youtube than she "
|
||||
"has on TikTok. How many followers does Lena have on all her social media?",
|
||||
1450.0,
|
||||
),
|
||||
(
|
||||
"At the family reunion, everyone ate too much food and gained weight. "
|
||||
"Ava gained 6 pounds. Ben gained four pounds more than twice what Ava "
|
||||
"gained. Cara gained 2 pounds less than half of what Ben gained. How much "
|
||||
"weight, in pounds, did the three family members gain at their reunion?",
|
||||
28.0,
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_sibling_generality(self, text: str, answer: float) -> None:
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(text).answer == pytest.approx(answer)
|
||||
|
||||
@pytest.mark.parametrize("case_id", ["0023", "0025", "0032", "0033", "0040", "0047"])
|
||||
def test_blocked_and_sealed_neighbors_refuse(self, case_id: str) -> None:
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(CASES[case_id]) is None
|
||||
|
||||
@pytest.mark.parametrize("case_id", ["0016", "0034"])
|
||||
def test_cross_family_refuses(self, case_id: str) -> None:
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(CASES[case_id]) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
CASES["0027"].replace("than he has on TikTok", "than he has on LinkedIn"),
|
||||
CASES["0027"].replace("all his social media", "TikTok"),
|
||||
CASES["0027"].replace("How many followers", "How many likes"),
|
||||
CASES["0027"].replace(
|
||||
"How many followers", "He has 9 followers on Mastodon. How many followers"
|
||||
),
|
||||
CASES["0039"].replace("what Orlando gained", "what Xavier gained"),
|
||||
CASES["0039"].replace("the three family members", "Jose"),
|
||||
CASES["0039"].replace("in pounds", "in years"),
|
||||
CASES["0039"].replace(
|
||||
"How much weight", "Maria gained 7 pounds. How much weight"
|
||||
),
|
||||
(
|
||||
"Malcolm listed 240 blue cards, 500 green cards, 3 red cards, and 510 "
|
||||
"yellow cards. How many followers does Malcolm have on all his social media?"
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_reference_target_unit_and_completeness_confusers_refuse(self, text: str) -> None:
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(text) is None
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
CASES["0027"].replace(
|
||||
"half the number of followers he has on Instagram and Facebook combined",
|
||||
"a third the number of followers he has on Instagram and Facebook combined",
|
||||
),
|
||||
CASES["0027"].replace(
|
||||
"half the number of followers he has on Instagram and Facebook combined",
|
||||
"one quarter the number of followers he has on Instagram and Facebook combined",
|
||||
),
|
||||
CASES["0027"].replace(
|
||||
"half the number of followers he has on Instagram and Facebook combined",
|
||||
"three quarters the number of followers he has on Instagram and Facebook combined",
|
||||
),
|
||||
CASES["0039"].replace(
|
||||
"half of what Jose gained",
|
||||
"a quarter of what Jose gained",
|
||||
),
|
||||
CASES["0039"].replace(
|
||||
"twice what Orlando gained",
|
||||
"three times what Orlando gained",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_fraction_word_and_nonlicensed_comparative_confusers_refuse(self, text: str) -> None:
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(text) is None
|
||||
|
||||
def test_weight_completeness_refuses_extra_actor_obligation(self) -> None:
|
||||
"""Non-vacuous: an extra named actor gain in the same family must refuse."""
|
||||
text = (
|
||||
"At the family reunion, everyone ate too much food and gained weight. "
|
||||
"Orlando gained 5 pounds. Jose gained two pounds more than twice what "
|
||||
"Orlando gained. Maria gained 7 pounds. Fernando gained 3 pounds less than "
|
||||
"half of what Jose gained. How much weight, in pounds, did the three family "
|
||||
"members gain at their reunion?"
|
||||
)
|
||||
assert resolve_promotable_closed_reference_affine_aggregate(text) is None
|
||||
|
||||
|
||||
def test_permanent_composition_guards_remain_refused() -> None:
|
||||
for line in _CV_PATH.read_text(encoding="utf-8").splitlines():
|
||||
row = json.loads(line)
|
||||
if row["gate"] == "permanent":
|
||||
assert parse_and_solve(row["question"]).answer is None, row["case_id"]
|
||||
|
||||
|
||||
class TestScoreAndHoldout:
|
||||
def test_train_sample_legendary_state_and_preservation(self) -> None:
|
||||
from evals.gsm8k_math.train_sample.v1.runner import _load_cases, build_report
|
||||
|
||||
report = build_report(_load_cases(_TRAIN_PATH))
|
||||
counts = report["counts"]
|
||||
assert counts == {"correct": 30, "wrong": 0, "refused": 20}
|
||||
correct = {
|
||||
row["case_id"].split("-")[-1]
|
||||
for row in report["per_case"]
|
||||
if row["verdict"] == "correct"
|
||||
}
|
||||
assert PRESERVED_SOLVED <= correct
|
||||
assert {"0016", "0027", "0034", "0039"} <= correct
|
||||
|
||||
def test_holdout_wrong_zero(self) -> None:
|
||||
from evals.gsm8k_math.holdout_dev.v1.runner import build_report
|
||||
|
||||
report = build_report()
|
||||
assert report["counts"]["wrong"] == 0
|
||||
Loading…
Reference in a new issue