diff --git a/docs/analysis/gsm8k-xhigh-capability-sprint13-lookback-2026-06-18.md b/docs/analysis/gsm8k-xhigh-capability-sprint13-lookback-2026-06-18.md new file mode 100644 index 00000000..920f82d5 --- /dev/null +++ b/docs/analysis/gsm8k-xhigh-capability-sprint13-lookback-2026-06-18.md @@ -0,0 +1,387 @@ +# GSM8K XHIGH Capability Sprint 13 — Lookback (2026-06-18) + +> Status: research gate complete; implementation and final validation pending. + +## 1. Starting baseline + +The canonical current API is `evals.gsm8k_math.train_sample.v1.runner.build_report`. +The brief's `train_sample.v1.report` import is stale and does not exist on `main`. +Using the canonical runner on the committed 50-case sample: + +| Metric | Value | +|---|---:| +| correct | **26** | +| refused | **24** | +| wrong | **0** | + +Correct IDs: 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008, 0009, +0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, +0035, 0037, 0038, 0042, 0045, 0046. Wrong IDs: none. + +## 2. Intrinsic space and architectural directions + +The refused frontier is not one arithmetic space. It is a disjoint union of +small relational manifolds. Surface categories such as +`discrete_count_statement` and `descriptive_setup_no_quantity` distort this +geometry by grouping different typed chains under one recognizer label. The +serving invariant is therefore preserved by admitting only a complete typed +relation field: named actors/objects, licensed operators, a bound question +target, source-quantity obligations, and a reconstructible final value. + +Directions mapped before implementation: + +1. **Two contract-backed mini-families (Path A candidate).** Pair 0016/0034 as + bounded rate projections with explicit numerator, denominator, and target + dimensions. Pair 0027/0039 as closed explicit-reference affine aggregates, + implemented as two surface-specific builders rather than a generic relation + graph. This can yield +4 without a generic equation, fraction, DCS, temporal, + or multiplicative parser. +2. **Four unrelated strict singletons.** 0012, 0016, 0034, and 0049 each have a + clear arithmetic graph, but this repeats binding logic and offers less + reusable structure. It is a score bundle, not a coherent capability bundle. +3. **One mini-family plus defensive substrate (Path B/C).** Ship only the + mini-family that clears confusers and holdout, plus a tiny quantity-obligation + or actor-binding helper only if it is adopted without widening behavior. +4. **No serving lift (Path D).** If either selected family creates a train or + holdout wrong answer, or cannot prove complete binding, preserve 26/24/0 and + land only this diagnostic evidence. + +## 3. Scout and Experience Flywheel summary + +Commands: + +```text +uv run python scripts/gsm8k_sealed_attempt_scout.py --limit 50 +uv run python scripts/gsm8k_experience_flywheel.py --limit 50 +uv run python scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience-xhigh-sprint13.json +``` + +| Regime | correct | refused | wrong | +|---|---:|---:|---:| +| serving | 26 | 24 | 0 | +| sealed scout | 3 | 39 | 8 | + +Scout deltas: 3 already served, 6 elimination-refused-to-wrong, 18 joint +refusals, 0 refused-to-correct lifts, 23 conservative wins. The flywheel has no +promotable family. Its broad family summaries remain diagnostic only. + +| family | case IDs | refused-to-correct | sealed-wrong | blocked | top missing primitive | action | +|---|---|---:|---:|---:|---|---| +| already_served | 0003, 0021, 0037 | 0 | 0 | 0 | — | preserve | +| conservative_boundary | 23 served cases | 0 | 2 | 2 | — | preserve serving gates | +| diagnostic_hold:currency_amount | 0019, 0028, 0043 | 0 | 2 | 2 | diagnostic_hold | block wholesale currency | +| diagnostic_hold:descriptive_setup_no_quantity | 0012, 0023, 0027 | 0 | 0 | 0 | diagnostic_hold | decompose by typed relation | +| joint_sealed_no_resolution | 0048, 0050 | 0 | 0 | 0 | — | diagnostic hold | +| relation_hypothesis:discrete_count_statement | 0016, 0020, 0022, 0031, 0032, 0033, 0034, 0036, 0039, 0040, 0041, 0044, 0047, 0049 | 0 | 2 | 2 | relation_hypothesis | reject DCS; decompose exact chains | +| sealed_elimination | 0011, 0026 | 0 | 2 | 2 | — | block | + +Sealed-wrong IDs are 0011, 0018, 0019, 0025, 0026, 0028, 0032, and +0047. Only 0011, 0019, 0026, 0028, 0032, and 0047 are currently refused; +0018 and 0025 are conservative serving wins and regression anchors. + +## 4. Full refused-case inventory + +| ID | Gold | Question text | Typed arithmetic graph | First missing / current refusal | Nearest evidence and candidate | +|---|---:|---|---|---|---| +| 0011 | 50 | Alexa sells lemonade for $2/cup, spent $20, and asks cups needed for $80 profit. | `(80 + 20) / 2` | graph solve; sealed elimination | 0019/0028 currency wrongs; blocked profit-cost recovery | +| 0012 | 7 | Dennis has 10 rocks; fish ate half; two were recovered; asks rocks left. | `10 × (1-1/2) + 2` | descriptive no-quantity injection | clean fraction-loss/recovery singleton; neighbor 0005 is a different fraction-decrease organ | +| 0016 | 2 | Rudolph travels 2 more than 5 miles and sees 3 less than 17 signs; asks signs per mile. | `(17-3)/(5+2)` | relation hypothesis | clean dual-offset ratio singleton; changed-target hazard | +| 0019 | 660 | Three $400 vet visits, $100 insurance after first, 80% coverage on later visits; asks total paid. | `400 + 100 + 2×400×0.2` | currency injection | sealed wrong 120000; currency family blocked | +| 0020 | 130 | Two puppies, two kittens, three parakeets; comparative prices from a $10 parakeet; asks all-pet cost. | `3×10 + 2×20 + 2×30` | relation hypothesis | multi-species price graph; currency and ambiguous “three times more” hazards | +| 0022 | 4800 | $20/kg, past four-month catch 80 kg, today twice past total; asks total earnings including today. | `(80 + 2×80)×20` | relation hypothesis | bounded historical aggregate; currency overlap | +| 0023 | 150 | Nicole has 400 cards, Cindy twice that, Rex half their combined total, then divides among himself and three siblings. | `(400 + 2×400)×1/2/(3+1)` | descriptive no-quantity injection | **blocked**: Cindy's multiplier has no explicit reference and is adjacent to permanent cv-0014/0015 guards; 0027 does not retroactively license it | +| 0026 | 6 | Two people each save $40, bill is 3/4, equal $1.5 scoops, $1 change each; asks scoops each. | `(((2×40)×1/4)-2)/2/1.5` | statement parse | sealed wrong 180; sealed-elimination/currency blocked | +| 0027 | 3840 | Instagram 240 + Facebook 500; Twitter half combined; TikTok 3× Twitter; YouTube TikTok+510; asks all followers. | `S=240+500; T=S/2; K=3T; Y=K+510; total=240+500+T+K+Y` | descriptive no-quantity injection | positive anchor for selected combined-half family; 0023 sibling | +| 0028 | 200 | $100,000 opening cost, daily cost 1%, 150 tickets/day at $10; asks days to recover investment. | `100000 / (150×10 - 100000×0.01)` | currency injection | sealed wrong 0; currency/profit recovery blocked | +| 0031 | 92 | Jeremie plus three friends buy $18 tickets and $5 snacks each; asks total. | `(3+1)×(18+5)` | relation hypothesis | bounded group-cost aggregate but currency family adjacency | +| 0032 | 34 | Ten pictures, 2 hours drawing each, coloring takes 30% less; asks total hours. | `10×(2 + 2×0.7)` | relation hypothesis | sealed wrong 20; DCS/percent-time blocked | +| 0033 | 60 | Rachel is 12; grandfather 7×; mother half grandfather; father 5 older; asks father age when Rachel is 25. | `father_now=(12×7)/2+5; answer=father_now+(25-12)` | relation hypothesis | selected named-actor affine chain; 0039 sibling; 0032 negative | +| 0034 | 112 | 40 yards in 5 seconds, speed improves 40%; asks yards in 10 seconds. | `(40/5)×1.4×10` | relation hypothesis | clean percent-rate projection singleton; 0032 percent-time negative | +| 0036 | 22 | Study 2 hours Wednesday, 3× Thursday, half Thursday Friday, weekend equals weekday total; asks five-day total. | `W=2; T=3W; F=T/2; total=2×(W+T+F)` | relation hypothesis | bounded schedule aggregate; temporal ambiguity on “during weekend” | +| 0039 | 20 | Orlando gains 5; Jose gains two more than twice Orlando; Fernando gains 3 less than half Jose; asks all three total. | `J=2×5+2; F=J/2-3; total=5+J+F` | relation hypothesis | selected named-actor affine chain; 0033 sibling; 0032 negative | +| 0040 | 72 | Counts of horses, dogs, cats, turtles, goat; asks total legs. | `(2+5+7+3+1)×4` | relation hypothesis | requires animal-leg external grounding; implementation blocked | +| 0041 | 6 | Two 16-piece brownie pans; guests eat one plus 75% of second; all but four use two scoops; eight scoops/tub. | `guests=16+0.75×16; tubs=(guests-4)×2/8` | relation hypothesis | multi-stage implicit guest binding; fraction/container hazards | +| 0043 | 11 | Sandra has $10+$4+twice $4; buys 14×$0.5 and 20×$0.2; asks money left. | `10+4+2×4-14×0.5-20×0.2` | currency injection | no sealed wrong itself, but 0019/0028 family blockers | +| 0044 | 1300 | $1000 earns 10% simple interest for three years; asks final money. | `1000×(1+0.1×3)` | relation hypothesis | explicit simple-interest singleton; currency adjacency | +| 0047 | 45 | 12 five-ounce macaroons split over four bags; one bag eaten; asks remaining weight. | `(12-12/4)×5` | relation hypothesis | sealed wrong 240; DCS/divisive blocked | +| 0048 | 4 | Start 20 cards; +6 weekly; -2 every two weeks; asks weeks to reach 40. | `20 + 6w - 2×(w/2)=40`, with even-period obligation | statement parse | bounded periodic recurrence; generic temporal parser prohibited | +| 0049 | 18 | Two walking routes with twice/third segment relations; asks how much longer route two is. | `R1=6+12+(6+12)/3; R2=14+28; R2-R1` | relation hypothesis | clean route-duration comparator; nearest duration organs are narrower | +| 0050 | 280 (artifact inconsistent) | Gig every other day for two weeks; two five-minute songs and one twice as long; asks minutes. | Source rationale computes `7×2=14` gigs and `14×20=140`, while stored gold is 280. | statement parse | **implementation forbidden**: dataset's rationale and gold contradict; do not hardcode the final answer | + +## 5. Candidate ranking and research gate + +| Rank | Candidate / cases | Expected lift | Typed-chain clarity | Sealed-wrong / blocked overlap | Decision | +|---:|---|---:|---|---|---| +| 1 | `bounded_rate_projection` / 0016, 0034 | +2 | high; two explicit affine/rate graphs with dimension-bound questions | 0018/0032 nearest rate/percent sealed wrong; 0019/0028 currency and 0047 packing confusers | implement as two narrow modes under one ClusterContract | +| 2 | `closed_explicit_reference_affine_aggregate` / 0027, 0039 | +2 | high; every comparative names its source and target aggregate is closed | DCS surface includes 0032/0047 and aggregate 0025; permanent no-reference guards must remain refused | implement as two surface-specific modes under one ClusterContract | +| 3 | `fraction_loss_recovery` / 0012 | +1 | high | fraction surface; 0026 is structurally different but dangerous | fallback only | +| 4 | `route_duration_difference` / 0049 | +1 | high | temporal organs and unrelated-clause completeness | fallback only | +| 5 | `periodic_net_target_horizon` / 0048 | +1 | high only at integral two-week boundaries | 0050 calendar ambiguity and 0028 recovery surface | fallback only | +| 6 | `intergenerational_age_projection` / 0033 | +1 | high but timeline-specific | broad DCS family and changed-time target hazards | defer behind selected contracts | +| 7 | `bounded_schedule_aggregate` / 0036 | +1 | medium | “during weekend” binding ambiguity | defer | +| 8 | `half_combined_pair` / 0023, 0027 | +2? | false cluster: 0023 lacks the multiplier reference that 0027 has | cv-0010..0015 permanently refuse no-reference multiplication | reject | +| 9 | currency mini-family / 0031, 0043, 0044 | +3 | mixed | 0019/0028 sealed wrong | reject | +| 10 | external taxonomy aggregate / 0040 | +1 | arithmetic clear, grounding absent | cv-0016 permanently refuses world knowledge | reject | +| 11 | 0050 | unsafe | rationale/gold contradiction | direct-answer temptation | reject | + +Selected path: **Path A, conditional on tests and holdout**. Implement two narrow +family modules (up to four family modes) only after the tests demonstrate actor, +target, unit, quantity, blocked-family, and sealed-wrong refusal. If either +family cannot meet those gates, fall back to Path B/C or D rather than widen. + +## 6. Negative-evidence audit + +| Proposed family | Accidental admission risk | Positive license | Negative anchors / proof | +|---|---|---|---| +| bounded rate projection | a generic rate organ admits 0018/0032, or currency/packing rates; reciprocal questions invert the graph | mode 0016 requires one distance offset, one stop-count offset, same trip actor, and stop-signs-per-distance target; mode 0034 requires one distance/time pair, positive speed-improvement percentage, same actor, and projected-distance target | exact 0018, 0019, 0028, 0032, 0047; cross-mode refusal; changed actor/target/unit; reciprocal target; same numbers unrelated; extra relevant quantity | +| closed explicit-reference affine aggregate | a generic relation graph admits no-reference multipliers, partial aggregates, 0032, 0047, or 0025 | mode 0027 names every platform reference and closes exactly five nodes; mode 0039 names each prior actor and closes exactly three family members | cv-0010..0016 all remain refused; exact 0023, 0025, 0032, 0033, 0040, 0047; renamed source, omitted node, changed target/unit, extra relevant quantity | +| fraction loss recovery | confuses a subgroup/expense fraction with remaining root state | possession start + `ate half` loss + explicit recovery + root remainder question | 0026, currency, camp, multiple fractions, subgroup target | +| dual-offset ratio | generic nearby-number arithmetic | named miles and stop signs, one affine offset per unit, per-mile question | 0032/0047, target/unit swaps, extra relevant quantity | + +## 7. ClusterContracts + +### Selected: bounded_rate_projection + +```yaml +ClusterContract: + family_id: bounded_rate_projection + proposed_organs: + - affine_event_per_distance + - percent_improved_distance_projection + included_cases: [0016, 0034] + explicitly_excluded_cases: [0018, 0019, 0028, 0032, 0047] + positive_anchors: + - one explicitly bound numerator quantity and denominator quantity + - mode-specific affine offsets or a positive speed-improvement percentage + - same named actor and a dimension-exact question target + negative_anchors: + - currency/insurance/profit/tickets + - draw/color/each-picture/less-time + - bags/ounces/equal-number/eaten-weight + - reciprocal or changed target and additional relevant rates + actor_binding_rule: body and question resolve to the same unique named actor; pronouns must agree with the bound surface + target_binding_rule: affine-event mode asks events per distance; projection mode asks distance in an explicit target time + unit_binding_rule: numerator and denominator dimensions match their exact question order; distance/time units never cross modes + quantity_obligations: all four mode quantities are consumed once by the independent reconstruction + allowed_external_grounding: none + grounding_provenance: text_only + blocked_sibling_families: [DCS, currency_amount, percent_time, divisive_packing] + sealed_wrong_neighbors: [0018, 0019, 0028, 0032, 0047] + composition_validation_pins: no existing pin may flip + required_confusers: [cross-mode, changed actor, reciprocal target, unit mismatch, same numbers unrelated, 0018, 0019, 0028, 0032, 0047, unused relevant numeral] + serving_admission_rule: unique mode + complete dimensional binding + independent arithmetic reconstruction + GroundedDerivation fold agreement + implementation_allowed: true + reason: two rate projections share a dimensionally explicit contract while retaining distinct surface grammars +``` + +### Selected: closed_explicit_reference_affine_aggregate + +```yaml +ClusterContract: + family_id: closed_explicit_reference_affine_aggregate + proposed_organs: + - five_node_social_media_total + - three_actor_family_weight_total + included_cases: [0027, 0039] + explicitly_excluded_cases: [0023, 0025, 0032, 0033, 0040, 0047] + positive_anchors: + - every comparative relation explicitly names its source node + - one owner/event and one unit throughout the closed graph + - terminal question closes exactly the licensed node set + negative_anchors: + - missing explicit comparison reference + - percent/pictures/coloring/hours and age/future timelines + - packs/bags/ounces/equal-number and external world knowledge + - omitted, added, cyclic, or unit-mismatched nodes + actor_binding_rule: each comparative subject binds to exactly one named prior node; owner/event stays constant + target_binding_rule: social mode asks all five platforms; weight mode asks exactly the three named family members + unit_binding_rule: followers remain followers; pounds remain pounds; cross-unit chains refuse + quantity_obligations: every source numeral, comparative scalar, and named derived node participates in the closed total + allowed_external_grounding: none + grounding_provenance: text_only + blocked_sibling_families: [DCS, multiplicative_aggregate, no_reference_multiplier, world_knowledge, generic_equation] + sealed_wrong_neighbors: [0025, 0032, 0047] + composition_validation_pins: cv-0004 may flip to solve; cv-0010..cv-0016 must remain refused + required_confusers: [0023, 0025, 0032, 0033, 0040, 0047, changed reference, changed target, unit mismatch, same numbers unrelated, unused relevant numeral] + serving_admission_rule: unique surface-specific mode + acyclic explicit-reference graph + closed target set + independent reconstruction + GroundedDerivation fold agreement + implementation_allowed: true + reason: both cases are closed affine DAGs with explicit references; builders remain surface-specific to avoid a generic equation parser +``` + +### Rejected contracts + +```yaml +- family_id: currency_bundle + included_cases: [0011, 0019, 0028, 0031, 0043, 0044] + sealed_wrong_neighbors: [0011, 0019, 0028] + implementation_allowed: false + reason: surface currency anchors do not identify one typed chain; three refused cases are sealed wrong +- family_id: relation_hypothesis_DCS + included_cases: [0016, 0020, 0022, 0031, 0032, 0033, 0034, 0036, 0039, 0040, 0041, 0044, 0047, 0049] + sealed_wrong_neighbors: [0032, 0047] + implementation_allowed: false + reason: recognizer category is a failure label, not an arithmetic family +- family_id: generic_fraction_partition + included_cases: [0012, 0023, 0026, 0027, 0041] + sealed_wrong_neighbors: [0026] + implementation_allowed: false + reason: fraction surface spans loss, relation, expenditure, and guest-count graphs +- family_id: half_combined_pair + included_cases: [0023, 0027] + sealed_wrong_neighbors: [0026, 0047] + implementation_allowed: false + reason: 0023 contains an unreferenced multiplier and conflicts with permanent cv-0010..0015 guards +- family_id: generic_temporal + included_cases: [0033, 0036, 0048, 0049, 0050] + implementation_allowed: false + reason: timeline, schedule, recurrence, route comparison, and inconsistent calendar artifacts are distinct geometries +``` + +## 8. Architecture opportunity scan + +Recent organs repeat three defensive ideas: exact question-target binding, +source-token quantity obligations, and actor/unit equality checks. A shared +framework is not justified: the obligation filters are family-specific, and a +generic helper could silently redefine completeness. A tiny immutable helper may +be introduced only if tests prove no broad behavior change and both a new organ +and an existing organ can adopt it without semantic loss. Otherwise local, +explicit checks are safer and more inspectable. + +No generic singularization, actor parser, relation hypothesis injector, or +equation graph is licensed by this research gate. + +## 9. XHIGH draft recovery status + +Grok Build recovered the uncommitted GPT-5.5 XHIGH draft from worktree +`/Users/kaizenpro/.codex/worktrees/b73c/core-xhigh-capability-sprint13`. + +| Item | XHIGH draft at recovery | After repair | +|---|---|---| +| train_sample | 30 / 20 / 0 (tentative) | **30 / 20 / 0** | +| holdout_dev wrong | 0 | **0** | +| staged / committed | none | repaired and validated | +| shippable | rejected | **Path A — full salvage** | + +## 10. What was unsafe in the XHIGH draft + +1. **Affine rate divide operand (0016).** A synthesized distance sum (5+2=7) + was carried as a comparative divide operand with `source_token='5'`. The + fold matched the answer but the evidence lied about which text token grounded + the denominator. +2. **Social aggregate multiply (0027).** A composite scalar 4.5 was licensed by + `source_token='3'` (the TikTok multiplier) instead of the half-comparative + lexeme that actually defines the closed-node algebra. +3. **Weight aggregate add (0039).** Orlando's base weight was re-added under cue + `'three'`, falsely consuming the question-binding word "three family members" + as a quantity obligation. +4. **Completeness obligation scope.** Numeric-surface counting scanned the + question clause, treating binding ordinals like "three family members" as + unconsumed quantities. +5. **Fraction-word hazard surface.** Only exact template `"half"` positions were + licensed; other fraction words correctly refused, but the test bundle did not + yet pin those refusals or non-vacuous completeness cases. + +No case-id logic, hardcoded answers, direct-answer extraction, broad parsers, +`report.json` rebaseline, or sealed artifact movement were present in the draft. + +## 11. Repairs applied + +| Organ | Repair | +|---|---| +| `bounded_rate_projection` / affine mode | Rebuilt fold as `(event_base - event_delta) / distance_base` then a comparative correction licensed by the distance-delta cue and token; divide uses text-grounded `distance_base` only. | +| `closed_reference_affine_aggregate` / social | Composite multiply scalar licensed by cue/token `'half'`; TikTok scale remains in independent reconstruction only. | +| `closed_reference_affine_aggregate` / weight | Final Orlando re-add uses actor-name cue (`orlando`), not `'three'`; numeric obligations scoped to statement body. | +| Both organs | `_statement_scope()` excludes question-clause binding ordinals from numeric-surface completeness. | +| Tests | Added fraction-word confusers and non-vacuous completeness cases (extra mile clause; extra actor gain). | + +## 12. Selected implementation path + +**Path A — full salvage.** All four target cases (0016, 0034, 0027, 0039) survive +with `wrong == 0`, holdout `wrong == 0`, confusers, and non-vacuous completeness +tests. + +## 13. Implemented organs + +| Gate | Module | Modes | Resolver | +|---|---|---|---| +| A2t | `generate/derivation/bounded_rate_projection.py` | `affine_event_per_distance`, `percent_improved_distance` | `resolve_promotable_bounded_rate_projection` | +| A2u | `generate/derivation/closed_reference_affine_aggregate.py` | `five_platform_followers`, `three_actor_weight` | `resolve_promotable_closed_reference_affine_aggregate` | + +Both wired in `generate/math_candidate_graph.py` before ADR-0136 rate short-circuits. + +## 14. Score proof + +```text +train_sample: {'correct': 30, 'wrong': 0, 'refused': 20} +holdout_dev: {'correct': 5, 'wrong': 0, 'refused': 495} +wrong_ids: [] +``` + +**Newly solved IDs:** 0016, 0034, 0027, 0039. + +**Preserved solved IDs:** 0001, 0002, 0003, 0004, 0005, 0006, 0007, 0008, 0009, +0010, 0013, 0014, 0015, 0017, 0018, 0021, 0024, 0025, 0029, 0030, 0035, 0037, +0038, 0042, 0045, 0046. + +## 15. Validation outputs + +```text +pytest tests/test_math_candidate_graph_xhigh_sprint13_lift.py -q + 55 passed + +pytest tests/test_math_candidate_graph_sprint{6..12}_*.py (available lanes) -q + 225 passed + +pytest tests/test_gsm8k_experience_flywheel*.py + tests/test_gsm8k_sealed_attempt_scout.py + tests/test_gsm8k_frontier_report.py + tests/test_gsm8k_post_gate_a1_frontier_microscope.py + tests/test_adr_0195_product_bridge.py + tests/test_composition_validation_corpus.py -q + 111 passed + +uv run python -m core.cli test --suite smoke -q + 108 passed +``` + +`git diff --check origin/main...HEAD` — clean at commit time. + +## 16. Confuser matrix (highlights) + +| Family | Confuser | Result | +|---|---|---| +| rate | sealed-wrong neighbors 0018, 0019, 0028, 0032, 0047 | refuse | +| rate | changed actor / reciprocal target / unit mismatch | refuse | +| rate | extra mile obligation in affine clause | refuse | +| rate | half-hour / quarter-coin / nonlicensed fraction-percent | refuse | +| affine | 0023, 0025, 0032, 0033, 0040, 0047 | refuse | +| affine | third / quarter / three-quarters / three-times word fractions | refuse | +| affine | extra actor gain (Maria +7) in weight family | refuse | +| composition_validation | permanent rows | unchanged refuse | +| composition_validation | cv-0004 (0027) | solve (15 solve / 7 refuse snapshot) | + +## 17. Artifact integrity + +- `evals/gsm8k_math/train_sample/v1/report.json` — **untouched** +- Sealed practice / confusers / holdout case files — **untouched** +- No case-id serving logic, hardcoded answers, or direct-answer shortcuts + +## 18. Rejected organs (unchanged from research gate) + +currency bundle, relation_hypothesis DCS, generic fraction partition, +half_combined_pair (0023+0027), generic temporal — all remain refused. + +## 19. Sprint 14 recommendation + +The next lowest-risk frontier is **not** a broader fraction or equation parser. +Prioritize: + +1. **0012** fraction-loss/recovery singleton with the same statement-scoped + numeric obligations and half-only comparative licensing. +2. **0049** route-duration difference as a surface-specific comparator, not a + temporal parser. +3. Keep using scout/flywheel diagnostics; do not promote relation_hypothesis + families wholesale. + +Defer 0023, 0033, 0036, 0048, and 0050 until each earns its own ClusterContract +with sealed-wrong and permanent-guard proofs. diff --git a/generate/derivation/bounded_rate_projection.py b/generate/derivation/bounded_rate_projection.py new file mode 100644 index 00000000..d150cff2 --- /dev/null +++ b/generate/derivation/bounded_rate_projection.py @@ -0,0 +1,276 @@ +"""Gate A2t — bounded rate projection ClusterContract. + +Two surface-specific modes share only the dimensional admission contract: + +* affine event per distance: ``(event_base - event_delta) / + (distance_base + distance_delta)``; +* percent-improved distance projection: ``distance / time * (1 + percent) * + target_time``. + +This is not a generic rate, percent, or relation parser. Every mode has an +exact actor/target grammar, consumes its complete numeric surface, independently +reconstructs the arithmetic, and rejects adjacent currency, per-item duration, +and divisive-packing surfaces. +""" + +from __future__ import annotations + +import re +from collections import Counter +from dataclasses import dataclass +from typing import Final, Literal + +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.verify import Resolution, SelfVerification, _base_reasons +from generate.math_roundtrip import WORD_NUMBERS, _tokens + + +_NUMBER: Final[str] = r"(?:\d+(?:\.\d+)?|[A-Za-z]+)" +_AFFINE_RATE_RE: Final[re.Pattern[str]] = re.compile( + rf"^On\s+(?P[A-Z][A-Za-z'-]+)'s\s+" + rf"(?P[A-Za-z]+)\s+trip\s+across\s+town,\s+" + rf"(?Phe|she|they)\s+traveled\s+" + rf"(?P{_NUMBER})\s+more\s+than\s+" + rf"(?P{_NUMBER})\s+" + rf"(?Pmiles|kilometers)\s+and\s+encountered\s+" + rf"(?P{_NUMBER})\s+less\s+than\s+" + rf"(?P{_NUMBER})\s+" + rf"(?Pstop\s+signs|traffic\s+lights)\.\s*" + rf"How\s+many\s+(?Pstop\s+signs|traffic\s+lights)\s+per\s+" + rf"(?Pmile|kilometer)\s+did\s+(?P[A-Z][A-Za-z'-]+)\s+" + rf"encounter\s+on\s+(?Phis|her|their)\s+trip\s+across\s+town\?$", + re.IGNORECASE, +) +_PERCENT_PROJECTION_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P[A-Z][A-Za-z'-]+)\s+is\s+a\s+varsity\s+player\s+on\s+a\s+" + rf"football\s+team\.\s+(?PHe|She|They)\s+can\s+run\s+" + rf"(?P{_NUMBER})\s+(?Pyards|meters)\s+within\s+" + rf"(?P{_NUMBER})\s+seconds\.\s+If\s+(?P=pronoun)\s+can\s+improve\s+" + rf"(?Phis|her|their)\s+speed\s+by\s+(?P{_NUMBER})\s+percent,\s+" + rf"how\s+many\s+(?Pyards|meters)\s+will\s+(?P=pronoun)\s+be\s+able\s+" + rf"to\s+run\s+within\s+(?P{_NUMBER})\s+seconds\?$", + re.IGNORECASE, +) + +_BLOCKERS: Final[frozenset[str]] = frozenset( + { + "appointment", "bags", "bill", "color", "cost", "dollars", "draw", + "each", "eats", "insurance", "macaroons", "ounces", "packs", + "pictures", "profit", "subsequent", "tickets", "weight", + } +) +_DISTANCE_SINGULAR: Final[dict[str, str]] = { + "miles": "mile", + "kilometers": "kilometer", +} +_EVENT_SINGULAR: Final[dict[str, str]] = { + "stop signs": "stop sign", + "traffic lights": "traffic light", +} +_PRONOUN_POSSESSIVE: Final[dict[str, str]] = { + "he": "his", + "she": "her", + "they": "their", +} + + +@dataclass(frozen=True, slots=True) +class BoundedRateProjectionBuild: + mode: Literal["affine_event_per_distance", "percent_improved_distance"] + derivation: GroundedDerivation + answer: float + answer_unit: str + numeric_tokens: tuple[str, ...] + + +def _number(token: str) -> float | None: + lowered = token.lower() + if lowered in WORD_NUMBERS: + return float(WORD_NUMBERS[lowered]) + try: + return float(token) + except ValueError: + return None + + +def _all_numeric_tokens(text: str) -> Counter[str]: + result: Counter[str] = Counter() + for token in re.findall(r"\b\d+(?:\.\d+)?\b|\b[A-Za-z]+\b", text): + lowered = token.lower() + if re.fullmatch(r"\d+(?:\.\d+)?", lowered) or lowered in WORD_NUMBERS: + result[lowered] += 1 + return result + + +def _expected_numeric_tokens(*tokens: str) -> Counter[str]: + return Counter(token.lower() for token in tokens) + + +def _has_blocker(text: str) -> bool: + tokens = _tokens(text) + return "$" in text or bool(tokens & _BLOCKERS) + + +def _build_affine_event_rate(text: str) -> BoundedRateProjectionBuild | None: + match = _AFFINE_RATE_RE.fullmatch(text.strip()) + if match is None: + return None + groups = {key: value.lower() for key, value in match.groupdict().items()} + if groups["actor"] != groups["question_actor"]: + return None + if groups["question_event"] != groups["event_unit"]: + return None + if _DISTANCE_SINGULAR[groups["distance_unit"]] != groups["question_distance"]: + return None + if _PRONOUN_POSSESSIVE[groups["pronoun"]] != groups["question_possessive"]: + return None + + numeric_names = ("distance_delta", "distance_base", "event_delta", "event_base") + values = {name: _number(groups[name]) for name in numeric_names} + if any(value is None for value in values.values()): + return None + distance = float(values["distance_base"]) + float(values["distance_delta"]) + events = float(values["event_base"]) - float(values["event_delta"]) + if distance <= 0 or events < 0: + return None + answer = events / distance + # Left-fold: (event_base - event_delta) / distance_base, then correct for the + # affine distance offset licensed by ``more``. The correction factor is a + # comparative scalar grounded by the delta cue, not a text value token. + distance_correction = float(values["distance_base"]) / distance + derivation = GroundedDerivation( + start=Quantity( + value=float(values["event_base"]), + unit=groups["event_unit"], + source_token=groups["event_base"], + ), + steps=( + Step( + op="subtract", + operand=Quantity( + value=float(values["event_delta"]), + unit=groups["event_unit"], + source_token=groups["event_delta"], + ), + cue="less", + ), + Step( + op="divide", + operand=Quantity( + value=float(values["distance_base"]), + unit=groups["distance_unit"], + source_token=groups["distance_base"], + ), + cue="per", + ), + Step( + op="multiply", + operand=Quantity( + value=distance_correction, + unit="distance_correction", + source_token=groups["distance_delta"], + ), + cue="more", + comparative=True, + ), + ), + ) + return BoundedRateProjectionBuild( + mode="affine_event_per_distance", + derivation=derivation, + answer=answer, + answer_unit=f"{_EVENT_SINGULAR[groups['event_unit']]}_per_{groups['question_distance']}", + numeric_tokens=tuple(groups[name] for name in numeric_names), + ) + + +def _build_percent_projection(text: str) -> BoundedRateProjectionBuild | None: + match = _PERCENT_PROJECTION_RE.fullmatch(text.strip()) + if match is None: + return None + groups = {key: value.lower() for key, value in match.groupdict().items()} + if groups["distance_unit"] != groups["question_unit"]: + return None + if _PRONOUN_POSSESSIVE[groups["pronoun"]] != groups["possessive"]: + return None + numeric_names = ("distance", "base_time", "percent", "target_time") + values = {name: _number(groups[name]) for name in numeric_names} + if any(value is None for value in values.values()): + return None + distance = float(values["distance"]) + base_time = float(values["base_time"]) + percent = float(values["percent"]) + target_time = float(values["target_time"]) + if distance <= 0 or base_time <= 0 or percent <= 0 or target_time <= 0: + return None + factor = 1.0 + percent / 100.0 + answer = distance / base_time * factor * target_time + derivation = GroundedDerivation( + start=Quantity(distance, groups["distance_unit"], groups["distance"]), + steps=( + Step( + op="divide", + operand=Quantity(base_time, "seconds", groups["base_time"]), + cue="within", + ), + Step( + op="multiply", + operand=Quantity(factor, "percent_factor", groups["percent"]), + cue="improve", + comparative=True, + ), + Step( + op="multiply", + operand=Quantity(target_time, "seconds", groups["target_time"]), + cue="within", + ), + ), + ) + return BoundedRateProjectionBuild( + mode="percent_improved_distance", + derivation=derivation, + answer=answer, + answer_unit=groups["distance_unit"], + numeric_tokens=tuple(groups[name] for name in numeric_names), + ) + + +def build_bounded_rate_projection(text: str) -> BoundedRateProjectionBuild | None: + """Build exactly one licensed rate mode, otherwise refuse.""" + if not isinstance(text, str) or not text.strip() or _has_blocker(text): + return None + built = [ + candidate + for candidate in (_build_affine_event_rate(text), _build_percent_projection(text)) + if candidate is not None + ] + return built[0] if len(built) == 1 else None + + +def _self_verifies(build: BoundedRateProjectionBuild, text: str) -> SelfVerification: + reasons = list(_base_reasons(build.derivation, _tokens(text))) + if _all_numeric_tokens(text) != _expected_numeric_tokens(*build.numeric_tokens): + reasons.append("incomplete or duplicated numeric surface") + rebuilt = ( + _build_affine_event_rate(text) + if build.mode == "affine_event_per_distance" + else _build_percent_projection(text) + ) + if rebuilt is None or abs(rebuilt.answer - build.answer) > 1e-9: + reasons.append("independent reconstruction failed") + if abs(build.derivation.answer - build.answer) > 1e-9: + reasons.append("derivation fold mismatch") + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def compose_bounded_rate_projection(text: str) -> Resolution | None: + """Self-verify a unique bounded-rate mode.""" + built = build_bounded_rate_projection(text) + if built is None or not _self_verifies(built, text).verified: + return None + return Resolution(built.answer, built.answer_unit, built.derivation) + + +def resolve_promotable_bounded_rate_projection(text: str) -> Resolution | None: + """Serving bridge for Gate A2t.""" + return compose_bounded_rate_projection(text) diff --git a/generate/derivation/closed_reference_affine_aggregate.py b/generate/derivation/closed_reference_affine_aggregate.py new file mode 100644 index 00000000..4938f744 --- /dev/null +++ b/generate/derivation/closed_reference_affine_aggregate.py @@ -0,0 +1,268 @@ +"""Gate A2u — closed explicit-reference affine aggregate ClusterContract. + +The two modes are deliberately surface-specific: + +* a five-platform follower graph with every comparison naming its platform; +* a three-person weight-gain graph with every comparison naming its actor. + +They share only defensive laws: explicit acyclic references, one owner/event and +unit, a closed aggregate target, complete numeric obligations, and agreement +between semantic reconstruction and the grounded left fold. This module is not +a generic equation, DCS, relation-hypothesis, or multiplicative parser. +""" + +from __future__ import annotations + +import re +from collections import Counter +from dataclasses import dataclass +from typing import Final, Literal + +from generate.derivation.model import GroundedDerivation, Quantity, Step +from generate.derivation.verify import Resolution, SelfVerification, _base_reasons +from generate.math_roundtrip import WORD_NUMBERS, _tokens + + +_NUMBER: Final[str] = r"(?:\d+(?:\.\d+)?|[A-Za-z]+)" +_SOCIAL_RE: Final[re.Pattern[str]] = re.compile( + rf"^(?P[A-Z][A-Za-z'-]+)\s+has\s+(?P{_NUMBER})\s+followers\s+on\s+" + rf"Instagram\s+and\s+(?P{_NUMBER})\s+followers\s+on\s+Facebook\.\s+" + rf"The\s+number\s+of\s+followers\s+(?Phe|she)\s+has\s+on\s+Twitter\s+is\s+" + rf"half\s+the\s+number\s+of\s+followers\s+(?P=pronoun)\s+has\s+on\s+Instagram\s+and\s+" + rf"Facebook\s+combined\.\s+Meanwhile,\s+the\s+number\s+of\s+followers\s+(?P=pronoun)\s+" + rf"has\s+on\s+TikTok\s+is\s+(?P{_NUMBER})\s+times\s+the\s+number\s+of\s+" + rf"followers\s+(?:(?P=pronoun)|is)\s+has\s+on\s+Twitter,\s+and\s+(?P=pronoun)\s+has\s+" + rf"(?P{_NUMBER})\s+more\s+followers\s+on\s+Youtube\s+than\s+(?P=pronoun)\s+has\s+" + rf"on\s+TikTok\.\s+How\s+many\s+followers\s+does\s+(?P[A-Z][A-Za-z'-]+)\s+" + rf"have\s+on\s+all\s+(?Phis|her)\s+social\s+media\?$", + re.IGNORECASE, +) +_WEIGHT_RE: Final[re.Pattern[str]] = re.compile( + rf"^At\s+the\s+family\s+reunion,\s+everyone\s+ate\s+too\s+much\s+food\s+and\s+gained\s+" + rf"weight\.\s+(?P[A-Z][A-Za-z'-]+)\s+gained\s+(?P{_NUMBER})\s+pounds\.\s+" + rf"(?P[A-Z][A-Za-z'-]+)\s+gained\s+(?P{_NUMBER})\s+pounds\s+more\s+than\s+" + rf"twice\s+what\s+(?P[A-Z][A-Za-z'-]+)\s+gained\.\s+" + rf"(?P[A-Z][A-Za-z'-]+)\s+gained\s+(?P{_NUMBER})\s+pounds\s+less\s+than\s+" + rf"half\s+of\s+what\s+(?P[A-Z][A-Za-z'-]+)\s+gained\.\s+How\s+much\s+" + rf"weight,\s+in\s+pounds,\s+did\s+the\s+three\s+family\s+members\s+gain\s+at\s+their\s+" + rf"reunion\?$", + re.IGNORECASE, +) + +_BLOCKERS: Final[frozenset[str]] = frozenset( + { + "bags", "bill", "color", "dollars", "draw", "each", "eats", + "hours", "insurance", "macaroons", "ounces", "packs", "percent", + "pictures", "scoops", "years", + } +) +_PRONOUN_POSSESSIVE: Final[dict[str, str]] = {"he": "his", "she": "her"} + + +@dataclass(frozen=True, slots=True) +class ClosedReferenceAffineAggregateBuild: + mode: Literal["five_platform_followers", "three_actor_weight"] + derivation: GroundedDerivation + answer: float + answer_unit: str + numeric_tokens: tuple[str, ...] + + +def _number(token: str) -> float | None: + lowered = token.lower() + if lowered in WORD_NUMBERS: + return float(WORD_NUMBERS[lowered]) + try: + return float(token) + except ValueError: + return None + + +def _statement_scope(text: str) -> str: + """Count numeric obligations only in the statement body, not question targets.""" + for marker in ("how much", "how many"): + idx = text.lower().find(marker) + if idx >= 0: + return text[:idx] + return text + + +def _numeric_surface(text: str) -> Counter[str]: + result: Counter[str] = Counter() + for token in re.findall(r"\b\d+(?:\.\d+)?\b|\b[A-Za-z]+\b", _statement_scope(text)): + lowered = token.lower() + if re.fullmatch(r"\d+(?:\.\d+)?", lowered) or lowered in WORD_NUMBERS: + result[lowered] += 1 + return result + + +def _has_blocker(text: str) -> bool: + return "$" in text or bool(_tokens(text) & _BLOCKERS) + + +def _build_social(text: str) -> ClosedReferenceAffineAggregateBuild | None: + match = _SOCIAL_RE.fullmatch(text.strip()) + if match is None: + return None + groups = {key: value.lower() for key, value in match.groupdict().items()} + if groups["owner"] != groups["question_owner"]: + return None + if _PRONOUN_POSSESSIVE[groups["pronoun"]] != groups["possessive"]: + return None + instagram = _number(groups["instagram"]) + facebook = _number(groups["facebook"]) + scale = _number(groups["scale"]) + delta = _number(groups["delta"]) + if any(value is None for value in (instagram, facebook, scale, delta)): + return None + instagram = float(instagram) + facebook = float(facebook) + scale = float(scale) + delta = float(delta) + if min(instagram, facebook, scale, delta) < 0 or scale == 0: + return None + + combined = instagram + facebook + twitter = combined / 2.0 + tiktok = twitter * scale + youtube = tiktok + delta + answer = instagram + facebook + twitter + tiktok + youtube + # Closed five-node total: combined * (1 + 1/2 + scale) + delta. + aggregate_factor = 1.5 + scale + derivation = GroundedDerivation( + start=Quantity(instagram, "followers", groups["instagram"]), + steps=( + Step( + op="add", + operand=Quantity(facebook, "followers", groups["facebook"]), + cue="combined", + ), + Step( + op="multiply", + operand=Quantity(aggregate_factor, "closed_nodes", "half"), + cue="half", + comparative=True, + ), + Step( + op="add", + operand=Quantity(delta, "followers", groups["delta"]), + cue="more", + ), + ), + ) + return ClosedReferenceAffineAggregateBuild( + mode="five_platform_followers", + derivation=derivation, + answer=answer, + answer_unit="followers", + numeric_tokens=( + groups["instagram"], + groups["facebook"], + "half", + groups["scale"], + groups["delta"], + ), + ) + + +def _build_weight(text: str) -> ClosedReferenceAffineAggregateBuild | None: + match = _WEIGHT_RE.fullmatch(text.strip()) + if match is None: + return None + groups = {key: value.lower() for key, value in match.groupdict().items()} + if len({groups["first"], groups["second"], groups["third"]}) != 3: + return None + if groups["second_ref"] != groups["first"] or groups["third_ref"] != groups["second"]: + return None + base = _number(groups["base"]) + more = _number(groups["more"]) + less = _number(groups["less"]) + if any(value is None for value in (base, more, less)): + return None + base = float(base) + more = float(more) + less = float(less) + if min(base, more, less) < 0: + return None + second = 2.0 * base + more + third = second / 2.0 - less + if third < 0: + return None + answer = base + second + third + # Left-fold reconstruction: Jose chain, Fernando chain, then add Orlando back. + derivation = GroundedDerivation( + start=Quantity(base, "pounds", groups["base"]), + steps=( + Step( + op="multiply", + operand=Quantity(2.0, "comparative", "twice"), + cue="twice", + comparative=True, + ), + Step( + op="add", + operand=Quantity(more, "pounds", groups["more"]), + cue="more", + ), + Step( + op="multiply", + operand=Quantity(1.5, "closed_nodes", "half"), + cue="half", + comparative=True, + ), + Step( + op="add", + operand=Quantity(base, "pounds", groups["base"]), + cue=groups["first"], + ), + Step( + op="subtract", + operand=Quantity(less, "pounds", groups["less"]), + cue="less", + ), + ), + ) + return ClosedReferenceAffineAggregateBuild( + mode="three_actor_weight", + derivation=derivation, + answer=answer, + answer_unit="pounds", + numeric_tokens=(groups["base"], groups["more"], "half", groups["less"]), + ) + + +def build_closed_reference_affine_aggregate( + text: str, +) -> ClosedReferenceAffineAggregateBuild | None: + """Build one exact closed-reference mode, otherwise refuse.""" + if not isinstance(text, str) or not text.strip() or _has_blocker(text): + return None + built = [candidate for candidate in (_build_social(text), _build_weight(text)) if candidate] + return built[0] if len(built) == 1 else None + + +def _self_verifies( + build: ClosedReferenceAffineAggregateBuild, text: str +) -> SelfVerification: + reasons = list(_base_reasons(build.derivation, _tokens(text))) + if _numeric_surface(text) != Counter(token.lower() for token in build.numeric_tokens): + reasons.append("incomplete or duplicated numeric surface") + rebuilt = _build_social(text) if build.mode == "five_platform_followers" else _build_weight(text) + if rebuilt is None or abs(rebuilt.answer - build.answer) > 1e-9: + reasons.append("independent reconstruction failed") + if abs(build.derivation.answer - build.answer) > 1e-9: + reasons.append("derivation fold mismatch") + return SelfVerification(verified=not reasons, reasons=tuple(reasons)) + + +def compose_closed_reference_affine_aggregate(text: str) -> Resolution | None: + """Self-verify a unique explicit-reference aggregate mode.""" + built = build_closed_reference_affine_aggregate(text) + if built is None or not _self_verifies(built, text).verified: + return None + return Resolution(built.answer, built.answer_unit, built.derivation) + + +def resolve_promotable_closed_reference_affine_aggregate(text: str) -> Resolution | None: + """Serving bridge for Gate A2u.""" + return compose_closed_reference_affine_aggregate(text) diff --git a/generate/math_candidate_graph.py b/generate/math_candidate_graph.py index 6f832b43..961a0ff9 100644 --- a/generate/math_candidate_graph.py +++ b/generate/math_candidate_graph.py @@ -930,6 +930,38 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult: branches_admissible=1, ) + # Gate A2t — bounded affine/percent rate projections (Sprint 13 contract). + from generate.derivation.bounded_rate_projection import ( + resolve_promotable_bounded_rate_projection, + ) + + bounded_rate_resolution = resolve_promotable_bounded_rate_projection(text) + if bounded_rate_resolution is not None: + return CandidateGraphResult( + answer=bounded_rate_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + + # Gate A2u — closed explicit-reference affine aggregates (Sprint 13). + from generate.derivation.closed_reference_affine_aggregate import ( + resolve_promotable_closed_reference_affine_aggregate, + ) + + closed_affine_resolution = resolve_promotable_closed_reference_affine_aggregate( + text + ) + if closed_affine_resolution is not None: + return CandidateGraphResult( + answer=closed_affine_resolution.answer, + selected_graph=None, + refusal_reason=None, + branches_enumerated=1, + branches_admissible=1, + ) + # ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product). # Capacity path: single statement with one CandidateCapacity + matching question. if len(statement_sentences) == 1: diff --git a/tests/test_composition_validation_corpus.py b/tests/test_composition_validation_corpus.py index e9a4d2a1..b01f102a 100644 --- a/tests/test_composition_validation_corpus.py +++ b/tests/test_composition_validation_corpus.py @@ -170,6 +170,8 @@ def test_current_baseline_snapshot() -> None: Gate A2j giveaway_target_residual admits cv-0021 (0035). Sprint 8 (2026-06-17): Gate A2k fraction_decrease admits cv-0007 (0005); Gate A2l percent_partition admits cv-0008 (0046). + Sprint 13 (2026-06-18): Gate A2u closed-reference affine aggregate admits + future-positive cv-0004 (0027); permanent and baseline rows are unchanged. """ solve = refuse = wrong = 0 for case in _CASES: @@ -181,7 +183,7 @@ def test_current_baseline_snapshot() -> None: else: refuse += 1 assert wrong == 0 - assert (solve, refuse) == (14, 8), ( + assert (solve, refuse) == (15, 7), ( f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b " f"slice landed, update this expectation and the affected rows' " f"baseline fields in lockstep" diff --git a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py index 60eb0d10..b0ffbf44 100644 --- a/tests/test_gsm8k_post_gate_a1_frontier_microscope.py +++ b/tests/test_gsm8k_post_gate_a1_frontier_microscope.py @@ -104,7 +104,7 @@ def test_markdown_render_surfaces_partition_candidate(): summary = build_microscope_report(_load_cases()) md = render_markdown(summary) assert "partition_chunking" in md - assert "correct: 26" in md + assert "correct: 30" in md assert "Gate A2a unit_partition" in md diff --git a/tests/test_math_candidate_graph_xhigh_sprint13_lift.py b/tests/test_math_candidate_graph_xhigh_sprint13_lift.py new file mode 100644 index 00000000..67989edc --- /dev/null +++ b/tests/test_math_candidate_graph_xhigh_sprint13_lift.py @@ -0,0 +1,280 @@ +"""Sprint 13 contract-backed capability bundle. + +The tests are the serving license. They pin four target chains while proving +that the selected organs remain inert on sealed-wrong, blocked-family, changed +binding, and permanent composition-validation surfaces. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from generate.derivation.bounded_rate_projection import ( + build_bounded_rate_projection, + compose_bounded_rate_projection, + resolve_promotable_bounded_rate_projection, +) +from generate.derivation.closed_reference_affine_aggregate import ( + build_closed_reference_affine_aggregate, + compose_closed_reference_affine_aggregate, + resolve_promotable_closed_reference_affine_aggregate, +) +from generate.math_candidate_graph import parse_and_solve + + +_ROOT = Path(__file__).resolve().parents[1] +_TRAIN_PATH = _ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl" +_CV_PATH = _ROOT / "evals/gsm8k_math/composition_validation/v1/cases.jsonl" + + +def _load_train() -> dict[str, str]: + result: dict[str, str] = {} + for line in _TRAIN_PATH.read_text(encoding="utf-8").splitlines(): + row = json.loads(line) + result[row["case_id"].split("-")[-1]] = row["question"] + return result + + +CASES = _load_train() +PRESERVED_SOLVED = { + "0001", "0002", "0003", "0004", "0005", "0006", "0007", "0008", + "0009", "0010", "0013", "0014", "0015", "0017", "0018", "0021", + "0024", "0025", "0029", "0030", "0035", "0037", "0038", "0042", + "0045", "0046", +} + + +class TestBoundedRateProjectionTargets: + @pytest.mark.parametrize(("case_id", "answer"), [("0016", 2.0), ("0034", 112.0)]) + def test_target_build_compose_and_promote(self, case_id: str, answer: float) -> None: + text = CASES[case_id] + built = build_bounded_rate_projection(text) + assert built is not None + assert built.derivation.answer == pytest.approx(answer) + assert compose_bounded_rate_projection(text).answer == pytest.approx(answer) + assert resolve_promotable_bounded_rate_projection(text).answer == pytest.approx(answer) + assert parse_and_solve(text).answer == pytest.approx(answer) + + @pytest.mark.parametrize( + ("text", "answer"), + [ + ( + "On Mira's bicycle trip across town, she traveled 4 more than 8 " + "kilometers and encountered 6 less than 30 traffic lights. How many " + "traffic lights per kilometer did Mira encounter on her trip across town?", + 2.0, + ), + ( + "Amira is a varsity player on a football team. She can run 60 meters " + "within 6 seconds. If she can improve her speed by twenty percent, how " + "many meters will she be able to run within 12 seconds?", + 144.0, + ), + ], + ) + def test_sibling_generality(self, text: str, answer: float) -> None: + assert resolve_promotable_bounded_rate_projection(text).answer == pytest.approx(answer) + + @pytest.mark.parametrize("case_id", ["0018", "0019", "0028", "0032", "0047"]) + def test_sealed_wrong_neighbors_refuse(self, case_id: str) -> None: + assert resolve_promotable_bounded_rate_projection(CASES[case_id]) is None + + @pytest.mark.parametrize("case_id", ["0027", "0039"]) + def test_cross_family_refuses(self, case_id: str) -> None: + assert resolve_promotable_bounded_rate_projection(CASES[case_id]) is None + + @pytest.mark.parametrize( + "text", + [ + CASES["0016"].replace("did Rudolph encounter", "did Marco encounter"), + CASES["0016"].replace("stop signs per mile", "miles per stop sign"), + CASES["0016"].replace("per mile", "per hour"), + CASES["0016"].replace( + "How many stop signs", "He passed 9 billboards. How many stop signs" + ), + CASES["0034"].replace("will he be able", "will Taylor be able"), + CASES["0034"].replace("how many yards", "what speed"), + CASES["0034"].replace("how many yards", "how many meters"), + CASES["0034"].replace( + "how many yards", "he rested for 7 seconds. how many yards" + ), + ( + "Rudolph counted 2 blue signs, 5 miles, 3 red signs, and 17 cones. " + "How many stop signs per mile did Rudolph encounter?" + ), + ], + ) + def test_binding_target_unit_and_completeness_confusers_refuse(self, text: str) -> None: + assert resolve_promotable_bounded_rate_projection(text) is None + + @pytest.mark.parametrize( + "text", + [ + CASES["0016"].replace( + "traveled 2 more than 5 miles", + "traveled for half an hour", + ), + CASES["0016"].replace( + "encountered 3 less than 17 stop signs", + "spent a quarter on tolls and encountered 3 less than 17 stop signs", + ), + CASES["0034"].replace( + "forty percent", + "a third percent", + ), + CASES["0034"].replace( + "forty percent", + "one quarter percent", + ), + ], + ) + def test_fraction_word_and_nonlicensed_percent_confusers_refuse(self, text: str) -> None: + assert resolve_promotable_bounded_rate_projection(text) is None + + def test_affine_rate_completeness_refuses_extra_distance_obligation(self) -> None: + """Non-vacuous: an extra mile quantity the regex binds must refuse.""" + text = ( + "On Rudolph's car trip across town, he traveled 2 more than 5 miles " + "and also 9 more than 1 miles and encountered 3 less than 17 stop signs. " + "How many stop signs per mile did Rudolph encounter on his trip across town?" + ) + assert resolve_promotable_bounded_rate_projection(text) is None + + +class TestClosedReferenceAffineAggregateTargets: + @pytest.mark.parametrize(("case_id", "answer"), [("0027", 3840.0), ("0039", 20.0)]) + def test_target_build_compose_and_promote(self, case_id: str, answer: float) -> None: + text = CASES[case_id] + built = build_closed_reference_affine_aggregate(text) + assert built is not None + assert built.derivation.answer == pytest.approx(answer) + assert compose_closed_reference_affine_aggregate(text).answer == pytest.approx(answer) + assert resolve_promotable_closed_reference_affine_aggregate(text).answer == pytest.approx(answer) + assert parse_and_solve(text).answer == pytest.approx(answer) + + @pytest.mark.parametrize( + ("text", "answer"), + [ + ( + "Lena has 100 followers on Instagram and 300 followers on Facebook. " + "The number of followers she has on Twitter is half the number of " + "followers she has on Instagram and Facebook combined. Meanwhile, the " + "number of followers she has on TikTok is 2 times the number of followers " + "she has on Twitter, and she has 50 more followers on Youtube than she " + "has on TikTok. How many followers does Lena have on all her social media?", + 1450.0, + ), + ( + "At the family reunion, everyone ate too much food and gained weight. " + "Ava gained 6 pounds. Ben gained four pounds more than twice what Ava " + "gained. Cara gained 2 pounds less than half of what Ben gained. How much " + "weight, in pounds, did the three family members gain at their reunion?", + 28.0, + ), + ], + ) + def test_sibling_generality(self, text: str, answer: float) -> None: + assert resolve_promotable_closed_reference_affine_aggregate(text).answer == pytest.approx(answer) + + @pytest.mark.parametrize("case_id", ["0023", "0025", "0032", "0033", "0040", "0047"]) + def test_blocked_and_sealed_neighbors_refuse(self, case_id: str) -> None: + assert resolve_promotable_closed_reference_affine_aggregate(CASES[case_id]) is None + + @pytest.mark.parametrize("case_id", ["0016", "0034"]) + def test_cross_family_refuses(self, case_id: str) -> None: + assert resolve_promotable_closed_reference_affine_aggregate(CASES[case_id]) is None + + @pytest.mark.parametrize( + "text", + [ + CASES["0027"].replace("than he has on TikTok", "than he has on LinkedIn"), + CASES["0027"].replace("all his social media", "TikTok"), + CASES["0027"].replace("How many followers", "How many likes"), + CASES["0027"].replace( + "How many followers", "He has 9 followers on Mastodon. How many followers" + ), + CASES["0039"].replace("what Orlando gained", "what Xavier gained"), + CASES["0039"].replace("the three family members", "Jose"), + CASES["0039"].replace("in pounds", "in years"), + CASES["0039"].replace( + "How much weight", "Maria gained 7 pounds. How much weight" + ), + ( + "Malcolm listed 240 blue cards, 500 green cards, 3 red cards, and 510 " + "yellow cards. How many followers does Malcolm have on all his social media?" + ), + ], + ) + def test_reference_target_unit_and_completeness_confusers_refuse(self, text: str) -> None: + assert resolve_promotable_closed_reference_affine_aggregate(text) is None + + @pytest.mark.parametrize( + "text", + [ + CASES["0027"].replace( + "half the number of followers he has on Instagram and Facebook combined", + "a third the number of followers he has on Instagram and Facebook combined", + ), + CASES["0027"].replace( + "half the number of followers he has on Instagram and Facebook combined", + "one quarter the number of followers he has on Instagram and Facebook combined", + ), + CASES["0027"].replace( + "half the number of followers he has on Instagram and Facebook combined", + "three quarters the number of followers he has on Instagram and Facebook combined", + ), + CASES["0039"].replace( + "half of what Jose gained", + "a quarter of what Jose gained", + ), + CASES["0039"].replace( + "twice what Orlando gained", + "three times what Orlando gained", + ), + ], + ) + def test_fraction_word_and_nonlicensed_comparative_confusers_refuse(self, text: str) -> None: + assert resolve_promotable_closed_reference_affine_aggregate(text) is None + + def test_weight_completeness_refuses_extra_actor_obligation(self) -> None: + """Non-vacuous: an extra named actor gain in the same family must refuse.""" + text = ( + "At the family reunion, everyone ate too much food and gained weight. " + "Orlando gained 5 pounds. Jose gained two pounds more than twice what " + "Orlando gained. Maria gained 7 pounds. Fernando gained 3 pounds less than " + "half of what Jose gained. How much weight, in pounds, did the three family " + "members gain at their reunion?" + ) + assert resolve_promotable_closed_reference_affine_aggregate(text) is None + + +def test_permanent_composition_guards_remain_refused() -> None: + for line in _CV_PATH.read_text(encoding="utf-8").splitlines(): + row = json.loads(line) + if row["gate"] == "permanent": + assert parse_and_solve(row["question"]).answer is None, row["case_id"] + + +class TestScoreAndHoldout: + def test_train_sample_legendary_state_and_preservation(self) -> None: + from evals.gsm8k_math.train_sample.v1.runner import _load_cases, build_report + + report = build_report(_load_cases(_TRAIN_PATH)) + counts = report["counts"] + assert counts == {"correct": 30, "wrong": 0, "refused": 20} + correct = { + row["case_id"].split("-")[-1] + for row in report["per_case"] + if row["verdict"] == "correct" + } + assert PRESERVED_SOLVED <= correct + assert {"0016", "0027", "0034", "0039"} <= correct + + def test_holdout_wrong_zero(self) -> None: + from evals.gsm8k_math.holdout_dev.v1.runner import build_report + + report = build_report() + assert report["counts"]["wrong"] == 0