diff --git a/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md b/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md index d4e62854..fa4eb2dc 100644 --- a/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md +++ b/docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md @@ -1,53 +1,83 @@ # CORE Problem-Solving Capability Roadmap v2 — 2026-06-17 -**Status:** Living document (docs-only update) +**Status:** Living document (post-Gate-A1 calibration update) **Date:** 2026-06-17 -**Context:** Post PR #797 (rate injection), #798 (Grok governance), #799 (Inc3 rate connector); Inc3 evidence closure is the current hygiene step before Gate A1 comparative injection. +**Context:** Post PR #797 (rate injection), #798 (Grok governance), #799 (Inc3 rate connector), #801–#805 (Inc3 evidence closure, practice-lane monotonic counts, Gate A1 ratification + implementation). ## Overview -This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate injection delivered by PR #797. +This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate and comparative injector closures. -As of 2026-06-17: PR #797, #798, and #799 are merged. +As of 2026-06-17 on `main` @ `bb083004`: -## GSM8K Workstream A +| PR | Status | Role | +|---|---|---| +| #797 | merged | Inc2 narrow rate injection | +| #798 | merged | Grok governance | +| #799 | merged | Inc3 `"one"` rate connector | +| #801 | merged | Inc3 evidence closure lookback | +| #802 | merged | Practice-lane monotonic counts | +| #803 | merged | Gate A1 multiplicative comparative ratification | +| #804 | merged | Test doctrine: score vs truth preservation | +| #805 | merged | Gate A1 multiplicative comparative injection (`bb083004`) | -- Inc 1: reader/recognizer baseline lift (discrete etc.) -- Inc 2: frontier measurement + stale doctrine repair + narrow rate injection (PR #797) -- Inc 3: `"one"` connector for rate_with_currency (PR #799) — **shipped** -- **Current hygiene step:** Inc3 evidence closure (lookback + frontier test semantics; pinned `report.json` remains historical; live ephemeral frontier shows `rate_with_currency` no-injection = 0) +Live ephemeral train_sample (current code, no `report.json` write): **6 correct / 44 refused / 0 wrong**. Pinned `report.json` remains historical **6/44/0**. -### Recommended Inc3 target (narrow) +## GSM8K Workstream A — closed increments -Make the rate frontier evidence actionable by resolving the next narrow blocker exposed by #797. +- **Inc 1:** reader/recognizer baseline lift (discrete etc.) +- **Inc 2:** frontier measurement + stale doctrine repair + narrow rate injection (PR #797) +- **Inc 3:** `"one"` connector for `rate_with_currency` (PR #799) — **closed** (#801 evidence) +- **Gate A1:** multiplicative entity-comparison comparative injection (PR #805) — **closed** -Scope candidates (in preference order for this increment): -1. Denominator-state support for rate application (if failures surface as "actor has rate but no denom-unit quantity reachable"). -2. Safe connector expansion only if frontier proves "for one cup" is a dominant blocker. -3. Measurement-only frontier report refresh if artifacts stale. +Injector-frontier closures verified live: -Inc3 selected #2 (connector for "for one cup"/"one" token) because live debug on the pinned report + cases showed the exact remaining rate injector deferral from Inc2 (matcher left rate_anchor_token=None for "one"; spec unresolved_notes explicitly called it out for the Alexa surface). This was the minimal change that reclassifies the rate_with_currency no-injection bucket (making evidence actionable) while preserving all guards. Denom production is larger future work (see ratification for rationale and out-of-scope). +- `rate_with_currency` recognized_no_injection = **0** (Inc3) +- `comparative_with_unit` recognized_no_injection = **0** (Gate A1) -"Complete and harden PR #797" is revised as: Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection. +Aggregate proxy unchanged at 6/44/0; refusal reclassification downstream is expected. -Explicitly: Inc3 did not broaden to full rate language family, comparative injection, or non-rate categories. +## Gate naming (do not conflate) -## Gate A1 / Comparative Injection +| Name | Meaning | Status | +|---|---|---| +| **Gate A1** | Multiplicative comparative recognizer injection (`compare_multiplicative` v1 template) | **Closed** (#803 ratification, #805 implementation) | +| **Gate A1b / Comparative-A2** | Additive comparative injection (`compare_additive`, “X more/less than Y”) | Deferred — not Gate A2 | +| **Gate A2 (roadmap)** | Partition / chunking (`unit_partition`, split-into-sections, aggregate-then-divide) | **Ratification candidate pending microscope (Gate A2a)** | +| **Inc4 denom-state** | Rate denominator / hour-kg-cup Initial production | Deferred — not started | -**Next** after Inc3 evidence closure merges. Ratify-first (docs/analysis gate-a1 ratification) before any comparative injector code. +Additive comparative is **not** roadmap Gate A2. Roadmap Gate A2 remains partition/chunking. -## Success Criteria (Inc3 — met on main; evidence closure completes governance) +## Current hygiene step (this PR) -- Inc3 ratified change merged (#799). -- Live ephemeral frontier: `rate_with_currency` no-injection = 0; aggregate proxy 6/44/0, wrong=0. -- Pinned `report.json` may remain historical; live measurement via ephemeral runner or synthetic frontier tests. -- Inc3 lookback committed (docs/analysis/gsm8k-workstream-a-increment-3-lookback-2026-06-17.md). -- No rebaseline of sealed lanes or SHA movement without separate ratification. +**Post-Gate-A1 frontier microscope** — docs/tooling only: -## Out of Scope (Inc3 — held) +- Ephemeral `build_report(cases)` classification of all 44 refusals +- Lookback: `docs/analysis/gsm8k-post-gate-a1-frontier-microscope-2026-06-17.md` +- Tool: `scripts/gsm8k_post_gate_a1_frontier_microscope.py` -- Full comparative (Gate A1) implementation. -- Denominator-state production, report.json rebaseline, broad recognizer work. -- Changes to serving sealed paths, identity, policy, or algebra invariants. +**Next ratification candidate:** Gate A2a unit partition / chunking primitive (microscope recommendation; not an implementation decision in this PR). -See Inc3 lookback for loop-closure criterion before Gate A1. +## Success Criteria (Inc3 + Gate A1 — met) + +- Inc3 ratified change merged (#799); evidence closure #801 +- Gate A1 ratified (#803) and implemented (#805) +- Live ephemeral: `rate_with_currency` and `comparative_with_unit` no-injection = **0**; wrong = **0** +- Monotonic contract: `correct >= 6`, `refused <= 44` +- Pinned `report.json` historical; no rebaseline without separate ratification +- Inc3 lookback: `docs/analysis/gsm8k-workstream-a-increment-3-lookback-2026-06-17.md` +- Gate A1 lookback: `docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md` + +## Out of Scope (held) + +- Gate A2 partition/chunking **implementation** (microscope first) +- Gate A1b / Comparative-A2 additive comparative **implementation** +- Inc4 denominator-state production +- `report.json` rebaseline, broad recognizer work, sealed-lane movement +- Changes to serving sealed paths, identity, policy, or algebra invariants + +## Sequencing after microscope + +1. **Ratify** Gate A2a unit partition / chunking primitive (shared narrow primitive — see lookback evidence). Implementation follows ratification only. +2. Preserve wrong=0 and monotonic counts on every future slice. +3. Gate A1b / Comparative-A2 (additive comparative) and Inc4 denom-state remain deferred until separately ratified. diff --git a/docs/analysis/gsm8k-post-gate-a1-frontier-microscope-2026-06-17.md b/docs/analysis/gsm8k-post-gate-a1-frontier-microscope-2026-06-17.md new file mode 100644 index 00000000..cea5ca18 --- /dev/null +++ b/docs/analysis/gsm8k-post-gate-a1-frontier-microscope-2026-06-17.md @@ -0,0 +1,199 @@ +# GSM8K post-Gate-A1 frontier microscope lookback + +**Date:** 2026-06-17 +**Branch:** `docs/gsm8k-post-gate-a1-frontier-microscope` +**Main merge anchor:** `bb083004` (PR #805 Gate A1 implementation) +**Tool:** `scripts/gsm8k_post_gate_a1_frontier_microscope.py` +**Scope:** docs/tooling only — no runtime, no `report.json` rebaseline, no implementation ratification + +## Purpose + +After Inc3 (#799/#801) and Gate A1 (#803/#805), the injector frontiers for +`rate_with_currency` and `comparative_with_unit` are closed. Aggregate +train_sample proxy remains **6/44/0** with **wrong=0**. This lookback runs a +deterministic live microscope over ephemeral `build_report(cases)` to name the +**exact remaining refusal taxonomy** and nominate the next **ratification +candidate** (not an implementation decision). + +## Gate naming (fixed) + +| Name | Meaning | +|---|---| +| **Gate A2 (roadmap)** | Partition / chunking family | +| **Gate A2a** | Narrow ratification candidate within Gate A2: unit partition / chunking primitive | +| **Gate A1b / Comparative-A2** | Additive comparative (`compare_additive`) — **not** roadmap Gate A2 | + +## Measurement method + +```bash +PYTHONPATH=. .venv/bin/python scripts/gsm8k_post_gate_a1_frontier_microscope.py +``` + +- Reads `evals/gsm8k_math/train_sample/v1/cases.jsonl` (50 cases) +- Builds ephemeral report via `evals.gsm8k_math.train_sample.v1.runner.build_report` +- **Does not write** `evals/gsm8k_math/train_sample/v1/report.json` +- Emits aggregate buckets plus a **44-row `refusal_table`** with per-case blocking layer, + candidate primitive, expected movement, and evidence snippet + +Pinned artifact unchanged: `report.json` stays Inc1-era **6/44/0**. + +## Live counts (ephemeral main @ bb083004) + +| Metric | Value | +|---|---:| +| correct | 6 | +| refused | 44 | +| wrong | 0 | +| `rate_with_currency` no-injection | **0** | +| `comparative_with_unit` no-injection | **0** | +| total `recognized_no_injection` | 31 | + +### Aggregate buckets (summary) + +| Top bucket | Count | +|---|---:| +| `recognized_no_injection` | 31 | +| `no_admissible_statement` | 7 | +| `no_admissible_question` | 5 | +| `no_solvable_branch` | 1 | + +**`recognized_no_injection` by category:** `discrete_count_statement` 19, +`descriptive_setup_no_quantity` 4, `currency_amount` 3, `multiplicative_aggregation` 3, +`temporal_aggregation` 2 + +**DCS subfamilies (19):** `dcs_composition_wall` 14, `dcs_misroute_fraction_change` 2, +`dcs_misroute_unit_partition` 1 (**0002**), `dcs_misroute_comparative_multiplicative` 1 (0033), +`dcs_misroute_comparative_additive` 1 (0016) + +**Expected movement histogram:** `diagnostic_only` 25, `downstream_reclassification` 19, +`correct_lift` 0 (none nominated — composition/multi-step dominates) + +## Case-level refusal table (all 44 refused cases) + +Full per-case `reason` strings are in the JSON `refusal_table` emitted by the microscope. +Truncated reasons shown here for doc readability. + +| case_id | verdict | top bucket | subfamily | recognizer cat | blocking layer | next primitive | movement | evidence | reason (truncated) | +|---|---|---|---|---|---|---|---|---|---| +| 0001 | refused | recognized_no_injection | no_injection_temporal_aggregation | temporal_aggregation | recognizer_injector | temporal_tariff | diagnostic_only | If she works more than 8 hours per shift, she is eligible for overtime, which is paid by your... | candidate_graph: recognizer matched but produced no injection for sta... | +| 0002 | refused | recognized_no_injection | dcs_misroute_unit_partition | discrete_count_statement | recognizer_injector | unit_partition | downstream_reclassification | She splits it up into 25-foot sections. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0003 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | The local bookstore donated 48 boxes of erasers. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0004 | refused | no_admissible_statement | nested_fraction_partition | — | statement_parser | fraction_partition | diagnostic_only | Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going... | candidate_graph: no admissible candidate for statement: 'Half of the ... | +| 0005 | refused | no_admissible_statement | affine_equation_fraction_target | — | statement_parser | fractional_delta | diagnostic_only | In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. | candidate_graph: no admissible candidate for statement: "In one hour,... | +| 0006 | refused | recognized_no_injection | no_injection_multiplicative_aggregation | multiplicative_aggregation | recognizer_injector | multiplicative_aggregate | diagnostic_only | Mandy started reading books with only 8 pages when she was 6 years old. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0007 | refused | no_admissible_question | inverse_residual_more | — | question_parser | inverse_residual_question | downstream_reclassification | How many more boxes do they need if Francine has a total of 85 crayons? | candidate_graph: no admissible candidate for question: 'How many more... | +| 0008 | refused | no_admissible_question | production_yield_question | — | question_parser | yield_question_frame | downstream_reclassification | If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out... | candidate_graph: no admissible candidate for question: 'If 50 beads a... | +| 0009 | refused | no_admissible_question | conditional_aggregate_total | — | question_parser | conditional_aggregate_question | downstream_reclassification | If Jen has 150 ducks, how many total birds does she have? | candidate_graph: no admissible candidate for question: 'If Jen has 15... | +| 0010 | refused | no_admissible_statement | affine_equation_fraction_delta | — | statement_parser | affine_equation | diagnostic_only | Marion has 1/4 more than what Yun currently has, plus 7. | candidate_graph: no admissible candidate for statement: 'Marion has 1... | +| 0011 | refused | no_solvable_branch | rate_graph_unsolvable | — | graph_composition | rate_composition | downstream_reclassification | Alexa has a lemonade stand where she sells lemonade for $2 for one cup. | candidate_graph: no branch produced a solvable graph | +| 0012 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | He put all of them in his aquarium but his fish ate half of them. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0013 | refused | recognized_no_injection | no_injection_multiplicative_aggregation | multiplicative_aggregation | recognizer_injector | multiplicative_aggregate | diagnostic_only | Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0015 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train... | candidate_graph: recognizer matched but produced no injection for sta... | +| 0016 | refused | recognized_no_injection | dcs_misroute_comparative_additive | discrete_count_statement | recognizer_injector | compare_additive | downstream_reclassification | On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less tha... | candidate_graph: recognizer matched but produced no injection for sta... | +| 0017 | refused | recognized_no_injection | no_injection_temporal_aggregation | temporal_aggregation | recognizer_injector | temporal_tariff | diagnostic_only | He's charging $50.00 per day or $500.00 for 14 days. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0019 | refused | recognized_no_injection | no_injection_currency_amount | currency_amount | recognizer_injector | currency_mutation | diagnostic_only | After the first appointment, John paid $100 for pet insurance that covers 80% of the subseque... | candidate_graph: recognizer matched but produced no injection for sta... | +| 0020 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Two puppies, two kittens, and three parakeets were for sale at the pet shop. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0021 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He bench presses 15 pounds for 10 reps and does 3 sets. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0022 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | She goes out fishing today and catches twice as many fish as she caught in total in the past... | candidate_graph: recognizer matched but produced no injection for sta... | +| 0023 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | Cindy collected twice as many, and Rex collected half of Nicole and Cindy's combined total. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0025 | refused | no_admissible_question | multiplicative_peer_pick | — | question_parser | peer_partition_question | downstream_reclassification | If three of Lilibeth's friends pick the same amount as her, how many strawberries do Lilibeth... | candidate_graph: no admissible candidate for question: "If three of L... | +| 0026 | refused | no_admissible_statement | each_binding_currency | — | statement_parser | each_entity_binding | downstream_reclassification | Aaron and his brother Carson each saved up $40 to go to dinner. | candidate_graph: no admissible candidate for statement: 'Aaron and hi... | +| 0027 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | The number of followers he has on Twitter is half the number of followers he has on Instagram... | candidate_graph: recognizer matched but produced no injection for sta... | +| 0028 | refused | recognized_no_injection | no_injection_currency_amount | currency_amount | recognizer_injector | currency_mutation | diagnostic_only | It cost $100,000 to open initially. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0030 | refused | no_admissible_statement | numeric_expression_duration | — | statement_parser | duration_multiplier | downstream_reclassification | It is a 2-hour drive each way. | candidate_graph: no admissible candidate for statement: 'It is a 2-ho... | +| 0031 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Jeremie wants to go to an amusement park with 3 friends at the end of summer. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0032 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He draws and colors 10 pictures. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0033 | refused | recognized_no_injection | dcs_misroute_comparative_multiplicative | discrete_count_statement | recognizer_injector | compare_multiplicative | downstream_reclassification | Rachel is 12 years old, and her grandfather is 7 times her age. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0034 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He can run 40 yards within 5 seconds. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0035 | refused | no_admissible_question | inverse_residual_more | — | question_parser | inverse_residual_question | downstream_reclassification | How many more apples would Martha need to give away to be left with only 4 of them? | candidate_graph: no admissible candidate for question: 'How many more... | +| 0036 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | She studied for 2 hours on Wednesday and three times as long on Thursday. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0037 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Michael wants to lose 10 pounds by June. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0039 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Orlando gained 5 pounds. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0040 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0041 | refused | recognized_no_injection | dcs_misroute_fraction_change | discrete_count_statement | recognizer_injector | fraction_of_prior | diagnostic_only | The guests eat all of 1 pan, and 75% of the 2nd pan. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0043 | refused | recognized_no_injection | no_injection_currency_amount | currency_amount | recognizer_injector | currency_mutation | diagnostic_only | Her mother gave her an additional $4, and her father twice as much as her mother. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0044 | refused | recognized_no_injection | dcs_misroute_fraction_change | discrete_count_statement | recognizer_injector | fraction_of_prior | diagnostic_only | John invests in a bank and gets 10% simple interest. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0045 | refused | recognized_no_injection | no_injection_multiplicative_aggregation | multiplicative_aggregation | recognizer_injector | multiplicative_aggregate | diagnostic_only | Each survey has 10 questions. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0046 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | Half of the students are girls, the other half are boys. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0047 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He then packs an equal number of the macaroons in 4 different brown bags, ready for delivery. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0048 | refused | no_admissible_statement | numeric_expression_recurrence | — | statement_parser | recurrence_frame | downstream_reclassification | Every week, he gets 6 cards. | candidate_graph: no admissible candidate for statement: 'Every week, ... | +| 0049 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Malcolm is trying to find the fastest walk to school and is currently comparing two routes. | candidate_graph: recognizer matched but produced no injection for sta... | +| 0050 | refused | no_admissible_statement | numeric_expression_recurrence | — | statement_parser | recurrence_frame | downstream_reclassification | Mark does a gig every other day for 2 weeks. | candidate_graph: no admissible candidate for statement: 'Mark does a ... | + +## Recommended next ratification candidate + +**Gate A2a unit partition / chunking primitive** (within roadmap Gate A2 partition/chunking). + +This is a **ratification candidate**, not an implementation decision. Implementation +follows only after a ratification doc proves a shared narrow primitive with +wrong=0 admission tests. + +### Why Gate A2a (evidence) + +1. **One confirmed DCS unit-partition misroute:** case **0002** — `1000 feet` split into + `25-foot sections`, blocked at `recognizer_injector` with + `dcs_misroute_unit_partition` → candidate primitive `unit_partition`, movement + `downstream_reclassification`. +2. **Broader partition/chunk overlap exists but is not unified:** heuristic tag + `partition_chunking` also touches 0003 (multiplicative product) and 0035 (inverse + residual question); parser-layer partition-adjacent cases include 0004, 0025, 0047. + These do **not** share one primitive yet — ratification must name what is in/out of + Gate A2a v1. +3. **Inc3/Gate A1 precedent:** closed injector buckets first with a ratified v1 template; + aggregate 6/44/0 unchanged; movement is reclassification visibility. + +### Why this is ratification, not implementation + +- Only **one** DCS misroute is confirmed for `unit_partition` on current serving code. +- Partition/chunk **candidates** appear across statement parser, question parser, and + downstream composition buckets — implementation without ratification would smuggle + scope (affine fractions, peer-pick questions, DCS composition wall). +- Ratification must prove a **shared narrow primitive** (total → chunk count → per-chunk + unit) with exemplars, confusers, and wrong=0 replay before any injector/parser code. + +## Not next (explicit deferrals) + +| Track | Defer because | +|---|---| +| **Gate A1b / Comparative-A2 (additive comparative)** | Only one simple same-sentence DCS misroute (0016); no cluster of additive entity-comparison refusals at injector. Defer unless a future microscope pass finds ≥3 same-sentence `more than` / `less than` entity-comparison refusals with shared template. | +| **Inc4 rate denom-state** | Live rate cases refuse on overtime (0001), tariff (0017), profit/composition (0011 graph), not merely missing hour/kg/cup Initial. Downstream rate refusals = 2. Denom-state alone does not explain the frontier. | +| **Broad DCS widening** | 14/19 DCS refusals are `dcs_composition_wall` → `derivation_composer`. Forbidden without a composition-wall ratification; metric-inert injector widening risks incomplete readings under wrong=0. | +| **`report.json` rebaseline** | Separate ratified PR only; pinned artifact remains historical 6/44/0 in this PR. | + +## Gate A2a ratification seed (docs only — not ratified) + +**Candidate primitive names:** `unit_partition` / `partition_count` / `unitize` + +**Template examples (from train_sample surfaces):** + +| Surface pattern | Case hint | Notes | +|---|---|---| +| `N total-units split into M-unit sections` | 0002: 1000 feet → 25-foot sections | Canonical Gate A2a exemplar | +| `N beads per bracelet` (yield/partition question) | 0008: 50 beads → bracelets | Question-layer binding — may be out of v1 | +| `N crayons per box` / residual box count | 0007: boxes vs loose crayons | Inverse residual — likely out of v1 | + +**Open design questions (must be resolved in ratification):** + +1. **Output unit:** chunk count vs retained measure unit vs dual unknowns? +2. **Divisibility:** exact division required (0002: 1000/25) vs refuse non-integer chunk counts? +3. **Fractional results:** 1/4 given away + half stored (0002 full problem) — in v1 scope or composition follow-up? +4. **Source grounding:** total quantity referent vs chunk-size referent — which entity keys are grounded from the statement? +5. **Inverse direction:** “how many more boxes” (0007) — partition primitive or question-frame primitive? +6. **Downstream question binding:** per-chunk query (`how much does she keep on hand`) requires partition+remainder composition — v1 injector only or paired composition ratification? + +## Tests / validation (this PR) + +```bash +git diff --check origin/main...HEAD +PYTHONPATH=. .venv/bin/python -m pytest tests/test_gsm8k_post_gate_a1_frontier_microscope.py -q +PYTHONPATH=. .venv/bin/python -m pytest tests/test_gsm8k_frontier_report.py -q +.venv/bin/core test --suite smoke -q +``` + +## Explicit non-changes + +- No Gate A2a implementation +- No Gate A1b / Comparative-A2 implementation +- No Inc4 denom-state +- No runtime / generate / solver / recognizer edits +- No `report.json` rebaseline \ No newline at end of file diff --git a/docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md b/docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md index 66a61d33..5c9917f9 100644 --- a/docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md +++ b/docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md @@ -1,11 +1,11 @@ # GSM8K Workstream A Gate A1 — multiplicative comparative injection lookback **Date:** 2026-06-17 (post-implementation evidence closure) -**Branch:** `feat/gsm8k-workstream-a-gate-a1-comparative-multiplicative-injection` -**Head (implementation + patch):** `64ab58a7fa01ccfb1c707573b1be044296f5fe38` -**Base implementation commit:** `e578ec72` +**Branch:** `feat/gsm8k-workstream-a-gate-a1-comparative-multiplicative-injection` (merged #805) +**Head (final PR #805):** `c37f97bec36a19bd60a3921c2e1a78b294e00e7b` +**Merge commit on main:** `bb0830046aa4da23ae58eac8fe2368da5f0be6b5` **Governing ratification:** `docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-ratification-2026-06-17.md` (merged #803) -**Scope:** **First subfamily only** — multiplicative entity comparison with explicit same-sentence reference. Additive comparative (Gate A2) deferred. +**Scope:** **First subfamily only** — multiplicative entity comparison with explicit same-sentence reference. Additive comparative deferred to **Gate A1b / Comparative-A2** (roadmap **Gate A2** is partition/chunking). ## What shipped @@ -95,7 +95,7 @@ Live `recognized_no_injection_by_category` (post-Gate A1): ## Explicit non-changes - No `report.json` rebaseline -- No additive comparative (`compare_additive`) — Gate A2 +- No additive comparative (`compare_additive`) — Gate A1b / Comparative-A2 - No `double` / `one-third` / hyphenated N-times factors - No sealed-lane movement - No solver `_apply_compare_multiplicative` semantic change @@ -103,13 +103,13 @@ Live `recognized_no_injection_by_category` (post-Gate A1): ## Known caveats -1. **Not full Gate A1 family** — multiplicative entity-comparison subfamily only; additive deferred to Gate A2. +1. **Not full Gate A1 family** — multiplicative entity-comparison subfamily only; additive deferred to Gate A1b / Comparative-A2. 2. **No guaranteed correct-count lift** — monotonic contract holds (`correct>=6`, `refused<=44`); primary deliverable is injector closure + reclassification visibility. 3. **DCS/parser path for bare `Jerry has 3 times`** — standalone no-reference surface may still interact with parser initials on some question shapes; Gate A1 injector refuses; completeness guard covers multi-clause confabulation class. Not claimed solved in this slice. 4. **Matcher regex vs factor narrowness** — `_COMPARE_MULT_NTIMES_RE` still uses broad `_VALUE` at template level; extraction refuses unsafe factors via `_parse_comparative_v1_count_factor`. 5. **Pinned frontier tests** — historical `report.json` fixture unchanged; live behavior tested ephemerally. -## Deferred work (Gate A1b / A2) +## Deferred work (Gate A1b / Comparative-A2; roadmap Gate A2 = partition/chunking) - `double`, `one-third`, `as much`, unit ellipsis - Additive comparative injection diff --git a/scripts/gsm8k_post_gate_a1_frontier_microscope.py b/scripts/gsm8k_post_gate_a1_frontier_microscope.py new file mode 100644 index 00000000..e6b2e1a6 --- /dev/null +++ b/scripts/gsm8k_post_gate_a1_frontier_microscope.py @@ -0,0 +1,588 @@ +#!/usr/bin/env python3 +"""Post-Gate-A1 live frontier microscope for GSM8K train_sample. + +Builds an ephemeral ``build_report(cases)`` snapshot (never writes +``report.json``) and classifies every refused case into stable buckets +and implementation-slice candidate families. + +Docs/tooling only — no runtime mutation. Rules are deterministic +string predicates; no LLM, no embeddings, no clocks. + +Usage: + uv run python scripts/gsm8k_post_gate_a1_frontier_microscope.py +""" +from __future__ import annotations + +import json +import re +import sys +from collections import defaultdict +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from evals.gsm8k_math.train_sample.v1.runner import build_report +from evals.refusal_taxonomy.shape_categories import ShapeCategory, categorize +from scripts.gsm8k_frontier_report import _extract_category + +_RECOGNIZED_NO_INJ = "candidate_graph: recognizer matched but produced no injection" + +_MICROSCOPE_BUCKET_PATTERNS: tuple[tuple[str, str], ...] = ( + ("wrong", "wrong"), + ("fast-path", "fast_path_correct"), + ("no admissible candidate for question", "no_admissible_question"), + ("no admissible candidate for statement", "no_admissible_statement"), + ("no branch produced a solvable graph", "no_solvable_branch"), + ("no solvable branch", "no_solvable_branch"), + ("incomplete reading", "incomplete_reading"), + (_RECOGNIZED_NO_INJ, "recognized_no_injection"), +) + + +def _classify_reason(reason: str) -> str: + if not reason: + return "other_refused" + lowered = reason.lower() + for needle, bucket in _MICROSCOPE_BUCKET_PATTERNS: + if needle.lower() in lowered: + return bucket + if "refused" in lowered or not reason.strip(): + return "other_refused" + return "other" + +_REPO_ROOT = Path(__file__).resolve().parents[1] +_CASES_PATH = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl" +_EXPECTED_COUNT = 50 + +# --- subfamily needles (stable, sorted for determinism) ------------------- + +_NO_ADM_STMT_SUBFAMILIES: tuple[tuple[str, str], ...] = ( + ("nested_fraction_partition", r"half of .+ (?:going|are)"), + ("affine_equation_fraction_delta", r"1/\d+ more than|more than what .+ has"), + ("affine_equation_fraction_target", r"decrease to \d+/\d+\s+of"), + ("each_binding_currency", r"each saved up \$"), + ("numeric_expression_duration", r"\d+-hour (?:drive|trip|walk)"), + ("numeric_expression_recurrence", r"every (?:other )?(?:day|week)"), + ("partition_statement_parser_gap", r"split|splits|sections|pieces"), +) + +_NO_ADM_QUEST_SUBFAMILIES: tuple[tuple[str, str], ...] = ( + ("inverse_residual_more", r"how many more"), + ("multiplicative_peer_pick", r"friends? pick"), + ("ratio_times_entity", r"times (?:the )?number of"), + ("production_yield_question", r"how many \w+ (?:will|be able to)"), + ("conditional_aggregate_total", r" if .+, how many"), +) + +_NO_SOLVABLE_SUBFAMILIES: tuple[tuple[str, str], ...] = ( + ("rate_graph_unsolvable", r"\$|per (?:day|hour|kg|cup)"), + ("composition_branch_exhausted", r".*"), +) + +_DCS_NO_INJ_SUBFAMILIES: tuple[tuple[str, str], ...] = ( + ("dcs_misroute_unit_partition", r"\d+-(?:foot|feet|inch|yard|mile|meter)"), + ("dcs_misroute_comparative_multiplicative", r"times (?:her|his|their|as many|as much)"), + ("dcs_misroute_comparative_additive", r" more than | less than "), + ("dcs_misroute_rate_surface", r"\$| per | an hour|/hour|/day|/kg"), + ("dcs_misroute_temporal", r"each day|every day|per day|one-hour|in \d+ (?:minute|hour)"), + ("dcs_misroute_fraction_change", r"half of|75%|10%|ate half"), + ("dcs_composition_wall", r".*"), +) + +_SLICE_CANDIDATE_RULES: tuple[tuple[str, str], ...] = ( + ("partition_chunking", r"split|sections|\d+-(?:foot|feet)|partition|per box|in each"), + ("additive_comparative", r" more than | less than |as much as|as many as"), + ("affine_equation", r"1/\d+ more than|decrease to \d+/\d+ of|half of .+ (?:going|are)"), + ("numeric_expression", r"\d+-hour|every (?:other )?(?:day|week)|each way"), + ("rate_follow_up", r"\$| per |overtime|an hour|/day|/kg|for one cup"), + ("multiplicative_comparative_follow_up", r"twice|thrice|\d+ times (?:her|his|their)"), +) + +_BLOCKING_LAYER_BY_BUCKET: dict[str, str] = { + "recognized_no_injection": "recognizer_injector", + "no_admissible_statement": "statement_parser", + "no_admissible_question": "question_parser", + "no_solvable_branch": "graph_composition", + "incomplete_reading": "reading_completeness", + "other_refused": "unknown", + "other": "unknown", +} + +_PRIMITIVE_BY_SUBFAMILY: dict[str, str] = { + "dcs_misroute_unit_partition": "unit_partition", + "dcs_misroute_comparative_additive": "compare_additive", + "dcs_misroute_comparative_multiplicative": "compare_multiplicative", + "dcs_misroute_rate_surface": "rate_composition", + "dcs_misroute_temporal": "temporal_aggregation", + "dcs_misroute_fraction_change": "fraction_of_prior", + "dcs_composition_wall": "derivation_composer", + "no_injection_currency_amount": "currency_mutation", + "no_injection_descriptive_setup_no_quantity": "relation_hypothesis", + "no_injection_multiplicative_aggregation": "multiplicative_aggregate", + "no_injection_temporal_aggregation": "temporal_tariff", + "nested_fraction_partition": "fraction_partition", + "affine_equation_fraction_delta": "affine_equation", + "affine_equation_fraction_target": "fractional_delta", + "each_binding_currency": "each_entity_binding", + "numeric_expression_duration": "duration_multiplier", + "numeric_expression_recurrence": "recurrence_frame", + "partition_statement_parser_gap": "unit_partition", + "unclassified": "diagnostic_hold", + "inverse_residual_more": "inverse_residual_question", + "ratio_times_entity": "ratio_question_frame", + "production_yield_question": "yield_question_frame", + "conditional_aggregate_total": "conditional_aggregate_question", + "multiplicative_peer_pick": "peer_partition_question", + "rate_graph_unsolvable": "rate_composition", +} + +_EVIDENCE_SNIPPET_MAX = 96 + + +def _padded_lower(text: str) -> str: + return " " + text.lower().replace("\n", " ") + " " + + +def _first_subfamily( + text: str, + rules: tuple[tuple[str, str], ...], +) -> str: + hay = _padded_lower(text) + for name, pattern in rules: + if re.search(pattern, hay, flags=re.IGNORECASE): + return name + if rules and rules[-1][1] == r".*": + return rules[-1][0] + return "unclassified" + + +def _extract_quoted_clause(reason: str, label: str) -> str: + """Extract ``for