docs(tooling): add post-Gate-A1 frontier microscope (#806)
* docs(gsm8k): post-Gate-A1 frontier microscope and roadmap calibration Add ephemeral live refusal taxonomy with 44-row case-level table, blocking-layer/primitive/movement classification, and Gate A2a ratification-candidate recommendation (not implementation). Update roadmap v2 and Gate A1 lookback metadata (#805 headc37f97be, mergebb083004). No runtime or report.json changes. * fix(gsm8k): soften microscope live pins and hygiene for PR #806 Remove exact live top-bucket count assertions per #804 doctrine; keep structural invariants and case 0002 Gate A2a seed pin. Fix EOF newlines and duplicate _PRIMITIVE_BY_SUBFAMILY key.
This commit is contained in:
Shay
GitHub
parent
7e1951915a
commit
e134e53cca
6 changed files with 995 additions and 38 deletions
|
|
@ -1,53 +1,83 @@
|
|||
# CORE Problem-Solving Capability Roadmap v2 — 2026-06-17
|
||||
|
||||
**Status:** Living document (docs-only update)
|
||||
**Status:** Living document (post-Gate-A1 calibration update)
|
||||
**Date:** 2026-06-17
|
||||
**Context:** Post PR #797 (rate injection), #798 (Grok governance), #799 (Inc3 rate connector); Inc3 evidence closure is the current hygiene step before Gate A1 comparative injection.
|
||||
**Context:** Post PR #797 (rate injection), #798 (Grok governance), #799 (Inc3 rate connector), #801–#805 (Inc3 evidence closure, practice-lane monotonic counts, Gate A1 ratification + implementation).
|
||||
|
||||
## Overview
|
||||
|
||||
This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate injection delivered by PR #797.
|
||||
This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate and comparative injector closures.
|
||||
|
||||
As of 2026-06-17: PR #797, #798, and #799 are merged.
|
||||
As of 2026-06-17 on `main` @ `bb083004`:
|
||||
|
||||
## GSM8K Workstream A
|
||||
| PR | Status | Role |
|
||||
|---|---|---|
|
||||
| #797 | merged | Inc2 narrow rate injection |
|
||||
| #798 | merged | Grok governance |
|
||||
| #799 | merged | Inc3 `"one"` rate connector |
|
||||
| #801 | merged | Inc3 evidence closure lookback |
|
||||
| #802 | merged | Practice-lane monotonic counts |
|
||||
| #803 | merged | Gate A1 multiplicative comparative ratification |
|
||||
| #804 | merged | Test doctrine: score vs truth preservation |
|
||||
| #805 | merged | Gate A1 multiplicative comparative injection (`bb083004`) |
|
||||
|
||||
- Inc 1: reader/recognizer baseline lift (discrete etc.)
|
||||
- Inc 2: frontier measurement + stale doctrine repair + narrow rate injection (PR #797)
|
||||
- Inc 3: `"one"` connector for rate_with_currency (PR #799) — **shipped**
|
||||
- **Current hygiene step:** Inc3 evidence closure (lookback + frontier test semantics; pinned `report.json` remains historical; live ephemeral frontier shows `rate_with_currency` no-injection = 0)
|
||||
Live ephemeral train_sample (current code, no `report.json` write): **6 correct / 44 refused / 0 wrong**. Pinned `report.json` remains historical **6/44/0**.
|
||||
|
||||
### Recommended Inc3 target (narrow)
|
||||
## GSM8K Workstream A — closed increments
|
||||
|
||||
Make the rate frontier evidence actionable by resolving the next narrow blocker exposed by #797.
|
||||
- **Inc 1:** reader/recognizer baseline lift (discrete etc.)
|
||||
- **Inc 2:** frontier measurement + stale doctrine repair + narrow rate injection (PR #797)
|
||||
- **Inc 3:** `"one"` connector for `rate_with_currency` (PR #799) — **closed** (#801 evidence)
|
||||
- **Gate A1:** multiplicative entity-comparison comparative injection (PR #805) — **closed**
|
||||
|
||||
Scope candidates (in preference order for this increment):
|
||||
1. Denominator-state support for rate application (if failures surface as "actor has rate but no denom-unit quantity reachable").
|
||||
2. Safe connector expansion only if frontier proves "for one cup" is a dominant blocker.
|
||||
3. Measurement-only frontier report refresh if artifacts stale.
|
||||
Injector-frontier closures verified live:
|
||||
|
||||
Inc3 selected #2 (connector for "for one cup"/"one" token) because live debug on the pinned report + cases showed the exact remaining rate injector deferral from Inc2 (matcher left rate_anchor_token=None for "one"; spec unresolved_notes explicitly called it out for the Alexa surface). This was the minimal change that reclassifies the rate_with_currency no-injection bucket (making evidence actionable) while preserving all guards. Denom production is larger future work (see ratification for rationale and out-of-scope).
|
||||
- `rate_with_currency` recognized_no_injection = **0** (Inc3)
|
||||
- `comparative_with_unit` recognized_no_injection = **0** (Gate A1)
|
||||
|
||||
"Complete and harden PR #797" is revised as: Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection.
|
||||
Aggregate proxy unchanged at 6/44/0; refusal reclassification downstream is expected.
|
||||
|
||||
Explicitly: Inc3 did not broaden to full rate language family, comparative injection, or non-rate categories.
|
||||
## Gate naming (do not conflate)
|
||||
|
||||
## Gate A1 / Comparative Injection
|
||||
| Name | Meaning | Status |
|
||||
|---|---|---|
|
||||
| **Gate A1** | Multiplicative comparative recognizer injection (`compare_multiplicative` v1 template) | **Closed** (#803 ratification, #805 implementation) |
|
||||
| **Gate A1b / Comparative-A2** | Additive comparative injection (`compare_additive`, “X more/less than Y”) | Deferred — not Gate A2 |
|
||||
| **Gate A2 (roadmap)** | Partition / chunking (`unit_partition`, split-into-sections, aggregate-then-divide) | **Ratification candidate pending microscope (Gate A2a)** |
|
||||
| **Inc4 denom-state** | Rate denominator / hour-kg-cup Initial production | Deferred — not started |
|
||||
|
||||
**Next** after Inc3 evidence closure merges. Ratify-first (docs/analysis gate-a1 ratification) before any comparative injector code.
|
||||
Additive comparative is **not** roadmap Gate A2. Roadmap Gate A2 remains partition/chunking.
|
||||
|
||||
## Success Criteria (Inc3 — met on main; evidence closure completes governance)
|
||||
## Current hygiene step (this PR)
|
||||
|
||||
- Inc3 ratified change merged (#799).
|
||||
- Live ephemeral frontier: `rate_with_currency` no-injection = 0; aggregate proxy 6/44/0, wrong=0.
|
||||
- Pinned `report.json` may remain historical; live measurement via ephemeral runner or synthetic frontier tests.
|
||||
- Inc3 lookback committed (docs/analysis/gsm8k-workstream-a-increment-3-lookback-2026-06-17.md).
|
||||
- No rebaseline of sealed lanes or SHA movement without separate ratification.
|
||||
**Post-Gate-A1 frontier microscope** — docs/tooling only:
|
||||
|
||||
## Out of Scope (Inc3 — held)
|
||||
- Ephemeral `build_report(cases)` classification of all 44 refusals
|
||||
- Lookback: `docs/analysis/gsm8k-post-gate-a1-frontier-microscope-2026-06-17.md`
|
||||
- Tool: `scripts/gsm8k_post_gate_a1_frontier_microscope.py`
|
||||
|
||||
- Full comparative (Gate A1) implementation.
|
||||
- Denominator-state production, report.json rebaseline, broad recognizer work.
|
||||
- Changes to serving sealed paths, identity, policy, or algebra invariants.
|
||||
**Next ratification candidate:** Gate A2a unit partition / chunking primitive (microscope recommendation; not an implementation decision in this PR).
|
||||
|
||||
See Inc3 lookback for loop-closure criterion before Gate A1.
|
||||
## Success Criteria (Inc3 + Gate A1 — met)
|
||||
|
||||
- Inc3 ratified change merged (#799); evidence closure #801
|
||||
- Gate A1 ratified (#803) and implemented (#805)
|
||||
- Live ephemeral: `rate_with_currency` and `comparative_with_unit` no-injection = **0**; wrong = **0**
|
||||
- Monotonic contract: `correct >= 6`, `refused <= 44`
|
||||
- Pinned `report.json` historical; no rebaseline without separate ratification
|
||||
- Inc3 lookback: `docs/analysis/gsm8k-workstream-a-increment-3-lookback-2026-06-17.md`
|
||||
- Gate A1 lookback: `docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-lookback-2026-06-17.md`
|
||||
|
||||
## Out of Scope (held)
|
||||
|
||||
- Gate A2 partition/chunking **implementation** (microscope first)
|
||||
- Gate A1b / Comparative-A2 additive comparative **implementation**
|
||||
- Inc4 denominator-state production
|
||||
- `report.json` rebaseline, broad recognizer work, sealed-lane movement
|
||||
- Changes to serving sealed paths, identity, policy, or algebra invariants
|
||||
|
||||
## Sequencing after microscope
|
||||
|
||||
1. **Ratify** Gate A2a unit partition / chunking primitive (shared narrow primitive — see lookback evidence). Implementation follows ratification only.
|
||||
2. Preserve wrong=0 and monotonic counts on every future slice.
|
||||
3. Gate A1b / Comparative-A2 (additive comparative) and Inc4 denom-state remain deferred until separately ratified.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,199 @@
|
|||
# GSM8K post-Gate-A1 frontier microscope lookback
|
||||
|
||||
**Date:** 2026-06-17
|
||||
**Branch:** `docs/gsm8k-post-gate-a1-frontier-microscope`
|
||||
**Main merge anchor:** `bb083004` (PR #805 Gate A1 implementation)
|
||||
**Tool:** `scripts/gsm8k_post_gate_a1_frontier_microscope.py`
|
||||
**Scope:** docs/tooling only — no runtime, no `report.json` rebaseline, no implementation ratification
|
||||
|
||||
## Purpose
|
||||
|
||||
After Inc3 (#799/#801) and Gate A1 (#803/#805), the injector frontiers for
|
||||
`rate_with_currency` and `comparative_with_unit` are closed. Aggregate
|
||||
train_sample proxy remains **6/44/0** with **wrong=0**. This lookback runs a
|
||||
deterministic live microscope over ephemeral `build_report(cases)` to name the
|
||||
**exact remaining refusal taxonomy** and nominate the next **ratification
|
||||
candidate** (not an implementation decision).
|
||||
|
||||
## Gate naming (fixed)
|
||||
|
||||
| Name | Meaning |
|
||||
|---|---|
|
||||
| **Gate A2 (roadmap)** | Partition / chunking family |
|
||||
| **Gate A2a** | Narrow ratification candidate within Gate A2: unit partition / chunking primitive |
|
||||
| **Gate A1b / Comparative-A2** | Additive comparative (`compare_additive`) — **not** roadmap Gate A2 |
|
||||
|
||||
## Measurement method
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. .venv/bin/python scripts/gsm8k_post_gate_a1_frontier_microscope.py
|
||||
```
|
||||
|
||||
- Reads `evals/gsm8k_math/train_sample/v1/cases.jsonl` (50 cases)
|
||||
- Builds ephemeral report via `evals.gsm8k_math.train_sample.v1.runner.build_report`
|
||||
- **Does not write** `evals/gsm8k_math/train_sample/v1/report.json`
|
||||
- Emits aggregate buckets plus a **44-row `refusal_table`** with per-case blocking layer,
|
||||
candidate primitive, expected movement, and evidence snippet
|
||||
|
||||
Pinned artifact unchanged: `report.json` stays Inc1-era **6/44/0**.
|
||||
|
||||
## Live counts (ephemeral main @ bb083004)
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| correct | 6 |
|
||||
| refused | 44 |
|
||||
| wrong | 0 |
|
||||
| `rate_with_currency` no-injection | **0** |
|
||||
| `comparative_with_unit` no-injection | **0** |
|
||||
| total `recognized_no_injection` | 31 |
|
||||
|
||||
### Aggregate buckets (summary)
|
||||
|
||||
| Top bucket | Count |
|
||||
|---|---:|
|
||||
| `recognized_no_injection` | 31 |
|
||||
| `no_admissible_statement` | 7 |
|
||||
| `no_admissible_question` | 5 |
|
||||
| `no_solvable_branch` | 1 |
|
||||
|
||||
**`recognized_no_injection` by category:** `discrete_count_statement` 19,
|
||||
`descriptive_setup_no_quantity` 4, `currency_amount` 3, `multiplicative_aggregation` 3,
|
||||
`temporal_aggregation` 2
|
||||
|
||||
**DCS subfamilies (19):** `dcs_composition_wall` 14, `dcs_misroute_fraction_change` 2,
|
||||
`dcs_misroute_unit_partition` 1 (**0002**), `dcs_misroute_comparative_multiplicative` 1 (0033),
|
||||
`dcs_misroute_comparative_additive` 1 (0016)
|
||||
|
||||
**Expected movement histogram:** `diagnostic_only` 25, `downstream_reclassification` 19,
|
||||
`correct_lift` 0 (none nominated — composition/multi-step dominates)
|
||||
|
||||
## Case-level refusal table (all 44 refused cases)
|
||||
|
||||
Full per-case `reason` strings are in the JSON `refusal_table` emitted by the microscope.
|
||||
Truncated reasons shown here for doc readability.
|
||||
|
||||
| case_id | verdict | top bucket | subfamily | recognizer cat | blocking layer | next primitive | movement | evidence | reason (truncated) |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| 0001 | refused | recognized_no_injection | no_injection_temporal_aggregation | temporal_aggregation | recognizer_injector | temporal_tariff | diagnostic_only | If she works more than 8 hours per shift, she is eligible for overtime, which is paid by your... | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0002 | refused | recognized_no_injection | dcs_misroute_unit_partition | discrete_count_statement | recognizer_injector | unit_partition | downstream_reclassification | She splits it up into 25-foot sections. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0003 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | The local bookstore donated 48 boxes of erasers. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0004 | refused | no_admissible_statement | nested_fraction_partition | — | statement_parser | fraction_partition | diagnostic_only | Half of the kids are going to soccer camp, and 1/4 of the kids going to soccer camp are going... | candidate_graph: no admissible candidate for statement: 'Half of the ... |
|
||||
| 0005 | refused | no_admissible_statement | affine_equation_fraction_target | — | statement_parser | fractional_delta | diagnostic_only | In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. | candidate_graph: no admissible candidate for statement: "In one hour,... |
|
||||
| 0006 | refused | recognized_no_injection | no_injection_multiplicative_aggregation | multiplicative_aggregation | recognizer_injector | multiplicative_aggregate | diagnostic_only | Mandy started reading books with only 8 pages when she was 6 years old. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0007 | refused | no_admissible_question | inverse_residual_more | — | question_parser | inverse_residual_question | downstream_reclassification | How many more boxes do they need if Francine has a total of 85 crayons? | candidate_graph: no admissible candidate for question: 'How many more... |
|
||||
| 0008 | refused | no_admissible_question | production_yield_question | — | question_parser | yield_question_frame | downstream_reclassification | If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out... | candidate_graph: no admissible candidate for question: 'If 50 beads a... |
|
||||
| 0009 | refused | no_admissible_question | conditional_aggregate_total | — | question_parser | conditional_aggregate_question | downstream_reclassification | If Jen has 150 ducks, how many total birds does she have? | candidate_graph: no admissible candidate for question: 'If Jen has 15... |
|
||||
| 0010 | refused | no_admissible_statement | affine_equation_fraction_delta | — | statement_parser | affine_equation | diagnostic_only | Marion has 1/4 more than what Yun currently has, plus 7. | candidate_graph: no admissible candidate for statement: 'Marion has 1... |
|
||||
| 0011 | refused | no_solvable_branch | rate_graph_unsolvable | — | graph_composition | rate_composition | downstream_reclassification | Alexa has a lemonade stand where she sells lemonade for $2 for one cup. | candidate_graph: no branch produced a solvable graph |
|
||||
| 0012 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | He put all of them in his aquarium but his fish ate half of them. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0013 | refused | recognized_no_injection | no_injection_multiplicative_aggregation | multiplicative_aggregation | recognizer_injector | multiplicative_aggregate | diagnostic_only | Allison, a YouTuber, uploads 10 one-hour videos of food reviews each day to her channel. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0015 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train... | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0016 | refused | recognized_no_injection | dcs_misroute_comparative_additive | discrete_count_statement | recognizer_injector | compare_additive | downstream_reclassification | On Rudolph's car trip across town, he traveled 2 more than 5 miles and encountered 3 less tha... | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0017 | refused | recognized_no_injection | no_injection_temporal_aggregation | temporal_aggregation | recognizer_injector | temporal_tariff | diagnostic_only | He's charging $50.00 per day or $500.00 for 14 days. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0019 | refused | recognized_no_injection | no_injection_currency_amount | currency_amount | recognizer_injector | currency_mutation | diagnostic_only | After the first appointment, John paid $100 for pet insurance that covers 80% of the subseque... | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0020 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Two puppies, two kittens, and three parakeets were for sale at the pet shop. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0021 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He bench presses 15 pounds for 10 reps and does 3 sets. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0022 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | She goes out fishing today and catches twice as many fish as she caught in total in the past... | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0023 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | Cindy collected twice as many, and Rex collected half of Nicole and Cindy's combined total. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0025 | refused | no_admissible_question | multiplicative_peer_pick | — | question_parser | peer_partition_question | downstream_reclassification | If three of Lilibeth's friends pick the same amount as her, how many strawberries do Lilibeth... | candidate_graph: no admissible candidate for question: "If three of L... |
|
||||
| 0026 | refused | no_admissible_statement | each_binding_currency | — | statement_parser | each_entity_binding | downstream_reclassification | Aaron and his brother Carson each saved up $40 to go to dinner. | candidate_graph: no admissible candidate for statement: 'Aaron and hi... |
|
||||
| 0027 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | The number of followers he has on Twitter is half the number of followers he has on Instagram... | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0028 | refused | recognized_no_injection | no_injection_currency_amount | currency_amount | recognizer_injector | currency_mutation | diagnostic_only | It cost $100,000 to open initially. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0030 | refused | no_admissible_statement | numeric_expression_duration | — | statement_parser | duration_multiplier | downstream_reclassification | It is a 2-hour drive each way. | candidate_graph: no admissible candidate for statement: 'It is a 2-ho... |
|
||||
| 0031 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Jeremie wants to go to an amusement park with 3 friends at the end of summer. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0032 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He draws and colors 10 pictures. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0033 | refused | recognized_no_injection | dcs_misroute_comparative_multiplicative | discrete_count_statement | recognizer_injector | compare_multiplicative | downstream_reclassification | Rachel is 12 years old, and her grandfather is 7 times her age. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0034 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He can run 40 yards within 5 seconds. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0035 | refused | no_admissible_question | inverse_residual_more | — | question_parser | inverse_residual_question | downstream_reclassification | How many more apples would Martha need to give away to be left with only 4 of them? | candidate_graph: no admissible candidate for question: 'How many more... |
|
||||
| 0036 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | She studied for 2 hours on Wednesday and three times as long on Thursday. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0037 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Michael wants to lose 10 pounds by June. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0039 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Orlando gained 5 pounds. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0040 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He now has 2 horses, 5 dogs, 7 cats, 3 turtles, and 1 goat. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0041 | refused | recognized_no_injection | dcs_misroute_fraction_change | discrete_count_statement | recognizer_injector | fraction_of_prior | diagnostic_only | The guests eat all of 1 pan, and 75% of the 2nd pan. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0043 | refused | recognized_no_injection | no_injection_currency_amount | currency_amount | recognizer_injector | currency_mutation | diagnostic_only | Her mother gave her an additional $4, and her father twice as much as her mother. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0044 | refused | recognized_no_injection | dcs_misroute_fraction_change | discrete_count_statement | recognizer_injector | fraction_of_prior | diagnostic_only | John invests in a bank and gets 10% simple interest. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0045 | refused | recognized_no_injection | no_injection_multiplicative_aggregation | multiplicative_aggregation | recognizer_injector | multiplicative_aggregate | diagnostic_only | Each survey has 10 questions. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0046 | refused | recognized_no_injection | no_injection_descriptive_setup_no_quantity | descriptive_setup_no_quantity | recognizer_injector | relation_hypothesis | diagnostic_only | Half of the students are girls, the other half are boys. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0047 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | He then packs an equal number of the macaroons in 4 different brown bags, ready for delivery. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0048 | refused | no_admissible_statement | numeric_expression_recurrence | — | statement_parser | recurrence_frame | downstream_reclassification | Every week, he gets 6 cards. | candidate_graph: no admissible candidate for statement: 'Every week, ... |
|
||||
| 0049 | refused | recognized_no_injection | dcs_composition_wall | discrete_count_statement | recognizer_injector | derivation_composer | diagnostic_only | Malcolm is trying to find the fastest walk to school and is currently comparing two routes. | candidate_graph: recognizer matched but produced no injection for sta... |
|
||||
| 0050 | refused | no_admissible_statement | numeric_expression_recurrence | — | statement_parser | recurrence_frame | downstream_reclassification | Mark does a gig every other day for 2 weeks. | candidate_graph: no admissible candidate for statement: 'Mark does a ... |
|
||||
|
||||
## Recommended next ratification candidate
|
||||
|
||||
**Gate A2a unit partition / chunking primitive** (within roadmap Gate A2 partition/chunking).
|
||||
|
||||
This is a **ratification candidate**, not an implementation decision. Implementation
|
||||
follows only after a ratification doc proves a shared narrow primitive with
|
||||
wrong=0 admission tests.
|
||||
|
||||
### Why Gate A2a (evidence)
|
||||
|
||||
1. **One confirmed DCS unit-partition misroute:** case **0002** — `1000 feet` split into
|
||||
`25-foot sections`, blocked at `recognizer_injector` with
|
||||
`dcs_misroute_unit_partition` → candidate primitive `unit_partition`, movement
|
||||
`downstream_reclassification`.
|
||||
2. **Broader partition/chunk overlap exists but is not unified:** heuristic tag
|
||||
`partition_chunking` also touches 0003 (multiplicative product) and 0035 (inverse
|
||||
residual question); parser-layer partition-adjacent cases include 0004, 0025, 0047.
|
||||
These do **not** share one primitive yet — ratification must name what is in/out of
|
||||
Gate A2a v1.
|
||||
3. **Inc3/Gate A1 precedent:** closed injector buckets first with a ratified v1 template;
|
||||
aggregate 6/44/0 unchanged; movement is reclassification visibility.
|
||||
|
||||
### Why this is ratification, not implementation
|
||||
|
||||
- Only **one** DCS misroute is confirmed for `unit_partition` on current serving code.
|
||||
- Partition/chunk **candidates** appear across statement parser, question parser, and
|
||||
downstream composition buckets — implementation without ratification would smuggle
|
||||
scope (affine fractions, peer-pick questions, DCS composition wall).
|
||||
- Ratification must prove a **shared narrow primitive** (total → chunk count → per-chunk
|
||||
unit) with exemplars, confusers, and wrong=0 replay before any injector/parser code.
|
||||
|
||||
## Not next (explicit deferrals)
|
||||
|
||||
| Track | Defer because |
|
||||
|---|---|
|
||||
| **Gate A1b / Comparative-A2 (additive comparative)** | Only one simple same-sentence DCS misroute (0016); no cluster of additive entity-comparison refusals at injector. Defer unless a future microscope pass finds ≥3 same-sentence `more than` / `less than` entity-comparison refusals with shared template. |
|
||||
| **Inc4 rate denom-state** | Live rate cases refuse on overtime (0001), tariff (0017), profit/composition (0011 graph), not merely missing hour/kg/cup Initial. Downstream rate refusals = 2. Denom-state alone does not explain the frontier. |
|
||||
| **Broad DCS widening** | 14/19 DCS refusals are `dcs_composition_wall` → `derivation_composer`. Forbidden without a composition-wall ratification; metric-inert injector widening risks incomplete readings under wrong=0. |
|
||||
| **`report.json` rebaseline** | Separate ratified PR only; pinned artifact remains historical 6/44/0 in this PR. |
|
||||
|
||||
## Gate A2a ratification seed (docs only — not ratified)
|
||||
|
||||
**Candidate primitive names:** `unit_partition` / `partition_count` / `unitize`
|
||||
|
||||
**Template examples (from train_sample surfaces):**
|
||||
|
||||
| Surface pattern | Case hint | Notes |
|
||||
|---|---|---|
|
||||
| `N total-units split into M-unit sections` | 0002: 1000 feet → 25-foot sections | Canonical Gate A2a exemplar |
|
||||
| `N beads per bracelet` (yield/partition question) | 0008: 50 beads → bracelets | Question-layer binding — may be out of v1 |
|
||||
| `N crayons per box` / residual box count | 0007: boxes vs loose crayons | Inverse residual — likely out of v1 |
|
||||
|
||||
**Open design questions (must be resolved in ratification):**
|
||||
|
||||
1. **Output unit:** chunk count vs retained measure unit vs dual unknowns?
|
||||
2. **Divisibility:** exact division required (0002: 1000/25) vs refuse non-integer chunk counts?
|
||||
3. **Fractional results:** 1/4 given away + half stored (0002 full problem) — in v1 scope or composition follow-up?
|
||||
4. **Source grounding:** total quantity referent vs chunk-size referent — which entity keys are grounded from the statement?
|
||||
5. **Inverse direction:** “how many more boxes” (0007) — partition primitive or question-frame primitive?
|
||||
6. **Downstream question binding:** per-chunk query (`how much does she keep on hand`) requires partition+remainder composition — v1 injector only or paired composition ratification?
|
||||
|
||||
## Tests / validation (this PR)
|
||||
|
||||
```bash
|
||||
git diff --check origin/main...HEAD
|
||||
PYTHONPATH=. .venv/bin/python -m pytest tests/test_gsm8k_post_gate_a1_frontier_microscope.py -q
|
||||
PYTHONPATH=. .venv/bin/python -m pytest tests/test_gsm8k_frontier_report.py -q
|
||||
.venv/bin/core test --suite smoke -q
|
||||
```
|
||||
|
||||
## Explicit non-changes
|
||||
|
||||
- No Gate A2a implementation
|
||||
- No Gate A1b / Comparative-A2 implementation
|
||||
- No Inc4 denom-state
|
||||
- No runtime / generate / solver / recognizer edits
|
||||
- No `report.json` rebaseline
|
||||
|
|
@ -1,11 +1,11 @@
|
|||
# GSM8K Workstream A Gate A1 — multiplicative comparative injection lookback
|
||||
|
||||
**Date:** 2026-06-17 (post-implementation evidence closure)
|
||||
**Branch:** `feat/gsm8k-workstream-a-gate-a1-comparative-multiplicative-injection`
|
||||
**Head (implementation + patch):** `64ab58a7fa01ccfb1c707573b1be044296f5fe38`
|
||||
**Base implementation commit:** `e578ec72`
|
||||
**Branch:** `feat/gsm8k-workstream-a-gate-a1-comparative-multiplicative-injection` (merged #805)
|
||||
**Head (final PR #805):** `c37f97bec36a19bd60a3921c2e1a78b294e00e7b`
|
||||
**Merge commit on main:** `bb0830046aa4da23ae58eac8fe2368da5f0be6b5`
|
||||
**Governing ratification:** `docs/analysis/gsm8k-workstream-a-gate-a1-comparative-multiplicative-ratification-2026-06-17.md` (merged #803)
|
||||
**Scope:** **First subfamily only** — multiplicative entity comparison with explicit same-sentence reference. Additive comparative (Gate A2) deferred.
|
||||
**Scope:** **First subfamily only** — multiplicative entity comparison with explicit same-sentence reference. Additive comparative deferred to **Gate A1b / Comparative-A2** (roadmap **Gate A2** is partition/chunking).
|
||||
|
||||
## What shipped
|
||||
|
||||
|
|
@ -95,7 +95,7 @@ Live `recognized_no_injection_by_category` (post-Gate A1):
|
|||
## Explicit non-changes
|
||||
|
||||
- No `report.json` rebaseline
|
||||
- No additive comparative (`compare_additive`) — Gate A2
|
||||
- No additive comparative (`compare_additive`) — Gate A1b / Comparative-A2
|
||||
- No `double` / `one-third` / hyphenated N-times factors
|
||||
- No sealed-lane movement
|
||||
- No solver `_apply_compare_multiplicative` semantic change
|
||||
|
|
@ -103,13 +103,13 @@ Live `recognized_no_injection_by_category` (post-Gate A1):
|
|||
|
||||
## Known caveats
|
||||
|
||||
1. **Not full Gate A1 family** — multiplicative entity-comparison subfamily only; additive deferred to Gate A2.
|
||||
1. **Not full Gate A1 family** — multiplicative entity-comparison subfamily only; additive deferred to Gate A1b / Comparative-A2.
|
||||
2. **No guaranteed correct-count lift** — monotonic contract holds (`correct>=6`, `refused<=44`); primary deliverable is injector closure + reclassification visibility.
|
||||
3. **DCS/parser path for bare `Jerry has 3 times`** — standalone no-reference surface may still interact with parser initials on some question shapes; Gate A1 injector refuses; completeness guard covers multi-clause confabulation class. Not claimed solved in this slice.
|
||||
4. **Matcher regex vs factor narrowness** — `_COMPARE_MULT_NTIMES_RE` still uses broad `_VALUE` at template level; extraction refuses unsafe factors via `_parse_comparative_v1_count_factor`.
|
||||
5. **Pinned frontier tests** — historical `report.json` fixture unchanged; live behavior tested ephemerally.
|
||||
|
||||
## Deferred work (Gate A1b / A2)
|
||||
## Deferred work (Gate A1b / Comparative-A2; roadmap Gate A2 = partition/chunking)
|
||||
|
||||
- `double`, `one-third`, `as much`, unit ellipsis
|
||||
- Additive comparative injection
|
||||
|
|
|
|||
588
scripts/gsm8k_post_gate_a1_frontier_microscope.py
Normal file
588
scripts/gsm8k_post_gate_a1_frontier_microscope.py
Normal file
|
|
@ -0,0 +1,588 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Post-Gate-A1 live frontier microscope for GSM8K train_sample.
|
||||
|
||||
Builds an ephemeral ``build_report(cases)`` snapshot (never writes
|
||||
``report.json``) and classifies every refused case into stable buckets
|
||||
and implementation-slice candidate families.
|
||||
|
||||
Docs/tooling only — no runtime mutation. Rules are deterministic
|
||||
string predicates; no LLM, no embeddings, no clocks.
|
||||
|
||||
Usage:
|
||||
uv run python scripts/gsm8k_post_gate_a1_frontier_microscope.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report
|
||||
from evals.refusal_taxonomy.shape_categories import ShapeCategory, categorize
|
||||
from scripts.gsm8k_frontier_report import _extract_category
|
||||
|
||||
_RECOGNIZED_NO_INJ = "candidate_graph: recognizer matched but produced no injection"
|
||||
|
||||
_MICROSCOPE_BUCKET_PATTERNS: tuple[tuple[str, str], ...] = (
|
||||
("wrong", "wrong"),
|
||||
("fast-path", "fast_path_correct"),
|
||||
("no admissible candidate for question", "no_admissible_question"),
|
||||
("no admissible candidate for statement", "no_admissible_statement"),
|
||||
("no branch produced a solvable graph", "no_solvable_branch"),
|
||||
("no solvable branch", "no_solvable_branch"),
|
||||
("incomplete reading", "incomplete_reading"),
|
||||
(_RECOGNIZED_NO_INJ, "recognized_no_injection"),
|
||||
)
|
||||
|
||||
|
||||
def _classify_reason(reason: str) -> str:
|
||||
if not reason:
|
||||
return "other_refused"
|
||||
lowered = reason.lower()
|
||||
for needle, bucket in _MICROSCOPE_BUCKET_PATTERNS:
|
||||
if needle.lower() in lowered:
|
||||
return bucket
|
||||
if "refused" in lowered or not reason.strip():
|
||||
return "other_refused"
|
||||
return "other"
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
_CASES_PATH = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
|
||||
_EXPECTED_COUNT = 50
|
||||
|
||||
# --- subfamily needles (stable, sorted for determinism) -------------------
|
||||
|
||||
_NO_ADM_STMT_SUBFAMILIES: tuple[tuple[str, str], ...] = (
|
||||
("nested_fraction_partition", r"half of .+ (?:going|are)"),
|
||||
("affine_equation_fraction_delta", r"1/\d+ more than|more than what .+ has"),
|
||||
("affine_equation_fraction_target", r"decrease to \d+/\d+\s+of"),
|
||||
("each_binding_currency", r"each saved up \$"),
|
||||
("numeric_expression_duration", r"\d+-hour (?:drive|trip|walk)"),
|
||||
("numeric_expression_recurrence", r"every (?:other )?(?:day|week)"),
|
||||
("partition_statement_parser_gap", r"split|splits|sections|pieces"),
|
||||
)
|
||||
|
||||
_NO_ADM_QUEST_SUBFAMILIES: tuple[tuple[str, str], ...] = (
|
||||
("inverse_residual_more", r"how many more"),
|
||||
("multiplicative_peer_pick", r"friends? pick"),
|
||||
("ratio_times_entity", r"times (?:the )?number of"),
|
||||
("production_yield_question", r"how many \w+ (?:will|be able to)"),
|
||||
("conditional_aggregate_total", r" if .+, how many"),
|
||||
)
|
||||
|
||||
_NO_SOLVABLE_SUBFAMILIES: tuple[tuple[str, str], ...] = (
|
||||
("rate_graph_unsolvable", r"\$|per (?:day|hour|kg|cup)"),
|
||||
("composition_branch_exhausted", r".*"),
|
||||
)
|
||||
|
||||
_DCS_NO_INJ_SUBFAMILIES: tuple[tuple[str, str], ...] = (
|
||||
("dcs_misroute_unit_partition", r"\d+-(?:foot|feet|inch|yard|mile|meter)"),
|
||||
("dcs_misroute_comparative_multiplicative", r"times (?:her|his|their|as many|as much)"),
|
||||
("dcs_misroute_comparative_additive", r" more than | less than "),
|
||||
("dcs_misroute_rate_surface", r"\$| per | an hour|/hour|/day|/kg"),
|
||||
("dcs_misroute_temporal", r"each day|every day|per day|one-hour|in \d+ (?:minute|hour)"),
|
||||
("dcs_misroute_fraction_change", r"half of|75%|10%|ate half"),
|
||||
("dcs_composition_wall", r".*"),
|
||||
)
|
||||
|
||||
_SLICE_CANDIDATE_RULES: tuple[tuple[str, str], ...] = (
|
||||
("partition_chunking", r"split|sections|\d+-(?:foot|feet)|partition|per box|in each"),
|
||||
("additive_comparative", r" more than | less than |as much as|as many as"),
|
||||
("affine_equation", r"1/\d+ more than|decrease to \d+/\d+ of|half of .+ (?:going|are)"),
|
||||
("numeric_expression", r"\d+-hour|every (?:other )?(?:day|week)|each way"),
|
||||
("rate_follow_up", r"\$| per |overtime|an hour|/day|/kg|for one cup"),
|
||||
("multiplicative_comparative_follow_up", r"twice|thrice|\d+ times (?:her|his|their)"),
|
||||
)
|
||||
|
||||
_BLOCKING_LAYER_BY_BUCKET: dict[str, str] = {
|
||||
"recognized_no_injection": "recognizer_injector",
|
||||
"no_admissible_statement": "statement_parser",
|
||||
"no_admissible_question": "question_parser",
|
||||
"no_solvable_branch": "graph_composition",
|
||||
"incomplete_reading": "reading_completeness",
|
||||
"other_refused": "unknown",
|
||||
"other": "unknown",
|
||||
}
|
||||
|
||||
_PRIMITIVE_BY_SUBFAMILY: dict[str, str] = {
|
||||
"dcs_misroute_unit_partition": "unit_partition",
|
||||
"dcs_misroute_comparative_additive": "compare_additive",
|
||||
"dcs_misroute_comparative_multiplicative": "compare_multiplicative",
|
||||
"dcs_misroute_rate_surface": "rate_composition",
|
||||
"dcs_misroute_temporal": "temporal_aggregation",
|
||||
"dcs_misroute_fraction_change": "fraction_of_prior",
|
||||
"dcs_composition_wall": "derivation_composer",
|
||||
"no_injection_currency_amount": "currency_mutation",
|
||||
"no_injection_descriptive_setup_no_quantity": "relation_hypothesis",
|
||||
"no_injection_multiplicative_aggregation": "multiplicative_aggregate",
|
||||
"no_injection_temporal_aggregation": "temporal_tariff",
|
||||
"nested_fraction_partition": "fraction_partition",
|
||||
"affine_equation_fraction_delta": "affine_equation",
|
||||
"affine_equation_fraction_target": "fractional_delta",
|
||||
"each_binding_currency": "each_entity_binding",
|
||||
"numeric_expression_duration": "duration_multiplier",
|
||||
"numeric_expression_recurrence": "recurrence_frame",
|
||||
"partition_statement_parser_gap": "unit_partition",
|
||||
"unclassified": "diagnostic_hold",
|
||||
"inverse_residual_more": "inverse_residual_question",
|
||||
"ratio_times_entity": "ratio_question_frame",
|
||||
"production_yield_question": "yield_question_frame",
|
||||
"conditional_aggregate_total": "conditional_aggregate_question",
|
||||
"multiplicative_peer_pick": "peer_partition_question",
|
||||
"rate_graph_unsolvable": "rate_composition",
|
||||
}
|
||||
|
||||
_EVIDENCE_SNIPPET_MAX = 96
|
||||
|
||||
|
||||
def _padded_lower(text: str) -> str:
|
||||
return " " + text.lower().replace("\n", " ") + " "
|
||||
|
||||
|
||||
def _first_subfamily(
|
||||
text: str,
|
||||
rules: tuple[tuple[str, str], ...],
|
||||
) -> str:
|
||||
hay = _padded_lower(text)
|
||||
for name, pattern in rules:
|
||||
if re.search(pattern, hay, flags=re.IGNORECASE):
|
||||
return name
|
||||
if rules and rules[-1][1] == r".*":
|
||||
return rules[-1][0]
|
||||
return "unclassified"
|
||||
|
||||
|
||||
def _extract_quoted_clause(reason: str, label: str) -> str:
|
||||
"""Extract ``for <label>: '...'`` / ``"..."`` including embedded apostrophes."""
|
||||
patterns = (
|
||||
rf"for {label}: '(.+?)'(?: \(category=|\s*$)",
|
||||
rf'for {label}: "(.+?)"(?: \(category=|\s*$)',
|
||||
)
|
||||
for pattern in patterns:
|
||||
m = re.search(pattern, reason, flags=re.DOTALL)
|
||||
if m:
|
||||
return m.group(1).replace("\\'", "'").replace('\\"', '"')
|
||||
return ""
|
||||
|
||||
|
||||
def _extract_statement(reason: str) -> str:
|
||||
return _extract_quoted_clause(reason, "statement")
|
||||
|
||||
|
||||
def _extract_question(reason: str) -> str:
|
||||
return _extract_quoted_clause(reason, "question")
|
||||
|
||||
|
||||
def _load_cases(path: Path = _CASES_PATH) -> list[dict[str, Any]]:
|
||||
records = [
|
||||
json.loads(line)
|
||||
for line in path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
if len(records) != _EXPECTED_COUNT:
|
||||
raise ValueError(
|
||||
f"train sample must contain exactly {_EXPECTED_COUNT} cases; "
|
||||
f"found {len(records)} at {path}"
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RefusalRecord:
|
||||
case_id: str
|
||||
verdict: str
|
||||
reason: str
|
||||
question: str
|
||||
top_bucket: str
|
||||
no_injection_category: str | None
|
||||
failing_text: str
|
||||
statement_shape: str
|
||||
question_shape: str
|
||||
subfamily: str
|
||||
slice_candidates: tuple[str, ...]
|
||||
first_blocking_layer: str
|
||||
candidate_next_primitive: str
|
||||
expected_movement: str
|
||||
evidence_snippet: str
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"case_id": self.case_id,
|
||||
"verdict": self.verdict,
|
||||
"reason": self.reason,
|
||||
"top_refusal_bucket": self.top_bucket,
|
||||
"subfamily": self.subfamily,
|
||||
"matched_recognizer_category": self.no_injection_category,
|
||||
"first_blocking_layer": self.first_blocking_layer,
|
||||
"candidate_next_primitive": self.candidate_next_primitive,
|
||||
"expected_movement": self.expected_movement,
|
||||
"evidence_snippet": self.evidence_snippet,
|
||||
"failing_text": self.failing_text,
|
||||
"statement_shape": self.statement_shape,
|
||||
"question_shape": self.question_shape,
|
||||
"slice_candidates": list(self.slice_candidates),
|
||||
}
|
||||
|
||||
|
||||
def _evidence_snippet(question: str, failing_text: str) -> str:
|
||||
source = (failing_text or question).strip()
|
||||
if not source:
|
||||
return ""
|
||||
# Prefer the sentence containing the load-bearing surface.
|
||||
for part in re.split(r"(?<=[.!?])\s+", source):
|
||||
part = part.strip()
|
||||
if part:
|
||||
source = part
|
||||
break
|
||||
source = re.sub(r"\s+", " ", source)
|
||||
if len(source) <= _EVIDENCE_SNIPPET_MAX:
|
||||
return source
|
||||
return source[: _EVIDENCE_SNIPPET_MAX - 3].rstrip() + "..."
|
||||
|
||||
|
||||
def _candidate_next_primitive(subfamily: str, slice_candidates: tuple[str, ...]) -> str:
|
||||
if subfamily in _PRIMITIVE_BY_SUBFAMILY:
|
||||
return _PRIMITIVE_BY_SUBFAMILY[subfamily]
|
||||
if "partition_chunking" in slice_candidates:
|
||||
return "unit_partition"
|
||||
if "additive_comparative" in slice_candidates:
|
||||
return "compare_additive"
|
||||
if "affine_equation" in slice_candidates:
|
||||
return "affine_equation"
|
||||
if "numeric_expression" in slice_candidates:
|
||||
return "numeric_expression_frame"
|
||||
if "rate_follow_up" in slice_candidates:
|
||||
return "rate_composition"
|
||||
if "multiplicative_comparative_follow_up" in slice_candidates:
|
||||
return "compare_multiplicative"
|
||||
return "diagnostic_hold"
|
||||
|
||||
|
||||
def _expected_movement(
|
||||
*,
|
||||
subfamily: str,
|
||||
top_bucket: str,
|
||||
candidate_primitive: str,
|
||||
) -> str:
|
||||
if subfamily == "dcs_misroute_unit_partition":
|
||||
return "downstream_reclassification"
|
||||
if subfamily == "dcs_composition_wall":
|
||||
return "diagnostic_only"
|
||||
if top_bucket == "no_solvable_branch":
|
||||
return "downstream_reclassification"
|
||||
if top_bucket == "no_admissible_question" and subfamily in {
|
||||
"inverse_residual_more",
|
||||
"ratio_times_entity",
|
||||
"multiplicative_peer_pick",
|
||||
"production_yield_question",
|
||||
"conditional_aggregate_total",
|
||||
}:
|
||||
return "downstream_reclassification"
|
||||
if top_bucket == "no_admissible_statement" and candidate_primitive in {
|
||||
"unit_partition",
|
||||
"duration_multiplier",
|
||||
"recurrence_frame",
|
||||
"each_entity_binding",
|
||||
}:
|
||||
return "downstream_reclassification"
|
||||
if subfamily in {
|
||||
"dcs_misroute_comparative_additive",
|
||||
"dcs_misroute_comparative_multiplicative",
|
||||
}:
|
||||
return "downstream_reclassification"
|
||||
if candidate_primitive == "derivation_composer":
|
||||
return "diagnostic_only"
|
||||
return "diagnostic_only"
|
||||
|
||||
|
||||
def _slice_candidates(question: str, failing_text: str) -> tuple[str, ...]:
|
||||
hay = _padded_lower(f"{question} {failing_text}")
|
||||
hits: list[str] = []
|
||||
for name, pattern in _SLICE_CANDIDATE_RULES:
|
||||
if re.search(pattern, hay, flags=re.IGNORECASE):
|
||||
hits.append(name)
|
||||
return tuple(hits)
|
||||
|
||||
|
||||
def _assign_subfamily(
|
||||
*,
|
||||
top_bucket: str,
|
||||
no_inj_cat: str | None,
|
||||
failing_text: str,
|
||||
question: str,
|
||||
) -> str:
|
||||
text = failing_text or question
|
||||
if top_bucket == "recognized_no_injection" and no_inj_cat == "discrete_count_statement":
|
||||
return _first_subfamily(text, _DCS_NO_INJ_SUBFAMILIES)
|
||||
if top_bucket == "no_admissible_statement":
|
||||
return _first_subfamily(text, _NO_ADM_STMT_SUBFAMILIES)
|
||||
if top_bucket == "no_admissible_question":
|
||||
q = failing_text or question
|
||||
return _first_subfamily(q, _NO_ADM_QUEST_SUBFAMILIES)
|
||||
if top_bucket == "no_solvable_branch":
|
||||
return _first_subfamily(question, _NO_SOLVABLE_SUBFAMILIES)
|
||||
if top_bucket == "recognized_no_injection" and no_inj_cat:
|
||||
return f"no_injection_{no_inj_cat}"
|
||||
return top_bucket
|
||||
|
||||
|
||||
def classify_refusal(
|
||||
*,
|
||||
case_id: str,
|
||||
reason: str,
|
||||
question: str,
|
||||
verdict: str = "refused",
|
||||
) -> RefusalRecord:
|
||||
top_bucket = _classify_reason(reason)
|
||||
no_inj_cat = _extract_category(reason) if top_bucket == "recognized_no_injection" else None
|
||||
failing_stmt = _extract_statement(reason)
|
||||
failing_q = _extract_question(reason)
|
||||
failing_text = failing_stmt or failing_q or question
|
||||
subfamily = _assign_subfamily(
|
||||
top_bucket=top_bucket,
|
||||
no_inj_cat=no_inj_cat,
|
||||
failing_text=failing_text,
|
||||
question=question,
|
||||
)
|
||||
slices = _slice_candidates(question, failing_text)
|
||||
primitive = _candidate_next_primitive(subfamily, slices)
|
||||
return RefusalRecord(
|
||||
case_id=case_id,
|
||||
verdict=verdict,
|
||||
reason=reason,
|
||||
question=question,
|
||||
top_bucket=top_bucket,
|
||||
no_injection_category=no_inj_cat,
|
||||
failing_text=failing_text,
|
||||
statement_shape=categorize(failing_stmt).value if failing_stmt else "",
|
||||
question_shape=categorize(question).value,
|
||||
subfamily=subfamily,
|
||||
slice_candidates=slices,
|
||||
first_blocking_layer=_BLOCKING_LAYER_BY_BUCKET.get(top_bucket, "unknown"),
|
||||
candidate_next_primitive=primitive,
|
||||
expected_movement=_expected_movement(
|
||||
subfamily=subfamily,
|
||||
top_bucket=top_bucket,
|
||||
candidate_primitive=primitive,
|
||||
),
|
||||
evidence_snippet=_evidence_snippet(question, failing_text),
|
||||
)
|
||||
|
||||
|
||||
def build_microscope_report(
|
||||
cases: list[dict[str, Any]] | None = None,
|
||||
*,
|
||||
cases_path: Path = _CASES_PATH,
|
||||
) -> dict[str, Any]:
|
||||
"""Ephemeral live frontier microscope — never writes report.json."""
|
||||
loaded = cases if cases is not None else _load_cases(cases_path)
|
||||
report = build_report(loaded)
|
||||
case_by_id = {c["case_id"]: c for c in loaded}
|
||||
|
||||
records: list[RefusalRecord] = []
|
||||
for row in report.get("per_case", []):
|
||||
if str(row.get("verdict", "")).lower() != "refused":
|
||||
continue
|
||||
cid = str(row["case_id"])
|
||||
records.append(
|
||||
classify_refusal(
|
||||
case_id=cid,
|
||||
reason=str(row.get("reason", "") or ""),
|
||||
question=str(case_by_id[cid]["question"]),
|
||||
)
|
||||
)
|
||||
records.sort(key=lambda r: r.case_id)
|
||||
|
||||
def _count_by(getter) -> dict[str, int]:
|
||||
tallies: dict[str, int] = defaultdict(int)
|
||||
for rec in records:
|
||||
tallies[str(getter(rec))] += 1
|
||||
return dict(sorted(tallies.items()))
|
||||
|
||||
def _cases_for(predicate) -> list[str]:
|
||||
return [r.case_id for r in records if predicate(r)]
|
||||
|
||||
recognized_by_cat = _count_by(
|
||||
lambda r: (
|
||||
r.no_injection_category or "uncategorized"
|
||||
if r.top_bucket == "recognized_no_injection"
|
||||
else "__skip__"
|
||||
)
|
||||
)
|
||||
recognized_by_cat.pop("__skip__", None)
|
||||
downstream_rate = _cases_for(
|
||||
lambda r: (
|
||||
r.top_bucket != "recognized_no_injection"
|
||||
and (
|
||||
r.question_shape == ShapeCategory.RATE_WITH_CURRENCY.value
|
||||
or ShapeCategory.RATE_WITH_CURRENCY.value in r.slice_candidates
|
||||
or "rate_follow_up" in r.slice_candidates
|
||||
)
|
||||
)
|
||||
)
|
||||
downstream_comparative = _cases_for(
|
||||
lambda r: (
|
||||
r.top_bucket != "recognized_no_injection"
|
||||
and (
|
||||
r.statement_shape == ShapeCategory.COMPARATIVE_WITH_UNIT.value
|
||||
or r.question_shape == ShapeCategory.COMPARATIVE_WITH_UNIT.value
|
||||
or "additive_comparative" in r.slice_candidates
|
||||
or "multiplicative_comparative_follow_up" in r.slice_candidates
|
||||
)
|
||||
)
|
||||
)
|
||||
partition_cases = _cases_for(lambda r: "partition_chunking" in r.slice_candidates)
|
||||
numeric_cases = _cases_for(lambda r: "numeric_expression" in r.slice_candidates)
|
||||
affine_cases = _cases_for(lambda r: "affine_equation" in r.slice_candidates)
|
||||
additive_cases = _cases_for(lambda r: "additive_comparative" in r.slice_candidates)
|
||||
rate_follow_cases = _cases_for(lambda r: "rate_follow_up" in r.slice_candidates)
|
||||
|
||||
dcs_subfamilies = _count_by(
|
||||
lambda r: r.subfamily
|
||||
if r.no_injection_category == "discrete_count_statement"
|
||||
else "__skip__"
|
||||
)
|
||||
dcs_subfamilies.pop("__skip__", None)
|
||||
|
||||
no_adm_stmt = _count_by(
|
||||
lambda r: r.subfamily if r.top_bucket == "no_admissible_statement" else "__skip__"
|
||||
)
|
||||
no_adm_stmt.pop("__skip__", None)
|
||||
|
||||
no_adm_q = _count_by(
|
||||
lambda r: r.subfamily if r.top_bucket == "no_admissible_question" else "__skip__"
|
||||
)
|
||||
no_adm_q.pop("__skip__", None)
|
||||
|
||||
no_solvable = _count_by(
|
||||
lambda r: r.subfamily if r.top_bucket == "no_solvable_branch" else "__skip__"
|
||||
)
|
||||
no_solvable.pop("__skip__", None)
|
||||
|
||||
slice_histogram = _count_by(
|
||||
lambda r: "|".join(r.slice_candidates) if r.slice_candidates else "none"
|
||||
)
|
||||
|
||||
refusal_table = [r.as_dict() for r in records]
|
||||
|
||||
return {
|
||||
"schema_version": "gsm8k_post_gate_a1_frontier_microscope_v2",
|
||||
"recommended_next_ratification_candidate": "Gate A2a unit_partition / chunking primitive",
|
||||
"source": "ephemeral_build_report",
|
||||
"cases_path": str(cases_path),
|
||||
"counts": report["counts"],
|
||||
"top_buckets": _count_by(lambda r: r.top_bucket),
|
||||
"recognized_no_injection_by_category": recognized_by_cat,
|
||||
"no_admissible_statement_subfamilies": no_adm_stmt,
|
||||
"no_admissible_question_subfamilies": no_adm_q,
|
||||
"no_solvable_branch_subfamilies": no_solvable,
|
||||
"dcs_no_injection_subfamilies": dcs_subfamilies,
|
||||
"downstream_rate_refusals": {
|
||||
"count": len(downstream_rate),
|
||||
"case_ids": downstream_rate,
|
||||
},
|
||||
"downstream_comparative_refusals": {
|
||||
"count": len(downstream_comparative),
|
||||
"case_ids": downstream_comparative,
|
||||
},
|
||||
"implementation_slice_candidates": {
|
||||
"partition_chunking": {
|
||||
"count": len(partition_cases),
|
||||
"case_ids": partition_cases,
|
||||
},
|
||||
"numeric_expression": {
|
||||
"count": len(numeric_cases),
|
||||
"case_ids": numeric_cases,
|
||||
},
|
||||
"affine_equation": {"count": len(affine_cases), "case_ids": affine_cases},
|
||||
"additive_comparative_gate_a1b": {
|
||||
"count": len(additive_cases),
|
||||
"case_ids": additive_cases,
|
||||
},
|
||||
"rate_follow_up": {
|
||||
"count": len(rate_follow_cases),
|
||||
"case_ids": rate_follow_cases,
|
||||
},
|
||||
},
|
||||
"slice_candidate_histogram": slice_histogram,
|
||||
"refusal_table": refusal_table,
|
||||
"per_case": refusal_table,
|
||||
"closed_injector_buckets": {
|
||||
"rate_with_currency_no_injection": recognized_by_cat.get(
|
||||
"rate_with_currency", 0
|
||||
),
|
||||
"comparative_with_unit_no_injection": recognized_by_cat.get(
|
||||
"comparative_with_unit", 0
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(summary: dict[str, Any]) -> str:
|
||||
lines: list[str] = []
|
||||
lines.append("# GSM8K post-Gate-A1 frontier microscope (live ephemeral)")
|
||||
lines.append("")
|
||||
c = summary["counts"]
|
||||
lines.append(f"- correct: {c.get('correct', 0)}")
|
||||
lines.append(f"- refused: {c.get('refused', 0)}")
|
||||
lines.append(f"- wrong: {c.get('wrong', 0)}")
|
||||
lines.append("")
|
||||
lines.append("## Top refusal buckets")
|
||||
for k, v in summary["top_buckets"].items():
|
||||
lines.append(f"- {k}: {v}")
|
||||
lines.append("")
|
||||
lines.append("## recognized_no_injection by category")
|
||||
for k, v in summary["recognized_no_injection_by_category"].items():
|
||||
lines.append(f"- {k}: {v}")
|
||||
lines.append("")
|
||||
lines.append("## DCS no-injection subfamilies")
|
||||
for k, v in summary["dcs_no_injection_subfamilies"].items():
|
||||
lines.append(f"- {k}: {v}")
|
||||
lines.append("")
|
||||
lines.append("## Implementation slice candidates (overlap allowed)")
|
||||
for name, block in summary["implementation_slice_candidates"].items():
|
||||
lines.append(f"- {name}: {block['count']} ({', '.join(block['case_ids'])})")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
f"## Recommended next ratification candidate: "
|
||||
f"{summary.get('recommended_next_ratification_candidate', 'pending')}"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("## Case-level refusal table")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"| case_id | verdict | top bucket | subfamily | recognizer cat | "
|
||||
"blocking layer | next primitive | movement | evidence | reason (truncated) |"
|
||||
)
|
||||
lines.append("|---|---|---|---|---|---|---|---|---|---|")
|
||||
for row in summary.get("refusal_table", []):
|
||||
short_id = row["case_id"].rsplit("-", 1)[-1]
|
||||
cat = row.get("matched_recognizer_category") or "—"
|
||||
reason = row.get("reason", "")
|
||||
if len(reason) > 72:
|
||||
reason = reason[:69] + "..."
|
||||
lines.append(
|
||||
f"| {short_id} | {row['verdict']} | {row['top_refusal_bucket']} | "
|
||||
f"{row['subfamily']} | {cat} | {row['first_blocking_layer']} | "
|
||||
f"{row['candidate_next_primitive']} | {row['expected_movement']} | "
|
||||
f"{row['evidence_snippet']} | {reason} |"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
argv = argv if argv is not None else sys.argv[1:]
|
||||
del argv
|
||||
summary = build_microscope_report()
|
||||
print(json.dumps(summary, indent=2, sort_keys=True))
|
||||
print("\n---\n")
|
||||
print(render_markdown(summary))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -215,6 +215,25 @@ def test_markdown_render_is_stable_and_mentions_rate():
|
|||
assert render_markdown(summary) == md
|
||||
|
||||
|
||||
def test_frontier_report_aligns_with_post_gate_a1_microscope_structure():
|
||||
"""Live microscope stays structurally complete with closed injector buckets."""
|
||||
from scripts.gsm8k_post_gate_a1_frontier_microscope import build_microscope_report
|
||||
|
||||
cases_path = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
|
||||
cases = [
|
||||
json.loads(line)
|
||||
for line in cases_path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
microscope = build_microscope_report(cases)
|
||||
assert_monotonic_serving_counts(microscope["counts"])
|
||||
refused = microscope["counts"]["refused"]
|
||||
assert len(microscope["refusal_table"]) == refused
|
||||
assert sum(microscope["top_buckets"].values()) == refused
|
||||
assert microscope["closed_injector_buckets"]["rate_with_currency_no_injection"] == 0
|
||||
assert microscope["closed_injector_buckets"]["comparative_with_unit_no_injection"] == 0
|
||||
|
||||
|
||||
def test_inc3_connector_makes_rate_no_injection_actionable():
|
||||
"""Inc3 effect: supporting 'one' (and prior 'an'/'per') means rate_with_currency
|
||||
surfaces no longer contribute to recognized_no_injection bucket when injector
|
||||
|
|
|
|||
121
tests/test_gsm8k_post_gate_a1_frontier_microscope.py
Normal file
121
tests/test_gsm8k_post_gate_a1_frontier_microscope.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
"""Tests for post-Gate-A1 live frontier microscope (docs/tooling only)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.gsm8k_post_gate_a1_frontier_microscope import (
|
||||
build_microscope_report,
|
||||
classify_refusal,
|
||||
render_markdown,
|
||||
)
|
||||
from tests.gsm8k_train_sample_baseline import assert_monotonic_serving_counts
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
_CASES_PATH = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
|
||||
|
||||
_APPROVED_TOP_BUCKETS = frozenset({
|
||||
"recognized_no_injection",
|
||||
"no_admissible_statement",
|
||||
"no_admissible_question",
|
||||
"no_solvable_branch",
|
||||
"incomplete_reading",
|
||||
"other_refused",
|
||||
"other",
|
||||
})
|
||||
|
||||
_REFUSAL_TABLE_FIELDS = frozenset({
|
||||
"case_id",
|
||||
"verdict",
|
||||
"reason",
|
||||
"top_refusal_bucket",
|
||||
"subfamily",
|
||||
"matched_recognizer_category",
|
||||
"first_blocking_layer",
|
||||
"candidate_next_primitive",
|
||||
"expected_movement",
|
||||
"evidence_snippet",
|
||||
})
|
||||
|
||||
|
||||
def _load_cases() -> list[dict]:
|
||||
return [
|
||||
json.loads(line)
|
||||
for line in _CASES_PATH.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def test_live_microscope_meets_monotonic_contract_and_closed_injectors():
|
||||
summary = build_microscope_report(_load_cases())
|
||||
|
||||
assert_monotonic_serving_counts(summary["counts"])
|
||||
closed = summary["closed_injector_buckets"]
|
||||
assert closed["rate_with_currency_no_injection"] == 0
|
||||
assert closed["comparative_with_unit_no_injection"] == 0
|
||||
|
||||
|
||||
def test_live_microscope_refusal_partition_is_complete():
|
||||
summary = build_microscope_report(_load_cases())
|
||||
|
||||
refused = summary["counts"]["refused"]
|
||||
assert len(summary["refusal_table"]) == refused
|
||||
assert len(summary["per_case"]) == refused
|
||||
assert sum(summary["top_buckets"].values()) == refused
|
||||
assert set(summary["top_buckets"]) <= _APPROVED_TOP_BUCKETS
|
||||
assert summary["top_buckets"].get("recognized_no_injection", 0) >= 0
|
||||
|
||||
for row in summary["refusal_table"]:
|
||||
assert _REFUSAL_TABLE_FIELDS <= set(row.keys())
|
||||
assert row["verdict"] == "refused"
|
||||
assert row["reason"]
|
||||
assert row["evidence_snippet"]
|
||||
|
||||
|
||||
def test_live_microscope_partition_seed_case_is_tagged():
|
||||
summary = build_microscope_report(_load_cases())
|
||||
assert (
|
||||
"gsm8k-train-sample-v1-0002"
|
||||
in summary["implementation_slice_candidates"]["partition_chunking"]["case_ids"]
|
||||
)
|
||||
|
||||
|
||||
def test_microscope_output_is_deterministic():
|
||||
cases = _load_cases()
|
||||
a = build_microscope_report(cases)
|
||||
b = build_microscope_report(cases)
|
||||
assert json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True)
|
||||
assert render_markdown(a) == render_markdown(b)
|
||||
|
||||
|
||||
def test_classify_refusal_extracts_no_solvable_branch():
|
||||
rec = classify_refusal(
|
||||
case_id="gsm8k-train-sample-v1-0011",
|
||||
reason="candidate_graph: no branch produced a solvable graph",
|
||||
question="Alexa has a lemonade stand where she sells lemonade for $2 for one cup.",
|
||||
)
|
||||
assert rec.top_bucket == "no_solvable_branch"
|
||||
assert rec.subfamily == "rate_graph_unsolvable"
|
||||
|
||||
|
||||
def test_markdown_render_surfaces_partition_candidate():
|
||||
summary = build_microscope_report(_load_cases())
|
||||
md = render_markdown(summary)
|
||||
assert "partition_chunking" in md
|
||||
assert "| 0002 |" in md
|
||||
assert "Gate A2a unit_partition" in md
|
||||
|
||||
|
||||
def test_case_0002_ratification_candidate_fields():
|
||||
summary = build_microscope_report(_load_cases())
|
||||
row = next(
|
||||
r for r in summary["refusal_table"] if r["case_id"].endswith("0002")
|
||||
)
|
||||
assert row["subfamily"] == "dcs_misroute_unit_partition"
|
||||
assert row["candidate_next_primitive"] == "unit_partition"
|
||||
assert row["expected_movement"] == "downstream_reclassification"
|
||||
assert (
|
||||
summary["recommended_next_ratification_candidate"]
|
||||
== "Gate A2a unit_partition / chunking primitive"
|
||||
)
|
||||
Loading…
Reference in a new issue