docs(analysis): ratify Inc3 rate followup + v2 roadmap update (docs-first, pre any rate logic)
This commit is contained in:
Shay
parent
e36591ed46
commit
e9ece64ddd
2 changed files with 141 additions and 0 deletions
|
|
@ -0,0 +1,56 @@
|
|||
# CORE Problem-Solving Capability Roadmap v2 — 2026-06-17
|
||||
|
||||
**Status:** Living document (docs-only update)
|
||||
**Date:** 2026-06-17
|
||||
**Context:** Post PR #797 (rate injection) + #798; preparing Inc3 rate follow-up before Gate A1 comparative injection.
|
||||
|
||||
## Overview
|
||||
|
||||
This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate injection delivered by PR #797.
|
||||
|
||||
As of 2026-06-17, PR #797 is merged and #798 is merged.
|
||||
|
||||
## GSM8K Workstream A
|
||||
|
||||
- Inc 1: reader/recognizer baseline lift (discrete etc.)
|
||||
- Inc 2: frontier measurement + stale doctrine repair + narrow rate injection (PR #797)
|
||||
- **Inc 3 (current seam):** Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection.
|
||||
|
||||
### Recommended Inc3 target (narrow)
|
||||
|
||||
Make the rate frontier evidence actionable by resolving the next narrow blocker exposed by #797.
|
||||
|
||||
Scope candidates (in preference order for this increment):
|
||||
1. Denominator-state support for rate application (if failures surface as "actor has rate but no denom-unit quantity reachable").
|
||||
2. Safe connector expansion only if frontier proves "for one cup" is a dominant blocker.
|
||||
3. Measurement-only frontier report refresh if artifacts stale.
|
||||
|
||||
Inc3 selected #2 (connector for "for one cup"/"one" token) because live debug on the pinned report + cases showed the exact remaining rate injector deferral from Inc2 (matcher left rate_anchor_token=None for "one"; spec unresolved_notes explicitly called it out for the Alexa surface). This was the minimal change that reclassifies the rate_with_currency no-injection bucket (making evidence actionable) while preserving all guards. Denom production is larger future work (see ratification for rationale and out-of-scope).
|
||||
|
||||
"Complete and harden PR #797" is revised as: Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection.
|
||||
|
||||
As of 2026-06-17, PR #797 is merged and #798 is merged.
|
||||
|
||||
Explicitly: do not broaden to full rate language family, comparative injection, or non-rate categories in this increment.
|
||||
|
||||
## Gate A1 / Comparative Injection
|
||||
|
||||
Deferred until after the post-#797 rate follow-up loop is closed with Inc3 measurement.
|
||||
|
||||
## Success Criteria for This Phase
|
||||
|
||||
- Frontier report run on current main (train-sample proxy).
|
||||
- One narrow ratified Inc3 change.
|
||||
- Wrong=0 preserved on train_sample, practice, and relevant confusers.
|
||||
- Rate-family "recognized_no_injection" bucket reduced or its refusal mode made actionable (e.g. surfaces the true next blocker like denom reachability).
|
||||
- No rebaseline of sealed lanes or SHA movement without separate ratification.
|
||||
- Documentation (this roadmap + Inc3 ratification) committed as docs-first.
|
||||
|
||||
## Out of Scope (for Inc3)
|
||||
|
||||
- Full comparative (Gate A1) implementation.
|
||||
- Broad recognizer anchor work or other shape categories.
|
||||
- Changes to serving sealed paths.
|
||||
- Any mutation of identity, policy, or algebra invariants.
|
||||
|
||||
Follow the ratified Inc3 doc for the exact bounded change.
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
# GSM8K Workstream A Increment 3 — rate followup (post-#797) ratification
|
||||
|
||||
**Date:** 2026-06-17
|
||||
**Workstream:** A
|
||||
**Increment:** 3 — post-#797 rate frontier evidence loop closure (narrow)
|
||||
**Status:** Ratified for implementation (BEFORE code changes)
|
||||
**Scope lock:** Bounded to making the rate "recognized_no_injection" bucket produce actionable evidence by resolving the explicit remaining connector blocker left open in #797. One smallest change only.
|
||||
|
||||
## 1. Which exact refusal bucket is being attacked?
|
||||
|
||||
From frontier report run on the (stale but authoritative post-#797) committed proxy `evals/gsm8k_math/train_sample/v1/report.json`:
|
||||
|
||||
- Overall: 6 correct / 44 refused / 0 wrong (passed=false)
|
||||
- recognized_no_injection: 32
|
||||
- recognized_no_injection_by_category (rate relevant): rate_with_currency: 3
|
||||
|
||||
The three rate_with_currency cases still emitting the "recognizer matched but produced no injection" (category=rate_with_currency) are exactly the ones referencing the surfaces left partially unhandled after Inc2:
|
||||
- 'Tina makes $18.00 an hour.' (category=rate_with_currency)
|
||||
- 'Alexa has a lemonade stand where she sells lemonade for $2 for one cup.' (category=rate_with_currency)
|
||||
- 'Erica lives near a lake where most locals sell fish as their main source of income, earning $20 per kg of fish.' (category=rate_with_currency)
|
||||
|
||||
Post-#797, the matcher fires for all three and "an"/"per" surfaces now reach the injector and emit a CandidateOperation (verified by live debug on current main). The "for one cup" explicitly sets `rate_anchor_token: None` (see matcher comment and spec unresolved_notes: "Non-canonical 'for one X' framing").
|
||||
|
||||
The injector returns () for the "one" case (and any elimination downstream for the others surfaces as the same top-level refusal reason because the statement-level inject did not contribute an admitted choice).
|
||||
|
||||
This is the narrow remaining rate-family blocker visible in the rate bucket of the frontier analyzer.
|
||||
|
||||
## 2. Which cases are expected to lift?
|
||||
|
||||
- On the train_sample proxy: expected 0 net lift in correct count (the Alexa "for one cup" case uses inverse semantics — target cups from known revenue, not forward apply_rate on a held cup count; Tina/Erica denom qty statements use verbs/shapes that do not yet emit the required Initial for "hour"/"kg" unit). The change makes injection succeed for the "one" framing; the case will surface a downstream refusal reason ("no branch produced a solvable graph", "no admissible...", or "requires ... state") instead of the "no injection" one.
|
||||
- The rate_with_currency slice of recognized_no_injection is expected to drop from 3 (at least the connector case will no longer refuse at the injector boundary).
|
||||
- No change to non-rate buckets. Wrong remains 0.
|
||||
|
||||
The primary deliverable is **actionable evidence**: after the change the frontier report will show the rate category either empty or reclassified to the true next blocker (denom state reachability), closing the post-#797 measurement loop without claiming a correct-count jump.
|
||||
|
||||
## 3. Which confusers must still refuse?
|
||||
|
||||
All existing confusers from the Inc2 ratification and test suite:
|
||||
- No denom state for the actor (e.g. isolated rate sentence).
|
||||
- Wrong actor (rate stated for A, quantity held by B).
|
||||
- Multiple rates in one sentence.
|
||||
- Time-unit without conversion (days vs hours).
|
||||
- Any surface that would produce ambiguous or ungrounded Rate / actor / verb.
|
||||
|
||||
The change adds "one" only in the exact "for <one> <unit>" rate framing already present in the ratified rate_with_currency exemplars; no broadening of actor binding, no pronoun support, no new verbs outside the rate anchor list.
|
||||
|
||||
## 4. What is the wrong=0 guard?
|
||||
|
||||
- All paths still go through the existing five-layer net (matcher narrowness, source grounding in anchors, injector returns () on any construction failure, roundtrip_admissible + constraint propagation elimination, candidate-graph multi-branch disagreement + completeness).
|
||||
- New surfaces exercise the same `CandidateOperation` + `roundtrip_admissible` + `KIND_TO_VERBS["apply_rate"]` checks.
|
||||
- "one" treated as a surface alias only for the already-ratified "for one X" exemplar in the rate proposal; added to RATE_ANCHORS and injector allow-list with no other semantic change.
|
||||
- No sealed path touched (train_sample runner + serving use sealed=False).
|
||||
- Pre/post change: run the frontier script + `parse_and_solve` on the three rate surfaces + full proxy cases; assert wrong==0 on all.
|
||||
- The `tests/test_math_candidate_graph_rate_injection.py` and `test_gsm8k_frontier_report.py` continue to pass (the existing test already tolerates non-"no injection" refusals for the Alexa stmt).
|
||||
- If after change any train_sample case flips from refused to wrong, revert.
|
||||
|
||||
## 5. Does this touch serving, sealed lanes, report.json, or solver semantics?
|
||||
|
||||
- No changes to sealed injector lane (`_SEALED_INJECTORS` remains empty for this).
|
||||
- No write of updated report.json in this increment (proxy remains at 6/44/0 unless a later runner run is separately committed; the ratification does not require rebaseline).
|
||||
- No solver changes (`_apply_rate` unchanged; still requires denom state).
|
||||
- No graph construction or cartesian changes.
|
||||
- Touches only: the rate anchor token allow-list (matcher + roundtrip set + injector guard) + comments. This is the minimal patch to retire the explicit "narrow for Inc 2" deferral.
|
||||
- The frontier report script, ratification, and roadmap update are docs/evidence only.
|
||||
|
||||
## 6. What is explicitly out of scope?
|
||||
|
||||
- Denominator-state production (seeding Initials for "hour", "kg", "cup" from "works N hours", "trawled 80 kg", etc.). That is future work once the connector surface is closed and the frontier reclassifies the bucket.
|
||||
- Any change that would allow apply_rate without prior denom state.
|
||||
- "for one cup" solving (would require inverse/division op or goal-residual style for price-per).
|
||||
- Expansion to other temporal_aggregation, currency_amount, or non-rate categories.
|
||||
- Comparative injection / Gate A1.
|
||||
- Any movement of sealed SHAs, practice lane, or CLAIMS.
|
||||
- Broad verb or subject binding relaxations.
|
||||
- Re-running the train_sample runner and committing a new report.json as part of this PR (measurement-only refresh is out; the script run on the committed report is the evidence).
|
||||
|
||||
## Implementation Notes (for the PR)
|
||||
|
||||
- Smallest diff: 3 locations (RATE_ANCHORS, matcher "one" case, injector allow-list) + doc updates.
|
||||
- Update comments that say "narrow for Inc 2" or "deferred".
|
||||
- Run `uv run python scripts/gsm8k_frontier_report.py evals/gsm8k_math/train_sample/v1/report.json` before/after for the artifact.
|
||||
- `core test --suite gsm8k` or equivalent lane (pytest on the rate graph test + frontier test) + full `core test --suite full -q` before merge when practical.
|
||||
- Preserve `passed=false` on the proxy.
|
||||
|
||||
This Inc3 closes the rate follow-up loop narrowly so that Ladder A has a clean evidence boundary before any comparative work.
|
||||
Loading…
Reference in a new issue