docs(analysis): close Inc3 rate-followup evidence loop (#801)
Add Inc3 lookback documenting merged #799 behavior, stale pinned report.json vs live ephemeral frontier (rate no-injection = 0), and loop-closure criteria before Gate A1. Update roadmap v2 status. Split frontier tests: historical pinned artifact vs live post-Inc3 runner invariant.
This commit is contained in:
Shay
GitHub
parent
cdd135801d
commit
525793abcc
3 changed files with 178 additions and 24 deletions
|
|
@ -2,19 +2,20 @@
|
|||
|
||||
**Status:** Living document (docs-only update)
|
||||
**Date:** 2026-06-17
|
||||
**Context:** Post PR #797 (rate injection) + #798; preparing Inc3 rate follow-up before Gate A1 comparative injection.
|
||||
**Context:** Post PR #797 (rate injection), #798 (Grok governance), #799 (Inc3 rate connector); Inc3 evidence closure is the current hygiene step before Gate A1 comparative injection.
|
||||
|
||||
## Overview
|
||||
|
||||
This v2 roadmap refines the GSM8K Workstream A path and the broader capability sequencing after the rate injection delivered by PR #797.
|
||||
|
||||
As of 2026-06-17, PR #797 is merged and #798 is merged.
|
||||
As of 2026-06-17: PR #797, #798, and #799 are merged.
|
||||
|
||||
## GSM8K Workstream A
|
||||
|
||||
- Inc 1: reader/recognizer baseline lift (discrete etc.)
|
||||
- Inc 2: frontier measurement + stale doctrine repair + narrow rate injection (PR #797)
|
||||
- **Inc 3 (current seam):** Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection.
|
||||
- Inc 3: `"one"` connector for rate_with_currency (PR #799) — **shipped**
|
||||
- **Current hygiene step:** Inc3 evidence closure (lookback + frontier test semantics; pinned `report.json` remains historical; live ephemeral frontier shows `rate_with_currency` no-injection = 0)
|
||||
|
||||
### Recommended Inc3 target (narrow)
|
||||
|
||||
|
|
@ -29,28 +30,24 @@ Inc3 selected #2 (connector for "for one cup"/"one" token) because live debug on
|
|||
|
||||
"Complete and harden PR #797" is revised as: Complete the post-#797 rate-follow-up evidence loop: run frontier report from current main, identify the remaining rate-family blocker, and ship at most one narrow Inc3 increment before comparative injection.
|
||||
|
||||
As of 2026-06-17, PR #797 is merged and #798 is merged.
|
||||
|
||||
Explicitly: do not broaden to full rate language family, comparative injection, or non-rate categories in this increment.
|
||||
Explicitly: Inc3 did not broaden to full rate language family, comparative injection, or non-rate categories.
|
||||
|
||||
## Gate A1 / Comparative Injection
|
||||
|
||||
Deferred until after the post-#797 rate follow-up loop is closed with Inc3 measurement.
|
||||
**Next** after Inc3 evidence closure merges. Ratify-first (docs/analysis gate-a1 ratification) before any comparative injector code.
|
||||
|
||||
## Success Criteria for This Phase
|
||||
## Success Criteria (Inc3 — met on main; evidence closure completes governance)
|
||||
|
||||
- Frontier report run on current main (train-sample proxy).
|
||||
- One narrow ratified Inc3 change.
|
||||
- Wrong=0 preserved on train_sample, practice, and relevant confusers.
|
||||
- Rate-family "recognized_no_injection" bucket reduced or its refusal mode made actionable (e.g. surfaces the true next blocker like denom reachability).
|
||||
- Inc3 ratified change merged (#799).
|
||||
- Live ephemeral frontier: `rate_with_currency` no-injection = 0; aggregate proxy 6/44/0, wrong=0.
|
||||
- Pinned `report.json` may remain historical; live measurement via ephemeral runner or synthetic frontier tests.
|
||||
- Inc3 lookback committed (docs/analysis/gsm8k-workstream-a-increment-3-lookback-2026-06-17.md).
|
||||
- No rebaseline of sealed lanes or SHA movement without separate ratification.
|
||||
- Documentation (this roadmap + Inc3 ratification) committed as docs-first.
|
||||
|
||||
## Out of Scope (for Inc3)
|
||||
## Out of Scope (Inc3 — held)
|
||||
|
||||
- Full comparative (Gate A1) implementation.
|
||||
- Broad recognizer anchor work or other shape categories.
|
||||
- Changes to serving sealed paths.
|
||||
- Any mutation of identity, policy, or algebra invariants.
|
||||
- Denominator-state production, report.json rebaseline, broad recognizer work.
|
||||
- Changes to serving sealed paths, identity, policy, or algebra invariants.
|
||||
|
||||
Follow the ratified Inc3 doc for the exact bounded change.
|
||||
See Inc3 lookback for loop-closure criterion before Gate A1.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,117 @@
|
|||
# GSM8K Workstream A Increment 3 — rate followup lookback
|
||||
|
||||
**Date:** 2026-06-17 (post-merge evidence closure)
|
||||
**Branch (implementation):** feat/gsm8k-workstream-a-inc3-rate-followup (merged as PR #799)
|
||||
**Head (merged on main):** 8a981e66c716f020b5558c584d39084fadee5011
|
||||
**Governing ratification:** docs/analysis/gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md
|
||||
**Evidence-closure PR branch:** docs/gsm8k-workstream-a-inc3-lookback (this lookback + test semantics; no runtime logic)
|
||||
|
||||
## What shipped (PR #799 merged)
|
||||
|
||||
Inc3 closed the post-#797 rate connector blocker: the `"one"` token in **for one <unit>** rate surfaces is now a valid `apply_rate` rate anchor.
|
||||
|
||||
### Exact semantic path
|
||||
|
||||
1. **Matcher** (`generate/recognizer_match.py`): when the currency-per-unit regex captures connector `q`, if `q in ("each", "every", "a", "one")` then `connector = q` and `rate_anchor_token` is set to that literal (e.g. `"one"` for *for one cup*).
|
||||
2. **Roundtrip whitelist** (`generate/math_roundtrip.py`): `RATE_ANCHORS` includes `"one"` alongside `per`, `each`, `every`, `a`, `an`.
|
||||
3. **Injector** (`generate/recognizer_anchor_inject.py`): `inject_rate_with_currency` accepts `rate_anchor_token in ("per", "each", "every", "a", "an", "one")`; on success emits `CandidateOperation(kind="apply_rate", operand=Rate(...))` with `matched_verb` grounded to the literal connector.
|
||||
|
||||
No other runtime modules changed solver semantics, sealed lanes, graph construction, or serving policy.
|
||||
|
||||
### Files in the merged implementation (#799)
|
||||
|
||||
- generate/math_roundtrip.py
|
||||
- generate/recognizer_match.py
|
||||
- generate/recognizer_anchor_inject.py
|
||||
- tests/test_recognizer_anchor_inject.py
|
||||
- tests/test_math_candidate_graph_rate_injection.py
|
||||
- tests/test_gsm8k_frontier_report.py
|
||||
- tests/test_candidate_graph_recognizer_wiring.py
|
||||
- docs/analysis/gsm8k-workstream-a-increment-3-rate-followup-ratification-2026-06-17.md
|
||||
- docs/analysis/core-problem-solving-capability-roadmap-v2-2026-06-17.md (partial update at merge time)
|
||||
|
||||
## Measurement truth (pinned vs live)
|
||||
|
||||
### Pinned committed artifact (historical / stale for rate bucket)
|
||||
|
||||
`evals/gsm8k_math/train_sample/v1/report.json` remains the **pre-Inc2/Inc3 runner output** committed at the Inc1 rebaseline:
|
||||
|
||||
- **6 correct / 44 refused / 0 wrong**
|
||||
- `recognized_no_injection_by_category.rate_with_currency: 3` (Tina *an hour*, Alexa *for one cup*, Erica *per kg* still show injector-level no-injection strings)
|
||||
|
||||
This artifact is **intentionally not rebaselined** in Inc3 or this evidence-closure PR. Frontier script analysis on the pinned file still reports `rate_with_currency: 3` — that is expected and documents the historical measurement surface Inc2 targeted.
|
||||
|
||||
### Live ephemeral runner (current main code, not committed)
|
||||
|
||||
`build_report(cases)` on current main (ephemeral invocation; no `report.json` write):
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| correct | 6 |
|
||||
| refused | 44 |
|
||||
| wrong | 0 |
|
||||
| `rate_with_currency` in `recognized_no_injection` | **0** |
|
||||
| total `recognized_no_injection` | 31 (was 32 on pinned artifact) |
|
||||
| live `recognized_no_injection_by_category` | `discrete_count_statement: 19`, `temporal_aggregation: 2`, `multiplicative_aggregation: 3`, `descriptive_setup_no_quantity: 4`, `currency_amount: 3` |
|
||||
|
||||
**Interpretation:** rate surfaces no longer refuse at the injector boundary. Refusal moved **downstream** (e.g. `no admissible candidate for statement`, `no branch produced a solvable graph`) because denom-state / composition blockers remain. This matches Inc3 ratification §2: actionable evidence, not correct-count lift.
|
||||
|
||||
`evals/gsm8k_math/train_sample/v1/refusal_rescan_v3.json` corroborates Tina/Alexa cases: `current_refusal_reason` is `no admissible candidate for statement`, not `produced no injection`.
|
||||
|
||||
## wrong=0
|
||||
|
||||
- Pinned artifact: `wrong: 0`
|
||||
- Live ephemeral runner: `wrong: 0`
|
||||
- Inc3 unit/graph tests exercise confusers; no wrong answers admitted on exercised paths.
|
||||
|
||||
## Explicit non-changes (Inc3 scope held)
|
||||
|
||||
- **No** `report.json` rebaseline committed
|
||||
- **No** denominator-state production (hour/kg/cup Initial seeding)
|
||||
- **No** comparative injection / Gate A1
|
||||
- **No** sealed-lane movement (`_SEALED_INJECTORS` remains empty; train_sample uses `sealed=False`)
|
||||
- **No** serving SHA / CLAIMS movement
|
||||
- **No** `determine()` / FrameVerdict / CLOSE interaction
|
||||
- **No** corpus/pack/policy/identity mutation
|
||||
- **No** solver `_apply_rate` semantic change
|
||||
|
||||
## Known caveats
|
||||
|
||||
1. **Aggregate proxy unchanged:** still 6/44/0 and `passed=false` (`correct_min=10`). Inc3 did not claim net correct lift.
|
||||
2. **Refusal-family reclassification:** rate cases may move from `recognized_no_injection` to `no_admissible_statement` or graph unsolvability — not to `correct`.
|
||||
3. **Pinned frontier tests:** tests that read committed `report.json` encode the **historical** rate bucket (3× no-injection). Live-behavior tests use ephemeral `build_report` or synthetic fixtures (see `tests/test_gsm8k_frontier_report.py`).
|
||||
4. **Erica / denom reachability:** third rate case may still refuse downstream for actor/denom reasons; connector fix does not solve inverse *for one cup* pricing semantics.
|
||||
5. **Inc3 lookback was missing until this PR:** ratification landed with #799; lookback + roadmap status update closes the governance loop before Gate A1 ratification.
|
||||
|
||||
## Loop-closure criterion (for Gate A1 readiness)
|
||||
|
||||
The post-#797 rate follow-up loop is **closed** when all of the following hold:
|
||||
|
||||
1. PR #799 merged with `"one"` connector in matcher + roundtrip + injector (**done**).
|
||||
2. Live ephemeral frontier shows `rate_with_currency` no-injection = **0** (**done** on current main).
|
||||
3. `wrong_total == 0` on train_sample proxy (**done**).
|
||||
4. This lookback + roadmap status committed; frontier tests distinguish pinned historical artifact from live post-Inc3 behavior (**this PR**).
|
||||
5. No `report.json` rebaseline unless separately ratified (**held**).
|
||||
|
||||
Gate A1 comparative ratification may proceed after this evidence-closure PR merges.
|
||||
|
||||
## Tests (evidence-closure PR adds/clarifies)
|
||||
|
||||
Focused lane (from worktree):
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest tests/test_gsm8k_frontier_report.py -q
|
||||
.venv/bin/python -m pytest tests/test_recognizer_anchor_inject.py -q
|
||||
.venv/bin/python -m pytest tests/test_math_candidate_graph_rate_injection.py -q
|
||||
.venv/bin/python -m pytest tests/test_candidate_graph_recognizer_wiring.py -q
|
||||
```
|
||||
|
||||
See PR body for exact captured outputs at commit time.
|
||||
|
||||
## Post-merge obligations
|
||||
|
||||
- Ratify Gate A1 multiplicative comparative injection (docs-only) before any comparative code.
|
||||
- Preserve wrong=0, sealed discipline, and ratify-first on all Workstream A follow-ons.
|
||||
- Do not treat pinned `report.json` rate bucket as live-state without ephemeral runner corroboration.
|
||||
|
||||
This lookback is truthful to merged #799 behavior, the stale pinned artifact, and ephemeral live measurement. No benchmark theater; no hidden rebaseline.
|
||||
|
|
@ -1,9 +1,12 @@
|
|||
"""Tests for the deterministic GSM8K frontier report analyzer (Inc 2).
|
||||
"""Tests for the deterministic GSM8K frontier report analyzer (Inc 2 + Inc3 evidence).
|
||||
|
||||
These tests pin:
|
||||
- Stable bucketing of the exact refusal reasons emitted by the candidate graph.
|
||||
- Correct extraction of category=... from "recognizer matched but produced no injection" strings.
|
||||
- rate_with_currency appears as a prominent recognized_no_injection category on the committed train-sample report (the measurement target of Inc 2).
|
||||
- **Pinned historical artifact:** committed report.json still encodes pre-Inc2/Inc3 rate
|
||||
no-injection (Inc 2 measurement target); not rebaselined by policy.
|
||||
- **Live post-Inc3 behavior:** ephemeral train_sample runner shows rate_with_currency
|
||||
no-injection = 0 after #799 connector support.
|
||||
- Fully deterministic output (sorted keys, no timestamps, repeatable across runs).
|
||||
"""
|
||||
|
||||
|
|
@ -22,8 +25,13 @@ _REPO_ROOT = Path(__file__).resolve().parents[1]
|
|||
_REPORT = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/report.json"
|
||||
|
||||
|
||||
def test_analyze_report_is_deterministic_and_has_expected_buckets():
|
||||
"""Run on the real post-Inc1 report; assert structure and rate frontier presence."""
|
||||
def test_pinned_report_json_is_historical_pre_inc3_artifact():
|
||||
"""Committed report.json is the stale Inc1-era measurement artifact.
|
||||
|
||||
Inc2 used this file to surface rate_with_currency no-injection (3 cases).
|
||||
Inc3 (#799) changed live code but did not rebaseline report.json — this test
|
||||
preserves the historical frontier pin for replayability of Inc2 evidence.
|
||||
"""
|
||||
assert _REPORT.exists(), "report.json must be present for frontier measurement"
|
||||
|
||||
summary = analyze_report(_REPORT)
|
||||
|
|
@ -37,18 +45,50 @@ def test_analyze_report_is_deterministic_and_has_expected_buckets():
|
|||
c = summary["counts"]
|
||||
assert c["correct"] == 6
|
||||
assert c["refused"] == 44
|
||||
assert c.get("wrong", 0) == 0
|
||||
assert c.get("recognized_no_injection", 0) > 0
|
||||
|
||||
# The Inc-2 target: rate_with_currency must be visible in the no-injection frontier
|
||||
# Historical Inc-2 target: rate_with_currency visible in pinned no-injection frontier
|
||||
no_inj = summary["recognized_no_injection_by_category"]
|
||||
assert "rate_with_currency" in no_inj
|
||||
assert no_inj["rate_with_currency"] >= 1 # at minimum the Tina case and peers
|
||||
assert no_inj["rate_with_currency"] == 3
|
||||
|
||||
# Determinism: re-running produces byte-identical structure (keys sorted)
|
||||
summary2 = analyze_report(_REPORT)
|
||||
assert json.dumps(summary, sort_keys=True) == json.dumps(summary2, sort_keys=True)
|
||||
|
||||
|
||||
def test_post_inc3_live_runner_has_zero_rate_no_injection():
|
||||
"""Live train_sample scoring on current code: rate bucket no longer at injector."""
|
||||
import re
|
||||
from collections import Counter
|
||||
|
||||
from evals.gsm8k_math.train_sample.v1.runner import build_report
|
||||
|
||||
cases_path = _REPO_ROOT / "evals/gsm8k_math/train_sample/v1/cases.jsonl"
|
||||
cases = [
|
||||
json.loads(line)
|
||||
for line in cases_path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
report = build_report(cases)
|
||||
|
||||
assert report["counts"]["correct"] == 6
|
||||
assert report["counts"]["refused"] == 44
|
||||
assert report["counts"]["wrong"] == 0
|
||||
|
||||
cats: Counter[str] = Counter()
|
||||
for row in report["per_case"]:
|
||||
reason = row.get("reason", "")
|
||||
if "produced no injection" not in reason:
|
||||
continue
|
||||
m = re.search(r"category=(\w+)", reason)
|
||||
if m:
|
||||
cats[m.group(1)] += 1
|
||||
|
||||
assert cats.get("rate_with_currency", 0) == 0
|
||||
|
||||
|
||||
def test_classify_and_extract_category_logic():
|
||||
"""Unit the internal classification on the exact reason strings the graph emits."""
|
||||
# We exercise via the public analyze path with a tiny synthetic report
|
||||
|
|
|
|||
Loading…
Reference in a new issue