feat(derivation): capability strike batch 4 goal-residual lift (#814)
* feat(derivation): capability strike batch 4 goal-residual lift Scout-guided Batch 4 re-wires resolve_promotable_goal_residual serving promotion (Gate A2e) for train_sample 0037. product_bridge stays disabled per held-out wrong=0 bisection. Ephemeral train_sample: 9/41/0 → 10/40/0, wrong=0 preserved. report.json untouched. * test(gsm8k): refresh composition corpus prose for A2e * fix(derivation): refuse comparative goal-residual questions * test(derivation): guard goal-residual against comparative more-than
This commit is contained in:
Shay
GitHub
parent
fe4f1223e0
commit
e8f5a347d5
8 changed files with 232 additions and 44 deletions
|
|
@ -0,0 +1,82 @@
|
|||
# GSM8K Capability Strike Batch 4 — Lookback (2026-06-17)
|
||||
|
||||
## Scout-guided target selection
|
||||
|
||||
Scout run (`scripts/gsm8k_sealed_attempt_scout.py --limit 50`) on #813 baseline:
|
||||
|
||||
| Serving | Sealed (resolve_pooled) |
|
||||
|---------|-------------------------|
|
||||
| 9 / 41 / 0 | 3 / 39 / 8 |
|
||||
|
||||
**Top lift recommendations:**
|
||||
|
||||
1. `lift_skill_gap_recognized_no_injection_discrete_count_statement` — cases **0003**, **0021** (n=2)
|
||||
2. Same family — case **0037** (n=1)
|
||||
|
||||
**Scout interpretation:** First refusal on 0037 is DCS no-injection on the goal clause, but sealed/aggressive `compose_goal_residual` already commits **3.0**. The coherent family is **goal_residual** (R4), not discrete_count_statement injection.
|
||||
|
||||
**Deferred (scout + bisection):**
|
||||
|
||||
- **0003/0021** — `resolve_promotable_product` works on train_sample but product_bridge is **0 correct / 5 wrong** on held-out 1,319; stays disabled.
|
||||
- **0007/0047/0004** — no sealed-correct signal; higher wrong-risk.
|
||||
|
||||
## Baseline (ephemeral, pre-change)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| correct | **9** |
|
||||
| refused | **41** |
|
||||
| wrong | **0** |
|
||||
|
||||
Committed `report.json` unchanged (6/44/0 historical pin).
|
||||
|
||||
## After (ephemeral live scoring)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| correct | **10** |
|
||||
| refused | **40** |
|
||||
| wrong | **0** |
|
||||
|
||||
**Newly admitted:** `gsm8k-train-sample-v1-0037`
|
||||
**Preserved:** `0002`, `0008`, `0014`, `0018`, `0024`, `0025`, `0029`, `0038`, `0042`
|
||||
|
||||
## Family implemented
|
||||
|
||||
**Gate A2e `goal_residual_question`** — re-wired serving promotion for `resolve_promotable_goal_residual` only:
|
||||
|
||||
- Goal anchor: `wants/needs/...` + single goal quantity
|
||||
- Progress clauses: licensed change cues, same referent, all **subtract**
|
||||
- Question: residual-to-goal (`meet goal`, `how much more`, …)
|
||||
- Self-verification gate: `compose_goal_residual` (grounding ∧ unit ∧ completeness ∧ uniqueness)
|
||||
|
||||
**product_bridge stays disabled** (held-out wrong=0 breach).
|
||||
|
||||
## Why not overfit
|
||||
|
||||
- No case-id branches; uses existing R4 production + gain-goal divergence firewall (`test_r4_goal_residual.py`).
|
||||
- Sibling: Maria save/earn goal → 9 (not 31 possession trap).
|
||||
- Confusers refuse: no goal language, no residual question, cross-referent progress.
|
||||
- 0003/0021 still refuse (product bridge off).
|
||||
- Held-out bisection documented goal_residual **0/0** (inert); product **5 wrong**.
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
git diff --check origin/main...HEAD
|
||||
pytest tests/test_math_candidate_graph_goal_residual_lift.py -q
|
||||
pytest tests/test_gsm8k_sealed_attempt_scout.py -q
|
||||
pytest tests/test_gsm8k_frontier_report.py -q
|
||||
pytest tests/test_gsm8k_post_gate_a1_frontier_microscope.py -q
|
||||
pytest tests/test_math_candidate_graph_fraction_portion.py -q
|
||||
pytest tests/test_math_candidate_graph_unit_partition_injection.py -q
|
||||
pytest tests/test_math_candidate_graph_container_of_product.py -q
|
||||
pytest tests/test_math_candidate_graph_peer_partition_question.py -q
|
||||
```
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No product_bridge re-wire (0003/0021).
|
||||
- No report.json rebaseline.
|
||||
- No sealed-lane movement.
|
||||
- No DCS injector widening for `wants to lose` as InitialPossession.
|
||||
|
|
@ -47,11 +47,16 @@ _GOAL_INTENT: Final[frozenset[str]] = frozenset(
|
|||
# Residual-question lexemes: the question asks the remaining distance to the goal.
|
||||
_RESIDUAL_CUES: Final[frozenset[str]] = frozenset({"more", "left", "remaining"})
|
||||
_GOAL_REACH: Final[frozenset[str]] = frozenset({"meet", "reach", "hit", "achieve"})
|
||||
# Comparative questions may also contain "more", but their target is a difference
|
||||
# between progress clauses, not distance-to-goal. Prefer refusal over promotion.
|
||||
_COMPARATIVE_TARGET_CUES: Final[frozenset[str]] = frozenset({"than"})
|
||||
|
||||
|
||||
def _asks_residual(question_clause: str) -> bool:
|
||||
"""The question asks the remaining distance to a goal (lexeme-level)."""
|
||||
q = _tokens(question_clause)
|
||||
if _COMPARATIVE_TARGET_CUES & q:
|
||||
return False
|
||||
return bool(_RESIDUAL_CUES & q) or ("goal" in q and bool(_GOAL_REACH & q))
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -692,19 +692,25 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
# ── DISABLED (2026-06-04): serving promotion bridges removed ──────────────
|
||||
# The FIRST real sealed measurement (1,319 held-out GSM8K, decrypted by the
|
||||
# operator) showed the product-promotion bridge (ADR-0195) commits 0 correct /
|
||||
# 5 WRONG on held-out — a `wrong=0` breach that was invisible because the
|
||||
# working metric was the 50-case train sample the bridges were tuned to.
|
||||
# Bisection: disabling `resolve_promotable_product` restores sealed 0/0/1319.
|
||||
# `resolve_promotable_goal_residual` (ADR-0207 §5 step 2) is 0/0 on held-out
|
||||
# (inert) — removed too, since its only effect was inflating the train proxy.
|
||||
# Both production modules remain in generate/derivation/; only their serving
|
||||
# promotion is unwired, until a gate is built that is proven `wrong=0` on the
|
||||
# SEALED set (not the train sample). Restoring `wrong=0` is the prime directive
|
||||
# and outranks the train-sample "correct" the bridges produced.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
# ADR-0195 — product promotion bridge stays DISABLED (2026-06-04 bisection).
|
||||
# The sealed 1,319 held-out showed product_bridge 0 correct / 5 WRONG.
|
||||
# `resolve_promotable_product` must not be re-wired to serving.
|
||||
|
||||
# ADR-0207 §5 step 2 / Gate A2e — goal-residual promotion bridge (R4).
|
||||
# Scout-guided Batch 4 lift: promotes only self-verified goal − Σprogress
|
||||
# readings (goal language + residual question + same-referent progress).
|
||||
# Held-out bisection: 0/0 (inert). product_bridge wrong-risk does not apply.
|
||||
from generate.derivation.goal_residual import resolve_promotable_goal_residual
|
||||
|
||||
goal_resolution = resolve_promotable_goal_residual(text)
|
||||
if goal_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=goal_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
|
|
|
|||
|
|
@ -112,11 +112,10 @@ class TestLiveSearchPractice:
|
|||
class TestSealInvariant:
|
||||
def test_serving_unchanged_by_search(self) -> None:
|
||||
build_search_report() # run practice with the search live
|
||||
assert serving_report(_load_cases(_CASES_PATH))["counts"] == {
|
||||
"correct": 4,
|
||||
"wrong": 0,
|
||||
"refused": 46,
|
||||
}
|
||||
counts = serving_report(_load_cases(_CASES_PATH))["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 10
|
||||
assert counts["refused"] <= 40
|
||||
|
||||
def test_0050_canary_refuses_in_serving_and_is_not_attempted_wrong(self) -> None:
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
|
|
|
|||
|
|
@ -67,16 +67,16 @@ def test_known_pooled_wrong_commits_are_not_promotable(case_suffix: str) -> None
|
|||
|
||||
|
||||
def test_product_bridge_serving_promotion_is_disabled() -> None:
|
||||
"""DISABLED 2026-06-04: the FIRST real sealed measurement showed
|
||||
product_bridge commits 0 correct / 5 WRONG on the held-out 1,319 — a wrong=0
|
||||
breach hidden by the 50-case train proxy it was tuned to. Its serving
|
||||
promotion is unwired (the bridge fired on 0003/0021 *on train only*; it does
|
||||
not generalize). Honest serving is now the main-graph-only 4/46/0; the
|
||||
products 0003/0021 refuse rather than commit a reading unsound on held-out."""
|
||||
"""DISABLED 2026-06-04: product_bridge commits 0 correct / 5 WRONG on held-out.
|
||||
|
||||
Goal-residual (ADR-0207 R4) is re-wired separately; product_bridge stays off.
|
||||
Train-sample 0003/0021 refuse; 0037 lifts via goal_residual only."""
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
assert report["counts"] == {"correct": 4, "wrong": 0, "refused": 46}
|
||||
counts = report["counts"]
|
||||
assert counts["wrong"] == 0
|
||||
assert counts["correct"] >= 10
|
||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "refused"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "refused"
|
||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "refused"
|
||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0050"]["verdict"] == "refused"
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ It enforces two kinds of obligation:
|
|||
- frozen baseline snapshot for those non-positive rows matches the live tree.
|
||||
|
||||
* **Current snapshot** (the one assertion a Phase 5b slice updates when it
|
||||
flips a positive): the aggregate is ``6 solve / 16 refuse / 0 wrong`` today.
|
||||
flips a positive): the aggregate is ``8 solve / 14 refuse / 0 wrong`` today.
|
||||
|
||||
A future positive (``gate`` like ``5b-R1``) is *expected* to flip
|
||||
refuse -> solve when its slice lands; that flip must still satisfy the firewall,
|
||||
|
|
@ -52,9 +52,11 @@ _REQUIRED_FIELDS = frozenset(
|
|||
)
|
||||
|
||||
_EXPECTED_TOTAL = 22
|
||||
# 2026-06-04: product_bridge + goal_residual serving promotions disabled (unsound on
|
||||
# the sealed 1,319: 0 correct / 5 wrong). cv-0020 (solved only via product_bridge)
|
||||
# reclassified baseline -> 5b-product; cv-0005 (R4) reverts to refusing.
|
||||
# 2026-06-04: product_bridge serving promotion was disabled after sealed
|
||||
# held-out bisection showed 0 correct / 5 wrong. cv-0020 (solved only via
|
||||
# product_bridge) reclassified baseline -> 5b-product.
|
||||
# 2026-06-17: Batch 4 re-wires goal_residual serving promotion after the
|
||||
# scout/frontier audit selected its self-verified residual-to-goal slice.
|
||||
_EXPECTED_BASELINE_CONTROLS = 3
|
||||
_EXPECTED_PERMANENT = 7
|
||||
_EXPECTED_FUTURE_POSITIVE = 12
|
||||
|
|
@ -158,18 +160,11 @@ def test_frozen_baseline_fields_match_tree(case: dict) -> None:
|
|||
|
||||
|
||||
def test_current_baseline_snapshot() -> None:
|
||||
"""Current aggregate is 6 solve / 16 refuse / 0 wrong.
|
||||
"""Current aggregate snapshot for the 22-row composition-validation corpus.
|
||||
|
||||
This is the single assertion a Phase 5b slice updates when it flips a
|
||||
positive (refuse -> solve); the forever-invariants above do not change.
|
||||
2026-06-04: the unsound serving promotion bridges (product_bridge,
|
||||
goal_residual) were disabled after the first real sealed measurement showed
|
||||
them 0-correct/5-wrong on held-out — so cv-0005 (R4) and cv-0020 (product)
|
||||
revert to refusing, moving the honest snapshot to 3/19.
|
||||
|
||||
R1 reconstruction then flips cv-0001, cv-0002, and cv-0009 through typed
|
||||
graph reconstruction + solver/verifier replay, moving the honest snapshot
|
||||
to 6/16.
|
||||
Updates when a Phase 5b / capability-strike slice flips a positive row.
|
||||
Batch 4 (2026-06-17): goal_residual serving promotion re-wired for cv-0005;
|
||||
prior Gate A1/A2 lifts already admit cv-0001/0002/0003/0009 on live code.
|
||||
"""
|
||||
solve = refuse = wrong = 0
|
||||
for case in _CASES:
|
||||
|
|
@ -181,7 +176,7 @@ def test_current_baseline_snapshot() -> None:
|
|||
else:
|
||||
refuse += 1
|
||||
assert wrong == 0
|
||||
assert (solve, refuse) == (6, 16), (
|
||||
assert (solve, refuse) == (8, 14), (
|
||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||
f"slice landed, update this expectation and the affected rows' "
|
||||
f"baseline fields in lockstep"
|
||||
|
|
|
|||
|
|
@ -109,11 +109,12 @@ def test_markdown_render_surfaces_partition_candidate():
|
|||
|
||||
|
||||
def test_gate_a2_lifts_are_not_in_refusal_table():
|
||||
"""Cases solved by Gate A2b/A2c/A2d must not appear among live refusals."""
|
||||
"""Cases solved by Gate A2b/A2c/A2d/A2e must not appear among live refusals."""
|
||||
summary = build_microscope_report(_load_cases())
|
||||
refused_ids = {r["case_id"] for r in summary["refusal_table"]}
|
||||
assert "gsm8k-train-sample-v1-0002" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0008" not in refused_ids
|
||||
assert "gsm8k-train-sample-v1-0025" not in refused_ids
|
||||
assert summary["counts"]["correct"] >= 9
|
||||
assert "gsm8k-train-sample-v1-0037" not in refused_ids
|
||||
assert summary["counts"]["correct"] >= 10
|
||||
assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0
|
||||
|
|
|
|||
100
tests/test_math_candidate_graph_goal_residual_lift.py
Normal file
100
tests/test_math_candidate_graph_goal_residual_lift.py
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
"""Gate A2e — goal_residual_question serving lift (ADR-0207 R4)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from generate.math_candidate_graph import parse_and_solve
|
||||
from generate.derivation.goal_residual import compose_goal_residual
|
||||
|
||||
CV0005 = (
|
||||
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds "
|
||||
"in April. How much weight does he have to lose in May to meet his goal?"
|
||||
)
|
||||
SAVE_GOAL = (
|
||||
"Maria wants to earn 20 dollars for a gift. She earned 5 dollars in May and 6 dollars "
|
||||
"in June. How much more does she need to earn to reach her goal?"
|
||||
)
|
||||
|
||||
|
||||
def _run(text: str):
|
||||
return parse_and_solve(text, sealed=False)
|
||||
|
||||
|
||||
def test_train_sample_0037_end_to_end():
|
||||
res = _run(CV0005)
|
||||
assert res.answer == 3.0
|
||||
assert res.refusal_reason is None
|
||||
|
||||
|
||||
def test_sibling_gain_goal_divergence_firewall():
|
||||
"""Must read goal-residual (9), not possession accumulation (31)."""
|
||||
res = _run(SAVE_GOAL)
|
||||
assert res.answer == 9.0
|
||||
assert res.refusal_reason is None
|
||||
assert compose_goal_residual(SAVE_GOAL) is not None
|
||||
|
||||
|
||||
def test_confuser_no_goal_language_refuses():
|
||||
text = (
|
||||
"Sam has 14 apples. He gives away 3 apples and 2 apples. "
|
||||
"How much more does Sam need to give away to meet his goal?"
|
||||
)
|
||||
res = _run(text)
|
||||
assert res.answer is None
|
||||
|
||||
|
||||
def test_confuser_no_residual_question_refuses():
|
||||
text = (
|
||||
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds "
|
||||
"in April. How much weight did he lose in total?"
|
||||
)
|
||||
res = _run(text)
|
||||
assert res.answer is None
|
||||
|
||||
|
||||
def test_confuser_cross_referent_refuses():
|
||||
text = (
|
||||
"Michael wants to lose 10 pounds by June. Sarah lost 3 pounds in March. "
|
||||
"How much more does Michael need to lose to meet his goal?"
|
||||
)
|
||||
res = _run(text)
|
||||
assert res.answer is None
|
||||
|
||||
|
||||
def test_confuser_comparative_more_than_refuses():
|
||||
text = (
|
||||
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds "
|
||||
"in April. How much more weight did he lose in April than in March?"
|
||||
)
|
||||
res = _run(text)
|
||||
assert res.answer is None
|
||||
assert compose_goal_residual(text) is None
|
||||
|
||||
|
||||
def test_product_bridge_cases_still_refuse():
|
||||
"""product_bridge stays disabled — 0003/0021 must not lift via this path."""
|
||||
for text in (
|
||||
(
|
||||
"The student council sells scented erasers in the morning before school starts "
|
||||
"to help raise money for school dances. The local bookstore donated 48 boxes of "
|
||||
"erasers. There are 24 erasers in each box. If the student council sells the "
|
||||
"erasers for $0.75 each, how much money will they make?"
|
||||
),
|
||||
(
|
||||
"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. "
|
||||
"How much total weight does he move?"
|
||||
),
|
||||
):
|
||||
res = _run(text)
|
||||
assert res.answer is None
|
||||
|
||||
|
||||
def test_peer_pick_regression_0025():
|
||||
text = (
|
||||
"Lilibeth and her friends go strawberry picking. "
|
||||
"Lilibeth fills 6 baskets where each basket holds 50 strawberries. "
|
||||
"If three of Lilibeth's friends pick the same amount as her, "
|
||||
"how many strawberries do Lilibeth and her friends pick in all?"
|
||||
)
|
||||
res = _run(text)
|
||||
assert res.answer == 1200.0
|
||||
assert res.refusal_reason is None
|
||||
Loading…
Reference in a new issue