feat(derivation): capability paradigm sprint 5 question-bound product lift
Practice/scout showed 0003/0021 refuse on DCS no-injection while sealed resolve_pooled commits correctly. Build Gate A2f typed revenue/weight product organs narrower than disabled product_bridge; lifts train_sample 10/40/0 to 12/38/0 with wrong=0 and zero holdout_dev admissions.
This commit is contained in:
parent
e8f5a347d5
commit
f2b21c3eeb
9 changed files with 669 additions and 29 deletions
|
|
@ -0,0 +1,131 @@
|
||||||
|
# GSM8K Capability Paradigm Sprint 5 — Lookback (2026-06-17)
|
||||||
|
|
||||||
|
## 1. Starting baseline
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|------:|
|
||||||
|
| correct | **10** |
|
||||||
|
| refused | **40** |
|
||||||
|
| wrong | **0** |
|
||||||
|
|
||||||
|
After #814 (Gate A2e goal_residual). Preserved solved: 0002, 0008, 0014, 0018, 0024, 0025, 0029, 0037, 0038, 0042.
|
||||||
|
|
||||||
|
## 2. Scout / practice evidence
|
||||||
|
|
||||||
|
Scout (`scripts/gsm8k_sealed_attempt_scout.py --limit 50`):
|
||||||
|
|
||||||
|
| Regime | correct | refused | wrong |
|
||||||
|
|--------|--------:|--------:|------:|
|
||||||
|
| Serving | 10 | 40 | 0 |
|
||||||
|
| Sealed (resolve_pooled) | 3 | 39 | 8 |
|
||||||
|
|
||||||
|
**Top lift cluster:** `lift_skill_gap_recognized_no_injection_discrete_count_statement` — cases **0003**, **0021** (n=2). Sealed commits 864 and 450 via `relation_hypothesis:discrete_count_statement`; serving stops at DCS no-injection.
|
||||||
|
|
||||||
|
**Elimination signal (sealed wrong, serving refused):** 0011 (3200 vs 50), 0047 (240 vs 45), plus 6 `elimination_refused_to_wrong` — not promotable.
|
||||||
|
|
||||||
|
## 3. Candidate family clusters
|
||||||
|
|
||||||
|
| family_bucket | case_ids | serving failure | sealed behavior | missing primitive | expected lift | wrong-risk |
|
||||||
|
|---------------|----------|-----------------|-----------------|-------------------|---------------|------------|
|
||||||
|
| discrete_count_statement → product | 0003, 0021 | DCS no-injection | sealed correct | typed product chain | +2 | medium via product_bridge |
|
||||||
|
| goal_residual (served) | 0037 | — | already served | — | 0 | low |
|
||||||
|
| joint_sealed_no_resolution | 32 | parse/injection | refused | various | 0 | low |
|
||||||
|
| sealed_elimination | 0011, 0047, … | refused | sealed wrong | not product | 0 | high |
|
||||||
|
|
||||||
|
## 4. Why selected paradigm
|
||||||
|
|
||||||
|
**`question_bound_product_aggregate` (Gate A2f)** — practice proved the reasoning chain is valid for 0003/0021, but broad `product_bridge` over `resolve_pooled` admitted held-out wrong (e.g. holdout-dev 0345: fraction surface + `total weight` → 6720 vs 595).
|
||||||
|
|
||||||
|
Built a **first-principles organ** (not a pooled filter):
|
||||||
|
|
||||||
|
1. **Revenue chain:** container leaf × `in each` per-container leaf × `$`/`dollar`+`each` price (price may live in question clause); question binds `money` + `make|earn|raise|collect`.
|
||||||
|
2. **Weight-work chain:** single clause with exactly 3 distinct units, mass anchor, `for` cue, `search_chain` multiply-only; question binds `weight` + `total|move|lift|press`.
|
||||||
|
|
||||||
|
## 5. Rejected candidates
|
||||||
|
|
||||||
|
| Candidate | Why unsafe / lower leverage |
|
||||||
|
|-----------|----------------------------|
|
||||||
|
| Re-wire `product_bridge` | Held-out 5 wrong; admits fraction/comparative `total weight` surfaces |
|
||||||
|
| Broad DCS injection | Would multiply arbitrary co-occurring numbers |
|
||||||
|
| 0007 / 0047 / 0004 chunk | No sealed-correct signal; sealed wrong on 0047 |
|
||||||
|
| `resolve_pooled` wholesale | 17% wrong on holdout_dev per 2026-06-04 measurement |
|
||||||
|
|
||||||
|
## 6. Typed paradigm implemented
|
||||||
|
|
||||||
|
Module: `generate/derivation/question_bound_product.py`
|
||||||
|
|
||||||
|
- `build_question_bound_product()` — typed chain construction
|
||||||
|
- `compose_question_bound_product()` — self-verification gate
|
||||||
|
- `resolve_promotable_question_bound_product()` — serving promotion (Gate A2f)
|
||||||
|
- Wired in `generate/math_candidate_graph.py` after goal_residual, before rate short-circuits
|
||||||
|
|
||||||
|
## 7. Before / after score
|
||||||
|
|
||||||
|
| | Before | After |
|
||||||
|
|---|------:|------:|
|
||||||
|
| correct | 10 | **12** |
|
||||||
|
| refused | 40 | **38** |
|
||||||
|
| wrong | 0 | **0** |
|
||||||
|
|
||||||
|
## 8. Newly solved IDs
|
||||||
|
|
||||||
|
- `gsm8k-train-sample-v1-0003`
|
||||||
|
- `gsm8k-train-sample-v1-0021`
|
||||||
|
|
||||||
|
## 9. Preserved solved IDs
|
||||||
|
|
||||||
|
0002, 0008, 0014, 0018, 0024, 0025, 0029, 0037, 0038, 0042
|
||||||
|
|
||||||
|
## 10. Wrong=0 proof
|
||||||
|
|
||||||
|
Full ephemeral train_sample: `12 / 38 / 0`. Holdout_dev scan: **0 admissions** for `resolve_promotable_question_bound_product`.
|
||||||
|
|
||||||
|
## 11. Confuser matrix
|
||||||
|
|
||||||
|
| Confuser | Result |
|
||||||
|
|----------|--------|
|
||||||
|
| Fraction surface (holdout 0345 shape) | refuse |
|
||||||
|
| Tom donates / Sam sells | refuse (seller≠supplier) |
|
||||||
|
| Additive distractor (loose pens) | refuse |
|
||||||
|
| Rate without money/weight target | refuse |
|
||||||
|
| `more than` comparative | refuse |
|
||||||
|
| Revenue without `in each` | refuse |
|
||||||
|
| Weight without `for` cue | refuse |
|
||||||
|
| Known product_bridge held-out hazard | refuse |
|
||||||
|
|
||||||
|
## 12. Sibling generality proof
|
||||||
|
|
||||||
|
- Revenue sibling: 12 cartons × 30 in each × $1.25 → 450
|
||||||
|
- Weight sibling: 20 kg × 8 reps × 4 sets → 640
|
||||||
|
- No case-id logic; no hardcoded answers
|
||||||
|
|
||||||
|
## 13. product_bridge status
|
||||||
|
|
||||||
|
**Broad `product_bridge` stays disabled** in `math_candidate_graph.py`. Function still resolves 0003/0021 in isolation; serving uses Gate A2f only.
|
||||||
|
|
||||||
|
## 14–19. Confirmations
|
||||||
|
|
||||||
|
- `report.json` untouched (historical 6/44/0 pin)
|
||||||
|
- Sealed-lane artifacts untouched
|
||||||
|
- No case-id logic
|
||||||
|
- No hardcoded final answers
|
||||||
|
- No direct-answer extraction
|
||||||
|
- **Non-goals:** broad product_bridge, report rebaseline, sealed movement, determine()/CLOSE, generic multiplication of co-occurring numbers
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git diff --check origin/main
|
||||||
|
pytest tests/test_math_candidate_graph_question_bound_product_lift.py -q
|
||||||
|
pytest tests/test_gsm8k_sealed_attempt_scout.py -q
|
||||||
|
pytest tests/test_gsm8k_frontier_report.py -q
|
||||||
|
pytest tests/test_gsm8k_post_gate_a1_frontier_microscope.py -q
|
||||||
|
pytest tests/test_math_candidate_graph_fraction_portion.py -q
|
||||||
|
pytest tests/test_math_candidate_graph_unit_partition_injection.py -q
|
||||||
|
pytest tests/test_math_candidate_graph_container_of_product.py -q
|
||||||
|
pytest tests/test_math_candidate_graph_peer_partition_question.py -q
|
||||||
|
pytest tests/test_math_candidate_graph_goal_residual_lift.py -q
|
||||||
|
pytest tests/test_adr_0195_product_bridge.py -q
|
||||||
|
pytest tests/test_composition_validation_corpus.py -q
|
||||||
|
# 151 passed
|
||||||
|
```
|
||||||
309
generate/derivation/question_bound_product.py
Normal file
309
generate/derivation/question_bound_product.py
Normal file
|
|
@ -0,0 +1,309 @@
|
||||||
|
"""Gate A2f — question-bound product aggregate (typed cross-clause / in-clause chain).
|
||||||
|
|
||||||
|
Practice/scout evidence (Batch 5) showed train_sample **0003** and **0021** refuse on
|
||||||
|
serving because discrete_count_statement injection stops before a product chain can
|
||||||
|
commit, while sealed ``resolve_pooled`` already derives the correct readings. The
|
||||||
|
broad ``product_bridge`` wrapper over ``resolve_pooled`` was disabled after it
|
||||||
|
committed **5 wrong** on the sealed 1,319 — it admitted structurally similar but
|
||||||
|
non-product problems (fraction/comparative surfaces with a ``total weight`` target).
|
||||||
|
|
||||||
|
This module is a **first-principles organ**, not a filter over the pooled reader.
|
||||||
|
It constructs only two explicit shapes:
|
||||||
|
|
||||||
|
1. **Revenue chain** — container count × per-container count (``in each``) × unit
|
||||||
|
price (``$``/``dollar`` + ``each``, price may live in the question clause).
|
||||||
|
2. **Weight-work chain** — mass-per-rep × reps × sets in a **single** statement
|
||||||
|
clause, licensed by ``for``, question asks ``total``/``move`` + ``weight``.
|
||||||
|
|
||||||
|
Promotion requires question-target binding, distinct non-empty units across factors
|
||||||
|
(where known), hazard refusal (fractions, comparatives, residual/profit cues), and
|
||||||
|
the unchanged self-verification gate (grounding ∧ cue ∧ unit ∧ completeness ∧
|
||||||
|
uniqueness). Deterministic; sealed module (no ``chat/`` import).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
from generate.derivation.clauses import segment_clauses
|
||||||
|
from generate.derivation.extract import extract_quantities
|
||||||
|
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||||
|
from generate.derivation.multistep import search_chain
|
||||||
|
from generate.derivation.state.bind import leading_subject_token
|
||||||
|
from generate.derivation.target import _question_clause
|
||||||
|
from generate.derivation.verify import Resolution, select_self_verified
|
||||||
|
from generate.math_roundtrip import _tokens
|
||||||
|
|
||||||
|
_FRACTION_RE: Final[re.Pattern[str]] = re.compile(r"\d+/\d+")
|
||||||
|
_COMPARATIVE_CUES: Final[frozenset[str]] = frozenset(
|
||||||
|
{"twice", "thrice", "double", "doubled", "half", "triple", "tripled"}
|
||||||
|
)
|
||||||
|
_MASS_UNITS: Final[frozenset[str]] = frozenset(
|
||||||
|
{
|
||||||
|
"gram",
|
||||||
|
"grams",
|
||||||
|
"kg",
|
||||||
|
"kilogram",
|
||||||
|
"kilograms",
|
||||||
|
"ounce",
|
||||||
|
"ounces",
|
||||||
|
"pound",
|
||||||
|
"pounds",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_REVENUE_QUESTION_VERBS: Final[frozenset[str]] = frozenset(
|
||||||
|
{"make", "earn", "raise", "collect"}
|
||||||
|
)
|
||||||
|
_WEIGHT_QUESTION_VERBS: Final[frozenset[str]] = frozenset(
|
||||||
|
{"total", "move", "lift", "press"}
|
||||||
|
)
|
||||||
|
_TEXT_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||||
|
{
|
||||||
|
"back",
|
||||||
|
"cover",
|
||||||
|
"covered",
|
||||||
|
"covers",
|
||||||
|
"eat",
|
||||||
|
"eats",
|
||||||
|
"gave",
|
||||||
|
"give",
|
||||||
|
"given",
|
||||||
|
"gives",
|
||||||
|
"insurance",
|
||||||
|
"kept",
|
||||||
|
"left",
|
||||||
|
"less",
|
||||||
|
"long",
|
||||||
|
"more",
|
||||||
|
"profit",
|
||||||
|
"remain",
|
||||||
|
"remaining",
|
||||||
|
"rest",
|
||||||
|
"same",
|
||||||
|
"spent",
|
||||||
|
"spend",
|
||||||
|
"subsequent",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_QUESTION_BLOCKERS: Final[frozenset[str]] = frozenset(
|
||||||
|
{
|
||||||
|
"after",
|
||||||
|
"before",
|
||||||
|
"left",
|
||||||
|
"long",
|
||||||
|
"per",
|
||||||
|
"profit",
|
||||||
|
"remaining",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_revenue_total(question_clause: str) -> bool:
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
return "money" in tokens and bool(_REVENUE_QUESTION_VERBS & tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def _asks_weight_total(question_clause: str) -> bool:
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
return "weight" in tokens and bool(_WEIGHT_QUESTION_VERBS & tokens)
|
||||||
|
|
||||||
|
|
||||||
|
def _has_hazard_surface(problem_text: str, question_clause: str) -> bool:
|
||||||
|
if _FRACTION_RE.search(problem_text):
|
||||||
|
return True
|
||||||
|
text_tokens = _tokens(problem_text)
|
||||||
|
question_tokens = _tokens(question_clause)
|
||||||
|
if _COMPARATIVE_CUES & text_tokens:
|
||||||
|
return True
|
||||||
|
if "%" in problem_text or "percent" in text_tokens or "percentage" in text_tokens:
|
||||||
|
return True
|
||||||
|
if text_tokens & _TEXT_BLOCKERS:
|
||||||
|
return True
|
||||||
|
if question_tokens & _QUESTION_BLOCKERS:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _distinct_known_units(derivation: GroundedDerivation) -> bool:
|
||||||
|
units = [derivation.start.unit, *(step.operand.unit for step in derivation.steps)]
|
||||||
|
non_empty = [unit for unit in units if unit]
|
||||||
|
return len(non_empty) == len(set(non_empty))
|
||||||
|
|
||||||
|
|
||||||
|
_IMPERSONAL_SUBJECTS: Final[frozenset[str]] = frozenset(
|
||||||
|
{
|
||||||
|
"a",
|
||||||
|
"an",
|
||||||
|
"each",
|
||||||
|
"every",
|
||||||
|
"if",
|
||||||
|
"it",
|
||||||
|
"the",
|
||||||
|
"there",
|
||||||
|
"this",
|
||||||
|
"that",
|
||||||
|
"when",
|
||||||
|
"while",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _named_actor_subjects(problem_text: str) -> set[str]:
|
||||||
|
"""Collect likely named-actor subjects from quantity-bearing clauses."""
|
||||||
|
subjects: set[str] = set()
|
||||||
|
for clause in segment_clauses(problem_text):
|
||||||
|
if not extract_quantities(clause):
|
||||||
|
continue
|
||||||
|
subject = leading_subject_token(clause)
|
||||||
|
if subject is None:
|
||||||
|
continue
|
||||||
|
key = subject.lower()
|
||||||
|
if key in _IMPERSONAL_SUBJECTS:
|
||||||
|
continue
|
||||||
|
subjects.add(key)
|
||||||
|
return subjects
|
||||||
|
|
||||||
|
|
||||||
|
_SELL_VERBS: Final[frozenset[str]] = frozenset({"sell", "sells", "sold", "selling"})
|
||||||
|
|
||||||
|
|
||||||
|
def _question_named_seller(question_clause: str) -> str | None:
|
||||||
|
"""Named seller in a conditional question clause, if present."""
|
||||||
|
tokens = _tokens(question_clause)
|
||||||
|
if "if" not in tokens:
|
||||||
|
return None
|
||||||
|
if not (_SELL_VERBS & tokens):
|
||||||
|
return None
|
||||||
|
# ``If Sam sells ...`` — the token immediately after ``if``.
|
||||||
|
token_list = question_clause.split()
|
||||||
|
for index, word in enumerate(token_list):
|
||||||
|
if word.lower() == "if" and index + 1 < len(token_list):
|
||||||
|
candidate = token_list[index + 1].strip("',.")
|
||||||
|
key = candidate.lower()
|
||||||
|
if key not in _IMPERSONAL_SUBJECTS:
|
||||||
|
return key
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _has_conflicting_actor_subjects(problem_text: str) -> bool:
|
||||||
|
"""Refuse when named suppliers and sellers disagree across clauses."""
|
||||||
|
question_clause = _question_clause(problem_text)
|
||||||
|
statement_actors = set()
|
||||||
|
for clause in segment_clauses(problem_text):
|
||||||
|
if clause == question_clause:
|
||||||
|
continue
|
||||||
|
statement_actors |= _named_actor_subjects(clause)
|
||||||
|
seller = _question_named_seller(question_clause)
|
||||||
|
if seller is not None and statement_actors and seller not in statement_actors:
|
||||||
|
return True
|
||||||
|
all_actors = _named_actor_subjects(problem_text)
|
||||||
|
return len(all_actors) > 1
|
||||||
|
|
||||||
|
|
||||||
|
def _build_revenue_product_chain(problem_text: str) -> GroundedDerivation | None:
|
||||||
|
"""Container × per-container (``in each``) × unit price — revenue target only."""
|
||||||
|
question_clause = _question_clause(problem_text)
|
||||||
|
if not _asks_revenue_total(question_clause):
|
||||||
|
return None
|
||||||
|
|
||||||
|
factors: list[Quantity] = []
|
||||||
|
in_each_seen = False
|
||||||
|
price_seen = False
|
||||||
|
|
||||||
|
for clause in segment_clauses(problem_text):
|
||||||
|
if clause == question_clause:
|
||||||
|
continue
|
||||||
|
quantities = extract_quantities(clause)
|
||||||
|
if len(quantities) != 1:
|
||||||
|
continue
|
||||||
|
factors.append(quantities[0])
|
||||||
|
if "in each" in clause.lower():
|
||||||
|
in_each_seen = True
|
||||||
|
|
||||||
|
question_quantities = extract_quantities(question_clause)
|
||||||
|
if (
|
||||||
|
len(question_quantities) == 1
|
||||||
|
and ("$" in question_clause or "dollar" in _tokens(question_clause))
|
||||||
|
and "each" in _tokens(question_clause)
|
||||||
|
):
|
||||||
|
factors.append(question_quantities[0])
|
||||||
|
price_seen = True
|
||||||
|
|
||||||
|
if len(factors) != 3 or not in_each_seen or not price_seen:
|
||||||
|
return None
|
||||||
|
|
||||||
|
start, *rest = factors
|
||||||
|
return GroundedDerivation(
|
||||||
|
start=start,
|
||||||
|
steps=tuple(Step(op="multiply", operand=q, cue="each") for q in rest),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_weight_product_chain(problem_text: str) -> GroundedDerivation | None:
|
||||||
|
"""Mass-per-rep × reps × sets in one clause — total-weight target only."""
|
||||||
|
question_clause = _question_clause(problem_text)
|
||||||
|
if not _asks_weight_total(question_clause):
|
||||||
|
return None
|
||||||
|
|
||||||
|
statement_clauses = [
|
||||||
|
clause
|
||||||
|
for clause in segment_clauses(problem_text)
|
||||||
|
if clause != question_clause and not clause.rstrip().endswith("?")
|
||||||
|
]
|
||||||
|
quantity_clauses = [
|
||||||
|
clause for clause in statement_clauses if len(extract_quantities(clause)) >= 2
|
||||||
|
]
|
||||||
|
if len(quantity_clauses) != 1:
|
||||||
|
return None
|
||||||
|
|
||||||
|
clause = quantity_clauses[0]
|
||||||
|
quantities = extract_quantities(clause)
|
||||||
|
if len(quantities) != 3:
|
||||||
|
return None
|
||||||
|
if not (_MASS_UNITS & {quantities[0].unit}):
|
||||||
|
return None
|
||||||
|
if "for" not in _tokens(clause):
|
||||||
|
return None
|
||||||
|
|
||||||
|
resolution = search_chain(clause)
|
||||||
|
if resolution is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
derivation = resolution.derivation
|
||||||
|
if not all(step.op == "multiply" and step.cue == "for" for step in derivation.steps):
|
||||||
|
return None
|
||||||
|
if not _distinct_known_units(derivation):
|
||||||
|
return None
|
||||||
|
return derivation
|
||||||
|
|
||||||
|
|
||||||
|
def build_question_bound_product(problem_text: str) -> GroundedDerivation | None:
|
||||||
|
"""Construct the ungated question-bound product chain, or ``None``."""
|
||||||
|
question_clause = _question_clause(problem_text)
|
||||||
|
if _has_hazard_surface(problem_text, question_clause):
|
||||||
|
return None
|
||||||
|
if _has_conflicting_actor_subjects(problem_text):
|
||||||
|
return None
|
||||||
|
return _build_revenue_product_chain(problem_text) or _build_weight_product_chain(
|
||||||
|
problem_text
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def compose_question_bound_product(problem_text: str) -> Resolution | None:
|
||||||
|
"""Gate the typed product chain through self-verification."""
|
||||||
|
derivation = build_question_bound_product(problem_text)
|
||||||
|
if derivation is None:
|
||||||
|
return None
|
||||||
|
return select_self_verified([derivation], problem_text, target_units=())
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_promotable_question_bound_product(problem_text: str) -> Resolution | None:
|
||||||
|
"""Serving promotion bridge (Gate A2f).
|
||||||
|
|
||||||
|
Narrower than :func:`generate.derivation.product_bridge.resolve_promotable_product`:
|
||||||
|
does not call ``resolve_pooled``; admits only the two typed shapes above with
|
||||||
|
explicit question binding and hazard refusal. Held-out dev scan (2026-06-17):
|
||||||
|
0 admissions on 500 cases; train_sample lifts 0003 + 0021 only.
|
||||||
|
"""
|
||||||
|
return compose_question_bound_product(problem_text)
|
||||||
|
|
@ -712,6 +712,23 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
||||||
branches_admissible=1,
|
branches_admissible=1,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Gate A2f — question-bound product aggregate (typed revenue/weight chains).
|
||||||
|
# Paradigm Sprint 5: practice/scout lift for 0003/0021 without re-wiring the
|
||||||
|
# broad product_bridge over resolve_pooled (sealed 1,319 had 5 wrong).
|
||||||
|
from generate.derivation.question_bound_product import (
|
||||||
|
resolve_promotable_question_bound_product,
|
||||||
|
)
|
||||||
|
|
||||||
|
product_resolution = resolve_promotable_question_bound_product(text)
|
||||||
|
if product_resolution is not None:
|
||||||
|
return CandidateGraphResult(
|
||||||
|
answer=product_resolution.answer,
|
||||||
|
selected_graph=None,
|
||||||
|
refusal_reason=None,
|
||||||
|
branches_enumerated=1,
|
||||||
|
branches_admissible=1,
|
||||||
|
)
|
||||||
|
|
||||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||||
if len(statement_sentences) == 1:
|
if len(statement_sentences) == 1:
|
||||||
|
|
|
||||||
|
|
@ -114,8 +114,8 @@ class TestSealInvariant:
|
||||||
build_search_report() # run practice with the search live
|
build_search_report() # run practice with the search live
|
||||||
counts = serving_report(_load_cases(_CASES_PATH))["counts"]
|
counts = serving_report(_load_cases(_CASES_PATH))["counts"]
|
||||||
assert counts["wrong"] == 0
|
assert counts["wrong"] == 0
|
||||||
assert counts["correct"] >= 10
|
assert counts["correct"] >= 12
|
||||||
assert counts["refused"] <= 40
|
assert counts["refused"] <= 38
|
||||||
|
|
||||||
def test_0050_canary_refuses_in_serving_and_is_not_attempted_wrong(self) -> None:
|
def test_0050_canary_refuses_in_serving_and_is_not_attempted_wrong(self) -> None:
|
||||||
from generate.math_candidate_graph import parse_and_solve
|
from generate.math_candidate_graph import parse_and_solve
|
||||||
|
|
|
||||||
|
|
@ -41,11 +41,10 @@ def test_promotable_product_cases_resolve(case_suffix: str, expected: float) ->
|
||||||
assert resolution is not None
|
assert resolution is not None
|
||||||
assert resolution.answer == expected
|
assert resolution.answer == expected
|
||||||
|
|
||||||
# ...but its SERVING promotion is DISABLED (2026-06-04): the bridge committed
|
# ...but broad product_bridge SERVING promotion stays DISABLED (2026-06-04).
|
||||||
# 0 correct / 5 WRONG on the sealed 1,319, so serving now REFUSES rather than
|
# Gate A2f question_bound_product lifts 0003/0021 via a narrower typed organ.
|
||||||
# promote a product reading unsound on held-out. wrong=0 over train "correct".
|
|
||||||
result = parse_and_solve(row["question"])
|
result = parse_and_solve(row["question"])
|
||||||
assert result.answer is None
|
assert result.answer == expected
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
|
|
@ -67,16 +66,13 @@ def test_known_pooled_wrong_commits_are_not_promotable(case_suffix: str) -> None
|
||||||
|
|
||||||
|
|
||||||
def test_product_bridge_serving_promotion_is_disabled() -> None:
|
def test_product_bridge_serving_promotion_is_disabled() -> None:
|
||||||
"""DISABLED 2026-06-04: product_bridge commits 0 correct / 5 WRONG on held-out.
|
"""Broad product_bridge stays off; Gate A2f typed organ promotes 0003/0021."""
|
||||||
|
|
||||||
Goal-residual (ADR-0207 R4) is re-wired separately; product_bridge stays off.
|
|
||||||
Train-sample 0003/0021 refuse; 0037 lifts via goal_residual only."""
|
|
||||||
report = build_report(_load_cases(_CASES_PATH))
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
counts = report["counts"]
|
counts = report["counts"]
|
||||||
assert counts["wrong"] == 0
|
assert counts["wrong"] == 0
|
||||||
assert counts["correct"] >= 10
|
assert counts["correct"] >= 12
|
||||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "refused"
|
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "refused"
|
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "correct"
|
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "correct"
|
||||||
assert by_case["gsm8k-train-sample-v1-0050"]["verdict"] == "refused"
|
assert by_case["gsm8k-train-sample-v1-0050"]["verdict"] == "refused"
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ It enforces two kinds of obligation:
|
||||||
- frozen baseline snapshot for those non-positive rows matches the live tree.
|
- frozen baseline snapshot for those non-positive rows matches the live tree.
|
||||||
|
|
||||||
* **Current snapshot** (the one assertion a Phase 5b slice updates when it
|
* **Current snapshot** (the one assertion a Phase 5b slice updates when it
|
||||||
flips a positive): the aggregate is ``8 solve / 14 refuse / 0 wrong`` today.
|
flips a positive): the aggregate is ``9 solve / 13 refuse / 0 wrong`` today.
|
||||||
|
|
||||||
A future positive (``gate`` like ``5b-R1``) is *expected* to flip
|
A future positive (``gate`` like ``5b-R1``) is *expected* to flip
|
||||||
refuse -> solve when its slice lands; that flip must still satisfy the firewall,
|
refuse -> solve when its slice lands; that flip must still satisfy the firewall,
|
||||||
|
|
@ -163,8 +163,8 @@ def test_current_baseline_snapshot() -> None:
|
||||||
"""Current aggregate snapshot for the 22-row composition-validation corpus.
|
"""Current aggregate snapshot for the 22-row composition-validation corpus.
|
||||||
|
|
||||||
Updates when a Phase 5b / capability-strike slice flips a positive row.
|
Updates when a Phase 5b / capability-strike slice flips a positive row.
|
||||||
Batch 4 (2026-06-17): goal_residual serving promotion re-wired for cv-0005;
|
Batch 5 (2026-06-17): Gate A2f question_bound_product admits cv-0020 (0021);
|
||||||
prior Gate A1/A2 lifts already admit cv-0001/0002/0003/0009 on live code.
|
Batch 4 goal_residual admits cv-0005; prior Gate A1/A2 lifts admit others.
|
||||||
"""
|
"""
|
||||||
solve = refuse = wrong = 0
|
solve = refuse = wrong = 0
|
||||||
for case in _CASES:
|
for case in _CASES:
|
||||||
|
|
@ -176,7 +176,7 @@ def test_current_baseline_snapshot() -> None:
|
||||||
else:
|
else:
|
||||||
refuse += 1
|
refuse += 1
|
||||||
assert wrong == 0
|
assert wrong == 0
|
||||||
assert (solve, refuse) == (8, 14), (
|
assert (solve, refuse) == (9, 13), (
|
||||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||||
f"slice landed, update this expectation and the affected rows' "
|
f"slice landed, update this expectation and the affected rows' "
|
||||||
f"baseline fields in lockstep"
|
f"baseline fields in lockstep"
|
||||||
|
|
|
||||||
|
|
@ -75,11 +75,11 @@ def test_live_microscope_refusal_partition_is_complete():
|
||||||
|
|
||||||
|
|
||||||
def test_live_microscope_partition_seed_case_is_tagged():
|
def test_live_microscope_partition_seed_case_is_tagged():
|
||||||
|
"""0003 is solved by Gate A2f; partition_chunking still tags refused peers."""
|
||||||
summary = build_microscope_report(_load_cases())
|
summary = build_microscope_report(_load_cases())
|
||||||
assert (
|
partition_ids = summary["implementation_slice_candidates"]["partition_chunking"]["case_ids"]
|
||||||
"gsm8k-train-sample-v1-0003"
|
assert "gsm8k-train-sample-v1-0003" not in {r["case_id"] for r in summary["refusal_table"]}
|
||||||
in summary["implementation_slice_candidates"]["partition_chunking"]["case_ids"]
|
assert len(partition_ids) >= 1
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_microscope_output_is_deterministic():
|
def test_microscope_output_is_deterministic():
|
||||||
|
|
@ -104,17 +104,19 @@ def test_markdown_render_surfaces_partition_candidate():
|
||||||
summary = build_microscope_report(_load_cases())
|
summary = build_microscope_report(_load_cases())
|
||||||
md = render_markdown(summary)
|
md = render_markdown(summary)
|
||||||
assert "partition_chunking" in md
|
assert "partition_chunking" in md
|
||||||
assert "| 0003 |" in md
|
assert "correct: 12" in md
|
||||||
assert "Gate A2a unit_partition" in md
|
assert "Gate A2a unit_partition" in md
|
||||||
|
|
||||||
|
|
||||||
def test_gate_a2_lifts_are_not_in_refusal_table():
|
def test_gate_a2_lifts_are_not_in_refusal_table():
|
||||||
"""Cases solved by Gate A2b/A2c/A2d/A2e must not appear among live refusals."""
|
"""Cases solved by Gate A2b–A2f must not appear among live refusals."""
|
||||||
summary = build_microscope_report(_load_cases())
|
summary = build_microscope_report(_load_cases())
|
||||||
refused_ids = {r["case_id"] for r in summary["refusal_table"]}
|
refused_ids = {r["case_id"] for r in summary["refusal_table"]}
|
||||||
assert "gsm8k-train-sample-v1-0002" not in refused_ids
|
assert "gsm8k-train-sample-v1-0002" not in refused_ids
|
||||||
|
assert "gsm8k-train-sample-v1-0003" not in refused_ids
|
||||||
assert "gsm8k-train-sample-v1-0008" not in refused_ids
|
assert "gsm8k-train-sample-v1-0008" not in refused_ids
|
||||||
|
assert "gsm8k-train-sample-v1-0021" not in refused_ids
|
||||||
assert "gsm8k-train-sample-v1-0025" not in refused_ids
|
assert "gsm8k-train-sample-v1-0025" not in refused_ids
|
||||||
assert "gsm8k-train-sample-v1-0037" not in refused_ids
|
assert "gsm8k-train-sample-v1-0037" not in refused_ids
|
||||||
assert summary["counts"]["correct"] >= 10
|
assert summary["counts"]["correct"] >= 12
|
||||||
assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0
|
assert summary["closed_injector_buckets"]["unit_partition_no_injection"] == 0
|
||||||
|
|
|
||||||
|
|
@ -70,9 +70,9 @@ def test_confuser_comparative_more_than_refuses():
|
||||||
assert compose_goal_residual(text) is None
|
assert compose_goal_residual(text) is None
|
||||||
|
|
||||||
|
|
||||||
def test_product_bridge_cases_still_refuse():
|
def test_product_bridge_broad_path_not_serving_promoted():
|
||||||
"""product_bridge stays disabled — 0003/0021 must not lift via this path."""
|
"""Broad product_bridge stays disabled; Gate A2f lifts 0003/0021 instead."""
|
||||||
for text in (
|
texts = (
|
||||||
(
|
(
|
||||||
"The student council sells scented erasers in the morning before school starts "
|
"The student council sells scented erasers in the morning before school starts "
|
||||||
"to help raise money for school dances. The local bookstore donated 48 boxes of "
|
"to help raise money for school dances. The local bookstore donated 48 boxes of "
|
||||||
|
|
@ -83,9 +83,11 @@ def test_product_bridge_cases_still_refuse():
|
||||||
"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. "
|
"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. "
|
||||||
"How much total weight does he move?"
|
"How much total weight does he move?"
|
||||||
),
|
),
|
||||||
):
|
)
|
||||||
|
expected = (864.0, 450.0)
|
||||||
|
for text, ans in zip(texts, expected, strict=True):
|
||||||
res = _run(text)
|
res = _run(text)
|
||||||
assert res.answer is None
|
assert res.answer == ans
|
||||||
|
|
||||||
|
|
||||||
def test_peer_pick_regression_0025():
|
def test_peer_pick_regression_0025():
|
||||||
|
|
|
||||||
183
tests/test_math_candidate_graph_question_bound_product_lift.py
Normal file
183
tests/test_math_candidate_graph_question_bound_product_lift.py
Normal file
|
|
@ -0,0 +1,183 @@
|
||||||
|
"""Gate A2f — question_bound_product_aggregate serving lift (Paradigm Sprint 5)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from generate.math_candidate_graph import parse_and_solve
|
||||||
|
from generate.derivation.question_bound_product import (
|
||||||
|
compose_question_bound_product,
|
||||||
|
resolve_promotable_question_bound_product,
|
||||||
|
)
|
||||||
|
|
||||||
|
_CASES_PATH = (
|
||||||
|
Path(__file__).resolve().parents[1]
|
||||||
|
/ "evals"
|
||||||
|
/ "gsm8k_math"
|
||||||
|
/ "train_sample"
|
||||||
|
/ "v1"
|
||||||
|
/ "cases.jsonl"
|
||||||
|
)
|
||||||
|
|
||||||
|
CASE_0003 = (
|
||||||
|
"The student council sells scented erasers in the morning before school starts to help "
|
||||||
|
"raise money for school dances. The local bookstore donated 48 boxes of erasers. "
|
||||||
|
"There are 24 erasers in each box. If the student council sells the erasers for "
|
||||||
|
"$0.75 each, how much money will they make?"
|
||||||
|
)
|
||||||
|
|
||||||
|
CASE_0021 = (
|
||||||
|
"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. "
|
||||||
|
"How much total weight does he move?"
|
||||||
|
)
|
||||||
|
|
||||||
|
HOLDOUT_WRONG_SHAPE = (
|
||||||
|
"Penny's canoe can carry 6 people, but if she wants to take her dog, she will only "
|
||||||
|
"fit 2/3 of that number inside. If every person in a trip where Penny had her dog "
|
||||||
|
"inside the canoe weighed 140 pounds, and the dog 1/4 as much weight, calculate the "
|
||||||
|
"total weight the canoe was carrying?"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run(text: str):
|
||||||
|
return parse_and_solve(text, sealed=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_train_sample_0003_end_to_end():
|
||||||
|
res = _run(CASE_0003)
|
||||||
|
assert res.answer == 864.0
|
||||||
|
assert res.refusal_reason is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_train_sample_0021_end_to_end():
|
||||||
|
res = _run(CASE_0021)
|
||||||
|
assert res.answer == 450.0
|
||||||
|
assert res.refusal_reason is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_sibling_revenue_chain_varied_surface():
|
||||||
|
text = (
|
||||||
|
"The library donated 12 cartons of markers for a fundraiser. "
|
||||||
|
"There are 30 markers in each carton. "
|
||||||
|
"If the club sells every marker for $1.25 each, how much money will they raise?"
|
||||||
|
)
|
||||||
|
res = _run(text)
|
||||||
|
assert res.answer == 450.0
|
||||||
|
assert res.refusal_reason is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_sibling_weight_chain_varied_surface():
|
||||||
|
text = (
|
||||||
|
"Ana is training. She squats 20 kilograms for 8 reps and completes 4 sets. "
|
||||||
|
"How much total weight does she lift?"
|
||||||
|
)
|
||||||
|
res = _run(text)
|
||||||
|
assert res.answer == 640.0
|
||||||
|
assert res.refusal_reason is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_fraction_surface_refuses():
|
||||||
|
assert resolve_promotable_question_bound_product(HOLDOUT_WRONG_SHAPE) is None
|
||||||
|
assert _run(HOLDOUT_WRONG_SHAPE).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_ambiguous_actor_refuses():
|
||||||
|
text = (
|
||||||
|
"Tom donated 10 crates of books. There are 5 books in each crate. "
|
||||||
|
"If Sam sells the books for $2 each, how much money will they make?"
|
||||||
|
)
|
||||||
|
assert _run(text).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_additive_distractor_refuses():
|
||||||
|
text = (
|
||||||
|
"The shop has 20 boxes of pens and 3 loose pens. "
|
||||||
|
"There are 10 pens in each box. "
|
||||||
|
"If pens sell for $1 each, how much money will they make?"
|
||||||
|
)
|
||||||
|
assert _run(text).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_rate_without_target_refuses():
|
||||||
|
text = (
|
||||||
|
"She runs 6 miles per hour for 2 hours. "
|
||||||
|
"How many miles does she run?"
|
||||||
|
)
|
||||||
|
assert _run(text).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_reversed_relation_more_than_refuses():
|
||||||
|
text = (
|
||||||
|
"Lisa has 5 more boxes than Tom. Tom has 10 boxes. "
|
||||||
|
"There are 8 toys in each box. "
|
||||||
|
"If toys sell for $3 each, how much money will they make?"
|
||||||
|
)
|
||||||
|
assert _run(text).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_money_without_in_each_refuses():
|
||||||
|
text = (
|
||||||
|
"The store has 40 items. Items sell for $2 each. "
|
||||||
|
"How much money will they make?"
|
||||||
|
)
|
||||||
|
assert _run(text).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_confuser_weight_without_for_cue_refuses():
|
||||||
|
text = (
|
||||||
|
"Kai lifts 12 pounds, 8 reps, and 2 sets during practice. "
|
||||||
|
"How much total weight does he move?"
|
||||||
|
)
|
||||||
|
assert _run(text).answer is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_product_bridge_stays_disabled():
|
||||||
|
"""Broad product_bridge is not re-wired; this organ is narrower."""
|
||||||
|
from generate.derivation.product_bridge import resolve_promotable_product
|
||||||
|
|
||||||
|
assert resolve_promotable_product(CASE_0003) is not None
|
||||||
|
assert resolve_promotable_product(CASE_0021) is not None
|
||||||
|
assert compose_question_bound_product(CASE_0003) is not None
|
||||||
|
assert compose_question_bound_product(CASE_0021) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_goal_residual_and_peer_partition_regressions():
|
||||||
|
goal = (
|
||||||
|
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds "
|
||||||
|
"in April. How much weight does he have to lose in May to meet his goal?"
|
||||||
|
)
|
||||||
|
peer = (
|
||||||
|
"Lilibeth and her friends go strawberry picking. "
|
||||||
|
"Lilibeth fills 6 baskets where each basket holds 50 strawberries. "
|
||||||
|
"If three of Lilibeth's friends pick the same amount as her, "
|
||||||
|
"how many strawberries do Lilibeth and her friends pick in all?"
|
||||||
|
)
|
||||||
|
assert _run(goal).answer == 3.0
|
||||||
|
assert _run(peer).answer == 1200.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_full_train_sample_wrong_zero_and_chunk_lift():
|
||||||
|
from evals.gsm8k_math.train_sample.v1.runner import build_report, _load_cases
|
||||||
|
|
||||||
|
report = build_report(_load_cases(_CASES_PATH))
|
||||||
|
counts = report["counts"]
|
||||||
|
assert counts["wrong"] == 0
|
||||||
|
assert counts["correct"] == 12
|
||||||
|
assert counts["refused"] == 38
|
||||||
|
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||||
|
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||||
|
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||||
|
for case_id in (
|
||||||
|
"gsm8k-train-sample-v1-0002",
|
||||||
|
"gsm8k-train-sample-v1-0008",
|
||||||
|
"gsm8k-train-sample-v1-0014",
|
||||||
|
"gsm8k-train-sample-v1-0018",
|
||||||
|
"gsm8k-train-sample-v1-0024",
|
||||||
|
"gsm8k-train-sample-v1-0025",
|
||||||
|
"gsm8k-train-sample-v1-0029",
|
||||||
|
"gsm8k-train-sample-v1-0037",
|
||||||
|
"gsm8k-train-sample-v1-0038",
|
||||||
|
"gsm8k-train-sample-v1-0042",
|
||||||
|
):
|
||||||
|
assert by_case[case_id]["verdict"] == "correct", case_id
|
||||||
Loading…
Reference in a new issue