Merge pull request #544 from AssetOverflow/feat/r4-goal-residual
feat(r4): cv-0005 goal-residual to serving — train_sample 6/44/0 → 7/43/0 (ADR-0207 §5 step 2)
This commit is contained in:
commit
7160e5e77f
16 changed files with 297 additions and 24 deletions
|
|
@ -39,7 +39,7 @@ is where the shape is blocked (see §3). Golds are dataset-sourced (corpus invar
|
|||
| cv-0007 | R6 | 21 | 3 | None | 0/0/0 | none | **B** target |
|
||||
| cv-0008 | R6 | 15 | 1 | None | 0/0/0 | none | **A**+**B** extraction+target |
|
||||
| cv-0009 | compare_mult | 60 | 1 | altogether | 0/0/0 | none | **A** extraction |
|
||||
| cv-0018 | compare_mult | 28 | 2 | None | 0/0/2 | wrong-only | **C** production |
|
||||
| cv-0018 | *(control)* | 28 | 2 | None | 0/0/2 | wrong-only (derivation) | control — `gate=baseline`, **already solves on serving** (not a lift target) |
|
||||
| cv-0019 | additive | 1200 | 2 | None | 0/0/0 | none | **B** target |
|
||||
| cv-0017 | *(control)* | 438 | 5 | None | 0/0/2 | wrong-only | control — solves on serving |
|
||||
| cv-0020 | *(control)* | 450 | 3 | total | 0/1/2 | **GOLD-BUILT** | control — **D** gate exemplar |
|
||||
|
|
|
|||
46
docs/handoff/sealed-measurement-obligation-2026-06-04.md
Normal file
46
docs/handoff/sealed-measurement-obligation-2026-06-04.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
<!-- handoff | sealed-measurement-obligation | 2026-06-04 | OPEN obligation — operator/CI action required | ADR-0207 §5 step 2 / §6 -->
|
||||
|
||||
# ⚠ OPEN OBLIGATION — sealed-1,319 measurement for the R4 goal-residual serving flip
|
||||
|
||||
**Status: REQUIRED, NOT YET DONE.** Blocks the claim that the GSM8K substrate is
|
||||
still `wrong=0` on held-out.
|
||||
|
||||
## What changed and why this note exists
|
||||
|
||||
ADR-0207 §5 step 2 landed the first composition lift on serving: **cv-0005 / train_sample
|
||||
0037** (R4 goal-residual) now solves, moving the serving metric **6/44/0 → 7/43/0**
|
||||
(`wrong=0` preserved on every surface this lane *can* run). The new production
|
||||
(`generate/derivation/goal_residual.py`) is wired to serving via
|
||||
`resolve_promotable_goal_residual` at `generate/math_candidate_graph.py` (mirroring the
|
||||
`product_bridge` pattern).
|
||||
|
||||
**The sealed holdout (1,319 cases, `evals/gsm8k_math/holdouts/v1/cases.jsonl.age`) was NOT
|
||||
re-measured.** It is age-encrypted and not CI-reproducible, so the agent that built this
|
||||
could not run it. Per ADR-0207 §6, **the sealed number is the real bar** — a serving gain
|
||||
that does not hold on the sealed set is overfitting and does not count.
|
||||
|
||||
## The obligation
|
||||
|
||||
The operator (key-holder) or a CI job with decryption access **must**:
|
||||
|
||||
1. Decrypt + run the sealed 1,319 through the current serving path
|
||||
(`generate.math_candidate_graph.parse_and_solve`) at the merge SHA.
|
||||
2. Confirm sealed **`wrong == 0`** still holds (prior recorded: `0/0/1319`).
|
||||
3. Record the verdict in `docs/claims_ledger.md` (row A / ADR-0119.7) and resolve this file.
|
||||
|
||||
## If sealed `wrong > 0`
|
||||
|
||||
`goal_residual` fired-and-committed a wrong reading on a held-out case. **Revert is trivial
|
||||
and isolated:** remove the `resolve_promotable_goal_residual` block from
|
||||
`math_candidate_graph.py` (the serving bridge) — the production and its tests can stay; only
|
||||
the serving promotion is rolled back. Then tighten the production's gate before re-flipping.
|
||||
|
||||
## Evidence the agent *could* gather (strong, not sufficient)
|
||||
|
||||
- Fires on **2 / 455** visible GSM8K-style cases, **both correct, zero wrong**.
|
||||
- Self-verify gate (grounding ∧ unit ∧ completeness) + extreme narrowness (goal-intent
|
||||
lexeme + residual question + same-referent licensed progress).
|
||||
- Gain-goal divergence firewall (`test_reads_goal_not_possession`) proves it reads the
|
||||
**goal**, not a possession (the coincidental-correctness trap).
|
||||
|
||||
None of that is the sealed verdict. **This note stays OPEN until step 2 above is done.**
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0002","composition":"R1","gate":"5b-R1","gold":400,"note":"Derived second-floor count = 3 * first-floor count, then total floors.","question":"In a building, there are a hundred ladies on the first-floor studying. There are three times that many girls at a party being held on the second floor of the building. How many ladies are on the two floors in total?","source":"gsm8k_train_sample:0038","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0003","composition":"R1","gate":"5b-R1","gold":9,"note":"Derived total beads from two products, then divide by beads per bracelet.","question":"Marnie makes bead bracelets. She bought 5 bags of 50 beads and 2 bags of 100 beads. If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?","source":"gsm8k_train_sample:0008","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0004","composition":"R1","gate":"5b-R1","gold":3840,"note":"Chained intermediate symbols: combined IG+Facebook, Twitter, TikTok, YouTube, then total.","question":"Malcolm has 240 followers on Instagram and 500 followers on Facebook. The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined. Meanwhile, the number of followers he has on TikTok is 3 times the number of followers is has on Twitter, and he has 510 more followers on Youtube than he has on TikTok. How many followers does Malcolm have on all his social media?","source":"gsm8k_train_sample:0027","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss.","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
|
||||
{"baseline_answer":3,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss. (flipped to solve on serving by ADR-0207 \u00a75 step 2 goal-residual)","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0006","composition":"R5","gate":"5b-R5","gold":14,"note":"Multi-step duration: round-trip drive time, scalar beach time, total trip time.","question":"Jake decides to go to the beach for a fun day. It is a 2-hour drive each way. He then spends 2.5 times at long at the beach as his total driving time. How much time does the trip take?","source":"gsm8k_train_sample:0030","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0007","composition":"R6","gate":"5b-R6","gold":21,"note":"Fraction mutation: final temperature = 3/4 of current, decrease = current - final.","question":"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?","source":"gsm8k_train_sample:0005","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0008","composition":"R6","gate":"5b-R6","gold":15,"note":"Percent partition: girls/boys split, percent of each, then total owners.","question":"A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?","source":"gsm8k_train_sample:0046","target_verdict":"solve"}
|
||||
|
|
|
|||
|
|
@ -62,12 +62,12 @@ Each JSONL row has these fields:
|
|||
|
||||
## Baseline
|
||||
|
||||
At creation on `origin/main` lineage after PR #534 plus the docs-only runway
|
||||
correction, and extended with the second R4/R5 positives (cv-0021/cv-0022), the
|
||||
intended baseline is:
|
||||
At creation the baseline was 4 solve / 18 refuse / 0 wrong. ADR-0207 §5 step 2
|
||||
landed the first Phase-5b flip — cv-0005 (R4 goal-residual) now solves on serving
|
||||
— moving the snapshot to:
|
||||
|
||||
- 4 solve
|
||||
- 18 refuse
|
||||
- 5 solve
|
||||
- 17 refuse
|
||||
- 0 wrong
|
||||
|
||||
The corpus deliberately includes both future positives and permanent
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
{
|
||||
"adr": "0126",
|
||||
"counts": {
|
||||
"correct": 6,
|
||||
"refused": 44,
|
||||
"correct": 7,
|
||||
"refused": 43,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
|
|
@ -193,8 +193,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0037",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Michael wants to lose 10 pounds by June.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0038",
|
||||
|
|
|
|||
127
generate/derivation/goal_residual.py
Normal file
127
generate/derivation/goal_residual.py
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
"""ADR-0207 §5 step 2 / R4 — goal-residual reading (single-referent).
|
||||
|
||||
The first **residual-to-target** comprehension reading, distinct from GB-3b.1
|
||||
accumulation (:mod:`generate.derivation.accumulate`):
|
||||
|
||||
- accumulation reads a **possession** that changes — ``start ± changes`` (``Sam has
|
||||
14 apples. He buys 9 more.`` -> ``14 + 9``);
|
||||
- this reads a **goal** and the **progress** toward it — ``goal − Σprogress``
|
||||
(``Michael wants to lose 10 pounds. He lost 3, then 4. How much more to meet his
|
||||
goal?`` -> ``10 − 3 − 4``).
|
||||
|
||||
**Critical wrong=0 distinction — the coincidental-correctness trap.** Goal-residual is
|
||||
*not* possession-accumulation. For a **loss** goal the two arithmetics coincide
|
||||
(``10 − 3 − 4`` == ``10 − (3+4)``), so a naive "make accumulation fire" change would
|
||||
pass ``cv-0005`` for the wrong reason — reading the goal ``10`` as a possession start.
|
||||
For a **gain/save** goal they **diverge**: ``wants to save 20, saved 5, saved 6`` ->
|
||||
residual ``20 − 5 − 6 = 9``, whereas possession-accumulation gives ``20 + 5 + 6 = 31``.
|
||||
This production fires **only** on goal-language and **always subtracts** progress
|
||||
(progress reduces the residual regardless of its world-polarity), so it reads the goal,
|
||||
never the possession. The gain-goal divergence is the wrong=0 firewall test.
|
||||
|
||||
Lexeme-level (ADR-0165): goal-intent and residual-question cues are closed token sets,
|
||||
not sentence-shape grammar. The constructed chain runs through the unchanged
|
||||
self-verification gate (grounding ∧ unit ∧ completeness ∧ uniqueness) — refuse-
|
||||
preferring; this only proposes a structurally-licensed candidate.
|
||||
|
||||
Sealed (no ``chat/`` import); deterministic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Final
|
||||
|
||||
from generate.derivation.clauses import segment_clauses
|
||||
from generate.derivation.extract import extract_quantities
|
||||
from generate.derivation.model import GroundedDerivation, Quantity, Step
|
||||
from generate.derivation.state.bind import continues_anchor_referent, leading_subject_token
|
||||
from generate.derivation.state.change import classify_change_polarity, select_change_cue
|
||||
from generate.derivation.target import _question_clause
|
||||
from generate.derivation.verify import Resolution, select_self_verified
|
||||
from generate.math_roundtrip import _tokens
|
||||
|
||||
# Goal-intent lexemes: the anchor quantity is a *target*, not a possession. Closed set.
|
||||
_GOAL_INTENT: Final[frozenset[str]] = frozenset(
|
||||
{"want", "wants", "wanted", "need", "needs", "hoping", "hopes", "plans", "aims", "goal"}
|
||||
)
|
||||
# Residual-question lexemes: the question asks the remaining distance to the goal.
|
||||
_RESIDUAL_CUES: Final[frozenset[str]] = frozenset({"more", "left", "remaining"})
|
||||
_GOAL_REACH: Final[frozenset[str]] = frozenset({"meet", "reach", "hit", "achieve"})
|
||||
|
||||
|
||||
def _asks_residual(question_clause: str) -> bool:
|
||||
"""The question asks the remaining distance to a goal (lexeme-level)."""
|
||||
q = _tokens(question_clause)
|
||||
return bool(_RESIDUAL_CUES & q) or ("goal" in q and bool(_GOAL_REACH & q))
|
||||
|
||||
|
||||
def build_goal_residual(problem_text: str) -> GroundedDerivation | None:
|
||||
"""Construct the ungated goal-residual chain ``goal − Σprogress``, or ``None``.
|
||||
|
||||
Fires only when: (a) the question asks a residual-to-goal; (b) the first
|
||||
quantity-clause carries a goal-intent lexeme and establishes exactly one goal
|
||||
quantity; (c) every later quantity-clause stays on the same referent and carries a
|
||||
licensed change cue (its quantities are *progress*). Progress is **subtracted**
|
||||
(it reduces the residual) regardless of the change's world-polarity — this is what
|
||||
makes the reading goal-residual, not possession-accumulation.
|
||||
"""
|
||||
if not _asks_residual(_question_clause(problem_text)):
|
||||
return None
|
||||
|
||||
clauses = segment_clauses(problem_text)
|
||||
quantity_clauses = [c for c in clauses if extract_quantities(c)]
|
||||
if len(quantity_clauses) < 2:
|
||||
return None
|
||||
|
||||
anchor_clause, *progress_clauses = quantity_clauses
|
||||
if not (_GOAL_INTENT & _tokens(anchor_clause)):
|
||||
return None # the anchor must be goal-language, not a possession
|
||||
anchor_quantities = extract_quantities(anchor_clause)
|
||||
if len(anchor_quantities) != 1:
|
||||
return None # exactly one goal quantity
|
||||
goal = anchor_quantities[0]
|
||||
anchor_subject = leading_subject_token(anchor_clause)
|
||||
|
||||
steps: list[Step] = []
|
||||
for clause in progress_clauses:
|
||||
if not continues_anchor_referent(clause, anchor_subject):
|
||||
return None # new named actor -> referent hazard -> refuse
|
||||
polarity = classify_change_polarity(clause)
|
||||
if polarity is None:
|
||||
return None # progress must carry a licensed change cue
|
||||
cue = select_change_cue(clause, polarity)
|
||||
progress = [q for q in extract_quantities(clause) if (not q.unit) or q.unit == goal.unit]
|
||||
if not progress:
|
||||
return None # no same-unit progress quantity -> refuse
|
||||
for q in progress:
|
||||
operand = Quantity(value=q.value, unit=goal.unit, source_token=q.source_token)
|
||||
steps.append(Step(op="subtract", operand=operand, cue=cue))
|
||||
|
||||
if not steps:
|
||||
return None
|
||||
return GroundedDerivation(start=goal, steps=tuple(steps))
|
||||
|
||||
|
||||
def compose_goal_residual(problem_text: str) -> Resolution | None:
|
||||
"""R4 goal-residual composer. Refuse-preferring: gates the chain through the
|
||||
unchanged self-verification gate (grounding ∧ unit ∧ completeness ∧ uniqueness)."""
|
||||
derivation = build_goal_residual(problem_text)
|
||||
if derivation is None:
|
||||
return None
|
||||
return select_self_verified([derivation], problem_text, target_units=())
|
||||
|
||||
|
||||
def resolve_promotable_goal_residual(problem_text: str) -> Resolution | None:
|
||||
"""Serving promotion bridge (ADR-0207 §5 step 2), parallel to
|
||||
:func:`generate.derivation.product_bridge.resolve_promotable_product`.
|
||||
|
||||
Returns a serving-safe goal-residual resolution, or ``None``. The promotion
|
||||
invariant is the production's own narrowness (goal-anchor + residual question +
|
||||
same-referent licensed progress) **and** the self-verification gate
|
||||
(:func:`compose_goal_residual`): grounding ∧ unit ∧ completeness. Empirically
|
||||
(2026-06-04) this fires on 2/455 visible GSM8K cases, both correct, zero wrong;
|
||||
the gain-goal divergence firewall proves it reads the goal, not a possession.
|
||||
The sealed 1,319 verdict (ADR-0207 §6) is the operator/CI bar this gate cannot
|
||||
self-check.
|
||||
"""
|
||||
return compose_goal_residual(problem_text)
|
||||
|
|
@ -30,6 +30,7 @@ never invoked here).
|
|||
from __future__ import annotations
|
||||
|
||||
from generate.derivation.accumulate import accumulation_candidates
|
||||
from generate.derivation.goal_residual import build_goal_residual
|
||||
from generate.derivation.model import GroundedDerivation
|
||||
from generate.derivation.multistep import candidate_chains
|
||||
from generate.derivation.search import multiplicative_candidates
|
||||
|
|
@ -42,10 +43,12 @@ def pooled_candidates(problem_text: str) -> list[GroundedDerivation]:
|
|||
order (accumulation, then multiplicative, then chain)."""
|
||||
seen: set[tuple[object, ...]] = set()
|
||||
pooled: list[GroundedDerivation] = []
|
||||
_goal_residual = build_goal_residual(problem_text)
|
||||
for derivation in (
|
||||
*accumulation_candidates(problem_text),
|
||||
*multiplicative_candidates(problem_text),
|
||||
*candidate_chains(problem_text),
|
||||
*((_goal_residual,) if _goal_residual is not None else ()),
|
||||
):
|
||||
key = (
|
||||
round(derivation.answer, 9),
|
||||
|
|
|
|||
|
|
@ -539,6 +539,24 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0207 §5 step 2 — goal-residual promotion bridge (R4). The pooled reader
|
||||
# builds `goal - Σprogress` for residual-to-target questions; this gate promotes
|
||||
# only the self-verified single-referent goal-residual reading, which reads the
|
||||
# GOAL (not a possession — proven by the gain-goal divergence firewall). Narrow
|
||||
# and refuse-preferring (2/455 visible cases, 0 wrong); the sealed 1,319 is the
|
||||
# operator/CI bar (ADR-0207 §6).
|
||||
from generate.derivation.goal_residual import resolve_promotable_goal_residual
|
||||
|
||||
goal_resolution = resolve_promotable_goal_residual(text)
|
||||
if goal_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=goal_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ Acceptance tests:
|
|||
4. Refusal-preferring discipline: a held statement with no discourse
|
||||
antecedent emits a "no_antecedent" trace event and drops cleanly.
|
||||
|
||||
5. wrong=0 preserved on train_sample/v1 (score unchanged at 6/44/0).
|
||||
5. wrong=0 preserved on train_sample/v1 (score now 7/43/0 after ADR-0207 §5 step 2).
|
||||
|
||||
Phase 3a substrate scope: this PR builds the reevaluate operator and
|
||||
wires pronoun resolution into the recognizer-injection branch of
|
||||
|
|
|
|||
|
|
@ -160,15 +160,16 @@ class TestRunPractice:
|
|||
|
||||
class TestLiveLane:
|
||||
def test_live_practice_mirrors_serving_today(self) -> None:
|
||||
# With the refuse-preferring engine, practice == serving (6/44/0).
|
||||
# With the refuse-preferring engine, practice == serving (7/43/0 after
|
||||
# ADR-0207 §5 step 2 lifted cv-0005/0037 via goal-residual).
|
||||
# Attempts/eliminations go live in Phase 3.
|
||||
rep = build_report()
|
||||
assert rep.counts == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
assert rep.counts == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
assert len(rep.elimination_records) == 0 # no wrongs yet
|
||||
|
||||
def test_every_refusal_is_diagnosed(self) -> None:
|
||||
rep = build_report()
|
||||
assert len(rep.refusal_diagnoses) == 44
|
||||
assert len(rep.refusal_diagnoses) == 43
|
||||
assert all(d in REFUSAL_DIAGNOSES for d in rep.refusal_diagnoses.values())
|
||||
|
||||
|
||||
|
|
@ -185,7 +186,7 @@ class TestSealInvariant:
|
|||
)
|
||||
build_report() # run practice
|
||||
serving = serving_build_report(_load_cases(_CASES_PATH))
|
||||
assert serving["counts"] == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
assert serving["counts"] == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
|
||||
def test_no_serving_module_imports_the_practice_lane(self) -> None:
|
||||
import subprocess
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ Covers:
|
|||
- extraction + search behaviour;
|
||||
- the ADR-0114a generality guard (renumbered/reworded variants flip too — the
|
||||
capability is not memorised to the 0021 surface);
|
||||
- invariant #1 (seal): serving stays 6/44/0, the 0050 canary refuses in serving
|
||||
- invariant #1 (seal): serving stays 7/43/0 (ADR-0207 §5 step 2 cv-0005 flip), the 0050 canary refuses in serving
|
||||
and is not attempted-wrong in practice;
|
||||
- invariant #3 (determinism).
|
||||
"""
|
||||
|
|
@ -113,9 +113,9 @@ class TestSealInvariant:
|
|||
def test_serving_unchanged_by_search(self) -> None:
|
||||
build_search_report() # run practice with the search live
|
||||
assert serving_report(_load_cases(_CASES_PATH))["counts"] == {
|
||||
"correct": 6,
|
||||
"correct": 7,
|
||||
"wrong": 0,
|
||||
"refused": 44,
|
||||
"refused": 43,
|
||||
}
|
||||
|
||||
def test_0050_canary_refuses_in_serving_and_is_not_attempted_wrong(self) -> None:
|
||||
|
|
|
|||
|
|
@ -48,7 +48,8 @@ class TestWrongZeroPreserved:
|
|||
)
|
||||
|
||||
counts = build_report(_load_cases(_CASES_PATH))["counts"]
|
||||
assert counts == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
# ADR-0207 §5 step 2: serving baseline is now 7/43/0 (cv-0005 goal-residual).
|
||||
assert counts == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
|
||||
|
||||
class TestUnblocksDecimalProduct:
|
||||
|
|
|
|||
|
|
@ -94,4 +94,4 @@ def test_frozen_train_sample_byte_identical() -> None:
|
|||
report = json.loads(
|
||||
Path("evals/gsm8k_math/train_sample/v1/report.json").read_text()
|
||||
)
|
||||
assert report["counts"] == {"correct": 6, "refused": 44, "wrong": 0}
|
||||
assert report["counts"] == {"correct": 7, "refused": 43, "wrong": 0}
|
||||
|
|
|
|||
|
|
@ -65,8 +65,10 @@ def test_known_pooled_wrong_commits_are_not_promotable(case_suffix: str) -> None
|
|||
|
||||
def test_train_sample_lifts_two_products_without_wrong() -> None:
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
assert report["counts"] == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
# ADR-0207 §5 step 2: serving lifted 6/44/0 -> 7/43/0 (cv-0005/0037 goal-residual).
|
||||
assert report["counts"] == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0050"]["verdict"] == "refused"
|
||||
|
|
|
|||
|
|
@ -155,10 +155,12 @@ def test_frozen_baseline_fields_match_tree(case: dict) -> None:
|
|||
|
||||
|
||||
def test_current_baseline_snapshot() -> None:
|
||||
"""Current aggregate is 4 solve / 18 refuse / 0 wrong.
|
||||
"""Current aggregate is 5 solve / 17 refuse / 0 wrong.
|
||||
|
||||
This is the single assertion a Phase 5b slice updates when it flips a
|
||||
positive (refuse -> solve); the forever-invariants above do not change.
|
||||
ADR-0207 §5 step 2 landed the first such flip: cv-0005 (R4 goal-residual),
|
||||
moving 4/18 -> 5/17.
|
||||
"""
|
||||
solve = refuse = wrong = 0
|
||||
for case in _CASES:
|
||||
|
|
@ -170,7 +172,7 @@ def test_current_baseline_snapshot() -> None:
|
|||
else:
|
||||
refuse += 1
|
||||
assert wrong == 0
|
||||
assert (solve, refuse) == (4, 18), (
|
||||
assert (solve, refuse) == (5, 17), (
|
||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||
f"slice landed, update this expectation and the affected rows' "
|
||||
f"baseline fields in lockstep"
|
||||
|
|
|
|||
73
tests/test_r4_goal_residual.py
Normal file
73
tests/test_r4_goal_residual.py
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
"""ADR-0207 §5 step 2 / R4 — goal-residual production + its wrong=0 firewall.
|
||||
|
||||
The corpus lift-target cv-0005 (train_sample 0037) must read `goal − Σprogress`.
|
||||
The load-bearing test is the **gain-goal divergence** (`test_reads_goal_not_possession`):
|
||||
it proves the production reads the goal, not the possession, on a case where the two
|
||||
arithmetics differ — the coincidental-correctness trap cv-0005 alone cannot catch.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from generate.derivation.goal_residual import build_goal_residual, compose_goal_residual
|
||||
|
||||
# cv-0005 / train_sample 0037 (loss goal — residual and possession coincide at 3).
|
||||
CV0005 = (
|
||||
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds "
|
||||
"in April. How much weight does he have to lose in May to meet his goal?"
|
||||
)
|
||||
# Adversarial gain goal — residual (20-5-6=9) DIVERGES from possession (20+5+6=31).
|
||||
# Uses a recognized gain verb ("earned") so the progress cue is licensed.
|
||||
SAVE_GOAL = (
|
||||
"Maria wants to earn 20 dollars for a gift. She earned 5 dollars in May and 6 dollars "
|
||||
"in June. How much more does she need to earn to reach her goal?"
|
||||
)
|
||||
|
||||
|
||||
def _answer(text):
|
||||
res = compose_goal_residual(text)
|
||||
return None if res is None else res.answer
|
||||
|
||||
|
||||
def test_cv0005_goal_residual_solves() -> None:
|
||||
"""cv-0005: goal 10 − (3 + 4) = 3, read as a goal (start=10, all-subtract)."""
|
||||
deriv = build_goal_residual(CV0005)
|
||||
assert deriv is not None
|
||||
assert deriv.start.value == 10.0
|
||||
assert all(s.op == "subtract" for s in deriv.steps)
|
||||
assert _answer(CV0005) == 3.0
|
||||
|
||||
|
||||
def test_reads_goal_not_possession() -> None:
|
||||
"""WRONG=0 FIREWALL. On a gain goal, goal-residual (9) diverges from possession-
|
||||
accumulation (31). The production must give 9 (reads the goal) — never 31, and the
|
||||
chain must be all-subtract (progress reduces the residual regardless of polarity)."""
|
||||
deriv = build_goal_residual(SAVE_GOAL)
|
||||
assert deriv is not None
|
||||
assert _answer(SAVE_GOAL) == 9.0, "must read goal-residual, not possession 31"
|
||||
assert all(s.op == "subtract" for s in deriv.steps), "progress always subtracts"
|
||||
assert deriv.start.value == 20.0
|
||||
|
||||
|
||||
def test_no_goal_language_does_not_fire() -> None:
|
||||
"""A possession case (no goal-intent lexeme) must NOT fire this production —
|
||||
it belongs to accumulation, not goal-residual."""
|
||||
possession = "Sam has 14 apples. He gives away 3 apples and 2 apples. How many are left?"
|
||||
assert build_goal_residual(possession) is None
|
||||
|
||||
|
||||
def test_no_residual_question_does_not_fire() -> None:
|
||||
"""Goal language but no residual question → does not fire."""
|
||||
no_residual = (
|
||||
"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds "
|
||||
"in April. How much weight did he lose in total?"
|
||||
)
|
||||
assert build_goal_residual(no_residual) is None
|
||||
|
||||
|
||||
def test_incomplete_reading_refuses() -> None:
|
||||
"""A progress clause with a new named actor (referent hazard) must refuse — the
|
||||
same-referent guard is inherited, never weakened."""
|
||||
cross_referent = (
|
||||
"Michael wants to lose 10 pounds by June. Sarah lost 3 pounds in March. "
|
||||
"How much more does Michael need to lose to meet his goal?"
|
||||
)
|
||||
assert build_goal_residual(cross_referent) is None
|
||||
Loading…
Reference in a new issue