feat(r4): flip cv-0005 to serving — train_sample 6/44/0 -> 7/43/0 (ADR-0207 §5 step 2)
Wires the R4 goal-residual production to serving via resolve_promotable_goal_residual (math_candidate_graph.py, mirroring product_bridge). cv-0005 / train_sample 0037 now solves on serving as goal - Σprogress = 10 - 3 - 4 = 3. First Phase-5b composition lift on serving. wrong=0 preserved on every runnable surface: - train_sample 6/44/0 -> 7/43/0 (0037 added; 6 prior correct intact; wrong=0). - Fires on 2/455 visible GSM8K cases, both correct, ZERO wrong. - Gain-goal divergence firewall proves it reads the GOAL, not a possession. - smoke 73, math+invariants 53, derivation/pool/practice 341, corpus, all green. Lockstep updates (the ratified metric move, 6/44/0 -> 7/43/0): - report.json; 7 ADR test lanes that pinned 6/44/0; corpus cv-0005 baseline fields + snapshot (4/18 -> 5/17) + contract; plan-doc cv-0018 control fix. ⚠ SEALED MEASUREMENT REQUIRED — NOT DONE. The sealed 1,319 (encrypted, not CI-reproducible) is the real bar (ADR-0207 §6) and was NOT re-measured. The operator/CI must decrypt+run it and confirm sealed wrong==0; if wrong>0, revert the resolve_promotable_goal_residual block (isolated). See docs/handoff/sealed-measurement-obligation-2026-06-04.md.
This commit is contained in:
Shay
parent
94d8137ad1
commit
ad9cf57069
14 changed files with 110 additions and 24 deletions
|
|
@ -39,7 +39,7 @@ is where the shape is blocked (see §3). Golds are dataset-sourced (corpus invar
|
|||
| cv-0007 | R6 | 21 | 3 | None | 0/0/0 | none | **B** target |
|
||||
| cv-0008 | R6 | 15 | 1 | None | 0/0/0 | none | **A**+**B** extraction+target |
|
||||
| cv-0009 | compare_mult | 60 | 1 | altogether | 0/0/0 | none | **A** extraction |
|
||||
| cv-0018 | compare_mult | 28 | 2 | None | 0/0/2 | wrong-only | **C** production |
|
||||
| cv-0018 | *(control)* | 28 | 2 | None | 0/0/2 | wrong-only (derivation) | control — `gate=baseline`, **already solves on serving** (not a lift target) |
|
||||
| cv-0019 | additive | 1200 | 2 | None | 0/0/0 | none | **B** target |
|
||||
| cv-0017 | *(control)* | 438 | 5 | None | 0/0/2 | wrong-only | control — solves on serving |
|
||||
| cv-0020 | *(control)* | 450 | 3 | total | 0/1/2 | **GOLD-BUILT** | control — **D** gate exemplar |
|
||||
|
|
|
|||
46
docs/handoff/sealed-measurement-obligation-2026-06-04.md
Normal file
46
docs/handoff/sealed-measurement-obligation-2026-06-04.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
<!-- handoff | sealed-measurement-obligation | 2026-06-04 | OPEN obligation — operator/CI action required | ADR-0207 §5 step 2 / §6 -->
|
||||
|
||||
# ⚠ OPEN OBLIGATION — sealed-1,319 measurement for the R4 goal-residual serving flip
|
||||
|
||||
**Status: REQUIRED, NOT YET DONE.** Blocks the claim that the GSM8K substrate is
|
||||
still `wrong=0` on held-out.
|
||||
|
||||
## What changed and why this note exists
|
||||
|
||||
ADR-0207 §5 step 2 landed the first composition lift on serving: **cv-0005 / train_sample
|
||||
0037** (R4 goal-residual) now solves, moving the serving metric **6/44/0 → 7/43/0**
|
||||
(`wrong=0` preserved on every surface this lane *can* run). The new production
|
||||
(`generate/derivation/goal_residual.py`) is wired to serving via
|
||||
`resolve_promotable_goal_residual` at `generate/math_candidate_graph.py` (mirroring the
|
||||
`product_bridge` pattern).
|
||||
|
||||
**The sealed holdout (1,319 cases, `evals/gsm8k_math/holdouts/v1/cases.jsonl.age`) was NOT
|
||||
re-measured.** It is age-encrypted and not CI-reproducible, so the agent that built this
|
||||
could not run it. Per ADR-0207 §6, **the sealed number is the real bar** — a serving gain
|
||||
that does not hold on the sealed set is overfitting and does not count.
|
||||
|
||||
## The obligation
|
||||
|
||||
The operator (key-holder) or a CI job with decryption access **must**:
|
||||
|
||||
1. Decrypt + run the sealed 1,319 through the current serving path
|
||||
(`generate.math_candidate_graph.parse_and_solve`) at the merge SHA.
|
||||
2. Confirm sealed **`wrong == 0`** still holds (prior recorded: `0/0/1319`).
|
||||
3. Record the verdict in `docs/claims_ledger.md` (row A / ADR-0119.7) and resolve this file.
|
||||
|
||||
## If sealed `wrong > 0`
|
||||
|
||||
`goal_residual` fired-and-committed a wrong reading on a held-out case. **Revert is trivial
|
||||
and isolated:** remove the `resolve_promotable_goal_residual` block from
|
||||
`math_candidate_graph.py` (the serving bridge) — the production and its tests can stay; only
|
||||
the serving promotion is rolled back. Then tighten the production's gate before re-flipping.
|
||||
|
||||
## Evidence the agent *could* gather (strong, not sufficient)
|
||||
|
||||
- Fires on **2 / 455** visible GSM8K-style cases, **both correct, zero wrong**.
|
||||
- Self-verify gate (grounding ∧ unit ∧ completeness) + extreme narrowness (goal-intent
|
||||
lexeme + residual question + same-referent licensed progress).
|
||||
- Gain-goal divergence firewall (`test_reads_goal_not_possession`) proves it reads the
|
||||
**goal**, not a possession (the coincidental-correctness trap).
|
||||
|
||||
None of that is the sealed verdict. **This note stays OPEN until step 2 above is done.**
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0002","composition":"R1","gate":"5b-R1","gold":400,"note":"Derived second-floor count = 3 * first-floor count, then total floors.","question":"In a building, there are a hundred ladies on the first-floor studying. There are three times that many girls at a party being held on the second floor of the building. How many ladies are on the two floors in total?","source":"gsm8k_train_sample:0038","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0003","composition":"R1","gate":"5b-R1","gold":9,"note":"Derived total beads from two products, then divide by beads per bracelet.","question":"Marnie makes bead bracelets. She bought 5 bags of 50 beads and 2 bags of 100 beads. If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?","source":"gsm8k_train_sample:0008","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0004","composition":"R1","gate":"5b-R1","gold":3840,"note":"Chained intermediate symbols: combined IG+Facebook, Twitter, TikTok, YouTube, then total.","question":"Malcolm has 240 followers on Instagram and 500 followers on Facebook. The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined. Meanwhile, the number of followers he has on TikTok is 3 times the number of followers is has on Twitter, and he has 510 more followers on Youtube than he has on TikTok. How many followers does Malcolm have on all his social media?","source":"gsm8k_train_sample:0027","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss.","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
|
||||
{"baseline_answer":3,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss. (flipped to solve on serving by ADR-0207 \u00a75 step 2 goal-residual)","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0006","composition":"R5","gate":"5b-R5","gold":14,"note":"Multi-step duration: round-trip drive time, scalar beach time, total trip time.","question":"Jake decides to go to the beach for a fun day. It is a 2-hour drive each way. He then spends 2.5 times at long at the beach as his total driving time. How much time does the trip take?","source":"gsm8k_train_sample:0030","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0007","composition":"R6","gate":"5b-R6","gold":21,"note":"Fraction mutation: final temperature = 3/4 of current, decrease = current - final.","question":"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?","source":"gsm8k_train_sample:0005","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0008","composition":"R6","gate":"5b-R6","gold":15,"note":"Percent partition: girls/boys split, percent of each, then total owners.","question":"A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?","source":"gsm8k_train_sample:0046","target_verdict":"solve"}
|
||||
|
|
|
|||
|
|
@ -62,12 +62,12 @@ Each JSONL row has these fields:
|
|||
|
||||
## Baseline
|
||||
|
||||
At creation on `origin/main` lineage after PR #534 plus the docs-only runway
|
||||
correction, and extended with the second R4/R5 positives (cv-0021/cv-0022), the
|
||||
intended baseline is:
|
||||
At creation the baseline was 4 solve / 18 refuse / 0 wrong. ADR-0207 §5 step 2
|
||||
landed the first Phase-5b flip — cv-0005 (R4 goal-residual) now solves on serving
|
||||
— moving the snapshot to:
|
||||
|
||||
- 4 solve
|
||||
- 18 refuse
|
||||
- 5 solve
|
||||
- 17 refuse
|
||||
- 0 wrong
|
||||
|
||||
The corpus deliberately includes both future positives and permanent
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
{
|
||||
"adr": "0126",
|
||||
"counts": {
|
||||
"correct": 6,
|
||||
"refused": 44,
|
||||
"correct": 7,
|
||||
"refused": 43,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
|
|
@ -193,8 +193,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0037",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Michael wants to lose 10 pounds by June.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0038",
|
||||
|
|
|
|||
|
|
@ -109,3 +109,19 @@ def compose_goal_residual(problem_text: str) -> Resolution | None:
|
|||
if derivation is None:
|
||||
return None
|
||||
return select_self_verified([derivation], problem_text, target_units=())
|
||||
|
||||
|
||||
def resolve_promotable_goal_residual(problem_text: str) -> Resolution | None:
|
||||
"""Serving promotion bridge (ADR-0207 §5 step 2), parallel to
|
||||
:func:`generate.derivation.product_bridge.resolve_promotable_product`.
|
||||
|
||||
Returns a serving-safe goal-residual resolution, or ``None``. The promotion
|
||||
invariant is the production's own narrowness (goal-anchor + residual question +
|
||||
same-referent licensed progress) **and** the self-verification gate
|
||||
(:func:`compose_goal_residual`): grounding ∧ unit ∧ completeness. Empirically
|
||||
(2026-06-04) this fires on 2/455 visible GSM8K cases, both correct, zero wrong;
|
||||
the gain-goal divergence firewall proves it reads the goal, not a possession.
|
||||
The sealed 1,319 verdict (ADR-0207 §6) is the operator/CI bar this gate cannot
|
||||
self-check.
|
||||
"""
|
||||
return compose_goal_residual(problem_text)
|
||||
|
|
|
|||
|
|
@ -539,6 +539,24 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0207 §5 step 2 — goal-residual promotion bridge (R4). The pooled reader
|
||||
# builds `goal - Σprogress` for residual-to-target questions; this gate promotes
|
||||
# only the self-verified single-referent goal-residual reading, which reads the
|
||||
# GOAL (not a possession — proven by the gain-goal divergence firewall). Narrow
|
||||
# and refuse-preferring (2/455 visible cases, 0 wrong); the sealed 1,319 is the
|
||||
# operator/CI bar (ADR-0207 §6).
|
||||
from generate.derivation.goal_residual import resolve_promotable_goal_residual
|
||||
|
||||
goal_resolution = resolve_promotable_goal_residual(text)
|
||||
if goal_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=goal_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
if len(statement_sentences) == 1:
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ Acceptance tests:
|
|||
4. Refusal-preferring discipline: a held statement with no discourse
|
||||
antecedent emits a "no_antecedent" trace event and drops cleanly.
|
||||
|
||||
5. wrong=0 preserved on train_sample/v1 (score unchanged at 6/44/0).
|
||||
5. wrong=0 preserved on train_sample/v1 (score now 7/43/0 after ADR-0207 §5 step 2).
|
||||
|
||||
Phase 3a substrate scope: this PR builds the reevaluate operator and
|
||||
wires pronoun resolution into the recognizer-injection branch of
|
||||
|
|
|
|||
|
|
@ -160,15 +160,16 @@ class TestRunPractice:
|
|||
|
||||
class TestLiveLane:
|
||||
def test_live_practice_mirrors_serving_today(self) -> None:
|
||||
# With the refuse-preferring engine, practice == serving (6/44/0).
|
||||
# With the refuse-preferring engine, practice == serving (7/43/0 after
|
||||
# ADR-0207 §5 step 2 lifted cv-0005/0037 via goal-residual).
|
||||
# Attempts/eliminations go live in Phase 3.
|
||||
rep = build_report()
|
||||
assert rep.counts == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
assert rep.counts == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
assert len(rep.elimination_records) == 0 # no wrongs yet
|
||||
|
||||
def test_every_refusal_is_diagnosed(self) -> None:
|
||||
rep = build_report()
|
||||
assert len(rep.refusal_diagnoses) == 44
|
||||
assert len(rep.refusal_diagnoses) == 43
|
||||
assert all(d in REFUSAL_DIAGNOSES for d in rep.refusal_diagnoses.values())
|
||||
|
||||
|
||||
|
|
@ -185,7 +186,7 @@ class TestSealInvariant:
|
|||
)
|
||||
build_report() # run practice
|
||||
serving = serving_build_report(_load_cases(_CASES_PATH))
|
||||
assert serving["counts"] == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
assert serving["counts"] == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
|
||||
def test_no_serving_module_imports_the_practice_lane(self) -> None:
|
||||
import subprocess
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ Covers:
|
|||
- extraction + search behaviour;
|
||||
- the ADR-0114a generality guard (renumbered/reworded variants flip too — the
|
||||
capability is not memorised to the 0021 surface);
|
||||
- invariant #1 (seal): serving stays 6/44/0, the 0050 canary refuses in serving
|
||||
- invariant #1 (seal): serving stays 7/43/0 (ADR-0207 §5 step 2 cv-0005 flip), the 0050 canary refuses in serving
|
||||
and is not attempted-wrong in practice;
|
||||
- invariant #3 (determinism).
|
||||
"""
|
||||
|
|
@ -113,9 +113,9 @@ class TestSealInvariant:
|
|||
def test_serving_unchanged_by_search(self) -> None:
|
||||
build_search_report() # run practice with the search live
|
||||
assert serving_report(_load_cases(_CASES_PATH))["counts"] == {
|
||||
"correct": 6,
|
||||
"correct": 7,
|
||||
"wrong": 0,
|
||||
"refused": 44,
|
||||
"refused": 43,
|
||||
}
|
||||
|
||||
def test_0050_canary_refuses_in_serving_and_is_not_attempted_wrong(self) -> None:
|
||||
|
|
|
|||
|
|
@ -48,7 +48,8 @@ class TestWrongZeroPreserved:
|
|||
)
|
||||
|
||||
counts = build_report(_load_cases(_CASES_PATH))["counts"]
|
||||
assert counts == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
# ADR-0207 §5 step 2: serving baseline is now 7/43/0 (cv-0005 goal-residual).
|
||||
assert counts == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
|
||||
|
||||
class TestUnblocksDecimalProduct:
|
||||
|
|
|
|||
|
|
@ -94,4 +94,4 @@ def test_frozen_train_sample_byte_identical() -> None:
|
|||
report = json.loads(
|
||||
Path("evals/gsm8k_math/train_sample/v1/report.json").read_text()
|
||||
)
|
||||
assert report["counts"] == {"correct": 6, "refused": 44, "wrong": 0}
|
||||
assert report["counts"] == {"correct": 7, "refused": 43, "wrong": 0}
|
||||
|
|
|
|||
|
|
@ -65,8 +65,10 @@ def test_known_pooled_wrong_commits_are_not_promotable(case_suffix: str) -> None
|
|||
|
||||
def test_train_sample_lifts_two_products_without_wrong() -> None:
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
assert report["counts"] == {"correct": 6, "wrong": 0, "refused": 44}
|
||||
# ADR-0207 §5 step 2: serving lifted 6/44/0 -> 7/43/0 (cv-0005/0037 goal-residual).
|
||||
assert report["counts"] == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0050"]["verdict"] == "refused"
|
||||
|
|
|
|||
|
|
@ -155,10 +155,12 @@ def test_frozen_baseline_fields_match_tree(case: dict) -> None:
|
|||
|
||||
|
||||
def test_current_baseline_snapshot() -> None:
|
||||
"""Current aggregate is 4 solve / 18 refuse / 0 wrong.
|
||||
"""Current aggregate is 5 solve / 17 refuse / 0 wrong.
|
||||
|
||||
This is the single assertion a Phase 5b slice updates when it flips a
|
||||
positive (refuse -> solve); the forever-invariants above do not change.
|
||||
ADR-0207 §5 step 2 landed the first such flip: cv-0005 (R4 goal-residual),
|
||||
moving 4/18 -> 5/17.
|
||||
"""
|
||||
solve = refuse = wrong = 0
|
||||
for case in _CASES:
|
||||
|
|
@ -170,7 +172,7 @@ def test_current_baseline_snapshot() -> None:
|
|||
else:
|
||||
refuse += 1
|
||||
assert wrong == 0
|
||||
assert (solve, refuse) == (4, 18), (
|
||||
assert (solve, refuse) == (5, 17), (
|
||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||
f"slice landed, update this expectation and the affected rows' "
|
||||
f"baseline fields in lockstep"
|
||||
|
|
|
|||
Loading…
Reference in a new issue