fix(gsm8k): disable unsound serving bridges — restore sealed wrong=0 (0/5 -> 0/0)
The FIRST real sealed measurement (operator-decrypted 1,319 held-out GSM8K) found `0 correct / 5 WRONG` — a wrong=0 breach hidden for weeks because the working metric was the 50-case train sample the bridges were tuned to. Bisection isolated it to the product_bridge serving promotion (ADR-0195). - generate/math_candidate_graph.py: REMOVE both serving promotion bridges (product_bridge + goal_residual/ADR-0207 §5 step 2). Serving = main-graph-only. Restores sealed 0/0/1319 (verified by bisect: disabling product_bridge -> 0 wrong). Production modules remain in generate/derivation/; only serving promotion is unwired, until a gate is proven wrong=0 on the SEALED set (never the train sample). - Honest numbers everywhere: train_sample 7/43/0 -> 4/46/0 (the bridges' "correct" was train-overfit). report.json + coverage probe regenerated. 7 ADR test lanes de-pinned from the inflated count. corpus: cv-0005 (R4) reverts to refuse; cv-0020 (a "baseline control" that solved ONLY via product_bridge) reclassified. - docs/claims_ledger.md: dated wrong=0-breach-and-remediation note + the rule: the train_sample number had ZERO predictive validity for the exam; never the score. - docs/analysis/gsm8k-lift-program-strategy: the program to actually move the 1,319. NOTE the exit gate stays `correct>=10 AND wrong==0` — refusing-everything is an explicit FAIL, not a wrong=0 pass; serving still commits (main graph). Verified: broad regression 848 passed, smoke 73 passed.
This commit is contained in:
Shay
parent
3a72d69678
commit
763c46d2f4
14 changed files with 260 additions and 85 deletions
162
docs/analysis/gsm8k-lift-program-strategy-2026-06-04.md
Normal file
162
docs/analysis/gsm8k-lift-program-strategy-2026-06-04.md
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
<!-- CANONICAL | gsm8k-lift-program-strategy-2026-06-04.md | 2026-06-04 | strategy (Opus) lane | program plan for sizeable GSM8K lift under ADR-0207 | grounded against origin/main 3a72d69 | NOT a new design — sequences existing substrate -->
|
||||
|
||||
# GSM8K Lift Program — Strategy for Sizeable, Sealed-Verified Lift
|
||||
|
||||
**Goal (stated honestly):** move the **real** number — the sealed 1,319 (and full GSM8K),
|
||||
not the 50-case train_sample — by *meaningful* chunks, with `wrong=0` preserved at every
|
||||
step. This is the program plan, sequenced from the substrate ADR-0207 ratified.
|
||||
|
||||
> ## ⚠ The brutal baseline (read this first)
|
||||
>
|
||||
> The **real external number is `0 / 0 / 1319`** — sealed real GSM8K test (HuggingFace
|
||||
> `openai/gsm8k`), claims_ledger row A: **0 correct out of 1,319. CORE solves *nothing* on
|
||||
> held-out GSM8K.** The train_sample `7/43/0` is on a **50-case unsealed sample CORE was
|
||||
> effectively built against** — the ledger's own rule: *"never present A as an accuracy."*
|
||||
>
|
||||
> So "sizeable lift" means **getting the sealed number off zero** — a fundamentally harder
|
||||
> thing than the train_sample increments suggest. The R4 win (cv-0005) moved a *proxy*; it
|
||||
> may be **+0 on the real 1,319** (goal-residual fires on 2/455 visible cases; whether it
|
||||
> fires on *any* sealed case is unknown until Stream 0 runs). Every magnitude claim below is
|
||||
> against this: **we are at zero on the bar that counts.**
|
||||
|
||||
**Grounding (origin/main `3a72d69`, 2026-06-04):** train_sample 7/43/0 (R4 goal-residual
|
||||
landed, sealed-pending). Composition class frequencies across the 44 refusals (multi-tagged,
|
||||
`composition-capability-scope §8`): **R5 multi-step 27, R1 derived-symbol 24, R6 percent 18,
|
||||
R4 residual 10, R2 inverse 6, R3 partition 3.** Only R4 is landed.
|
||||
|
||||
---
|
||||
|
||||
## 0. The leverage equation (why this plan is shaped as it is)
|
||||
|
||||
> **lift ≈ Σ over shapes of [ class_frequency × tractability × sealed_transfer ]**, divided by
|
||||
> **per-shape cost** — and per-shape cost is dominated by the *hand-built promotion bridge*,
|
||||
> not the composer.
|
||||
|
||||
Three consequences fall straight out, and they set the streams:
|
||||
|
||||
1. **R4 gave +1 because it is rare (10/44, and most are multi-referent).** Chasing more R4-like
|
||||
rare shapes one-by-one is a *trickle*. The big frequencies are **R1 (24)** and **R5 (27)** —
|
||||
but both need hard productions (R1 = quantity **reuse / DAG**; R5 = multi-step rate/duration).
|
||||
2. **Per-shape cost is the bridge, not the reading.** Today every shape needs a *hand-coded*
|
||||
serving promotion gate (`product_bridge`, `resolve_promotable_goal_residual`). That is the
|
||||
tax that makes lift one-shape-at-a-time. Removing it (Stream A) is the **force multiplier**.
|
||||
3. **No lift counts until the sealed set says so.** train_sample is a 50-case proxy; the 7/43/0
|
||||
win is *unverified on the real bar*. Sealed measurement is the **prerequisite**, not a
|
||||
formality (Stream 0).
|
||||
|
||||
**Honest magnitude expectation:** there is no single move that jumps the sealed number by a big
|
||||
chunk *cheaply and safely*. Sizeable lift is **compounding**: build the flywheel + the general
|
||||
consumption bridge so each subsequent shape is cheap, then spend the expensive research on the
|
||||
**high-frequency** shapes (R1/R5). The curve bends up when per-shape cost drops, not from any
|
||||
one production.
|
||||
|
||||
---
|
||||
|
||||
## Stream 0 — Sealed baseline (PREREQUISITE, blocks everything)
|
||||
|
||||
Until this runs, every number below is train_sample theater.
|
||||
|
||||
- **0.1** Resolve `docs/handoff/sealed-measurement-obligation-2026-06-04.md`: operator/CI
|
||||
decrypts + runs the sealed 1,319 at HEAD, confirms **sealed `wrong==0`** and records the
|
||||
sealed **correct** count. *Did R4 (cv-0005) actually move the sealed number, or was it +0 on
|
||||
held-out?* This answer calibrates the whole program.
|
||||
- **0.2** Stand up a **repeatable sealed-measurement gate** the operator can run per-increment
|
||||
(decrypt → `parse_and_solve` → counts → ledger row). Without a per-increment sealed check,
|
||||
the program cannot tell real lift from overfitting. This is the single most important
|
||||
infrastructure item — the program's measuring stick.
|
||||
- **Exit:** a known sealed baseline `(correct, 0, refused)` and a one-command sealed re-measure.
|
||||
|
||||
## Stream A — The force multiplier: general composition-promotion consumer
|
||||
|
||||
**The highest-leverage infrastructure in the program.** Replace N hand-built promotion bridges
|
||||
with one gated consumer.
|
||||
|
||||
- **A.1** Design a **ratified-composition-frame → structural-promotion-gate → serving** bridge:
|
||||
a single serving consumer that takes a *ratified frame* (shape + op-class + target signature)
|
||||
and promotes any reading that passes the **generalized gate** (`extract_target` + `target_units`
|
||||
+ self-verify grounding∧unit∧completeness + the divergence-firewall pattern). `product_bridge`
|
||||
and `goal_residual` become *instances* of this consumer, not bespoke code.
|
||||
- **A.2** The `wrong=0` firewall is the entire risk: an auto-promotion consumer that admits a
|
||||
wrong frame is the prime-directive violation. Gate it harder than any single bridge — every
|
||||
promoted frame carries its own divergence-firewall test (the goal-vs-possession pattern,
|
||||
generalized) and a sealed-gated ratification.
|
||||
- **A.3** Wire **cue-precision (ADR-0177)** as the *ranking* signal into this consumer (it is
|
||||
currently inert / consumed nowhere on serving) — it ranks which frame promotes when several
|
||||
self-verify, replacing per-shape disagreement-refusal with learned precision.
|
||||
- **Payoff:** after A, a new shape is "ratify a frame + its firewall test," not "write a bridge."
|
||||
Per-shape cost collapses; the flywheel's output finally reaches serving.
|
||||
|
||||
## Stream B — Harvest at scale (the trickle, industrialized)
|
||||
|
||||
The cheap wins, mined from the real corpus instead of the 50.
|
||||
|
||||
- **B.1** Acquire **full GSM8K train (7,473 cases, public)** into a harvest lane (the repo has
|
||||
only the 50-sample + the encrypted sealed holdout — the harvest pool must be added).
|
||||
- **B.2** Point the existing **practice / contemplation / propose** loop (`evals/gsm8k_math/practice/v1/`,
|
||||
`propose_runner.py`) at the full train set: attempt → diagnose refusals → for each, classify
|
||||
*structural match but lexeme/frame miss* vs *needs new production*.
|
||||
- **B.3** The **lexical-variant harvest** (your idea, grounded): structurally-built shapes that
|
||||
refuse only on a closed-set miss — a goal verb, a progress verb (`saved` is a live example —
|
||||
not in the change-cue vocab), a residual cue. Each becomes a one-lexeme ratification through
|
||||
the contemplation→HITL corridor, firewall-gated, landed via Stream A. The visible proxy showed
|
||||
a **~58-case remainder-question family** (9% of visible) — at full-train scale this is the
|
||||
steady trickle, *if* A makes landing cheap.
|
||||
- **Honest cap:** B is a trickle, not a flood. Its value is *steady + cheap*, and it compounds
|
||||
only once Stream A removes the landing tax.
|
||||
|
||||
## Stream C — The high-frequency research bets (where the big chunk lives)
|
||||
|
||||
The expensive, uncertain work — but the only path to a *sizeable* single move.
|
||||
|
||||
- **C.1 — R1 derived/intermediate symbol (24/44, the biggest single class).** Needs **quantity
|
||||
reuse** (`base + multiplier×base`, the value used twice) — a **DAG**, which the linear chain
|
||||
cannot express (flagged GB-5). This is the genuine research. Cracking it unblocks the largest
|
||||
class at once. Build the DAG composer + its promotion frame (via A), prove on the near-pure
|
||||
exemplars (0027/0008/0029/0038), **measure on sealed**.
|
||||
- **C.2 — R5 multi-step rate/duration/scalar (27/44, biggest overall).** Multi-step chains with
|
||||
a scalar-of-a-prior-stage and referent binding (0030/0015). Medium-hard; large frequency.
|
||||
- **C.3 — Sequence by frequency × tractability:** the Stage-C investigation
|
||||
(`docs/handoff/stage-c-composition-investigation-2026-06-03.md`) is the input that ranks the
|
||||
cheapest real entry into C; run it first to pick C.1-vs-C.2 ordering on evidence, not guess.
|
||||
- **Risk discipline:** each C build is a hypothesis tested against the **sealed** set, not the
|
||||
corpus. A corpus flip that does not move sealed is overfitting (ADR-0207 §6) and is reverted.
|
||||
|
||||
## Stream D — Measurement & `wrong=0` discipline (the spine)
|
||||
|
||||
- Every increment: train_sample (fast proxy) **and** the Stream-0 sealed gate (the real bar).
|
||||
- The serving metric "moves only via ratified PRs" (CLAUDE.md) — each lift PR carries its sealed
|
||||
delta or an explicit sealed-pending obligation note (as R4 did).
|
||||
- Track the **sealed correct count** as *the* program metric. train_sample is a smoke proxy.
|
||||
|
||||
---
|
||||
|
||||
## Sequencing & priorities
|
||||
|
||||
```
|
||||
NOW Stream 0 (sealed baseline + per-increment gate) ── blocks all claims
|
||||
THEN Stream A (general promotion consumer) ║ Stream B.1-2 (full-train harvest lane)
|
||||
— A is the force multiplier; B feeds it cheap wins
|
||||
NEXT Stream C.3 → C.1/C.2 (the big-frequency research, ranked by the Stage-C investigation)
|
||||
ALWAYS Stream D (sealed-gated, wrong=0, ratified-PR discipline)
|
||||
```
|
||||
|
||||
**Priority order if forced to pick one:** **Stream 0**, then **Stream A**. Reason: without 0 we
|
||||
are flying blind on the real number; without A every win costs a hand-built bridge and the
|
||||
flywheel never compounds. C is where the big chunk lives, but it is wasted effort until 0 can
|
||||
measure it and A can land it cheaply.
|
||||
|
||||
## What "meaningful lift in sizeable numbers" honestly requires
|
||||
|
||||
1. The sealed gate exists and the R4 win is confirmed real (Stream 0).
|
||||
2. The landing tax is gone (Stream A) — so the flywheel and each new shape are cheap.
|
||||
3. The expensive research lands on the **high-frequency** shapes R1/R5 (Stream C), each
|
||||
sealed-verified.
|
||||
|
||||
That is the honest path to a curve that bends up. It is a program, not a patch — and the first
|
||||
real milestone is **Stream 0**, because we do not yet know whether today's 7/43/0 moved the
|
||||
number that actually counts.
|
||||
|
||||
## Cross-references
|
||||
- Substrate: ADR-0207 (ratify · freeze · execute), §5 lever order, §6 gates.
|
||||
- Inputs: `composition-capability-scope.md` (§8 class frequencies), `composition-wall-execution-plan-2026-06-03.md` (stage taxonomy), `stage-c-composition-investigation-2026-06-03.md` (Stream C ranking), `sealed-measurement-obligation-2026-06-04.md` (Stream 0.1).
|
||||
- Landed exemplar of the build-then-gate pattern Stream A generalizes: `generate/derivation/goal_residual.py` + `resolve_promotable_goal_residual`.
|
||||
|
|
@ -45,6 +45,17 @@ material that blurs them is a credibility risk.
|
|||
CI-reproducible without the holdout key. Never present A as an accuracy
|
||||
achievement — present it as zero-confabulation discipline plus an honest
|
||||
coverage gap.
|
||||
|
||||
> **⚠ 2026-06-04 — `wrong=0` breach found and remediated.** The first re-run of
|
||||
> this sealed lane since the original record measured **`0 / 5 / 1314`** — the
|
||||
> `product_bridge` serving promotion (ADR-0195) had been silently committing **5
|
||||
> wrong** on held-out, invisible because the working metric was the 50-case train
|
||||
> sample it was tuned to. Bisection isolated it; both serving promotion bridges
|
||||
> (`product_bridge`, `goal_residual`/ADR-0207 §5 step 2) were **disabled**
|
||||
> (`generate/math_candidate_graph.py`), restoring `0/0/1319`. **The bridges must
|
||||
> not be re-enabled without a gate proven `wrong=0` on this sealed set, not the
|
||||
> train sample.** Lesson: the train_sample number had **zero predictive validity**
|
||||
> for the exam; never treat it as the score again.
|
||||
- **(B) A frontier *comparison*, not a benchmark result.** 150/150 is on a split
|
||||
**CORE wrote**, designed to exercise its own grammar without data
|
||||
contamination. `frontier.json` itself flags the "apples-vs-oranges" caveat:
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0002","composition":"R1","gate":"5b-R1","gold":400,"note":"Derived second-floor count = 3 * first-floor count, then total floors.","question":"In a building, there are a hundred ladies on the first-floor studying. There are three times that many girls at a party being held on the second floor of the building. How many ladies are on the two floors in total?","source":"gsm8k_train_sample:0038","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0003","composition":"R1","gate":"5b-R1","gold":9,"note":"Derived total beads from two products, then divide by beads per bracelet.","question":"Marnie makes bead bracelets. She bought 5 bags of 50 beads and 2 bags of 100 beads. If 50 beads are used to make one bracelet, how many bracelets will Marnie be able to make out of the beads she bought?","source":"gsm8k_train_sample:0008","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0004","composition":"R1","gate":"5b-R1","gold":3840,"note":"Chained intermediate symbols: combined IG+Facebook, Twitter, TikTok, YouTube, then total.","question":"Malcolm has 240 followers on Instagram and 500 followers on Facebook. The number of followers he has on Twitter is half the number of followers he has on Instagram and Facebook combined. Meanwhile, the number of followers he has on TikTok is 3 times the number of followers is has on Twitter, and he has 510 more followers on Youtube than he has on TikTok. How many followers does Malcolm have on all his social media?","source":"gsm8k_train_sample:0027","target_verdict":"solve"}
|
||||
{"baseline_answer":3,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual target: goal minus accumulated March and April loss. (flipped to solve on serving by ADR-0207 \u00a75 step 2 goal-residual)","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0005","composition":"R4","gate":"5b-R4","gold":3,"note":"Residual-to-target give-away: needed-to-give = start - desired remainder, minus already given (Jane 5, James 5+2). (R4 serving promotion reverted 2026-06-04 \u2014 unsound bridges disabled; refuses again.)","question":"Michael wants to lose 10 pounds by June. He lost 3 pounds in March and 4 pounds in April. How much weight does he have to lose in May to meet his goal?","source":"gsm8k_train_sample:0037","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0006","composition":"R5","gate":"5b-R5","gold":14,"note":"Multi-step duration: round-trip drive time, scalar beach time, total trip time.","question":"Jake decides to go to the beach for a fun day. It is a 2-hour drive each way. He then spends 2.5 times at long at the beach as his total driving time. How much time does the trip take?","source":"gsm8k_train_sample:0030","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0007","composition":"R6","gate":"5b-R6","gold":21,"note":"Fraction mutation: final temperature = 3/4 of current, decrease = current - final.","question":"In one hour, Addison mountain's temperature will decrease to 3/4 of its temperature. If the current temperature of the mountain is 84 degrees, what will the temperature decrease by?","source":"gsm8k_train_sample:0005","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0008","composition":"R6","gate":"5b-R6","gold":15,"note":"Percent partition: girls/boys split, percent of each, then total owners.","question":"A school has 100 students. Half of the students are girls, the other half are boys. 20% of the girls have dogs at home and 10% of the boys have dogs at home. How many students own dogs?","source":"gsm8k_train_sample:0046","target_verdict":"solve"}
|
||||
|
|
@ -17,6 +17,6 @@
|
|||
{"baseline_answer":438,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0017","composition":"compare_mult_additive","gate":"baseline","gold":438,"note":"Existing graph-path correct case; completeness guard must not over-refuse it.","question":"Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday. Brooke does three times as many jumping jacks as Sidney. How many jumping jacks did Brooke do?","source":"gsm8k_train_sample:0024","target_verdict":"solve"}
|
||||
{"baseline_answer":28,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0018","composition":"compare_mult","gate":"baseline","gold":28,"note":"Referenced multiplicative comparison already solves; protects against over-broad no-reference refusal.","question":"Tom has 7 apples. Jerry has 3 times as many apples as Tom. How many apples do they have together?","source":"guard:n_times_with_reference:3_as_many_as_tom","target_verdict":"solve"}
|
||||
{"baseline_answer":1200,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0019","composition":"additive","gate":"baseline","gold":1200,"note":"Stated-sum contrastive control for the derived-symbol Nicole/Cindy form.","question":"Nicole has 400 cards. Cindy has 800 cards. How many cards do they have together?","source":"analysis:composition-capability-scope:R1_contrastive_control","target_verdict":"solve"}
|
||||
{"baseline_answer":450,"baseline_branches_enumerated":1,"baseline_verdict":"solve","case_id":"cv-0020","composition":"product_additive","gate":"baseline","gold":450,"note":"Existing guarded product bridge correct case; protects current 6/44/0 lift.","question":"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. How much total weight does he move?","source":"gsm8k_train_sample:0021","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0020","composition":"product_additive","gate":"5b-product","gold":450,"note":"Existing guarded product bridge correct case; protects current 6/44/0 lift. (Reclassified 2026-06-04: solved only via product_bridge, disabled as unsound on the sealed set; refuses until a sealed-verified product gate exists.)","question":"John is lifting weights. He bench presses 15 pounds for 10 reps and does 3 sets. How much total weight does he move?","source":"gsm8k_train_sample:0021","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0021","composition":"R4","gate":"5b-R4","gold":4,"note":"Residual-to-target give-away: needed-to-give = start - desired remainder, minus already given (Jane 5, James 5+2).","question":"Martha has 20 apples. She decided to split them among her friends. Jane got 5 apples from her, and James got 2 more than Jane. How many more apples would Martha need to give away to be left with only 4 of them?","source":"gsm8k_train_sample:0035","target_verdict":"solve"}
|
||||
{"baseline_answer":null,"baseline_branches_enumerated":0,"baseline_verdict":"refuse","case_id":"cv-0022","composition":"R5","gate":"5b-R5","gold":38,"note":"Multi-step duration: train leg = 2x subway leg (scalar of prior stage), then sum of the three legs.","question":"Traveling from Manhattan to the Bronx, Andrew rides the subway for 10 hours, takes the train and rides for twice as much time as the subway ride, and then bikes the remaining distance for 8 hours. What's the total time he takes to reach the Bronx from Manhattan?","source":"gsm8k_train_sample:0015","target_verdict":"solve"}
|
||||
|
|
|
|||
|
|
@ -62,12 +62,15 @@ Each JSONL row has these fields:
|
|||
|
||||
## Baseline
|
||||
|
||||
At creation the baseline was 4 solve / 18 refuse / 0 wrong. ADR-0207 §5 step 2
|
||||
landed the first Phase-5b flip — cv-0005 (R4 goal-residual) now solves on serving
|
||||
— moving the snapshot to:
|
||||
At creation the baseline was 4 solve / 18 refuse / 0 wrong. **2026-06-04: the
|
||||
serving promotion bridges (product_bridge, goal_residual) were disabled** after the
|
||||
first real sealed measurement found them 0-correct / 5-wrong on the held-out 1,319
|
||||
(a `wrong=0` breach hidden by the train proxy). cv-0005 (R4) reverts to refusing and
|
||||
cv-0020 (solved only via product_bridge) was reclassified `baseline -> 5b-product`,
|
||||
so the honest snapshot is:
|
||||
|
||||
- 5 solve
|
||||
- 17 refuse
|
||||
- 3 solve
|
||||
- 19 refuse
|
||||
- 0 wrong
|
||||
|
||||
The corpus deliberately includes both future positives and permanent
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
{
|
||||
"adr": "0126",
|
||||
"counts": {
|
||||
"correct": 7,
|
||||
"refused": 43,
|
||||
"correct": 4,
|
||||
"refused": 46,
|
||||
"wrong": 0
|
||||
},
|
||||
"exit_criterion": {
|
||||
|
|
@ -23,8 +23,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0003",
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The local bookstore donated 48 boxes of erasers.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0004",
|
||||
|
|
@ -113,8 +113,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0021",
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He bench presses 15 pounds for 10 reps and does 3 sets.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0022",
|
||||
|
|
@ -193,8 +193,8 @@
|
|||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0037",
|
||||
"reason": "fast-path",
|
||||
"verdict": "correct"
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Michael wants to lose 10 pounds by June.' (category=discrete_count_statement)",
|
||||
"verdict": "refused"
|
||||
},
|
||||
{
|
||||
"case_id": "gsm8k-train-sample-v1-0038",
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
{
|
||||
"adr": "0131.G",
|
||||
"metrics": {
|
||||
"admission_rate": 0.12,
|
||||
"admitted_solved": 6,
|
||||
"admission_rate": 0.08,
|
||||
"admitted_solved": 4,
|
||||
"admitted_wrong": 0,
|
||||
"cases_total": 50,
|
||||
"refused": 44,
|
||||
"refused_rate": 0.88,
|
||||
"refused": 46,
|
||||
"refused_rate": 0.92,
|
||||
"safety_rail_intact": true,
|
||||
"wrong_count_is_zero": true,
|
||||
"wrong_rate": 0.0
|
||||
|
|
@ -35,14 +35,14 @@
|
|||
"trace_hash": null
|
||||
},
|
||||
{
|
||||
"actual_answer": 864.0,
|
||||
"actual_answer": null,
|
||||
"actual_unit": null,
|
||||
"case_id": "gsm8k-train-sample-v1-0003",
|
||||
"expected_answer": 864.0,
|
||||
"expected_unit": "",
|
||||
"outcome": "correct",
|
||||
"outcome": "refused",
|
||||
"realized_prose": null,
|
||||
"reason": "fast-path",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The local bookstore donated 48 boxes of erasers.' (category=discrete_count_statement)",
|
||||
"trace_hash": null
|
||||
},
|
||||
{
|
||||
|
|
@ -233,14 +233,14 @@
|
|||
"trace_hash": null
|
||||
},
|
||||
{
|
||||
"actual_answer": 450.0,
|
||||
"actual_answer": null,
|
||||
"actual_unit": null,
|
||||
"case_id": "gsm8k-train-sample-v1-0021",
|
||||
"expected_answer": 450.0,
|
||||
"expected_unit": "",
|
||||
"outcome": "correct",
|
||||
"outcome": "refused",
|
||||
"realized_prose": null,
|
||||
"reason": "fast-path",
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He bench presses 15 pounds for 10 reps and does 3 sets.' (category=discrete_count_statement)",
|
||||
"trace_hash": null
|
||||
},
|
||||
{
|
||||
|
|
@ -645,6 +645,10 @@
|
|||
"count": 1,
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'Half of the students are girls, the other "
|
||||
},
|
||||
{
|
||||
"count": 1,
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He bench presses 15 pounds for 10 reps and"
|
||||
},
|
||||
{
|
||||
"count": 1,
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'He can run 40 yards within 5 seconds.' (ca"
|
||||
|
|
@ -721,6 +725,10 @@
|
|||
"count": 1,
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The guests eat all of 1 pan, and 75% of th"
|
||||
},
|
||||
{
|
||||
"count": 1,
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The local bookstore donated 48 boxes of er"
|
||||
},
|
||||
{
|
||||
"count": 1,
|
||||
"reason": "candidate_graph: recognizer matched but produced no injection for statement: 'The number of followers he has on Twitter "
|
||||
|
|
|
|||
|
|
@ -521,41 +521,19 @@ def parse_and_solve(text: str, *, sealed: bool = False) -> CandidateGraphResult:
|
|||
branches_enumerated=0, branches_admissible=0,
|
||||
)
|
||||
|
||||
# ADR-0195 — product promotion bridge. The sealed pooled derivation
|
||||
# reader can solve complete multiplicative aggregates that the
|
||||
# candidate-graph recognizer still refuses at the injector boundary,
|
||||
# but the pooled reader is not safe to promote wholesale. Expose only
|
||||
# complete pure-product readings whose target/hazard gate proves this
|
||||
# is an aggregate product, not a rate/residual/percentage/equation.
|
||||
from generate.derivation.product_bridge import resolve_promotable_product
|
||||
|
||||
product_resolution = resolve_promotable_product(text)
|
||||
if product_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=product_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
|
||||
# ADR-0207 §5 step 2 — goal-residual promotion bridge (R4). The pooled reader
|
||||
# builds `goal - Σprogress` for residual-to-target questions; this gate promotes
|
||||
# only the self-verified single-referent goal-residual reading, which reads the
|
||||
# GOAL (not a possession — proven by the gain-goal divergence firewall). Narrow
|
||||
# and refuse-preferring (2/455 visible cases, 0 wrong); the sealed 1,319 is the
|
||||
# operator/CI bar (ADR-0207 §6).
|
||||
from generate.derivation.goal_residual import resolve_promotable_goal_residual
|
||||
|
||||
goal_resolution = resolve_promotable_goal_residual(text)
|
||||
if goal_resolution is not None:
|
||||
return CandidateGraphResult(
|
||||
answer=goal_resolution.answer,
|
||||
selected_graph=None,
|
||||
refusal_reason=None,
|
||||
branches_enumerated=1,
|
||||
branches_admissible=1,
|
||||
)
|
||||
# ── DISABLED (2026-06-04): serving promotion bridges removed ──────────────
|
||||
# The FIRST real sealed measurement (1,319 held-out GSM8K, decrypted by the
|
||||
# operator) showed the product-promotion bridge (ADR-0195) commits 0 correct /
|
||||
# 5 WRONG on held-out — a `wrong=0` breach that was invisible because the
|
||||
# working metric was the 50-case train sample the bridges were tuned to.
|
||||
# Bisection: disabling `resolve_promotable_product` restores sealed 0/0/1319.
|
||||
# `resolve_promotable_goal_residual` (ADR-0207 §5 step 2) is 0/0 on held-out
|
||||
# (inert) — removed too, since its only effect was inflating the train proxy.
|
||||
# Both production modules remain in generate/derivation/; only their serving
|
||||
# promotion is unwired, until a gate is built that is proven `wrong=0` on the
|
||||
# SEALED set (not the train sample). Restoring `wrong=0` is the prime directive
|
||||
# and outranks the train-sample "correct" the bridges produced.
|
||||
# ──────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# ADR-0136.S.1 — Rate/event short-circuit paths (before Cartesian product).
|
||||
# Capacity path: single statement with one CandidateCapacity + matching question.
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ Acceptance tests:
|
|||
4. Refusal-preferring discipline: a held statement with no discourse
|
||||
antecedent emits a "no_antecedent" trace event and drops cleanly.
|
||||
|
||||
5. wrong=0 preserved on train_sample/v1 (score now 7/43/0 after ADR-0207 §5 step 2).
|
||||
5. wrong=0 preserved on train_sample/v1 (score now 4/46/0 after ADR-0207 §5 step 2).
|
||||
|
||||
Phase 3a substrate scope: this PR builds the reevaluate operator and
|
||||
wires pronoun resolution into the recognizer-injection branch of
|
||||
|
|
|
|||
|
|
@ -160,16 +160,16 @@ class TestRunPractice:
|
|||
|
||||
class TestLiveLane:
|
||||
def test_live_practice_mirrors_serving_today(self) -> None:
|
||||
# With the refuse-preferring engine, practice == serving (7/43/0 after
|
||||
# With the refuse-preferring engine, practice == serving (4/46/0 after
|
||||
# ADR-0207 §5 step 2 lifted cv-0005/0037 via goal-residual).
|
||||
# Attempts/eliminations go live in Phase 3.
|
||||
rep = build_report()
|
||||
assert rep.counts == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
assert rep.counts == {"correct": 4, "wrong": 0, "refused": 46}
|
||||
assert len(rep.elimination_records) == 0 # no wrongs yet
|
||||
|
||||
def test_every_refusal_is_diagnosed(self) -> None:
|
||||
rep = build_report()
|
||||
assert len(rep.refusal_diagnoses) == 43
|
||||
assert len(rep.refusal_diagnoses) == 46
|
||||
assert all(d in REFUSAL_DIAGNOSES for d in rep.refusal_diagnoses.values())
|
||||
|
||||
|
||||
|
|
@ -186,7 +186,7 @@ class TestSealInvariant:
|
|||
)
|
||||
build_report() # run practice
|
||||
serving = serving_build_report(_load_cases(_CASES_PATH))
|
||||
assert serving["counts"] == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
assert serving["counts"] == {"correct": 4, "wrong": 0, "refused": 46}
|
||||
|
||||
def test_no_serving_module_imports_the_practice_lane(self) -> None:
|
||||
import subprocess
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ Covers:
|
|||
- extraction + search behaviour;
|
||||
- the ADR-0114a generality guard (renumbered/reworded variants flip too — the
|
||||
capability is not memorised to the 0021 surface);
|
||||
- invariant #1 (seal): serving stays 7/43/0 (ADR-0207 §5 step 2 cv-0005 flip), the 0050 canary refuses in serving
|
||||
- invariant #1 (seal): serving stays 4/46/0 (ADR-0207 §5 step 2 cv-0005 flip), the 0050 canary refuses in serving
|
||||
and is not attempted-wrong in practice;
|
||||
- invariant #3 (determinism).
|
||||
"""
|
||||
|
|
@ -113,9 +113,9 @@ class TestSealInvariant:
|
|||
def test_serving_unchanged_by_search(self) -> None:
|
||||
build_search_report() # run practice with the search live
|
||||
assert serving_report(_load_cases(_CASES_PATH))["counts"] == {
|
||||
"correct": 7,
|
||||
"correct": 4,
|
||||
"wrong": 0,
|
||||
"refused": 43,
|
||||
"refused": 46,
|
||||
}
|
||||
|
||||
def test_0050_canary_refuses_in_serving_and_is_not_attempted_wrong(self) -> None:
|
||||
|
|
|
|||
|
|
@ -48,8 +48,8 @@ class TestWrongZeroPreserved:
|
|||
)
|
||||
|
||||
counts = build_report(_load_cases(_CASES_PATH))["counts"]
|
||||
# ADR-0207 §5 step 2: serving baseline is now 7/43/0 (cv-0005 goal-residual).
|
||||
assert counts == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
# ADR-0207 §5 step 2: serving baseline is now 4/46/0 (cv-0005 goal-residual).
|
||||
assert counts == {"correct": 4, "wrong": 0, "refused": 46}
|
||||
|
||||
|
||||
class TestUnblocksDecimalProduct:
|
||||
|
|
|
|||
|
|
@ -94,4 +94,4 @@ def test_frozen_train_sample_byte_identical() -> None:
|
|||
report = json.loads(
|
||||
Path("evals/gsm8k_math/train_sample/v1/report.json").read_text()
|
||||
)
|
||||
assert report["counts"] == {"correct": 7, "refused": 43, "wrong": 0}
|
||||
assert report["counts"] == {"correct": 4, "refused": 46, "wrong": 0}
|
||||
|
|
|
|||
|
|
@ -36,13 +36,16 @@ def _case(case_suffix: str) -> dict:
|
|||
@pytest.mark.parametrize("case_suffix, expected", [("0003", 864.0), ("0021", 450.0)])
|
||||
def test_promotable_product_cases_resolve(case_suffix: str, expected: float) -> None:
|
||||
row = _case(case_suffix)
|
||||
# The bridge FUNCTION still resolves these in isolation...
|
||||
resolution = resolve_promotable_product(row["question"])
|
||||
assert resolution is not None
|
||||
assert resolution.answer == expected
|
||||
|
||||
# ...but its SERVING promotion is DISABLED (2026-06-04): the bridge committed
|
||||
# 0 correct / 5 WRONG on the sealed 1,319, so serving now REFUSES rather than
|
||||
# promote a product reading unsound on held-out. wrong=0 over train "correct".
|
||||
result = parse_and_solve(row["question"])
|
||||
assert result.refusal_reason is None
|
||||
assert result.answer == expected
|
||||
assert result.answer is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
|
@ -63,12 +66,17 @@ def test_known_pooled_wrong_commits_are_not_promotable(case_suffix: str) -> None
|
|||
assert resolve_promotable_product(row["question"]) is None
|
||||
|
||||
|
||||
def test_train_sample_lifts_two_products_without_wrong() -> None:
|
||||
def test_product_bridge_serving_promotion_is_disabled() -> None:
|
||||
"""DISABLED 2026-06-04: the FIRST real sealed measurement showed
|
||||
product_bridge commits 0 correct / 5 WRONG on the held-out 1,319 — a wrong=0
|
||||
breach hidden by the 50-case train proxy it was tuned to. Its serving
|
||||
promotion is unwired (the bridge fired on 0003/0021 *on train only*; it does
|
||||
not generalize). Honest serving is now the main-graph-only 4/46/0; the
|
||||
products 0003/0021 refuse rather than commit a reading unsound on held-out."""
|
||||
report = build_report(_load_cases(_CASES_PATH))
|
||||
# ADR-0207 §5 step 2: serving lifted 6/44/0 -> 7/43/0 (cv-0005/0037 goal-residual).
|
||||
assert report["counts"] == {"correct": 7, "wrong": 0, "refused": 43}
|
||||
assert report["counts"] == {"correct": 4, "wrong": 0, "refused": 46}
|
||||
by_case = {row["case_id"]: row for row in report["per_case"]}
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "correct"
|
||||
assert by_case["gsm8k-train-sample-v1-0003"]["verdict"] == "refused"
|
||||
assert by_case["gsm8k-train-sample-v1-0021"]["verdict"] == "refused"
|
||||
assert by_case["gsm8k-train-sample-v1-0037"]["verdict"] == "refused"
|
||||
assert by_case["gsm8k-train-sample-v1-0050"]["verdict"] == "refused"
|
||||
|
|
|
|||
|
|
@ -52,9 +52,12 @@ _REQUIRED_FIELDS = frozenset(
|
|||
)
|
||||
|
||||
_EXPECTED_TOTAL = 22
|
||||
_EXPECTED_BASELINE_CONTROLS = 4
|
||||
# 2026-06-04: product_bridge + goal_residual serving promotions disabled (unsound on
|
||||
# the sealed 1,319: 0 correct / 5 wrong). cv-0020 (solved only via product_bridge)
|
||||
# reclassified baseline -> 5b-product; cv-0005 (R4) reverts to refusing.
|
||||
_EXPECTED_BASELINE_CONTROLS = 3
|
||||
_EXPECTED_PERMANENT = 7
|
||||
_EXPECTED_FUTURE_POSITIVE = 11
|
||||
_EXPECTED_FUTURE_POSITIVE = 12
|
||||
|
||||
|
||||
def _load_cases() -> list[dict]:
|
||||
|
|
@ -155,12 +158,14 @@ def test_frozen_baseline_fields_match_tree(case: dict) -> None:
|
|||
|
||||
|
||||
def test_current_baseline_snapshot() -> None:
|
||||
"""Current aggregate is 5 solve / 17 refuse / 0 wrong.
|
||||
"""Current aggregate is 3 solve / 19 refuse / 0 wrong.
|
||||
|
||||
This is the single assertion a Phase 5b slice updates when it flips a
|
||||
positive (refuse -> solve); the forever-invariants above do not change.
|
||||
ADR-0207 §5 step 2 landed the first such flip: cv-0005 (R4 goal-residual),
|
||||
moving 4/18 -> 5/17.
|
||||
2026-06-04: the unsound serving promotion bridges (product_bridge,
|
||||
goal_residual) were disabled after the first real sealed measurement showed
|
||||
them 0-correct/5-wrong on held-out — so cv-0005 (R4) and cv-0020 (product)
|
||||
revert to refusing, moving the honest snapshot to 3/19.
|
||||
"""
|
||||
solve = refuse = wrong = 0
|
||||
for case in _CASES:
|
||||
|
|
@ -172,7 +177,7 @@ def test_current_baseline_snapshot() -> None:
|
|||
else:
|
||||
refuse += 1
|
||||
assert wrong == 0
|
||||
assert (solve, refuse) == (5, 17), (
|
||||
assert (solve, refuse) == (3, 19), (
|
||||
f"snapshot moved to {solve} solve / {refuse} refuse — if a Phase 5b "
|
||||
f"slice landed, update this expectation and the affected rows' "
|
||||
f"baseline fields in lockstep"
|
||||
|
|
|
|||
Loading…
Reference in a new issue