core/docs/analysis/composition-wall-execution-plan-2026-06-03.md
Shay ad9cf57069 feat(r4): flip cv-0005 to serving — train_sample 6/44/0 -> 7/43/0 (ADR-0207 §5 step 2)
Wires the R4 goal-residual production to serving via
resolve_promotable_goal_residual (math_candidate_graph.py, mirroring
product_bridge). cv-0005 / train_sample 0037 now solves on serving as
goal - Σprogress = 10 - 3 - 4 = 3. First Phase-5b composition lift on serving.

wrong=0 preserved on every runnable surface:
- train_sample 6/44/0 -> 7/43/0 (0037 added; 6 prior correct intact; wrong=0).
- Fires on 2/455 visible GSM8K cases, both correct, ZERO wrong.
- Gain-goal divergence firewall proves it reads the GOAL, not a possession.
- smoke 73, math+invariants 53, derivation/pool/practice 341, corpus, all green.

Lockstep updates (the ratified metric move, 6/44/0 -> 7/43/0):
- report.json; 7 ADR test lanes that pinned 6/44/0; corpus cv-0005 baseline
  fields + snapshot (4/18 -> 5/17) + contract; plan-doc cv-0018 control fix.

⚠ SEALED MEASUREMENT REQUIRED — NOT DONE. The sealed 1,319 (encrypted, not
CI-reproducible) is the real bar (ADR-0207 §6) and was NOT re-measured. The
operator/CI must decrypt+run it and confirm sealed wrong==0; if wrong>0, revert
the resolve_promotable_goal_residual block (isolated). See
docs/handoff/sealed-measurement-obligation-2026-06-04.md.
2026-06-03 22:20:12 -07:00

172 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!-- CANONICAL | composition-wall-execution-plan-2026-06-03.md | 2026-06-03 | spec-and-review (Opus) lane | read-only analysis / execution plan for ADR-0207 §5 step 2 | verified against origin/main d9fc7f9 | NOT a new design -->
# Composition Wall — Execution Plan (R1/R4 first)
This is the execution scope for **ADR-0207 §5 step 2 (COMPOSITION)***not* a new design. It
takes the ratified substrate as given and answers the only question that matters next: *what,
concretely, must change in the built `generate/derivation/` composer to move the live metric,
and where is each target shape actually blocked?* Every per-case claim below was reproduced
read-only against the tree on 2026-06-03 by running the actual composer on the corpus cases.
## 1. The wall, restated from reproduction (folds in the wiring brief)
- `resolve_pooled` (the general pooled composer, `pool.py:62`) **refuses all ten R1/R4/R5/R6
corpus lift-targets**, so wiring it to serving yields **+0 correct**; and it **wrong-commits
`0016 → 510`** (gold 2), so wiring it wholesale is a live `wrong=0` regression.
- The only derivation symbol on serving is `product_bridge.resolve_promotable_product`
(`math_candidate_graph.py:530`), and its safety is a **two-token target near-whitelist**
(`money`+`make/earn`, `weight`+`total/move`) — it does not generalize.
- Therefore **WIRING is trivial and already done for the safe shapes**; the actual work is
making the composer *build the right chain for a new shape and certify it*. The wire is the
last, one-line step (mirror the `:530` block) once a shape builds-and-certifies.
## 2. Grounded per-shape diagnosis (all 15 corpus positives)
Ran `extract_quantities``extract_target` → the three composers
(`accumulation_candidates` / `multiplicative_candidates` / `candidate_chains`) on each. "Stage"
is where the shape is blocked (see §3). Golds are dataset-sourced (corpus invariant 6).
| case | comp | gold | #qty | target-agg | acc/mul/chn | result | **stage** |
|---|---|---|---|---|---|---|---|
| cv-0001 | R1 | 64 | 1 | None | 0/0/0 | none | **A** extraction (`$16` not extracted) |
| cv-0003 | R1 | 9 | 6 | None | 0/0/0 | none | **B** target |
| cv-0002 | R1 | 400 | 3 | total | 0/0/4 | wrong-only | **C** production |
| cv-0004 | R1 | 3840 | 4 | None | 0/1/4 | wrong-only | **C** production |
| cv-0005 | R4 | 3 | 3 | None | 0/0/0 | none | **B** target |
| cv-0021 | R4 | 4 | 4 | None | 0/0/0 | none | **B** target |
| cv-0006 | R5 | 14 | 2 | None | 3/0/4 | wrong-only | **C** production |
| cv-0022 | R5 | 38 | 2 | total | 0/1/4 | wrong-only | **C** production |
| cv-0007 | R6 | 21 | 3 | None | 0/0/0 | none | **B** target |
| cv-0008 | R6 | 15 | 1 | None | 0/0/0 | none | **A**+**B** extraction+target |
| cv-0009 | compare_mult | 60 | 1 | altogether | 0/0/0 | none | **A** extraction |
| cv-0018 | *(control)* | 28 | 2 | None | 0/0/2 | wrong-only (derivation) | control — `gate=baseline`, **already solves on serving** (not a lift target) |
| cv-0019 | additive | 1200 | 2 | None | 0/0/0 | none | **B** target |
| cv-0017 | *(control)* | 438 | 5 | None | 0/0/2 | wrong-only | control — solves on serving |
| cv-0020 | *(control)* | 450 | 3 | total | 0/1/2 | **GOLD-BUILT** | control — **D** gate exemplar |
Controls (cv-0017=case 0024, cv-0020=case 0021) already solve on serving via other paths; they
are regression anchors, not lift targets. cv-0020 is the **proof the build-then-gate pattern
works**: the pool *builds the gold* (450) but only commits it on serving because
`product_bridge`'s gate promotes it.
## 3. The four-stage failure taxonomy (and the `wrong=0` firewall at each)
The composer is a pipeline: **extract → characterize target → build production → certify (gate)**.
A shape can be blocked at any stage; each stage refuses (never commits) when incomplete, so
`wrong=0` holds throughout.
- **Stage A — extraction-thin.** The composer is starved of a needed quantity, so no chain is
possible. *cv-0001* (currency literal `$16` not extracted — the derivation reader has no `$N`
production; this is the gap I earlier *overstated as closed*), *cv-0008*, *cv-0009*. Firewall:
no quantity → no candidate → refuse.
- **Stage B — target-uncharacterized.** Quantities are present but `extract_target` returns
`aggregation=None`, so the composer doesn't know *what* to build. *cv-0003, cv-0005, cv-0007,
cv-0019, cv-0021*. **Both R4 cases live here.** Firewall: no target → no admissible chain → refuse.
- **Stage C — production-wrong.** The composer builds chains, but the *wrong* ones (none equals
gold), and the disagreement rule correctly refuses them. *cv-0002, cv-0004, cv-0006, cv-0018,
cv-0022.* This is the hard core — "which quantities group, via which ops, in what order."
Firewall: disagreement among wrong candidates → refuse.
- **Stage D — gold-built, needs gate.** The composer builds the gold (possibly among rivals); a
*structural* promotion gate certifies and commits it. Exemplar: *cv-0020* via `product_bridge`.
Firewall: the gate must prove the reading; a lone wrong chain with no rival (the *0016* pattern)
is exactly what disagreement cannot catch, so **the gate must be structural, never
disagreement alone.**
## 4. Execution sequence — one shape, end-to-end, instrumented
Each shape is a **hypothesis to test against the gates**, not a guaranteed win. Drive *one* shape
through its blocked stages to green on the corpus, then check the sealed 1,319, before starting
the next.
1. **R4 first — but it is NEW PRODUCTION, not a target tweak** (verified 2026-06-03 by code-read
of `accumulate.py`: `_build_accumulation` is single-referent gain/loss running-total with **no
goal/target/residual concept** — target characterization alone does **not** unblock it). cv-0005
(single-referent `goal Σchanges`) is the nearest extension of GB-3b.1 and the first build;
cv-0021 (multi-referent give-away to a remainder) is two layers harder — separate, do not bundle.
Goal-recognition is a new `wrong=0` surface (must refuse on ambiguous goal-language), and beware
cv-0005 passing for the wrong reason (`1034=3` also falls out of treating the goal as a start);
the gate must consume the goal *as a goal*, and the real test is the sealed set, not cv-0005.
Detailed first-build scope: **§10**.
2. **R1 next (cv-0001) — Stage A (extraction).** Add currency-literal (`$N`) extraction to the
derivation reader, then test whether the comparative-multiplicative production (multiplier ×
base) fires and builds `4×16=64`. Extraction-first; different stage from R4, so it exercises a
different pipeline leg.
3. **Then Stage C shapes (cv-0002/0004/0006/0022)** — the genuine wall: building the correct
grouping/op-order. Highest risk, do last, with the most instrumentation. Investigation task:
`docs/handoff/stage-c-composition-investigation-2026-06-03.md`.
## 5. The promotion gate (generalize `product_bridge`, do not relax disagreement)
Every new production needs a co-designed structural gate, modeled on `product_bridge` but real:
- Use `extract_target` + `target_units` to prove the built chain answers *the asked question in
the asked units* (op-class + target + unit agreement), replacing the two-token whitelist with a
general op-class/target proof.
- A chain may commit only if it is the unique reading that *structurally* certifies; otherwise
refuse. **Disagreement is not the defense***cv-0016/0016* shows a lone wrong chain commits
with no rival to disagree. The gate, not the pool, is the `wrong=0` firewall for committed
compositions.
## 6. Acceptance gates (inherit ADR-0207 §6 verbatim)
train_sample stays `wrong=0` (6/44/0 or better); the no-reference `<N> times` hazard stays
refused; no partial chain commits with an unbound quantity; **progress measured on the 22-case
corpus AND the sealed 1,319** — a train_sample/corpus gain that does not move held-out does not
count; each gain lands as composer/extractor enrichment behind a gate, audited to confirm it is
**not** a new serving recognizer/injector branch (the §4 freeze).
## 7. Honest uncertainty
This is the research wall, not engineering. The payoff against the **sealed 1,319** is genuinely
unknown: the composer building gold on a corpus case does not mean the same production fires on
held-out paraphrases, and Stage C (correct grouping/op-order) is an open research problem, not a
known build. I can scope the work and the gates; I cannot promise R1/R4 transfer until the
productions exist and the sealed number actually moves. The gates (§6) are what keep that honest
— they will tell us the truth either way.
## 8. Correction recorded
I earlier asserted "extraction is substantially done; only EX-3 deferred; extraction is not the
open lever." Stage A (cv-0001: `$16` present in text, not extracted) shows that was overstated —
the ADR-0179 lever was largely shipped *for its four cases*, but per-shape extraction gaps remain
(currency literals at minimum). This does not unwind ADR-0207; the per-shape gaps are execution
detail and live here, not in the ratification ADR.
## 9. Reproduction-methodology correction
The `target-agg` column was first computed by passing the **full problem text** to
`extract_target`; serving passes only the **question clause**. For cv-0004 and cv-0006 the
aggregation word is in the body, not the question, so full-text wrongly read `combined`/`total`
where the serving-faithful call returns `None` (now corrected above). The other agg cells are
unaffected (their agg word is in the question). This changes no stage (both are Stage C) and no
claim. Methodology fix carried forward: reproduce serving behaviour with serving's inputs —
`extract_target` gets the question clause, the composers get the full problem.
## 10. R4 first-build scope (from code-read of `accumulate.py`)
R4 is **new production, not a target tweak** — confirmed by reading the source (not a trace):
`compose_accumulation` / `_build_accumulation` (GB-3b.1) computes single-referent gain/loss
*running-total* `start ± changes` and consults **no goal/target/residual concept**. Hard bails:
the anchor must establish exactly one quantity (`accumulate.py:70`); a new named actor refuses
(`continues_anchor_referent`, `:76`/`:143`); no unambiguous licensed change cue refuses
(`:87`/`:149`). So R4's `goal accumulated` cannot be expressed regardless of target — both R4
cases build **0 candidates** at `d9fc7f9e` (latent, nothing passes for any reason yet).
- **cv-0005 first (single-referent, nearest extension of GB-3b.1).** A new production recognizing
(a) a **goal** quantity as a distinct anchor type (*not* a running total), (b) accumulated
same-referent changes, (c) a **residual** question target, computing `goal Σchanges`. Single
referent (Michael), so it reuses GB-3b.1's referent/cue guards.
- **cv-0021 second (harder).** Multi-referent give-away to a remainder —
`(start desired_remainder) Σgiven`. Outside GB-3b.1's single-referent scope (refused at
`continues_anchor_referent`); needs multi-referent accumulation **and** residual handling. Two
layers beyond cv-0005; do not bundle.
- **Gate (mandatory, co-designed).** The residual chain commits only if `extract_target`
structurally proves the residual question + op-class + unit agreement, and completeness consumes
the **goal** quantity. Never disagreement alone (the 0016 firewall).
- **`wrong=0` hazard (latent, build-time guard).** `10 3 4 = 3` also falls out of
*mis-anchoring* the goal as a running-total start — coincidental correctness. The production must
consume the goal *as a goal*; the real test is the **sealed set**, not cv-0005's arithmetic.
- **Build-time discipline.** Whoever builds instruments the exact `_build_accumulation` interaction
and the new production's path through `self_verifies` (completeness must require the goal
consumed) **in-tree** — the code-read scopes *what* to build; the runtime path confirms *that it
fires* before the one-line wire.