core/docs/analysis/composition-wall-execution-plan-2026-06-03.md
Shay ad9cf57069 feat(r4): flip cv-0005 to serving — train_sample 6/44/0 -> 7/43/0 (ADR-0207 §5 step 2)
Wires the R4 goal-residual production to serving via
resolve_promotable_goal_residual (math_candidate_graph.py, mirroring
product_bridge). cv-0005 / train_sample 0037 now solves on serving as
goal - Σprogress = 10 - 3 - 4 = 3. First Phase-5b composition lift on serving.

wrong=0 preserved on every runnable surface:
- train_sample 6/44/0 -> 7/43/0 (0037 added; 6 prior correct intact; wrong=0).
- Fires on 2/455 visible GSM8K cases, both correct, ZERO wrong.
- Gain-goal divergence firewall proves it reads the GOAL, not a possession.
- smoke 73, math+invariants 53, derivation/pool/practice 341, corpus, all green.

Lockstep updates (the ratified metric move, 6/44/0 -> 7/43/0):
- report.json; 7 ADR test lanes that pinned 6/44/0; corpus cv-0005 baseline
  fields + snapshot (4/18 -> 5/17) + contract; plan-doc cv-0018 control fix.

⚠ SEALED MEASUREMENT REQUIRED — NOT DONE. The sealed 1,319 (encrypted, not
CI-reproducible) is the real bar (ADR-0207 §6) and was NOT re-measured. The
operator/CI must decrypt+run it and confirm sealed wrong==0; if wrong>0, revert
the resolve_promotable_goal_residual block (isolated). See
docs/handoff/sealed-measurement-obligation-2026-06-04.md.
2026-06-03 22:20:12 -07:00

12 KiB
Raw Blame History

Composition Wall — Execution Plan (R1/R4 first)

This is the execution scope for ADR-0207 §5 step 2 (COMPOSITION)not a new design. It takes the ratified substrate as given and answers the only question that matters next: what, concretely, must change in the built generate/derivation/ composer to move the live metric, and where is each target shape actually blocked? Every per-case claim below was reproduced read-only against the tree on 2026-06-03 by running the actual composer on the corpus cases.

1. The wall, restated from reproduction (folds in the wiring brief)

  • resolve_pooled (the general pooled composer, pool.py:62) refuses all ten R1/R4/R5/R6 corpus lift-targets, so wiring it to serving yields +0 correct; and it wrong-commits 0016 → 510 (gold 2), so wiring it wholesale is a live wrong=0 regression.
  • The only derivation symbol on serving is product_bridge.resolve_promotable_product (math_candidate_graph.py:530), and its safety is a two-token target near-whitelist (money+make/earn, weight+total/move) — it does not generalize.
  • Therefore WIRING is trivial and already done for the safe shapes; the actual work is making the composer build the right chain for a new shape and certify it. The wire is the last, one-line step (mirror the :530 block) once a shape builds-and-certifies.

2. Grounded per-shape diagnosis (all 15 corpus positives)

Ran extract_quantitiesextract_target → the three composers (accumulation_candidates / multiplicative_candidates / candidate_chains) on each. "Stage" is where the shape is blocked (see §3). Golds are dataset-sourced (corpus invariant 6).

case comp gold #qty target-agg acc/mul/chn result stage
cv-0001 R1 64 1 None 0/0/0 none A extraction ($16 not extracted)
cv-0003 R1 9 6 None 0/0/0 none B target
cv-0002 R1 400 3 total 0/0/4 wrong-only C production
cv-0004 R1 3840 4 None 0/1/4 wrong-only C production
cv-0005 R4 3 3 None 0/0/0 none B target
cv-0021 R4 4 4 None 0/0/0 none B target
cv-0006 R5 14 2 None 3/0/4 wrong-only C production
cv-0022 R5 38 2 total 0/1/4 wrong-only C production
cv-0007 R6 21 3 None 0/0/0 none B target
cv-0008 R6 15 1 None 0/0/0 none A+B extraction+target
cv-0009 compare_mult 60 1 altogether 0/0/0 none A extraction
cv-0018 (control) 28 2 None 0/0/2 wrong-only (derivation) control — gate=baseline, already solves on serving (not a lift target)
cv-0019 additive 1200 2 None 0/0/0 none B target
cv-0017 (control) 438 5 None 0/0/2 wrong-only control — solves on serving
cv-0020 (control) 450 3 total 0/1/2 GOLD-BUILT control — D gate exemplar

Controls (cv-0017=case 0024, cv-0020=case 0021) already solve on serving via other paths; they are regression anchors, not lift targets. cv-0020 is the proof the build-then-gate pattern works: the pool builds the gold (450) but only commits it on serving because product_bridge's gate promotes it.

3. The four-stage failure taxonomy (and the wrong=0 firewall at each)

The composer is a pipeline: extract → characterize target → build production → certify (gate). A shape can be blocked at any stage; each stage refuses (never commits) when incomplete, so wrong=0 holds throughout.

  • Stage A — extraction-thin. The composer is starved of a needed quantity, so no chain is possible. cv-0001 (currency literal $16 not extracted — the derivation reader has no $N production; this is the gap I earlier overstated as closed), cv-0008, cv-0009. Firewall: no quantity → no candidate → refuse.
  • Stage B — target-uncharacterized. Quantities are present but extract_target returns aggregation=None, so the composer doesn't know what to build. cv-0003, cv-0005, cv-0007, cv-0019, cv-0021. Both R4 cases live here. Firewall: no target → no admissible chain → refuse.
  • Stage C — production-wrong. The composer builds chains, but the wrong ones (none equals gold), and the disagreement rule correctly refuses them. cv-0002, cv-0004, cv-0006, cv-0018, cv-0022. This is the hard core — "which quantities group, via which ops, in what order." Firewall: disagreement among wrong candidates → refuse.
  • Stage D — gold-built, needs gate. The composer builds the gold (possibly among rivals); a structural promotion gate certifies and commits it. Exemplar: cv-0020 via product_bridge. Firewall: the gate must prove the reading; a lone wrong chain with no rival (the 0016 pattern) is exactly what disagreement cannot catch, so the gate must be structural, never disagreement alone.

4. Execution sequence — one shape, end-to-end, instrumented

Each shape is a hypothesis to test against the gates, not a guaranteed win. Drive one shape through its blocked stages to green on the corpus, then check the sealed 1,319, before starting the next.

  1. R4 first — but it is NEW PRODUCTION, not a target tweak (verified 2026-06-03 by code-read of accumulate.py: _build_accumulation is single-referent gain/loss running-total with no goal/target/residual concept — target characterization alone does not unblock it). cv-0005 (single-referent goal Σchanges) is the nearest extension of GB-3b.1 and the first build; cv-0021 (multi-referent give-away to a remainder) is two layers harder — separate, do not bundle. Goal-recognition is a new wrong=0 surface (must refuse on ambiguous goal-language), and beware cv-0005 passing for the wrong reason (1034=3 also falls out of treating the goal as a start); the gate must consume the goal as a goal, and the real test is the sealed set, not cv-0005. Detailed first-build scope: §10.
  2. R1 next (cv-0001) — Stage A (extraction). Add currency-literal ($N) extraction to the derivation reader, then test whether the comparative-multiplicative production (multiplier × base) fires and builds 4×16=64. Extraction-first; different stage from R4, so it exercises a different pipeline leg.
  3. Then Stage C shapes (cv-0002/0004/0006/0022) — the genuine wall: building the correct grouping/op-order. Highest risk, do last, with the most instrumentation. Investigation task: docs/handoff/stage-c-composition-investigation-2026-06-03.md.

5. The promotion gate (generalize product_bridge, do not relax disagreement)

Every new production needs a co-designed structural gate, modeled on product_bridge but real:

  • Use extract_target + target_units to prove the built chain answers the asked question in the asked units (op-class + target + unit agreement), replacing the two-token whitelist with a general op-class/target proof.
  • A chain may commit only if it is the unique reading that structurally certifies; otherwise refuse. Disagreement is not the defensecv-0016/0016 shows a lone wrong chain commits with no rival to disagree. The gate, not the pool, is the wrong=0 firewall for committed compositions.

6. Acceptance gates (inherit ADR-0207 §6 verbatim)

train_sample stays wrong=0 (6/44/0 or better); the no-reference <N> times hazard stays refused; no partial chain commits with an unbound quantity; progress measured on the 22-case corpus AND the sealed 1,319 — a train_sample/corpus gain that does not move held-out does not count; each gain lands as composer/extractor enrichment behind a gate, audited to confirm it is not a new serving recognizer/injector branch (the §4 freeze).

7. Honest uncertainty

This is the research wall, not engineering. The payoff against the sealed 1,319 is genuinely unknown: the composer building gold on a corpus case does not mean the same production fires on held-out paraphrases, and Stage C (correct grouping/op-order) is an open research problem, not a known build. I can scope the work and the gates; I cannot promise R1/R4 transfer until the productions exist and the sealed number actually moves. The gates (§6) are what keep that honest — they will tell us the truth either way.

8. Correction recorded

I earlier asserted "extraction is substantially done; only EX-3 deferred; extraction is not the open lever." Stage A (cv-0001: $16 present in text, not extracted) shows that was overstated — the ADR-0179 lever was largely shipped for its four cases, but per-shape extraction gaps remain (currency literals at minimum). This does not unwind ADR-0207; the per-shape gaps are execution detail and live here, not in the ratification ADR.

9. Reproduction-methodology correction

The target-agg column was first computed by passing the full problem text to extract_target; serving passes only the question clause. For cv-0004 and cv-0006 the aggregation word is in the body, not the question, so full-text wrongly read combined/total where the serving-faithful call returns None (now corrected above). The other agg cells are unaffected (their agg word is in the question). This changes no stage (both are Stage C) and no claim. Methodology fix carried forward: reproduce serving behaviour with serving's inputs — extract_target gets the question clause, the composers get the full problem.

10. R4 first-build scope (from code-read of accumulate.py)

R4 is new production, not a target tweak — confirmed by reading the source (not a trace): compose_accumulation / _build_accumulation (GB-3b.1) computes single-referent gain/loss running-total start ± changes and consults no goal/target/residual concept. Hard bails: the anchor must establish exactly one quantity (accumulate.py:70); a new named actor refuses (continues_anchor_referent, :76/:143); no unambiguous licensed change cue refuses (:87/:149). So R4's goal accumulated cannot be expressed regardless of target — both R4 cases build 0 candidates at d9fc7f9e (latent, nothing passes for any reason yet).

  • cv-0005 first (single-referent, nearest extension of GB-3b.1). A new production recognizing (a) a goal quantity as a distinct anchor type (not a running total), (b) accumulated same-referent changes, (c) a residual question target, computing goal Σchanges. Single referent (Michael), so it reuses GB-3b.1's referent/cue guards.
  • cv-0021 second (harder). Multi-referent give-away to a remainder — (start desired_remainder) Σgiven. Outside GB-3b.1's single-referent scope (refused at continues_anchor_referent); needs multi-referent accumulation and residual handling. Two layers beyond cv-0005; do not bundle.
  • Gate (mandatory, co-designed). The residual chain commits only if extract_target structurally proves the residual question + op-class + unit agreement, and completeness consumes the goal quantity. Never disagreement alone (the 0016 firewall).
  • wrong=0 hazard (latent, build-time guard). 10 3 4 = 3 also falls out of mis-anchoring the goal as a running-total start — coincidental correctness. The production must consume the goal as a goal; the real test is the sealed set, not cv-0005's arithmetic.
  • Build-time discipline. Whoever builds instruments the exact _build_accumulation interaction and the new production's path through self_verifies (completeness must require the goal consumed) in-tree — the code-read scopes what to build; the runtime path confirms that it fires before the one-line wire.