# Composition-Capability Scope — v2 (re-anchored to shipped reality) Status: Proposed analysis / build-scoping. No serving or eval edits. Source of truth: `docs/claims_ledger.md`, `evals/gsm8k_math/train_sample/v1/report.json`, ADR-0174 **read in full (644 lines, incl. Implementation Notes + amended Phase 5, lines 300–533)**. Every empirical claim below reproduced live against `main` @ `3e29559`. > **v1 correction.** v1 aimed the plan at "land Phase 1; the unlanded value is > Phase 3/4." That was wrong — scoped from ADR-0174's forward sections only, > missing the ~225 lines documenting that **Phases 1–4 already shipped and are > wired into serving**. Verified on `main`: > `generate/comprehension/{state,lookback,contemplate,constraint_propagation}.py` > exist with `HYPOTHESIS_CAP`/`open_hypotheses`/`Hypothesis`/`UnknownHeld`/ > `reevaluate`/`contemplate`, and `reevaluate`+`contemplate` are imported and > invoked inside `generate/math_candidate_graph.py::parse_and_solve`. Metric still > **6/44/0**. v2 re-anchors to the ADR's own amended Phase 5. ## 0. The finding that scopes everything (corrected) The held-hypothesis machinery (ADR-0174 Phases 1–4) is **live in serving and the metric did not move.** A live `parse_and_solve` audit of all 44 refusals (§1) locates the wall precisely: **44/44 refuse at `branches_enumerated = 0` — upstream of the solver, which is never reached.** So the lift is **not** more reader machinery, and **not** solver operation coverage (the 8 op-kinds are never reached; see §1). The wall is **emission/representation**: the recognizer/parser/binding layer cannot construct an admissible multi-step candidate for these shapes, so nothing is fed to the (already-capable) solver. ADR-0174's "removing the 3–5 narrowness layers per case" is that emission/representation work. This inverts v1's thesis *and* supersedes the intermediate "operation coverage" thesis: the composition wall is at **emission/representation upstream of the solver**, not at the operation layer. ## 1. Verified current state on `main` | Layer | State | |---|---| | Held-hypothesis reader (P1–P4: state, constraint-propagation, lookback, contemplate) | **Shipped, wired into serving** candidate-graph | | `HYPOTHESIS_CAP`, vault>packs>audit precedence | already set/enforced (0174 OQ#1/#3 moot) | | `lifecycle.py` GSM8K-scoring dispatch | **inert** (admits 0/50) — retirable in 5a | | `lifecycle.py` / `audit.py` reader surface | **load-bearing** for the ADR-0172 teaching corridor — **keep** | | Solver operation kinds | **8 already exist** (verified by discovery): `add`, `subtract`, `transfer`, `multiply`, `divide`, `apply_rate`, `compare_additive`, `compare_multiplicative`. Missing arithmetic op kinds is **not** the blocker. | | **Live audit of all 44 refusals (ran `parse_and_solve` on `main`)** | **44/44 refuse at `branches_enumerated = 0`** — i.e. *upstream of the solver*. No branch is built, so the 8 operation kinds are **never reached**. `rate`/`ratio` also exist as first-class `SEMANTIC_ROLES`/`QUESTION_FORMS` (`binding_graph/model.py:42–65`); only `percent`/`accumulate` lack a named op-kind — moot while nothing reaches the solver. | | train_sample metric | **6 / 44 / 0** (`correct_min=10` not yet passed) | ## 2. The actual remaining work (ADR-0174 amended Phase 5) ### Phase 5a — retire the inert parallel parser (structural, ~0 lift) Retire the GSM8K-scoring-only inert dispatch (`_try_comprehension_reader` / `_try_reader_for_question`) + adapter + `use_reader` plumbing. Net **−1,038 LOC** (code + tests). **Keep** `lifecycle.py`/`audit.py` — their reader *refusals* feed the ADR-0172 math-contemplation teaching corridor (`teaching/math_*`, `evals/flywheel_demo`, `core/cli.py`). Low-risk; single-path serving. Acceptance: 6/44/0 byte-identical, capability-axis lanes 100% `wrong=0`, pinned-SHAs pass. ### Phase 5b — emission / representation buildout (semantic — the real lift) "This is where `correct` climbs toward 25. It is not a refactor." **Verified diagnosis (live `parse_and_solve` audit of all 44, on `main`):** every refusal occurs at `branches_enumerated = 0` — *upstream of the solver*. The 8 operation kinds are never reached. So 5b is **not** an operation-execution problem and **not** chiefly a new-primitive problem. It is an **emission/representation** problem: the recognizer/parser/binding layer cannot construct an admissible multi-step candidate for these shapes, so nothing is fed to the (already-capable) solver. The 44 split into two emission failure modes (verified, matches `report.json`): - **32 "recognizer matched but produced no injection"** — an anchor fired but the hypothesis-emitter declined to construct a candidate (cannot represent the multi-clause / derived / scoped structure). - **12 "no admissible candidate"** — the parser produced nothing admissible. **Sequencing implication:** fix emission/representation first; the `percent`/`accumulate` primitive question is **secondary** — answerable only after cases actually reach the solver. This *lowers* the wrong=0 risk: emitting existing ops behind the same admissibility predicates is lower-risk than new solver math. **5b first PR (smallest verified-tractable slice):** pick the emission failure mode with the highest count whose representation is bounded — and prove it reaches branch enumeration on ≥1 case without breaking `wrong=0`. **But the §9 completeness-gate precondition lands before any emission PR.** ## 3. Test anchors (reusable, relabeled to 5a/5b) 1. **Serving gate (`gsm8k_math`):** 5a byte-identical 6/44/0; 5b climbs (`correct ≥ 10` clears the ADR-0126 exit, target → 15 → 25), `wrong = 0` at every step. 2. **Regression nets (100% `wrong=0`):** G1–G5, S1; `anti_regression`; case-0050 pin. 3. **Determinism:** `trace_hash` over `open_hypotheses`; replay-equivalence. 4. **Serving freeze:** `verify_lane_shas.py` passes each step. 5. **Still-needed new asset (0174 OQ#5):** the curated ≤20-case multi-step validation sub-corpus — **unbuilt, still required for 5b measurement.** The §9 hard negatives are the first deposits into it. Author before 5b measurement. ## 4. Corrected sequencing 0. **Precondition (§9):** harden the comparative-multiplicative completeness tripwire so a dropped `×` clause refuses (as `twice` does). 5b emission on these shapes is unsafe until this lands. Driver test is FINAL and RED on `main` (see §9). 1. **5b gating analysis — done (§8):** the **emission/representation** audit ran and the 32 no-injection cases are sub-classified by representation. Highest-count bounded gap = **R1 derived/intermediate symbol (24/44)**. 2. **Author the ≤20-case multi-step validation sub-corpus** (test anchor for 5b). 3. **Phase 5a** (optional, can run in parallel — structural, low-risk). 4. **Phase 5b** — build the **emitter/representation** for R1 (derived-symbol) on the §8 near-pure exemplars (prove ≥1 case reaches `branches_enumerated > 0` and admits), each behind §3 gates **and the §9 precondition**. Not new operations — the 8 exist and are unreached. (Do **not** "land Phase 1" — shipped. Do **not** treat P3/P4 as the lever — shipped, metric flat.) ## 5. Relation to cross-subject testing 5b maturing the math `DomainSolver`'s emission/representation is still the precondition for wiring symbolic_logic as arena #2; the held-hypothesis reader + disagreement gate remain the shared, arena-portable primitives. `cross_domain_transfer` / `monotonic_learning` (both exist, with `contract.md`/`holdouts/`) become live tests once arena #2 exists. ## 6. Honest ceiling (corrected) The ≥15 climb is a **5b** outcome (emission/representation), **not** a P4 outcome — P4 shipped and we are at 6. Cases needing world knowledge (0040 legs) or representations 5b does not add stay refused = `wrong = 0` holding, not failure. ## 7. Open questions for the build lane 1. **Answered by the live audit:** the binding constraint is emission upstream of the solver (44/44 at `branches_enumerated=0`), not operation execution. Remaining: rank the §8 R-classes by the size of the bounded-representation slice per class. 2. Of the 44, which are emission-fixable with existing ops vs (a) genuinely need a `percent`/`accumulate` primitive *after* emission, vs (b) world-knowledge/out-of- scope permanent refusals (e.g. 0040 legs-per-animal). The audit gives the upstream verdict; this split needs per-case reader-trace reads. 3. 5a pre-flight: confirm the exact inert-dispatch LOC and that no teaching-corridor consumer breaks when the GSM8K-scoring dispatch is retired. --- ## 8. Verified per-case representation classification (live audit, all 44) Structural reading of each of the 44 (what representation the emitter must build for emission to succeed), grounded in the case text + the live `parse_and_solve` locus. Multi-tagged; a case usually needs several. Frequency across the 44: | Representation gap | # cases (multi-tagged) | |---|---:| | R5 — multi-step rate/duration/scalar | 27 | | R1 — derived/intermediate symbol | 24 | | R6 — percent/fraction mutation (no op-kind) | 18 | | R4 — accumulation/residual | 10 | | R2 — inverse target | 6 | | R3 — subset/partition scope | 3 | | R7 — world-knowledge (permanent refusal) | 1 | **Highest-leverage gap: R1 (derived/intermediate symbol)** — 24/44 need to compute an intermediate quantity and reuse it downstream. Its lowest-arity exemplars are nearly pure R1: 0027 (Twitter = half of IG+FB), 0008 (total beads → ÷ per bracelet), 0029 (keyboard = 3× mouse → sum), 0038 (×3 → sum). Contrastive proof it is the emission gap (parseable aggregate question form): `"Nicole has 400 cards. Cindy has 800 cards. How many cards do they have together?"` **admits (1200)**; `"…Cindy has twice as many cards. How many cards do they have together?"` reaches `branches_enumerated=1` but **refuses** (completeness: scalar `2.0` unconsumed). The stated-sum reaches the solver; the derived form does not. R7 (0040 legs-per-animal) is a permanent refusal — world knowledge. **First 5b slice (recommended):** derived/intermediate-symbol emission, validated on the four near-pure exemplars above — move them from refused to admitted, `wrong=0` preserved. **But see §9 first — it is a hard precondition.** ## 9. ⚠ Latent `wrong=0` hazard surfaced by the live audit (gate gap) Contrastive probes (not in the 50-case sample) surface a reproducible **admitted-wrong** path. **Use the parseable aggregate question form `"...do they have together?"`** — the short form `"How many apples together?"` refuses upstream at question-parse (`branches_enumerated=0`) and does NOT exercise this `be=1` completeness gap. ``` "Tom has 7 apples. Jerry has 3 times as many apples. How many apples do they have together?" → admitted=True, answer=7, branches_enumerated=1 (correct = 28) # base returned, clause dropped "...Jerry has five times as many apples. ..." → admitted, answer=7 (correct = 42) "...Jerry has twice as many apples. ..." → refused, be=1 (completeness: scalar 2.0 unconsumed) # safe ``` **Broadened surface (all verified RED on `main`).** The hazard is the **no-reference** comparative-multiplier surface across all four connectives and both cardinal forms: ``` times as many times more times the number of times the ``` …for `` ∈ {digit 2/3/5/…, word two/three/five/…, N≥2}. Every one admits the base. `twice` / `double` / `double the` (no digit/cardinal) refuse safely. **Root cause (verified by reading the emitted `MathProblemGraph`).** The admitted graph contains a spurious initial: `InitialPossession(entity='Jerry', quantity=Quantity(value=3, unit='times'))`. 1. `quantity_values_in_text` is **symmetric** (registers 2.0 for `twice` *and* 3.0 for `3 times`) — **not** a quantity-extraction asymmetry (this overturns an earlier guess). 2. For the no-ref surface, **neither** serving comparative regex fires (`_COMPARE_MULT_ANCHOR_RE` / `_COMPARE_MULT_NTIMES_RE` both require an "as ``" tail the probe lacks). `comparatives.py::_N_TIMES_RE` is the **disjoint** `derivation/` reader — off the serving path — so it is *not* the locus either. 3. Instead `recognizer_match.py::_match_discrete_count_statement` (open regex `_extract_discrete_count_re_open`) captures the multiplier cardinal as a **count**, and `recognizer_anchor_inject.py::_build_initial_from_discrete_count` builds `CandidateInitial(value=N, unit='times', entity=)` — note the unit is the literal `'times'` token, **not** the counted unit. 4. That bogus initial **consumes** the scalar N, so completeness sees `uncovered = {N, base} − {N, base} = ∅` → admits. The answer sums only the counted unit (`apples`) and returns the base. `twice`/`double` carry no cardinal the discrete-count regex grabs → no spurious initial → scalar unconsumed → existing guard refuses. **Recommended guard shape (refusal-only, `wrong=0` preserving).** Make the discrete-count recognizer **decline** when its cardinal sits in a ` times {as many | more | the number of | the} ` comparative-multiplier context — equivalently, refuse to build a discrete-count initial whose unit is the literal `'times'`. The no-ref form then refuses (like no-ref `twice`) until real no-ref comparative-multiplicative *emission* lands in 5b. The fix MUST NOT regress the controls below. **Controls (verified on `main`).** - **With-ref ` times` already solves — must stay green.** train_sample **case 0024**: *"Sidney does 20 jumping jacks on Monday, 36 on Tuesday, 40 on Wednesday, and 50 on Thursday. Brooke does three times as many jumping jacks as Sidney. How many jumping jacks did Brooke do?"* → **438** (committed verdict: correct). Also `dice 3×` → 80. - **Existing safe refusals — must stay refused.** no-ref `twice`, no-ref `double the`, with-ref `twice` (Ivan/Jerry dice). **Driver test (FINAL, RED on `main`).** `test_completeness_guard_ntimes_noref_hazard.py` — **10 no-ref hazard cases** (the broadened surface × digit/word) that MUST refuse, plus **2 with-ref must-solve controls** (0024 → 438, dice → 80) and **3 must-still-refuse controls**. Verified live: **10 failed, 5 passed** on `main` (hazard RED, controls green). This is the first concrete 5b PR, ahead of any R1 emission work. **Scope — stated precisely.** This is a **latent gate gap, not a live-metric violation.** The two layers are distinct: the §1 wall is at `branches_enumerated=0` (emission/question-parse); this hazard lives at `branches_enumerated=1` (the completeness gate). The real 44 all refuse at `be=0` — upstream of `be=1` — so none reach this gap today; that is *why* it is latent. train_sample 6/44/0 is reproduced live. The full-test `0/0/1319` is **not** re-verified here — it is a sealed, recorded measurement per `docs/claims_ledger.md` row A / `ADR-0119.7` (ciphertext `evals/gsm8k_math/holdouts/v1/cases.jsonl.age`), not a CI-reproducible artifact; cited, not re-verified (the citation and ciphertext both exist on `main`). **Why it gates 5b.** Phase 5b's explicit goal is to make comparative-multiplicative cases *reach the graph*. If emission improves while this completeness gap remains, 5b would **convert latent into live** — admitting wrong answers on exactly the ` times` shape it is trying to solve. Therefore: **hardening the comparative-multiplicative completeness tripwire (so a dropped `×` clause refuses, as `twice` does) is a hard precondition for any 5b emission PR.** The §9 driver test is the ready-made hard negative for the ADR-0174 OQ#5 validation sub-corpus.