core/docs/decisions/ADR-0177-cue-precision-learning.md

ADR-0177 — Cue-Precision Learning: from practice eliminations to trusted cue→op patterns

Status: Proposed Date: 2026-05-28 Author: Shay Anchor: thesis-decoding-not-generating Builds on: ADR-0175 — Calibrated Attempt-and-Eliminate Learning (the reliability ledger + conservative_floor + θ ceilings + the sealed practice loop — reused, keyed by cue-pattern) and ADR-0176 — Multi-Step Composition (the search whose gold-checked candidate chains are the training signal)

---

Context — the lever MS-1→MS-3 proved, and the honesty it forces

The multi-step search (MS-3) is built, deterministic, and wrong=0-safe, but low-coverage by design: when several arithmetic shapes self-verify and disagree, the uniqueness rule refuses, because broad cues cannot tell which operation the text actually licenses. The lever, repeatedly, is cue precision: learning, from the practice eliminations, which (cue → op) readings are reliable.

ADR-0175 §"Phase 3b finding" already named the prerequisite: self-verification is necessary but not sufficient (9/13 self-verified attempts were wrong). Before Phase 5 may let self-verification gate proposals, the gate must be made sufficient — and that is exactly what a learned cue-pattern reliability provides.

This ADR scopes that learning. It is the self-supervised ("learn-from-questions") half of the learning system; the packs half (comparatives, superordinate units) supplies the irreducible world-facts (ADR-0175 §10).

Two distinct gaps the eliminations expose

The MS-3 eliminations are not uniformly "wrong cue→op." Profiling them:

• Gap A — cue→op precision. Given a present cue, which op does it license here? "for 10 reps" → multiply; "works for 3 hours" → not. "and" → sometimes sum, sometimes mere conjunction. (0021: "for"→multiply was right.)
• Gap B — compositional structure. Which quantities group, in what order/op tree. The dominant MS-3 failure: product-of-all when the answer needs a sub-grouping or a mixed chain (0019 120000 vs 660; 0041 2048 vs 6). The op may be right; the structure is wrong.

Cue-precision is Gap A. It is necessary but, on its own, does not close Gap B.

The mechanism

A per-cue-pattern reliability ledger (reusing ADR-0175's ClassTally + conservative_floor, keyed by a cue-pattern string instead of a capability axis), fed by gold-labeling the search's candidate chains in the sealed practice lane.

Pattern key: (cue, op, unit_shape) where unit_shape ∈ {cross_unit, same_unit} — e.g. ("per", multiply, cross_unit). The unit_shape dimension captures the most load-bearing precision (cross-unit multiplication is the aggregate signal) without the instant starvation of keying on full operand-unit pairs. Finer context (neighbouring lexemes) is a scale-dependent refinement, not v1.

Credit assignment (per-case, contrastive via gold): for each practice case, the search emits candidate chains; label each by gold (value == answer); for every step's pattern in a chain, record +correct if the chain matched gold else +wrong. Reliability per pattern = conservative_floor(correct, correct+wrong). The pessimistic floor + N_min suppress the noise of coarse attribution (a pattern earns trust only after many clean appearances). Learning does not depend on the search resolving — it learns from labelling candidates, separate from the resolve/refuse decision.

Three uses, increasing risk:

• U1 — self-verification trust (the near-term value). A chain may produce a serving proposal only if every step's cue-pattern reliability ≥ θ_serve. This makes self-verification sufficient (closes the ADR-0175 3b gap). With a cold ledger nothing clears θ_serve → no proposals → safe: it prevents the 3b "propose junk 70% of the time" disaster by construction. Its value is correctness/ trust, not coverage.
• U2 — search guidance. Prefer/try high-reliability patterns first; deprioritise unproven shapes. Reduces wrong attempts. Refuse-preferring (pruning a right-but- unproven shape only costs coverage, never wrong=0).
• U3 — disagreement resolution (the coverage lever). When shapes disagree, resolve to the one whose patterns decisively dominate in reliability instead of refusing — hard-gated: only when the winner ≥ θ AND beats the alternatives by a margin; ties and near-ties refuse. Relaxes uniqueness using earned evidence, not a guess. Sealed practice checks it against gold; serving additionally requires ratification (Phase 5).

The bottleneck — why cue-precision cannot stand alone yet (the load-bearing honesty)

A cue-pattern earns positive signal only from a chain that matches gold. On the current blunt shapes (product-of-all / sum-of-all), only ~4 of 50 cases produce a gold-matching candidate chain. The other ~43 produce only wrong chains, so:

1. The ledger is starved of positive signal — dominated by +wrong. Almost no pattern reaches N_min of clean appearances → reliabilities stay near zero → U1 trusts nothing, U3 resolves nothing. The mechanism runs but learns little.
2. Structure failures (Gap B) pollute cue→op credit — a (cue, multiply) whose op was right but appeared in a product-of-all chain that was structurally wrong gets +wrong. Coarse attribution conflates Gap A and Gap B, so a correct op is penalised for a structure error.
3. Data starvation — 50 cases, each cue appearing in a handful → even uncorrupted, the counts are far below N_min. Compounding needs volume.

Consequence — cue-precision is tightly coupled to richer compositional shapes (Gap B) and to scale. Patterns can only earn reliability once the search can produce gold-matching chains for them; that requires richer, guided shapes (Gap B). And richer shapes explode combinatorially without cue-precision to prune them. They co-evolve: Gap B supplies gold-matching candidates → cue-precision earns signal → cue-precision prunes Gap B's search. Neither standalone closes coverage on the current substrate.

Recommended sequencing (the honest answer)

1. Build the cue-precision substrate now (CP-1, CP-2 = U1). The mechanism + the self-verification trust gate. Near-term value is correctness: it makes the Phase 5 proposal gate honest (only earned-reliability patterns may propose; cold ledger ⇒ refuse), permanently closing the 3b "necessary-not-sufficient" hazard. Low risk, no coverage promise.
2. Then richer guided compositional shapes (Gap B, a sibling to ADR-0176 / its own ADR), pruned by the cue-precision ledger. This is what produces gold-matching chains for more cases → gives cue-precision positive signal → and is the actual flip-count lever.
3. Scale (more practice problems, ADR-0163 §Phase F) is what makes the learning compound. On 50 cases this is mechanism-demonstration, not payoff.

So: cue-precision learning is the trust substrate and the pruning engine, not the coverage unlock by itself. Coverage = Gap B (richer guided search) × scale, with cue-precision as the safety gate and the prune.

wrong=0 obligations (must be proven, not asserted)

Each needs a failing-under-violation test (CLAUDE.md §Schema-Defined Proof Obligations):

1. Cold ledger ⇒ no regression. With an empty/low ledger, U1 trusts nothing and U3 resolves nothing — behaviour identical to today's refuse-on-disagreement. A test fails if a cold ledger resolves a previously-refused disagreement.
2. Ties refuse. U3 with two patterns at equal (or within-margin) reliability + disagreeing chains → refuse. A test fails if a tie resolves.
3. θ-gated serving. No pattern below θ_serve may contribute to a serving proposal; serving stays wrong=0; the search stays sealed (no serving import).
4. Credit noise cannot flip a served answer. The conservative floor + N_min + margin + ratification (Phase 5) gate it; the ADR-0175 gold tether audits per-pattern reliability against gold and contracts appetite on divergence.
5. Determinism/replay. Ledger updates, the floor, and the tiebreak are deterministic; byte-stable across runs.

Sub-phases

• CP-1 — cue-pattern ledger + credit assignment. (cue, op, unit_shape) ledger; per-case gold-labelling of candidate chains → per-pattern counts. Sealed practice. Tests: credit attribution; determinism; cold-ledger reliabilities are 0.
• CP-2 — self-verification trust (U1) + search guidance (U2). A chain proposes only if its patterns clear θ; the search orders/prunes by reliability. Tests: invariant #1 (cold ⇒ no proposals, no regression); U2 never causes a wrong=0 violation.
  • CP-2a — ledger training + measurement (landed). The training step (generate/cue_precision/trainer.py) folds gold-labelled candidate readings from the real search enumerators (search._sentence_candidates + multistep.candidate_chains) into the CP-1 ledger; the measurement (evals/gsm8k_math/practice/v1/cue_precision_report.py) reports per-pattern reliability over the 200 sealed cases (50 train_sample + 150 ADR-0163-F additive). Inert: trains/reports only, consulted by nobody — serving 3/47/0 byte-identical, practice counts unchanged. search_chain now delegates enumeration to the public candidate_chains (verified byte-identical).
  • CP-2a finding (load-bearing): no cue is reliable yet — CP-2b is blocked on candidate generation, not on the ledger. Trained over 200 cases, every (cue, op, unit_shape) pattern floors at ≈ 0.0 (best: for·multiply·cross_unit = 0.0116 at 2/34; each·multiply ≈ 0.006; times·multiply 0/57, total·add 0/47). The blunt product/sum-of-all readings the search proposes are almost always wrong vs gold, so the conservative floor correctly trusts nothing. The lever is therefore not "trust high-reliability cues" (there are none) — it is that the candidate readings must get less crude (clause structure + referent-awareness, i.e. ADR-0178 GB-3b) before any pattern earns reliability. Cue-precision (CP-2b) and compositional structure (GB-3b) are coupled, and structure comes first. This is the ADR-0177 §"bottleneck" honesty, now measured rather than asserted. (Table reproducible via the report; deterministic.)
• CP-3 — disagreement resolution (U3), wrong=0-first. Margin+θ-gated resolution; prove ties refuse before enabling resolution. Measure any coverage delta.
• CP-4 — measurement + scale dependency. Per-pattern reliability table; the (data-starved) compounding curve; honest report that flip-count payoff awaits Gap B + scale.

Acceptance criteria (Proposed → Accepted)

1. CP-1/CP-2 land; invariant #1 (cold ⇒ no regression) and θ-gating proven; serving wrong=0 unchanged; the self-verification trust gate is demonstrable (a chain with earned patterns proposes; one without refuses).
2. CP-3 proves ties/near-ties refuse before any reliability-based resolution.
3. Determinism/replay + seal invariants hold; capability lanes G1–G5/S1 stay 100% wrong=0.
4. The measurement honestly reports the data-starvation/Gap-B bottleneck rather than a coverage claim the 50-case substrate cannot support.

Cross-references

• Substrate: ADR-0175 (ClassTally, conservative_floor, θ ceilings, gold tether, the sealed practice lane) — reused, keyed by cue-pattern.
• Signal source: ADR-0176 (search_chain candidate chains, gold-labelled in practice).
• Co-requisite (the flip lever): richer guided compositional shapes (Gap B) — a follow-on ADR; cue-precision prunes its search and learns from its gold-matching chains.
• Scale: ADR-0163 §Phase F — the volume that makes the loop compound.
• Thesis: thesis-decoding-not-generating — the engine learns which readings are true by elimination against gold; it is not handed a library of founds.