docs(adr-0177): scope cue-precision learning (the self-supervised half)
The lever MS-1->MS-3 proved: learn which (cue->op) readings are reliable from practice eliminations, closing ADR-0175's self-verification 'necessary-not- sufficient' gap before Phase 5. Mechanism: per-(cue, op, unit_shape) reliability ledger (reuses ADR-0175 ClassTally + conservative_floor), fed by gold-labelling search candidate chains. Three uses: U1 self-verification TRUST (near-term value: makes the Phase 5 proposal gate honest; cold ledger => refuse, no junk proposals), U2 search guidance, U3 disagreement resolution (coverage lever, hard-gated: margin+theta, ties refuse). Load-bearing honesty (the bottleneck): a pattern earns POSITIVE signal only from a gold-MATCHING chain; current blunt shapes match gold for ~4/50 cases, so the ledger is starved (all-blame) AND structure failures (Gap B) pollute cue->op credit (a correct op penalised for a product-of-ALL structure error). Plus data starvation (50 cases << N_min). So cue-precision is COUPLED to richer guided shapes (Gap B) + scale; it co-evolves (Gap B supplies gold-matching candidates -> cue-precision earns signal -> prunes Gap B's search). It is the TRUST substrate + pruning engine, NOT the coverage unlock by itself. Recommended sequencing: CP-1/CP-2 (mechanism + self-verification trust, near-term correctness value) now; richer guided shapes (Gap B) next as the flip lever; scale makes it compound. wrong=0 obligations: cold=>no regression, ties refuse, theta-gated serving, credit-noise can't flip serving (floor+N_min+margin+ratification+gold-tether), determinism. Sub-phases CP-1..CP-4.
This commit is contained in:
parent
ebaa603b68
commit
b57e5dca9a
1 changed files with 178 additions and 0 deletions
178
docs/decisions/ADR-0177-cue-precision-learning.md
Normal file
178
docs/decisions/ADR-0177-cue-precision-learning.md
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
# ADR-0177 — Cue-Precision Learning: from practice eliminations to trusted cue→op patterns
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-28
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Builds on:** [ADR-0175 — Calibrated Attempt-and-Eliminate Learning](./ADR-0175-calibrated-attempt-and-eliminate-learning.md) (the reliability ledger + `conservative_floor` + θ ceilings + the sealed practice loop — reused, keyed by cue-pattern) and [ADR-0176 — Multi-Step Composition](./ADR-0176-multistep-composition-question-targeting.md) (the search whose gold-checked candidate chains are the training signal)
|
||||
|
||||
---
|
||||
|
||||
## Context — the lever MS-1→MS-3 proved, and the honesty it forces
|
||||
|
||||
The multi-step search (MS-3) is built, deterministic, and wrong=0-safe, but
|
||||
**low-coverage by design**: when several arithmetic shapes self-verify and
|
||||
disagree, the uniqueness rule refuses, because broad cues cannot tell which
|
||||
operation the text actually licenses. The lever, repeatedly, is **cue precision**:
|
||||
learning, from the practice eliminations, which `(cue → op)` readings are reliable.
|
||||
|
||||
ADR-0175 §"Phase 3b finding" already named the prerequisite: self-verification is
|
||||
**necessary but not sufficient** (9/13 self-verified attempts were wrong). Before
|
||||
Phase 5 may let self-verification gate proposals, the gate must be made *sufficient*
|
||||
— and that is exactly what a learned cue-pattern reliability provides.
|
||||
|
||||
This ADR scopes that learning. It is the **self-supervised ("learn-from-questions")**
|
||||
half of the learning system; the **packs** half (comparatives, superordinate units)
|
||||
supplies the irreducible world-facts (ADR-0175 §10).
|
||||
|
||||
### Two distinct gaps the eliminations expose
|
||||
|
||||
The MS-3 eliminations are *not* uniformly "wrong cue→op." Profiling them:
|
||||
|
||||
- **Gap A — cue→op precision.** Given a present cue, which op does it license *here*?
|
||||
"for 10 reps" → multiply; "works for 3 hours" → not. "and" → sometimes sum,
|
||||
sometimes mere conjunction. (0021: "for"→multiply was right.)
|
||||
- **Gap B — compositional structure.** *Which* quantities group, in what order/op
|
||||
tree. The dominant MS-3 failure: product-of-**all** when the answer needs a
|
||||
sub-grouping or a mixed chain (0019 `120000` vs `660`; 0041 `2048` vs `6`). The
|
||||
op may be right; the *structure* is wrong.
|
||||
|
||||
Cue-precision is **Gap A**. It is necessary but, on its own, does not close Gap B.
|
||||
|
||||
## The mechanism
|
||||
|
||||
A **per-cue-pattern reliability ledger** (reusing ADR-0175's `ClassTally` +
|
||||
`conservative_floor`, keyed by a cue-pattern string instead of a capability axis),
|
||||
fed by gold-labeling the search's candidate chains in the sealed practice lane.
|
||||
|
||||
**Pattern key:** `(cue, op, unit_shape)` where `unit_shape ∈ {cross_unit, same_unit}`
|
||||
— e.g. `("per", multiply, cross_unit)`. The `unit_shape` dimension captures the most
|
||||
load-bearing precision (cross-unit multiplication is the *aggregate* signal)
|
||||
without the instant starvation of keying on full operand-unit pairs. Finer context
|
||||
(neighbouring lexemes) is a scale-dependent refinement, not v1.
|
||||
|
||||
**Credit assignment (per-case, contrastive via gold):** for each practice case,
|
||||
the search emits candidate chains; label each by gold (value == answer); for every
|
||||
step's pattern in a chain, record `+correct` if the chain matched gold else
|
||||
`+wrong`. Reliability per pattern = `conservative_floor(correct, correct+wrong)`.
|
||||
The pessimistic floor + `N_min` suppress the noise of coarse attribution (a pattern
|
||||
earns trust only after many clean appearances). Learning does **not** depend on the
|
||||
search *resolving* — it learns from labelling candidates, separate from the
|
||||
resolve/refuse decision.
|
||||
|
||||
**Three uses, increasing risk:**
|
||||
|
||||
- **U1 — self-verification *trust* (the near-term value).** A chain may produce a
|
||||
*serving* proposal only if every step's cue-pattern reliability ≥ `θ_serve`. This
|
||||
makes self-verification **sufficient** (closes the ADR-0175 3b gap). With a cold
|
||||
ledger nothing clears `θ_serve` → no proposals → **safe**: it prevents the 3b
|
||||
"propose junk 70% of the time" disaster by construction. Its value is *correctness/
|
||||
trust*, not coverage.
|
||||
- **U2 — search guidance.** Prefer/try high-reliability patterns first; deprioritise
|
||||
unproven shapes. Reduces wrong attempts. Refuse-preferring (pruning a right-but-
|
||||
unproven shape only costs coverage, never wrong=0).
|
||||
- **U3 — disagreement resolution (the coverage lever).** When shapes disagree,
|
||||
resolve to the one whose patterns *decisively dominate* in reliability instead of
|
||||
refusing — **hard-gated**: only when the winner ≥ `θ` AND beats the alternatives by
|
||||
a margin; ties and near-ties refuse. Relaxes uniqueness using *earned evidence*,
|
||||
not a guess. Sealed practice checks it against gold; serving additionally requires
|
||||
ratification (Phase 5).
|
||||
|
||||
## The bottleneck — why cue-precision cannot stand alone yet (the load-bearing honesty)
|
||||
|
||||
A cue-pattern earns **positive** signal only from a chain that **matches gold**. On
|
||||
the current blunt shapes (product-of-all / sum-of-all), only ~4 of 50 cases produce
|
||||
a gold-matching candidate chain. The other ~43 produce only wrong chains, so:
|
||||
|
||||
1. **The ledger is starved of positive signal** — dominated by `+wrong`. Almost no
|
||||
pattern reaches `N_min` of *clean* appearances → reliabilities stay near zero →
|
||||
U1 trusts nothing, U3 resolves nothing. The mechanism runs but learns little.
|
||||
2. **Structure failures (Gap B) pollute cue→op credit** — a `(cue, multiply)` whose
|
||||
op was *right* but appeared in a product-of-*all* chain that was structurally
|
||||
*wrong* gets `+wrong`. Coarse attribution conflates Gap A and Gap B, so a correct
|
||||
op is penalised for a structure error.
|
||||
3. **Data starvation** — 50 cases, each cue appearing in a handful → even uncorrupted,
|
||||
the counts are far below `N_min`. Compounding needs **volume**.
|
||||
|
||||
**Consequence — cue-precision is tightly coupled to richer compositional shapes
|
||||
(Gap B) and to scale.** Patterns can only earn reliability once the search can
|
||||
produce gold-matching chains for them; that requires richer, *guided* shapes (Gap B).
|
||||
And richer shapes explode combinatorially without cue-precision to prune them. They
|
||||
**co-evolve**: Gap B supplies gold-matching candidates → cue-precision earns signal →
|
||||
cue-precision prunes Gap B's search. Neither standalone closes coverage on the
|
||||
current substrate.
|
||||
|
||||
## Recommended sequencing (the honest answer)
|
||||
|
||||
1. **Build the cue-precision substrate now (CP-1, CP-2 = U1).** The *mechanism* + the
|
||||
**self-verification trust gate**. Near-term value is **correctness**: it makes the
|
||||
Phase 5 proposal gate honest (only earned-reliability patterns may propose; cold
|
||||
ledger ⇒ refuse), permanently closing the 3b "necessary-not-sufficient" hazard.
|
||||
Low risk, no coverage promise.
|
||||
2. **Then richer guided compositional shapes (Gap B, a sibling to ADR-0176 / its own
|
||||
ADR), pruned by the cue-precision ledger.** This is what produces gold-matching
|
||||
chains for more cases → gives cue-precision positive signal → and is the actual
|
||||
**flip-count** lever.
|
||||
3. **Scale (more practice problems, ADR-0163 §Phase F)** is what makes the learning
|
||||
*compound*. On 50 cases this is mechanism-demonstration, not payoff.
|
||||
|
||||
So: cue-precision learning is the **trust substrate and the pruning engine**, not the
|
||||
coverage unlock by itself. Coverage = Gap B (richer guided search) × scale, with
|
||||
cue-precision as the safety gate and the prune.
|
||||
|
||||
## wrong=0 obligations (must be *proven*, not asserted)
|
||||
|
||||
Each needs a failing-under-violation test (CLAUDE.md §Schema-Defined Proof Obligations):
|
||||
|
||||
1. **Cold ledger ⇒ no regression.** With an empty/low ledger, U1 trusts nothing and
|
||||
U3 resolves nothing — behaviour identical to today's refuse-on-disagreement. A
|
||||
test fails if a cold ledger resolves a previously-refused disagreement.
|
||||
2. **Ties refuse.** U3 with two patterns at equal (or within-margin) reliability +
|
||||
disagreeing chains → refuse. A test fails if a tie resolves.
|
||||
3. **θ-gated serving.** No pattern below `θ_serve` may contribute to a serving
|
||||
proposal; serving stays `wrong=0`; the search stays sealed (no serving import).
|
||||
4. **Credit noise cannot flip a served answer.** The conservative floor + `N_min` +
|
||||
margin + ratification (Phase 5) gate it; the ADR-0175 **gold tether** audits
|
||||
per-pattern reliability against gold and contracts appetite on divergence.
|
||||
5. **Determinism/replay.** Ledger updates, the floor, and the tiebreak are
|
||||
deterministic; byte-stable across runs.
|
||||
|
||||
## Sub-phases
|
||||
|
||||
- **CP-1 — cue-pattern ledger + credit assignment.** `(cue, op, unit_shape)` ledger;
|
||||
per-case gold-labelling of candidate chains → per-pattern counts. Sealed practice.
|
||||
Tests: credit attribution; determinism; cold-ledger reliabilities are 0.
|
||||
- **CP-2 — self-verification trust (U1) + search guidance (U2).** A chain proposes
|
||||
only if its patterns clear `θ`; the search orders/prunes by reliability. Tests:
|
||||
invariant #1 (cold ⇒ no proposals, no regression); U2 never causes a wrong=0
|
||||
violation.
|
||||
- **CP-3 — disagreement resolution (U3), wrong=0-first.** Margin+θ-gated resolution;
|
||||
**prove ties refuse before enabling resolution.** Measure any coverage delta.
|
||||
- **CP-4 — measurement + scale dependency.** Per-pattern reliability table; the
|
||||
(data-starved) compounding curve; honest report that flip-count payoff awaits
|
||||
Gap B + scale.
|
||||
|
||||
## Acceptance criteria (Proposed → Accepted)
|
||||
|
||||
1. CP-1/CP-2 land; invariant #1 (cold ⇒ no regression) and θ-gating proven; serving
|
||||
`wrong=0` unchanged; the self-verification *trust* gate is demonstrable (a chain
|
||||
with earned patterns proposes; one without refuses).
|
||||
2. CP-3 proves ties/near-ties refuse before any reliability-based resolution.
|
||||
3. Determinism/replay + seal invariants hold; capability lanes G1–G5/S1 stay 100%
|
||||
`wrong=0`.
|
||||
4. The measurement honestly reports the data-starvation/Gap-B bottleneck rather than
|
||||
a coverage claim the 50-case substrate cannot support.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Substrate:** [ADR-0175](./ADR-0175-calibrated-attempt-and-eliminate-learning.md)
|
||||
(`ClassTally`, `conservative_floor`, θ ceilings, gold tether, the sealed practice
|
||||
lane) — reused, keyed by cue-pattern.
|
||||
- **Signal source:** [ADR-0176](./ADR-0176-multistep-composition-question-targeting.md)
|
||||
(`search_chain` candidate chains, gold-labelled in practice).
|
||||
- **Co-requisite (the flip lever):** richer *guided* compositional shapes (Gap B) —
|
||||
a follow-on ADR; cue-precision prunes its search and learns from its gold-matching
|
||||
chains.
|
||||
- **Scale:** ADR-0163 §Phase F — the volume that makes the loop compound.
|
||||
- **Thesis:** [[thesis-decoding-not-generating]] — the engine learns which readings
|
||||
are true by elimination against gold; it is not handed a library of founds.
|
||||
Loading…
Reference in a new issue