docs(adr-0175): calibrated attempt-and-eliminate learning architecture
Proposed ADR + session derivation doc capturing the 2026-05-28 design discussion that took GSM8K Phase 5b from 'build another matcher' to a self-calibrating problem solver. Session doc (docs/sessions/SESSION-2026-05-28-...): the full journey — problem (per-shape matchers can't compound; 79% need mul, 0% single-step), dead-ends (brute-force spurious matches; 0021 is the only single-sentence case and it's idiosyncratic), and the four pivots that converged on the solution. ADR-0175 (Proposed): the decision — - two regimes: serving (wrong=0, unchanged) vs sealed practice (attempt-and-eliminate; wrong is the learning signal) - proof-carrying seal: practice never writes serving; ratification only - deterministic attempt/refuse gate: reliability(C) / theta_required >= 1 (NOT RL; regimes collapse the reward side so only reliability is quantified) - per-class calibration ledger of replayable COUNTS + conservative lower bound; human-set theta ceilings raised only on evidence - checkability ladder (gold > convergent self-verification > consistency-only), privilege proportional to reversibility; provenance + gold tether against correlated self-delusion - diagnostic refusal routes skill vs knowledge vs ambiguity; three compounding stores (vault/packs/pruning); self-proving acquisition narrows human input without bypassing the gate - five proof-obligation invariants (wrong=0 on serving, no spurious banking, determinism, no self-authorization, retractability) Supersedes the matcher-oriented ADR-0174 5b sub-phases; repoints the 0174 eliminate/reevaluate/contemplate substrate from reading to solving. Open question: shape of conservative_floor + N_min.
This commit is contained in:
Shay
parent
d1dbda24fc
commit
8c2e469be0
2 changed files with 583 additions and 0 deletions
|
|
@ -0,0 +1,277 @@
|
|||
# ADR-0175 — Calibrated Attempt-and-Eliminate Learning: Two Regimes Under wrong=0
|
||||
|
||||
**Status:** Proposed
|
||||
**Date:** 2026-05-28
|
||||
**Author:** Shay
|
||||
**Anchor:** [[thesis-decoding-not-generating]]
|
||||
**Discussion / derivation:** [SESSION-2026-05-28 — From "Another Matcher" to a Self-Calibrating Problem Solver](../sessions/SESSION-2026-05-28-risk-reward-learning-architecture.md)
|
||||
**Builds on:** [ADR-0174 — Held-Hypothesis Comprehension](./ADR-0174-held-hypothesis-comprehension.md) (the `eliminate_violating` / `reevaluate` / `contemplate` substrate — here **repointed from reading to solving**), the calibration module, capability axes G1–G5, the round-trip filter + multi-branch disagreement rule, teaching-safety (proposal-only / reviewed = the seal)
|
||||
**Supersedes:** the matcher-oriented Phase 5b sub-phases (5b.1 single-sentence / 5b.2 cross-sentence / 5b.3 deep) in ADR-0174 — they collapse into *instances* of this architecture.
|
||||
|
||||
---
|
||||
|
||||
## Context — why per-shape matching cannot compound, and the contradiction underneath
|
||||
|
||||
ADR-0163/0164/0174 moved the engine from `0/50/0` to `3/47/0` on GSM8K
|
||||
train_sample by building, one at a time, recognizers and injectors for specific
|
||||
sentence shapes. Each addition lifts `correct` by 0–2 cases. The 2026-05-28
|
||||
measurement (GSM8K's own `<<a*b=c>>` calculator annotations over the 47 refused
|
||||
cases) explains why this is structural, not incidental:
|
||||
|
||||
- **37/47 (79%)** of refused cases need multiplication; **43/47** need mul-or-div;
|
||||
**0/47** are single-step (median 3 steps).
|
||||
- The *single-sentence* multiplicative aggregate — the supposed "simplest" target
|
||||
— is exactly **one idiosyncratic case** (0021); the regular in-clause shape
|
||||
already works.
|
||||
|
||||
So the needed *operations* are few and general (the solver already supports
|
||||
`{add, subtract, transfer, multiply, divide, apply_rate, compare_additive,
|
||||
compare_multiplicative}` with pack lemmas), but the *phrasings* are unbounded. A
|
||||
matcher per phrasing buys a handful of cases and the curve flattens. **This is
|
||||
overfitting by construction** — the library-of-founds trap the thesis forbids
|
||||
([[thesis-decoding-not-generating]]).
|
||||
|
||||
The deeper problem surfaced once we asked "let the solver *attempt* instead": a
|
||||
solver that **refuses whenever uncertain is safe but frozen** — it can only ever
|
||||
learn what a human hands it. Autonomous learning *requires* attempting through
|
||||
uncertainty, which is exactly where being wrong lives. **You cannot have
|
||||
autonomous learning and a global "never be wrong" at the same time.** The current
|
||||
design applies `wrong=0` even to gold-labeled *practice* data — i.e. it is built
|
||||
to give up on the very corpus meant to teach it.
|
||||
|
||||
The full derivation (problem → dead-ends → pivots → solution) is in the
|
||||
[session doc](../sessions/SESSION-2026-05-28-risk-reward-learning-architecture.md).
|
||||
This ADR records the decision.
|
||||
|
||||
## Decision
|
||||
|
||||
Relocate the engine's intelligence from front-end pattern-matching into a
|
||||
**problem solver that attempts a grounded derivation and learns by elimination**,
|
||||
and govern *when it may attempt vs. must refuse* with a **deterministic,
|
||||
per-context risk/reward gate grounded in earned calibration** — while preserving
|
||||
`wrong=0` on everything served, by construction.
|
||||
|
||||
### 1. Two regimes (the "mode")
|
||||
|
||||
- **Serving** — anything the engine commits to a consumer who will act on it
|
||||
(chat runtime, held-out/generalization measurement). Cost of error ≈ ∞.
|
||||
**`wrong=0`, absolute, unchanged.** Refuse unless certain. This ADR does not
|
||||
weaken the field/answer-integrity invariant in any way.
|
||||
- **Practice** — attempting on material where being wrong is **checkable and not
|
||||
served**. Here **wrong is the elimination signal**, not a failure. This is the
|
||||
only place autonomous learning occurs.
|
||||
|
||||
### 2. The proof-carrying seal (why wrong=0 survives a hot practice loop)
|
||||
|
||||
Practice never writes to serving. It emits **proposals that carry their own
|
||||
proof** (round-trips, forced-unique, introduces zero wrong, replay-stable);
|
||||
nothing crosses into the serving path except through the existing **proposal-only,
|
||||
reviewed** teaching gate. Practice may be as bold as its calibration allows
|
||||
*because* the seal makes its mistakes structurally unable to become a served
|
||||
answer. This reuses, and must not bypass, teaching-safety.
|
||||
|
||||
### 3. The attempt/refuse gate — a deterministic ratio (NOT reinforcement learning)
|
||||
|
||||
Per attempt, the action is licensed iff measured reliability meets the human-set
|
||||
ceiling for that action's blast-radius:
|
||||
|
||||
```
|
||||
license(action, C) := reliability_of_relevant_checker(C) / θ_required(action, C) ≥ 1
|
||||
```
|
||||
|
||||
- The two-regime structure **collapses the reward side**: in serving only
|
||||
reliability matters (learning value is irrelevant to a served answer); in sealed
|
||||
practice the threshold is "is it checkable?" (`θ_practice = 0`). We therefore
|
||||
never have to assign units to "value" or "learning" — **the only thing
|
||||
quantified is reliability vs. ceiling.**
|
||||
- `θ` are **human-set, version-controlled constants** per class and blast-radius
|
||||
(`θ_practice = 0`; `θ_propose`; `θ_serve` strict, e.g. `.99`). Raising autonomy =
|
||||
a human lowering `θ_serve(C)` for a class the ledger has earned. **The engine
|
||||
never sets or raises its own ceiling.**
|
||||
|
||||
### 4. The per-class calibration ledger (counts, not learned weights)
|
||||
|
||||
Per **class** (= capability axis G1–G5), a **replayable ledger of counts** —
|
||||
nothing learned, nothing stochastic, every figure a tally over deterministic
|
||||
attempts or an explicit human constant:
|
||||
|
||||
- `n(C), correct(C), wrong(C), refused(C)` — already produced by the eval harness.
|
||||
- `t2_verified(C), t2_agrees_gold(C)` — on the live gold anchor set.
|
||||
- `reliability(C) = conservative_floor(correct(C), n(C))` — a deterministic
|
||||
**lower bound**, pessimistic at small `n`, so luck cannot grant appetite.
|
||||
- `t2_precision(C) = conservative_floor(t2_agrees_gold(C), t2_verified(C))` — how
|
||||
trustworthy self-verification is on `C`; the number that licenses widening past
|
||||
gold.
|
||||
|
||||
This lives in the **calibration module**; `conservative_floor` is fixed arithmetic
|
||||
(see Open Questions for its shape).
|
||||
|
||||
### 5. The checkability ladder — privilege ∝ reversibility
|
||||
|
||||
Checkability is not a line but a confidence-stratified ladder. **Governing rule:
|
||||
require check-strength proportional to the reversibility/blast-radius of the action
|
||||
it licenses**, because a false positive in the checker (a wrongly-"verified"
|
||||
belief) is a *persistent contaminant* and is far worse than a missed learning
|
||||
opportunity.
|
||||
|
||||
| Tier | Checker | May change |
|
||||
|---|---|---|
|
||||
| **1 — External truth** | gold label / known answer | serving-bound knowledge (via ratification) — *anchors* |
|
||||
| **2 — Convergent self-verification** | round-trip **∧** ≥2 *structurally-distinct* derivations agree **∧** unit/dimensional consistency **∧** no contradiction with vault/packs (conjunctive) | provisional, **retractable** knowledge (still ratified before serving) |
|
||||
| **3 — Consistency-only** | merely no contradiction with the known | **practice-internal pruning only** — never crosses the seal |
|
||||
|
||||
**Tier 2 is the operating median**: the widest checkability still strong enough to
|
||||
*create* knowledge, needing no human and no label, so the practice arena scales
|
||||
toward open-world. Tier 3 keeps the arena wide (attempt anything; learn search
|
||||
shape) while being reversible and sealed.
|
||||
|
||||
### 6. Provenance + retractability
|
||||
|
||||
Every learned belief stores `(tier, n_at_admission)`. Retraction is deterministic:
|
||||
a Tier-1 (or stronger Tier-2) contradiction → retract, and decrement
|
||||
`t2_agrees_gold` → `t2_precision` falls → the `θ`-gate tightens. Provenance is what
|
||||
makes widening past gold *safe* — weak beliefs are quarantined by confidence and
|
||||
reversible. Extends CORE's existing provenance / exact-recall discipline to beliefs.
|
||||
|
||||
### 7. Gold tether — defense against correlated self-delusion
|
||||
|
||||
Tier-2 agreement only helps if derivations are **independent**; a shared wrong
|
||||
premise (the engine misunderstands "twice") makes them all agree *and* round-trip
|
||||
while all being wrong. Defenses:
|
||||
- **Independence is counted, not assumed**: Tier-2 requires **≥2 structurally
|
||||
distinct paths** (different operation multiset or different intermediate
|
||||
quantities).
|
||||
- **A live Tier-1 anchor set always runs**, measuring `t2_precision(C)` per class.
|
||||
When it drifts below a floor, appetite contracts. Gold doesn't just teach — it
|
||||
*audits whether self-verification is trustworthy*, which is the calibration loop
|
||||
closing.
|
||||
|
||||
### 8. Diagnostic refusal — the router between skill and knowledge
|
||||
|
||||
Every refusal must **name the missing piece**, so effort routes to the right axis:
|
||||
- *"quantities extracted, units consistent, no grounded derivation reaches target"*
|
||||
→ **skill** gap → solver search / elimination practice.
|
||||
- *"unknown relation / unit relationship"* → **knowledge** gap → acquire a
|
||||
world-fact.
|
||||
- *"two grounded derivations disagree"* → **genuine ambiguity** → stay refused.
|
||||
|
||||
Quantified: if `reliability(C)` still climbs with practice → skill gap (keep
|
||||
practicing); if it has stalled → knowledge gap (needs a new world-fact). Extends
|
||||
typed refusals + the OOV gradient + the math-reader-refusal audit corridor.
|
||||
|
||||
### 9. Three compounding stores
|
||||
|
||||
Each diagnosis routes to where learning accumulates: **experience → vault** (exact,
|
||||
deterministic recall), **world-knowledge → ratified packs**, **skill → the
|
||||
solver's elimination-learned pruning**. The flywheel: stronger solver → more vault
|
||||
experience + sharper pruning → fewer knowledge gaps to ask about → less
|
||||
contemplation per problem → fewer hand-authored packs → compounds.
|
||||
|
||||
### 10. Self-proving acquisition and the narrowing of human input
|
||||
|
||||
Autonomy does **not** mean "no human input" — the engine cannot conjure world-facts
|
||||
from nothing; facts enter from an ingested data/experience stream. The human role
|
||||
*shifts* from hand-authoring meaning to **curating what it ingests + ratifying
|
||||
what it has already self-proven**. The bridge is **self-proving acquisition**: new
|
||||
knowledge is proposed *with a mechanical proof attached* — the schema-proof-
|
||||
obligation discipline (CLAUDE.md) pointed at learning. The ratification gate never
|
||||
opens to ungrounded learning; it simply has less to do as the engine's proofs get
|
||||
stronger.
|
||||
|
||||
### 11. "Creative," defined for a deterministic engine
|
||||
|
||||
Not stochastic invention. **A willingness to leap a gap in known structure** — a
|
||||
recombination no stored pattern directly licenses — always a step from given
|
||||
ground, never from the void. The checkability tier is the leap dial: a Tier-3 leap
|
||||
stays a hypothesis; a Tier-2 leap becomes provisional knowledge; a Tier-1-confirmed
|
||||
leap becomes an anchor.
|
||||
|
||||
## Non-negotiable invariants (must be *proven*, not asserted)
|
||||
|
||||
Per CLAUDE.md §Schema-Defined Proof Obligations, each of these requires a test that
|
||||
**fails** under the violation it names:
|
||||
|
||||
1. **wrong=0 on serving is untouched.** A test must fail if any practice-regime
|
||||
artifact reaches a served answer without crossing the ratification gate.
|
||||
2. **The search cannot bank a spurious answer.** A test must fail if a derivation
|
||||
that is *not* grounded+unique+round-tripping is admitted as knowledge (the
|
||||
`20/5` coincidence class).
|
||||
3. **Determinism / replay.** All ledger counts, the `conservative_floor`, the gate,
|
||||
and the search are deterministic and replayable; a test must fail on any
|
||||
run-to-run divergence. **No learned weights, no stochastic sampling, no
|
||||
approximate recall** — the vault stays exact.
|
||||
4. **No self-authorization.** A test must fail if the engine mutates any `θ`
|
||||
ceiling. Ceilings are human-set config only.
|
||||
5. **Retractability.** A test must fail if a Tier-1 contradiction does not retract
|
||||
the contradicted Tier-2 belief and tighten the gate.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **What it collapses:** the per-shape matcher backlog. Multiplicative/comparative/
|
||||
fraction cases become the **first practice arena** where attempt-and-eliminate is
|
||||
proven, not a set of shapes to hand-match. ADR-0174's 5b sub-phases are
|
||||
superseded.
|
||||
- **The train_sample double-duty is resolved.** It currently serves as *both*
|
||||
practice arena and serving-regression canary. Decouple: practice may attempt all
|
||||
47 (scored correct/wrong/refused, wrongs feed elimination) while `wrong=0` stays
|
||||
absolute on the serving contract + held-out generalization, and the hazard
|
||||
canaries (0050) keep guarding serving.
|
||||
- **Risk concentrates in the search + checker.** This is where the project's
|
||||
correctness mandate is most stressed; invariants #1–#2 are the load-bearing work.
|
||||
- **Mostly composition, not new machinery:** classes = capability axes; counts =
|
||||
eval harness; reliability + lower bound = calibration module; replay guarantees
|
||||
reproducibility; `θ` = a small config table; seal = teaching-safety; elimination
|
||||
= ADR-0174's `eliminate_violating`/`reevaluate`/`contemplate` repointed to
|
||||
solving.
|
||||
|
||||
## Phasing (wrong=0-first; each phase ships its proof obligations)
|
||||
|
||||
1. **Ledger + gate substrate.** Per-class calibration ledger, `conservative_floor`,
|
||||
the ratio gate, `θ` config table. Invariants #3–#4 proven. Zero behavior change
|
||||
to serving.
|
||||
2. **Sealed practice regime on GSM8K train.** Run attempt-and-eliminate over the 47
|
||||
(Tier-1 gold checkable); score correct/wrong/refused as *practice* metrics;
|
||||
wrongs produce elimination records. Invariant #1 proven (nothing leaks to
|
||||
serving). Diagnostic refusal (§8) emitted.
|
||||
3. **Grounded derivation search.** Bounded, deterministic operation-chain search
|
||||
over extracted quantities, gated by grounding + unit + unique + round-trip.
|
||||
Invariant #2 proven (the spurious-answer test). Measure the flip-curve on the
|
||||
multiplicative chunk; require it to hold under ADR-0114a perturbation.
|
||||
4. **Tier-2 self-verification + provenance + tether.** Convergent self-verification,
|
||||
per-belief provenance, the live gold tether + `t2_precision`. Invariant #5
|
||||
proven. Widen the arena past gold-labeled material.
|
||||
5. **Self-proving proposals into the ratification corridor.** Practice emits
|
||||
proof-carrying proposals; the (narrowing) HITL gate admits to serving. Measure
|
||||
the serving-`correct` lift this produces with `wrong=0` held.
|
||||
|
||||
## Acceptance criteria (Proposed → Accepted)
|
||||
|
||||
1. Phase 1 substrate lands; invariants #3–#4 proven; serving byte-identical.
|
||||
2. A prototype grounded search demonstrably **refuses** the `20/5`-class spurious
|
||||
derivation (invariant #2) on a curated case.
|
||||
3. The practice regime is provably sealed from serving (invariant #1).
|
||||
4. Capability-axis lanes G1–G5, S1 remain 100% `wrong=0`; pinned lane SHAs pass.
|
||||
5. Cross-references to ADR-0174 (substrate), teaching-safety (seal), and the
|
||||
thesis reviewed and confirmed consistent.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Shape of `conservative_floor` and `N_min`** — how pessimistic at small `n`.
|
||||
This single choice sets how cautiously the system earns autonomy. Candidate:
|
||||
a deterministic Wilson/Wald-style lower bound, or a simpler `require n ≥ N_min
|
||||
AND wrong ≤ W_max` rule. Resolve before Phase 1 PR.
|
||||
2. **First practice-arena home.** GSM8K train (gold-labeled, checkable, already
|
||||
wired) is the obvious Phase 2/3 home; confirm no serving-path coupling remains
|
||||
after the train_sample double-duty decoupling.
|
||||
3. **Search bound + determinism budget.** The operation-chain search must be
|
||||
bounded and replay-stable; fix the enumeration order and depth cap before
|
||||
Phase 3.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Derivation:** [SESSION-2026-05-28](../sessions/SESSION-2026-05-28-risk-reward-learning-architecture.md).
|
||||
- **Substrate repointed:** [ADR-0174](./ADR-0174-held-hypothesis-comprehension.md)
|
||||
(`eliminate_violating` / `reevaluate` / `contemplate`), calibration module,
|
||||
capability axes G1–G5, round-trip filter + disagreement rule, teaching-safety.
|
||||
- **Anti-overfitting obligations:** ADR-0114a (perturbation / OOD / depth / adversarial axes apply to every flipped case).
|
||||
- **Thesis:** [[thesis-decoding-not-generating]] — find, comprehend, rationalize; not a library of founds.
|
||||
|
|
@ -0,0 +1,306 @@
|
|||
# Session 2026-05-28 — From "Another Matcher" to a Self-Calibrating Problem Solver
|
||||
|
||||
**Status:** Discussion captured ✓ — ADR-0175 to follow
|
||||
**Headline:** A scoping session for GSM8K Phase 5b turned into the discovery that the
|
||||
whole "build another recognizer per problem shape" strategy is structurally
|
||||
self-limiting (overfitting by construction), and converged on a different
|
||||
architecture: **move the intelligence into the solver (attempt + eliminate),
|
||||
separate a sealed practice regime from the wrong=0 serving regime, and gate
|
||||
attempts with a deterministic risk/reward ratio grounded in earned per-class
|
||||
calibration.** This document is the *journey* — the problem, the dead-ends, the
|
||||
pivots, and how we arrived at the solution. The formal decision is **ADR-0175**.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
We started trying to scope "Phase 5b.1 — single-sentence multiplicative
|
||||
aggregate." Measurement showed the target was **one idiosyncratic case (0021)**,
|
||||
and that the broader per-shape-matcher approach can never compound: GSM8K
|
||||
phrasings are unbounded, so each matcher buys a handful of cases and the curve
|
||||
flattens. That is overfitting wearing an engineering hat.
|
||||
|
||||
The turn (driven by Shay): stop asking the *front-end* to recognize every shape;
|
||||
give the *solver* room to **attempt** a derivation over extracted quantities and
|
||||
**learn by elimination** from what's wrong. The solver already supports the
|
||||
operations; the front-end was the bottleneck and the source of brittleness.
|
||||
|
||||
That exposed the real contradiction: a system that **refuses whenever uncertain
|
||||
is safe but frozen** — it can only ever learn what a human hands it. Autonomous
|
||||
learning *requires* attempting through uncertainty, which is exactly where being
|
||||
wrong lives. You cannot have autonomous learning and a global "never be wrong" at
|
||||
once.
|
||||
|
||||
Resolution: **two regimes** — *serving* (wrong=0, refuse-unless-certain,
|
||||
unchanged) and *sealed practice* (attempt-and-eliminate, where wrong is the
|
||||
learning signal, not a failure) — separated by a **proof-carrying seal**. Which
|
||||
regime applies is decided per-attempt by a **deterministic risk/reward ratio**:
|
||||
`measured_reliability(class) / required_reliability(action) ≥ 1`. Reliability is
|
||||
an **earned, per-class, conservatively-bounded count** from a live gold tether —
|
||||
not a learned weight, not a probability from nowhere. Checkability is a **tiered
|
||||
ladder** (gold > convergent self-verification > consistency-only) where the
|
||||
*privilege a passed check earns* scales with the check's strength, and everything
|
||||
weak is reversible. Humans raise the per-class ceiling on evidence; the engine
|
||||
never self-authorizes.
|
||||
|
||||
---
|
||||
|
||||
## Where we started: scoping Phase 5b.1
|
||||
|
||||
Phase 5a had just shipped (PR #430): the inert GSM8K reader retired, single
|
||||
canonical parse path, `3/47/0` byte-identical, net −1,038 LOC. We turned to
|
||||
Phase 5b — the *semantic* lift — and began with what the scope called the
|
||||
cleanest first slice: "single-sentence multiplicative aggregate" (e.g. case 0021,
|
||||
"He bench presses 15 pounds for 10 reps and does 3 sets" → 15×10×3 = 450).
|
||||
|
||||
## Dead-end #1 — the per-shape matcher reflex
|
||||
|
||||
Every instinct pulled toward *building another recognizer*: detect this shape,
|
||||
emit that primitive. Two measurements stopped that cold.
|
||||
|
||||
**The brute-force measurement that lied.** A bounded expression search over each
|
||||
refused case's numbers found "matches" — but several were *coincidences*
|
||||
(`20/5 = 4` for a case whose answer was 4 for unrelated reasons; `5−3 = 2`). The
|
||||
measurement itself overfit. Lesson banked: **number-matching without grounding
|
||||
manufactures spurious answers** — the same hazard a naive solver search would
|
||||
have.
|
||||
|
||||
**The ground-truth measurement.** Parsing GSM8K's own `<<a*b=c>>` calculator
|
||||
annotations (real operations, not guesses) over the 47 refused cases:
|
||||
|
||||
| Profile | Count | Share |
|
||||
|---|---|---|
|
||||
| Uses multiplication (`*`) | **37/47** | 79% |
|
||||
| Uses mul **or** div | 43/47 | 91% |
|
||||
| Pure add/subtract | 4/47 | 9% |
|
||||
| **Single-step solvable** | **0/47** | **0%** |
|
||||
|
||||
Step-count histogram: `2:13 · 3:12 · 4:8 · 5:8 · 6:3 · 7:3`.
|
||||
|
||||
And a scan for *single-sentence* multiplicative aggregates (all operands in one
|
||||
clause, gold = their product) returned exactly **one** case — 0021 — whose
|
||||
surface ("for 10 reps and does 3 sets", pronoun subject, three different units)
|
||||
is the *least* general multiplicative shape in the set, not the simplest.
|
||||
|
||||
**The conclusion that reframed everything:** multiplication is needed by 79% of
|
||||
the corpus (maximally general — a foundational capability, not a niche), but
|
||||
**no case is solvable in one step**, and the regular in-clause shape *already
|
||||
works* via existing Wave-A scaffolding. So a "single-sentence multiplicative
|
||||
matcher" would flip ~1 case via near-bespoke pattern matching — the textbook
|
||||
definition of overfitting the project has repeatedly ruled out ("lifting a few at
|
||||
a time → simply overfitting"). The per-shape path is unbounded because phrasings
|
||||
are unbounded. It cannot compound.
|
||||
|
||||
## Pivot #1 (Shay) — breadth-of-impact is the overfitting test
|
||||
|
||||
> "If solutions result in generally large chunks of failed results turning green,
|
||||
> then it's more likely that we are getting GENERALLY smarter... if tweaking the
|
||||
> injector fixes a big chunk or percentage then it COULD be a solution."
|
||||
|
||||
This corrected a sloppy framing ("tweaking the injector = overfitting"). The
|
||||
overfitting test isn't *where* the fix lives — it's **breadth**: a change that
|
||||
flips a large *general* chunk (and holds under perturbation) is a real capability;
|
||||
a per-phrasing patch that flips one case is overfitting. The measurement said the
|
||||
biggest, most general chunk is *multiplication itself* — so the question became
|
||||
"how do we get general multiplicative capability," not "how do we match 0021."
|
||||
|
||||
## Pivot #2 (Shay) — give the solver room to attempt; move the intelligence
|
||||
|
||||
> "Maybe the solution is to give it more room to attempt to solve the problem...
|
||||
> the better we make the problem solver, the better the model will be at
|
||||
> anything/everything... the less we will have to put into packs and the less it
|
||||
> will have to contemplate over time, and the faster it will learn what works and
|
||||
> it compounds. That's what intelligence does."
|
||||
|
||||
This is the architectural turn. Today the front-end (recognizer/injector) must
|
||||
hand the solver a *fully-typed, correct* graph or the solver does nothing — so
|
||||
all intelligence is forced into surface pattern-matching, and surface is infinite.
|
||||
The capable part sits idle:
|
||||
|
||||
- The **solver already supports** `{add, subtract, transfer, multiply, divide,
|
||||
apply_rate, compare_additive, compare_multiplicative}` with pack lemmas for
|
||||
each. It just waits passively.
|
||||
- The gap is entirely the **reader → injector → Operation** front-end (the
|
||||
recognizer matches a shape but extracts *zero* anchors on real sentences).
|
||||
|
||||
The redirection: shrink the front-end to **extract quantities + the relations the
|
||||
text licenses**, and grow the solver to **attempt** — search the bounded space of
|
||||
operation-chains for a derivation that reaches a *verifiable* answer. This
|
||||
generalizes across all phrasings because it never looks at phrasing — it looks at
|
||||
quantities and the goal. The compounding the project wants comes from the solver,
|
||||
not from a growing library of founds (the [[thesis-decoding-not-generating]]).
|
||||
|
||||
**The honest crux we named:** search makes wrong=0 *harder*, not easier — an
|
||||
active searcher has more chances to land on a spurious-but-valid answer (cf. the
|
||||
brute-force `20/5`). The whole approach rests on the derivation being **grounded**
|
||||
(each step licensed by text + unit-consistent), **unique-or-refuse** (the existing
|
||||
disagreement rule), and **round-trip-checked** (realize back to language, reject
|
||||
if it doesn't match the problem). Those gates already exist in CORE.
|
||||
|
||||
## Insight #3 — two failure modes, and the diagnosis that routes them
|
||||
|
||||
> "Either it has enough information... to work the problem with masterfully built
|
||||
> problem-solving skill, or it needs a wider understanding of what it's trying to
|
||||
> decode... it needs more pieces to the puzzle."
|
||||
|
||||
When the engine can't decode a problem, exactly one of three things is true:
|
||||
|
||||
1. **Skill gap** — has the pieces, can't compose the derivation → improve the
|
||||
reasoner.
|
||||
2. **Knowledge gap** — missing a world-fact (what "twice" means, that "each basket
|
||||
holds 50" is a per-container rate) → acquire knowledge.
|
||||
3. **Genuine ambiguity** — under-determined → refuse, correctly, forever.
|
||||
|
||||
The mechanism that makes "finding the pieces" efficient is **diagnostic refusal**:
|
||||
a refusal that *names the missing piece* routes itself to the right axis
|
||||
("quantities extracted but no grounded derivation" = skill; "unknown relation:
|
||||
twice" = knowledge; "two grounded derivations disagree" = ambiguity). CORE already
|
||||
has typed refusals, the OOV honesty gradient, and the math-reader-refusal audit
|
||||
corridor. Knowing what you don't know is most of knowing how to find it.
|
||||
|
||||
**Three compounding stores**, each fed by the matching diagnosis:
|
||||
- **experience → vault** (exact, deterministic recall),
|
||||
- **world-knowledge → ratified packs**,
|
||||
- **skill → solver's elimination-learned pruning**.
|
||||
|
||||
**The floor (Shay):** "it cannot conjure world-facts from nothing. not even a
|
||||
human can. only God himself." Autonomy doesn't mean "no input" — facts about the
|
||||
world must enter from a data/experience stream. The human role *shifts* from
|
||||
hand-authoring meaning to **curating what it ingests + ratifying what it has
|
||||
already self-proven**. The bridge is **self-proving acquisition**: a new fact is
|
||||
admitted with a *mechanical proof attached* (round-trips, forced-unique, zero
|
||||
wrong, replay-stable) — the schema-proof-obligation discipline, pointed at
|
||||
learning. Knowledge acquired with a proof needs less judgment to admit.
|
||||
|
||||
## Pivot #4 (Shay) — the contradiction, and the "mode"
|
||||
|
||||
> "If we intentionally build around it always 'giving up' whenever it doesn't
|
||||
> know, it will never learn without humans. Maybe there should be a 'mode' for
|
||||
> when it's allowed to work on a problem... a type of risk-to-reward RATIO system
|
||||
> (not... reinforcement learning; not that kind of risk/reward)."
|
||||
|
||||
This is the central contradiction stated exactly: **refuse-when-uncertain is safe
|
||||
and frozen.** The resolution can't be a global setting — it must be contextual.
|
||||
|
||||
**Two regimes (the concrete "mode"):**
|
||||
- **Serving** — committed to someone who will act on it; cost of error ≈ ∞;
|
||||
refuse unless certain; **wrong=0, absolute, unchanged.**
|
||||
- **Practice** — attempting where being wrong is *checkable and not served*; here
|
||||
**wrong is the elimination signal**, the most informative event possible. The
|
||||
only place autonomous learning can happen.
|
||||
|
||||
What keeps wrong=0 intact while practice runs hot: the **proof-carrying seal** —
|
||||
practice emits *proposals carrying their own proof*; nothing reaches serving
|
||||
except via the (narrowing) ratification gate. CORE already has this seal
|
||||
(proposal-only, reviewed teaching).
|
||||
|
||||
**"Creative," defined for a deterministic engine:** not stochastic invention — a
|
||||
**willingness to leap a gap in known structure** (a recombination no stored
|
||||
pattern directly licenses). The risk is the leap is unwarranted; the reward is
|
||||
that a *verified* leap becomes new structure. It is always a step from solid
|
||||
ground, never from the void — the engineering form of "only God conjures from
|
||||
nothing." The risk/reward gate decides *how far across an unlit gap calibration
|
||||
permits a step.*
|
||||
|
||||
## The checkability question — and the tiered answer
|
||||
|
||||
If practice can attempt without a human or a gold label, *what counts as checkable
|
||||
enough to learn from?* The asymmetry that governs the answer: **a false positive
|
||||
in the checker is far worse than a missed learning opportunity** — a wrongly
|
||||
"verified" belief is a persistent contaminant; a refusal is just a missed chance.
|
||||
|
||||
So the boundary isn't a line — it's a **confidence-stratified ladder**, with the
|
||||
rule: *require check-strength proportional to the reversibility/blast-radius of
|
||||
the action it licenses.*
|
||||
|
||||
| Tier | Checker | May change |
|
||||
|---|---|---|
|
||||
| **1 — External truth** | gold label / known answer | serving-bound knowledge (via ratification) — anchors |
|
||||
| **2 — Convergent self-verification** | round-trip **∧** ≥2 *structurally-distinct* derivations agree **∧** unit-consistent **∧** no contradiction with vault/packs | provisional, **retractable** knowledge (still ratified) |
|
||||
| **3 — Consistency-only** | merely no contradiction | **practice-internal pruning only** — never crosses the seal |
|
||||
|
||||
**Tier 2 is the median** — the widest checkability still strong enough to *create*
|
||||
knowledge, needing no label, so the arena scales toward open-world. Two
|
||||
safeguards make widening safe:
|
||||
- **Provenance per belief** `(tier, n_at_admission)` → Tier-2 beliefs are
|
||||
*retractable* when a stronger check later contradicts them.
|
||||
- **A live gold tether** measuring `t2_precision(class)` — catches *correlated
|
||||
self-delusion* (a shared wrong premise makes all derivations agree); when it
|
||||
drifts, appetite contracts. Independence is *counted* (≥2 structurally-distinct
|
||||
paths), not assumed.
|
||||
|
||||
## Quantifying it — the part that makes it engineering
|
||||
|
||||
The naive ratio needs units for "value" and "learning" — a quagmire. The regime
|
||||
structure **collapses the reward side**: in serving only reliability matters
|
||||
(learning is irrelevant to a served answer); in sealed practice the threshold is
|
||||
just "is it checkable?". So **the entire numeric system reduces to: measure
|
||||
reliability, compare to a ceiling.** Everything else is counting.
|
||||
|
||||
Per **class** (= capability axis G1–G5), a replayable **ledger of counts** —
|
||||
nothing learned, nothing stochastic:
|
||||
|
||||
- `n(C), correct(C), wrong(C), refused(C)` — the eval already counts these.
|
||||
- `t2_verified(C), t2_agrees_gold(C)` — on the live anchor set.
|
||||
- `reliability(C) = conservative_floor(correct, n)` — a deterministic **lower
|
||||
bound** (pessimistic at small n, so luck can't grant appetite).
|
||||
- `t2_precision(C) = conservative_floor(t2_agrees_gold, t2_verified)` — how
|
||||
trustworthy self-verification is on C; the number that licenses widening past
|
||||
gold.
|
||||
|
||||
**The gate is a literal ratio:**
|
||||
```
|
||||
license(action, C) := reliability_of_relevant_checker(C) / θ_required(action, C) ≥ 1
|
||||
```
|
||||
`θ` are **human-set, version-controlled constants** per class and blast-radius
|
||||
(`θ_practice = 0`; `θ_propose`; `θ_serve` strict, e.g. .99). "Humans ready to let
|
||||
it" = a human lowering `θ_serve(C)` for a class the ledger has earned.
|
||||
|
||||
**Compounding becomes a measurable curve:** `refused(C)` ↓ and `reliability(C)` ↑
|
||||
with practice volume, `wrong` on serving pinned at 0 throughout. And the
|
||||
**skill-vs-knowledge diagnosis is quantified**: if reliability still climbs with
|
||||
practice → skill gap (keep practicing); if it has stalled → knowledge gap (needs a
|
||||
new world-fact). Learning-rate = reliability gain per unit practice.
|
||||
|
||||
Almost none of this is new machinery — it's composition: classes = capability
|
||||
axes; counts = the eval harness; reliability + lower bound = the calibration
|
||||
module; replay guarantees reproducibility; `θ` = a small config table; seal +
|
||||
ratification = existing teaching-safety.
|
||||
|
||||
## The converged model (one line)
|
||||
|
||||
> attempt-or-refuse is a **per-context risk/reward decision**, not a global rule →
|
||||
> **two regimes** (serving wrong=0 / sealed practice attempt-and-eliminate)
|
||||
> separated by a **proof-carrying seal** → the ratio is
|
||||
> `measured_reliability / required_reliability ≥ 1` → reliability is an **earned,
|
||||
> conservatively-bounded, per-class count** from a live gold tether → checkability
|
||||
> is a **tiered ladder** with privilege ∝ reversibility and provenance making weak
|
||||
> beliefs retractable → humans **raise the ceiling on evidence**; the engine never
|
||||
> self-authorizes → "creative" = a calibrated leap across a gap, always from given
|
||||
> ground.
|
||||
|
||||
## Open question carried into the ADR
|
||||
|
||||
The one genuinely new number to pin: **the shape of the `conservative_floor`
|
||||
function and `N_min`** — how pessimistic to be at small n. That single choice sets
|
||||
how cautiously the whole system earns its autonomy.
|
||||
|
||||
## What this supersedes
|
||||
|
||||
The matcher-oriented Phase 5b sub-phases (5b.1 single-sentence / 5b.2
|
||||
cross-sentence / 5b.3 deep) in ADR-0174 collapse into *instances* of this
|
||||
architecture rather than standalone work. The multiplicative cases become the
|
||||
first **practice arena** where attempt-and-eliminate is proven, not a set of
|
||||
shapes to hand-match.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- **Decision:** ADR-0175 (to be written).
|
||||
- **Builds on:** ADR-0174 (held-hypothesis / `eliminate_violating` / `reevaluate`
|
||||
/ `contemplate` — the elimination substrate, here repointed from *reading* to
|
||||
*solving*); the calibration module; capability axes G1–G5; the round-trip
|
||||
filter and multi-branch disagreement rule (the wrong=0 gates that make search
|
||||
safe); teaching-safety (proposal-only, reviewed) = the seal.
|
||||
- **Thesis anchor:** [[thesis-decoding-not-generating]] — find/comprehend/
|
||||
rationalize, not a library of founds.
|
||||
- **Phase 5a (shipped):** PR #430.
|
||||
Loading…
Reference in a new issue