docs(session): practice-time attempts vs served answers — the servability blade
Records the wrong=0 refinement reached in discussion: wrong=0 is not a binary answer/refuse gate but 'no false presentation of epistemic status'. Practice/ contemplation may explore typed, isolated candidate guesses; served output needs a servability blade with more modes than verified|refuse. Proposes ProblemAttemptSession (PRA) and ServabilityDecision (SRV), sequenced PRA -> REL -> SRV, and traces the idea to the R1/R2/R3/CMB reader trajectory. Session doc, not an ADR yet.
This commit is contained in:
Shay
parent
52731c02aa
commit
f6292f1ecf
1 changed files with 344 additions and 0 deletions
|
|
@ -0,0 +1,344 @@
|
|||
# Session 2026-06-08 — Practice-time attempts vs served answers, and the servability blade
|
||||
|
||||
**Status:** discussion / design note — **session document, not an ADR yet.** No
|
||||
code shipped. This records a *refinement of doctrine* reached in conversation,
|
||||
deliberately preserved before any new capability work so the idea cannot be
|
||||
silently collapsed back into a binary gate. **Headline:** `wrong=0` has been
|
||||
operating as a hard binary — **verified answer OR refusal** — and that binary,
|
||||
correct for sealed evaluation, is wrong for collaborative reasoning. The real
|
||||
law is **"no false presentation of epistemic status."** From that reframing fall
|
||||
two proposed organs: a **ProblemAttemptSession** (where typed, isolated candidate
|
||||
guesses are *allowed* at practice time) and a **ServabilityBlade** emitting a
|
||||
**ServabilityDecision** (which chooses, at serving time, among *more than two*
|
||||
truthfully-labeled output modes).
|
||||
|
||||
`No serving path, algebra, versor, recall, or gate touched. This is the research`
|
||||
`trail — what we decided and how the conversation arrived there.`
|
||||
|
||||
> **Why this doc exists, and why now.** CORE is open-source research in a largely
|
||||
> uncharted corner of AI — a deterministic cognitive engine, not a transformer
|
||||
> wrapper. The *reasoning behind* a design is as load-bearing as the design.
|
||||
> Capturing it as a **session doc rather than an ADR** is deliberate: the idea is
|
||||
> a refinement, not yet a ratified architectural commitment. Writing it down now
|
||||
> protects the research trail and **keeps future agents (and the operator) from
|
||||
> collapsing the idea back into a binary gate** the next time the wrong=0
|
||||
> invariant is invoked. An ADR can follow once PRA/REL exist and the blade has a
|
||||
> concrete consumer.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — the refinement, in four lines
|
||||
|
||||
```text
|
||||
wrong=0 ≠ binary answer/refuse
|
||||
wrong=0 = no false presentation of epistemic status
|
||||
practice / contemplation MAY explore typed, isolated candidates (even guesses)
|
||||
served output NEEDS a servability blade (more modes than verified | refuse)
|
||||
```
|
||||
|
||||
1. **The binary we inherited.** The serving contract is, today, *emit a verified
|
||||
answer, or refuse.* That is the classic wrong=0 path
|
||||
([[project-self-check-soundness-not-correctness]]) and it is **correct for
|
||||
sealed eval.**
|
||||
2. **The over-reach.** Read literally, that binary also forbids partial
|
||||
reasoning, provisional insight, surfacing multiple admissible readings, and
|
||||
open-ended problem-solving — none of which are false presentations of truth.
|
||||
It makes the engine mute unless omniscient.
|
||||
3. **The reframing.** Truth discipline is **disclosure discipline**: *never hide
|
||||
uncertainty; never promote a candidate to a fact; never collapse a hypothesis
|
||||
into a verified answer; never omit the reason a conclusion is provisional.* A
|
||||
*labeled* provisional answer is honest, not a lie.
|
||||
4. **Two layers fall out.** A **practice/attempt layer** where typed, isolated
|
||||
candidates may be explored, and a **served-output blade** that decides what of
|
||||
that internal state is truthfully emittable in the current context.
|
||||
|
||||
---
|
||||
|
||||
## 1. The distinction: practice-time attempts vs served answers
|
||||
|
||||
This is the spine of the whole refinement. There are **two different lanes** with
|
||||
**two different epistemic licenses**, and the binary gate conflated them by
|
||||
applying the serving license everywhere.
|
||||
|
||||
| | **Practice / contemplation** | **Serving** |
|
||||
|---|---|---|
|
||||
| purpose | explore, attempt, eliminate, learn | emit to a user / downstream |
|
||||
| may contain | guesses, attempts, pattern-matches, candidate eliminations, partial derivations | only what is *truthfully labelable* in context |
|
||||
| failure mode it guards | *getting stuck / never learning* | *misrepresenting epistemic status* |
|
||||
| existing substrate | sealed practice lane (ADR-0175), `comprehension_attempt`, contemplation, failure families, proposal-only learning | wrong=0 verify gate, refusal taxonomy, safety/refusal policy |
|
||||
| license | **guessing allowed** (typed + isolated) | **no false presentation of status** |
|
||||
|
||||
The mistake was letting the *serving* license (prove-or-refuse) leak into the
|
||||
*practice* lane, which would forbid the engine from ever attempting something it
|
||||
cannot yet prove — i.e. forbid it from learning. And, symmetrically, letting the
|
||||
*practice* binary ("we either solved it or we didn't") leak into *serving*, which
|
||||
collapses everything that isn't a finished proof into silence.
|
||||
|
||||
**Practice may explore. Serving must disclose. Neither may lie.**
|
||||
|
||||
## 2. Why candidate guesses are allowed — *when typed and isolated*
|
||||
|
||||
A guess is dangerous only when it can be **mistaken for a fact** or **leak into
|
||||
ratified state**. Remove those two hazards and a guess becomes a legitimate
|
||||
research instrument. Two conditions make it safe:
|
||||
|
||||
- **Typed.** A candidate is not a free-floating string asserted as an answer. It
|
||||
is a *typed object* carrying its own epistemic status (`inferred`,
|
||||
`underdetermined`, `ambiguous`, …), its basis, and its eliminations. Its type
|
||||
*is* its honesty: nothing about a typed candidate claims to be verified.
|
||||
- **Isolated.** The candidate lives in the practice/attempt lane and the
|
||||
proposal-only learning path ([[project-next-batch-contemplation-prototype]]:
|
||||
`teaching/proposals/` emits, never self-installs). It cannot reach serving, and
|
||||
it cannot ratify itself into corpus or packs. The fail-closed boundary already
|
||||
exists; the guess simply lives behind it.
|
||||
|
||||
Under those two conditions the existing **sealed practice lane** (ADR-0175:
|
||||
attempt-and-eliminate, [[adr-0175-calibrated-learning-architecture]]) is already
|
||||
*doing this safely* — accumulating typed attempts and eliminations under a Wilson
|
||||
floor without ever touching the wrong=0 serving metric. The refinement is to name
|
||||
the principle explicitly so it is not re-litigated: **typed + isolated ⇒ a
|
||||
candidate guess is admissible at practice time.**
|
||||
|
||||
## 3. Why served output needs more than verified/refuse
|
||||
|
||||
The same internal evidence should produce *different* emissions depending on
|
||||
context — and "refuse" is frequently the *worst* truthful option available.
|
||||
Examples where a third mode is strictly more honest *and* more useful:
|
||||
|
||||
- **Partial progress** — *"I can establish these constraints but can't yet pin a
|
||||
unique answer; one variable is underdetermined."* (More useful than refusal,
|
||||
asserts nothing false.)
|
||||
- **Multiple candidates** — *"Two admissible readings: under A the answer is 12,
|
||||
under B it's 15; the text doesn't disambiguate."* (The honest output for
|
||||
genuinely ambiguous input — refusal *hides* the real finding.)
|
||||
- **Provisional working answer** — *"Best current setup is this; I haven't
|
||||
verified every assumption, so treat it as provisional."* (Honest disclosure for
|
||||
collaborative, low/medium-risk work.)
|
||||
- **Clarifying question** — *"Does 'total animals' include the dogs?"* (When the
|
||||
ambiguity is resolvable by the user.)
|
||||
|
||||
The unifying rule — **the engine may emit anything it can truthfully label:**
|
||||
|
||||
| May emit (truthfully labeled) | May **not** emit |
|
||||
|---|---|
|
||||
| verified fact **as** verified | hypothesis **as** fact |
|
||||
| inference **as** inference | guess **as** proof |
|
||||
| hypothesis **as** hypothesis | partial setup **as** complete |
|
||||
| partial result **as** partial | pattern-match **as** verification |
|
||||
| ambiguity **as** ambiguity | unreviewed proposal **as** ratified |
|
||||
| failure **as** failure | |
|
||||
|
||||
The right column is the real wrong=0. The binary was needlessly suppressing the
|
||||
left column.
|
||||
|
||||
## 4. Proposed: `ProblemAttemptSession` (PRA)
|
||||
|
||||
The practice-lane evidence object. It records the typed candidates, what's been
|
||||
eliminated, and what ambiguity remains — the substrate the blade later reads. (The
|
||||
comprehension-attempt substrate from [[project-next-batch-contemplation-prototype]]
|
||||
— `core/comprehension_attempt/`, setup router, failure-family registry — is the
|
||||
natural home; PRA firms up its *evidence shape*.)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class CandidateAttempt:
|
||||
value: str # the candidate answer/setup
|
||||
epistemic_state: Literal[
|
||||
"verified", "inferred", "underdetermined",
|
||||
"ambiguous", "contradicted", "unsupported",
|
||||
]
|
||||
basis: tuple[str, ...] # why this candidate was produced
|
||||
eliminated_by: tuple[str, ...] # checks that ruled it out (empty = live)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProblemAttemptSession:
|
||||
problem_id: str
|
||||
reader: Literal["R1", "R2", "R3", "CMB"] # which setup family produced it
|
||||
candidates: tuple[CandidateAttempt, ...] # typed; guesses allowed here
|
||||
eliminated: tuple[CandidateAttempt, ...]
|
||||
residual_ambiguity: tuple[str, ...] # unresolved readings/variables
|
||||
risk_context: Literal[
|
||||
"sealed_eval", "practice", "collaborative_work",
|
||||
"research", "tutoring", "safety_critical", "production_serving",
|
||||
]
|
||||
# frozen, inspectable, replayable — an auditable artifact like every gate decision
|
||||
```
|
||||
|
||||
Note what the type *guarantees*: a `CandidateAttempt` can never be mistaken for a
|
||||
served fact, because its `epistemic_state` travels with it and `eliminated_by`
|
||||
records its disqualification. This is the "typed" half of §2 made concrete.
|
||||
|
||||
## 5. Proposed: `ServabilityDecision` (SRV)
|
||||
|
||||
The serving-lane policy object. It consumes a `ProblemAttemptSession` plus risk
|
||||
and user intent, and selects a *truthfully-labeled* output mode.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ServabilityDecision:
|
||||
mode: Literal[
|
||||
"verified_answer", # proof complete — the classic wrong=0 path
|
||||
"provisional_working_answer", # strong-but-unratified, disclosed as such
|
||||
"partial_progress", # useful, can't finish
|
||||
"multiple_candidates", # several admissible readings survive
|
||||
"clarify", # ambiguity resolvable by the user
|
||||
"refuse_unsupported", # can't proceed without guessing
|
||||
"refuse_unsafe", # safety/high-risk threshold unmet
|
||||
"contradiction_report", # evidence conflicts
|
||||
]
|
||||
epistemic_state: Literal[
|
||||
"verified", "inferred", "underdetermined",
|
||||
"ambiguous", "contradicted", "unsupported",
|
||||
]
|
||||
risk_level: Literal["low", "medium", "high", "sealed_eval"]
|
||||
confidence_basis: tuple[str, ...] # why we believe what we believe
|
||||
required_disclosures: tuple[str, ...] # caveats that MUST accompany the surface
|
||||
prohibited_phrases: tuple[str, ...] # claims this mode may not make
|
||||
servable_claims: tuple[str, ...] # what may actually be emitted
|
||||
```
|
||||
|
||||
**Risk sets strictness, not the existence of the blade.** The *same* epistemic
|
||||
state yields different modes by context:
|
||||
|
||||
```text
|
||||
sealed_eval → VERIFIED_ANSWER | REFUSE only (no provisional)
|
||||
practice → (internal; not a served emission)
|
||||
collaborative_work → PROVISIONAL_WORKING_ANSWER allowed
|
||||
research → hypotheses / candidate paths / uncertainty valued
|
||||
tutoring → MULTIPLE_CANDIDATES with explanation
|
||||
safety_critical → strict proof/source threshold; else caveat or refuse
|
||||
production_serving → verified, else disclosed-provisional, else refuse
|
||||
```
|
||||
|
||||
The answer/refuse binary is now visible as **the strict end of a spectrum**
|
||||
(`sealed_eval` / `safety_critical`), retained verbatim there — not the whole
|
||||
contract. This is what resolves the practice-vs-serving tension cleanly: serving
|
||||
stops being binary and becomes **controlled disclosure**.
|
||||
|
||||
A provisional answer is truthful **iff** it carries its label: *"best current
|
||||
inference … not fully verified … these assumptions are load-bearing … here is
|
||||
what would change the answer."*
|
||||
|
||||
## 6. How this came from the R1 / R2 / R3 / CMB trajectory
|
||||
|
||||
The refinement was not abstract — it was forced by where the reader/compiler work
|
||||
actually went. Each step produced **typed candidates and typed failures in an
|
||||
isolated lane**, and that accumulation is what made the binary's coarseness
|
||||
visible:
|
||||
|
||||
- **R1 — reconstruction reader.** Derivation/reconstruction over GSM8K
|
||||
([[project-two-disjoint-gsm8k-readers]]). Committed serving sits at a few
|
||||
correct / many *refused* / **zero wrong** — i.e. the binary's "refuse" branch
|
||||
is doing almost all the work. The reader *knew things* about the refused cases
|
||||
it had no way to say.
|
||||
- **R2 — constraint compiler.** Finite-integer linear-constraint setup compiler
|
||||
(ADR-0217, [[project-r2-constraint-compiler-built]]): prose → reader →
|
||||
`ConstraintProblem` → solver → answer-verifier with a **contradiction flag**.
|
||||
Here the engine started producing *structured, typed* intermediate state
|
||||
(underdetermined systems, contradictions) that is precisely *not* "verified
|
||||
answer" and *not* "nothing" — the first hard evidence that two modes were too
|
||||
few. The reader owns *setup*; the solver owns *solvability* — already a graded
|
||||
split.
|
||||
- **R3 — single-rate / rates-time-state.** The rate family
|
||||
([[project-it-idle-integration-and-frontier]]): exact-rational time-unit
|
||||
conversion (R3.2), single-rate setup → `quantity = rate × time`. Gold
|
||||
`13/13`, reader `9/0/4`. Rate/time problems are ambiguity-rich, which is
|
||||
exactly where *partial progress* and *multiple candidates* become the honest
|
||||
outputs.
|
||||
- **CMB — the combined-rate ladder.** Combined rate is a *new semantic object*
|
||||
(two explicit rates over one shared unit, combined by an explicit `sum` /
|
||||
`difference` mode, then single-rate algebra over the result), with its own
|
||||
organ (`generate/combined_rate_comprehension/`) and ruler
|
||||
(`evals/combined_rate_oracle/`). Built as a staged ladder: **CMB-a** (model +
|
||||
gold + setup oracle — *"the ruler is defined and gold-valid"*, landed #650,
|
||||
[`cmb-a-combined-rate-ruler-2026-06-08`](../analysis/cmb-a-combined-rate-ruler-2026-06-08.md))
|
||||
→ **CMB-b** (exact solver, int-or-refuse) → **CMB-c** (reader: prose →
|
||||
`CombinedRateProblem | Refusal`) → **CMB-d** (router / contemplation wiring +
|
||||
failure-family) → **CMB-e** (capability ledger). CMB is the sharpest evidence
|
||||
for this whole session: its **2×2 domain-entry grid** does not produce a flat
|
||||
refuse — it produces *typed, graded* outcomes at the setup boundary itself:
|
||||
|
||||
| | combination cue | no cue |
|
||||
|----------------|------------------------|------------------------------|
|
||||
| **two rates** | solved (sum/difference)| `combine_mode_ambiguous` |
|
||||
| **one rate** | `missing_second_rate` | `not_combined_rate_shaped` |
|
||||
|
||||
`combine_mode_ambiguous` is explicitly *"a substantive refusal"* — the engine
|
||||
knows it is in combined-rate territory but cannot pick a unique mode. That is a
|
||||
textbook `MULTIPLE_CANDIDATES` / `clarify`, **not** the same epistemic object as
|
||||
`not_combined_rate_shaped` (out-of-domain, step aside to R3) or a clean solve.
|
||||
The binary gate has one bucket — "refuse" — for three structurally different
|
||||
states. CMB-a *already typed them apart*; it just has nowhere to send the
|
||||
distinction at serving.
|
||||
|
||||
**The throughline:** every rung — R1, R2, R3, CMB — kept generating *typed
|
||||
intermediate epistemic states* (refused-but-known, underdetermined, contradicted,
|
||||
**substantively-ambiguous**, out-of-domain), and the typed-contemplation lane
|
||||
([[project-next-batch-contemplation-prototype]]: `core/comprehension_attempt/`,
|
||||
failure-family registry, `teaching/proposals/` proposal-only emitter) kept
|
||||
*exploring typed candidates in isolation* without ever denting wrong=0 (the
|
||||
empirical proof of §2). The binary serving gate had nowhere to put any of it
|
||||
except "refuse." The ServabilityBlade is the organ that finally gives that
|
||||
accumulated, typed, isolated evidence a truthful way to reach the user.
|
||||
|
||||
---
|
||||
|
||||
## How it fits CORE (the ingredients already exist)
|
||||
|
||||
The blade is the **consumer** of organs already built, not a new subsystem:
|
||||
epistemic states (derivation/verify), safety/normative clearance
|
||||
([[safety-refusal-policy]] → `refuse_unsafe`, `required_disclosures`),
|
||||
setup/answer oracles (`verified` vs `underdetermined` — e.g. CMB-a's combined-rate
|
||||
ruler), the typed-contemplation / comprehension-attempt lane (→ the candidate
|
||||
evidence), proposal-only learning (`proposal`
|
||||
vs `ratified`), the existing [`refusal-taxonomy`](../refusal-taxonomy.md) (→ the
|
||||
`refuse_*` / `contradiction_report` branches), and the calibrated reliability gate
|
||||
(ADR-0175 Wilson floors → `risk_level` / `confidence_basis` for the provisional
|
||||
mode). The work is **integration + a policy object** — same character as the
|
||||
AGI-spine integration phase ([[milestone-agi-spine-A-to-E-complete]]).
|
||||
|
||||
## Why this matters for AGI-candidacy
|
||||
|
||||
A real intelligence does not only *answer* or *refuse*. It can say, distinctly and
|
||||
truthfully: *I know this. I think this. I suspect this. I can prove this part. I
|
||||
cannot prove that part. Here are the assumptions. Here is the risk. Here is what
|
||||
would settle it.* That is **more** truthful than the binary, not less — the binary
|
||||
discards true statements about the engine's own knowledge state by collapsing them
|
||||
into silence. It sits under the telos ([[project-core-is-one-continuous-life]]): a
|
||||
continuously-learning life that *thinks with* its operator must share in-progress
|
||||
thought honestly, not only ratified conclusions.
|
||||
|
||||
---
|
||||
|
||||
## Decision & sequencing
|
||||
|
||||
- **Decided (doctrine, not yet ADR):** `wrong=0` ≡ **no false presentation of
|
||||
epistemic status** — a disclosure discipline. Serving becomes graded controlled
|
||||
disclosure; the answer/refuse binary is retained verbatim only at the strict end
|
||||
(`sealed_eval` / `safety_critical`).
|
||||
- **Decided:** typed + isolated candidate guesses are admissible at **practice
|
||||
time** (they already are, via the sealed lane + CMB; we are naming the law).
|
||||
- **Proposed:** `ProblemAttemptSession` (PRA) as the practice-lane evidence
|
||||
object; `ServabilityDecision` (SRV) as the serving-lane policy object.
|
||||
- **Sequencing (do not invert): PRA → REL → SRV.** The blade consumes candidate
|
||||
evidence, so it cannot precede the attempt layer. Data models may be drafted now
|
||||
(above); wiring waits.
|
||||
- **Preserved / not touched:** the wrong=0 verify gate, determinism/replay, the
|
||||
fail-closed safety/refusal path. The blade *consumes* these; it never weakens
|
||||
them. Any future wiring PR bundles `docs/runtime_contracts.md` + contract-test
|
||||
updates (the surface contract changes shape), and gives the `context → admissible
|
||||
modes` table a meaningfully-failing test per branch (per the
|
||||
schema-defined-proof-obligation rule in CLAUDE.md).
|
||||
|
||||
## Open / next
|
||||
|
||||
- **PRA** — firm the `ProblemAttemptSession` evidence shape on the existing
|
||||
`core/comprehension_attempt/` substrate (verified + eliminated candidates +
|
||||
residual ambiguity, explicit).
|
||||
- **REL** — relevance / distractor grounding, so `confidence_basis` and
|
||||
`servable_claims` are grounded, not asserted.
|
||||
- **SRV** — implement the blade against `ServabilityDecision`; wire as the
|
||||
realizer's input; unify the refusal taxonomy under its `refuse_*` /
|
||||
`contradiction_report` branches.
|
||||
- **ADR** — promote this doctrine to an ADR *only* once SRV has a concrete
|
||||
consumer and the risk-tier mapping is a closed, tested contract.
|
||||
Loading…
Reference in a new issue