diff --git a/docs/analysis/ask-serving-integration-scoping-2026-06-08.md b/docs/analysis/ask-serving-integration-scoping-2026-06-08.md new file mode 100644 index 00000000..0b59cf3d --- /dev/null +++ b/docs/analysis/ask-serving-integration-scoping-2026-06-08.md @@ -0,0 +1,199 @@ +# ASK serving-integration — `ask_serving_enabled` — scoping brief + +**Date:** 2026-06-08 · **Status:** scoping (NO CODE) · **HOLD for review** · +**Branch:** `docs/serving-integration-scoping` + +**What this brief is.** The scope for the *one* served-surface decision the ASK lane +deferred. The off-serving ASK lane is complete and on main: + +```text +Q1-B typed residue + ask classification (LimitationAssessment.missing_slots) +Q1-C grounded-only rendering (render_question → EpistemicQuestion) +Q1-D off-serving delivery artifact (deliver_ask → QUESTION_NEEDED + sink) +``` + +`deliver_ask` / `emit_question` exist and are tested, but **nothing calls them from a +live path** — they are produced-but-not-emitted, exactly as P1-B's verified producer +is built-but-not-served. This brief scopes the step that closes that gap: letting a +`QUESTION_NEEDED` actually reach a user, behind a named gate. **No code here** — this +is the decision surface for review. + +> Companions: [[q1-d-ask-bus-delivery-scoping-2026-06-08]] (the off-serving rung, §7 +> defers exactly this), [[verified-serving-wiring-scoping-2026-06-08]] (the VERIFIED +> half of the same "where off-serving stops" line). Design of record: the session doc +> §5 / §1.5.8 (the disclosure bus). + +--- + +## 1. The five things this decision must pin + +The steer named five; each is grounded against shipped code below. + +```text +1. ask_serving_enabled — the kill switch +2. the pass_manager integration point — where deliver_ask is actually called +3. the Q1B_ASK_CARVE_OUT retirement gate + registry flip +4. served-surface behaviour for a QUESTION_NEEDED reaching a user +5. the no-question/no-proposal dead-zone proof +``` + +--- + +## 2. The integration point (grounded) + +`generate/contemplation/pass_manager.py::contemplate(text, *, proposal_root=...)` is +the live contemplation entry. When every attempt is refused it calls +`_classify_all_refused(text, attempts, findings, proposal_root)`, which is where +`emit_proposal` fires for a `proposal_allowed` family and terminates +`PROPOSAL_EMITTED`. **That is the exact analogous site for ASK:** when a refused +attempt's family maps to `ask_question` (via `assess_from_attempt`), call `deliver_ask` +and — on a renderable result — `emit_question` to the sink and terminate +`QUESTION_NEEDED`; on the D2 fallback, fall through to today's proposal/refuse path. + +`contemplate()` is called from `chat/runtime.py` at three sites (827, 901, 1364) — so +the contemplation `Terminal` is already on the served path. The served question text +travels the same way `PROPOSAL_EMITTED` already does. + +**Two-layer split (the recommendation), mirroring VERIFIED:** + +- **Layer A — pass-manager emission (off-serving still).** `contemplate` emits + `QUESTION_NEEDED` + writes the `teaching/questions/` artifact, exactly as it emits + `PROPOSAL_EMITTED` today. This changes the *contemplation* terminal but **reaches no + user** — the teaching loop is off-serving. This layer is buildable behind no gate + (it only adds a terminal the pass can reach), but see §3: it interacts with the + carve-out and so should still wait for the gate, to avoid double-emission churn. +- **Layer B — served delivery (the gated surface).** `chat/runtime.py`, gated by + `ask_serving_enabled`, renders the `DeliveredQuestion.question.text` to the user as + the served response when the terminal is `QUESTION_NEEDED`. This is the only layer + that actually asks the user anything. + +Decision to confirm: **do Layer A and Layer B land together behind one gate, or does +Layer A land first (pass emits, nothing served) and Layer B follow?** Recommend +**together, behind `ask_serving_enabled`** — Layer A alone creates the double-emission +state in §3 with no compensating benefit. + +--- + +## 3. The kill switch + the carve-out retirement gate (the coupled core) + +### 3.1 `ask_serving_enabled` + +Add `ask_serving_enabled: bool = False` to `core/config.py`, the sibling of the +existing `estimation_enabled = False` kill-switch pattern. Default **off**: the +served question path is dark until deliberately enabled, holdout-gated (§5). + +### 3.2 Why the carve-out flip is *coupled* to the gate (not to Q1-D) + +Q1-B introduced `Q1B_ASK_CARVE_OUT` for `missing_total_count` / `missing_weighted_total`: +the disclosure layer classifies them `ask_question`, but the shipped `REGISTRY` keeps +`proposal_allowed = True` so the proposal pile keeps working. The carve-out's +*retirement condition* is written into `limitation.py`: *"Once ASK is serving, flip +`proposal_allowed = False` on these two families, drop the carve-out set, amend the +test."* The operative word is **serving** — not "delivery exists" (Q1-D already shipped +that off-serving). So: + +```text +carve-out retires ⟺ ask_serving_enabled is the ruling + AND a QUESTION_NEEDED is actually served for these families + AND the §4 dead-zone proof holds +``` + +Until then, `proposal_allowed` stays `True`. During the gate's "off" state both signals +coexist (the off-serving question artifact + the proposal) — intentional, no loss. + +### 3.3 The flip, as a single reviewed act + +When `ask_serving_enabled` is turned on for these families, in **one** change: +1. `proposal_allowed = False` for `missing_total_count` / `missing_weighted_total`. +2. Drop them from `Q1B_ASK_CARVE_OUT` (empty the set, or remove the constant). +3. Amend the `proposal_allowed` invariant test + the carve-out test. +4. The §4 dead-zone proof test must already be green. + +This is the "conscious act, not a silent re-key" the carve-out was built to force. + +--- + +## 4. The no-question/no-proposal dead-zone proof (the wrong=0-adjacent guard) + +**The hazard.** The flip removes the proposal signal for these families. If, for some +input class, the family classifies `ask_question` BUT the question is unrenderable +(D2), AND the proposal is now off, the family would terminate `NO_PROGRESS` with **no +served question and no proposal** — a dead zone where a user-resolvable gap produces +*nothing*. That is the ASK-side wrong=0 hazard: not a false answer, but a silent loss +of a capability that previously at least proposed. + +**The proof obligation (before any flip).** For every family being flipped, prove that +**no input class lands in the dead zone** — i.e. for every reachable assessment of that +family, the question renders (so `QUESTION_NEEDED` is served), OR the proposal is still +on. Concretely: + +- The `missing_*` families have pinned single slots in `_FAMILY_TO_MISSING_SLOTS` with + mapped types (`count_int` / `measured_unit_int`) → they **always render** today, so + the dead zone is currently empty. The proof must show this is *structural*, not + incidental: a test that asserts `deliver_ask` returns `QUESTION_NEEDED` (never a + fallback) for every reachable assessment of the flipped families. +- If any future residue change could make one of these unrenderable (multi-slot, + unmapped type), the flip must be blocked for that family until either the renderer + covers it or the proposal stays on. + +**The rule:** `proposal_allowed` may flip `True → False` for a family **only** if a +test proves every reachable ask of that family renders. The dead-zone proof is a +precondition of the flip, enforced like the D2 guard (it must *fail* if a fallback path +is reachable for a flipped family). + +--- + +## 5. Served-surface behaviour + holdout gating + +### 5.1 What a served `QUESTION_NEEDED` looks like + +When `ask_serving_enabled` and the terminal is `QUESTION_NEEDED`, the served surface +returns the `DeliveredQuestion.question.text` (the grounded-only rendered question) as +the response — distinct from a committed answer, an `[approximate]` disclosure, or a +refusal. It is an **intake request**: the disposition is `ServedDisposition.ASK` +(already mapped in `disposition.py`). The question names nothing ungrounded (Q1-C +guarantee), so it cannot leak a fabricated entity even on the served path. + +Open sub-decision: **the served prefix/marker.** VERIFIED gets a distinct `[verified]` +prefix; APPROXIMATE gets `[approximate]`. ASK should get its own surface marker (a +question is neither). Recommend a distinct, tested marker; pin the exact string at +build, not here. + +### 5.2 Holdout gate (no quiet widening) + +Like VERIFIED, ASK serving must be proven on a holdout before it widens live: a +validate-first probe over a held-out set confirming (a) served questions are +grounded-only (no fabrication escapes on the served path), (b) no family in the flip +set hits the dead zone, (c) the GSM8K serving seal is byte-identical (ASK is +off-the-metric — it asks, it does not answer — but the probe proves it). Only then does +`ask_serving_enabled` go on, one surface at a time. + +--- + +## 6. What this is NOT + +- **Not** a dialogue manager / multi-turn state machine — one grounded question, then + the existing flow; the answer round-trip is Q2 (`AnswerBinding`, a separate batch). +- **Not** a re-render — the served path emits the Q1-D `DeliveredQuestion.question.text` + verbatim; no second prose surface. +- **Not** a GSM8K-metric move — ASK asks, it never answers; the pinned SHAs and + `CLAIMS.md` are untouched. The holdout probe proves it. +- **Not** the carve-out flip *yet* — the flip is the terminal act of *this* decision, + gated on `ask_serving_enabled` + the §4 dead-zone proof, not on Q1-D. + +--- + +## 7. The questions for the ruling + +1. **Gate:** add `ask_serving_enabled = False` (sibling of `estimation_enabled`)? (rec: yes) +2. **Layering:** land pass-emission (Layer A) and served delivery (Layer B) together + behind the one gate? (rec: yes — Layer A alone only adds double-emission churn) +3. **Carve-out flip:** retire `Q1B_ASK_CARVE_OUT` + flip `proposal_allowed` as a single + reviewed act, gated on the gate AND the §4 dead-zone proof? (rec: yes) +4. **Dead-zone proof:** require a passing "every reachable ask of a flipped family + renders" test as a precondition of any flip? (rec: yes — this is the ASK wrong=0 guard) +5. **Served marker:** a distinct ASK surface marker (not `[verified]` / `[approximate]`)? (rec: yes) +6. **Holdout:** validate-first probe (grounded-only on served path + no dead zone + + GSM8K seal byte-identical) before the gate goes on? (rec: yes) + +No served-surface code until this brief is reviewed. diff --git a/docs/analysis/verified-serving-wiring-scoping-2026-06-08.md b/docs/analysis/verified-serving-wiring-scoping-2026-06-08.md new file mode 100644 index 00000000..9e1e662f --- /dev/null +++ b/docs/analysis/verified-serving-wiring-scoping-2026-06-08.md @@ -0,0 +1,183 @@ +# VERIFIED serving-wiring — `verified_serving_enabled` — scoping brief + +**Date:** 2026-06-08 · **Status:** scoping (NO CODE) · **HOLD for review** · +**Branch:** `docs/serving-integration-scoping` + +**What this brief is.** The scope for the served-surface decision the VERIFIED lane +deferred. The off-serving VERIFIED lane is complete and on main: + +```text +P1-A the VERIFIED contract (verified_contract.py — two independent reads converge) +P1-B gold-setup-backed producer (evals/constraint_oracle/verified_producer.py — OFF-SERVING) +P1-C bound_slots_digest (a separable, load-bearing proof obligation) +``` + +P1-B verifies 7/13 real R2 problems with wrong=0 — but it is **gold-setup-backed**, so +it is structurally off-serving: the independent read is the INV-25 hand-authored gold +SETUP, which is not available at serving time. This brief scopes what a *serving-time* +VERIFIED would require — and why it cannot reuse P1-B. **No code here.** + +> Companions: [[ask-serving-integration-scoping-2026-06-08]] (the ASK half of the same +> "where off-serving stops" line), [[VERIFIED-canonical-comparison-scoping-2026-06-06]] +> (the validate-first probe that already KILLED the naive fold-reader producer), +> [[stage2-epistemic-disclosure-bus-verified-v1-scoping-2026-06-08]] (Doc 1). + +--- + +## 1. The seam (grounded) — and why it is still inert by design + +`generate/derivation/verify.py::_canonically_verified(verified, problem_text, policy)` +is the ADR-0206 §5 VERIFIED gate — *the only thing that may license a math answer past +gold* (resolve a disagreement STRICT refuses). It **returns `None` today**, so the +widening is structurally inert: disagreement refuses regardless of `policy`, preserving +absolute `wrong == 0`. Its own docstring states the bright line this brief must honour: + +> "A reliability *license* (statistical) must NEVER substitute here: math serving is +> absolute-wrong=0, not disclosed like the cognition path." + +So VERIFIED serving = replacing that `return None` with a producer that returns a +derivation **only when it is canonically VERIFIED** (proven correct, not merely sound), +behind a kill switch, proven on a holdout. Everything below is the eligibility bar for +that producer. + +--- + +## 2. The five things this decision must pin + +```text +1. the gold-free independent-reader requirement — the crux +2. verified_serving_enabled — the kill switch +3. holdout gates — validate-first, no quiet widening +4. proof-producer eligibility — what may plug into _canonically_verified +5. the explicit ban on eval-gold-backed serving — why P1-B cannot serve +``` + +--- + +## 3. The gold-free independent-reader requirement (the crux) + +VERIFIED means **two independent reads that converge on one canonical structure** +(P1-A): a faithful solve of a *wrong read* is caught because the independent read +disagrees. P1-B's two reads are `read_constraint_problem` (the engine reader) vs the +**gold-authored setup**. At serving time there is no gold. So serving-time VERIFIED +needs a **second, gold-free reader** whose disagreement is the safety mechanism. + +The hard constraints, from the killed-probe doc and CLAUDE.md: + +- **Independence must be in the READING, not the solving.** Back-substitution catches + solve-errors, never read-errors. Two solvers over one reading is *fake* independence. + The second reader must parse the problem into the same `ConstraintProblem` structure + by a **genuinely different route** (different lineage, asserted by the + `SAME_READER_LINEAGE` firewall / INV-27 reader-disjointness), and converge on the + same canonical signature. +- **No eval gold in the read** (§7). The second reader may not consult, hash, or be + derived from any gold artifact — not the answer, not the setup. If it needs gold to + read, it is P1-B, and P1-B does not serve. +- **Conservative refuse-on-doubt carries wrong=0.** The second reader, like the first, + refuses when uncertain. VERIFIED fires only on *agreement of two confident, + independent reads*; any refusal or disagreement → STRICT refuses (today's behaviour). + +**Eligibility, stated as a gate:** a serving-time VERIFIED producer is eligible **only +if** it exhibits two reads with (a) distinct, firewall-asserted lineages, (b) neither +read derived from gold, (c) convergence on one canonical `ConstraintProblem` signature, +(d) back-substitution + boundary-clear + bound-slots (the P1-A/P1-C obligations), and +(e) refuse-preferring failure. Absent any one, it stays `None`. + +Whether such a second R2 reader *exists yet* is the open empirical question — the killed +fold-reader probe is the cautionary precedent: complementary readers were ~98% wrong on +the refused set. **This brief does not assume one exists; it sets the bar a candidate +must clear, validate-first, before any wiring.** + +--- + +## 4. `verified_serving_enabled` — the kill switch + +Add `verified_serving_enabled: bool = False` to `core/config.py` (sibling of +`estimation_enabled`). Default **off**. When off, `_canonically_verified` returns `None` +unconditionally (today's inert state) regardless of any producer being present. The +switch is the single audited place that lights the seam, and it stays off until §5. + +--- + +## 5. Holdout gates — validate-first, no quiet widening + +VERIFIED may not widen live until proven on a **held-out** set it never trained or +tuned on (INV-25 discipline; the killed-probe doc's validate-first rule). The gate, in +order: + +1. **Holdout probe (off-serving):** run the candidate producer over a held-out R2 set + with gold answers withheld; require **wrong == 0** on everything it marks VERIFIED, + and that everything it cannot independently verify it *refuses* (over-refusal is + acceptable; one wrong is disqualifying). +2. **Seal byte-identity:** the GSM8K candidate-graph serving seal (pinned SHAs) stays + byte-identical — VERIFIED widens a *different* surface (R2 constraint answers), it + must not perturb the sealed lane. +3. **Disagreement-still-refuses:** a test proving that with the producer wired and the + switch ON, a faithful solve of a deliberately wrong read still refuses (the P1-A + poison test, now on the served path). +4. Only then does `verified_serving_enabled` go on, **one surface at a time** (R2 + first; R4 second), each with its own holdout pass. + +--- + +## 6. The served surface — a distinct `[verified]` disclosure + +When wired and enabled, a VERIFIED R2 answer is served under its **own** disclosure +claim/marker — the locked decision from Doc 1's review: *VERIFIED gets a distinct +served disclosure mode + `[verified]` prefix, NEVER reused from `[approximate]`*. +VERIFIED is a *license* claim (more licensed than gold-strict), not a *speculation* +claim. The route to it is the only sanctioned one: `disclosure_for_verification(result)` +→ `(EpistemicState.VERIFIED, DisclosureClaim.VERIFIED)` → `ServedDisposition.DISCLOSE` +under the P0-3 guard (which already degrades an unbacked VERIFIED claim to COMMIT). + +--- + +## 7. The explicit ban on eval-gold-backed serving (the load-bearing non-claim) + +**P1-B must never become a serving producer.** It is gold-setup-backed: its independent +read is the hand-authored gold SETUP. Wiring it into `_canonically_verified` would mean +the engine "verifies" by consulting the answer key's structure — circular, and a +wrong=0 fiction (it would serve VERIFIED on exactly the problems it was handed the setup +for). The ban, stated so a test can enforce it: + +- `_canonically_verified` (and any serving producer behind it) may import **nothing** + from `evals/` — no gold loader, no `gold_to_problem`, no `r2_gold`. An AST/import test + asserts the serving path is gold-free, the mirror of the off-serving AST tests. +- P1-B stays in `evals/` precisely so this boundary is structural: `evals → core` is the + allowed direction; `core/generate serving → evals` is forbidden. +- "Verified on a holdout" (§5) is **not** the same as "verified by gold at serving" — + the holdout *measures* a gold-free producer; it never *feeds* gold into one. + +This is the VERIFIED analogue of the GSM8K "candidate-graph owns the metric; no bridge +re-enables without sealed/independent wrong=0" discipline. + +--- + +## 8. What this is NOT + +- **Not** a claim that a serving-time VERIFIED producer exists — §3 sets the eligibility + bar; whether an R2 second reader clears it is an open, validate-first question. +- **Not** a reliability/statistical license at the math seam — that path is the + cognition `[approximate]` disclosure; math serving is absolute wrong=0 (§1 bright line). +- **Not** a GSM8K-seal move — VERIFIED widens R2 constraint answers, seal stays + byte-identical (§5.2). +- **Not** P1-B promoted to serving — explicitly banned (§7); P1-B stays off-serving. + +--- + +## 9. The questions for the ruling + +1. **Eligibility bar:** adopt §3 (a)–(e) as the gate any serving VERIFIED producer must + clear — gold-free second reader, independence-in-the-reading, refuse-preferring? (rec: yes) +2. **Gate:** add `verified_serving_enabled = False` (sibling of `estimation_enabled`), + `_canonically_verified` stays `None` while off? (rec: yes) +3. **Holdout:** require the §5 validate-first sequence (wrong=0 on holdout + seal + byte-identity + disagreement-still-refuses) before the switch, R2-first? (rec: yes) +4. **Gold ban:** enforce the §7 import ban (serving path imports nothing from `evals/`) + with a test, keeping P1-B off-serving forever? (rec: yes) +5. **Surface:** serve VERIFIED under its distinct `[verified]` marker via + `disclosure_for_verification` only? (rec: yes — the locked Doc 1 decision) + +No served-surface code until this brief is reviewed. Pairs with +[[ask-serving-integration-scoping-2026-06-08]] — together they draw the full line where +off-serving stops.