docs(epistemic): scope ASK and VERIFIED serving gates
Accepts the ASK + VERIFIED serving-integration scoping briefs. No served-surface code is introduced. ASK is identified as the next gated serving slice; VERIFIED remains blocked on a gold-free second reader and explicit eval-gold serving ban.
This commit is contained in:
Shay
GitHub
parent
07149c208a
commit
28cb42d0dd
2 changed files with 382 additions and 0 deletions
199
docs/analysis/ask-serving-integration-scoping-2026-06-08.md
Normal file
199
docs/analysis/ask-serving-integration-scoping-2026-06-08.md
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
# ASK serving-integration — `ask_serving_enabled` — scoping brief
|
||||
|
||||
**Date:** 2026-06-08 · **Status:** scoping (NO CODE) · **HOLD for review** ·
|
||||
**Branch:** `docs/serving-integration-scoping`
|
||||
|
||||
**What this brief is.** The scope for the *one* served-surface decision the ASK lane
|
||||
deferred. The off-serving ASK lane is complete and on main:
|
||||
|
||||
```text
|
||||
Q1-B typed residue + ask classification (LimitationAssessment.missing_slots)
|
||||
Q1-C grounded-only rendering (render_question → EpistemicQuestion)
|
||||
Q1-D off-serving delivery artifact (deliver_ask → QUESTION_NEEDED + sink)
|
||||
```
|
||||
|
||||
`deliver_ask` / `emit_question` exist and are tested, but **nothing calls them from a
|
||||
live path** — they are produced-but-not-emitted, exactly as P1-B's verified producer
|
||||
is built-but-not-served. This brief scopes the step that closes that gap: letting a
|
||||
`QUESTION_NEEDED` actually reach a user, behind a named gate. **No code here** — this
|
||||
is the decision surface for review.
|
||||
|
||||
> Companions: [[q1-d-ask-bus-delivery-scoping-2026-06-08]] (the off-serving rung, §7
|
||||
> defers exactly this), [[verified-serving-wiring-scoping-2026-06-08]] (the VERIFIED
|
||||
> half of the same "where off-serving stops" line). Design of record: the session doc
|
||||
> §5 / §1.5.8 (the disclosure bus).
|
||||
|
||||
---
|
||||
|
||||
## 1. The five things this decision must pin
|
||||
|
||||
The steer named five; each is grounded against shipped code below.
|
||||
|
||||
```text
|
||||
1. ask_serving_enabled — the kill switch
|
||||
2. the pass_manager integration point — where deliver_ask is actually called
|
||||
3. the Q1B_ASK_CARVE_OUT retirement gate + registry flip
|
||||
4. served-surface behaviour for a QUESTION_NEEDED reaching a user
|
||||
5. the no-question/no-proposal dead-zone proof
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. The integration point (grounded)
|
||||
|
||||
`generate/contemplation/pass_manager.py::contemplate(text, *, proposal_root=...)` is
|
||||
the live contemplation entry. When every attempt is refused it calls
|
||||
`_classify_all_refused(text, attempts, findings, proposal_root)`, which is where
|
||||
`emit_proposal` fires for a `proposal_allowed` family and terminates
|
||||
`PROPOSAL_EMITTED`. **That is the exact analogous site for ASK:** when a refused
|
||||
attempt's family maps to `ask_question` (via `assess_from_attempt`), call `deliver_ask`
|
||||
and — on a renderable result — `emit_question` to the sink and terminate
|
||||
`QUESTION_NEEDED`; on the D2 fallback, fall through to today's proposal/refuse path.
|
||||
|
||||
`contemplate()` is called from `chat/runtime.py` at three sites (827, 901, 1364) — so
|
||||
the contemplation `Terminal` is already on the served path. The served question text
|
||||
travels the same way `PROPOSAL_EMITTED` already does.
|
||||
|
||||
**Two-layer split (the recommendation), mirroring VERIFIED:**
|
||||
|
||||
- **Layer A — pass-manager emission (off-serving still).** `contemplate` emits
|
||||
`QUESTION_NEEDED` + writes the `teaching/questions/` artifact, exactly as it emits
|
||||
`PROPOSAL_EMITTED` today. This changes the *contemplation* terminal but **reaches no
|
||||
user** — the teaching loop is off-serving. This layer is buildable behind no gate
|
||||
(it only adds a terminal the pass can reach), but see §3: it interacts with the
|
||||
carve-out and so should still wait for the gate, to avoid double-emission churn.
|
||||
- **Layer B — served delivery (the gated surface).** `chat/runtime.py`, gated by
|
||||
`ask_serving_enabled`, renders the `DeliveredQuestion.question.text` to the user as
|
||||
the served response when the terminal is `QUESTION_NEEDED`. This is the only layer
|
||||
that actually asks the user anything.
|
||||
|
||||
Decision to confirm: **do Layer A and Layer B land together behind one gate, or does
|
||||
Layer A land first (pass emits, nothing served) and Layer B follow?** Recommend
|
||||
**together, behind `ask_serving_enabled`** — Layer A alone creates the double-emission
|
||||
state in §3 with no compensating benefit.
|
||||
|
||||
---
|
||||
|
||||
## 3. The kill switch + the carve-out retirement gate (the coupled core)
|
||||
|
||||
### 3.1 `ask_serving_enabled`
|
||||
|
||||
Add `ask_serving_enabled: bool = False` to `core/config.py`, the sibling of the
|
||||
existing `estimation_enabled = False` kill-switch pattern. Default **off**: the
|
||||
served question path is dark until deliberately enabled, holdout-gated (§5).
|
||||
|
||||
### 3.2 Why the carve-out flip is *coupled* to the gate (not to Q1-D)
|
||||
|
||||
Q1-B introduced `Q1B_ASK_CARVE_OUT` for `missing_total_count` / `missing_weighted_total`:
|
||||
the disclosure layer classifies them `ask_question`, but the shipped `REGISTRY` keeps
|
||||
`proposal_allowed = True` so the proposal pile keeps working. The carve-out's
|
||||
*retirement condition* is written into `limitation.py`: *"Once ASK is serving, flip
|
||||
`proposal_allowed = False` on these two families, drop the carve-out set, amend the
|
||||
test."* The operative word is **serving** — not "delivery exists" (Q1-D already shipped
|
||||
that off-serving). So:
|
||||
|
||||
```text
|
||||
carve-out retires ⟺ ask_serving_enabled is the ruling
|
||||
AND a QUESTION_NEEDED is actually served for these families
|
||||
AND the §4 dead-zone proof holds
|
||||
```
|
||||
|
||||
Until then, `proposal_allowed` stays `True`. During the gate's "off" state both signals
|
||||
coexist (the off-serving question artifact + the proposal) — intentional, no loss.
|
||||
|
||||
### 3.3 The flip, as a single reviewed act
|
||||
|
||||
When `ask_serving_enabled` is turned on for these families, in **one** change:
|
||||
1. `proposal_allowed = False` for `missing_total_count` / `missing_weighted_total`.
|
||||
2. Drop them from `Q1B_ASK_CARVE_OUT` (empty the set, or remove the constant).
|
||||
3. Amend the `proposal_allowed` invariant test + the carve-out test.
|
||||
4. The §4 dead-zone proof test must already be green.
|
||||
|
||||
This is the "conscious act, not a silent re-key" the carve-out was built to force.
|
||||
|
||||
---
|
||||
|
||||
## 4. The no-question/no-proposal dead-zone proof (the wrong=0-adjacent guard)
|
||||
|
||||
**The hazard.** The flip removes the proposal signal for these families. If, for some
|
||||
input class, the family classifies `ask_question` BUT the question is unrenderable
|
||||
(D2), AND the proposal is now off, the family would terminate `NO_PROGRESS` with **no
|
||||
served question and no proposal** — a dead zone where a user-resolvable gap produces
|
||||
*nothing*. That is the ASK-side wrong=0 hazard: not a false answer, but a silent loss
|
||||
of a capability that previously at least proposed.
|
||||
|
||||
**The proof obligation (before any flip).** For every family being flipped, prove that
|
||||
**no input class lands in the dead zone** — i.e. for every reachable assessment of that
|
||||
family, the question renders (so `QUESTION_NEEDED` is served), OR the proposal is still
|
||||
on. Concretely:
|
||||
|
||||
- The `missing_*` families have pinned single slots in `_FAMILY_TO_MISSING_SLOTS` with
|
||||
mapped types (`count_int` / `measured_unit_int`) → they **always render** today, so
|
||||
the dead zone is currently empty. The proof must show this is *structural*, not
|
||||
incidental: a test that asserts `deliver_ask` returns `QUESTION_NEEDED` (never a
|
||||
fallback) for every reachable assessment of the flipped families.
|
||||
- If any future residue change could make one of these unrenderable (multi-slot,
|
||||
unmapped type), the flip must be blocked for that family until either the renderer
|
||||
covers it or the proposal stays on.
|
||||
|
||||
**The rule:** `proposal_allowed` may flip `True → False` for a family **only** if a
|
||||
test proves every reachable ask of that family renders. The dead-zone proof is a
|
||||
precondition of the flip, enforced like the D2 guard (it must *fail* if a fallback path
|
||||
is reachable for a flipped family).
|
||||
|
||||
---
|
||||
|
||||
## 5. Served-surface behaviour + holdout gating
|
||||
|
||||
### 5.1 What a served `QUESTION_NEEDED` looks like
|
||||
|
||||
When `ask_serving_enabled` and the terminal is `QUESTION_NEEDED`, the served surface
|
||||
returns the `DeliveredQuestion.question.text` (the grounded-only rendered question) as
|
||||
the response — distinct from a committed answer, an `[approximate]` disclosure, or a
|
||||
refusal. It is an **intake request**: the disposition is `ServedDisposition.ASK`
|
||||
(already mapped in `disposition.py`). The question names nothing ungrounded (Q1-C
|
||||
guarantee), so it cannot leak a fabricated entity even on the served path.
|
||||
|
||||
Open sub-decision: **the served prefix/marker.** VERIFIED gets a distinct `[verified]`
|
||||
prefix; APPROXIMATE gets `[approximate]`. ASK should get its own surface marker (a
|
||||
question is neither). Recommend a distinct, tested marker; pin the exact string at
|
||||
build, not here.
|
||||
|
||||
### 5.2 Holdout gate (no quiet widening)
|
||||
|
||||
Like VERIFIED, ASK serving must be proven on a holdout before it widens live: a
|
||||
validate-first probe over a held-out set confirming (a) served questions are
|
||||
grounded-only (no fabrication escapes on the served path), (b) no family in the flip
|
||||
set hits the dead zone, (c) the GSM8K serving seal is byte-identical (ASK is
|
||||
off-the-metric — it asks, it does not answer — but the probe proves it). Only then does
|
||||
`ask_serving_enabled` go on, one surface at a time.
|
||||
|
||||
---
|
||||
|
||||
## 6. What this is NOT
|
||||
|
||||
- **Not** a dialogue manager / multi-turn state machine — one grounded question, then
|
||||
the existing flow; the answer round-trip is Q2 (`AnswerBinding`, a separate batch).
|
||||
- **Not** a re-render — the served path emits the Q1-D `DeliveredQuestion.question.text`
|
||||
verbatim; no second prose surface.
|
||||
- **Not** a GSM8K-metric move — ASK asks, it never answers; the pinned SHAs and
|
||||
`CLAIMS.md` are untouched. The holdout probe proves it.
|
||||
- **Not** the carve-out flip *yet* — the flip is the terminal act of *this* decision,
|
||||
gated on `ask_serving_enabled` + the §4 dead-zone proof, not on Q1-D.
|
||||
|
||||
---
|
||||
|
||||
## 7. The questions for the ruling
|
||||
|
||||
1. **Gate:** add `ask_serving_enabled = False` (sibling of `estimation_enabled`)? (rec: yes)
|
||||
2. **Layering:** land pass-emission (Layer A) and served delivery (Layer B) together
|
||||
behind the one gate? (rec: yes — Layer A alone only adds double-emission churn)
|
||||
3. **Carve-out flip:** retire `Q1B_ASK_CARVE_OUT` + flip `proposal_allowed` as a single
|
||||
reviewed act, gated on the gate AND the §4 dead-zone proof? (rec: yes)
|
||||
4. **Dead-zone proof:** require a passing "every reachable ask of a flipped family
|
||||
renders" test as a precondition of any flip? (rec: yes — this is the ASK wrong=0 guard)
|
||||
5. **Served marker:** a distinct ASK surface marker (not `[verified]` / `[approximate]`)? (rec: yes)
|
||||
6. **Holdout:** validate-first probe (grounded-only on served path + no dead zone +
|
||||
GSM8K seal byte-identical) before the gate goes on? (rec: yes)
|
||||
|
||||
No served-surface code until this brief is reviewed.
|
||||
183
docs/analysis/verified-serving-wiring-scoping-2026-06-08.md
Normal file
183
docs/analysis/verified-serving-wiring-scoping-2026-06-08.md
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
# VERIFIED serving-wiring — `verified_serving_enabled` — scoping brief
|
||||
|
||||
**Date:** 2026-06-08 · **Status:** scoping (NO CODE) · **HOLD for review** ·
|
||||
**Branch:** `docs/serving-integration-scoping`
|
||||
|
||||
**What this brief is.** The scope for the served-surface decision the VERIFIED lane
|
||||
deferred. The off-serving VERIFIED lane is complete and on main:
|
||||
|
||||
```text
|
||||
P1-A the VERIFIED contract (verified_contract.py — two independent reads converge)
|
||||
P1-B gold-setup-backed producer (evals/constraint_oracle/verified_producer.py — OFF-SERVING)
|
||||
P1-C bound_slots_digest (a separable, load-bearing proof obligation)
|
||||
```
|
||||
|
||||
P1-B verifies 7/13 real R2 problems with wrong=0 — but it is **gold-setup-backed**, so
|
||||
it is structurally off-serving: the independent read is the INV-25 hand-authored gold
|
||||
SETUP, which is not available at serving time. This brief scopes what a *serving-time*
|
||||
VERIFIED would require — and why it cannot reuse P1-B. **No code here.**
|
||||
|
||||
> Companions: [[ask-serving-integration-scoping-2026-06-08]] (the ASK half of the same
|
||||
> "where off-serving stops" line), [[VERIFIED-canonical-comparison-scoping-2026-06-06]]
|
||||
> (the validate-first probe that already KILLED the naive fold-reader producer),
|
||||
> [[stage2-epistemic-disclosure-bus-verified-v1-scoping-2026-06-08]] (Doc 1).
|
||||
|
||||
---
|
||||
|
||||
## 1. The seam (grounded) — and why it is still inert by design
|
||||
|
||||
`generate/derivation/verify.py::_canonically_verified(verified, problem_text, policy)`
|
||||
is the ADR-0206 §5 VERIFIED gate — *the only thing that may license a math answer past
|
||||
gold* (resolve a disagreement STRICT refuses). It **returns `None` today**, so the
|
||||
widening is structurally inert: disagreement refuses regardless of `policy`, preserving
|
||||
absolute `wrong == 0`. Its own docstring states the bright line this brief must honour:
|
||||
|
||||
> "A reliability *license* (statistical) must NEVER substitute here: math serving is
|
||||
> absolute-wrong=0, not disclosed like the cognition path."
|
||||
|
||||
So VERIFIED serving = replacing that `return None` with a producer that returns a
|
||||
derivation **only when it is canonically VERIFIED** (proven correct, not merely sound),
|
||||
behind a kill switch, proven on a holdout. Everything below is the eligibility bar for
|
||||
that producer.
|
||||
|
||||
---
|
||||
|
||||
## 2. The five things this decision must pin
|
||||
|
||||
```text
|
||||
1. the gold-free independent-reader requirement — the crux
|
||||
2. verified_serving_enabled — the kill switch
|
||||
3. holdout gates — validate-first, no quiet widening
|
||||
4. proof-producer eligibility — what may plug into _canonically_verified
|
||||
5. the explicit ban on eval-gold-backed serving — why P1-B cannot serve
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. The gold-free independent-reader requirement (the crux)
|
||||
|
||||
VERIFIED means **two independent reads that converge on one canonical structure**
|
||||
(P1-A): a faithful solve of a *wrong read* is caught because the independent read
|
||||
disagrees. P1-B's two reads are `read_constraint_problem` (the engine reader) vs the
|
||||
**gold-authored setup**. At serving time there is no gold. So serving-time VERIFIED
|
||||
needs a **second, gold-free reader** whose disagreement is the safety mechanism.
|
||||
|
||||
The hard constraints, from the killed-probe doc and CLAUDE.md:
|
||||
|
||||
- **Independence must be in the READING, not the solving.** Back-substitution catches
|
||||
solve-errors, never read-errors. Two solvers over one reading is *fake* independence.
|
||||
The second reader must parse the problem into the same `ConstraintProblem` structure
|
||||
by a **genuinely different route** (different lineage, asserted by the
|
||||
`SAME_READER_LINEAGE` firewall / INV-27 reader-disjointness), and converge on the
|
||||
same canonical signature.
|
||||
- **No eval gold in the read** (§7). The second reader may not consult, hash, or be
|
||||
derived from any gold artifact — not the answer, not the setup. If it needs gold to
|
||||
read, it is P1-B, and P1-B does not serve.
|
||||
- **Conservative refuse-on-doubt carries wrong=0.** The second reader, like the first,
|
||||
refuses when uncertain. VERIFIED fires only on *agreement of two confident,
|
||||
independent reads*; any refusal or disagreement → STRICT refuses (today's behaviour).
|
||||
|
||||
**Eligibility, stated as a gate:** a serving-time VERIFIED producer is eligible **only
|
||||
if** it exhibits two reads with (a) distinct, firewall-asserted lineages, (b) neither
|
||||
read derived from gold, (c) convergence on one canonical `ConstraintProblem` signature,
|
||||
(d) back-substitution + boundary-clear + bound-slots (the P1-A/P1-C obligations), and
|
||||
(e) refuse-preferring failure. Absent any one, it stays `None`.
|
||||
|
||||
Whether such a second R2 reader *exists yet* is the open empirical question — the killed
|
||||
fold-reader probe is the cautionary precedent: complementary readers were ~98% wrong on
|
||||
the refused set. **This brief does not assume one exists; it sets the bar a candidate
|
||||
must clear, validate-first, before any wiring.**
|
||||
|
||||
---
|
||||
|
||||
## 4. `verified_serving_enabled` — the kill switch
|
||||
|
||||
Add `verified_serving_enabled: bool = False` to `core/config.py` (sibling of
|
||||
`estimation_enabled`). Default **off**. When off, `_canonically_verified` returns `None`
|
||||
unconditionally (today's inert state) regardless of any producer being present. The
|
||||
switch is the single audited place that lights the seam, and it stays off until §5.
|
||||
|
||||
---
|
||||
|
||||
## 5. Holdout gates — validate-first, no quiet widening
|
||||
|
||||
VERIFIED may not widen live until proven on a **held-out** set it never trained or
|
||||
tuned on (INV-25 discipline; the killed-probe doc's validate-first rule). The gate, in
|
||||
order:
|
||||
|
||||
1. **Holdout probe (off-serving):** run the candidate producer over a held-out R2 set
|
||||
with gold answers withheld; require **wrong == 0** on everything it marks VERIFIED,
|
||||
and that everything it cannot independently verify it *refuses* (over-refusal is
|
||||
acceptable; one wrong is disqualifying).
|
||||
2. **Seal byte-identity:** the GSM8K candidate-graph serving seal (pinned SHAs) stays
|
||||
byte-identical — VERIFIED widens a *different* surface (R2 constraint answers), it
|
||||
must not perturb the sealed lane.
|
||||
3. **Disagreement-still-refuses:** a test proving that with the producer wired and the
|
||||
switch ON, a faithful solve of a deliberately wrong read still refuses (the P1-A
|
||||
poison test, now on the served path).
|
||||
4. Only then does `verified_serving_enabled` go on, **one surface at a time** (R2
|
||||
first; R4 second), each with its own holdout pass.
|
||||
|
||||
---
|
||||
|
||||
## 6. The served surface — a distinct `[verified]` disclosure
|
||||
|
||||
When wired and enabled, a VERIFIED R2 answer is served under its **own** disclosure
|
||||
claim/marker — the locked decision from Doc 1's review: *VERIFIED gets a distinct
|
||||
served disclosure mode + `[verified]` prefix, NEVER reused from `[approximate]`*.
|
||||
VERIFIED is a *license* claim (more licensed than gold-strict), not a *speculation*
|
||||
claim. The route to it is the only sanctioned one: `disclosure_for_verification(result)`
|
||||
→ `(EpistemicState.VERIFIED, DisclosureClaim.VERIFIED)` → `ServedDisposition.DISCLOSE`
|
||||
under the P0-3 guard (which already degrades an unbacked VERIFIED claim to COMMIT).
|
||||
|
||||
---
|
||||
|
||||
## 7. The explicit ban on eval-gold-backed serving (the load-bearing non-claim)
|
||||
|
||||
**P1-B must never become a serving producer.** It is gold-setup-backed: its independent
|
||||
read is the hand-authored gold SETUP. Wiring it into `_canonically_verified` would mean
|
||||
the engine "verifies" by consulting the answer key's structure — circular, and a
|
||||
wrong=0 fiction (it would serve VERIFIED on exactly the problems it was handed the setup
|
||||
for). The ban, stated so a test can enforce it:
|
||||
|
||||
- `_canonically_verified` (and any serving producer behind it) may import **nothing**
|
||||
from `evals/` — no gold loader, no `gold_to_problem`, no `r2_gold`. An AST/import test
|
||||
asserts the serving path is gold-free, the mirror of the off-serving AST tests.
|
||||
- P1-B stays in `evals/` precisely so this boundary is structural: `evals → core` is the
|
||||
allowed direction; `core/generate serving → evals` is forbidden.
|
||||
- "Verified on a holdout" (§5) is **not** the same as "verified by gold at serving" —
|
||||
the holdout *measures* a gold-free producer; it never *feeds* gold into one.
|
||||
|
||||
This is the VERIFIED analogue of the GSM8K "candidate-graph owns the metric; no bridge
|
||||
re-enables without sealed/independent wrong=0" discipline.
|
||||
|
||||
---
|
||||
|
||||
## 8. What this is NOT
|
||||
|
||||
- **Not** a claim that a serving-time VERIFIED producer exists — §3 sets the eligibility
|
||||
bar; whether an R2 second reader clears it is an open, validate-first question.
|
||||
- **Not** a reliability/statistical license at the math seam — that path is the
|
||||
cognition `[approximate]` disclosure; math serving is absolute wrong=0 (§1 bright line).
|
||||
- **Not** a GSM8K-seal move — VERIFIED widens R2 constraint answers, seal stays
|
||||
byte-identical (§5.2).
|
||||
- **Not** P1-B promoted to serving — explicitly banned (§7); P1-B stays off-serving.
|
||||
|
||||
---
|
||||
|
||||
## 9. The questions for the ruling
|
||||
|
||||
1. **Eligibility bar:** adopt §3 (a)–(e) as the gate any serving VERIFIED producer must
|
||||
clear — gold-free second reader, independence-in-the-reading, refuse-preferring? (rec: yes)
|
||||
2. **Gate:** add `verified_serving_enabled = False` (sibling of `estimation_enabled`),
|
||||
`_canonically_verified` stays `None` while off? (rec: yes)
|
||||
3. **Holdout:** require the §5 validate-first sequence (wrong=0 on holdout + seal
|
||||
byte-identity + disagreement-still-refuses) before the switch, R2-first? (rec: yes)
|
||||
4. **Gold ban:** enforce the §7 import ban (serving path imports nothing from `evals/`)
|
||||
with a test, keeping P1-B off-serving forever? (rec: yes)
|
||||
5. **Surface:** serve VERIFIED under its distinct `[verified]` marker via
|
||||
`disclosure_for_verification` only? (rec: yes — the locked Doc 1 decision)
|
||||
|
||||
No served-surface code until this brief is reviewed. Pairs with
|
||||
[[ask-serving-integration-scoping-2026-06-08]] — together they draw the full line where
|
||||
off-serving stops.
|
||||
Loading…
Reference in a new issue