core/docs/analysis/ask-serving-integration-scoping-2026-06-08.md
Shay 28cb42d0dd
docs(epistemic): scope ASK and VERIFIED serving gates
Accepts the ASK + VERIFIED serving-integration scoping briefs.

No served-surface code is introduced. ASK is identified as the next gated serving slice; VERIFIED remains blocked on a gold-free second reader and explicit eval-gold serving ban.
2026-06-08 22:22:35 -07:00

10 KiB

ASK serving-integration — ask_serving_enabled — scoping brief

Date: 2026-06-08 · Status: scoping (NO CODE) · HOLD for review · Branch: docs/serving-integration-scoping

What this brief is. The scope for the one served-surface decision the ASK lane deferred. The off-serving ASK lane is complete and on main:

Q1-B  typed residue + ask classification         (LimitationAssessment.missing_slots)
Q1-C  grounded-only rendering                     (render_question → EpistemicQuestion)
Q1-D  off-serving delivery artifact               (deliver_ask → QUESTION_NEEDED + sink)

deliver_ask / emit_question exist and are tested, but nothing calls them from a live path — they are produced-but-not-emitted, exactly as P1-B's verified producer is built-but-not-served. This brief scopes the step that closes that gap: letting a QUESTION_NEEDED actually reach a user, behind a named gate. No code here — this is the decision surface for review.

Companions: q1-d-ask-bus-delivery-scoping-2026-06-08 (the off-serving rung, §7 defers exactly this), verified-serving-wiring-scoping-2026-06-08 (the VERIFIED half of the same "where off-serving stops" line). Design of record: the session doc §5 / §1.5.8 (the disclosure bus).


1. The five things this decision must pin

The steer named five; each is grounded against shipped code below.

1. ask_serving_enabled              — the kill switch
2. the pass_manager integration point  — where deliver_ask is actually called
3. the Q1B_ASK_CARVE_OUT retirement gate + registry flip
4. served-surface behaviour for a QUESTION_NEEDED reaching a user
5. the no-question/no-proposal dead-zone proof

2. The integration point (grounded)

generate/contemplation/pass_manager.py::contemplate(text, *, proposal_root=...) is the live contemplation entry. When every attempt is refused it calls _classify_all_refused(text, attempts, findings, proposal_root), which is where emit_proposal fires for a proposal_allowed family and terminates PROPOSAL_EMITTED. That is the exact analogous site for ASK: when a refused attempt's family maps to ask_question (via assess_from_attempt), call deliver_ask and — on a renderable result — emit_question to the sink and terminate QUESTION_NEEDED; on the D2 fallback, fall through to today's proposal/refuse path.

contemplate() is called from chat/runtime.py at three sites (827, 901, 1364) — so the contemplation Terminal is already on the served path. The served question text travels the same way PROPOSAL_EMITTED already does.

Two-layer split (the recommendation), mirroring VERIFIED:

  • Layer A — pass-manager emission (off-serving still). contemplate emits QUESTION_NEEDED + writes the teaching/questions/ artifact, exactly as it emits PROPOSAL_EMITTED today. This changes the contemplation terminal but reaches no user — the teaching loop is off-serving. This layer is buildable behind no gate (it only adds a terminal the pass can reach), but see §3: it interacts with the carve-out and so should still wait for the gate, to avoid double-emission churn.
  • Layer B — served delivery (the gated surface). chat/runtime.py, gated by ask_serving_enabled, renders the DeliveredQuestion.question.text to the user as the served response when the terminal is QUESTION_NEEDED. This is the only layer that actually asks the user anything.

Decision to confirm: do Layer A and Layer B land together behind one gate, or does Layer A land first (pass emits, nothing served) and Layer B follow? Recommend together, behind ask_serving_enabled — Layer A alone creates the double-emission state in §3 with no compensating benefit.


3. The kill switch + the carve-out retirement gate (the coupled core)

3.1 ask_serving_enabled

Add ask_serving_enabled: bool = False to core/config.py, the sibling of the existing estimation_enabled = False kill-switch pattern. Default off: the served question path is dark until deliberately enabled, holdout-gated (§5).

3.2 Why the carve-out flip is coupled to the gate (not to Q1-D)

Q1-B introduced Q1B_ASK_CARVE_OUT for missing_total_count / missing_weighted_total: the disclosure layer classifies them ask_question, but the shipped REGISTRY keeps proposal_allowed = True so the proposal pile keeps working. The carve-out's retirement condition is written into limitation.py: "Once ASK is serving, flip proposal_allowed = False on these two families, drop the carve-out set, amend the test." The operative word is serving — not "delivery exists" (Q1-D already shipped that off-serving). So:

carve-out retires  ⟺  ask_serving_enabled is the ruling
                       AND a QUESTION_NEEDED is actually served for these families
                       AND the §4 dead-zone proof holds

Until then, proposal_allowed stays True. During the gate's "off" state both signals coexist (the off-serving question artifact + the proposal) — intentional, no loss.

3.3 The flip, as a single reviewed act

When ask_serving_enabled is turned on for these families, in one change:

  1. proposal_allowed = False for missing_total_count / missing_weighted_total.
  2. Drop them from Q1B_ASK_CARVE_OUT (empty the set, or remove the constant).
  3. Amend the proposal_allowed invariant test + the carve-out test.
  4. The §4 dead-zone proof test must already be green.

This is the "conscious act, not a silent re-key" the carve-out was built to force.


4. The no-question/no-proposal dead-zone proof (the wrong=0-adjacent guard)

The hazard. The flip removes the proposal signal for these families. If, for some input class, the family classifies ask_question BUT the question is unrenderable (D2), AND the proposal is now off, the family would terminate NO_PROGRESS with no served question and no proposal — a dead zone where a user-resolvable gap produces nothing. That is the ASK-side wrong=0 hazard: not a false answer, but a silent loss of a capability that previously at least proposed.

The proof obligation (before any flip). For every family being flipped, prove that no input class lands in the dead zone — i.e. for every reachable assessment of that family, the question renders (so QUESTION_NEEDED is served), OR the proposal is still on. Concretely:

  • The missing_* families have pinned single slots in _FAMILY_TO_MISSING_SLOTS with mapped types (count_int / measured_unit_int) → they always render today, so the dead zone is currently empty. The proof must show this is structural, not incidental: a test that asserts deliver_ask returns QUESTION_NEEDED (never a fallback) for every reachable assessment of the flipped families.
  • If any future residue change could make one of these unrenderable (multi-slot, unmapped type), the flip must be blocked for that family until either the renderer covers it or the proposal stays on.

The rule: proposal_allowed may flip True → False for a family only if a test proves every reachable ask of that family renders. The dead-zone proof is a precondition of the flip, enforced like the D2 guard (it must fail if a fallback path is reachable for a flipped family).


5. Served-surface behaviour + holdout gating

5.1 What a served QUESTION_NEEDED looks like

When ask_serving_enabled and the terminal is QUESTION_NEEDED, the served surface returns the DeliveredQuestion.question.text (the grounded-only rendered question) as the response — distinct from a committed answer, an [approximate] disclosure, or a refusal. It is an intake request: the disposition is ServedDisposition.ASK (already mapped in disposition.py). The question names nothing ungrounded (Q1-C guarantee), so it cannot leak a fabricated entity even on the served path.

Open sub-decision: the served prefix/marker. VERIFIED gets a distinct [verified] prefix; APPROXIMATE gets [approximate]. ASK should get its own surface marker (a question is neither). Recommend a distinct, tested marker; pin the exact string at build, not here.

5.2 Holdout gate (no quiet widening)

Like VERIFIED, ASK serving must be proven on a holdout before it widens live: a validate-first probe over a held-out set confirming (a) served questions are grounded-only (no fabrication escapes on the served path), (b) no family in the flip set hits the dead zone, (c) the GSM8K serving seal is byte-identical (ASK is off-the-metric — it asks, it does not answer — but the probe proves it). Only then does ask_serving_enabled go on, one surface at a time.


6. What this is NOT

  • Not a dialogue manager / multi-turn state machine — one grounded question, then the existing flow; the answer round-trip is Q2 (AnswerBinding, a separate batch).
  • Not a re-render — the served path emits the Q1-D DeliveredQuestion.question.text verbatim; no second prose surface.
  • Not a GSM8K-metric move — ASK asks, it never answers; the pinned SHAs and CLAIMS.md are untouched. The holdout probe proves it.
  • Not the carve-out flip yet — the flip is the terminal act of this decision, gated on ask_serving_enabled + the §4 dead-zone proof, not on Q1-D.

7. The questions for the ruling

  1. Gate: add ask_serving_enabled = False (sibling of estimation_enabled)? (rec: yes)
  2. Layering: land pass-emission (Layer A) and served delivery (Layer B) together behind the one gate? (rec: yes — Layer A alone only adds double-emission churn)
  3. Carve-out flip: retire Q1B_ASK_CARVE_OUT + flip proposal_allowed as a single reviewed act, gated on the gate AND the §4 dead-zone proof? (rec: yes)
  4. Dead-zone proof: require a passing "every reachable ask of a flipped family renders" test as a precondition of any flip? (rec: yes — this is the ASK wrong=0 guard)
  5. Served marker: a distinct ASK surface marker (not [verified] / [approximate])? (rec: yes)
  6. Holdout: validate-first probe (grounded-only on served path + no dead zone + GSM8K seal byte-identical) before the gate goes on? (rec: yes)

No served-surface code until this brief is reviewed.