diff --git a/docs/analysis/ask-serving-integration-scoping-2026-06-09.md b/docs/analysis/ask-serving-integration-scoping-2026-06-09.md new file mode 100644 index 00000000..877f4f6c --- /dev/null +++ b/docs/analysis/ask-serving-integration-scoping-2026-06-09.md @@ -0,0 +1,144 @@ +# ASK Serving Integration Scoping — ask_serving_enabled, QUESTION_NEEDED, and Carve-Out Retirement + +## 1. Current State + +Currently, the ASK capability exists strictly off-serving. + +- **Residue Capture (Q1-B):** The `core/epistemic_disclosure/limitation.py` module captures `LimitationAssessment` with typed ASK residue including `MissingSlot` and `grounded_terms`. Specifically, `missing_total_count` and `missing_weighted_total` are classified as ask-oriented inside the disclosure layer. +- **Carve-Out Safety (Q1-B):** The transitional carve-out constant `Q1B_ASK_CARVE_OUT` is defined in `core/epistemic_disclosure/limitation.py`. The registry in `core/comprehension_attempt/failure_family.py` preserves `proposal_allowed=True` for these carve-out families. This guarantees that the proposal-pile signal remains active and uninterrupted until served ASK is fully wired and verified. +- **Grounded Rendering (Q1-C):** The `core/epistemic_questions/render.py` module safely renders `EpistemicQuestion` structures. It enforces structural rendering, ensuring no ungrounded problem-entity names or internal `snake_case` tokens escape to the user. Multi-slot, unmapped, or unsafe cases fall back to `question_unrenderable`. +- **Off-serving Delivery (Q1-D):** The delivery infrastructure in `core/epistemic_questions/delivery.py` defines `DeliveredQuestion` and ships the `Terminal.QUESTION_NEEDED` tenant. The resulting off-serving artifacts are written directly to the `teaching/questions/` sink. No served/user-facing surfaces are exposed. +- **Default-Dark Gate (Helper):** The helper function `ask_serving_enabled` is implemented in `core/epistemic_questions/serving_gate.py`. It operates in a fail-closed, default-dark manner. If the config field `ask_serving_enabled` is absent, it returns `False`. An explicit, truthy configuration is required to allow served ASK. +- **Integration gaps:** Currently, `generate/contemplation/pass_manager.py` does not emit served ASK, `chat/runtime.py` does not expose ASK, and the `Q1B_ASK_CARVE_OUT` remains active (unretired). + +## 2. Non-Negotiable Boundary + +Before ASK can be delivered to any user-facing surface, the following boundaries must be strictly enforced: + +- **Strict Gate Guard:** No served question may be shown without an explicit, active `ask_serving_enabled` configuration check. +- **Fail-Closed Default:** The `ask_serving_enabled` helper must default to `False`. A missing or `None` config attribute must evaluate to `False`. +- **Prose Encapsulation:** The serving layer must not construct or mutate question prose directly. It may only consume pre-rendered `EpistemicQuestion` / `DeliveredQuestion` structures produced by Q1-C/Q1-D. +- **No Contentless Delivery:** An unrenderable ASK must never be promoted to `QUESTION_NEEDED`. Contentless `QUESTION_NEEDED` outcomes are strictly forbidden. +- **Carve-Out Preservation:** The `Q1B_ASK_CARVE_OUT` must remain active and unchanged until served ASK is fully verified. No proposal signal may be lost before a served `QUESTION_NEEDED` is verified live. +- **Sink Distinction:** The `question_only` (teaching/questions) sink must remain logically and physically distinct from the `proposal_only` (teaching/proposals) sink. +- **Zero Impact on Claims:** No benchmark, `CLAIMS.md`, or performance metrics may be modified by this scoping documentation. + +## 3. Proposed Served Gate: ask_serving_enabled + +The serving gate helper exists under `core/epistemic_questions/serving_gate.py`. This document scoping defines how future served-surface code must interact with the gate: + +- **Helper Invariant:** Future code in the served-surface layer (e.g., `chat/runtime.py`) must verify `ask_serving_enabled(config)` before delivering any `QUESTION_NEEDED` response to a user. +- **Off-Serving Isolation:** The gate controls served (user-visible) output only. It must not disable or interfere with off-serving artifacts written to the `teaching/questions/` directory. + +### Serving Gate Policy + +| Config State | ask_serving_enabled(...) | Served ASK Allowed? | +| --- | --- | --- | +| Missing field | `False` | No | +| `None` / Default config | `False` | No | +| Explicit `False` | `False` | No | +| Explicit `True` | `True` | Only if rendering and delivery obligations pass | + +## 4. pass_manager Integration Boundary + +This section defines the future integration interface for `generate/contemplation/pass_manager.py`. This is scoping only; no execution is performed here. + +1. **Evaluation:** The refusal/comprehension flow yields a `ComprehensionAttempt`. +2. **Assessment:** A `LimitationAssessment` is derived from the attempt. +3. **Resolution Action:** If `resolution_action == "ask_question"`: + - The pipeline invokes `deliver_ask(assessment)`. + - `deliver_ask` calls `render_question` exactly once. + - If the question is renderable, the final `DeliveryOutcome` terminal becomes `QUESTION_NEEDED`. + - If the question is unrenderable, the outcome falls back to the standing disposition (e.g., proposal or refusal). +4. **Gate Enforced downstream:** `generate/contemplation/pass_manager.py` may produce/record ASK delivery outcomes later, but user-visible exposure remains gated downstream by `ask_serving_enabled` (e.g., in `chat/runtime.py`). + +- **Constraints:** + - `pass_manager` must not contain prose templates or formatting rules. + - `pass_manager` must not construct `DeliveredQuestion` manually (it must delegate to `deliver_ask`). + - `pass_manager` must never bypass `deliver_ask`. + - `pass_manager` must never emit a contentless `QUESTION_NEEDED`. + +## 5. Served Behavior Matrix + +| Assessment / Rendering | Gate Disabled | Gate Enabled | +| --- | --- | --- | +| Renderable `ask_question` | No served ASK; existing refusal/proposal behavior preserved | Served `QUESTION_NEEDED` is allowed | +| Unrenderable `ask_question` | Standing fallback; no `QUESTION_NEEDED` | Standing fallback; no `QUESTION_NEEDED` | +| Non-ASK limitation | Unaffected | Unaffected | +| `missing_total_count` / `missing_weighted_total` (carve-out) | Proposal signal preserved | Served ASK only after carve-out retirement proof | +| Multi-slot ASK | Unrenderable fallback | Unrenderable fallback | + +*Note: Gate enabled is a necessary but not sufficient condition for serving. The renderable checks and delivery invariants must also pass.* + +## 6. Q1B_ASK_CARVE_OUT Retirement Conditions + +The transitional carve-out constant `Q1B_ASK_CARVE_OUT` can only be retired and removed from `core/epistemic_disclosure/limitation.py` when the following milestones are met and verified: + +1. The `ask_serving_enabled` helper is tested and confirmed default-dark. +2. The `pass_manager.py` ASK integration is fully implemented behind the gate. +3. Served `QUESTION_NEEDED` terminal behavior is successfully tested using renderable `EpistemicQuestion` instances. +4. Unrenderable ASK paths are verified to fall back correctly, never emitting `QUESTION_NEEDED`. +5. Carve-out keys `missing_total_count` and `missing_weighted_total` are proven to yield safe renderable questions (or safe standing fallbacks). +6. A dedicated no-question/no-proposal dead-zone validation test is introduced and passes. +7. The registry in `core/comprehension_attempt/failure_family.py` flips `proposal_allowed=False` for the carve-out families without dropping signal. +8. The `teaching/questions/` (`question_only`) sink remains physically distinct from the `teaching/proposals/` (`proposal_only`) sink. +9. Smoke, contemplation, proposal, and disclosure test suites remain fully green. + +*Until all 9 conditions are met, the registry's `proposal_allowed=True` setting for these families must remain.* + +## 7. No-Dead-Zone Proof Obligation + +### The Dead-Zone Hazard +If `proposal_allowed` is flipped to `False` on a failure family before ASK serving is enabled/renderable, an input falling into that family would yield no proposal signal AND no served question. This results in a silent loss of capability (a dead zone), violating the `wrong=0` refusal-first discipline. + +### Required Proof Before Retirement +Before the carve-out is retired, tests must prove that for each carve-out family: +- If `proposal_allowed` is removed, either a served `QUESTION_NEEDED` is emitted safely, or the standing fallback preserves an admissible signal. +- The test suite must assert failure if an evaluation result yields neither a proposal nor a valid served question. + +### Suggested Test Names +- `test_ask_serving_disabled_preserves_existing_proposal_signal` +- `test_carveout_retirement_has_no_question_no_proposal_dead_zone` +- `test_unrenderable_ask_never_emits_question_needed` +- `test_pass_manager_uses_deliver_ask_not_direct_rendering` +- `test_question_only_not_proposal_only` + +## 8. Required Future Tests Before Wiring + +The implementation PR must include tests asserting the following behaviors: + +- **Gate Default:** `ask_serving_enabled` defaults to `False`. +- **Config Completeness:** A missing config field evaluates to `False`. +- **Opt-in Activation:** An explicit `True` configuration is required to allow served ASK. +- **Gate Override:** If the gate is disabled, no served ASK is emitted even if `deliver_ask` produces a `QUESTION_NEEDED` outcome. +- **Rendering Obligation:** If the gate is enabled, the output still requires a renderable `EpistemicQuestion`. +- **Unrenderable Fallback:** Unrenderable ASK instances are never served. +- **Prose Isolation:** `pass_manager` does not construct prose templates or strings. +- **Delegated Delivery:** `pass_manager` delegates entirely to `deliver_ask` and does not call `render_question` directly. +- **No Empty Terminals:** No contentless `QUESTION_NEEDED` outcomes are permitted. +- **Sink Safety:** The `question_only` artifact is written to the correct, distinct location. +- **Carve-out Lock:** `Q1B_ASK_CARVE_OUT` is preserved until both the gate and the dead-zone proofs are active. +- **Registry Guard:** Flipped registries are blocked until all retirement conditions are met. + +## 9. Non-Claims + +This scoping document establishes architectural boundaries only: + +- **No Implementation:** It does not implement served ASK. +- **No Wiring:** It does not wire `generate/contemplation/pass_manager.py` or `chat/runtime.py`. +- **No Retirement:** It does not retire `Q1B_ASK_CARVE_OUT` or flip any registry `proposal_allowed` flags. +- **No Metric Changes:** It does not alter GSM8K benchmark claims or refusal behaviors. +- **No General Intelligence:** It does not make ASK generally intelligent; it simply bounds the served-surface gate. + +## 10. Recommended Next Slice + +To safely approach the implementation, the next sequential slices are recommended: + +### Slice 1: Configuration Definition (No Wiring) +- Add a concrete configuration field `RuntimeConfig.ask_serving_enabled: bool = False` (if not already present). +- Ensure the helper `ask_serving_enabled(...)` references this configuration. +- Write configuration tests verifying the default and override values. +- Do not wire `pass_manager` or modify runtime loops. + +### Slice 2: pass_manager Integration +- Wire `pass_manager` to call `deliver_ask` without constructing prose; any user-visible ASK exposure remains behind `ask_serving_enabled`; preserve `Q1B_ASK_CARVE_OUT`.