core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Merge pull request #671 from AssetOverflow/docs/ask-serving-integration-scoping

Browse source 
docs(epistemic): scope ASK serving integration and carve-out retirement
This commit is contained in:
Shay 2026-06-09 09:38:04 -07:00 committed by GitHub
commit ffbaf0f27c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 144 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

144
docs/analysis/ask-serving-integration-scoping-2026-06-09.md Normal file
View file

@ -0,0 +1,144 @@
# ASK Serving Integration Scoping — ask_serving_enabled, QUESTION_NEEDED, and Carve-Out Retirement
## 1. Current State
Currently, the ASK capability exists strictly off-serving.
- **Residue Capture (Q1-B):** The `core/epistemic_disclosure/limitation.py` module captures `LimitationAssessment` with typed ASK residue including `MissingSlot` and `grounded_terms`. Specifically, `missing_total_count` and `missing_weighted_total` are classified as ask-oriented inside the disclosure layer.
- **Carve-Out Safety (Q1-B):** The transitional carve-out constant `Q1B_ASK_CARVE_OUT` is defined in `core/epistemic_disclosure/limitation.py`. The registry in `core/comprehension_attempt/failure_family.py` preserves `proposal_allowed=True` for these carve-out families. This guarantees that the proposal-pile signal remains active and uninterrupted until served ASK is fully wired and verified.
- **Grounded Rendering (Q1-C):** The `core/epistemic_questions/render.py` module safely renders `EpistemicQuestion` structures. It enforces structural rendering, ensuring no ungrounded problem-entity names or internal `snake_case` tokens escape to the user. Multi-slot, unmapped, or unsafe cases fall back to `question_unrenderable`.
- **Off-serving Delivery (Q1-D):** The delivery infrastructure in `core/epistemic_questions/delivery.py` defines `DeliveredQuestion` and ships the `Terminal.QUESTION_NEEDED` tenant. The resulting off-serving artifacts are written directly to the `teaching/questions/` sink. No served/user-facing surfaces are exposed.
- **Default-Dark Gate (Helper):** The helper function `ask_serving_enabled` is implemented in `core/epistemic_questions/serving_gate.py`. It operates in a fail-closed, default-dark manner. If the config field `ask_serving_enabled` is absent, it returns `False`. An explicit, truthy configuration is required to allow served ASK.
- **Integration gaps:** Currently, `generate/contemplation/pass_manager.py` does not emit served ASK, `chat/runtime.py` does not expose ASK, and the `Q1B_ASK_CARVE_OUT` remains active (unretired).
## 2. Non-Negotiable Boundary
Before ASK can be delivered to any user-facing surface, the following boundaries must be strictly enforced:
- **Strict Gate Guard:** No served question may be shown without an explicit, active `ask_serving_enabled` configuration check.
- **Fail-Closed Default:** The `ask_serving_enabled` helper must default to `False`. A missing or `None` config attribute must evaluate to `False`.
- **Prose Encapsulation:** The serving layer must not construct or mutate question prose directly. It may only consume pre-rendered `EpistemicQuestion` / `DeliveredQuestion` structures produced by Q1-C/Q1-D.
- **No Contentless Delivery:** An unrenderable ASK must never be promoted to `QUESTION_NEEDED`. Contentless `QUESTION_NEEDED` outcomes are strictly forbidden.
- **Carve-Out Preservation:** The `Q1B_ASK_CARVE_OUT` must remain active and unchanged until served ASK is fully verified. No proposal signal may be lost before a served `QUESTION_NEEDED` is verified live.
- **Sink Distinction:** The `question_only` (teaching/questions) sink must remain logically and physically distinct from the `proposal_only` (teaching/proposals) sink.
- **Zero Impact on Claims:** No benchmark, `CLAIMS.md`, or performance metrics may be modified by this scoping documentation.
## 3. Proposed Served Gate: ask_serving_enabled
The serving gate helper exists under `core/epistemic_questions/serving_gate.py`. This document scoping defines how future served-surface code must interact with the gate:
- **Helper Invariant:** Future code in the served-surface layer (e.g., `chat/runtime.py`) must verify `ask_serving_enabled(config)` before delivering any `QUESTION_NEEDED` response to a user.
- **Off-Serving Isolation:** The gate controls served (user-visible) output only. It must not disable or interfere with off-serving artifacts written to the `teaching/questions/` directory.
### Serving Gate Policy
| Config State | ask_serving_enabled(...) | Served ASK Allowed? |
| --- | --- | --- |
| Missing field | `False` | No |
| `None` / Default config | `False` | No |
| Explicit `False` | `False` | No |
| Explicit `True` | `True` | Only if rendering and delivery obligations pass |
## 4. pass_manager Integration Boundary
This section defines the future integration interface for `generate/contemplation/pass_manager.py`. This is scoping only; no execution is performed here.
1. **Evaluation:** The refusal/comprehension flow yields a `ComprehensionAttempt`.
2. **Assessment:** A `LimitationAssessment` is derived from the attempt.
3. **Resolution Action:** If `resolution_action == "ask_question"`:
- The pipeline invokes `deliver_ask(assessment)`.
- `deliver_ask` calls `render_question` exactly once.
- If the question is renderable, the final `DeliveryOutcome` terminal becomes `QUESTION_NEEDED`.
- If the question is unrenderable, the outcome falls back to the standing disposition (e.g., proposal or refusal).
4. **Gate Enforced downstream:** `generate/contemplation/pass_manager.py` may produce/record ASK delivery outcomes later, but user-visible exposure remains gated downstream by `ask_serving_enabled` (e.g., in `chat/runtime.py`).
- **Constraints:**
- `pass_manager` must not contain prose templates or formatting rules.
- `pass_manager` must not construct `DeliveredQuestion` manually (it must delegate to `deliver_ask`).
- `pass_manager` must never bypass `deliver_ask`.
- `pass_manager` must never emit a contentless `QUESTION_NEEDED`.
## 5. Served Behavior Matrix
| Assessment / Rendering | Gate Disabled | Gate Enabled |
| --- | --- | --- |
| Renderable `ask_question` | No served ASK; existing refusal/proposal behavior preserved | Served `QUESTION_NEEDED` is allowed |
| Unrenderable `ask_question` | Standing fallback; no `QUESTION_NEEDED` | Standing fallback; no `QUESTION_NEEDED` |
| Non-ASK limitation | Unaffected | Unaffected |
| `missing_total_count` / `missing_weighted_total` (carve-out) | Proposal signal preserved | Served ASK only after carve-out retirement proof |
| Multi-slot ASK | Unrenderable fallback | Unrenderable fallback |
*Note: Gate enabled is a necessary but not sufficient condition for serving. The renderable checks and delivery invariants must also pass.*
## 6. Q1B_ASK_CARVE_OUT Retirement Conditions
The transitional carve-out constant `Q1B_ASK_CARVE_OUT` can only be retired and removed from `core/epistemic_disclosure/limitation.py` when the following milestones are met and verified:
1. The `ask_serving_enabled` helper is tested and confirmed default-dark.
2. The `pass_manager.py` ASK integration is fully implemented behind the gate.
3. Served `QUESTION_NEEDED` terminal behavior is successfully tested using renderable `EpistemicQuestion` instances.
4. Unrenderable ASK paths are verified to fall back correctly, never emitting `QUESTION_NEEDED`.
5. Carve-out keys `missing_total_count` and `missing_weighted_total` are proven to yield safe renderable questions (or safe standing fallbacks).
6. A dedicated no-question/no-proposal dead-zone validation test is introduced and passes.
7. The registry in `core/comprehension_attempt/failure_family.py` flips `proposal_allowed=False` for the carve-out families without dropping signal.
8. The `teaching/questions/` (`question_only`) sink remains physically distinct from the `teaching/proposals/` (`proposal_only`) sink.
9. Smoke, contemplation, proposal, and disclosure test suites remain fully green.
*Until all 9 conditions are met, the registry's `proposal_allowed=True` setting for these families must remain.*
## 7. No-Dead-Zone Proof Obligation
### The Dead-Zone Hazard
If `proposal_allowed` is flipped to `False` on a failure family before ASK serving is enabled/renderable, an input falling into that family would yield no proposal signal AND no served question. This results in a silent loss of capability (a dead zone), violating the `wrong=0` refusal-first discipline.
### Required Proof Before Retirement
Before the carve-out is retired, tests must prove that for each carve-out family:
- If `proposal_allowed` is removed, either a served `QUESTION_NEEDED` is emitted safely, or the standing fallback preserves an admissible signal.
- The test suite must assert failure if an evaluation result yields neither a proposal nor a valid served question.
### Suggested Test Names
- `test_ask_serving_disabled_preserves_existing_proposal_signal`
- `test_carveout_retirement_has_no_question_no_proposal_dead_zone`
- `test_unrenderable_ask_never_emits_question_needed`
- `test_pass_manager_uses_deliver_ask_not_direct_rendering`
- `test_question_only_not_proposal_only`
## 8. Required Future Tests Before Wiring
The implementation PR must include tests asserting the following behaviors:
- **Gate Default:** `ask_serving_enabled` defaults to `False`.
- **Config Completeness:** A missing config field evaluates to `False`.
- **Opt-in Activation:** An explicit `True` configuration is required to allow served ASK.
- **Gate Override:** If the gate is disabled, no served ASK is emitted even if `deliver_ask` produces a `QUESTION_NEEDED` outcome.
- **Rendering Obligation:** If the gate is enabled, the output still requires a renderable `EpistemicQuestion`.
- **Unrenderable Fallback:** Unrenderable ASK instances are never served.
- **Prose Isolation:** `pass_manager` does not construct prose templates or strings.
- **Delegated Delivery:** `pass_manager` delegates entirely to `deliver_ask` and does not call `render_question` directly.
- **No Empty Terminals:** No contentless `QUESTION_NEEDED` outcomes are permitted.
- **Sink Safety:** The `question_only` artifact is written to the correct, distinct location.
- **Carve-out Lock:** `Q1B_ASK_CARVE_OUT` is preserved until both the gate and the dead-zone proofs are active.
- **Registry Guard:** Flipped registries are blocked until all retirement conditions are met.
## 9. Non-Claims
This scoping document establishes architectural boundaries only:
- **No Implementation:** It does not implement served ASK.
- **No Wiring:** It does not wire `generate/contemplation/pass_manager.py` or `chat/runtime.py`.
- **No Retirement:** It does not retire `Q1B_ASK_CARVE_OUT` or flip any registry `proposal_allowed` flags.
- **No Metric Changes:** It does not alter GSM8K benchmark claims or refusal behaviors.
- **No General Intelligence:** It does not make ASK generally intelligent; it simply bounds the served-surface gate.
## 10. Recommended Next Slice
To safely approach the implementation, the next sequential slices are recommended:
### Slice 1: Configuration Definition (No Wiring)
- Add a concrete configuration field `RuntimeConfig.ask_serving_enabled: bool = False` (if not already present).
- Ensure the helper `ask_serving_enabled(...)` references this configuration.
- Write configuration tests verifying the default and override values.
- Do not wire `pass_manager` or modify runtime loops.
### Slice 2: pass_manager Integration
- Wire `pass_manager` to call `deliver_ask` without constructing prose; any user-visible ASK exposure remains behind `ask_serving_enabled`; preserve `Q1B_ASK_CARVE_OUT`.