core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/docs/analysis/ask-serving-integration-scoping-2026-06-09.md

11 KiB
Raw Blame History

ASK Serving Integration Scoping — ask_serving_enabled, QUESTION_NEEDED, and Carve-Out Retirement

1. Current State

Currently, the ASK capability exists strictly off-serving.

  • Residue Capture (Q1-B): The core/epistemic_disclosure/limitation.py module captures LimitationAssessment with typed ASK residue including MissingSlot and grounded_terms. Specifically, missing_total_count and missing_weighted_total are classified as ask-oriented inside the disclosure layer.
  • Carve-Out Safety (Q1-B): The transitional carve-out constant Q1B_ASK_CARVE_OUT is defined in core/epistemic_disclosure/limitation.py. The registry in core/comprehension_attempt/failure_family.py preserves proposal_allowed=True for these carve-out families. This guarantees that the proposal-pile signal remains active and uninterrupted until served ASK is fully wired and verified.
  • Grounded Rendering (Q1-C): The core/epistemic_questions/render.py module safely renders EpistemicQuestion structures. It enforces structural rendering, ensuring no ungrounded problem-entity names or internal snake_case tokens escape to the user. Multi-slot, unmapped, or unsafe cases fall back to question_unrenderable.
  • Off-serving Delivery (Q1-D): The delivery infrastructure in core/epistemic_questions/delivery.py defines DeliveredQuestion and ships the Terminal.QUESTION_NEEDED tenant. The resulting off-serving artifacts are written directly to the teaching/questions/ sink. No served/user-facing surfaces are exposed.
  • Default-Dark Gate (Helper): The helper function ask_serving_enabled is implemented in core/epistemic_questions/serving_gate.py. It operates in a fail-closed, default-dark manner. If the config field ask_serving_enabled is absent, it returns False. An explicit, truthy configuration is required to allow served ASK.
  • Integration gaps: Currently, generate/contemplation/pass_manager.py does not emit served ASK, chat/runtime.py does not expose ASK, and the Q1B_ASK_CARVE_OUT remains active (unretired).

2. Non-Negotiable Boundary

Before ASK can be delivered to any user-facing surface, the following boundaries must be strictly enforced:

  • Strict Gate Guard: No served question may be shown without an explicit, active ask_serving_enabled configuration check.
  • Fail-Closed Default: The ask_serving_enabled helper must default to False. A missing or None config attribute must evaluate to False.
  • Prose Encapsulation: The serving layer must not construct or mutate question prose directly. It may only consume pre-rendered EpistemicQuestion / DeliveredQuestion structures produced by Q1-C/Q1-D.
  • No Contentless Delivery: An unrenderable ASK must never be promoted to QUESTION_NEEDED. Contentless QUESTION_NEEDED outcomes are strictly forbidden.
  • Carve-Out Preservation: The Q1B_ASK_CARVE_OUT must remain active and unchanged until served ASK is fully verified. No proposal signal may be lost before a served QUESTION_NEEDED is verified live.
  • Sink Distinction: The question_only (teaching/questions) sink must remain logically and physically distinct from the proposal_only (teaching/proposals) sink.
  • Zero Impact on Claims: No benchmark, CLAIMS.md, or performance metrics may be modified by this scoping documentation.

3. Proposed Served Gate: ask_serving_enabled

The serving gate helper exists under core/epistemic_questions/serving_gate.py. This document scoping defines how future served-surface code must interact with the gate:

  • Helper Invariant: Future code in the served-surface layer (e.g., chat/runtime.py) must verify ask_serving_enabled(config) before delivering any QUESTION_NEEDED response to a user.
  • Off-Serving Isolation: The gate controls served (user-visible) output only. It must not disable or interfere with off-serving artifacts written to the teaching/questions/ directory.

Serving Gate Policy

Config State ask_serving_enabled(...) Served ASK Allowed?
Missing field False No
None / Default config False No
Explicit False False No
Explicit True True Only if rendering and delivery obligations pass

4. pass_manager Integration Boundary

This section defines the future integration interface for generate/contemplation/pass_manager.py. This is scoping only; no execution is performed here.

  1. Evaluation: The refusal/comprehension flow yields a ComprehensionAttempt.
  2. Assessment: A LimitationAssessment is derived from the attempt.
  3. Resolution Action: If resolution_action == "ask_question":
    • The pipeline invokes deliver_ask(assessment).
    • deliver_ask calls render_question exactly once.
    • If the question is renderable, the final DeliveryOutcome terminal becomes QUESTION_NEEDED.
    • If the question is unrenderable, the outcome falls back to the standing disposition (e.g., proposal or refusal).
  4. Gate Enforced downstream: generate/contemplation/pass_manager.py may produce/record ASK delivery outcomes later, but user-visible exposure remains gated downstream by ask_serving_enabled (e.g., in chat/runtime.py).
  • Constraints:
    • pass_manager must not contain prose templates or formatting rules.
    • pass_manager must not construct DeliveredQuestion manually (it must delegate to deliver_ask).
    • pass_manager must never bypass deliver_ask.
    • pass_manager must never emit a contentless QUESTION_NEEDED.

5. Served Behavior Matrix

Assessment / Rendering Gate Disabled Gate Enabled
Renderable ask_question No served ASK; existing refusal/proposal behavior preserved Served QUESTION_NEEDED is allowed
Unrenderable ask_question Standing fallback; no QUESTION_NEEDED Standing fallback; no QUESTION_NEEDED
Non-ASK limitation Unaffected Unaffected
missing_total_count / missing_weighted_total (carve-out) Proposal signal preserved Served ASK only after carve-out retirement proof
Multi-slot ASK Unrenderable fallback Unrenderable fallback

Note: Gate enabled is a necessary but not sufficient condition for serving. The renderable checks and delivery invariants must also pass.

6. Q1B_ASK_CARVE_OUT Retirement Conditions

The transitional carve-out constant Q1B_ASK_CARVE_OUT can only be retired and removed from core/epistemic_disclosure/limitation.py when the following milestones are met and verified:

  1. The ask_serving_enabled helper is tested and confirmed default-dark.
  2. The pass_manager.py ASK integration is fully implemented behind the gate.
  3. Served QUESTION_NEEDED terminal behavior is successfully tested using renderable EpistemicQuestion instances.
  4. Unrenderable ASK paths are verified to fall back correctly, never emitting QUESTION_NEEDED.
  5. Carve-out keys missing_total_count and missing_weighted_total are proven to yield safe renderable questions (or safe standing fallbacks).
  6. A dedicated no-question/no-proposal dead-zone validation test is introduced and passes.
  7. The registry in core/comprehension_attempt/failure_family.py flips proposal_allowed=False for the carve-out families without dropping signal.
  8. The teaching/questions/ (question_only) sink remains physically distinct from the teaching/proposals/ (proposal_only) sink.
  9. Smoke, contemplation, proposal, and disclosure test suites remain fully green.

Until all 9 conditions are met, the registry's proposal_allowed=True setting for these families must remain.

7. No-Dead-Zone Proof Obligation

The Dead-Zone Hazard

If proposal_allowed is flipped to False on a failure family before ASK serving is enabled/renderable, an input falling into that family would yield no proposal signal AND no served question. This results in a silent loss of capability (a dead zone), violating the wrong=0 refusal-first discipline.

Required Proof Before Retirement

Before the carve-out is retired, tests must prove that for each carve-out family:

  • If proposal_allowed is removed, either a served QUESTION_NEEDED is emitted safely, or the standing fallback preserves an admissible signal.
  • The test suite must assert failure if an evaluation result yields neither a proposal nor a valid served question.

Suggested Test Names

  • test_ask_serving_disabled_preserves_existing_proposal_signal
  • test_carveout_retirement_has_no_question_no_proposal_dead_zone
  • test_unrenderable_ask_never_emits_question_needed
  • test_pass_manager_uses_deliver_ask_not_direct_rendering
  • test_question_only_not_proposal_only

8. Required Future Tests Before Wiring

The implementation PR must include tests asserting the following behaviors:

  • Gate Default: ask_serving_enabled defaults to False.
  • Config Completeness: A missing config field evaluates to False.
  • Opt-in Activation: An explicit True configuration is required to allow served ASK.
  • Gate Override: If the gate is disabled, no served ASK is emitted even if deliver_ask produces a QUESTION_NEEDED outcome.
  • Rendering Obligation: If the gate is enabled, the output still requires a renderable EpistemicQuestion.
  • Unrenderable Fallback: Unrenderable ASK instances are never served.
  • Prose Isolation: pass_manager does not construct prose templates or strings.
  • Delegated Delivery: pass_manager delegates entirely to deliver_ask and does not call render_question directly.
  • No Empty Terminals: No contentless QUESTION_NEEDED outcomes are permitted.
  • Sink Safety: The question_only artifact is written to the correct, distinct location.
  • Carve-out Lock: Q1B_ASK_CARVE_OUT is preserved until both the gate and the dead-zone proofs are active.
  • Registry Guard: Flipped registries are blocked until all retirement conditions are met.

9. Non-Claims

This scoping document establishes architectural boundaries only:

  • No Implementation: It does not implement served ASK.
  • No Wiring: It does not wire generate/contemplation/pass_manager.py or chat/runtime.py.
  • No Retirement: It does not retire Q1B_ASK_CARVE_OUT or flip any registry proposal_allowed flags.
  • No Metric Changes: It does not alter GSM8K benchmark claims or refusal behaviors.
  • No General Intelligence: It does not make ASK generally intelligent; it simply bounds the served-surface gate.

10. Recommended Next Slice

To safely approach the implementation, the next sequential slices are recommended:

Slice 1: Configuration Definition (No Wiring)

  • Add a concrete configuration field RuntimeConfig.ask_serving_enabled: bool = False (if not already present).
  • Ensure the helper ask_serving_enabled(...) references this configuration.
  • Write configuration tests verifying the default and override values.
  • Do not wire pass_manager or modify runtime loops.

Slice 2: pass_manager Integration

  • Wire pass_manager to call deliver_ask without constructing prose; any user-visible ASK exposure remains behind ask_serving_enabled; preserve Q1B_ASK_CARVE_OUT.