feat(epistemic-disclosure): Q1-B — typed ASK residue + missing_* reclassification (carve-out)
Reclassify missing_total_count / missing_weighted_total in the limitation layer
from (capability_gap, emit_proposal) to (missing_information, ask_question) —
the user CAN state the total, so these are missing data, not capability gaps.
Add typed ASK residue (MissingSlot, grounded_terms) on LimitationAssessment;
defaults to empty so existing P0-1 callers continue to work.
Q1B_ASK_CARVE_OUT: the shipped failure_family REGISTRY still flags both as
proposal_allowed=True so existing consumers (proposal pile, contemplation
pass_manager) keep emitting proposal artifacts until Q1-C/Q1-D wires ASK
delivery. Flipping the registry flag before ASK is served would create a
no-proposal/no-question dead zone — a capability-regression window with no
compensating intake. The carve-out is named, enumerated, and explicit; once
ASK delivery is live the REGISTRY flag flips and the carve-out retires.
Wrong=0 invariant on residue (scoping §2): MissingSlot carries family-typed
structural identifiers only (snake_case); grounded_terms is verbatim text from
ComprehensionAttempt.evidence SourceSpanLinks — never fabricated from family
or refusal_reason. Today classify_* leaves evidence empty for refusals, so
grounded_terms is empty everywhere; the wiring is ready for the residue
upgrade to readers later.
Off-serving: limitation.py imports no generate.derivation / core.reliability_gate
(AST-guarded by test_limitation_module_is_off_serving). No renderer, no bus
delivery — those are Q1-C/Q1-D.
Tests:
tests/test_limitation_assessment.py — 39 passing
+ test_proposal_allowed_iff_emit_proposal_with_q1b_ask_carveout (amended INV-A)
+ test_missing_{total,weighted}_count_limitation_is_ask_oriented
+ test_q1b_ask_carveout_is_explicit_and_enumerated
+ test_registry_proposal_allowed_preserved_until_ask_delivery
+ test_no_signal_loss_before_question_bus_is_serving (live pass_manager check)
+ test_missing_{total,weighted}_count_has_typed_missing_slot_residue
+ test_grounded_terms_populated_only_from_evidence_spans
+ test_residue_never_contains_ungrounded_terms
+ test_residue_empty_for_non_ask_attempts
+ test_missing_slots_default_empty_for_non_ask_families
+ test_missing_slot_keys_are_subset_of_ask_families
+ test_limitation_module_is_off_serving (AST guard)
Suites:
tests/test_epistemic_*.py, test_disclosure_claim.py, test_served_disposition.py,
test_verified_contract.py, test_failure_family.py, test_failure_proposal.py,
test_r3_router_contemplation.py, test_proposal_review_*.py,
test_idle_proposal_review.py — 135 passing
core test --suite smoke -q — 90 passing
This commit is contained in:
Shay
parent
05a044d9a5
commit
f90d84b4f7
3 changed files with 401 additions and 30 deletions
|
|
@ -36,9 +36,10 @@ from core.epistemic_disclosure.disposition import (
|
|||
choose_served_disposition,
|
||||
)
|
||||
from core.epistemic_disclosure.limitation import (
|
||||
PENDING_Q1B_RECLASSIFICATION,
|
||||
Q1B_ASK_CARVE_OUT,
|
||||
LimitationAssessment,
|
||||
LimitationKind,
|
||||
MissingSlot,
|
||||
ResolutionAction,
|
||||
assess_from_attempt,
|
||||
assess_from_family,
|
||||
|
|
@ -56,11 +57,12 @@ from core.epistemic_disclosure.verified_contract import (
|
|||
|
||||
__all__ = [
|
||||
"DEFAULT_DISCLOSURE_CLAIM",
|
||||
"PENDING_Q1B_RECLASSIFICATION",
|
||||
"Q1B_ASK_CARVE_OUT",
|
||||
"VERIFICATION_OBLIGATION",
|
||||
"DisclosureClaim",
|
||||
"LimitationAssessment",
|
||||
"LimitationKind",
|
||||
"MissingSlot",
|
||||
"ResolutionAction",
|
||||
"ServedDisposition",
|
||||
"VerificationObligation",
|
||||
|
|
|
|||
|
|
@ -16,20 +16,31 @@ DERIVES each from the already-shipped failure-family registry
|
|||
**Hard invariant (no fourth taxonomy).** Every assessment is mechanically derived
|
||||
from an existing ``FailureFamily``; the *only* genuinely new resolution action is
|
||||
``ask_question`` (the future Q1/ASK tenant — it is the one action with no shipped
|
||||
terminal, see :func:`terminal_for_action`). The *only* family reclassification this
|
||||
design calls for — ``missing_total_count`` / ``missing_weighted_total`` from
|
||||
``capability_gap`` to ``missing_information`` — is **deferred** to Q1-B and decided
|
||||
there with tests, NOT made here (see :data:`PENDING_Q1B_RECLASSIFICATION`). This
|
||||
slice maps them to their *current* shipped classification.
|
||||
terminal, see :func:`terminal_for_action`).
|
||||
|
||||
**Q1-B transitional carve-out (this slice).** Two shipped families —
|
||||
``missing_total_count`` and ``missing_weighted_total`` — are classified here as
|
||||
``missing_information`` / ``ask_question``: the user *could state the total* and
|
||||
unblock solving, so they are missing data, not capability gaps. The shipped
|
||||
:data:`REGISTRY` still flags them ``proposal_allowed = True`` so that existing
|
||||
consumers (:mod:`core.comprehension_attempt.proposal` /
|
||||
:mod:`generate.contemplation.pass_manager`) keep emitting proposal-only artifacts
|
||||
to the pile — until Q1-C/Q1-D wire ASK delivery to a served surface there is
|
||||
nowhere for an ``ask_question`` to go, and silently dropping the proposal signal
|
||||
would be a capability regression with no compensating intake. Once ASK is serving,
|
||||
the REGISTRY flag flips to ``False`` and the carve-out
|
||||
(:data:`Q1B_ASK_CARVE_OUT`) retires. The carve-out is named, enumerated, and
|
||||
covered by an explicit test (``tests/test_limitation_assessment.py``) so its
|
||||
removal is a conscious act, not a silent re-key.
|
||||
|
||||
**Off-serving.** Imports no ``generate.derivation`` / ``core.reliability_gate``; it
|
||||
cannot move the sealed GSM8K metric. Nothing consumes ``resolution_action`` to change
|
||||
serving yet — this slice only *classifies*.
|
||||
serving yet — this slice only *classifies* and *describes residue*.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Literal
|
||||
|
||||
from core.comprehension_attempt.failure_family import (
|
||||
|
|
@ -66,6 +77,26 @@ ResolutionAction = Literal[
|
|||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class MissingSlot:
|
||||
"""One typed slot the ASK limitation identifies as missing.
|
||||
|
||||
A *structural* description (NOT user-renderable prose): ``slot_name`` is the
|
||||
family-defined slot identifier (e.g. ``"total_count"``); ``expected_unit_or_type``
|
||||
is the family-typed expectation a future bound answer must satisfy (e.g.
|
||||
``"count_int"``); ``binding_target`` is the structural role the slot fills in
|
||||
the organ's setup (e.g. ``"collective_unit_total"``), which Q2 answer-binding
|
||||
re-enters the gate against (scoping §4). Renderable, user-facing strings come
|
||||
later from grounded text spans (:attr:`LimitationAssessment.grounded_terms`),
|
||||
NEVER from these snake_case identifiers — that split is what keeps the renderer
|
||||
wrong=0-safe (scoping §2).
|
||||
"""
|
||||
|
||||
slot_name: str
|
||||
expected_unit_or_type: str
|
||||
binding_target: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LimitationAssessment:
|
||||
"""One typed classification of the limitation blocking a comprehension attempt.
|
||||
|
|
@ -74,6 +105,13 @@ class LimitationAssessment:
|
|||
deliberately distinct: e.g. ``UNDETERMINED`` + ``ask_question`` for a missing
|
||||
datum. ``blocking_reason`` is the failure-family key (the partition key), so the
|
||||
assessment is back-traceable to the shipped registry.
|
||||
|
||||
``missing_slots`` and ``grounded_terms`` are the ASK *typed residue* — populated
|
||||
only for ``ask_question`` resolutions by :func:`assess_from_attempt`. Both
|
||||
default to empty so existing P0-1 callers using :func:`assess_from_family`
|
||||
continue to work unchanged. The wrong=0 invariant (scoping §2): a slot here
|
||||
carries family-typed structural identifiers only; renderable prose must come
|
||||
from ``grounded_terms`` (verbatim text spans), never fabricated.
|
||||
"""
|
||||
|
||||
limitation_kind: LimitationKind
|
||||
|
|
@ -81,6 +119,8 @@ class LimitationAssessment:
|
|||
epistemic_state: EpistemicState
|
||||
owner_organ: str | None
|
||||
blocking_reason: str
|
||||
missing_slots: tuple[MissingSlot, ...] = field(default_factory=tuple)
|
||||
grounded_terms: tuple[str, ...] = field(default_factory=tuple)
|
||||
|
||||
|
||||
# --- The consolidating mapping (derived from the shipped registry, not invented) ---
|
||||
|
|
@ -113,17 +153,23 @@ _ACTION_TO_TERMINAL: dict[ResolutionAction, Terminal | None] = {
|
|||
"ask_question": None,
|
||||
}
|
||||
|
||||
#: Families Q1-B will reclassify (``capability_gap`` → ``missing_information``): the
|
||||
#: user CAN supply the total, so it is missing information, not a capability gap. NOT
|
||||
#: changed here — this slice keeps the current classification; Q1-B flips it with its
|
||||
#: own tests. Listed so the deferral is explicit and greppable.
|
||||
PENDING_Q1B_RECLASSIFICATION: frozenset[str] = frozenset(
|
||||
#: **Transitional carve-out (Q1-B).** Families this slice classifies as
|
||||
#: ``ask_question`` in the limitation layer while their shipped
|
||||
#: :data:`REGISTRY` entries keep ``proposal_allowed = True`` so the contemplation
|
||||
#: pass and proposal pile keep working unchanged. This is an honest migration seam:
|
||||
#: the disclosure layer speaks the truthful classification now; the operational
|
||||
#: layer keeps the current signal so nothing regresses before Q1-C/Q1-D wires ASK
|
||||
#: delivery. Retirement: once ASK is serving, flip ``proposal_allowed = False`` on
|
||||
#: these two families in :mod:`core.comprehension_attempt.failure_family`, drop
|
||||
#: this set, and amend the ``proposal_allowed`` invariant in tests.
|
||||
Q1B_ASK_CARVE_OUT: frozenset[str] = frozenset(
|
||||
{"missing_total_count", "missing_weighted_total"}
|
||||
)
|
||||
|
||||
#: family name → (LimitationKind, ResolutionAction). Keys must equal the REGISTRY's
|
||||
#: family names exactly (asserted total by test). Classification rationale per family:
|
||||
#: - growth surfaces (``proposal_allowed``) → ``capability_gap`` / ``emit_proposal``
|
||||
#: EXCEPT :data:`Q1B_ASK_CARVE_OUT` — see that constant's docstring
|
||||
#: - underdetermined / missing-datum refusals → ``missing_information`` / ``ask_question``
|
||||
#: - same-unit-but-no-cue ambiguity the user could resolve → ``ambiguous_structure`` / ask
|
||||
#: - math/logic impossibility, incoherence, coarse-signal parse gaps → ``hard_boundary`` / refuse
|
||||
|
|
@ -144,9 +190,10 @@ _FAMILY_TO_LIMITATION: dict[str, tuple[LimitationKind, ResolutionAction]] = {
|
|||
"non_integer_solution": ("hard_boundary", "refuse_known_boundary"),
|
||||
"negative_solution": ("hard_boundary", "refuse_known_boundary"),
|
||||
"answer_choice_unresolved": ("ambiguous_structure", "refuse_known_boundary"),
|
||||
# R2 growth surfaces
|
||||
"missing_total_count": ("capability_gap", "emit_proposal"), # PENDING_Q1B → missing_information/ask
|
||||
"missing_weighted_total": ("capability_gap", "emit_proposal"), # PENDING_Q1B → missing_information/ask
|
||||
# R2 growth surfaces — Q1-B reclassification (see Q1B_ASK_CARVE_OUT):
|
||||
# disclosure says ask; REGISTRY still emits proposals to the pile.
|
||||
"missing_total_count": ("missing_information", "ask_question"),
|
||||
"missing_weighted_total": ("missing_information", "ask_question"),
|
||||
"missing_category_pair": ("capability_gap", "emit_proposal"),
|
||||
"missing_attribute_coefficient": ("capability_gap", "emit_proposal"),
|
||||
# R2 verdict
|
||||
|
|
@ -167,6 +214,34 @@ _FAMILY_TO_LIMITATION: dict[str, tuple[LimitationKind, ResolutionAction]] = {
|
|||
"cmb_unsupported_clock_interval": ("capability_gap", "emit_proposal"),
|
||||
}
|
||||
|
||||
#: family name → typed slots an ASK assessment for that family identifies as missing.
|
||||
#: Only families with a *known* slot structure appear here; an ask-mapped family without
|
||||
#: an entry yields an empty residue (the "minimal" stance — never fabricate a slot the
|
||||
#: family's contract has not pinned down yet). Slot semantics, per family:
|
||||
#: - ``missing_total_count`` : the collective-unit total count (R2 constraint C7);
|
||||
#: ``binding_target`` matches the equation slot the augmented input would fill
|
||||
#: (Q2 re-entry, scoping §4).
|
||||
#: - ``missing_weighted_total``: the measured-unit weighted total (R2 constraint C8).
|
||||
#: Other ask-mapped families (``ungrounded_base``, ``rate_underdetermined``,
|
||||
#: ``cmb_underdetermined``, ``cmb_combine_ambiguous``) get slots in later Q1 sub-PRs
|
||||
#: once their per-family slot signatures are pinned with tests.
|
||||
_FAMILY_TO_MISSING_SLOTS: dict[str, tuple[MissingSlot, ...]] = {
|
||||
"missing_total_count": (
|
||||
MissingSlot(
|
||||
slot_name="total_count",
|
||||
expected_unit_or_type="count_int",
|
||||
binding_target="collective_unit_total",
|
||||
),
|
||||
),
|
||||
"missing_weighted_total": (
|
||||
MissingSlot(
|
||||
slot_name="weighted_total",
|
||||
expected_unit_or_type="measured_unit_int",
|
||||
binding_target="weighted_total_value",
|
||||
),
|
||||
),
|
||||
}
|
||||
|
||||
# A conservative refusal for any family/reason that is not in the mapping. The total
|
||||
# mapping (asserted by test against REGISTRY) means this is dead in practice; if a new
|
||||
# family ever lands unmapped, it refuses — it NEVER silently becomes an answerable
|
||||
|
|
@ -181,7 +256,9 @@ def assess_from_family(family: FailureFamily) -> LimitationAssessment:
|
|||
"""The limitation a failure family expresses, as a typed assessment.
|
||||
|
||||
Total over :data:`REGISTRY` (asserted by test). An unmapped family falls to the
|
||||
conservative refuse default — never ``ask_question``.
|
||||
conservative refuse default — never ``ask_question``. Residue defaults to empty;
|
||||
populating typed slots / grounded terms requires a specific attempt (use
|
||||
:func:`assess_from_attempt`).
|
||||
"""
|
||||
kind, action = _FAMILY_TO_LIMITATION.get(family.name, _CONSERVATIVE_DEFAULT)
|
||||
return LimitationAssessment(
|
||||
|
|
@ -193,6 +270,29 @@ def assess_from_family(family: FailureFamily) -> LimitationAssessment:
|
|||
)
|
||||
|
||||
|
||||
def _residue_from_attempt(
|
||||
attempt: ComprehensionAttempt,
|
||||
family: FailureFamily,
|
||||
action: ResolutionAction,
|
||||
) -> tuple[tuple[MissingSlot, ...], tuple[str, ...]]:
|
||||
"""Typed ASK residue for an ask-mapped attempt — empty for any other action.
|
||||
|
||||
Wrong=0 invariant (scoping §2): ``grounded_terms`` carries only verbatim text from
|
||||
:attr:`ComprehensionAttempt.evidence` SourceSpanLinks. If the attempt carries no
|
||||
evidence (today: every refused attempt — :mod:`core.comprehension_attempt.classify`
|
||||
leaves ``evidence = ()``), ``grounded_terms`` is empty, NEVER fabricated from the
|
||||
family or the refusal reason. ``missing_slots`` is family-derived (snake_case
|
||||
structural identifiers, not renderable prose) so it is always safe to emit; absent
|
||||
from :data:`_FAMILY_TO_MISSING_SLOTS` ⇒ no slots (minimal stance — never invent a
|
||||
slot the family contract has not pinned down).
|
||||
"""
|
||||
if action != "ask_question":
|
||||
return ((), ())
|
||||
slots = _FAMILY_TO_MISSING_SLOTS.get(family.name, ())
|
||||
grounded = tuple(link.text for link in attempt.evidence)
|
||||
return (slots, grounded)
|
||||
|
||||
|
||||
def assess_from_attempt(attempt: ComprehensionAttempt) -> LimitationAssessment | None:
|
||||
"""Classify the limitation a comprehension attempt expresses, or ``None``.
|
||||
|
||||
|
|
@ -202,6 +302,10 @@ def assess_from_attempt(attempt: ComprehensionAttempt) -> LimitationAssessment |
|
|||
falls to the conservative refuse default (never ``ask_question``).
|
||||
- any non-limitation outcome (solved/admissible setup, or eval-only ``*_wrong``)
|
||||
→ ``None``: there is no resolvable limitation to act on.
|
||||
|
||||
For ask-mapped refusals, the returned assessment carries typed residue
|
||||
(:attr:`~LimitationAssessment.missing_slots` / ``grounded_terms``) per
|
||||
:func:`_residue_from_attempt`'s wrong=0 invariant.
|
||||
"""
|
||||
if attempt.outcome == "contradiction":
|
||||
return LimitationAssessment(
|
||||
|
|
@ -223,7 +327,21 @@ def assess_from_attempt(attempt: ComprehensionAttempt) -> LimitationAssessment |
|
|||
owner_organ=attempt.organ,
|
||||
blocking_reason=attempt.refusal_reason or "unclassified",
|
||||
)
|
||||
return assess_from_family(family)
|
||||
base = assess_from_family(family)
|
||||
missing_slots, grounded_terms = _residue_from_attempt(
|
||||
attempt, family, base.resolution_action
|
||||
)
|
||||
if not missing_slots and not grounded_terms:
|
||||
return base
|
||||
return LimitationAssessment(
|
||||
limitation_kind=base.limitation_kind,
|
||||
resolution_action=base.resolution_action,
|
||||
epistemic_state=base.epistemic_state,
|
||||
owner_organ=attempt.organ,
|
||||
blocking_reason=base.blocking_reason,
|
||||
missing_slots=missing_slots,
|
||||
grounded_terms=grounded_terms,
|
||||
)
|
||||
|
||||
|
||||
def terminal_for_action(action: ResolutionAction) -> Terminal | None:
|
||||
|
|
@ -237,9 +355,10 @@ def terminal_for_action(action: ResolutionAction) -> Terminal | None:
|
|||
|
||||
|
||||
__all__ = [
|
||||
"PENDING_Q1B_RECLASSIFICATION",
|
||||
"Q1B_ASK_CARVE_OUT",
|
||||
"LimitationAssessment",
|
||||
"LimitationKind",
|
||||
"MissingSlot",
|
||||
"ResolutionAction",
|
||||
"assess_from_attempt",
|
||||
"assess_from_family",
|
||||
|
|
|
|||
|
|
@ -6,10 +6,19 @@ wrong=0-relevant cases: a hard boundary never becomes an answerable question, a
|
|||
proposal stays a proposal, a contradiction reports, foreign text steps aside. Each
|
||||
test would fail under a single mis-keyed mapping entry — none passes under a broken
|
||||
impl (CLAUDE.md "Schema-Defined Proof Obligations").
|
||||
|
||||
Q1-B tests (added 2026-06-08) cover the typed ASK residue (``MissingSlot``,
|
||||
``grounded_terms``) and the transitional carve-out (:data:`Q1B_ASK_CARVE_OUT`):
|
||||
the limitation layer classifies ``missing_total_count`` / ``missing_weighted_total``
|
||||
as ``ask_question`` while the shipped registry still flags them
|
||||
``proposal_allowed = True`` so existing proposal consumers keep working until
|
||||
Q1-C/Q1-D wires ASK delivery. The carve-out is named, enumerated, and explicit.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
from typing import get_args
|
||||
|
||||
import pytest
|
||||
|
|
@ -17,15 +26,18 @@ import pytest
|
|||
from core.comprehension_attempt.failure_family import REGISTRY, family_by_name
|
||||
from core.comprehension_attempt.model import ComprehensionAttempt
|
||||
from core.epistemic_disclosure.limitation import (
|
||||
PENDING_Q1B_RECLASSIFICATION,
|
||||
Q1B_ASK_CARVE_OUT,
|
||||
LimitationKind,
|
||||
MissingSlot,
|
||||
ResolutionAction,
|
||||
_FAMILY_TO_LIMITATION,
|
||||
_FAMILY_TO_MISSING_SLOTS,
|
||||
assess_from_attempt,
|
||||
assess_from_family,
|
||||
terminal_for_action,
|
||||
)
|
||||
from core.epistemic_state import EpistemicState
|
||||
from generate.binding_graph.model import SourceSpanLink
|
||||
from generate.contemplation.findings import Terminal
|
||||
|
||||
VALID_KINDS = frozenset(get_args(LimitationKind))
|
||||
|
|
@ -51,10 +63,21 @@ def test_assess_from_family_produces_valid_typed_values():
|
|||
|
||||
# --- flag-consistency invariants ----------------------------------------------------- #
|
||||
|
||||
def test_proposal_allowed_iff_emit_proposal():
|
||||
"""INV-A: the growth surfaces — and only those — propose."""
|
||||
def test_proposal_allowed_iff_emit_proposal_with_q1b_ask_carveout():
|
||||
"""INV-A (amended for Q1-B): the growth surfaces — and only those — propose,
|
||||
EXCEPT the Q1-B ASK carve-out (:data:`Q1B_ASK_CARVE_OUT`) which carries
|
||||
``proposal_allowed = True`` in the REGISTRY (to keep the proposal pile populated
|
||||
until Q1-C/Q1-D wires ASK delivery) while being classified as ``ask_question`` in
|
||||
the limitation layer. Retire the carve-out once ASK is serving (then flip the
|
||||
REGISTRY flag and this test reverts to the pre-Q1B form)."""
|
||||
for family in REGISTRY:
|
||||
a = assess_from_family(family)
|
||||
if family.name in Q1B_ASK_CARVE_OUT:
|
||||
# Carve-out: registry still proposal_allowed, limitation says ask.
|
||||
assert family.proposal_allowed is True, family.name
|
||||
assert a.resolution_action == "ask_question", family.name
|
||||
assert a.limitation_kind == "missing_information", family.name
|
||||
continue
|
||||
assert family.proposal_allowed == (a.resolution_action == "emit_proposal"), family.name
|
||||
|
||||
|
||||
|
|
@ -122,16 +145,217 @@ def test_specific_load_bearing_mappings(name, kind, action):
|
|||
assert (a.limitation_kind, a.resolution_action) == (kind, action)
|
||||
|
||||
|
||||
def test_pending_reclassification_is_currently_capability_gap():
|
||||
"""Documents the DEFERRED Q1-B change. Today these propose; Q1-B will flip them to
|
||||
missing_information/ask. When that lands this test changes deliberately — it is the
|
||||
tripwire proving the reclassification was a conscious, tested act, not a silent re-key."""
|
||||
for name in PENDING_Q1B_RECLASSIFICATION:
|
||||
# --- Q1-B reclassification — disclosure layer (named, tested, never silent) ---------- #
|
||||
|
||||
def test_missing_total_count_limitation_is_ask_oriented():
|
||||
"""Q1-B: ``missing_total_count`` is classified as ``missing_information`` /
|
||||
``ask_question`` in the limitation layer — the user CAN state the total. This is
|
||||
the disclosure-spine's truthful classification; see
|
||||
:func:`test_registry_proposal_allowed_preserved_until_ask_delivery` for the
|
||||
transitional carve-out that keeps proposal emission alive until Q1-C/D."""
|
||||
family = family_by_name("missing_total_count")
|
||||
assert family is not None
|
||||
a = assess_from_family(family)
|
||||
assert a.limitation_kind == "missing_information"
|
||||
assert a.resolution_action == "ask_question"
|
||||
assert a.epistemic_state == EpistemicState.UNDETERMINED
|
||||
assert a.owner_organ == "r2"
|
||||
assert a.blocking_reason == "missing_total_count"
|
||||
|
||||
|
||||
def test_missing_weighted_total_limitation_is_ask_oriented():
|
||||
"""Q1-B: ``missing_weighted_total`` is classified as ``missing_information`` /
|
||||
``ask_question`` in the limitation layer — the user CAN state the weighted
|
||||
total. Same carve-out story as ``missing_total_count``."""
|
||||
family = family_by_name("missing_weighted_total")
|
||||
assert family is not None
|
||||
a = assess_from_family(family)
|
||||
assert a.limitation_kind == "missing_information"
|
||||
assert a.resolution_action == "ask_question"
|
||||
assert a.epistemic_state == EpistemicState.UNDETERMINED
|
||||
assert a.owner_organ == "r2"
|
||||
assert a.blocking_reason == "missing_weighted_total"
|
||||
|
||||
|
||||
def test_q1b_ask_carveout_is_explicit_and_enumerated():
|
||||
"""The carve-out set is named, frozen, and contains exactly the two families
|
||||
Q1-B reclassifies — so its retirement (once ASK is serving) is a conscious,
|
||||
grep-discoverable act, not a silent re-key."""
|
||||
assert Q1B_ASK_CARVE_OUT == {"missing_total_count", "missing_weighted_total"}
|
||||
assert isinstance(Q1B_ASK_CARVE_OUT, frozenset)
|
||||
for name in Q1B_ASK_CARVE_OUT:
|
||||
assert family_by_name(name) is not None, name
|
||||
|
||||
|
||||
# --- Q1-B carve-out — no signal loss before ASK delivery ----------------------------- #
|
||||
|
||||
def test_registry_proposal_allowed_preserved_until_ask_delivery():
|
||||
"""The transitional carve-out's *operational* half: the shipped REGISTRY MUST
|
||||
still flag ``missing_total_count`` / ``missing_weighted_total`` as
|
||||
``proposal_allowed = True`` so existing consumers
|
||||
(:mod:`core.comprehension_attempt.proposal`,
|
||||
:mod:`generate.contemplation.pass_manager`) keep emitting proposal-only artifacts
|
||||
to the pile. Until Q1-C/D wires ASK delivery to a served surface, flipping this
|
||||
flag would create a no-proposal/no-question dead zone (capability regression).
|
||||
Retirement: this test inverts once ASK is serving."""
|
||||
for name in Q1B_ASK_CARVE_OUT:
|
||||
family = family_by_name(name)
|
||||
assert family is not None, name
|
||||
assert family.proposal_allowed is True, (
|
||||
f"{name}: REGISTRY proposal_allowed flipped before ASK delivery — "
|
||||
f"would silently drop the proposal pile signal. See Q1B_ASK_CARVE_OUT."
|
||||
)
|
||||
assert family.must_remain_refused is False, name
|
||||
assert family.proposal_target == "r2_gold_fixture", name
|
||||
|
||||
|
||||
def test_no_signal_loss_before_question_bus_is_serving():
|
||||
"""End-to-end: the contemplation pass MUST still reach ``PROPOSAL_EMITTED`` for
|
||||
Q1-B-reclassified families until ASK delivery is wired. Proves the disclosure-
|
||||
layer reclassification did not silently drop proposal emission in the live
|
||||
contemplation pass — the no-regression guarantee made above."""
|
||||
from generate.contemplation.pass_manager import contemplate
|
||||
|
||||
# The R2 fixture for the missing_total_count case is the same one
|
||||
# tests/test_r3_router_contemplation.py uses for its R3-not-blocking check.
|
||||
from evals.constraint_oracle.runner import _load_r2_gold
|
||||
|
||||
fx = next(f for f in _load_r2_gold() if f["id"] == "r2-011-missing-total-count")
|
||||
result = contemplate(fx["text"])
|
||||
assert result.terminal == Terminal.PROPOSAL_EMITTED, (
|
||||
f"missing_total_count contemplation terminal={result.terminal} — Q1-B carve-out "
|
||||
f"breach: proposal emission lost before ASK delivery is wired."
|
||||
)
|
||||
assert result.family == "missing_total_count"
|
||||
|
||||
|
||||
# --- Q1-B typed residue — MissingSlot + grounded_terms ------------------------------- #
|
||||
|
||||
def test_missing_slots_default_empty_for_non_ask_families():
|
||||
"""Backward compatibility: ``assess_from_family`` returns empty residue (so
|
||||
existing P0-1 callers continue to work). Only :func:`assess_from_attempt` for
|
||||
an ask-mapped attempt populates residue."""
|
||||
for family in REGISTRY:
|
||||
a = assess_from_family(family)
|
||||
assert (a.limitation_kind, a.resolution_action) == ("capability_gap", "emit_proposal"), name
|
||||
assert PENDING_Q1B_RECLASSIFICATION == {"missing_total_count", "missing_weighted_total"}
|
||||
assert a.missing_slots == ()
|
||||
assert a.grounded_terms == ()
|
||||
|
||||
|
||||
def test_missing_total_count_has_typed_missing_slot_residue():
|
||||
"""Ask-mapped attempt for ``missing_total_count`` carries the family's typed
|
||||
slot (structural identifier — NOT renderable prose). ``grounded_terms`` is empty
|
||||
because today's :func:`classify_r2` leaves ``evidence = ()``; the wrong=0
|
||||
invariant (scoping §2) forbids fabricating renderable terms."""
|
||||
attempt = ComprehensionAttempt(
|
||||
organ="r2_constraints",
|
||||
outcome="setup_refused",
|
||||
refusal_reason="missing_total_count",
|
||||
)
|
||||
a = assess_from_attempt(attempt)
|
||||
assert a is not None
|
||||
assert a.resolution_action == "ask_question"
|
||||
assert a.missing_slots == (
|
||||
MissingSlot(
|
||||
slot_name="total_count",
|
||||
expected_unit_or_type="count_int",
|
||||
binding_target="collective_unit_total",
|
||||
),
|
||||
)
|
||||
assert a.grounded_terms == ()
|
||||
|
||||
|
||||
def test_missing_weighted_total_has_typed_missing_slot_residue():
|
||||
"""Same shape as ``missing_total_count``, different slot."""
|
||||
attempt = ComprehensionAttempt(
|
||||
organ="r2_constraints",
|
||||
outcome="setup_refused",
|
||||
refusal_reason="missing_weighted_total",
|
||||
)
|
||||
a = assess_from_attempt(attempt)
|
||||
assert a is not None
|
||||
assert a.resolution_action == "ask_question"
|
||||
assert a.missing_slots == (
|
||||
MissingSlot(
|
||||
slot_name="weighted_total",
|
||||
expected_unit_or_type="measured_unit_int",
|
||||
binding_target="weighted_total_value",
|
||||
),
|
||||
)
|
||||
assert a.grounded_terms == ()
|
||||
|
||||
|
||||
def test_grounded_terms_populated_only_from_evidence_spans():
|
||||
"""When the attempt carries evidence, ``grounded_terms`` reads verbatim text
|
||||
from those spans — never the family name, never the refusal reason. This is the
|
||||
grounded-rendering wrong=0 invariant in action."""
|
||||
spans = (
|
||||
SourceSpanLink(source_id="src", start=0, end=8, text="chickens"),
|
||||
SourceSpanLink(source_id="src", start=10, end=17, text="rabbits"),
|
||||
)
|
||||
attempt = ComprehensionAttempt(
|
||||
organ="r2_constraints",
|
||||
outcome="setup_refused",
|
||||
refusal_reason="missing_total_count",
|
||||
evidence=spans,
|
||||
)
|
||||
a = assess_from_attempt(attempt)
|
||||
assert a is not None
|
||||
assert a.resolution_action == "ask_question"
|
||||
assert a.grounded_terms == ("chickens", "rabbits")
|
||||
# And missing_slots is still populated from the family.
|
||||
assert len(a.missing_slots) == 1
|
||||
|
||||
|
||||
def test_residue_empty_for_non_ask_attempts():
|
||||
"""A refusal that classifies to ``refuse_known_boundary`` carries no residue
|
||||
even if it has evidence — only ask-mapped attempts get residue."""
|
||||
spans = (SourceSpanLink(source_id="src", start=0, end=5, text="dummy"),)
|
||||
attempt = ComprehensionAttempt(
|
||||
organ="r4_combined_rate",
|
||||
outcome="setup_refused",
|
||||
refusal_reason="cmb_non_positive_net_rate",
|
||||
evidence=spans,
|
||||
)
|
||||
a = assess_from_attempt(attempt)
|
||||
assert a is not None
|
||||
assert a.resolution_action == "refuse_known_boundary"
|
||||
assert a.missing_slots == ()
|
||||
assert a.grounded_terms == ()
|
||||
|
||||
|
||||
def test_residue_never_contains_ungrounded_terms():
|
||||
"""The hard wrong=0 invariant: nothing in ``grounded_terms`` may be invented
|
||||
from the family/reason. Across the registry, an attempt with no evidence yields
|
||||
no grounded terms, period."""
|
||||
for family in REGISTRY:
|
||||
if not family.refusal_reasons:
|
||||
continue
|
||||
reason = family.refusal_reasons[0]
|
||||
attempt = ComprehensionAttempt(
|
||||
organ="r2_constraints",
|
||||
outcome="setup_refused",
|
||||
refusal_reason=reason,
|
||||
)
|
||||
a = assess_from_attempt(attempt)
|
||||
if a is None:
|
||||
continue
|
||||
# Empty evidence ⇒ empty grounded_terms, full stop.
|
||||
assert a.grounded_terms == (), (family.name, reason)
|
||||
# Slot identifiers are structural — they may not appear verbatim in the
|
||||
# source text, so they never count as "grounded terms" here.
|
||||
for slot in a.missing_slots:
|
||||
assert slot.slot_name not in a.grounded_terms
|
||||
assert slot.binding_target not in a.grounded_terms
|
||||
|
||||
|
||||
def test_missing_slot_keys_are_subset_of_ask_families():
|
||||
"""``_FAMILY_TO_MISSING_SLOTS`` may only list ask-mapped families — slots are an
|
||||
ask-only artifact. Non-ask families with slots would be a category error."""
|
||||
for name in _FAMILY_TO_MISSING_SLOTS:
|
||||
family = family_by_name(name)
|
||||
assert family is not None, name
|
||||
kind, action = _FAMILY_TO_LIMITATION[name]
|
||||
assert action == "ask_question", (name, action)
|
||||
|
||||
|
||||
# --- the epistemic-state axis -------------------------------------------------------- #
|
||||
|
|
@ -210,3 +434,29 @@ def test_assess_attempt_unclassified_reason_refuses_never_asks():
|
|||
assert a is not None
|
||||
assert a.resolution_action == "refuse_known_boundary"
|
||||
assert a.resolution_action != "ask_question"
|
||||
|
||||
|
||||
# --- off-serving AST guard ----------------------------------------------------------- #
|
||||
|
||||
def test_limitation_module_is_off_serving():
|
||||
"""The limitation module must import nothing from ``generate.derivation`` or
|
||||
``core.reliability_gate`` — that is the discipline that lets the disclosure spine
|
||||
move without touching the sealed GSM8K serving metric. Enforced AST-side so a
|
||||
regression is caught at module-load time, not by metric drift."""
|
||||
forbidden = ("generate.derivation", "core.reliability_gate")
|
||||
path = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "core" / "epistemic_disclosure" / "limitation.py"
|
||||
)
|
||||
tree = ast.parse(path.read_text())
|
||||
seen: list[str] = []
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
seen.extend(alias.name for alias in node.names)
|
||||
elif isinstance(node, ast.ImportFrom) and node.module is not None:
|
||||
seen.append(node.module)
|
||||
for module in seen:
|
||||
for forbidden_prefix in forbidden:
|
||||
assert not (module == forbidden_prefix or module.startswith(forbidden_prefix + ".")), (
|
||||
f"off-serving breach: limitation.py imports {module!r}"
|
||||
)
|
||||
|
|
|
|||
Loading…
Reference in a new issue