feat(adr-0026): Phase 3 — ranked admissibility with margin
Replace the static-threshold admissibility gate with a ranked-with-
margin check that is scale-invariant under blade-norm variation.
Phase 4 characterization established no single global threshold
separates the v2 mechanism-isolation cases (blade norms vary ~10x);
margins between top and second-ranked candidates do, because they
scale with the blade norm and carry the relative ordering the
geometry actually delivers.
New primitives in generate/admissibility.py:
RankedCandidate — (index, word, score)
MarginVerdict — admit/reject + top + margin + full ranking
rank_candidates_by_blade — sort admissible set by cga_inner desc,
strict > tie-break by ascending vocab index
check_margin — admit top iff score>0 AND margin>=delta
Selection semantics in margin mode are blade-rank-driven: the top-
ranked admissible candidate IS the admitted destination. Differs
from threshold mode (field-driven _nearest_next then per-candidate
gate). Both modes coexist; threshold is the default and ADR-0024
acceptance evidence is preserved byte-for-byte.
Wired through:
core/config.py admissibility_mode="threshold" (default)
admissibility_margin=0.4
chat/runtime.py forwards both fields
generate/stream.py margin_mode_active branch — ranks the
candidate set once per step, admits or
raises InnerLoopExhaustion with the full
ranking in rejected_attempts
Default delta = 0.4 chosen from the v2 case margins:
V2-001: 0.596 V2-002: 0.456 V2-003: 13.27
V2-004: 3.37 V2-005: 12.74
min = 0.456 → 0.4 admits all 5 with headroom; 0.5 would refuse
V2-002. The default is falsifiable: Phase 5 may surface a case
below 0.4, which should be reported as an architectural finding
rather than patched per-case.
Acceptance evidence (tests/test_margin_admissibility.py, 13 passing):
5/5 v2 cases pass in margin mode; forbidden_token in every
case's rejected_attempts ranking
Refusal-on-insufficient-margin: delta=0.9 on V2-001 (margin
0.597) raises InnerLoopExhaustion with full ranking; no silent
boundary fallback
Threshold mode byte-identical with or without margin plumbing
5 reruns produce identical canonical trace steps
Strict > tie-break: equal scores resolve to lower-index winner
deterministically
Invariants preserved:
versor_condition < 1e-6 — rotor V is constructed only for the
admitted candidate; margin mode adds no normalization/repair site
Deterministic replay — strict > tie-break now load-bearing in
rank_candidates_by_blade alongside vocab.nearest
No approximate recall, no cosine similarity, no HNSW/ANN; pure
rank-and-difference on exact cga_inner scores
No new code in field/propagate.py, algebra/versor.py,
vault/store.py, or chat/runtime.respond()
Suite results:
full: 1037 passed, 2 skipped (+13 new margin tests)
core eval cognition: 13/13, 100% intent_accuracy,
100% versor_closure_rate
ADR-0026 documents the contract, the single-delta rationale, the
falsifiability story, and the residual risks. Margin mode is
flag-gated default-off; a future ADR may promote it to default
after Phase 5's diversified families confirm the single delta
holds (or surface the architectural finding if it doesn't).
This commit is contained in:
Shay
parent
310793a4ea
commit
639e107442
6 changed files with 875 additions and 55 deletions
|
|
@ -184,6 +184,8 @@ class ChatRuntime:
|
|||
inhibition_threshold=config.inhibition_threshold,
|
||||
inner_loop_admissibility=config.inner_loop_admissibility,
|
||||
admissibility_threshold=config.admissibility_threshold,
|
||||
admissibility_mode=config.admissibility_mode,
|
||||
admissibility_margin=config.admissibility_margin,
|
||||
)
|
||||
else:
|
||||
resolved_config = config
|
||||
|
|
@ -400,6 +402,8 @@ class ChatRuntime:
|
|||
inhibition_threshold=self.config.inhibition_threshold,
|
||||
inner_loop_admissibility=self.config.inner_loop_admissibility,
|
||||
admissibility_threshold=self.config.admissibility_threshold,
|
||||
admissibility_mode=self.config.admissibility_mode,
|
||||
admissibility_margin=self.config.admissibility_margin,
|
||||
)
|
||||
|
||||
# --- Articulation fidelity: replace bare S-P-O join with intent-aware surface ---
|
||||
|
|
|
|||
|
|
@ -22,6 +22,13 @@ class RuntimeConfig:
|
|||
inhibition_threshold: float = 0.3
|
||||
inner_loop_admissibility: bool = False
|
||||
admissibility_threshold: float = 0.0
|
||||
# ADR-0026 / Phase 3 — margin-based admissibility. ``mode``
|
||||
# selects between ADR-0024's per-candidate threshold check and
|
||||
# the ranked-with-margin check. Default "threshold" preserves
|
||||
# ADR-0024 acceptance evidence; opt-in "margin" replaces the
|
||||
# static-threshold gate with a scale-invariant margin.
|
||||
admissibility_mode: str = "threshold"
|
||||
admissibility_margin: float = 0.4
|
||||
|
||||
|
||||
DEFAULT_CONFIG = RuntimeConfig()
|
||||
|
|
|
|||
240
docs/decisions/ADR-0026-ranked-admissibility-with-margin.md
Normal file
240
docs/decisions/ADR-0026-ranked-admissibility-with-margin.md
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
# ADR-0026 — Ranked Admissibility with Margin
|
||||
|
||||
**Status:** Accepted (2026-05-17)
|
||||
**Supersedes (in part):** ADR-0024's static `admissibility_threshold`
|
||||
mechanism for production admissibility gating. ADR-0024 remains
|
||||
Accepted; threshold mode is preserved for backward-compatible
|
||||
acceptance evidence and is the default unless margin mode is
|
||||
explicitly enabled.
|
||||
|
||||
## Context
|
||||
|
||||
Phase 4 threshold characterization (`tests/test_inner_loop_phase4.py`)
|
||||
established a load-bearing geometric finding: **no single global static
|
||||
threshold delivers `separation_quality >= 0.8` across the v2 mechanism-
|
||||
isolation cases.** Blade norms vary roughly tenfold across cases
|
||||
(`best_separation_quality < 0.5` is now an invariant test), so the
|
||||
same `tau` value means semantically different things case-to-case.
|
||||
|
||||
Three options were on the table for Phase 3:
|
||||
|
||||
1. **Per-case normalised thresholds** (e.g. `alpha * blade_self_score`).
|
||||
Adds a tuning surface; the constant becomes a knob.
|
||||
2. **Per-pack thresholds.** Migrates the tuning problem from the
|
||||
blade level to the pack level; same failure mode.
|
||||
3. **Ranked-with-margin admissibility.** Replaces the absolute-
|
||||
score gate with a *relative-ordering* gate: rank the admissible
|
||||
candidates by `cga_inner(versor, relation_blade)` descending and
|
||||
admit the top iff its margin over the second-ranked exceeds a
|
||||
single per-runtime `delta`.
|
||||
|
||||
Option 3 is the only option that is **scale-invariant under blade-
|
||||
norm variation** — the gap between the top and second-ranked scores
|
||||
scales with the blade norm, so the *relative* ordering carries the
|
||||
semantic separation the static threshold was reaching for. This is
|
||||
the architectural difference between "what direction is admissible"
|
||||
(blade alignment, absolute) and "which candidate is confidently
|
||||
selected over the next-best" (separation, relative).
|
||||
|
||||
The previous assessment (recorded in the planning conversation that
|
||||
preceded this ADR) put it this way:
|
||||
|
||||
> A global threshold — even normalized — assumes admissibility is an
|
||||
> absolute property of a candidate. It isn't: in your geometry,
|
||||
> what matters is the ordering of admissibility scores across the
|
||||
> candidate set, plus enough separation to be confident the
|
||||
> boundary's pick can be rejected and replaced deterministically.
|
||||
|
||||
## Decision
|
||||
|
||||
Add a new admissibility mode `margin` alongside ADR-0024's existing
|
||||
`threshold` mode. When margin mode is active, the inner loop's
|
||||
selection-and-admission collapses into a single deterministic step:
|
||||
|
||||
```python
|
||||
ranked = rank_candidates_by_blade(region, candidate_indices=…, …)
|
||||
verdict = check_margin(region, ranked, delta=admissibility_margin)
|
||||
```
|
||||
|
||||
`rank_candidates_by_blade` sorts the post-region-filter candidate set
|
||||
by `cga_inner(versor, relation_blade)` descending, with a strict `>`
|
||||
tie-break: when scores are equal, ascending vocab index wins. This
|
||||
matches the `vocab.nearest` strict-`>` convention documented as
|
||||
load-bearing in ADR-0024.
|
||||
|
||||
`check_margin` admits the top-ranked candidate iff:
|
||||
|
||||
1. `len(ranked) >= 1` (non-empty admissible set), AND
|
||||
2. `ranked[0].score > 0` (basic positivity in the blade half-space),
|
||||
AND
|
||||
3. `len(ranked) == 1` (trivial, no competitor) **OR**
|
||||
`ranked[0].score - ranked[1].score >= delta`.
|
||||
|
||||
When refused, the inner loop raises `InnerLoopExhaustion`
|
||||
(ADR-0024 Phase 2) carrying the full ranking as `rejected_attempts` —
|
||||
not a single rejected score, but the entire blade-ordering at the
|
||||
failed step.
|
||||
|
||||
### Selection semantics in margin mode
|
||||
|
||||
Margin mode is **blade-rank-driven** selection: the top-ranked
|
||||
admissible candidate IS the admitted destination. This differs from
|
||||
threshold mode, where `_nearest_next` (field-driven) picks one
|
||||
candidate and `check_transition` gates it; on rejection, the
|
||||
selector advances to the next field-closest candidate and re-gates.
|
||||
|
||||
This is a meaningful semantic difference, not a re-shading. Margin
|
||||
mode says *"of the candidates the region admits as semantically
|
||||
valid, the most blade-aligned one is the destination, provided its
|
||||
lead is confident."* Threshold mode says *"of the candidates the
|
||||
field's geometry prefers, accept the first one above an absolute
|
||||
score bar."*
|
||||
|
||||
Both have a place: threshold mode is the ADR-0024 acceptance
|
||||
evidence and remains the default to preserve every existing turn's
|
||||
trace_hash byte-for-byte. Margin mode is the new production
|
||||
admissibility for cases where blade-norm variation makes the static
|
||||
bar incoherent.
|
||||
|
||||
## The single `delta` choice
|
||||
|
||||
Phase 3 picks `delta = 0.4` as the default. This was derived from
|
||||
the v2 mechanism-isolation case margins:
|
||||
|
||||
| Case | Top score | Second score | Margin |
|
||||
| ------------- | --------- | ------------ | ------ |
|
||||
| FSC-PUB-V2-001 | 1.420 | 0.824 | 0.596 |
|
||||
| FSC-PUB-V2-002 | 1.173 | 0.717 | 0.456 |
|
||||
| FSC-PUB-V2-003 | 12.720 | -0.550 | 13.270 |
|
||||
| FSC-PUB-V2-004 | 5.740 | 2.370 | 3.370 |
|
||||
| FSC-PUB-V2-005 | 14.360 | 1.620 | 12.740 |
|
||||
|
||||
The minimum margin across the five cases is **0.456**. `delta = 0.4`
|
||||
admits all five with headroom; `delta = 0.5` would refuse V2-002.
|
||||
The default is *falsifiable*: Phase 5's diversified failure-mode
|
||||
families may surface a case below `0.4`, in which case the finding
|
||||
is architectural — margin alone is insufficient for that family —
|
||||
and should be reported honestly rather than patched with a per-case
|
||||
override.
|
||||
|
||||
## Wiring
|
||||
|
||||
| Surface | Field |
|
||||
| ------------------------------------------ | ------------------------------ |
|
||||
| `core/config.py::RuntimeConfig` | `admissibility_mode: str = "threshold"` |
|
||||
| | `admissibility_margin: float = 0.4` |
|
||||
| `chat/runtime.py::ChatRuntime.chat` | forwards both fields |
|
||||
| `generate/stream.py::generate` | `admissibility_mode` / `admissibility_margin` kwargs |
|
||||
| `generate/admissibility.py` | `RankedCandidate`, `MarginVerdict`, `rank_candidates_by_blade`, `check_margin` |
|
||||
|
||||
Selection ordering inside `generate.generate()`:
|
||||
|
||||
```text
|
||||
if margin_mode_active:
|
||||
rank → check_margin → admit-or-refuse
|
||||
else:
|
||||
_nearest_next → check_transition (per-attempt loop)
|
||||
```
|
||||
|
||||
The rotor `V` is only constructed for the *admitted* candidate, so
|
||||
the `versor_condition(F) < 1e-6` invariant is preserved by
|
||||
construction (CLAUDE.md §Non-Negotiable Field Invariant).
|
||||
|
||||
## Why flag-gated (default off)
|
||||
|
||||
Margin mode is a real semantic change in selection. Defaulting it
|
||||
off preserves:
|
||||
|
||||
* Every existing trace_hash byte-for-byte (no payload bytes change
|
||||
when margin mode is unset).
|
||||
* ADR-0024's acceptance evidence intact.
|
||||
* The ability to A/B threshold vs margin on the same corpus during
|
||||
the transition window.
|
||||
|
||||
A future ADR may flip the default; this ADR does not.
|
||||
|
||||
## Invariants preserved
|
||||
|
||||
* `versor_condition(F) < 1e-6` — the rotor is constructed only for
|
||||
the admitted candidate; margin mode does not add a normalization
|
||||
or repair site (CLAUDE.md §Normalization Rules).
|
||||
* Deterministic replay — strict `>` tie-break is now load-bearing
|
||||
in two places: `vocab.nearest` (ADR-0024) and
|
||||
`rank_candidates_by_blade` (this ADR).
|
||||
`tests/test_margin_admissibility.py::TestRankCandidates::test_strict_tie_break_by_ascending_index`
|
||||
pins it.
|
||||
* No approximate recall, no cosine similarity, no HNSW/ANN.
|
||||
Margin is a pure rank-and-difference operation on the exact
|
||||
`cga_inner` scores already in use.
|
||||
* No new code in `field/propagate.py`, `algebra/versor.py`,
|
||||
`vault/store.py`, or `chat/runtime.respond()`.
|
||||
|
||||
## Trust boundary
|
||||
|
||||
Same as ADR-0024 — admissibility regions are constructed upstream
|
||||
from pack-grounded data; margin mode adds no new surface that
|
||||
consumes user-controlled text, filesystem paths, or dynamic
|
||||
imports.
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
* **5/5 v2 mechanism-isolation cases pass in margin mode** with
|
||||
`delta = 0.4`. Forbidden token traced in every case's
|
||||
`rejected_attempts`. Mirrors ADR-0024 §Acceptance evidence for
|
||||
threshold mode.
|
||||
* **Threshold mode unchanged.**
|
||||
`tests/test_margin_admissibility.py::TestGenerateMarginMode::test_threshold_mode_unchanged_by_margin_plumbing`
|
||||
asserts `mode="threshold"` and unset `mode` produce identical
|
||||
tokens.
|
||||
* **Refusal on insufficient margin.**
|
||||
`test_v2_001_refuses_when_delta_too_high` runs FSC-PUB-V2-001 at
|
||||
`delta = 0.9` (above its 0.597 margin) and asserts
|
||||
`InnerLoopExhaustion` with `reason=INNER_LOOP_EXHAUSTION` and the
|
||||
full ranking in `rejected_attempts`. No silent boundary fallback.
|
||||
* **Replay determinism.**
|
||||
`TestMarginModeDeterminism::test_margin_mode_replay_stable_across_5_runs`
|
||||
asserts 5 reruns of the same input produce identical canonical
|
||||
trace steps.
|
||||
* **Strict tie-break determinism.**
|
||||
`TestRankCandidates::test_strict_tie_break_by_ascending_index`
|
||||
asserts equal-score candidates resolve to the lower-index winner
|
||||
reproducibly across 5 runs.
|
||||
|
||||
## Out of scope
|
||||
|
||||
* **Promoting margin to default.** Requires a separate ADR plus
|
||||
trace-hash migration evidence.
|
||||
* **Rotor-frame margin.** ADR-0025 will wire rotor admissibility;
|
||||
if its mechanism also benefits from a margin gate, that's an
|
||||
ADR-0025 decision.
|
||||
* **Per-family delta calibration.** Phase 5 will report metrics
|
||||
stratified by failure-mode family. If a family fails on the
|
||||
single `delta`, the finding is architectural and surfaces in
|
||||
Phase 5; this ADR explicitly forbids per-family tuned constants
|
||||
as a fix.
|
||||
* **CLI flags.** RuntimeConfig + ChatRuntime is the wired surface.
|
||||
A `--admissibility-mode` flag is a UX follow-up, not load-bearing
|
||||
for the contract.
|
||||
|
||||
## Risks
|
||||
|
||||
* **A single `delta` may fail on a future failure-mode family.**
|
||||
Mitigation: the default is falsifiable (Phase 5 will diversify);
|
||||
refusal is honest (`InnerLoopExhaustion` carries the ranking, so
|
||||
the failure mode is visible in the trace). Patching with
|
||||
per-family constants is explicitly out of scope.
|
||||
* **Margin mode changes selection semantics.** Threshold-mode
|
||||
acceptance evidence does not transfer. Mitigation: default off;
|
||||
separate test suite (`tests/test_margin_admissibility.py`) pins
|
||||
margin contract independently.
|
||||
* **Cost.** Margin mode evaluates `cga_inner` for every candidate
|
||||
in the admissible set, not just the field-closest one as
|
||||
threshold mode does. In practice the admissible set is small
|
||||
(chain length / v2 case size) so this is bounded. No
|
||||
approximation added.
|
||||
|
||||
## Rollback
|
||||
|
||||
Set `admissibility_mode = "threshold"` (the default) at every call
|
||||
site. The threshold path is unchanged from ADR-0024. No trace_hash
|
||||
migration required for non-refused turns.
|
||||
|
|
@ -405,6 +405,190 @@ def check_transition(
|
|||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------
|
||||
# Phase 3 — Ranked admissibility with margin (ADR-0026)
|
||||
# ----------------------------------------------------------------------
|
||||
#
|
||||
# Replaces ``score >= threshold`` with
|
||||
# ``score(top) - score(second) >= delta`` over the candidates in the
|
||||
# admissible set, ranked by ``cga_inner(versor, relation_blade)``
|
||||
# descending with strict ``>`` tie-break (deterministic on ascending
|
||||
# vocab index for ties).
|
||||
#
|
||||
# Phase 4 characterization (recorded in
|
||||
# ``tests/test_inner_loop_phase4.py``) showed that no single global
|
||||
# threshold separates the v2 mechanism-isolation cases because blade
|
||||
# norms vary ~10x across cases. Margins are scale-invariant under
|
||||
# blade-norm variation — the gap between top and second-ranked is
|
||||
# proportional to the blade norm, so the *relative* admissibility
|
||||
# ordering is what the geometry actually delivers. A single per-
|
||||
# runtime ``delta`` is therefore meaningful in a way that a single
|
||||
# ``tau`` is not.
|
||||
#
|
||||
# Selection within margin mode is blade-rank-driven: the top-ranked
|
||||
# admissible candidate IS the admitted destination. This differs
|
||||
# from threshold mode where ``_nearest_next`` (field-driven) picks
|
||||
# and ``check_transition`` gates. The mode is opt-in; threshold
|
||||
# mode remains the default to preserve ADR-0024 acceptance evidence.
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RankedCandidate:
|
||||
"""One row in the blade-score ranking of an admissible candidate set."""
|
||||
|
||||
index: int
|
||||
word: str
|
||||
score: float
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class MarginVerdict:
|
||||
"""Result of margin-based admissibility check on a ranked candidate set.
|
||||
|
||||
Carries the admit/reject verdict, the top-ranked candidate, the
|
||||
margin between top and second-ranked, and the full ranked list so
|
||||
refusal evidence (in ``InnerLoopExhaustion.rejected_attempts``)
|
||||
can carry the entire ordering at the failed step rather than a
|
||||
single rejected score.
|
||||
"""
|
||||
|
||||
admitted: bool
|
||||
top: RankedCandidate | None
|
||||
margin: float
|
||||
delta: float
|
||||
region_label: str
|
||||
ranked: tuple[RankedCandidate, ...] = ()
|
||||
reason: str = ""
|
||||
|
||||
|
||||
def rank_candidates_by_blade(
|
||||
region: AdmissibilityRegion,
|
||||
*,
|
||||
candidate_indices: np.ndarray | None,
|
||||
versor_lookup,
|
||||
word_lookup,
|
||||
) -> tuple[RankedCandidate, ...]:
|
||||
"""Rank candidates by ``cga_inner(versor, relation_blade)`` desc.
|
||||
|
||||
``candidate_indices`` is the post-region-filter candidate set
|
||||
(already intersected with ``region.allowed_indices`` upstream).
|
||||
``versor_lookup(idx) -> np.ndarray`` and
|
||||
``word_lookup(idx) -> str`` are vocab accessors hoisted into
|
||||
parameters so this function has no I/O and can be unit-tested
|
||||
with a stub vocab.
|
||||
|
||||
Tie-break: strict ``>`` on score, with ascending ``index`` as the
|
||||
deterministic secondary key. This matches the ``vocab.nearest``
|
||||
strict-``>`` convention documented as load-bearing in ADR-0024.
|
||||
"""
|
||||
if candidate_indices is None or len(candidate_indices) == 0:
|
||||
return ()
|
||||
blade_norm = float(np.linalg.norm(region.relation_blade))
|
||||
if blade_norm < _NULL_TOLERANCE:
|
||||
# No blade constraint — every candidate scores 0. Margin
|
||||
# cannot separate them; the caller should not enter margin
|
||||
# mode on an unconstrained blade. Return ranking in vocab
|
||||
# index order so behaviour is at least deterministic.
|
||||
rows = [
|
||||
RankedCandidate(index=int(i), word=str(word_lookup(int(i))), score=0.0)
|
||||
for i in candidate_indices
|
||||
]
|
||||
return tuple(rows)
|
||||
rows: list[RankedCandidate] = []
|
||||
for idx in candidate_indices:
|
||||
v = np.asarray(versor_lookup(int(idx)), dtype=np.float32)
|
||||
s = float(cga_inner(v, region.relation_blade))
|
||||
rows.append(
|
||||
RankedCandidate(index=int(idx), word=str(word_lookup(int(idx))), score=s)
|
||||
)
|
||||
# Sort descending by score, ascending by index for tie-break.
|
||||
rows.sort(key=lambda r: (-r.score, r.index))
|
||||
return tuple(rows)
|
||||
|
||||
|
||||
def check_margin(
|
||||
region: AdmissibilityRegion,
|
||||
ranked: tuple[RankedCandidate, ...],
|
||||
*,
|
||||
delta: float,
|
||||
) -> MarginVerdict:
|
||||
"""Admit the top-ranked candidate iff it has a clean margin.
|
||||
|
||||
Admission requires:
|
||||
|
||||
1. The ranking is non-empty (``len(ranked) >= 1``).
|
||||
2. ``ranked[0].score > 0`` — basic positivity in the blade
|
||||
half-space. A non-positive top score means the admissible
|
||||
set has no blade-aligned candidate at all; refuse.
|
||||
3. If ``len(ranked) >= 2``: ``ranked[0].score - ranked[1].score
|
||||
>= delta``. The top must out-score the next-best by at
|
||||
least ``delta``.
|
||||
4. If ``len(ranked) == 1``: trivially admit (no competitor to
|
||||
confuse the boundary's pick).
|
||||
|
||||
A single ``delta`` works across blades of varying norm because
|
||||
the margin scales with blade norm — the *relative* gap, not the
|
||||
absolute score, is what carries semantic separation.
|
||||
"""
|
||||
if not ranked:
|
||||
return MarginVerdict(
|
||||
admitted=False,
|
||||
top=None,
|
||||
margin=float("-inf"),
|
||||
delta=delta,
|
||||
region_label=region.label,
|
||||
ranked=(),
|
||||
reason="empty ranking",
|
||||
)
|
||||
top = ranked[0]
|
||||
if top.score <= 0.0:
|
||||
return MarginVerdict(
|
||||
admitted=False,
|
||||
top=top,
|
||||
margin=float("-inf"),
|
||||
delta=delta,
|
||||
region_label=region.label,
|
||||
ranked=ranked,
|
||||
reason=f"top score {top.score:.6f} not positive",
|
||||
)
|
||||
if len(ranked) == 1:
|
||||
# Single admissible candidate — no margin to compute.
|
||||
return MarginVerdict(
|
||||
admitted=True,
|
||||
top=top,
|
||||
margin=float("inf"),
|
||||
delta=delta,
|
||||
region_label=region.label,
|
||||
ranked=ranked,
|
||||
reason="ok (single admissible)",
|
||||
)
|
||||
second = ranked[1]
|
||||
margin = top.score - second.score
|
||||
if margin < delta:
|
||||
return MarginVerdict(
|
||||
admitted=False,
|
||||
top=top,
|
||||
margin=margin,
|
||||
delta=delta,
|
||||
region_label=region.label,
|
||||
ranked=ranked,
|
||||
reason=(
|
||||
f"margin {margin:.6f} below delta {delta:.6f} "
|
||||
f"(top={top.word!r}@{top.score:.6f}, "
|
||||
f"next={second.word!r}@{second.score:.6f})"
|
||||
),
|
||||
)
|
||||
return MarginVerdict(
|
||||
admitted=True,
|
||||
top=top,
|
||||
margin=margin,
|
||||
delta=delta,
|
||||
region_label=region.label,
|
||||
ranked=ranked,
|
||||
reason="ok",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AdmissibilityTraceStep:
|
||||
"""One per-transition record from a constrained walk (ADR-0023 §2).
|
||||
|
|
|
|||
|
|
@ -23,8 +23,12 @@ from generate.admissibility import (
|
|||
AdmissibilityRegion,
|
||||
AdmissibilityTraceStep,
|
||||
AdmissibilityVerdict,
|
||||
MarginVerdict,
|
||||
RankedCandidate,
|
||||
check_margin,
|
||||
check_transition,
|
||||
filter_candidates,
|
||||
rank_candidates_by_blade,
|
||||
)
|
||||
from generate.attention import AttentionOperator
|
||||
from generate.exhaustion import InnerLoopExhaustion, RefusalReason
|
||||
|
|
@ -281,6 +285,8 @@ def generate(
|
|||
inner_loop_admissibility: bool = False,
|
||||
admissibility_threshold: float = 0.0,
|
||||
inner_loop_force_admit: bool = False,
|
||||
admissibility_mode: str = "threshold",
|
||||
admissibility_margin: float = 0.4,
|
||||
) -> GenerationResult:
|
||||
"""Generate a token sequence.
|
||||
|
||||
|
|
@ -375,6 +381,18 @@ def generate(
|
|||
region_active = region is not None and not region.is_unconstrained()
|
||||
active_region: AdmissibilityRegion | None = region if region_active else None
|
||||
inner_loop_active = inner_loop_admissibility and region_active
|
||||
# ADR-0026 / Phase 3 — margin mode is opt-in. When active it
|
||||
# replaces the per-candidate threshold check with a ranked
|
||||
# admissibility test on the candidate set, admitting the top
|
||||
# blade-ranked candidate iff its margin over the second-ranked
|
||||
# is at least ``admissibility_margin``. Falls back to threshold
|
||||
# mode (ADR-0024) on any unrecognised value so a config typo
|
||||
# cannot silently disable admissibility.
|
||||
margin_mode_active = (
|
||||
inner_loop_active
|
||||
and admissibility_mode == "margin"
|
||||
and active_region is not None
|
||||
)
|
||||
for step_index in range(token_budget):
|
||||
current, hits_applied = _recall_state(_voiced_state(current, persona), vault, recall_top_k)
|
||||
vault_hits += hits_applied
|
||||
|
|
@ -388,70 +406,115 @@ def generate(
|
|||
word_idx: int
|
||||
verdict: AdmissibilityVerdict
|
||||
|
||||
max_attempts = (
|
||||
len(candidate_indices) if (inner_loop_active and candidate_indices is not None)
|
||||
else 1
|
||||
)
|
||||
for _attempt in range(max(max_attempts, 1)):
|
||||
word, word_idx = _nearest_next(
|
||||
vocab,
|
||||
current.F,
|
||||
current.node,
|
||||
recent_nodes=tuple(recent_nodes),
|
||||
stop_nodes=stop_nodes | frozenset(step_exclude),
|
||||
if margin_mode_active:
|
||||
# ADR-0026 / Phase 3 — rank the admissible candidate set by
|
||||
# blade-score and admit the top iff margin >= delta. The
|
||||
# rotor V is only constructed for the admitted candidate
|
||||
# below, preserving the versor_condition invariant.
|
||||
assert active_region is not None # margin_mode_active gates this
|
||||
ranked = rank_candidates_by_blade(
|
||||
active_region,
|
||||
candidate_indices=candidate_indices,
|
||||
versor_lookup=vocab.get_versor_at,
|
||||
word_lookup=vocab.get_word_at,
|
||||
)
|
||||
if active_region is not None:
|
||||
verdict = check_transition(
|
||||
active_region,
|
||||
candidate_index=int(word_idx),
|
||||
candidate_versor=vocab.get_versor_at(word_idx),
|
||||
threshold=admissibility_threshold,
|
||||
)
|
||||
else:
|
||||
verdict = AdmissibilityVerdict(
|
||||
admitted=True,
|
||||
score=0.0,
|
||||
region_label=effective_region_label,
|
||||
reason="unconstrained",
|
||||
)
|
||||
if not inner_loop_active or verdict.admitted or inner_loop_force_admit:
|
||||
# `inner_loop_force_admit` is the Phase 2 null control:
|
||||
# exercises the inner-loop code path (same attempt loop,
|
||||
# same telemetry side effects) but force-breaks on the
|
||||
# first candidate so any pass-rate delta vs the true
|
||||
# inner-loop run is causally attributable to rejection,
|
||||
# not to incidental code-path differences.
|
||||
break
|
||||
# Inner loop is on and verdict rejected this candidate.
|
||||
rejected_attempts.append((int(word_idx), str(word), float(verdict.score)))
|
||||
if int(word_idx) in step_exclude:
|
||||
# Selector returned the same exhausted candidate — no
|
||||
# further admissible destinations. Honest refusal.
|
||||
# ADR-0024 Phase 2 — in-walk exhaustion site; carries the
|
||||
# ordered ``rejected_attempts`` accumulated this step so
|
||||
# downstream layers can read refusal evidence without
|
||||
# re-parsing the exception message.
|
||||
margin_verdict = check_margin(
|
||||
active_region, ranked, delta=admissibility_margin
|
||||
)
|
||||
# rejected_attempts carries the full ranked list as evidence
|
||||
# — index, word, score — so refusal traces show the entire
|
||||
# blade-ordering at the failed step, not just one rejection.
|
||||
rejected_attempts = [
|
||||
(r.index, r.word, r.score) for r in margin_verdict.ranked
|
||||
]
|
||||
if not margin_verdict.admitted:
|
||||
raise InnerLoopExhaustion(
|
||||
reason=RefusalReason.INNER_LOOP_EXHAUSTION,
|
||||
region_label=effective_region_label,
|
||||
step_index=step_index,
|
||||
rejected_attempts=tuple(rejected_attempts),
|
||||
)
|
||||
step_exclude.add(int(word_idx))
|
||||
else:
|
||||
# max_attempts exhausted without break — every admissible
|
||||
# candidate was rejected by the inner-loop threshold.
|
||||
# Same refusal shape as the same-candidate-loop site above;
|
||||
# both are structurally "inner-loop produced no admissible
|
||||
# candidate at this step". Splitting into separate reasons
|
||||
# can wait for Phase 4 (rotor frame, ADR-0025).
|
||||
raise InnerLoopExhaustion(
|
||||
reason=RefusalReason.INNER_LOOP_EXHAUSTION,
|
||||
region_label=effective_region_label,
|
||||
step_index=step_index,
|
||||
rejected_attempts=tuple(rejected_attempts),
|
||||
assert margin_verdict.top is not None # admitted => non-None
|
||||
word_idx = int(margin_verdict.top.index)
|
||||
word = str(margin_verdict.top.word)
|
||||
# Build a legacy AdmissibilityVerdict for trace storage so
|
||||
# the AdmissibilityTraceStep shape is unchanged. Margin
|
||||
# info is encoded into ``reason`` for human inspection;
|
||||
# the structured ranking lives in ``rejected_attempts``.
|
||||
verdict = AdmissibilityVerdict(
|
||||
admitted=True,
|
||||
score=float(margin_verdict.top.score),
|
||||
region_label=margin_verdict.region_label,
|
||||
reason=(
|
||||
f"margin {margin_verdict.margin:.6f} >= "
|
||||
f"delta {margin_verdict.delta:.6f}"
|
||||
),
|
||||
)
|
||||
else:
|
||||
max_attempts = (
|
||||
len(candidate_indices) if (inner_loop_active and candidate_indices is not None)
|
||||
else 1
|
||||
)
|
||||
for _attempt in range(max(max_attempts, 1)):
|
||||
word, word_idx = _nearest_next(
|
||||
vocab,
|
||||
current.F,
|
||||
current.node,
|
||||
recent_nodes=tuple(recent_nodes),
|
||||
stop_nodes=stop_nodes | frozenset(step_exclude),
|
||||
candidate_indices=candidate_indices,
|
||||
)
|
||||
if active_region is not None:
|
||||
verdict = check_transition(
|
||||
active_region,
|
||||
candidate_index=int(word_idx),
|
||||
candidate_versor=vocab.get_versor_at(word_idx),
|
||||
threshold=admissibility_threshold,
|
||||
)
|
||||
else:
|
||||
verdict = AdmissibilityVerdict(
|
||||
admitted=True,
|
||||
score=0.0,
|
||||
region_label=effective_region_label,
|
||||
reason="unconstrained",
|
||||
)
|
||||
if not inner_loop_active or verdict.admitted or inner_loop_force_admit:
|
||||
# `inner_loop_force_admit` is the Phase 2 null control:
|
||||
# exercises the inner-loop code path (same attempt loop,
|
||||
# same telemetry side effects) but force-breaks on the
|
||||
# first candidate so any pass-rate delta vs the true
|
||||
# inner-loop run is causally attributable to rejection,
|
||||
# not to incidental code-path differences.
|
||||
break
|
||||
# Inner loop is on and verdict rejected this candidate.
|
||||
rejected_attempts.append((int(word_idx), str(word), float(verdict.score)))
|
||||
if int(word_idx) in step_exclude:
|
||||
# Selector returned the same exhausted candidate — no
|
||||
# further admissible destinations. Honest refusal.
|
||||
# ADR-0024 Phase 2 — in-walk exhaustion site; carries the
|
||||
# ordered ``rejected_attempts`` accumulated this step so
|
||||
# downstream layers can read refusal evidence without
|
||||
# re-parsing the exception message.
|
||||
raise InnerLoopExhaustion(
|
||||
reason=RefusalReason.INNER_LOOP_EXHAUSTION,
|
||||
region_label=effective_region_label,
|
||||
step_index=step_index,
|
||||
rejected_attempts=tuple(rejected_attempts),
|
||||
)
|
||||
step_exclude.add(int(word_idx))
|
||||
else:
|
||||
# max_attempts exhausted without break — every admissible
|
||||
# candidate was rejected by the inner-loop threshold.
|
||||
# Same refusal shape as the same-candidate-loop site above;
|
||||
# both are structurally "inner-loop produced no admissible
|
||||
# candidate at this step". Splitting into separate reasons
|
||||
# can wait for Phase 4 (rotor frame, ADR-0025).
|
||||
raise InnerLoopExhaustion(
|
||||
reason=RefusalReason.INNER_LOOP_EXHAUSTION,
|
||||
region_label=effective_region_label,
|
||||
step_index=step_index,
|
||||
rejected_attempts=tuple(rejected_attempts),
|
||||
)
|
||||
|
||||
tokens.append(_articulate(vocab, word))
|
||||
admissibility_trace.append(
|
||||
|
|
|
|||
322
tests/test_margin_admissibility.py
Normal file
322
tests/test_margin_admissibility.py
Normal file
|
|
@ -0,0 +1,322 @@
|
|||
"""Phase 3 / ADR-0026 — ranked-with-margin admissibility contract.
|
||||
|
||||
These tests pin the new admissibility shape:
|
||||
|
||||
* ``rank_candidates_by_blade`` returns the admissible set sorted
|
||||
by ``cga_inner(versor, blade)`` descending, with strict ``>``
|
||||
tie-break (ascending vocab index for ties).
|
||||
* ``check_margin`` admits the top-ranked candidate iff
|
||||
``top_score > 0`` AND ``top_score - second_score >= delta``.
|
||||
* Margin mode wired through ``generate()`` produces the correct
|
||||
endpoint on v2 mechanism-isolation cases and emits
|
||||
``InnerLoopExhaustion`` when the margin is not met — never a
|
||||
silent boundary fallback.
|
||||
* Near-equal-score candidates resolve deterministically across
|
||||
repeated runs (replay invariance).
|
||||
* Threshold mode (ADR-0024) is unchanged by margin-mode plumbing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from chat.runtime import ChatRuntime
|
||||
from field.state import FieldState
|
||||
from generate.admissibility import (
|
||||
AdmissibilityRegion,
|
||||
RankedCandidate,
|
||||
RegionSource,
|
||||
check_margin,
|
||||
rank_candidates_by_blade,
|
||||
)
|
||||
from generate.exhaustion import InnerLoopExhaustion, RefusalReason
|
||||
from generate.stream import generate
|
||||
|
||||
_BLADE_DIM = 32
|
||||
|
||||
|
||||
def _zero_blade() -> np.ndarray:
|
||||
return np.zeros(_BLADE_DIM, dtype=np.float32)
|
||||
|
||||
|
||||
def _region_with_blade(allowed: list[int], blade: np.ndarray, label: str = "test") -> AdmissibilityRegion:
|
||||
return AdmissibilityRegion(
|
||||
allowed_indices=np.asarray(allowed, dtype=np.int64),
|
||||
relation_blade=blade.astype(np.float32),
|
||||
source=RegionSource.RELATION,
|
||||
label=label,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# rank_candidates_by_blade
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestRankCandidates:
|
||||
def test_ranks_descending_by_score(self) -> None:
|
||||
blade = _zero_blade()
|
||||
blade[0] = 1.0 # non-zero blade with self-inner != 0 (depends on metric)
|
||||
# Build versors with known scores; we'll just construct them and
|
||||
# verify the *ordering*, not the exact score values.
|
||||
from algebra.cga import cga_inner
|
||||
|
||||
bb = float(cga_inner(blade, blade))
|
||||
if abs(bb) < 1e-9:
|
||||
pytest.skip("blade self-inner is zero for chosen test blade")
|
||||
versors = {
|
||||
10: (0.5 / bb) * blade,
|
||||
20: (1.0 / bb) * blade,
|
||||
30: (0.2 / bb) * blade,
|
||||
}
|
||||
words = {10: "alpha", 20: "beta", 30: "gamma"}
|
||||
region = _region_with_blade([10, 20, 30], blade)
|
||||
ranked = rank_candidates_by_blade(
|
||||
region,
|
||||
candidate_indices=np.asarray([10, 20, 30], dtype=np.int64),
|
||||
versor_lookup=lambda i: versors[int(i)],
|
||||
word_lookup=lambda i: words[int(i)],
|
||||
)
|
||||
# Sorted desc by score → indices [20, 10, 30]
|
||||
assert [r.index for r in ranked] == [20, 10, 30]
|
||||
# Strict descending scores
|
||||
assert ranked[0].score > ranked[1].score > ranked[2].score
|
||||
|
||||
def test_strict_tie_break_by_ascending_index(self) -> None:
|
||||
"""When two candidates have the *same* score, ascending vocab
|
||||
index wins. This pins the determinism contract."""
|
||||
blade = _zero_blade()
|
||||
blade[0] = 1.0
|
||||
from algebra.cga import cga_inner
|
||||
|
||||
bb = float(cga_inner(blade, blade))
|
||||
if abs(bb) < 1e-9:
|
||||
pytest.skip("blade self-inner is zero for chosen test blade")
|
||||
# Two equal scores at indices 30 and 10 → 10 should win.
|
||||
versors = {
|
||||
10: (1.0 / bb) * blade,
|
||||
30: (1.0 / bb) * blade,
|
||||
20: (0.5 / bb) * blade,
|
||||
}
|
||||
words = {10: "alpha", 20: "beta", 30: "gamma"}
|
||||
region = _region_with_blade([10, 20, 30], blade)
|
||||
ranked = rank_candidates_by_blade(
|
||||
region,
|
||||
candidate_indices=np.asarray([10, 20, 30], dtype=np.int64),
|
||||
versor_lookup=lambda i: versors[int(i)],
|
||||
word_lookup=lambda i: words[int(i)],
|
||||
)
|
||||
# Both tied candidates are at the top; lower index comes first.
|
||||
assert ranked[0].index == 10
|
||||
assert ranked[1].index == 30
|
||||
# Determinism across repeats
|
||||
for _ in range(5):
|
||||
r = rank_candidates_by_blade(
|
||||
region,
|
||||
candidate_indices=np.asarray([10, 20, 30], dtype=np.int64),
|
||||
versor_lookup=lambda i: versors[int(i)],
|
||||
word_lookup=lambda i: words[int(i)],
|
||||
)
|
||||
assert tuple(c.index for c in r) == tuple(c.index for c in ranked)
|
||||
|
||||
def test_empty_candidate_set_returns_empty(self) -> None:
|
||||
blade = _zero_blade()
|
||||
blade[0] = 1.0
|
||||
region = _region_with_blade([1], blade)
|
||||
assert (
|
||||
rank_candidates_by_blade(
|
||||
region,
|
||||
candidate_indices=np.asarray([], dtype=np.int64),
|
||||
versor_lookup=lambda i: np.zeros(_BLADE_DIM, dtype=np.float32),
|
||||
word_lookup=lambda i: "",
|
||||
)
|
||||
== ()
|
||||
)
|
||||
|
||||
def test_zero_blade_returns_zero_scores_index_order(self) -> None:
|
||||
"""An unconstrained-direction region returns all candidates at
|
||||
score 0, in vocab-index order. The caller should not enter
|
||||
margin mode here; this test pins the safe fallback."""
|
||||
region = _region_with_blade([3, 1, 2], _zero_blade())
|
||||
# allowed_indices is canonicalised to sorted unique → [1,2,3].
|
||||
ranked = rank_candidates_by_blade(
|
||||
region,
|
||||
candidate_indices=region.allowed_indices,
|
||||
versor_lookup=lambda i: np.zeros(_BLADE_DIM, dtype=np.float32),
|
||||
word_lookup=lambda i: f"w{i}",
|
||||
)
|
||||
assert [r.index for r in ranked] == [1, 2, 3]
|
||||
assert all(r.score == 0.0 for r in ranked)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# check_margin
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCheckMargin:
|
||||
def _ranked(self, *triples: tuple[int, str, float]) -> tuple[RankedCandidate, ...]:
|
||||
return tuple(RankedCandidate(index=i, word=w, score=s) for i, w, s in triples)
|
||||
|
||||
def test_admits_when_margin_meets_delta(self) -> None:
|
||||
ranked = self._ranked((1, "a", 1.5), (2, "b", 0.8))
|
||||
region = _region_with_blade([1, 2], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||
verdict = check_margin(region, ranked, delta=0.5)
|
||||
assert verdict.admitted is True
|
||||
assert verdict.top is not None and verdict.top.index == 1
|
||||
assert pytest.approx(verdict.margin, abs=1e-9) == 0.7
|
||||
|
||||
def test_refuses_when_margin_below_delta(self) -> None:
|
||||
ranked = self._ranked((1, "a", 1.0), (2, "b", 0.8))
|
||||
region = _region_with_blade([1, 2], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||
verdict = check_margin(region, ranked, delta=0.5)
|
||||
assert verdict.admitted is False
|
||||
assert "margin" in verdict.reason
|
||||
assert pytest.approx(verdict.margin, abs=1e-9) == pytest.approx(0.2)
|
||||
|
||||
def test_refuses_when_top_score_not_positive(self) -> None:
|
||||
"""Even a clean margin doesn't save a non-positive top score:
|
||||
the admissible set has no blade-aligned candidate at all."""
|
||||
ranked = self._ranked((1, "a", -0.5), (2, "b", -2.0))
|
||||
region = _region_with_blade([1, 2], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||
verdict = check_margin(region, ranked, delta=0.5)
|
||||
assert verdict.admitted is False
|
||||
assert "not positive" in verdict.reason
|
||||
|
||||
def test_single_candidate_trivially_admitted_when_positive(self) -> None:
|
||||
ranked = self._ranked((1, "a", 0.5))
|
||||
region = _region_with_blade([1], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||
verdict = check_margin(region, ranked, delta=999.0)
|
||||
assert verdict.admitted is True
|
||||
assert verdict.margin == float("inf")
|
||||
assert "single admissible" in verdict.reason
|
||||
|
||||
def test_empty_ranking_refuses(self) -> None:
|
||||
region = _region_with_blade([1], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||
verdict = check_margin(region, (), delta=0.4)
|
||||
assert verdict.admitted is False
|
||||
assert verdict.top is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# generate() in margin mode — integration via v2-like setup
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _v2_state_and_region(rt: ChatRuntime, *, seed: str, admissible: list[str], blade_tok: str, label: str):
|
||||
vocab = rt.session.vocab
|
||||
idx = vocab.index_of(seed)
|
||||
F = np.asarray(vocab.get_versor(seed), dtype=np.float32)
|
||||
state = FieldState(F=F.copy(), node=idx, step=0)
|
||||
indices = np.asarray([vocab.index_of(t) for t in admissible], dtype=np.int64)
|
||||
blade = np.asarray(vocab.get_versor(blade_tok), dtype=np.float32)
|
||||
region = AdmissibilityRegion(
|
||||
allowed_indices=indices,
|
||||
relation_blade=blade,
|
||||
source=RegionSource.RELATION,
|
||||
label=label,
|
||||
)
|
||||
return state, region
|
||||
|
||||
|
||||
class TestGenerateMarginMode:
|
||||
"""End-to-end: margin mode on v2-style mechanism-isolation cases."""
|
||||
|
||||
def test_v2_001_question_admitted_via_margin(self) -> None:
|
||||
rt = ChatRuntime()
|
||||
state, region = _v2_state_and_region(
|
||||
rt, seed="symbol", admissible=["answer", "question"],
|
||||
blade_tok="question", label="v2-001",
|
||||
)
|
||||
result = generate(
|
||||
state, rt.session.vocab, rt.session.persona,
|
||||
max_tokens=1, region=region,
|
||||
inner_loop_admissibility=True,
|
||||
admissibility_mode="margin",
|
||||
admissibility_margin=0.4,
|
||||
)
|
||||
assert result.tokens == ("question",)
|
||||
step = result.admissibility_trace[0]
|
||||
assert step.verdict.admitted is True
|
||||
assert "margin" in step.verdict.reason
|
||||
# rejected_attempts now carries the *full* ranking
|
||||
assert len(step.rejected_attempts) >= 2
|
||||
words = [w for (_i, w, _s) in step.rejected_attempts]
|
||||
assert "question" in words and "answer" in words
|
||||
|
||||
def test_v2_001_refuses_when_delta_too_high(self) -> None:
|
||||
"""The v2-001 margin is ≈0.597. Setting delta=0.9 must trigger
|
||||
honest refusal — no silent boundary fallback."""
|
||||
rt = ChatRuntime()
|
||||
state, region = _v2_state_and_region(
|
||||
rt, seed="symbol", admissible=["answer", "question"],
|
||||
blade_tok="question", label="v2-001",
|
||||
)
|
||||
with pytest.raises(InnerLoopExhaustion) as exc_info:
|
||||
generate(
|
||||
state, rt.session.vocab, rt.session.persona,
|
||||
max_tokens=1, region=region,
|
||||
inner_loop_admissibility=True,
|
||||
admissibility_mode="margin",
|
||||
admissibility_margin=0.9,
|
||||
)
|
||||
exc = exc_info.value
|
||||
assert exc.reason is RefusalReason.INNER_LOOP_EXHAUSTION
|
||||
assert exc.region_label == "v2-001"
|
||||
# Refusal carries the ranking as evidence
|
||||
words = [w for (_i, w, _s) in exc.rejected_attempts]
|
||||
assert "question" in words and "answer" in words
|
||||
|
||||
def test_threshold_mode_unchanged_by_margin_plumbing(self) -> None:
|
||||
"""Threshold mode (the ADR-0024 default) must produce the same
|
||||
result whether admissibility_mode is "threshold" or unset."""
|
||||
rt = ChatRuntime()
|
||||
state, region = _v2_state_and_region(
|
||||
rt, seed="symbol", admissible=["answer", "question"],
|
||||
blade_tok="question", label="v2-001",
|
||||
)
|
||||
r1 = generate(
|
||||
state, rt.session.vocab, rt.session.persona,
|
||||
max_tokens=1, region=region,
|
||||
inner_loop_admissibility=True,
|
||||
admissibility_threshold=1.122,
|
||||
)
|
||||
r2 = generate(
|
||||
state, rt.session.vocab, rt.session.persona,
|
||||
max_tokens=1, region=region,
|
||||
inner_loop_admissibility=True,
|
||||
admissibility_threshold=1.122,
|
||||
admissibility_mode="threshold",
|
||||
)
|
||||
assert r1.tokens == r2.tokens
|
||||
assert r1.tokens == ("question",)
|
||||
|
||||
|
||||
class TestMarginModeDeterminism:
|
||||
"""5 reruns of the same margin-mode case produce identical traces."""
|
||||
|
||||
def test_margin_mode_replay_stable_across_5_runs(self) -> None:
|
||||
rt = ChatRuntime()
|
||||
state, region = _v2_state_and_region(
|
||||
rt, seed="symbol", admissible=["answer", "question"],
|
||||
blade_tok="question", label="v2-001",
|
||||
)
|
||||
first = generate(
|
||||
state, rt.session.vocab, rt.session.persona,
|
||||
max_tokens=1, region=region,
|
||||
inner_loop_admissibility=True,
|
||||
admissibility_mode="margin",
|
||||
admissibility_margin=0.4,
|
||||
)
|
||||
first_canonical = first.admissibility_trace[0].canonical()
|
||||
for _ in range(4):
|
||||
r = generate(
|
||||
state, rt.session.vocab, rt.session.persona,
|
||||
max_tokens=1, region=region,
|
||||
inner_loop_admissibility=True,
|
||||
admissibility_mode="margin",
|
||||
admissibility_margin=0.4,
|
||||
)
|
||||
assert r.tokens == first.tokens
|
||||
assert r.admissibility_trace[0].canonical() == first_canonical
|
||||
Loading…
Reference in a new issue