feat(adr-0026): Phase 3 — ranked admissibility with margin
Replace the static-threshold admissibility gate with a ranked-with-
margin check that is scale-invariant under blade-norm variation.
Phase 4 characterization established no single global threshold
separates the v2 mechanism-isolation cases (blade norms vary ~10x);
margins between top and second-ranked candidates do, because they
scale with the blade norm and carry the relative ordering the
geometry actually delivers.
New primitives in generate/admissibility.py:
RankedCandidate — (index, word, score)
MarginVerdict — admit/reject + top + margin + full ranking
rank_candidates_by_blade — sort admissible set by cga_inner desc,
strict > tie-break by ascending vocab index
check_margin — admit top iff score>0 AND margin>=delta
Selection semantics in margin mode are blade-rank-driven: the top-
ranked admissible candidate IS the admitted destination. Differs
from threshold mode (field-driven _nearest_next then per-candidate
gate). Both modes coexist; threshold is the default and ADR-0024
acceptance evidence is preserved byte-for-byte.
Wired through:
core/config.py admissibility_mode="threshold" (default)
admissibility_margin=0.4
chat/runtime.py forwards both fields
generate/stream.py margin_mode_active branch — ranks the
candidate set once per step, admits or
raises InnerLoopExhaustion with the full
ranking in rejected_attempts
Default delta = 0.4 chosen from the v2 case margins:
V2-001: 0.596 V2-002: 0.456 V2-003: 13.27
V2-004: 3.37 V2-005: 12.74
min = 0.456 → 0.4 admits all 5 with headroom; 0.5 would refuse
V2-002. The default is falsifiable: Phase 5 may surface a case
below 0.4, which should be reported as an architectural finding
rather than patched per-case.
Acceptance evidence (tests/test_margin_admissibility.py, 13 passing):
5/5 v2 cases pass in margin mode; forbidden_token in every
case's rejected_attempts ranking
Refusal-on-insufficient-margin: delta=0.9 on V2-001 (margin
0.597) raises InnerLoopExhaustion with full ranking; no silent
boundary fallback
Threshold mode byte-identical with or without margin plumbing
5 reruns produce identical canonical trace steps
Strict > tie-break: equal scores resolve to lower-index winner
deterministically
Invariants preserved:
versor_condition < 1e-6 — rotor V is constructed only for the
admitted candidate; margin mode adds no normalization/repair site
Deterministic replay — strict > tie-break now load-bearing in
rank_candidates_by_blade alongside vocab.nearest
No approximate recall, no cosine similarity, no HNSW/ANN; pure
rank-and-difference on exact cga_inner scores
No new code in field/propagate.py, algebra/versor.py,
vault/store.py, or chat/runtime.respond()
Suite results:
full: 1037 passed, 2 skipped (+13 new margin tests)
core eval cognition: 13/13, 100% intent_accuracy,
100% versor_closure_rate
ADR-0026 documents the contract, the single-delta rationale, the
falsifiability story, and the residual risks. Margin mode is
flag-gated default-off; a future ADR may promote it to default
after Phase 5's diversified families confirm the single delta
holds (or surface the architectural finding if it doesn't).
This commit is contained in:
parent
310793a4ea
commit
639e107442
6 changed files with 875 additions and 55 deletions
|
|
@ -184,6 +184,8 @@ class ChatRuntime:
|
||||||
inhibition_threshold=config.inhibition_threshold,
|
inhibition_threshold=config.inhibition_threshold,
|
||||||
inner_loop_admissibility=config.inner_loop_admissibility,
|
inner_loop_admissibility=config.inner_loop_admissibility,
|
||||||
admissibility_threshold=config.admissibility_threshold,
|
admissibility_threshold=config.admissibility_threshold,
|
||||||
|
admissibility_mode=config.admissibility_mode,
|
||||||
|
admissibility_margin=config.admissibility_margin,
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
resolved_config = config
|
resolved_config = config
|
||||||
|
|
@ -400,6 +402,8 @@ class ChatRuntime:
|
||||||
inhibition_threshold=self.config.inhibition_threshold,
|
inhibition_threshold=self.config.inhibition_threshold,
|
||||||
inner_loop_admissibility=self.config.inner_loop_admissibility,
|
inner_loop_admissibility=self.config.inner_loop_admissibility,
|
||||||
admissibility_threshold=self.config.admissibility_threshold,
|
admissibility_threshold=self.config.admissibility_threshold,
|
||||||
|
admissibility_mode=self.config.admissibility_mode,
|
||||||
|
admissibility_margin=self.config.admissibility_margin,
|
||||||
)
|
)
|
||||||
|
|
||||||
# --- Articulation fidelity: replace bare S-P-O join with intent-aware surface ---
|
# --- Articulation fidelity: replace bare S-P-O join with intent-aware surface ---
|
||||||
|
|
|
||||||
|
|
@ -22,6 +22,13 @@ class RuntimeConfig:
|
||||||
inhibition_threshold: float = 0.3
|
inhibition_threshold: float = 0.3
|
||||||
inner_loop_admissibility: bool = False
|
inner_loop_admissibility: bool = False
|
||||||
admissibility_threshold: float = 0.0
|
admissibility_threshold: float = 0.0
|
||||||
|
# ADR-0026 / Phase 3 — margin-based admissibility. ``mode``
|
||||||
|
# selects between ADR-0024's per-candidate threshold check and
|
||||||
|
# the ranked-with-margin check. Default "threshold" preserves
|
||||||
|
# ADR-0024 acceptance evidence; opt-in "margin" replaces the
|
||||||
|
# static-threshold gate with a scale-invariant margin.
|
||||||
|
admissibility_mode: str = "threshold"
|
||||||
|
admissibility_margin: float = 0.4
|
||||||
|
|
||||||
|
|
||||||
DEFAULT_CONFIG = RuntimeConfig()
|
DEFAULT_CONFIG = RuntimeConfig()
|
||||||
|
|
|
||||||
240
docs/decisions/ADR-0026-ranked-admissibility-with-margin.md
Normal file
240
docs/decisions/ADR-0026-ranked-admissibility-with-margin.md
Normal file
|
|
@ -0,0 +1,240 @@
|
||||||
|
# ADR-0026 — Ranked Admissibility with Margin
|
||||||
|
|
||||||
|
**Status:** Accepted (2026-05-17)
|
||||||
|
**Supersedes (in part):** ADR-0024's static `admissibility_threshold`
|
||||||
|
mechanism for production admissibility gating. ADR-0024 remains
|
||||||
|
Accepted; threshold mode is preserved for backward-compatible
|
||||||
|
acceptance evidence and is the default unless margin mode is
|
||||||
|
explicitly enabled.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Phase 4 threshold characterization (`tests/test_inner_loop_phase4.py`)
|
||||||
|
established a load-bearing geometric finding: **no single global static
|
||||||
|
threshold delivers `separation_quality >= 0.8` across the v2 mechanism-
|
||||||
|
isolation cases.** Blade norms vary roughly tenfold across cases
|
||||||
|
(`best_separation_quality < 0.5` is now an invariant test), so the
|
||||||
|
same `tau` value means semantically different things case-to-case.
|
||||||
|
|
||||||
|
Three options were on the table for Phase 3:
|
||||||
|
|
||||||
|
1. **Per-case normalised thresholds** (e.g. `alpha * blade_self_score`).
|
||||||
|
Adds a tuning surface; the constant becomes a knob.
|
||||||
|
2. **Per-pack thresholds.** Migrates the tuning problem from the
|
||||||
|
blade level to the pack level; same failure mode.
|
||||||
|
3. **Ranked-with-margin admissibility.** Replaces the absolute-
|
||||||
|
score gate with a *relative-ordering* gate: rank the admissible
|
||||||
|
candidates by `cga_inner(versor, relation_blade)` descending and
|
||||||
|
admit the top iff its margin over the second-ranked exceeds a
|
||||||
|
single per-runtime `delta`.
|
||||||
|
|
||||||
|
Option 3 is the only option that is **scale-invariant under blade-
|
||||||
|
norm variation** — the gap between the top and second-ranked scores
|
||||||
|
scales with the blade norm, so the *relative* ordering carries the
|
||||||
|
semantic separation the static threshold was reaching for. This is
|
||||||
|
the architectural difference between "what direction is admissible"
|
||||||
|
(blade alignment, absolute) and "which candidate is confidently
|
||||||
|
selected over the next-best" (separation, relative).
|
||||||
|
|
||||||
|
The previous assessment (recorded in the planning conversation that
|
||||||
|
preceded this ADR) put it this way:
|
||||||
|
|
||||||
|
> A global threshold — even normalized — assumes admissibility is an
|
||||||
|
> absolute property of a candidate. It isn't: in your geometry,
|
||||||
|
> what matters is the ordering of admissibility scores across the
|
||||||
|
> candidate set, plus enough separation to be confident the
|
||||||
|
> boundary's pick can be rejected and replaced deterministically.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Add a new admissibility mode `margin` alongside ADR-0024's existing
|
||||||
|
`threshold` mode. When margin mode is active, the inner loop's
|
||||||
|
selection-and-admission collapses into a single deterministic step:
|
||||||
|
|
||||||
|
```python
|
||||||
|
ranked = rank_candidates_by_blade(region, candidate_indices=…, …)
|
||||||
|
verdict = check_margin(region, ranked, delta=admissibility_margin)
|
||||||
|
```
|
||||||
|
|
||||||
|
`rank_candidates_by_blade` sorts the post-region-filter candidate set
|
||||||
|
by `cga_inner(versor, relation_blade)` descending, with a strict `>`
|
||||||
|
tie-break: when scores are equal, ascending vocab index wins. This
|
||||||
|
matches the `vocab.nearest` strict-`>` convention documented as
|
||||||
|
load-bearing in ADR-0024.
|
||||||
|
|
||||||
|
`check_margin` admits the top-ranked candidate iff:
|
||||||
|
|
||||||
|
1. `len(ranked) >= 1` (non-empty admissible set), AND
|
||||||
|
2. `ranked[0].score > 0` (basic positivity in the blade half-space),
|
||||||
|
AND
|
||||||
|
3. `len(ranked) == 1` (trivial, no competitor) **OR**
|
||||||
|
`ranked[0].score - ranked[1].score >= delta`.
|
||||||
|
|
||||||
|
When refused, the inner loop raises `InnerLoopExhaustion`
|
||||||
|
(ADR-0024 Phase 2) carrying the full ranking as `rejected_attempts` —
|
||||||
|
not a single rejected score, but the entire blade-ordering at the
|
||||||
|
failed step.
|
||||||
|
|
||||||
|
### Selection semantics in margin mode
|
||||||
|
|
||||||
|
Margin mode is **blade-rank-driven** selection: the top-ranked
|
||||||
|
admissible candidate IS the admitted destination. This differs from
|
||||||
|
threshold mode, where `_nearest_next` (field-driven) picks one
|
||||||
|
candidate and `check_transition` gates it; on rejection, the
|
||||||
|
selector advances to the next field-closest candidate and re-gates.
|
||||||
|
|
||||||
|
This is a meaningful semantic difference, not a re-shading. Margin
|
||||||
|
mode says *"of the candidates the region admits as semantically
|
||||||
|
valid, the most blade-aligned one is the destination, provided its
|
||||||
|
lead is confident."* Threshold mode says *"of the candidates the
|
||||||
|
field's geometry prefers, accept the first one above an absolute
|
||||||
|
score bar."*
|
||||||
|
|
||||||
|
Both have a place: threshold mode is the ADR-0024 acceptance
|
||||||
|
evidence and remains the default to preserve every existing turn's
|
||||||
|
trace_hash byte-for-byte. Margin mode is the new production
|
||||||
|
admissibility for cases where blade-norm variation makes the static
|
||||||
|
bar incoherent.
|
||||||
|
|
||||||
|
## The single `delta` choice
|
||||||
|
|
||||||
|
Phase 3 picks `delta = 0.4` as the default. This was derived from
|
||||||
|
the v2 mechanism-isolation case margins:
|
||||||
|
|
||||||
|
| Case | Top score | Second score | Margin |
|
||||||
|
| ------------- | --------- | ------------ | ------ |
|
||||||
|
| FSC-PUB-V2-001 | 1.420 | 0.824 | 0.596 |
|
||||||
|
| FSC-PUB-V2-002 | 1.173 | 0.717 | 0.456 |
|
||||||
|
| FSC-PUB-V2-003 | 12.720 | -0.550 | 13.270 |
|
||||||
|
| FSC-PUB-V2-004 | 5.740 | 2.370 | 3.370 |
|
||||||
|
| FSC-PUB-V2-005 | 14.360 | 1.620 | 12.740 |
|
||||||
|
|
||||||
|
The minimum margin across the five cases is **0.456**. `delta = 0.4`
|
||||||
|
admits all five with headroom; `delta = 0.5` would refuse V2-002.
|
||||||
|
The default is *falsifiable*: Phase 5's diversified failure-mode
|
||||||
|
families may surface a case below `0.4`, in which case the finding
|
||||||
|
is architectural — margin alone is insufficient for that family —
|
||||||
|
and should be reported honestly rather than patched with a per-case
|
||||||
|
override.
|
||||||
|
|
||||||
|
## Wiring
|
||||||
|
|
||||||
|
| Surface | Field |
|
||||||
|
| ------------------------------------------ | ------------------------------ |
|
||||||
|
| `core/config.py::RuntimeConfig` | `admissibility_mode: str = "threshold"` |
|
||||||
|
| | `admissibility_margin: float = 0.4` |
|
||||||
|
| `chat/runtime.py::ChatRuntime.chat` | forwards both fields |
|
||||||
|
| `generate/stream.py::generate` | `admissibility_mode` / `admissibility_margin` kwargs |
|
||||||
|
| `generate/admissibility.py` | `RankedCandidate`, `MarginVerdict`, `rank_candidates_by_blade`, `check_margin` |
|
||||||
|
|
||||||
|
Selection ordering inside `generate.generate()`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
if margin_mode_active:
|
||||||
|
rank → check_margin → admit-or-refuse
|
||||||
|
else:
|
||||||
|
_nearest_next → check_transition (per-attempt loop)
|
||||||
|
```
|
||||||
|
|
||||||
|
The rotor `V` is only constructed for the *admitted* candidate, so
|
||||||
|
the `versor_condition(F) < 1e-6` invariant is preserved by
|
||||||
|
construction (CLAUDE.md §Non-Negotiable Field Invariant).
|
||||||
|
|
||||||
|
## Why flag-gated (default off)
|
||||||
|
|
||||||
|
Margin mode is a real semantic change in selection. Defaulting it
|
||||||
|
off preserves:
|
||||||
|
|
||||||
|
* Every existing trace_hash byte-for-byte (no payload bytes change
|
||||||
|
when margin mode is unset).
|
||||||
|
* ADR-0024's acceptance evidence intact.
|
||||||
|
* The ability to A/B threshold vs margin on the same corpus during
|
||||||
|
the transition window.
|
||||||
|
|
||||||
|
A future ADR may flip the default; this ADR does not.
|
||||||
|
|
||||||
|
## Invariants preserved
|
||||||
|
|
||||||
|
* `versor_condition(F) < 1e-6` — the rotor is constructed only for
|
||||||
|
the admitted candidate; margin mode does not add a normalization
|
||||||
|
or repair site (CLAUDE.md §Normalization Rules).
|
||||||
|
* Deterministic replay — strict `>` tie-break is now load-bearing
|
||||||
|
in two places: `vocab.nearest` (ADR-0024) and
|
||||||
|
`rank_candidates_by_blade` (this ADR).
|
||||||
|
`tests/test_margin_admissibility.py::TestRankCandidates::test_strict_tie_break_by_ascending_index`
|
||||||
|
pins it.
|
||||||
|
* No approximate recall, no cosine similarity, no HNSW/ANN.
|
||||||
|
Margin is a pure rank-and-difference operation on the exact
|
||||||
|
`cga_inner` scores already in use.
|
||||||
|
* No new code in `field/propagate.py`, `algebra/versor.py`,
|
||||||
|
`vault/store.py`, or `chat/runtime.respond()`.
|
||||||
|
|
||||||
|
## Trust boundary
|
||||||
|
|
||||||
|
Same as ADR-0024 — admissibility regions are constructed upstream
|
||||||
|
from pack-grounded data; margin mode adds no new surface that
|
||||||
|
consumes user-controlled text, filesystem paths, or dynamic
|
||||||
|
imports.
|
||||||
|
|
||||||
|
## Acceptance evidence
|
||||||
|
|
||||||
|
* **5/5 v2 mechanism-isolation cases pass in margin mode** with
|
||||||
|
`delta = 0.4`. Forbidden token traced in every case's
|
||||||
|
`rejected_attempts`. Mirrors ADR-0024 §Acceptance evidence for
|
||||||
|
threshold mode.
|
||||||
|
* **Threshold mode unchanged.**
|
||||||
|
`tests/test_margin_admissibility.py::TestGenerateMarginMode::test_threshold_mode_unchanged_by_margin_plumbing`
|
||||||
|
asserts `mode="threshold"` and unset `mode` produce identical
|
||||||
|
tokens.
|
||||||
|
* **Refusal on insufficient margin.**
|
||||||
|
`test_v2_001_refuses_when_delta_too_high` runs FSC-PUB-V2-001 at
|
||||||
|
`delta = 0.9` (above its 0.597 margin) and asserts
|
||||||
|
`InnerLoopExhaustion` with `reason=INNER_LOOP_EXHAUSTION` and the
|
||||||
|
full ranking in `rejected_attempts`. No silent boundary fallback.
|
||||||
|
* **Replay determinism.**
|
||||||
|
`TestMarginModeDeterminism::test_margin_mode_replay_stable_across_5_runs`
|
||||||
|
asserts 5 reruns of the same input produce identical canonical
|
||||||
|
trace steps.
|
||||||
|
* **Strict tie-break determinism.**
|
||||||
|
`TestRankCandidates::test_strict_tie_break_by_ascending_index`
|
||||||
|
asserts equal-score candidates resolve to the lower-index winner
|
||||||
|
reproducibly across 5 runs.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
* **Promoting margin to default.** Requires a separate ADR plus
|
||||||
|
trace-hash migration evidence.
|
||||||
|
* **Rotor-frame margin.** ADR-0025 will wire rotor admissibility;
|
||||||
|
if its mechanism also benefits from a margin gate, that's an
|
||||||
|
ADR-0025 decision.
|
||||||
|
* **Per-family delta calibration.** Phase 5 will report metrics
|
||||||
|
stratified by failure-mode family. If a family fails on the
|
||||||
|
single `delta`, the finding is architectural and surfaces in
|
||||||
|
Phase 5; this ADR explicitly forbids per-family tuned constants
|
||||||
|
as a fix.
|
||||||
|
* **CLI flags.** RuntimeConfig + ChatRuntime is the wired surface.
|
||||||
|
A `--admissibility-mode` flag is a UX follow-up, not load-bearing
|
||||||
|
for the contract.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
* **A single `delta` may fail on a future failure-mode family.**
|
||||||
|
Mitigation: the default is falsifiable (Phase 5 will diversify);
|
||||||
|
refusal is honest (`InnerLoopExhaustion` carries the ranking, so
|
||||||
|
the failure mode is visible in the trace). Patching with
|
||||||
|
per-family constants is explicitly out of scope.
|
||||||
|
* **Margin mode changes selection semantics.** Threshold-mode
|
||||||
|
acceptance evidence does not transfer. Mitigation: default off;
|
||||||
|
separate test suite (`tests/test_margin_admissibility.py`) pins
|
||||||
|
margin contract independently.
|
||||||
|
* **Cost.** Margin mode evaluates `cga_inner` for every candidate
|
||||||
|
in the admissible set, not just the field-closest one as
|
||||||
|
threshold mode does. In practice the admissible set is small
|
||||||
|
(chain length / v2 case size) so this is bounded. No
|
||||||
|
approximation added.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Set `admissibility_mode = "threshold"` (the default) at every call
|
||||||
|
site. The threshold path is unchanged from ADR-0024. No trace_hash
|
||||||
|
migration required for non-refused turns.
|
||||||
|
|
@ -405,6 +405,190 @@ def check_transition(
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------
|
||||||
|
# Phase 3 — Ranked admissibility with margin (ADR-0026)
|
||||||
|
# ----------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# Replaces ``score >= threshold`` with
|
||||||
|
# ``score(top) - score(second) >= delta`` over the candidates in the
|
||||||
|
# admissible set, ranked by ``cga_inner(versor, relation_blade)``
|
||||||
|
# descending with strict ``>`` tie-break (deterministic on ascending
|
||||||
|
# vocab index for ties).
|
||||||
|
#
|
||||||
|
# Phase 4 characterization (recorded in
|
||||||
|
# ``tests/test_inner_loop_phase4.py``) showed that no single global
|
||||||
|
# threshold separates the v2 mechanism-isolation cases because blade
|
||||||
|
# norms vary ~10x across cases. Margins are scale-invariant under
|
||||||
|
# blade-norm variation — the gap between top and second-ranked is
|
||||||
|
# proportional to the blade norm, so the *relative* admissibility
|
||||||
|
# ordering is what the geometry actually delivers. A single per-
|
||||||
|
# runtime ``delta`` is therefore meaningful in a way that a single
|
||||||
|
# ``tau`` is not.
|
||||||
|
#
|
||||||
|
# Selection within margin mode is blade-rank-driven: the top-ranked
|
||||||
|
# admissible candidate IS the admitted destination. This differs
|
||||||
|
# from threshold mode where ``_nearest_next`` (field-driven) picks
|
||||||
|
# and ``check_transition`` gates. The mode is opt-in; threshold
|
||||||
|
# mode remains the default to preserve ADR-0024 acceptance evidence.
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class RankedCandidate:
|
||||||
|
"""One row in the blade-score ranking of an admissible candidate set."""
|
||||||
|
|
||||||
|
index: int
|
||||||
|
word: str
|
||||||
|
score: float
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class MarginVerdict:
|
||||||
|
"""Result of margin-based admissibility check on a ranked candidate set.
|
||||||
|
|
||||||
|
Carries the admit/reject verdict, the top-ranked candidate, the
|
||||||
|
margin between top and second-ranked, and the full ranked list so
|
||||||
|
refusal evidence (in ``InnerLoopExhaustion.rejected_attempts``)
|
||||||
|
can carry the entire ordering at the failed step rather than a
|
||||||
|
single rejected score.
|
||||||
|
"""
|
||||||
|
|
||||||
|
admitted: bool
|
||||||
|
top: RankedCandidate | None
|
||||||
|
margin: float
|
||||||
|
delta: float
|
||||||
|
region_label: str
|
||||||
|
ranked: tuple[RankedCandidate, ...] = ()
|
||||||
|
reason: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
def rank_candidates_by_blade(
|
||||||
|
region: AdmissibilityRegion,
|
||||||
|
*,
|
||||||
|
candidate_indices: np.ndarray | None,
|
||||||
|
versor_lookup,
|
||||||
|
word_lookup,
|
||||||
|
) -> tuple[RankedCandidate, ...]:
|
||||||
|
"""Rank candidates by ``cga_inner(versor, relation_blade)`` desc.
|
||||||
|
|
||||||
|
``candidate_indices`` is the post-region-filter candidate set
|
||||||
|
(already intersected with ``region.allowed_indices`` upstream).
|
||||||
|
``versor_lookup(idx) -> np.ndarray`` and
|
||||||
|
``word_lookup(idx) -> str`` are vocab accessors hoisted into
|
||||||
|
parameters so this function has no I/O and can be unit-tested
|
||||||
|
with a stub vocab.
|
||||||
|
|
||||||
|
Tie-break: strict ``>`` on score, with ascending ``index`` as the
|
||||||
|
deterministic secondary key. This matches the ``vocab.nearest``
|
||||||
|
strict-``>`` convention documented as load-bearing in ADR-0024.
|
||||||
|
"""
|
||||||
|
if candidate_indices is None or len(candidate_indices) == 0:
|
||||||
|
return ()
|
||||||
|
blade_norm = float(np.linalg.norm(region.relation_blade))
|
||||||
|
if blade_norm < _NULL_TOLERANCE:
|
||||||
|
# No blade constraint — every candidate scores 0. Margin
|
||||||
|
# cannot separate them; the caller should not enter margin
|
||||||
|
# mode on an unconstrained blade. Return ranking in vocab
|
||||||
|
# index order so behaviour is at least deterministic.
|
||||||
|
rows = [
|
||||||
|
RankedCandidate(index=int(i), word=str(word_lookup(int(i))), score=0.0)
|
||||||
|
for i in candidate_indices
|
||||||
|
]
|
||||||
|
return tuple(rows)
|
||||||
|
rows: list[RankedCandidate] = []
|
||||||
|
for idx in candidate_indices:
|
||||||
|
v = np.asarray(versor_lookup(int(idx)), dtype=np.float32)
|
||||||
|
s = float(cga_inner(v, region.relation_blade))
|
||||||
|
rows.append(
|
||||||
|
RankedCandidate(index=int(idx), word=str(word_lookup(int(idx))), score=s)
|
||||||
|
)
|
||||||
|
# Sort descending by score, ascending by index for tie-break.
|
||||||
|
rows.sort(key=lambda r: (-r.score, r.index))
|
||||||
|
return tuple(rows)
|
||||||
|
|
||||||
|
|
||||||
|
def check_margin(
|
||||||
|
region: AdmissibilityRegion,
|
||||||
|
ranked: tuple[RankedCandidate, ...],
|
||||||
|
*,
|
||||||
|
delta: float,
|
||||||
|
) -> MarginVerdict:
|
||||||
|
"""Admit the top-ranked candidate iff it has a clean margin.
|
||||||
|
|
||||||
|
Admission requires:
|
||||||
|
|
||||||
|
1. The ranking is non-empty (``len(ranked) >= 1``).
|
||||||
|
2. ``ranked[0].score > 0`` — basic positivity in the blade
|
||||||
|
half-space. A non-positive top score means the admissible
|
||||||
|
set has no blade-aligned candidate at all; refuse.
|
||||||
|
3. If ``len(ranked) >= 2``: ``ranked[0].score - ranked[1].score
|
||||||
|
>= delta``. The top must out-score the next-best by at
|
||||||
|
least ``delta``.
|
||||||
|
4. If ``len(ranked) == 1``: trivially admit (no competitor to
|
||||||
|
confuse the boundary's pick).
|
||||||
|
|
||||||
|
A single ``delta`` works across blades of varying norm because
|
||||||
|
the margin scales with blade norm — the *relative* gap, not the
|
||||||
|
absolute score, is what carries semantic separation.
|
||||||
|
"""
|
||||||
|
if not ranked:
|
||||||
|
return MarginVerdict(
|
||||||
|
admitted=False,
|
||||||
|
top=None,
|
||||||
|
margin=float("-inf"),
|
||||||
|
delta=delta,
|
||||||
|
region_label=region.label,
|
||||||
|
ranked=(),
|
||||||
|
reason="empty ranking",
|
||||||
|
)
|
||||||
|
top = ranked[0]
|
||||||
|
if top.score <= 0.0:
|
||||||
|
return MarginVerdict(
|
||||||
|
admitted=False,
|
||||||
|
top=top,
|
||||||
|
margin=float("-inf"),
|
||||||
|
delta=delta,
|
||||||
|
region_label=region.label,
|
||||||
|
ranked=ranked,
|
||||||
|
reason=f"top score {top.score:.6f} not positive",
|
||||||
|
)
|
||||||
|
if len(ranked) == 1:
|
||||||
|
# Single admissible candidate — no margin to compute.
|
||||||
|
return MarginVerdict(
|
||||||
|
admitted=True,
|
||||||
|
top=top,
|
||||||
|
margin=float("inf"),
|
||||||
|
delta=delta,
|
||||||
|
region_label=region.label,
|
||||||
|
ranked=ranked,
|
||||||
|
reason="ok (single admissible)",
|
||||||
|
)
|
||||||
|
second = ranked[1]
|
||||||
|
margin = top.score - second.score
|
||||||
|
if margin < delta:
|
||||||
|
return MarginVerdict(
|
||||||
|
admitted=False,
|
||||||
|
top=top,
|
||||||
|
margin=margin,
|
||||||
|
delta=delta,
|
||||||
|
region_label=region.label,
|
||||||
|
ranked=ranked,
|
||||||
|
reason=(
|
||||||
|
f"margin {margin:.6f} below delta {delta:.6f} "
|
||||||
|
f"(top={top.word!r}@{top.score:.6f}, "
|
||||||
|
f"next={second.word!r}@{second.score:.6f})"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return MarginVerdict(
|
||||||
|
admitted=True,
|
||||||
|
top=top,
|
||||||
|
margin=margin,
|
||||||
|
delta=delta,
|
||||||
|
region_label=region.label,
|
||||||
|
ranked=ranked,
|
||||||
|
reason="ok",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
@dataclass(frozen=True, slots=True)
|
||||||
class AdmissibilityTraceStep:
|
class AdmissibilityTraceStep:
|
||||||
"""One per-transition record from a constrained walk (ADR-0023 §2).
|
"""One per-transition record from a constrained walk (ADR-0023 §2).
|
||||||
|
|
|
||||||
|
|
@ -23,8 +23,12 @@ from generate.admissibility import (
|
||||||
AdmissibilityRegion,
|
AdmissibilityRegion,
|
||||||
AdmissibilityTraceStep,
|
AdmissibilityTraceStep,
|
||||||
AdmissibilityVerdict,
|
AdmissibilityVerdict,
|
||||||
|
MarginVerdict,
|
||||||
|
RankedCandidate,
|
||||||
|
check_margin,
|
||||||
check_transition,
|
check_transition,
|
||||||
filter_candidates,
|
filter_candidates,
|
||||||
|
rank_candidates_by_blade,
|
||||||
)
|
)
|
||||||
from generate.attention import AttentionOperator
|
from generate.attention import AttentionOperator
|
||||||
from generate.exhaustion import InnerLoopExhaustion, RefusalReason
|
from generate.exhaustion import InnerLoopExhaustion, RefusalReason
|
||||||
|
|
@ -281,6 +285,8 @@ def generate(
|
||||||
inner_loop_admissibility: bool = False,
|
inner_loop_admissibility: bool = False,
|
||||||
admissibility_threshold: float = 0.0,
|
admissibility_threshold: float = 0.0,
|
||||||
inner_loop_force_admit: bool = False,
|
inner_loop_force_admit: bool = False,
|
||||||
|
admissibility_mode: str = "threshold",
|
||||||
|
admissibility_margin: float = 0.4,
|
||||||
) -> GenerationResult:
|
) -> GenerationResult:
|
||||||
"""Generate a token sequence.
|
"""Generate a token sequence.
|
||||||
|
|
||||||
|
|
@ -375,6 +381,18 @@ def generate(
|
||||||
region_active = region is not None and not region.is_unconstrained()
|
region_active = region is not None and not region.is_unconstrained()
|
||||||
active_region: AdmissibilityRegion | None = region if region_active else None
|
active_region: AdmissibilityRegion | None = region if region_active else None
|
||||||
inner_loop_active = inner_loop_admissibility and region_active
|
inner_loop_active = inner_loop_admissibility and region_active
|
||||||
|
# ADR-0026 / Phase 3 — margin mode is opt-in. When active it
|
||||||
|
# replaces the per-candidate threshold check with a ranked
|
||||||
|
# admissibility test on the candidate set, admitting the top
|
||||||
|
# blade-ranked candidate iff its margin over the second-ranked
|
||||||
|
# is at least ``admissibility_margin``. Falls back to threshold
|
||||||
|
# mode (ADR-0024) on any unrecognised value so a config typo
|
||||||
|
# cannot silently disable admissibility.
|
||||||
|
margin_mode_active = (
|
||||||
|
inner_loop_active
|
||||||
|
and admissibility_mode == "margin"
|
||||||
|
and active_region is not None
|
||||||
|
)
|
||||||
for step_index in range(token_budget):
|
for step_index in range(token_budget):
|
||||||
current, hits_applied = _recall_state(_voiced_state(current, persona), vault, recall_top_k)
|
current, hits_applied = _recall_state(_voiced_state(current, persona), vault, recall_top_k)
|
||||||
vault_hits += hits_applied
|
vault_hits += hits_applied
|
||||||
|
|
@ -388,6 +406,51 @@ def generate(
|
||||||
word_idx: int
|
word_idx: int
|
||||||
verdict: AdmissibilityVerdict
|
verdict: AdmissibilityVerdict
|
||||||
|
|
||||||
|
if margin_mode_active:
|
||||||
|
# ADR-0026 / Phase 3 — rank the admissible candidate set by
|
||||||
|
# blade-score and admit the top iff margin >= delta. The
|
||||||
|
# rotor V is only constructed for the admitted candidate
|
||||||
|
# below, preserving the versor_condition invariant.
|
||||||
|
assert active_region is not None # margin_mode_active gates this
|
||||||
|
ranked = rank_candidates_by_blade(
|
||||||
|
active_region,
|
||||||
|
candidate_indices=candidate_indices,
|
||||||
|
versor_lookup=vocab.get_versor_at,
|
||||||
|
word_lookup=vocab.get_word_at,
|
||||||
|
)
|
||||||
|
margin_verdict = check_margin(
|
||||||
|
active_region, ranked, delta=admissibility_margin
|
||||||
|
)
|
||||||
|
# rejected_attempts carries the full ranked list as evidence
|
||||||
|
# — index, word, score — so refusal traces show the entire
|
||||||
|
# blade-ordering at the failed step, not just one rejection.
|
||||||
|
rejected_attempts = [
|
||||||
|
(r.index, r.word, r.score) for r in margin_verdict.ranked
|
||||||
|
]
|
||||||
|
if not margin_verdict.admitted:
|
||||||
|
raise InnerLoopExhaustion(
|
||||||
|
reason=RefusalReason.INNER_LOOP_EXHAUSTION,
|
||||||
|
region_label=effective_region_label,
|
||||||
|
step_index=step_index,
|
||||||
|
rejected_attempts=tuple(rejected_attempts),
|
||||||
|
)
|
||||||
|
assert margin_verdict.top is not None # admitted => non-None
|
||||||
|
word_idx = int(margin_verdict.top.index)
|
||||||
|
word = str(margin_verdict.top.word)
|
||||||
|
# Build a legacy AdmissibilityVerdict for trace storage so
|
||||||
|
# the AdmissibilityTraceStep shape is unchanged. Margin
|
||||||
|
# info is encoded into ``reason`` for human inspection;
|
||||||
|
# the structured ranking lives in ``rejected_attempts``.
|
||||||
|
verdict = AdmissibilityVerdict(
|
||||||
|
admitted=True,
|
||||||
|
score=float(margin_verdict.top.score),
|
||||||
|
region_label=margin_verdict.region_label,
|
||||||
|
reason=(
|
||||||
|
f"margin {margin_verdict.margin:.6f} >= "
|
||||||
|
f"delta {margin_verdict.delta:.6f}"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
else:
|
||||||
max_attempts = (
|
max_attempts = (
|
||||||
len(candidate_indices) if (inner_loop_active and candidate_indices is not None)
|
len(candidate_indices) if (inner_loop_active and candidate_indices is not None)
|
||||||
else 1
|
else 1
|
||||||
|
|
|
||||||
322
tests/test_margin_admissibility.py
Normal file
322
tests/test_margin_admissibility.py
Normal file
|
|
@ -0,0 +1,322 @@
|
||||||
|
"""Phase 3 / ADR-0026 — ranked-with-margin admissibility contract.
|
||||||
|
|
||||||
|
These tests pin the new admissibility shape:
|
||||||
|
|
||||||
|
* ``rank_candidates_by_blade`` returns the admissible set sorted
|
||||||
|
by ``cga_inner(versor, blade)`` descending, with strict ``>``
|
||||||
|
tie-break (ascending vocab index for ties).
|
||||||
|
* ``check_margin`` admits the top-ranked candidate iff
|
||||||
|
``top_score > 0`` AND ``top_score - second_score >= delta``.
|
||||||
|
* Margin mode wired through ``generate()`` produces the correct
|
||||||
|
endpoint on v2 mechanism-isolation cases and emits
|
||||||
|
``InnerLoopExhaustion`` when the margin is not met — never a
|
||||||
|
silent boundary fallback.
|
||||||
|
* Near-equal-score candidates resolve deterministically across
|
||||||
|
repeated runs (replay invariance).
|
||||||
|
* Threshold mode (ADR-0024) is unchanged by margin-mode plumbing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from chat.runtime import ChatRuntime
|
||||||
|
from field.state import FieldState
|
||||||
|
from generate.admissibility import (
|
||||||
|
AdmissibilityRegion,
|
||||||
|
RankedCandidate,
|
||||||
|
RegionSource,
|
||||||
|
check_margin,
|
||||||
|
rank_candidates_by_blade,
|
||||||
|
)
|
||||||
|
from generate.exhaustion import InnerLoopExhaustion, RefusalReason
|
||||||
|
from generate.stream import generate
|
||||||
|
|
||||||
|
_BLADE_DIM = 32
|
||||||
|
|
||||||
|
|
||||||
|
def _zero_blade() -> np.ndarray:
|
||||||
|
return np.zeros(_BLADE_DIM, dtype=np.float32)
|
||||||
|
|
||||||
|
|
||||||
|
def _region_with_blade(allowed: list[int], blade: np.ndarray, label: str = "test") -> AdmissibilityRegion:
|
||||||
|
return AdmissibilityRegion(
|
||||||
|
allowed_indices=np.asarray(allowed, dtype=np.int64),
|
||||||
|
relation_blade=blade.astype(np.float32),
|
||||||
|
source=RegionSource.RELATION,
|
||||||
|
label=label,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# rank_candidates_by_blade
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestRankCandidates:
|
||||||
|
def test_ranks_descending_by_score(self) -> None:
|
||||||
|
blade = _zero_blade()
|
||||||
|
blade[0] = 1.0 # non-zero blade with self-inner != 0 (depends on metric)
|
||||||
|
# Build versors with known scores; we'll just construct them and
|
||||||
|
# verify the *ordering*, not the exact score values.
|
||||||
|
from algebra.cga import cga_inner
|
||||||
|
|
||||||
|
bb = float(cga_inner(blade, blade))
|
||||||
|
if abs(bb) < 1e-9:
|
||||||
|
pytest.skip("blade self-inner is zero for chosen test blade")
|
||||||
|
versors = {
|
||||||
|
10: (0.5 / bb) * blade,
|
||||||
|
20: (1.0 / bb) * blade,
|
||||||
|
30: (0.2 / bb) * blade,
|
||||||
|
}
|
||||||
|
words = {10: "alpha", 20: "beta", 30: "gamma"}
|
||||||
|
region = _region_with_blade([10, 20, 30], blade)
|
||||||
|
ranked = rank_candidates_by_blade(
|
||||||
|
region,
|
||||||
|
candidate_indices=np.asarray([10, 20, 30], dtype=np.int64),
|
||||||
|
versor_lookup=lambda i: versors[int(i)],
|
||||||
|
word_lookup=lambda i: words[int(i)],
|
||||||
|
)
|
||||||
|
# Sorted desc by score → indices [20, 10, 30]
|
||||||
|
assert [r.index for r in ranked] == [20, 10, 30]
|
||||||
|
# Strict descending scores
|
||||||
|
assert ranked[0].score > ranked[1].score > ranked[2].score
|
||||||
|
|
||||||
|
def test_strict_tie_break_by_ascending_index(self) -> None:
|
||||||
|
"""When two candidates have the *same* score, ascending vocab
|
||||||
|
index wins. This pins the determinism contract."""
|
||||||
|
blade = _zero_blade()
|
||||||
|
blade[0] = 1.0
|
||||||
|
from algebra.cga import cga_inner
|
||||||
|
|
||||||
|
bb = float(cga_inner(blade, blade))
|
||||||
|
if abs(bb) < 1e-9:
|
||||||
|
pytest.skip("blade self-inner is zero for chosen test blade")
|
||||||
|
# Two equal scores at indices 30 and 10 → 10 should win.
|
||||||
|
versors = {
|
||||||
|
10: (1.0 / bb) * blade,
|
||||||
|
30: (1.0 / bb) * blade,
|
||||||
|
20: (0.5 / bb) * blade,
|
||||||
|
}
|
||||||
|
words = {10: "alpha", 20: "beta", 30: "gamma"}
|
||||||
|
region = _region_with_blade([10, 20, 30], blade)
|
||||||
|
ranked = rank_candidates_by_blade(
|
||||||
|
region,
|
||||||
|
candidate_indices=np.asarray([10, 20, 30], dtype=np.int64),
|
||||||
|
versor_lookup=lambda i: versors[int(i)],
|
||||||
|
word_lookup=lambda i: words[int(i)],
|
||||||
|
)
|
||||||
|
# Both tied candidates are at the top; lower index comes first.
|
||||||
|
assert ranked[0].index == 10
|
||||||
|
assert ranked[1].index == 30
|
||||||
|
# Determinism across repeats
|
||||||
|
for _ in range(5):
|
||||||
|
r = rank_candidates_by_blade(
|
||||||
|
region,
|
||||||
|
candidate_indices=np.asarray([10, 20, 30], dtype=np.int64),
|
||||||
|
versor_lookup=lambda i: versors[int(i)],
|
||||||
|
word_lookup=lambda i: words[int(i)],
|
||||||
|
)
|
||||||
|
assert tuple(c.index for c in r) == tuple(c.index for c in ranked)
|
||||||
|
|
||||||
|
def test_empty_candidate_set_returns_empty(self) -> None:
|
||||||
|
blade = _zero_blade()
|
||||||
|
blade[0] = 1.0
|
||||||
|
region = _region_with_blade([1], blade)
|
||||||
|
assert (
|
||||||
|
rank_candidates_by_blade(
|
||||||
|
region,
|
||||||
|
candidate_indices=np.asarray([], dtype=np.int64),
|
||||||
|
versor_lookup=lambda i: np.zeros(_BLADE_DIM, dtype=np.float32),
|
||||||
|
word_lookup=lambda i: "",
|
||||||
|
)
|
||||||
|
== ()
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_zero_blade_returns_zero_scores_index_order(self) -> None:
|
||||||
|
"""An unconstrained-direction region returns all candidates at
|
||||||
|
score 0, in vocab-index order. The caller should not enter
|
||||||
|
margin mode here; this test pins the safe fallback."""
|
||||||
|
region = _region_with_blade([3, 1, 2], _zero_blade())
|
||||||
|
# allowed_indices is canonicalised to sorted unique → [1,2,3].
|
||||||
|
ranked = rank_candidates_by_blade(
|
||||||
|
region,
|
||||||
|
candidate_indices=region.allowed_indices,
|
||||||
|
versor_lookup=lambda i: np.zeros(_BLADE_DIM, dtype=np.float32),
|
||||||
|
word_lookup=lambda i: f"w{i}",
|
||||||
|
)
|
||||||
|
assert [r.index for r in ranked] == [1, 2, 3]
|
||||||
|
assert all(r.score == 0.0 for r in ranked)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# check_margin
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestCheckMargin:
|
||||||
|
def _ranked(self, *triples: tuple[int, str, float]) -> tuple[RankedCandidate, ...]:
|
||||||
|
return tuple(RankedCandidate(index=i, word=w, score=s) for i, w, s in triples)
|
||||||
|
|
||||||
|
def test_admits_when_margin_meets_delta(self) -> None:
|
||||||
|
ranked = self._ranked((1, "a", 1.5), (2, "b", 0.8))
|
||||||
|
region = _region_with_blade([1, 2], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||||
|
verdict = check_margin(region, ranked, delta=0.5)
|
||||||
|
assert verdict.admitted is True
|
||||||
|
assert verdict.top is not None and verdict.top.index == 1
|
||||||
|
assert pytest.approx(verdict.margin, abs=1e-9) == 0.7
|
||||||
|
|
||||||
|
def test_refuses_when_margin_below_delta(self) -> None:
|
||||||
|
ranked = self._ranked((1, "a", 1.0), (2, "b", 0.8))
|
||||||
|
region = _region_with_blade([1, 2], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||||
|
verdict = check_margin(region, ranked, delta=0.5)
|
||||||
|
assert verdict.admitted is False
|
||||||
|
assert "margin" in verdict.reason
|
||||||
|
assert pytest.approx(verdict.margin, abs=1e-9) == pytest.approx(0.2)
|
||||||
|
|
||||||
|
def test_refuses_when_top_score_not_positive(self) -> None:
|
||||||
|
"""Even a clean margin doesn't save a non-positive top score:
|
||||||
|
the admissible set has no blade-aligned candidate at all."""
|
||||||
|
ranked = self._ranked((1, "a", -0.5), (2, "b", -2.0))
|
||||||
|
region = _region_with_blade([1, 2], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||||
|
verdict = check_margin(region, ranked, delta=0.5)
|
||||||
|
assert verdict.admitted is False
|
||||||
|
assert "not positive" in verdict.reason
|
||||||
|
|
||||||
|
def test_single_candidate_trivially_admitted_when_positive(self) -> None:
|
||||||
|
ranked = self._ranked((1, "a", 0.5))
|
||||||
|
region = _region_with_blade([1], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||||
|
verdict = check_margin(region, ranked, delta=999.0)
|
||||||
|
assert verdict.admitted is True
|
||||||
|
assert verdict.margin == float("inf")
|
||||||
|
assert "single admissible" in verdict.reason
|
||||||
|
|
||||||
|
def test_empty_ranking_refuses(self) -> None:
|
||||||
|
region = _region_with_blade([1], np.array([1.0] + [0.0] * (_BLADE_DIM - 1), dtype=np.float32))
|
||||||
|
verdict = check_margin(region, (), delta=0.4)
|
||||||
|
assert verdict.admitted is False
|
||||||
|
assert verdict.top is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# generate() in margin mode — integration via v2-like setup
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _v2_state_and_region(rt: ChatRuntime, *, seed: str, admissible: list[str], blade_tok: str, label: str):
|
||||||
|
vocab = rt.session.vocab
|
||||||
|
idx = vocab.index_of(seed)
|
||||||
|
F = np.asarray(vocab.get_versor(seed), dtype=np.float32)
|
||||||
|
state = FieldState(F=F.copy(), node=idx, step=0)
|
||||||
|
indices = np.asarray([vocab.index_of(t) for t in admissible], dtype=np.int64)
|
||||||
|
blade = np.asarray(vocab.get_versor(blade_tok), dtype=np.float32)
|
||||||
|
region = AdmissibilityRegion(
|
||||||
|
allowed_indices=indices,
|
||||||
|
relation_blade=blade,
|
||||||
|
source=RegionSource.RELATION,
|
||||||
|
label=label,
|
||||||
|
)
|
||||||
|
return state, region
|
||||||
|
|
||||||
|
|
||||||
|
class TestGenerateMarginMode:
|
||||||
|
"""End-to-end: margin mode on v2-style mechanism-isolation cases."""
|
||||||
|
|
||||||
|
def test_v2_001_question_admitted_via_margin(self) -> None:
|
||||||
|
rt = ChatRuntime()
|
||||||
|
state, region = _v2_state_and_region(
|
||||||
|
rt, seed="symbol", admissible=["answer", "question"],
|
||||||
|
blade_tok="question", label="v2-001",
|
||||||
|
)
|
||||||
|
result = generate(
|
||||||
|
state, rt.session.vocab, rt.session.persona,
|
||||||
|
max_tokens=1, region=region,
|
||||||
|
inner_loop_admissibility=True,
|
||||||
|
admissibility_mode="margin",
|
||||||
|
admissibility_margin=0.4,
|
||||||
|
)
|
||||||
|
assert result.tokens == ("question",)
|
||||||
|
step = result.admissibility_trace[0]
|
||||||
|
assert step.verdict.admitted is True
|
||||||
|
assert "margin" in step.verdict.reason
|
||||||
|
# rejected_attempts now carries the *full* ranking
|
||||||
|
assert len(step.rejected_attempts) >= 2
|
||||||
|
words = [w for (_i, w, _s) in step.rejected_attempts]
|
||||||
|
assert "question" in words and "answer" in words
|
||||||
|
|
||||||
|
def test_v2_001_refuses_when_delta_too_high(self) -> None:
|
||||||
|
"""The v2-001 margin is ≈0.597. Setting delta=0.9 must trigger
|
||||||
|
honest refusal — no silent boundary fallback."""
|
||||||
|
rt = ChatRuntime()
|
||||||
|
state, region = _v2_state_and_region(
|
||||||
|
rt, seed="symbol", admissible=["answer", "question"],
|
||||||
|
blade_tok="question", label="v2-001",
|
||||||
|
)
|
||||||
|
with pytest.raises(InnerLoopExhaustion) as exc_info:
|
||||||
|
generate(
|
||||||
|
state, rt.session.vocab, rt.session.persona,
|
||||||
|
max_tokens=1, region=region,
|
||||||
|
inner_loop_admissibility=True,
|
||||||
|
admissibility_mode="margin",
|
||||||
|
admissibility_margin=0.9,
|
||||||
|
)
|
||||||
|
exc = exc_info.value
|
||||||
|
assert exc.reason is RefusalReason.INNER_LOOP_EXHAUSTION
|
||||||
|
assert exc.region_label == "v2-001"
|
||||||
|
# Refusal carries the ranking as evidence
|
||||||
|
words = [w for (_i, w, _s) in exc.rejected_attempts]
|
||||||
|
assert "question" in words and "answer" in words
|
||||||
|
|
||||||
|
def test_threshold_mode_unchanged_by_margin_plumbing(self) -> None:
|
||||||
|
"""Threshold mode (the ADR-0024 default) must produce the same
|
||||||
|
result whether admissibility_mode is "threshold" or unset."""
|
||||||
|
rt = ChatRuntime()
|
||||||
|
state, region = _v2_state_and_region(
|
||||||
|
rt, seed="symbol", admissible=["answer", "question"],
|
||||||
|
blade_tok="question", label="v2-001",
|
||||||
|
)
|
||||||
|
r1 = generate(
|
||||||
|
state, rt.session.vocab, rt.session.persona,
|
||||||
|
max_tokens=1, region=region,
|
||||||
|
inner_loop_admissibility=True,
|
||||||
|
admissibility_threshold=1.122,
|
||||||
|
)
|
||||||
|
r2 = generate(
|
||||||
|
state, rt.session.vocab, rt.session.persona,
|
||||||
|
max_tokens=1, region=region,
|
||||||
|
inner_loop_admissibility=True,
|
||||||
|
admissibility_threshold=1.122,
|
||||||
|
admissibility_mode="threshold",
|
||||||
|
)
|
||||||
|
assert r1.tokens == r2.tokens
|
||||||
|
assert r1.tokens == ("question",)
|
||||||
|
|
||||||
|
|
||||||
|
class TestMarginModeDeterminism:
|
||||||
|
"""5 reruns of the same margin-mode case produce identical traces."""
|
||||||
|
|
||||||
|
def test_margin_mode_replay_stable_across_5_runs(self) -> None:
|
||||||
|
rt = ChatRuntime()
|
||||||
|
state, region = _v2_state_and_region(
|
||||||
|
rt, seed="symbol", admissible=["answer", "question"],
|
||||||
|
blade_tok="question", label="v2-001",
|
||||||
|
)
|
||||||
|
first = generate(
|
||||||
|
state, rt.session.vocab, rt.session.persona,
|
||||||
|
max_tokens=1, region=region,
|
||||||
|
inner_loop_admissibility=True,
|
||||||
|
admissibility_mode="margin",
|
||||||
|
admissibility_margin=0.4,
|
||||||
|
)
|
||||||
|
first_canonical = first.admissibility_trace[0].canonical()
|
||||||
|
for _ in range(4):
|
||||||
|
r = generate(
|
||||||
|
state, rt.session.vocab, rt.session.persona,
|
||||||
|
max_tokens=1, region=region,
|
||||||
|
inner_loop_admissibility=True,
|
||||||
|
admissibility_mode="margin",
|
||||||
|
admissibility_margin=0.4,
|
||||||
|
)
|
||||||
|
assert r.tokens == first.tokens
|
||||||
|
assert r.admissibility_trace[0].canonical() == first_canonical
|
||||||
Loading…
Reference in a new issue