e41a14f76c
Closes audit Finding 3 (2026-05-20).
Pre-fix ``ratify_intent`` defaulted to ``threshold=0.0``, which admits
anything with non-negative ``cga_inner(prompt, anchor)`` — the field
gate (ADR-0022 §TBD-1) was structurally live but semantically
transparent. RATIFIED was logged on essentially every turn because
the CGA inner product over conformal space is not sign-symmetric.
Measurement (``scripts/calibrate_ratification_threshold.py``):
* Runs every cognition eval prompt (45 cases = 13 public + 13 dev +
19 holdout) through a primed ``CognitiveTurnPipeline``.
* Captures the actual ``cga_inner(prompt, anchor)`` score from the
pipeline's own ``_ratify_intent`` via a temporary spy on the
imported ``ratify_intent`` binding.
Observed distribution:
* 34 RATIFIED: min=+1.1039 p10=+1.1039 median=+2.6820 max=+5.7508
* 11 PASSTHROUGH (no vocab-grounded anchor available; score=0.0)
* 0 DEMOTED at any threshold ≤ 1.10
Threshold = 0.5 chosen as the calibrated default:
* Well below the empirical floor of 1.10 — every currently-passing
case stays RATIFIED, byte-identically.
* Clearly non-trivially positive — random Cl(4,1) inner products
fluctuate around zero, so 0.5 demands genuine correlation with
the anchor rather than passive non-negativity.
* Leaves headroom for the gate to actually demote weakly-aligned
off-corpus / adversarial prompts to UNKNOWN and route them
through the honest-refusal surface.
Verification:
* ``core eval cognition`` — public 100/100/91.7/100, holdout
100/100/83.3/100, dev 100/100/78.6/100 — byte-identical to
MEMORY baselines.
* ``core test --suite cognition`` — 120/0/1
* ``core test --suite smoke`` — 67/0
* ``core test --suite runtime`` — 19/0
* 2 new tests in ``tests/test_ratification_threshold_default.py``
pin both the constant and the signature default so a future
change cannot silently regress to ``0.0``.
35 lines
1.3 KiB
Python
35 lines
1.3 KiB
Python
"""Ratification threshold default — Finding 3 (audit 2026-05-20).
|
|
|
|
Pre-fix the default threshold was ``0.0`` and the field gate was
|
|
semantically transparent (any non-negative projection passed). This
|
|
test pins the calibrated default so a future change cannot silently
|
|
revert to a transparent gate without updating the test and the
|
|
calibration evidence comment in
|
|
``generate/intent_ratifier.py:_DEFAULT_RATIFICATION_THRESHOLD``.
|
|
|
|
The calibration data the default rests on was measured by
|
|
``scripts/calibrate_ratification_threshold.py``:
|
|
|
|
* 34 RATIFIED cases across public+dev+holdout splits
|
|
* minimum ``cga_inner(prompt, anchor)`` observed: ~1.10
|
|
* 0.5 sits well below the floor (no regression on any passing case)
|
|
and clearly non-trivially positive (random Cl(4,1) inner products
|
|
fluctuate around zero).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import inspect
|
|
|
|
from generate.intent_ratifier import _DEFAULT_RATIFICATION_THRESHOLD, ratify_intent
|
|
|
|
|
|
def test_default_threshold_is_calibrated() -> None:
|
|
assert _DEFAULT_RATIFICATION_THRESHOLD == 0.5
|
|
|
|
|
|
def test_ratify_intent_signature_uses_calibrated_default() -> None:
|
|
sig = inspect.signature(ratify_intent)
|
|
threshold_param = sig.parameters["threshold"]
|
|
assert threshold_param.default == _DEFAULT_RATIFICATION_THRESHOLD
|
|
assert threshold_param.default != 0.0 # pre-fix value must not resurface
|