core/scripts
Shay e41a14f76c
chore(ratifier): calibrate default ratification threshold 0.0 → 0.5 (#86)
Closes audit Finding 3 (2026-05-20).

Pre-fix ``ratify_intent`` defaulted to ``threshold=0.0``, which admits
anything with non-negative ``cga_inner(prompt, anchor)`` — the field
gate (ADR-0022 §TBD-1) was structurally live but semantically
transparent.  RATIFIED was logged on essentially every turn because
the CGA inner product over conformal space is not sign-symmetric.

Measurement (``scripts/calibrate_ratification_threshold.py``):

  * Runs every cognition eval prompt (45 cases = 13 public + 13 dev +
    19 holdout) through a primed ``CognitiveTurnPipeline``.
  * Captures the actual ``cga_inner(prompt, anchor)`` score from the
    pipeline's own ``_ratify_intent`` via a temporary spy on the
    imported ``ratify_intent`` binding.

Observed distribution:

  * 34 RATIFIED:  min=+1.1039  p10=+1.1039  median=+2.6820  max=+5.7508
  * 11 PASSTHROUGH (no vocab-grounded anchor available; score=0.0)
  *  0 DEMOTED at any threshold ≤ 1.10

Threshold = 0.5 chosen as the calibrated default:

  * Well below the empirical floor of 1.10 — every currently-passing
    case stays RATIFIED, byte-identically.
  * Clearly non-trivially positive — random Cl(4,1) inner products
    fluctuate around zero, so 0.5 demands genuine correlation with
    the anchor rather than passive non-negativity.
  * Leaves headroom for the gate to actually demote weakly-aligned
    off-corpus / adversarial prompts to UNKNOWN and route them
    through the honest-refusal surface.

Verification:

  * ``core eval cognition`` — public 100/100/91.7/100, holdout
    100/100/83.3/100, dev 100/100/78.6/100 — byte-identical to
    MEMORY baselines.
  * ``core test --suite cognition`` — 120/0/1
  * ``core test --suite smoke`` — 67/0
  * ``core test --suite runtime`` — 19/0
  * 2 new tests in ``tests/test_ratification_threshold_default.py``
    pin both the constant and the signature default so a future
    change cannot silently regress to ``0.0``.
2026-05-20 19:59:25 -07:00
..
__init__.py
calibrate_ratification_threshold.py chore(ratifier): calibrate default ratification threshold 0.0 → 0.5 (#86) 2026-05-20 19:59:25 -07:00
generate_discourse_paragraph.py
generate_english_fluency_ood.py
generate_grammatical_v2.py
generate_identity_curriculum.py
generate_identity_test_cases.py
generate_monotonic_cases.py
generate_phase5_domain_lanes.py
generate_phase5_language_lanes.py
publish_pack_measurements.py chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
ratify_anchor_lens_packs.py chore(packs): re-seal register packs (ISSUED_AT bump + missing seals) 2026-05-20 07:39:38 -07:00
ratify_ethics_pack.py feat(packs): expansion round 2 — ethics ×3, anchor-lens ×3, relations-v3, register ×2 (#48) 2026-05-20 07:18:35 -07:00
ratify_identity_packs.py
ratify_register_packs.py chore(packs): re-seal register packs (ISSUED_AT bump + missing seals) 2026-05-20 07:39:38 -07:00
ratify_rhetorical_style_packs.py feat(adr-0087): rhetorical-style pack substrate (loader + default_unstyled_v1) (#74) 2026-05-20 16:19:36 -07:00
ratify_safety_pack.py
review_trace.py
run_examples.py
run_pulse.py
scaffold_phase5_domain_lanes.py
score_phase5_holdouts.py
verify_definitional_closure.py feat(adr-0084): definitional layer — proposal + substrate (schema/loader/closure) (#64) 2026-05-20 15:25:25 -07:00