core/scripts
Shay e41a14f76c
chore(ratifier): calibrate default ratification threshold 0.0 → 0.5 (#86)
Closes audit Finding 3 (2026-05-20).

Pre-fix ``ratify_intent`` defaulted to ``threshold=0.0``, which admits
anything with non-negative ``cga_inner(prompt, anchor)`` — the field
gate (ADR-0022 §TBD-1) was structurally live but semantically
transparent.  RATIFIED was logged on essentially every turn because
the CGA inner product over conformal space is not sign-symmetric.

Measurement (``scripts/calibrate_ratification_threshold.py``):

  * Runs every cognition eval prompt (45 cases = 13 public + 13 dev +
    19 holdout) through a primed ``CognitiveTurnPipeline``.
  * Captures the actual ``cga_inner(prompt, anchor)`` score from the
    pipeline's own ``_ratify_intent`` via a temporary spy on the
    imported ``ratify_intent`` binding.

Observed distribution:

  * 34 RATIFIED:  min=+1.1039  p10=+1.1039  median=+2.6820  max=+5.7508
  * 11 PASSTHROUGH (no vocab-grounded anchor available; score=0.0)
  *  0 DEMOTED at any threshold ≤ 1.10

Threshold = 0.5 chosen as the calibrated default:

  * Well below the empirical floor of 1.10 — every currently-passing
    case stays RATIFIED, byte-identically.
  * Clearly non-trivially positive — random Cl(4,1) inner products
    fluctuate around zero, so 0.5 demands genuine correlation with
    the anchor rather than passive non-negativity.
  * Leaves headroom for the gate to actually demote weakly-aligned
    off-corpus / adversarial prompts to UNKNOWN and route them
    through the honest-refusal surface.

Verification:

  * ``core eval cognition`` — public 100/100/91.7/100, holdout
    100/100/83.3/100, dev 100/100/78.6/100 — byte-identical to
    MEMORY baselines.
  * ``core test --suite cognition`` — 120/0/1
  * ``core test --suite smoke`` — 67/0
  * ``core test --suite runtime`` — 19/0
  * 2 new tests in ``tests/test_ratification_threshold_default.py``
    pin both the constant and the signature default so a future
    change cannot silently regress to ``0.0``.
2026-05-20 19:59:25 -07:00
..
__init__.py
calibrate_ratification_threshold.py chore(ratifier): calibrate default ratification threshold 0.0 → 0.5 (#86) 2026-05-20 19:59:25 -07:00
generate_discourse_paragraph.py feat(benchmarks): discourse_paragraph lane + pipeline profiler + word-selection tracer 2026-05-16 21:53:46 -07:00
generate_english_fluency_ood.py fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
generate_grammatical_v2.py feat(evals): grammatical-coverage v2 cases - 36 cases with deeper nesting and rarer vocabulary (100% pass) 2026-05-16 06:40:55 -07:00
generate_identity_curriculum.py feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
generate_identity_test_cases.py feat(evals): identity-divergence lane v1 - 93 curriculum events, two axis profiles (Precision/Generosity), divergence/coherence/causal metrics (all pass) 2026-05-16 06:48:13 -07:00
generate_monotonic_cases.py feat(evals): v3 lanes — monotonic-learning passes, adversarial-identity reveals gap 2026-05-16 13:42:47 -07:00
generate_phase5_domain_lanes.py feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
generate_phase5_language_lanes.py feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
publish_pack_measurements.py chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
ratify_anchor_lens_packs.py chore(packs): re-seal register packs (ISSUED_AT bump + missing seals) 2026-05-20 07:39:38 -07:00
ratify_ethics_pack.py feat(packs): expansion round 2 — ethics ×3, anchor-lens ×3, relations-v3, register ×2 (#48) 2026-05-20 07:18:35 -07:00
ratify_identity_packs.py feat(identity-packs): ADR-0027 Phase 5 — ratify all three v1 packs 2026-05-17 19:31:55 -07:00
ratify_register_packs.py chore(packs): re-seal register packs (ISSUED_AT bump + missing seals) 2026-05-20 07:39:38 -07:00
ratify_rhetorical_style_packs.py feat(adr-0087): rhetorical-style pack substrate (loader + default_unstyled_v1) (#74) 2026-05-20 16:19:36 -07:00
ratify_safety_pack.py feat(safety): ADR-0029 — always-loaded, never-replaceable safety pack 2026-05-17 19:56:29 -07:00
review_trace.py
run_examples.py
run_pulse.py feat: Full Proof — surface realizer join, Rust diffusion parity, benchmark harness 2026-05-15 17:39:14 -07:00
scaffold_phase5_domain_lanes.py feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
score_phase5_holdouts.py feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 2026-05-16 20:59:31 -07:00
verify_definitional_closure.py feat(adr-0084): definitional layer — proposal + substrate (schema/loader/closure) (#64) 2026-05-20 15:25:25 -07:00