8146844d90
Phase 2 — Corpus observation runner (inner_loop_runner.py):
- Four-condition matrix: boundary_only / null_control / inner_loop_t0 / inner_loop_tpos.
- Added `inner_loop_force_admit` to generate() — exercises the inner-loop
code path but force-breaks on first candidate. Eval-only null control:
isolates rejection as the causal factor for any pass-rate delta.
- Metrics: pass_rate, mean_rejection_count_per_turn,
non_empty_rejected_attempts_rate, exhaustion_rate (gated at 5%),
mean_admissibility_checks_per_turn, mean/p95 added_latency_ms,
trace_hash_stability across 5 reruns per case.
- Finding on v1+dev: causal_attribution_valid=True, code_path_residual=0.0,
but exhaustion_rate=0.33 at t=0 — chain outer-product blade is
geometrically blind to the active pack.
- Tests (tests/test_inner_loop_phase2.py, 5 pass): pin
causal-attribution and live-corpus trace-hash stability invariants.
Phase 3 — Mechanism-isolation v2 corpus (5 cases, v2_runner.py):
- Synthetic adversarial cases with controlled geometry — each case
specifies seed_token, admissible_tokens, relation_blade_token, and
admissibility_threshold. Field state is constructed directly from
the seed token versor, not via priming.
- For every case: boundary-only selects the forbidden decoy and
inner-loop selects the expected endpoint with the forbidden token
appearing in rejected_attempts.
- Result: mechanism_isolated=true on 5/5. boundary_decoy_rate=1.0,
rejection_traced_rate=1.0. Inner-loop rejection is demonstrably
doing causal semantic work on real packs.
- Tests (tests/test_inner_loop_phase3.py, 8 pass): GATE on
mechanism_isolated.
Phase 4 — Threshold characterization (threshold_characterization.py):
- Distribution mapping per-case AND globally on v1+dev, v2, combined.
- Per-threshold sweep over [-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0].
- Finding: per-case geometry separates cleanly (correct_min > incorrect_max
on every v2 case), BUT no global static threshold passes the
separation_quality >= 0.8 gate. Blade norms vary ~10x across cases.
- Static thresholds (global, relation-typed, or constant frame-derived)
are geometrically insufficient. Per-case-normalized thresholds
(e.g. fraction of blade self-score) are the recommended next step.
- v1 chain-token outer-product cases all skipped — the corpus's chain
tokens (alpha, beta, gamma, delta) are not grounded in the active
pack. Load-bearing finding for ADR-0025 region construction.
- Tests (tests/test_inner_loop_phase4.py, 5 pass): pin the finding
diagnostically (not gated).
Phase 5 — ADR-0025 design note (draft):
- No code changes proposed. Scopes three architectural questions:
(1) home (algebra/versor.py vs field/propagate.py vs generate/) —
preliminary stance: algebra/versor.py.
(2) threshold scheme (blade-normalized fraction recommended over
static; learned/adaptive rejected for determinism).
(3) teaching-loop boundary — Stance A confirmed: rejections are
runtime hygiene only, no entanglement with teaching/*.
- Decisions to be closed before Draft → Accepted.
Phase 1 acceptance criteria from previous commit (7fccf36) carry
forward: wired, deterministic-when-wired, legacy hash preserved.
Suite: 1014 passed, 0 failed, 2 skipped.
75 lines
2.9 KiB
Python
75 lines
2.9 KiB
Python
"""Phase 3 mechanism-isolation invariants (ADR-0024 v2 corpus).
|
|
|
|
These tests are the *load-bearing* proof contract: in synthetic
|
|
cases designed to exercise the rejection mechanism, the inner loop
|
|
must (a) actually reject the forbidden decoy, (b) select the
|
|
expected endpoint instead, and (c) leave a causal trail in
|
|
``rejected_attempts``.
|
|
|
|
Pass criteria are stricter than Phase 2 (which is observational):
|
|
Phase 3 *gates* on ``mechanism_isolated``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from evals.forward_semantic_control.v2_runner import run_lane
|
|
|
|
V2_CORPUS = Path("evals/forward_semantic_control/public/v2/cases.jsonl")
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def v2_report():
|
|
if not V2_CORPUS.exists():
|
|
pytest.skip("V2 corpus not available")
|
|
with V2_CORPUS.open() as fh:
|
|
cases = [json.loads(line) for line in fh if line.strip()]
|
|
if not cases:
|
|
pytest.skip("V2 corpus is empty")
|
|
return run_lane(cases)
|
|
|
|
|
|
class TestMechanismIsolated:
|
|
def test_mechanism_isolated_overall(self, v2_report) -> None:
|
|
"""The headline gate — every v2 case must isolate the mechanism."""
|
|
assert v2_report.metrics["mechanism_isolated"] is True
|
|
|
|
def test_pass_rate_is_one(self, v2_report) -> None:
|
|
assert v2_report.metrics["pass_rate"] == 1.0
|
|
|
|
def test_boundary_picks_decoy_every_case(self, v2_report) -> None:
|
|
"""If boundary doesn't pick the decoy on a v2 case, the case
|
|
is mis-constructed — the mechanism never gets exercised."""
|
|
assert v2_report.metrics["boundary_decoy_rate"] == 1.0
|
|
|
|
def test_rejection_causally_traced_every_case(self, v2_report) -> None:
|
|
"""The forbidden token must appear in rejected_attempts on
|
|
every case — this is the visible causal evidence."""
|
|
assert v2_report.metrics["rejection_traced_rate"] == 1.0
|
|
|
|
|
|
class TestPerCaseInvariants:
|
|
def test_no_case_was_skipped(self, v2_report) -> None:
|
|
assert v2_report.metrics["skipped_count"] == 0
|
|
|
|
def test_every_case_passed(self, v2_report) -> None:
|
|
for detail in v2_report.case_details:
|
|
assert detail.get("passed") is True, (
|
|
f"Case {detail.get('id')} failed: "
|
|
f"boundary={detail.get('boundary_selected')} "
|
|
f"inner={detail.get('inner_selected')} "
|
|
f"forbidden_traced={detail.get('rejection_in_trace')} "
|
|
f"inner_exhausted={detail.get('inner_exhausted')}"
|
|
)
|
|
|
|
def test_inner_selection_matches_expected_endpoint(self, v2_report) -> None:
|
|
for detail in v2_report.case_details:
|
|
assert detail.get("inner_selected") == detail.get("expected_endpoint")
|
|
|
|
def test_boundary_selection_matches_forbidden_token(self, v2_report) -> None:
|
|
for detail in v2_report.case_details:
|
|
assert detail.get("boundary_selected") == detail.get("forbidden_token")
|