8146844d90
Phase 2 — Corpus observation runner (inner_loop_runner.py):
- Four-condition matrix: boundary_only / null_control / inner_loop_t0 / inner_loop_tpos.
- Added `inner_loop_force_admit` to generate() — exercises the inner-loop
code path but force-breaks on first candidate. Eval-only null control:
isolates rejection as the causal factor for any pass-rate delta.
- Metrics: pass_rate, mean_rejection_count_per_turn,
non_empty_rejected_attempts_rate, exhaustion_rate (gated at 5%),
mean_admissibility_checks_per_turn, mean/p95 added_latency_ms,
trace_hash_stability across 5 reruns per case.
- Finding on v1+dev: causal_attribution_valid=True, code_path_residual=0.0,
but exhaustion_rate=0.33 at t=0 — chain outer-product blade is
geometrically blind to the active pack.
- Tests (tests/test_inner_loop_phase2.py, 5 pass): pin
causal-attribution and live-corpus trace-hash stability invariants.
Phase 3 — Mechanism-isolation v2 corpus (5 cases, v2_runner.py):
- Synthetic adversarial cases with controlled geometry — each case
specifies seed_token, admissible_tokens, relation_blade_token, and
admissibility_threshold. Field state is constructed directly from
the seed token versor, not via priming.
- For every case: boundary-only selects the forbidden decoy and
inner-loop selects the expected endpoint with the forbidden token
appearing in rejected_attempts.
- Result: mechanism_isolated=true on 5/5. boundary_decoy_rate=1.0,
rejection_traced_rate=1.0. Inner-loop rejection is demonstrably
doing causal semantic work on real packs.
- Tests (tests/test_inner_loop_phase3.py, 8 pass): GATE on
mechanism_isolated.
Phase 4 — Threshold characterization (threshold_characterization.py):
- Distribution mapping per-case AND globally on v1+dev, v2, combined.
- Per-threshold sweep over [-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0].
- Finding: per-case geometry separates cleanly (correct_min > incorrect_max
on every v2 case), BUT no global static threshold passes the
separation_quality >= 0.8 gate. Blade norms vary ~10x across cases.
- Static thresholds (global, relation-typed, or constant frame-derived)
are geometrically insufficient. Per-case-normalized thresholds
(e.g. fraction of blade self-score) are the recommended next step.
- v1 chain-token outer-product cases all skipped — the corpus's chain
tokens (alpha, beta, gamma, delta) are not grounded in the active
pack. Load-bearing finding for ADR-0025 region construction.
- Tests (tests/test_inner_loop_phase4.py, 5 pass): pin the finding
diagnostically (not gated).
Phase 5 — ADR-0025 design note (draft):
- No code changes proposed. Scopes three architectural questions:
(1) home (algebra/versor.py vs field/propagate.py vs generate/) —
preliminary stance: algebra/versor.py.
(2) threshold scheme (blade-normalized fraction recommended over
static; learned/adaptive rejected for determinism).
(3) teaching-loop boundary — Stance A confirmed: rejections are
runtime hygiene only, no entanglement with teaching/*.
- Decisions to be closed before Draft → Accepted.
Phase 1 acceptance criteria from previous commit (7fccf36) carry
forward: wired, deterministic-when-wired, legacy hash preserved.
Suite: 1014 passed, 0 failed, 2 skipped.
107 lines
4 KiB
Python
107 lines
4 KiB
Python
"""Phase 4 threshold characterization invariants (ADR-0024 follow-up).
|
|
|
|
These tests are diagnostic, not gates. They pin the finding so a
|
|
future change that silently improves (or breaks) the geometric
|
|
separability is visible in test output.
|
|
|
|
Findings recorded:
|
|
|
|
* Per-case the relation_blade DOES separate correct from incorrect
|
|
candidates (all five v2 cases pass mechanism-isolation), so the
|
|
blade construction is not geometrically blind.
|
|
* But globally NO STATIC threshold delivers separation_quality ≥ 0.8.
|
|
Blade norms vary across cases (~10x range), so the same threshold
|
|
value means different things case-to-case.
|
|
* The v1 chain-token outer-product blade is ungrounded in the active
|
|
pack — all 9 cases are skipped because chain_tokens (alpha, beta,
|
|
gamma, delta) are not in the en_core_cognition vocab. This is its
|
|
own load-bearing finding for ADR-0025: chain-token blades are
|
|
unsuitable as the default region construction.
|
|
|
|
ADR-0025 design implication: static thresholds (global, relation-typed,
|
|
or frame-derived) are insufficient. Per-case normalized thresholds
|
|
(e.g. fraction of blade self-score) are the next thing to investigate.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from evals.forward_semantic_control.threshold_characterization import characterize
|
|
|
|
V1 = Path("evals/forward_semantic_control/public/v1/cases.jsonl")
|
|
V2 = Path("evals/forward_semantic_control/public/v2/cases.jsonl")
|
|
DEV = Path("evals/forward_semantic_control/dev/cases.jsonl")
|
|
|
|
|
|
def _load(path: Path) -> list[dict]:
|
|
if not path.exists():
|
|
return []
|
|
with path.open() as fh:
|
|
return [json.loads(line) for line in fh if line.strip()]
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def v1_report():
|
|
cases = _load(V1) + _load(DEV)
|
|
if not cases:
|
|
pytest.skip("v1/dev corpus not available")
|
|
return characterize(cases)
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def v2_report():
|
|
cases = _load(V2)
|
|
if not cases:
|
|
pytest.skip("v2 corpus not available")
|
|
return characterize(cases)
|
|
|
|
|
|
class TestV1ChainBladeUngrounded:
|
|
"""V1 chain_tokens are synthetic (alpha, beta, gamma, delta) and
|
|
not present in the active pack. The characterization should
|
|
surface this by skipping every case.
|
|
"""
|
|
|
|
def test_all_v1_cases_skipped(self, v1_report) -> None:
|
|
assert v1_report.metrics["skipped_count"] == v1_report.metrics["case_count"]
|
|
|
|
def test_v1_reports_no_separation(self, v1_report) -> None:
|
|
# No candidates ⇒ best_separation_quality stays at zero.
|
|
assert v1_report.metrics["best_separation_quality"] == 0.0
|
|
|
|
|
|
class TestV2PerCaseSeparates:
|
|
"""Per-case, every v2 case has correct_min > incorrect_max."""
|
|
|
|
def test_every_v2_case_separates_locally(self, v2_report) -> None:
|
|
for detail in v2_report.case_details:
|
|
if detail.get("skipped"):
|
|
continue
|
|
correct = detail["correct_scores"]
|
|
incorrect = detail["incorrect_scores"]
|
|
assert correct, f"case {detail.get('id')} has no correct candidate"
|
|
assert min(correct) > max(incorrect), (
|
|
f"case {detail.get('id')} fails local separation: "
|
|
f"correct_min={min(correct)} ≤ incorrect_max={max(incorrect)}"
|
|
)
|
|
|
|
|
|
class TestV2GlobalNonSeparability:
|
|
"""Despite per-case separability, no static threshold works
|
|
globally — this is the load-bearing finding for ADR-0025."""
|
|
|
|
def test_no_static_threshold_passes_gate(self, v2_report) -> None:
|
|
# If a future change makes this pass, ADR-0025 design may
|
|
# need revision. Currently expected: False.
|
|
assert v2_report.metrics["geometry_supports_static_threshold"] is False
|
|
|
|
def test_score_distributions_overlap_globally(self, v2_report) -> None:
|
|
overlap = v2_report.score_distributions["overlap"]
|
|
# incorrect_max > correct_min ⇒ static threshold cannot
|
|
# separate. This is the geometric fact ADR-0025 must address.
|
|
assert overlap["separable_by_static_threshold"] is False
|
|
assert overlap["overlap_size"] > 0.0
|