8146844d90
Phase 2 — Corpus observation runner (inner_loop_runner.py):
- Four-condition matrix: boundary_only / null_control / inner_loop_t0 / inner_loop_tpos.
- Added `inner_loop_force_admit` to generate() — exercises the inner-loop
code path but force-breaks on first candidate. Eval-only null control:
isolates rejection as the causal factor for any pass-rate delta.
- Metrics: pass_rate, mean_rejection_count_per_turn,
non_empty_rejected_attempts_rate, exhaustion_rate (gated at 5%),
mean_admissibility_checks_per_turn, mean/p95 added_latency_ms,
trace_hash_stability across 5 reruns per case.
- Finding on v1+dev: causal_attribution_valid=True, code_path_residual=0.0,
but exhaustion_rate=0.33 at t=0 — chain outer-product blade is
geometrically blind to the active pack.
- Tests (tests/test_inner_loop_phase2.py, 5 pass): pin
causal-attribution and live-corpus trace-hash stability invariants.
Phase 3 — Mechanism-isolation v2 corpus (5 cases, v2_runner.py):
- Synthetic adversarial cases with controlled geometry — each case
specifies seed_token, admissible_tokens, relation_blade_token, and
admissibility_threshold. Field state is constructed directly from
the seed token versor, not via priming.
- For every case: boundary-only selects the forbidden decoy and
inner-loop selects the expected endpoint with the forbidden token
appearing in rejected_attempts.
- Result: mechanism_isolated=true on 5/5. boundary_decoy_rate=1.0,
rejection_traced_rate=1.0. Inner-loop rejection is demonstrably
doing causal semantic work on real packs.
- Tests (tests/test_inner_loop_phase3.py, 8 pass): GATE on
mechanism_isolated.
Phase 4 — Threshold characterization (threshold_characterization.py):
- Distribution mapping per-case AND globally on v1+dev, v2, combined.
- Per-threshold sweep over [-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0].
- Finding: per-case geometry separates cleanly (correct_min > incorrect_max
on every v2 case), BUT no global static threshold passes the
separation_quality >= 0.8 gate. Blade norms vary ~10x across cases.
- Static thresholds (global, relation-typed, or constant frame-derived)
are geometrically insufficient. Per-case-normalized thresholds
(e.g. fraction of blade self-score) are the recommended next step.
- v1 chain-token outer-product cases all skipped — the corpus's chain
tokens (alpha, beta, gamma, delta) are not grounded in the active
pack. Load-bearing finding for ADR-0025 region construction.
- Tests (tests/test_inner_loop_phase4.py, 5 pass): pin the finding
diagnostically (not gated).
Phase 5 — ADR-0025 design note (draft):
- No code changes proposed. Scopes three architectural questions:
(1) home (algebra/versor.py vs field/propagate.py vs generate/) —
preliminary stance: algebra/versor.py.
(2) threshold scheme (blade-normalized fraction recommended over
static; learned/adaptive rejected for determinism).
(3) teaching-loop boundary — Stance A confirmed: rejections are
runtime hygiene only, no entanglement with teaching/*.
- Decisions to be closed before Draft → Accepted.
Phase 1 acceptance criteria from previous commit (7fccf36) carry
forward: wired, deterministic-when-wired, legacy hash preserved.
Suite: 1014 passed, 0 failed, 2 skipped.
114 lines
4.1 KiB
Python
114 lines
4.1 KiB
Python
"""Phase 2 corpus-observation invariants (ADR-0024 follow-up).
|
|
|
|
These tests pin the causal-attribution and determinism contracts that
|
|
the Phase 2 runner must hold on the existing FSC corpus. They are
|
|
intentionally *not* gated on rejection_effect or exhaustion_rate —
|
|
those are findings to be characterised in Phase 4, not invariants.
|
|
|
|
What we *do* assert:
|
|
|
|
* ``causal_attribution_valid`` is True: the null control (inner-loop
|
|
code path on, force-admit on) matches boundary-only exactly. Any
|
|
pass-rate delta between inner_loop_t0 and boundary_only is then
|
|
attributable to rejection, not to incidental code-path effects.
|
|
* ``code_path_residual`` is zero (within float tolerance).
|
|
* Trace-hash stability holds for the inner-loop condition on every
|
|
non-skipped case (5 reruns produce identical hashes).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from evals.forward_semantic_control.inner_loop_runner import run_lane
|
|
|
|
_CORPUS_PATHS = (
|
|
Path("evals/forward_semantic_control/public/v1/cases.jsonl"),
|
|
Path("evals/forward_semantic_control/dev/cases.jsonl"),
|
|
)
|
|
|
|
|
|
def _load_corpus() -> list[dict]:
|
|
cases: list[dict] = []
|
|
for path in _CORPUS_PATHS:
|
|
if not path.exists():
|
|
continue
|
|
with path.open() as fh:
|
|
cases.extend(json.loads(line) for line in fh if line.strip())
|
|
return cases
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def phase2_report():
|
|
cases = _load_corpus()
|
|
if not cases:
|
|
pytest.skip("FSC corpus not available")
|
|
return run_lane(cases)
|
|
|
|
|
|
class TestCausalAttribution:
|
|
def test_null_control_matches_boundary_only(self, phase2_report) -> None:
|
|
"""Null control must reproduce boundary-only pass-rate exactly.
|
|
|
|
If this fails, the inner-loop code path is itself altering
|
|
selection (call ordering, telemetry side effects), and any
|
|
rejection_effect we measure is contaminated. ADR-0024 proof
|
|
depends on this invariant.
|
|
"""
|
|
assert phase2_report.metrics["causal_attribution_valid"] is True
|
|
assert phase2_report.metrics["code_path_residual"] == 0.0
|
|
|
|
def test_null_control_per_condition_metrics(self, phase2_report) -> None:
|
|
per = phase2_report.metrics["per_condition"]
|
|
assert per["null_control"]["pass_rate"] == per["boundary_only"]["pass_rate"]
|
|
# Null control must produce zero rejections by construction.
|
|
assert per["null_control"]["mean_rejection_count_per_turn"] == 0
|
|
assert per["null_control"]["non_empty_rejected_attempts_rate"] == 0.0
|
|
assert per["null_control"]["exhaustion_rate"] == 0.0
|
|
|
|
|
|
class TestInnerLoopDeterminismOnCorpus:
|
|
def test_inner_loop_t0_hash_stable_on_every_case(self, phase2_report) -> None:
|
|
"""Live-corpus version of the Phase 1 acceptance test.
|
|
|
|
Stub-vocab determinism is necessary but not sufficient — the
|
|
same property must hold on actual packs, actual field state,
|
|
actual rejection sequences. 5 reruns per case must hash
|
|
identically.
|
|
"""
|
|
rate = phase2_report.metrics["per_condition"]["inner_loop_t0"][
|
|
"trace_hash_stability_pass_rate"
|
|
]
|
|
assert rate == 1.0
|
|
|
|
|
|
class TestPhase2RecordsFindings:
|
|
"""These are not gates — they record the Phase 2 finding so a
|
|
future change that silently flips the sign of rejection_effect or
|
|
closes the exhaustion gap is visible in test output."""
|
|
|
|
def test_runner_emits_required_metric_keys(self, phase2_report) -> None:
|
|
required = {
|
|
"per_condition",
|
|
"rejection_effect",
|
|
"code_path_residual",
|
|
"causal_attribution_valid",
|
|
"exhaustion_ceiling",
|
|
"exhaustion_gate_pass",
|
|
"probe_threshold_positive",
|
|
"case_count",
|
|
"skipped_count",
|
|
}
|
|
assert required <= set(phase2_report.metrics.keys())
|
|
|
|
def test_all_four_conditions_present(self, phase2_report) -> None:
|
|
per = phase2_report.metrics["per_condition"]
|
|
assert set(per.keys()) == {
|
|
"boundary_only",
|
|
"null_control",
|
|
"inner_loop_t0",
|
|
"inner_loop_tpos",
|
|
}
|