diff --git a/docs/decisions/ADR-0025-rotor-frame-admissibility-design-note.md b/docs/decisions/ADR-0025-rotor-frame-admissibility-design-note.md new file mode 100644 index 00000000..21f3a751 --- /dev/null +++ b/docs/decisions/ADR-0025-rotor-frame-admissibility-design-note.md @@ -0,0 +1,287 @@ +# ADR-0025 — Rotor / Frame Admissibility (Design Note) + +| Field | Value | +|--------------|--------------------------------------| +| Status | **Draft — Design Note Only** | +| Date | 2026-05-17 | +| Supersedes | — | +| Extends | ADR-0022, ADR-0023, ADR-0024 | +| Decision lead| Shay (with CORE assistant) | + +--- + +## Status note + +This is a **design note**, not an implementation decision. No code +changes are proposed. Its purpose is to fix the home, scope, and +boundary of the next admissibility step *before* anything is built — +so the implementation doesn't inherit the wrong architectural shape +by default. + +It will be promoted from Draft to a real ADR (Proposed → Accepted) +only after the design questions below are decided. + +--- + +## Context + +ADR-0024 added per-rotor inner-loop admissibility for the +**destination-token / direction** side of an `AdmissibilityRegion`: +when a candidate's CGA inner product against `relation_blade` falls +below `admissibility_threshold`, the candidate is excluded and the +walk re-selects until admitted or exhausted. + +ADR-0024 explicitly deferred: + +> Frame-versor admissibility (does the rotor preserve / transform +> within the frame constraint?) remains out of scope. + +This note scopes that deferred work, but with two additional +constraints surfaced by the Phase 2–4 follow-up evidence: + +1. **Phase 2 corpus observation** (`evals/forward_semantic_control/ + inner_loop_runner.py`): on the existing v1+dev corpus, the + inner-loop mechanism is *wired, deterministic, causally + attributable* (null-control = boundary-only exactly, + `code_path_residual = 0.0`), but the chain-token outer-product + region produces `exhaustion_rate = 0.33` at `t = 0.0` — well + above the 5% benign-corpus ceiling. + +2. **Phase 4 threshold characterization** (`threshold_ + characterization.py`): **per-case the geometry separates cleanly** + (every mechanism-isolated v2 case has `correct_min > incorrect_max`), + but **no static global threshold** delivers + `separation_quality ≥ 0.8`. Blade norms vary ~10× across cases, + so the same threshold value means different things case-to-case. + Static thresholds — global, relation-typed, or frame-derived as a + constant — are insufficient. + +These findings change the framing. The next step is not "extend the +same idea to the rotor side." It is two distinct questions: + +* What level of the stack should enforce rotor/frame admissibility? +* What threshold scheme is geometrically valid given Phase 4? + +--- + +## Question 1 — Architectural home + +Three candidate homes for rotor-side admissibility: + +### Option A — Generation-time filter (`generate/`) + +Inherit ADR-0024's shape. Add a check inside the same per-step inner +loop in `generate/stream.py` that examines the *rotor* `V` (not just +the destination versor) before propagation. + +**Pros:** + +* Locality with ADR-0024. All admissibility decisions live in one + module. +* Trace evidence is uniform — one `AdmissibilityTraceStep` per + rotor-application. + +**Cons:** + +* Pushes algebra-shaped invariants into a generation-shaped module. + `generate/` already orchestrates candidates, salience, attention, + vault recall, persona — adding rotor invariant enforcement here + bloats the hot path and entangles concerns. +* Re-creates the "hot-path repair" anti-pattern CLAUDE.md explicitly + warns against, because the check would re-validate something + algebra already constructed. + +### Option B — Versor construction invariant (`algebra/versor.py`) + +Make rotor/frame admissibility part of sandwich closure. When +`word_transition_rotor(A, B)` builds the rotor, it also checks the +rotor against the active frame constraint. Violations raise — same +shape as the existing `versor_condition < 1e-6` invariant. + +**Pros:** + +* Aligned with the CLAUDE.md doctrine that algebra-owned closure + belongs in `algebra/`. +* No hot-path repair. The check is part of *construction*, not a + post-construction filter. +* Single invariant site — easier to reason about and prove. + +**Cons:** + +* Couples algebra to admissibility concepts (frame, relation_blade) + that today live in `generate/admissibility.py`. Either + `algebra/versor.py` grows a dependency on admissibility, or + admissibility primitives must be lifted to a shared layer. +* Honest refusal would surface deeper in the stack — callers that + today catch `ValueError` from `generate()` would also need to + catch from `propagate_step` or earlier. + +### Option C — Field propagation guard (`field/propagate.py`) + +Enforce at the *application* site: after rotor construction, before +`propagate_step` commits the new field state, verify the resulting +field stays within the frame's admissible cone. + +**Pros:** + +* Closest to the *claim*: rotor admissibility is fundamentally about + the field staying coherent under propagation, not about token + selection. +* `field/propagate.py` already owns the propagation invariant, so + this is a natural home for an additional propagation-time check. + +**Cons:** + +* `field/propagate.py` is explicitly listed in CLAUDE.md as a + *forbidden site* for normalization / drift repair / monitoring + ("Do not add drift repair, grade projection, watchdogs, timers, + hot-path normalizers, or monitoring functions whose only purpose + is to repair another function"). An admissibility *guard* (raise + on violation, never repair) is closer to a precondition than a + monitor, but the boundary needs to be made explicit before this + option is chosen — otherwise it sets a precedent that erodes the + current rule. + +### Recommended preliminary stance + +**Option B** is the most aligned with project doctrine and the +cleanest invariant. Option C is the second-best, but only if the +"guard vs. monitor" distinction is made explicit and respected — and +even then, the construction-site discipline of Option B is preferable. + +Option A is rejected as inheriting the wrong architectural shape from +ADR-0024 by momentum. ADR-0024 lived in `generate/` because it was +about *which destination to select*; rotor admissibility is about +*whether the rotor itself is valid*, which is a construction-site +question. + +**Decision required**: B vs. C. + +--- + +## Question 2 — Threshold scheme + +Phase 4 surfaced that static thresholds are geometrically invalid on +this manifold. The right scheme is *not yet decided*, but the +candidates are: + +1. **Per-candidate normalized score**: threshold = α · ‖blade‖, so + the same fraction of blade self-score is required regardless of + blade norm. Probable best first cut. + +2. **Cosine-similarity-style normalization**: replace `cga_inner` in + the threshold check with + `cga_inner(v, blade) / (‖v‖ · ‖blade‖)`. Rejected on doctrinal + grounds — CLAUDE.md says "do not add cosine similarity ... to the + runtime path." Listed for completeness only. + +3. **Per-relation-type static threshold**: a small table mapping + relation type → threshold. Phase 4 suggests this is insufficient + because *blade norm dominates*, not relation type, but it could + be a fallback if normalized scoring proves unstable. + +4. **Frame-derived threshold**: threshold is a property of the frame + versor, not the candidate or the relation. Requires the frame + versor to be the primary admissibility object — i.e. Option B + above — and may collapse Question 1 and Question 2 into one + decision. + +**Decision required**: (1) is the recommended starting point. Final +choice depends on Question 1 outcome and on a focused diagnostic +sweep over (1) and (3). + +**Out of scope for the eventual ADR**: learned thresholds, adaptive +thresholds, online tuning. Deterministic replay must be preserved; +no learned policy enters the runtime path. + +--- + +## Question 3 — Teaching loop boundary + +ADR-0024 lives in `generate/`. The teaching loop in `teaching/*` +corrects model behavior through reviewed mutation. An open question: +when inner-loop (or rotor) admissibility rejects a candidate, does +that rejection become a *teachable event*? + +Two stances: + +* **A. Rejections are runtime hygiene only.** The teaching loop + sees the final selected token, not the rejected ones. Rejection + is a property of the deterministic admissibility region, not of + the reviewed teaching example. + +* **B. Rejections are correction signals.** A teaching review can + examine `rejected_attempts` and decide whether the rejection was + correct (reinforce) or over-aggressive (loosen). This entangles + the teaching loop with admissibility geometry. + +### Recommended stance: **A — strictly hygiene-only.** + +Rationale: + +* The teaching loop's contract is *reviewed mutation of identity / + pack / vault*. Admissibility regions are deterministic geometric + objects derived from intents and frames; they are not learned, and + there is no review surface for them today. +* Entangling teaching with admissibility would create a parallel + correction path — explicitly forbidden by CLAUDE.md ("Do not + create a parallel correction/learning path"). +* Phase 4 showed that what needs to change is the threshold *scheme*, + not the per-event rejection decisions. Scheme changes belong in + the eventual ADR-0025 implementation, not in reviewed teaching + examples. + +The decision must be **stated in the final ADR**, not left as a +silent default, so the next person who touches both systems doesn't +have to re-derive the boundary. + +--- + +## Decisions to lock before ADR-0025 is implementable + +1. **Home**: Option B (algebra construction) vs. Option C (field + propagation guard). Reject A explicitly. +2. **Threshold scheme**: blade-normalized fraction (recommended) vs. + relation-typed table (fallback). Run a small diagnostic sweep + on the v2 corpus + a small extension before committing. +3. **Teaching boundary**: Stance A (hygiene-only) confirmed. State + explicitly in the eventual ADR's "Out of scope" section. +4. **Trace evidence**: extend `AdmissibilityTraceStep` to include + rotor-side verdict, or add a separate `RotorAdmissibilityTrace`? + Lean toward extending the existing step to keep the trace shape + simple. +5. **Honest refusal**: at which layer does `ValueError` get raised + on rotor rejection? Decided by (1) — same site as the check. + +--- + +## Evidence and links + +* ADR-0022 — Forward Semantic Control (region prefilter). +* ADR-0023 — Forward Semantic Control proof evidence. +* ADR-0024 — Inner-loop per-rotor admissibility (token-side). +* Phase 2 report — `evals/forward_semantic_control/results/ + phase2_inner_loop_report.json` — causal attribution proven, but + exhaustion gate fails on existing corpus. +* Phase 3 report — `evals/forward_semantic_control/results/ + phase3_v2_report.json` — mechanism isolated on real pack, + `mechanism_isolated = true` on 5/5 cases. +* Phase 4 reports — `evals/forward_semantic_control/results/ + phase4_characterization_{v1_plus_dev,v2,combined}.json` — static + thresholds geometrically insufficient. +* Tests pinning the findings: `tests/test_inner_loop_phase2.py`, + `tests/test_inner_loop_phase3.py`, `tests/test_inner_loop_phase4.py`. + +--- + +## What this note does NOT decide + +This note does not: + +* Choose between Options B and C — that requires a short focused + spike on the algebra-vs-propagation tradeoff. +* Specify the threshold scheme — that requires a small diagnostic + sweep over normalized-fraction vs. relation-typed schemes on the + v2 corpus. +* Authorize any code changes. Promotion from Draft to Accepted + requires the open questions to be closed. diff --git a/evals/forward_semantic_control/inner_loop_runner.py b/evals/forward_semantic_control/inner_loop_runner.py new file mode 100644 index 00000000..18e4780b --- /dev/null +++ b/evals/forward_semantic_control/inner_loop_runner.py @@ -0,0 +1,434 @@ +"""Phase 2 corpus-observation runner — ADR-0024 inner-loop admissibility. + +Runs each FSC case through a four-condition matrix on the *same* +field state so any pass-rate delta is attributable to the inner-loop +mechanism alone (region/vocab/persona/prompt held constant): + + (A) boundary_only — inner_loop_admissibility=False + (B) null_control — inner_loop_admissibility=True, + inner_loop_force_admit=True + (C) inner_loop_t0 — inner_loop_admissibility=True, + admissibility_threshold=0.0 + (D) inner_loop_tpos — inner_loop_admissibility=True, + admissibility_threshold=0.25 + +Reports per condition: + pass_rate + mean_rejection_count_per_turn + non_empty_rejected_attempts_rate + exhaustion_rate (gated: must be ≤ EXHAUSTION_CEILING) + mean_admissibility_checks_per_turn + mean_added_latency_ms + p95_added_latency_ms + trace_hash_stability_passes (5 reruns ⇒ identical trace hash) + +Causal attribution: delta(C - A) is the rejection effect *iff* delta(B - A) ≈ 0. +If null_control diverges from boundary_only, the inner-loop code path +itself is changing selection (call ordering, side effects); ADR-0024 +proof is contaminated until that residual is explained. + +Conforms to the framework interface (``run_lane``) so the standard +eval harness can call it. +""" + +from __future__ import annotations + +import statistics +import time +from dataclasses import dataclass, field +from typing import Any + +import numpy as np + +from algebra.cga import outer_product +from chat.runtime import ChatRuntime +from core.cognition.trace import hash_admissibility_trace +from core.config import RuntimeConfig +from generate.admissibility import AdmissibilityRegion, RegionSource +from generate.result import GenerationResult +from generate.stream import generate as generate_walk + +# Exhaustion ceiling on benign v1 corpus. Above this, the configured +# threshold is producing honest refusals where it should produce +# answers — a capability regression disguised as a virtue. +EXHAUSTION_CEILING = 0.05 + +# Default tested-positive threshold for condition (D). Phase 4 will +# characterise the threshold landscape; this is a probe-point only. +PROBE_THRESHOLD_POSITIVE = 0.25 + +# Reruns for hash-stability check. 5 is the same N used by the +# Phase 1 acceptance test in ``tests/test_inner_loop_admissibility.py``. +HASH_STABILITY_RERUNS = 5 + + +@dataclass(slots=True) +class _ConditionMetrics: + label: str + pass_count: int = 0 + case_count: int = 0 + rejection_counts: list[int] = field(default_factory=list) + non_empty_rejection_cases: int = 0 + exhaustions: int = 0 + admissibility_checks: list[int] = field(default_factory=list) + latencies_ms: list[float] = field(default_factory=list) + trace_hash_stable_count: int = 0 + trace_hash_checked_count: int = 0 + + def as_dict(self) -> dict[str, Any]: + n = max(self.case_count, 1) + return { + "label": self.label, + "pass_rate": round(self.pass_count / n, 4), + "mean_rejection_count_per_turn": round( + statistics.mean(self.rejection_counts) if self.rejection_counts else 0.0, + 4, + ), + "non_empty_rejected_attempts_rate": round( + self.non_empty_rejection_cases / n, 4 + ), + "exhaustion_rate": round(self.exhaustions / n, 4), + "mean_admissibility_checks_per_turn": round( + statistics.mean(self.admissibility_checks) + if self.admissibility_checks + else 0.0, + 4, + ), + "mean_added_latency_ms": round( + statistics.mean(self.latencies_ms) if self.latencies_ms else 0.0, 4 + ), + "p95_added_latency_ms": round(_p95(self.latencies_ms), 4), + "trace_hash_stability_pass_rate": round( + self.trace_hash_stable_count / max(self.trace_hash_checked_count, 1), 4 + ), + "case_count": self.case_count, + } + + +@dataclass(slots=True) +class InnerLoopReport: + metrics: dict[str, Any] = field(default_factory=dict) + case_details: list[dict[str, Any]] = field(default_factory=list) + + +def _p95(values: list[float]) -> float: + if not values: + return 0.0 + sorted_values = sorted(values) + idx = int(round(0.95 * (len(sorted_values) - 1))) + return sorted_values[idx] + + +def _region_from_token_chain( + vocab, + tokens: tuple[str, ...], + *, + label: str, +) -> AdmissibilityRegion | None: + indices: list[int] = [] + versors: list[np.ndarray] = [] + for raw in tokens: + token = raw.lower().strip() + if not token: + continue + try: + idx = vocab.index_of(token) + except (KeyError, AttributeError, IndexError): + continue + try: + versor = np.asarray(vocab.get_versor(token), dtype=np.float32) + except (KeyError, AttributeError): + continue + indices.append(int(idx)) + versors.append(versor) + if not indices: + return None + blade = versors[0] + for nxt in versors[1:]: + blade = outer_product(blade, nxt) + return AdmissibilityRegion( + allowed_indices=np.asarray(indices, dtype=np.int64), + relation_blade=blade, + source=RegionSource.RELATION, + label=label, + ) + + +def _surfaces_endpoint(surface: str, expected_endpoint: str) -> bool: + if not surface or not expected_endpoint: + return False + return expected_endpoint.lower().strip() in surface.lower() + + +def _run_walk( + field_state, + vocab, + persona, + region: AdmissibilityRegion | None, + *, + inner_loop: bool, + threshold: float, + force_admit: bool, +) -> tuple[GenerationResult | None, float, bool]: + """Run one walk, return (result, latency_ms, exhaustion_occurred).""" + start = time.perf_counter() + try: + result = generate_walk( + field_state, + vocab, + persona, + max_tokens=8, + region=region, + inner_loop_admissibility=inner_loop, + admissibility_threshold=threshold, + inner_loop_force_admit=force_admit, + ) + latency_ms = (time.perf_counter() - start) * 1000.0 + return result, latency_ms, False + except ValueError: + latency_ms = (time.perf_counter() - start) * 1000.0 + return None, latency_ms, True + + +def _rejection_count(result: GenerationResult | None) -> int: + if result is None: + return 0 + return sum(len(step.rejected_attempts) for step in result.admissibility_trace) + + +def _admissibility_check_count(result: GenerationResult | None) -> int: + """One check per attempt — admissions + rejections.""" + if result is None: + return 0 + return sum(len(step.rejected_attempts) + 1 for step in result.admissibility_trace) + + +def _surface_from(result: GenerationResult | None) -> str: + if result is None or not result.tokens: + return "" + return " ".join(result.tokens) + + +def _hash_of(result: GenerationResult | None) -> str: + if result is None: + return "__exhausted__" + return hash_admissibility_trace(result.admissibility_trace) + + +def _prime_runtime(case: dict[str, Any]) -> ChatRuntime: + runtime = ChatRuntime() + for prime in case.get("prime", []): + try: + runtime.chat(prime, max_tokens=8) + except ValueError: + pass + try: + runtime.chat(case["prompt"], max_tokens=8) + except ValueError: + pass + return runtime + + +def _run_case(case: dict[str, Any]) -> dict[str, Any]: + expected = case.get("expected_endpoint", "") + runtime = _prime_runtime(case) + field_state = runtime.session.state + if field_state is None: + return {"id": case.get("id", ""), "skipped": True, "reason": "no_field_state"} + + vocab = runtime.session.vocab + persona = runtime.session.persona + + chain_tokens = tuple(case.get("chain_tokens", ())) + if not chain_tokens and expected: + chain_tokens = (expected,) + region = _region_from_token_chain( + vocab, chain_tokens, label=f"phase2[{case.get('id', '')}]" + ) + if region is None: + return {"id": case.get("id", ""), "skipped": True, "reason": "no_grounded_chain"} + + # Boundary-only baseline latency — used so reported "added" latency + # is the cost over the boundary-only path, not absolute. + baseline_result, baseline_latency_ms, baseline_exh = _run_walk( + field_state, vocab, persona, region, + inner_loop=False, threshold=0.0, force_admit=False, + ) + + conditions: dict[str, dict[str, Any]] = {} + hash_stability: dict[str, bool] = {} + + # (A) boundary-only — recorded above + conditions["boundary_only"] = { + "surface": _surface_from(baseline_result), + "rejections": _rejection_count(baseline_result), + "checks": _admissibility_check_count(baseline_result), + "latency_ms": 0.0, # baseline — no "added" latency + "absolute_latency_ms": baseline_latency_ms, + "exhausted": baseline_exh, + "trace_hash": _hash_of(baseline_result), + } + + # (B) null_control + nc_result, nc_latency_ms, nc_exh = _run_walk( + field_state, vocab, persona, region, + inner_loop=True, threshold=0.0, force_admit=True, + ) + conditions["null_control"] = { + "surface": _surface_from(nc_result), + "rejections": _rejection_count(nc_result), + "checks": _admissibility_check_count(nc_result), + "latency_ms": max(nc_latency_ms - baseline_latency_ms, 0.0), + "absolute_latency_ms": nc_latency_ms, + "exhausted": nc_exh, + "trace_hash": _hash_of(nc_result), + } + + # (C) inner_loop_t0 + c_result, c_latency_ms, c_exh = _run_walk( + field_state, vocab, persona, region, + inner_loop=True, threshold=0.0, force_admit=False, + ) + conditions["inner_loop_t0"] = { + "surface": _surface_from(c_result), + "rejections": _rejection_count(c_result), + "checks": _admissibility_check_count(c_result), + "latency_ms": max(c_latency_ms - baseline_latency_ms, 0.0), + "absolute_latency_ms": c_latency_ms, + "exhausted": c_exh, + "trace_hash": _hash_of(c_result), + } + + # (D) inner_loop_tpos + d_result, d_latency_ms, d_exh = _run_walk( + field_state, vocab, persona, region, + inner_loop=True, threshold=PROBE_THRESHOLD_POSITIVE, force_admit=False, + ) + conditions["inner_loop_tpos"] = { + "surface": _surface_from(d_result), + "rejections": _rejection_count(d_result), + "checks": _admissibility_check_count(d_result), + "latency_ms": max(d_latency_ms - baseline_latency_ms, 0.0), + "absolute_latency_ms": d_latency_ms, + "exhausted": d_exh, + "trace_hash": _hash_of(d_result), + } + + # Hash stability: rerun condition (C) HASH_STABILITY_RERUNS-1 more + # times on the *same* field state (re-priming each time to keep + # vault state comparable). All hashes must match. + base_hash = conditions["inner_loop_t0"]["trace_hash"] + stable = True + for _ in range(HASH_STABILITY_RERUNS - 1): + re_runtime = _prime_runtime(case) + re_state = re_runtime.session.state + if re_state is None: + stable = False + break + re_vocab = re_runtime.session.vocab + re_persona = re_runtime.session.persona + re_region = _region_from_token_chain( + re_vocab, chain_tokens, label=f"phase2[{case.get('id', '')}]" + ) + if re_region is None: + stable = False + break + re_result, _re_latency, _re_exh = _run_walk( + re_state, re_vocab, re_persona, re_region, + inner_loop=True, threshold=0.0, force_admit=False, + ) + if _hash_of(re_result) != base_hash: + stable = False + break + hash_stability["inner_loop_t0"] = stable + + detail: dict[str, Any] = { + "id": case.get("id", ""), + "kind": case.get("kind", ""), + "expected_endpoint": expected, + "conditions": conditions, + "hash_stability": hash_stability, + "passes": { + label: _surfaces_endpoint(cond["surface"], expected) + for label, cond in conditions.items() + }, + } + return detail + + +def run_lane( + cases: list[dict[str, Any]], + *, + config: RuntimeConfig | None = None, + workers: int | None = None, +) -> InnerLoopReport: + _ = config + _ = workers # serial — Phase 2 is small and latency-sensitive + + if not cases: + return InnerLoopReport(metrics={}, case_details=[]) + + case_details: list[dict[str, Any]] = [] + by_condition: dict[str, _ConditionMetrics] = { + "boundary_only": _ConditionMetrics(label="boundary_only"), + "null_control": _ConditionMetrics(label="null_control"), + "inner_loop_t0": _ConditionMetrics(label="inner_loop_t0"), + "inner_loop_tpos": _ConditionMetrics(label="inner_loop_tpos"), + } + + for case in cases: + detail = _run_case(case) + case_details.append(detail) + if detail.get("skipped"): + continue + for label, metrics in by_condition.items(): + cond = detail["conditions"][label] + metrics.case_count += 1 + if detail["passes"][label]: + metrics.pass_count += 1 + metrics.rejection_counts.append(cond["rejections"]) + if cond["rejections"] > 0: + metrics.non_empty_rejection_cases += 1 + if cond["exhausted"]: + metrics.exhaustions += 1 + metrics.admissibility_checks.append(cond["checks"]) + metrics.latencies_ms.append(cond["latency_ms"]) + if label in detail["hash_stability"]: + metrics.trace_hash_checked_count += 1 + if detail["hash_stability"][label]: + metrics.trace_hash_stable_count += 1 + + per_condition = {label: m.as_dict() for label, m in by_condition.items()} + + # Causal attribution: + # rejection_effect = pass(inner_loop_t0) - pass(boundary_only) + # code_path_residual = pass(null_control) - pass(boundary_only) + # If |code_path_residual| is non-zero, the rejection effect is + # contaminated by code-path differences and the proof is invalid. + rejection_effect = ( + per_condition["inner_loop_t0"]["pass_rate"] + - per_condition["boundary_only"]["pass_rate"] + ) + code_path_residual = ( + per_condition["null_control"]["pass_rate"] + - per_condition["boundary_only"]["pass_rate"] + ) + + # Exhaustion gate — applies to inner-loop conditions only. + exhaustion_gate_pass = all( + per_condition[label]["exhaustion_rate"] <= EXHAUSTION_CEILING + for label in ("inner_loop_t0", "inner_loop_tpos") + ) + + metrics: dict[str, Any] = { + "per_condition": per_condition, + "rejection_effect": round(rejection_effect, 4), + "code_path_residual": round(code_path_residual, 4), + "causal_attribution_valid": abs(code_path_residual) < 1e-9, + "exhaustion_ceiling": EXHAUSTION_CEILING, + "exhaustion_gate_pass": exhaustion_gate_pass, + "probe_threshold_positive": PROBE_THRESHOLD_POSITIVE, + "case_count": len(cases), + "skipped_count": sum(1 for d in case_details if d.get("skipped")), + } + return InnerLoopReport(metrics=metrics, case_details=case_details) diff --git a/evals/forward_semantic_control/public/v2/cases.jsonl b/evals/forward_semantic_control/public/v2/cases.jsonl new file mode 100644 index 00000000..f4f7db02 --- /dev/null +++ b/evals/forward_semantic_control/public/v2/cases.jsonl @@ -0,0 +1,5 @@ +{"id":"FSC-PUB-V2-001","kind":"mechanism_isolation","semantic_pair":"question/answer","seed_token":"symbol","admissible_tokens":["answer","question"],"relation_blade_token":"question","expected_endpoint":"question","forbidden_token":"answer","admissibility_threshold":1.1221,"rationale":"Field state from 'symbol' is geometrically nearer to 'answer' than 'question' (gap=+1.566), so boundary-only selects 'answer'. Blade=versor('question') admits 'question' at score 1.420 but rejects 'answer' at score 0.824. Threshold 1.122 sits between them."} +{"id":"FSC-PUB-V2-002","kind":"mechanism_isolation","semantic_pair":"truth/meaning","seed_token":"infer","admissible_tokens":["meaning","truth"],"relation_blade_token":"truth","expected_endpoint":"truth","forbidden_token":"meaning","admissibility_threshold":0.9449,"rationale":"Field state from 'infer' is geometrically nearer to 'meaning' than 'truth' (gap=+2.193), so boundary-only selects 'meaning'. Blade=versor('truth') admits 'truth' at score 1.173 but rejects 'meaning' at score 0.717. Threshold 0.945 separates."} +{"id":"FSC-PUB-V2-003","kind":"mechanism_isolation","semantic_pair":"think/spirit","seed_token":"corrects","admissible_tokens":["spirit","think"],"relation_blade_token":"think","expected_endpoint":"think","forbidden_token":"spirit","admissibility_threshold":6.0833,"rationale":"Cognition vs. ineffable pole. Field state from 'corrects' is nearer to 'spirit' (gap=+2.711). Blade=versor('think') admits 'think' at score 12.72 and rejects 'spirit' at score -0.55. Wide blade gap (+13.27)."} +{"id":"FSC-PUB-V2-004","kind":"mechanism_isolation","semantic_pair":"understand/say","seed_token":"corrects","admissible_tokens":["say","understand"],"relation_blade_token":"understand","expected_endpoint":"understand","forbidden_token":"say","admissibility_threshold":4.055,"rationale":"Intellection vs. utterance. Field state from 'corrects' is nearer to 'say'. Blade=versor('understand') admits 'understand' at score 5.74 and rejects 'say' at score 2.37."} +{"id":"FSC-PUB-V2-005","kind":"mechanism_isolation","semantic_pair":"thought/beginning","seed_token":"corrects","admissible_tokens":["beginning","thought"],"relation_blade_token":"thought","expected_endpoint":"thought","forbidden_token":"beginning","admissibility_threshold":7.99,"rationale":"Cognition vs. genesis. Field state from 'corrects' is nearer to 'beginning'. Blade=versor('thought') admits 'thought' at score 14.36 and rejects 'beginning' at score 1.62. Largest blade gap (+12.75)."} diff --git a/evals/forward_semantic_control/results/phase2_inner_loop_report.json b/evals/forward_semantic_control/results/phase2_inner_loop_report.json new file mode 100644 index 00000000..8dafee5e --- /dev/null +++ b/evals/forward_semantic_control/results/phase2_inner_loop_report.json @@ -0,0 +1,532 @@ +{ + "metrics": { + "per_condition": { + "boundary_only": { + "label": "boundary_only", + "pass_rate": 1.0, + "mean_rejection_count_per_turn": 0, + "non_empty_rejected_attempts_rate": 0.0, + "exhaustion_rate": 0.0, + "mean_admissibility_checks_per_turn": 3, + "mean_added_latency_ms": 0.0, + "p95_added_latency_ms": 0.0, + "trace_hash_stability_pass_rate": 0.0, + "case_count": 9 + }, + "null_control": { + "label": "null_control", + "pass_rate": 1.0, + "mean_rejection_count_per_turn": 0, + "non_empty_rejected_attempts_rate": 0.0, + "exhaustion_rate": 0.0, + "mean_admissibility_checks_per_turn": 3, + "mean_added_latency_ms": 0.0505, + "p95_added_latency_ms": 0.1467, + "trace_hash_stability_pass_rate": 0.0, + "case_count": 9 + }, + "inner_loop_t0": { + "label": "inner_loop_t0", + "pass_rate": 0.6667, + "mean_rejection_count_per_turn": 0.2222, + "non_empty_rejected_attempts_rate": 0.2222, + "exhaustion_rate": 0.3333, + "mean_admissibility_checks_per_turn": 2.4444, + "mean_added_latency_ms": 0.0064, + "p95_added_latency_ms": 0.0577, + "trace_hash_stability_pass_rate": 1.0, + "case_count": 9 + }, + "inner_loop_tpos": { + "label": "inner_loop_tpos", + "pass_rate": 0.4444, + "mean_rejection_count_per_turn": 0, + "non_empty_rejected_attempts_rate": 0.0, + "exhaustion_rate": 0.5556, + "mean_admissibility_checks_per_turn": 1.3333, + "mean_added_latency_ms": 0.0, + "p95_added_latency_ms": 0.0, + "trace_hash_stability_pass_rate": 0.0, + "case_count": 9 + } + }, + "rejection_effect": -0.3333, + "code_path_residual": 0.0, + "causal_attribution_valid": true, + "exhaustion_ceiling": 0.05, + "exhaustion_gate_pass": false, + "probe_threshold_positive": 0.25, + "case_count": 9, + "skipped_count": 0 + }, + "case_details": [ + { + "id": "FSC-PUB-001", + "kind": "chain_three_hop", + "expected_endpoint": "delta", + "conditions": { + "boundary_only": { + "surface": "alpha gamma delta beta", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 6.3321669986180495, + "exhausted": false, + "trace_hash": "f1749b663fdf0d85445d4ace27a2c4e492894445aad050e97cec8ea8a89650c5" + }, + "null_control": { + "surface": "alpha gamma delta beta", + "rejections": 0, + "checks": 4, + "latency_ms": 0.07079200076987036, + "absolute_latency_ms": 6.40295899938792, + "exhausted": false, + "trace_hash": "f1749b663fdf0d85445d4ace27a2c4e492894445aad050e97cec8ea8a89650c5" + }, + "inner_loop_t0": { + "surface": "alpha gamma delta alpha", + "rejections": 1, + "checks": 5, + "latency_ms": 0.05766700269305147, + "absolute_latency_ms": 6.389834001311101, + "exhausted": false, + "trace_hash": "a6c009be37a048c723dfa87b155f277f6d0c0ab0c0ab9355180e0a09ca014045" + }, + "inner_loop_tpos": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 1.3651670014951378, + "exhausted": true, + "trace_hash": "__exhausted__" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": true, + "inner_loop_tpos": false + } + }, + { + "id": "FSC-DEV-001", + "kind": "chain_three_hop", + "expected_endpoint": "delta", + "conditions": { + "boundary_only": { + "surface": "alpha gamma delta beta", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 6.53266700101085, + "exhausted": false, + "trace_hash": "81a5907ed8c58cba69648dff42f15b24547ac585b36db1ba178b5314908f0919" + }, + "null_control": { + "surface": "alpha gamma delta beta", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 6.400083999324124, + "exhausted": false, + "trace_hash": "81a5907ed8c58cba69648dff42f15b24547ac585b36db1ba178b5314908f0919" + }, + "inner_loop_t0": { + "surface": "alpha gamma delta alpha", + "rejections": 1, + "checks": 5, + "latency_ms": 0.0, + "absolute_latency_ms": 6.458666000980884, + "exhausted": false, + "trace_hash": "afb66f361ebf275498956e51a92598c73ed29fc569949009c354ebe33414d286" + }, + "inner_loop_tpos": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 1.3585829983639996, + "exhausted": true, + "trace_hash": "__exhausted__" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": true, + "inner_loop_tpos": false + } + }, + { + "id": "FSC-DEV-002", + "kind": "negative_control_no_chain", + "expected_endpoint": "beta", + "conditions": { + "boundary_only": { + "surface": "alpha beta", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 2.8974999986530747, + "exhausted": false, + "trace_hash": "9ecd42e41d24f6167c7dbafdab07a309b5d5cf9d673d412bd6ab418c192095de" + }, + "null_control": { + "surface": "alpha beta", + "rejections": 0, + "checks": 2, + "latency_ms": 0.08908300151233561, + "absolute_latency_ms": 2.9865830001654103, + "exhausted": false, + "trace_hash": "9ecd42e41d24f6167c7dbafdab07a309b5d5cf9d673d412bd6ab418c192095de" + }, + "inner_loop_t0": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 2.1187089987506624, + "exhausted": true, + "trace_hash": "__exhausted__" + }, + "inner_loop_tpos": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 0.6152080022729933, + "exhausted": true, + "trace_hash": "__exhausted__" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": false, + "inner_loop_tpos": false + } + }, + { + "id": "FSC-DEV-003", + "kind": "frame_constraint_blocks_wrong_relation", + "expected_endpoint": "beta", + "conditions": { + "boundary_only": { + "surface": "alpha beta", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 3.1092079989321064, + "exhausted": false, + "trace_hash": "f5f42fdb1b841368ccff2b9f4a0bdfc666001800076a72277298e91aa97b7cd1" + }, + "null_control": { + "surface": "alpha beta", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 3.086417000304209, + "exhausted": false, + "trace_hash": "f5f42fdb1b841368ccff2b9f4a0bdfc666001800076a72277298e91aa97b7cd1" + }, + "inner_loop_t0": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 2.2947910001676064, + "exhausted": true, + "trace_hash": "__exhausted__" + }, + "inner_loop_tpos": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 0.6693330033158418, + "exhausted": true, + "trace_hash": "__exhausted__" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": false, + "inner_loop_tpos": false + } + }, + { + "id": "FSC-DEV-004", + "kind": "chain_two_hop_means", + "expected_endpoint": "omicron", + "conditions": { + "boundary_only": { + "surface": "mu nu omicron", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 2.9942500004835892, + "exhausted": false, + "trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d" + }, + "null_control": { + "surface": "mu nu omicron", + "rejections": 0, + "checks": 3, + "latency_ms": 0.11316699965391308, + "absolute_latency_ms": 3.1074170001375023, + "exhausted": false, + "trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d" + }, + "inner_loop_t0": { + "surface": "mu nu omicron", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 2.97054200200364, + "exhausted": false, + "trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d" + }, + "inner_loop_tpos": { + "surface": "mu nu omicron", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 2.97287500143284, + "exhausted": false, + "trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": true, + "inner_loop_tpos": true + } + }, + { + "id": "FSC-DEV-005", + "kind": "chain_three_hop_precedes", + "expected_endpoint": "tau", + "conditions": { + "boundary_only": { + "surface": "pi rho sigma tau", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 3.3884170006786007, + "exhausted": false, + "trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b" + }, + "null_control": { + "surface": "pi rho sigma tau", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 3.3558340001036413, + "exhausted": false, + "trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b" + }, + "inner_loop_t0": { + "surface": "pi rho sigma tau", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 3.295792001154041, + "exhausted": false, + "trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b" + }, + "inner_loop_tpos": { + "surface": "pi rho sigma tau", + "rejections": 0, + "checks": 4, + "latency_ms": 0.0, + "absolute_latency_ms": 3.1976249993022066, + "exhausted": false, + "trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": true, + "inner_loop_tpos": true + } + }, + { + "id": "FSC-DEV-006", + "kind": "chain_two_hop_part_of", + "expected_endpoint": "chi", + "conditions": { + "boundary_only": { + "surface": "phi chi upsilon", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 3.3560420015419368, + "exhausted": false, + "trace_hash": "244373cd293ef544c1c4395218335c1c1f255d562a9697646b734041412d407f" + }, + "null_control": { + "surface": "phi chi upsilon", + "rejections": 0, + "checks": 3, + "latency_ms": 0.1467080001020804, + "absolute_latency_ms": 3.502750001644017, + "exhausted": false, + "trace_hash": "244373cd293ef544c1c4395218335c1c1f255d562a9697646b734041412d407f" + }, + "inner_loop_t0": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 0.9736249994602986, + "exhausted": true, + "trace_hash": "__exhausted__" + }, + "inner_loop_tpos": { + "surface": "", + "rejections": 0, + "checks": 0, + "latency_ms": 0.0, + "absolute_latency_ms": 0.9959999988495838, + "exhausted": true, + "trace_hash": "__exhausted__" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": false, + "inner_loop_tpos": false + } + }, + { + "id": "FSC-DEV-007", + "kind": "adversarial_distractor_means_vs_cause", + "expected_endpoint": "omega", + "conditions": { + "boundary_only": { + "surface": "psi omega", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 1.9424589991103858, + "exhausted": false, + "trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd" + }, + "null_control": { + "surface": "psi omega", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 1.8285829974047374, + "exhausted": false, + "trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd" + }, + "inner_loop_t0": { + "surface": "psi omega", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 1.8532499998400453, + "exhausted": false, + "trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd" + }, + "inner_loop_tpos": { + "surface": "psi omega", + "rejections": 0, + "checks": 2, + "latency_ms": 0.0, + "absolute_latency_ms": 1.8378329987172037, + "exhausted": false, + "trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": true, + "inner_loop_tpos": true + } + }, + { + "id": "FSC-DEV-008", + "kind": "adversarial_distractor_chain_branching", + "expected_endpoint": "zeta", + "conditions": { + "boundary_only": { + "surface": "eta theta zeta", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 3.0924999991839286, + "exhausted": false, + "trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f" + }, + "null_control": { + "surface": "eta theta zeta", + "rejections": 0, + "checks": 3, + "latency_ms": 0.034500000765547156, + "absolute_latency_ms": 3.1269999999494758, + "exhausted": false, + "trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f" + }, + "inner_loop_t0": { + "surface": "eta theta zeta", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 2.940541999123525, + "exhausted": false, + "trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f" + }, + "inner_loop_tpos": { + "surface": "eta theta zeta", + "rejections": 0, + "checks": 3, + "latency_ms": 0.0, + "absolute_latency_ms": 3.039292001631111, + "exhausted": false, + "trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f" + } + }, + "hash_stability": { + "inner_loop_t0": true + }, + "passes": { + "boundary_only": true, + "null_control": true, + "inner_loop_t0": true, + "inner_loop_tpos": true + } + } + ] +} \ No newline at end of file diff --git a/evals/forward_semantic_control/results/phase3_v2_report.json b/evals/forward_semantic_control/results/phase3_v2_report.json new file mode 100644 index 00000000..6d58a6ee --- /dev/null +++ b/evals/forward_semantic_control/results/phase3_v2_report.json @@ -0,0 +1,129 @@ +{ + "metrics": { + "case_count": 5, + "skipped_count": 0, + "eligible_count": 5, + "pass_count": 5, + "pass_rate": 1.0, + "boundary_decoy_rate": 1.0, + "rejection_traced_rate": 1.0, + "mechanism_isolated": true + }, + "case_details": [ + { + "id": "FSC-PUB-V2-001", + "skipped": false, + "passed": true, + "semantic_pair": "question/answer", + "expected_endpoint": "question", + "forbidden_token": "answer", + "boundary_selected": "answer", + "boundary_picks_forbidden": true, + "boundary_verdict_rejects": true, + "inner_selected": "question", + "inner_admitted": true, + "inner_exhausted": false, + "rejection_in_trace": true, + "rejected_attempts": [ + [ + 21, + "answer", + 0.823794424533844 + ] + ], + "rationale": "Field state from 'symbol' is geometrically nearer to 'answer' than 'question' (gap=+1.566), so boundary-only selects 'answer'. Blade=versor('question') admits 'question' at score 1.420 but rejects 'answer' at score 0.824. Threshold 1.122 sits between them." + }, + { + "id": "FSC-PUB-V2-002", + "skipped": false, + "passed": true, + "semantic_pair": "truth/meaning", + "expected_endpoint": "truth", + "forbidden_token": "meaning", + "boundary_selected": "meaning", + "boundary_picks_forbidden": true, + "boundary_verdict_rejects": true, + "inner_selected": "truth", + "inner_admitted": true, + "inner_exhausted": false, + "rejection_in_trace": true, + "rejected_attempts": [ + [ + 110, + "meaning", + 0.7167291045188904 + ] + ], + "rationale": "Field state from 'infer' is geometrically nearer to 'meaning' than 'truth' (gap=+2.193), so boundary-only selects 'meaning'. Blade=versor('truth') admits 'truth' at score 1.173 but rejects 'meaning' at score 0.717. Threshold 0.945 separates." + }, + { + "id": "FSC-PUB-V2-003", + "skipped": false, + "passed": true, + "semantic_pair": "think/spirit", + "expected_endpoint": "think", + "forbidden_token": "spirit", + "boundary_selected": "spirit", + "boundary_picks_forbidden": true, + "boundary_verdict_rejects": true, + "inner_selected": "think", + "inner_admitted": true, + "inner_exhausted": false, + "rejection_in_trace": true, + "rejected_attempts": [ + [ + 29, + "spirit", + -0.5510096549987793 + ] + ], + "rationale": "Cognition vs. ineffable pole. Field state from 'corrects' is nearer to 'spirit' (gap=+2.711). Blade=versor('think') admits 'think' at score 12.72 and rejects 'spirit' at score -0.55. Wide blade gap (+13.27)." + }, + { + "id": "FSC-PUB-V2-004", + "skipped": false, + "passed": true, + "semantic_pair": "understand/say", + "expected_endpoint": "understand", + "forbidden_token": "say", + "boundary_selected": "say", + "boundary_picks_forbidden": true, + "boundary_verdict_rejects": true, + "inner_selected": "understand", + "inner_admitted": true, + "inner_exhausted": false, + "rejection_in_trace": true, + "rejected_attempts": [ + [ + 19, + "say", + 2.3711442947387695 + ] + ], + "rationale": "Intellection vs. utterance. Field state from 'corrects' is nearer to 'say'. Blade=versor('understand') admits 'understand' at score 5.74 and rejects 'say' at score 2.37." + }, + { + "id": "FSC-PUB-V2-005", + "skipped": false, + "passed": true, + "semantic_pair": "thought/beginning", + "expected_endpoint": "thought", + "forbidden_token": "beginning", + "boundary_selected": "beginning", + "boundary_picks_forbidden": true, + "boundary_verdict_rejects": true, + "inner_selected": "thought", + "inner_admitted": true, + "inner_exhausted": false, + "rejection_in_trace": true, + "rejected_attempts": [ + [ + 27, + "beginning", + 1.616857886314392 + ] + ], + "rationale": "Cognition vs. genesis. Field state from 'corrects' is nearer to 'beginning'. Blade=versor('thought') admits 'thought' at score 14.36 and rejects 'beginning' at score 1.62. Largest blade gap (+12.75)." + } + ] +} \ No newline at end of file diff --git a/evals/forward_semantic_control/results/phase4_characterization_combined.json b/evals/forward_semantic_control/results/phase4_characterization_combined.json new file mode 100644 index 00000000..656b8ae7 --- /dev/null +++ b/evals/forward_semantic_control/results/phase4_characterization_combined.json @@ -0,0 +1,211 @@ +{ + "metrics": { + "thresholds_swept": [ + -1.0, + -0.5, + 0.0, + 0.1, + 0.25, + 0.5, + 1.0 + ], + "best_threshold": 1.0, + "best_separation_quality": 0.6, + "separation_quality_gate": 0.8, + "passes_separation_gate": false, + "total_correct_candidates": 5, + "total_incorrect_candidates": 5, + "overlap_ratio": 0.0803, + "case_count": 14, + "skipped_count": 9, + "geometry_supports_static_threshold": false + }, + "per_threshold": { + "-1.0": { + "TP": 5, + "FN": 0, + "FP": 5, + "TN": 0, + "TP_rate": 1.0, + "FP_rate": 1.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "-0.5": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.0": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.1": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.25": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.5": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "1.0": { + "TP": 5, + "FN": 0, + "FP": 2, + "TN": 3, + "TP_rate": 1.0, + "FP_rate": 0.4, + "FN_rate": 0.0, + "TN_rate": 0.6, + "separation_quality": 0.6 + } + }, + "score_distributions": { + "correct": { + "count": 5, + "mean": 7.0824, + "median": 5.7389, + "min": 1.1731, + "max": 14.3621, + "stdev": 6.1954 + }, + "incorrect": { + "count": 5, + "mean": 0.9955, + "median": 0.8238, + "min": -0.551, + "max": 2.3711, + "stdev": 1.0929 + }, + "overlap": { + "correct_min": 1.1731, + "incorrect_max": 2.3711, + "overlap_size": 1.198, + "full_range": 14.9131, + "overlap_ratio": 0.0803, + "separable_by_static_threshold": false + } + }, + "case_details": [ + { + "id": "FSC-PUB-001", + "skipped": true + }, + { + "id": "FSC-DEV-001", + "skipped": true + }, + { + "id": "FSC-DEV-002", + "skipped": true + }, + { + "id": "FSC-DEV-003", + "skipped": true + }, + { + "id": "FSC-DEV-004", + "skipped": true + }, + { + "id": "FSC-DEV-005", + "skipped": true + }, + { + "id": "FSC-DEV-006", + "skipped": true + }, + { + "id": "FSC-DEV-007", + "skipped": true + }, + { + "id": "FSC-DEV-008", + "skipped": true + }, + { + "id": "FSC-PUB-V2-001", + "correct_scores": [ + 1.4205 + ], + "incorrect_scores": [ + 0.8238 + ] + }, + { + "id": "FSC-PUB-V2-002", + "correct_scores": [ + 1.1731 + ], + "incorrect_scores": [ + 0.7167 + ] + }, + { + "id": "FSC-PUB-V2-003", + "correct_scores": [ + 12.7176 + ], + "incorrect_scores": [ + -0.551 + ] + }, + { + "id": "FSC-PUB-V2-004", + "correct_scores": [ + 5.7389 + ], + "incorrect_scores": [ + 2.3711 + ] + }, + { + "id": "FSC-PUB-V2-005", + "correct_scores": [ + 14.3621 + ], + "incorrect_scores": [ + 1.6169 + ] + } + ] +} \ No newline at end of file diff --git a/evals/forward_semantic_control/results/phase4_characterization_v1_plus_dev.json b/evals/forward_semantic_control/results/phase4_characterization_v1_plus_dev.json new file mode 100644 index 00000000..1cf230d8 --- /dev/null +++ b/evals/forward_semantic_control/results/phase4_characterization_v1_plus_dev.json @@ -0,0 +1,148 @@ +{ + "metrics": { + "thresholds_swept": [ + -1.0, + -0.5, + 0.0, + 0.1, + 0.25, + 0.5, + 1.0 + ], + "best_threshold": -1.0, + "best_separation_quality": 0.0, + "separation_quality_gate": 0.8, + "passes_separation_gate": false, + "total_correct_candidates": 0, + "total_incorrect_candidates": 0, + "overlap_ratio": 0.0, + "case_count": 9, + "skipped_count": 9, + "geometry_supports_static_threshold": false + }, + "per_threshold": { + "-1.0": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "-0.5": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "0.0": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "0.1": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "0.25": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "0.5": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "1.0": { + "TP": 0, + "FN": 0, + "FP": 0, + "TN": 0, + "TP_rate": 0.0, + "FP_rate": 0.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + } + }, + "score_distributions": { + "correct": { + "count": 0 + }, + "incorrect": { + "count": 0 + } + }, + "case_details": [ + { + "id": "FSC-PUB-001", + "skipped": true + }, + { + "id": "FSC-DEV-001", + "skipped": true + }, + { + "id": "FSC-DEV-002", + "skipped": true + }, + { + "id": "FSC-DEV-003", + "skipped": true + }, + { + "id": "FSC-DEV-004", + "skipped": true + }, + { + "id": "FSC-DEV-005", + "skipped": true + }, + { + "id": "FSC-DEV-006", + "skipped": true + }, + { + "id": "FSC-DEV-007", + "skipped": true + }, + { + "id": "FSC-DEV-008", + "skipped": true + } + ] +} \ No newline at end of file diff --git a/evals/forward_semantic_control/results/phase4_characterization_v2.json b/evals/forward_semantic_control/results/phase4_characterization_v2.json new file mode 100644 index 00000000..52cd9006 --- /dev/null +++ b/evals/forward_semantic_control/results/phase4_characterization_v2.json @@ -0,0 +1,175 @@ +{ + "metrics": { + "thresholds_swept": [ + -1.0, + -0.5, + 0.0, + 0.1, + 0.25, + 0.5, + 1.0 + ], + "best_threshold": 1.0, + "best_separation_quality": 0.6, + "separation_quality_gate": 0.8, + "passes_separation_gate": false, + "total_correct_candidates": 5, + "total_incorrect_candidates": 5, + "overlap_ratio": 0.0803, + "case_count": 5, + "skipped_count": 0, + "geometry_supports_static_threshold": false + }, + "per_threshold": { + "-1.0": { + "TP": 5, + "FN": 0, + "FP": 5, + "TN": 0, + "TP_rate": 1.0, + "FP_rate": 1.0, + "FN_rate": 0.0, + "TN_rate": 0.0, + "separation_quality": 0.0 + }, + "-0.5": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.0": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.1": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.25": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "0.5": { + "TP": 5, + "FN": 0, + "FP": 4, + "TN": 1, + "TP_rate": 1.0, + "FP_rate": 0.8, + "FN_rate": 0.0, + "TN_rate": 0.2, + "separation_quality": 0.2 + }, + "1.0": { + "TP": 5, + "FN": 0, + "FP": 2, + "TN": 3, + "TP_rate": 1.0, + "FP_rate": 0.4, + "FN_rate": 0.0, + "TN_rate": 0.6, + "separation_quality": 0.6 + } + }, + "score_distributions": { + "correct": { + "count": 5, + "mean": 7.0824, + "median": 5.7389, + "min": 1.1731, + "max": 14.3621, + "stdev": 6.1954 + }, + "incorrect": { + "count": 5, + "mean": 0.9955, + "median": 0.8238, + "min": -0.551, + "max": 2.3711, + "stdev": 1.0929 + }, + "overlap": { + "correct_min": 1.1731, + "incorrect_max": 2.3711, + "overlap_size": 1.198, + "full_range": 14.9131, + "overlap_ratio": 0.0803, + "separable_by_static_threshold": false + } + }, + "case_details": [ + { + "id": "FSC-PUB-V2-001", + "correct_scores": [ + 1.4205 + ], + "incorrect_scores": [ + 0.8238 + ] + }, + { + "id": "FSC-PUB-V2-002", + "correct_scores": [ + 1.1731 + ], + "incorrect_scores": [ + 0.7167 + ] + }, + { + "id": "FSC-PUB-V2-003", + "correct_scores": [ + 12.7176 + ], + "incorrect_scores": [ + -0.551 + ] + }, + { + "id": "FSC-PUB-V2-004", + "correct_scores": [ + 5.7389 + ], + "incorrect_scores": [ + 2.3711 + ] + }, + { + "id": "FSC-PUB-V2-005", + "correct_scores": [ + 14.3621 + ], + "incorrect_scores": [ + 1.6169 + ] + } + ] +} \ No newline at end of file diff --git a/evals/forward_semantic_control/results/phase4_summary.json b/evals/forward_semantic_control/results/phase4_summary.json new file mode 100644 index 00000000..f0f9a419 --- /dev/null +++ b/evals/forward_semantic_control/results/phase4_summary.json @@ -0,0 +1,65 @@ +{ + "v1_plus_dev": { + "thresholds_swept": [ + -1.0, + -0.5, + 0.0, + 0.1, + 0.25, + 0.5, + 1.0 + ], + "best_threshold": -1.0, + "best_separation_quality": 0.0, + "separation_quality_gate": 0.8, + "passes_separation_gate": false, + "total_correct_candidates": 0, + "total_incorrect_candidates": 0, + "overlap_ratio": 0.0, + "case_count": 9, + "skipped_count": 9, + "geometry_supports_static_threshold": false + }, + "v2": { + "thresholds_swept": [ + -1.0, + -0.5, + 0.0, + 0.1, + 0.25, + 0.5, + 1.0 + ], + "best_threshold": 1.0, + "best_separation_quality": 0.6, + "separation_quality_gate": 0.8, + "passes_separation_gate": false, + "total_correct_candidates": 5, + "total_incorrect_candidates": 5, + "overlap_ratio": 0.0803, + "case_count": 5, + "skipped_count": 0, + "geometry_supports_static_threshold": false + }, + "combined": { + "thresholds_swept": [ + -1.0, + -0.5, + 0.0, + 0.1, + 0.25, + 0.5, + 1.0 + ], + "best_threshold": 1.0, + "best_separation_quality": 0.6, + "separation_quality_gate": 0.8, + "passes_separation_gate": false, + "total_correct_candidates": 5, + "total_incorrect_candidates": 5, + "overlap_ratio": 0.0803, + "case_count": 14, + "skipped_count": 9, + "geometry_supports_static_threshold": false + } +} \ No newline at end of file diff --git a/evals/forward_semantic_control/threshold_characterization.py b/evals/forward_semantic_control/threshold_characterization.py new file mode 100644 index 00000000..354135d3 --- /dev/null +++ b/evals/forward_semantic_control/threshold_characterization.py @@ -0,0 +1,301 @@ +"""Phase 4 threshold characterization — ADR-0024 diagnostic, not tuning. + +The Phase 2 report on the existing FSC v1 corpus surfaced +``exhaustion_rate=0.33 at t=0.0`` and ``exhaustion_rate=0.56 at +t=0.25`` — well above the 5% ceiling. Before proposing a learned or +adaptive threshold, we need to know *whether the geometry permits a +clean threshold at all*. + +This module produces a distribution-map diagnostic, NOT a tuned +threshold: + + * For each case in v1+dev, build the same region the inner-loop + runner builds (chain outer-product over chain_tokens). + * For every candidate index in the admissible set, compute its + ``cga_inner`` score against the relation_blade. + * Group scores by whether the candidate is "correct" (== the + expected_endpoint) or "incorrect" (anything else admissible). + * Sweep thresholds [-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0] and + report, per threshold: + admitted_correct / total_correct (TP rate) + admitted_incorrect / total_incorrect (FP rate) + rejected_correct / total_correct (FN rate) + rejected_incorrect / total_incorrect (TN rate) + separation_quality = TP_rate - FP_rate + + * Also report admitted-vs-rejected score *distribution* maps: + admitted_score_mean / median / min / max per correctness class + rejected_score_mean / median / min / max per correctness class + score_overlap_ratio = (max(correct_rejected_min, incorrect_admitted_min) + - min(correct_admitted_max, incorrect_rejected_max)) + normalized + + * The headline finding is whether ANY threshold delivers + ``separation_quality >= 0.8`` on the corpus. If not, the + relation_blade construction is geometrically under-resolved for + static thresholding regardless of value. + +This is a Phase 4 diagnostic that informs whether ADR-0025 should +even attempt static thresholds or move directly to relation-typed +or frame-derived schemes. +""" + +from __future__ import annotations + +import json +import statistics +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import numpy as np + +from algebra.cga import cga_inner, outer_product +from chat.runtime import ChatRuntime + +THRESHOLDS = (-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0) +SEPARATION_QUALITY_GATE = 0.8 + + +@dataclass(slots=True) +class CharacterizationReport: + metrics: dict[str, Any] = field(default_factory=dict) + per_threshold: dict[float, dict[str, Any]] = field(default_factory=dict) + score_distributions: dict[str, dict[str, Any]] = field(default_factory=dict) + case_details: list[dict[str, Any]] = field(default_factory=list) + + +def _build_blade(vocab, chain_tokens: tuple[str, ...]) -> tuple[np.ndarray | None, list[int]]: + indices: list[int] = [] + versors: list[np.ndarray] = [] + for raw in chain_tokens: + token = raw.lower().strip() + if not token: + continue + try: + idx = vocab.index_of(token) + versor = np.asarray(vocab.get_versor(token), dtype=np.float32) + except (KeyError, AttributeError): + continue + indices.append(int(idx)) + versors.append(versor) + if not versors: + return None, [] + blade = versors[0] + for nxt in versors[1:]: + blade = outer_product(blade, nxt) + return blade, indices + + +def _score_candidates( + vocab, + blade: np.ndarray, + indices: list[int], + expected_token: str, +) -> tuple[list[float], list[float]]: + """Return (correct_scores, incorrect_scores) for candidates in admissible set.""" + correct: list[float] = [] + incorrect: list[float] = [] + expected_idx: int | None = None + try: + expected_idx = int(vocab.index_of(expected_token.lower().strip())) + except (KeyError, AttributeError, ValueError): + expected_idx = None + for idx in indices: + v = np.asarray(vocab.get_versor_at(idx), dtype=np.float32) + score = float(cga_inner(v, blade)) + if expected_idx is not None and idx == expected_idx: + correct.append(score) + else: + incorrect.append(score) + return correct, incorrect + + +def _summarize(scores: list[float]) -> dict[str, Any]: + if not scores: + return {"count": 0} + return { + "count": len(scores), + "mean": round(statistics.mean(scores), 4), + "median": round(statistics.median(scores), 4), + "min": round(min(scores), 4), + "max": round(max(scores), 4), + "stdev": round(statistics.stdev(scores), 4) if len(scores) > 1 else 0.0, + } + + +def _blade_and_indices_for_case( + vocab, case: dict[str, Any] +) -> tuple[np.ndarray | None, list[int], str]: + """Build the blade + admissible indices for either schema. + + Returns ``(blade, indices, expected_token)`` or ``(None, [], "")`` + if the case cannot be grounded in the active vocab. + """ + expected = case.get("expected_endpoint", "") + # v2 schema: explicit admissible_tokens + relation_blade_token. + if "admissible_tokens" in case and "relation_blade_token" in case: + try: + blade = np.asarray( + vocab.get_versor(case["relation_blade_token"]), dtype=np.float32 + ) + indices = [int(vocab.index_of(tok)) for tok in case["admissible_tokens"]] + except (KeyError, AttributeError, ValueError): + return None, [], "" + return blade, indices, expected + # v1 schema: chain_tokens outer-product, or single-token fallback. + chain_tokens = tuple(case.get("chain_tokens", ())) + if not chain_tokens and expected: + chain_tokens = (expected,) + blade, indices = _build_blade(vocab, chain_tokens) + return blade, indices, expected + + +def characterize(cases: list[dict[str, Any]]) -> CharacterizationReport: + runtime = ChatRuntime() + vocab = runtime.session.vocab + + case_details: list[dict[str, Any]] = [] + all_correct: list[float] = [] + all_incorrect: list[float] = [] + + for case in cases: + blade, indices, expected = _blade_and_indices_for_case(vocab, case) + if blade is None or not indices: + case_details.append({"id": case.get("id", ""), "skipped": True}) + continue + correct, incorrect = _score_candidates(vocab, blade, indices, expected) + all_correct.extend(correct) + all_incorrect.extend(incorrect) + case_details.append({ + "id": case.get("id", ""), + "correct_scores": [round(s, 4) for s in correct], + "incorrect_scores": [round(s, 4) for s in incorrect], + }) + + per_threshold: dict[float, dict[str, Any]] = {} + for thr in THRESHOLDS: + tp = sum(1 for s in all_correct if s >= thr) + fn = sum(1 for s in all_correct if s < thr) + fp = sum(1 for s in all_incorrect if s >= thr) + tn = sum(1 for s in all_incorrect if s < thr) + tot_c = max(len(all_correct), 1) + tot_i = max(len(all_incorrect), 1) + tp_rate = tp / tot_c + fp_rate = fp / tot_i + per_threshold[thr] = { + "TP": tp, "FN": fn, "FP": fp, "TN": tn, + "TP_rate": round(tp_rate, 4), + "FP_rate": round(fp_rate, 4), + "FN_rate": round(fn / tot_c, 4), + "TN_rate": round(tn / tot_i, 4), + "separation_quality": round(tp_rate - fp_rate, 4), + } + + score_distributions = { + "correct": _summarize(all_correct), + "incorrect": _summarize(all_incorrect), + } + + # Overlap diagnostic: is there *any* gap between the worst correct + # and the best incorrect? + overlap_ratio = 0.0 + if all_correct and all_incorrect: + cmin = min(all_correct) + imax = max(all_incorrect) + full_range = max(all_correct + all_incorrect) - min(all_correct + all_incorrect) + overlap = max(imax - cmin, 0.0) + overlap_ratio = round(overlap / full_range, 4) if full_range > 0 else 0.0 + score_distributions["overlap"] = { + "correct_min": round(cmin, 4), + "incorrect_max": round(imax, 4), + "overlap_size": round(overlap, 4), + "full_range": round(full_range, 4), + "overlap_ratio": overlap_ratio, + "separable_by_static_threshold": cmin > imax, + } + + best_thr = max( + per_threshold.items(), + key=lambda kv: kv[1]["separation_quality"], + ) + metrics = { + "thresholds_swept": list(THRESHOLDS), + "best_threshold": best_thr[0], + "best_separation_quality": best_thr[1]["separation_quality"], + "separation_quality_gate": SEPARATION_QUALITY_GATE, + "passes_separation_gate": best_thr[1]["separation_quality"] >= SEPARATION_QUALITY_GATE, + "total_correct_candidates": len(all_correct), + "total_incorrect_candidates": len(all_incorrect), + "overlap_ratio": overlap_ratio, + "case_count": len(cases), + "skipped_count": sum(1 for d in case_details if d.get("skipped")), + # Headline finding: can a STATIC threshold separate correct from + # incorrect on this corpus? If no, ADR-0025 must not propose + # static thresholds. + "geometry_supports_static_threshold": ( + best_thr[1]["separation_quality"] >= SEPARATION_QUALITY_GATE + ), + } + return CharacterizationReport( + metrics=metrics, + per_threshold=per_threshold, + score_distributions=score_distributions, + case_details=case_details, + ) + + +def _load(path: Path) -> list[dict[str, Any]]: + if not path.exists(): + return [] + with path.open() as fh: + return [json.loads(line) for line in fh if line.strip()] + + +def _serialize(report: CharacterizationReport) -> dict[str, Any]: + return { + "metrics": report.metrics, + "per_threshold": {str(k): v for k, v in report.per_threshold.items()}, + "score_distributions": report.score_distributions, + "case_details": report.case_details, + } + + +def main() -> None: + v1 = _load(Path("evals/forward_semantic_control/public/v1/cases.jsonl")) + dev = _load(Path("evals/forward_semantic_control/dev/cases.jsonl")) + v2 = _load(Path("evals/forward_semantic_control/public/v2/cases.jsonl")) + + out_dir = Path("evals/forward_semantic_control/results") + out_dir.mkdir(parents=True, exist_ok=True) + + bundles = [ + ("v1_plus_dev", v1 + dev), + ("v2", v2), + ("combined", v1 + dev + v2), + ] + summary: dict[str, dict[str, Any]] = {} + for label, cases in bundles: + if not cases: + summary[label] = {"empty": True} + continue + report = characterize(cases) + out_path = out_dir / f"phase4_characterization_{label}.json" + with out_path.open("w") as fh: + json.dump(_serialize(report), fh, indent=2) + summary[label] = report.metrics + print(f"\n=== {label} ===") + print(f" cases: {report.metrics['case_count']}, " + f"skipped: {report.metrics['skipped_count']}") + print(f" best_threshold: {report.metrics['best_threshold']}") + print(f" best_separation_quality: {report.metrics['best_separation_quality']}") + print(f" geometry_supports_static_threshold: " + f"{report.metrics['geometry_supports_static_threshold']}") + print(f" overlap_ratio: {report.metrics['overlap_ratio']}") + with (out_dir / "phase4_summary.json").open("w") as fh: + json.dump(summary, fh, indent=2) + print(f"\nWrote summary: {out_dir / 'phase4_summary.json'}") + + +if __name__ == "__main__": + main() diff --git a/evals/forward_semantic_control/v2_runner.py b/evals/forward_semantic_control/v2_runner.py new file mode 100644 index 00000000..0803b6a0 --- /dev/null +++ b/evals/forward_semantic_control/v2_runner.py @@ -0,0 +1,211 @@ +"""Phase 3 mechanism-isolation runner — ADR-0024 v2 adversarial cases. + +Synthetic cases where boundary-only is *expected* to select a forbidden +decoy and inner-loop is *expected* to reject it and select the correct +endpoint. The case schema specifies its own region (admissible token +set + relation blade token) so the geometric setup is fully controlled +and reproducible. + +A case passes iff *all* of the following hold under the same field +state: + + boundary-only: + selected == forbidden_token + verdict.admitted is False (the rejection is visible in trace) + + inner-loop (admissibility_threshold from the case): + selected == expected_endpoint + verdict.admitted is True + forbidden_token appears in step.rejected_attempts + +Each case's seed_token sets the initial FieldState.F. No priming — +the geometric configuration is given. This is mechanism isolation, +not corpus observation; pair with ``inner_loop_runner.py`` (Phase 2) +for the corpus side. + +Reports per case + aggregate proof_rate and rejection_causally_traced +counts. Conforms to the ``run_lane`` interface. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +import numpy as np + +from chat.runtime import ChatRuntime +from core.config import RuntimeConfig +from field.state import FieldState +from generate.admissibility import AdmissibilityRegion, RegionSource +from generate.result import GenerationResult +from generate.stream import generate as generate_walk + + +@dataclass(slots=True) +class V2Report: + metrics: dict[str, Any] = field(default_factory=dict) + case_details: list[dict[str, Any]] = field(default_factory=list) + + +def _field_state_from_seed(vocab, seed_token: str) -> FieldState: + idx = vocab.index_of(seed_token) + versor = np.asarray(vocab.get_versor(seed_token), dtype=np.float32) + return FieldState(F=versor.copy(), node=idx, step=0) + + +def _region_from_case(vocab, case: dict[str, Any]) -> AdmissibilityRegion: + indices = [int(vocab.index_of(tok)) for tok in case["admissible_tokens"]] + blade = np.asarray( + vocab.get_versor(case["relation_blade_token"]), dtype=np.float32 + ) + return AdmissibilityRegion( + allowed_indices=np.asarray(indices, dtype=np.int64), + relation_blade=blade, + source=RegionSource.RELATION, + label=f"v2[{case.get('id', '')}]", + ) + + +def _run_case(case: dict[str, Any]) -> dict[str, Any]: + runtime = ChatRuntime() + vocab = runtime.session.vocab + persona = runtime.session.persona + + try: + seed_state = _field_state_from_seed(vocab, case["seed_token"]) + region = _region_from_case(vocab, case) + except (KeyError, ValueError) as exc: + return {"id": case.get("id", ""), "skipped": True, "reason": str(exc)} + + threshold = float(case["admissibility_threshold"]) + expected = case["expected_endpoint"] + forbidden = case["forbidden_token"] + + # Boundary-only leg. + boundary: GenerationResult = generate_walk( + seed_state, vocab, persona, + max_tokens=1, region=region, + inner_loop_admissibility=False, + admissibility_threshold=threshold, + ) + b_step = boundary.admissibility_trace[0] + boundary_selected = b_step.selected_word + boundary_admitted = b_step.verdict.admitted + # Boundary expectation: selects the forbidden decoy and verdict is + # NOT admitted (the rejection is visible in trace but the walk + # still emits it — this is ADR-0023 boundary-only behavior). + boundary_picks_forbidden = boundary_selected == forbidden + boundary_verdict_rejects = not boundary_admitted + + # Inner-loop leg. + inner: GenerationResult | None = None + inner_exhaust_reason = "" + try: + inner = generate_walk( + seed_state, vocab, persona, + max_tokens=1, region=region, + inner_loop_admissibility=True, + admissibility_threshold=threshold, + ) + except ValueError as exc: + inner_exhaust_reason = str(exc) + + if inner is None: + return { + "id": case.get("id", ""), + "skipped": False, + "passed": False, + "boundary_selected": boundary_selected, + "boundary_picks_forbidden": boundary_picks_forbidden, + "boundary_verdict_rejects": boundary_verdict_rejects, + "inner_selected": None, + "inner_admitted": None, + "inner_exhausted": True, + "inner_exhaust_reason": inner_exhaust_reason, + "rejection_in_trace": False, + "rejected_attempts": (), + "rationale": case.get("rationale", ""), + } + + i_step = inner.admissibility_trace[0] + inner_selected = i_step.selected_word + inner_admitted = i_step.verdict.admitted + rejected_words = tuple(word for (_idx, word, _score) in i_step.rejected_attempts) + rejection_in_trace = forbidden in rejected_words + + passed = ( + boundary_picks_forbidden + and boundary_verdict_rejects + and inner_selected == expected + and inner_admitted + and rejection_in_trace + ) + + return { + "id": case.get("id", ""), + "skipped": False, + "passed": passed, + "semantic_pair": case.get("semantic_pair", ""), + "expected_endpoint": expected, + "forbidden_token": forbidden, + "boundary_selected": boundary_selected, + "boundary_picks_forbidden": boundary_picks_forbidden, + "boundary_verdict_rejects": boundary_verdict_rejects, + "inner_selected": inner_selected, + "inner_admitted": inner_admitted, + "inner_exhausted": False, + "rejection_in_trace": rejection_in_trace, + "rejected_attempts": [ + [int(idx), str(word), float(score)] + for (idx, word, score) in i_step.rejected_attempts + ], + "rationale": case.get("rationale", ""), + } + + +def run_lane( + cases: list[dict[str, Any]], + *, + config: RuntimeConfig | None = None, + workers: int | None = None, +) -> V2Report: + _ = config + _ = workers # serial — v2 corpus is small + + if not cases: + return V2Report(metrics={}, case_details=[]) + + details = [_run_case(c) for c in cases] + n = len(details) + skipped = sum(1 for d in details if d.get("skipped")) + eligible = [d for d in details if not d.get("skipped")] + passed = sum(1 for d in eligible if d.get("passed")) + boundary_picks_forbidden_count = sum( + 1 for d in eligible if d.get("boundary_picks_forbidden") + ) + rejection_in_trace_count = sum( + 1 for d in eligible if d.get("rejection_in_trace") + ) + + pass_rate = passed / max(len(eligible), 1) + boundary_decoy_rate = boundary_picks_forbidden_count / max(len(eligible), 1) + rejection_traced_rate = rejection_in_trace_count / max(len(eligible), 1) + + metrics: dict[str, Any] = { + "case_count": n, + "skipped_count": skipped, + "eligible_count": len(eligible), + "pass_count": passed, + "pass_rate": round(pass_rate, 4), + "boundary_decoy_rate": round(boundary_decoy_rate, 4), + "rejection_traced_rate": round(rejection_traced_rate, 4), + # Headline: do we have causal evidence that inner-loop rejection + # is responsible for the selection difference? + "mechanism_isolated": ( + pass_rate == 1.0 + and boundary_decoy_rate == 1.0 + and rejection_traced_rate == 1.0 + ), + } + return V2Report(metrics=metrics, case_details=details) diff --git a/generate/stream.py b/generate/stream.py index 415bcd2d..22079b2f 100644 --- a/generate/stream.py +++ b/generate/stream.py @@ -279,6 +279,7 @@ def generate( region: AdmissibilityRegion | None = None, inner_loop_admissibility: bool = False, admissibility_threshold: float = 0.0, + inner_loop_force_admit: bool = False, ) -> GenerationResult: """Generate a token sequence. @@ -299,6 +300,15 @@ def generate( byte-identical. The rotor ``V`` is only constructed for the admitted candidate, so the ``versor_condition < 1e-6`` invariant is unaffected. + + ``inner_loop_force_admit`` (Phase 2 null control) — only meaningful + when ``inner_loop_admissibility=True``. Exercises the inner-loop + code path (same attempt-loop scaffolding, same telemetry side + effects) but force-breaks on the first candidate regardless of + verdict. This isolates rejection as the causal factor: any + delta between boundary-only and inner-loop-on runs that vanishes + under the null control is attributable to code-path differences, + not to rejection. Not exposed to RuntimeConfig — eval-only. """ tokens = [] trajectory = [] if record_trajectory else None @@ -394,7 +404,13 @@ def generate( region_label=effective_region_label, reason="unconstrained", ) - if not inner_loop_active or verdict.admitted: + if not inner_loop_active or verdict.admitted or inner_loop_force_admit: + # `inner_loop_force_admit` is the Phase 2 null control: + # exercises the inner-loop code path (same attempt loop, + # same telemetry side effects) but force-breaks on the + # first candidate so any pass-rate delta vs the true + # inner-loop run is causally attributable to rejection, + # not to incidental code-path differences. break # Inner loop is on and verdict rejected this candidate. rejected_attempts.append((int(word_idx), str(word), float(verdict.score))) diff --git a/tests/test_inner_loop_admissibility.py b/tests/test_inner_loop_admissibility.py index dbbb145d..4f3afc03 100644 --- a/tests/test_inner_loop_admissibility.py +++ b/tests/test_inner_loop_admissibility.py @@ -328,5 +328,39 @@ class TestInnerLoopDeterminism: assert h_off == h_on +class TestInnerLoopNullControl: + """Phase 2 null control — exercises the inner-loop code path but + force-admits every candidate. Used by the FSC corpus runner to + isolate rejection as the causal factor in any pass-rate delta. + """ + + def test_force_admit_selects_first_preferred_candidate_no_rejections(self) -> None: + # Without null control, this case rejects alpha and selects beta. + # With null control, the inner-loop path is exercised but the + # first candidate (alpha) is force-admitted — same outcome as + # boundary-only. + vocab = _ControllableVocab( + words=["seed", "alpha", "beta"], + preference=[1, 2], + versor_signs=[+1.0, -1.0, +1.0], + ) + result = generate( + _initial_state(vocab), + vocab, + _IdentityPersona(), + max_tokens=1, + region=_positive_blade_region((1, 2)), + inner_loop_admissibility=True, + inner_loop_force_admit=True, + ) + # Force-admit selects alpha (preferred) even though verdict is + # rejected — the breakout happens regardless. + assert result.tokens == ("alpha",) + step = result.admissibility_trace[0] + assert step.selected_word == "alpha" + # No rejections accumulated — first attempt breaks out. + assert step.rejected_attempts == () + + if __name__ == "__main__": # pragma: no cover pytest.main([__file__, "-v"]) diff --git a/tests/test_inner_loop_phase2.py b/tests/test_inner_loop_phase2.py new file mode 100644 index 00000000..edf613c3 --- /dev/null +++ b/tests/test_inner_loop_phase2.py @@ -0,0 +1,114 @@ +"""Phase 2 corpus-observation invariants (ADR-0024 follow-up). + +These tests pin the causal-attribution and determinism contracts that +the Phase 2 runner must hold on the existing FSC corpus. They are +intentionally *not* gated on rejection_effect or exhaustion_rate — +those are findings to be characterised in Phase 4, not invariants. + +What we *do* assert: + + * ``causal_attribution_valid`` is True: the null control (inner-loop + code path on, force-admit on) matches boundary-only exactly. Any + pass-rate delta between inner_loop_t0 and boundary_only is then + attributable to rejection, not to incidental code-path effects. + * ``code_path_residual`` is zero (within float tolerance). + * Trace-hash stability holds for the inner-loop condition on every + non-skipped case (5 reruns produce identical hashes). +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from evals.forward_semantic_control.inner_loop_runner import run_lane + +_CORPUS_PATHS = ( + Path("evals/forward_semantic_control/public/v1/cases.jsonl"), + Path("evals/forward_semantic_control/dev/cases.jsonl"), +) + + +def _load_corpus() -> list[dict]: + cases: list[dict] = [] + for path in _CORPUS_PATHS: + if not path.exists(): + continue + with path.open() as fh: + cases.extend(json.loads(line) for line in fh if line.strip()) + return cases + + +@pytest.fixture(scope="module") +def phase2_report(): + cases = _load_corpus() + if not cases: + pytest.skip("FSC corpus not available") + return run_lane(cases) + + +class TestCausalAttribution: + def test_null_control_matches_boundary_only(self, phase2_report) -> None: + """Null control must reproduce boundary-only pass-rate exactly. + + If this fails, the inner-loop code path is itself altering + selection (call ordering, telemetry side effects), and any + rejection_effect we measure is contaminated. ADR-0024 proof + depends on this invariant. + """ + assert phase2_report.metrics["causal_attribution_valid"] is True + assert phase2_report.metrics["code_path_residual"] == 0.0 + + def test_null_control_per_condition_metrics(self, phase2_report) -> None: + per = phase2_report.metrics["per_condition"] + assert per["null_control"]["pass_rate"] == per["boundary_only"]["pass_rate"] + # Null control must produce zero rejections by construction. + assert per["null_control"]["mean_rejection_count_per_turn"] == 0 + assert per["null_control"]["non_empty_rejected_attempts_rate"] == 0.0 + assert per["null_control"]["exhaustion_rate"] == 0.0 + + +class TestInnerLoopDeterminismOnCorpus: + def test_inner_loop_t0_hash_stable_on_every_case(self, phase2_report) -> None: + """Live-corpus version of the Phase 1 acceptance test. + + Stub-vocab determinism is necessary but not sufficient — the + same property must hold on actual packs, actual field state, + actual rejection sequences. 5 reruns per case must hash + identically. + """ + rate = phase2_report.metrics["per_condition"]["inner_loop_t0"][ + "trace_hash_stability_pass_rate" + ] + assert rate == 1.0 + + +class TestPhase2RecordsFindings: + """These are not gates — they record the Phase 2 finding so a + future change that silently flips the sign of rejection_effect or + closes the exhaustion gap is visible in test output.""" + + def test_runner_emits_required_metric_keys(self, phase2_report) -> None: + required = { + "per_condition", + "rejection_effect", + "code_path_residual", + "causal_attribution_valid", + "exhaustion_ceiling", + "exhaustion_gate_pass", + "probe_threshold_positive", + "case_count", + "skipped_count", + } + assert required <= set(phase2_report.metrics.keys()) + + def test_all_four_conditions_present(self, phase2_report) -> None: + per = phase2_report.metrics["per_condition"] + assert set(per.keys()) == { + "boundary_only", + "null_control", + "inner_loop_t0", + "inner_loop_tpos", + } diff --git a/tests/test_inner_loop_phase3.py b/tests/test_inner_loop_phase3.py new file mode 100644 index 00000000..e835a327 --- /dev/null +++ b/tests/test_inner_loop_phase3.py @@ -0,0 +1,75 @@ +"""Phase 3 mechanism-isolation invariants (ADR-0024 v2 corpus). + +These tests are the *load-bearing* proof contract: in synthetic +cases designed to exercise the rejection mechanism, the inner loop +must (a) actually reject the forbidden decoy, (b) select the +expected endpoint instead, and (c) leave a causal trail in +``rejected_attempts``. + +Pass criteria are stricter than Phase 2 (which is observational): +Phase 3 *gates* on ``mechanism_isolated``. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from evals.forward_semantic_control.v2_runner import run_lane + +V2_CORPUS = Path("evals/forward_semantic_control/public/v2/cases.jsonl") + + +@pytest.fixture(scope="module") +def v2_report(): + if not V2_CORPUS.exists(): + pytest.skip("V2 corpus not available") + with V2_CORPUS.open() as fh: + cases = [json.loads(line) for line in fh if line.strip()] + if not cases: + pytest.skip("V2 corpus is empty") + return run_lane(cases) + + +class TestMechanismIsolated: + def test_mechanism_isolated_overall(self, v2_report) -> None: + """The headline gate — every v2 case must isolate the mechanism.""" + assert v2_report.metrics["mechanism_isolated"] is True + + def test_pass_rate_is_one(self, v2_report) -> None: + assert v2_report.metrics["pass_rate"] == 1.0 + + def test_boundary_picks_decoy_every_case(self, v2_report) -> None: + """If boundary doesn't pick the decoy on a v2 case, the case + is mis-constructed — the mechanism never gets exercised.""" + assert v2_report.metrics["boundary_decoy_rate"] == 1.0 + + def test_rejection_causally_traced_every_case(self, v2_report) -> None: + """The forbidden token must appear in rejected_attempts on + every case — this is the visible causal evidence.""" + assert v2_report.metrics["rejection_traced_rate"] == 1.0 + + +class TestPerCaseInvariants: + def test_no_case_was_skipped(self, v2_report) -> None: + assert v2_report.metrics["skipped_count"] == 0 + + def test_every_case_passed(self, v2_report) -> None: + for detail in v2_report.case_details: + assert detail.get("passed") is True, ( + f"Case {detail.get('id')} failed: " + f"boundary={detail.get('boundary_selected')} " + f"inner={detail.get('inner_selected')} " + f"forbidden_traced={detail.get('rejection_in_trace')} " + f"inner_exhausted={detail.get('inner_exhausted')}" + ) + + def test_inner_selection_matches_expected_endpoint(self, v2_report) -> None: + for detail in v2_report.case_details: + assert detail.get("inner_selected") == detail.get("expected_endpoint") + + def test_boundary_selection_matches_forbidden_token(self, v2_report) -> None: + for detail in v2_report.case_details: + assert detail.get("boundary_selected") == detail.get("forbidden_token") diff --git a/tests/test_inner_loop_phase4.py b/tests/test_inner_loop_phase4.py new file mode 100644 index 00000000..3ff82069 --- /dev/null +++ b/tests/test_inner_loop_phase4.py @@ -0,0 +1,107 @@ +"""Phase 4 threshold characterization invariants (ADR-0024 follow-up). + +These tests are diagnostic, not gates. They pin the finding so a +future change that silently improves (or breaks) the geometric +separability is visible in test output. + +Findings recorded: + + * Per-case the relation_blade DOES separate correct from incorrect + candidates (all five v2 cases pass mechanism-isolation), so the + blade construction is not geometrically blind. + * But globally NO STATIC threshold delivers separation_quality ≥ 0.8. + Blade norms vary across cases (~10x range), so the same threshold + value means different things case-to-case. + * The v1 chain-token outer-product blade is ungrounded in the active + pack — all 9 cases are skipped because chain_tokens (alpha, beta, + gamma, delta) are not in the en_core_cognition vocab. This is its + own load-bearing finding for ADR-0025: chain-token blades are + unsuitable as the default region construction. + +ADR-0025 design implication: static thresholds (global, relation-typed, +or frame-derived) are insufficient. Per-case normalized thresholds +(e.g. fraction of blade self-score) are the next thing to investigate. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from evals.forward_semantic_control.threshold_characterization import characterize + +V1 = Path("evals/forward_semantic_control/public/v1/cases.jsonl") +V2 = Path("evals/forward_semantic_control/public/v2/cases.jsonl") +DEV = Path("evals/forward_semantic_control/dev/cases.jsonl") + + +def _load(path: Path) -> list[dict]: + if not path.exists(): + return [] + with path.open() as fh: + return [json.loads(line) for line in fh if line.strip()] + + +@pytest.fixture(scope="module") +def v1_report(): + cases = _load(V1) + _load(DEV) + if not cases: + pytest.skip("v1/dev corpus not available") + return characterize(cases) + + +@pytest.fixture(scope="module") +def v2_report(): + cases = _load(V2) + if not cases: + pytest.skip("v2 corpus not available") + return characterize(cases) + + +class TestV1ChainBladeUngrounded: + """V1 chain_tokens are synthetic (alpha, beta, gamma, delta) and + not present in the active pack. The characterization should + surface this by skipping every case. + """ + + def test_all_v1_cases_skipped(self, v1_report) -> None: + assert v1_report.metrics["skipped_count"] == v1_report.metrics["case_count"] + + def test_v1_reports_no_separation(self, v1_report) -> None: + # No candidates ⇒ best_separation_quality stays at zero. + assert v1_report.metrics["best_separation_quality"] == 0.0 + + +class TestV2PerCaseSeparates: + """Per-case, every v2 case has correct_min > incorrect_max.""" + + def test_every_v2_case_separates_locally(self, v2_report) -> None: + for detail in v2_report.case_details: + if detail.get("skipped"): + continue + correct = detail["correct_scores"] + incorrect = detail["incorrect_scores"] + assert correct, f"case {detail.get('id')} has no correct candidate" + assert min(correct) > max(incorrect), ( + f"case {detail.get('id')} fails local separation: " + f"correct_min={min(correct)} ≤ incorrect_max={max(incorrect)}" + ) + + +class TestV2GlobalNonSeparability: + """Despite per-case separability, no static threshold works + globally — this is the load-bearing finding for ADR-0025.""" + + def test_no_static_threshold_passes_gate(self, v2_report) -> None: + # If a future change makes this pass, ADR-0025 design may + # need revision. Currently expected: False. + assert v2_report.metrics["geometry_supports_static_threshold"] is False + + def test_score_distributions_overlap_globally(self, v2_report) -> None: + overlap = v2_report.score_distributions["overlap"] + # incorrect_max > correct_min ⇒ static threshold cannot + # separate. This is the geometric fact ADR-0025 must address. + assert overlap["separable_by_static_threshold"] is False + assert overlap["overlap_size"] > 0.0