core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat(adr-0024): Phases 2-5 — corpus eval, v2 adversarial, threshold characterization, ADR-0025 design note

Browse source 
Phase 2 — Corpus observation runner (inner_loop_runner.py):
- Four-condition matrix: boundary_only / null_control / inner_loop_t0 / inner_loop_tpos.
- Added `inner_loop_force_admit` to generate() — exercises the inner-loop
  code path but force-breaks on first candidate.  Eval-only null control:
  isolates rejection as the causal factor for any pass-rate delta.
- Metrics: pass_rate, mean_rejection_count_per_turn,
  non_empty_rejected_attempts_rate, exhaustion_rate (gated at 5%),
  mean_admissibility_checks_per_turn, mean/p95 added_latency_ms,
  trace_hash_stability across 5 reruns per case.
- Finding on v1+dev: causal_attribution_valid=True, code_path_residual=0.0,
  but exhaustion_rate=0.33 at t=0 — chain outer-product blade is
  geometrically blind to the active pack.
- Tests (tests/test_inner_loop_phase2.py, 5 pass): pin
  causal-attribution and live-corpus trace-hash stability invariants.

Phase 3 — Mechanism-isolation v2 corpus (5 cases, v2_runner.py):
- Synthetic adversarial cases with controlled geometry — each case
  specifies seed_token, admissible_tokens, relation_blade_token, and
  admissibility_threshold.  Field state is constructed directly from
  the seed token versor, not via priming.
- For every case: boundary-only selects the forbidden decoy and
  inner-loop selects the expected endpoint with the forbidden token
  appearing in rejected_attempts.
- Result: mechanism_isolated=true on 5/5.  boundary_decoy_rate=1.0,
  rejection_traced_rate=1.0.  Inner-loop rejection is demonstrably
  doing causal semantic work on real packs.
- Tests (tests/test_inner_loop_phase3.py, 8 pass): GATE on
  mechanism_isolated.

Phase 4 — Threshold characterization (threshold_characterization.py):
- Distribution mapping per-case AND globally on v1+dev, v2, combined.
- Per-threshold sweep over [-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0].
- Finding: per-case geometry separates cleanly (correct_min > incorrect_max
  on every v2 case), BUT no global static threshold passes the
  separation_quality >= 0.8 gate.  Blade norms vary ~10x across cases.
- Static thresholds (global, relation-typed, or constant frame-derived)
  are geometrically insufficient.  Per-case-normalized thresholds
  (e.g. fraction of blade self-score) are the recommended next step.
- v1 chain-token outer-product cases all skipped — the corpus's chain
  tokens (alpha, beta, gamma, delta) are not grounded in the active
  pack.  Load-bearing finding for ADR-0025 region construction.
- Tests (tests/test_inner_loop_phase4.py, 5 pass): pin the finding
  diagnostically (not gated).

Phase 5 — ADR-0025 design note (draft):
- No code changes proposed.  Scopes three architectural questions:
  (1) home (algebra/versor.py vs field/propagate.py vs generate/) —
      preliminary stance: algebra/versor.py.
  (2) threshold scheme (blade-normalized fraction recommended over
      static; learned/adaptive rejected for determinism).
  (3) teaching-loop boundary — Stance A confirmed: rejections are
      runtime hygiene only, no entanglement with teaching/*.
- Decisions to be closed before Draft → Accepted.

Phase 1 acceptance criteria from previous commit (7fccf36) carry
forward: wired, deterministic-when-wired, legacy hash preserved.

Suite: 1014 passed, 0 failed, 2 skipped.
This commit is contained in:
Shay 2026-05-17 14:07:50 -07:00
parent 7fccf368fb
commit 8146844d90
16 changed files with 2845 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

287
docs/decisions/ADR-0025-rotor-frame-admissibility-design-note.md Normal file
View file

@ -0,0 +1,287 @@
# ADR-0025 — Rotor / Frame Admissibility (Design Note)
| Field | Value |
|--------------|--------------------------------------|
| Status | **Draft — Design Note Only** |
| Date | 2026-05-17 |
| Supersedes | — |
| Extends | ADR-0022, ADR-0023, ADR-0024 |
| Decision lead| Shay (with CORE assistant) |
---
## Status note
This is a **design note**, not an implementation decision. No code
changes are proposed. Its purpose is to fix the home, scope, and
boundary of the next admissibility step *before* anything is built —
so the implementation doesn't inherit the wrong architectural shape
by default.
It will be promoted from Draft to a real ADR (Proposed → Accepted)
only after the design questions below are decided.
---
## Context
ADR-0024 added per-rotor inner-loop admissibility for the
**destination-token / direction** side of an `AdmissibilityRegion`:
when a candidate's CGA inner product against `relation_blade` falls
below `admissibility_threshold`, the candidate is excluded and the
walk re-selects until admitted or exhausted.
ADR-0024 explicitly deferred:
> Frame-versor admissibility (does the rotor preserve / transform
> within the frame constraint?) remains out of scope.
This note scopes that deferred work, but with two additional
constraints surfaced by the Phase 2–4 follow-up evidence:
1. **Phase 2 corpus observation** (`evals/forward_semantic_control/
inner_loop_runner.py`): on the existing v1+dev corpus, the
inner-loop mechanism is *wired, deterministic, causally
attributable* (null-control = boundary-only exactly,
`code_path_residual = 0.0`), but the chain-token outer-product
region produces `exhaustion_rate = 0.33` at `t = 0.0` — well
above the 5% benign-corpus ceiling.
2. **Phase 4 threshold characterization** (`threshold_
characterization.py`): **per-case the geometry separates cleanly**
(every mechanism-isolated v2 case has `correct_min > incorrect_max`),
but **no static global threshold** delivers
`separation_quality ≥ 0.8`. Blade norms vary ~10× across cases,
so the same threshold value means different things case-to-case.
Static thresholds — global, relation-typed, or frame-derived as a
constant — are insufficient.
These findings change the framing. The next step is not "extend the
same idea to the rotor side." It is two distinct questions:
* What level of the stack should enforce rotor/frame admissibility?
* What threshold scheme is geometrically valid given Phase 4?
---
## Question 1 — Architectural home
Three candidate homes for rotor-side admissibility:
### Option A — Generation-time filter (`generate/`)
Inherit ADR-0024's shape. Add a check inside the same per-step inner
loop in `generate/stream.py` that examines the *rotor* `V` (not just
the destination versor) before propagation.
**Pros:**
* Locality with ADR-0024. All admissibility decisions live in one
module.
* Trace evidence is uniform — one `AdmissibilityTraceStep` per
rotor-application.
**Cons:**
* Pushes algebra-shaped invariants into a generation-shaped module.
`generate/` already orchestrates candidates, salience, attention,
vault recall, persona — adding rotor invariant enforcement here
bloats the hot path and entangles concerns.
* Re-creates the "hot-path repair" anti-pattern CLAUDE.md explicitly
warns against, because the check would re-validate something
algebra already constructed.
### Option B — Versor construction invariant (`algebra/versor.py`)
Make rotor/frame admissibility part of sandwich closure. When
`word_transition_rotor(A, B)` builds the rotor, it also checks the
rotor against the active frame constraint. Violations raise — same
shape as the existing `versor_condition < 1e-6` invariant.
**Pros:**
* Aligned with the CLAUDE.md doctrine that algebra-owned closure
belongs in `algebra/`.
* No hot-path repair. The check is part of *construction*, not a
post-construction filter.
* Single invariant site — easier to reason about and prove.
**Cons:**
* Couples algebra to admissibility concepts (frame, relation_blade)
that today live in `generate/admissibility.py`. Either
`algebra/versor.py` grows a dependency on admissibility, or
admissibility primitives must be lifted to a shared layer.
* Honest refusal would surface deeper in the stack — callers that
today catch `ValueError` from `generate()` would also need to
catch from `propagate_step` or earlier.
### Option C — Field propagation guard (`field/propagate.py`)
Enforce at the *application* site: after rotor construction, before
`propagate_step` commits the new field state, verify the resulting
field stays within the frame's admissible cone.
**Pros:**
* Closest to the *claim*: rotor admissibility is fundamentally about
the field staying coherent under propagation, not about token
selection.
* `field/propagate.py` already owns the propagation invariant, so
this is a natural home for an additional propagation-time check.
**Cons:**
* `field/propagate.py` is explicitly listed in CLAUDE.md as a
*forbidden site* for normalization / drift repair / monitoring
("Do not add drift repair, grade projection, watchdogs, timers,
hot-path normalizers, or monitoring functions whose only purpose
is to repair another function"). An admissibility *guard* (raise
on violation, never repair) is closer to a precondition than a
monitor, but the boundary needs to be made explicit before this
option is chosen — otherwise it sets a precedent that erodes the
current rule.
### Recommended preliminary stance
**Option B** is the most aligned with project doctrine and the
cleanest invariant. Option C is the second-best, but only if the
"guard vs. monitor" distinction is made explicit and respected — and
even then, the construction-site discipline of Option B is preferable.
Option A is rejected as inheriting the wrong architectural shape from
ADR-0024 by momentum. ADR-0024 lived in `generate/` because it was
about *which destination to select*; rotor admissibility is about
*whether the rotor itself is valid*, which is a construction-site
question.
**Decision required**: B vs. C.
---
## Question 2 — Threshold scheme
Phase 4 surfaced that static thresholds are geometrically invalid on
this manifold. The right scheme is *not yet decided*, but the
candidates are:
1. **Per-candidate normalized score**: threshold = α · ‖blade‖, so
the same fraction of blade self-score is required regardless of
blade norm. Probable best first cut.
2. **Cosine-similarity-style normalization**: replace `cga_inner` in
the threshold check with
`cga_inner(v, blade) / (‖v‖ · ‖blade‖)`. Rejected on doctrinal
grounds — CLAUDE.md says "do not add cosine similarity ... to the
runtime path." Listed for completeness only.
3. **Per-relation-type static threshold**: a small table mapping
relation type → threshold. Phase 4 suggests this is insufficient
because *blade norm dominates*, not relation type, but it could
be a fallback if normalized scoring proves unstable.
4. **Frame-derived threshold**: threshold is a property of the frame
versor, not the candidate or the relation. Requires the frame
versor to be the primary admissibility object — i.e. Option B
above — and may collapse Question 1 and Question 2 into one
decision.
**Decision required**: (1) is the recommended starting point. Final
choice depends on Question 1 outcome and on a focused diagnostic
sweep over (1) and (3).
**Out of scope for the eventual ADR**: learned thresholds, adaptive
thresholds, online tuning. Deterministic replay must be preserved;
no learned policy enters the runtime path.
---
## Question 3 — Teaching loop boundary
ADR-0024 lives in `generate/`. The teaching loop in `teaching/*`
corrects model behavior through reviewed mutation. An open question:
when inner-loop (or rotor) admissibility rejects a candidate, does
that rejection become a *teachable event*?
Two stances:
* **A. Rejections are runtime hygiene only.** The teaching loop
sees the final selected token, not the rejected ones. Rejection
is a property of the deterministic admissibility region, not of
the reviewed teaching example.
* **B. Rejections are correction signals.** A teaching review can
examine `rejected_attempts` and decide whether the rejection was
correct (reinforce) or over-aggressive (loosen). This entangles
the teaching loop with admissibility geometry.
### Recommended stance: **A — strictly hygiene-only.**
Rationale:
* The teaching loop's contract is *reviewed mutation of identity /
pack / vault*. Admissibility regions are deterministic geometric
objects derived from intents and frames; they are not learned, and
there is no review surface for them today.
* Entangling teaching with admissibility would create a parallel
correction path — explicitly forbidden by CLAUDE.md ("Do not
create a parallel correction/learning path").
* Phase 4 showed that what needs to change is the threshold *scheme*,
not the per-event rejection decisions. Scheme changes belong in
the eventual ADR-0025 implementation, not in reviewed teaching
examples.
The decision must be **stated in the final ADR**, not left as a
silent default, so the next person who touches both systems doesn't
have to re-derive the boundary.
---
## Decisions to lock before ADR-0025 is implementable
1. **Home**: Option B (algebra construction) vs. Option C (field
propagation guard). Reject A explicitly.
2. **Threshold scheme**: blade-normalized fraction (recommended) vs.
relation-typed table (fallback). Run a small diagnostic sweep
on the v2 corpus + a small extension before committing.
3. **Teaching boundary**: Stance A (hygiene-only) confirmed. State
explicitly in the eventual ADR's "Out of scope" section.
4. **Trace evidence**: extend `AdmissibilityTraceStep` to include
rotor-side verdict, or add a separate `RotorAdmissibilityTrace`?
Lean toward extending the existing step to keep the trace shape
simple.
5. **Honest refusal**: at which layer does `ValueError` get raised
on rotor rejection? Decided by (1) — same site as the check.
---
## Evidence and links
* ADR-0022 — Forward Semantic Control (region prefilter).
* ADR-0023 — Forward Semantic Control proof evidence.
* ADR-0024 — Inner-loop per-rotor admissibility (token-side).
* Phase 2 report — `evals/forward_semantic_control/results/
phase2_inner_loop_report.json` — causal attribution proven, but
exhaustion gate fails on existing corpus.
* Phase 3 report — `evals/forward_semantic_control/results/
phase3_v2_report.json` — mechanism isolated on real pack,
`mechanism_isolated = true` on 5/5 cases.
* Phase 4 reports — `evals/forward_semantic_control/results/
phase4_characterization_{v1_plus_dev,v2,combined}.json` — static
thresholds geometrically insufficient.
* Tests pinning the findings: `tests/test_inner_loop_phase2.py`,
`tests/test_inner_loop_phase3.py`, `tests/test_inner_loop_phase4.py`.
---
## What this note does NOT decide
This note does not:
* Choose between Options B and C — that requires a short focused
spike on the algebra-vs-propagation tradeoff.
* Specify the threshold scheme — that requires a small diagnostic
sweep over normalized-fraction vs. relation-typed schemes on the
v2 corpus.
* Authorize any code changes. Promotion from Draft to Accepted
requires the open questions to be closed.

434
evals/forward_semantic_control/inner_loop_runner.py Normal file
View file

@ -0,0 +1,434 @@
"""Phase 2 corpus-observation runner — ADR-0024 inner-loop admissibility.
Runs each FSC case through a four-condition matrix on the *same*
field state so any pass-rate delta is attributable to the inner-loop
mechanism alone (region/vocab/persona/prompt held constant):
(A) boundary_only — inner_loop_admissibility=False
(B) null_control — inner_loop_admissibility=True,
inner_loop_force_admit=True
(C) inner_loop_t0 — inner_loop_admissibility=True,
admissibility_threshold=0.0
(D) inner_loop_tpos — inner_loop_admissibility=True,
admissibility_threshold=0.25
Reports per condition:
pass_rate
mean_rejection_count_per_turn
non_empty_rejected_attempts_rate
exhaustion_rate (gated: must be ≤ EXHAUSTION_CEILING)
mean_admissibility_checks_per_turn
mean_added_latency_ms
p95_added_latency_ms
trace_hash_stability_passes (5 reruns ⇒ identical trace hash)
Causal attribution: delta(C - A) is the rejection effect *iff* delta(B - A) ≈ 0.
If null_control diverges from boundary_only, the inner-loop code path
itself is changing selection (call ordering, side effects); ADR-0024
proof is contaminated until that residual is explained.
Conforms to the framework interface (``run_lane``) so the standard
eval harness can call it.
"""
from __future__ import annotations
import statistics
import time
from dataclasses import dataclass, field
from typing import Any
import numpy as np
from algebra.cga import outer_product
from chat.runtime import ChatRuntime
from core.cognition.trace import hash_admissibility_trace
from core.config import RuntimeConfig
from generate.admissibility import AdmissibilityRegion, RegionSource
from generate.result import GenerationResult
from generate.stream import generate as generate_walk
# Exhaustion ceiling on benign v1 corpus. Above this, the configured
# threshold is producing honest refusals where it should produce
# answers — a capability regression disguised as a virtue.
EXHAUSTION_CEILING = 0.05
# Default tested-positive threshold for condition (D). Phase 4 will
# characterise the threshold landscape; this is a probe-point only.
PROBE_THRESHOLD_POSITIVE = 0.25
# Reruns for hash-stability check. 5 is the same N used by the
# Phase 1 acceptance test in ``tests/test_inner_loop_admissibility.py``.
HASH_STABILITY_RERUNS = 5
@dataclass(slots=True)
class _ConditionMetrics:
label: str
pass_count: int = 0
case_count: int = 0
rejection_counts: list[int] = field(default_factory=list)
non_empty_rejection_cases: int = 0
exhaustions: int = 0
admissibility_checks: list[int] = field(default_factory=list)
latencies_ms: list[float] = field(default_factory=list)
trace_hash_stable_count: int = 0
trace_hash_checked_count: int = 0
def as_dict(self) -> dict[str, Any]:
n = max(self.case_count, 1)
return {
"label": self.label,
"pass_rate": round(self.pass_count / n, 4),
"mean_rejection_count_per_turn": round(
statistics.mean(self.rejection_counts) if self.rejection_counts else 0.0,
4,
),
"non_empty_rejected_attempts_rate": round(
self.non_empty_rejection_cases / n, 4
),
"exhaustion_rate": round(self.exhaustions / n, 4),
"mean_admissibility_checks_per_turn": round(
statistics.mean(self.admissibility_checks)
if self.admissibility_checks
else 0.0,
4,
),
"mean_added_latency_ms": round(
statistics.mean(self.latencies_ms) if self.latencies_ms else 0.0, 4
),
"p95_added_latency_ms": round(_p95(self.latencies_ms), 4),
"trace_hash_stability_pass_rate": round(
self.trace_hash_stable_count / max(self.trace_hash_checked_count, 1), 4
),
"case_count": self.case_count,
}
@dataclass(slots=True)
class InnerLoopReport:
metrics: dict[str, Any] = field(default_factory=dict)
case_details: list[dict[str, Any]] = field(default_factory=list)
def _p95(values: list[float]) -> float:
if not values:
return 0.0
sorted_values = sorted(values)
idx = int(round(0.95 * (len(sorted_values) - 1)))
return sorted_values[idx]
def _region_from_token_chain(
vocab,
tokens: tuple[str, ...],
*,
label: str,
) -> AdmissibilityRegion | None:
indices: list[int] = []
versors: list[np.ndarray] = []
for raw in tokens:
token = raw.lower().strip()
if not token:
continue
try:
idx = vocab.index_of(token)
except (KeyError, AttributeError, IndexError):
continue
try:
versor = np.asarray(vocab.get_versor(token), dtype=np.float32)
except (KeyError, AttributeError):
continue
indices.append(int(idx))
versors.append(versor)
if not indices:
return None
blade = versors[0]
for nxt in versors[1:]:
blade = outer_product(blade, nxt)
return AdmissibilityRegion(
allowed_indices=np.asarray(indices, dtype=np.int64),
relation_blade=blade,
source=RegionSource.RELATION,
label=label,
)
def _surfaces_endpoint(surface: str, expected_endpoint: str) -> bool:
if not surface or not expected_endpoint:
return False
return expected_endpoint.lower().strip() in surface.lower()
def _run_walk(
field_state,
vocab,
persona,
region: AdmissibilityRegion | None,
*,
inner_loop: bool,
threshold: float,
force_admit: bool,
) -> tuple[GenerationResult | None, float, bool]:
"""Run one walk, return (result, latency_ms, exhaustion_occurred)."""
start = time.perf_counter()
try:
result = generate_walk(
field_state,
vocab,
persona,
max_tokens=8,
region=region,
inner_loop_admissibility=inner_loop,
admissibility_threshold=threshold,
inner_loop_force_admit=force_admit,
)
latency_ms = (time.perf_counter() - start) * 1000.0
return result, latency_ms, False
except ValueError:
latency_ms = (time.perf_counter() - start) * 1000.0
return None, latency_ms, True
def _rejection_count(result: GenerationResult | None) -> int:
if result is None:
return 0
return sum(len(step.rejected_attempts) for step in result.admissibility_trace)
def _admissibility_check_count(result: GenerationResult | None) -> int:
"""One check per attempt — admissions + rejections."""
if result is None:
return 0
return sum(len(step.rejected_attempts) + 1 for step in result.admissibility_trace)
def _surface_from(result: GenerationResult | None) -> str:
if result is None or not result.tokens:
return ""
return " ".join(result.tokens)
def _hash_of(result: GenerationResult | None) -> str:
if result is None:
return "__exhausted__"
return hash_admissibility_trace(result.admissibility_trace)
def _prime_runtime(case: dict[str, Any]) -> ChatRuntime:
runtime = ChatRuntime()
for prime in case.get("prime", []):
try:
runtime.chat(prime, max_tokens=8)
except ValueError:
pass
try:
runtime.chat(case["prompt"], max_tokens=8)
except ValueError:
pass
return runtime
def _run_case(case: dict[str, Any]) -> dict[str, Any]:
expected = case.get("expected_endpoint", "")
runtime = _prime_runtime(case)
field_state = runtime.session.state
if field_state is None:
return {"id": case.get("id", ""), "skipped": True, "reason": "no_field_state"}
vocab = runtime.session.vocab
persona = runtime.session.persona
chain_tokens = tuple(case.get("chain_tokens", ()))
if not chain_tokens and expected:
chain_tokens = (expected,)
region = _region_from_token_chain(
vocab, chain_tokens, label=f"phase2[{case.get('id', '')}]"
)
if region is None:
return {"id": case.get("id", ""), "skipped": True, "reason": "no_grounded_chain"}
# Boundary-only baseline latency — used so reported "added" latency
# is the cost over the boundary-only path, not absolute.
baseline_result, baseline_latency_ms, baseline_exh = _run_walk(
field_state, vocab, persona, region,
inner_loop=False, threshold=0.0, force_admit=False,
)
conditions: dict[str, dict[str, Any]] = {}
hash_stability: dict[str, bool] = {}
# (A) boundary-only — recorded above
conditions["boundary_only"] = {
"surface": _surface_from(baseline_result),
"rejections": _rejection_count(baseline_result),
"checks": _admissibility_check_count(baseline_result),
"latency_ms": 0.0, # baseline — no "added" latency
"absolute_latency_ms": baseline_latency_ms,
"exhausted": baseline_exh,
"trace_hash": _hash_of(baseline_result),
}
# (B) null_control
nc_result, nc_latency_ms, nc_exh = _run_walk(
field_state, vocab, persona, region,
inner_loop=True, threshold=0.0, force_admit=True,
)
conditions["null_control"] = {
"surface": _surface_from(nc_result),
"rejections": _rejection_count(nc_result),
"checks": _admissibility_check_count(nc_result),
"latency_ms": max(nc_latency_ms - baseline_latency_ms, 0.0),
"absolute_latency_ms": nc_latency_ms,
"exhausted": nc_exh,
"trace_hash": _hash_of(nc_result),
}
# (C) inner_loop_t0
c_result, c_latency_ms, c_exh = _run_walk(
field_state, vocab, persona, region,
inner_loop=True, threshold=0.0, force_admit=False,
)
conditions["inner_loop_t0"] = {
"surface": _surface_from(c_result),
"rejections": _rejection_count(c_result),
"checks": _admissibility_check_count(c_result),
"latency_ms": max(c_latency_ms - baseline_latency_ms, 0.0),
"absolute_latency_ms": c_latency_ms,
"exhausted": c_exh,
"trace_hash": _hash_of(c_result),
}
# (D) inner_loop_tpos
d_result, d_latency_ms, d_exh = _run_walk(
field_state, vocab, persona, region,
inner_loop=True, threshold=PROBE_THRESHOLD_POSITIVE, force_admit=False,
)
conditions["inner_loop_tpos"] = {
"surface": _surface_from(d_result),
"rejections": _rejection_count(d_result),
"checks": _admissibility_check_count(d_result),
"latency_ms": max(d_latency_ms - baseline_latency_ms, 0.0),
"absolute_latency_ms": d_latency_ms,
"exhausted": d_exh,
"trace_hash": _hash_of(d_result),
}
# Hash stability: rerun condition (C) HASH_STABILITY_RERUNS-1 more
# times on the *same* field state (re-priming each time to keep
# vault state comparable). All hashes must match.
base_hash = conditions["inner_loop_t0"]["trace_hash"]
stable = True
for _ in range(HASH_STABILITY_RERUNS - 1):
re_runtime = _prime_runtime(case)
re_state = re_runtime.session.state
if re_state is None:
stable = False
break
re_vocab = re_runtime.session.vocab
re_persona = re_runtime.session.persona
re_region = _region_from_token_chain(
re_vocab, chain_tokens, label=f"phase2[{case.get('id', '')}]"
)
if re_region is None:
stable = False
break
re_result, _re_latency, _re_exh = _run_walk(
re_state, re_vocab, re_persona, re_region,
inner_loop=True, threshold=0.0, force_admit=False,
)
if _hash_of(re_result) != base_hash:
stable = False
break
hash_stability["inner_loop_t0"] = stable
detail: dict[str, Any] = {
"id": case.get("id", ""),
"kind": case.get("kind", ""),
"expected_endpoint": expected,
"conditions": conditions,
"hash_stability": hash_stability,
"passes": {
label: _surfaces_endpoint(cond["surface"], expected)
for label, cond in conditions.items()
},
}
return detail
def run_lane(
cases: list[dict[str, Any]],
*,
config: RuntimeConfig | None = None,
workers: int | None = None,
) -> InnerLoopReport:
_ = config
_ = workers # serial — Phase 2 is small and latency-sensitive
if not cases:
return InnerLoopReport(metrics={}, case_details=[])
case_details: list[dict[str, Any]] = []
by_condition: dict[str, _ConditionMetrics] = {
"boundary_only": _ConditionMetrics(label="boundary_only"),
"null_control": _ConditionMetrics(label="null_control"),
"inner_loop_t0": _ConditionMetrics(label="inner_loop_t0"),
"inner_loop_tpos": _ConditionMetrics(label="inner_loop_tpos"),
}
for case in cases:
detail = _run_case(case)
case_details.append(detail)
if detail.get("skipped"):
continue
for label, metrics in by_condition.items():
cond = detail["conditions"][label]
metrics.case_count += 1
if detail["passes"][label]:
metrics.pass_count += 1
metrics.rejection_counts.append(cond["rejections"])
if cond["rejections"] > 0:
metrics.non_empty_rejection_cases += 1
if cond["exhausted"]:
metrics.exhaustions += 1
metrics.admissibility_checks.append(cond["checks"])
metrics.latencies_ms.append(cond["latency_ms"])
if label in detail["hash_stability"]:
metrics.trace_hash_checked_count += 1
if detail["hash_stability"][label]:
metrics.trace_hash_stable_count += 1
per_condition = {label: m.as_dict() for label, m in by_condition.items()}
# Causal attribution:
# rejection_effect = pass(inner_loop_t0) - pass(boundary_only)
# code_path_residual = pass(null_control) - pass(boundary_only)
# If |code_path_residual| is non-zero, the rejection effect is
# contaminated by code-path differences and the proof is invalid.
rejection_effect = (
per_condition["inner_loop_t0"]["pass_rate"]
- per_condition["boundary_only"]["pass_rate"]
)
code_path_residual = (
per_condition["null_control"]["pass_rate"]
- per_condition["boundary_only"]["pass_rate"]
)
# Exhaustion gate — applies to inner-loop conditions only.
exhaustion_gate_pass = all(
per_condition[label]["exhaustion_rate"] <= EXHAUSTION_CEILING
for label in ("inner_loop_t0", "inner_loop_tpos")
)
metrics: dict[str, Any] = {
"per_condition": per_condition,
"rejection_effect": round(rejection_effect, 4),
"code_path_residual": round(code_path_residual, 4),
"causal_attribution_valid": abs(code_path_residual) < 1e-9,
"exhaustion_ceiling": EXHAUSTION_CEILING,
"exhaustion_gate_pass": exhaustion_gate_pass,
"probe_threshold_positive": PROBE_THRESHOLD_POSITIVE,
"case_count": len(cases),
"skipped_count": sum(1 for d in case_details if d.get("skipped")),
}
return InnerLoopReport(metrics=metrics, case_details=case_details)

5
evals/forward_semantic_control/public/v2/cases.jsonl Normal file
View file

@ -0,0 +1,5 @@
{"id":"FSC-PUB-V2-001","kind":"mechanism_isolation","semantic_pair":"question/answer","seed_token":"symbol","admissible_tokens":["answer","question"],"relation_blade_token":"question","expected_endpoint":"question","forbidden_token":"answer","admissibility_threshold":1.1221,"rationale":"Field state from 'symbol' is geometrically nearer to 'answer' than 'question' (gap=+1.566), so boundary-only selects 'answer'. Blade=versor('question') admits 'question' at score 1.420 but rejects 'answer' at score 0.824. Threshold 1.122 sits between them."}
{"id":"FSC-PUB-V2-002","kind":"mechanism_isolation","semantic_pair":"truth/meaning","seed_token":"infer","admissible_tokens":["meaning","truth"],"relation_blade_token":"truth","expected_endpoint":"truth","forbidden_token":"meaning","admissibility_threshold":0.9449,"rationale":"Field state from 'infer' is geometrically nearer to 'meaning' than 'truth' (gap=+2.193), so boundary-only selects 'meaning'. Blade=versor('truth') admits 'truth' at score 1.173 but rejects 'meaning' at score 0.717. Threshold 0.945 separates."}
{"id":"FSC-PUB-V2-003","kind":"mechanism_isolation","semantic_pair":"think/spirit","seed_token":"corrects","admissible_tokens":["spirit","think"],"relation_blade_token":"think","expected_endpoint":"think","forbidden_token":"spirit","admissibility_threshold":6.0833,"rationale":"Cognition vs. ineffable pole. Field state from 'corrects' is nearer to 'spirit' (gap=+2.711). Blade=versor('think') admits 'think' at score 12.72 and rejects 'spirit' at score -0.55. Wide blade gap (+13.27)."}
{"id":"FSC-PUB-V2-004","kind":"mechanism_isolation","semantic_pair":"understand/say","seed_token":"corrects","admissible_tokens":["say","understand"],"relation_blade_token":"understand","expected_endpoint":"understand","forbidden_token":"say","admissibility_threshold":4.055,"rationale":"Intellection vs. utterance. Field state from 'corrects' is nearer to 'say'. Blade=versor('understand') admits 'understand' at score 5.74 and rejects 'say' at score 2.37."}
{"id":"FSC-PUB-V2-005","kind":"mechanism_isolation","semantic_pair":"thought/beginning","seed_token":"corrects","admissible_tokens":["beginning","thought"],"relation_blade_token":"thought","expected_endpoint":"thought","forbidden_token":"beginning","admissibility_threshold":7.99,"rationale":"Cognition vs. genesis. Field state from 'corrects' is nearer to 'beginning'. Blade=versor('thought') admits 'thought' at score 14.36 and rejects 'beginning' at score 1.62. Largest blade gap (+12.75)."}

532
evals/forward_semantic_control/results/phase2_inner_loop_report.json Normal file
View file

@ -0,0 +1,532 @@
{
"metrics": {
"per_condition": {
"boundary_only": {
"label": "boundary_only",
"pass_rate": 1.0,
"mean_rejection_count_per_turn": 0,
"non_empty_rejected_attempts_rate": 0.0,
"exhaustion_rate": 0.0,
"mean_admissibility_checks_per_turn": 3,
"mean_added_latency_ms": 0.0,
"p95_added_latency_ms": 0.0,
"trace_hash_stability_pass_rate": 0.0,
"case_count": 9
},
"null_control": {
"label": "null_control",
"pass_rate": 1.0,
"mean_rejection_count_per_turn": 0,
"non_empty_rejected_attempts_rate": 0.0,
"exhaustion_rate": 0.0,
"mean_admissibility_checks_per_turn": 3,
"mean_added_latency_ms": 0.0505,
"p95_added_latency_ms": 0.1467,
"trace_hash_stability_pass_rate": 0.0,
"case_count": 9
},
"inner_loop_t0": {
"label": "inner_loop_t0",
"pass_rate": 0.6667,
"mean_rejection_count_per_turn": 0.2222,
"non_empty_rejected_attempts_rate": 0.2222,
"exhaustion_rate": 0.3333,
"mean_admissibility_checks_per_turn": 2.4444,
"mean_added_latency_ms": 0.0064,
"p95_added_latency_ms": 0.0577,
"trace_hash_stability_pass_rate": 1.0,
"case_count": 9
},
"inner_loop_tpos": {
"label": "inner_loop_tpos",
"pass_rate": 0.4444,
"mean_rejection_count_per_turn": 0,
"non_empty_rejected_attempts_rate": 0.0,
"exhaustion_rate": 0.5556,
"mean_admissibility_checks_per_turn": 1.3333,
"mean_added_latency_ms": 0.0,
"p95_added_latency_ms": 0.0,
"trace_hash_stability_pass_rate": 0.0,
"case_count": 9
}
},
"rejection_effect": -0.3333,
"code_path_residual": 0.0,
"causal_attribution_valid": true,
"exhaustion_ceiling": 0.05,
"exhaustion_gate_pass": false,
"probe_threshold_positive": 0.25,
"case_count": 9,
"skipped_count": 0
},
"case_details": [
{
"id": "FSC-PUB-001",
"kind": "chain_three_hop",
"expected_endpoint": "delta",
"conditions": {
"boundary_only": {
"surface": "alpha gamma delta beta",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 6.3321669986180495,
"exhausted": false,
"trace_hash": "f1749b663fdf0d85445d4ace27a2c4e492894445aad050e97cec8ea8a89650c5"
},
"null_control": {
"surface": "alpha gamma delta beta",
"rejections": 0,
"checks": 4,
"latency_ms": 0.07079200076987036,
"absolute_latency_ms": 6.40295899938792,
"exhausted": false,
"trace_hash": "f1749b663fdf0d85445d4ace27a2c4e492894445aad050e97cec8ea8a89650c5"
},
"inner_loop_t0": {
"surface": "alpha gamma delta alpha",
"rejections": 1,
"checks": 5,
"latency_ms": 0.05766700269305147,
"absolute_latency_ms": 6.389834001311101,
"exhausted": false,
"trace_hash": "a6c009be37a048c723dfa87b155f277f6d0c0ab0c0ab9355180e0a09ca014045"
},
"inner_loop_tpos": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 1.3651670014951378,
"exhausted": true,
"trace_hash": "__exhausted__"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": true,
"inner_loop_tpos": false
}
},
{
"id": "FSC-DEV-001",
"kind": "chain_three_hop",
"expected_endpoint": "delta",
"conditions": {
"boundary_only": {
"surface": "alpha gamma delta beta",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 6.53266700101085,
"exhausted": false,
"trace_hash": "81a5907ed8c58cba69648dff42f15b24547ac585b36db1ba178b5314908f0919"
},
"null_control": {
"surface": "alpha gamma delta beta",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 6.400083999324124,
"exhausted": false,
"trace_hash": "81a5907ed8c58cba69648dff42f15b24547ac585b36db1ba178b5314908f0919"
},
"inner_loop_t0": {
"surface": "alpha gamma delta alpha",
"rejections": 1,
"checks": 5,
"latency_ms": 0.0,
"absolute_latency_ms": 6.458666000980884,
"exhausted": false,
"trace_hash": "afb66f361ebf275498956e51a92598c73ed29fc569949009c354ebe33414d286"
},
"inner_loop_tpos": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 1.3585829983639996,
"exhausted": true,
"trace_hash": "__exhausted__"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": true,
"inner_loop_tpos": false
}
},
{
"id": "FSC-DEV-002",
"kind": "negative_control_no_chain",
"expected_endpoint": "beta",
"conditions": {
"boundary_only": {
"surface": "alpha beta",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 2.8974999986530747,
"exhausted": false,
"trace_hash": "9ecd42e41d24f6167c7dbafdab07a309b5d5cf9d673d412bd6ab418c192095de"
},
"null_control": {
"surface": "alpha beta",
"rejections": 0,
"checks": 2,
"latency_ms": 0.08908300151233561,
"absolute_latency_ms": 2.9865830001654103,
"exhausted": false,
"trace_hash": "9ecd42e41d24f6167c7dbafdab07a309b5d5cf9d673d412bd6ab418c192095de"
},
"inner_loop_t0": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 2.1187089987506624,
"exhausted": true,
"trace_hash": "__exhausted__"
},
"inner_loop_tpos": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 0.6152080022729933,
"exhausted": true,
"trace_hash": "__exhausted__"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": false,
"inner_loop_tpos": false
}
},
{
"id": "FSC-DEV-003",
"kind": "frame_constraint_blocks_wrong_relation",
"expected_endpoint": "beta",
"conditions": {
"boundary_only": {
"surface": "alpha beta",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 3.1092079989321064,
"exhausted": false,
"trace_hash": "f5f42fdb1b841368ccff2b9f4a0bdfc666001800076a72277298e91aa97b7cd1"
},
"null_control": {
"surface": "alpha beta",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 3.086417000304209,
"exhausted": false,
"trace_hash": "f5f42fdb1b841368ccff2b9f4a0bdfc666001800076a72277298e91aa97b7cd1"
},
"inner_loop_t0": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 2.2947910001676064,
"exhausted": true,
"trace_hash": "__exhausted__"
},
"inner_loop_tpos": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 0.6693330033158418,
"exhausted": true,
"trace_hash": "__exhausted__"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": false,
"inner_loop_tpos": false
}
},
{
"id": "FSC-DEV-004",
"kind": "chain_two_hop_means",
"expected_endpoint": "omicron",
"conditions": {
"boundary_only": {
"surface": "mu nu omicron",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 2.9942500004835892,
"exhausted": false,
"trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d"
},
"null_control": {
"surface": "mu nu omicron",
"rejections": 0,
"checks": 3,
"latency_ms": 0.11316699965391308,
"absolute_latency_ms": 3.1074170001375023,
"exhausted": false,
"trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d"
},
"inner_loop_t0": {
"surface": "mu nu omicron",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 2.97054200200364,
"exhausted": false,
"trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d"
},
"inner_loop_tpos": {
"surface": "mu nu omicron",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 2.97287500143284,
"exhausted": false,
"trace_hash": "48e86b2d95103d6218d4f01871d9b0a2d3ebbbcb8fc7583646bd19bf6570aa9d"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": true,
"inner_loop_tpos": true
}
},
{
"id": "FSC-DEV-005",
"kind": "chain_three_hop_precedes",
"expected_endpoint": "tau",
"conditions": {
"boundary_only": {
"surface": "pi rho sigma tau",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 3.3884170006786007,
"exhausted": false,
"trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b"
},
"null_control": {
"surface": "pi rho sigma tau",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 3.3558340001036413,
"exhausted": false,
"trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b"
},
"inner_loop_t0": {
"surface": "pi rho sigma tau",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 3.295792001154041,
"exhausted": false,
"trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b"
},
"inner_loop_tpos": {
"surface": "pi rho sigma tau",
"rejections": 0,
"checks": 4,
"latency_ms": 0.0,
"absolute_latency_ms": 3.1976249993022066,
"exhausted": false,
"trace_hash": "893e663ec87151db1538d090a2244c5a61cd40a8c64d6d16f9f1e39bece2479b"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": true,
"inner_loop_tpos": true
}
},
{
"id": "FSC-DEV-006",
"kind": "chain_two_hop_part_of",
"expected_endpoint": "chi",
"conditions": {
"boundary_only": {
"surface": "phi chi upsilon",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 3.3560420015419368,
"exhausted": false,
"trace_hash": "244373cd293ef544c1c4395218335c1c1f255d562a9697646b734041412d407f"
},
"null_control": {
"surface": "phi chi upsilon",
"rejections": 0,
"checks": 3,
"latency_ms": 0.1467080001020804,
"absolute_latency_ms": 3.502750001644017,
"exhausted": false,
"trace_hash": "244373cd293ef544c1c4395218335c1c1f255d562a9697646b734041412d407f"
},
"inner_loop_t0": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 0.9736249994602986,
"exhausted": true,
"trace_hash": "__exhausted__"
},
"inner_loop_tpos": {
"surface": "",
"rejections": 0,
"checks": 0,
"latency_ms": 0.0,
"absolute_latency_ms": 0.9959999988495838,
"exhausted": true,
"trace_hash": "__exhausted__"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": false,
"inner_loop_tpos": false
}
},
{
"id": "FSC-DEV-007",
"kind": "adversarial_distractor_means_vs_cause",
"expected_endpoint": "omega",
"conditions": {
"boundary_only": {
"surface": "psi omega",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 1.9424589991103858,
"exhausted": false,
"trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd"
},
"null_control": {
"surface": "psi omega",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 1.8285829974047374,
"exhausted": false,
"trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd"
},
"inner_loop_t0": {
"surface": "psi omega",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 1.8532499998400453,
"exhausted": false,
"trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd"
},
"inner_loop_tpos": {
"surface": "psi omega",
"rejections": 0,
"checks": 2,
"latency_ms": 0.0,
"absolute_latency_ms": 1.8378329987172037,
"exhausted": false,
"trace_hash": "8805362f4f24b1c50272287bbcb71f8300837adceb8c1214ab3fe7a5260307fd"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": true,
"inner_loop_tpos": true
}
},
{
"id": "FSC-DEV-008",
"kind": "adversarial_distractor_chain_branching",
"expected_endpoint": "zeta",
"conditions": {
"boundary_only": {
"surface": "eta theta zeta",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 3.0924999991839286,
"exhausted": false,
"trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f"
},
"null_control": {
"surface": "eta theta zeta",
"rejections": 0,
"checks": 3,
"latency_ms": 0.034500000765547156,
"absolute_latency_ms": 3.1269999999494758,
"exhausted": false,
"trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f"
},
"inner_loop_t0": {
"surface": "eta theta zeta",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 2.940541999123525,
"exhausted": false,
"trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f"
},
"inner_loop_tpos": {
"surface": "eta theta zeta",
"rejections": 0,
"checks": 3,
"latency_ms": 0.0,
"absolute_latency_ms": 3.039292001631111,
"exhausted": false,
"trace_hash": "cedc7183c1ae90cae5eb538f1116b30231a98ed89f1d8c8728d4835df657323f"
}
},
"hash_stability": {
"inner_loop_t0": true
},
"passes": {
"boundary_only": true,
"null_control": true,
"inner_loop_t0": true,
"inner_loop_tpos": true
}
}
]
}

129
evals/forward_semantic_control/results/phase3_v2_report.json Normal file
View file

@ -0,0 +1,129 @@
{
"metrics": {
"case_count": 5,
"skipped_count": 0,
"eligible_count": 5,
"pass_count": 5,
"pass_rate": 1.0,
"boundary_decoy_rate": 1.0,
"rejection_traced_rate": 1.0,
"mechanism_isolated": true
},
"case_details": [
{
"id": "FSC-PUB-V2-001",
"skipped": false,
"passed": true,
"semantic_pair": "question/answer",
"expected_endpoint": "question",
"forbidden_token": "answer",
"boundary_selected": "answer",
"boundary_picks_forbidden": true,
"boundary_verdict_rejects": true,
"inner_selected": "question",
"inner_admitted": true,
"inner_exhausted": false,
"rejection_in_trace": true,
"rejected_attempts": [
[
21,
"answer",
0.823794424533844
]
],
"rationale": "Field state from 'symbol' is geometrically nearer to 'answer' than 'question' (gap=+1.566), so boundary-only selects 'answer'. Blade=versor('question') admits 'question' at score 1.420 but rejects 'answer' at score 0.824. Threshold 1.122 sits between them."
},
{
"id": "FSC-PUB-V2-002",
"skipped": false,
"passed": true,
"semantic_pair": "truth/meaning",
"expected_endpoint": "truth",
"forbidden_token": "meaning",
"boundary_selected": "meaning",
"boundary_picks_forbidden": true,
"boundary_verdict_rejects": true,
"inner_selected": "truth",
"inner_admitted": true,
"inner_exhausted": false,
"rejection_in_trace": true,
"rejected_attempts": [
[
110,
"meaning",
0.7167291045188904
]
],
"rationale": "Field state from 'infer' is geometrically nearer to 'meaning' than 'truth' (gap=+2.193), so boundary-only selects 'meaning'. Blade=versor('truth') admits 'truth' at score 1.173 but rejects 'meaning' at score 0.717. Threshold 0.945 separates."
},
{
"id": "FSC-PUB-V2-003",
"skipped": false,
"passed": true,
"semantic_pair": "think/spirit",
"expected_endpoint": "think",
"forbidden_token": "spirit",
"boundary_selected": "spirit",
"boundary_picks_forbidden": true,
"boundary_verdict_rejects": true,
"inner_selected": "think",
"inner_admitted": true,
"inner_exhausted": false,
"rejection_in_trace": true,
"rejected_attempts": [
[
29,
"spirit",
-0.5510096549987793
]
],
"rationale": "Cognition vs. ineffable pole. Field state from 'corrects' is nearer to 'spirit' (gap=+2.711). Blade=versor('think') admits 'think' at score 12.72 and rejects 'spirit' at score -0.55. Wide blade gap (+13.27)."
},
{
"id": "FSC-PUB-V2-004",
"skipped": false,
"passed": true,
"semantic_pair": "understand/say",
"expected_endpoint": "understand",
"forbidden_token": "say",
"boundary_selected": "say",
"boundary_picks_forbidden": true,
"boundary_verdict_rejects": true,
"inner_selected": "understand",
"inner_admitted": true,
"inner_exhausted": false,
"rejection_in_trace": true,
"rejected_attempts": [
[
19,
"say",
2.3711442947387695
]
],
"rationale": "Intellection vs. utterance. Field state from 'corrects' is nearer to 'say'. Blade=versor('understand') admits 'understand' at score 5.74 and rejects 'say' at score 2.37."
},
{
"id": "FSC-PUB-V2-005",
"skipped": false,
"passed": true,
"semantic_pair": "thought/beginning",
"expected_endpoint": "thought",
"forbidden_token": "beginning",
"boundary_selected": "beginning",
"boundary_picks_forbidden": true,
"boundary_verdict_rejects": true,
"inner_selected": "thought",
"inner_admitted": true,
"inner_exhausted": false,
"rejection_in_trace": true,
"rejected_attempts": [
[
27,
"beginning",
1.616857886314392
]
],
"rationale": "Cognition vs. genesis. Field state from 'corrects' is nearer to 'beginning'. Blade=versor('thought') admits 'thought' at score 14.36 and rejects 'beginning' at score 1.62. Largest blade gap (+12.75)."
}
]
}

211
evals/forward_semantic_control/results/phase4_characterization_combined.json Normal file
View file

@ -0,0 +1,211 @@
{
"metrics": {
"thresholds_swept": [
-1.0,
-0.5,
0.0,
0.1,
0.25,
0.5,
1.0
],
"best_threshold": 1.0,
"best_separation_quality": 0.6,
"separation_quality_gate": 0.8,
"passes_separation_gate": false,
"total_correct_candidates": 5,
"total_incorrect_candidates": 5,
"overlap_ratio": 0.0803,
"case_count": 14,
"skipped_count": 9,
"geometry_supports_static_threshold": false
},
"per_threshold": {
"-1.0": {
"TP": 5,
"FN": 0,
"FP": 5,
"TN": 0,
"TP_rate": 1.0,
"FP_rate": 1.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"-0.5": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.0": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.1": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.25": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.5": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"1.0": {
"TP": 5,
"FN": 0,
"FP": 2,
"TN": 3,
"TP_rate": 1.0,
"FP_rate": 0.4,
"FN_rate": 0.0,
"TN_rate": 0.6,
"separation_quality": 0.6
}
},
"score_distributions": {
"correct": {
"count": 5,
"mean": 7.0824,
"median": 5.7389,
"min": 1.1731,
"max": 14.3621,
"stdev": 6.1954
},
"incorrect": {
"count": 5,
"mean": 0.9955,
"median": 0.8238,
"min": -0.551,
"max": 2.3711,
"stdev": 1.0929
},
"overlap": {
"correct_min": 1.1731,
"incorrect_max": 2.3711,
"overlap_size": 1.198,
"full_range": 14.9131,
"overlap_ratio": 0.0803,
"separable_by_static_threshold": false
}
},
"case_details": [
{
"id": "FSC-PUB-001",
"skipped": true
},
{
"id": "FSC-DEV-001",
"skipped": true
},
{
"id": "FSC-DEV-002",
"skipped": true
},
{
"id": "FSC-DEV-003",
"skipped": true
},
{
"id": "FSC-DEV-004",
"skipped": true
},
{
"id": "FSC-DEV-005",
"skipped": true
},
{
"id": "FSC-DEV-006",
"skipped": true
},
{
"id": "FSC-DEV-007",
"skipped": true
},
{
"id": "FSC-DEV-008",
"skipped": true
},
{
"id": "FSC-PUB-V2-001",
"correct_scores": [
1.4205
],
"incorrect_scores": [
0.8238
]
},
{
"id": "FSC-PUB-V2-002",
"correct_scores": [
1.1731
],
"incorrect_scores": [
0.7167
]
},
{
"id": "FSC-PUB-V2-003",
"correct_scores": [
12.7176
],
"incorrect_scores": [
-0.551
]
},
{
"id": "FSC-PUB-V2-004",
"correct_scores": [
5.7389
],
"incorrect_scores": [
2.3711
]
},
{
"id": "FSC-PUB-V2-005",
"correct_scores": [
14.3621
],
"incorrect_scores": [
1.6169
]
}
]
}

148
evals/forward_semantic_control/results/phase4_characterization_v1_plus_dev.json Normal file
View file

@ -0,0 +1,148 @@
{
"metrics": {
"thresholds_swept": [
-1.0,
-0.5,
0.0,
0.1,
0.25,
0.5,
1.0
],
"best_threshold": -1.0,
"best_separation_quality": 0.0,
"separation_quality_gate": 0.8,
"passes_separation_gate": false,
"total_correct_candidates": 0,
"total_incorrect_candidates": 0,
"overlap_ratio": 0.0,
"case_count": 9,
"skipped_count": 9,
"geometry_supports_static_threshold": false
},
"per_threshold": {
"-1.0": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"-0.5": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"0.0": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"0.1": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"0.25": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"0.5": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"1.0": {
"TP": 0,
"FN": 0,
"FP": 0,
"TN": 0,
"TP_rate": 0.0,
"FP_rate": 0.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
}
},
"score_distributions": {
"correct": {
"count": 0
},
"incorrect": {
"count": 0
}
},
"case_details": [
{
"id": "FSC-PUB-001",
"skipped": true
},
{
"id": "FSC-DEV-001",
"skipped": true
},
{
"id": "FSC-DEV-002",
"skipped": true
},
{
"id": "FSC-DEV-003",
"skipped": true
},
{
"id": "FSC-DEV-004",
"skipped": true
},
{
"id": "FSC-DEV-005",
"skipped": true
},
{
"id": "FSC-DEV-006",
"skipped": true
},
{
"id": "FSC-DEV-007",
"skipped": true
},
{
"id": "FSC-DEV-008",
"skipped": true
}
]
}

175
evals/forward_semantic_control/results/phase4_characterization_v2.json Normal file
View file

@ -0,0 +1,175 @@
{
"metrics": {
"thresholds_swept": [
-1.0,
-0.5,
0.0,
0.1,
0.25,
0.5,
1.0
],
"best_threshold": 1.0,
"best_separation_quality": 0.6,
"separation_quality_gate": 0.8,
"passes_separation_gate": false,
"total_correct_candidates": 5,
"total_incorrect_candidates": 5,
"overlap_ratio": 0.0803,
"case_count": 5,
"skipped_count": 0,
"geometry_supports_static_threshold": false
},
"per_threshold": {
"-1.0": {
"TP": 5,
"FN": 0,
"FP": 5,
"TN": 0,
"TP_rate": 1.0,
"FP_rate": 1.0,
"FN_rate": 0.0,
"TN_rate": 0.0,
"separation_quality": 0.0
},
"-0.5": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.0": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.1": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.25": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"0.5": {
"TP": 5,
"FN": 0,
"FP": 4,
"TN": 1,
"TP_rate": 1.0,
"FP_rate": 0.8,
"FN_rate": 0.0,
"TN_rate": 0.2,
"separation_quality": 0.2
},
"1.0": {
"TP": 5,
"FN": 0,
"FP": 2,
"TN": 3,
"TP_rate": 1.0,
"FP_rate": 0.4,
"FN_rate": 0.0,
"TN_rate": 0.6,
"separation_quality": 0.6
}
},
"score_distributions": {
"correct": {
"count": 5,
"mean": 7.0824,
"median": 5.7389,
"min": 1.1731,
"max": 14.3621,
"stdev": 6.1954
},
"incorrect": {
"count": 5,
"mean": 0.9955,
"median": 0.8238,
"min": -0.551,
"max": 2.3711,
"stdev": 1.0929
},
"overlap": {
"correct_min": 1.1731,
"incorrect_max": 2.3711,
"overlap_size": 1.198,
"full_range": 14.9131,
"overlap_ratio": 0.0803,
"separable_by_static_threshold": false
}
},
"case_details": [
{
"id": "FSC-PUB-V2-001",
"correct_scores": [
1.4205
],
"incorrect_scores": [
0.8238
]
},
{
"id": "FSC-PUB-V2-002",
"correct_scores": [
1.1731
],
"incorrect_scores": [
0.7167
]
},
{
"id": "FSC-PUB-V2-003",
"correct_scores": [
12.7176
],
"incorrect_scores": [
-0.551
]
},
{
"id": "FSC-PUB-V2-004",
"correct_scores": [
5.7389
],
"incorrect_scores": [
2.3711
]
},
{
"id": "FSC-PUB-V2-005",
"correct_scores": [
14.3621
],
"incorrect_scores": [
1.6169
]
}
]
}

65
evals/forward_semantic_control/results/phase4_summary.json Normal file
View file

@ -0,0 +1,65 @@
{
"v1_plus_dev": {
"thresholds_swept": [
-1.0,
-0.5,
0.0,
0.1,
0.25,
0.5,
1.0
],
"best_threshold": -1.0,
"best_separation_quality": 0.0,
"separation_quality_gate": 0.8,
"passes_separation_gate": false,
"total_correct_candidates": 0,
"total_incorrect_candidates": 0,
"overlap_ratio": 0.0,
"case_count": 9,
"skipped_count": 9,
"geometry_supports_static_threshold": false
},
"v2": {
"thresholds_swept": [
-1.0,
-0.5,
0.0,
0.1,
0.25,
0.5,
1.0
],
"best_threshold": 1.0,
"best_separation_quality": 0.6,
"separation_quality_gate": 0.8,
"passes_separation_gate": false,
"total_correct_candidates": 5,
"total_incorrect_candidates": 5,
"overlap_ratio": 0.0803,
"case_count": 5,
"skipped_count": 0,
"geometry_supports_static_threshold": false
},
"combined": {
"thresholds_swept": [
-1.0,
-0.5,
0.0,
0.1,
0.25,
0.5,
1.0
],
"best_threshold": 1.0,
"best_separation_quality": 0.6,
"separation_quality_gate": 0.8,
"passes_separation_gate": false,
"total_correct_candidates": 5,
"total_incorrect_candidates": 5,
"overlap_ratio": 0.0803,
"case_count": 14,
"skipped_count": 9,
"geometry_supports_static_threshold": false
}
}

301
evals/forward_semantic_control/threshold_characterization.py Normal file
View file

@ -0,0 +1,301 @@
"""Phase 4 threshold characterization — ADR-0024 diagnostic, not tuning.
The Phase 2 report on the existing FSC v1 corpus surfaced
``exhaustion_rate=0.33 at t=0.0`` and ``exhaustion_rate=0.56 at
t=0.25`` — well above the 5% ceiling. Before proposing a learned or
adaptive threshold, we need to know *whether the geometry permits a
clean threshold at all*.
This module produces a distribution-map diagnostic, NOT a tuned
threshold:
* For each case in v1+dev, build the same region the inner-loop
runner builds (chain outer-product over chain_tokens).
* For every candidate index in the admissible set, compute its
``cga_inner`` score against the relation_blade.
* Group scores by whether the candidate is "correct" (== the
expected_endpoint) or "incorrect" (anything else admissible).
* Sweep thresholds [-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0] and
report, per threshold:
admitted_correct / total_correct (TP rate)
admitted_incorrect / total_incorrect (FP rate)
rejected_correct / total_correct (FN rate)
rejected_incorrect / total_incorrect (TN rate)
separation_quality = TP_rate - FP_rate
* Also report admitted-vs-rejected score *distribution* maps:
admitted_score_mean / median / min / max per correctness class
rejected_score_mean / median / min / max per correctness class
score_overlap_ratio = (max(correct_rejected_min, incorrect_admitted_min)
- min(correct_admitted_max, incorrect_rejected_max))
normalized
* The headline finding is whether ANY threshold delivers
``separation_quality >= 0.8`` on the corpus. If not, the
relation_blade construction is geometrically under-resolved for
static thresholding regardless of value.
This is a Phase 4 diagnostic that informs whether ADR-0025 should
even attempt static thresholds or move directly to relation-typed
or frame-derived schemes.
"""
from __future__ import annotations
import json
import statistics
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
import numpy as np
from algebra.cga import cga_inner, outer_product
from chat.runtime import ChatRuntime
THRESHOLDS = (-1.0, -0.5, 0.0, 0.1, 0.25, 0.5, 1.0)
SEPARATION_QUALITY_GATE = 0.8
@dataclass(slots=True)
class CharacterizationReport:
metrics: dict[str, Any] = field(default_factory=dict)
per_threshold: dict[float, dict[str, Any]] = field(default_factory=dict)
score_distributions: dict[str, dict[str, Any]] = field(default_factory=dict)
case_details: list[dict[str, Any]] = field(default_factory=list)
def _build_blade(vocab, chain_tokens: tuple[str, ...]) -> tuple[np.ndarray | None, list[int]]:
indices: list[int] = []
versors: list[np.ndarray] = []
for raw in chain_tokens:
token = raw.lower().strip()
if not token:
continue
try:
idx = vocab.index_of(token)
versor = np.asarray(vocab.get_versor(token), dtype=np.float32)
except (KeyError, AttributeError):
continue
indices.append(int(idx))
versors.append(versor)
if not versors:
return None, []
blade = versors[0]
for nxt in versors[1:]:
blade = outer_product(blade, nxt)
return blade, indices
def _score_candidates(
vocab,
blade: np.ndarray,
indices: list[int],
expected_token: str,
) -> tuple[list[float], list[float]]:
"""Return (correct_scores, incorrect_scores) for candidates in admissible set."""
correct: list[float] = []
incorrect: list[float] = []
expected_idx: int | None = None
try:
expected_idx = int(vocab.index_of(expected_token.lower().strip()))
except (KeyError, AttributeError, ValueError):
expected_idx = None
for idx in indices:
v = np.asarray(vocab.get_versor_at(idx), dtype=np.float32)
score = float(cga_inner(v, blade))
if expected_idx is not None and idx == expected_idx:
correct.append(score)
else:
incorrect.append(score)
return correct, incorrect
def _summarize(scores: list[float]) -> dict[str, Any]:
if not scores:
return {"count": 0}
return {
"count": len(scores),
"mean": round(statistics.mean(scores), 4),
"median": round(statistics.median(scores), 4),
"min": round(min(scores), 4),
"max": round(max(scores), 4),
"stdev": round(statistics.stdev(scores), 4) if len(scores) > 1 else 0.0,
}
def _blade_and_indices_for_case(
vocab, case: dict[str, Any]
) -> tuple[np.ndarray | None, list[int], str]:
"""Build the blade + admissible indices for either schema.
Returns ``(blade, indices, expected_token)`` or ``(None, [], "")``
if the case cannot be grounded in the active vocab.
"""
expected = case.get("expected_endpoint", "")
# v2 schema: explicit admissible_tokens + relation_blade_token.
if "admissible_tokens" in case and "relation_blade_token" in case:
try:
blade = np.asarray(
vocab.get_versor(case["relation_blade_token"]), dtype=np.float32
)
indices = [int(vocab.index_of(tok)) for tok in case["admissible_tokens"]]
except (KeyError, AttributeError, ValueError):
return None, [], ""
return blade, indices, expected
# v1 schema: chain_tokens outer-product, or single-token fallback.
chain_tokens = tuple(case.get("chain_tokens", ()))
if not chain_tokens and expected:
chain_tokens = (expected,)
blade, indices = _build_blade(vocab, chain_tokens)
return blade, indices, expected
def characterize(cases: list[dict[str, Any]]) -> CharacterizationReport:
runtime = ChatRuntime()
vocab = runtime.session.vocab
case_details: list[dict[str, Any]] = []
all_correct: list[float] = []
all_incorrect: list[float] = []
for case in cases:
blade, indices, expected = _blade_and_indices_for_case(vocab, case)
if blade is None or not indices:
case_details.append({"id": case.get("id", ""), "skipped": True})
continue
correct, incorrect = _score_candidates(vocab, blade, indices, expected)
all_correct.extend(correct)
all_incorrect.extend(incorrect)
case_details.append({
"id": case.get("id", ""),
"correct_scores": [round(s, 4) for s in correct],
"incorrect_scores": [round(s, 4) for s in incorrect],
})
per_threshold: dict[float, dict[str, Any]] = {}
for thr in THRESHOLDS:
tp = sum(1 for s in all_correct if s >= thr)
fn = sum(1 for s in all_correct if s < thr)
fp = sum(1 for s in all_incorrect if s >= thr)
tn = sum(1 for s in all_incorrect if s < thr)
tot_c = max(len(all_correct), 1)
tot_i = max(len(all_incorrect), 1)
tp_rate = tp / tot_c
fp_rate = fp / tot_i
per_threshold[thr] = {
"TP": tp, "FN": fn, "FP": fp, "TN": tn,
"TP_rate": round(tp_rate, 4),
"FP_rate": round(fp_rate, 4),
"FN_rate": round(fn / tot_c, 4),
"TN_rate": round(tn / tot_i, 4),
"separation_quality": round(tp_rate - fp_rate, 4),
}
score_distributions = {
"correct": _summarize(all_correct),
"incorrect": _summarize(all_incorrect),
}
# Overlap diagnostic: is there *any* gap between the worst correct
# and the best incorrect?
overlap_ratio = 0.0
if all_correct and all_incorrect:
cmin = min(all_correct)
imax = max(all_incorrect)
full_range = max(all_correct + all_incorrect) - min(all_correct + all_incorrect)
overlap = max(imax - cmin, 0.0)
overlap_ratio = round(overlap / full_range, 4) if full_range > 0 else 0.0
score_distributions["overlap"] = {
"correct_min": round(cmin, 4),
"incorrect_max": round(imax, 4),
"overlap_size": round(overlap, 4),
"full_range": round(full_range, 4),
"overlap_ratio": overlap_ratio,
"separable_by_static_threshold": cmin > imax,
}
best_thr = max(
per_threshold.items(),
key=lambda kv: kv[1]["separation_quality"],
)
metrics = {
"thresholds_swept": list(THRESHOLDS),
"best_threshold": best_thr[0],
"best_separation_quality": best_thr[1]["separation_quality"],
"separation_quality_gate": SEPARATION_QUALITY_GATE,
"passes_separation_gate": best_thr[1]["separation_quality"] >= SEPARATION_QUALITY_GATE,
"total_correct_candidates": len(all_correct),
"total_incorrect_candidates": len(all_incorrect),
"overlap_ratio": overlap_ratio,
"case_count": len(cases),
"skipped_count": sum(1 for d in case_details if d.get("skipped")),
# Headline finding: can a STATIC threshold separate correct from
# incorrect on this corpus? If no, ADR-0025 must not propose
# static thresholds.
"geometry_supports_static_threshold": (
best_thr[1]["separation_quality"] >= SEPARATION_QUALITY_GATE
),
}
return CharacterizationReport(
metrics=metrics,
per_threshold=per_threshold,
score_distributions=score_distributions,
case_details=case_details,
)
def _load(path: Path) -> list[dict[str, Any]]:
if not path.exists():
return []
with path.open() as fh:
return [json.loads(line) for line in fh if line.strip()]
def _serialize(report: CharacterizationReport) -> dict[str, Any]:
return {
"metrics": report.metrics,
"per_threshold": {str(k): v for k, v in report.per_threshold.items()},
"score_distributions": report.score_distributions,
"case_details": report.case_details,
}
def main() -> None:
v1 = _load(Path("evals/forward_semantic_control/public/v1/cases.jsonl"))
dev = _load(Path("evals/forward_semantic_control/dev/cases.jsonl"))
v2 = _load(Path("evals/forward_semantic_control/public/v2/cases.jsonl"))
out_dir = Path("evals/forward_semantic_control/results")
out_dir.mkdir(parents=True, exist_ok=True)
bundles = [
("v1_plus_dev", v1 + dev),
("v2", v2),
("combined", v1 + dev + v2),
]
summary: dict[str, dict[str, Any]] = {}
for label, cases in bundles:
if not cases:
summary[label] = {"empty": True}
continue
report = characterize(cases)
out_path = out_dir / f"phase4_characterization_{label}.json"
with out_path.open("w") as fh:
json.dump(_serialize(report), fh, indent=2)
summary[label] = report.metrics
print(f"\n=== {label} ===")
print(f" cases: {report.metrics['case_count']}, "
f"skipped: {report.metrics['skipped_count']}")
print(f" best_threshold: {report.metrics['best_threshold']}")
print(f" best_separation_quality: {report.metrics['best_separation_quality']}")
print(f" geometry_supports_static_threshold: "
f"{report.metrics['geometry_supports_static_threshold']}")
print(f" overlap_ratio: {report.metrics['overlap_ratio']}")
with (out_dir / "phase4_summary.json").open("w") as fh:
json.dump(summary, fh, indent=2)
print(f"\nWrote summary: {out_dir / 'phase4_summary.json'}")
if __name__ == "__main__":
main()

211
evals/forward_semantic_control/v2_runner.py Normal file
View file

@ -0,0 +1,211 @@
"""Phase 3 mechanism-isolation runner — ADR-0024 v2 adversarial cases.
Synthetic cases where boundary-only is *expected* to select a forbidden
decoy and inner-loop is *expected* to reject it and select the correct
endpoint. The case schema specifies its own region (admissible token
set + relation blade token) so the geometric setup is fully controlled
and reproducible.
A case passes iff *all* of the following hold under the same field
state:
boundary-only:
selected == forbidden_token
verdict.admitted is False (the rejection is visible in trace)
inner-loop (admissibility_threshold from the case):
selected == expected_endpoint
verdict.admitted is True
forbidden_token appears in step.rejected_attempts
Each case's seed_token sets the initial FieldState.F. No priming —
the geometric configuration is given. This is mechanism isolation,
not corpus observation; pair with ``inner_loop_runner.py`` (Phase 2)
for the corpus side.
Reports per case + aggregate proof_rate and rejection_causally_traced
counts. Conforms to the ``run_lane`` interface.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any
import numpy as np
from chat.runtime import ChatRuntime
from core.config import RuntimeConfig
from field.state import FieldState
from generate.admissibility import AdmissibilityRegion, RegionSource
from generate.result import GenerationResult
from generate.stream import generate as generate_walk
@dataclass(slots=True)
class V2Report:
metrics: dict[str, Any] = field(default_factory=dict)
case_details: list[dict[str, Any]] = field(default_factory=list)
def _field_state_from_seed(vocab, seed_token: str) -> FieldState:
idx = vocab.index_of(seed_token)
versor = np.asarray(vocab.get_versor(seed_token), dtype=np.float32)
return FieldState(F=versor.copy(), node=idx, step=0)
def _region_from_case(vocab, case: dict[str, Any]) -> AdmissibilityRegion:
indices = [int(vocab.index_of(tok)) for tok in case["admissible_tokens"]]
blade = np.asarray(
vocab.get_versor(case["relation_blade_token"]), dtype=np.float32
)
return AdmissibilityRegion(
allowed_indices=np.asarray(indices, dtype=np.int64),
relation_blade=blade,
source=RegionSource.RELATION,
label=f"v2[{case.get('id', '')}]",
)
def _run_case(case: dict[str, Any]) -> dict[str, Any]:
runtime = ChatRuntime()
vocab = runtime.session.vocab
persona = runtime.session.persona
try:
seed_state = _field_state_from_seed(vocab, case["seed_token"])
region = _region_from_case(vocab, case)
except (KeyError, ValueError) as exc:
return {"id": case.get("id", ""), "skipped": True, "reason": str(exc)}
threshold = float(case["admissibility_threshold"])
expected = case["expected_endpoint"]
forbidden = case["forbidden_token"]
# Boundary-only leg.
boundary: GenerationResult = generate_walk(
seed_state, vocab, persona,
max_tokens=1, region=region,
inner_loop_admissibility=False,
admissibility_threshold=threshold,
)
b_step = boundary.admissibility_trace[0]
boundary_selected = b_step.selected_word
boundary_admitted = b_step.verdict.admitted
# Boundary expectation: selects the forbidden decoy and verdict is
# NOT admitted (the rejection is visible in trace but the walk
# still emits it — this is ADR-0023 boundary-only behavior).
boundary_picks_forbidden = boundary_selected == forbidden
boundary_verdict_rejects = not boundary_admitted
# Inner-loop leg.
inner: GenerationResult | None = None
inner_exhaust_reason = ""
try:
inner = generate_walk(
seed_state, vocab, persona,
max_tokens=1, region=region,
inner_loop_admissibility=True,
admissibility_threshold=threshold,
)
except ValueError as exc:
inner_exhaust_reason = str(exc)
if inner is None:
return {
"id": case.get("id", ""),
"skipped": False,
"passed": False,
"boundary_selected": boundary_selected,
"boundary_picks_forbidden": boundary_picks_forbidden,
"boundary_verdict_rejects": boundary_verdict_rejects,
"inner_selected": None,
"inner_admitted": None,
"inner_exhausted": True,
"inner_exhaust_reason": inner_exhaust_reason,
"rejection_in_trace": False,
"rejected_attempts": (),
"rationale": case.get("rationale", ""),
}
i_step = inner.admissibility_trace[0]
inner_selected = i_step.selected_word
inner_admitted = i_step.verdict.admitted
rejected_words = tuple(word for (_idx, word, _score) in i_step.rejected_attempts)
rejection_in_trace = forbidden in rejected_words
passed = (
boundary_picks_forbidden
and boundary_verdict_rejects
and inner_selected == expected
and inner_admitted
and rejection_in_trace
)
return {
"id": case.get("id", ""),
"skipped": False,
"passed": passed,
"semantic_pair": case.get("semantic_pair", ""),
"expected_endpoint": expected,
"forbidden_token": forbidden,
"boundary_selected": boundary_selected,
"boundary_picks_forbidden": boundary_picks_forbidden,
"boundary_verdict_rejects": boundary_verdict_rejects,
"inner_selected": inner_selected,
"inner_admitted": inner_admitted,
"inner_exhausted": False,
"rejection_in_trace": rejection_in_trace,
"rejected_attempts": [
[int(idx), str(word), float(score)]
for (idx, word, score) in i_step.rejected_attempts
],
"rationale": case.get("rationale", ""),
}
def run_lane(
cases: list[dict[str, Any]],
*,
config: RuntimeConfig | None = None,
workers: int | None = None,
) -> V2Report:
_ = config
_ = workers # serial — v2 corpus is small
if not cases:
return V2Report(metrics={}, case_details=[])
details = [_run_case(c) for c in cases]
n = len(details)
skipped = sum(1 for d in details if d.get("skipped"))
eligible = [d for d in details if not d.get("skipped")]
passed = sum(1 for d in eligible if d.get("passed"))
boundary_picks_forbidden_count = sum(
1 for d in eligible if d.get("boundary_picks_forbidden")
)
rejection_in_trace_count = sum(
1 for d in eligible if d.get("rejection_in_trace")
)
pass_rate = passed / max(len(eligible), 1)
boundary_decoy_rate = boundary_picks_forbidden_count / max(len(eligible), 1)
rejection_traced_rate = rejection_in_trace_count / max(len(eligible), 1)
metrics: dict[str, Any] = {
"case_count": n,
"skipped_count": skipped,
"eligible_count": len(eligible),
"pass_count": passed,
"pass_rate": round(pass_rate, 4),
"boundary_decoy_rate": round(boundary_decoy_rate, 4),
"rejection_traced_rate": round(rejection_traced_rate, 4),
# Headline: do we have causal evidence that inner-loop rejection
# is responsible for the selection difference?
"mechanism_isolated": (
pass_rate == 1.0
and boundary_decoy_rate == 1.0
and rejection_traced_rate == 1.0
),
}
return V2Report(metrics=metrics, case_details=details)

18
generate/stream.py
View file

@ -279,6 +279,7 @@ def generate(
region: AdmissibilityRegion | None = None,
inner_loop_admissibility: bool = False,
admissibility_threshold: float = 0.0,
inner_loop_force_admit: bool = False,
) -> GenerationResult:
"""Generate a token sequence.
@ -299,6 +300,15 @@ def generate(
byte-identical. The rotor ``V`` is only constructed for the
admitted candidate, so the ``versor_condition < 1e-6`` invariant
is unaffected.
``inner_loop_force_admit`` (Phase 2 null control) — only meaningful
when ``inner_loop_admissibility=True``. Exercises the inner-loop
code path (same attempt-loop scaffolding, same telemetry side
effects) but force-breaks on the first candidate regardless of
verdict. This isolates rejection as the causal factor: any
delta between boundary-only and inner-loop-on runs that vanishes
under the null control is attributable to code-path differences,
not to rejection. Not exposed to RuntimeConfig — eval-only.
"""
tokens = []
trajectory = [] if record_trajectory else None
@ -394,7 +404,13 @@ def generate(
region_label=effective_region_label,
reason="unconstrained",
)
if not inner_loop_active or verdict.admitted:
if not inner_loop_active or verdict.admitted or inner_loop_force_admit:
# `inner_loop_force_admit` is the Phase 2 null control:
# exercises the inner-loop code path (same attempt loop,
# same telemetry side effects) but force-breaks on the
# first candidate so any pass-rate delta vs the true
# inner-loop run is causally attributable to rejection,
# not to incidental code-path differences.
break
# Inner loop is on and verdict rejected this candidate.
rejected_attempts.append((int(word_idx), str(word), float(verdict.score)))

34
tests/test_inner_loop_admissibility.py
View file

@ -328,5 +328,39 @@ class TestInnerLoopDeterminism:
assert h_off == h_on
class TestInnerLoopNullControl:
"""Phase 2 null control — exercises the inner-loop code path but
force-admits every candidate. Used by the FSC corpus runner to
isolate rejection as the causal factor in any pass-rate delta.
"""
def test_force_admit_selects_first_preferred_candidate_no_rejections(self) -> None:
# Without null control, this case rejects alpha and selects beta.
# With null control, the inner-loop path is exercised but the
# first candidate (alpha) is force-admitted — same outcome as
# boundary-only.
vocab = _ControllableVocab(
words=["seed", "alpha", "beta"],
preference=[1, 2],
versor_signs=[+1.0, -1.0, +1.0],
)
result = generate(
_initial_state(vocab),
vocab,
_IdentityPersona(),
max_tokens=1,
region=_positive_blade_region((1, 2)),
inner_loop_admissibility=True,
inner_loop_force_admit=True,
)
# Force-admit selects alpha (preferred) even though verdict is
# rejected — the breakout happens regardless.
assert result.tokens == ("alpha",)
step = result.admissibility_trace[0]
assert step.selected_word == "alpha"
# No rejections accumulated — first attempt breaks out.
assert step.rejected_attempts == ()
if __name__ == "__main__": # pragma: no cover
pytest.main([__file__, "-v"])

114
tests/test_inner_loop_phase2.py Normal file
View file

@ -0,0 +1,114 @@
"""Phase 2 corpus-observation invariants (ADR-0024 follow-up).
These tests pin the causal-attribution and determinism contracts that
the Phase 2 runner must hold on the existing FSC corpus. They are
intentionally *not* gated on rejection_effect or exhaustion_rate —
those are findings to be characterised in Phase 4, not invariants.
What we *do* assert:
* ``causal_attribution_valid`` is True: the null control (inner-loop
code path on, force-admit on) matches boundary-only exactly. Any
pass-rate delta between inner_loop_t0 and boundary_only is then
attributable to rejection, not to incidental code-path effects.
* ``code_path_residual`` is zero (within float tolerance).
* Trace-hash stability holds for the inner-loop condition on every
non-skipped case (5 reruns produce identical hashes).
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.forward_semantic_control.inner_loop_runner import run_lane
_CORPUS_PATHS = (
Path("evals/forward_semantic_control/public/v1/cases.jsonl"),
Path("evals/forward_semantic_control/dev/cases.jsonl"),
)
def _load_corpus() -> list[dict]:
cases: list[dict] = []
for path in _CORPUS_PATHS:
if not path.exists():
continue
with path.open() as fh:
cases.extend(json.loads(line) for line in fh if line.strip())
return cases
@pytest.fixture(scope="module")
def phase2_report():
cases = _load_corpus()
if not cases:
pytest.skip("FSC corpus not available")
return run_lane(cases)
class TestCausalAttribution:
def test_null_control_matches_boundary_only(self, phase2_report) -> None:
"""Null control must reproduce boundary-only pass-rate exactly.
If this fails, the inner-loop code path is itself altering
selection (call ordering, telemetry side effects), and any
rejection_effect we measure is contaminated. ADR-0024 proof
depends on this invariant.
"""
assert phase2_report.metrics["causal_attribution_valid"] is True
assert phase2_report.metrics["code_path_residual"] == 0.0
def test_null_control_per_condition_metrics(self, phase2_report) -> None:
per = phase2_report.metrics["per_condition"]
assert per["null_control"]["pass_rate"] == per["boundary_only"]["pass_rate"]
# Null control must produce zero rejections by construction.
assert per["null_control"]["mean_rejection_count_per_turn"] == 0
assert per["null_control"]["non_empty_rejected_attempts_rate"] == 0.0
assert per["null_control"]["exhaustion_rate"] == 0.0
class TestInnerLoopDeterminismOnCorpus:
def test_inner_loop_t0_hash_stable_on_every_case(self, phase2_report) -> None:
"""Live-corpus version of the Phase 1 acceptance test.
Stub-vocab determinism is necessary but not sufficient — the
same property must hold on actual packs, actual field state,
actual rejection sequences. 5 reruns per case must hash
identically.
"""
rate = phase2_report.metrics["per_condition"]["inner_loop_t0"][
"trace_hash_stability_pass_rate"
]
assert rate == 1.0
class TestPhase2RecordsFindings:
"""These are not gates — they record the Phase 2 finding so a
future change that silently flips the sign of rejection_effect or
closes the exhaustion gap is visible in test output."""
def test_runner_emits_required_metric_keys(self, phase2_report) -> None:
required = {
"per_condition",
"rejection_effect",
"code_path_residual",
"causal_attribution_valid",
"exhaustion_ceiling",
"exhaustion_gate_pass",
"probe_threshold_positive",
"case_count",
"skipped_count",
}
assert required <= set(phase2_report.metrics.keys())
def test_all_four_conditions_present(self, phase2_report) -> None:
per = phase2_report.metrics["per_condition"]
assert set(per.keys()) == {
"boundary_only",
"null_control",
"inner_loop_t0",
"inner_loop_tpos",
}

75
tests/test_inner_loop_phase3.py Normal file
View file

@ -0,0 +1,75 @@
"""Phase 3 mechanism-isolation invariants (ADR-0024 v2 corpus).
These tests are the *load-bearing* proof contract: in synthetic
cases designed to exercise the rejection mechanism, the inner loop
must (a) actually reject the forbidden decoy, (b) select the
expected endpoint instead, and (c) leave a causal trail in
``rejected_attempts``.
Pass criteria are stricter than Phase 2 (which is observational):
Phase 3 *gates* on ``mechanism_isolated``.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.forward_semantic_control.v2_runner import run_lane
V2_CORPUS = Path("evals/forward_semantic_control/public/v2/cases.jsonl")
@pytest.fixture(scope="module")
def v2_report():
if not V2_CORPUS.exists():
pytest.skip("V2 corpus not available")
with V2_CORPUS.open() as fh:
cases = [json.loads(line) for line in fh if line.strip()]
if not cases:
pytest.skip("V2 corpus is empty")
return run_lane(cases)
class TestMechanismIsolated:
def test_mechanism_isolated_overall(self, v2_report) -> None:
"""The headline gate — every v2 case must isolate the mechanism."""
assert v2_report.metrics["mechanism_isolated"] is True
def test_pass_rate_is_one(self, v2_report) -> None:
assert v2_report.metrics["pass_rate"] == 1.0
def test_boundary_picks_decoy_every_case(self, v2_report) -> None:
"""If boundary doesn't pick the decoy on a v2 case, the case
is mis-constructed — the mechanism never gets exercised."""
assert v2_report.metrics["boundary_decoy_rate"] == 1.0
def test_rejection_causally_traced_every_case(self, v2_report) -> None:
"""The forbidden token must appear in rejected_attempts on
every case — this is the visible causal evidence."""
assert v2_report.metrics["rejection_traced_rate"] == 1.0
class TestPerCaseInvariants:
def test_no_case_was_skipped(self, v2_report) -> None:
assert v2_report.metrics["skipped_count"] == 0
def test_every_case_passed(self, v2_report) -> None:
for detail in v2_report.case_details:
assert detail.get("passed") is True, (
f"Case {detail.get('id')} failed: "
f"boundary={detail.get('boundary_selected')} "
f"inner={detail.get('inner_selected')} "
f"forbidden_traced={detail.get('rejection_in_trace')} "
f"inner_exhausted={detail.get('inner_exhausted')}"
)
def test_inner_selection_matches_expected_endpoint(self, v2_report) -> None:
for detail in v2_report.case_details:
assert detail.get("inner_selected") == detail.get("expected_endpoint")
def test_boundary_selection_matches_forbidden_token(self, v2_report) -> None:
for detail in v2_report.case_details:
assert detail.get("boundary_selected") == detail.get("forbidden_token")

107
tests/test_inner_loop_phase4.py Normal file
View file

@ -0,0 +1,107 @@
"""Phase 4 threshold characterization invariants (ADR-0024 follow-up).
These tests are diagnostic, not gates. They pin the finding so a
future change that silently improves (or breaks) the geometric
separability is visible in test output.
Findings recorded:
* Per-case the relation_blade DOES separate correct from incorrect
candidates (all five v2 cases pass mechanism-isolation), so the
blade construction is not geometrically blind.
* But globally NO STATIC threshold delivers separation_quality ≥ 0.8.
Blade norms vary across cases (~10x range), so the same threshold
value means different things case-to-case.
* The v1 chain-token outer-product blade is ungrounded in the active
pack — all 9 cases are skipped because chain_tokens (alpha, beta,
gamma, delta) are not in the en_core_cognition vocab. This is its
own load-bearing finding for ADR-0025: chain-token blades are
unsuitable as the default region construction.
ADR-0025 design implication: static thresholds (global, relation-typed,
or frame-derived) are insufficient. Per-case normalized thresholds
(e.g. fraction of blade self-score) are the next thing to investigate.
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from evals.forward_semantic_control.threshold_characterization import characterize
V1 = Path("evals/forward_semantic_control/public/v1/cases.jsonl")
V2 = Path("evals/forward_semantic_control/public/v2/cases.jsonl")
DEV = Path("evals/forward_semantic_control/dev/cases.jsonl")
def _load(path: Path) -> list[dict]:
if not path.exists():
return []
with path.open() as fh:
return [json.loads(line) for line in fh if line.strip()]
@pytest.fixture(scope="module")
def v1_report():
cases = _load(V1) + _load(DEV)
if not cases:
pytest.skip("v1/dev corpus not available")
return characterize(cases)
@pytest.fixture(scope="module")
def v2_report():
cases = _load(V2)
if not cases:
pytest.skip("v2 corpus not available")
return characterize(cases)
class TestV1ChainBladeUngrounded:
"""V1 chain_tokens are synthetic (alpha, beta, gamma, delta) and
not present in the active pack. The characterization should
surface this by skipping every case.
"""
def test_all_v1_cases_skipped(self, v1_report) -> None:
assert v1_report.metrics["skipped_count"] == v1_report.metrics["case_count"]
def test_v1_reports_no_separation(self, v1_report) -> None:
# No candidates ⇒ best_separation_quality stays at zero.
assert v1_report.metrics["best_separation_quality"] == 0.0
class TestV2PerCaseSeparates:
"""Per-case, every v2 case has correct_min > incorrect_max."""
def test_every_v2_case_separates_locally(self, v2_report) -> None:
for detail in v2_report.case_details:
if detail.get("skipped"):
continue
correct = detail["correct_scores"]
incorrect = detail["incorrect_scores"]
assert correct, f"case {detail.get('id')} has no correct candidate"
assert min(correct) > max(incorrect), (
f"case {detail.get('id')} fails local separation: "
f"correct_min={min(correct)} ≤ incorrect_max={max(incorrect)}"
)
class TestV2GlobalNonSeparability:
"""Despite per-case separability, no static threshold works
globally — this is the load-bearing finding for ADR-0025."""
def test_no_static_threshold_passes_gate(self, v2_report) -> None:
# If a future change makes this pass, ADR-0025 design may
# need revision. Currently expected: False.
assert v2_report.metrics["geometry_supports_static_threshold"] is False
def test_score_distributions_overlap_globally(self, v2_report) -> None:
overlap = v2_report.score_distributions["overlap"]
# incorrect_max > correct_min ⇒ static threshold cannot
# separate. This is the geometric fact ADR-0025 must address.
assert overlap["separable_by_static_threshold"] is False
assert overlap["overlap_size"] > 0.0