diff --git a/docs/decisions/ADR-0024-inner-loop-admissibility.md b/docs/decisions/ADR-0024-inner-loop-admissibility.md index b1a445bc..ae46bd58 100644 --- a/docs/decisions/ADR-0024-inner-loop-admissibility.md +++ b/docs/decisions/ADR-0024-inner-loop-admissibility.md @@ -182,3 +182,111 @@ user-controlled text enters the trace path. Set `inner_loop_admissibility=False` (the default) at every call site. The trace hash remains byte-identical to ADR-0023, so deterministic replay over the existing corpus is unaffected. + +--- + +## Addendum — Phase 1 v1/dev fixture retirement (2026-05-17) + +### Finding + +Phase 4's characterization recorded that v1 chain-token outer-product +blades skipped 9/9 cases because `chain_tokens` (`alpha`, `beta`, +`gamma`, `delta`, `mu`, `nu`, `omicron`, `pi`, `rho`, `sigma`, `tau`, +`upsilon`, `phi`, `chi`, `psi`, `omega`, `iota`, `lambda`, `eta`, +`theta`, `zeta`, `xenon`, `ytterbium`, `kappa`) were synthetic and +ungrounded in `en_core_cognition_v1`. The Phase 2 corpus-observation +runner inherited the same fixtures, surfacing `exhaustion_rate = 0.33` +at `t = 0` on v1+dev — above the 5 % benign-corpus ceiling. + +Original assessment (pre-Phase-1): "v1 chain construction can produce +ungrounded regions; fix the cause, not the symptom." + +### What Phase 1 did + +The 1 v1 case and 8 dev cases were rewritten with pack-grounded tokens +drawn from `en_core_cognition_v1`: + +* `tone → evidence → memory → wisdom` (causes chain, replacing + `alpha → beta → gamma → delta`) +* `voice → memory → wisdom` (means chain, replacing + `mu → nu → omicron`) +* `question → answer → understanding → wisdom` (precedes chain, + replacing `pi → rho → sigma → tau`) +* `word → discourse → narrative` (part_of chain, replacing + `upsilon → phi → chi`) +* `symbol → word → meaning` with `image`/`light` distractors + (adversarial branching, replacing `eta/theta/zeta` plus + `beta/rho` means-distractors) + +All token-level grammar adjustments (e.g. *"register cause memory"* → +*"register causes memory"*) were corrected as part of the rewrite. + +### What changed about the finding + +| Metric | Pre-rewrite | Post-rewrite | +| --------------------------------------- | ----------- | ------------ | +| v1+dev cases skipped (region builds) | 9 / 9 | 0 / 9 | +| `causal_attribution_valid` | True | True | +| `code_path_residual` | 0.0 | 0.0 | +| `inner_loop_t0` hash stability | 1.0 | 1.0 | +| `best_separation_quality` (Phase 4) | 0.0 | 0.056 | +| `geometry_supports_static_threshold` | False | False | +| `inner_loop_t0` exhaustion (Phase 2) | 0.33 | 0.67 | +| `inner_loop_tpos` exhaustion (Phase 2) | (skipped) | 1.00 | + +The original Phase 4 finding ("v1 chain-token blades unsuitable as the +default region construction") was caused by two layered issues: + +1. **Surface fixture rot** — synthetic tokens prevented region + construction at all. This is now fixed. +2. **Deeper architectural mismatch** — v1's case schema + (`prime` + `chain_tokens`) probes *teaching-driven walk* through a + relation injected at runtime (ADR-0022 / ADR-0023's mechanism). + The inner-loop admissibility lane (ADR-0024) tests + *blade-geometric region constraint*. These are distinct + mechanisms; the same case schema cannot meaningfully exercise both. + +Exhaustion *increased* post-rewrite because the chain-blade region is +now constructible *and* actively rejecting the boundary's off-chain +picks. That is honest behavior of the inner-loop on cases it was not +designed to evaluate. + +### Architectural conclusion + +* **v1 / dev cases belong to the ADR-0022 / ADR-0023 boundary-walk + lane (`evals/forward_semantic_control/runner.py`).** The rewrite + improves that lane (real semantic substrate for teaching priming) and + does not affect this ADR. +* **v2 cases belong to the ADR-0024 inner-loop admissibility lane + (`evals/forward_semantic_control/v2_runner.py`).** v2's schema + (`seed_token` + `admissible_tokens` + `relation_blade_token`) is the + correct fixture shape for blade-admissibility evaluation. +* **The Phase 2 corpus-observation runner's reuse of v1+dev cases was + a categorical error.** The runner remains useful as a corpus-style + observation harness; it now needs a proper benign inner-loop corpus, + authored with v2's schema. Authoring that corpus is **Phase 5's + job** (stratified mechanism-isolation families subsume benign-corpus + observation). + +### What this ADR addendum does *not* claim + +* The exhaustion finding is not "closed" — it is reattributed. v1/dev + inner-loop exhaustion at 67 % is an honest consequence of running the + wrong mechanism on the wrong fixture, not a defect in the inner-loop + implementation. +* The Phase 2 runner's exhaustion gate (`EXHAUSTION_CEILING = 0.05`) + remains valid as a contract; it cannot be satisfied on the current + fixture mix. Phase 5 will produce the benign-corpus fixtures the + gate was designed against. +* Phase 4's "no static threshold passes" finding stands unchanged. + +### Tests pinning the new state + +* `tests/test_inner_loop_phase4.py::TestV1ChainBladePostGrounding` — + asserts `skipped_count == 0` (replacing the pre-rewrite assertion + `skipped_count == case_count`) and + `best_separation_quality < 0.5` (replacing `== 0.0`). +* `tests/test_inner_loop_phase2.py` (unchanged) — continues to assert + `causal_attribution_valid` and inner-loop hash stability. No + exhaustion gate assertion; exhaustion remains a recorded finding, + not an invariant. diff --git a/evals/forward_semantic_control/dev/cases.jsonl b/evals/forward_semantic_control/dev/cases.jsonl index 5c942c7f..5e238052 100644 --- a/evals/forward_semantic_control/dev/cases.jsonl +++ b/evals/forward_semantic_control/dev/cases.jsonl @@ -1,8 +1,8 @@ -{"id":"FSC-DEV-001","kind":"chain_three_hop","prime":["What does alpha cause?","Actually alpha causes beta.","What does beta cause?","Actually beta causes gamma.","What does gamma cause?","Actually gamma causes delta."],"prompt":"What does alpha cause?","expected_endpoint":"delta","chain_tokens":["alpha","beta","gamma","delta"],"baseline_must_fail":true} -{"id":"FSC-DEV-002","kind":"negative_control_no_chain","prime":["What does alpha cause?","Actually alpha causes beta.","What does xenon cause?","Actually xenon causes ytterbium."],"prompt":"What does alpha cause?","expected_endpoint":"beta","chain_tokens":["alpha","beta"],"baseline_must_fail":false} -{"id":"FSC-DEV-003","kind":"frame_constraint_blocks_wrong_relation","prime":["What does alpha cause?","Actually alpha causes beta.","What does alpha mean?","Actually alpha means kappa."],"prompt":"What does alpha cause?","expected_endpoint":"beta","forbidden_token":"kappa","chain_tokens":["alpha","beta"],"baseline_must_fail":false} -{"id":"FSC-DEV-004","kind":"chain_two_hop_means","prime":["What does mu mean?","Actually mu means nu.","What does nu mean?","Actually nu means omicron."],"prompt":"What does mu mean?","expected_endpoint":"omicron","chain_tokens":["mu","nu","omicron"],"baseline_must_fail":true} -{"id":"FSC-DEV-005","kind":"chain_three_hop_precedes","prime":["What does pi precede?","Actually pi precedes rho.","What does rho precede?","Actually rho precedes sigma.","What does sigma precede?","Actually sigma precedes tau."],"prompt":"What does pi precede?","expected_endpoint":"tau","chain_tokens":["pi","rho","sigma","tau"],"baseline_must_fail":true} -{"id":"FSC-DEV-006","kind":"chain_two_hop_part_of","prime":["What is upsilon part of?","Actually upsilon is part of phi.","What is phi part of?","Actually phi is part of chi."],"prompt":"What is upsilon part of?","expected_endpoint":"chi","chain_tokens":["upsilon","phi","chi"],"baseline_must_fail":true} -{"id":"FSC-DEV-007","kind":"adversarial_distractor_means_vs_cause","prime":["What does psi cause?","Actually psi causes omega.","What does psi mean?","Actually psi means iota.","What does psi precede?","Actually psi precedes lambda."],"prompt":"What does psi cause?","expected_endpoint":"omega","forbidden_token":"iota","chain_tokens":["psi","omega"],"baseline_must_fail":false} -{"id":"FSC-DEV-008","kind":"adversarial_distractor_chain_branching","prime":["What does eta cause?","Actually eta causes theta.","What does theta cause?","Actually theta causes zeta.","What does eta mean?","Actually eta means beta.","What does theta mean?","Actually theta means rho."],"prompt":"What does eta cause?","expected_endpoint":"zeta","forbidden_token":"rho","chain_tokens":["eta","theta","zeta"],"baseline_must_fail":true} +{"id":"FSC-DEV-001","kind":"chain_three_hop","prime":["What does tone cause?","Actually tone causes evidence.","What does evidence cause?","Actually evidence causes memory.","What does memory cause?","Actually memory causes wisdom."],"prompt":"What does tone cause?","expected_endpoint":"wisdom","chain_tokens":["tone","evidence","memory","wisdom"],"baseline_must_fail":true,"grounding_note":"All chain tokens grounded in en_core_cognition_v1; relations are taught via priming, not pack-derived."} +{"id":"FSC-DEV-002","kind":"negative_control_no_chain","prime":["What does tone cause?","Actually tone causes evidence.","What does register cause?","Actually register causes memory."],"prompt":"What does tone cause?","expected_endpoint":"evidence","chain_tokens":["tone","evidence"],"baseline_must_fail":false,"grounding_note":"Second priming chain (register/memory) is unrelated to the queried chain (tone/evidence); both branches grounded."} +{"id":"FSC-DEV-003","kind":"frame_constraint_blocks_wrong_relation","prime":["What does tone cause?","Actually tone causes evidence.","What does tone mean?","Actually tone means voice."],"prompt":"What does tone cause?","expected_endpoint":"evidence","forbidden_token":"voice","chain_tokens":["tone","evidence"],"baseline_must_fail":false,"grounding_note":"Same source (tone) carries two relations (causes/means); admissibility must restrict to the causes-relation branch."} +{"id":"FSC-DEV-004","kind":"chain_two_hop_means","prime":["What does voice mean?","Actually voice means memory.","What does memory mean?","Actually memory means wisdom."],"prompt":"What does voice mean?","expected_endpoint":"wisdom","chain_tokens":["voice","memory","wisdom"],"baseline_must_fail":true,"grounding_note":"Means-relation chain over three grounded tokens; tested relations are taught, not pack-asserted."} +{"id":"FSC-DEV-005","kind":"chain_three_hop_precedes","prime":["What does question precede?","Actually question precedes answer.","What does answer precede?","Actually answer precedes understanding.","What does understanding precede?","Actually understanding precedes wisdom."],"prompt":"What does question precede?","expected_endpoint":"wisdom","chain_tokens":["question","answer","understanding","wisdom"],"baseline_must_fail":true,"grounding_note":"Precedes-relation chain; tokens grounded though the chain itself is taught."} +{"id":"FSC-DEV-006","kind":"chain_two_hop_part_of","prime":["What is word part of?","Actually word is part of discourse.","What is discourse part of?","Actually discourse is part of narrative."],"prompt":"What is word part of?","expected_endpoint":"narrative","chain_tokens":["word","discourse","narrative"],"baseline_must_fail":true,"grounding_note":"belongs_to/part_of relation chain over three grounded tokens."} +{"id":"FSC-DEV-007","kind":"adversarial_distractor_means_vs_cause","prime":["What does symbol cause?","Actually symbol causes word.","What does symbol mean?","Actually symbol means image.","What does symbol precede?","Actually symbol precedes truth."],"prompt":"What does symbol cause?","expected_endpoint":"word","forbidden_token":"image","chain_tokens":["symbol","word"],"baseline_must_fail":false,"grounding_note":"Three taught relations on the same source; admissibility must select the causes-target and reject the means-target."} +{"id":"FSC-DEV-008","kind":"adversarial_distractor_chain_branching","prime":["What does symbol cause?","Actually symbol causes word.","What does word cause?","Actually word causes meaning.","What does symbol mean?","Actually symbol means image.","What does word mean?","Actually word means light."],"prompt":"What does symbol cause?","expected_endpoint":"meaning","forbidden_token":"light","chain_tokens":["symbol","word","meaning"],"baseline_must_fail":true,"grounding_note":"Branching adversarial: causes-chain (symbol→word→meaning) competes with means-distractors (symbol→image, word→light)."} diff --git a/evals/forward_semantic_control/public/v1/cases.jsonl b/evals/forward_semantic_control/public/v1/cases.jsonl index a3bbd037..e930d5a5 100644 --- a/evals/forward_semantic_control/public/v1/cases.jsonl +++ b/evals/forward_semantic_control/public/v1/cases.jsonl @@ -1 +1 @@ -{"id":"FSC-PUB-001","kind":"chain_three_hop","prime":["What does alpha cause?","Actually alpha causes beta.","What does beta cause?","Actually beta causes gamma.","What does gamma cause?","Actually gamma causes delta."],"prompt":"What does alpha cause?","expected_endpoint":"delta","chain_tokens":["alpha","beta","gamma","delta"],"baseline_must_fail":true} +{"id":"FSC-PUB-001","kind":"chain_three_hop","prime":["What does tone cause?","Actually tone causes evidence.","What does evidence cause?","Actually evidence causes memory.","What does memory cause?","Actually memory causes wisdom."],"prompt":"What does tone cause?","expected_endpoint":"wisdom","chain_tokens":["tone","evidence","memory","wisdom"],"baseline_must_fail":true,"grounding_note":"All chain tokens grounded in en_core_cognition_v1; relations are taught via priming, not pack-derived."} diff --git a/tests/test_inner_loop_phase4.py b/tests/test_inner_loop_phase4.py index 3ff82069..bf0faa69 100644 --- a/tests/test_inner_loop_phase4.py +++ b/tests/test_inner_loop_phase4.py @@ -12,11 +12,16 @@ Findings recorded: * But globally NO STATIC threshold delivers separation_quality ≥ 0.8. Blade norms vary across cases (~10x range), so the same threshold value means different things case-to-case. - * The v1 chain-token outer-product blade is ungrounded in the active - pack — all 9 cases are skipped because chain_tokens (alpha, beta, - gamma, delta) are not in the en_core_cognition vocab. This is its - own load-bearing finding for ADR-0025: chain-token blades are - unsuitable as the default region construction. + * Post-Phase-1 (ADR-0024 addendum): v1/dev cases were rewritten with + pack-grounded tokens (e.g. tone/evidence/memory/wisdom), so the + chain-token outer-product blade now constructs successfully (0/9 + skipped vs the pre-rewrite 9/9). But the chain-blade geometry on + v1/dev still does NOT separate cleanly (best_separation_quality + ≈ 0.06), reinforcing the deeper Phase 4 finding: v1/dev chain + blades probe teaching-driven walk (ADR-0022/0023), not the + inner-loop's blade-admissibility mechanism. v1/dev belong to the + boundary-walk lane (runner.py); v2's seed_token + relation_blade_token + schema is the proper inner-loop fixture. ADR-0025 design implication: static thresholds (global, relation-typed, or frame-derived) are insufficient. Per-case normalized thresholds @@ -60,18 +65,26 @@ def v2_report(): return characterize(cases) -class TestV1ChainBladeUngrounded: - """V1 chain_tokens are synthetic (alpha, beta, gamma, delta) and - not present in the active pack. The characterization should - surface this by skipping every case. +class TestV1ChainBladePostGrounding: + """Post-Phase-1: v1/dev chain_tokens were rewritten with pack-grounded + tokens (ADR-0024 addendum). Region construction now succeeds — but + the chain-blade geometry remains a poor fit for the inner-loop lane. + These tests pin the new finding: v1/dev is constructible but probes + a different mechanism than v2. """ - def test_all_v1_cases_skipped(self, v1_report) -> None: - assert v1_report.metrics["skipped_count"] == v1_report.metrics["case_count"] + def test_no_v1_cases_skipped_after_grounding(self, v1_report) -> None: + # Phase 1 retired synthetic chain tokens; every case now grounds. + assert v1_report.metrics["skipped_count"] == 0 - def test_v1_reports_no_separation(self, v1_report) -> None: - # No candidates ⇒ best_separation_quality stays at zero. - assert v1_report.metrics["best_separation_quality"] == 0.0 + def test_v1_chain_blade_geometry_remains_unsuitable(self, v1_report) -> None: + # Constructible but not separable: chain-blade outer-product + # geometry produces near-zero separation_quality on v1/dev, + # confirming the architectural finding that v1/dev belong to + # the boundary-walk lane, not the inner-loop lane. If a future + # change pushes this above 0.5, ADR-0024's lane-assignment + # decision may need revisiting. + assert v1_report.metrics["best_separation_quality"] < 0.5 class TestV2PerCaseSeparates: