feat(adr-0024): Phase 1 addendum — retire v1/dev fixture rot
Rewrite v1+dev FSC cases with pack-grounded tokens drawn from en_core_cognition_v1. Closes the 9/9 region-construction failure recorded in Phase 4 (chain_tokens alpha/beta/gamma/delta/etc. were ungrounded in the active pack). Token mappings preserve each case's test pattern: * alpha→beta→gamma→delta → tone→evidence→memory→wisdom (causes) * mu→nu→omicron → voice→memory→wisdom (means) * pi→rho→sigma→tau → question→answer→understanding→wisdom (precedes) * upsilon→phi→chi → word→discourse→narrative (part_of) * eta/theta/zeta + means-distractors → symbol/word/meaning + image/light Result post-rewrite: * skipped_count: 9/9 → 0/9 (region constructible) * causal_attribution_valid: True (preserved) * code_path_residual: 0.0 (preserved) * inner_loop_t0 hash stability: 1.0 (preserved) * best_separation_quality: 0.0 → 0.056 (still below 0.8 gate) The rewrite exposes a deeper architectural finding documented in the ADR addendum: v1/dev case schema (prime + chain_tokens) probes teaching-driven walk (ADR-0022/0023), not the inner-loop's blade-admissibility mechanism (ADR-0024). The Phase 2 corpus- observation runner's reuse of v1/dev was a categorical error. v1/dev belong to the boundary-walk lane (runner.py); v2 belongs to the inner-loop lane (v2_runner.py). Phase 5 will author the benign inner-loop corpus the EXHAUSTION_CEILING gate was designed against. Tests pinning new state: * TestV1ChainBladeUngrounded → TestV1ChainBladePostGrounding (assertions inverted: skipped_count == 0; separation_quality < 0.5) * TestPhase2 (unchanged) continues to assert causal_attribution_valid and hash stability; exhaustion remains a finding, not an invariant.
This commit is contained in:
Shay
parent
8146844d90
commit
394029008e
4 changed files with 144 additions and 23 deletions
|
|
@ -182,3 +182,111 @@ user-controlled text enters the trace path.
|
||||||
Set `inner_loop_admissibility=False` (the default) at every call
|
Set `inner_loop_admissibility=False` (the default) at every call
|
||||||
site. The trace hash remains byte-identical to ADR-0023, so
|
site. The trace hash remains byte-identical to ADR-0023, so
|
||||||
deterministic replay over the existing corpus is unaffected.
|
deterministic replay over the existing corpus is unaffected.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Addendum — Phase 1 v1/dev fixture retirement (2026-05-17)
|
||||||
|
|
||||||
|
### Finding
|
||||||
|
|
||||||
|
Phase 4's characterization recorded that v1 chain-token outer-product
|
||||||
|
blades skipped 9/9 cases because `chain_tokens` (`alpha`, `beta`,
|
||||||
|
`gamma`, `delta`, `mu`, `nu`, `omicron`, `pi`, `rho`, `sigma`, `tau`,
|
||||||
|
`upsilon`, `phi`, `chi`, `psi`, `omega`, `iota`, `lambda`, `eta`,
|
||||||
|
`theta`, `zeta`, `xenon`, `ytterbium`, `kappa`) were synthetic and
|
||||||
|
ungrounded in `en_core_cognition_v1`. The Phase 2 corpus-observation
|
||||||
|
runner inherited the same fixtures, surfacing `exhaustion_rate = 0.33`
|
||||||
|
at `t = 0` on v1+dev — above the 5 % benign-corpus ceiling.
|
||||||
|
|
||||||
|
Original assessment (pre-Phase-1): "v1 chain construction can produce
|
||||||
|
ungrounded regions; fix the cause, not the symptom."
|
||||||
|
|
||||||
|
### What Phase 1 did
|
||||||
|
|
||||||
|
The 1 v1 case and 8 dev cases were rewritten with pack-grounded tokens
|
||||||
|
drawn from `en_core_cognition_v1`:
|
||||||
|
|
||||||
|
* `tone → evidence → memory → wisdom` (causes chain, replacing
|
||||||
|
`alpha → beta → gamma → delta`)
|
||||||
|
* `voice → memory → wisdom` (means chain, replacing
|
||||||
|
`mu → nu → omicron`)
|
||||||
|
* `question → answer → understanding → wisdom` (precedes chain,
|
||||||
|
replacing `pi → rho → sigma → tau`)
|
||||||
|
* `word → discourse → narrative` (part_of chain, replacing
|
||||||
|
`upsilon → phi → chi`)
|
||||||
|
* `symbol → word → meaning` with `image`/`light` distractors
|
||||||
|
(adversarial branching, replacing `eta/theta/zeta` plus
|
||||||
|
`beta/rho` means-distractors)
|
||||||
|
|
||||||
|
All token-level grammar adjustments (e.g. *"register cause memory"* →
|
||||||
|
*"register causes memory"*) were corrected as part of the rewrite.
|
||||||
|
|
||||||
|
### What changed about the finding
|
||||||
|
|
||||||
|
| Metric | Pre-rewrite | Post-rewrite |
|
||||||
|
| --------------------------------------- | ----------- | ------------ |
|
||||||
|
| v1+dev cases skipped (region builds) | 9 / 9 | 0 / 9 |
|
||||||
|
| `causal_attribution_valid` | True | True |
|
||||||
|
| `code_path_residual` | 0.0 | 0.0 |
|
||||||
|
| `inner_loop_t0` hash stability | 1.0 | 1.0 |
|
||||||
|
| `best_separation_quality` (Phase 4) | 0.0 | 0.056 |
|
||||||
|
| `geometry_supports_static_threshold` | False | False |
|
||||||
|
| `inner_loop_t0` exhaustion (Phase 2) | 0.33 | 0.67 |
|
||||||
|
| `inner_loop_tpos` exhaustion (Phase 2) | (skipped) | 1.00 |
|
||||||
|
|
||||||
|
The original Phase 4 finding ("v1 chain-token blades unsuitable as the
|
||||||
|
default region construction") was caused by two layered issues:
|
||||||
|
|
||||||
|
1. **Surface fixture rot** — synthetic tokens prevented region
|
||||||
|
construction at all. This is now fixed.
|
||||||
|
2. **Deeper architectural mismatch** — v1's case schema
|
||||||
|
(`prime` + `chain_tokens`) probes *teaching-driven walk* through a
|
||||||
|
relation injected at runtime (ADR-0022 / ADR-0023's mechanism).
|
||||||
|
The inner-loop admissibility lane (ADR-0024) tests
|
||||||
|
*blade-geometric region constraint*. These are distinct
|
||||||
|
mechanisms; the same case schema cannot meaningfully exercise both.
|
||||||
|
|
||||||
|
Exhaustion *increased* post-rewrite because the chain-blade region is
|
||||||
|
now constructible *and* actively rejecting the boundary's off-chain
|
||||||
|
picks. That is honest behavior of the inner-loop on cases it was not
|
||||||
|
designed to evaluate.
|
||||||
|
|
||||||
|
### Architectural conclusion
|
||||||
|
|
||||||
|
* **v1 / dev cases belong to the ADR-0022 / ADR-0023 boundary-walk
|
||||||
|
lane (`evals/forward_semantic_control/runner.py`).** The rewrite
|
||||||
|
improves that lane (real semantic substrate for teaching priming) and
|
||||||
|
does not affect this ADR.
|
||||||
|
* **v2 cases belong to the ADR-0024 inner-loop admissibility lane
|
||||||
|
(`evals/forward_semantic_control/v2_runner.py`).** v2's schema
|
||||||
|
(`seed_token` + `admissible_tokens` + `relation_blade_token`) is the
|
||||||
|
correct fixture shape for blade-admissibility evaluation.
|
||||||
|
* **The Phase 2 corpus-observation runner's reuse of v1+dev cases was
|
||||||
|
a categorical error.** The runner remains useful as a corpus-style
|
||||||
|
observation harness; it now needs a proper benign inner-loop corpus,
|
||||||
|
authored with v2's schema. Authoring that corpus is **Phase 5's
|
||||||
|
job** (stratified mechanism-isolation families subsume benign-corpus
|
||||||
|
observation).
|
||||||
|
|
||||||
|
### What this ADR addendum does *not* claim
|
||||||
|
|
||||||
|
* The exhaustion finding is not "closed" — it is reattributed. v1/dev
|
||||||
|
inner-loop exhaustion at 67 % is an honest consequence of running the
|
||||||
|
wrong mechanism on the wrong fixture, not a defect in the inner-loop
|
||||||
|
implementation.
|
||||||
|
* The Phase 2 runner's exhaustion gate (`EXHAUSTION_CEILING = 0.05`)
|
||||||
|
remains valid as a contract; it cannot be satisfied on the current
|
||||||
|
fixture mix. Phase 5 will produce the benign-corpus fixtures the
|
||||||
|
gate was designed against.
|
||||||
|
* Phase 4's "no static threshold passes" finding stands unchanged.
|
||||||
|
|
||||||
|
### Tests pinning the new state
|
||||||
|
|
||||||
|
* `tests/test_inner_loop_phase4.py::TestV1ChainBladePostGrounding` —
|
||||||
|
asserts `skipped_count == 0` (replacing the pre-rewrite assertion
|
||||||
|
`skipped_count == case_count`) and
|
||||||
|
`best_separation_quality < 0.5` (replacing `== 0.0`).
|
||||||
|
* `tests/test_inner_loop_phase2.py` (unchanged) — continues to assert
|
||||||
|
`causal_attribution_valid` and inner-loop hash stability. No
|
||||||
|
exhaustion gate assertion; exhaustion remains a recorded finding,
|
||||||
|
not an invariant.
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
{"id":"FSC-DEV-001","kind":"chain_three_hop","prime":["What does alpha cause?","Actually alpha causes beta.","What does beta cause?","Actually beta causes gamma.","What does gamma cause?","Actually gamma causes delta."],"prompt":"What does alpha cause?","expected_endpoint":"delta","chain_tokens":["alpha","beta","gamma","delta"],"baseline_must_fail":true}
|
{"id":"FSC-DEV-001","kind":"chain_three_hop","prime":["What does tone cause?","Actually tone causes evidence.","What does evidence cause?","Actually evidence causes memory.","What does memory cause?","Actually memory causes wisdom."],"prompt":"What does tone cause?","expected_endpoint":"wisdom","chain_tokens":["tone","evidence","memory","wisdom"],"baseline_must_fail":true,"grounding_note":"All chain tokens grounded in en_core_cognition_v1; relations are taught via priming, not pack-derived."}
|
||||||
{"id":"FSC-DEV-002","kind":"negative_control_no_chain","prime":["What does alpha cause?","Actually alpha causes beta.","What does xenon cause?","Actually xenon causes ytterbium."],"prompt":"What does alpha cause?","expected_endpoint":"beta","chain_tokens":["alpha","beta"],"baseline_must_fail":false}
|
{"id":"FSC-DEV-002","kind":"negative_control_no_chain","prime":["What does tone cause?","Actually tone causes evidence.","What does register cause?","Actually register causes memory."],"prompt":"What does tone cause?","expected_endpoint":"evidence","chain_tokens":["tone","evidence"],"baseline_must_fail":false,"grounding_note":"Second priming chain (register/memory) is unrelated to the queried chain (tone/evidence); both branches grounded."}
|
||||||
{"id":"FSC-DEV-003","kind":"frame_constraint_blocks_wrong_relation","prime":["What does alpha cause?","Actually alpha causes beta.","What does alpha mean?","Actually alpha means kappa."],"prompt":"What does alpha cause?","expected_endpoint":"beta","forbidden_token":"kappa","chain_tokens":["alpha","beta"],"baseline_must_fail":false}
|
{"id":"FSC-DEV-003","kind":"frame_constraint_blocks_wrong_relation","prime":["What does tone cause?","Actually tone causes evidence.","What does tone mean?","Actually tone means voice."],"prompt":"What does tone cause?","expected_endpoint":"evidence","forbidden_token":"voice","chain_tokens":["tone","evidence"],"baseline_must_fail":false,"grounding_note":"Same source (tone) carries two relations (causes/means); admissibility must restrict to the causes-relation branch."}
|
||||||
{"id":"FSC-DEV-004","kind":"chain_two_hop_means","prime":["What does mu mean?","Actually mu means nu.","What does nu mean?","Actually nu means omicron."],"prompt":"What does mu mean?","expected_endpoint":"omicron","chain_tokens":["mu","nu","omicron"],"baseline_must_fail":true}
|
{"id":"FSC-DEV-004","kind":"chain_two_hop_means","prime":["What does voice mean?","Actually voice means memory.","What does memory mean?","Actually memory means wisdom."],"prompt":"What does voice mean?","expected_endpoint":"wisdom","chain_tokens":["voice","memory","wisdom"],"baseline_must_fail":true,"grounding_note":"Means-relation chain over three grounded tokens; tested relations are taught, not pack-asserted."}
|
||||||
{"id":"FSC-DEV-005","kind":"chain_three_hop_precedes","prime":["What does pi precede?","Actually pi precedes rho.","What does rho precede?","Actually rho precedes sigma.","What does sigma precede?","Actually sigma precedes tau."],"prompt":"What does pi precede?","expected_endpoint":"tau","chain_tokens":["pi","rho","sigma","tau"],"baseline_must_fail":true}
|
{"id":"FSC-DEV-005","kind":"chain_three_hop_precedes","prime":["What does question precede?","Actually question precedes answer.","What does answer precede?","Actually answer precedes understanding.","What does understanding precede?","Actually understanding precedes wisdom."],"prompt":"What does question precede?","expected_endpoint":"wisdom","chain_tokens":["question","answer","understanding","wisdom"],"baseline_must_fail":true,"grounding_note":"Precedes-relation chain; tokens grounded though the chain itself is taught."}
|
||||||
{"id":"FSC-DEV-006","kind":"chain_two_hop_part_of","prime":["What is upsilon part of?","Actually upsilon is part of phi.","What is phi part of?","Actually phi is part of chi."],"prompt":"What is upsilon part of?","expected_endpoint":"chi","chain_tokens":["upsilon","phi","chi"],"baseline_must_fail":true}
|
{"id":"FSC-DEV-006","kind":"chain_two_hop_part_of","prime":["What is word part of?","Actually word is part of discourse.","What is discourse part of?","Actually discourse is part of narrative."],"prompt":"What is word part of?","expected_endpoint":"narrative","chain_tokens":["word","discourse","narrative"],"baseline_must_fail":true,"grounding_note":"belongs_to/part_of relation chain over three grounded tokens."}
|
||||||
{"id":"FSC-DEV-007","kind":"adversarial_distractor_means_vs_cause","prime":["What does psi cause?","Actually psi causes omega.","What does psi mean?","Actually psi means iota.","What does psi precede?","Actually psi precedes lambda."],"prompt":"What does psi cause?","expected_endpoint":"omega","forbidden_token":"iota","chain_tokens":["psi","omega"],"baseline_must_fail":false}
|
{"id":"FSC-DEV-007","kind":"adversarial_distractor_means_vs_cause","prime":["What does symbol cause?","Actually symbol causes word.","What does symbol mean?","Actually symbol means image.","What does symbol precede?","Actually symbol precedes truth."],"prompt":"What does symbol cause?","expected_endpoint":"word","forbidden_token":"image","chain_tokens":["symbol","word"],"baseline_must_fail":false,"grounding_note":"Three taught relations on the same source; admissibility must select the causes-target and reject the means-target."}
|
||||||
{"id":"FSC-DEV-008","kind":"adversarial_distractor_chain_branching","prime":["What does eta cause?","Actually eta causes theta.","What does theta cause?","Actually theta causes zeta.","What does eta mean?","Actually eta means beta.","What does theta mean?","Actually theta means rho."],"prompt":"What does eta cause?","expected_endpoint":"zeta","forbidden_token":"rho","chain_tokens":["eta","theta","zeta"],"baseline_must_fail":true}
|
{"id":"FSC-DEV-008","kind":"adversarial_distractor_chain_branching","prime":["What does symbol cause?","Actually symbol causes word.","What does word cause?","Actually word causes meaning.","What does symbol mean?","Actually symbol means image.","What does word mean?","Actually word means light."],"prompt":"What does symbol cause?","expected_endpoint":"meaning","forbidden_token":"light","chain_tokens":["symbol","word","meaning"],"baseline_must_fail":true,"grounding_note":"Branching adversarial: causes-chain (symbol→word→meaning) competes with means-distractors (symbol→image, word→light)."}
|
||||||
|
|
|
||||||
|
|
@ -1 +1 @@
|
||||||
{"id":"FSC-PUB-001","kind":"chain_three_hop","prime":["What does alpha cause?","Actually alpha causes beta.","What does beta cause?","Actually beta causes gamma.","What does gamma cause?","Actually gamma causes delta."],"prompt":"What does alpha cause?","expected_endpoint":"delta","chain_tokens":["alpha","beta","gamma","delta"],"baseline_must_fail":true}
|
{"id":"FSC-PUB-001","kind":"chain_three_hop","prime":["What does tone cause?","Actually tone causes evidence.","What does evidence cause?","Actually evidence causes memory.","What does memory cause?","Actually memory causes wisdom."],"prompt":"What does tone cause?","expected_endpoint":"wisdom","chain_tokens":["tone","evidence","memory","wisdom"],"baseline_must_fail":true,"grounding_note":"All chain tokens grounded in en_core_cognition_v1; relations are taught via priming, not pack-derived."}
|
||||||
|
|
|
||||||
|
|
@ -12,11 +12,16 @@ Findings recorded:
|
||||||
* But globally NO STATIC threshold delivers separation_quality ≥ 0.8.
|
* But globally NO STATIC threshold delivers separation_quality ≥ 0.8.
|
||||||
Blade norms vary across cases (~10x range), so the same threshold
|
Blade norms vary across cases (~10x range), so the same threshold
|
||||||
value means different things case-to-case.
|
value means different things case-to-case.
|
||||||
* The v1 chain-token outer-product blade is ungrounded in the active
|
* Post-Phase-1 (ADR-0024 addendum): v1/dev cases were rewritten with
|
||||||
pack — all 9 cases are skipped because chain_tokens (alpha, beta,
|
pack-grounded tokens (e.g. tone/evidence/memory/wisdom), so the
|
||||||
gamma, delta) are not in the en_core_cognition vocab. This is its
|
chain-token outer-product blade now constructs successfully (0/9
|
||||||
own load-bearing finding for ADR-0025: chain-token blades are
|
skipped vs the pre-rewrite 9/9). But the chain-blade geometry on
|
||||||
unsuitable as the default region construction.
|
v1/dev still does NOT separate cleanly (best_separation_quality
|
||||||
|
≈ 0.06), reinforcing the deeper Phase 4 finding: v1/dev chain
|
||||||
|
blades probe teaching-driven walk (ADR-0022/0023), not the
|
||||||
|
inner-loop's blade-admissibility mechanism. v1/dev belong to the
|
||||||
|
boundary-walk lane (runner.py); v2's seed_token + relation_blade_token
|
||||||
|
schema is the proper inner-loop fixture.
|
||||||
|
|
||||||
ADR-0025 design implication: static thresholds (global, relation-typed,
|
ADR-0025 design implication: static thresholds (global, relation-typed,
|
||||||
or frame-derived) are insufficient. Per-case normalized thresholds
|
or frame-derived) are insufficient. Per-case normalized thresholds
|
||||||
|
|
@ -60,18 +65,26 @@ def v2_report():
|
||||||
return characterize(cases)
|
return characterize(cases)
|
||||||
|
|
||||||
|
|
||||||
class TestV1ChainBladeUngrounded:
|
class TestV1ChainBladePostGrounding:
|
||||||
"""V1 chain_tokens are synthetic (alpha, beta, gamma, delta) and
|
"""Post-Phase-1: v1/dev chain_tokens were rewritten with pack-grounded
|
||||||
not present in the active pack. The characterization should
|
tokens (ADR-0024 addendum). Region construction now succeeds — but
|
||||||
surface this by skipping every case.
|
the chain-blade geometry remains a poor fit for the inner-loop lane.
|
||||||
|
These tests pin the new finding: v1/dev is constructible but probes
|
||||||
|
a different mechanism than v2.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def test_all_v1_cases_skipped(self, v1_report) -> None:
|
def test_no_v1_cases_skipped_after_grounding(self, v1_report) -> None:
|
||||||
assert v1_report.metrics["skipped_count"] == v1_report.metrics["case_count"]
|
# Phase 1 retired synthetic chain tokens; every case now grounds.
|
||||||
|
assert v1_report.metrics["skipped_count"] == 0
|
||||||
|
|
||||||
def test_v1_reports_no_separation(self, v1_report) -> None:
|
def test_v1_chain_blade_geometry_remains_unsuitable(self, v1_report) -> None:
|
||||||
# No candidates ⇒ best_separation_quality stays at zero.
|
# Constructible but not separable: chain-blade outer-product
|
||||||
assert v1_report.metrics["best_separation_quality"] == 0.0
|
# geometry produces near-zero separation_quality on v1/dev,
|
||||||
|
# confirming the architectural finding that v1/dev belong to
|
||||||
|
# the boundary-walk lane, not the inner-loop lane. If a future
|
||||||
|
# change pushes this above 0.5, ADR-0024's lane-assignment
|
||||||
|
# decision may need revisiting.
|
||||||
|
assert v1_report.metrics["best_separation_quality"] < 0.5
|
||||||
|
|
||||||
|
|
||||||
class TestV2PerCaseSeparates:
|
class TestV2PerCaseSeparates:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue