Compare commits

...

2 commits

Author SHA1 Message Date
Shay
e15b5d0b8d audit(L9): epistemic state + verdicts - PARTIAL 2026-05-24 19:30:03 -07:00
Shay
c4f7a0430d docs(audit): ratchet v3 — add W-015 from L6 finding
W-015: session/context.py:207-246 post-generation unitize is
test-covered but not ADR-documented as an allowed normalization
boundary. Surfaced by L6 audit (#246) answering L1's forward note
(#237).

Per CLAUDE.md normalization rules, sanctioned unitize sites are
ingest/gate.py, language_packs/compiler.py, and algebra/versor.py.
The session/context.py site is not in that list — either an
undocumented allowed boundary or a discipline violation.

Recommended resolution path: investigate root cause first; if
unitize is masking an upstream construction violation, fix upstream.
If it's a legitimate boundary, write ADR sanctioning it. If it's
pure drift repair, refactor to remove (per CLAUDE.md "do not add
drift repair").

Updated dependency graph and suggested-sequence list to include
W-015 as a discipline-question entry.

Progress note updated: L0-L7 audited (7 of 9 layers); L8-L9 pending.
2026-05-24 19:18:02 -07:00
2 changed files with 174 additions and 3 deletions

View file

@ -1,4 +1,4 @@
# Substrate Liveness Ratchet — v2 (partial; informed by L0-L5 + L10 scope)
# Substrate Liveness Ratchet — v3 (partial; informed by L0-L7 + L10 scope)
**Scope:** [substrate-liveness-audit-scope](../decisions/substrate-liveness-audit-scope.md) (v2)
**Companion:** [substrate-liveness-registry](./substrate-liveness-registry.md)
@ -278,6 +278,37 @@ home (ADR-XXXX or new).
consumer (evals); the question is whether it should be promoted to
runtime use.
### W-015 — `session/context.py` post-generation unitize undocumented
- **Surfaced by:** L6 audit (PR #246), answering L1 audit's forward
note (PR #237).
- **Gap:** `session/context.py:207-246` performs final-turn hemisphere
correction and anchor pull with `unitize_versor()`. The site is
test-covered, but no ADR documents it as an allowed normalization
boundary. Per CLAUDE.md normalization rules, the only sanctioned
unitize sites are `ingest/gate.py`, `language_packs/compiler.py`,
and `algebra/versor.py`. The `session/context.py` site does not
appear in that list — it is either an undocumented allowed boundary
or a discipline violation.
- **Dependency:** none — purely a documentation/discipline question.
- **Resolution paths:**
- **(a)** Write a small ADR amendment or new ADR sanctioning
`session/context.py:207-246` as an allowed normalization boundary
(the "final-turn anchor pull" boundary), with rationale.
- **(b)** If the site is NOT a sanctioned boundary, refactor to
remove the unitize call and surface any resulting closure
failures rather than silently repairing them. Per CLAUDE.md:
*"Do not add drift repair... whose only purpose is to repair
another function."*
- **(c)** Investigate whether the unitize is masking an upstream
construction violation. If so, fix upstream and remove the
site.
- **Recommended:** start with (c) — investigate root cause. (a) is the
fallback if the unitize turns out to be a legitimate boundary.
(b) is the fallback if it's pure drift repair.
- **Status:** ⏳ OPEN — discipline question; small ADR or small
refactor depending on investigation outcome.
---
## Dependency graph (Mermaid-style, ASCII)
@ -296,6 +327,7 @@ W-012 ⏳ ──── (independent, mechanical) — sibling of W-011
W-010 ⏳ ──── (operator decision: intentional or wire L3 vocab)
W-013 ⏳ ──── (operator decision: wire, relocate, or delete)
W-014 ⏳ ──── (operator decision: lighter than W-013)
W-015 ⏳ ──── (discipline question: investigate root cause first)
W-008 (L10 ADR) ⏳
├──→ W-003 (VaultPromotionPolicy wiring) ⏳
@ -334,6 +366,9 @@ the bigger L10 unit.**
6. **W-010 — L4 recognition vocabulary: token-level intentional, or
wire L3 vocab.** Operator decision; affects whether recognition
pulls in pack-resident domain types.
7. **W-015 — `session/context.py` unitize: ADR-sanction, remove, or
fix-upstream.** Investigate root cause first; resolution shape
depends on what's found.
### Then user-observable second-order changes
@ -368,7 +403,7 @@ demonstrates the audit-to-fix loop actually closes.
[[project-engine-identity-candidate]]. Trigger to un-shelve: L10
runtime-model ADR commits to cross-reboot identity verification as
a sub-question 3 requirement.
- **L6-L9 wiring debt.** L0-L5 audited (5 of 9 layers); L6-L9 pending.
- **L8-L9 wiring debt.** L0-L7 audited (7 of 9 layers); L8-L9 pending.
Ratchet will be revised as remaining entries land.
- **Drop-off sibling ADR for recognizers.** Named in recognizer-
storage-scope v2; depends on W-008 + recognizer-storage ADR + W-009.

View file

@ -17,7 +17,8 @@
| L6 — Chat runtime + surface composition | ✅ Audited | **PARTIAL** | None — no unambiguous dead chat code found | 2026-05-24 |
| L7 — Teaching loop | ✅ Audited | **PARTIAL** | None — no unambiguously dead modules found | 2026-05-24 |
| L8 — Inter-session memory + contemplation | ⏳ Pending | — | — | — |
| L9 — Epistemic state + verdicts | ⏳ Pending | — | — | — |
| L9 — Epistemic state + verdicts | ✅ Audited | **PARTIAL** | None — no dead code found | 2026-05-24 |
L10 / L11 are not audit targets (no design to audit; flagged in scope as
gaps the audit will surface need for).
@ -1411,3 +1412,138 @@ Subsequent engine invocations parse the new active corpus via `chat.teaching_gro
---
## L9 — Epistemic state + verdicts
**Audit date:** 2026-05-24
**Auditor:** primary agent (Gemini)
**Verdict:** **PARTIAL**
### Scope-hypothesis correction (per audit step 0)
The scope cited files like `chat/safety.py` and `chat/ethics.py` as starting points. Reality: safety check (`SafetyCheck`) and ethics check (`EthicsCheck`) reside in `packs/safety/check.py` and `packs/ethics/check.py` respectively, while the runtime orchestrator `chat/runtime.py` invokes them and constructs the `TurnVerdicts` bundle defined in `chat/verdicts.py`.
### ADRs in scope for L9
Triaged from decisions and ADRs:
| ADR | Title | Status | Belongs at L9? |
|---|---|---|---|
| ADR-0029 | Safety Packs — Always-Loaded, Never-Replaceable Boundaries | Accepted | Yes |
| ADR-0032 | Safety Check Surface | Accepted | Yes |
| ADR-0033 | Ethics Packs — Swappable Domain Commitments | Accepted | Yes |
| ADR-0034 | Ethics Check Surface | Accepted | Yes |
| ADR-0035 | Turn-Loop Verdict Surfacing | Accepted | Yes |
| ADR-0036 | Safety-Only Typed Refusal Policy | Accepted | Yes |
| ADR-0037 | Per-Predicate Ethics Refusal Opt-In | Accepted | Yes |
| ADR-0038 | Hedge Injection as Runtime Affordance | Accepted | Yes |
| ADR-0039 | Audit Completeness / `TurnVerdicts` Bundle | Accepted | Yes |
| ADR-0040 | Structured-Logging Sink | Accepted | Yes |
| ADR-0041 | CLI Verdicts and Fan-Out | Accepted | Yes |
| ADR-0044 | Medical / clinical ethics pack | Accepted | Yes |
| ADR-0142 | Epistemic State Taxonomy | Accepted | Yes |
### Modules in scope for L9
| Module | Lines | Live-import sites (outside own package, outside `tests/`) | Test-import sites | Status |
|---|---|---|---|---|
| `packs/safety/check.py` | 333 | 2 (`chat/runtime.py`, `core/cognition/result.py`) | 9 | Live |
| `packs/safety/loader.py` | 260 | 1 (`chat/runtime.py`) | 4 | Live |
| `packs/ethics/check.py` | 409 | 2 (`chat/runtime.py`, `core/cognition/result.py`) | 8 | Live |
| `packs/ethics/loader.py` | 410 | 1 (`chat/runtime.py`) | 5 | Live |
| `chat/verdicts.py` | 50 | 1 (`chat/runtime.py`) | 3 | Live |
| `chat/refusal.py` | 178 | 2 (`chat/runtime.py`, `core/cognition/pipeline.py`) | 4 | Live |
| `chat/dispatch_trace.py` | 13 | 2 (`chat/runtime.py`, `core/cognition/result.py`) | 1 | Live |
| `core/epistemic_state.py` | 146 | 5 | 7 | Live |
### Caller-trace evidence
Exposed symbols of the layer are cleanly resolved through static imports across multiple layers:
- `packs.safety.check.SafetyCheck` and `packs.ethics.check.EthicsCheck` are instantiated in `chat/runtime.py:536-537` and executed on every turn.
- `chat.refusal.build_refusal_surface` and `chat.refusal.should_inject_hedge` are invoked on both the stub and main paths of `chat/runtime.py:1413,1892` to compute remediation surfaces.
- `chat.verdicts.TurnVerdicts` is constructed per turn (`chat/runtime.py:1517,2058`) and attached to `ChatResponse` and `TurnEvent`.
- `core.epistemic_state` enums and helper mappings (`EpistemicState`, `NormativeClearance`, `clearance_from_verdicts`, `epistemic_state_for_grounding_source`) are invoked by `chat/runtime.py` and `core/cognition/pipeline.py` to tag results and events.
- Telemetry events are structured and written in `chat/telemetry.py:46` by serializing `TurnEvent.verdicts` and mapping state enums deterministic-by-default.
### Exercising suite lane
- `core test --suite packs` — Exercises loaders, validators, checks, self-seal report verifications:
```bash
python3 -m core.cli test --suite packs -q
```
**Verification:** 13 passed, 0 skipped.
- `core test --suite smoke` — Exercises end-to-end turn loop composition, stub-path TurnEvent logging, and hedge injection:
```bash
python3 -m core.cli test --suite smoke -q
```
**Verification:** 67 passed, 0 skipped.
- `core test --suite cognition` — Exercises pipeline orchestration, trace-hash conditional folding, and state-tagging:
```bash
python3 -m core.cli test --suite cognition -q
```
**Verification:** 120 passed, 1 skipped.
- `verify_lane_shas.py` — Exercises all 7 pinned lanes:
```bash
python3 scripts/verify_lane_shas.py
```
**Verification:** 7/7 match pinned SHAs.
### Cross-layer contract check
**Pass 1 — mechanical (consumer-exists per exposed symbol):**
| Exposed symbol | Consumer evidence |
|---|---|
| `TurnVerdicts` | `ChatResponse`, `TurnEvent`, `chat/telemetry.py` (serialized in `serialize_turn_event`), `core/epistemic_state.py` (`clearance_from_verdicts`), and tests |
| `build_refusal_surface` | `chat/runtime.py:1413,1892`, and tests |
| `should_inject_hedge` / `build_hedge_prefix` / `inject_hedge` | `chat/runtime.py:1981-1985`, and tests |
| `EpistemicState` / `NormativeClearance` | `ChatResponse`, `TurnEvent`, `core/cognition/pipeline.py`, `chat/telemetry.py`, and tests |
| `clearance_from_verdicts` / `normative_detail_from_verdicts` | `chat/runtime.py:1525,2068`, and tests |
| `epistemic_state_for_grounding_source` | `chat/runtime.py:1524,2065`, and tests |
| `SafetyCheck` / `EthicsCheck` | `chat/runtime.py:536-537`, and tests |
| `load_safety_pack` / `load_ethics_pack` | `chat/runtime.py:541-542`, and tests |
**Pass 2 — semantic (six L9-specific invariants checked):**
1. **Safety verdicts fail-closed (ADR-0029, ADR-0036).** Verified. Safety violations replace the response surface with a typed refusal completely. Downstream composer attempts are skipped with `"refusal_emitted"`.
2. **Ethics refusals opt-in (ADR-0037).** Verified. Failed commitments only trigger refusal if specified in `EthicsPack.refusal_commitments`. Other failed commitments remain audit-only.
3. **Epistemic state taxonomy completeness (ADR-0142).** Verified. `TurnEvent` and `CognitiveTurnResult` carry the `epistemic_state` and `normative_clearance` enums. No silent defaults: ungrounded/unknown runs resolve to `undetermined` or `epistemic_state_needed`.
4. **Refusal_reason materialization matrix.** Mapping confirmed: safety and opt-in ethics violations propagate to both `ChatResponse.refusal_reason` and `CognitiveTurnResult.refusal_reason`. However, recognition refusals (W-011) and Inner-loop admissibility exhaustions (W-012) fail to propagate to refusal reason.
5. **Audit completeness (ADR-0039).** Verified. `TurnVerdicts` aggregates safety/ethics verdicts and remediation flags (`refusal_emitted`, `hedge_injected`) correctly.
6. **Telemetry sink redaction (ADR-0040).** Verified. `chat/telemetry.py` JsonlFileSink redact-by-default stance is honored: when `include_content=False` (default), surface text, input/walk surfaces, and canonical surfaces are omitted.
### Semantic mismatches flagged for human review
- **W-011 and W-012 open debt:** These remain active. Recognition refusals are dropped at the pipeline boundary (W-011); `InnerLoopExhaustion` exceptions during walk generation are uncaught and propagate as raw exceptions instead of populating `refusal_reason` (W-012).
- **`dispatch_trace` is observability-only:** Confirmed. It is not folded into the trace hash, which is the correct design intent to preserve replay stability.
### Closure criteria scorecard
| Criterion | Status | Evidence |
|---|---|---|
| 1. Design artifact | ✅ | ADR-0029, ADR-0032..0041, ADR-0044, ADR-0142 |
| 2. Code artifact | ✅ | `packs/safety/check.py`, `packs/safety/loader.py`, `packs/ethics/check.py`, `packs/ethics/loader.py`, `chat/verdicts.py`, `chat/refusal.py`, `chat/dispatch_trace.py`, `core/epistemic_state.py` |
| 3. Live caller | ⚠️ PARTIAL | Safety, ethics, and verdict-surface checks live; recognition refusal (W-011) and InnerLoopExhaustion refusal (W-012) propagation remain open |
| 4. Exercised by suite lane | ✅ | `packs`, `smoke`, and `cognition` lanes pass; `verify_lane_shas.py` is 7/7 |
| 5. Cross-layer consistency | ⚠️ PARTIAL | Invariants are consistent, but wiring debt (W-011 and W-012) prevents complete epistemic refusal tracking |
**Verdict:** **PARTIAL** (due to active W-011 and W-012 wiring debts).
### Cleanup performed
**None.** Inspected `packs/safety/`, `packs/ethics/`, `chat/refusal.py`, `chat/verdicts.py`, and `core/epistemic_state.py`. No redundant, legacy, or dead modules found.
### Forward notes to the operator (Final audit layer 9/9 complete)
As Layer L9 is the final audit layer of the substrate, all 9/9 layers of the CORE engine have been verified. The forward notes address the operator directly on what the ratchet needs to resolve:
1. **Wiring Debt Resolution (CLOSED-eligible pathways):**
- **W-011 (Recognition Refusal):** Wire `_rec_outcome.refusal_reason` to the turn result instead of dropping it at the pipeline boundary.
- **W-012 (Inner-Loop Exhaustion):** Catch `InnerLoopExhaustion` in `ChatRuntime.chat()` and populate `ChatResponse.refusal_reason` with the machine-readable refusal code so it is folded into the trace hash and results objects.
- **L2 / L3 Readback Modulation:** Implement energy-based tense and hedging modulations specified in ADR-0006/0007 (currently E0 recalled vault crystal is formatted the same as E2/E3 warmed field regions).
2. **Audit Method Feedback (v3 suggestions):**
- **Replay Equality Boundaries:** Future audit scopes should explicitly define which provenance fields (such as `dispatch_trace`) must participate in replay equality checks versus remaining purely observability-focused.
- **Static vs. Runtime Invariants**: Highlight that static check-only boundaries (e.g., `no_hot_path_repair` or `no_manipulation`) represent a different class of constraints that require static analysis or compile-time checks rather than dynamic runtime checking.
---