core/docs/decisions/ADR-0039-audit-completeness.md
Shay f3cc408f82 feat(adr-0039): audit completeness — TurnVerdicts bundle, stub TurnEvent, hedge_injected
Closes three audit gaps left by the ADR-0035→ADR-0038 pack-layer
surface:

1. TurnVerdicts bundle (chat/verdicts.py) — frozen dataclass
   aggregating identity_score + safety_verdict + ethics_verdict +
   refusal_emitted + hedge_injected.  Attached to both
   ChatResponse.verdicts and TurnEvent.verdicts.  Fields typed as
   object for the same module-coupling reason as
   TurnEvent.safety_verdict.

2. Stub-path TurnEvent emission — _stub_response accepts optional
   tokens kwarg and appends a TurnEvent to turn_log when invoked
   from a real turn.  Audit consumers can now iterate turn_log
   end-to-end without missing stub paths.  Defensive call sites
   (correct() fallback) bypass the append by omitting tokens.

3. refusal_emitted / hedge_injected flags — runtime tracks whether
   it actually mutated the surface this turn.  hedge_injected uses
   idempotent-on-prefix semantics (True iff the runtime ADDED a
   hedge, not iff a hedge happens to be present).

Test-pattern note: previous "gate on rt.turn_log to detect main vs
stub" pattern is now broken; updated to gate on walk_surface ==
_UNKNOWN_DOMAIN_SURFACE.  One existing hedge-injection test gate
updated accordingly.

Back-compat: ADR-0035→0038 per-field accessors
(response.safety_verdict, etc.) still work.  New consumers should
read response.verdicts.

Files:
- chat/verdicts.py (new) — TurnVerdicts dataclass
- chat/runtime.py — _stub_response tokens kwarg + stub TurnEvent
  append + hedge_injected tracking + bundle construction
- core/physics/identity.py — TurnEvent.verdicts: object = None
- tests/test_turn_verdicts_bundle.py (new) — 16 tests
- tests/test_hedge_injection.py — gate fix for stub detection
- docs/decisions/ADR-0039-audit-completeness.md (new)

Verification:
- Combined pack-layer suite: 170 green (was 154 after ADR-0038)
- CLI suites unchanged: smoke 67, runtime 19, cognition 121
- core eval cognition: intent 100%, versor_closure 100% (baseline)
2026-05-17 21:32:46 -07:00

186 lines
8.4 KiB
Markdown

# ADR-0039: Audit Completeness — `TurnVerdicts` Bundle, Stub-Path `TurnEvent`, `hedge_injected` Signal
**Status:** Accepted (2026-05-17)
**Author:** Joshua Shay + planner pass
**Companion docs:** [`ADR-0035-turn-loop-verdict-surfacing.md`](ADR-0035-turn-loop-verdict-surfacing.md), [`ADR-0036-safety-refusal-policy.md`](ADR-0036-safety-refusal-policy.md), [`ADR-0037-per-predicate-ethics-refusal.md`](ADR-0037-per-predicate-ethics-refusal.md), [`ADR-0038-hedge-injection.md`](ADR-0038-hedge-injection.md)
## Context
After ADR-0035 → ADR-0038, the runtime emits three verdicts per turn
(identity / safety / ethics) and two possible remediations (typed
refusal, hedge injection). But auditing it had three rough edges:
1. **Per-field correlation.** An audit consumer had to read three
separate fields (`identity_score`, `safety_verdict`,
`ethics_verdict`) and *infer* the remediation by inspecting the
surface text (does it start with the refusal prefix? with the
hedge phrase?). Surface-text inference is brittle.
2. **Stub-path TurnEvent gap.** Stub turns (cold start, unknown
domain) bypassed `turn_log.append()`. ADR-0035 already noted
this as a known limit. The result: audit consumers reading the
turn stream couldn't see stub turns at all, even though stub turns
*did* carry a `ChatResponse.safety_verdict` and
`ChatResponse.ethics_verdict`.
3. **`hedge_injected` invisibility.** Whether the runtime actually
prepended a hedge this turn could only be detected by surface
inspection (does it start with `preferred_hedge_soft`?). An
audit consumer wanting "count hedged turns this hour" had to
re-implement runtime decision logic.
The three rough edges share a root cause: the runtime knew the
answers but didn't *surface* them in a form callers could read.
## Decision
Three changes land together. Each is small; together they close the
audit gap.
### 1. `TurnVerdicts` bundle type
A new frozen dataclass in `chat/verdicts.py`:
```python
@dataclass(frozen=True, slots=True)
class TurnVerdicts:
identity_score: object # IdentityScore | None
safety_verdict: object # SafetyVerdict | None
ethics_verdict: object # EthicsVerdict | None
refusal_emitted: bool # ADR-0036 / ADR-0037
hedge_injected: bool # ADR-0038
```
Fields are typed `object` for the same reason `TurnEvent.safety_verdict`
was: avoid coupling `chat/verdicts.py` to packs.* at module-resolution
time. Audit consumers downcast at use site.
The bundle is attached to both `ChatResponse.verdicts` and
`TurnEvent.verdicts`. The pre-existing individual fields
(`safety_verdict`, `ethics_verdict`) remain — back-compat with
ADR-0035 callers — but new consumers should read the bundle.
### 2. Stub-path `TurnEvent` emission
`_stub_response` now accepts an optional `tokens` kwarg. When invoked
from a real turn (with `tokens` non-empty), it constructs and appends
a `TurnEvent` to `turn_log` before returning the `ChatResponse`.
The stub event records:
| Field | Value |
|---|---|
| `turn` | `self._context.turn - 1` (after `finalize_turn` already ran) |
| `input_tokens` | tokens passed in |
| `surface` | typed refusal if it fired, else the unknown-domain marker |
| `walk_surface` | unknown-domain marker (preserved) |
| `articulation_surface` | unknown-domain marker (preserved) |
| `identity_score` | `None` (no trajectory ran) |
| `cycle_cost_total` | `0.0` |
| `vault_hits` | `0` |
| `versor_condition` | `versor_condition(field_state.F)` |
| `flagged` | `False` |
| `safety_verdict` / `ethics_verdict` / `verdicts` | computed verdicts |
The `correct()` fallback path still calls `_stub_response` without
tokens — that's a defensive call where no real "turn" happened, and
appending a `TurnEvent` would mis-record the audit stream.
### 3. `hedge_injected` signal
The main turn path tracks whether the runtime actually mutated the
surface during hedge injection:
```python
before = response_surface
response_surface = inject_hedge(response_surface, hedge_prefix)
hedge_injected = response_surface != before
```
`inject_hedge()` is idempotent on prefix (ADR-0038) — if the surface
already begins with the hedge phrase, the function returns it
unchanged and `hedge_injected` stays `False`. This is the correct
audit semantic: "did the runtime ADD a hedge this turn?", not "is
there a hedge somewhere in the surface?"
Stub paths always report `hedge_injected=False` (ADR-0038 prohibits
hedge on stub).
## Consequences
### Positive
* **One field, full picture.** An audit consumer reads
`response.verdicts` (or `event.verdicts`) and gets identity +
safety + ethics + remediation flags. No correlation across fields.
* **Surface-inspection no longer needed.** `refusal_emitted` and
`hedge_injected` answer the runtime-decision questions directly.
* **Stub turns are now first-class audit events.** `turn_log` now
covers the entire turn stream; downstream telemetry and replay
systems can iterate over `turn_log` without missing stub paths.
* **Mutual exclusion is verifiable.** Existing test
(`test_refusal_and_hedge_never_both_true`) confirms the runtime
contract holds at the bundle level.
* **No back-compat breakage.** ADR-0035 / ADR-0036 / ADR-0037 /
ADR-0038 individual fields and helpers still work.
### Negative / risks
* **Stub turns now have `identity_score=None` in the audit stream.**
Downstream consumers that assumed every TurnEvent carried an
IdentityScore need to handle `None`. Mitigated: this was already
true on `ChatResponse.identity_score` for stub paths; consumers
reading the stream just had no stream entries at all before, and
now they have entries with `identity_score=None`. The change is
additive.
* **The bundle duplicates per-field state.** `safety_verdict` is
reachable as both `response.safety_verdict` and
`response.verdicts.safety_verdict`. Acceptable cost for
back-compat; a future ADR could remove the per-field accessors if
no in-tree consumer still uses them.
* **`hedge_injected` doesn't distinguish "would have fired but
idempotent" from "didn't fire at all."** Two distinct cases
collapse to `hedge_injected=False`. Audit consumers wanting that
distinction can read the ethics_verdict directly.
* **Hedge-injection test gate updated.** The pre-ADR-0039 gate
("skip if `turn_log` empty") no longer discriminates stub vs main
path; the test updated to gate on `walk_surface ==
_UNKNOWN_DOMAIN_SURFACE` instead. No semantic change.
## Verification
* `tests/test_turn_verdicts_bundle.py` — 16 tests covering: bundle
shape and frozen contract; `ChatResponse` and `TurnEvent` carry
the bundle; stub path appends a `TurnEvent` with input tokens,
unknown walk/articulation surfaces, `identity_score=None`,
recorded versor condition; `refusal_emitted` flag toggles on
forced safety violation, appears symmetrically on
`ChatResponse.verdicts` and `TurnEvent.verdicts`; `hedge_injected`
flag defaults False, stays False on stub paths even with opt-in;
mutual exclusion (refusal supersedes hedge); response and event
bundles agree on remediation flags and reference the same
underlying verdicts.
* Combined pack-layer suite: **170 tests, all green** (was 154 after
ADR-0038).
* CLI suites unchanged: smoke 67, runtime 19, cognition 121.
* `core eval cognition`: intent 100%, versor_closure 100% — baseline
preserved.
## Open questions deferred to a future ADR
1. **Structured-logging sink that consumes `turn_log`.** Now that
stub turns participate in the stream, a structured emitter could
produce one log line per turn with `pack_id`, `refusal_emitted`,
`hedge_injected`, and violated boundary ids.
2. **`core chat --show-verdicts` CLI flag.** Print per-turn verdict
bundle summaries to stdout/stderr for manual audit.
3. **Drop per-field accessors after a deprecation cycle.**
`response.safety_verdict` could become `response.verdicts.safety_verdict`-only
once internal callers migrate.
4. **`identity_score` on stub turns.** Today stub paths skip the
trajectory operator entirely and report `None`. A future ADR
could compute a degenerate-but-valid IdentityScore for stubs so
the audit stream is fully populated.
5. **Bundle versioning.** As more remediation tiers land (per-domain
default policies, score-decomposition surfaces), the bundle may
grow. Frozen-dataclass + optional defaults should scale fine,
but a `schema_version` field could be added if downstream consumers
need explicit versioning.