core/docs/safety_packs.md
Shay 6f67e9a616 feat(safety): ADR-0032 — SafetyCheck structural surface
Closes the 'boundaries are checked at scattered call sites' gap noted
in ADR-0029.  Adds a centralized observational surface parallel in
shape to IdentityCheck — produces a verdict, does not refuse.  Wiring
verdicts into refusal paths is a future ADR.

Shape (parallel to IdentityCheck, different in mechanism):

  SafetyContext     — duck-typed input bag (field_state, citations,
                       refusal-was-typed flag, identity manifold hashes
                       before/after).  Every field optional with safe
                       defaults; absence of evidence is not evidence of
                       violation.
  SafetyCheckResult — per-boundary: boundary_id, upheld, reason,
                       runtime_checkable, evidence tuple.
  SafetyVerdict     — aggregate: pack_id, results (lex order on
                       boundary_id), upheld, violated_boundaries,
                       runtime_checkable_count.
  SafetyCheck       — registry of predicates; check(ctx, pack) returns
                       SafetyVerdict.  register(boundary_id, predicate)
                       adds custom predicates.

Five default predicates for v1 boundaries:

  preserve_versor_closure   runtime_checkable=True   field.versor_condition < 1e-6
  no_fabricated_source      runtime_checkable=True*  cited ⊆ allowed
  no_silent_correction      runtime_checkable=True   last refusal was typed
  no_identity_override      runtime_checkable=True*  hash before == hash after
  no_hot_path_repair        runtime_checkable=FALSE  code-path; static-analysis

  *Conditional on the caller supplying the necessary fields.

The honest answer on no_hot_path_repair: it is a code-path boundary
enforced by static analysis + code review.  Runtime cannot judge it.
A predicate that silently reported upheld=True would be a small lie —
exactly the kind of thing CLAUDE.md forbids.  SafetyCheck reports
runtime_checkable=False with a clear reason so auditors see the truth.

ChatRuntime integration:
  ChatRuntime.__init__ now constructs self.safety_check = SafetyCheck()
  alongside self._identity_check.  Turn loop does NOT auto-invoke at
  v1 — operators and future ADRs decide when/where to call it.

Files:
  packs/safety/check.py            new — SafetyCheck + value types +
                                   default predicates
  packs/safety/__init__.py         re-exports the new public surface
  chat/runtime.py                  constructs self.safety_check
  tests/test_safety_check.py       new — 20 tests covering each
                                   default predicate (positive +
                                   negative), unknown-boundary
                                   fallback, custom registration,
                                   defensive boundary-id rebinding,
                                   verdict aggregation, ChatRuntime
                                   integration
  docs/decisions/ADR-0032-safety-check-surface.md  Accepted
  docs/safety_packs.md             §SafetyCheck section added,
                                   known-limit #1 struck through
  memory/safety-pack.md            refreshed; new follow-up about
                                   turn-loop auto-invocation

Suite status (all green):
  cognition 121, teaching 17, runtime 19, formation 182, smoke 67
  identity / safety / surface divergence suites: 108 tests passing
  (was 88 before this ADR; +20 safety-check tests)

Scope limits (documented):
  - No auto-invocation in the turn loop.
  - No refusal wiring on violation.
  - No refactoring of existing scattered enforcement sites.
  - Defensive boundary-id rebinding masks predicate bugs; debug-mode
    surfacing is a future enhancement.
2026-05-17 20:25:22 -07:00

9.8 KiB
Raw Permalink Blame History

Safety Packs — Reference

Status: Operational reference. Update when pack format, loader contract, or composition rules change. Last updated: 2026-05-17 Companion docs: decisions/ADR-0029-safety-packs.md, identity_packs.md

What a safety pack is

A safety pack carries the boundaries CORE will never cross, regardless of which identity pack is selected. Where identity packs encode who CORE is, safety packs encode what CORE will not do. The two layers compose at runtime: manifold.boundary_ids = safety.boundary_ids identity.boundary_ids.

Three properties distinguish safety packs from identity packs:

Property Identity pack Safety pack
Swappable at runtime Yes (--identity X) No
Multiple packs available Yes Exactly one
Failure to load Falls back to default; warns Fail-closed; refuses startup
Schema value_axes, surface_preferences, etc. boundary_ids, boundary_descriptions
Directory packs/identity/ packs/safety/

Shipping safety pack (v1)

Pack id Description Ratified
core_safety_axes_v1 Always-loaded core boundaries: no fabricated source, no hot-path repair, no identity override, no silent correction, preserve versor closure. ee1249acdf8c273aeb656d803c37ef915e536d85f177f5cc18c6e2f6c995ce29

Pack format (v1)

{
  "pack_id": "core_safety_axes_v1",
  "version": "1.0.0",
  "description": "Always-loaded, never-replaceable core safety boundaries.",
  "schema_version": "1.0.0",
  "mastery_report_sha256": "...",
  "boundary_ids": [
    "no_fabricated_source",
    "no_hot_path_repair",
    "no_identity_override",
    "no_silent_correction",
    "preserve_versor_closure"
  ],
  "boundary_descriptions": {
    "no_fabricated_source": "Citations must point to a real source span; the system never invents provenance.",
    "no_hot_path_repair": "...",
    "no_identity_override": "...",
    "no_silent_correction": "...",
    "preserve_versor_closure": "..."
  }
}

Field semantics

Field Required Meaning
pack_id yes Pack identifier. Convention: <slug>_v<major>.
version yes Semver.
description yes Human-facing one-liner.
schema_version yes Currently "1.0.0".
mastery_report_sha256 yes (production) SHA of the companion <pack_id>.mastery_report.json. Empty only in development; production refuses.
boundary_ids yes Non-empty list of unique boundary identifier strings.
boundary_descriptions yes Dict mapping each boundary_id to a human-readable rationale.

Loader bounds (enforced)

  • boundary_ids must be a non-empty list of unique non-empty strings.
  • pack_id must not contain / or ...
  • schema_version must equal "1.0.0".
  • In production mode (default), mastery_report_sha256 must be non-empty, the companion report must exist, its report_sha256 must match, and its self-seal must verify via formation.hashing.verify_seal.

Loader contract

from packs.safety.loader import load_safety_pack, SafetyPackError, DEFAULT_SAFETY_PACK

pack = load_safety_pack(
    pack_id=DEFAULT_SAFETY_PACK,    # default — callers should rarely pass anything else
    search_paths=None,               # default: ["./packs/safety"]
    require_ratified=True,           # production default
)

Returns a SafetyPack (frozen dataclass) with fields pack_id, version, description, boundary_ids (frozenset), boundary_descriptions (dict), mastery_report_sha256, ratified.

SafetyPackError inherits from RuntimeError, not ValueError. Missing safety pack is a fail-closed runtime condition, not a recoverable input error. Do not catch and continue.

Development override

CORE_ALLOW_UNRATIFIED_SAFETY=1 python -m core.cli chat

Bypasses only the seal-verification check. Missing file / empty boundaries / malformed JSON still fail closed. Use only while authoring or editing the safety pack; never set in production.

Composition rule

At ChatRuntime startup:

identity_manifold = load_identity_manifold(config.identity_pack or DEFAULT_IDENTITY_PACK)
safety_pack       = load_safety_pack()                # fail-closed
final_manifold    = IdentityManifold(
    value_axes        = identity_manifold.value_axes,
    boundary_ids      = identity_manifold.boundary_ids | safety_pack.boundary_ids,
    alignment_threshold = identity_manifold.alignment_threshold,
    surface_preferences = identity_manifold.surface_preferences,
)

Safety contributes boundaries only. Identity contributes axes, threshold, surface preferences, and may add further boundaries. The runtime exposes the loaded safety pack as ChatRuntime.safety_pack for audit.

Authoring a new safety pack

A safety pack is unique to a deployment. The shipping default is core_safety_axes_v1; downstream deployments may author their own stricter pack and place it at packs/safety/<deployment_safety_id>.json.

  1. Author the pack JSON. List the boundary ids your deployment requires; supply descriptions explaining each.
  2. Run python scripts/ratify_safety_pack.py (idempotent). Produces the companion .mastery_report.json and embeds the SHA in the pack.
  3. Test it under fail-closed semantics. Run python -m pytest tests/test_safety_pack.py and verify all 15 tests pass.
  4. Commit both files (<pack_id>.json and <pack_id>.mastery_report.json) atomically.

Anti-patterns

  • Don't catch SafetyPackError. A missing safety pack should crash the runtime, not silently degrade. The exception class deliberately doesn't inherit from ValueError.
  • Don't carry value axes in a safety pack. Safety boundaries are not directional preferences. If you find yourself wanting axes, you want an identity pack.
  • Don't make boundary text user-facing without curation. boundary_descriptions is for audit and operator visibility, not end-user prose.
  • Don't ship multiple safety packs. The design is "exactly one shipping safety pack per CORE installation." Per-tenant safety packs are an architectural change requiring a future ADR.

Versioning policy

Change Version bump
Description text edits Patch (v1.0.0v1.0.1)
Adding a boundary Minor (v1.0.0v1.1.0)
Removing a boundary Major + new ADR justifying the removal (core_safety_axes_v2)
Schema format change Major + new schema_version

A new major version means a new pack_id. The old pack remains in the repo for replay and audit; the runtime loads whichever pack id is shipped (currently hardcoded in packs.safety.loader.DEFAULT_SAFETY_PACK).

SafetyCheck — structural surface (ADR-0032)

A centralized, observational surface for evaluating safety boundaries at runtime, parallel in shape to IdentityCheck. Produces a SafetyVerdict; does not refuse. Wiring violations into refusal paths is a future ADR.

from packs.safety import SafetyCheck, SafetyContext

check = SafetyCheck()  # ships with default predicates for the five v1 boundaries
ctx = SafetyContext(
    field_state=current_field_state,
    cited_source_shas=frozenset({...}),
    allowed_source_shas=frozenset({...}),
    last_refusal_was_typed=True,
    identity_manifold_hash_before=before_hash,
    identity_manifold_hash_after=after_hash,
)
verdict = check.check(ctx, safety_pack)
# verdict.upheld: bool, verdict.violated_boundaries: frozenset[str]
# verdict.results: tuple of per-boundary SafetyCheckResult

Every field on SafetyContext is optional. Predicates over fields the caller didn't populate default to upheld=True, runtime_checkable=False — absence of evidence is not evidence of violation. ChatRuntime exposes a pre-constructed instance as runtime.safety_check; the turn loop does not auto-invoke it at v1.

Default predicates per v1 boundary

Boundary Runtime-checkable? What it checks
preserve_versor_closure Yes field_state.versor_condition < 1.0e-6
no_fabricated_source Yes (when allowlist supplied) cited_source_shas ⊆ allowed_source_shas
no_silent_correction Yes last_refusal_was_typed flag
no_identity_override Yes (when both hashes supplied) identity-manifold hash before == after
no_hot_path_repair No code-path boundary; enforced by static analysis + code review

The no_hot_path_repair predicate reports runtime_checkable=False and upheld=True honestly. A predicate that silently reported upheld=True would be a small lie — the surface acknowledges what it cannot judge.

Custom predicates

check = SafetyCheck()
check.register("my_boundary_id", my_predicate)

Unknown boundaries (declared in the pack but no predicate registered) default to upheld=True, runtime_checkable=False, reason="no predicate registered for boundary". Doesn't crash; surfaces in audit.

Known limits

  1. No SafetyCheck parallel to IdentityCheck. Closed by ADR-0032 (2026-05-17). See §SafetyCheck above. v1 is observational; turn-loop auto-invocation and refusal wiring are future ADRs.
  2. No per-tenant safety packs. Multi-tenant CORE deployments share one safety pack.
  3. No human-in-the-loop ratification step. Operational discipline lives in PR review, not the code.
  4. English-only boundary descriptions at v1.

Cross-reference index

  • Pack format spec: this doc §"Pack format (v1)".
  • Loader contract: this doc §"Loader contract".
  • Decision record: ADR-0029.
  • Identity pack composition: identity_packs.md.
  • Trust-boundary doctrine: runtime_contracts.md, CLAUDE.md "Security and Trust Boundaries".
  • The formation template used for ratification: formation/templates/identity_anchor.py.