core/docs
Shay 573ef98aba
docs(audit): scope substrate liveness audit (system-of-systems closure) (#234)
* docs(audit): scope substrate liveness audit (system-of-systems closure)

The recognizer-storage v1→v2 revision surfaced a pattern: CORE
contains ~140 ADRs, many marked Implemented, but several have
spec-in-code that nothing live calls (e.g., VaultPromotionPolicy in
core/physics/learning.py — imported by no module outside its package).
The engine today executes a subset of its own design.

Per the operator's system-of-systems framing (human body / universe /
ecosystem: subsystems achieve closure together; a half-built layer
degrades the whole organism silently): this scope defines a layered
audit that walks from the foundation outward to identify, per ADR
and per module, which subsystems are closed (designed + wired +
exercised + cross-layer consistent), which are partial, and which
are open.

The audit method is mechanical: grep + caller-trace + end-to-end test
verification + cross-layer contract check. Two reviewers running the
audit should produce identical verdicts. No refactoring, no new ADRs,
no subjective judgment — just evidence.

The output is two artifacts: a closure registry (per-layer, per-ADR
verdicts with evidence) and a ratchet plan (wiring sequence in
dependency order). Both append-only / revisable; both committed to
the repo as audit artifacts.

First-pass layering (L0 algebra primitives → L11 forever-running
engine, with L10 runtime model named as the missing prerequisite)
is a hypothesis the audit will refine. Layers L0–L3 are expected to
be closed (foundation); L4–L9 are expected to be partial; L10–L11
are explicitly open and depend on the audit + the runtime-model
scope.

Applies feedback-adr-cross-reference-discipline (the memory entry
this revision flagged): explicit cross-references to ADR-0006/0014/
0055/0056/0057/0142/0143/0144 and the existing scope docs.

This is a scope, not an audit. Audit deliverables (registry, ratchet)
are separate work.

* docs(audit): revise substrate-liveness-audit scope to v2 (self-review fixes)

Self-review surfaced two HIGH, three MEDIUM gaps in v1. Notably,
v1 of the scope that creates cross-reference discipline still
committed the documented mistake — third consecutive iteration of
the same failure mode in one session (recognizer-storage v1
substrate overclaim → recognizer-storage v2 drop-off invention →
audit-scope v1 ADR range mis-grouping). New "Self-review
acknowledgment" section records the pattern's durability and
states the structural mitigation: the audit's mechanical
deliverables make the discipline impossible to skip silently,
which is more rigorous than the memory entry alone.

HIGH-1 — ADR range mis-grouping. v1 layering table listed
"ADR-0055..0064" as L7 (teaching loop); verification showed
ADR-0058-0064 are predominantly L6 (surface composition,
correction telemetry, cross-pack resolution). Fixed L7 to cite
only ADR-0057; added explicit note that ADR-range citations
are starting points and the audit's first act per layer is
re-enumeration.

HIGH-2 — Audit tractability buried in risks. ~140 ADRs requires
structural handling, not just a risk warning. Promoted "per-layer
commits + per-layer handoff to subagents + progress tracking in
registry + optional per-layer file splitting" to a first-class
Step 0 in the audit method. The audit is explicitly framed as the
archetypal parallel-agent handoff candidate.

MEDIUM-1 — Expected-status column anchored the auditor. v1's
table had my predictions ("Closed (foundation)", "Live but
session-bounded"). Removed; replaced with a "Where to look first"
column. Explicit note: "No expected-status column intentionally
— predictions are the failure mode this scope was meant to
prevent."

MEDIUM-2 — "End-to-end test" criterion maps awkwardly onto CORE's
suite-lane organization. Reframed Step 4 to "Identify the
exercising suite lane" with concrete `core test --suite {…}` /
`core eval …` invocations. A module whose only test coverage is
in `tests/` files not reached by any suite lane is a closure gap.

MEDIUM-3 — Cross-layer contract check was hand-wavy. Made
Step 5 explicitly two-pass: mechanical (grep for at least one
consumer per exposed field/method) carries full verdict authority;
judgment-required semantic mismatches are flagged for operator
review rather than verdicted mechanically.

LOW fixes: softened "two reviewers identical" claim; L10/L11
explicitly marked not-audit-targets; per-layer file splitting
flagged as auditor's choice; closure-criteria item 4 wording
aligned with new Step 4.

Frontmatter status bumped to "Draft v2"; date line records
revision provenance.
2026-05-24 16:02:10 -07:00
..
adr
architecture
briefs/parallel-2026-05-23 docs(ADR-0136): statement-layer corridor, S.0 taxonomy, 2026-05-23 brief archive (#200) 2026-05-23 20:25:23 -07:00
curriculum
decisions docs(audit): scope substrate liveness audit (system-of-systems closure) (#234) 2026-05-24 16:02:10 -07:00
evals
handoff
implementation docs: propose semantic-symbolic binding graph layer (#170) 2026-05-23 09:58:39 -07:00
sessions docs: pedagogy research review + 2 deferred teaching-loop ADRs 2026-05-23 07:01:55 -07:00
capability_roadmap.md docs: Phase 5 GSM8K-math substrate completion sweep (ADR-0119 + sub-phases + ADR-0114a 10/10) 2026-05-22 20:39:24 -07:00
ethics_packs.md
EVAL_AUDIT_2026-05-20.md
eval_methodology.md
formation_pipeline_plan.md
frontier_baselines.md
gaps.md
holdout_recipients.txt
identity_packs.md
pack_inventory_2026-05-21.md
PROGRESS.md docs: Phase 5 GSM8K-math substrate completion sweep (ADR-0119 + sub-phases + ADR-0114a 10/10) 2026-05-22 20:39:24 -07:00
reviewers.yaml docs(ADR-0136): statement-layer corridor + S.0 taxonomy + resolve math-file conflicts 2026-05-23 20:21:32 -07:00
runtime_contracts.md docs: Phase 5 GSM8K-math substrate completion sweep (ADR-0119 + sub-phases + ADR-0114a 10/10) 2026-05-22 20:39:24 -07:00
RUST.md
safety_packs.md
teaching_order.md
truth_seeking_schema.md
Whitepaper.md docs: Phase 5 GSM8K-math substrate completion sweep (ADR-0119 + sub-phases + ADR-0114a 10/10) 2026-05-22 20:39:24 -07:00
Yellowpaper.md docs: Phase 5 GSM8K-math substrate completion sweep (ADR-0119 + sub-phases + ADR-0114a 10/10) 2026-05-22 20:39:24 -07:00