diff --git a/docs/workbench/acceptance-gates.md b/docs/workbench/acceptance-gates.md index d5ba2c6b..5884ac08 100644 --- a/docs/workbench/acceptance-gates.md +++ b/docs/workbench/acceptance-gates.md @@ -15,7 +15,7 @@ The planning branch is acceptable when it includes: - data-shape contract - UI component map - explicit non-goals -- explicit mutation boundaries +- explicit mutation boundaries and admitted-corridor requirements - work queue for W-026 through W-031 ## W-026 — Read-only API gates @@ -29,7 +29,7 @@ Required: - path traversal tests for artifact readers - proposal event-log read-model tests using `ProposalLog.current_state()` - unknown trace ids return `404`, not placeholder success payloads -- no proposal accept/reject route +- no proposal accept/reject route outside an admitted corridor - no corpus mutation route - no pack mutation route - no workflow dispatch route @@ -64,7 +64,7 @@ Required: - top runtime bar - empty states for all modules - API client shape aligned to `data-shapes-v1.md` -- no mutation buttons +- no mutation buttons outside an admitted corridor - no fake/mock runtime as permanent architecture Acceptance command candidates: @@ -103,13 +103,18 @@ Required: - display replay evidence - display proposed chain - display suggested CLI copy command +- mutation only through admitted corridors: + ADR-governed path, visible preconditions, auditable telemetry, and replay + evidence before action Forbidden: -- accept button -- reject button +- unadmitted accept/reject button - workflow dispatch -- direct proposal mutation +- direct proposal mutation outside an admitted corridor +- corpus mutation +- pack mutation +- arbitrary file writes ## W-030 — Eval Center gates @@ -150,17 +155,18 @@ The v1 workbench is releasable when an operator can: 1. start the local API, 2. open the UI, 3. see runtime status, -4. inspect a proposal without mutating it, +4. inspect a proposal and see whether an admitted corridor applies, 5. run/view contemplation-quality, 6. inspect trace/replay metadata, 7. perform a basic chat turn, -8. verify from UI copy and docs that no hidden ratification/mutation path exists. +8. verify from UI copy and docs that no unadmitted ratification/mutation path exists. ## Red flags Any of these should block merge: -- UI can accept/reject proposals in v1 +- UI can accept/reject proposals without an admitted corridor, visible + preconditions, auditable telemetry, and replay evidence - API exposes arbitrary path reads - API invokes shell from user input - frontend hardcodes fake eval/proposal data as product path diff --git a/docs/workbench/implementation-plan.md b/docs/workbench/implementation-plan.md index 44ebd3da..4e92dba9 100644 --- a/docs/workbench/implementation-plan.md +++ b/docs/workbench/implementation-plan.md @@ -1,8 +1,10 @@ # CORE Workbench v1 — Implementation Plan This plan intentionally starts with planning and read-only observability before -any mutating workflow. The purpose is to preserve CORE's existing ADR trust +any mutating workflow. The purpose is to preserve CORE's existing ADR trust boundaries while making the system legible to operators, engineers, and auditors. +When mutation is admitted, it must pass through an explicit corridor rather than +appearing as an ambient UI capability. ## Work queue @@ -11,7 +13,7 @@ boundaries while making the system legible to operators, engineers, and auditors | W-026 | Read-only API contract | endpoint schemas + stdlib local server surface only | No | | W-027 | Frontend shell | navigation, layout, design tokens, empty states | No | | W-028 | Chat + trace drawer | basic chat, turn metadata, trace drawer | Runtime turn only | -| W-029 | Proposal queue | proposal-log read model, detail view, CLI-copy affordance | No | +| W-029 | Proposal queue | proposal-log read model, detail view, CLI-copy affordance, admitted math ratification corridor | Math ratification only | | W-030 | Eval center | lane discovery, run/read evals, result viewer | Eval result writes only if CLI already supports it | | W-031 | Replay theater | artifact selection, digest compare, replay status UI | No | @@ -32,7 +34,8 @@ No code beyond docs. ### Phase 1 — Read-only API skeleton -Goal: expose a narrow local workbench API without introducing new mutation paths. +Goal: expose a narrow local workbench API without introducing unadmitted +mutation paths. Deliverables: @@ -61,7 +64,8 @@ Strict rules: live chat endpoint, or replay theater. - Use the Python standard-library HTTP server for W-026. Defer FastAPI or any other web framework unless a later ADR admits the dependency. -- No proposal accept/reject endpoints. +- No proposal accept/reject endpoints unless a later ADR admits the specific + corridor and its preconditions. - No synthetic trace/replay evidence. Routes for later phases must return `unsupported` or `not_found` until a real evidence path exists. @@ -113,6 +117,31 @@ A live chat turn may use the existing runtime path. The workbench must label any runtime checkpointing clearly and must not add hidden persistence beyond runtime behavior already governed by ADR-0146/0150. +## Mutation doctrine + +The Workbench mutation rule is not "no mutation ever." It is: no mutation +without an admitted corridor, explicit preconditions, auditable telemetry, and +replay evidence. + +An admitted corridor must satisfy all four requirements: + +1. **ADR-governed path.** The path is named and bounded by an accepted ADR or + follow-on work item. Chat turns use the existing runtime checkpoint path + governed by ADR-0146/0150. Math ratification is the first proposal corridor, + governed by the ADR-0172 math proposal path. +2. **Visible preconditions.** The UI shows what must be true before action is + enabled. The math corridor requires pending state, replay equivalence, and an + admitted handler before ratification is available. +3. **Auditable telemetry.** Every action emits an operator event with enough + information to reconstruct what happened. +4. **Replay evidence before action.** The operator sees replay-equivalence + status and provenance before committing a change. + +Everything outside an admitted corridor remains forbidden. In particular, +corpus mutation, pack mutation, arbitrary file writes, arbitrary workflow +dispatch, hidden background workers, and unreviewed proposal mutation remain +out of bounds for v1. + ### Phase 4 — Proposal Queue Goal: make proposal lifecycle visible. @@ -128,10 +157,11 @@ Deliverables: Forbidden in v1: -- accept button -- reject button +- unadmitted accept/reject buttons - workflow dispatch - direct corpus mutation +- pack mutation +- arbitrary file writes ### Phase 5 — Eval Center @@ -170,7 +200,7 @@ Deliverables: - Artifact readers must be rooted under known repo directories. - Every response should contain enough metadata to audit source path, digest, and timestamp when available. -- Mutation endpoints are forbidden in v1. +- Unadmitted mutation endpoints are forbidden in v1. - Proposal reads must derive state from `ProposalLog.current_state()`, not by treating append-only JSONL events as proposal rows. @@ -191,7 +221,7 @@ Each implementation PR must include: - schema serialization tests - permission/path traversal tests for artifact readers - frontend smoke/build check once UI exists -- no-mutation regression checks for read-only routes +- no-unadmitted-mutation regression checks for read-only routes ## Release criteria for v1 @@ -201,10 +231,10 @@ The v1 workbench is accepted when an operator can: 2. open the UI, 3. inspect runtime status, 4. run or view contemplation-quality, -5. inspect a proposal without accepting it, +5. inspect a proposal and see whether any admitted corridor applies, 6. inspect a trace/replay artifact, 7. perform a basic chat turn, -8. verify no hidden mutation path exists. +8. verify no unadmitted mutation path exists. ## Explicit deferrals @@ -213,8 +243,10 @@ The v1 workbench is accepted when an operator can: - mobile UI - auth, unless a separate ADR admits it after the local read-only boundary is proven -- proposal accept/reject buttons +- unadmitted proposal accept/reject buttons - workflow dispatch +- corpus or pack mutation +- arbitrary file writes - packaged desktop app - public marketing site - arbitrary plugin/tool execution diff --git a/docs/workbench/wave-1-evidence-spine.md b/docs/workbench/wave-1-evidence-spine.md new file mode 100644 index 00000000..a837e885 --- /dev/null +++ b/docs/workbench/wave-1-evidence-spine.md @@ -0,0 +1,390 @@ +# Wave 1 — Evidence Spine + +Status: approved plan +Date: 2026-06-12 +Supersedes: implementation-plan.md phases W-032+ +Peer-reviewed by: Codex (architectural path selection), GPT5.5-Thinking (trace +honesty correction), Claude Opus 4.6 (original plan + synthesis) + +## Governing idea + +Evidence context is the intrinsic UI space. Every route is a projection of the +same evidence manifold. The workbench is not organized around routes (boxes on +screen); it is organized around the evidence chain: + +``` +operator intent + -> selected evidence subject + -> provenance + -> admissibility + -> replay + -> authority + -> allowed action +``` + +Chat shows the newest turn. Trace deepens it. Replay tests it. Proposals ask +whether it may alter reviewed memory. Packs/Vault/Audit reveal the substrate +that made it possible. Once the spine exists, remaining routes become inevitable +projections — not independent features. + +## Three governing principles (ADR-0160) + +1. **Audit-native, not analytics theater.** Every panel answers: what happened, + why was it allowed, what evidence exists, can it replay, who holds authority. +2. **Calm default, infinite depth.** Quiet surface by default; the deeper you + inspect, the more transparent it becomes. +3. **Replay before persuasion.** Show deterministic evidence before asking the + operator to trust anything. + +--- + +## Wave 1 deliverables + +Six pieces, one architectural idea. + +### 1A. Command Registry + Full Navigation + +- [ ] Replace hardcoded command list in `CommandPalette.tsx` with a + route-registered command registry +- [ ] Each route registers its commands on mount via a shared context/provider +- [ ] Fuzzy search across routes, recent resources (turns, proposals, artifacts), + and actions (run eval, copy hash) +- [ ] `Cmd+K` opens, type-ahead filters, arrow keys navigate, `Enter` executes, + `Esc` closes +- [ ] Recent items: last 10 visited resources (turns, proposals, artifacts) +- [ ] Test: palette finds and navigates to every route and registered command + +**Current state:** `CommandPalette.tsx` has only Chat/Proposals/Evals hardcoded. + +**Key files:** +- `workbench-ui/src/design/components/primitives/CommandPalette.tsx` +- New: `workbench-ui/src/app/commandRegistry.ts` (or context provider) + +--- + +### 1B. RightInspector as Evidence Drawer + +- [ ] Remove permanent `collapsed=true` from `Shell.tsx` +- [ ] Create shared evidence-subject context: `useEvidenceSubject()` hook + + provider in Shell +- [ ] Each route pushes its selection (turn, proposal, artifact, pack, eval + result) into the shared context +- [ ] Inspector renders the appropriate evidence projection for the selected + subject type +- [ ] Toggle via `Cmd+I` or "inspect" affordance on selectable items +- [ ] Stays open across route transitions (operator opened it deliberately) +- [ ] Collapsed by default on fresh load (calm default) +- [ ] Resizable width via drag handle +- [ ] Test: inspector opens, shows correct context for Chat selection, closes, + persists across route change + +**Current state:** `RightInspector.tsx` returns null. `Shell.tsx` hardcodes +`collapsed={true}`. + +**Key files:** +- `workbench-ui/src/app/RightInspector.tsx` +- `workbench-ui/src/app/Shell.tsx` +- New: `workbench-ui/src/app/evidenceContext.ts` + +--- + +### 1C. Minimal Evidence Primitives + +Build only the six components the spine needs. Defer everything else until a +route proves it requires the component. + +| Component | Purpose | Used by | +|---|---|---| +| `SplitPane` | Resizable horizontal/vertical split for list-detail | Trace, Proposals, Evals, Replay | +| `TabBar` | Accessible tab switching (see dependency note below) | Trace evidence sections, Inspector | +| `MetadataTable` | Key-value pair display for structured metadata | Trace, Proposals, Artifacts | +| `DigestBadge` | Copyable hash/digest with truncation + verify indicator | Trace, Replay, everywhere | +| `Timestamp` | Relative + absolute time, timezone-aware (PST/PDT) | All list views, Inspector | +| `SearchInput` | Filtered search with keyboard shortcut binding (`/`) | Command palette, list views | + +For each component: + +- [x] Built with design tokens only (no raw hex/rgb) +- [x] Motion via `--motion-duration-*` and `--motion-ease-*` tokens +- [x] `prefers-reduced-motion` collapses to instant (global rule in tokens.css) +- [x] `:focus-visible` ring via `--color-focus-ring` +- [x] Renders in PreviewPage (`/preview`) +- [x] Unit test + +**TabBar dependency decision:** Implemented with native ARIA tab semantics +(no `@radix-ui/react-tabs` added). Full `role="tablist"`/`role="tab"`/ +`role="tabpanel"` with `ArrowLeft`/`ArrowRight`/`Home`/`End` keyboard nav. + +**Deferred primitives** (build when a route needs them): +- DataTable, TreeView, Timeline, CodeViewer, Drawer, Toast, SkeletonLoader, Kbd + +--- + +### 1D. Workbench Turn Evidence Journal + +**Critical correction (GPT5.5 review):** The workbench backend does NOT +currently attach a telemetry sink to chat turns. `WorkbenchApi()` constructs +with no sink, `_run_chat_turn()` creates a bare `ChatRuntime()`, and +`serialize_turn_event` redacts content by default. Raw runtime telemetry does +not contain the three surfaces needed for Trace. + +**Solution:** A Workbench Turn Evidence Journal — a local, append-only, +content-bearing record of the `ChatTurnResult` envelope already returned by +`/chat/turn`. + +This is a **read model**, not a cognitive runtime fork. It does not replace +runtime telemetry; it records the exact evidence the operator already saw. + +#### Backend + +- [ ] New module: `workbench/journal.py` +- [ ] `TurnJournal` class: append-only JSONL writer +- [ ] Each `/chat/turn` response is journaled with: + - `turn_id` (stable, sequential) + - `timestamp` (ISO-8601 UTC) + - `trace_hash` (from ChatTurnResult) + - `prompt` (content-bearing — explicit in schema) + - `surface`, `articulation_surface`, `walk_surface` (all three, kept + separate per api-contract-v1.md line 231) + - `grounding_source`, `epistemic_state`, `normative_clearance` + - `verdicts` (identity, safety, ethics) + - `refusal_emitted`, `hedge_injected` + - `proposal_candidates` + - `turn_cost_ms` + - `journal_digest` (SHA-256 of the serialized entry) +- [ ] Journal path: `workbench_data/turn_journal.jsonl` (under repo root, not + under `engine_state/` or `teaching/`) +- [ ] Content-bearing warning: journal entries contain user prompts and engine + surfaces. Document this in `workbench_data/README.md` (not as a text + header in the JSONL file — every line must be valid JSON) +- [ ] Path confinement: journal writes only to `workbench_data/` +- [ ] No journal writes to `teaching/`, `packs/`, `language_packs/data/`, or + `engine_state/`. Note: existing chat turns DO write `engine_state/` + through the normal runtime checkpoint path governed by ADR-0146/0150 — + that is existing behavior, not journal behavior. + +#### New API endpoints + +- [ ] `GET /trace/turns` — list journal entries (summary: turn_id, timestamp, + prompt excerpt, surface excerpt, trace_hash, grounding_source) +- [ ] `GET /trace/{turn_id}` — full journal entry for a turn (replaces the + current 404 behavior) +- [ ] Pagination: `?limit=50&offset=0` (default limit 50) +- [ ] Unknown turn_id returns 404 (not synthetic data) + +#### Tests + +- [ ] Append-only behavior: entries are never modified or deleted +- [ ] Stable ordering: entries are sequential by turn_id +- [ ] Prompt/content size limits respected (max 4096 chars prompt) +- [ ] Path confinement: journal cannot write outside `workbench_data/` +- [ ] No journal writes to `teaching/`, `packs/`, `language_packs/data/`; no + NEW writes to `engine_state/` beyond existing chat checkpoint behavior + (ADR-0146/0150) +- [ ] Journal digest is deterministic for identical content +- [ ] Round-trip: `/chat/turn` response -> journal -> `/trace/{turn_id}` -> + identical evidence fields + +#### Optional linkage to runtime telemetry + +If `ChatRuntime` is later configured with a `JsonlFileSink` +(`include_content=True`), the Trace route can cross-reference journal entries +with runtime telemetry events by `trace_hash`. This is additive — the journal +is the primary read model. + +#### Public interfaces and types + +Backend (Python): +- `workbench/journal.py` — `TurnJournalEntry` dataclass, `TurnJournalSummary` + dataclass, `TurnJournal` class (append, list, get) +- `workbench/schemas.py` — add `TurnJournalEntrySchema`, `TurnJournalSummarySchema` + for API serialization + +Frontend (TypeScript): +- `workbench-ui/src/types/api.ts` — add `TurnJournalEntry`, + `TurnJournalSummary` interfaces mirroring the Python shapes +- `workbench-ui/src/api/client.ts` — add `fetchTraceTurns(limit?, offset?)`, + `fetchTraceTurn(turnId)` +- `workbench-ui/src/api/queries.ts` — add `useTraceTurns()`, + `useTraceTurn(turnId)` React Query hooks +- `workbench-ui/src/app/evidenceContext.ts` — `EvidenceSubject` union type: + `{ kind: 'turn', data: TurnJournalEntry }` | `{ kind: 'proposal', ... }` | + `{ kind: 'artifact', ... }` | `{ kind: 'eval_result', ... }` | + `{ kind: 'none' }` + +All new API responses use the existing `{ ok, generated_at, data/error }` +envelope. + +--- + +### 1E. Trace Route + +- [ ] Replace `TraceRoutePlaceholder.tsx` with real Trace route +- [ ] Layout: `SplitPane` — turn timeline (left) + trace evidence panel (right) +- [ ] Turn timeline: list of journal entries with `DigestBadge` (trace_hash + thumbnail), `Timestamp`, prompt excerpt +- [ ] Trace evidence panel: `TabBar` with sections: + - **Surfaces** — all three, labeled explicitly (surface = user response, + walk_surface = telemetry evidence, articulation_surface = realizer + output). This IS the canonical proof of the api-contract-v1.md surface + separation contract. + - **Grounding** — source, epistemic state, normative clearance + - **Verdicts** — identity, safety, ethics verdicts with badge indicators + - **Metadata** — `MetadataTable` showing turn_cost_ms, checkpoint_emitted, + refusal/hedge status, proposal candidates + - **Raw** — collapsed-by-default full JSON viewer (StableJsonViewer) +- [ ] Selection pushes turn into evidence-subject context (RightInspector shows + same turn from inspector angle) +- [ ] `SearchInput` for filtering turns by prompt text or trace_hash prefix +- [ ] Empty state when journal is empty: "No turns recorded yet. Use Chat to + create evidence." +- [ ] Test: navigate to Trace, see real journal entries, select one, see evidence + panel, inspect raw JSON + +**Key design rules:** +- Versor condition (when available): green < 1e-6, red >= 1e-6 +- Walk surface labeled "telemetry/evidence" — never confused with user surface +- Trace hash always visible and copyable (replay before persuasion) +- Raw trace behind explicit expand (calm default, infinite depth) + +--- + +### 1F. Mutation Doctrine Reconciliation + +- [x] Update `docs/workbench/implementation-plan.md` mutation section to match + reality +- [x] Update `docs/workbench/acceptance-gates.md` to reflect admitted corridors +- [x] Document the honest rule: + +**The mutation rule is not "no buttons ever." It is:** + +1. **Admitted corridor** — mutation only through an ADR-governed path + (math ratification via ADR-0172, chat turns via ADR-0146/0150). +2. **Explicit preconditions** — the UI shows what must be true before + mutation is allowed. +3. **Telemetry** — every mutation emits an auditable event. +4. **Replay evidence** — the operator can see replay-equivalence status + before acting. + +`RatificationCommandPanel.tsx` already implements this pattern for math +proposals. This is the template for future mutation surfaces, not an exception +to a "no mutation" rule. + +- [x] Record what already exists: `ratify_math_proposal`, `reject`, `defer` + in `workbench/api.py` lines 112+; `RatificationCommandPanel.tsx` with + precondition gates +- [x] No new mutation endpoints in Wave 1 beyond what exists + +--- + +## Wave 2 — Projections (after spine is live) + +Once the evidence spine exists, each remaining route becomes a projection. +These are parallelizable. + +### Packs Route + +- [ ] Backend: `GET /packs`, `GET /packs/{pack_id}` +- [ ] Frontend: pack list with verification badges, lexicon browser +- [ ] New primitive: `TreeView` (pack hierarchy) +- [ ] Pushes selected pack into evidence-subject context + +### Vault Route + +- [ ] Backend: `GET /vault/summary`, `GET /vault/entries`, + `GET /vault/entries/{entry_id}` +- [ ] Frontend: entry list with epistemic state badges, recall history +- [ ] Pushes selected entry into evidence-subject context + +### Audit Route + +- [ ] Backend: `GET /audit/events`, `GET /audit/events/{event_id}` +- [ ] Frontend: vertical event timeline, mutation boundary highlighting +- [ ] New primitive: `Timeline` +- [ ] Pushes selected event into evidence-subject context + +### Runs Route + +- [ ] Backend: `GET /runs`, `GET /runs/{session_id}` +- [ ] Frontend: session list with checkpoint badges, turn history +- [ ] Cross-links to Trace for any turn +- [ ] Pushes selected session into evidence-subject context + +### Settings Route + +- [ ] Frontend (mostly localStorage): inspector default, JSON depth, timestamp + format, API connection +- [ ] Runtime config display (read-only) +- [ ] No dangerous mutations — engine config changes require CLI + +--- + +## Wave 3 — Polish + Demo Theater + +### Existing module polish + +- [ ] Chat: multi-line composer, submission history, richer evidence strip +- [ ] Proposals: visual chain diagram, provenance links, metric deltas +- [ ] Eval Center: failure-first display, lane health overview, run progress +- [ ] Replay Theater: synchronized side-by-side diff, multi-artifact comparison + +### Demo Theater route + +- [ ] Backend: `GET /demos`, `POST /demos/{demo_id}/run`, + `GET /demos/{demo_id}/scenarios` +- [ ] Frontend: demo list, scenario results with evidence, "what this proves" / + "what this does not prove" honesty cards +- [ ] Evidence class badges: substrate-capability vs interface-contract +- [ ] "Proposer was wrong" scenarios visually highlighted + +--- + +## Keyboard map (global, shipped with Wave 1) + +| Shortcut | Action | +|---|---| +| `Cmd+K` | Command palette | +| `Cmd+I` | Toggle inspector | +| `Cmd+1`..`Cmd+0` | Navigate to route 1-10 | +| `j/k` or Up/Down | Navigate lists | +| `Enter` | Open selected item | +| `Esc` | Close drawer/palette/inspector | +| `/` | Focus search input | +| `?` | Show keyboard shortcut overlay | + +--- + +## Dependencies + +``` +Wave 1 (evidence spine) --> Wave 2 (projections, parallelizable) + --> Wave 3 (polish + demos) +``` + +Wave 1 must complete before Wave 2 work begins. Wave 2 routes are independent +of each other. Wave 3 can overlap with late Wave 2 work. + +--- + +## Explicit exclusions + +Per ADR-0160 doctrine and CLAUDE.md: + +- No multi-user auth, cloud deployment, or SaaS surface +- No animated "thinking" indicators or decorative motion +- No dashboard analytics walls +- No plugin/agent marketplace +- No corpus/pack mutation from the UI +- No mobile layout (engineering workstation only) +- No Deephaven, heavy JVM dependencies, or streaming database engines +- No framework upgrades (stays React 18 + stdlib HTTP) + +--- + +## Peer review record + +| Reviewer | Key contribution | +|---|---| +| Claude Opus 4.6 | Original 7-phase plan; synthesis into evidence spine after critique | +| Codex | Path selection: "evidence spine first" over routes-first; evidence chain as intrinsic UI manifold; mutation doctrine correction (admitted corridors, not "no buttons") | +| GPT5.5-Thinking | Trace honesty correction: workbench chat turns do not attach telemetry sink; `ChatTurnResult` content is not in runtime telemetry; solution = Workbench Turn Evidence Journal as honest read model |