Compare commits

...

5 commits

Author SHA1 Message Date
Shay
37b04d504b feat(workbench/evals): polish states, numeric alignment, lane URL state (wave 1d) 2026-05-28 08:11:06 -07:00
Shay
e954c660f7 Merge branch 'feat/workbench-ui-continuation' into polish-workbench-evals-flow 2026-05-28 08:07:23 -07:00
Shay
a4a24de836 docs(workbench): add wave-1 brief pack for Phase 1 polish
Four parallel-safe briefs covering Chat/Proposals/Replay/Evals polish,
each with operator assignment, dispatch line, doctrine refs, owned-files
scope, acceptance criteria, and validation commands.

Operator assignment:
- 1a Chat polish     -> Opus 4.6 (surface contract sensitive)
- 1b Proposals polish-> Sonnet 4.6 (tight-scope lifecycle)
- 1c Replay polish   -> Gemini 3.1 Pro (cross-file presentation)
- 1d Evals polish    -> GPT-OSS-120B / Copilot (mechanical)

All four are dispatchable simultaneously. No order dependency.
2026-05-28 07:58:44 -07:00
Shay
685aaae40e docs(workbench): add CORE-doctrine addendum to UI continuation plan
Binds Phase 5+ to per-route trust classification, runtime surface
contract obligations, audit fidelity guarantees, pack-mutation
proposal-only discipline, backend contract documentation, and the
CLAUDE.md PR Checklist questions.

Phases 1-4 remain safe to start; Phase 5+ gated on this addendum.
2026-05-28 07:46:15 -07:00
Shay
4d7c605256 docs(workbench): add temporary UI continuation plan 2026-05-28 07:29:02 -07:00
12 changed files with 1554 additions and 138 deletions

View file

@ -0,0 +1,112 @@
# Brief 1a — Chat route polish
> **Operator:** Opus 4.6 (foundations / surface-contract sensitive)
> **Branch:** `workbench/wave-1a-chat-polish`
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
> **Estimated diff:** ~150300 lines, UI only
> **Why this operator:** Chat is the one Phase 1 brief that touches the
> Runtime Surface Contract (`surface` / `walk_surface` / `articulation_surface`).
> Distinguishing those correctly under the addendum §2 is high-stakes.
---
## Dispatch line
```bash
git fetch origin && \
git worktree add /Users/kaizenpro/Projects/core-w1a-chat \
-b workbench/wave-1a-chat-polish origin/feat/workbench-ui-continuation && \
cd /Users/kaizenpro/Projects/core-w1a-chat/workbench-ui && \
pnpm install
```
Then open the worktree in your editor and start.
---
## Doctrine refs (read before touching code)
- `docs/plans/workbench-ui-continuation.md` §Phase 1a
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Chat trust class),
§2 (Trace surface contract — applies to Chat's trace drawer), §7 (PR checklist)
- `docs/runtime_contracts.md` — surface distinction and `trace_hash` stability
## Scope (owned files)
- `src/app/chat/**` — everything in this folder
- Test files colocated with the above
## May read (no edits)
- `src/types/api.ts` (append-only at the bottom of the chat section if
new types are needed)
- `src/api/queries.ts` (no edits; this brief does not add hooks)
## Off-limits
- `src/design/**`
- `src/app/Shell.tsx`, `App.tsx`, `LeftNav.tsx`, `TopBar.tsx`, `StatusFooter.tsx`
- Any backend Python file
- Any placeholder file in `src/routes/`
## Tasks
- [ ] **State audit.** Walk every async surface in `src/app/chat/` and confirm
it has loading, empty, and error states. Where missing, add them using
`EmptyState` from `src/design/`.
- [ ] **Surface distinction.** Inspect the trace drawer / evidence panel. The
three surfaces — `surface`, `walk_surface`, `articulation_surface`
must render as three distinct UI elements with labels the operator can
tell apart at a glance. If currently conflated, separate them.
- [ ] **Navigation hardening.** From a turn response:
- clicking `trace_hash` opens the trace drawer at that turn,
- any proposal candidate links to its detail in the Proposals route
(use the existing route path; do not invent a new one).
- [ ] **⌘K commands.** Confirm or add palette entries for `New chat session`
and `Jump to proposal <id>`. Entries belong to chat; do not edit other
features' entries.
- [ ] **Trace-hash stability test.** Add a test asserting that mounting the
trace drawer, navigating nodes, and unmounting does not mutate any
`trace_hash` value held in component state. This is the addendum §2
proof obligation.
- [ ] **Tests for new interaction paths.** Use the existing `Shell.test.tsx`
pattern: render under MemoryRouter + QueryClientProvider, drive
interactions through `@testing-library/react`.
## Acceptance criteria
- Every async surface in `src/app/chat/` has loading + empty + error states.
- Trace drawer labels `surface` / `walk_surface` / `articulation_surface`
distinctly. A reviewer should be able to point at each on screen.
- `trace_hash` stability test exists and passes.
- ⌘K entries for `New chat session` and `Jump to proposal` are present and
keyboard-activatable.
- No edits to off-limits files. `git diff --stat` shows only `src/app/chat/`
and (optionally) `src/types/api.ts` append-only.
- `pnpm test` and `pnpm test:enum-coverage` green.
## Validation
```bash
cd workbench-ui
pnpm install
pnpm test
pnpm test:enum-coverage
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
```
The diff-stat output must include only `src/app/chat/**` and optionally
appended lines in `src/types/api.ts`. Anything else is out of scope.
## PR
Title: `feat(workbench/chat): polish states, surface distinction, ⌘K (wave 1a)`
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
## When done
- [ ] PR open against `feat/workbench-ui-continuation`
- [ ] CI green
- [ ] Mark this brief complete by checking the box in this file in a follow-up
commit on the same PR (single source of truth: this file)

View file

@ -0,0 +1,111 @@
# Brief 1b — Proposals route polish
> **Operator:** Sonnet 4.6 (tight-scope route work)
> **Branch:** `workbench/wave-1b-proposals-polish`
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
> **Estimated diff:** ~150250 lines, UI only
> **Why this operator:** Proposals has a well-defined lifecycle
> (pending / accepted / rejected / withdrawn) and clear color tokens
> already in `src/design/`. Tight contract, no surface-contract surprises.
---
## Dispatch line
```bash
git fetch origin && \
git worktree add /Users/kaizenpro/Projects/core-w1b-proposals \
-b workbench/wave-1b-proposals-polish origin/feat/workbench-ui-continuation && \
cd /Users/kaizenpro/Projects/core-w1b-proposals/workbench-ui && \
pnpm install
```
---
## Doctrine refs
- `docs/plans/workbench-ui-continuation.md` §Phase 1b
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Proposals is
proposal-only — no direct accept/reject endpoints introduced; UI only
calls existing API), §7 (PR checklist)
## Scope (owned files)
- `src/app/proposals/**`
- Test files colocated with the above
## May read (no edits)
- `src/api/queries.ts` — observe existing `useProposals` / `useProposal` shapes
- `src/design/tokens.css` — read review-lifecycle tokens; do not edit
- `src/types/api.ts` — append-only at the bottom of proposals section if needed
## Off-limits
- `src/design/**` (including tokens)
- Shell / TopBar / LeftNav / StatusFooter / App.tsx
- Any backend Python file
- Any placeholder file
- Any new proposal-mutation endpoint or hook (proposal-only discipline)
## Tasks
- [ ] **State audit.** Confirm queue list and proposal detail handle loading,
empty, and error consistently. Wire `EmptyState` from `src/design/`
where missing.
- [ ] **Lifecycle color consistency.** The four states (`pending`,
`accepted`, `rejected`, `withdrawn`) must render with the same color
tokens in both list and detail. Verify each token is referenced via
CSS custom property — no hex literals.
- [ ] **Cross-route links.** From proposal detail:
- link to the source artifact (use existing Runs/artifact route path
if present; otherwise the artifact route Phase 2 will introduce —
for now link to the Replay-comparison entry-point if that exists),
- link to the replay-comparison view for the proposal if applicable.
Document any link that cannot land because the target route is a
placeholder; leave a TODO with the route name.
- [ ] **Suggested-CLI block.** Make the suggested-CLI snippet block
`user-select: text` (or a copy button) so the operator can copy it.
- [ ] **⌘K command entries.** Add `Open proposal <id>` palette entry
driven by the existing `useProposals` query. Only proposals entries —
do not touch other features'.
- [ ] **Route tests.** Cover at least:
- queue renders with loading → success transition,
- empty state when zero proposals returned,
- error state when query errors,
- lifecycle color class applied per state.
## Acceptance criteria
- Loading / empty / error covered everywhere in `src/app/proposals/`.
- All four lifecycle states render with the documented tokens, no hex
literals anywhere in this brief's diff (`grep -E '#[0-9a-fA-F]{3,6}'`
returns nothing in the diff).
- Suggested-CLI block is selectable or has a copy button.
- ⌘K entries for proposals are present and keyboard-activatable.
- No backend changes. No off-limits files touched.
- `pnpm test` and `pnpm test:enum-coverage` green.
## Validation
```bash
cd workbench-ui
pnpm install
pnpm test
pnpm test:enum-coverage
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
# Verify no hex literals introduced
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/proposals/**' | grep -E '^\+.*#[0-9a-fA-F]{3,6}' && echo "FAIL: hex literal introduced" || echo "ok: no hex literals"
```
## PR
Title: `feat(workbench/proposals): polish states, lifecycle colors, links (wave 1b)`
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
## When done
- [ ] PR open against `feat/workbench-ui-continuation`
- [ ] CI green
- [ ] This brief checked complete on the PR

View file

@ -0,0 +1,111 @@
# Brief 1c — Replay route polish
> **Operator:** Gemini 3.1 Pro (survey + cross-file presentation work)
> **Branch:** `workbench/wave-1c-replay-polish`
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
> **Estimated diff:** ~150250 lines, UI only
> **Why this operator:** Replay polish requires reading divergence
> rendering across multiple files and improving clarity of metadata
> presentation. Gemini's strength is cross-file survey + presentation
> reasoning over a moderate diff surface.
---
## Dispatch line
```bash
git fetch origin && \
git worktree add /Users/kaizenpro/Projects/core-w1c-replay \
-b workbench/wave-1c-replay-polish origin/feat/workbench-ui-continuation && \
cd /Users/kaizenpro/Projects/core-w1c-replay/workbench-ui && \
pnpm install
```
---
## Doctrine refs
- `docs/plans/workbench-ui-continuation.md` §Phase 1c
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Replay = read-only;
no "fix" buttons, no artifact writes), §7 (PR checklist)
- The Replay route should never present an affordance to mutate an
artifact. Read + render + link out only.
## Scope (owned files)
- `src/app/replay/**`
- Test files colocated with the above
## May read (no edits)
- `src/api/queries.ts` — observe `useReplayComparison`, `useArtifacts`, `useArtifact`
- `src/design/tokens.css` — divergence-severity tokens
- `src/types/api.ts` — append-only at the bottom of replay section if needed
## Off-limits
- `src/design/**`
- Shell / TopBar / LeftNav / StatusFooter / App.tsx
- Any backend Python file
- Any placeholder file
- **No mutating affordances of any kind** — no "rerun", "save", "fix",
"accept divergence" buttons. Read-only doctrine.
## Tasks
- [ ] **Survey current state.** Read every file in `src/app/replay/` and
produce a short comment block at the top of the route (or in this
brief's PR description) summarizing what each component does. This
forces a complete read before edits.
- [ ] **State audit.** Artifact-selector and comparison view: loading,
empty (no artifacts available; selection invalid), error states.
- [ ] **Divergence-severity clarity.** Currently the divergence severity
may be coded via color alone. Add a textual severity label next to
the color band (e.g. `low`, `material`, `breaking`) so the meaning
is unambiguous without the color legend.
- [ ] **Metadata presentation.** Artifact metadata (timestamp, trace_hash,
run id, lane) should be presented in a key-value table or definition
list with consistent alignment. No "wall of text" rendering.
- [ ] **Back-link to proposal.** When the comparison view is reached from
a proposal, show a "Back to proposal #N" link. When reached directly,
do not show the link. Drive this from URL state, not props.
- [ ] **Route tests.** Cover:
- artifact selection populating comparison view,
- empty state when no artifacts,
- error state when comparison query fails,
- severity label rendering for each severity bucket.
## Acceptance criteria
- Every async surface in `src/app/replay/` has loading + empty + error.
- Divergence severity is conveyed by **both** color **and** text label.
- Artifact metadata renders as a structured key-value layout.
- Back-link to proposal appears only when route was entered from a proposal.
- No mutating affordances anywhere in the diff.
- No backend changes. No off-limits files touched.
- `pnpm test` and `pnpm test:enum-coverage` green.
## Validation
```bash
cd workbench-ui
pnpm install
pnpm test
pnpm test:enum-coverage
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
# Confirm no write paths introduced (no new mutation hooks)
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/replay/**' | grep -E '^\+.*(useMutation|fetch\(|axios\.(post|put|patch|delete))' && echo "FAIL: mutation introduced" || echo "ok: read-only preserved"
```
## PR
Title: `feat(workbench/replay): polish states, severity labels, metadata layout (wave 1c)`
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
## When done
- [ ] PR open against `feat/workbench-ui-continuation`
- [ ] CI green
- [ ] This brief checked complete on the PR

View file

@ -0,0 +1,114 @@
# Brief 1d — Evals route polish
> **Operator:** GPT-OSS-120B or GitHub Copilot cloud agent (mechanical)
> **Branch:** `workbench/wave-1d-evals-polish`
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
> **Estimated diff:** ~120200 lines, UI only
> **Why this operator:** Evals polish is mostly mechanical: state audits,
> numeric column alignment, route tests. Clear acceptance criteria, narrow
> file scope, no surface-contract subtlety. Ideal for a local or
> cloud-mechanical operator.
---
## Dispatch line
```bash
git fetch origin && \
git worktree add /Users/kaizenpro/Projects/core-w1d-evals \
-b workbench/wave-1d-evals-polish origin/feat/workbench-ui-continuation && \
cd /Users/kaizenpro/Projects/core-w1d-evals/workbench-ui && \
pnpm install
```
---
## Doctrine refs
- `docs/plans/workbench-ui-continuation.md` §Phase 1d
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Evals = read-only;
no lane mutation, no "suppress failing case" affordance), §7 (PR checklist)
- Evals UI must not hide or suppress failing cases. Filters that change
view are fine; filters that remove failing cases from the underlying
count are not.
## Scope (owned files)
- `src/app/evals/**`
- Test files colocated with the above
## May read (no edits)
- `src/api/queries.ts``useEvalLanes`, `useEvalLane`, `useEvalExecution`
- `src/design/tokens.css` — read; do not edit
- `src/types/api.ts` — append-only at the bottom of evals section if needed
## Off-limits
- `src/design/**`
- Shell / TopBar / LeftNav / StatusFooter / App.tsx
- Any backend Python file
- Any placeholder file
- Any affordance that mutates a lane definition or hides a failing case
## Tasks
- [ ] **State audit.** Lane list and eval-run view: loading, empty
(no lanes; no runs for selected lane), error states.
- [ ] **Numeric column alignment.** Wherever numeric metrics render in a
list or table, they must be right-aligned, monospace-numeric, and
use tabular-nums for consistent digit width. Use existing tokens.
- [ ] **Metric readability.** Where a metric is shown alongside a target,
render `actual / target` with the actual value emphasized. If a
metric is failing (actual < target where higher-is-better, or
actual > target where lower-is-better), apply the existing
failure color token. No new tokens.
- [ ] **Lane selection persistence.** Selected lane should be reflected in
the URL (query string is fine). Reloading the page should restore the
selection.
- [ ] **⌘K command entry.** Add `Open eval lane <name>` palette entries
driven by the existing `useEvalLanes` query. Only evals entries.
- [ ] **Route tests.** Cover:
- lane list loading → success,
- empty state when no lanes,
- run view rendering metrics with correct alignment,
- failure color applied when metric does not meet target.
## Acceptance criteria
- Every async surface in `src/app/evals/` has loading + empty + error.
- All numeric metric columns are right-aligned with `tabular-nums`.
- Failing metrics use the existing failure token, no hex literals.
- Lane selection survives reload via URL state.
- ⌘K `Open eval lane` entries present and keyboard-activatable.
- No backend changes. No off-limits files touched. No affordance that
suppresses or hides failing cases.
- `pnpm test` and `pnpm test:enum-coverage` green.
## Validation
```bash
cd workbench-ui
pnpm install
pnpm test
pnpm test:enum-coverage
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
# No hex literals in the diff
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/evals/**' | grep -E '^\+.*#[0-9a-fA-F]{3,6}' && echo "FAIL: hex literal introduced" || echo "ok: no hex literals"
# No mutation hooks introduced
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/evals/**' | grep -E '^\+.*useMutation' && echo "FAIL: mutation introduced" || echo "ok: read-only preserved"
```
## PR
Title: `feat(workbench/evals): polish states, numeric alignment, lane URL state (wave 1d)`
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
## When done
- [ ] PR open against `feat/workbench-ui-continuation`
- [ ] CI green
- [ ] This brief checked complete on the PR

View file

@ -0,0 +1,110 @@
# Workbench UI — Wave 1 Brief Pack
> **Parent plan:** `docs/plans/workbench-ui-continuation.md`
> **Binding addendum:** `docs/plans/workbench-ui-continuation-addendum.md`
> **Base branch:** `feat/workbench-ui-continuation` @ `685aaae`
> **Scope:** Phase 1 only — polish the four implemented routes
## What this wave does
Polish Chat, Proposals, Replay, and Evals so they are cohesive,
cross-linked, and handle loading/empty/error states consistently —
before any new route lands. All four routes are already implemented;
this is hardening, not feature work.
## Parallel safety
| Operator brief | Owns | Read-only touches |
|-----------------------|-----------------------------------|----------------------------------|
| 1a — Chat polish | `src/app/chat/**` | `src/types/api.ts` (append only) |
| 1b — Proposals polish | `src/app/proposals/**` | `src/types/api.ts` (append only) |
| 1c — Replay polish | `src/app/replay/**` | `src/types/api.ts` (append only) |
| 1d — Evals polish | `src/app/evals/**` | `src/types/api.ts` (append only) |
**Off-limits to every brief:**
- `src/design/**` (protected substrate — plan constraint)
- `src/app/Shell.tsx`, `LeftNav.tsx`, `TopBar.tsx`, `StatusFooter.tsx`
- `src/app/App.tsx` route table (no swaps in Wave 1)
- Any placeholder file in `src/routes/` (those belong to Phase 2+)
- `workbench-ui/src/design/tokens.css` and friends
- Backend Python (this is a UI-only wave)
**Coordinated surface:**
- CommandPalette entries: each brief adds **only** entries for its
own feature; no brief edits another's entries. If a conflict appears
at merge, the second-merging brief rebases additively.
- `src/types/api.ts`: append-only additions to the bottom of the
matching section. No edits to existing types.
## Dispatch DAG
```
┌── 1a Chat polish (Opus 4.6) ───────┐
main ──>│ │── merge any order
│── 1b Proposals polish (Sonnet 4.6)─│ (all four parallel-safe)
│ │
│── 1c Replay polish (Gemini 3.1) ───│
│ │
└── 1d Evals polish (GPT-OSS/Copilot)┘
```
All four briefs are dispatchable simultaneously. No order dependency.
## Doctrine that applies to every brief
From `docs/plans/workbench-ui-continuation-addendum.md`:
- **§1 trust classification.** Chat = read + turn-submit; Proposals = proposal-only;
Replay = read-only; Evals = read-only. No brief introduces a write path.
- **§2 surface contract (Chat brief only).** `surface`, `walk_surface`, and
`articulation_surface` rendered distinctly; `trace_hash` stability test required.
- **§7 per-PR checklist.** Every PR answers the four CLAUDE.md §PR Checklist
questions in its description.
- **§8 proof obligations.** New TS unions must have a test that fails under
their named violation, or the type is decoration.
## Validation each brief must run
```bash
cd workbench-ui
pnpm install # if first time on this worktree
pnpm test # all green
pnpm test:enum-coverage
```
Backend-touching changes are **out of scope** for Wave 1 and must be
rejected at review. Any "while I'm here" backend edit means the PR
fails the trust-classification check.
## PR template (every brief uses this)
```markdown
## Wave 1 brief: <1a|1b|1c|1d><feature> polish
### Trust classification
<read-only | read + turn-submit | proposal-only>
### CLAUDE.md §PR Checklist
- Capability/property/boundary added: <one line>
- Invariant preserved: <one line, e.g. trace_hash stability, no design-substrate edit>
- CLI/test lane that proves it: pnpm test + pnpm test:enum-coverage (UI-only)
- Trust boundary enforced: <one line; "no write paths introduced">
### Diff scope
- Owned: <files>
- Read-only touched: <files>
- Off-limits respected: yes
### Tests
- Added: <tests>
- Passing: pnpm test green; pnpm test:enum-coverage green
```
## Done criteria for the wave
- [ ] All four PRs merged to `feat/workbench-ui-continuation`
- [ ] No off-limits files modified across the merged set
- [ ] No backend changes across the merged set
- [ ] `pnpm test` and `pnpm test:enum-coverage` green on the merged tip
- [ ] CommandPalette has entries for the new interactions added by each brief
- [ ] All four briefs in this folder marked done in their respective files

View file

@ -0,0 +1,190 @@
# Workbench UI Continuation — CORE-Doctrine Addendum
> **Status:** binding addendum to `docs/plans/workbench-ui-continuation.md`
> **Applies to:** all Phase 5+ work; back-applies to Phases 14 where noted
> **Authority:** CLAUDE.md (§Security, §Teaching Safety, §Runtime Surface Contract, §PR Checklist, §Schema-Defined Proof Obligations), `docs/runtime_contracts.md`, ADR-0051
The parent plan is sound as frontend hygiene. This addendum supplies the
trust-boundary, determinism, and proof-obligation language CORE requires before
any route that touches packs, vault, audit, settings, or trace can ship.
---
## 1. Per-route trust classification
Every route must declare one classification before its first implementation PR.
The classification gates what the UI is allowed to do.
| Route | Classification | Allowed UI operations | Forbidden |
|-------------|------------------|-----------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
| Chat | read + turn-submit | Submit turn; render `surface`; expose `walk_surface` / `articulation_surface` distinctly | Mutating identity axes, runtime policy, or operator code from user text |
| Proposals | proposal-only | View queue/detail; accept/reject through existing proposal API | Bypassing review; direct pack/vault mutation |
| Replay | read-only | Compare artifacts; render divergence | Writing artifacts; rewriting trace_hash; "fix" buttons |
| Evals | read-only | Browse lanes/runs/metrics | Mutating lane definitions; suppressing failing cases from view |
| Runs | read-only | List/filter artifacts; deep-link to Replay | Deletion; metadata edits |
| Trace | read-only | Render three surfaces distinctly; node-by-node walk | Surface conflation; any write path |
| Inspector | read-only | Display selected entity by type | Inline edits; cross-route mutation |
| Vault | read-only | Hierarchical browse; grounding-source coloring | Entry creation/edit/delete; recall-semantic changes |
| Packs | proposal-only | List; inspect contents; **propose** re-ingestion via existing CLI/proposal path | Direct re-ingestion buttons; pack_id strings reaching the filesystem without `_validate_pack_id` |
| Audit | read-only | Browse `TurnVerdicts`; filter by clearance state; paginate | Aggregating/suppressing per-turn verdicts; "mark resolved" |
| Settings | mutating (allowlisted) | Read/write only fields on the allowlist below | Any setting that affects closure thresholds, normalization sites, recall semantics, or identity packs |
### Settings allowlist (initial)
The Settings route may read **and** write only these fields. Anything outside
this list is read-only or absent from the UI entirely.
- API endpoint override (UI ↔ backend wiring only)
- Operator display preferences (theme, density, default route)
- Telemetry sink toggles already exposed by `chat/telemetry.py`
- Eval lane selection / default suite
Out of scope for the UI (operator CLI only):
- `versor_condition` threshold (1e-6, non-negotiable)
- Algebra backend selection
- Identity pack selection / `DEFAULT_IDENTITY_PACK`
- Safety/ethics pack mounting
- Vault recall thresholds and indexing parameters
- Any normalization site (`ingest/gate.py`, `language_packs/compiler.py`, `algebra/versor.py`)
Adding a field to the allowlist requires an ADR.
---
## 2. Runtime Surface Contract obligations (Phase 3 — Trace route)
The Trace route is the surface where doctrine violations are easiest to commit
silently. Three obligations:
1. **Distinct rendering.** `surface`, `walk_surface`, and `articulation_surface`
are visually and semantically distinct in the UI. They are not collapsed into
a single "response" pane.
2. **Read-only.** No node, no edge, no turn is editable in v1. Drill-down to
evidence is fine; mutation is not.
3. **Trace-hash stability.** Rendering must not depend on or alter values that
feed `trace_hash`. The Trace route PR must include a contract test asserting
`trace_hash` is byte-identical before and after the route fetches and renders
a session.
Reference: `docs/runtime_contracts.md`. Any change to the surface contract lands
the doc update and contract test in the same PR (per CLAUDE.md §Runtime Surface
Contract).
---
## 3. Audit fidelity guarantee (Phase 6 — Audit route)
The Audit route is the operator's window into `SafetyCheck` / `EthicsCheck`
verdicts. It is load-bearing for catching regressions in fail-closed safety
behavior. Therefore:
- Per-turn verdict cardinality is preserved. Pagination is allowed; aggregation
that hides individual verdicts is not.
- Clearance state coloring uses the existing tokens (`cleared`, `violated`,
`unassessable`, `suppressed`) without remapping or merging.
- No UI affordance "resolves," "suppresses," or "dismisses" a verdict. Operator
response to verdicts happens in the engine, not the workbench.
- Search/filter is permitted; saved views are permitted; mutation is not.
Reference: `audit-completeness`, `TurnVerdicts`, CLAUDE.md §Teaching Safety.
---
## 4. Pack route discipline (Phase 6 — Packs)
`language_packs/compiler.py` is one of three allowed normalization sites and is
guarded by `_validate_pack_id` (ADR-0051) against path traversal.
- Pack re-ingestion is **proposal-only** in the UI. The UI may surface a
"propose re-ingestion" affordance; it may not invoke the compiler directly.
- Any `pack_id` displayed in the UI is treated as untrusted display data:
centralized safe-display path, no filesystem operations driven by UI input.
- Pack mutation proposals follow the existing reviewed-teaching pipeline.
No parallel correction/learning path (CLAUDE.md §Teaching Safety).
---
## 5. Backend contract documentation
"Confirm with backend" (parent plan, Phase 5) is replaced by:
- Every new endpoint consumed by a workbench route is documented in
`docs/runtime_contracts.md` (or a named sibling under `docs/contracts/`) in
the same PR that introduces the TypeScript mirror in `src/types/api.ts`.
- The Python handler and the TS mirror cite the same contract section.
- Contract drift between Python and TS is a CI failure, not a code-review
catch — add a parity test if one does not exist.
---
## 6. CLI-lane requirement
Any PR that adds or modifies a backend endpoint to support a workbench route
must include the CLI lane that exercises it:
- Runtime-touching: `core test --suite runtime`
- Cognition-touching: `core eval cognition`
- Pack-touching: `core test --suite packs`
- Teaching-touching: `core test --suite teaching`
UI-only PRs (no backend change) are exempt — `pnpm test` is sufficient.
---
## 7. Per-PR checklist (replaces the parent plan's "Validation checklist" for Phase 5+)
```text
[ ] Route trust classification matches §1 of this addendum
[ ] If Settings: every writable field is on the §1 allowlist (or an ADR adds it)
[ ] If Trace: trace_hash stability test included; three surfaces rendered distinctly
[ ] If Audit: no aggregation/suppression of per-turn verdicts
[ ] If Packs: re-ingestion is proposal-only; pack_id flows through safe display
[ ] New backend endpoints documented in docs/runtime_contracts.md (or sibling)
[ ] TS mirror in src/types/api.ts cites the same contract section
[ ] Relevant CLI lane added or extended (§6); pnpm test green
[ ] PR description answers the four CLAUDE.md §PR Checklist questions:
- What capability/property/boundary did this add/protect?
- Which invariant proves the field remains valid?
- Which CLI suite/eval proves the lane?
- What trust boundary was enforced?
```
---
## 8. Proof-obligation discipline (CLAUDE.md §Schema-Defined Proof Obligations)
Schemas and types added to `src/types/api.ts` that nominally guarantee a
structural property (e.g. a discriminated union for verdict states, a typed
proposal lifecycle) carry the same obligation as backend schemas: an executing
test must fail under the violations the type is written to catch.
If no such test exists, the type is decoration. Record the gap in the PR
description rather than treating the type as load-bearing.
---
## 9. Phase applicability
| Phase | Addendum sections that apply |
|--------------------------------------|-------------------------------------|
| 1 — Polish existing routes | §2 (Chat surface distinction), §7 |
| 2 — Runs | §1 (read-only), §7 |
| 3 — Trace | §1, §2, §7 |
| 4 — Inspector | §1 (read-only), §7 |
| 5 — Contracts for placeholders | §1, §5, §7 |
| 6 — Packs / Vault / Audit / Settings | §1, §3, §4, §5, §6, §7, §8 |
| 7 — Command palette | §1 (no command bypasses §1 rules) |
---
## 10. Delete-when-done
This addendum is deleted together with the parent plan, after all phases merge
and its rules have been folded into:
- `docs/runtime_contracts.md` (surface contract additions)
- ADRs for any Settings-allowlist extensions
- Workbench-specific contributing notes if needed
Until then, both files travel together.

View file

@ -0,0 +1,246 @@
# Workbench UI — Continuation Plan
> **Status:** temporary working plan — delete when all phases are merged
> **Scope:** `workbench-ui/` only
> **Tracks against:** current `main` @ `89defef`
---
## Context
The CORE Workbench is a React + Vite + TanStack Query frontend that serves as the
primary operator interface for the CORE engine. The shell is structurally sound.
Four feature routes are implemented. Six routes remain placeholders. The right
inspector is a stub kept collapsed. This plan sequences the remaining work.
### What exists today
| Area | Status | Notes |
|---|---|---|
| Shell grid (TopBar / LeftNav / StatusFooter) | ✅ Live | Token-driven, tested |
| Design token substrate (`src/design/`) | ✅ Complete | **Do not expand** except `EmptyState` / `CommandPalette` |
| CommandPalette (⌘K) | ✅ Live | Wired in TopBar |
| ApiErrorBoundary | ✅ Live | Wraps `<Outlet />` |
| Chat route | ✅ Implemented | `src/app/chat/` — turn flow, trace drawer, evidence |
| Proposals route | ✅ Implemented | `src/app/proposals/` — queue, detail, review lifecycle |
| Replay route | ✅ Implemented | `src/app/replay/` — artifact comparison |
| Evals route | ✅ Implemented | `src/app/evals/` — lanes, runs, metrics |
| Trace route | 🔴 Placeholder | `src/routes/TraceRoutePlaceholder.tsx` |
| Runs route | 🔴 Placeholder | `src/routes/RunsRoutePlaceholder.tsx` |
| Packs route | 🔴 Placeholder | `src/routes/PacksRoutePlaceholder.tsx` |
| Vault route | 🔴 Placeholder | `src/routes/VaultRoutePlaceholder.tsx` |
| Audit route | 🔴 Placeholder | `src/routes/AuditRoutePlaceholder.tsx` |
| Settings route | 🔴 Placeholder | `src/routes/SettingsRoutePlaceholder.tsx` |
| RightInspector | 🔴 Stub | Hardcoded collapsed in `Shell.tsx` (W-027) |
### Existing query hooks (api/queries.ts)
- `useRuntimeStatus``/runtime/status`
- `useChatTurn``/chat/turn`
- `useArtifacts` — artifact listing
- `useArtifact` — artifact detail
- `useReplayComparison``/replay/compare`
- `useProposals` — proposals listing
- `useProposal` — proposal detail
- `useEvalLanes` — eval lanes listing
- `useEvalLane` — eval lane detail
- `useEvalExecution` — eval run execution
**No hooks yet exist** for: runs, trace sessions, packs, vault, audit log, or settings.
---
## Constraints
- Documentation stays in Markdown, never HTML dashboards.
- `src/design/` is a protected Branch 1 substrate. Only `EmptyState` and `CommandPalette` may be edited there.
- New UI pieces go feature-local: `src/app/<feature>/` or `src/routes/<Feature>Route.tsx`.
- Zero hardcoded hex values — always reference CSS custom properties from `tokens.css`.
- No new state management libraries. Prefer TanStack Query + React `useState` / `useReducer` + `useContext` where needed.
- All new routes need at minimum a smoke test and a loading/empty/error state.
- Schema/type mirrors in `src/types/api.ts` must stay in sync with the Python backend.
- PR discipline: small, load-bearing, each with tests. Follow existing ADR checklist.
---
## Implementation Sequence
### Phase 1 — Polish the four implemented routes
**Branch prefix:** `workbench/WIP-1-*`
**Goal:** Make Chat, Proposals, Replay, and Evals cohesive and cross-linked before adding new surfaces.
#### 1a — Chat
- [ ] Audit loading / empty / error states for consistency
- [ ] Improve trace-hash and proposal-candidate navigation from within a turn response
- [ ] Verify ⌘K command entries exist for `New chat session` and `Jump to proposal`
- [ ] Extend `Shell.test.tsx`-style tests for any new interaction paths
#### 1b — Proposals
- [ ] Audit queue list loading / empty / error states
- [ ] Strengthen proposal detail: add direct link to its source artifact and replay-comparison entry
- [ ] Expose review-lifecycle color tokens (`review-pending`, `accepted`, `rejected`, `withdrawn`) consistently across list and detail views
- [ ] Ensure suggested-CLI block is selectable / copyable
#### 1c — Replay
- [ ] Audit artifact-selector loading / empty / error states
- [ ] Improve divergence-severity and metadata presentation clarity
- [ ] Add a direct link from replay comparison back to the originating proposal where applicable
- [ ] Add route tests for primary comparison path
#### 1d — Evals
- [ ] Audit lane list and eval run view states
- [ ] Improve metric readability — consider aligned numeric columns
- [ ] Add route tests for lane selection and run inspection
---
### Phase 2 — Build the Runs route
**Branch prefix:** `workbench/WIP-2-runs`
**Goal:** Replace `RunsRoutePlaceholder.tsx` with a real first-version route.
**Why this is next:** The existing `useArtifacts` and `useArtifact` query hooks already provide the needed read paths. Runs is the closest unfinished route to having backend support.
- [ ] Create `src/app/runs/` directory
- [ ] Build `RunsRoute.tsx`: artifact listing table — sortable by date, filterable by status
- [ ] Build `RunDetail.tsx`: artifact detail panel — metadata, trace hash, proposal links, replay entry-point
- [ ] Wire deep link from RunDetail → Replay comparison
- [ ] Add loading / empty / error states
- [ ] Update `App.tsx` to swap placeholder for the real route
- [ ] Add route tests
---
### Phase 3 — Build the Trace route
**Branch prefix:** `workbench/WIP-3-trace`
**Goal:** Replace `TraceRoutePlaceholder.tsx` with a first-version standalone trace explorer built by promoting and reusing the chat trace model.
**Why this is next:** Chat already contains trace-oriented UI logic. Trace should be grown by promotion, not by inventing a separate graph system from scratch.
- [ ] Audit what `src/app/chat/` already exposes for trace viewing
- [ ] Extract or adapt the trace model into a reusable `src/app/trace/` home
- [ ] Build `TraceRoute.tsx`: trace session list / selector
- [ ] Build `TraceDetail.tsx`: node-by-node state exploration using existing state-color tokens (`decoded`, `inferred`, `evidenced`, `contradicted`, etc.)
- [ ] Read-only first — no mutations in v1
- [ ] Add `useTraceSession` and `useTraceNode` query hooks to `queries.ts` once backend contract is confirmed
- [ ] Update `App.tsx` to swap placeholder
- [ ] Add route tests
---
### Phase 4 — Activate the RightInspector
**Branch prefix:** `workbench/WIP-4-inspector`
**Goal:** Replace the inspector stub and the hardcoded `inspectorCollapsed = true` in `Shell.tsx` with a real, read-only detail surface.
**Why this comes after Runs + Trace:** Entity shapes must be concrete before the inspector can meaningfully propagate them. Doing this first would mean speculating on schema.
- [ ] Add a minimal selection model: a `useInspectorStore` context/reducer that routes can publish into
- [ ] Lift the inspector toggle into `TopBar` (icon button, rightmost, `aria-expanded`)
- [ ] Replace `const inspectorCollapsed = true` with reactive state wired to the store
- [ ] Implement `RightInspector.tsx` — reads selected entity from the store, renders detail by entity type
- [ ] Support initial entity types: `artifact`, `proposal`, `trace-node`, `replay-diff`
- [ ] Read-only in v1
- [ ] Add tests for toggle behavior and entity rendering
---
### Phase 5 — Define contracts for remaining placeholder routes
**Branch prefix:** `workbench/WIP-5-contracts`
**Goal:** Before building Packs, Vault, Audit, and Settings, confirm Python schema shapes and add TypeScript mirrors.
**These routes are still true placeholders with no corresponding query hooks. Contract work precedes UI depth.**
#### Packs
- [ ] Confirm pack listing and pack-detail API shapes with backend
- [ ] Add `usePackList` and `usePack` hooks to `queries.ts`
- [ ] Add types to `src/types/api.ts`
#### Vault
- [ ] Confirm vault entry listing and entry-detail API shapes
- [ ] Add `useVaultEntries` and `useVaultEntry` hooks
- [ ] Add types to `src/types/api.ts`
#### Audit
- [ ] Confirm clearance-event log API shape (pagination, filters)
- [ ] Add `useAuditLog` hook
- [ ] Add types to `src/types/api.ts`
#### Settings
- [ ] Enumerate mutable runtime settings and confirm read/write API contract
- [ ] Decide which settings are operator-only (read only in UI) vs. writable from the workbench
- [ ] Add `useSettings` and `useUpdateSettings` hooks
---
### Phase 6 — Implement remaining routes
**Branch prefix:** `workbench/WIP-6-*`
**Goal:** One route per PR, in this priority order based on anticipated backend readiness.
1. **Vault** — hierarchical grounded knowledge browser; uses grounding-source color tokens (`grounding-teaching`, `grounding-pack`, `grounding-vault`, `grounding-oov`, etc.)
2. **Packs** — pack management: list, inspect contents, trigger re-ingestion; piggybacks on vault color system
3. **Audit** — temporal clearance-event log; uses clearance tokens (`cleared`, `violated`, `unassessable`, `suppressed`); time-sorted, paginated table
4. **Settings** — runtime config surface: model selection, threshold sliders, API endpoint override; write paths gated behind explicit confirmation
Each must have:
- [ ] Loading / empty / error states
- [ ] Route tests
- [ ] Updated `App.tsx` wiring (placeholder swap)
- [ ] ⌘K command entries
---
### Phase 7 — ⌘K command registry completion
**Branch prefix:** `workbench/WIP-7-command-palette`
**Goal:** Fill the CommandPalette command registry with live, route-aware entries.
- [ ] Route navigation: all 10 nav items accessible via ⌘K
- [ ] Context-sensitive: `Jump to run #N`, `Open vault entry`, `Inspect proposal`, `Compare artifacts`
- [ ] Live search over recent runs and sessions via existing query hooks
- [ ] Keyboard-first: full arrow-key navigation, Escape to close, Enter to activate
---
## Validation checklist (per PR)
```
cd workbench-ui
pnpm test
pnpm test:enum-coverage
```
Before merging a placeholder-swap PR:
- [ ] Old placeholder file removed from `src/routes/`
- [ ] New route wired in `App.tsx`
- [ ] New route has at minimum: loading state, empty state, error state, one smoke test
- [ ] No hardcoded hex values
- [ ] No new dependencies added without discussion
---
## Done criteria
This plan is complete when:
- [ ] All four existing routes are cohesive, cross-linked, and consistently handle all async states
- [ ] `Runs` is a live route backed by artifact/replay query hooks
- [ ] `Trace` is a live route promoted from the chat trace model
- [ ] `RightInspector` is a real, toggleable, read-only inspection surface
- [ ] `Packs`, `Vault`, `Audit`, `Settings` have confirmed backend contracts and TypeScript mirrors
- [ ] All six placeholder files in `src/routes/` are replaced and deleted
- [ ] ⌘K command registry covers all 10 routes and key entity-jump commands
- [ ] `src/design/` remains untouched except for `EmptyState` / `CommandPalette` allowed edits
- [ ] Every route has tests
---
*Delete this file once all items above are checked and the final PR in each phase is merged.*

View file

@ -5,7 +5,7 @@ export function EvalMetricGrid({
}: {
metrics: Record<string, unknown>;
}) {
const sortedKeys = Object.keys(metrics).sort();
const allKeys = Object.keys(metrics);
function formatValue(value: unknown): string {
if (typeof value === "boolean") {
@ -34,6 +34,20 @@ export function EvalMetricGrid({
return undefined;
}
function isLowerIsBetter(key: string): boolean {
const k = key.toLowerCase();
return (
k.includes("latency") ||
k.includes("duration") ||
k.includes("time") ||
k.includes("error") ||
k.includes("fail") ||
k.includes("fabrication") ||
k.includes("divergence") ||
k.includes("cost")
);
}
function renderPassFailBadge(key: string, value: unknown) {
const isBool = typeof value === "boolean";
const k = key.toLowerCase();
@ -65,33 +79,159 @@ export function EvalMetricGrid({
return null;
}
// Find pairs
const pairedKeys = new Set<string>();
const pairs: Array<{
key: string;
actual: unknown;
target: unknown;
}> = [];
// 1. Group "passed" and "total"
if (allKeys.includes("passed") && allKeys.includes("total")) {
pairs.push({
key: "passed",
actual: metrics["passed"],
target: metrics["total"],
});
pairedKeys.add("passed");
pairedKeys.add("total");
}
// 2. Group "X" and "X_target" or "target_X"
allKeys.forEach((k) => {
if (pairedKeys.has(k)) return;
if (k.endsWith("_target")) {
const baseKey = k.slice(0, -7);
if (allKeys.includes(baseKey)) {
pairs.push({
key: baseKey,
actual: metrics[baseKey],
target: metrics[k],
});
pairedKeys.add(baseKey);
pairedKeys.add(k);
}
} else if (k.startsWith("target_")) {
const baseKey = k.slice(7);
if (allKeys.includes(baseKey)) {
pairs.push({
key: baseKey,
actual: metrics[baseKey],
target: metrics[k],
});
pairedKeys.add(baseKey);
pairedKeys.add(k);
}
}
});
// 3. Remaining singles
const singles: Array<{ key: string; value: unknown }> = [];
allKeys.forEach((k) => {
if (!pairedKeys.has(k)) {
singles.push({ key: k, value: metrics[k] });
}
});
interface DisplayMetric {
key: string;
isPair: boolean;
actual: unknown;
target?: unknown;
value?: unknown;
}
const list: DisplayMetric[] = [
...pairs.map((p) => ({ key: p.key, isPair: true, actual: p.actual, target: p.target })),
...singles.map((s) => ({ key: s.key, isPair: false, actual: s.value, value: s.value })),
];
// Sort lexicographically
list.sort((a, b) => a.key.localeCompare(b.key));
return (
<div className="grid gap-3 sm:grid-cols-2 md:grid-cols-3" data-testid="eval-metric-grid">
{sortedKeys.map((key) => {
const val = metrics[key];
{list.map((item) => {
const key = item.key;
const unit = getUnit(key);
const badge = renderPassFailBadge(key, val);
const formatted = formatValue(val);
const isObj = typeof val === "object" && val !== null;
return (
<div
key={key}
className="flex flex-col justify-between rounded-lg border border-[var(--color-border-subtle)] bg-[var(--color-surface-raised)] p-3 shadow-sm"
data-testid="metric-card"
>
<div>
<div className="text-[10px] font-semibold text-[var(--color-text-muted)] uppercase tracking-wider truncate" title={key}>
{key.replaceAll("_", " ")}
</div>
<div className={`mt-1 font-mono text-sm font-semibold text-[var(--color-text-primary)] ${isObj ? "whitespace-pre overflow-x-auto text-[10px] bg-[var(--color-surface-inset)] p-1.5 rounded" : ""}`}>
{formatted}
{unit && <span className="ml-1 text-xs text-[var(--color-text-muted)] font-sans font-normal">{unit}</span>}
if (item.isPair) {
const actualVal = item.actual;
const targetVal = item.target;
const badge = renderPassFailBadge(key, actualVal);
const actualNum = Number(actualVal);
const targetNum = Number(targetVal);
const hasNumericValues = !isNaN(actualNum) && !isNaN(targetNum);
const isFailing =
hasNumericValues &&
(isLowerIsBetter(key) ? actualNum > targetNum : actualNum < targetNum);
const formattedActual = formatValue(actualVal);
const formattedTarget = formatValue(targetVal);
return (
<div
key={key}
className="flex flex-col justify-between rounded-lg border border-[var(--color-border-subtle)] bg-[var(--color-surface-raised)] p-3 shadow-sm"
data-testid="metric-card"
>
<div>
<div className="text-[10px] font-semibold text-[var(--color-text-muted)] uppercase tracking-wider truncate" title={key}>
{key.replaceAll("_", " ")}
</div>
<div className="mt-1 font-mono text-sm font-semibold text-right tabular-nums">
<span
className={
isFailing
? "text-[var(--color-state-contradicted)] font-bold"
: "text-[var(--color-text-primary)]"
}
>
{formattedActual}
</span>
<span className="text-[var(--color-text-muted)] mx-1">/</span>
<span className="text-[var(--color-text-secondary)] font-normal">
{formattedTarget}
</span>
{unit && <span className="ml-1 text-xs text-[var(--color-text-muted)] font-sans font-normal">{unit}</span>}
</div>
</div>
{badge && <div className="mt-2 flex justify-start">{badge}</div>}
</div>
{badge && <div className="mt-2 flex justify-start">{badge}</div>}
</div>
);
);
} else {
const val = item.value;
const badge = renderPassFailBadge(key, val);
const formatted = formatValue(val);
const isObj = typeof val === "object" && val !== null;
const isNum = typeof val === "number";
return (
<div
key={key}
className="flex flex-col justify-between rounded-lg border border-[var(--color-border-subtle)] bg-[var(--color-surface-raised)] p-3 shadow-sm"
data-testid="metric-card"
>
<div>
<div className="text-[10px] font-semibold text-[var(--color-text-muted)] uppercase tracking-wider truncate" title={key}>
{key.replaceAll("_", " ")}
</div>
<div
className={`mt-1 font-mono text-sm font-semibold text-[var(--color-text-primary)] ${
isNum ? "text-right tabular-nums" : ""
} ${isObj ? "whitespace-pre overflow-x-auto text-[10px] bg-[var(--color-surface-inset)] p-1.5 rounded" : ""}`}
>
{formatted}
{unit && <span className="ml-1 text-xs text-[var(--color-text-muted)] font-sans font-normal">{unit}</span>}
</div>
</div>
{badge && <div className="mt-2 flex justify-start">{badge}</div>}
</div>
);
}
})}
</div>
);

View file

@ -12,10 +12,33 @@ import { LoadingState } from "../../design/components/states/LoadingState";
import type { EvalRunResult } from "../../types/api";
import { WorkbenchApiError } from "../../api/client";
import { useState, useEffect } from "react";
import { useSearchParams } from "react-router-dom";
import { useEvalLanes, useEvalLane } from "../../api/queries";
import { EvalLaneCard } from "./EvalLaneCard";
import { EvalRunButton } from "./EvalRunButton";
import { EvalMetricGrid } from "./EvalMetricGrid";
import { EvalFailureViewer } from "./EvalFailureViewer";
import { EvalArtifactLink } from "./EvalArtifactLink";
import { EmptyState } from "../../design/components/states/EmptyState";
import { ErrorState } from "../../design/components/states/ErrorState";
import { LoadingState } from "../../design/components/states/LoadingState";
import type { EvalRunResult } from "../../types/api";
import { WorkbenchApiError } from "../../api/client";
import { useQueryClient } from "@tanstack/react-query";
export function EvalsRoute() {
const { data: lanes, isLoading, isError, error } = useEvalLanes();
const [searchParams, setSearchParams] = useSearchParams();
const selectedLaneName = searchParams.get("lane") || "";
const queryClient = useQueryClient();
const {
data: lastRunResult,
isLoading: isLaneLoading,
isError: isLaneError,
error: laneError,
} = useEvalLane(selectedLaneName);
// Maintain per-lane run states (pending, result, error)
const [runStates, setRunStates] = useState<
@ -29,6 +52,13 @@ export function EvalsRoute() {
>
>({});
// Auto-select the first lane on load if no selection is in the URL
useEffect(() => {
if (!selectedLaneName && lanes && lanes.length > 0) {
setSearchParams({ lane: lanes[0].lane }, { replace: true });
}
}, [selectedLaneName, lanes, setSearchParams]);
if (isLoading) {
return <LoadingState label="Loading eval lanes..." />;
}
@ -46,6 +76,178 @@ export function EvalsRoute() {
const selectedLane = lanes?.find((l) => l.lane === selectedLaneName) || null;
const currentRunState = selectedLaneName ? runStates[selectedLaneName] : null;
const isNotFoundError =
laneError instanceof WorkbenchApiError && laneError.code === "not_found";
function renderHeaderAndForm(lane: typeof selectedLane) {
if (!lane) return null;
return (
<>
{/* Header info */}
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-[var(--color-border-subtle)] pb-3">
<div>
<h2 className="text-lg font-semibold text-[var(--color-text-primary)] font-mono">
{lane.lane}
</h2>
{lane.description && (
<p className="mt-1 text-xs text-[var(--color-text-secondary)]">
{lane.description}
</p>
)}
</div>
</div>
{/* Run Button configuration if read-only */}
{lane.read_only ? (
<EvalRunButton
lane={lane}
onRunStart={() => {
setRunStates((prev) => ({
...prev,
[lane.lane]: { isPending: true },
}));
}}
onRunSuccess={(result) => {
setRunStates((prev) => ({
...prev,
[lane.lane]: { isPending: false, result },
}));
queryClient.invalidateQueries({ queryKey: ["api", "eval", lane.lane] });
}}
onRunError={(err) => {
setRunStates((prev) => ({
...prev,
[lane.lane]: { isPending: false, error: err },
}));
}}
/>
) : (
<div className="text-xs text-[var(--color-text-muted)] bg-[var(--color-surface-inset)] p-3 rounded-lg border border-[var(--color-border-subtle)] font-medium">
API runs disabled for write-active lane. Use local CLI to execute this lane:
<code className="block mt-1 font-mono text-[var(--color-text-primary)] bg-[var(--color-surface-base)] p-1.5 rounded">
core eval --lane {lane.lane}
</code>
</div>
)}
</>
);
}
const renderRightPane = () => {
if (!selectedLaneName) {
return (
<EmptyState
statement="Select an eval lane from the list to view results or run checks."
nextAction={{ kind: "cli", command: "core eval --list" }}
/>
);
}
if (!selectedLane) {
return (
<EmptyState
statement={`Selected lane "${selectedLaneName}" not found.`}
nextAction={{ kind: "cli", command: "core eval --list" }}
/>
);
}
// 1. Loading state for fetching the lane's last run details
if (isLaneLoading && !lastRunResult) {
return <LoadingState label="Loading eval lane details..." />;
}
// 2. Error state for fetching the lane details (excluding not_found)
if (isLaneError && !isNotFoundError) {
return (
<ErrorState
whatFailed={laneError instanceof Error ? laneError.message : "Failed to load eval lane details."}
mutationStatus="No corpus mutation occurred."
reproducer={`core eval --lane ${selectedLaneName}`}
retrySafety="Retry: safe"
/>
);
}
// 3. Current run execution state
if (currentRunState?.isPending) {
return (
<div className="flex flex-col gap-4">
{renderHeaderAndForm(selectedLane)}
<LoadingState label="Running eval lane..." />
</div>
);
}
if (currentRunState?.error) {
return (
<div className="flex flex-col gap-4">
{renderHeaderAndForm(selectedLane)}
<ErrorState
whatFailed={currentRunState.error.message}
mutationStatus="No corpus mutation occurred."
reproducer={`core eval --lane ${selectedLane.lane}`}
retrySafety="Retry: safe"
/>
</div>
);
}
// 4. Success or Empty state
const result = currentRunState?.result || (lastRunResult as EvalRunResult | undefined);
if (result && result.metrics) {
return (
<div className="flex flex-col gap-4">
{renderHeaderAndForm(selectedLane)}
<div className="flex flex-wrap items-center gap-3">
<span className="text-xs text-[var(--color-text-secondary)] font-semibold">
Status:
</span>
<span
className={`rounded-md px-2 py-0.5 text-xs font-semibold ${
result.passed
? "bg-[var(--color-state-success-bg)] text-[var(--color-state-success-text)] border border-[var(--color-state-success-border)]"
: "bg-[var(--color-state-danger-bg)] text-[var(--color-state-danger-text)] border border-[var(--color-state-danger-border)]"
}`}
>
{result.passed ? "Passed" : "Failed"}
</span>
{result.source_digest && (
<EvalArtifactLink
lane={selectedLane.lane}
sourceDigest={result.source_digest}
/>
)}
</div>
<div className="flex flex-col gap-2">
<h3 className="text-sm font-semibold text-[var(--color-text-primary)]">Metrics</h3>
<EvalMetricGrid metrics={result.metrics} />
</div>
<EvalFailureViewer
cases={result.cases}
passed={result.passed}
laneName={selectedLane.lane}
/>
</div>
);
}
return (
<div className="flex flex-col gap-4">
{renderHeaderAndForm(selectedLane)}
<EmptyState
statement={
selectedLane.read_only
? `No run results for lane "${selectedLane.lane}" in this session. Trigger a run above.`
: `Eval lane "${selectedLane.lane}" is CLI-only. No session results available.`
}
nextAction={{ kind: "cli", command: `core eval --lane ${selectedLane.lane}` }}
/>
</div>
);
};
return (
<div className="grid h-full grid-cols-1 gap-4 md:grid-cols-[18rem_1fr]" data-testid="evals-route">
@ -73,115 +275,7 @@ export function EvalsRoute() {
{/* Right Pane: Results / Form */}
<div className="flex flex-col gap-4 overflow-y-auto pl-2">
{selectedLane ? (
<>
{/* Header info */}
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-[var(--color-border-subtle)] pb-3">
<div>
<h2 className="text-lg font-semibold text-[var(--color-text-primary)] font-mono">
{selectedLane.lane}
</h2>
{selectedLane.description && (
<p className="mt-1 text-xs text-[var(--color-text-secondary)]">
{selectedLane.description}
</p>
)}
</div>
</div>
{/* Run Button configuration if read-only */}
{selectedLane.read_only ? (
<EvalRunButton
lane={selectedLane}
onRunStart={() => {
setRunStates((prev) => ({
...prev,
[selectedLane.lane]: { isPending: true },
}));
}}
onRunSuccess={(result) => {
setRunStates((prev) => ({
...prev,
[selectedLane.lane]: { isPending: false, result },
}));
}}
onRunError={(err) => {
setRunStates((prev) => ({
...prev,
[selectedLane.lane]: { isPending: false, error: err },
}));
}}
/>
) : (
<div className="text-xs text-[var(--color-text-muted)] bg-[var(--color-surface-inset)] p-3 rounded-lg border border-[var(--color-border-subtle)] font-medium">
API runs disabled for write-active lane. Use local CLI to execute this lane:
<code className="block mt-1 font-mono text-[var(--color-text-primary)] bg-[var(--color-surface-base)] p-1.5 rounded">
core eval --lane {selectedLane.lane}
</code>
</div>
)}
{/* Result display */}
{currentRunState?.isPending ? (
<LoadingState label="Running eval lane..." />
) : currentRunState?.error ? (
<ErrorState
whatFailed={currentRunState.error.message}
mutationStatus="No corpus mutation occurred."
reproducer={`core eval --lane ${selectedLane.lane}`}
retrySafety="Retry: safe"
/>
) : currentRunState?.result ? (
<div className="flex flex-col gap-4">
<div className="flex flex-wrap items-center gap-3">
<span className="text-xs text-[var(--color-text-secondary)] font-semibold">
Status:
</span>
<span
className={`rounded-md px-2 py-0.5 text-xs font-semibold ${
currentRunState.result.passed
? "bg-[var(--color-state-success-bg)] text-[var(--color-state-success-text)] border border-[var(--color-state-success-border)]"
: "bg-[var(--color-state-danger-bg)] text-[var(--color-state-danger-text)] border border-[var(--color-state-danger-border)]"
}`}
>
{currentRunState.result.passed ? "Passed" : "Failed"}
</span>
{currentRunState.result.source_digest && (
<EvalArtifactLink
lane={selectedLane.lane}
sourceDigest={currentRunState.result.source_digest}
/>
)}
</div>
<div className="flex flex-col gap-2">
<h3 className="text-sm font-semibold text-[var(--color-text-primary)]">Metrics</h3>
<EvalMetricGrid metrics={currentRunState.result.metrics} />
</div>
<EvalFailureViewer
cases={currentRunState.result.cases}
passed={currentRunState.result.passed}
laneName={selectedLane.lane}
/>
</div>
) : (
<EmptyState
statement={
selectedLane.read_only
? `No run results for lane "${selectedLane.lane}" in this session. Trigger a run above.`
: `Eval lane "${selectedLane.lane}" is CLI-only. No session results available.`
}
nextAction={{ kind: "cli", command: `core eval --lane ${selectedLane.lane}` }}
/>
)}
</>
) : (
<EmptyState
statement="Select an eval lane from the list to view results or run checks."
nextAction={{ kind: "cli", command: "core eval --list" }}
/>
)}
{renderRightPane()}
</div>
</div>
);

View file

@ -16,11 +16,13 @@ vi.mock("../../api/queries", async (importOriginal) => {
return {
...actual,
useEvalLanes: vi.fn(),
useEvalLane: vi.fn(),
useEvalRun: vi.fn(),
};
});
import { useEvalLanes, useEvalRun } from "../../api/queries";
import { useEvalLanes, useEvalLane, useEvalRun } from "../../api/queries";
import { CommandPalette } from "../../design/components/primitives/CommandPalette";
const mockLanes: EvalLaneSummary[] = [
{ lane: "contemplation_quality", versions: ["v1", "v2"], read_only: true, description: "Contemplation checks" },
@ -49,6 +51,27 @@ describe("W-030 Component Tests", () => {
beforeEach(() => {
vi.resetAllMocks();
vi.mocked(useEvalLanes).mockReturnValue({
data: mockLanes,
isLoading: false,
isError: false,
error: null,
} as any);
vi.mocked(useEvalLane).mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
error: new WorkbenchApiError("not_found", "No run history found"),
} as any);
vi.mocked(useEvalRun).mockReturnValue({
mutate: vi.fn(),
isPending: false,
isError: false,
isSuccess: false,
error: null,
} as any);
fetchMock.mockImplementation((url: any) => {
const urlStr = typeof url === "string" ? url : String(url?.url || url || "");
if (urlStr.endsWith("/evals")) {
@ -162,10 +185,45 @@ describe("W-030 Component Tests", () => {
// Verify lexicographical order
expect(cards1).toEqual(cards2);
expect(cards1.length).toBe(3);
expect(cards1[0]).toContain("accuracy");
expect(cards1[1]).toContain("latency ms");
expect(cards1[2]).toContain("passed");
expect(cards1[3]).toContain("total");
expect(cards1[2]).toContain("9/10");
});
it("renders actual / target, emphasizes actual, and applies failure color token when target is not met", () => {
const metrics = {
passed: 4,
total: 5,
latency_ms: 150,
latency_ms_target: 100, // failing because actual > target
accuracy: 0.9,
accuracy_target: 0.95, // failing because actual < target
};
const { container } = render(<EvalMetricGrid metrics={metrics} />);
const cards = container.querySelectorAll('[data-testid="metric-card"]');
expect(cards.length).toBe(3);
const passedCard = Array.from(cards).find(c => c.textContent?.includes("passed"));
expect(passedCard).toBeDefined();
expect(passedCard?.querySelector(".text-\\[var\\(--color-state-contradicted\\)\\]")).toBeInTheDocument();
expect(passedCard?.textContent).toContain("4");
expect(passedCard?.textContent).toContain("5");
const latencyCard = Array.from(cards).find(c => c.textContent?.includes("latency ms"));
expect(latencyCard).toBeDefined();
expect(latencyCard?.querySelector(".text-\\[var\\(--color-state-contradicted\\)\\]")).toBeInTheDocument();
expect(latencyCard?.textContent).toContain("150");
expect(latencyCard?.textContent).toContain("100");
const accuracyCard = Array.from(cards).find(c => c.textContent?.includes("accuracy"));
expect(accuracyCard).toBeDefined();
expect(accuracyCard?.querySelector(".text-\\[var\\(--color-state-contradicted\\)\\]")).toBeInTheDocument();
expect(accuracyCard?.textContent).toContain("0.9");
expect(accuracyCard?.textContent).toContain("0.95");
});
});
@ -295,5 +353,115 @@ describe("W-030 Component Tests", () => {
const res = await runEvalLane({ lane: "contemplation_quality", split: "holdout" });
expect(res).toEqual(mockResult);
});
it("renders loading state when lane list is loading", () => {
vi.mocked(useEvalLanes).mockReturnValue({
data: null,
isLoading: true,
isError: false,
error: null,
} as any);
const client = makeClient();
render(
<QueryClientProvider client={client}>
<MemoryRouter initialEntries={["/evals"]}>
<EvalsRoute />
</MemoryRouter>
</QueryClientProvider>
);
expect(screen.getByText("Loading eval lanes...")).toBeInTheDocument();
});
it("renders empty state when there are no lanes", () => {
vi.mocked(useEvalLanes).mockReturnValue({
data: [],
isLoading: false,
isError: false,
error: null,
} as any);
const client = makeClient();
render(
<QueryClientProvider client={client}>
<MemoryRouter initialEntries={["/evals"]}>
<EvalsRoute />
</MemoryRouter>
</QueryClientProvider>
);
expect(screen.getByText("No eval lanes discovered.")).toBeInTheDocument();
});
it("renders loading and error states for the selected lane detail view", () => {
vi.mocked(useEvalLanes).mockReturnValue({
data: mockLanes,
isLoading: false,
isError: false,
error: null,
} as any);
// Mock useEvalLane to return loading
vi.mocked(useEvalLane).mockReturnValue({
data: undefined,
isLoading: true,
isError: false,
error: null,
} as any);
const client = makeClient();
const { rerender } = render(
<QueryClientProvider client={client}>
<MemoryRouter initialEntries={["/evals?lane=contemplation_quality"]}>
<EvalsRoute />
</MemoryRouter>
</QueryClientProvider>
);
expect(screen.getByText("Loading eval lane details...")).toBeInTheDocument();
// Mock useEvalLane to return error (other than not_found)
vi.mocked(useEvalLane).mockReturnValue({
data: undefined,
isLoading: false,
isError: true,
error: new WorkbenchApiError("read_error", "Disk read error"),
} as any);
rerender(
<QueryClientProvider client={client}>
<MemoryRouter initialEntries={["/evals?lane=contemplation_quality"]}>
<EvalsRoute />
</MemoryRouter>
</QueryClientProvider>
);
expect(screen.getByText("Disk read error")).toBeInTheDocument();
});
it("CommandPalette registers dynamic command entries driven by useEvalLanes", () => {
vi.mocked(useEvalLanes).mockReturnValue({
data: mockLanes,
isLoading: false,
isError: false,
error: null,
} as any);
const client = makeClient();
render(
<QueryClientProvider client={client}>
<MemoryRouter>
<CommandPalette open={true} onOpenChange={vi.fn()} />
</MemoryRouter>
</QueryClientProvider>
);
// Verify static commands
expect(screen.getByRole("button", { name: "Open Chat" })).toBeInTheDocument();
// Verify dynamic commands
expect(screen.getByRole("button", { name: "Open eval lane contemplation_quality" })).toBeInTheDocument();
expect(screen.getByRole("button", { name: "Open eval lane unsafe_lane" })).toBeInTheDocument();
});
});
});

View file

@ -1,10 +1,18 @@
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import { MemoryRouter } from "react-router-dom";
import { useState } from "react";
import { CommandPalette } from "./CommandPalette";
vi.mock("../../../api/queries", () => ({
useEvalLanes: () => ({
data: [],
isLoading: false,
isError: false,
}),
}));
function PaletteHarness({ initialOpen = false }: { initialOpen?: boolean }) {
const [open, setOpen] = useState(initialOpen);
return (

View file

@ -2,13 +2,14 @@ import * as Dialog from "@radix-ui/react-dialog";
import { Search } from "lucide-react";
import { useRef, useState, useEffect, useCallback } from "react";
import { useNavigate, useInRouterContext } from "react-router-dom";
import { useEvalLanes } from "../../../api/queries";
interface Command {
name: string;
path: string;
}
const COMMANDS: Command[] = [
const STATIC_COMMANDS: Command[] = [
{ name: "Open Chat", path: "/chat" },
{ name: "Open Proposals", path: "/proposals" },
{ name: "Open Evals", path: "/evals" },
@ -20,6 +21,15 @@ function RouterCommandPalette(props: {
onOpenChange: (open: boolean) => void;
}) {
const navigate = useNavigate();
const { data: lanes } = useEvalLanes();
const dynamicCommands = (lanes || []).map((lane) => ({
name: `Open eval lane ${lane.lane}`,
path: `/evals?lane=${lane.lane}`,
}));
const commands = [...STATIC_COMMANDS, ...dynamicCommands];
const activate = useCallback(
(cmd: Command) => {
navigate(cmd.path);
@ -27,7 +37,7 @@ function RouterCommandPalette(props: {
},
[navigate, props],
);
return <CommandPaletteContent {...props} onActivate={activate} />;
return <CommandPaletteContent {...props} commands={commands} onActivate={activate} />;
}
// Fallback for design-system preview (no Router).
@ -41,23 +51,25 @@ function FallbackCommandPalette(props: {
},
[props],
);
return <CommandPaletteContent {...props} onActivate={activate} />;
return <CommandPaletteContent {...props} commands={STATIC_COMMANDS} onActivate={activate} />;
}
function CommandPaletteContent({
open,
onOpenChange,
onActivate,
commands,
}: {
open: boolean;
onOpenChange: (open: boolean) => void;
onActivate: (cmd: Command) => void;
commands: Command[];
}) {
const [query, setQuery] = useState("");
const [focusedIndex, setFocusedIndex] = useState(0);
const inputRef = useRef<HTMLInputElement>(null);
const filtered = COMMANDS.filter((cmd) =>
const filtered = commands.filter((cmd) =>
cmd.name.toLowerCase().includes(query.toLowerCase()),
);