Compare commits
5 commits
add-ask-ac
...
polish-wor
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
37b04d504b | ||
|
|
e954c660f7 | ||
|
|
a4a24de836 | ||
|
|
685aaae40e | ||
|
|
4d7c605256 |
12 changed files with 1554 additions and 138 deletions
112
docs/handoff/workbench-ui/wave-1/1a-chat-polish.md
Normal file
112
docs/handoff/workbench-ui/wave-1/1a-chat-polish.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# Brief 1a — Chat route polish
|
||||
|
||||
> **Operator:** Opus 4.6 (foundations / surface-contract sensitive)
|
||||
> **Branch:** `workbench/wave-1a-chat-polish`
|
||||
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
|
||||
> **Estimated diff:** ~150–300 lines, UI only
|
||||
> **Why this operator:** Chat is the one Phase 1 brief that touches the
|
||||
> Runtime Surface Contract (`surface` / `walk_surface` / `articulation_surface`).
|
||||
> Distinguishing those correctly under the addendum §2 is high-stakes.
|
||||
|
||||
---
|
||||
|
||||
## Dispatch line
|
||||
|
||||
```bash
|
||||
git fetch origin && \
|
||||
git worktree add /Users/kaizenpro/Projects/core-w1a-chat \
|
||||
-b workbench/wave-1a-chat-polish origin/feat/workbench-ui-continuation && \
|
||||
cd /Users/kaizenpro/Projects/core-w1a-chat/workbench-ui && \
|
||||
pnpm install
|
||||
```
|
||||
|
||||
Then open the worktree in your editor and start.
|
||||
|
||||
---
|
||||
|
||||
## Doctrine refs (read before touching code)
|
||||
|
||||
- `docs/plans/workbench-ui-continuation.md` §Phase 1a
|
||||
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Chat trust class),
|
||||
§2 (Trace surface contract — applies to Chat's trace drawer), §7 (PR checklist)
|
||||
- `docs/runtime_contracts.md` — surface distinction and `trace_hash` stability
|
||||
|
||||
## Scope (owned files)
|
||||
|
||||
- `src/app/chat/**` — everything in this folder
|
||||
- Test files colocated with the above
|
||||
|
||||
## May read (no edits)
|
||||
|
||||
- `src/types/api.ts` (append-only at the bottom of the chat section if
|
||||
new types are needed)
|
||||
- `src/api/queries.ts` (no edits; this brief does not add hooks)
|
||||
|
||||
## Off-limits
|
||||
|
||||
- `src/design/**`
|
||||
- `src/app/Shell.tsx`, `App.tsx`, `LeftNav.tsx`, `TopBar.tsx`, `StatusFooter.tsx`
|
||||
- Any backend Python file
|
||||
- Any placeholder file in `src/routes/`
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **State audit.** Walk every async surface in `src/app/chat/` and confirm
|
||||
it has loading, empty, and error states. Where missing, add them using
|
||||
`EmptyState` from `src/design/`.
|
||||
- [ ] **Surface distinction.** Inspect the trace drawer / evidence panel. The
|
||||
three surfaces — `surface`, `walk_surface`, `articulation_surface` —
|
||||
must render as three distinct UI elements with labels the operator can
|
||||
tell apart at a glance. If currently conflated, separate them.
|
||||
- [ ] **Navigation hardening.** From a turn response:
|
||||
- clicking `trace_hash` opens the trace drawer at that turn,
|
||||
- any proposal candidate links to its detail in the Proposals route
|
||||
(use the existing route path; do not invent a new one).
|
||||
- [ ] **⌘K commands.** Confirm or add palette entries for `New chat session`
|
||||
and `Jump to proposal <id>`. Entries belong to chat; do not edit other
|
||||
features' entries.
|
||||
- [ ] **Trace-hash stability test.** Add a test asserting that mounting the
|
||||
trace drawer, navigating nodes, and unmounting does not mutate any
|
||||
`trace_hash` value held in component state. This is the addendum §2
|
||||
proof obligation.
|
||||
- [ ] **Tests for new interaction paths.** Use the existing `Shell.test.tsx`
|
||||
pattern: render under MemoryRouter + QueryClientProvider, drive
|
||||
interactions through `@testing-library/react`.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Every async surface in `src/app/chat/` has loading + empty + error states.
|
||||
- Trace drawer labels `surface` / `walk_surface` / `articulation_surface`
|
||||
distinctly. A reviewer should be able to point at each on screen.
|
||||
- `trace_hash` stability test exists and passes.
|
||||
- ⌘K entries for `New chat session` and `Jump to proposal` are present and
|
||||
keyboard-activatable.
|
||||
- No edits to off-limits files. `git diff --stat` shows only `src/app/chat/`
|
||||
and (optionally) `src/types/api.ts` append-only.
|
||||
- `pnpm test` and `pnpm test:enum-coverage` green.
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
cd workbench-ui
|
||||
pnpm install
|
||||
pnpm test
|
||||
pnpm test:enum-coverage
|
||||
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
|
||||
```
|
||||
|
||||
The diff-stat output must include only `src/app/chat/**` and optionally
|
||||
appended lines in `src/types/api.ts`. Anything else is out of scope.
|
||||
|
||||
## PR
|
||||
|
||||
Title: `feat(workbench/chat): polish states, surface distinction, ⌘K (wave 1a)`
|
||||
|
||||
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
|
||||
|
||||
## When done
|
||||
|
||||
- [ ] PR open against `feat/workbench-ui-continuation`
|
||||
- [ ] CI green
|
||||
- [ ] Mark this brief complete by checking the box in this file in a follow-up
|
||||
commit on the same PR (single source of truth: this file)
|
||||
111
docs/handoff/workbench-ui/wave-1/1b-proposals-polish.md
Normal file
111
docs/handoff/workbench-ui/wave-1/1b-proposals-polish.md
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
# Brief 1b — Proposals route polish
|
||||
|
||||
> **Operator:** Sonnet 4.6 (tight-scope route work)
|
||||
> **Branch:** `workbench/wave-1b-proposals-polish`
|
||||
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
|
||||
> **Estimated diff:** ~150–250 lines, UI only
|
||||
> **Why this operator:** Proposals has a well-defined lifecycle
|
||||
> (pending / accepted / rejected / withdrawn) and clear color tokens
|
||||
> already in `src/design/`. Tight contract, no surface-contract surprises.
|
||||
|
||||
---
|
||||
|
||||
## Dispatch line
|
||||
|
||||
```bash
|
||||
git fetch origin && \
|
||||
git worktree add /Users/kaizenpro/Projects/core-w1b-proposals \
|
||||
-b workbench/wave-1b-proposals-polish origin/feat/workbench-ui-continuation && \
|
||||
cd /Users/kaizenpro/Projects/core-w1b-proposals/workbench-ui && \
|
||||
pnpm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Doctrine refs
|
||||
|
||||
- `docs/plans/workbench-ui-continuation.md` §Phase 1b
|
||||
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Proposals is
|
||||
proposal-only — no direct accept/reject endpoints introduced; UI only
|
||||
calls existing API), §7 (PR checklist)
|
||||
|
||||
## Scope (owned files)
|
||||
|
||||
- `src/app/proposals/**`
|
||||
- Test files colocated with the above
|
||||
|
||||
## May read (no edits)
|
||||
|
||||
- `src/api/queries.ts` — observe existing `useProposals` / `useProposal` shapes
|
||||
- `src/design/tokens.css` — read review-lifecycle tokens; do not edit
|
||||
- `src/types/api.ts` — append-only at the bottom of proposals section if needed
|
||||
|
||||
## Off-limits
|
||||
|
||||
- `src/design/**` (including tokens)
|
||||
- Shell / TopBar / LeftNav / StatusFooter / App.tsx
|
||||
- Any backend Python file
|
||||
- Any placeholder file
|
||||
- Any new proposal-mutation endpoint or hook (proposal-only discipline)
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **State audit.** Confirm queue list and proposal detail handle loading,
|
||||
empty, and error consistently. Wire `EmptyState` from `src/design/`
|
||||
where missing.
|
||||
- [ ] **Lifecycle color consistency.** The four states (`pending`,
|
||||
`accepted`, `rejected`, `withdrawn`) must render with the same color
|
||||
tokens in both list and detail. Verify each token is referenced via
|
||||
CSS custom property — no hex literals.
|
||||
- [ ] **Cross-route links.** From proposal detail:
|
||||
- link to the source artifact (use existing Runs/artifact route path
|
||||
if present; otherwise the artifact route Phase 2 will introduce —
|
||||
for now link to the Replay-comparison entry-point if that exists),
|
||||
- link to the replay-comparison view for the proposal if applicable.
|
||||
Document any link that cannot land because the target route is a
|
||||
placeholder; leave a TODO with the route name.
|
||||
- [ ] **Suggested-CLI block.** Make the suggested-CLI snippet block
|
||||
`user-select: text` (or a copy button) so the operator can copy it.
|
||||
- [ ] **⌘K command entries.** Add `Open proposal <id>` palette entry
|
||||
driven by the existing `useProposals` query. Only proposals entries —
|
||||
do not touch other features'.
|
||||
- [ ] **Route tests.** Cover at least:
|
||||
- queue renders with loading → success transition,
|
||||
- empty state when zero proposals returned,
|
||||
- error state when query errors,
|
||||
- lifecycle color class applied per state.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Loading / empty / error covered everywhere in `src/app/proposals/`.
|
||||
- All four lifecycle states render with the documented tokens, no hex
|
||||
literals anywhere in this brief's diff (`grep -E '#[0-9a-fA-F]{3,6}'`
|
||||
returns nothing in the diff).
|
||||
- Suggested-CLI block is selectable or has a copy button.
|
||||
- ⌘K entries for proposals are present and keyboard-activatable.
|
||||
- No backend changes. No off-limits files touched.
|
||||
- `pnpm test` and `pnpm test:enum-coverage` green.
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
cd workbench-ui
|
||||
pnpm install
|
||||
pnpm test
|
||||
pnpm test:enum-coverage
|
||||
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
|
||||
# Verify no hex literals introduced
|
||||
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/proposals/**' | grep -E '^\+.*#[0-9a-fA-F]{3,6}' && echo "FAIL: hex literal introduced" || echo "ok: no hex literals"
|
||||
```
|
||||
|
||||
## PR
|
||||
|
||||
Title: `feat(workbench/proposals): polish states, lifecycle colors, links (wave 1b)`
|
||||
|
||||
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
|
||||
|
||||
## When done
|
||||
|
||||
- [ ] PR open against `feat/workbench-ui-continuation`
|
||||
- [ ] CI green
|
||||
- [ ] This brief checked complete on the PR
|
||||
111
docs/handoff/workbench-ui/wave-1/1c-replay-polish.md
Normal file
111
docs/handoff/workbench-ui/wave-1/1c-replay-polish.md
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
# Brief 1c — Replay route polish
|
||||
|
||||
> **Operator:** Gemini 3.1 Pro (survey + cross-file presentation work)
|
||||
> **Branch:** `workbench/wave-1c-replay-polish`
|
||||
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
|
||||
> **Estimated diff:** ~150–250 lines, UI only
|
||||
> **Why this operator:** Replay polish requires reading divergence
|
||||
> rendering across multiple files and improving clarity of metadata
|
||||
> presentation. Gemini's strength is cross-file survey + presentation
|
||||
> reasoning over a moderate diff surface.
|
||||
|
||||
---
|
||||
|
||||
## Dispatch line
|
||||
|
||||
```bash
|
||||
git fetch origin && \
|
||||
git worktree add /Users/kaizenpro/Projects/core-w1c-replay \
|
||||
-b workbench/wave-1c-replay-polish origin/feat/workbench-ui-continuation && \
|
||||
cd /Users/kaizenpro/Projects/core-w1c-replay/workbench-ui && \
|
||||
pnpm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Doctrine refs
|
||||
|
||||
- `docs/plans/workbench-ui-continuation.md` §Phase 1c
|
||||
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Replay = read-only;
|
||||
no "fix" buttons, no artifact writes), §7 (PR checklist)
|
||||
- The Replay route should never present an affordance to mutate an
|
||||
artifact. Read + render + link out only.
|
||||
|
||||
## Scope (owned files)
|
||||
|
||||
- `src/app/replay/**`
|
||||
- Test files colocated with the above
|
||||
|
||||
## May read (no edits)
|
||||
|
||||
- `src/api/queries.ts` — observe `useReplayComparison`, `useArtifacts`, `useArtifact`
|
||||
- `src/design/tokens.css` — divergence-severity tokens
|
||||
- `src/types/api.ts` — append-only at the bottom of replay section if needed
|
||||
|
||||
## Off-limits
|
||||
|
||||
- `src/design/**`
|
||||
- Shell / TopBar / LeftNav / StatusFooter / App.tsx
|
||||
- Any backend Python file
|
||||
- Any placeholder file
|
||||
- **No mutating affordances of any kind** — no "rerun", "save", "fix",
|
||||
"accept divergence" buttons. Read-only doctrine.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **Survey current state.** Read every file in `src/app/replay/` and
|
||||
produce a short comment block at the top of the route (or in this
|
||||
brief's PR description) summarizing what each component does. This
|
||||
forces a complete read before edits.
|
||||
- [ ] **State audit.** Artifact-selector and comparison view: loading,
|
||||
empty (no artifacts available; selection invalid), error states.
|
||||
- [ ] **Divergence-severity clarity.** Currently the divergence severity
|
||||
may be coded via color alone. Add a textual severity label next to
|
||||
the color band (e.g. `low`, `material`, `breaking`) so the meaning
|
||||
is unambiguous without the color legend.
|
||||
- [ ] **Metadata presentation.** Artifact metadata (timestamp, trace_hash,
|
||||
run id, lane) should be presented in a key-value table or definition
|
||||
list with consistent alignment. No "wall of text" rendering.
|
||||
- [ ] **Back-link to proposal.** When the comparison view is reached from
|
||||
a proposal, show a "Back to proposal #N" link. When reached directly,
|
||||
do not show the link. Drive this from URL state, not props.
|
||||
- [ ] **Route tests.** Cover:
|
||||
- artifact selection populating comparison view,
|
||||
- empty state when no artifacts,
|
||||
- error state when comparison query fails,
|
||||
- severity label rendering for each severity bucket.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Every async surface in `src/app/replay/` has loading + empty + error.
|
||||
- Divergence severity is conveyed by **both** color **and** text label.
|
||||
- Artifact metadata renders as a structured key-value layout.
|
||||
- Back-link to proposal appears only when route was entered from a proposal.
|
||||
- No mutating affordances anywhere in the diff.
|
||||
- No backend changes. No off-limits files touched.
|
||||
- `pnpm test` and `pnpm test:enum-coverage` green.
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
cd workbench-ui
|
||||
pnpm install
|
||||
pnpm test
|
||||
pnpm test:enum-coverage
|
||||
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
|
||||
|
||||
# Confirm no write paths introduced (no new mutation hooks)
|
||||
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/replay/**' | grep -E '^\+.*(useMutation|fetch\(|axios\.(post|put|patch|delete))' && echo "FAIL: mutation introduced" || echo "ok: read-only preserved"
|
||||
```
|
||||
|
||||
## PR
|
||||
|
||||
Title: `feat(workbench/replay): polish states, severity labels, metadata layout (wave 1c)`
|
||||
|
||||
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
|
||||
|
||||
## When done
|
||||
|
||||
- [ ] PR open against `feat/workbench-ui-continuation`
|
||||
- [ ] CI green
|
||||
- [ ] This brief checked complete on the PR
|
||||
114
docs/handoff/workbench-ui/wave-1/1d-evals-polish.md
Normal file
114
docs/handoff/workbench-ui/wave-1/1d-evals-polish.md
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
# Brief 1d — Evals route polish
|
||||
|
||||
> **Operator:** GPT-OSS-120B or GitHub Copilot cloud agent (mechanical)
|
||||
> **Branch:** `workbench/wave-1d-evals-polish`
|
||||
> **Base:** `feat/workbench-ui-continuation` @ `685aaae`
|
||||
> **Estimated diff:** ~120–200 lines, UI only
|
||||
> **Why this operator:** Evals polish is mostly mechanical: state audits,
|
||||
> numeric column alignment, route tests. Clear acceptance criteria, narrow
|
||||
> file scope, no surface-contract subtlety. Ideal for a local or
|
||||
> cloud-mechanical operator.
|
||||
|
||||
---
|
||||
|
||||
## Dispatch line
|
||||
|
||||
```bash
|
||||
git fetch origin && \
|
||||
git worktree add /Users/kaizenpro/Projects/core-w1d-evals \
|
||||
-b workbench/wave-1d-evals-polish origin/feat/workbench-ui-continuation && \
|
||||
cd /Users/kaizenpro/Projects/core-w1d-evals/workbench-ui && \
|
||||
pnpm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Doctrine refs
|
||||
|
||||
- `docs/plans/workbench-ui-continuation.md` §Phase 1d
|
||||
- `docs/plans/workbench-ui-continuation-addendum.md` §1 (Evals = read-only;
|
||||
no lane mutation, no "suppress failing case" affordance), §7 (PR checklist)
|
||||
- Evals UI must not hide or suppress failing cases. Filters that change
|
||||
view are fine; filters that remove failing cases from the underlying
|
||||
count are not.
|
||||
|
||||
## Scope (owned files)
|
||||
|
||||
- `src/app/evals/**`
|
||||
- Test files colocated with the above
|
||||
|
||||
## May read (no edits)
|
||||
|
||||
- `src/api/queries.ts` — `useEvalLanes`, `useEvalLane`, `useEvalExecution`
|
||||
- `src/design/tokens.css` — read; do not edit
|
||||
- `src/types/api.ts` — append-only at the bottom of evals section if needed
|
||||
|
||||
## Off-limits
|
||||
|
||||
- `src/design/**`
|
||||
- Shell / TopBar / LeftNav / StatusFooter / App.tsx
|
||||
- Any backend Python file
|
||||
- Any placeholder file
|
||||
- Any affordance that mutates a lane definition or hides a failing case
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **State audit.** Lane list and eval-run view: loading, empty
|
||||
(no lanes; no runs for selected lane), error states.
|
||||
- [ ] **Numeric column alignment.** Wherever numeric metrics render in a
|
||||
list or table, they must be right-aligned, monospace-numeric, and
|
||||
use tabular-nums for consistent digit width. Use existing tokens.
|
||||
- [ ] **Metric readability.** Where a metric is shown alongside a target,
|
||||
render `actual / target` with the actual value emphasized. If a
|
||||
metric is failing (actual < target where higher-is-better, or
|
||||
actual > target where lower-is-better), apply the existing
|
||||
failure color token. No new tokens.
|
||||
- [ ] **Lane selection persistence.** Selected lane should be reflected in
|
||||
the URL (query string is fine). Reloading the page should restore the
|
||||
selection.
|
||||
- [ ] **⌘K command entry.** Add `Open eval lane <name>` palette entries
|
||||
driven by the existing `useEvalLanes` query. Only evals entries.
|
||||
- [ ] **Route tests.** Cover:
|
||||
- lane list loading → success,
|
||||
- empty state when no lanes,
|
||||
- run view rendering metrics with correct alignment,
|
||||
- failure color applied when metric does not meet target.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Every async surface in `src/app/evals/` has loading + empty + error.
|
||||
- All numeric metric columns are right-aligned with `tabular-nums`.
|
||||
- Failing metrics use the existing failure token, no hex literals.
|
||||
- Lane selection survives reload via URL state.
|
||||
- ⌘K `Open eval lane` entries present and keyboard-activatable.
|
||||
- No backend changes. No off-limits files touched. No affordance that
|
||||
suppresses or hides failing cases.
|
||||
- `pnpm test` and `pnpm test:enum-coverage` green.
|
||||
|
||||
## Validation
|
||||
|
||||
```bash
|
||||
cd workbench-ui
|
||||
pnpm install
|
||||
pnpm test
|
||||
pnpm test:enum-coverage
|
||||
git diff --stat $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD
|
||||
|
||||
# No hex literals in the diff
|
||||
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/evals/**' | grep -E '^\+.*#[0-9a-fA-F]{3,6}' && echo "FAIL: hex literal introduced" || echo "ok: no hex literals"
|
||||
|
||||
# No mutation hooks introduced
|
||||
git diff $(git merge-base HEAD origin/feat/workbench-ui-continuation)..HEAD -- 'src/app/evals/**' | grep -E '^\+.*useMutation' && echo "FAIL: mutation introduced" || echo "ok: read-only preserved"
|
||||
```
|
||||
|
||||
## PR
|
||||
|
||||
Title: `feat(workbench/evals): polish states, numeric alignment, lane URL state (wave 1d)`
|
||||
|
||||
Body uses the template in `docs/handoff/workbench-ui/wave-1/README.md`.
|
||||
|
||||
## When done
|
||||
|
||||
- [ ] PR open against `feat/workbench-ui-continuation`
|
||||
- [ ] CI green
|
||||
- [ ] This brief checked complete on the PR
|
||||
110
docs/handoff/workbench-ui/wave-1/README.md
Normal file
110
docs/handoff/workbench-ui/wave-1/README.md
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
# Workbench UI — Wave 1 Brief Pack
|
||||
|
||||
> **Parent plan:** `docs/plans/workbench-ui-continuation.md`
|
||||
> **Binding addendum:** `docs/plans/workbench-ui-continuation-addendum.md`
|
||||
> **Base branch:** `feat/workbench-ui-continuation` @ `685aaae`
|
||||
> **Scope:** Phase 1 only — polish the four implemented routes
|
||||
|
||||
## What this wave does
|
||||
|
||||
Polish Chat, Proposals, Replay, and Evals so they are cohesive,
|
||||
cross-linked, and handle loading/empty/error states consistently —
|
||||
before any new route lands. All four routes are already implemented;
|
||||
this is hardening, not feature work.
|
||||
|
||||
## Parallel safety
|
||||
|
||||
| Operator brief | Owns | Read-only touches |
|
||||
|-----------------------|-----------------------------------|----------------------------------|
|
||||
| 1a — Chat polish | `src/app/chat/**` | `src/types/api.ts` (append only) |
|
||||
| 1b — Proposals polish | `src/app/proposals/**` | `src/types/api.ts` (append only) |
|
||||
| 1c — Replay polish | `src/app/replay/**` | `src/types/api.ts` (append only) |
|
||||
| 1d — Evals polish | `src/app/evals/**` | `src/types/api.ts` (append only) |
|
||||
|
||||
**Off-limits to every brief:**
|
||||
- `src/design/**` (protected substrate — plan constraint)
|
||||
- `src/app/Shell.tsx`, `LeftNav.tsx`, `TopBar.tsx`, `StatusFooter.tsx`
|
||||
- `src/app/App.tsx` route table (no swaps in Wave 1)
|
||||
- Any placeholder file in `src/routes/` (those belong to Phase 2+)
|
||||
- `workbench-ui/src/design/tokens.css` and friends
|
||||
- Backend Python (this is a UI-only wave)
|
||||
|
||||
**Coordinated surface:**
|
||||
- CommandPalette entries: each brief adds **only** entries for its
|
||||
own feature; no brief edits another's entries. If a conflict appears
|
||||
at merge, the second-merging brief rebases additively.
|
||||
- `src/types/api.ts`: append-only additions to the bottom of the
|
||||
matching section. No edits to existing types.
|
||||
|
||||
## Dispatch DAG
|
||||
|
||||
```
|
||||
┌── 1a Chat polish (Opus 4.6) ───────┐
|
||||
main ──>│ │── merge any order
|
||||
│── 1b Proposals polish (Sonnet 4.6)─│ (all four parallel-safe)
|
||||
│ │
|
||||
│── 1c Replay polish (Gemini 3.1) ───│
|
||||
│ │
|
||||
└── 1d Evals polish (GPT-OSS/Copilot)┘
|
||||
```
|
||||
|
||||
All four briefs are dispatchable simultaneously. No order dependency.
|
||||
|
||||
## Doctrine that applies to every brief
|
||||
|
||||
From `docs/plans/workbench-ui-continuation-addendum.md`:
|
||||
|
||||
- **§1 trust classification.** Chat = read + turn-submit; Proposals = proposal-only;
|
||||
Replay = read-only; Evals = read-only. No brief introduces a write path.
|
||||
- **§2 surface contract (Chat brief only).** `surface`, `walk_surface`, and
|
||||
`articulation_surface` rendered distinctly; `trace_hash` stability test required.
|
||||
- **§7 per-PR checklist.** Every PR answers the four CLAUDE.md §PR Checklist
|
||||
questions in its description.
|
||||
- **§8 proof obligations.** New TS unions must have a test that fails under
|
||||
their named violation, or the type is decoration.
|
||||
|
||||
## Validation each brief must run
|
||||
|
||||
```bash
|
||||
cd workbench-ui
|
||||
pnpm install # if first time on this worktree
|
||||
pnpm test # all green
|
||||
pnpm test:enum-coverage
|
||||
```
|
||||
|
||||
Backend-touching changes are **out of scope** for Wave 1 and must be
|
||||
rejected at review. Any "while I'm here" backend edit means the PR
|
||||
fails the trust-classification check.
|
||||
|
||||
## PR template (every brief uses this)
|
||||
|
||||
```markdown
|
||||
## Wave 1 brief: <1a|1b|1c|1d> — <feature> polish
|
||||
|
||||
### Trust classification
|
||||
<read-only | read + turn-submit | proposal-only>
|
||||
|
||||
### CLAUDE.md §PR Checklist
|
||||
- Capability/property/boundary added: <one line>
|
||||
- Invariant preserved: <one line, e.g. trace_hash stability, no design-substrate edit>
|
||||
- CLI/test lane that proves it: pnpm test + pnpm test:enum-coverage (UI-only)
|
||||
- Trust boundary enforced: <one line; "no write paths introduced">
|
||||
|
||||
### Diff scope
|
||||
- Owned: <files>
|
||||
- Read-only touched: <files>
|
||||
- Off-limits respected: yes
|
||||
|
||||
### Tests
|
||||
- Added: <tests>
|
||||
- Passing: pnpm test green; pnpm test:enum-coverage green
|
||||
```
|
||||
|
||||
## Done criteria for the wave
|
||||
|
||||
- [ ] All four PRs merged to `feat/workbench-ui-continuation`
|
||||
- [ ] No off-limits files modified across the merged set
|
||||
- [ ] No backend changes across the merged set
|
||||
- [ ] `pnpm test` and `pnpm test:enum-coverage` green on the merged tip
|
||||
- [ ] CommandPalette has entries for the new interactions added by each brief
|
||||
- [ ] All four briefs in this folder marked done in their respective files
|
||||
190
docs/plans/workbench-ui-continuation-addendum.md
Normal file
190
docs/plans/workbench-ui-continuation-addendum.md
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
# Workbench UI Continuation — CORE-Doctrine Addendum
|
||||
|
||||
> **Status:** binding addendum to `docs/plans/workbench-ui-continuation.md`
|
||||
> **Applies to:** all Phase 5+ work; back-applies to Phases 1–4 where noted
|
||||
> **Authority:** CLAUDE.md (§Security, §Teaching Safety, §Runtime Surface Contract, §PR Checklist, §Schema-Defined Proof Obligations), `docs/runtime_contracts.md`, ADR-0051
|
||||
|
||||
The parent plan is sound as frontend hygiene. This addendum supplies the
|
||||
trust-boundary, determinism, and proof-obligation language CORE requires before
|
||||
any route that touches packs, vault, audit, settings, or trace can ship.
|
||||
|
||||
---
|
||||
|
||||
## 1. Per-route trust classification
|
||||
|
||||
Every route must declare one classification before its first implementation PR.
|
||||
The classification gates what the UI is allowed to do.
|
||||
|
||||
| Route | Classification | Allowed UI operations | Forbidden |
|
||||
|-------------|------------------|-----------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
|
||||
| Chat | read + turn-submit | Submit turn; render `surface`; expose `walk_surface` / `articulation_surface` distinctly | Mutating identity axes, runtime policy, or operator code from user text |
|
||||
| Proposals | proposal-only | View queue/detail; accept/reject through existing proposal API | Bypassing review; direct pack/vault mutation |
|
||||
| Replay | read-only | Compare artifacts; render divergence | Writing artifacts; rewriting trace_hash; "fix" buttons |
|
||||
| Evals | read-only | Browse lanes/runs/metrics | Mutating lane definitions; suppressing failing cases from view |
|
||||
| Runs | read-only | List/filter artifacts; deep-link to Replay | Deletion; metadata edits |
|
||||
| Trace | read-only | Render three surfaces distinctly; node-by-node walk | Surface conflation; any write path |
|
||||
| Inspector | read-only | Display selected entity by type | Inline edits; cross-route mutation |
|
||||
| Vault | read-only | Hierarchical browse; grounding-source coloring | Entry creation/edit/delete; recall-semantic changes |
|
||||
| Packs | proposal-only | List; inspect contents; **propose** re-ingestion via existing CLI/proposal path | Direct re-ingestion buttons; pack_id strings reaching the filesystem without `_validate_pack_id` |
|
||||
| Audit | read-only | Browse `TurnVerdicts`; filter by clearance state; paginate | Aggregating/suppressing per-turn verdicts; "mark resolved" |
|
||||
| Settings | mutating (allowlisted) | Read/write only fields on the allowlist below | Any setting that affects closure thresholds, normalization sites, recall semantics, or identity packs |
|
||||
|
||||
### Settings allowlist (initial)
|
||||
|
||||
The Settings route may read **and** write only these fields. Anything outside
|
||||
this list is read-only or absent from the UI entirely.
|
||||
|
||||
- API endpoint override (UI ↔ backend wiring only)
|
||||
- Operator display preferences (theme, density, default route)
|
||||
- Telemetry sink toggles already exposed by `chat/telemetry.py`
|
||||
- Eval lane selection / default suite
|
||||
|
||||
Out of scope for the UI (operator CLI only):
|
||||
|
||||
- `versor_condition` threshold (1e-6, non-negotiable)
|
||||
- Algebra backend selection
|
||||
- Identity pack selection / `DEFAULT_IDENTITY_PACK`
|
||||
- Safety/ethics pack mounting
|
||||
- Vault recall thresholds and indexing parameters
|
||||
- Any normalization site (`ingest/gate.py`, `language_packs/compiler.py`, `algebra/versor.py`)
|
||||
|
||||
Adding a field to the allowlist requires an ADR.
|
||||
|
||||
---
|
||||
|
||||
## 2. Runtime Surface Contract obligations (Phase 3 — Trace route)
|
||||
|
||||
The Trace route is the surface where doctrine violations are easiest to commit
|
||||
silently. Three obligations:
|
||||
|
||||
1. **Distinct rendering.** `surface`, `walk_surface`, and `articulation_surface`
|
||||
are visually and semantically distinct in the UI. They are not collapsed into
|
||||
a single "response" pane.
|
||||
2. **Read-only.** No node, no edge, no turn is editable in v1. Drill-down to
|
||||
evidence is fine; mutation is not.
|
||||
3. **Trace-hash stability.** Rendering must not depend on or alter values that
|
||||
feed `trace_hash`. The Trace route PR must include a contract test asserting
|
||||
`trace_hash` is byte-identical before and after the route fetches and renders
|
||||
a session.
|
||||
|
||||
Reference: `docs/runtime_contracts.md`. Any change to the surface contract lands
|
||||
the doc update and contract test in the same PR (per CLAUDE.md §Runtime Surface
|
||||
Contract).
|
||||
|
||||
---
|
||||
|
||||
## 3. Audit fidelity guarantee (Phase 6 — Audit route)
|
||||
|
||||
The Audit route is the operator's window into `SafetyCheck` / `EthicsCheck`
|
||||
verdicts. It is load-bearing for catching regressions in fail-closed safety
|
||||
behavior. Therefore:
|
||||
|
||||
- Per-turn verdict cardinality is preserved. Pagination is allowed; aggregation
|
||||
that hides individual verdicts is not.
|
||||
- Clearance state coloring uses the existing tokens (`cleared`, `violated`,
|
||||
`unassessable`, `suppressed`) without remapping or merging.
|
||||
- No UI affordance "resolves," "suppresses," or "dismisses" a verdict. Operator
|
||||
response to verdicts happens in the engine, not the workbench.
|
||||
- Search/filter is permitted; saved views are permitted; mutation is not.
|
||||
|
||||
Reference: `audit-completeness`, `TurnVerdicts`, CLAUDE.md §Teaching Safety.
|
||||
|
||||
---
|
||||
|
||||
## 4. Pack route discipline (Phase 6 — Packs)
|
||||
|
||||
`language_packs/compiler.py` is one of three allowed normalization sites and is
|
||||
guarded by `_validate_pack_id` (ADR-0051) against path traversal.
|
||||
|
||||
- Pack re-ingestion is **proposal-only** in the UI. The UI may surface a
|
||||
"propose re-ingestion" affordance; it may not invoke the compiler directly.
|
||||
- Any `pack_id` displayed in the UI is treated as untrusted display data:
|
||||
centralized safe-display path, no filesystem operations driven by UI input.
|
||||
- Pack mutation proposals follow the existing reviewed-teaching pipeline.
|
||||
No parallel correction/learning path (CLAUDE.md §Teaching Safety).
|
||||
|
||||
---
|
||||
|
||||
## 5. Backend contract documentation
|
||||
|
||||
"Confirm with backend" (parent plan, Phase 5) is replaced by:
|
||||
|
||||
- Every new endpoint consumed by a workbench route is documented in
|
||||
`docs/runtime_contracts.md` (or a named sibling under `docs/contracts/`) in
|
||||
the same PR that introduces the TypeScript mirror in `src/types/api.ts`.
|
||||
- The Python handler and the TS mirror cite the same contract section.
|
||||
- Contract drift between Python and TS is a CI failure, not a code-review
|
||||
catch — add a parity test if one does not exist.
|
||||
|
||||
---
|
||||
|
||||
## 6. CLI-lane requirement
|
||||
|
||||
Any PR that adds or modifies a backend endpoint to support a workbench route
|
||||
must include the CLI lane that exercises it:
|
||||
|
||||
- Runtime-touching: `core test --suite runtime`
|
||||
- Cognition-touching: `core eval cognition`
|
||||
- Pack-touching: `core test --suite packs`
|
||||
- Teaching-touching: `core test --suite teaching`
|
||||
|
||||
UI-only PRs (no backend change) are exempt — `pnpm test` is sufficient.
|
||||
|
||||
---
|
||||
|
||||
## 7. Per-PR checklist (replaces the parent plan's "Validation checklist" for Phase 5+)
|
||||
|
||||
```text
|
||||
[ ] Route trust classification matches §1 of this addendum
|
||||
[ ] If Settings: every writable field is on the §1 allowlist (or an ADR adds it)
|
||||
[ ] If Trace: trace_hash stability test included; three surfaces rendered distinctly
|
||||
[ ] If Audit: no aggregation/suppression of per-turn verdicts
|
||||
[ ] If Packs: re-ingestion is proposal-only; pack_id flows through safe display
|
||||
[ ] New backend endpoints documented in docs/runtime_contracts.md (or sibling)
|
||||
[ ] TS mirror in src/types/api.ts cites the same contract section
|
||||
[ ] Relevant CLI lane added or extended (§6); pnpm test green
|
||||
[ ] PR description answers the four CLAUDE.md §PR Checklist questions:
|
||||
- What capability/property/boundary did this add/protect?
|
||||
- Which invariant proves the field remains valid?
|
||||
- Which CLI suite/eval proves the lane?
|
||||
- What trust boundary was enforced?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Proof-obligation discipline (CLAUDE.md §Schema-Defined Proof Obligations)
|
||||
|
||||
Schemas and types added to `src/types/api.ts` that nominally guarantee a
|
||||
structural property (e.g. a discriminated union for verdict states, a typed
|
||||
proposal lifecycle) carry the same obligation as backend schemas: an executing
|
||||
test must fail under the violations the type is written to catch.
|
||||
|
||||
If no such test exists, the type is decoration. Record the gap in the PR
|
||||
description rather than treating the type as load-bearing.
|
||||
|
||||
---
|
||||
|
||||
## 9. Phase applicability
|
||||
|
||||
| Phase | Addendum sections that apply |
|
||||
|--------------------------------------|-------------------------------------|
|
||||
| 1 — Polish existing routes | §2 (Chat surface distinction), §7 |
|
||||
| 2 — Runs | §1 (read-only), §7 |
|
||||
| 3 — Trace | §1, §2, §7 |
|
||||
| 4 — Inspector | §1 (read-only), §7 |
|
||||
| 5 — Contracts for placeholders | §1, §5, §7 |
|
||||
| 6 — Packs / Vault / Audit / Settings | §1, §3, §4, §5, §6, §7, §8 |
|
||||
| 7 — Command palette | §1 (no command bypasses §1 rules) |
|
||||
|
||||
---
|
||||
|
||||
## 10. Delete-when-done
|
||||
|
||||
This addendum is deleted together with the parent plan, after all phases merge
|
||||
and its rules have been folded into:
|
||||
|
||||
- `docs/runtime_contracts.md` (surface contract additions)
|
||||
- ADRs for any Settings-allowlist extensions
|
||||
- Workbench-specific contributing notes if needed
|
||||
|
||||
Until then, both files travel together.
|
||||
246
docs/plans/workbench-ui-continuation.md
Normal file
246
docs/plans/workbench-ui-continuation.md
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
# Workbench UI — Continuation Plan
|
||||
|
||||
> **Status:** temporary working plan — delete when all phases are merged
|
||||
> **Scope:** `workbench-ui/` only
|
||||
> **Tracks against:** current `main` @ `89defef`
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
The CORE Workbench is a React + Vite + TanStack Query frontend that serves as the
|
||||
primary operator interface for the CORE engine. The shell is structurally sound.
|
||||
Four feature routes are implemented. Six routes remain placeholders. The right
|
||||
inspector is a stub kept collapsed. This plan sequences the remaining work.
|
||||
|
||||
### What exists today
|
||||
|
||||
| Area | Status | Notes |
|
||||
|---|---|---|
|
||||
| Shell grid (TopBar / LeftNav / StatusFooter) | ✅ Live | Token-driven, tested |
|
||||
| Design token substrate (`src/design/`) | ✅ Complete | **Do not expand** except `EmptyState` / `CommandPalette` |
|
||||
| CommandPalette (⌘K) | ✅ Live | Wired in TopBar |
|
||||
| ApiErrorBoundary | ✅ Live | Wraps `<Outlet />` |
|
||||
| Chat route | ✅ Implemented | `src/app/chat/` — turn flow, trace drawer, evidence |
|
||||
| Proposals route | ✅ Implemented | `src/app/proposals/` — queue, detail, review lifecycle |
|
||||
| Replay route | ✅ Implemented | `src/app/replay/` — artifact comparison |
|
||||
| Evals route | ✅ Implemented | `src/app/evals/` — lanes, runs, metrics |
|
||||
| Trace route | 🔴 Placeholder | `src/routes/TraceRoutePlaceholder.tsx` |
|
||||
| Runs route | 🔴 Placeholder | `src/routes/RunsRoutePlaceholder.tsx` |
|
||||
| Packs route | 🔴 Placeholder | `src/routes/PacksRoutePlaceholder.tsx` |
|
||||
| Vault route | 🔴 Placeholder | `src/routes/VaultRoutePlaceholder.tsx` |
|
||||
| Audit route | 🔴 Placeholder | `src/routes/AuditRoutePlaceholder.tsx` |
|
||||
| Settings route | 🔴 Placeholder | `src/routes/SettingsRoutePlaceholder.tsx` |
|
||||
| RightInspector | 🔴 Stub | Hardcoded collapsed in `Shell.tsx` (W-027) |
|
||||
|
||||
### Existing query hooks (api/queries.ts)
|
||||
|
||||
- `useRuntimeStatus` — `/runtime/status`
|
||||
- `useChatTurn` — `/chat/turn`
|
||||
- `useArtifacts` — artifact listing
|
||||
- `useArtifact` — artifact detail
|
||||
- `useReplayComparison` — `/replay/compare`
|
||||
- `useProposals` — proposals listing
|
||||
- `useProposal` — proposal detail
|
||||
- `useEvalLanes` — eval lanes listing
|
||||
- `useEvalLane` — eval lane detail
|
||||
- `useEvalExecution` — eval run execution
|
||||
|
||||
**No hooks yet exist** for: runs, trace sessions, packs, vault, audit log, or settings.
|
||||
|
||||
---
|
||||
|
||||
## Constraints
|
||||
|
||||
- Documentation stays in Markdown, never HTML dashboards.
|
||||
- `src/design/` is a protected Branch 1 substrate. Only `EmptyState` and `CommandPalette` may be edited there.
|
||||
- New UI pieces go feature-local: `src/app/<feature>/` or `src/routes/<Feature>Route.tsx`.
|
||||
- Zero hardcoded hex values — always reference CSS custom properties from `tokens.css`.
|
||||
- No new state management libraries. Prefer TanStack Query + React `useState` / `useReducer` + `useContext` where needed.
|
||||
- All new routes need at minimum a smoke test and a loading/empty/error state.
|
||||
- Schema/type mirrors in `src/types/api.ts` must stay in sync with the Python backend.
|
||||
- PR discipline: small, load-bearing, each with tests. Follow existing ADR checklist.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
### Phase 1 — Polish the four implemented routes
|
||||
|
||||
**Branch prefix:** `workbench/WIP-1-*`
|
||||
**Goal:** Make Chat, Proposals, Replay, and Evals cohesive and cross-linked before adding new surfaces.
|
||||
|
||||
#### 1a — Chat
|
||||
|
||||
- [ ] Audit loading / empty / error states for consistency
|
||||
- [ ] Improve trace-hash and proposal-candidate navigation from within a turn response
|
||||
- [ ] Verify ⌘K command entries exist for `New chat session` and `Jump to proposal`
|
||||
- [ ] Extend `Shell.test.tsx`-style tests for any new interaction paths
|
||||
|
||||
#### 1b — Proposals
|
||||
|
||||
- [ ] Audit queue list loading / empty / error states
|
||||
- [ ] Strengthen proposal detail: add direct link to its source artifact and replay-comparison entry
|
||||
- [ ] Expose review-lifecycle color tokens (`review-pending`, `accepted`, `rejected`, `withdrawn`) consistently across list and detail views
|
||||
- [ ] Ensure suggested-CLI block is selectable / copyable
|
||||
|
||||
#### 1c — Replay
|
||||
|
||||
- [ ] Audit artifact-selector loading / empty / error states
|
||||
- [ ] Improve divergence-severity and metadata presentation clarity
|
||||
- [ ] Add a direct link from replay comparison back to the originating proposal where applicable
|
||||
- [ ] Add route tests for primary comparison path
|
||||
|
||||
#### 1d — Evals
|
||||
|
||||
- [ ] Audit lane list and eval run view states
|
||||
- [ ] Improve metric readability — consider aligned numeric columns
|
||||
- [ ] Add route tests for lane selection and run inspection
|
||||
|
||||
---
|
||||
|
||||
### Phase 2 — Build the Runs route
|
||||
|
||||
**Branch prefix:** `workbench/WIP-2-runs`
|
||||
**Goal:** Replace `RunsRoutePlaceholder.tsx` with a real first-version route.
|
||||
|
||||
**Why this is next:** The existing `useArtifacts` and `useArtifact` query hooks already provide the needed read paths. Runs is the closest unfinished route to having backend support.
|
||||
|
||||
- [ ] Create `src/app/runs/` directory
|
||||
- [ ] Build `RunsRoute.tsx`: artifact listing table — sortable by date, filterable by status
|
||||
- [ ] Build `RunDetail.tsx`: artifact detail panel — metadata, trace hash, proposal links, replay entry-point
|
||||
- [ ] Wire deep link from RunDetail → Replay comparison
|
||||
- [ ] Add loading / empty / error states
|
||||
- [ ] Update `App.tsx` to swap placeholder for the real route
|
||||
- [ ] Add route tests
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 — Build the Trace route
|
||||
|
||||
**Branch prefix:** `workbench/WIP-3-trace`
|
||||
**Goal:** Replace `TraceRoutePlaceholder.tsx` with a first-version standalone trace explorer built by promoting and reusing the chat trace model.
|
||||
|
||||
**Why this is next:** Chat already contains trace-oriented UI logic. Trace should be grown by promotion, not by inventing a separate graph system from scratch.
|
||||
|
||||
- [ ] Audit what `src/app/chat/` already exposes for trace viewing
|
||||
- [ ] Extract or adapt the trace model into a reusable `src/app/trace/` home
|
||||
- [ ] Build `TraceRoute.tsx`: trace session list / selector
|
||||
- [ ] Build `TraceDetail.tsx`: node-by-node state exploration using existing state-color tokens (`decoded`, `inferred`, `evidenced`, `contradicted`, etc.)
|
||||
- [ ] Read-only first — no mutations in v1
|
||||
- [ ] Add `useTraceSession` and `useTraceNode` query hooks to `queries.ts` once backend contract is confirmed
|
||||
- [ ] Update `App.tsx` to swap placeholder
|
||||
- [ ] Add route tests
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 — Activate the RightInspector
|
||||
|
||||
**Branch prefix:** `workbench/WIP-4-inspector`
|
||||
**Goal:** Replace the inspector stub and the hardcoded `inspectorCollapsed = true` in `Shell.tsx` with a real, read-only detail surface.
|
||||
|
||||
**Why this comes after Runs + Trace:** Entity shapes must be concrete before the inspector can meaningfully propagate them. Doing this first would mean speculating on schema.
|
||||
|
||||
- [ ] Add a minimal selection model: a `useInspectorStore` context/reducer that routes can publish into
|
||||
- [ ] Lift the inspector toggle into `TopBar` (icon button, rightmost, `aria-expanded`)
|
||||
- [ ] Replace `const inspectorCollapsed = true` with reactive state wired to the store
|
||||
- [ ] Implement `RightInspector.tsx` — reads selected entity from the store, renders detail by entity type
|
||||
- [ ] Support initial entity types: `artifact`, `proposal`, `trace-node`, `replay-diff`
|
||||
- [ ] Read-only in v1
|
||||
- [ ] Add tests for toggle behavior and entity rendering
|
||||
|
||||
---
|
||||
|
||||
### Phase 5 — Define contracts for remaining placeholder routes
|
||||
|
||||
**Branch prefix:** `workbench/WIP-5-contracts`
|
||||
**Goal:** Before building Packs, Vault, Audit, and Settings, confirm Python schema shapes and add TypeScript mirrors.
|
||||
|
||||
**These routes are still true placeholders with no corresponding query hooks. Contract work precedes UI depth.**
|
||||
|
||||
#### Packs
|
||||
- [ ] Confirm pack listing and pack-detail API shapes with backend
|
||||
- [ ] Add `usePackList` and `usePack` hooks to `queries.ts`
|
||||
- [ ] Add types to `src/types/api.ts`
|
||||
|
||||
#### Vault
|
||||
- [ ] Confirm vault entry listing and entry-detail API shapes
|
||||
- [ ] Add `useVaultEntries` and `useVaultEntry` hooks
|
||||
- [ ] Add types to `src/types/api.ts`
|
||||
|
||||
#### Audit
|
||||
- [ ] Confirm clearance-event log API shape (pagination, filters)
|
||||
- [ ] Add `useAuditLog` hook
|
||||
- [ ] Add types to `src/types/api.ts`
|
||||
|
||||
#### Settings
|
||||
- [ ] Enumerate mutable runtime settings and confirm read/write API contract
|
||||
- [ ] Decide which settings are operator-only (read only in UI) vs. writable from the workbench
|
||||
- [ ] Add `useSettings` and `useUpdateSettings` hooks
|
||||
|
||||
---
|
||||
|
||||
### Phase 6 — Implement remaining routes
|
||||
|
||||
**Branch prefix:** `workbench/WIP-6-*`
|
||||
**Goal:** One route per PR, in this priority order based on anticipated backend readiness.
|
||||
|
||||
1. **Vault** — hierarchical grounded knowledge browser; uses grounding-source color tokens (`grounding-teaching`, `grounding-pack`, `grounding-vault`, `grounding-oov`, etc.)
|
||||
2. **Packs** — pack management: list, inspect contents, trigger re-ingestion; piggybacks on vault color system
|
||||
3. **Audit** — temporal clearance-event log; uses clearance tokens (`cleared`, `violated`, `unassessable`, `suppressed`); time-sorted, paginated table
|
||||
4. **Settings** — runtime config surface: model selection, threshold sliders, API endpoint override; write paths gated behind explicit confirmation
|
||||
|
||||
Each must have:
|
||||
- [ ] Loading / empty / error states
|
||||
- [ ] Route tests
|
||||
- [ ] Updated `App.tsx` wiring (placeholder swap)
|
||||
- [ ] ⌘K command entries
|
||||
|
||||
---
|
||||
|
||||
### Phase 7 — ⌘K command registry completion
|
||||
|
||||
**Branch prefix:** `workbench/WIP-7-command-palette`
|
||||
**Goal:** Fill the CommandPalette command registry with live, route-aware entries.
|
||||
|
||||
- [ ] Route navigation: all 10 nav items accessible via ⌘K
|
||||
- [ ] Context-sensitive: `Jump to run #N`, `Open vault entry`, `Inspect proposal`, `Compare artifacts`
|
||||
- [ ] Live search over recent runs and sessions via existing query hooks
|
||||
- [ ] Keyboard-first: full arrow-key navigation, Escape to close, Enter to activate
|
||||
|
||||
---
|
||||
|
||||
## Validation checklist (per PR)
|
||||
|
||||
```
|
||||
cd workbench-ui
|
||||
pnpm test
|
||||
pnpm test:enum-coverage
|
||||
```
|
||||
|
||||
Before merging a placeholder-swap PR:
|
||||
- [ ] Old placeholder file removed from `src/routes/`
|
||||
- [ ] New route wired in `App.tsx`
|
||||
- [ ] New route has at minimum: loading state, empty state, error state, one smoke test
|
||||
- [ ] No hardcoded hex values
|
||||
- [ ] No new dependencies added without discussion
|
||||
|
||||
---
|
||||
|
||||
## Done criteria
|
||||
|
||||
This plan is complete when:
|
||||
|
||||
- [ ] All four existing routes are cohesive, cross-linked, and consistently handle all async states
|
||||
- [ ] `Runs` is a live route backed by artifact/replay query hooks
|
||||
- [ ] `Trace` is a live route promoted from the chat trace model
|
||||
- [ ] `RightInspector` is a real, toggleable, read-only inspection surface
|
||||
- [ ] `Packs`, `Vault`, `Audit`, `Settings` have confirmed backend contracts and TypeScript mirrors
|
||||
- [ ] All six placeholder files in `src/routes/` are replaced and deleted
|
||||
- [ ] ⌘K command registry covers all 10 routes and key entity-jump commands
|
||||
- [ ] `src/design/` remains untouched except for `EmptyState` / `CommandPalette` allowed edits
|
||||
- [ ] Every route has tests
|
||||
|
||||
---
|
||||
|
||||
*Delete this file once all items above are checked and the final PR in each phase is merged.*
|
||||
|
|
@ -5,7 +5,7 @@ export function EvalMetricGrid({
|
|||
}: {
|
||||
metrics: Record<string, unknown>;
|
||||
}) {
|
||||
const sortedKeys = Object.keys(metrics).sort();
|
||||
const allKeys = Object.keys(metrics);
|
||||
|
||||
function formatValue(value: unknown): string {
|
||||
if (typeof value === "boolean") {
|
||||
|
|
@ -34,6 +34,20 @@ export function EvalMetricGrid({
|
|||
return undefined;
|
||||
}
|
||||
|
||||
function isLowerIsBetter(key: string): boolean {
|
||||
const k = key.toLowerCase();
|
||||
return (
|
||||
k.includes("latency") ||
|
||||
k.includes("duration") ||
|
||||
k.includes("time") ||
|
||||
k.includes("error") ||
|
||||
k.includes("fail") ||
|
||||
k.includes("fabrication") ||
|
||||
k.includes("divergence") ||
|
||||
k.includes("cost")
|
||||
);
|
||||
}
|
||||
|
||||
function renderPassFailBadge(key: string, value: unknown) {
|
||||
const isBool = typeof value === "boolean";
|
||||
const k = key.toLowerCase();
|
||||
|
|
@ -65,33 +79,159 @@ export function EvalMetricGrid({
|
|||
return null;
|
||||
}
|
||||
|
||||
// Find pairs
|
||||
const pairedKeys = new Set<string>();
|
||||
const pairs: Array<{
|
||||
key: string;
|
||||
actual: unknown;
|
||||
target: unknown;
|
||||
}> = [];
|
||||
|
||||
// 1. Group "passed" and "total"
|
||||
if (allKeys.includes("passed") && allKeys.includes("total")) {
|
||||
pairs.push({
|
||||
key: "passed",
|
||||
actual: metrics["passed"],
|
||||
target: metrics["total"],
|
||||
});
|
||||
pairedKeys.add("passed");
|
||||
pairedKeys.add("total");
|
||||
}
|
||||
|
||||
// 2. Group "X" and "X_target" or "target_X"
|
||||
allKeys.forEach((k) => {
|
||||
if (pairedKeys.has(k)) return;
|
||||
|
||||
if (k.endsWith("_target")) {
|
||||
const baseKey = k.slice(0, -7);
|
||||
if (allKeys.includes(baseKey)) {
|
||||
pairs.push({
|
||||
key: baseKey,
|
||||
actual: metrics[baseKey],
|
||||
target: metrics[k],
|
||||
});
|
||||
pairedKeys.add(baseKey);
|
||||
pairedKeys.add(k);
|
||||
}
|
||||
} else if (k.startsWith("target_")) {
|
||||
const baseKey = k.slice(7);
|
||||
if (allKeys.includes(baseKey)) {
|
||||
pairs.push({
|
||||
key: baseKey,
|
||||
actual: metrics[baseKey],
|
||||
target: metrics[k],
|
||||
});
|
||||
pairedKeys.add(baseKey);
|
||||
pairedKeys.add(k);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// 3. Remaining singles
|
||||
const singles: Array<{ key: string; value: unknown }> = [];
|
||||
allKeys.forEach((k) => {
|
||||
if (!pairedKeys.has(k)) {
|
||||
singles.push({ key: k, value: metrics[k] });
|
||||
}
|
||||
});
|
||||
|
||||
interface DisplayMetric {
|
||||
key: string;
|
||||
isPair: boolean;
|
||||
actual: unknown;
|
||||
target?: unknown;
|
||||
value?: unknown;
|
||||
}
|
||||
|
||||
const list: DisplayMetric[] = [
|
||||
...pairs.map((p) => ({ key: p.key, isPair: true, actual: p.actual, target: p.target })),
|
||||
...singles.map((s) => ({ key: s.key, isPair: false, actual: s.value, value: s.value })),
|
||||
];
|
||||
|
||||
// Sort lexicographically
|
||||
list.sort((a, b) => a.key.localeCompare(b.key));
|
||||
|
||||
return (
|
||||
<div className="grid gap-3 sm:grid-cols-2 md:grid-cols-3" data-testid="eval-metric-grid">
|
||||
{sortedKeys.map((key) => {
|
||||
const val = metrics[key];
|
||||
{list.map((item) => {
|
||||
const key = item.key;
|
||||
const unit = getUnit(key);
|
||||
const badge = renderPassFailBadge(key, val);
|
||||
const formatted = formatValue(val);
|
||||
const isObj = typeof val === "object" && val !== null;
|
||||
|
||||
return (
|
||||
<div
|
||||
key={key}
|
||||
className="flex flex-col justify-between rounded-lg border border-[var(--color-border-subtle)] bg-[var(--color-surface-raised)] p-3 shadow-sm"
|
||||
data-testid="metric-card"
|
||||
>
|
||||
<div>
|
||||
<div className="text-[10px] font-semibold text-[var(--color-text-muted)] uppercase tracking-wider truncate" title={key}>
|
||||
{key.replaceAll("_", " ")}
|
||||
</div>
|
||||
<div className={`mt-1 font-mono text-sm font-semibold text-[var(--color-text-primary)] ${isObj ? "whitespace-pre overflow-x-auto text-[10px] bg-[var(--color-surface-inset)] p-1.5 rounded" : ""}`}>
|
||||
{formatted}
|
||||
{unit && <span className="ml-1 text-xs text-[var(--color-text-muted)] font-sans font-normal">{unit}</span>}
|
||||
if (item.isPair) {
|
||||
const actualVal = item.actual;
|
||||
const targetVal = item.target;
|
||||
const badge = renderPassFailBadge(key, actualVal);
|
||||
|
||||
const actualNum = Number(actualVal);
|
||||
const targetNum = Number(targetVal);
|
||||
const hasNumericValues = !isNaN(actualNum) && !isNaN(targetNum);
|
||||
const isFailing =
|
||||
hasNumericValues &&
|
||||
(isLowerIsBetter(key) ? actualNum > targetNum : actualNum < targetNum);
|
||||
|
||||
const formattedActual = formatValue(actualVal);
|
||||
const formattedTarget = formatValue(targetVal);
|
||||
|
||||
return (
|
||||
<div
|
||||
key={key}
|
||||
className="flex flex-col justify-between rounded-lg border border-[var(--color-border-subtle)] bg-[var(--color-surface-raised)] p-3 shadow-sm"
|
||||
data-testid="metric-card"
|
||||
>
|
||||
<div>
|
||||
<div className="text-[10px] font-semibold text-[var(--color-text-muted)] uppercase tracking-wider truncate" title={key}>
|
||||
{key.replaceAll("_", " ")}
|
||||
</div>
|
||||
<div className="mt-1 font-mono text-sm font-semibold text-right tabular-nums">
|
||||
<span
|
||||
className={
|
||||
isFailing
|
||||
? "text-[var(--color-state-contradicted)] font-bold"
|
||||
: "text-[var(--color-text-primary)]"
|
||||
}
|
||||
>
|
||||
{formattedActual}
|
||||
</span>
|
||||
<span className="text-[var(--color-text-muted)] mx-1">/</span>
|
||||
<span className="text-[var(--color-text-secondary)] font-normal">
|
||||
{formattedTarget}
|
||||
</span>
|
||||
{unit && <span className="ml-1 text-xs text-[var(--color-text-muted)] font-sans font-normal">{unit}</span>}
|
||||
</div>
|
||||
</div>
|
||||
{badge && <div className="mt-2 flex justify-start">{badge}</div>}
|
||||
</div>
|
||||
{badge && <div className="mt-2 flex justify-start">{badge}</div>}
|
||||
</div>
|
||||
);
|
||||
);
|
||||
} else {
|
||||
const val = item.value;
|
||||
const badge = renderPassFailBadge(key, val);
|
||||
const formatted = formatValue(val);
|
||||
const isObj = typeof val === "object" && val !== null;
|
||||
const isNum = typeof val === "number";
|
||||
|
||||
return (
|
||||
<div
|
||||
key={key}
|
||||
className="flex flex-col justify-between rounded-lg border border-[var(--color-border-subtle)] bg-[var(--color-surface-raised)] p-3 shadow-sm"
|
||||
data-testid="metric-card"
|
||||
>
|
||||
<div>
|
||||
<div className="text-[10px] font-semibold text-[var(--color-text-muted)] uppercase tracking-wider truncate" title={key}>
|
||||
{key.replaceAll("_", " ")}
|
||||
</div>
|
||||
<div
|
||||
className={`mt-1 font-mono text-sm font-semibold text-[var(--color-text-primary)] ${
|
||||
isNum ? "text-right tabular-nums" : ""
|
||||
} ${isObj ? "whitespace-pre overflow-x-auto text-[10px] bg-[var(--color-surface-inset)] p-1.5 rounded" : ""}`}
|
||||
>
|
||||
{formatted}
|
||||
{unit && <span className="ml-1 text-xs text-[var(--color-text-muted)] font-sans font-normal">{unit}</span>}
|
||||
</div>
|
||||
</div>
|
||||
{badge && <div className="mt-2 flex justify-start">{badge}</div>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
|
|
|
|||
|
|
@ -12,10 +12,33 @@ import { LoadingState } from "../../design/components/states/LoadingState";
|
|||
import type { EvalRunResult } from "../../types/api";
|
||||
import { WorkbenchApiError } from "../../api/client";
|
||||
|
||||
import { useState, useEffect } from "react";
|
||||
import { useSearchParams } from "react-router-dom";
|
||||
import { useEvalLanes, useEvalLane } from "../../api/queries";
|
||||
import { EvalLaneCard } from "./EvalLaneCard";
|
||||
import { EvalRunButton } from "./EvalRunButton";
|
||||
import { EvalMetricGrid } from "./EvalMetricGrid";
|
||||
import { EvalFailureViewer } from "./EvalFailureViewer";
|
||||
import { EvalArtifactLink } from "./EvalArtifactLink";
|
||||
import { EmptyState } from "../../design/components/states/EmptyState";
|
||||
import { ErrorState } from "../../design/components/states/ErrorState";
|
||||
import { LoadingState } from "../../design/components/states/LoadingState";
|
||||
import type { EvalRunResult } from "../../types/api";
|
||||
import { WorkbenchApiError } from "../../api/client";
|
||||
import { useQueryClient } from "@tanstack/react-query";
|
||||
|
||||
export function EvalsRoute() {
|
||||
const { data: lanes, isLoading, isError, error } = useEvalLanes();
|
||||
const [searchParams, setSearchParams] = useSearchParams();
|
||||
const selectedLaneName = searchParams.get("lane") || "";
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
const {
|
||||
data: lastRunResult,
|
||||
isLoading: isLaneLoading,
|
||||
isError: isLaneError,
|
||||
error: laneError,
|
||||
} = useEvalLane(selectedLaneName);
|
||||
|
||||
// Maintain per-lane run states (pending, result, error)
|
||||
const [runStates, setRunStates] = useState<
|
||||
|
|
@ -29,6 +52,13 @@ export function EvalsRoute() {
|
|||
>
|
||||
>({});
|
||||
|
||||
// Auto-select the first lane on load if no selection is in the URL
|
||||
useEffect(() => {
|
||||
if (!selectedLaneName && lanes && lanes.length > 0) {
|
||||
setSearchParams({ lane: lanes[0].lane }, { replace: true });
|
||||
}
|
||||
}, [selectedLaneName, lanes, setSearchParams]);
|
||||
|
||||
if (isLoading) {
|
||||
return <LoadingState label="Loading eval lanes..." />;
|
||||
}
|
||||
|
|
@ -46,6 +76,178 @@ export function EvalsRoute() {
|
|||
|
||||
const selectedLane = lanes?.find((l) => l.lane === selectedLaneName) || null;
|
||||
const currentRunState = selectedLaneName ? runStates[selectedLaneName] : null;
|
||||
const isNotFoundError =
|
||||
laneError instanceof WorkbenchApiError && laneError.code === "not_found";
|
||||
|
||||
function renderHeaderAndForm(lane: typeof selectedLane) {
|
||||
if (!lane) return null;
|
||||
return (
|
||||
<>
|
||||
{/* Header info */}
|
||||
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-[var(--color-border-subtle)] pb-3">
|
||||
<div>
|
||||
<h2 className="text-lg font-semibold text-[var(--color-text-primary)] font-mono">
|
||||
{lane.lane}
|
||||
</h2>
|
||||
{lane.description && (
|
||||
<p className="mt-1 text-xs text-[var(--color-text-secondary)]">
|
||||
{lane.description}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Run Button configuration if read-only */}
|
||||
{lane.read_only ? (
|
||||
<EvalRunButton
|
||||
lane={lane}
|
||||
onRunStart={() => {
|
||||
setRunStates((prev) => ({
|
||||
...prev,
|
||||
[lane.lane]: { isPending: true },
|
||||
}));
|
||||
}}
|
||||
onRunSuccess={(result) => {
|
||||
setRunStates((prev) => ({
|
||||
...prev,
|
||||
[lane.lane]: { isPending: false, result },
|
||||
}));
|
||||
queryClient.invalidateQueries({ queryKey: ["api", "eval", lane.lane] });
|
||||
}}
|
||||
onRunError={(err) => {
|
||||
setRunStates((prev) => ({
|
||||
...prev,
|
||||
[lane.lane]: { isPending: false, error: err },
|
||||
}));
|
||||
}}
|
||||
/>
|
||||
) : (
|
||||
<div className="text-xs text-[var(--color-text-muted)] bg-[var(--color-surface-inset)] p-3 rounded-lg border border-[var(--color-border-subtle)] font-medium">
|
||||
⚠️ API runs disabled for write-active lane. Use local CLI to execute this lane:
|
||||
<code className="block mt-1 font-mono text-[var(--color-text-primary)] bg-[var(--color-surface-base)] p-1.5 rounded">
|
||||
core eval --lane {lane.lane}
|
||||
</code>
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
const renderRightPane = () => {
|
||||
if (!selectedLaneName) {
|
||||
return (
|
||||
<EmptyState
|
||||
statement="Select an eval lane from the list to view results or run checks."
|
||||
nextAction={{ kind: "cli", command: "core eval --list" }}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (!selectedLane) {
|
||||
return (
|
||||
<EmptyState
|
||||
statement={`Selected lane "${selectedLaneName}" not found.`}
|
||||
nextAction={{ kind: "cli", command: "core eval --list" }}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
// 1. Loading state for fetching the lane's last run details
|
||||
if (isLaneLoading && !lastRunResult) {
|
||||
return <LoadingState label="Loading eval lane details..." />;
|
||||
}
|
||||
|
||||
// 2. Error state for fetching the lane details (excluding not_found)
|
||||
if (isLaneError && !isNotFoundError) {
|
||||
return (
|
||||
<ErrorState
|
||||
whatFailed={laneError instanceof Error ? laneError.message : "Failed to load eval lane details."}
|
||||
mutationStatus="No corpus mutation occurred."
|
||||
reproducer={`core eval --lane ${selectedLaneName}`}
|
||||
retrySafety="Retry: safe"
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
// 3. Current run execution state
|
||||
if (currentRunState?.isPending) {
|
||||
return (
|
||||
<div className="flex flex-col gap-4">
|
||||
{renderHeaderAndForm(selectedLane)}
|
||||
<LoadingState label="Running eval lane..." />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (currentRunState?.error) {
|
||||
return (
|
||||
<div className="flex flex-col gap-4">
|
||||
{renderHeaderAndForm(selectedLane)}
|
||||
<ErrorState
|
||||
whatFailed={currentRunState.error.message}
|
||||
mutationStatus="No corpus mutation occurred."
|
||||
reproducer={`core eval --lane ${selectedLane.lane}`}
|
||||
retrySafety="Retry: safe"
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
// 4. Success or Empty state
|
||||
const result = currentRunState?.result || (lastRunResult as EvalRunResult | undefined);
|
||||
if (result && result.metrics) {
|
||||
return (
|
||||
<div className="flex flex-col gap-4">
|
||||
{renderHeaderAndForm(selectedLane)}
|
||||
<div className="flex flex-wrap items-center gap-3">
|
||||
<span className="text-xs text-[var(--color-text-secondary)] font-semibold">
|
||||
Status:
|
||||
</span>
|
||||
<span
|
||||
className={`rounded-md px-2 py-0.5 text-xs font-semibold ${
|
||||
result.passed
|
||||
? "bg-[var(--color-state-success-bg)] text-[var(--color-state-success-text)] border border-[var(--color-state-success-border)]"
|
||||
: "bg-[var(--color-state-danger-bg)] text-[var(--color-state-danger-text)] border border-[var(--color-state-danger-border)]"
|
||||
}`}
|
||||
>
|
||||
{result.passed ? "Passed" : "Failed"}
|
||||
</span>
|
||||
{result.source_digest && (
|
||||
<EvalArtifactLink
|
||||
lane={selectedLane.lane}
|
||||
sourceDigest={result.source_digest}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-2">
|
||||
<h3 className="text-sm font-semibold text-[var(--color-text-primary)]">Metrics</h3>
|
||||
<EvalMetricGrid metrics={result.metrics} />
|
||||
</div>
|
||||
|
||||
<EvalFailureViewer
|
||||
cases={result.cases}
|
||||
passed={result.passed}
|
||||
laneName={selectedLane.lane}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex flex-col gap-4">
|
||||
{renderHeaderAndForm(selectedLane)}
|
||||
<EmptyState
|
||||
statement={
|
||||
selectedLane.read_only
|
||||
? `No run results for lane "${selectedLane.lane}" in this session. Trigger a run above.`
|
||||
: `Eval lane "${selectedLane.lane}" is CLI-only. No session results available.`
|
||||
}
|
||||
nextAction={{ kind: "cli", command: `core eval --lane ${selectedLane.lane}` }}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="grid h-full grid-cols-1 gap-4 md:grid-cols-[18rem_1fr]" data-testid="evals-route">
|
||||
|
|
@ -73,115 +275,7 @@ export function EvalsRoute() {
|
|||
|
||||
{/* Right Pane: Results / Form */}
|
||||
<div className="flex flex-col gap-4 overflow-y-auto pl-2">
|
||||
{selectedLane ? (
|
||||
<>
|
||||
{/* Header info */}
|
||||
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-[var(--color-border-subtle)] pb-3">
|
||||
<div>
|
||||
<h2 className="text-lg font-semibold text-[var(--color-text-primary)] font-mono">
|
||||
{selectedLane.lane}
|
||||
</h2>
|
||||
{selectedLane.description && (
|
||||
<p className="mt-1 text-xs text-[var(--color-text-secondary)]">
|
||||
{selectedLane.description}
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Run Button configuration if read-only */}
|
||||
{selectedLane.read_only ? (
|
||||
<EvalRunButton
|
||||
lane={selectedLane}
|
||||
onRunStart={() => {
|
||||
setRunStates((prev) => ({
|
||||
...prev,
|
||||
[selectedLane.lane]: { isPending: true },
|
||||
}));
|
||||
}}
|
||||
onRunSuccess={(result) => {
|
||||
setRunStates((prev) => ({
|
||||
...prev,
|
||||
[selectedLane.lane]: { isPending: false, result },
|
||||
}));
|
||||
}}
|
||||
onRunError={(err) => {
|
||||
setRunStates((prev) => ({
|
||||
...prev,
|
||||
[selectedLane.lane]: { isPending: false, error: err },
|
||||
}));
|
||||
}}
|
||||
/>
|
||||
) : (
|
||||
<div className="text-xs text-[var(--color-text-muted)] bg-[var(--color-surface-inset)] p-3 rounded-lg border border-[var(--color-border-subtle)] font-medium">
|
||||
⚠️ API runs disabled for write-active lane. Use local CLI to execute this lane:
|
||||
<code className="block mt-1 font-mono text-[var(--color-text-primary)] bg-[var(--color-surface-base)] p-1.5 rounded">
|
||||
core eval --lane {selectedLane.lane}
|
||||
</code>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Result display */}
|
||||
{currentRunState?.isPending ? (
|
||||
<LoadingState label="Running eval lane..." />
|
||||
) : currentRunState?.error ? (
|
||||
<ErrorState
|
||||
whatFailed={currentRunState.error.message}
|
||||
mutationStatus="No corpus mutation occurred."
|
||||
reproducer={`core eval --lane ${selectedLane.lane}`}
|
||||
retrySafety="Retry: safe"
|
||||
/>
|
||||
) : currentRunState?.result ? (
|
||||
<div className="flex flex-col gap-4">
|
||||
<div className="flex flex-wrap items-center gap-3">
|
||||
<span className="text-xs text-[var(--color-text-secondary)] font-semibold">
|
||||
Status:
|
||||
</span>
|
||||
<span
|
||||
className={`rounded-md px-2 py-0.5 text-xs font-semibold ${
|
||||
currentRunState.result.passed
|
||||
? "bg-[var(--color-state-success-bg)] text-[var(--color-state-success-text)] border border-[var(--color-state-success-border)]"
|
||||
: "bg-[var(--color-state-danger-bg)] text-[var(--color-state-danger-text)] border border-[var(--color-state-danger-border)]"
|
||||
}`}
|
||||
>
|
||||
{currentRunState.result.passed ? "Passed" : "Failed"}
|
||||
</span>
|
||||
{currentRunState.result.source_digest && (
|
||||
<EvalArtifactLink
|
||||
lane={selectedLane.lane}
|
||||
sourceDigest={currentRunState.result.source_digest}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-2">
|
||||
<h3 className="text-sm font-semibold text-[var(--color-text-primary)]">Metrics</h3>
|
||||
<EvalMetricGrid metrics={currentRunState.result.metrics} />
|
||||
</div>
|
||||
|
||||
<EvalFailureViewer
|
||||
cases={currentRunState.result.cases}
|
||||
passed={currentRunState.result.passed}
|
||||
laneName={selectedLane.lane}
|
||||
/>
|
||||
</div>
|
||||
) : (
|
||||
<EmptyState
|
||||
statement={
|
||||
selectedLane.read_only
|
||||
? `No run results for lane "${selectedLane.lane}" in this session. Trigger a run above.`
|
||||
: `Eval lane "${selectedLane.lane}" is CLI-only. No session results available.`
|
||||
}
|
||||
nextAction={{ kind: "cli", command: `core eval --lane ${selectedLane.lane}` }}
|
||||
/>
|
||||
)}
|
||||
</>
|
||||
) : (
|
||||
<EmptyState
|
||||
statement="Select an eval lane from the list to view results or run checks."
|
||||
nextAction={{ kind: "cli", command: "core eval --list" }}
|
||||
/>
|
||||
)}
|
||||
{renderRightPane()}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
|
|
|
|||
|
|
@ -16,11 +16,13 @@ vi.mock("../../api/queries", async (importOriginal) => {
|
|||
return {
|
||||
...actual,
|
||||
useEvalLanes: vi.fn(),
|
||||
useEvalLane: vi.fn(),
|
||||
useEvalRun: vi.fn(),
|
||||
};
|
||||
});
|
||||
|
||||
import { useEvalLanes, useEvalRun } from "../../api/queries";
|
||||
import { useEvalLanes, useEvalLane, useEvalRun } from "../../api/queries";
|
||||
import { CommandPalette } from "../../design/components/primitives/CommandPalette";
|
||||
|
||||
const mockLanes: EvalLaneSummary[] = [
|
||||
{ lane: "contemplation_quality", versions: ["v1", "v2"], read_only: true, description: "Contemplation checks" },
|
||||
|
|
@ -49,6 +51,27 @@ describe("W-030 Component Tests", () => {
|
|||
|
||||
beforeEach(() => {
|
||||
vi.resetAllMocks();
|
||||
|
||||
vi.mocked(useEvalLanes).mockReturnValue({
|
||||
data: mockLanes,
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
error: null,
|
||||
} as any);
|
||||
vi.mocked(useEvalLane).mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
error: new WorkbenchApiError("not_found", "No run history found"),
|
||||
} as any);
|
||||
vi.mocked(useEvalRun).mockReturnValue({
|
||||
mutate: vi.fn(),
|
||||
isPending: false,
|
||||
isError: false,
|
||||
isSuccess: false,
|
||||
error: null,
|
||||
} as any);
|
||||
|
||||
fetchMock.mockImplementation((url: any) => {
|
||||
const urlStr = typeof url === "string" ? url : String(url?.url || url || "");
|
||||
if (urlStr.endsWith("/evals")) {
|
||||
|
|
@ -162,10 +185,45 @@ describe("W-030 Component Tests", () => {
|
|||
|
||||
// Verify lexicographical order
|
||||
expect(cards1).toEqual(cards2);
|
||||
expect(cards1.length).toBe(3);
|
||||
expect(cards1[0]).toContain("accuracy");
|
||||
expect(cards1[1]).toContain("latency ms");
|
||||
expect(cards1[2]).toContain("passed");
|
||||
expect(cards1[3]).toContain("total");
|
||||
expect(cards1[2]).toContain("9/10");
|
||||
});
|
||||
|
||||
it("renders actual / target, emphasizes actual, and applies failure color token when target is not met", () => {
|
||||
const metrics = {
|
||||
passed: 4,
|
||||
total: 5,
|
||||
latency_ms: 150,
|
||||
latency_ms_target: 100, // failing because actual > target
|
||||
accuracy: 0.9,
|
||||
accuracy_target: 0.95, // failing because actual < target
|
||||
};
|
||||
|
||||
const { container } = render(<EvalMetricGrid metrics={metrics} />);
|
||||
|
||||
const cards = container.querySelectorAll('[data-testid="metric-card"]');
|
||||
expect(cards.length).toBe(3);
|
||||
|
||||
const passedCard = Array.from(cards).find(c => c.textContent?.includes("passed"));
|
||||
expect(passedCard).toBeDefined();
|
||||
expect(passedCard?.querySelector(".text-\\[var\\(--color-state-contradicted\\)\\]")).toBeInTheDocument();
|
||||
expect(passedCard?.textContent).toContain("4");
|
||||
expect(passedCard?.textContent).toContain("5");
|
||||
|
||||
const latencyCard = Array.from(cards).find(c => c.textContent?.includes("latency ms"));
|
||||
expect(latencyCard).toBeDefined();
|
||||
expect(latencyCard?.querySelector(".text-\\[var\\(--color-state-contradicted\\)\\]")).toBeInTheDocument();
|
||||
expect(latencyCard?.textContent).toContain("150");
|
||||
expect(latencyCard?.textContent).toContain("100");
|
||||
|
||||
const accuracyCard = Array.from(cards).find(c => c.textContent?.includes("accuracy"));
|
||||
expect(accuracyCard).toBeDefined();
|
||||
expect(accuracyCard?.querySelector(".text-\\[var\\(--color-state-contradicted\\)\\]")).toBeInTheDocument();
|
||||
expect(accuracyCard?.textContent).toContain("0.9");
|
||||
expect(accuracyCard?.textContent).toContain("0.95");
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -295,5 +353,115 @@ describe("W-030 Component Tests", () => {
|
|||
const res = await runEvalLane({ lane: "contemplation_quality", split: "holdout" });
|
||||
expect(res).toEqual(mockResult);
|
||||
});
|
||||
|
||||
it("renders loading state when lane list is loading", () => {
|
||||
vi.mocked(useEvalLanes).mockReturnValue({
|
||||
data: null,
|
||||
isLoading: true,
|
||||
isError: false,
|
||||
error: null,
|
||||
} as any);
|
||||
|
||||
const client = makeClient();
|
||||
render(
|
||||
<QueryClientProvider client={client}>
|
||||
<MemoryRouter initialEntries={["/evals"]}>
|
||||
<EvalsRoute />
|
||||
</MemoryRouter>
|
||||
</QueryClientProvider>
|
||||
);
|
||||
|
||||
expect(screen.getByText("Loading eval lanes...")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("renders empty state when there are no lanes", () => {
|
||||
vi.mocked(useEvalLanes).mockReturnValue({
|
||||
data: [],
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
error: null,
|
||||
} as any);
|
||||
|
||||
const client = makeClient();
|
||||
render(
|
||||
<QueryClientProvider client={client}>
|
||||
<MemoryRouter initialEntries={["/evals"]}>
|
||||
<EvalsRoute />
|
||||
</MemoryRouter>
|
||||
</QueryClientProvider>
|
||||
);
|
||||
|
||||
expect(screen.getByText("No eval lanes discovered.")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("renders loading and error states for the selected lane detail view", () => {
|
||||
vi.mocked(useEvalLanes).mockReturnValue({
|
||||
data: mockLanes,
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
error: null,
|
||||
} as any);
|
||||
|
||||
// Mock useEvalLane to return loading
|
||||
vi.mocked(useEvalLane).mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: true,
|
||||
isError: false,
|
||||
error: null,
|
||||
} as any);
|
||||
|
||||
const client = makeClient();
|
||||
const { rerender } = render(
|
||||
<QueryClientProvider client={client}>
|
||||
<MemoryRouter initialEntries={["/evals?lane=contemplation_quality"]}>
|
||||
<EvalsRoute />
|
||||
</MemoryRouter>
|
||||
</QueryClientProvider>
|
||||
);
|
||||
|
||||
expect(screen.getByText("Loading eval lane details...")).toBeInTheDocument();
|
||||
|
||||
// Mock useEvalLane to return error (other than not_found)
|
||||
vi.mocked(useEvalLane).mockReturnValue({
|
||||
data: undefined,
|
||||
isLoading: false,
|
||||
isError: true,
|
||||
error: new WorkbenchApiError("read_error", "Disk read error"),
|
||||
} as any);
|
||||
|
||||
rerender(
|
||||
<QueryClientProvider client={client}>
|
||||
<MemoryRouter initialEntries={["/evals?lane=contemplation_quality"]}>
|
||||
<EvalsRoute />
|
||||
</MemoryRouter>
|
||||
</QueryClientProvider>
|
||||
);
|
||||
|
||||
expect(screen.getByText("Disk read error")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("CommandPalette registers dynamic command entries driven by useEvalLanes", () => {
|
||||
vi.mocked(useEvalLanes).mockReturnValue({
|
||||
data: mockLanes,
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
error: null,
|
||||
} as any);
|
||||
|
||||
const client = makeClient();
|
||||
render(
|
||||
<QueryClientProvider client={client}>
|
||||
<MemoryRouter>
|
||||
<CommandPalette open={true} onOpenChange={vi.fn()} />
|
||||
</MemoryRouter>
|
||||
</QueryClientProvider>
|
||||
);
|
||||
|
||||
// Verify static commands
|
||||
expect(screen.getByRole("button", { name: "Open Chat" })).toBeInTheDocument();
|
||||
// Verify dynamic commands
|
||||
expect(screen.getByRole("button", { name: "Open eval lane contemplation_quality" })).toBeInTheDocument();
|
||||
expect(screen.getByRole("button", { name: "Open eval lane unsafe_lane" })).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,10 +1,18 @@
|
|||
import { render, screen } from "@testing-library/react";
|
||||
import userEvent from "@testing-library/user-event";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { MemoryRouter } from "react-router-dom";
|
||||
import { useState } from "react";
|
||||
import { CommandPalette } from "./CommandPalette";
|
||||
|
||||
vi.mock("../../../api/queries", () => ({
|
||||
useEvalLanes: () => ({
|
||||
data: [],
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
}),
|
||||
}));
|
||||
|
||||
function PaletteHarness({ initialOpen = false }: { initialOpen?: boolean }) {
|
||||
const [open, setOpen] = useState(initialOpen);
|
||||
return (
|
||||
|
|
|
|||
|
|
@ -2,13 +2,14 @@ import * as Dialog from "@radix-ui/react-dialog";
|
|||
import { Search } from "lucide-react";
|
||||
import { useRef, useState, useEffect, useCallback } from "react";
|
||||
import { useNavigate, useInRouterContext } from "react-router-dom";
|
||||
import { useEvalLanes } from "../../../api/queries";
|
||||
|
||||
interface Command {
|
||||
name: string;
|
||||
path: string;
|
||||
}
|
||||
|
||||
const COMMANDS: Command[] = [
|
||||
const STATIC_COMMANDS: Command[] = [
|
||||
{ name: "Open Chat", path: "/chat" },
|
||||
{ name: "Open Proposals", path: "/proposals" },
|
||||
{ name: "Open Evals", path: "/evals" },
|
||||
|
|
@ -20,6 +21,15 @@ function RouterCommandPalette(props: {
|
|||
onOpenChange: (open: boolean) => void;
|
||||
}) {
|
||||
const navigate = useNavigate();
|
||||
const { data: lanes } = useEvalLanes();
|
||||
|
||||
const dynamicCommands = (lanes || []).map((lane) => ({
|
||||
name: `Open eval lane ${lane.lane}`,
|
||||
path: `/evals?lane=${lane.lane}`,
|
||||
}));
|
||||
|
||||
const commands = [...STATIC_COMMANDS, ...dynamicCommands];
|
||||
|
||||
const activate = useCallback(
|
||||
(cmd: Command) => {
|
||||
navigate(cmd.path);
|
||||
|
|
@ -27,7 +37,7 @@ function RouterCommandPalette(props: {
|
|||
},
|
||||
[navigate, props],
|
||||
);
|
||||
return <CommandPaletteContent {...props} onActivate={activate} />;
|
||||
return <CommandPaletteContent {...props} commands={commands} onActivate={activate} />;
|
||||
}
|
||||
|
||||
// Fallback for design-system preview (no Router).
|
||||
|
|
@ -41,23 +51,25 @@ function FallbackCommandPalette(props: {
|
|||
},
|
||||
[props],
|
||||
);
|
||||
return <CommandPaletteContent {...props} onActivate={activate} />;
|
||||
return <CommandPaletteContent {...props} commands={STATIC_COMMANDS} onActivate={activate} />;
|
||||
}
|
||||
|
||||
function CommandPaletteContent({
|
||||
open,
|
||||
onOpenChange,
|
||||
onActivate,
|
||||
commands,
|
||||
}: {
|
||||
open: boolean;
|
||||
onOpenChange: (open: boolean) => void;
|
||||
onActivate: (cmd: Command) => void;
|
||||
commands: Command[];
|
||||
}) {
|
||||
const [query, setQuery] = useState("");
|
||||
const [focusedIndex, setFocusedIndex] = useState(0);
|
||||
const inputRef = useRef<HTMLInputElement>(null);
|
||||
|
||||
const filtered = COMMANDS.filter((cmd) =>
|
||||
const filtered = commands.filter((cmd) =>
|
||||
cmd.name.toLowerCase().includes(query.toLowerCase()),
|
||||
);
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue