feat(workbench): reconcile capability mastery surfaces (#826)
* feat(workbench): reconcile capability mastery surfaces onto main Reconcile draft PR 821 onto current origin/main after PR 824 without porting stale queries/types deletions or fake live API wiring. - Add Experience Flywheel panel with honest empty state and CLI guidance - Add Capability Paradigm panel with documented gate ladder A2e–A2q - Integrate CapabilityMasterySection into EvalsRoute landing and lane view - Preserve ADR-0160/0162 evidence-first posture; no mutation endpoints - Add capability-mastery-implementation-plan and panel tests Invariant protected: wrong_total == 0 serving honesty — documented 24/26/0 from lookbacks, never fabricated as live workbench metrics. * fix(workbench): keep eval landing guidance above mastery panels * fix(workbench): document capability milestones through sprint12 * fix(workbench): render capability ladder through sprint12 * test(workbench): assert sprint12 capability milestones * docs(workbench): update capability mastery plan through sprint12 * fix(workbench): make flywheel empty-state command runnable * test(workbench): expect runnable flywheel command * docs(workbench): use runnable flywheel command
This commit is contained in:
Shay
GitHub
parent
6daa5dfdea
commit
69febc6c0e
7 changed files with 708 additions and 4 deletions
164
docs/workbench/capability-mastery-implementation-plan.md
Normal file
164
docs/workbench/capability-mastery-implementation-plan.md
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
# Workbench Capability Mastery — Implementation Plan
|
||||
|
||||
**Branch:** `feat/workbench-capability-mastery-reconcile`
|
||||
**Reconciles:** Draft PR #821 (`feat/workbench-capability-mastery`) onto current `origin/main` after #825
|
||||
**Status:** Reconciled UI surfaces — honest documented milestones, no fake live data
|
||||
|
||||
## 1. What #821 originally attempted
|
||||
|
||||
PR #821 (`feat/workbench-capability-mastery`) added first Workbench surfaces on the Evals route:
|
||||
|
||||
- `ExperienceFlywheelPanel` — practice-memory record inspection
|
||||
- `CapabilityParadigmPanel` — derivation lift / Gate A* visibility
|
||||
- Integration in `EvalsRoute.tsx`
|
||||
- A draft `capability-mastery-implementation-plan.md`
|
||||
|
||||
The draft branch also **deleted large sections** of `workbench-ui/src/api/queries.ts` and `workbench-ui/src/types/api.ts`, introduced stale API assumptions (`runtime.capability_health`, `GET /flywheel`), and left `ExperienceFlywheelPanel` as an incomplete stub. Workbench-ui CI failed at build.
|
||||
|
||||
## 2. What changed since #821 was cut
|
||||
|
||||
Capability sprints 9–12 landed on main:
|
||||
|
||||
| PR | Sprint | Gates / organs | train_sample after |
|
||||
|----|--------|----------------|-------------------:|
|
||||
| #816 | Flywheel PR-1 | measurement-only adapter | (no serving change) |
|
||||
| #817 | 6 | A2g duration_segment_total, A2h survey_rate_earnings | 14/36/0 |
|
||||
| #818 | 7 | A2i round_trip_trip_duration, A2j giveaway_target_residual | 16/34/0 |
|
||||
| #819 | 8 | A2k fraction_decrease, A2l percent_partition | 18/32/0 |
|
||||
| #820 / #822 | 9 + hardening | A2m temporal_tariff, A2n affine_fraction_delta | 21/29/0 |
|
||||
| #823 | 10 | A2o affine_comparative_inversion_total, A2p sequential_comparative_scale | 23/27/0 |
|
||||
| #824 | 11 | A2q calendar_grounded_piecewise_daily_hours_total + ClusterContract | 24/26/0 |
|
||||
| #825 | 12 | A2r nested_fraction_remainder_total, A2s loose_crayon_box_capacity | **26/24/0** |
|
||||
|
||||
Lookback analyses through Sprint 12 are committed under `docs/analysis/gsm8k-capability-paradigm-sprint*-lookback-2026-06-17.md`.
|
||||
|
||||
## 3. Current capability baseline
|
||||
|
||||
**Documented serving state (train_sample after #825):** `26 correct / 24 refused / 0 wrong`
|
||||
|
||||
This is shown in the UI as **documented** evidence from lookbacks — not as a live workbench API read.
|
||||
|
||||
## 4. Gate ladder (A2e → A2s)
|
||||
|
||||
| Gate | Organ | Sprint |
|
||||
|------|-------|--------|
|
||||
| A2e | goal_residual_question | Strike #814 |
|
||||
| A2f | question_bound_product_aggregate | PR #815 |
|
||||
| A2g | duration_segment_total | 6 (#817) |
|
||||
| A2h | survey_rate_earnings | 6 (#817) |
|
||||
| A2i | round_trip_trip_duration | 7 (#818) |
|
||||
| A2j | giveaway_target_residual | 7 (#818) |
|
||||
| A2k | fraction_decrease | 8 (#819) |
|
||||
| A2l | percent_partition | 8 (#819) |
|
||||
| A2m | temporal_tariff | 9 (#820) |
|
||||
| A2n | affine_fraction_delta | 9 (#820) |
|
||||
| A2o | affine_comparative_inversion_total | 10 (#823) |
|
||||
| A2p | sequential_comparative_scale | 10 (#823) |
|
||||
| A2q | calendar_grounded_piecewise_daily_hours_total | 11 (#824) |
|
||||
| A2r | nested_fraction_remainder_total | 12 (#825) |
|
||||
| A2s | loose_crayon_box_capacity | 12 (#825) |
|
||||
|
||||
## 5. Experience Flywheel is measurement-only
|
||||
|
||||
PR-1 (#816) adds `scripts/gsm8k_experience_flywheel.py` and `evals/gsm8k_math/train_sample/v1/experience.py`:
|
||||
|
||||
- Read-only report / sealed-practice inputs
|
||||
- Explicit `--out` path only
|
||||
- No serving mutation, no `report.json` writes, no pack/corpus mutation
|
||||
- No auto-promotion into teaching or serving
|
||||
|
||||
## 6. ClusterContracts / singleton contracts
|
||||
|
||||
Sprint 11 ratified `calendar_grounded_piecewise_daily_hours_total` with:
|
||||
|
||||
- `piecewise_daily_hours_total` (Gate A2q)
|
||||
- `calendar_grounding.py` — `civil_month_day_count_table` with `calendar_table:{month}` provenance
|
||||
- Included case: `gsm8k-train-sample-v1-0013`
|
||||
|
||||
Sprint 12 extended the contract-first discipline with two singleton contracts:
|
||||
|
||||
- `nested_fraction_remainder_total` (Gate A2r) for `0004`, blocked against `0026` sealed-elimination confusers
|
||||
- `loose_crayon_box_capacity` (Gate A2s) for `0007`, blocked against `0047` DCS/divisive confusers
|
||||
|
||||
See:
|
||||
|
||||
- `docs/analysis/gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md`
|
||||
- `docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md`
|
||||
|
||||
## 7. What is UI-live now
|
||||
|
||||
| Surface | Live? | Source |
|
||||
|---------|-------|--------|
|
||||
| Eval lane list | Yes | `GET /evals` |
|
||||
| Eval lane run (read-only lanes) | Yes | `POST /evals/run` |
|
||||
| Wrong-zero ledger from session run | Yes | `EvalRunResult` in session |
|
||||
| Experience Flywheel records | **No** | Honest empty + CLI guidance |
|
||||
| Capability paradigm milestones | **Documented static** | Committed lookback markdown |
|
||||
| train_sample 26/24/0 headline | **Documented** | Lookbacks through #825, not live poll |
|
||||
|
||||
## 8. Honest empty states
|
||||
|
||||
- **Experience Flywheel:** states no workbench endpoint; points to `uv run python scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience.json`
|
||||
- **Capability Paradigm:** labeled "Documented" badge; sprint cards cite lookback doc paths
|
||||
- **No lane selected:** lane-selection guidance appears first; capability section follows as documented context
|
||||
|
||||
## 9. Backend endpoints still pending
|
||||
|
||||
- `GET /experience-flywheel` (or equivalent) — read-only compacted record feed from explicit artifact path or repo-local cache
|
||||
- `GET /capability-paradigm` — optional; could mirror lookback JSON if generated deterministically
|
||||
- Any endpoint must be read-only, drift-gated, and typed in `workbench/schemas.py` before client wiring
|
||||
|
||||
## 10. Explicitly not implemented
|
||||
|
||||
- Mutation endpoints for practice memory, serving, or reports
|
||||
- Fake / seeded flywheel records in the UI
|
||||
- `runtime.capability_health` footer theater
|
||||
- Wholesale copy of #821 `types/api.ts` / `queries.ts` deletions
|
||||
- Changes to `evals/gsm8k_math/train_sample/v1/report.json`
|
||||
- Changes to sealed practice artifacts
|
||||
- Live Sprint 12 metrics; Sprint 12 is documented from committed lookback evidence only
|
||||
|
||||
## 11. Invariants preserved
|
||||
|
||||
- ADR-0160 / ADR-0162 evidence-first, read-only Workbench posture
|
||||
- No cognitive theater or invented mastery scores
|
||||
- Route conformance unchanged (`EvalsRoute` empty/loading strings preserved for palette tests)
|
||||
- `StatusFooter` unchanged on main (no fake capability_health)
|
||||
|
||||
## 12. Files in this reconciliation
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `workbench-ui/src/app/evals/capabilityMasteryData.ts` | New — documented milestones through #825 |
|
||||
| `workbench-ui/src/app/evals/ExperienceFlywheelPanel.tsx` | New — honest empty state |
|
||||
| `workbench-ui/src/app/evals/CapabilityParadigmPanel.tsx` | New — documented gate ladder |
|
||||
| `workbench-ui/src/app/evals/CapabilityMasterySection.tsx` | New — composition |
|
||||
| `workbench-ui/src/app/evals/EvalsRoute.tsx` | Integrate section; lane guidance remains first on landing |
|
||||
| `workbench-ui/src/app/evals/capabilityMastery.test.tsx` | New tests |
|
||||
| `docs/workbench/capability-mastery-implementation-plan.md` | This plan |
|
||||
|
||||
**Not ported from #821:** `queries.ts` / `types/api.ts` mass deletions, `StatusFooter` capability_health stub.
|
||||
|
||||
## 13. Next Workbench steps
|
||||
|
||||
1. Add read-only `GET /experience-flywheel` backed by explicit artifact path config (no default repo writes)
|
||||
2. Optional deterministic `capability-paradigm.json` generator from lookback tables for API parity
|
||||
3. Wire `ExperienceFlywheelPanel` to real records when endpoint exists; keep empty state as primary until then
|
||||
4. Close #821 in favor of this PR after review
|
||||
|
||||
## 14. Verification
|
||||
|
||||
```bash
|
||||
cd workbench-ui
|
||||
pnpm install --frozen-lockfile
|
||||
pnpm exec tsc -b
|
||||
pnpm build
|
||||
pnpm test -- --run src/app/evals
|
||||
pnpm test -- --run src/app
|
||||
```
|
||||
|
||||
From repo root:
|
||||
|
||||
```bash
|
||||
uv run python -m core.cli test --suite smoke -q
|
||||
```
|
||||
16
workbench-ui/src/app/evals/CapabilityMasterySection.tsx
Normal file
16
workbench-ui/src/app/evals/CapabilityMasterySection.tsx
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
import { CapabilityParadigmPanel } from "./CapabilityParadigmPanel";
|
||||
import { ExperienceFlywheelPanel } from "./ExperienceFlywheelPanel";
|
||||
|
||||
/** Capability mastery surfaces for the Evals route (reconciled from PR 821). */
|
||||
export function CapabilityMasterySection() {
|
||||
return (
|
||||
<section
|
||||
className="flex flex-col gap-4"
|
||||
data-testid="capability-mastery-section"
|
||||
aria-label="Capability mastery surfaces"
|
||||
>
|
||||
<ExperienceFlywheelPanel />
|
||||
<CapabilityParadigmPanel />
|
||||
</section>
|
||||
);
|
||||
}
|
||||
181
workbench-ui/src/app/evals/CapabilityParadigmPanel.tsx
Normal file
181
workbench-ui/src/app/evals/CapabilityParadigmPanel.tsx
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
import { InfoBadge } from "../../design/components/badges/Badge";
|
||||
import { MetadataTable } from "../../design/components/MetadataTable/MetadataTable";
|
||||
import { Panel } from "../../design/components/Panel/Panel";
|
||||
import { TruncatedCell } from "../../design/components/TruncatedCell";
|
||||
import {
|
||||
BLOCKED_FAMILIES,
|
||||
CLUSTER_CONTRACT_SPRINT11,
|
||||
CLUSTER_CONTRACT_SPRINT12,
|
||||
DOCUMENTED_BASELINE_LABEL,
|
||||
DOCUMENTED_TRAIN_SAMPLE_BASELINE,
|
||||
GATE_LADDER_A2E_A2S,
|
||||
} from "./capabilityMasteryData";
|
||||
|
||||
function formatScore({ correct, refused, wrong }: { correct: number; refused: number; wrong: number }) {
|
||||
return `${correct} / ${refused} / ${wrong}`;
|
||||
}
|
||||
|
||||
export function CapabilityParadigmPanel() {
|
||||
const baseline = DOCUMENTED_TRAIN_SAMPLE_BASELINE;
|
||||
|
||||
return (
|
||||
<Panel
|
||||
title="Capability Paradigm"
|
||||
toolbar={
|
||||
<InfoBadge
|
||||
label="Documented"
|
||||
colorToken="--color-text-secondary"
|
||||
meaning="Milestone cards are sourced from committed lookback docs on main through PR 825 — not a live workbench API."
|
||||
adr="ADR-0160 / ADR-0162"
|
||||
evidence="No GET /capability endpoint exists; values mirror lookback analyses."
|
||||
/>
|
||||
}
|
||||
>
|
||||
<div className="flex flex-col gap-4">
|
||||
<div>
|
||||
<p className="m-0 text-xs text-[var(--color-text-secondary)] [text-wrap:balance]">
|
||||
{DOCUMENTED_BASELINE_LABEL}
|
||||
</p>
|
||||
<MetadataTable
|
||||
rows={[
|
||||
{
|
||||
key: "train_sample (serving)",
|
||||
value: (
|
||||
<span className="font-mono text-sm text-[var(--color-text-primary)]">
|
||||
{formatScore(baseline)}
|
||||
</span>
|
||||
),
|
||||
mono: true,
|
||||
},
|
||||
{
|
||||
key: "wrong invariant",
|
||||
value: (
|
||||
<span className="text-[var(--color-state-success-text)] font-medium">
|
||||
wrong = 0 (documented)
|
||||
</span>
|
||||
),
|
||||
},
|
||||
{
|
||||
key: "evidence through",
|
||||
value: "PR 825 — Sprint 12 singleton contracts",
|
||||
},
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<h3 className="m-0 mb-2 text-xs font-semibold uppercase tracking-wide text-[var(--color-text-secondary)]">
|
||||
Gate ladder (A2e → A2s)
|
||||
</h3>
|
||||
<div className="overflow-x-auto rounded border border-[var(--color-border-subtle)]">
|
||||
<table className="w-full min-w-[36rem] border-collapse text-left text-xs">
|
||||
<thead className="bg-[var(--color-surface-inset)] text-[var(--color-text-secondary)]">
|
||||
<tr>
|
||||
<th className="px-2 py-1.5 font-medium">Gate</th>
|
||||
<th className="px-2 py-1.5 font-medium">Organ</th>
|
||||
<th className="px-2 py-1.5 font-medium">Sprint</th>
|
||||
<th className="px-2 py-1.5 font-medium">Lifted</th>
|
||||
<th className="px-2 py-1.5 font-medium">Score after</th>
|
||||
<th className="px-2 py-1.5 font-medium">Lookback</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{GATE_LADDER_A2E_A2S.map((row) => (
|
||||
<tr
|
||||
key={row.gate}
|
||||
className="border-t border-[var(--color-border-subtle)]"
|
||||
data-testid={`gate-row-${row.gate}`}
|
||||
>
|
||||
<td className="px-2 py-1.5 font-mono text-[var(--color-text-primary)]">{row.gate}</td>
|
||||
<td className="px-2 py-1.5 text-[var(--color-text-primary)]">
|
||||
<TruncatedCell value={row.organ} label="organ" />
|
||||
</td>
|
||||
<td className="px-2 py-1.5 text-[var(--color-text-secondary)]">{row.sprint}</td>
|
||||
<td className="px-2 py-1.5 font-mono text-[var(--color-text-secondary)]">
|
||||
{row.newlySolved.length > 0 ? row.newlySolved.join(", ") : "—"}
|
||||
</td>
|
||||
<td className="px-2 py-1.5 font-mono text-[var(--color-text-primary)]">
|
||||
{formatScore(row.scoreAfter)}
|
||||
</td>
|
||||
<td className="px-2 py-1.5 font-mono text-[10px] text-[var(--color-text-tertiary)]">
|
||||
<TruncatedCell value={row.lookbackDoc} label="lookback doc" mono />
|
||||
</td>
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div
|
||||
className="rounded border border-[var(--color-border-subtle)] bg-[var(--color-surface-inset)] p-3"
|
||||
data-testid="cluster-contract-sprint11"
|
||||
>
|
||||
<h3 className="m-0 text-xs font-semibold text-[var(--color-text-primary)]">
|
||||
ClusterContract — Sprint 11 (PR 824)
|
||||
</h3>
|
||||
<p className="mt-1 mb-2 text-xs text-[var(--color-text-secondary)] [text-wrap:balance]">
|
||||
First ClusterContract sprint: typed chain + explicit{" "}
|
||||
<code className="font-mono">calendar_table</code> provenance for Gate A2q.
|
||||
</p>
|
||||
<MetadataTable
|
||||
rows={[
|
||||
{ key: "family_id", value: CLUSTER_CONTRACT_SPRINT11.familyId, mono: true },
|
||||
{ key: "organs", value: CLUSTER_CONTRACT_SPRINT11.organs.join(" + ") },
|
||||
{ key: "included case", value: CLUSTER_CONTRACT_SPRINT11.includedCase, mono: true },
|
||||
{ key: "provenance", value: CLUSTER_CONTRACT_SPRINT11.provenance, mono: true },
|
||||
{ key: "lookback", value: CLUSTER_CONTRACT_SPRINT11.lookbackDoc, mono: true },
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div
|
||||
className="rounded border border-[var(--color-border-subtle)] bg-[var(--color-surface-inset)] p-3"
|
||||
data-testid="cluster-contract-sprint12"
|
||||
>
|
||||
<h3 className="m-0 text-xs font-semibold text-[var(--color-text-primary)]">
|
||||
Singleton contracts — Sprint 12 (PR 825)
|
||||
</h3>
|
||||
<p className="mt-1 mb-2 text-xs text-[var(--color-text-secondary)] [text-wrap:balance]">
|
||||
Contract-backed singleton lift: A2r/A2s admitted only after 0026 and 0047 neighbor confuser matrices.
|
||||
</p>
|
||||
<MetadataTable
|
||||
rows={[
|
||||
{ key: "family_ids", value: CLUSTER_CONTRACT_SPRINT12.familyIds.join(" + "), mono: true },
|
||||
{ key: "organs", value: CLUSTER_CONTRACT_SPRINT12.organs.join(" + ") },
|
||||
{ key: "included cases", value: CLUSTER_CONTRACT_SPRINT12.includedCases.join(", "), mono: true },
|
||||
{ key: "blocked neighbors", value: CLUSTER_CONTRACT_SPRINT12.blockedNeighbors.join(", ") },
|
||||
{ key: "lookback", value: CLUSTER_CONTRACT_SPRINT12.lookbackDoc, mono: true },
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<h3 className="m-0 mb-2 text-xs font-semibold uppercase tracking-wide text-[var(--color-text-secondary)]">
|
||||
Blocked families (documented)
|
||||
</h3>
|
||||
<ul className="m-0 flex list-none flex-col gap-2 p-0">
|
||||
{BLOCKED_FAMILIES.map((blocked) => (
|
||||
<li
|
||||
key={blocked.family}
|
||||
className="rounded border border-[var(--color-border-subtle)] p-2 text-xs"
|
||||
data-testid={`blocked-${blocked.family.split(" ")[0]}`}
|
||||
>
|
||||
<div className="font-medium text-[var(--color-text-primary)]">{blocked.family}</div>
|
||||
<div className="mt-0.5 font-mono text-[var(--color-text-secondary)]">
|
||||
cases: {blocked.cases.join(", ")}
|
||||
</div>
|
||||
<div className="mt-1 text-[var(--color-text-tertiary)]">{blocked.reason}</div>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<p className="m-0 text-[10px] text-[var(--color-text-tertiary)] [text-wrap:balance]">
|
||||
Sprint 12 milestones are documented from committed lookback markdown. Re-run eval lanes for session
|
||||
evidence; this panel does not fabricate live serving counts.
|
||||
</p>
|
||||
</div>
|
||||
</Panel>
|
||||
);
|
||||
}
|
||||
|
|
@ -11,6 +11,7 @@ import { EvalMetricGrid } from "./EvalMetricGrid";
|
|||
import { EvalFailureViewer } from "./EvalFailureViewer";
|
||||
import { EvalArtifactLink } from "./EvalArtifactLink";
|
||||
import { EvalWrongZeroLedger } from "./EvalWrongZeroLedger";
|
||||
import { CapabilityMasterySection } from "./CapabilityMasterySection";
|
||||
import { EmptyState } from "../../design/components/states/EmptyState";
|
||||
import { ErrorState } from "../../design/components/states/ErrorState";
|
||||
import { LoadingState } from "../../design/components/states/LoadingState";
|
||||
|
|
@ -254,12 +255,17 @@ export function EvalsRoute() {
|
|||
nextAction={{ kind: "cli", command: `core eval --lane ${selectedLane.lane}` }}
|
||||
/>
|
||||
)}
|
||||
|
||||
<CapabilityMasterySection />
|
||||
</>
|
||||
) : (
|
||||
<EmptyState
|
||||
statement="Select an eval lane from the list to view results or run checks."
|
||||
nextAction={{ kind: "cli", command: "core eval --list" }}
|
||||
/>
|
||||
<>
|
||||
<EmptyState
|
||||
statement="Select an eval lane from the list to view lane metrics or run read-only checks."
|
||||
nextAction={{ kind: "cli", command: "core eval --list" }}
|
||||
/>
|
||||
<CapabilityMasterySection />
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
66
workbench-ui/src/app/evals/ExperienceFlywheelPanel.tsx
Normal file
66
workbench-ui/src/app/evals/ExperienceFlywheelPanel.tsx
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
import { InfoBadge } from "../../design/components/badges/Badge";
|
||||
import { MetadataTable } from "../../design/components/MetadataTable/MetadataTable";
|
||||
import { Panel } from "../../design/components/Panel/Panel";
|
||||
import { EmptyState } from "../../design/components/states/EmptyState";
|
||||
import { EXPERIENCE_FLYWHEEL_CLI, EXPERIENCE_RECORD_FIELDS } from "./capabilityMasteryData";
|
||||
|
||||
export const EXPERIENCE_FLYWHEEL_ABSENCE_STATEMENT =
|
||||
"No Experience Flywheel records in this Workbench session. PR-1 (PR 816) is measurement-only — records are emitted by CLI to an explicit --out path; there is no GET /flywheel workbench endpoint yet.";
|
||||
|
||||
/**
|
||||
* Experience Flywheel — measurement-only sealed-practice memory (PR-1 / PR 816).
|
||||
* Honest empty state until a read-only backend surface lands.
|
||||
*/
|
||||
export function ExperienceFlywheelPanel() {
|
||||
return (
|
||||
<Panel
|
||||
title="Experience Flywheel"
|
||||
toolbar={
|
||||
<InfoBadge
|
||||
label="Measurement-only"
|
||||
colorToken="--color-text-secondary"
|
||||
meaning="PR-1 adapter reads report.json and sealed practice artifacts — never mutates serving, corpus, or packs."
|
||||
adr="ADR-0160 / ADR-0162"
|
||||
evidence="docs/analysis/gsm8k-experience-flywheel-pr1-lookback-2026-06-17.md"
|
||||
/>
|
||||
}
|
||||
>
|
||||
<div className="flex flex-col gap-4" data-testid="experience-flywheel-panel">
|
||||
<EmptyState
|
||||
statement={EXPERIENCE_FLYWHEEL_ABSENCE_STATEMENT}
|
||||
nextAction={{ kind: "cli", command: EXPERIENCE_FLYWHEEL_CLI }}
|
||||
/>
|
||||
|
||||
<div>
|
||||
<h3 className="m-0 mb-2 text-xs font-semibold uppercase tracking-wide text-[var(--color-text-secondary)]">
|
||||
Compact record shape (when CLI emits --out)
|
||||
</h3>
|
||||
<MetadataTable
|
||||
rows={EXPERIENCE_RECORD_FIELDS.map((field) => ({
|
||||
key: field.key,
|
||||
value: field.value,
|
||||
}))}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<MetadataTable
|
||||
rows={[
|
||||
{
|
||||
key: "Trust boundary",
|
||||
value: "No serving mutation • report.json read-only • no auto-promotion",
|
||||
},
|
||||
{
|
||||
key: "Backend status",
|
||||
value: "Pending — no workbench GET endpoint; do not invent live records",
|
||||
},
|
||||
{
|
||||
key: "Lookback",
|
||||
value: "docs/analysis/gsm8k-experience-flywheel-pr1-lookback-2026-06-17.md",
|
||||
mono: true,
|
||||
},
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
</Panel>
|
||||
);
|
||||
}
|
||||
46
workbench-ui/src/app/evals/capabilityMastery.test.tsx
Normal file
46
workbench-ui/src/app/evals/capabilityMastery.test.tsx
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
import { render, screen } from "@testing-library/react";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { CapabilityParadigmPanel } from "./CapabilityParadigmPanel";
|
||||
import {
|
||||
EXPERIENCE_FLYWHEEL_ABSENCE_STATEMENT,
|
||||
ExperienceFlywheelPanel,
|
||||
} from "./ExperienceFlywheelPanel";
|
||||
import { CapabilityMasterySection } from "./CapabilityMasterySection";
|
||||
import { DOCUMENTED_TRAIN_SAMPLE_BASELINE } from "./capabilityMasteryData";
|
||||
|
||||
describe("Capability mastery surfaces", () => {
|
||||
it("ExperienceFlywheelPanel renders honest empty state and CLI guidance", () => {
|
||||
render(<ExperienceFlywheelPanel />);
|
||||
expect(screen.getByTestId("experience-flywheel-panel")).toBeInTheDocument();
|
||||
expect(screen.getByText(EXPERIENCE_FLYWHEEL_ABSENCE_STATEMENT)).toBeInTheDocument();
|
||||
expect(
|
||||
screen.getByText("uv run python scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience.json"),
|
||||
).toBeInTheDocument();
|
||||
expect(screen.getByText("Measurement-only")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("CapabilityParadigmPanel shows documented baseline and gate ladder through A2s", () => {
|
||||
render(<CapabilityParadigmPanel />);
|
||||
const { correct, refused, wrong } = DOCUMENTED_TRAIN_SAMPLE_BASELINE;
|
||||
expect(screen.getAllByText(`${correct} / ${refused} / ${wrong}`).length).toBeGreaterThanOrEqual(1);
|
||||
expect(screen.getByText("Documented")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("gate-row-A2e")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("gate-row-A2f")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("gate-row-A2q")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("gate-row-A2r")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("gate-row-A2s")).toBeInTheDocument();
|
||||
expect(screen.getByText("0004")).toBeInTheDocument();
|
||||
expect(screen.getByText("0007")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("cluster-contract-sprint11")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("cluster-contract-sprint12")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("blocked-DCS")).toBeInTheDocument();
|
||||
expect(screen.getByTestId("blocked-multiplicative_aggregate")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("CapabilityMasterySection composes both panels", () => {
|
||||
render(<CapabilityMasterySection />);
|
||||
expect(screen.getByTestId("capability-mastery-section")).toBeInTheDocument();
|
||||
expect(screen.getByText("Experience Flywheel")).toBeInTheDocument();
|
||||
expect(screen.getByText("Capability Paradigm")).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
225
workbench-ui/src/app/evals/capabilityMasteryData.ts
Normal file
225
workbench-ui/src/app/evals/capabilityMasteryData.ts
Normal file
|
|
@ -0,0 +1,225 @@
|
|||
/**
|
||||
* Documented GSM8K capability milestones — sourced from committed lookback
|
||||
* analyses on main through PR 825. These are NOT live API values; labels must
|
||||
* stay honest (ADR-0160 / ADR-0162).
|
||||
*/
|
||||
|
||||
export type DocumentedScore = {
|
||||
correct: number;
|
||||
refused: number;
|
||||
wrong: number;
|
||||
};
|
||||
|
||||
export type GateMilestone = {
|
||||
gate: string;
|
||||
organ: string;
|
||||
sprint: string;
|
||||
pr: string;
|
||||
newlySolved: string[];
|
||||
scoreAfter: DocumentedScore;
|
||||
lookbackDoc: string;
|
||||
};
|
||||
|
||||
export type BlockedFamily = {
|
||||
family: string;
|
||||
cases: string[];
|
||||
reason: string;
|
||||
};
|
||||
|
||||
/** Serving baseline after PR 825 (Sprint 12), from lookback evidence. */
|
||||
export const DOCUMENTED_TRAIN_SAMPLE_BASELINE: DocumentedScore = {
|
||||
correct: 26,
|
||||
refused: 24,
|
||||
wrong: 0,
|
||||
};
|
||||
|
||||
export const DOCUMENTED_BASELINE_LABEL =
|
||||
"Documented baseline (train_sample after PR 825 — not a live API read)";
|
||||
|
||||
export const GATE_LADDER_A2E_A2S: GateMilestone[] = [
|
||||
{
|
||||
gate: "A2e",
|
||||
organ: "goal_residual_question",
|
||||
sprint: "Strike batch 4 (PR 814)",
|
||||
pr: "PR 814",
|
||||
newlySolved: ["0037"],
|
||||
scoreAfter: { correct: 10, refused: 40, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint5-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2f",
|
||||
organ: "question_bound_product_aggregate",
|
||||
sprint: "Sprint 5 follow-on (PR 815)",
|
||||
pr: "PR 815",
|
||||
newlySolved: ["0003", "0021"],
|
||||
scoreAfter: { correct: 12, refused: 38, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint5-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2g",
|
||||
organ: "duration_segment_total",
|
||||
sprint: "Sprint 6 (PR 817)",
|
||||
pr: "PR 817",
|
||||
newlySolved: ["0015"],
|
||||
scoreAfter: { correct: 14, refused: 36, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint6-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2h",
|
||||
organ: "survey_rate_earnings",
|
||||
sprint: "Sprint 6 (PR 817)",
|
||||
pr: "PR 817",
|
||||
newlySolved: ["0045"],
|
||||
scoreAfter: { correct: 14, refused: 36, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint6-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2i",
|
||||
organ: "round_trip_trip_duration",
|
||||
sprint: "Sprint 7 (PR 818)",
|
||||
pr: "PR 818",
|
||||
newlySolved: ["0030"],
|
||||
scoreAfter: { correct: 16, refused: 34, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint7-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2j",
|
||||
organ: "giveaway_target_residual",
|
||||
sprint: "Sprint 7 (PR 818)",
|
||||
pr: "PR 818",
|
||||
newlySolved: ["0035"],
|
||||
scoreAfter: { correct: 16, refused: 34, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint7-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2k",
|
||||
organ: "fraction_decrease",
|
||||
sprint: "Sprint 8 (PR 819)",
|
||||
pr: "PR 819",
|
||||
newlySolved: ["0005"],
|
||||
scoreAfter: { correct: 18, refused: 32, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint8-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2l",
|
||||
organ: "percent_partition",
|
||||
sprint: "Sprint 8 (PR 819)",
|
||||
pr: "PR 819",
|
||||
newlySolved: ["0046"],
|
||||
scoreAfter: { correct: 18, refused: 32, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint8-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2m",
|
||||
organ: "temporal_tariff",
|
||||
sprint: "Sprint 9 (PR 820/PR 822)",
|
||||
pr: "PR 820",
|
||||
newlySolved: ["0001", "0017"],
|
||||
scoreAfter: { correct: 21, refused: 29, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint9-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2n",
|
||||
organ: "affine_fraction_delta",
|
||||
sprint: "Sprint 9 (PR 820/PR 822)",
|
||||
pr: "PR 820",
|
||||
newlySolved: ["0010"],
|
||||
scoreAfter: { correct: 21, refused: 29, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint9-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2o",
|
||||
organ: "affine_comparative_inversion_total",
|
||||
sprint: "Sprint 10 (PR 823)",
|
||||
pr: "PR 823",
|
||||
newlySolved: ["0009"],
|
||||
scoreAfter: { correct: 23, refused: 27, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint10-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2p",
|
||||
organ: "sequential_comparative_scale",
|
||||
sprint: "Sprint 10 (PR 823)",
|
||||
pr: "PR 823",
|
||||
newlySolved: ["0006"],
|
||||
scoreAfter: { correct: 23, refused: 27, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint10-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2q",
|
||||
organ: "calendar_grounded_piecewise_daily_hours_total",
|
||||
sprint: "Sprint 11 (PR 824)",
|
||||
pr: "PR 824",
|
||||
newlySolved: ["0013"],
|
||||
scoreAfter: { correct: 24, refused: 26, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2r",
|
||||
organ: "nested_fraction_remainder_total",
|
||||
sprint: "Sprint 12 (PR 825)",
|
||||
pr: "PR 825",
|
||||
newlySolved: ["0004"],
|
||||
scoreAfter: { correct: 26, refused: 24, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md",
|
||||
},
|
||||
{
|
||||
gate: "A2s",
|
||||
organ: "loose_crayon_box_capacity",
|
||||
sprint: "Sprint 12 (PR 825)",
|
||||
pr: "PR 825",
|
||||
newlySolved: ["0007"],
|
||||
scoreAfter: { correct: 26, refused: 24, wrong: 0 },
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md",
|
||||
},
|
||||
];
|
||||
|
||||
export const BLOCKED_FAMILIES: BlockedFamily[] = [
|
||||
{
|
||||
family: "DCS / relation_hypothesis",
|
||||
cases: ["0032", "0047"],
|
||||
reason: "Sealed-wrong neighbors; divisive/count surfaces blocked until typed chain clears wrong-risk.",
|
||||
},
|
||||
{
|
||||
family: "currency_amount",
|
||||
cases: ["0019", "0028"],
|
||||
reason: "Currency grounding not ratified for serving; conservative refusal preserved.",
|
||||
},
|
||||
{
|
||||
family: "sealed_elimination",
|
||||
cases: ["0011", "0026"],
|
||||
reason: "Sealed practice elimination surfaces; serving admission blocked.",
|
||||
},
|
||||
{
|
||||
family: "multiplicative_aggregate (wholesale)",
|
||||
cases: ["0006", "0013", "0025", "0047"],
|
||||
reason: "Rejected broad MA promotion — 0006/0013 solved only by narrow typed organs; 0025/0047 remain blocked neighbors.",
|
||||
},
|
||||
];
|
||||
|
||||
export const CLUSTER_CONTRACT_SPRINT11 = {
|
||||
familyId: "calendar_grounded_piecewise_daily_hours_total",
|
||||
organs: ["piecewise_daily_hours_total (Gate A2q)", "calendar_grounding (civil_month_day_count_table)"],
|
||||
includedCase: "gsm8k-train-sample-v1-0013",
|
||||
provenance: "calendar_table:{month_name}",
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint11-lookback-2026-06-17.md",
|
||||
};
|
||||
|
||||
export const CLUSTER_CONTRACT_SPRINT12 = {
|
||||
familyIds: ["nested_fraction_remainder_total", "loose_crayon_box_capacity"],
|
||||
organs: ["nested_fraction_remainder_total (Gate A2r)", "loose_crayon_box_capacity (Gate A2s)"],
|
||||
includedCases: ["gsm8k-train-sample-v1-0004", "gsm8k-train-sample-v1-0007"],
|
||||
blockedNeighbors: ["0026 sealed_elimination", "0047 DCS/divisive"],
|
||||
lookbackDoc: "docs/analysis/gsm8k-capability-paradigm-sprint12-lookback-2026-06-17.md",
|
||||
};
|
||||
|
||||
export const EXPERIENCE_FLYWHEEL_CLI =
|
||||
"uv run python scripts/gsm8k_experience_flywheel.py --limit 50 --out /tmp/gsm8k-experience.json";
|
||||
|
||||
export const EXPERIENCE_RECORD_FIELDS: { key: string; value: string }[] = [
|
||||
{ key: "record_id", value: "SHA-256 of load-bearing fields" },
|
||||
{ key: "case_id / serving_status / sealed_status", value: "Per-case diagnostic posture" },
|
||||
{ key: "candidate_family / first_missing_primitive", value: "Typed lift hypothesis" },
|
||||
{ key: "hazard_tags / promotion_status", value: "Retention and promotion gates" },
|
||||
{ key: "source_report_hash", value: "Read-only report digest — report.json never mutated" },
|
||||
];
|
||||
Loading…
Reference in a new issue