From 02f02c804a2aa15b7f09b8f16f22d89f342004ea Mon Sep 17 00:00:00 2001 From: Shay Date: Fri, 12 Jun 2026 23:51:57 -0700 Subject: [PATCH] docs(workbench): Wave M plan (mastery & worthiness) + Phase B calibration brief pack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wave M takes the workbench to mastery and closes its biggest design gap: it surfaces the teaching/ratification loop but is blind to the calibrated-learning / serving-discipline loop (gold-tether arena, reliability gate, Wilson floor vs θ ceiling, 'the engine earns the right to guess') and to cognition itself (pipeline stages, field substrate, identity continuity). Lens: Anthropic + xAI as target users who'd WANT to use it. - wave-M-worthiness.md: full plan, Phases A–E, the missing-surfaces table, the backend-reader-first / never-re-implement-engine-math disciplines, execution order (B→C→D, A parallel). - wave-M-phaseB-calibration-briefs: executable Phase B pack grounded in the real core/reliability_gate shapes (ClassTally / conservative_floor / license_for / Action θ) and the committed report.json evidence — B1 readers (GATING, Python), B2 Calibration route, B3 wrong=0 global frame, B4 leeway wiring. Dependency DAG + STOP gates + no-theater rules. --- ...-M-phaseB-calibration-briefs-2026-06-13.md | 189 ++++++++++++++++++ docs/workbench/wave-M-worthiness.md | 179 +++++++++++++++++ 2 files changed, 368 insertions(+) create mode 100644 docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md create mode 100644 docs/workbench/wave-M-worthiness.md diff --git a/docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md b/docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md new file mode 100644 index 00000000..299335ac --- /dev/null +++ b/docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md @@ -0,0 +1,189 @@ +# Wave M · Phase B — Calibration / Serving-Discipline Brief Pack + +Date: 2026-06-13 +Plan: `docs/workbench/wave-M-worthiness.md` § Phase B. +Goal: make the calibrated-learning / serving-discipline loop *visible* — the +gold-tether arena, the reliability gate, "the engine earns the right to +guess." This is the widest worthiness gap. + +## Dependency DAG + +``` +B1 (backend readers) ──┬──→ B2 (Calibration route) ──→ B4 (leeway wiring) + └──→ B3 (wrong=0 global frame) +``` + +**B1 gates everything** — it merges first. B2/B3 are parallel-safe after B1 +(disjoint files except the usual train: App.tsx / types/api.ts / +routeConformance / NOT_YET_MIRRORED → strictly sequential merges, union +rebase). B4 last (touches Proposals + Replay rails). + +## Standing constraints (all briefs) + +- Worktree off fresh `origin/main`; green-local (`pnpm build && pnpm test`, + plus the Python lane for B1) before push; **STOP after checks green; + Shay merges.** +- **NEVER re-implement engine math.** Import and call + `core.reliability_gate` (`conservative_floor`, `license_for`, `Ceilings`, + `Action`); never reproduce the Wilson floor or θ logic in the workbench. +- **Read-only.** No new mutation endpoints; no execution. The reader reads + committed artifacts + computes derived numbers via the engine's own + functions. A calibration view never changes a license. +- Token-only styling (hexScan); schema mirrored + snapshot regenerated + + drift gate; enum coverage if a new badge enum is added; conformance rows + (ADR-0162 §6); no invented data — absent calibration evidence renders + honest absence. + +--- + +## Brief B1 — Calibration readers + endpoints (Python only; GATING) + +### Worktree + gates +```bash +cd /Users/kaizenpro/Projects/core +git fetch origin +git worktree add ../core-wb-m-b1 origin/main -b feat/wb-m-calibration-readers +cd ../core-wb-m-b1 +ls core/reliability_gate/ledger.py || echo "STOP: reliability_gate missing" +``` + +### Read first (do not wander) +- `core/reliability_gate/ledger.py` — `ClassTally(class_name, correct, + wrong, refused, t2_verified, t2_agrees_gold)`; derived `committed` + (=correct+wrong), `attempted`, `reliability()` (=`conservative_floor`), + `coverage()`. +- `core/reliability_gate/{floor,gate,ceilings,propose}.py` — + `conservative_floor(successes, committed)` (Wilson, `N_MIN=10`, 0.0 below); + `license_for(tally, ceilings, action) -> LicenseDecision(licensed, + measured, required)`; `Action.PROPOSE` (θ=0.85) / `Action.SERVE` (θ=0.99); + `Ceilings.required(class_name, action)`. +- `workbench/readers.py` (list_/read_ + `_page` + `_is_allowed` patterns), + `workbench/api.py` (route wiring), `workbench/schemas.py` + the two snapshot + generators (`scripts/dump-schemas.py`, `scripts/dump-enums.py`). +- `evals/gsm8k_math/train_sample/v1/report.json` + + `train_sample_coverage_report.json` — the **persisted calibration + evidence**. + +### Investigate first (decides the reader's source) +Is the live `ClassTally` ledger persisted anywhere at rest? Grep for a +written ledger jsonl/json. **If not** (likely), the reader reconstructs +per-class `ClassTally` from the committed `report.json` per-class outcomes, +then applies the *real* `conservative_floor` + `license_for`. Document which +source you used in the reader docstring + the PR. + +### Deliverables +1. `workbench/calibration.py` (new): pure functions that load the committed + report artifact(s), build `ClassTally` per class, and produce per-class + rows via the real engine functions — no math re-implemented here. +2. Schemas (`workbench/schemas.py`): `CalibrationClass` (class_name, correct, + wrong, refused, committed, attempted, reliability_floor, coverage, + propose_licensed, propose_required, serve_licensed, serve_required) and + `ServingMetrics` (lane, correct, refused, wrong, source_path, + source_digest). Mirror in `types/api.ts`; regenerate both snapshots; the + drift gate must pass. +3. Endpoints (`workbench/api.py`): `GET /calibration/classes` → + `{items: CalibrationClass[]}`; `GET /serving/metrics` → + `{items: ServingMetrics[]}` (read `train_sample` + `holdout_dev` committed + reports; **never** run a lane). Path-validate any id; reads only inside + allowed eval roots. +4. Trust boundary stated in the PR: read-only over committed artifacts + + engine-owned derivation; no execution, no mutation. + +### Verification +```bash +cd ../core-wb-m-b1 +.venv/bin/python -m pytest tests/ -k "workbench_calibration or workbench_schemas or workbench_api" -q +.venv/bin/python scripts/dump-schemas.py | diff - workbench-ui/schema-snapshot.json +``` +Add `tests/test_workbench_calibration.py`: a class that cleared SERVE shows +`serve_licensed=true`; a class below `N_MIN` shows `reliability_floor=0.0` +and `propose_licensed=false`; the reader's numbers equal a direct +`conservative_floor`/`license_for` call (proves no re-implementation). + +--- + +## Brief B2 — Calibration / Gold-Tether route (frontend; after B1) + +### Gates +```bash +git worktree add ../core-wb-m-b2 origin/main -b feat/wb-m-calibration-route +grep -q "CalibrationClass" workbench-ui/src/types/api.ts || echo "STOP: B1 not merged" +``` + +### Deliverables +- TS mirrors already landed in B1; add `useCalibrationClasses` / + `useServingMetrics` query hooks. +- `app/calibration/CalibrationRoute.tsx` — per class: a coverage-vs-Wilson + bar (reliability_floor vs the cleared θ), correct/refused/**wrong** counts + (wrong load-bearing), and a plain verdict pill: "earned SERVE", "earned + PROPOSE", or "not yet licensed". **Failures-first** ordering (lowest + reliability / un-licensed at top). VirtualizedList + useListNavigation + + SearchInput; Panel/TabBar detail (Counts / License math / Raw). +- The "License math" tab shows the honest derivation: committed N, Wilson + floor, θ required, measured ≥ required → licensed — read from B1, not + computed in the UI. +- Nav entry; Calibration row in `routeConformance` (loading "Loading + calibration...", empty "No calibration evidence yet." + the practice-lane + CLI, error). Selection publishes an evidence subject (new `calibration_class` + kind, inspect-param) — or, if that's too much for one PR, local selection + + flag the subject-kind as a follow-up. +- Tests: failures-first ordering, the un-licensed/below-N_MIN class renders + "not yet licensed", the wrong count renders, j/k spine. + +### Verify: `cd workbench-ui && pnpm build && pnpm test` + +--- + +## Brief B3 — wrong=0 as a felt global presence (frontend; parallel with B2) + +### Gates +```bash +git worktree add ../core-wb-m-b3 origin/main -b feat/wb-m-wrong-zero-frame +grep -q "ServingMetrics" workbench-ui/src/types/api.ts || echo "STOP: B1 not merged" +``` + +### Deliverables +- A small always-present invariant element in the `Shell` chrome (header + strip): live **N correct · N refused · 0 wrong**, the zero rendered hard + (verified token) — sourced from `/serving/metrics`, never invented; when + unavailable, render an honest "metrics unavailable", never a fake zero. +- It links to the Calibration route (B2) and the Evals wrong=0 ledger. +- **Doctrine line:** the strip states an invariant, it does not *claim* + correctness it can't read — if the committed report shows wrong>0 it shows + wrong>0 in the contradicted token (the strip must be able to show a + non-zero wrong honestly; it is a mirror, not a slogan). +- Tests: renders the triplet from a stubbed metrics fetch; renders a + non-zero wrong honestly (no hard-coded zero); honest absence on fetch + error. + +### Verify: `cd workbench-ui && pnpm build && pnpm test` + +--- + +## Brief B4 — The leeway story (frontend; after B1 + B2) + +### Gates +```bash +git worktree add ../core-wb-m-b4 origin/main -b feat/wb-m-leeway-wiring +grep -q "CalibrationClass" workbench-ui/src/types/api.ts || echo "STOP: B1 not merged" +``` + +### Deliverables +- In the Replay / Proposals evidence rails, when a turn or proposal carries an + approximate/served result, surface *why latitude was granted*: the class, + its license (PROPOSE/SERVE), the θ it cleared, and the `[approximate]` + disclosure — joining the existing HITL ratification to the calibration that + grants it. Read from B1; link to the Calibration route. +- No new mutation; purely a read-only cross-link/annotation. +- Tests: a served-with-leeway fixture renders its class + θ + license; a + fully-verified turn renders no leeway annotation (absence is honest). + +### Verify: `cd workbench-ui && pnpm build && pnpm test` + +--- + +## After this pack + +Phase C brief pack (cognitive-pipeline visualizer, contemplation-as-process, +field substrate, identity continuity) is authored once Phase B lands — +C1/C3 are also backend-reader-first and Python-gated. diff --git a/docs/workbench/wave-M-worthiness.md b/docs/workbench/wave-M-worthiness.md new file mode 100644 index 00000000..f0eebb15 --- /dev/null +++ b/docs/workbench/wave-M-worthiness.md @@ -0,0 +1,179 @@ +# Wave M — CORE Workbench: Mastery & Worthiness + +Date: 2026-06-13 +Status: approved plan (Shay, 2026-06-13). Predecessor: Wave R complete +(#702–#723; 11 routes real, Replay Moment, trace integrity, DAG/Demo/wrong=0). +Execution: committed brief packs in `docs/handoff/`, parallel-safe DAGs, +dispatched between Fable 5 and GPT5.5 — the same production line that +shipped R2 + R3. + +## Thesis + +Two asks, one lens. + +1. **Mastery** — take the shipped surface from very good to best-in-class. +2. **Worthiness** — add what's *missing* so the workbench is undeniably + worthy of the deterministic cognitive engine beneath it. + +The lens: **Anthropic and xAI as target users who would *want* to use it.** +They build the opaque transformer this engine defines itself *against*. What +impresses them is not prettier charts — it is a UI that makes +**determinism, refusal-discipline, and geometric coherence inspectable and +felt.** Standard: ADR-0160's three pillars — audit-native (not analytics +theater), calm default / infinite depth, replay before persuasion. + +## Diagnosis — the two blind spots + +The workbench today is excellent at **evidence browsing**: every route +projects an evidence manifold, the Evidence Chain Rail threads provenance, +the Replay Moment makes hash-equality felt. But it is blind to the two most +*distinctive* parts of the organism: + +1. **It shows the teaching/ratification loop and is blind to the + calibrated-learning / serving-discipline loop.** You can ratify a + proposal, but you cannot *see* the gold-tether arena, the reliability + gate, the Wilson floor vs the θ ceiling, or the moment "the engine earns + the right to guess." That discipline — *the engine refuses rather than + guesses wrong* — is the single most impressive idea in the project, and + it is invisible. +2. **It shows outputs and evidence but not cognition itself.** The + `CognitiveTurnPipeline` stages, the contemplation *process*, the CL(4,1) + field substrate, `versor_condition`, identity continuity — none are + legible. For an audience that lives inside opaque models, *legible + deterministic cognition* is the wow. + +Everything below closes those two gaps on top of a mastery polish. + +## Non-negotiable disciplines (bind every phase) + +- **Backend-reader-first, no theater.** Every new surface reads *real* + engine data through a new read-only reader; no dashboard over invented or + recomputed numbers. The calibration and field readers do not exist yet — + that gating work is Python, not React. +- **Never re-implement engine math in the workbench.** The calibration + reader *imports and uses* `core.reliability_gate` (`conservative_floor`, + `license_for`, `Ceilings`, `Action`); the field reader uses the engine's + real `versor_condition`/`cga_inner`. The workbench computes nothing the + engine owns. +- **Read-only doctrine holds.** No new mutation endpoints; execution stays + the existing allowlisted set (`/evals/run`, ratify, `/demos/{id}/run`). A + calibration view never *changes* a license. +- **Determinism in the UI too.** No force-directed / nondeterministic + layout, no decorative motion-as-cognition. Golden-file layout tests for + every new visualizer (like the DAG). The honesty *is* the impressiveness. +- **Doctrine gates extend to every new surface**: schema mirrored, enums + covered, route conformant, readers SHA-pinned where they assert a metric. + +## Phases (priority-ordered) + +### Phase A — Mastery polish of the shipped surface (scope: M; parallel) +No new concepts; make the 11 routes undeniable. +- Design-system full expression: semantic token roles, elevation, **density + modes actually wired** (the deferred Settings density pref), `tabular-nums` + on all numerics, `[text-wrap:balance]` on all statements, motion-discipline + audit (only state-transition affordances). +- Cross-route consistency sweep: every list = `VirtualizedList` + + `useListNavigation` + `SearchInput` + selection tokens; every detail = + `Panel` + `TabBar`; calm-honest prose audit on every state. +- **DAG viewer: finish its consumers.** It shipped wired only to proposal + chains; wire the **PCCP proof-promotion 8 scenarios** and **entailment + traces** (the other two the brief named). A primitive with one consumer is + half-built. +- Command/keyboard completeness: a palette verb for every route action; + registry-driven help stays the exhaustive contract. +- Accessibility pass: focus-visible audit, SR labels on every evidence + badge, reduced-motion honored. + +### Phase B — Calibrated-Learning / Serving-Discipline surfaces (scope: L) ← the heart +The "worthy of the model" core. Backend-reader-first (none exist; data lives +in `core/reliability_gate/` + the committed `evals/gsm8k_math/*/report.json`). +Detailed brief pack: `docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md`. +- **B1 (Python):** read-only readers/endpoints over the real ledger — + `GET /calibration/classes` (per-class `ClassTally` counts + the Wilson + `conservative_floor` reliability + PROPOSE/SERVE `license_for` verdicts via + the real `core.reliability_gate`), `GET /serving/metrics` (the committed + `train_sample/v1/report.json` numbers — read the artifact, never re-run an + unsafe lane). Schema mirrors + snapshots + drift gate. +- **B2 — Calibration / Gold-Tether route:** per class, a + coverage-vs-Wilson-floor bar, the θ ceiling, and a plain-language "earned + PROPOSE / SERVE / neither" verdict. Failures-first. Where you *see* "the + engine earns the right to guess." +- **B3 — wrong=0 as a felt global presence:** an always-present invariant + element (N correct / N refused / **0 wrong**, the zero load-bearing), + elevating the per-run Evals ledger to the project's thesis made constant. +- **B4 — the leeway story:** wire the calibration verdict into the Proposals + / Replay rails so a reviewer sees *why* a turn was granted latitude (which + class license, which θ, the `[approximate]` disclosure) — connecting the + HITL ratification you already have to the calibration that grants it. + +### Phase C — Make cognition legible (scope: L) ← the wow for Anthropic/xAI +- **C1 — Cognitive Pipeline visualizer:** for a selected turn, render the + real `CognitiveTurnPipeline` stages (intent → PropositionGraph → + ArticulationTarget → realizer → walk telemetry → trace hash) as a + deterministic staged view (reuse the DAG primitive). *The* "real, + replayable path, not animated fake cognition" surface. Reader-first over + existing trace/walk telemetry. +- **C2 — Contemplation as a process, not just outputs:** the contemplation + *loop* (attempt → gold-tether → ClassTally → propose), connecting + Demos/Proposals/Calibration into one story. +- **C3 — Field substrate (honest, read-only, hard):** `GET /field/state` + over real `FieldState` + `versor_condition` for a turn, rendered as + **inspectable exact numbers and invariant status** — `versor_condition < + 1e-6` as a live "field is valid" assertion, `cga_inner` coherence as exact + values. **NOT** a decorative 3D blob; no force-directed/nondeterministic + motion. The honesty is the impressiveness: "this is the geometry, it's + exact, it can't fake coherence." +- **C4 — Identity continuity (L10/L11):** surface the engine-identity hash, + lineage chain, reboot-verification status — "the same continuous life + across restart," the deepest telos, currently invisible. + +### Phase D — The "they'd want to use it" layer (scope: M) +- **Guided Determinism Tour** — elevate Demo Theater into a first-run + narrative: pick a demo, watch the proposer get disciplined, see + hash-to-hash replay, see a wrong answer *refused*. "What this proves / what + this does not prove" honesty cards on every scenario. +- **Provider-agnostic framing** — the pitch for Anthropic *and* xAI: "bring + your own model's claim; watch the deterministic engine decide, refuse, and + replay it." The Tool-Authority / Hybrid-Verification demos already embody + this; make it the tour's spine. +- **Shareable evidence bundles** — deterministic export of a turn + its + trace + replay + calibration verdict as a single citable artifact. + Reproducibility *as a deliverable*. + +### Phase E — Robustness pillars (scope: S; continuous) +- Extend doctrine gates to every new surface; SHA-pin the calibration/field + readers where they assert a metric. +- Performance budget (resolve the Vite chunk-size warning via route + code-split), error-boundary discipline, golden-file regime for the + pipeline/field visualizers. + +## What's missing in the design (the second ask, distilled) + +| Missing surface | Why it matters for worthiness | Reader exists? | +|---|---|---| +| Calibration / gold-tether arena | Makes wrong=0 *earned*, not asserted — the most distinctive idea, invisible | **No** — build first | +| Serving-vs-learning regime frame | Names the two-regime architecture; without it the UI reads as a chatbot | No | +| wrong=0 as a felt global presence | The thesis itself; today only per-eval-run | Partial (ledger) | +| Cognitive pipeline visualizer | "Real replayable cognition" vs animated fake — the core wow | Trace exists; needs staging reader | +| Contemplation-as-process | The learning flywheel, today only its outputs | Partial | +| Field substrate / versor_condition | The geometry that *can't fake coherence* — honest, exact | **No** — build first | +| Identity continuity (L10/L11) | "One continuous life" — the deepest telos | No | +| Serving metrics reachable | The actual capability numbers (gsm8k) aren't viewable | No | + +## Risks + +- **Theater is risk #1** — mitigated by backend-reader-first + never + re-implementing engine math. The gating work (B1, C1, C3 readers) is + Python and parallel-safe. +- **The field surface must stay honest** — read-only over real + `versor_condition`/`cga_inner`, no decorative geometry, no motion theater. +- **Scope is large** — several PR trains. Sequences as readers → routes → + cross-wiring → tour. Phase A runs in parallel as polish. +- No timelines — phases/priorities/scope-sizes; sequencing is the dependency + DAG, not a clock. + +## Execution order + +**B → C → D**, with **A in parallel**. The worthiness gap is widest at B; the +tour (D) lands hardest once B and C exist to show off. Phase B brief pack is +authored first (this commit); subsequent phase packs follow as each lands.