docs(workbench): Wave M plan (mastery & worthiness) + Phase B calibration brief pack
Wave M takes the workbench to mastery and closes its biggest design gap: it surfaces the teaching/ratification loop but is blind to the calibrated-learning / serving-discipline loop (gold-tether arena, reliability gate, Wilson floor vs θ ceiling, 'the engine earns the right to guess') and to cognition itself (pipeline stages, field substrate, identity continuity). Lens: Anthropic + xAI as target users who'd WANT to use it. - wave-M-worthiness.md: full plan, Phases A–E, the missing-surfaces table, the backend-reader-first / never-re-implement-engine-math disciplines, execution order (B→C→D, A parallel). - wave-M-phaseB-calibration-briefs: executable Phase B pack grounded in the real core/reliability_gate shapes (ClassTally / conservative_floor / license_for / Action θ) and the committed report.json evidence — B1 readers (GATING, Python), B2 Calibration route, B3 wrong=0 global frame, B4 leeway wiring. Dependency DAG + STOP gates + no-theater rules.
This commit is contained in:
Shay
parent
f0225b1f6b
commit
02f02c804a
2 changed files with 368 additions and 0 deletions
189
docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md
Normal file
189
docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
# Wave M · Phase B — Calibration / Serving-Discipline Brief Pack
|
||||
|
||||
Date: 2026-06-13
|
||||
Plan: `docs/workbench/wave-M-worthiness.md` § Phase B.
|
||||
Goal: make the calibrated-learning / serving-discipline loop *visible* — the
|
||||
gold-tether arena, the reliability gate, "the engine earns the right to
|
||||
guess." This is the widest worthiness gap.
|
||||
|
||||
## Dependency DAG
|
||||
|
||||
```
|
||||
B1 (backend readers) ──┬──→ B2 (Calibration route) ──→ B4 (leeway wiring)
|
||||
└──→ B3 (wrong=0 global frame)
|
||||
```
|
||||
|
||||
**B1 gates everything** — it merges first. B2/B3 are parallel-safe after B1
|
||||
(disjoint files except the usual train: App.tsx / types/api.ts /
|
||||
routeConformance / NOT_YET_MIRRORED → strictly sequential merges, union
|
||||
rebase). B4 last (touches Proposals + Replay rails).
|
||||
|
||||
## Standing constraints (all briefs)
|
||||
|
||||
- Worktree off fresh `origin/main`; green-local (`pnpm build && pnpm test`,
|
||||
plus the Python lane for B1) before push; **STOP after checks green;
|
||||
Shay merges.**
|
||||
- **NEVER re-implement engine math.** Import and call
|
||||
`core.reliability_gate` (`conservative_floor`, `license_for`, `Ceilings`,
|
||||
`Action`); never reproduce the Wilson floor or θ logic in the workbench.
|
||||
- **Read-only.** No new mutation endpoints; no execution. The reader reads
|
||||
committed artifacts + computes derived numbers via the engine's own
|
||||
functions. A calibration view never changes a license.
|
||||
- Token-only styling (hexScan); schema mirrored + snapshot regenerated +
|
||||
drift gate; enum coverage if a new badge enum is added; conformance rows
|
||||
(ADR-0162 §6); no invented data — absent calibration evidence renders
|
||||
honest absence.
|
||||
|
||||
---
|
||||
|
||||
## Brief B1 — Calibration readers + endpoints (Python only; GATING)
|
||||
|
||||
### Worktree + gates
|
||||
```bash
|
||||
cd /Users/kaizenpro/Projects/core
|
||||
git fetch origin
|
||||
git worktree add ../core-wb-m-b1 origin/main -b feat/wb-m-calibration-readers
|
||||
cd ../core-wb-m-b1
|
||||
ls core/reliability_gate/ledger.py || echo "STOP: reliability_gate missing"
|
||||
```
|
||||
|
||||
### Read first (do not wander)
|
||||
- `core/reliability_gate/ledger.py` — `ClassTally(class_name, correct,
|
||||
wrong, refused, t2_verified, t2_agrees_gold)`; derived `committed`
|
||||
(=correct+wrong), `attempted`, `reliability()` (=`conservative_floor`),
|
||||
`coverage()`.
|
||||
- `core/reliability_gate/{floor,gate,ceilings,propose}.py` —
|
||||
`conservative_floor(successes, committed)` (Wilson, `N_MIN=10`, 0.0 below);
|
||||
`license_for(tally, ceilings, action) -> LicenseDecision(licensed,
|
||||
measured, required)`; `Action.PROPOSE` (θ=0.85) / `Action.SERVE` (θ=0.99);
|
||||
`Ceilings.required(class_name, action)`.
|
||||
- `workbench/readers.py` (list_/read_ + `_page` + `_is_allowed` patterns),
|
||||
`workbench/api.py` (route wiring), `workbench/schemas.py` + the two snapshot
|
||||
generators (`scripts/dump-schemas.py`, `scripts/dump-enums.py`).
|
||||
- `evals/gsm8k_math/train_sample/v1/report.json` +
|
||||
`train_sample_coverage_report.json` — the **persisted calibration
|
||||
evidence**.
|
||||
|
||||
### Investigate first (decides the reader's source)
|
||||
Is the live `ClassTally` ledger persisted anywhere at rest? Grep for a
|
||||
written ledger jsonl/json. **If not** (likely), the reader reconstructs
|
||||
per-class `ClassTally` from the committed `report.json` per-class outcomes,
|
||||
then applies the *real* `conservative_floor` + `license_for`. Document which
|
||||
source you used in the reader docstring + the PR.
|
||||
|
||||
### Deliverables
|
||||
1. `workbench/calibration.py` (new): pure functions that load the committed
|
||||
report artifact(s), build `ClassTally` per class, and produce per-class
|
||||
rows via the real engine functions — no math re-implemented here.
|
||||
2. Schemas (`workbench/schemas.py`): `CalibrationClass` (class_name, correct,
|
||||
wrong, refused, committed, attempted, reliability_floor, coverage,
|
||||
propose_licensed, propose_required, serve_licensed, serve_required) and
|
||||
`ServingMetrics` (lane, correct, refused, wrong, source_path,
|
||||
source_digest). Mirror in `types/api.ts`; regenerate both snapshots; the
|
||||
drift gate must pass.
|
||||
3. Endpoints (`workbench/api.py`): `GET /calibration/classes` →
|
||||
`{items: CalibrationClass[]}`; `GET /serving/metrics` →
|
||||
`{items: ServingMetrics[]}` (read `train_sample` + `holdout_dev` committed
|
||||
reports; **never** run a lane). Path-validate any id; reads only inside
|
||||
allowed eval roots.
|
||||
4. Trust boundary stated in the PR: read-only over committed artifacts +
|
||||
engine-owned derivation; no execution, no mutation.
|
||||
|
||||
### Verification
|
||||
```bash
|
||||
cd ../core-wb-m-b1
|
||||
.venv/bin/python -m pytest tests/ -k "workbench_calibration or workbench_schemas or workbench_api" -q
|
||||
.venv/bin/python scripts/dump-schemas.py | diff - workbench-ui/schema-snapshot.json
|
||||
```
|
||||
Add `tests/test_workbench_calibration.py`: a class that cleared SERVE shows
|
||||
`serve_licensed=true`; a class below `N_MIN` shows `reliability_floor=0.0`
|
||||
and `propose_licensed=false`; the reader's numbers equal a direct
|
||||
`conservative_floor`/`license_for` call (proves no re-implementation).
|
||||
|
||||
---
|
||||
|
||||
## Brief B2 — Calibration / Gold-Tether route (frontend; after B1)
|
||||
|
||||
### Gates
|
||||
```bash
|
||||
git worktree add ../core-wb-m-b2 origin/main -b feat/wb-m-calibration-route
|
||||
grep -q "CalibrationClass" workbench-ui/src/types/api.ts || echo "STOP: B1 not merged"
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
- TS mirrors already landed in B1; add `useCalibrationClasses` /
|
||||
`useServingMetrics` query hooks.
|
||||
- `app/calibration/CalibrationRoute.tsx` — per class: a coverage-vs-Wilson
|
||||
bar (reliability_floor vs the cleared θ), correct/refused/**wrong** counts
|
||||
(wrong load-bearing), and a plain verdict pill: "earned SERVE", "earned
|
||||
PROPOSE", or "not yet licensed". **Failures-first** ordering (lowest
|
||||
reliability / un-licensed at top). VirtualizedList + useListNavigation +
|
||||
SearchInput; Panel/TabBar detail (Counts / License math / Raw).
|
||||
- The "License math" tab shows the honest derivation: committed N, Wilson
|
||||
floor, θ required, measured ≥ required → licensed — read from B1, not
|
||||
computed in the UI.
|
||||
- Nav entry; Calibration row in `routeConformance` (loading "Loading
|
||||
calibration...", empty "No calibration evidence yet." + the practice-lane
|
||||
CLI, error). Selection publishes an evidence subject (new `calibration_class`
|
||||
kind, inspect-param) — or, if that's too much for one PR, local selection
|
||||
+ flag the subject-kind as a follow-up.
|
||||
- Tests: failures-first ordering, the un-licensed/below-N_MIN class renders
|
||||
"not yet licensed", the wrong count renders, j/k spine.
|
||||
|
||||
### Verify: `cd workbench-ui && pnpm build && pnpm test`
|
||||
|
||||
---
|
||||
|
||||
## Brief B3 — wrong=0 as a felt global presence (frontend; parallel with B2)
|
||||
|
||||
### Gates
|
||||
```bash
|
||||
git worktree add ../core-wb-m-b3 origin/main -b feat/wb-m-wrong-zero-frame
|
||||
grep -q "ServingMetrics" workbench-ui/src/types/api.ts || echo "STOP: B1 not merged"
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
- A small always-present invariant element in the `Shell` chrome (header
|
||||
strip): live **N correct · N refused · 0 wrong**, the zero rendered hard
|
||||
(verified token) — sourced from `/serving/metrics`, never invented; when
|
||||
unavailable, render an honest "metrics unavailable", never a fake zero.
|
||||
- It links to the Calibration route (B2) and the Evals wrong=0 ledger.
|
||||
- **Doctrine line:** the strip states an invariant, it does not *claim*
|
||||
correctness it can't read — if the committed report shows wrong>0 it shows
|
||||
wrong>0 in the contradicted token (the strip must be able to show a
|
||||
non-zero wrong honestly; it is a mirror, not a slogan).
|
||||
- Tests: renders the triplet from a stubbed metrics fetch; renders a
|
||||
non-zero wrong honestly (no hard-coded zero); honest absence on fetch
|
||||
error.
|
||||
|
||||
### Verify: `cd workbench-ui && pnpm build && pnpm test`
|
||||
|
||||
---
|
||||
|
||||
## Brief B4 — The leeway story (frontend; after B1 + B2)
|
||||
|
||||
### Gates
|
||||
```bash
|
||||
git worktree add ../core-wb-m-b4 origin/main -b feat/wb-m-leeway-wiring
|
||||
grep -q "CalibrationClass" workbench-ui/src/types/api.ts || echo "STOP: B1 not merged"
|
||||
```
|
||||
|
||||
### Deliverables
|
||||
- In the Replay / Proposals evidence rails, when a turn or proposal carries an
|
||||
approximate/served result, surface *why latitude was granted*: the class,
|
||||
its license (PROPOSE/SERVE), the θ it cleared, and the `[approximate]`
|
||||
disclosure — joining the existing HITL ratification to the calibration that
|
||||
grants it. Read from B1; link to the Calibration route.
|
||||
- No new mutation; purely a read-only cross-link/annotation.
|
||||
- Tests: a served-with-leeway fixture renders its class + θ + license; a
|
||||
fully-verified turn renders no leeway annotation (absence is honest).
|
||||
|
||||
### Verify: `cd workbench-ui && pnpm build && pnpm test`
|
||||
|
||||
---
|
||||
|
||||
## After this pack
|
||||
|
||||
Phase C brief pack (cognitive-pipeline visualizer, contemplation-as-process,
|
||||
field substrate, identity continuity) is authored once Phase B lands —
|
||||
C1/C3 are also backend-reader-first and Python-gated.
|
||||
179
docs/workbench/wave-M-worthiness.md
Normal file
179
docs/workbench/wave-M-worthiness.md
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
# Wave M — CORE Workbench: Mastery & Worthiness
|
||||
|
||||
Date: 2026-06-13
|
||||
Status: approved plan (Shay, 2026-06-13). Predecessor: Wave R complete
|
||||
(#702–#723; 11 routes real, Replay Moment, trace integrity, DAG/Demo/wrong=0).
|
||||
Execution: committed brief packs in `docs/handoff/`, parallel-safe DAGs,
|
||||
dispatched between Fable 5 and GPT5.5 — the same production line that
|
||||
shipped R2 + R3.
|
||||
|
||||
## Thesis
|
||||
|
||||
Two asks, one lens.
|
||||
|
||||
1. **Mastery** — take the shipped surface from very good to best-in-class.
|
||||
2. **Worthiness** — add what's *missing* so the workbench is undeniably
|
||||
worthy of the deterministic cognitive engine beneath it.
|
||||
|
||||
The lens: **Anthropic and xAI as target users who would *want* to use it.**
|
||||
They build the opaque transformer this engine defines itself *against*. What
|
||||
impresses them is not prettier charts — it is a UI that makes
|
||||
**determinism, refusal-discipline, and geometric coherence inspectable and
|
||||
felt.** Standard: ADR-0160's three pillars — audit-native (not analytics
|
||||
theater), calm default / infinite depth, replay before persuasion.
|
||||
|
||||
## Diagnosis — the two blind spots
|
||||
|
||||
The workbench today is excellent at **evidence browsing**: every route
|
||||
projects an evidence manifold, the Evidence Chain Rail threads provenance,
|
||||
the Replay Moment makes hash-equality felt. But it is blind to the two most
|
||||
*distinctive* parts of the organism:
|
||||
|
||||
1. **It shows the teaching/ratification loop and is blind to the
|
||||
calibrated-learning / serving-discipline loop.** You can ratify a
|
||||
proposal, but you cannot *see* the gold-tether arena, the reliability
|
||||
gate, the Wilson floor vs the θ ceiling, or the moment "the engine earns
|
||||
the right to guess." That discipline — *the engine refuses rather than
|
||||
guesses wrong* — is the single most impressive idea in the project, and
|
||||
it is invisible.
|
||||
2. **It shows outputs and evidence but not cognition itself.** The
|
||||
`CognitiveTurnPipeline` stages, the contemplation *process*, the CL(4,1)
|
||||
field substrate, `versor_condition`, identity continuity — none are
|
||||
legible. For an audience that lives inside opaque models, *legible
|
||||
deterministic cognition* is the wow.
|
||||
|
||||
Everything below closes those two gaps on top of a mastery polish.
|
||||
|
||||
## Non-negotiable disciplines (bind every phase)
|
||||
|
||||
- **Backend-reader-first, no theater.** Every new surface reads *real*
|
||||
engine data through a new read-only reader; no dashboard over invented or
|
||||
recomputed numbers. The calibration and field readers do not exist yet —
|
||||
that gating work is Python, not React.
|
||||
- **Never re-implement engine math in the workbench.** The calibration
|
||||
reader *imports and uses* `core.reliability_gate` (`conservative_floor`,
|
||||
`license_for`, `Ceilings`, `Action`); the field reader uses the engine's
|
||||
real `versor_condition`/`cga_inner`. The workbench computes nothing the
|
||||
engine owns.
|
||||
- **Read-only doctrine holds.** No new mutation endpoints; execution stays
|
||||
the existing allowlisted set (`/evals/run`, ratify, `/demos/{id}/run`). A
|
||||
calibration view never *changes* a license.
|
||||
- **Determinism in the UI too.** No force-directed / nondeterministic
|
||||
layout, no decorative motion-as-cognition. Golden-file layout tests for
|
||||
every new visualizer (like the DAG). The honesty *is* the impressiveness.
|
||||
- **Doctrine gates extend to every new surface**: schema mirrored, enums
|
||||
covered, route conformant, readers SHA-pinned where they assert a metric.
|
||||
|
||||
## Phases (priority-ordered)
|
||||
|
||||
### Phase A — Mastery polish of the shipped surface (scope: M; parallel)
|
||||
No new concepts; make the 11 routes undeniable.
|
||||
- Design-system full expression: semantic token roles, elevation, **density
|
||||
modes actually wired** (the deferred Settings density pref), `tabular-nums`
|
||||
on all numerics, `[text-wrap:balance]` on all statements, motion-discipline
|
||||
audit (only state-transition affordances).
|
||||
- Cross-route consistency sweep: every list = `VirtualizedList` +
|
||||
`useListNavigation` + `SearchInput` + selection tokens; every detail =
|
||||
`Panel` + `TabBar`; calm-honest prose audit on every state.
|
||||
- **DAG viewer: finish its consumers.** It shipped wired only to proposal
|
||||
chains; wire the **PCCP proof-promotion 8 scenarios** and **entailment
|
||||
traces** (the other two the brief named). A primitive with one consumer is
|
||||
half-built.
|
||||
- Command/keyboard completeness: a palette verb for every route action;
|
||||
registry-driven help stays the exhaustive contract.
|
||||
- Accessibility pass: focus-visible audit, SR labels on every evidence
|
||||
badge, reduced-motion honored.
|
||||
|
||||
### Phase B — Calibrated-Learning / Serving-Discipline surfaces (scope: L) ← the heart
|
||||
The "worthy of the model" core. Backend-reader-first (none exist; data lives
|
||||
in `core/reliability_gate/` + the committed `evals/gsm8k_math/*/report.json`).
|
||||
Detailed brief pack: `docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.md`.
|
||||
- **B1 (Python):** read-only readers/endpoints over the real ledger —
|
||||
`GET /calibration/classes` (per-class `ClassTally` counts + the Wilson
|
||||
`conservative_floor` reliability + PROPOSE/SERVE `license_for` verdicts via
|
||||
the real `core.reliability_gate`), `GET /serving/metrics` (the committed
|
||||
`train_sample/v1/report.json` numbers — read the artifact, never re-run an
|
||||
unsafe lane). Schema mirrors + snapshots + drift gate.
|
||||
- **B2 — Calibration / Gold-Tether route:** per class, a
|
||||
coverage-vs-Wilson-floor bar, the θ ceiling, and a plain-language "earned
|
||||
PROPOSE / SERVE / neither" verdict. Failures-first. Where you *see* "the
|
||||
engine earns the right to guess."
|
||||
- **B3 — wrong=0 as a felt global presence:** an always-present invariant
|
||||
element (N correct / N refused / **0 wrong**, the zero load-bearing),
|
||||
elevating the per-run Evals ledger to the project's thesis made constant.
|
||||
- **B4 — the leeway story:** wire the calibration verdict into the Proposals
|
||||
/ Replay rails so a reviewer sees *why* a turn was granted latitude (which
|
||||
class license, which θ, the `[approximate]` disclosure) — connecting the
|
||||
HITL ratification you already have to the calibration that grants it.
|
||||
|
||||
### Phase C — Make cognition legible (scope: L) ← the wow for Anthropic/xAI
|
||||
- **C1 — Cognitive Pipeline visualizer:** for a selected turn, render the
|
||||
real `CognitiveTurnPipeline` stages (intent → PropositionGraph →
|
||||
ArticulationTarget → realizer → walk telemetry → trace hash) as a
|
||||
deterministic staged view (reuse the DAG primitive). *The* "real,
|
||||
replayable path, not animated fake cognition" surface. Reader-first over
|
||||
existing trace/walk telemetry.
|
||||
- **C2 — Contemplation as a process, not just outputs:** the contemplation
|
||||
*loop* (attempt → gold-tether → ClassTally → propose), connecting
|
||||
Demos/Proposals/Calibration into one story.
|
||||
- **C3 — Field substrate (honest, read-only, hard):** `GET /field/state`
|
||||
over real `FieldState` + `versor_condition` for a turn, rendered as
|
||||
**inspectable exact numbers and invariant status** — `versor_condition <
|
||||
1e-6` as a live "field is valid" assertion, `cga_inner` coherence as exact
|
||||
values. **NOT** a decorative 3D blob; no force-directed/nondeterministic
|
||||
motion. The honesty is the impressiveness: "this is the geometry, it's
|
||||
exact, it can't fake coherence."
|
||||
- **C4 — Identity continuity (L10/L11):** surface the engine-identity hash,
|
||||
lineage chain, reboot-verification status — "the same continuous life
|
||||
across restart," the deepest telos, currently invisible.
|
||||
|
||||
### Phase D — The "they'd want to use it" layer (scope: M)
|
||||
- **Guided Determinism Tour** — elevate Demo Theater into a first-run
|
||||
narrative: pick a demo, watch the proposer get disciplined, see
|
||||
hash-to-hash replay, see a wrong answer *refused*. "What this proves / what
|
||||
this does not prove" honesty cards on every scenario.
|
||||
- **Provider-agnostic framing** — the pitch for Anthropic *and* xAI: "bring
|
||||
your own model's claim; watch the deterministic engine decide, refuse, and
|
||||
replay it." The Tool-Authority / Hybrid-Verification demos already embody
|
||||
this; make it the tour's spine.
|
||||
- **Shareable evidence bundles** — deterministic export of a turn + its
|
||||
trace + replay + calibration verdict as a single citable artifact.
|
||||
Reproducibility *as a deliverable*.
|
||||
|
||||
### Phase E — Robustness pillars (scope: S; continuous)
|
||||
- Extend doctrine gates to every new surface; SHA-pin the calibration/field
|
||||
readers where they assert a metric.
|
||||
- Performance budget (resolve the Vite chunk-size warning via route
|
||||
code-split), error-boundary discipline, golden-file regime for the
|
||||
pipeline/field visualizers.
|
||||
|
||||
## What's missing in the design (the second ask, distilled)
|
||||
|
||||
| Missing surface | Why it matters for worthiness | Reader exists? |
|
||||
|---|---|---|
|
||||
| Calibration / gold-tether arena | Makes wrong=0 *earned*, not asserted — the most distinctive idea, invisible | **No** — build first |
|
||||
| Serving-vs-learning regime frame | Names the two-regime architecture; without it the UI reads as a chatbot | No |
|
||||
| wrong=0 as a felt global presence | The thesis itself; today only per-eval-run | Partial (ledger) |
|
||||
| Cognitive pipeline visualizer | "Real replayable cognition" vs animated fake — the core wow | Trace exists; needs staging reader |
|
||||
| Contemplation-as-process | The learning flywheel, today only its outputs | Partial |
|
||||
| Field substrate / versor_condition | The geometry that *can't fake coherence* — honest, exact | **No** — build first |
|
||||
| Identity continuity (L10/L11) | "One continuous life" — the deepest telos | No |
|
||||
| Serving metrics reachable | The actual capability numbers (gsm8k) aren't viewable | No |
|
||||
|
||||
## Risks
|
||||
|
||||
- **Theater is risk #1** — mitigated by backend-reader-first + never
|
||||
re-implementing engine math. The gating work (B1, C1, C3 readers) is
|
||||
Python and parallel-safe.
|
||||
- **The field surface must stay honest** — read-only over real
|
||||
`versor_condition`/`cga_inner`, no decorative geometry, no motion theater.
|
||||
- **Scope is large** — several PR trains. Sequences as readers → routes →
|
||||
cross-wiring → tour. Phase A runs in parallel as polish.
|
||||
- No timelines — phases/priorities/scope-sizes; sequencing is the dependency
|
||||
DAG, not a clock.
|
||||
|
||||
## Execution order
|
||||
|
||||
**B → C → D**, with **A in parallel**. The worthiness gap is widest at B; the
|
||||
tour (D) lands hardest once B and C exist to show off. Phase B brief pack is
|
||||
authored first (this commit); subsequent phase packs follow as each lands.
|
||||
Loading…
Reference in a new issue