Merge pull request #729 from AssetOverflow/docs/wave-m-plan-corrections

docs(workbench): B3.5 D2 runner-reproducibility + C1-a persistence constraint
This commit is contained in:
Shay 2026-06-13 07:20:53 -07:00 committed by GitHub
commit 5a1266af48
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 32 additions and 6 deletions

View file

@ -111,8 +111,28 @@ Detailed brief pack: `docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.m
real `CognitiveTurnPipeline` stages (intent → PropositionGraph →
ArticulationTarget → realizer → walk telemetry → trace hash) as a
deterministic staged view (reuse the DAG primitive). *The* "real,
replayable path, not animated fake cognition" surface. Reader-first over
existing trace/walk telemetry.
replayable path, not animated fake cognition" surface.
**Persistence-first constraint (verified 2026-06-13):** this is NOT
reader-first over existing telemetry — the journal does not persist the
stage internals. `CognitiveTurnResult` carries ~25 fields (field_state,
proposition, proposition_graph, intent, admissibility_trace,
operator_invocation, dropped_compound_clauses, versor_condition, …) but
`TurnJournalEntry` persists only ~12 surface fields, and `_run_chat_turn`
discards the rest; there is no `/trace/{id}/pipeline` endpoint and the data
was never written. So **C1-a's real first deliverable is a persistence
change, not an API route**: persist a *curated* `CognitivePipelineRecord`
at turn-write time, then the endpoint is a trivial read-only projection.
Two hard constraints: (1) persist the cheap structured stage records +
`versor_condition` as a scalar (and at most a field *digest*) — **never the
raw `field_state_before/after` multivectors**, which would resurrect the
deferred per-turn O(n²) persistence cost and contradict the L10
discard-on-exit design; (2) it touches the runtime surface contract
(`CognitiveTurnResult → ChatTurnResult → TurnJournalEntry` is two narrowing
hops), so the PR updates `docs/runtime_contracts.md` + a **non-vacuous
fail-closed test** that a silently-dropped stage fails loudly (CLAUDE.md
Schema-Defined Proof Obligations), and pre-widening turns show
`missing_evidence`, not green. The replay-reconstruction fallback (Option B)
recomputes rather than reads recorded state — fallback only, never primary.
- **C2 — Contemplation as a process, not just outputs:** the contemplation
*loop* (attempt → gold-tether → ClassTally → propose), connecting
Demos/Proposals/Calibration into one story.

View file

@ -45,6 +45,8 @@ That means calibration is still partly a page-local experience, not fully part o
**Addendum (review, 2026-06-13): the centerpiece currently undersells its own thesis.** The reader reads `evals/gsm8k_math/practice/v1/report.json` `per_class`, whose committed copy is a sub-`N_MIN` baseline (`additive` committed=0) — so on live data the route shows three classes, all "not yet licensed", with empty reliability bars. It never shows a class *crossing θ*, which is the entire point. Meanwhile the *earned* state (`additive` committed=100, measured 0.861, **PROPOSE-licensed**) lives in the separately-committed `ratification_queue.json`, and the two artifacts disagree (correct 3 vs 95, committed in different commits). An evaluator opening Calibration today sees the discipline's scaffolding but not its moment. The fix is data-side, not a re-derivation: make the committed practice artifacts coherent (regenerate from one deterministic practice run via the runner) so `report.json` `per_class` shows ≥1 earned class and matches the queue; the reader stays unchanged. Sealed-practice may carry wrong>0 (attempt-and-eliminate) — that is *not* the serving wrong=0; do not conflate. Folds into Deliverable 2.
**Correction (review, 2026-06-13b) — coherence is necessary but NOT sufficient; the bar is runner-reproducibility.** A first attempt at this fix made `report.json` *agree with* the existing `ratification_queue.json` (both `95/50/5`, `additive` PROPOSE @0.861) — but verification showed those numbers are a **fossil**: the queue was committed back in `b82897a0` when the now-disabled `resolve_pooled` scorer was active (every new elimination record is tagged `reason: "resolve_pooled"`), and **no current runner path reproduces them**. The canonical `runner.py main()` (`build_report()` over the train sample) yields `6/44/0`; `build_practice_report()` over the 150 practice cases with the default candidate-graph scorer yields `0/1/149` — additive does **not** earn PROPOSE today. Making the report agree with a stale queue achieved artifact-to-artifact coherence while breaking artifact-to-runner coherence, surfacing an *earned PROPOSE license the engine cannot currently reproduce* — the precise false-epistemic-status the workbench exists to refuse. **D2 acceptance is therefore strengthened: the committed `report.json` MUST be byte-reproducible by a documented, deterministic runner entry point** (e.g. `python -m evals.gsm8k_math.practice.v1.runner` after repointing `main()` to `build_practice_report()`), and the queue must be regenerated from that *same* pass. Three admissible routes: (A) **honest floor** — commit the real candidate-graph practice output, let the route show "not yet licensed" until a class genuinely crosses θ; (B) **earn it for real** — stand up a reproducible, deterministic practice scorer that legitimately earns additive PROPOSE, wired into the runner, regenerating report+queue from one pass; (C) defer the "earned" demo and keep the route honest meanwhile. Take (A) now; file (B) as the follow-up that actually delivers the θ-crossing moment. Copying numbers between committed artifacts is **not** an admissible fix.
### 3. B4 is conceptually right but data-shape risky
B4 wants the Replay/Proposals evidence rails to explain why approximation/leeway was granted: class, license, theta, disclosure, and relation to HITL ratification.
@ -186,10 +188,14 @@ Show only fields carried by the backend read model:
- RightInspector handles the subject without falling to raw unknown.
- Evidence Chain Rail names calibration as serving-discipline evidence, not runtime truth.
- Missing data renders as missing/unknown, not as green.
- The committed practice artifacts are coherent (`report.json` `per_class`
agrees with `ratification_queue.json`) and the route shows ≥1 earned
(licensed) class — the centerpiece demonstrates "earns the right to guess",
not only the un-earned floor.
- The committed `report.json` is **byte-reproducible by a documented,
deterministic runner entry point** (re-running it overwrites the file with
identical bytes) — coherence with `ratification_queue.json` alone is
insufficient; the queue is regenerated from the *same* pass, never copied
into the report. See the "runner-reproducibility" correction above.
- If a class is shown as earned (licensed), the current runner reproduces that
earned state; the route never asserts a license the engine cannot currently
reproduce. (If none earns yet, the route honestly shows the un-earned floor.)
---