feat(workbench): Wave M D3 — shareable evidence bundles (reproducibility as a deliverable)
The "they'd want to use it" deliverable: a turn's evidence exported as ONE
deterministic, content-addressed, citable artifact — composing the Phase-C
evidence (pipeline + field) with the trace and the calibration leeway verdict.
Backend (read-only, no engine execution):
- schemas.py: EvidenceBundle.
- workbench/evidence_bundle.py: build_evidence_bundle(entry) assembles a turn
journal entry into the bundle and computes bundle_digest. The digest
content-addresses the DETERMINISTIC cognitive evidence only — journal
position + wall-clock (turn_id, journal_digest, replay_reproducer,
generated_from) are carried for provenance but EXCLUDED, so the same turn
content reproduces the same digest regardless of journal position.
- api.py: GET /trace/{turn_id}/bundle. schema-snapshot.json regenerated.
The bundle carries a replay *reproducer* command (how to verify) rather than a
live-run replay, so the artifact itself stays deterministic — verification is
the consumer's step: re-run the prompt sealed, confirm trace_hash, recompute the
bundle, check the digest.
Frontend:
- types/api.ts EvidenceBundle; client/query hook; Trace route **Bundle** tab —
citable digest, "what this proves / does not prove" honesty note, the
reproducer, and a deterministic JSON download (Blob anchor, leak-safe).
Validation: 135 workbench Python tests incl. non-vacuous bundle guards (digest
reproducible, journal-position/wall-clock excluded, any evidence change flips
the digest, missing Phase-C evidence honest); 468/468 frontend incl. schemaDrift
+ the citable-download bundle test; pnpm build clean; git diff --check clean. No
serving-path imports.
This commit is contained in:
parent
dedaa0b7dd
commit
5061a0804b
12 changed files with 500 additions and 0 deletions
|
|
@ -190,6 +190,42 @@ first-class read endpoint is `GET /trace/{turn_id}/field`; it returns a
|
|||
measured value against the ceiling, the `cga_inner` transition, and the digests
|
||||
— inspectable exact numbers and invariant status, no decorative geometry.
|
||||
|
||||
## EvidenceBundle (D3 shareable evidence bundle)
|
||||
|
||||
```ts
|
||||
export type EvidenceBundle = {
|
||||
schema_version: "evidence_bundle_v1";
|
||||
turn_id: number;
|
||||
generated_from: "turn_journal";
|
||||
trace_hash: string | null;
|
||||
trace_integrity: "pipeline_trace" | "legacy_unhashed";
|
||||
prompt: string;
|
||||
surface: string;
|
||||
grounding_source: GroundingSource;
|
||||
epistemic_state: EpistemicState;
|
||||
normative_clearance: NormativeClearance;
|
||||
refusal_emitted: boolean;
|
||||
journal_digest: string; // provenance — NOT in the content digest
|
||||
pipeline_record: CognitivePipelineRecord | null;
|
||||
field_evidence: FieldEvidence | null;
|
||||
leeway_evidence: LeewayEvidence | null;
|
||||
replay_reproducer: string; // how to verify — NOT in the content digest
|
||||
bundle_digest: string; // sha256 content address of the evidence
|
||||
};
|
||||
```
|
||||
|
||||
A turn's evidence exported as one citable artifact — *reproducibility as a
|
||||
deliverable*. `bundle_digest` content-addresses the **deterministic cognitive
|
||||
evidence** only (prompt, surface, trace_hash, grounding/epistemic/normative
|
||||
state, pipeline + field evidence, leeway verdict). Journal position and
|
||||
wall-clock metadata (`turn_id`, `journal_digest`, `replay_reproducer`,
|
||||
`generated_from`) are carried for provenance but **excluded** from the digest,
|
||||
so the same turn content always yields the same digest regardless of where it
|
||||
landed in a journal. Read-only — a pure projection of a persisted journal
|
||||
entry, no engine execution. Endpoint: `GET /trace/{turn_id}/bundle`. The Trace
|
||||
route's **Bundle** tab shows the citable digest, a "what this proves / does not
|
||||
prove" honesty note, the reproducer, and a deterministic JSON download.
|
||||
|
||||
---
|
||||
|
||||
# Proposal
|
||||
|
|
|
|||
|
|
@ -213,6 +213,21 @@ Detailed brief pack: `docs/handoff/wave-M-phaseB-calibration-briefs-2026-06-13.m
|
|||
- **Shareable evidence bundles** — deterministic export of a turn + its
|
||||
trace + replay + calibration verdict as a single citable artifact.
|
||||
Reproducibility *as a deliverable*.
|
||||
**D3 implementation note (2026-06-13):** BUILT. `workbench/evidence_bundle.py`
|
||||
assembles a turn journal entry into a content-addressed `EvidenceBundle`
|
||||
composing the Phase-C evidence (pipeline + field) with the trace and the
|
||||
calibration leeway verdict. The `bundle_digest` content-addresses the
|
||||
**deterministic cognitive evidence only** — journal position + wall-clock
|
||||
(`turn_id`, `journal_digest`, `replay_reproducer`) are carried for provenance
|
||||
but excluded, so the same turn content reproduces the same digest (verified by
|
||||
test: identical content → identical digest; different journal position → same
|
||||
digest; any evidence change → different digest). Read endpoint
|
||||
`GET /trace/{turn_id}/bundle`; surfaced as the Trace **Bundle** tab (citable
|
||||
digest, "what this proves / does not prove" honesty note, reproducer command,
|
||||
deterministic JSON download). Read-only, no engine execution. The bundle
|
||||
carries the replay *reproducer* rather than a live-run replay so the artifact
|
||||
itself stays deterministic — verification is the consumer's step. Phase D
|
||||
remaining: guided determinism tour (D1) + provider-agnostic framing (D2).
|
||||
|
||||
### Phase E — Robustness pillars (scope: S; continuous)
|
||||
- Extend doctrine gates to every new surface; SHA-pin the calibration/field
|
||||
|
|
|
|||
97
tests/test_workbench_evidence_bundle.py
Normal file
97
tests/test_workbench_evidence_bundle.py
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
"""D3 shareable evidence bundle — non-vacuous determinism guards.
|
||||
|
||||
The bundle is the citable claim; the ``bundle_digest`` is its content address.
|
||||
These tests fail under the violations that would make it a lie: a digest that
|
||||
drifts on identical content, a digest that leaks journal position / wall-clock,
|
||||
or a digest that does NOT move when the cognitive evidence changes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from workbench.evidence_bundle import build_evidence_bundle
|
||||
from workbench.journal import TurnJournalEntry
|
||||
|
||||
|
||||
def _entry(**overrides: object) -> TurnJournalEntry:
|
||||
base: dict[str, object] = dict(
|
||||
turn_id=1,
|
||||
timestamp="2026-06-13T00:00:00Z",
|
||||
trace_hash="th-abc",
|
||||
prompt="why",
|
||||
surface="because",
|
||||
articulation_surface=None,
|
||||
walk_surface=None,
|
||||
grounding_source="pack",
|
||||
epistemic_state="verified",
|
||||
normative_clearance="cleared",
|
||||
verdicts={},
|
||||
refusal_emitted=False,
|
||||
hedge_injected=False,
|
||||
proposal_candidates=[],
|
||||
turn_cost_ms=12,
|
||||
checkpoint_emitted=False,
|
||||
journal_digest="jd-1",
|
||||
)
|
||||
base.update(overrides)
|
||||
return TurnJournalEntry(**base) # type: ignore[arg-type]
|
||||
|
||||
|
||||
class TestBundleDigest:
|
||||
def test_same_content_is_reproducible(self) -> None:
|
||||
assert (
|
||||
build_evidence_bundle(_entry()).bundle_digest
|
||||
== build_evidence_bundle(_entry()).bundle_digest
|
||||
)
|
||||
|
||||
def test_journal_position_and_wall_clock_are_excluded(self) -> None:
|
||||
# Same cognitive evidence, different journal position / wall-clock ->
|
||||
# SAME digest. This is the "reproducibility as a deliverable" claim.
|
||||
base = build_evidence_bundle(_entry())
|
||||
moved = build_evidence_bundle(
|
||||
_entry(
|
||||
turn_id=99,
|
||||
journal_digest="jd-DIFFERENT",
|
||||
timestamp="2099-01-01T00:00:00Z",
|
||||
turn_cost_ms=9999,
|
||||
)
|
||||
)
|
||||
assert base.bundle_digest == moved.bundle_digest
|
||||
# ...but the provenance fields ARE carried on the bundle.
|
||||
assert moved.turn_id == 99
|
||||
assert moved.journal_digest == "jd-DIFFERENT"
|
||||
|
||||
def test_surface_change_flips_digest(self) -> None:
|
||||
assert (
|
||||
build_evidence_bundle(_entry()).bundle_digest
|
||||
!= build_evidence_bundle(_entry(surface="something else")).bundle_digest
|
||||
)
|
||||
|
||||
def test_trace_hash_change_flips_digest(self) -> None:
|
||||
assert (
|
||||
build_evidence_bundle(_entry()).bundle_digest
|
||||
!= build_evidence_bundle(_entry(trace_hash="th-OTHER")).bundle_digest
|
||||
)
|
||||
|
||||
def test_grounding_change_flips_digest(self) -> None:
|
||||
assert (
|
||||
build_evidence_bundle(_entry()).bundle_digest
|
||||
!= build_evidence_bundle(_entry(grounding_source="oov")).bundle_digest
|
||||
)
|
||||
|
||||
|
||||
class TestBundleShape:
|
||||
def test_carries_deterministic_evidence_and_reproducer(self) -> None:
|
||||
bundle = build_evidence_bundle(_entry())
|
||||
assert bundle.schema_version == "evidence_bundle_v1"
|
||||
assert bundle.generated_from == "turn_journal"
|
||||
assert bundle.trace_hash == "th-abc"
|
||||
assert bundle.bundle_digest.startswith("sha256:")
|
||||
assert "core replay turn 1" in bundle.replay_reproducer
|
||||
assert "th-abc" in bundle.replay_reproducer
|
||||
|
||||
def test_missing_phase_c_evidence_is_honest_not_fabricated(self) -> None:
|
||||
bundle = build_evidence_bundle(_entry())
|
||||
# No pipeline/field/leeway persisted on this bare entry -> carried as null.
|
||||
assert bundle.pipeline_record is None
|
||||
assert bundle.field_evidence is None
|
||||
assert bundle.leeway_evidence is None
|
||||
|
|
@ -181,6 +181,25 @@
|
|||
"cases",
|
||||
"source_digest"
|
||||
],
|
||||
"EvidenceBundle": [
|
||||
"schema_version",
|
||||
"turn_id",
|
||||
"generated_from",
|
||||
"trace_hash",
|
||||
"trace_integrity",
|
||||
"prompt",
|
||||
"surface",
|
||||
"grounding_source",
|
||||
"epistemic_state",
|
||||
"normative_clearance",
|
||||
"refusal_emitted",
|
||||
"journal_digest",
|
||||
"pipeline_record",
|
||||
"field_evidence",
|
||||
"leeway_evidence",
|
||||
"replay_reproducer",
|
||||
"bundle_digest"
|
||||
],
|
||||
"FieldEvidence": [
|
||||
"schema_version",
|
||||
"status",
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ import type {
|
|||
ServingMetrics,
|
||||
CognitivePipelineRecord,
|
||||
FieldEvidence,
|
||||
EvidenceBundle,
|
||||
TurnJournalEntry,
|
||||
TurnJournalSummary,
|
||||
ContemplationRunDetail,
|
||||
|
|
@ -193,6 +194,12 @@ export async function fetchTraceField(turnId: number): Promise<FieldEvidence> {
|
|||
);
|
||||
}
|
||||
|
||||
export async function fetchTraceBundle(turnId: number): Promise<EvidenceBundle> {
|
||||
return apiFetch<EvidenceBundle>(
|
||||
`/trace/${encodeURIComponent(String(turnId))}/bundle`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function fetchAuditEvents(
|
||||
limit?: number,
|
||||
offset?: number,
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ import {
|
|||
fetchTraceTurn,
|
||||
fetchTracePipeline,
|
||||
fetchTraceField,
|
||||
fetchTraceBundle,
|
||||
fetchTraceTurns,
|
||||
fetchContemplationRun,
|
||||
fetchContemplationRuns,
|
||||
|
|
@ -57,6 +58,7 @@ import type {
|
|||
ServingMetrics,
|
||||
CognitivePipelineRecord,
|
||||
FieldEvidence,
|
||||
EvidenceBundle,
|
||||
TurnJournalEntry,
|
||||
TurnJournalSummary,
|
||||
EvalLaneSummary,
|
||||
|
|
@ -222,6 +224,16 @@ export function useTraceField(turnId?: number | null) {
|
|||
});
|
||||
}
|
||||
|
||||
export function useTraceBundle(turnId?: number | null) {
|
||||
return useQuery<EvidenceBundle, WorkbenchApiError>({
|
||||
queryKey: ["api", "trace", "bundle", turnId ?? null],
|
||||
queryFn: () => fetchTraceBundle(turnId as number),
|
||||
enabled: typeof turnId === "number",
|
||||
staleTime: 30_000,
|
||||
refetchOnWindowFocus: false,
|
||||
});
|
||||
}
|
||||
|
||||
export function useAuditEvents(limit?: number, offset?: number) {
|
||||
return useQuery<{ items: AuditEvent[] }, WorkbenchApiError>({
|
||||
queryKey: ["api", "audit", "events", limit ?? null, offset ?? null],
|
||||
|
|
|
|||
|
|
@ -183,6 +183,36 @@ function stubTraceFetch(
|
|||
}),
|
||||
});
|
||||
}
|
||||
const bundleMatch = path.match(/^\/trace\/(\d+)\/bundle$/);
|
||||
if (bundleMatch) {
|
||||
const id = Number(bundleMatch[1]);
|
||||
return Promise.resolve({
|
||||
json: () =>
|
||||
Promise.resolve({
|
||||
ok: true,
|
||||
generated_at: "now",
|
||||
data: {
|
||||
schema_version: "evidence_bundle_v1",
|
||||
turn_id: id,
|
||||
generated_from: "turn_journal",
|
||||
trace_hash: `sha256:trace${id}`,
|
||||
trace_integrity: "pipeline_trace",
|
||||
prompt: "p",
|
||||
surface: "s",
|
||||
grounding_source: "pack",
|
||||
epistemic_state: "evidenced",
|
||||
normative_clearance: "cleared",
|
||||
refusal_emitted: false,
|
||||
journal_digest: `sha256:journal${id}`,
|
||||
pipeline_record: null,
|
||||
field_evidence: null,
|
||||
leeway_evidence: null,
|
||||
replay_reproducer: `core replay turn ${id} # re-run sealed; expect trace_hash == sha256:trace${id}`,
|
||||
bundle_digest: `sha256:bundle${id}abcdef0123456789`,
|
||||
},
|
||||
}),
|
||||
});
|
||||
}
|
||||
const match = path.match(/^\/trace\/(\d+)$/);
|
||||
if (match) {
|
||||
return Promise.resolve({
|
||||
|
|
@ -281,6 +311,36 @@ describe("TraceRoute", () => {
|
|||
expect(screen.getByText("Walk Surface (telemetry/evidence)")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("exports a citable, downloadable evidence bundle", async () => {
|
||||
// Set only the two object-URL methods (jsdom lacks them); leave the URL
|
||||
// constructor intact so the fetch mock's `new URL()` keeps working.
|
||||
const createObjectURL = vi.fn(() => "blob:mock-bundle");
|
||||
const revokeObjectURL = vi.fn();
|
||||
const originalCreate = URL.createObjectURL;
|
||||
const originalRevoke = URL.revokeObjectURL;
|
||||
URL.createObjectURL = createObjectURL as typeof URL.createObjectURL;
|
||||
URL.revokeObjectURL = revokeObjectURL as typeof URL.revokeObjectURL;
|
||||
stubTraceFetch();
|
||||
const user = userEvent.setup();
|
||||
renderRoute("/trace/2");
|
||||
|
||||
await user.click(await screen.findByRole("tab", { name: "Bundle" }));
|
||||
|
||||
const bundle = await screen.findByTestId("evidence-bundle");
|
||||
// The citable digest is shown (truncated form of the content address).
|
||||
expect(within(bundle).getByText(/bundle2abcdef0123/)).toBeInTheDocument();
|
||||
// "What this proves / does not prove" honesty is present.
|
||||
expect(within(bundle).getByText(/Does not prove:/)).toBeInTheDocument();
|
||||
// A deterministic download anchor is offered.
|
||||
const download = within(bundle).getByTestId("bundle-download");
|
||||
expect(download).toHaveAttribute("href", "blob:mock-bundle");
|
||||
expect(download.getAttribute("download")).toContain("evidence-bundle-turn-2");
|
||||
expect(createObjectURL).toHaveBeenCalled();
|
||||
|
||||
URL.createObjectURL = originalCreate;
|
||||
URL.revokeObjectURL = originalRevoke;
|
||||
});
|
||||
|
||||
it("renders the persisted cognitive pipeline as a deterministic DAG", async () => {
|
||||
stubTraceFetch();
|
||||
renderRoute("/trace/2");
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import { Eye } from "lucide-react";
|
|||
import { useNavigate, useParams } from "react-router-dom";
|
||||
import { WorkbenchApiError } from "../../api/client";
|
||||
import {
|
||||
useTraceBundle,
|
||||
useTraceField,
|
||||
useTracePipeline,
|
||||
useTraceTurn,
|
||||
|
|
@ -34,6 +35,7 @@ import { LoadingState } from "../../design/components/states/LoadingState";
|
|||
import type {
|
||||
CognitivePipelineRecord,
|
||||
CognitivePipelineStage,
|
||||
EvidenceBundle,
|
||||
FieldEvidence,
|
||||
TraceIntegrity,
|
||||
TurnJournalEntry,
|
||||
|
|
@ -46,6 +48,7 @@ import { useEvidenceSubject } from "../evidenceContext";
|
|||
const TRACE_TABS: readonly Tab[] = [
|
||||
{ id: "pipeline", label: "Pipeline" },
|
||||
{ id: "field", label: "Field" },
|
||||
{ id: "bundle", label: "Bundle" },
|
||||
{ id: "surfaces", label: "Surfaces" },
|
||||
{ id: "grounding", label: "Grounding" },
|
||||
{ id: "verdicts", label: "Verdicts" },
|
||||
|
|
@ -442,6 +445,92 @@ function FieldTab({
|
|||
return <FieldInvariantCard record={record} />;
|
||||
}
|
||||
|
||||
function BundleTab({
|
||||
bundle,
|
||||
isLoading,
|
||||
error,
|
||||
turnId,
|
||||
}: {
|
||||
bundle?: EvidenceBundle | null;
|
||||
isLoading: boolean;
|
||||
error: unknown;
|
||||
turnId: number;
|
||||
}) {
|
||||
// Deterministic export: a stable object URL over the bundle JSON, revoked on
|
||||
// unmount. The bundle is already content-addressed by the backend.
|
||||
const downloadUrl = useMemo(() => {
|
||||
if (!bundle) return null;
|
||||
return URL.createObjectURL(
|
||||
new Blob([JSON.stringify(bundle, null, 2)], { type: "application/json" }),
|
||||
);
|
||||
}, [bundle]);
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
if (downloadUrl) URL.revokeObjectURL(downloadUrl);
|
||||
};
|
||||
}, [downloadUrl]);
|
||||
|
||||
if (isLoading) {
|
||||
return <LoadingState label="Assembling evidence bundle..." />;
|
||||
}
|
||||
if (error) {
|
||||
return (
|
||||
<ErrorState
|
||||
whatFailed={errorMessage(error)}
|
||||
mutationStatus="No trace mutation occurred."
|
||||
reproducer={`curl /trace/${turnId}/bundle`}
|
||||
retrySafety="Retry: safe"
|
||||
/>
|
||||
);
|
||||
}
|
||||
if (!bundle) {
|
||||
return <LoadingState label="Assembling evidence bundle..." />;
|
||||
}
|
||||
|
||||
const digest = digestPayload(bundle.bundle_digest);
|
||||
const fileName = `evidence-bundle-turn-${bundle.turn_id}-${(digest ?? "").slice(0, 12)}.json`;
|
||||
return (
|
||||
<section className="flex flex-col gap-3" data-testid="evidence-bundle">
|
||||
<div className="rounded-md border border-[var(--color-border-subtle)] bg-[var(--color-surface-inset)] p-3">
|
||||
<div className="flex flex-wrap items-center justify-between gap-2">
|
||||
<span className="text-xs font-semibold uppercase tracking-wide text-[var(--color-text-secondary)]">
|
||||
citable digest
|
||||
</span>
|
||||
{digest ? <DigestBadge digest={digest} truncate={16} /> : null}
|
||||
</div>
|
||||
<p className="m-0 mt-2 text-xs text-[var(--color-text-secondary)] [text-wrap:balance]">
|
||||
<span className="font-semibold text-[var(--color-text-primary)]">Proves:</span> this
|
||||
exact evidence (trace, pipeline, field, leeway) is reproducible — re-run the prompt over a
|
||||
sealed runtime, confirm the trace hash, recompute the bundle, and this digest matches.{" "}
|
||||
<span className="font-semibold text-[var(--color-text-primary)]">Does not prove:</span>{" "}
|
||||
anything about a different prompt, model, or runtime.
|
||||
</p>
|
||||
{downloadUrl ? (
|
||||
<a
|
||||
className="mt-3 inline-flex items-center gap-2 text-sm underline"
|
||||
href={downloadUrl}
|
||||
download={fileName}
|
||||
data-testid="bundle-download"
|
||||
>
|
||||
Download evidence bundle
|
||||
</a>
|
||||
) : null}
|
||||
</div>
|
||||
<div>
|
||||
<div className="mb-1 text-xs font-semibold text-[var(--color-text-secondary)]">
|
||||
reproducer
|
||||
</div>
|
||||
<code className="block overflow-auto rounded-md border border-[var(--color-border-subtle)] bg-[var(--color-surface-inset)] p-2 font-mono text-xs text-[var(--color-text-primary)]">
|
||||
{bundle.replay_reproducer}
|
||||
</code>
|
||||
</div>
|
||||
<div className="max-h-[28rem] overflow-auto rounded-md border border-[var(--color-border-subtle)] bg-[var(--color-surface-inset)] p-2">
|
||||
<StableJsonViewer source={JSON.stringify(bundle, null, 2)} />
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
function TraceRow({
|
||||
turn,
|
||||
selected,
|
||||
|
|
@ -602,6 +691,9 @@ function TraceDetail({
|
|||
fieldEvidence,
|
||||
fieldLoading,
|
||||
fieldError,
|
||||
bundle,
|
||||
bundleLoading,
|
||||
bundleError,
|
||||
}: {
|
||||
turn: TurnJournalEntry;
|
||||
pipelineRecord?: CognitivePipelineRecord | null;
|
||||
|
|
@ -610,6 +702,9 @@ function TraceDetail({
|
|||
fieldEvidence?: FieldEvidence | null;
|
||||
fieldLoading: boolean;
|
||||
fieldError: unknown;
|
||||
bundle?: EvidenceBundle | null;
|
||||
bundleLoading: boolean;
|
||||
bundleError: unknown;
|
||||
}) {
|
||||
const [activeTab, setActiveTab] = useState("pipeline");
|
||||
return (
|
||||
|
|
@ -640,6 +735,14 @@ function TraceDetail({
|
|||
turnId={turn.turn_id}
|
||||
/>
|
||||
) : null}
|
||||
{activeTab === "bundle" ? (
|
||||
<BundleTab
|
||||
bundle={bundle}
|
||||
isLoading={bundleLoading}
|
||||
error={bundleError}
|
||||
turnId={turn.turn_id}
|
||||
/>
|
||||
) : null}
|
||||
{activeTab === "surfaces" ? <SurfacesTab turn={turn} /> : null}
|
||||
{activeTab === "grounding" ? <GroundingTab turn={turn} /> : null}
|
||||
{activeTab === "verdicts" ? <VerdictsTab turn={turn} /> : null}
|
||||
|
|
@ -661,6 +764,7 @@ export function TraceRoute() {
|
|||
const turnQuery = useTraceTurn(selectedTurnId);
|
||||
const pipelineQuery = useTracePipeline(selectedTurnId);
|
||||
const fieldQuery = useTraceField(selectedTurnId);
|
||||
const bundleQuery = useTraceBundle(selectedTurnId);
|
||||
|
||||
const turns = turnsQuery.data ?? [];
|
||||
const filteredTurns = useMemo(() => {
|
||||
|
|
@ -774,6 +878,9 @@ export function TraceRoute() {
|
|||
fieldEvidence={fieldQuery.data}
|
||||
fieldLoading={fieldQuery.isLoading}
|
||||
fieldError={fieldQuery.isError ? fieldQuery.error : null}
|
||||
bundle={bundleQuery.data}
|
||||
bundleLoading={bundleQuery.isLoading}
|
||||
bundleError={bundleQuery.isError ? bundleQuery.error : null}
|
||||
/>
|
||||
) : null}
|
||||
</section>
|
||||
|
|
|
|||
|
|
@ -114,6 +114,32 @@ export interface FieldEvidence {
|
|||
transition_inner_product: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* D3 shareable evidence bundle — a turn's deterministic evidence exported as one
|
||||
* content-addressed, citable artifact. `bundle_digest` content-addresses the
|
||||
* cognitive evidence only (journal position + wall-clock are carried but
|
||||
* excluded from the digest), so the same turn reproduces the same digest.
|
||||
*/
|
||||
export interface EvidenceBundle {
|
||||
schema_version: "evidence_bundle_v1";
|
||||
turn_id: number;
|
||||
generated_from: "turn_journal";
|
||||
trace_hash: string | null;
|
||||
trace_integrity: TraceIntegrity;
|
||||
prompt: string;
|
||||
surface: string;
|
||||
grounding_source: GroundingSource;
|
||||
epistemic_state: EpistemicState;
|
||||
normative_clearance: NormativeClearance;
|
||||
refusal_emitted: boolean;
|
||||
journal_digest: string;
|
||||
pipeline_record: CognitivePipelineRecord | null;
|
||||
field_evidence: FieldEvidence | null;
|
||||
leeway_evidence: LeewayEvidence | null;
|
||||
replay_reproducer: string;
|
||||
bundle_digest: string;
|
||||
}
|
||||
|
||||
export type LeewayLicense = "PROPOSE" | "SERVE" | "blocked" | "unknown";
|
||||
export type ClaimDisclosure = "approximate" | "verified" | "proposal_only" | "none";
|
||||
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ from core.epistemic_state import (
|
|||
)
|
||||
from workbench import calibration, readers
|
||||
from workbench.journal import DEFAULT_JOURNAL_DIR, TurnJournal, TurnJournalEntry
|
||||
from workbench.evidence_bundle import build_evidence_bundle
|
||||
from workbench.field_evidence import (
|
||||
field_evidence_from_journal_entry,
|
||||
field_evidence_from_result,
|
||||
|
|
@ -360,6 +361,24 @@ class WorkbenchApi:
|
|||
404, error("not_found", f"trace field not found: {turn_id}")
|
||||
)
|
||||
return ApiResponse(200, ok(field_evidence_from_journal_entry(entry)))
|
||||
if method == "GET" and path.startswith("/trace/") and path.endswith("/bundle"):
|
||||
raw_turn_id = unquote(
|
||||
path.removeprefix("/trace/").removesuffix("/bundle").strip("/")
|
||||
)
|
||||
try:
|
||||
turn_id = int(raw_turn_id)
|
||||
except ValueError:
|
||||
return ApiResponse(
|
||||
404,
|
||||
error("not_found", f"trace bundle not found: {raw_turn_id}"),
|
||||
)
|
||||
try:
|
||||
entry = self._journal.get_entry(turn_id)
|
||||
except FileNotFoundError:
|
||||
return ApiResponse(
|
||||
404, error("not_found", f"trace bundle not found: {turn_id}")
|
||||
)
|
||||
return ApiResponse(200, ok(build_evidence_bundle(entry)))
|
||||
if method == "GET" and path.startswith("/trace/"):
|
||||
raw_turn_id = unquote(path.removeprefix("/trace/"))
|
||||
try:
|
||||
|
|
|
|||
69
workbench/evidence_bundle.py
Normal file
69
workbench/evidence_bundle.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
"""D3 shareable evidence bundle — reproducibility as a deliverable.
|
||||
|
||||
A turn's evidence, exported as ONE deterministic, content-addressed, citable
|
||||
artifact. The ``bundle_digest`` content-addresses the turn's deterministic
|
||||
cognitive evidence (prompt, surface, trace_hash, grounding / epistemic /
|
||||
normative state, the Phase-C pipeline + field evidence, and the calibration
|
||||
leeway verdict). Journal-position and wall-clock metadata (``turn_id``,
|
||||
``journal_digest``, the reproducer string) are carried for provenance but
|
||||
EXCLUDED from the digest, so the same turn content always yields the same
|
||||
digest — a reviewer can re-run the prompt over a sealed runtime, confirm the
|
||||
trace_hash, recompute the bundle, and check the digest.
|
||||
|
||||
Read-only: a pure projection of an already-persisted journal entry. No engine
|
||||
execution, no mutation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from dataclasses import replace
|
||||
from typing import Any
|
||||
|
||||
from workbench.schemas import EvidenceBundle, to_data
|
||||
|
||||
# Provenance / position / wall-clock fields: carried on the bundle but NOT part
|
||||
# of the content address, so the digest identifies the cognitive evidence
|
||||
# rather than where it happened to land in a journal.
|
||||
_DIGEST_EXCLUDED: frozenset[str] = frozenset(
|
||||
{"turn_id", "generated_from", "journal_digest", "replay_reproducer", "bundle_digest"}
|
||||
)
|
||||
|
||||
|
||||
def _bundle_digest(bundle: EvidenceBundle) -> str:
|
||||
payload = to_data(bundle)
|
||||
for key in _DIGEST_EXCLUDED:
|
||||
payload.pop(key, None)
|
||||
canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
|
||||
return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def build_evidence_bundle(entry: Any) -> EvidenceBundle:
|
||||
"""Assemble a turn journal entry into a content-addressed evidence bundle."""
|
||||
|
||||
trace_hash = entry.trace_hash
|
||||
reproducer = (
|
||||
f"core replay turn {entry.turn_id} "
|
||||
f"# re-run sealed; expect trace_hash == {trace_hash or 'none'}"
|
||||
)
|
||||
bundle = EvidenceBundle(
|
||||
schema_version="evidence_bundle_v1",
|
||||
turn_id=entry.turn_id,
|
||||
generated_from="turn_journal",
|
||||
trace_hash=trace_hash,
|
||||
trace_integrity=entry.trace_integrity or "legacy_unhashed",
|
||||
prompt=entry.prompt,
|
||||
surface=entry.surface,
|
||||
grounding_source=entry.grounding_source,
|
||||
epistemic_state=entry.epistemic_state,
|
||||
normative_clearance=entry.normative_clearance,
|
||||
refusal_emitted=entry.refusal_emitted,
|
||||
journal_digest=entry.journal_digest,
|
||||
pipeline_record=to_data(entry.pipeline_record),
|
||||
field_evidence=to_data(entry.field_evidence),
|
||||
leeway_evidence=to_data(entry.leeway_evidence),
|
||||
replay_reproducer=reproducer,
|
||||
bundle_digest="", # filled below; excluded from its own computation
|
||||
)
|
||||
return replace(bundle, bundle_digest=_bundle_digest(bundle))
|
||||
|
|
@ -170,6 +170,39 @@ class FieldEvidence:
|
|||
transition_inner_product: float | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EvidenceBundle:
|
||||
"""D3 shareable evidence bundle — a turn's deterministic evidence, citable.
|
||||
|
||||
Reproducibility as a deliverable: a content-addressed export of the
|
||||
DETERMINISTIC subset of a turn (the wall-clock fields — timestamp,
|
||||
turn_cost_ms — are deliberately omitted) so the same turn always yields the
|
||||
same bytes and the same ``bundle_digest``. Anyone can re-run the prompt
|
||||
over a sealed runtime and check the trace_hash, then recompute the bundle
|
||||
and check ``bundle_digest`` — the bundle is the citable claim, the
|
||||
reproducer is how to verify it. It composes the Phase-C evidence
|
||||
(pipeline + field) with the trace and the calibration leeway verdict.
|
||||
"""
|
||||
|
||||
schema_version: Literal["evidence_bundle_v1"]
|
||||
turn_id: int
|
||||
generated_from: Literal["turn_journal"]
|
||||
trace_hash: str | None
|
||||
trace_integrity: TraceIntegrity
|
||||
prompt: str
|
||||
surface: str
|
||||
grounding_source: GroundingSource
|
||||
epistemic_state: EpistemicStateValue
|
||||
normative_clearance: NormativeClearanceValue
|
||||
refusal_emitted: bool
|
||||
journal_digest: str
|
||||
pipeline_record: CognitivePipelineRecord | None
|
||||
field_evidence: FieldEvidence | None
|
||||
leeway_evidence: LeewayEvidence | None
|
||||
replay_reproducer: str
|
||||
bundle_digest: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ChatTurnResult:
|
||||
prompt: str
|
||||
|
|
|
|||
Loading…
Reference in a new issue