Shay bfb54fb015 feat(demos): implement ADR-0099 — Public Showcase Demo

Single 30-second artifact composing four CORE invariants
(determinism, honest unknown, reviewed learning, multi-hop with
trace) by delegating to existing DemoCommand adapters. **No new
mechanism** — every claim is backed by an already-shipped,
separately-tested adapter. Closes the 8-ADR scale-up slate.

- new core/demos/learning_loop_adapter.py: LearningLoopDemo wraps
  ADR-0056 reviewed-teaching loop; _strip_volatile_paths drops
  transient temp-dir paths from raw before serialization so the
  adapter's report_sha256 is content-stable across runs
- new core/demos/showcase_adapters.py:
  - FabricationControlPublicDemo: re-runs ADR-0096 public split,
    produces 3 claims (refusal_recall_meets_threshold,
    fabrication_rate_below_threshold, trace_evidence_present)
  - MultiHopTraceDemo: runs 'Does light reveal truth?' with
    transitive_surface=True + composed_surface=True against
    cognition pack; surfaces a 3-hop walk light→truth→knowledge→
    evidence; produces 3 claims (grounded_answer, depth_two_or_more,
    walk_evidence_present)
- new core/demos/showcase.py: run_showcase() composes 4 scenes,
  emits showcase.json + per-scene artifacts; render_html() produces
  presentation-only static HTML with no JS injection vector;
  ShowcaseScene dataclass; MAX_RUNTIME_SECONDS=30 hard ceiling
  with DemoContractError if exceeded
- CLI: 'showcase' added to demo target choices; --output-dir flag
  added; cmd_demo dispatch branch writes showcase.json + showcase.html
- new evals/public_demo/ lane with 4 cases:
  - all_claims_supported (each scene + composite)
  - determinism_run_to_run_byte_equality (two runs identical after
    stripping volatile keys: total_runtime_ms, json_path,
    transient_corpus)
  - runtime_under_budget (≤30s)
  - pure_composition_no_new_mechanism (grep gate over showcase
    imports — must come from core/chat/generate/language_packs/
    teaching/evals or allowed stdlib only)
- lane is itself byte-identical across runs (sha256 5707db8efc6a..);
  runtime case omits exact runtime_ms (it varies near bucket
  boundaries) but still asserts ≤ budget
- 8 unit tests with module-scoped fixture (showcase runs once,
  ~13s total) covering payload shape, scene order, runtime budget,
  HTML render absence of <script>, and the pure-composition import
  gate independently of the lane
- ADR-0099 measured: total_runtime_ms ~12.8s, well under 30s budget
- smoke 67/67, cognition eval byte-identical 100/100/100/100;
  all 6 ADR-0092..0099 lanes byte-identical:
    reviewer_registry        681a2aab..
    miner_loop_closure       9f071733..
    domain_contract_validation f9c06cde..
    fabrication_control sum  01e1b6b7..
    demo_composition         27d83824..
    public_demo              5707db8e..

2026-05-21 19:44:48 -07:00

1.5 KiB

Raw Permalink Blame History

evals/public_demo — Lane Contract

ADR: ADR-0099 Invariants:

public_showcase_pure_composition
public_showcase_all_claims_supported
public_showcase_json_byte_equality

Purpose

Prove that ADR-0099's core demo showcase is a single 30-second artifact composing four invariants (determinism, honest unknown, reviewed learning, multi-hop with trace) without introducing any new mechanism. Every claim it makes is backed by an existing, shipped, separately-tested adapter.

Cases

determinism_run_to_run_byte_equality — two consecutive showcase runs produce byte-identical JSON (after stripping total_runtime_ms). SHA-256 pinned.
all_claims_supported — single run reports all_claims_supported=True and every scene reports all_claims_supported=True.
runtime_under_budget — total runtime ≤ 30 seconds on the reference dev hardware.
pure_composition_no_new_mechanism — grep gate over core/demos/showcase.py's import graph refuses any symbol whose module path is not within the existing shipped packages (core/, chat/, generate/, language_packs/, teaching/, evals/ for adapter-lane bridges).

Determinism

Two showcase runs produce identical JSON bytes when total_runtime_ms is excluded (timing is the one legitimate piece of non-determinism — every other field is pinned by the showcase contract and the underlying adapter byte-equality from ADR-0098).

Exit code

Non-zero on any case whose actual outcome diverges from the case spec.

1.5 KiB Raw Permalink Blame History