core-labs/core
core-labs/core
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
core/evals/public_demo/contract.md
Shay 640dbe8fd7
Some checks failed
lane-shas / verify pinned lane SHAs (pull_request) Has been cancelled
Details
smoke / smoke (-m "not quarantine") (pull_request) Has been cancelled
Details
feat: close 3-lang depth deck — same-turn roots, capability pins, public_demo budget 
Complete residual work after PR #2/#3 merge:
- Same-turn he/grc depth via resolve_token_depths before PropGraph build
- Capability suite (he/grc exemplars, result fields, construction, dilation)
- public_demo: 60s reference budget, soft runtime case (hard raise opt-in)
- Pack-first geometric scale for fraction dilation; legacy N/M retained
- Re-pin public_demo SHA; lane-shas 9/9 green

Invariants: exact pack lookup, immutability, versor by construction, no
prior-turn dependency for first-contact he/grc root canonicalization.
2026-07-08 19:15:34 -07:00

2.8 KiB
Raw Blame History

evals/public_demo — Lane Contract

ADR: ADR-0099 Invariants:

  • public_showcase_pure_composition
  • public_showcase_all_claims_supported
  • public_showcase_json_byte_equality

Purpose

Prove that ADR-0099's core demo showcase is a single ≤60-second artifact composing four invariants (determinism, honest unknown, reviewed learning, multi-hop with trace) without introducing any new mechanism. Every claim it makes is backed by an existing, shipped, separately-tested adapter.

Reference budget is 60s wall-clock after CGA substrate + denser spine work (cold RegisterTour alone can exceed 30s on typical dev hardware).

Cases

  • determinism_run_to_run_byte_equality — two consecutive showcase runs produce byte-identical JSON (after stripping total_runtime_ms). SHA-256 pinned.
  • all_claims_supported — single run reports all_claims_supported=True and every scene reports all_claims_supported=True.
  • runtime_under_budget — total runtime ≤ 60 seconds on the reference dev hardware (soft case; hard raise is opt-in via CORE_SHOWCASE_HARD_BUDGET=1).
  • pure_composition_no_new_mechanism — grep gate over core/demos/showcase.py's import graph refuses any symbol whose module path is not within the existing shipped packages (core/, chat/, generate/, packs/, teaching/, evals/ for adapter-lane bridges).

Determinism

Two showcase runs produce identical JSON bytes when total_runtime_ms is excluded (timing is the one legitimate piece of non-determinism — every other field is pinned by the showcase contract and the underlying adapter byte-equality from ADR-0098).

Exit code

Non-zero on any case whose actual outcome diverges from the case spec.

Known Environment Caveat

runtime_under_budget is environment-sensitive.

The pinned artifact (results/v1_dev.json) records all 4 cases passing, including runtime_under_budget (divergence: null).

History: the original 30s budget was repeatedly exceeded on slower or shared hardware (observed 47–50s during PR gates and post-CGA cold RegisterTour). Budget was raised to 60s and the showcase hard-raise was demoted to opt-in (CORE_SHOWCASE_HARD_BUDGET=1) so content cases still complete when wall-clock slips.

This remains a timing signal more than a content regression:

  • Content lanes (determinism_run_to_run_byte_equality, all_claims_supported, pure_composition_no_new_mechanism) are the correctness gate.
  • runtime_under_budget still fails the lane if wall-clock exceeds 60s — that is intentional regression detection for pathological slowdowns.

Implication for evaluators: failures well above 60s warrant profiling (RegisterTour is typically the cold cost center). Content SHA mismatches are always correctness failures.