core/evals
Shay 4f640af40d feat(demos): implement ADR-0098 — Demo Composition Contract
DemoCommand Protocol + thin adapters retrofit shipped tours to a
typed composition contract. Composability becomes a structural
property: the ADR-0099 showcase will consume DemoResult through one
stable type rather than special-casing each tour. No demo behavior
changes — adapters wrap underlying run_tour() entry points.

- new core/demos/ package:
  - contract.py: frozen Claim / DemoResult dataclasses, runtime-checkable
    DemoCommand Protocol, canonical_json() sanctioned serializer
    (sorted keys, 2-space indent, trailing newline), CLAIM_CONTRACT_VERSION
  - audit_tour_adapter.py: AuditTourDemo (5 claims from ADR-0042 scenes
    1-4: identity_pack_swaps_visible, safety_typed_refusal,
    ethics_opt_in_deployment_fires, ethics_default_silent,
    replay_byte_identical)
  - tour_adapters.py: shared pattern for register/anchor-lens/orthogonality
    tours; _extract_claims walks the dict tree for *_supported booleans
    and builds Claim objects in deterministic sorted order

- global-state-mutation detector (ADR-0098 invariant #2):
  capture_state() snapshots a load-bearing subset of process state
  (CORE_* env vars + module identities for chat.telemetry,
  chat.runtime, language_packs.compiler);
  verify_no_global_state_mutation() ignores None→id transitions
  (benign lazy import) and only flags env-var changes or module
  identity rebindings

- new evals/demo_composition/ lane (ADR-0098 invariant proving):
  - 6 cases asserting byte-equality + no-state-mutation across the
    three fast adapters (audit-tour, register-tour, orthogonality-tour)
  - composition_read_only: confirms two adapter results compose into
    a composite claim set without mutating either
  - stateful_fixture_rejected: negative control — a deliberately
    stateful adapter MUST trigger divergence detection
  - anchor-lens-tour adapter is exercised by tests, not the lane,
    to keep wall time bounded
  - byte-identical across runs (sha256 27d838241bf3..)

- 26 unit tests covering Claim/DemoResult validation, canonical_json
  determinism, state-mutation detector (including the lazy-import
  benign case), Protocol conformance (isinstance check + claim
  contract version) for all four adapters, seed-rejection per
  adapter (all current adapters are fully deterministic), and an
  audit-tour integration smoke verifying 5 claims + byte-equality +
  no state mutation across two consecutive runs

- smoke 67/67, cognition eval byte-identical 100/100/100/100, all
  five lanes byte-identical (reviewer_registry 681a2aab..,
  miner_loop_closure 9f071733.., domain_contract_validation f9c06cde..,
  fabrication_control summary 01e1b6b7.., demo_composition 27d83824..)
2026-05-21 19:02:29 -07:00
..
adversarial_identity
anchor_lens_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
anti_regression chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status
audit_tour
calibration
classical_literature_ood
cognition parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
cold_start_grounding feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
compositionality
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
contradiction_detection
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer
demo_composition feat(demos): implement ADR-0098 — Demo Composition Contract 2026-05-21 19:02:29 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
discourse_paragraph
domain_contract_validation feat(capability): implement ADR-0093 — Domain Pack Contract v1 wired in 2026-05-21 18:33:23 -07:00
elementary_mathematics_ood
english_fluency_ood
fabrication_control feat(evals): implement ADR-0096 — Fabrication-Control Eval Lane 2026-05-21 18:44:25 -07:00
forward_semantic_control
foundational_biology_ood
foundational_physics_ood
frontier_compare feat(frontier): add replay variability suite and token-cost telemetry (#66) 2026-05-20 15:04:34 -07:00
grammatical_coverage
hebrew_fluency
identity_divergence
industry_demos evals/industry_demos: add run_all.py suite runner 2026-05-21 08:23:29 -07:00
inference_closure
introspection
koine_greek_fluency
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_loop chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
miner_loop_closure feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
monotonic_learning
multi_agent_composition
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning
orthogonality_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
provenance
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration
register_diagnostics feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
register_tour feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
reports
results feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
reviewer_registry feat(capability): implement ADR-0092 — Reviewer Registry v1 2026-05-21 18:01:24 -07:00
sample_efficiency
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
symbolic_logic
teaching_injection_resistance
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition
__init__.py
_parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
baseline_runner.py
CLAIMS.md chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
cognition_cases.jsonl
framework.py fix(evals): introspect run_lane signature before passing workers kwarg 2026-05-20 05:59:51 -07:00
holdout_runner.py
metrics.py
parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
run_cognition_eval.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00