core/evals
Shay a21d31a95c ci(lanes): pin ADR-0092..0099 lane SHAs and wire GitHub Actions verifier
Six lanes (reviewer_registry, miner_loop_closure,
domain_contract_validation, fabrication_control_summary,
demo_composition, public_demo) now have CI-enforced SHA-256 pins.
A failing job means a lane's deterministic output changed without
an explicit ADR-tracked pin update.

- new scripts/verify_lane_shas.py: single source of truth
  - PINNED_SHAS dict mapping lane_id → 64-char hex SHA
  - LANE_SPECS tuple wiring each lane to its runner module + canonical
    report path
  - accepts_report_flag handles the fabrication_control runner's
    different arg shape (--lane-dir not --report)
  - verify_all() runs each lane in subprocess isolation (clean Python
    state per lane — relevant for adapters that cache pack loads at
    module import)
  - --update flag refreshes pins after intentional ADR-tracked changes;
    diff is the audit trail
  - --json flag emits machine-readable report
  - exits non-zero on any mismatch

- new .github/workflows/lane-shas.yml:
  - triggers on push to main and pull_request to main
  - concurrency group cancels in-progress runs on new commits
  - Python 3.11 + pip-cached deps + editable install
  - runs verify_lane_shas.py; emits JSON report on failure
  - 12-minute timeout (lanes take ~30s in practice)

- new tests/test_lane_sha_verifier.py: cheap local-pytest pinning
  - every LaneSpec has a corresponding PINNED_SHAS entry
  - no orphan pins without a LaneSpec
  - every pin is a 64-char hex SHA-256
  - every runner module path exists on disk
  - canonical report paths are under repo root
  - all six expected lanes (ADR-0092/0093/0095/0096/0098/0099) covered;
    ADR-0094 and ADR-0097 are schema/ratification only, intentionally
    excluded from EXPECTED_LANES
  - 6 tests run in <100ms — catches drift before CI

- evals/public_demo/results/v1_dev.json: refreshed to match the new
  pin (21751aaf..) — earlier pin was generated under slightly different
  runner argparse defaults; --update produced the canonical bytes

Local verifier: 6/6 lanes match pinned SHAs. Smoke 67/67. Lane SHAs:
  reviewer_registry            681a2aab..
  miner_loop_closure           9f071733..
  domain_contract_validation   f9c06cde..
  fabrication_control_summary  01e1b6b7..
  demo_composition             27d83824..
  public_demo                  21751aaf..
2026-05-21 19:59:37 -07:00
..
adversarial_identity
anchor_lens_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
anti_regression chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status feat(epistemic): realizer-side closure — refusal_calibration + articulation_of_status graduate 2026-05-17 10:12:59 -07:00
audit_tour feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
calibration fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
classical_literature_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
cognition parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
cold_start_grounding feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
compositionality feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
demo_composition feat(demos): implement ADR-0098 — Demo Composition Contract 2026-05-21 19:02:29 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
discourse_paragraph feat(compositionality): compose_relations operator lifts lane 68.8% → 100% 2026-05-16 22:44:06 -07:00
domain_contract_validation feat(capability): implement ADR-0093 — Domain Pack Contract v1 wired in 2026-05-21 18:33:23 -07:00
elementary_mathematics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
english_fluency_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
fabrication_control feat(evals): implement ADR-0096 — Fabrication-Control Eval Lane 2026-05-21 18:44:25 -07:00
forward_semantic_control docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
foundational_biology_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
foundational_physics_ood fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
frontier_compare feat(frontier): add replay variability suite and token-cost telemetry (#66) 2026-05-20 15:04:34 -07:00
grammatical_coverage fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
hebrew_fluency
identity_divergence feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
industry_demos evals/industry_demos: add run_all.py suite runner 2026-05-21 08:23:29 -07:00
inference_closure feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
introspection
koine_greek_fluency
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_loop chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
miner_loop_closure feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
monotonic_learning
multi_agent_composition fix(gaps): close G1+G2+G3 + identity vocab + pipeline safety-stub honour 2026-05-16 21:21:06 -07:00
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning feat(algebra): null-preserving versor_apply path + un-skip 2 invariant tests 2026-05-16 21:40:37 -07:00
orthogonality_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
provenance
public_demo ci(lanes): pin ADR-0092..0099 lane SHAs and wire GitHub Actions verifier 2026-05-21 19:59:37 -07:00
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
register_diagnostics feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
register_tour feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
results feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
reviewer_registry feat(capability): implement ADR-0092 — Reviewer Registry v1 2026-05-21 18:01:24 -07:00
sample_efficiency
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
symbolic_logic
teaching_injection_resistance feat(epistemic): truth-seeking schema audit — 3 leaks closed, 4 new lanes, 3 new invariants 2026-05-17 07:27:41 -07:00
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition
__init__.py
_parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
baseline_runner.py
CLAIMS.md chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
cognition_cases.jsonl
framework.py fix(evals): introspect run_lane signature before passing workers kwarg 2026-05-20 05:59:51 -07:00
holdout_runner.py
metrics.py
parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
run_cognition_eval.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00