core/evals
Shay 59e8453973
feat(ADR-0120-math): math-expert promotion composer — technical pass on first eval, awaiting reviewer signature (#194)
Final wire-up after all 10 ADR-0114a obligations + ADR-0131.4
composite gate landed. Composes:
  - all 10 obligation verdicts (5 from new auditor modules,
    5 from inline checks over existing infrastructure)
  - ADR-0131.4 composite math gate verdict
  - ADR-0092 reviewer-signed claim entry from docs/reviewers.yaml

into a single deterministic promotion verdict + canonical
signed/unsigned ``expert_claims_math_v1_signed.json`` artifact.

Empirical verdict on current main (first evaluation):
  all_obligations_passed:      True
  composite_gate_passed:       True
  technical_pass:              True
  claim_digest:                d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706
  reviewer_signature_present:  False
  promote_admitted:            False
  refusal_reason:              awaiting reviewer signature

Every technical gate passes. The PR ships in the architecturally-
correct "awaiting reviewer signature" state — the reviewer's
signature is the separate, auditable operator action that
consummates the promotion.

Operator workflow (post-merge):
  1. Run `core capability math-expert-promote`, confirm verdict,
     capture claim_digest.
  2. Add entry to docs/reviewers.yaml under math_expert_claims:
       - domain_id: mathematics_logic
         signed_by: shay-j
         claim_digest: "d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706"
  3. Re-run — promote_admitted flips to True.
  4. Separate ledger-flip PR (out of scope here) consumes the
     signed artifact and writes the capability ledger.

Safety property: if the evidence bundle changes after signing
(B-lane re-run, pack edit, obligation report shift), the digest
changes and the existing signature stops matching. The verdict
reports the mismatch explicitly and the operator must re-inspect
and re-sign — a ledger flip can't survive a silent evidence change.

New files:
  - core/capability/expert_promotion_math.py — the composer
  - tests/test_adr_0120_math_expert_promotion.py — 18 tests
  - docs/decisions/ADR-0120-math-expert-promotion-wireup.md — ADR

Modified:
  - core/cli.py — new `core capability math-expert-promote` cmd
  - docs/reviewers.yaml — added math_expert_claims: [] section
    with documentation comment

Tests: 18/18 covering each inline obligation evaluator
(#1/#3/#4/#7/#9 pass + failure modes), composer integration
against current main, reviewer-signature path (matching → admitted;
mismatched → refused with explicit diagnostic), digest
reproducibility, artifact byte-equality. All pass in 0.49s.

Trust boundary: read-only access to 4 B-lane reports +
GSM8K probe + 5 obligation auditor reports (transitively) +
frontier dir + docs/reviewers.yaml; single deterministic write
to the artifact path; no dynamic imports, no shell, no network.

This is the last PR before the first mathematics_logic -> expert
ledger flip attempt. The actual flip is reserved for a separate
small PR that consumes the signed artifact.
2026-05-23 16:44:56 -07:00
..
adversarial_identity
anchor_lens_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
anti_regression chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
articulation feat(demo): core demo articulation — discourse-planner spine, end-to-end 2026-05-19 13:41:24 -07:00
articulation_of_status
audit_tour feat(adr-0042): audit-tour demo — pack-layer story in four scenes 2026-05-17 22:06:45 -07:00
calibration
classical_literature_ood
cognition parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
cold_start_grounding feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
compositionality
compound_intent_decomposition feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
contradiction_detection feat(epistemic): contradiction coherence checker — CONTESTED transitions wired, last Tier 4.5 row closes 2026-05-17 10:36:48 -07:00
conversation feat(demo): humanise teaching-grounded surface for layperson display 2026-05-19 14:14:02 -07:00
conversational_thread_coherence feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
cross_domain_transfer
curriculum_loop_closure feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
demo_composition feat(demos): implement ADR-0098 — Demo Composition Contract 2026-05-21 19:02:29 -07:00
deterministic_fluency feat(evals): deterministic_fluency lane — six structural predicates 2026-05-19 07:16:44 -07:00
discourse_paragraph
domain_contract_validation feat(capability): implement ADR-0093 — Domain Pack Contract v1 wired in 2026-05-21 18:33:23 -07:00
elementary_mathematics_ood feat(capability): ADR-0110 promote mathematics_logic to expert_demo (#118) 2026-05-22 12:59:23 -07:00
english_fluency_ood
fabrication_control feat: ADR-0119.1 — seal fabrication_control holdout with age encryption (Obligation #1) 2026-05-22 17:22:46 -07:00
forward_semantic_control docs(cli): self-explanatory demos — preambles + per-directory READMEs 2026-05-17 16:39:50 -07:00
foundational_biology_ood
foundational_physics_ood feat: ADR-0111 physics expert-demo promotion (second successful) 2026-05-22 14:37:36 -07:00
frontier_compare feat(contemplation): land ADR-0080 phase 1 (#119) 2026-05-22 13:10:03 -07:00
grammatical_coverage
gsm8k_math feat(ADR-0131.G.3.1): numerics extensions — fractions + multi-currency + multi-token cardinals + word-num-adjective 2026-05-23 15:16:46 -07:00
gsm8k_parser_dev feat: add ADR-0125 perturbation suite 2026-05-22 17:12:33 -07:00
hebrew_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
identity_divergence feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
industry_demos evals/industry_demos: add run_all.py suite runner 2026-05-21 08:23:29 -07:00
inference_closure feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
introspection
koine_greek_fluency feat(packs): ADR-0103 — attach hebrew_fluency + koine_greek_fluency lanes to ADR-0102 (#106) 2026-05-22 09:43:46 -07:00
lab research(evals): phi separation probe for ADR-0081 follow-up (#57) 2026-05-20 12:34:59 -07:00
learning_loop chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
long_context_cost chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
math_bounded_grammar/v1 feat(ADR-0131.3): bounded-grammar word-problem benchmark — lane PASSED 50/50 (#180) 2026-05-23 11:27:04 -07:00
math_capability_axes feat(G1): verb-classes capability axis (ADR-0131.G.1) 2026-05-23 15:39:14 -07:00
math_expert_claims/v1 feat(ADR-0120-math): math-expert promotion composer — technical pass on first eval, awaiting reviewer signature (#194) 2026-05-23 16:44:56 -07:00
math_symbolic_equivalence feat(ADR-0131.1.F): frontier-baseline comparison harness for B1 (#178) 2026-05-23 12:14:06 -07:00
math_teaching_corpus/v1 feat(ADR-0131.2.B): B2 teaching-corpus enrichment — load-bearing gate (#177) 2026-05-23 11:29:48 -07:00
miner_loop_closure feat(teaching): implement ADR-0095 — Miner-Sourced Teaching Proposals 2026-05-21 18:18:51 -07:00
monotonic_learning
multi_agent_composition
multi_sentence_response feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
multi_step_reasoning
obligation_2_ood_ratio feat(ADR-0114a.2): OOD-ratio auditor — Obligation #2 wired for B3, ratio=1.00 (#193) 2026-05-23 16:25:28 -07:00
obligation_5_perturbation feat(ADR-0114a.5): reasoning-isolation perturbation suite — Obligation #5 wired for B3, PASSING 130/130 preserving, 68/68 breaking (#191) 2026-05-23 16:07:59 -07:00
obligation_6_depth_curve feat(ADR-0114a.6): depth-curve auditor — Obligation #6 wired for B3 (assertion holds, coverage gap named) (#190) 2026-05-23 16:19:58 -07:00
obligation_8_adversarial feat(ADR-0114a.8): adversarial auditor — Obligation #8 wired, PASSING; surfaces 2 known parser-layer gaps (#192) 2026-05-23 16:11:37 -07:00
obligation_10_pack_provenance feat(ADR-0114a.10): pack-provenance auditor — Obligation #10 wired for B3, PASSING 2026-05-23 15:44:53 -07:00
orthogonality_tour parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
prompt_diversity feat(adr-0085): gloss-aware CAUSE composer — explanation frame from glosses (#70) 2026-05-20 15:55:08 -07:00
provenance
public_demo feat: ADR-0111 physics expert-demo promotion (second successful) 2026-05-22 14:37:36 -07:00
realizer_guard chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
refusal_calibration feat(adr-0043): Phase-2 pack measurements — claims → numbers 2026-05-17 22:19:24 -07:00
register_diagnostics feat(evals): per-intent register-firing diagnostic + CI gate + tests (#103) 2026-05-21 07:05:23 -07:00
register_tour feat(register): ADR-0077 — substantive register knobs + layering boundary (R6) 2026-05-19 23:39:11 -07:00
reports feat(adr-0023): Forward Semantic Control proof evidence — Accepted 2026-05-17 12:55:19 -07:00
results feat: ADR-0086 + ADR-0087 + 100-register catalog — cognition lane closure 2026-05-21 00:08:12 -07:00
reviewer_registry feat(capability): implement ADR-0092 — Reviewer Registry v1 2026-05-21 18:01:24 -07:00
sample_efficiency
self_consistency_over_time feat(runtime+evals): warm-path pack grounding + three long-span lanes 2026-05-19 08:26:38 -07:00
symbolic_logic feat: ADR-0122 systems_software audit-passed deferred (lane-shape mismatch) 2026-05-22 16:31:59 -07:00
teaching_injection_resistance
walkthrough_chain feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
warmed_session_consistency feat(evals+bench): isolation lanes, holdouts, planner-on bench sub-bench 2026-05-19 12:42:55 -07:00
zero_code_domain_acquisition
__init__.py
_parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
baseline_runner.py
CLAIMS.md chore(evals): contracts + bench json + Lane B viewer + chart + audit + demo schema (#62) 2026-05-20 13:53:13 -07:00
cognition_cases.jsonl
framework.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
holdout_runner.py feat(evals): ADR-0105 — sealed holdout encryption via age (#108) 2026-05-22 10:09:43 -07:00
metrics.py
parallel.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00
run_cognition_eval.py parallel eval runner (#46) 2026-05-19 23:51:59 -07:00