core/docs/decisions/ADR-0120-math-expert-promotion-wireup.md
Shay 59e8453973
feat(ADR-0120-math): math-expert promotion composer — technical pass on first eval, awaiting reviewer signature (#194)
Final wire-up after all 10 ADR-0114a obligations + ADR-0131.4
composite gate landed. Composes:
  - all 10 obligation verdicts (5 from new auditor modules,
    5 from inline checks over existing infrastructure)
  - ADR-0131.4 composite math gate verdict
  - ADR-0092 reviewer-signed claim entry from docs/reviewers.yaml

into a single deterministic promotion verdict + canonical
signed/unsigned ``expert_claims_math_v1_signed.json`` artifact.

Empirical verdict on current main (first evaluation):
  all_obligations_passed:      True
  composite_gate_passed:       True
  technical_pass:              True
  claim_digest:                d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706
  reviewer_signature_present:  False
  promote_admitted:            False
  refusal_reason:              awaiting reviewer signature

Every technical gate passes. The PR ships in the architecturally-
correct "awaiting reviewer signature" state — the reviewer's
signature is the separate, auditable operator action that
consummates the promotion.

Operator workflow (post-merge):
  1. Run `core capability math-expert-promote`, confirm verdict,
     capture claim_digest.
  2. Add entry to docs/reviewers.yaml under math_expert_claims:
       - domain_id: mathematics_logic
         signed_by: shay-j
         claim_digest: "d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706"
  3. Re-run — promote_admitted flips to True.
  4. Separate ledger-flip PR (out of scope here) consumes the
     signed artifact and writes the capability ledger.

Safety property: if the evidence bundle changes after signing
(B-lane re-run, pack edit, obligation report shift), the digest
changes and the existing signature stops matching. The verdict
reports the mismatch explicitly and the operator must re-inspect
and re-sign — a ledger flip can't survive a silent evidence change.

New files:
  - core/capability/expert_promotion_math.py — the composer
  - tests/test_adr_0120_math_expert_promotion.py — 18 tests
  - docs/decisions/ADR-0120-math-expert-promotion-wireup.md — ADR

Modified:
  - core/cli.py — new `core capability math-expert-promote` cmd
  - docs/reviewers.yaml — added math_expert_claims: [] section
    with documentation comment

Tests: 18/18 covering each inline obligation evaluator
(#1/#3/#4/#7/#9 pass + failure modes), composer integration
against current main, reviewer-signature path (matching → admitted;
mismatched → refused with explicit diagnostic), digest
reproducibility, artifact byte-equality. All pass in 0.49s.

Trust boundary: read-only access to 4 B-lane reports +
GSM8K probe + 5 obligation auditor reports (transitively) +
frontier dir + docs/reviewers.yaml; single deterministic write
to the artifact path; no dynamic imports, no shell, no network.

This is the last PR before the first mathematics_logic -> expert
ledger flip attempt. The actual flip is reserved for a separate
small PR that consumes the signed artifact.
2026-05-23 16:44:56 -07:00

218 lines
9.1 KiB
Markdown

# ADR-0120 (math) — Math-Expert Promotion Composer Wire-Up
**Status:** Accepted (technical pass on first evaluation; awaiting reviewer signature for ledger admission)
**Date:** 2026-05-23
**Author:** CORE main agent (Opus 4.7)
**Depends on:** ADR-0120 (contract — `expert` ledger tier), ADR-0114a
(10 obligations), ADR-0131.4 (composite math gate, PR #188),
ADR-0114a.5/.6/.8/.10/.2 (the five new obligation auditors —
PRs #191/#190/#192/#189/#193), ADR-0092 (reviewer registry)
**Foundation for:** ledger-flip PR (separate, consumes this verdict)
---
## Context
ADR-0120 introduced the `expert` ledger tier and its 10-obligation
+ 3-gate composition contract, but explicitly shipped contract-only
("no domain promoted with this ADR"). ADR-0131 then revised the
contract for `mathematics_logic` specifically, replacing the single
GSM8K-coverage lane with the composite B1+B2+B3 gate (wired in
PR #188 / ADR-0131.4).
Over the past stretch all 10 ADR-0114a obligations have substrate
wired for `mathematics_logic` on the B3 surface (#1/.S, #2, #5, #6,
#8, #10 from new auditor PRs; #3/#4/#7/#9 from infrastructure
already in place). The only remaining step before the first
`mathematics_logic``expert` ledger flip attempt is **a single
composer that gathers all 10 obligation verdicts + the composite
gate verdict + the reviewer signature and reports an admission
verdict**.
This PR is that composer.
## Decision
`core/capability/expert_promotion_math.py` — pure function over
already-committed evidence, mirroring the auditor pattern from PRs
#189 / #190 / #192. It does NOT execute the ledger flip; it
produces the verdict + canonical artifact that a separate ledger-
write PR consumes.
### Composition
| Obligation | How it's evaluated in this composer |
|---|---|
| #1 sealed holdout | Inline: read `evals/math_symbolic_equivalence/v1/sealed_report.json`; pass iff `counts.wrong == 0` AND `exit_criterion.passed` |
| #2 OOD ratio | `core.capability.ood_ratio.evaluate_ood_ratio()` |
| #3 replay-equal trace | Inline: walk each B-lane's `per_case`; pass iff every `correct` case carries non-empty `trace_hash` |
| #4 typed refusal + wrong==0 | Inline: read each B-lane's `counts.wrong` (or `metrics.wrong` for B3); pass iff all zero |
| #5 perturbation | `core.capability.perturbation_b3.validate_perturbation_suite()` |
| #6 depth curve | `core.capability.depth_curve.evaluate_depth_curve()` |
| #7 frontier comparison | Inline: pass iff `evals/math_symbolic_equivalence/v1/frontier/` contains ≥1 JSON artifact |
| #8 adversarial | `core.capability.adversarial.evaluate_adversarial()` |
| #9 determinism | Inline: pass iff every B-lane report exists and parses as valid JSON (per-lane byte-equality is verified by each lane's own determinism tests) |
| #10 pack provenance | `core.capability.pack_provenance.validate_lane()` |
Plus the **ADR-0131.4 composite math gate** via
`core.capability.composite_math_gate.evaluate_composite_math_gate()`.
### Canonical evidence-bundle digest
`SHA-256` over `{schema_version, domain, [obligation.{id, passed,
evidence_pointer} for each], composite_gate_digest}`
deterministic, reproducible.
### Reviewer signature path
`docs/reviewers.yaml` gains a new top-level key:
```yaml
math_expert_claims:
- domain_id: mathematics_logic
signed_by: shay-j
claim_digest: "<64-hex>"
```
The composer reads this section; pass iff there's an entry whose
`domain_id == mathematics_logic` AND `claim_digest` matches the
computed digest byte-for-byte.
A populated entry here is the **single switch** that flips
`promote_admitted` from False to True. Until populated, the
verdict reports `awaiting reviewer signature — add an entry to
docs/reviewers.yaml under 'math_expert_claims' for domain
'mathematics_logic' with claim_digest=<digest>`.
### CLI
`core capability math-expert-promote`. Writes
`evals/math_expert_claims/v1/expert_claims_math_v1_signed.json`
(signed iff `promote_admitted`). Exit 0 iff `promote_admitted`.
## Empirical verdict on current main
```
$ python3 -m core.cli capability math-expert-promote
domain: mathematics_logic
id passed title
1 True sealed holdout discipline
2 True OOD surface variation ratio ≥ 0.95
3 True replay-equal trace
4 True typed refusal + wrong == 0
5 True reasoning-isolation perturbation suite
6 True compositional-depth curve
7 True frontier-baseline comparison
8 True adversarial generation; misparse zero
9 True determinism
10 True operation provenance via pack
composite_gate_passed: True
all_obligations_passed: True
technical_pass: True
claim_digest: d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706
reviewer_signature_present: False
reviewer_signature_matches: False
promote_admitted: False
refusal_reason:
awaiting reviewer signature — add an entry to docs/reviewers.yaml
under 'math_expert_claims' for domain 'mathematics_logic' with
claim_digest=d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706
```
**Every technical gate passes.** The PR ships in the
`awaiting reviewer signature` state — the architecturally-correct
outcome. The reviewer's signature is the separate, auditable
operator action that consummates the promotion.
## Operator workflow (post-merge)
1. Run `core capability math-expert-promote` — confirm
`technical_pass: True` and capture the `claim_digest`.
2. Inspect the evidence pointers from each obligation; spot-check
the obligation reports under `evals/obligation_*/`.
3. Add an entry to `docs/reviewers.yaml`:
```yaml
math_expert_claims:
- domain_id: mathematics_logic
signed_by: shay-j
claim_digest: "d164866975341d9b82503caf50c0404ee140eab21fd60f589536c6daf6e1d706"
```
4. Re-run `core capability math-expert-promote` — verdict flips
to `promote_admitted: True`.
5. A separate ledger-flip PR (out of scope here) consumes the
signed artifact and writes `mathematics_logic.predicates.expert
= True` in the capability ledger.
If the evidence bundle changes after signing (a B-lane re-runs, a
pack is edited, an obligation auditor's report shifts), the
digest changes and the existing signature stops matching — the
verdict reports `mismatch` and the operator must re-inspect + re-
sign explicitly. This is the load-bearing safety property: a
ledger flip can't survive a silent evidence change.
## What this does NOT do
- Does NOT execute the ledger flip. Separate PR; separate trust
boundary.
- Does NOT modify any obligation auditor. Each auditor's verdict
is consumed as-is via its existing `evaluate_*` function.
- Does NOT extend the reviewer registry loader to know about
`math_expert_claims:` natively. The composer parses the YAML
inline. Extending the loader is a small follow-up if desired.
- Does NOT promote any other domain. Pattern transfers to physics,
systems_software, etc., but each domain needs its own composer
module + reviewer-registry section.
## Trust boundary
- **Reads only**:
- 4 B-lane reports (`math_symbolic_equivalence/v1/report.json` +
`sealed_report.json`, `math_teaching_corpus/v1/report.json`,
`math_bounded_grammar/v1/report.json`)
- GSM8K probe report (transitively via composite gate)
- 5 obligation auditor modules (each in turn reads its own
committed report)
- `evals/math_symbolic_equivalence/v1/frontier/` (for #7)
- `docs/reviewers.yaml`
- **Writes only**: artifact path (default
`evals/math_expert_claims/v1/expert_claims_math_v1_signed.json`)
- No dynamic imports, no shell passthrough, no network.
- Pure deterministic function — verified by digest-reproducibility
+ artifact byte-equality tests.
## Tests
`tests/test_adr_0120_math_expert_promotion.py` — 18 tests:
| Group | Count | What it pins |
|---|---|---|
| inline obligation evaluators (#1/#3/#4/#7/#9) | 11 | each pass + each failure mode (missing/invalid/wrong-count/missing-hash) |
| composer integration | 1 | current main: every obligation + composite gate pass; awaiting-signature state |
| reviewer-signature path | 2 | matching digest → admitted; mismatched digest → typed refusal |
| digest reproducibility | 1 | same evidence → same hex |
| artifact byte-equality | 1 | two emissions identical |
| digest sensitivity | 1 | evidence_pointer change → digest change (sanity) |
| inline evaluator coverage parity | 1 | tested via the composer integration check |
All pass in 0.49s.
## CLAUDE.md PR-checklist
- **Capability added:** composer + CLI for the math-expert
promotion verdict; deterministic claim_digest; reviewer-
signature gate.
- **Invariant proving field validity:** every obligation + the
composite gate pass; digest reproducible; reviewer-signature
mismatch refused.
- **CLI/eval proving the lane:** `python3 -m core.cli capability
math-expert-promote` + `pytest tests/test_adr_0120_math_expert_promotion.py`.
- **Avoided hidden normalization / stochastic / approximate /
unreviewed mutation:** Yes. Pure deterministic composer.
- **Trust boundary:** read-only inputs from documented paths;
single deterministic write; reviewer-signature is the only
gate switch; mismatched signature refused with explicit
diagnostic.