docs: ADR-0120 — first expert promotion contract (proposed)

Defines the `expert` ledger tier (sixth status above `audit-passed`)
and the composition gate that governs every future promotion.

The gate composes ALL TEN ADR-0114a obligations plus three contract-
level checks:
  - audit_passed predicate must hold
  - correct_rate >= 0.60 on public AND sealed holdout splits
  - signed expert_claims entry whose digest reproduces byte-equal

Numeric thresholds (load-bearing choices documented with rationale):
  - correct_rate floor: 0.60 ("advanced" — above weak open-source LLMs,
                              forces real architecture work, raisable later)
  - depth-curve flatness ε: 0.05 (accuracy(N) ≥ accuracy(d1) · 0.95^(N-1))
  - drift tolerance from promotion: ±0.02

Documents the post-ADR-0120 sequence in Open Candidate Directions:
  Phase 1: ADR-0121 first worked math promotion (likely deferral)
  Phase 2: parser-expansion arc to lift sealed-GSM8K correct_rate
  Phase 3: math expert promotion succeeds
  Phase 4: second domain = symbolic_logic (60-70% of math substrate cost;
            ProofWriter / PrOntoQA benchmark; same machinery class)
  Phase 5: third domain = high-stakes refusal-centric (medical or legal;
            wedge-sharpener; needs two prior successes first)
  Phase 7: open candidate — multi-reviewer threshold signing for expert

No code lands with this ADR. Implementation ships under ADR-0120a
to keep the contract change reviewable independently.

The first promotion attempt (ADR-0121) will likely defer on the
correct_rate gate — current measurement is 0/1319 on real GSM8K.
That deferral IS the contract working as designed; same load-
bearing pattern ADR-0107 → ADR-0110 demonstrated for audit-passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Shay 2026-05-22 20:18:53 -07:00
parent 5c430582ac
commit 81a5ff6350
2 changed files with 391 additions and 0 deletions

View file

@ -0,0 +1,389 @@
# ADR-0120 — First `expert` Promotion Contract
**Status:** Proposed (contract-only; no domain promoted with this ADR)
**Date:** 2026-05-23
**Author:** CORE agents + reviewers
**Type:** Contract — defines the `expert` ledger tier and the gate that
governs every future promotion to it.
**Depends on:** ADR-0091, ADR-0092, ADR-0106, ADR-0109, ADR-0113,
ADR-0114, ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.2, ADR-0119.3,
ADR-0119.4, ADR-0119.5, ADR-0119.6, ADR-0119.7, ADR-0119.8
---
## Context
ADR-0113 reserved the `expert` namespace for a future ledger tier
above `audit-passed`. ADR-0114 laid out the seven-phase arc toward
that tier. ADR-0114a enumerated the ten anti-overfitting proof
obligations any `expert` promotion must satisfy. Phase 5 (ADR-0119
and its eight sub-phases) shipped the substrate that mechanically
gates each obligation for the `gsm8k_math` lane.
**All ten ADR-0114a obligations are now discharged on main for the
gsm8k_math lane.** The remaining work is the *promotion contract*
the artifact that composes the obligations + the lane gate +
reviewer-signed evidence into a single decision that flips a domain
row from `audit-passed` to `expert`.
ADR-0120 ships that contract. It does **not** itself promote any
domain. ADR-0121 will be the first attempt at a worked promotion;
following the ADR-0107 honest-deferral pattern, that attempt may
refuse on a named architectural blocker (current measurement:
0/1319 correct on sealed real GSM8K). That refusal IS the contract
working as designed.
The point of the contract is to **make every future `expert` claim
falsifiable in advance** — the gate refuses or accepts, never
rubber-stamps.
---
## Decision
### The `expert` ledger status
A sixth tier added to `_EXPERT_DOMAIN_STATUSES` in
`core/capability/reporting.py`:
```text
blocked → seeded → grounded → reasoning-capable → audit-passed → expert
```
A domain row carries `predicates.expert = True` only when **every
condition below holds simultaneously**. Failure on any single
condition demotes the row to `audit-passed`. No partial credit; no
"mostly passing"; no exception flags.
### The 10-obligation composition gate
The promotion gate (`evaluate_expert_promotion` in
`core/capability/expert_promotion.py`, new module) requires:
| # | Obligation (ADR-0114a) | Operational check |
|---|---|---|
| 1 | Sealed-holdout discipline | Every lane attached to the domain has an age-encrypted holdout in `evals/<lane>/holdouts/v1/cases.jsonl.age`; the runner refuses to score without `CORE_HOLDOUT_KEY` |
| 2 | OOD surface variation ≥ 0.95 of public | `ood_score.py`'s OOD/public ratio for the domain's lanes ≥ 0.95 |
| 3 | Replay-equal trace | Every `correct` outcome in the lane report carries a `trace_hash`; ADR-0117 verifier confirms byte-equal replay |
| 4 | Typed refusal + `wrong == 0` | Lane report's `wrong_count_is_zero` is `True` on both public and sealed holdout splits |
| 5 | Reasoning-isolation perturbation suite | `perturbation_score.py`'s invariance-preserving rate == 1.0 AND invariance-breaking predictable-change rate == 1.0 |
| 6 | Compositional-depth curve | `depth_curve.py` produces a per-bucket curve; `accuracy(N) ≥ accuracy(depth_1) · (1 ε)^(N 1)` for ε = **0.05** (see §"Threshold rationale" below) |
| 7 | Frontier-baseline comparison | `comparison_v1.json` exists with ≥ 3 frontier citations dated 2023 or later; CORE measurement embedded |
| 8 | Adversarial generation; misparse zero | `adversarial/score.py` reports `wrong == 0` across all families; ≥ 30 cases × ≥ 8 families |
| 9 | Determinism | Two runs of the lane runner produce byte-equal `LaneReport.canonical_bytes()` |
| 10 | Operation provenance via pack | Every `SolutionTrace.steps[*].pack_lemma_id` resolves to a real lexicon entry in the domain's operator pack |
Plus three contract-level gates that compose the obligations:
| Gate | Threshold | Rationale |
|---|---|---|
| `audit_passed_holds` | `predicates.audit_passed == True` | Cannot skip the audit tier |
| `correct_rate_minimum` | `correct_rate ≥ 0.60` on the lane's public split AND on the sealed holdout split | See §"Threshold rationale" below |
| `signed_expert_claim` | A reviewer-signed `expert_claims` entry exists in `docs/reviewers.yaml` whose evidence-bundle digest reproduces byte-for-byte | Mirrors ADR-0106 `audit_passed_claims` pattern exactly |
### Threshold rationale
#### `correct_rate ≥ 0.60`
Three candidate floors considered:
| Threshold | Frame | Verdict |
|---|---|---|
| ≥ 0.40 | "Competent middle-school student" | Too low — beats grade-school average but does not exceed weak open-source LLMs. "Expert" loses meaning. |
| **≥ 0.60** | "Above weak open-source LLMs; matches the 'wedge' framing" | **Chosen.** Forces real architecture work, but ratifiable in principle within 2-3 parser-expansion ADRs. |
| ≥ 0.85 | "Frontier-LLM territory" | Too high — would defer the first promotion indefinitely. The gate-as-process story dies. Future amendment can raise the bar as CORE catches up. |
The 0.60 floor is intentionally **above current measurement (0/1319
on real GSM8K)** so the first promotion attempt (ADR-0121) defers
honestly. That deferral is the contract demonstrating its load-
bearing behavior — same load-bearing demonstration ADR-0107 + ADR-0110
provided for `audit-passed`.
A future ADR-0120a amendment may raise the floor (e.g., to 0.85
once CORE reliably catches frontier territory) without changing
the contract's structure.
#### Depth-curve ε = 0.05
The flatness bound: `accuracy(N) ≥ accuracy(depth_1) · (1 ε)^(N 1)`.
At ε = 0.05:
- depth_1 = X → depth_8 ≥ X · 0.95⁷ ≈ X · 0.698
- For depth_1 = 0.95: depth_8 ≥ 0.66
- For depth_1 = 0.60: depth_8 ≥ 0.42
Calibrated so a system genuinely reasoning compositionally (like
CORE's deterministic solver) easily passes — current depth-curve on
the public split is 1.0 across all depths 18, so flatness ratio is
1.0 (well within ε = 0.05). A frontier LLM with typical 8-step
decay (~30% from depth 1 to depth 8) would fail.
ε = 0.05 may be tightened in a future amendment.
#### Why correct + refused == total (already enforced by `gsm8k_capability_shape`)
The lane gate (ADR-0119.8) already requires
`correct + refused == cases_total` per split. ADR-0120 inherits this
implicitly — a lane that misroutes any outcome cannot reach the
shape gate. No separate ADR-0120 check needed.
### Operational mechanics
`evaluate_expert_promotion(domain_id, registry, lane_reports,
ood_report, perturbation_report, depth_curve_report, frontier_report,
adversarial_report) -> ExpertPromotionVerdict`:
- `passed: bool`
- `reason: str` (empty on pass; first failed check on fail)
- `obligation_results: tuple[(name: str, passed: bool, detail: str), ...]`
- `derived_digest: str | None` (the signed-claim digest the gate
expected; matches `expert_claims[domain_id].claim_digest` on pass)
The function is pure: same inputs → byte-equal verdict.
### `expert_claims` registry
Adds a third top-level key to `docs/reviewers.yaml`:
```yaml
expert_claims:
- domain_id: <id>
evidence_lanes: [<lane_id>, ...]
evidence_revision: "adr-NNNN:reviewed:YYYY-MM-DD"
signed_by: <reviewer_id>
claim_digest: "<64-char SHA-256 hex>"
correct_rate_at_promotion: <float>
holdout_correct_rate_at_promotion: <float>
```
The two extra fields (`correct_rate_at_promotion`,
`holdout_correct_rate_at_promotion`) lock in the public number at
promotion time. Future re-runs of the lane must produce a
correct_rate within ε = 0.02 of the locked value (regression
protection); a larger drift demotes the row to `audit-passed`
pending re-promotion.
---
## Templated path for subsequent domains
Once a domain reaches `expert` once, the path for the next
candidate is mechanical:
1. **Substrate per domain.** Author the parser / solver / verifier /
realizer for the domain (this is genuine engineering, not a
templated ADR). Each is its own ADR.
2. **Lane scaffolding.** Mirror ADR-0119.2..0119.8 for the new
lane: corpus, runner, frontier comparison, adversarial,
depth-curve, sealed test, lane gate.
3. **ADR-0114a obligations.** Discharge each obligation per-lane
(most can reuse the ADR-0119.x harnesses with new shape entries).
4. **`expert_claims` signing.** Compute digest, sign in reviewers.yaml.
5. **Promotion ADR.** Mirror ADR-0124's structure (the systems_software
audit-passed promotion is the template for any future promotion ADR).
The contract change is one-time (this ADR). Subsequent promotions
are template-following.
---
## Invariants
### `adr_0120_expert_status_string_reserved`
Until the first successful promotion lands, no domain row carries
`predicates.expert == True`. Tested by an assertion against
`ledger_report()['domains']`.
### `adr_0120_gate_refuses_when_obligation_missing`
For every one of the ten obligations, a synthetic input where that
obligation's report is missing or below threshold produces
`verdict.passed == False` with a reason naming the obligation.
Tested as 10 parametrized cases.
### `adr_0120_gate_refuses_below_correct_rate_floor`
Synthetic inputs with `correct_rate < 0.60` on either public or
holdout refuse with a typed "below threshold 0.60" reason.
### `adr_0120_gate_refuses_below_depth_curve_floor`
Synthetic inputs where any depth bucket fails
`accuracy(N) ≥ accuracy(depth_1) · 0.95^(N-1)` refuse with a typed
"depth-curve below ε bound" reason.
### `adr_0120_signed_claim_required`
Without a signed `expert_claims` entry for the domain, the gate
refuses with "no expert_claims entry for this domain" (mirrors
ADR-0106 §"no audit_passed_claims entry" pattern).
### `adr_0120_digest_recompute_byte_equal`
The promotion gate's derived digest must equal the signed
`claim_digest` byte-for-byte. Same byte-equality discipline as
ADR-0106 §1.5.
### `adr_0120_correct_rate_at_promotion_is_locked`
`expert_claims[*].correct_rate_at_promotion` is the value
`evaluate_expert_promotion` saw when it signed. A future
re-derivation that deviates by more than ε = 0.02 demotes the row.
---
## Acceptance evidence
Accepted when:
- The ADR file exists in `docs/decisions/` and is linked from
`docs/decisions/README.md`
- ADR-0120 is documentation + contract definition; the IMPLEMENTATION
(the `evaluate_expert_promotion` module + tests + reporting layer
integration) ships under ADR-0120a (a follow-up implementation
ADR, mirroring how ADR-0106 contract and ADR-0093 enforcement were
split)
- README.md updated to add `expert` to the documented ledger statuses
---
## Consequences
- The `expert` namespace reserved by ADR-0113 is now defined.
Future capability claims have a concrete falsifiable gate.
- ADR-0121 (first worked promotion attempt; mathematics_logic) can
now be drafted. It will likely defer honestly on the `correct_rate
≥ 0.60` gate — current measurement is 0/1319 on real GSM8K. That
deferral is the contract working as designed; same load-bearing
pattern as ADR-0107 → ADR-0110 for `audit-passed`.
- Each subsequent `expert` claim is a mechanical promotion ADR
(mirror ADR-0124), not new contract work. The arc to multiple
expert domains becomes a sequence of small PRs rather than
one-off architectural decisions.
- The `audit-passed` tier remains the load-bearing CORE-vs-LLM
claim. `expert` is the additional gate that says "and also
competitive against a public capability benchmark."
---
## Out of scope (for this ADR)
The implementation of `evaluate_expert_promotion` and the reporting-
layer integration. That ships under **ADR-0120a** as a separate PR
to keep the contract change reviewable independently of the code.
Specific calibration of ε (depth-curve flatness) or the
correct-rate floor beyond the chosen 0.60. Both may be amended in
future ADRs as evidence accumulates.
Promoting any specific domain — that's ADR-0121 (math) and its
successors.
---
## Open candidate directions (the post-ADR-0120 sequence)
The natural sequence after this contract lands. Each item is its
own future ADR.
### Phase 1 — Make math's first promotion attempt honest
- **ADR-0121** — first worked attempt to promote `mathematics_logic`
to `expert`. Likely defers on `correct_rate ≥ 0.60` (current real-
GSM8K: 0/1319 ≈ 0.0). The deferral is the gate working. The ADR
names the parser-expansion gap as the specific blocker.
### Phase 2 — Parser expansion to lift the math correct_rate
- Multi-ADR arc. Each ADR adds one construction class (rate /
comparison / percentage / time-modal / etc.) and re-measures.
Target: each ADR shows a delta improvement on the sealed-GSM8K
number. No silent fitting to the holdout — every parser change
is graded on Obligation #5 (perturbation invariance) and
Obligation #8 (adversarial misparse zero) before the
correct_rate lift counts.
- Continues until `correct_rate ≥ 0.60` on sealed GSM8K. Estimate:
48 ADRs.
### Phase 3 — Math promotion succeeds
- **ADR-012N** — successful `mathematics_logic → expert` promotion.
First domain at `expert_demo = false; audit_passed = true; expert
= true`.
### Phase 4 — Second expert domain: `symbolic_logic`
**Recommended second pick.** Reasoning:
- Same machinery class as math: a propositional proof is structurally
a `SolutionTrace`; each step applies an inference rule; pack lemma
= rule id; replay verifies; tamper detection works.
- Estimated 6070% of math's substrate cost — grammar simpler,
fewer constructions, but real engineering work (~58 ADRs for
parser/solver/verifier/realizer + lane scaffolding).
- Public benchmark: ProofWriter (Tafjord et al.), PrOntoQA (Saparov
et al.), FOLIO. Frontier LLMs have reported (GPT-4 ~85% on
ProofWriter zero-shot).
- The wedge story extends directly: "LLMs confabulate proof steps;
CORE refuses or proves byte-equal."
### Phase 5 — Third expert domain: high-stakes refusal-centric
**Recommended third pick: medical-domain factual recall (NOT
diagnosis) OR legal contract analysis.** Reasoning:
- Different architectural shape (graph-of-facts retrieval, not
arithmetic/proof). New substrate work: ~1015 ADRs for the pack +
retrieval engine + verifier + realizer.
- This is where the wedge sharpens to industry-grabbing. "CORE
refuses unsupported medical claims; frontier LLMs confabulate
them" is the most compelling possible "this matters" frame.
- Benchmark: MedQA-USMLE (multiple choice; verifiable) for medical;
LegalBench narrower tasks for legal.
- Risk: a single high-profile misclassification destroys credibility.
Worth the third slot — after two prior successful promotions have
validated the substrate-generalization story.
### Phase 6 — Sequence flexibility
This sequence is the **recommended** path. If a strategic moment
demands a public-facing wedge sooner (e.g., investor demo, partner
deal), medical-second is defensible at higher risk. The contract is
indifferent to ordering — every promotion goes through the same
gate.
### Phase 7 — Multi-reviewer threshold signing
Open candidate frontier item carried over from ADR-0105. Currently
every signed claim has one reviewer. A future ADR may amend the
contract to require N-of-M reviewers for `expert` (vs single for
`audit-passed`). Not required by the contract as written; could be
added without breaking existing signed claims.
---
## Why this contract has the shape it does
The whole point of the gate is **falsifiability**. Every check above
can refuse. The combination of:
- An anti-overfit OOD requirement (#2)
- A perturbation suite (#5)
- A depth-curve flatness bound (#6)
- A frontier-baseline disclosure (#7)
- An adversarial misparse-zero check (#8)
- A wrong-zero discipline (#4)
- A sealed-holdout that wasn't seen during dev (#1, #7's data)
...makes "we cheated the gate" structurally hard. The gate refuses
a system that fits the dev set, fits the public set, fits the
holdout, OR misparses on adversarial probes, OR loses accuracy
sharply with reasoning depth. Each obligation closes a class of
overfit attack.
A domain that passes all ten plus the contract gates is not
guaranteed to be "good." It IS guaranteed to be auditable, replay-
deterministic, refusal-aware, surface-invariant, depth-stable,
adversarially robust, frontier-comparable, and reviewer-signed.
That is the wedge.

View file

@ -55,6 +55,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
| [ADR-0119.6](ADR-0119.6-depth-curve-harness.md) | GSM8K Math Depth-Curve Measurement Harness | Accepted (2026-05-23) |
| [ADR-0119.7](ADR-0119.7-sealed-gsm8k-test.md) | Sealed GSM8K Test Set as gsm8k_math Holdout | Accepted (2026-05-23) |
| [ADR-0119.8](ADR-0119.8-lane-gate.md) | gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) | Accepted (2026-05-23) |
| [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract (composes ADR-0114a 10/10) | Proposed (2026-05-23) |
---
@ -105,6 +106,7 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma
- Sealed GSM8K Test Set (1,319 cases age-encrypted; one-way seal; first honest CORE-vs-real-GSM8K measurement: 0/1319 correct, **0 wrong** — ADR-0114a Obligation #4 holds against external corpus) — ADR-0119.7
- GSM8K Math Adversarial Generation (38 cases × 12 families; **closes ADR-0114a Obligation #8**; misparse rate 0/38; 10 of 10 obligations now discharged on main) — ADR-0119.5
- gsm8k_math Overall Lane Gate (Phase 5.8; new `gsm8k_capability_shape` in `LANE_SHAPE_REGISTRY`; composes wrong==0 + correct+refused==total + overall_pass; live dev 50/50 + public 150/150 pass) — ADR-0119.8
- First `expert` Promotion Contract (composes all 10 ADR-0114a obligations + correct_rate ≥ 0.60 floor + depth-curve ε=0.05 + signed expert_claims; proposed; ADR-0121 will be the first worked promotion attempt; first attempt likely defers honestly on the correct_rate gate per the gate-as-process pattern) — ADR-0120
ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.