diff --git a/docs/decisions/ADR-0120-expert-promotion-contract.md b/docs/decisions/ADR-0120-expert-promotion-contract.md new file mode 100644 index 00000000..18331bf9 --- /dev/null +++ b/docs/decisions/ADR-0120-expert-promotion-contract.md @@ -0,0 +1,389 @@ +# ADR-0120 — First `expert` Promotion Contract + +**Status:** Proposed (contract-only; no domain promoted with this ADR) +**Date:** 2026-05-23 +**Author:** CORE agents + reviewers +**Type:** Contract — defines the `expert` ledger tier and the gate that +governs every future promotion to it. +**Depends on:** ADR-0091, ADR-0092, ADR-0106, ADR-0109, ADR-0113, +ADR-0114, ADR-0114a, ADR-0119, ADR-0119.1, ADR-0119.2, ADR-0119.3, +ADR-0119.4, ADR-0119.5, ADR-0119.6, ADR-0119.7, ADR-0119.8 + +--- + +## Context + +ADR-0113 reserved the `expert` namespace for a future ledger tier +above `audit-passed`. ADR-0114 laid out the seven-phase arc toward +that tier. ADR-0114a enumerated the ten anti-overfitting proof +obligations any `expert` promotion must satisfy. Phase 5 (ADR-0119 +and its eight sub-phases) shipped the substrate that mechanically +gates each obligation for the `gsm8k_math` lane. + +**All ten ADR-0114a obligations are now discharged on main for the +gsm8k_math lane.** The remaining work is the *promotion contract* — +the artifact that composes the obligations + the lane gate + +reviewer-signed evidence into a single decision that flips a domain +row from `audit-passed` to `expert`. + +ADR-0120 ships that contract. It does **not** itself promote any +domain. ADR-0121 will be the first attempt at a worked promotion; +following the ADR-0107 honest-deferral pattern, that attempt may +refuse on a named architectural blocker (current measurement: +0/1319 correct on sealed real GSM8K). That refusal IS the contract +working as designed. + +The point of the contract is to **make every future `expert` claim +falsifiable in advance** — the gate refuses or accepts, never +rubber-stamps. + +--- + +## Decision + +### The `expert` ledger status + +A sixth tier added to `_EXPERT_DOMAIN_STATUSES` in +`core/capability/reporting.py`: + +```text +blocked → seeded → grounded → reasoning-capable → audit-passed → expert +``` + +A domain row carries `predicates.expert = True` only when **every +condition below holds simultaneously**. Failure on any single +condition demotes the row to `audit-passed`. No partial credit; no +"mostly passing"; no exception flags. + +### The 10-obligation composition gate + +The promotion gate (`evaluate_expert_promotion` in +`core/capability/expert_promotion.py`, new module) requires: + +| # | Obligation (ADR-0114a) | Operational check | +|---|---|---| +| 1 | Sealed-holdout discipline | Every lane attached to the domain has an age-encrypted holdout in `evals//holdouts/v1/cases.jsonl.age`; the runner refuses to score without `CORE_HOLDOUT_KEY` | +| 2 | OOD surface variation ≥ 0.95 of public | `ood_score.py`'s OOD/public ratio for the domain's lanes ≥ 0.95 | +| 3 | Replay-equal trace | Every `correct` outcome in the lane report carries a `trace_hash`; ADR-0117 verifier confirms byte-equal replay | +| 4 | Typed refusal + `wrong == 0` | Lane report's `wrong_count_is_zero` is `True` on both public and sealed holdout splits | +| 5 | Reasoning-isolation perturbation suite | `perturbation_score.py`'s invariance-preserving rate == 1.0 AND invariance-breaking predictable-change rate == 1.0 | +| 6 | Compositional-depth curve | `depth_curve.py` produces a per-bucket curve; `accuracy(N) ≥ accuracy(depth_1) · (1 − ε)^(N − 1)` for ε = **0.05** (see §"Threshold rationale" below) | +| 7 | Frontier-baseline comparison | `comparison_v1.json` exists with ≥ 3 frontier citations dated 2023 or later; CORE measurement embedded | +| 8 | Adversarial generation; misparse zero | `adversarial/score.py` reports `wrong == 0` across all families; ≥ 30 cases × ≥ 8 families | +| 9 | Determinism | Two runs of the lane runner produce byte-equal `LaneReport.canonical_bytes()` | +| 10 | Operation provenance via pack | Every `SolutionTrace.steps[*].pack_lemma_id` resolves to a real lexicon entry in the domain's operator pack | + +Plus three contract-level gates that compose the obligations: + +| Gate | Threshold | Rationale | +|---|---|---| +| `audit_passed_holds` | `predicates.audit_passed == True` | Cannot skip the audit tier | +| `correct_rate_minimum` | `correct_rate ≥ 0.60` on the lane's public split AND on the sealed holdout split | See §"Threshold rationale" below | +| `signed_expert_claim` | A reviewer-signed `expert_claims` entry exists in `docs/reviewers.yaml` whose evidence-bundle digest reproduces byte-for-byte | Mirrors ADR-0106 `audit_passed_claims` pattern exactly | + +### Threshold rationale + +#### `correct_rate ≥ 0.60` + +Three candidate floors considered: + +| Threshold | Frame | Verdict | +|---|---|---| +| ≥ 0.40 | "Competent middle-school student" | Too low — beats grade-school average but does not exceed weak open-source LLMs. "Expert" loses meaning. | +| **≥ 0.60** | "Above weak open-source LLMs; matches the 'wedge' framing" | **Chosen.** Forces real architecture work, but ratifiable in principle within 2-3 parser-expansion ADRs. | +| ≥ 0.85 | "Frontier-LLM territory" | Too high — would defer the first promotion indefinitely. The gate-as-process story dies. Future amendment can raise the bar as CORE catches up. | + +The 0.60 floor is intentionally **above current measurement (0/1319 +on real GSM8K)** so the first promotion attempt (ADR-0121) defers +honestly. That deferral is the contract demonstrating its load- +bearing behavior — same load-bearing demonstration ADR-0107 + ADR-0110 +provided for `audit-passed`. + +A future ADR-0120a amendment may raise the floor (e.g., to 0.85 +once CORE reliably catches frontier territory) without changing +the contract's structure. + +#### Depth-curve ε = 0.05 + +The flatness bound: `accuracy(N) ≥ accuracy(depth_1) · (1 − ε)^(N − 1)`. + +At ε = 0.05: +- depth_1 = X → depth_8 ≥ X · 0.95⁷ ≈ X · 0.698 +- For depth_1 = 0.95: depth_8 ≥ 0.66 +- For depth_1 = 0.60: depth_8 ≥ 0.42 + +Calibrated so a system genuinely reasoning compositionally (like +CORE's deterministic solver) easily passes — current depth-curve on +the public split is 1.0 across all depths 1–8, so flatness ratio is +1.0 (well within ε = 0.05). A frontier LLM with typical 8-step +decay (~30% from depth 1 to depth 8) would fail. + +ε = 0.05 may be tightened in a future amendment. + +#### Why correct + refused == total (already enforced by `gsm8k_capability_shape`) + +The lane gate (ADR-0119.8) already requires +`correct + refused == cases_total` per split. ADR-0120 inherits this +implicitly — a lane that misroutes any outcome cannot reach the +shape gate. No separate ADR-0120 check needed. + +### Operational mechanics + +`evaluate_expert_promotion(domain_id, registry, lane_reports, +ood_report, perturbation_report, depth_curve_report, frontier_report, +adversarial_report) -> ExpertPromotionVerdict`: + +- `passed: bool` +- `reason: str` (empty on pass; first failed check on fail) +- `obligation_results: tuple[(name: str, passed: bool, detail: str), ...]` +- `derived_digest: str | None` (the signed-claim digest the gate + expected; matches `expert_claims[domain_id].claim_digest` on pass) + +The function is pure: same inputs → byte-equal verdict. + +### `expert_claims` registry + +Adds a third top-level key to `docs/reviewers.yaml`: + +```yaml +expert_claims: + - domain_id: + evidence_lanes: [, ...] + evidence_revision: "adr-NNNN:reviewed:YYYY-MM-DD" + signed_by: + claim_digest: "<64-char SHA-256 hex>" + correct_rate_at_promotion: + holdout_correct_rate_at_promotion: +``` + +The two extra fields (`correct_rate_at_promotion`, +`holdout_correct_rate_at_promotion`) lock in the public number at +promotion time. Future re-runs of the lane must produce a +correct_rate within ε = 0.02 of the locked value (regression +protection); a larger drift demotes the row to `audit-passed` +pending re-promotion. + +--- + +## Templated path for subsequent domains + +Once a domain reaches `expert` once, the path for the next +candidate is mechanical: + +1. **Substrate per domain.** Author the parser / solver / verifier / + realizer for the domain (this is genuine engineering, not a + templated ADR). Each is its own ADR. +2. **Lane scaffolding.** Mirror ADR-0119.2..0119.8 for the new + lane: corpus, runner, frontier comparison, adversarial, + depth-curve, sealed test, lane gate. +3. **ADR-0114a obligations.** Discharge each obligation per-lane + (most can reuse the ADR-0119.x harnesses with new shape entries). +4. **`expert_claims` signing.** Compute digest, sign in reviewers.yaml. +5. **Promotion ADR.** Mirror ADR-0124's structure (the systems_software + audit-passed promotion is the template for any future promotion ADR). + +The contract change is one-time (this ADR). Subsequent promotions +are template-following. + +--- + +## Invariants + +### `adr_0120_expert_status_string_reserved` + +Until the first successful promotion lands, no domain row carries +`predicates.expert == True`. Tested by an assertion against +`ledger_report()['domains']`. + +### `adr_0120_gate_refuses_when_obligation_missing` + +For every one of the ten obligations, a synthetic input where that +obligation's report is missing or below threshold produces +`verdict.passed == False` with a reason naming the obligation. +Tested as 10 parametrized cases. + +### `adr_0120_gate_refuses_below_correct_rate_floor` + +Synthetic inputs with `correct_rate < 0.60` on either public or +holdout refuse with a typed "below threshold 0.60" reason. + +### `adr_0120_gate_refuses_below_depth_curve_floor` + +Synthetic inputs where any depth bucket fails +`accuracy(N) ≥ accuracy(depth_1) · 0.95^(N-1)` refuse with a typed +"depth-curve below ε bound" reason. + +### `adr_0120_signed_claim_required` + +Without a signed `expert_claims` entry for the domain, the gate +refuses with "no expert_claims entry for this domain" (mirrors +ADR-0106 §"no audit_passed_claims entry" pattern). + +### `adr_0120_digest_recompute_byte_equal` + +The promotion gate's derived digest must equal the signed +`claim_digest` byte-for-byte. Same byte-equality discipline as +ADR-0106 §1.5. + +### `adr_0120_correct_rate_at_promotion_is_locked` + +`expert_claims[*].correct_rate_at_promotion` is the value +`evaluate_expert_promotion` saw when it signed. A future +re-derivation that deviates by more than ε = 0.02 demotes the row. + +--- + +## Acceptance evidence + +Accepted when: + +- The ADR file exists in `docs/decisions/` and is linked from + `docs/decisions/README.md` +- ADR-0120 is documentation + contract definition; the IMPLEMENTATION + (the `evaluate_expert_promotion` module + tests + reporting layer + integration) ships under ADR-0120a (a follow-up implementation + ADR, mirroring how ADR-0106 contract and ADR-0093 enforcement were + split) +- README.md updated to add `expert` to the documented ledger statuses + +--- + +## Consequences + +- The `expert` namespace reserved by ADR-0113 is now defined. + Future capability claims have a concrete falsifiable gate. +- ADR-0121 (first worked promotion attempt; mathematics_logic) can + now be drafted. It will likely defer honestly on the `correct_rate + ≥ 0.60` gate — current measurement is 0/1319 on real GSM8K. That + deferral is the contract working as designed; same load-bearing + pattern as ADR-0107 → ADR-0110 for `audit-passed`. +- Each subsequent `expert` claim is a mechanical promotion ADR + (mirror ADR-0124), not new contract work. The arc to multiple + expert domains becomes a sequence of small PRs rather than + one-off architectural decisions. +- The `audit-passed` tier remains the load-bearing CORE-vs-LLM + claim. `expert` is the additional gate that says "and also + competitive against a public capability benchmark." + +--- + +## Out of scope (for this ADR) + +The implementation of `evaluate_expert_promotion` and the reporting- +layer integration. That ships under **ADR-0120a** as a separate PR +to keep the contract change reviewable independently of the code. + +Specific calibration of ε (depth-curve flatness) or the +correct-rate floor beyond the chosen 0.60. Both may be amended in +future ADRs as evidence accumulates. + +Promoting any specific domain — that's ADR-0121 (math) and its +successors. + +--- + +## Open candidate directions (the post-ADR-0120 sequence) + +The natural sequence after this contract lands. Each item is its +own future ADR. + +### Phase 1 — Make math's first promotion attempt honest + +- **ADR-0121** — first worked attempt to promote `mathematics_logic` + to `expert`. Likely defers on `correct_rate ≥ 0.60` (current real- + GSM8K: 0/1319 ≈ 0.0). The deferral is the gate working. The ADR + names the parser-expansion gap as the specific blocker. + +### Phase 2 — Parser expansion to lift the math correct_rate + +- Multi-ADR arc. Each ADR adds one construction class (rate / + comparison / percentage / time-modal / etc.) and re-measures. + Target: each ADR shows a delta improvement on the sealed-GSM8K + number. No silent fitting to the holdout — every parser change + is graded on Obligation #5 (perturbation invariance) and + Obligation #8 (adversarial misparse zero) before the + correct_rate lift counts. +- Continues until `correct_rate ≥ 0.60` on sealed GSM8K. Estimate: + 4–8 ADRs. + +### Phase 3 — Math promotion succeeds + +- **ADR-012N** — successful `mathematics_logic → expert` promotion. + First domain at `expert_demo = false; audit_passed = true; expert + = true`. + +### Phase 4 — Second expert domain: `symbolic_logic` + +**Recommended second pick.** Reasoning: + +- Same machinery class as math: a propositional proof is structurally + a `SolutionTrace`; each step applies an inference rule; pack lemma + = rule id; replay verifies; tamper detection works. +- Estimated 60–70% of math's substrate cost — grammar simpler, + fewer constructions, but real engineering work (~5–8 ADRs for + parser/solver/verifier/realizer + lane scaffolding). +- Public benchmark: ProofWriter (Tafjord et al.), PrOntoQA (Saparov + et al.), FOLIO. Frontier LLMs have reported (GPT-4 ~85% on + ProofWriter zero-shot). +- The wedge story extends directly: "LLMs confabulate proof steps; + CORE refuses or proves byte-equal." + +### Phase 5 — Third expert domain: high-stakes refusal-centric + +**Recommended third pick: medical-domain factual recall (NOT +diagnosis) OR legal contract analysis.** Reasoning: + +- Different architectural shape (graph-of-facts retrieval, not + arithmetic/proof). New substrate work: ~10–15 ADRs for the pack + + retrieval engine + verifier + realizer. +- This is where the wedge sharpens to industry-grabbing. "CORE + refuses unsupported medical claims; frontier LLMs confabulate + them" is the most compelling possible "this matters" frame. +- Benchmark: MedQA-USMLE (multiple choice; verifiable) for medical; + LegalBench narrower tasks for legal. +- Risk: a single high-profile misclassification destroys credibility. + Worth the third slot — after two prior successful promotions have + validated the substrate-generalization story. + +### Phase 6 — Sequence flexibility + +This sequence is the **recommended** path. If a strategic moment +demands a public-facing wedge sooner (e.g., investor demo, partner +deal), medical-second is defensible at higher risk. The contract is +indifferent to ordering — every promotion goes through the same +gate. + +### Phase 7 — Multi-reviewer threshold signing + +Open candidate frontier item carried over from ADR-0105. Currently +every signed claim has one reviewer. A future ADR may amend the +contract to require N-of-M reviewers for `expert` (vs single for +`audit-passed`). Not required by the contract as written; could be +added without breaking existing signed claims. + +--- + +## Why this contract has the shape it does + +The whole point of the gate is **falsifiability**. Every check above +can refuse. The combination of: + +- An anti-overfit OOD requirement (#2) +- A perturbation suite (#5) +- A depth-curve flatness bound (#6) +- A frontier-baseline disclosure (#7) +- An adversarial misparse-zero check (#8) +- A wrong-zero discipline (#4) +- A sealed-holdout that wasn't seen during dev (#1, #7's data) + +...makes "we cheated the gate" structurally hard. The gate refuses +a system that fits the dev set, fits the public set, fits the +holdout, OR misparses on adversarial probes, OR loses accuracy +sharply with reasoning depth. Each obligation closes a class of +overfit attack. + +A domain that passes all ten plus the contract gates is not +guaranteed to be "good." It IS guaranteed to be auditable, replay- +deterministic, refusal-aware, surface-invariant, depth-stable, +adversarially robust, frontier-comparable, and reviewer-signed. +That is the wedge. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 6fe7a30c..b462b65a 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -55,6 +55,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0119.6](ADR-0119.6-depth-curve-harness.md) | GSM8K Math Depth-Curve Measurement Harness | Accepted (2026-05-23) | | [ADR-0119.7](ADR-0119.7-sealed-gsm8k-test.md) | Sealed GSM8K Test Set as gsm8k_math Holdout | Accepted (2026-05-23) | | [ADR-0119.8](ADR-0119.8-lane-gate.md) | gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) | Accepted (2026-05-23) | +| [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract (composes ADR-0114a 10/10) | Proposed (2026-05-23) | --- @@ -105,6 +106,7 @@ The ADR-0091..0114 slate is fully accepted (0091..0113) plus one proposed-roadma - Sealed GSM8K Test Set (1,319 cases age-encrypted; one-way seal; first honest CORE-vs-real-GSM8K measurement: 0/1319 correct, **0 wrong** — ADR-0114a Obligation #4 holds against external corpus) — ADR-0119.7 - GSM8K Math Adversarial Generation (38 cases × 12 families; **closes ADR-0114a Obligation #8**; misparse rate 0/38; 10 of 10 obligations now discharged on main) — ADR-0119.5 - gsm8k_math Overall Lane Gate (Phase 5.8; new `gsm8k_capability_shape` in `LANE_SHAPE_REGISTRY`; composes wrong==0 + correct+refused==total + overall_pass; live dev 50/50 + public 150/150 pass) — ADR-0119.8 +- First `expert` Promotion Contract (composes all 10 ADR-0114a obligations + correct_rate ≥ 0.60 floor + depth-curve ε=0.05 + signed expert_claims; proposed; ADR-0121 will be the first worked promotion attempt; first attempt likely defers honestly on the correct_rate gate per the gate-as-process pattern) — ADR-0120 ADR-0080 has also landed: Contemplation Loop Phase 1 adds a read-only frontier-compare miner that emits `SPECULATIVE` findings only.