feat(adr): ADR-0107 mathematics_logic expert-demo promotion deferred (#114)
The ADR-0106 contract correctly refused promotion. ADR-0107 records the deferral and reserves two follow-up ADRs: - ADR-0109 (lane-shape-aware threshold amendment): ADR-0106 \xc2\xa71.2 prescribes cognition-pack-shape metrics uniformly, but math / physics / systems / hebrew-greek lanes carry native shapes (accuracy, passed_rate, all_pass_rate). Prerequisite for any future expert-demo promotion. - ADR-0110 (math re-attempt): conditional on ADR-0109 landing and inference_closure substantively passing (currently all_pass_rate=0.4 on public). tests/test_adr_0107_deferral.py pins adr_0107_no_silent_promotion: math stays at reasoning-capable, has no expert_demo_claims entry, and the ledger row carries a named refusal reason. No change to core/capability/expert_demo.py or reporting.py -- the contract is honored, not amended. README sequencing updated to reflect ADR-0107 acceptance and the new ADR-0109/0110 prerequisites.
This commit is contained in:
Shay
GitHub
parent
0493808215
commit
7cc2f7b422
3 changed files with 218 additions and 5 deletions
|
|
@ -0,0 +1,162 @@
|
|||
# ADR-0107 — `mathematics_logic` Expert-Demo Promotion: Deferred
|
||||
|
||||
**Status:** Accepted (decision: defer promotion)
|
||||
**Date:** 2026-05-22
|
||||
**Author:** CORE agents + reviewers
|
||||
**Depends on:** ADR-0091, ADR-0092, ADR-0097, ADR-0106, ADR-0108
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0108 reserved ADR-0107 as the first worked expert-demo promotion
|
||||
against the ADR-0106 contract, with `mathematics_logic` named as the
|
||||
smallest expert-demo proof surface across the four ratified domains.
|
||||
|
||||
On evaluation, the ADR-0106 gate correctly refused to promote
|
||||
`mathematics_logic`. ADR-0107 records that refusal honestly and reserves
|
||||
the follow-up ADRs needed before promotion can re-attempt.
|
||||
|
||||
This is the contract working as designed. ADR-0106 §Consequences
|
||||
predicted that "reasoning-capable" should not equal "demonstrated"; the
|
||||
first promotion attempt has now surfaced two specific evidence-side
|
||||
gaps that prove the gap is real.
|
||||
|
||||
---
|
||||
|
||||
## Evidence
|
||||
|
||||
`mathematics_logic` attaches three eval lanes via
|
||||
`language_packs/data/en_mathematics_logic_v1/manifest.json`:
|
||||
|
||||
| Lane | Public split | Holdout split | Notes |
|
||||
|---|---|---|---|
|
||||
| `elementary_mathematics_ood` | `accuracy=1.0`, all C01–C13 cases pass | `accuracy=1.0`, `passed=39/39` | Reports lane-shape metrics, not cognition-shape metrics |
|
||||
| `inference_closure` | `all_pass_rate=0.4`, `overall_pass=False` | requires `CORE_HOLDOUT_KEY` (sealed under ADR-0105) | Lane is currently below any reasonable promotion threshold |
|
||||
| `fabrication_control` | refusals clean (`fabricated=0` across phantom / cross-pack / sibling-collapse classes) | not formally re-run after ADR-0105 sealing | Required by ADR-0106 §1.3 |
|
||||
|
||||
Neither `elementary_mathematics_ood` nor `inference_closure` nor
|
||||
`fabrication_control` reports the four metric keys ADR-0106 §1.2 prescribes
|
||||
(`surface_groundedness`, `term_capture_rate`, `intent_accuracy`,
|
||||
`versor_closure_rate`). Those keys are cognition-pack-shape; the math
|
||||
lanes carry their own native shapes.
|
||||
|
||||
---
|
||||
|
||||
## Decision
|
||||
|
||||
`mathematics_logic` is **not** promoted to `expert_demo=true`. The
|
||||
ledger row stays at `reasoning-capable`. The reviewer registry receives
|
||||
**no** `expert_demo_claims` entry as part of this ADR.
|
||||
|
||||
This is a positive decision recording two concrete blocking gaps:
|
||||
|
||||
### Gap 1 — ADR-0106 metric-shape uniformity assumption
|
||||
|
||||
ADR-0106 §1.2 prescribes cognition-shape thresholds uniformly across
|
||||
every attached lane. In practice, each lane reports its own native
|
||||
metric shape (`accuracy`, `passed_rate`, `all_pass_rate`,
|
||||
`derived_recall_rate`). Enforcing cognition-shape keys uniformly causes
|
||||
every non-cognition lane to fail the gate by absence-of-key, not by
|
||||
substance.
|
||||
|
||||
**Resolution path:** ADR-0109 (reserved) — amend ADR-0106 with a
|
||||
lane-shape adaptation layer that maps each lane's native metric to a
|
||||
contract-bearing threshold. Candidate shapes already observable in the
|
||||
repo:
|
||||
|
||||
- `cognition_shape` — the original four keys, used by the cognition pack
|
||||
- `accuracy_shape` — single `accuracy` key, used by
|
||||
`elementary_mathematics_ood`
|
||||
- `refusal_shape` — `fabricated`/`refused` counts, used by
|
||||
`fabrication_control`
|
||||
- `inference_shape` — `all_pass_rate` + `replay_determinism`, used by
|
||||
`inference_closure`
|
||||
|
||||
ADR-0109 must specify the contract-bearing threshold per shape and pin
|
||||
the mapping so a future lane introduces its shape explicitly rather
|
||||
than silently broadening the gate.
|
||||
|
||||
### Gap 2 — `inference_closure` lane substantively fails
|
||||
|
||||
`inference_closure` reports `all_pass_rate=0.4` on the public v1 split.
|
||||
Whether the failure is a runner bug, a case-set issue, or a genuine
|
||||
substrate gap is unresolved. Either way, the lane is not a positive
|
||||
signal for promotion at present.
|
||||
|
||||
**Resolution path:** Triage `inference_closure` failures (5 of 5 cases
|
||||
on dev, 12 of 20 on public). Until that lane crosses an explicit
|
||||
threshold under the ADR-0109 shape rules, `mathematics_logic` cannot
|
||||
promote regardless of how the metric-shape question is resolved.
|
||||
|
||||
---
|
||||
|
||||
## Follow-up ADRs reserved
|
||||
|
||||
| ADR | Purpose |
|
||||
|---|---|
|
||||
| **ADR-0109** | Lane-shape-aware threshold rules amending ADR-0106. |
|
||||
| **ADR-0110** | Re-attempt `mathematics_logic` expert-demo promotion under the ADR-0109-amended contract, conditional on `inference_closure` substantively passing. |
|
||||
|
||||
ADR-0107 explicitly does **not** pre-decide ADR-0110's outcome. It only
|
||||
records that the current evidence cannot support promotion.
|
||||
|
||||
---
|
||||
|
||||
## Invariants
|
||||
|
||||
### `adr_0107_no_silent_promotion`
|
||||
|
||||
No `expert_demo_claims` entry for `mathematics_logic` exists in
|
||||
`docs/reviewers.yaml` as a result of this ADR. A pytest gate confirms
|
||||
the math row in `ledger_report()` reports
|
||||
`predicates.expert_demo == false`.
|
||||
|
||||
### `adr_0106_refusal_recorded`
|
||||
|
||||
The `expert_demo_reason` field for the math row must name at least one
|
||||
of the documented blockers (missing metric keys or
|
||||
`inference_closure` failure). This is informational; it gives operators
|
||||
a one-line legible reason without forcing them to read this ADR.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
Accepted when:
|
||||
|
||||
- this ADR lands at `Accepted (decision: defer promotion)`
|
||||
- README "Current frontier" updated: ADR-0107 moves out of the
|
||||
Proposed-ADR sequencing list; ADR-0109 and ADR-0110 enter as reserved
|
||||
- `tests/test_adr_0107_deferral.py` pins `adr_0107_no_silent_promotion`
|
||||
against the live ledger
|
||||
- no change to `core/capability/expert_demo.py` or
|
||||
`core/capability/reporting.py` — the contract is honored, not amended
|
||||
|
||||
---
|
||||
|
||||
## Consequences
|
||||
|
||||
- The ADR-0106 contract has demonstrated its load-bearing behavior: it
|
||||
refused a promotion attempt that the previous code-path would have
|
||||
silently failed-true on cognition metrics the math domain never
|
||||
produced.
|
||||
- "`reasoning-capable`" remains an honest ceiling for
|
||||
`mathematics_logic`. External readers can see in `ledger_report()`
|
||||
that the row sits at `reasoning-capable` with a named blocker.
|
||||
- The frontier shifts: ADR-0109 (metric-shape amendment) becomes the
|
||||
prerequisite work for any future expert-demo promotion across all
|
||||
four ratified domains, not just math.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- This ADR does not amend ADR-0106. The amendment is ADR-0109's scope.
|
||||
- This ADR does not investigate or fix `inference_closure`. That work
|
||||
is a prerequisite to ADR-0110, scoped to wherever the substrate gap
|
||||
actually lives.
|
||||
- This ADR does not change the ratification status of `physics`,
|
||||
`systems_software`, or `hebrew_greek_textual_reasoning`. Each will
|
||||
face the same metric-shape question and should be re-evaluated under
|
||||
ADR-0109 when that lands.
|
||||
|
|
@ -26,6 +26,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0104](ADR-0104-curriculum-sourced-teaching-proposals.md) | Curriculum-Sourced Teaching Proposals | Accepted (2026-05-22) |
|
||||
| [ADR-0105](ADR-0105-sealed-holdout-encryption.md) | Sealed Holdout Encryption via age | Accepted (2026-05-22) |
|
||||
| [ADR-0106](ADR-0106-expert-demo-promotion-contract.md) | Expert-Demo Promotion Contract | Accepted (2026-05-22) |
|
||||
| [ADR-0107](ADR-0107-mathematics-logic-expert-demo-deferred.md) | `mathematics_logic` Expert-Demo Promotion — Deferred | Accepted (2026-05-22) |
|
||||
| [ADR-0108](ADR-0108-proposed-adr-sequencing.md) | Proposed-ADR Sequencing Post-ADR-0105 | Accepted (2026-05-22) |
|
||||
|
||||
---
|
||||
|
|
@ -64,10 +65,11 @@ Seven lanes are SHA-pinned in `scripts/verify_lane_shas.py` and gated by the `la
|
|||
|
||||
Sequencing per ADR-0108. Listed in priority order:
|
||||
|
||||
1. **ADR-0107 (reserved) — `mathematics_logic` expert-demo promotion.** First worked promotion against the now-Accepted ADR-0106 contract. Smallest expert-demo proof surface across the four ratified domains.
|
||||
2. **[ADR-0080](ADR-0080-contemplation-loop.md) — Contemplation Loop.** Sandboxed, read-only Phase 1 self-interrogation; emits `SPECULATIVE` findings from `frontier_compare` reports. Converts gap-finding from human-driven to system-emitted-and-reviewed.
|
||||
3. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred pending ADR-0107 — value surfaces during a worked expert promotion that needs definitional depth.
|
||||
4. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
|
||||
1. **ADR-0109 (reserved) — ADR-0106 lane-shape-aware threshold amendment.** ADR-0107 surfaced that ADR-0106 §1.2 prescribes cognition-pack-shape metrics uniformly, but math/physics/systems/hebrew-greek lanes carry their own native shapes (`accuracy`, `passed_rate`, `all_pass_rate`). Prerequisite to any future expert-demo promotion.
|
||||
2. **ADR-0110 (reserved) — `mathematics_logic` expert-demo re-attempt.** Conditional on ADR-0109 landing AND `inference_closure` substantively passing (currently `all_pass_rate=0.4` on public).
|
||||
3. **[ADR-0080](ADR-0080-contemplation-loop.md) — Contemplation Loop.** Sandboxed, read-only Phase 1 self-interrogation; emits `SPECULATIVE` findings from `frontier_compare` reports. Converts gap-finding from human-driven to system-emitted-and-reviewed.
|
||||
4. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred — value surfaces during a worked expert promotion that needs definitional depth.
|
||||
5. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
|
||||
|
||||
### Open candidate directions (no ADR yet)
|
||||
|
||||
|
|
@ -77,7 +79,7 @@ Sequencing per ADR-0108. Listed in priority order:
|
|||
|
||||
## Accepted reasoning-capable domains
|
||||
|
||||
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. All four currently-ratified domains sit at `expert_demo=false` until a worked promotion ADR (ADR-0107+) lands.
|
||||
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion (`mathematics_logic`) and the contract refused — recording two evidence-side blockers now reserved for follow-up as ADR-0109 (lane-shape amendment) and ADR-0110 (re-attempt). All four ratified domains sit at `expert_demo=false`.
|
||||
|
||||
| Domain | Ratification ADR | Pack(s) | Evidence summary |
|
||||
|---|---|---|---|
|
||||
|
|
|
|||
49
tests/test_adr_0107_deferral.py
Normal file
49
tests/test_adr_0107_deferral.py
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
"""ADR-0107 — `mathematics_logic` expert-demo deferral invariant.
|
||||
|
||||
Pins `adr_0107_no_silent_promotion`: the ADR-0107 decision is *defer*,
|
||||
not promote. If a later change silently flips the math row to
|
||||
`expert_demo=true` without ADR-0109 + ADR-0110 landing first, this gate
|
||||
must fail.
|
||||
|
||||
This test reads the *live* ledger, not a fixture. A green test asserts
|
||||
that the contract is behaving as ADR-0107 records: math sits at
|
||||
`reasoning-capable`, no `expert_demo_claims` entry for math is present,
|
||||
and the row carries a named refusal reason.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from core.capability.reporting import ledger_report
|
||||
from core.capability.reviewers import load_reviewer_registry
|
||||
from core.capability.sources import LEDGER_SOURCES
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def _math_row() -> dict:
|
||||
report = ledger_report()
|
||||
for row in report["domains"]:
|
||||
if row["domain"] == "mathematics_logic":
|
||||
return row
|
||||
raise AssertionError("mathematics_logic row missing from ledger_report()")
|
||||
|
||||
|
||||
class TestAdr0107NoSilentPromotion:
|
||||
def test_math_row_is_reasoning_capable_not_expert_demo(self) -> None:
|
||||
row = _math_row()
|
||||
assert row["predicates"]["reasoning_capable"] is True
|
||||
assert row["predicates"]["expert_demo"] is False
|
||||
assert row["status"] == "reasoning-capable"
|
||||
|
||||
def test_math_has_no_expert_demo_claim(self) -> None:
|
||||
registry_path = _REPO_ROOT / LEDGER_SOURCES.reviewers
|
||||
registry = load_reviewer_registry(registry_path)
|
||||
assert registry.expert_demo_claim_for("mathematics_logic") is None
|
||||
|
||||
def test_refusal_reason_is_named(self) -> None:
|
||||
row = _math_row()
|
||||
reason = row.get("expert_demo_reason", "")
|
||||
assert reason, "expert_demo_reason must be populated"
|
||||
assert reason != "all expert-demo predicates satisfied"
|
||||
Loading…
Reference in a new issue