feat(adr): ADR-0107 mathematics_logic expert-demo promotion deferred (#114)
The ADR-0106 contract correctly refused promotion. ADR-0107 records the deferral and reserves two follow-up ADRs: - ADR-0109 (lane-shape-aware threshold amendment): ADR-0106 \xc2\xa71.2 prescribes cognition-pack-shape metrics uniformly, but math / physics / systems / hebrew-greek lanes carry native shapes (accuracy, passed_rate, all_pass_rate). Prerequisite for any future expert-demo promotion. - ADR-0110 (math re-attempt): conditional on ADR-0109 landing and inference_closure substantively passing (currently all_pass_rate=0.4 on public). tests/test_adr_0107_deferral.py pins adr_0107_no_silent_promotion: math stays at reasoning-capable, has no expert_demo_claims entry, and the ledger row carries a named refusal reason. No change to core/capability/expert_demo.py or reporting.py -- the contract is honored, not amended. README sequencing updated to reflect ADR-0107 acceptance and the new ADR-0109/0110 prerequisites.
This commit is contained in:
parent
0493808215
commit
7cc2f7b422
3 changed files with 218 additions and 5 deletions
|
|
@ -0,0 +1,162 @@
|
||||||
|
# ADR-0107 — `mathematics_logic` Expert-Demo Promotion: Deferred
|
||||||
|
|
||||||
|
**Status:** Accepted (decision: defer promotion)
|
||||||
|
**Date:** 2026-05-22
|
||||||
|
**Author:** CORE agents + reviewers
|
||||||
|
**Depends on:** ADR-0091, ADR-0092, ADR-0097, ADR-0106, ADR-0108
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
ADR-0108 reserved ADR-0107 as the first worked expert-demo promotion
|
||||||
|
against the ADR-0106 contract, with `mathematics_logic` named as the
|
||||||
|
smallest expert-demo proof surface across the four ratified domains.
|
||||||
|
|
||||||
|
On evaluation, the ADR-0106 gate correctly refused to promote
|
||||||
|
`mathematics_logic`. ADR-0107 records that refusal honestly and reserves
|
||||||
|
the follow-up ADRs needed before promotion can re-attempt.
|
||||||
|
|
||||||
|
This is the contract working as designed. ADR-0106 §Consequences
|
||||||
|
predicted that "reasoning-capable" should not equal "demonstrated"; the
|
||||||
|
first promotion attempt has now surfaced two specific evidence-side
|
||||||
|
gaps that prove the gap is real.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
`mathematics_logic` attaches three eval lanes via
|
||||||
|
`language_packs/data/en_mathematics_logic_v1/manifest.json`:
|
||||||
|
|
||||||
|
| Lane | Public split | Holdout split | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `elementary_mathematics_ood` | `accuracy=1.0`, all C01–C13 cases pass | `accuracy=1.0`, `passed=39/39` | Reports lane-shape metrics, not cognition-shape metrics |
|
||||||
|
| `inference_closure` | `all_pass_rate=0.4`, `overall_pass=False` | requires `CORE_HOLDOUT_KEY` (sealed under ADR-0105) | Lane is currently below any reasonable promotion threshold |
|
||||||
|
| `fabrication_control` | refusals clean (`fabricated=0` across phantom / cross-pack / sibling-collapse classes) | not formally re-run after ADR-0105 sealing | Required by ADR-0106 §1.3 |
|
||||||
|
|
||||||
|
Neither `elementary_mathematics_ood` nor `inference_closure` nor
|
||||||
|
`fabrication_control` reports the four metric keys ADR-0106 §1.2 prescribes
|
||||||
|
(`surface_groundedness`, `term_capture_rate`, `intent_accuracy`,
|
||||||
|
`versor_closure_rate`). Those keys are cognition-pack-shape; the math
|
||||||
|
lanes carry their own native shapes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
`mathematics_logic` is **not** promoted to `expert_demo=true`. The
|
||||||
|
ledger row stays at `reasoning-capable`. The reviewer registry receives
|
||||||
|
**no** `expert_demo_claims` entry as part of this ADR.
|
||||||
|
|
||||||
|
This is a positive decision recording two concrete blocking gaps:
|
||||||
|
|
||||||
|
### Gap 1 — ADR-0106 metric-shape uniformity assumption
|
||||||
|
|
||||||
|
ADR-0106 §1.2 prescribes cognition-shape thresholds uniformly across
|
||||||
|
every attached lane. In practice, each lane reports its own native
|
||||||
|
metric shape (`accuracy`, `passed_rate`, `all_pass_rate`,
|
||||||
|
`derived_recall_rate`). Enforcing cognition-shape keys uniformly causes
|
||||||
|
every non-cognition lane to fail the gate by absence-of-key, not by
|
||||||
|
substance.
|
||||||
|
|
||||||
|
**Resolution path:** ADR-0109 (reserved) — amend ADR-0106 with a
|
||||||
|
lane-shape adaptation layer that maps each lane's native metric to a
|
||||||
|
contract-bearing threshold. Candidate shapes already observable in the
|
||||||
|
repo:
|
||||||
|
|
||||||
|
- `cognition_shape` — the original four keys, used by the cognition pack
|
||||||
|
- `accuracy_shape` — single `accuracy` key, used by
|
||||||
|
`elementary_mathematics_ood`
|
||||||
|
- `refusal_shape` — `fabricated`/`refused` counts, used by
|
||||||
|
`fabrication_control`
|
||||||
|
- `inference_shape` — `all_pass_rate` + `replay_determinism`, used by
|
||||||
|
`inference_closure`
|
||||||
|
|
||||||
|
ADR-0109 must specify the contract-bearing threshold per shape and pin
|
||||||
|
the mapping so a future lane introduces its shape explicitly rather
|
||||||
|
than silently broadening the gate.
|
||||||
|
|
||||||
|
### Gap 2 — `inference_closure` lane substantively fails
|
||||||
|
|
||||||
|
`inference_closure` reports `all_pass_rate=0.4` on the public v1 split.
|
||||||
|
Whether the failure is a runner bug, a case-set issue, or a genuine
|
||||||
|
substrate gap is unresolved. Either way, the lane is not a positive
|
||||||
|
signal for promotion at present.
|
||||||
|
|
||||||
|
**Resolution path:** Triage `inference_closure` failures (5 of 5 cases
|
||||||
|
on dev, 12 of 20 on public). Until that lane crosses an explicit
|
||||||
|
threshold under the ADR-0109 shape rules, `mathematics_logic` cannot
|
||||||
|
promote regardless of how the metric-shape question is resolved.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Follow-up ADRs reserved
|
||||||
|
|
||||||
|
| ADR | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| **ADR-0109** | Lane-shape-aware threshold rules amending ADR-0106. |
|
||||||
|
| **ADR-0110** | Re-attempt `mathematics_logic` expert-demo promotion under the ADR-0109-amended contract, conditional on `inference_closure` substantively passing. |
|
||||||
|
|
||||||
|
ADR-0107 explicitly does **not** pre-decide ADR-0110's outcome. It only
|
||||||
|
records that the current evidence cannot support promotion.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
### `adr_0107_no_silent_promotion`
|
||||||
|
|
||||||
|
No `expert_demo_claims` entry for `mathematics_logic` exists in
|
||||||
|
`docs/reviewers.yaml` as a result of this ADR. A pytest gate confirms
|
||||||
|
the math row in `ledger_report()` reports
|
||||||
|
`predicates.expert_demo == false`.
|
||||||
|
|
||||||
|
### `adr_0106_refusal_recorded`
|
||||||
|
|
||||||
|
The `expert_demo_reason` field for the math row must name at least one
|
||||||
|
of the documented blockers (missing metric keys or
|
||||||
|
`inference_closure` failure). This is informational; it gives operators
|
||||||
|
a one-line legible reason without forcing them to read this ADR.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acceptance evidence
|
||||||
|
|
||||||
|
Accepted when:
|
||||||
|
|
||||||
|
- this ADR lands at `Accepted (decision: defer promotion)`
|
||||||
|
- README "Current frontier" updated: ADR-0107 moves out of the
|
||||||
|
Proposed-ADR sequencing list; ADR-0109 and ADR-0110 enter as reserved
|
||||||
|
- `tests/test_adr_0107_deferral.py` pins `adr_0107_no_silent_promotion`
|
||||||
|
against the live ledger
|
||||||
|
- no change to `core/capability/expert_demo.py` or
|
||||||
|
`core/capability/reporting.py` — the contract is honored, not amended
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The ADR-0106 contract has demonstrated its load-bearing behavior: it
|
||||||
|
refused a promotion attempt that the previous code-path would have
|
||||||
|
silently failed-true on cognition metrics the math domain never
|
||||||
|
produced.
|
||||||
|
- "`reasoning-capable`" remains an honest ceiling for
|
||||||
|
`mathematics_logic`. External readers can see in `ledger_report()`
|
||||||
|
that the row sits at `reasoning-capable` with a named blocker.
|
||||||
|
- The frontier shifts: ADR-0109 (metric-shape amendment) becomes the
|
||||||
|
prerequisite work for any future expert-demo promotion across all
|
||||||
|
four ratified domains, not just math.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- This ADR does not amend ADR-0106. The amendment is ADR-0109's scope.
|
||||||
|
- This ADR does not investigate or fix `inference_closure`. That work
|
||||||
|
is a prerequisite to ADR-0110, scoped to wherever the substrate gap
|
||||||
|
actually lives.
|
||||||
|
- This ADR does not change the ratification status of `physics`,
|
||||||
|
`systems_software`, or `hebrew_greek_textual_reasoning`. Each will
|
||||||
|
face the same metric-shape question and should be re-evaluated under
|
||||||
|
ADR-0109 when that lands.
|
||||||
|
|
@ -26,6 +26,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
||||||
| [ADR-0104](ADR-0104-curriculum-sourced-teaching-proposals.md) | Curriculum-Sourced Teaching Proposals | Accepted (2026-05-22) |
|
| [ADR-0104](ADR-0104-curriculum-sourced-teaching-proposals.md) | Curriculum-Sourced Teaching Proposals | Accepted (2026-05-22) |
|
||||||
| [ADR-0105](ADR-0105-sealed-holdout-encryption.md) | Sealed Holdout Encryption via age | Accepted (2026-05-22) |
|
| [ADR-0105](ADR-0105-sealed-holdout-encryption.md) | Sealed Holdout Encryption via age | Accepted (2026-05-22) |
|
||||||
| [ADR-0106](ADR-0106-expert-demo-promotion-contract.md) | Expert-Demo Promotion Contract | Accepted (2026-05-22) |
|
| [ADR-0106](ADR-0106-expert-demo-promotion-contract.md) | Expert-Demo Promotion Contract | Accepted (2026-05-22) |
|
||||||
|
| [ADR-0107](ADR-0107-mathematics-logic-expert-demo-deferred.md) | `mathematics_logic` Expert-Demo Promotion — Deferred | Accepted (2026-05-22) |
|
||||||
| [ADR-0108](ADR-0108-proposed-adr-sequencing.md) | Proposed-ADR Sequencing Post-ADR-0105 | Accepted (2026-05-22) |
|
| [ADR-0108](ADR-0108-proposed-adr-sequencing.md) | Proposed-ADR Sequencing Post-ADR-0105 | Accepted (2026-05-22) |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -64,10 +65,11 @@ Seven lanes are SHA-pinned in `scripts/verify_lane_shas.py` and gated by the `la
|
||||||
|
|
||||||
Sequencing per ADR-0108. Listed in priority order:
|
Sequencing per ADR-0108. Listed in priority order:
|
||||||
|
|
||||||
1. **ADR-0107 (reserved) — `mathematics_logic` expert-demo promotion.** First worked promotion against the now-Accepted ADR-0106 contract. Smallest expert-demo proof surface across the four ratified domains.
|
1. **ADR-0109 (reserved) — ADR-0106 lane-shape-aware threshold amendment.** ADR-0107 surfaced that ADR-0106 §1.2 prescribes cognition-pack-shape metrics uniformly, but math/physics/systems/hebrew-greek lanes carry their own native shapes (`accuracy`, `passed_rate`, `all_pass_rate`). Prerequisite to any future expert-demo promotion.
|
||||||
2. **[ADR-0080](ADR-0080-contemplation-loop.md) — Contemplation Loop.** Sandboxed, read-only Phase 1 self-interrogation; emits `SPECULATIVE` findings from `frontier_compare` reports. Converts gap-finding from human-driven to system-emitted-and-reviewed.
|
2. **ADR-0110 (reserved) — `mathematics_logic` expert-demo re-attempt.** Conditional on ADR-0109 landing AND `inference_closure` substantively passing (currently `all_pass_rate=0.4` on public).
|
||||||
3. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred pending ADR-0107 — value surfaces during a worked expert promotion that needs definitional depth.
|
3. **[ADR-0080](ADR-0080-contemplation-loop.md) — Contemplation Loop.** Sandboxed, read-only Phase 1 self-interrogation; emits `SPECULATIVE` findings from `frontier_compare` reports. Converts gap-finding from human-driven to system-emitted-and-reviewed.
|
||||||
4. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
|
4. **[ADR-0084](ADR-0084-definitional-layer.md) — Definitional Layer for Lexicon Packs.** Optional per-entry definitional block. Deferred — value surfaces during a worked expert promotion that needs definitional depth.
|
||||||
|
5. **[ADR-0087](ADR-0087-rhetorical-style-axis.md) — Rhetorical Style Axis.** A third substantive selection axis sibling to anchor-lens. Lowest current priority — no active downstream consumer; register + anchor-lens already demonstrate the orthogonality pattern.
|
||||||
|
|
||||||
### Open candidate directions (no ADR yet)
|
### Open candidate directions (no ADR yet)
|
||||||
|
|
||||||
|
|
@ -77,7 +79,7 @@ Sequencing per ADR-0108. Listed in priority order:
|
||||||
|
|
||||||
## Accepted reasoning-capable domains
|
## Accepted reasoning-capable domains
|
||||||
|
|
||||||
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. All four currently-ratified domains sit at `expert_demo=false` until a worked promotion ADR (ADR-0107+) lands.
|
Per ADR-0106, `expert_demo` is **contract-gated**, not threshold-only: a domain row may carry `expert_demo=true` only when a reviewer-signed `expert_demo_claims` entry exists whose evidence-bundle digest reproduces byte-for-byte. ADR-0107 attempted the first worked promotion (`mathematics_logic`) and the contract refused — recording two evidence-side blockers now reserved for follow-up as ADR-0109 (lane-shape amendment) and ADR-0110 (re-attempt). All four ratified domains sit at `expert_demo=false`.
|
||||||
|
|
||||||
| Domain | Ratification ADR | Pack(s) | Evidence summary |
|
| Domain | Ratification ADR | Pack(s) | Evidence summary |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
|
|
|
||||||
49
tests/test_adr_0107_deferral.py
Normal file
49
tests/test_adr_0107_deferral.py
Normal file
|
|
@ -0,0 +1,49 @@
|
||||||
|
"""ADR-0107 — `mathematics_logic` expert-demo deferral invariant.
|
||||||
|
|
||||||
|
Pins `adr_0107_no_silent_promotion`: the ADR-0107 decision is *defer*,
|
||||||
|
not promote. If a later change silently flips the math row to
|
||||||
|
`expert_demo=true` without ADR-0109 + ADR-0110 landing first, this gate
|
||||||
|
must fail.
|
||||||
|
|
||||||
|
This test reads the *live* ledger, not a fixture. A green test asserts
|
||||||
|
that the contract is behaving as ADR-0107 records: math sits at
|
||||||
|
`reasoning-capable`, no `expert_demo_claims` entry for math is present,
|
||||||
|
and the row carries a named refusal reason.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from core.capability.reporting import ledger_report
|
||||||
|
from core.capability.reviewers import load_reviewer_registry
|
||||||
|
from core.capability.sources import LEDGER_SOURCES
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
|
|
||||||
|
def _math_row() -> dict:
|
||||||
|
report = ledger_report()
|
||||||
|
for row in report["domains"]:
|
||||||
|
if row["domain"] == "mathematics_logic":
|
||||||
|
return row
|
||||||
|
raise AssertionError("mathematics_logic row missing from ledger_report()")
|
||||||
|
|
||||||
|
|
||||||
|
class TestAdr0107NoSilentPromotion:
|
||||||
|
def test_math_row_is_reasoning_capable_not_expert_demo(self) -> None:
|
||||||
|
row = _math_row()
|
||||||
|
assert row["predicates"]["reasoning_capable"] is True
|
||||||
|
assert row["predicates"]["expert_demo"] is False
|
||||||
|
assert row["status"] == "reasoning-capable"
|
||||||
|
|
||||||
|
def test_math_has_no_expert_demo_claim(self) -> None:
|
||||||
|
registry_path = _REPO_ROOT / LEDGER_SOURCES.reviewers
|
||||||
|
registry = load_reviewer_registry(registry_path)
|
||||||
|
assert registry.expert_demo_claim_for("mathematics_logic") is None
|
||||||
|
|
||||||
|
def test_refusal_reason_is_named(self) -> None:
|
||||||
|
row = _math_row()
|
||||||
|
reason = row.get("expert_demo_reason", "")
|
||||||
|
assert reason, "expert_demo_reason must be populated"
|
||||||
|
assert reason != "all expert-demo predicates satisfied"
|
||||||
Loading…
Reference in a new issue