docs(claims): ADR-0200 reconciliation — expert claim to audit-passed truth
Reconcile every artifact that asserted the (since auto-reverted) mathematics_logic expert promotion to the live machine state. Determinism proven intact (Week-1a): the digest divergence is genuine single-source evidence-drift (GSM8K coverage probe 3/47 -> 4/46 via #310/#488), not a non-determinism defect. ADR-0120's fail-closed property fired as designed; CORE revoked its own expert claim. History keeps receipts; current-state reconciles to truth: - Regenerate expert_claims_math_v1_signed.json -> promote_admitted:false, reviewer_signature_matches:false, digest 02f6d3c8. - reviewers.yaml math_expert_claims: quarantine note; entry kept (mismatch-refusal keeps firing); intentionally NOT re-signed. - ADR-0120-math-expert-ledger-flip: dated valid-at/auto-reverted header note. - README: "next gate" narrative -> built-attempted-reverted; refresh stale count. - docs/decisions/README: revert note + ADR-0200 index row. - 3 fail-closed tests (2 files): "is-expert" -> fail-closed-revert assertions. Were RED on main; now green (30 passed). No eval gate, threshold, or safety boundary changed.
This commit is contained in:
Shay
parent
96b6c5e4b2
commit
2d18976fa4
7 changed files with 66 additions and 28 deletions
17
README.md
17
README.md
|
|
@ -78,7 +78,7 @@ English establishes the operational base. Hebrew and Koine Greek bring the hidde
|
|||
```bash
|
||||
pip install -e ".[dev]"
|
||||
pytest tests/test_versor_closure.py # the core invariant — must pass first
|
||||
pytest tests/ # full suite (~4 minutes, 1099 tests)
|
||||
pytest tests/ # full suite (~8,337 tests; some pre-existing reds — see docs/test-debt-quarantine.md)
|
||||
```
|
||||
|
||||
### Watch the flywheel turn — one command
|
||||
|
|
@ -308,11 +308,16 @@ ADR-0114a's 10 anti-overfitting proof obligations are all discharged for the
|
|||
it does not confabulate. The zero-confabulation property holds against the external
|
||||
benchmark.
|
||||
|
||||
**ADR-0120 (first `expert` promotion contract) is the next gate.** It will set the
|
||||
numeric expert threshold and ε, require all 10 ADR-0114a obligations as hard gates,
|
||||
and sign the first `expert_claims` entry — or defer honestly if the correct_rate
|
||||
gate is not yet met. **No domain is at `expert` today.** That status string remains
|
||||
reserved namespace.
|
||||
**ADR-0120 (the first `expert` promotion contract) has since been built and
|
||||
exercised.** On 2026-05-23 `mathematics_logic` was signed and briefly promoted to
|
||||
`expert` — then **auto-reverted to `audit-passed`** when its evidence bundle
|
||||
drifted: a *non-gating* GSM8K coverage metric moved, which changed the
|
||||
evidence-derived digest and invalidated the signature. That revert is the
|
||||
contract's fail-closed property working as designed — CORE revoked its own expert
|
||||
claim rather than carry a stale one. **No domain is at `expert` today**, and when
|
||||
`expert` is held at all it rests on CORE-authored lanes, not external GSM8K. Full
|
||||
record: [ADR-0200](docs/decisions/ADR-0200-expert-claim-reconciliation.md) and
|
||||
[`docs/claims_ledger.md`](docs/claims_ledger.md).
|
||||
|
||||
To run the GSM8K math eval lane:
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,18 @@
|
|||
ADR-0114a.{1,2,3,4,5,6,7,8,9,10} (obligation auditors)
|
||||
**Predecessor PRs:** #173, #176-#180, #182-#194
|
||||
|
||||
> **Reconciliation note (ADR-0200, 2026-06-02).** This flip was **valid when it
|
||||
> was made** (2026-05-23 / #267): code + evidence reproduced the signed digest
|
||||
> `4c46f530...` exactly, and the empirical verdict below was correct *at that
|
||||
> commit*. It has since **auto-reverted to `audit-passed`.** The GSM8K coverage
|
||||
> probe drifted (`3/47 -> 4/46`, PRs #310/#488), changing the evidence-derived
|
||||
> digest to `02f6d3c8...`; the signature no longer matches and the composer
|
||||
> refuses — ADR-0120's documented fail-closed property firing as designed.
|
||||
> **Current ledger status: `audit-passed`.** The empirical block below is
|
||||
> preserved as the historical record. See
|
||||
> [ADR-0200](ADR-0200-expert-claim-reconciliation.md) and the
|
||||
> [claims ledger](../claims_ledger.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
|
|
|||
|
|
@ -143,7 +143,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0119.7](ADR-0119.7-sealed-gsm8k-test.md) | Sealed GSM8K Test Set as gsm8k_math Holdout | Accepted |
|
||||
| [ADR-0119.8](ADR-0119.8-lane-gate.md) | gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) | Accepted |
|
||||
| [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract | Proposed (contract-only; no domain promoted with this ADR) |
|
||||
| [ADR-0120](ADR-0120-math-expert-ledger-flip.md) | ADR-0120 (math, ledger flip) — Mathematics-Logic Domain Promoted to `expert` | Accepted — first `expert`-tier domain in the capability ledger |
|
||||
| [ADR-0120](ADR-0120-math-expert-ledger-flip.md) | ADR-0120 (math, ledger flip) — Mathematics-Logic Domain Promoted to `expert` | Accepted 2026-05-23; **auto-reverted to `audit-passed`** when evidence drifted — current status audit-passed (see [ADR-0200](ADR-0200-expert-claim-reconciliation.md)) |
|
||||
| [ADR-0120](ADR-0120-math-expert-promotion-wireup.md) | ADR-0120 (math) — Math-Expert Promotion Composer Wire-Up | Accepted (technical pass on first evaluation; awaiting reviewer signature for ledger admission) |
|
||||
| [ADR-0121](ADR-0121-mathematics-logic-expert-deferred.md) | `mathematics_logic` `expert` Promotion — Deferred (first attempt) | Accepted (the deferral is the decision) |
|
||||
| [ADR-0122](ADR-0122-parser-rate-per-unit.md) | Parser Expansion: Rate / Per-Unit Reasoning (substrate-only; lift deferred) | Accepted (substrate landed; sealed-lift gate deferred — the |
|
||||
|
|
@ -252,6 +252,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0197](ADR-0197-vision-compiler-delta-crdt.md) | CORE-native Vision Compiler over the Delta-CRDT Substrate | Proposed |
|
||||
| [ADR-0198](ADR-0198-motor-efferent-decoder-spike.md) | Motor as Efferent Modality — Protocol Gap & Governance (Design Spike) | Proposed (design spike — no implementation) |
|
||||
| [ADR-0199](ADR-0199-cross-domain-learning-arena-contract.md) | Cross-Domain Learning Arena Contract | Proposed |
|
||||
| [ADR-0200](ADR-0200-expert-claim-reconciliation.md) | Expert-Claim Reconciliation: record the math fail-closed revert as designed behavior | Proposed |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -51,6 +51,20 @@ audit_passed_claims:
|
|||
evidence_revision: "adr-0124:reviewed:2026-05-22"
|
||||
signed_by: shay-j
|
||||
claim_digest: "17e24436b6875b89f6d1a5c2992557413c7ef456250f549d463159f54438c407"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QUARANTINED (ADR-0200, 2026-06-02). The math_expert_claims entry below was a
|
||||
# VALID signature when made (2026-05-23 / #267): code + evidence reproduced
|
||||
# claim_digest 4c46f530... exactly. It is RETAINED as a historical receipt and
|
||||
# is intentionally NOT re-signed. It no longer validates: the GSM8K coverage
|
||||
# probe drifted (3/47 -> 4/46, PRs #310/#488), so the evidence-derived digest
|
||||
# is now 02f6d3c8... and the expert composer REFUSES promotion. The live ledger
|
||||
# reports mathematics_logic = audit-passed. Keeping this mismatch in place is
|
||||
# ADR-0120's fail-closed property firing by design. DO NOT re-sign to force the
|
||||
# ledger flip; the honest current status is audit-passed (expert rests on
|
||||
# CORE-authored lanes, not external GSM8K).
|
||||
# See docs/decisions/ADR-0200-expert-claim-reconciliation.md.
|
||||
# ---------------------------------------------------------------------------
|
||||
math_expert_claims:
|
||||
- domain_id: mathematics_logic
|
||||
evidence_lanes:
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"adr": "0120-math",
|
||||
"all_obligations_passed": true,
|
||||
"claim_digest": "4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4",
|
||||
"claim_digest": "02f6d3c8803b0a3154b84e1ba6a292556ae6a7ed4765507e617576fe342a9c94",
|
||||
"composite_gate_passed": true,
|
||||
"composite_gate_refusal": "",
|
||||
"domain": "mathematics_logic",
|
||||
|
|
@ -77,8 +77,8 @@
|
|||
"title": "operation provenance via pack"
|
||||
}
|
||||
],
|
||||
"promote_admitted": true,
|
||||
"refusal_reason": "",
|
||||
"promote_admitted": false,
|
||||
"refusal_reason": "reviewer claim_digest mismatch \u2014 registry has '4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4', evidence-derived digest is '02f6d3c8803b0a3154b84e1ba6a292556ae6a7ed4765507e617576fe342a9c94'; the evidence bundle has changed since the signature was added",
|
||||
"reviewer_signature": {
|
||||
"claim_digest": "4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4",
|
||||
"domain_id": "mathematics_logic",
|
||||
|
|
@ -91,7 +91,7 @@
|
|||
"evidence_revision": "adr-0120-math:reviewed:2026-05-23",
|
||||
"signed_by": "shay-j"
|
||||
},
|
||||
"reviewer_signature_matches": true,
|
||||
"reviewer_signature_matches": false,
|
||||
"schema_version": 1,
|
||||
"technical_pass": true
|
||||
}
|
||||
|
|
|
|||
|
|
@ -159,11 +159,13 @@ def test_obligation_9_refuses_when_a_lane_is_invalid_json(tmp_path: Path) -> Non
|
|||
|
||||
|
||||
def test_composer_runs_on_current_main_with_all_obligations_passing() -> None:
|
||||
"""The load-bearing snapshot. Updated by the ledger-flip PR after
|
||||
the reviewer added the signed entry to ``docs/reviewers.yaml`` —
|
||||
every obligation auditor passes, the composite gate passes,
|
||||
technical_pass is True, the reviewer signature is present and
|
||||
matches the computed digest, so promote_admitted is True.
|
||||
"""Fail-closed revert snapshot (ADR-0200). On current main every
|
||||
obligation auditor passes, the composite gate passes, technical_pass is
|
||||
True, and the reviewer signature is present — but a non-gating GSM8K
|
||||
coverage metric drifted after signing, so the signature no longer matches
|
||||
the evidence-derived digest. The composer therefore refuses:
|
||||
promote_admitted is False with a digest-mismatch reason. This is
|
||||
ADR-0120's load-bearing fail-closed safety property.
|
||||
"""
|
||||
v = evaluate_math_expert_promotion()
|
||||
assert v.all_obligations_passed is True, (
|
||||
|
|
@ -174,9 +176,9 @@ def test_composer_runs_on_current_main_with_all_obligations_passing() -> None:
|
|||
assert v.technical_pass is True
|
||||
assert v.reviewer_signature is not None
|
||||
assert v.reviewer_signature.get("signed_by") == "shay-j"
|
||||
assert v.reviewer_signature_matches is True
|
||||
assert v.promote_admitted is True
|
||||
assert v.refusal_reason == ""
|
||||
assert v.reviewer_signature_matches is False
|
||||
assert v.promote_admitted is False
|
||||
assert "mismatch" in v.refusal_reason.lower()
|
||||
assert v.claim_digest # non-empty
|
||||
assert len(v.claim_digest) == 64 # SHA-256 hex
|
||||
|
||||
|
|
|
|||
|
|
@ -125,19 +125,21 @@ def _math_row(report: dict) -> dict:
|
|||
raise AssertionError("mathematics_logic row missing from ledger")
|
||||
|
||||
|
||||
def test_mathlogic_status_is_expert_with_signed_evidence() -> None:
|
||||
"""The load-bearing snapshot. Given the signed entry in
|
||||
docs/reviewers.yaml AND every obligation auditor passing AND the
|
||||
composite gate passing, the ledger reports
|
||||
``mathematics_logic.status == "expert"``."""
|
||||
def test_mathlogic_status_is_audit_passed_after_evidence_drift() -> None:
|
||||
"""Fail-closed revert snapshot (ADR-0200). The signed expert entry in
|
||||
docs/reviewers.yaml no longer re-derives — a non-gating GSM8K coverage
|
||||
metric drifted after signing — so the composer refuses and the ledger
|
||||
reports ``mathematics_logic.status == "audit-passed"``, not "expert".
|
||||
This is ADR-0120's fail-closed property firing by design."""
|
||||
report = ledger_report()
|
||||
row = _math_row(report)
|
||||
assert row["status"] == "expert", (
|
||||
f"expected status=expert; got {row['status']!r}; "
|
||||
assert row["status"] == "audit-passed", (
|
||||
f"expected status=audit-passed (expert fail-closed-reverted); "
|
||||
f"got {row['status']!r}; "
|
||||
f"audit_passed_reason={row.get('audit_passed_reason')}; "
|
||||
f"expert_reason={row.get('expert_reason')}"
|
||||
)
|
||||
assert row["predicates"]["expert"] is True
|
||||
assert row["predicates"]["expert"] is False
|
||||
assert row["predicates"]["audit_passed"] is True
|
||||
|
||||
|
||||
|
|
@ -145,7 +147,9 @@ def test_mathlogic_row_carries_expert_reason() -> None:
|
|||
report = ledger_report()
|
||||
row = _math_row(report)
|
||||
assert "expert_reason" in row
|
||||
assert "admitted" in row["expert_reason"].lower()
|
||||
# Fail-closed revert (ADR-0200): the reason explains the digest mismatch,
|
||||
# not an admission.
|
||||
assert "mismatch" in row["expert_reason"].lower()
|
||||
|
||||
|
||||
def test_other_domains_have_expert_false_no_composer_wired() -> None:
|
||||
|
|
|
|||
Loading…
Reference in a new issue