From 2d18976fa4dff3adf45f362480c9331b0cc945ef Mon Sep 17 00:00:00 2001 From: Shay Date: Tue, 2 Jun 2026 10:06:16 -0700 Subject: [PATCH] =?UTF-8?q?docs(claims):=20ADR-0200=20reconciliation=20?= =?UTF-8?q?=E2=80=94=20expert=20claim=20to=20audit-passed=20truth?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconcile every artifact that asserted the (since auto-reverted) mathematics_logic expert promotion to the live machine state. Determinism proven intact (Week-1a): the digest divergence is genuine single-source evidence-drift (GSM8K coverage probe 3/47 -> 4/46 via #310/#488), not a non-determinism defect. ADR-0120's fail-closed property fired as designed; CORE revoked its own expert claim. History keeps receipts; current-state reconciles to truth: - Regenerate expert_claims_math_v1_signed.json -> promote_admitted:false, reviewer_signature_matches:false, digest 02f6d3c8. - reviewers.yaml math_expert_claims: quarantine note; entry kept (mismatch-refusal keeps firing); intentionally NOT re-signed. - ADR-0120-math-expert-ledger-flip: dated valid-at/auto-reverted header note. - README: "next gate" narrative -> built-attempted-reverted; refresh stale count. - docs/decisions/README: revert note + ADR-0200 index row. - 3 fail-closed tests (2 files): "is-expert" -> fail-closed-revert assertions. Were RED on main; now green (30 passed). No eval gate, threshold, or safety boundary changed. --- README.md | 17 +++++++++----- .../ADR-0120-math-expert-ledger-flip.md | 12 ++++++++++ docs/decisions/README.md | 3 ++- docs/reviewers.yaml | 14 ++++++++++++ .../v1/expert_claims_math_v1_signed.json | 8 +++---- tests/test_adr_0120_math_expert_promotion.py | 18 ++++++++------- tests/test_mathlogic_expert_ledger_flip.py | 22 +++++++++++-------- 7 files changed, 66 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index 158beb88..17262ec4 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ English establishes the operational base. Hebrew and Koine Greek bring the hidde ```bash pip install -e ".[dev]" pytest tests/test_versor_closure.py # the core invariant — must pass first -pytest tests/ # full suite (~4 minutes, 1099 tests) +pytest tests/ # full suite (~8,337 tests; some pre-existing reds — see docs/test-debt-quarantine.md) ``` ### Watch the flywheel turn — one command @@ -308,11 +308,16 @@ ADR-0114a's 10 anti-overfitting proof obligations are all discharged for the it does not confabulate. The zero-confabulation property holds against the external benchmark. -**ADR-0120 (first `expert` promotion contract) is the next gate.** It will set the -numeric expert threshold and ε, require all 10 ADR-0114a obligations as hard gates, -and sign the first `expert_claims` entry — or defer honestly if the correct_rate -gate is not yet met. **No domain is at `expert` today.** That status string remains -reserved namespace. +**ADR-0120 (the first `expert` promotion contract) has since been built and +exercised.** On 2026-05-23 `mathematics_logic` was signed and briefly promoted to +`expert` — then **auto-reverted to `audit-passed`** when its evidence bundle +drifted: a *non-gating* GSM8K coverage metric moved, which changed the +evidence-derived digest and invalidated the signature. That revert is the +contract's fail-closed property working as designed — CORE revoked its own expert +claim rather than carry a stale one. **No domain is at `expert` today**, and when +`expert` is held at all it rests on CORE-authored lanes, not external GSM8K. Full +record: [ADR-0200](docs/decisions/ADR-0200-expert-claim-reconciliation.md) and +[`docs/claims_ledger.md`](docs/claims_ledger.md). To run the GSM8K math eval lane: diff --git a/docs/decisions/ADR-0120-math-expert-ledger-flip.md b/docs/decisions/ADR-0120-math-expert-ledger-flip.md index 1b04a07c..20c8f19e 100644 --- a/docs/decisions/ADR-0120-math-expert-ledger-flip.md +++ b/docs/decisions/ADR-0120-math-expert-ledger-flip.md @@ -8,6 +8,18 @@ ADR-0114a.{1,2,3,4,5,6,7,8,9,10} (obligation auditors) **Predecessor PRs:** #173, #176-#180, #182-#194 +> **Reconciliation note (ADR-0200, 2026-06-02).** This flip was **valid when it +> was made** (2026-05-23 / #267): code + evidence reproduced the signed digest +> `4c46f530...` exactly, and the empirical verdict below was correct *at that +> commit*. It has since **auto-reverted to `audit-passed`.** The GSM8K coverage +> probe drifted (`3/47 -> 4/46`, PRs #310/#488), changing the evidence-derived +> digest to `02f6d3c8...`; the signature no longer matches and the composer +> refuses — ADR-0120's documented fail-closed property firing as designed. +> **Current ledger status: `audit-passed`.** The empirical block below is +> preserved as the historical record. See +> [ADR-0200](ADR-0200-expert-claim-reconciliation.md) and the +> [claims ledger](../claims_ledger.md). + --- ## Context diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 22899b85..ec01ceb8 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -143,7 +143,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0119.7](ADR-0119.7-sealed-gsm8k-test.md) | Sealed GSM8K Test Set as gsm8k_math Holdout | Accepted | | [ADR-0119.8](ADR-0119.8-lane-gate.md) | gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) | Accepted | | [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract | Proposed (contract-only; no domain promoted with this ADR) | -| [ADR-0120](ADR-0120-math-expert-ledger-flip.md) | ADR-0120 (math, ledger flip) — Mathematics-Logic Domain Promoted to `expert` | Accepted — first `expert`-tier domain in the capability ledger | +| [ADR-0120](ADR-0120-math-expert-ledger-flip.md) | ADR-0120 (math, ledger flip) — Mathematics-Logic Domain Promoted to `expert` | Accepted 2026-05-23; **auto-reverted to `audit-passed`** when evidence drifted — current status audit-passed (see [ADR-0200](ADR-0200-expert-claim-reconciliation.md)) | | [ADR-0120](ADR-0120-math-expert-promotion-wireup.md) | ADR-0120 (math) — Math-Expert Promotion Composer Wire-Up | Accepted (technical pass on first evaluation; awaiting reviewer signature for ledger admission) | | [ADR-0121](ADR-0121-mathematics-logic-expert-deferred.md) | `mathematics_logic` `expert` Promotion — Deferred (first attempt) | Accepted (the deferral is the decision) | | [ADR-0122](ADR-0122-parser-rate-per-unit.md) | Parser Expansion: Rate / Per-Unit Reasoning (substrate-only; lift deferred) | Accepted (substrate landed; sealed-lift gate deferred — the | @@ -252,6 +252,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt | [ADR-0197](ADR-0197-vision-compiler-delta-crdt.md) | CORE-native Vision Compiler over the Delta-CRDT Substrate | Proposed | | [ADR-0198](ADR-0198-motor-efferent-decoder-spike.md) | Motor as Efferent Modality — Protocol Gap & Governance (Design Spike) | Proposed (design spike — no implementation) | | [ADR-0199](ADR-0199-cross-domain-learning-arena-contract.md) | Cross-Domain Learning Arena Contract | Proposed | +| [ADR-0200](ADR-0200-expert-claim-reconciliation.md) | Expert-Claim Reconciliation: record the math fail-closed revert as designed behavior | Proposed | --- diff --git a/docs/reviewers.yaml b/docs/reviewers.yaml index 4ea0ddf2..8a235912 100644 --- a/docs/reviewers.yaml +++ b/docs/reviewers.yaml @@ -51,6 +51,20 @@ audit_passed_claims: evidence_revision: "adr-0124:reviewed:2026-05-22" signed_by: shay-j claim_digest: "17e24436b6875b89f6d1a5c2992557413c7ef456250f549d463159f54438c407" + +# --------------------------------------------------------------------------- +# QUARANTINED (ADR-0200, 2026-06-02). The math_expert_claims entry below was a +# VALID signature when made (2026-05-23 / #267): code + evidence reproduced +# claim_digest 4c46f530... exactly. It is RETAINED as a historical receipt and +# is intentionally NOT re-signed. It no longer validates: the GSM8K coverage +# probe drifted (3/47 -> 4/46, PRs #310/#488), so the evidence-derived digest +# is now 02f6d3c8... and the expert composer REFUSES promotion. The live ledger +# reports mathematics_logic = audit-passed. Keeping this mismatch in place is +# ADR-0120's fail-closed property firing by design. DO NOT re-sign to force the +# ledger flip; the honest current status is audit-passed (expert rests on +# CORE-authored lanes, not external GSM8K). +# See docs/decisions/ADR-0200-expert-claim-reconciliation.md. +# --------------------------------------------------------------------------- math_expert_claims: - domain_id: mathematics_logic evidence_lanes: diff --git a/evals/math_expert_claims/v1/expert_claims_math_v1_signed.json b/evals/math_expert_claims/v1/expert_claims_math_v1_signed.json index 3696ce84..873e8af3 100644 --- a/evals/math_expert_claims/v1/expert_claims_math_v1_signed.json +++ b/evals/math_expert_claims/v1/expert_claims_math_v1_signed.json @@ -1,7 +1,7 @@ { "adr": "0120-math", "all_obligations_passed": true, - "claim_digest": "4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4", + "claim_digest": "02f6d3c8803b0a3154b84e1ba6a292556ae6a7ed4765507e617576fe342a9c94", "composite_gate_passed": true, "composite_gate_refusal": "", "domain": "mathematics_logic", @@ -77,8 +77,8 @@ "title": "operation provenance via pack" } ], - "promote_admitted": true, - "refusal_reason": "", + "promote_admitted": false, + "refusal_reason": "reviewer claim_digest mismatch \u2014 registry has '4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4', evidence-derived digest is '02f6d3c8803b0a3154b84e1ba6a292556ae6a7ed4765507e617576fe342a9c94'; the evidence bundle has changed since the signature was added", "reviewer_signature": { "claim_digest": "4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4", "domain_id": "mathematics_logic", @@ -91,7 +91,7 @@ "evidence_revision": "adr-0120-math:reviewed:2026-05-23", "signed_by": "shay-j" }, - "reviewer_signature_matches": true, + "reviewer_signature_matches": false, "schema_version": 1, "technical_pass": true } diff --git a/tests/test_adr_0120_math_expert_promotion.py b/tests/test_adr_0120_math_expert_promotion.py index 3e521aa0..70a10396 100644 --- a/tests/test_adr_0120_math_expert_promotion.py +++ b/tests/test_adr_0120_math_expert_promotion.py @@ -159,11 +159,13 @@ def test_obligation_9_refuses_when_a_lane_is_invalid_json(tmp_path: Path) -> Non def test_composer_runs_on_current_main_with_all_obligations_passing() -> None: - """The load-bearing snapshot. Updated by the ledger-flip PR after - the reviewer added the signed entry to ``docs/reviewers.yaml`` — - every obligation auditor passes, the composite gate passes, - technical_pass is True, the reviewer signature is present and - matches the computed digest, so promote_admitted is True. + """Fail-closed revert snapshot (ADR-0200). On current main every + obligation auditor passes, the composite gate passes, technical_pass is + True, and the reviewer signature is present — but a non-gating GSM8K + coverage metric drifted after signing, so the signature no longer matches + the evidence-derived digest. The composer therefore refuses: + promote_admitted is False with a digest-mismatch reason. This is + ADR-0120's load-bearing fail-closed safety property. """ v = evaluate_math_expert_promotion() assert v.all_obligations_passed is True, ( @@ -174,9 +176,9 @@ def test_composer_runs_on_current_main_with_all_obligations_passing() -> None: assert v.technical_pass is True assert v.reviewer_signature is not None assert v.reviewer_signature.get("signed_by") == "shay-j" - assert v.reviewer_signature_matches is True - assert v.promote_admitted is True - assert v.refusal_reason == "" + assert v.reviewer_signature_matches is False + assert v.promote_admitted is False + assert "mismatch" in v.refusal_reason.lower() assert v.claim_digest # non-empty assert len(v.claim_digest) == 64 # SHA-256 hex diff --git a/tests/test_mathlogic_expert_ledger_flip.py b/tests/test_mathlogic_expert_ledger_flip.py index a5ebb437..de855e8e 100644 --- a/tests/test_mathlogic_expert_ledger_flip.py +++ b/tests/test_mathlogic_expert_ledger_flip.py @@ -125,19 +125,21 @@ def _math_row(report: dict) -> dict: raise AssertionError("mathematics_logic row missing from ledger") -def test_mathlogic_status_is_expert_with_signed_evidence() -> None: - """The load-bearing snapshot. Given the signed entry in - docs/reviewers.yaml AND every obligation auditor passing AND the - composite gate passing, the ledger reports - ``mathematics_logic.status == "expert"``.""" +def test_mathlogic_status_is_audit_passed_after_evidence_drift() -> None: + """Fail-closed revert snapshot (ADR-0200). The signed expert entry in + docs/reviewers.yaml no longer re-derives — a non-gating GSM8K coverage + metric drifted after signing — so the composer refuses and the ledger + reports ``mathematics_logic.status == "audit-passed"``, not "expert". + This is ADR-0120's fail-closed property firing by design.""" report = ledger_report() row = _math_row(report) - assert row["status"] == "expert", ( - f"expected status=expert; got {row['status']!r}; " + assert row["status"] == "audit-passed", ( + f"expected status=audit-passed (expert fail-closed-reverted); " + f"got {row['status']!r}; " f"audit_passed_reason={row.get('audit_passed_reason')}; " f"expert_reason={row.get('expert_reason')}" ) - assert row["predicates"]["expert"] is True + assert row["predicates"]["expert"] is False assert row["predicates"]["audit_passed"] is True @@ -145,7 +147,9 @@ def test_mathlogic_row_carries_expert_reason() -> None: report = ledger_report() row = _math_row(report) assert "expert_reason" in row - assert "admitted" in row["expert_reason"].lower() + # Fail-closed revert (ADR-0200): the reason explains the digest mismatch, + # not an admission. + assert "mismatch" in row["expert_reason"].lower() def test_other_domains_have_expert_false_no_composer_wired() -> None: