Merge pull request #521 from AssetOverflow/docs/claims-ledger-and-reconciliation
docs(claims): claims ledger + ADR-0200 expert-claim reconciliation
This commit is contained in:
commit
dcb5172f00
9 changed files with 380 additions and 28 deletions
17
README.md
17
README.md
|
|
@ -78,7 +78,7 @@ English establishes the operational base. Hebrew and Koine Greek bring the hidde
|
|||
```bash
|
||||
pip install -e ".[dev]"
|
||||
pytest tests/test_versor_closure.py # the core invariant — must pass first
|
||||
pytest tests/ # full suite (~4 minutes, 1099 tests)
|
||||
pytest tests/ # full suite (~8,337 tests; some pre-existing reds — see docs/test-debt-quarantine.md)
|
||||
```
|
||||
|
||||
### Watch the flywheel turn — one command
|
||||
|
|
@ -308,11 +308,16 @@ ADR-0114a's 10 anti-overfitting proof obligations are all discharged for the
|
|||
it does not confabulate. The zero-confabulation property holds against the external
|
||||
benchmark.
|
||||
|
||||
**ADR-0120 (first `expert` promotion contract) is the next gate.** It will set the
|
||||
numeric expert threshold and ε, require all 10 ADR-0114a obligations as hard gates,
|
||||
and sign the first `expert_claims` entry — or defer honestly if the correct_rate
|
||||
gate is not yet met. **No domain is at `expert` today.** That status string remains
|
||||
reserved namespace.
|
||||
**ADR-0120 (the first `expert` promotion contract) has since been built and
|
||||
exercised.** On 2026-05-23 `mathematics_logic` was signed and briefly promoted to
|
||||
`expert` — then **auto-reverted to `audit-passed`** when its evidence bundle
|
||||
drifted: a *non-gating* GSM8K coverage metric moved, which changed the
|
||||
evidence-derived digest and invalidated the signature. That revert is the
|
||||
contract's fail-closed property working as designed — CORE revoked its own expert
|
||||
claim rather than carry a stale one. **No domain is at `expert` today**, and when
|
||||
`expert` is held at all it rests on CORE-authored lanes, not external GSM8K. Full
|
||||
record: [ADR-0200](docs/decisions/ADR-0200-expert-claim-reconciliation.md) and
|
||||
[`docs/claims_ledger.md`](docs/claims_ledger.md).
|
||||
|
||||
To run the GSM8K math eval lane:
|
||||
|
||||
|
|
|
|||
182
docs/claims_ledger.md
Normal file
182
docs/claims_ledger.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# Claims Ledger — Single Source of Truth
|
||||
|
||||
**Purpose.** Every public or material claim CORE makes — in `README.md`,
|
||||
`CLAIMS.md`, ADRs, eval summaries, and any outward-facing brief or site copy —
|
||||
maps here to its exact in-repo evidence and its *honest* framing. When a number
|
||||
or status appears in any external material, it must reconcile against this file.
|
||||
If a claim cannot be substantiated from the repo, it is softened or cut, not
|
||||
defended.
|
||||
|
||||
> **Discipline:** Honesty is the product. CORE refuses rather than guesses; that
|
||||
> discipline applies to how we describe ourselves, not only to what the engine
|
||||
> outputs. Overclaiming to hit a deadline is failure, not success.
|
||||
|
||||
- **Base commit:** `c058d96` (origin/main).
|
||||
- **Verification date:** 2026-06-02.
|
||||
- **Reproduce the machine-derived rows:**
|
||||
```bash
|
||||
uv run core capability ledger # Tier-1 status (audit-passed ×3; NO expert)
|
||||
uv run python scripts/verify_lane_shas.py # Tier-2 pinned lane SHAs
|
||||
uv run python scripts/generate_claims.py --check # CLAIMS.md is current
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. The Four GSM8K Numbers — never conflate these
|
||||
|
||||
CORE has **four** distinct GSM8K-related measurements. They measure different
|
||||
things on different data, and only **one** is the real external benchmark. Any
|
||||
material that blurs them is a credibility risk.
|
||||
|
||||
| Tag | What it is | Result | Real GSM8K? | Gates expert? | Evidence |
|
||||
|---|---|---|---|---|---|
|
||||
| **A** | **Sealed real GSM8K test** (1,319 cases, HuggingFace `openai/gsm8k` test split) | **0 correct / 0 wrong / 1,319 refused** | **Yes** | No | `ADR-0119.7`; ciphertext `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` |
|
||||
| **B** | **CORE-authored synthetic "public" split** (150 cases, rule-based, written to exercise the grammar) | **150 correct / 0 wrong / 0 refused** (rate 1.0) | **No** | No | `evals/gsm8k_math/baselines/comparison_v1.json`, `frontier.json` |
|
||||
| **C** | **Real `train_sample`** (50-case dev sample of real GSM8K) | **6 correct / 44 refused / 0 wrong** | Yes (sample) | No | `evals/gsm8k_math/train_sample/v1/report.json` |
|
||||
| **D** | **Expert-promotion composite gate** (CORE-authored lanes B1/B2/B3) | **185/185 + 14/14 + 40/40 + 50/50, wrong=0** | **No** | Yes (but reverted — §2) | `ADR-0131.4`; `core/capability/composite_math_gate.py` |
|
||||
|
||||
**Per-number honest framing:**
|
||||
|
||||
- **(A) The honest external number.** `0/0/1319` — CORE's grammar covers **zero**
|
||||
real GSM8K test problems today. This is the truthful gap. The load-bearing
|
||||
property is **`wrong == 0` against the external corpus**: CORE *refuses* what it
|
||||
cannot grammar-handle; it does not confabulate. The seal is one-way
|
||||
(encrypted to an off-repo key); the number is a **recorded measurement**, not
|
||||
CI-reproducible without the holdout key. Never present A as an accuracy
|
||||
achievement — present it as zero-confabulation discipline plus an honest
|
||||
coverage gap.
|
||||
- **(B) A frontier *comparison*, not a benchmark result.** 150/150 is on a split
|
||||
**CORE wrote**, designed to exercise its own grammar without data
|
||||
contamination. `frontier.json` itself flags the "apples-vs-oranges" caveat:
|
||||
frontier LLMs (Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%) are
|
||||
scored on *real* GSM8K. **Never cite B as "CORE scores 100% on GSM8K."** The
|
||||
qualitative differentiator is `wrong=0`, not the rate.
|
||||
- **(C) The dev-sample reality.** 6/44/0 on 50 real cases — the lane's
|
||||
`exit_criterion` (`correct_min: 10`) is **NOT met**. A stricter candidate-graph
|
||||
coverage probe reports **4/46/0** on the same 50 cases
|
||||
(`train_sample_coverage_report.json`); both preserve `wrong=0`. The two readers
|
||||
differ on 2 fast-path cases. The probe's movement (`3→4`) is precisely what
|
||||
reverted the expert claim (§2).
|
||||
- **(D) CORE-authored, and currently NOT conferring expert.** All four B-lanes are
|
||||
written and scored by CORE; none is external GSM8K. ADR-0131.4 replaced the
|
||||
original real-GSM8K gate (`correct_rate ≥ 0.60`) with this composite; ADR-0131.5
|
||||
retired the GSM8K probe from per-iteration gating. **Expert status therefore
|
||||
rests on CORE-authored evals, not external GSM8K — and must always be stated
|
||||
that way.** It is also *currently reverted* (§2).
|
||||
|
||||
---
|
||||
|
||||
## 2. Capability ledger status — and the expert fail-closed revert
|
||||
|
||||
**No domain is at `expert`.** The live ledger (`core capability ledger`) reports:
|
||||
|
||||
| Domain | Status | Note |
|
||||
|---|---|---|
|
||||
| `mathematics_logic` | **audit-passed** | expert promotion **fail-closed-reverted** — see below |
|
||||
| `physics` | audit-passed | no expert composer wired |
|
||||
| `systems_software` | audit-passed | no expert composer wired |
|
||||
| `hebrew_greek_textual_reasoning` | reasoning-capable | — |
|
||||
| `philosophy_theology` | reasoning-capable | — |
|
||||
|
||||
**What `audit-passed` means (and does NOT).** Per `ADR-0113`, `audit-passed`
|
||||
verifies *CORE claim-shape compliance* — signed digest, replay determinism, typed
|
||||
refusal, exact recall, grounding provenance. These are shapes a transformer LLM
|
||||
cannot structurally produce regardless of raw accuracy. **It is explicitly NOT a
|
||||
raw-capability claim.** A frontier LLM might score higher on the same benchmark
|
||||
and still fail this contract.
|
||||
|
||||
**The expert revert (the honesty story).** On 2026-05-23 `mathematics_logic` was
|
||||
signed and promoted to `expert` (the first-ever flip). It **auto-reverted to
|
||||
`audit-passed`** when its evidence bundle later drifted. The live composer refuses:
|
||||
|
||||
> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived
|
||||
> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was
|
||||
> added.`
|
||||
|
||||
Root cause (proven 2026-06-02, Week-1a): **genuine single-source evidence-drift,
|
||||
not a determinism defect.** The digest is byte-stable across processes and
|
||||
`PYTHONHASHSEED`; all digest/obligation modules are unchanged since signing; the
|
||||
signature was valid at its commit; the **only** moved input is the GSM8K coverage
|
||||
probe (`train_sample_coverage_report.json`, `3/47 → 4/46` via PRs #310/#488), and
|
||||
restoring those bytes in-place reproduces `4c46f530` exactly. This is the
|
||||
ADR-0120 fail-closed property working as designed — **CORE revoked its own expert
|
||||
claim when its evidence moved.** Full record: `ADR-0200`.
|
||||
|
||||
> **Known safe-direction wrinkle (documented, not yet fixed):** a *non-gating*
|
||||
> GSM8K disclosure value is committed into the *gating* digest, so improving
|
||||
> coverage invalidates the signature. It fails toward `audit-passed`, never toward
|
||||
> a false `expert` — so it is not a `wrong=0` hazard. Digest-scoping refinement is
|
||||
> deferred to a future ADR (`ADR-0200` §Consequences).
|
||||
|
||||
---
|
||||
|
||||
## 3. Verified invariants (safe to assert)
|
||||
|
||||
| Claim | Evidence | Reproduce |
|
||||
|---|---|---|
|
||||
| Versor closure `‖F·reverse(F) − 1‖_F < 1e-6` at all times | `algebra/`, `tests/test_versor_closure.py` | `uv run pytest tests/test_versor_closure.py` |
|
||||
| **Byte-identical replay / determinism** (same inputs → same trace hash) | `core/cognition/trace.py`; lane reports byte-equal across runs | digest determinism proven Week-1a (stable across processes/seeds) |
|
||||
| **Refuse-rather-than-guess / `wrong = 0`** | typed refusal across all four GSM8K numbers; `verify.py` gate | A/B/C/D all `wrong=0` |
|
||||
| Exact CGA recall (no ANN / HNSW / cosine) | `vault/store.py`; `docs/Yellowpaper.md` recall section | `uv run core test --suite algebra` |
|
||||
| Safety pack: add-but-never-remove, fail-closed on load | `packs/safety/core_safety_axes_v1.json`; `ADR-0029` | `uv run core test --suite smoke` |
|
||||
| On-device, non-LLM, no sampling/gradient/tokenizer | architecture; `README.md` "Third Door" | — |
|
||||
|
||||
---
|
||||
|
||||
## 4. Multimodal status — text only is *capability*; the rest is substrate or proposal
|
||||
|
||||
The patent application's "multimodal" title is broader than what is demonstrated
|
||||
in-repo. **External materials must not imply working vision or motor.**
|
||||
|
||||
| Modality | Status | Evidence |
|
||||
|---|---|---|
|
||||
| **Text** (English, Hebrew, Koine Greek) | **Active capability** | `sensorium/adapters/text.py`; language packs |
|
||||
| **Audio** | **Substrate landed, gate CLOSED (no capability claim)** | `sensorium/audio/*`; `make_audio_pack(gate_engaged=False)`; determinism + order-invariant CRDT merge + no-PCM-in-trace all gate-tested |
|
||||
| **Vision** (`ADR-0197`) | **Proposed only** — no code, no pack, no eval | `Modality.VISION` enum exists; no `sensorium/vision/` |
|
||||
| **Motor** (`ADR-0198`) | **Proposed only** — "design spike, no implementation" | registry has no `decode()` path; `persona/motor.py` is an *internal CGA screw motion*, not an actuator |
|
||||
|
||||
Honest line: *text is a working modality; audio is a determinism-proven substrate
|
||||
with its capability gate deliberately closed; vision and motor are design
|
||||
proposals with no implementation.*
|
||||
|
||||
---
|
||||
|
||||
## 5. Attestation — single-signer (a known boundary)
|
||||
|
||||
The reviewer registry has exactly **one** signer: `shay-j`, `domains: ["*"]`,
|
||||
`role: primary` (`docs/reviewers.yaml`; `reviewer_count: 1`). Every
|
||||
`audit_passed_claims` and the (now-quarantined) `math_expert_claims` entry is
|
||||
signed by the same identity. This is a real single-point-of-capture a partner may
|
||||
probe. The migration toward a multi-reviewer / threshold-signing registry is
|
||||
scoped as a Week-4 deliverable (`ADR-02xx`, design-only — no fake signers).
|
||||
|
||||
---
|
||||
|
||||
## 6. Claims we explicitly do NOT make
|
||||
|
||||
- ❌ "A domain is at `expert`." → No domain is; `mathematics_logic` is
|
||||
`audit-passed` (expert reverted).
|
||||
- ❌ "CORE is externally validated on GSM8K." → The expert gate rests on
|
||||
**CORE-authored** lanes; the real external number is A = `0/0/1319`.
|
||||
- ❌ "CORE scores ~100% on GSM8K." → That is B, a CORE-authored synthetic split.
|
||||
- ❌ "CORE has vision / motor / working multimodal perception." → Audio is
|
||||
substrate-only (gate closed); vision/motor are proposals.
|
||||
- ❌ "`audit-passed` means expert-level capability." → It is claim-shape
|
||||
compliance, not raw capability (`ADR-0113`).
|
||||
|
||||
---
|
||||
|
||||
## 7. Reconciliation log (proposed — pending operator ratification)
|
||||
|
||||
These reconcile stale artifacts to the truth above. **History-describing**
|
||||
artifacts keep their content with a dated "valid-at … auto-reverted … current =
|
||||
audit-passed" note (keep the receipt; keep the mismatch-refusal firing).
|
||||
**Current-machine-state** artifacts reconcile to the truth. Tracked in `ADR-0200`.
|
||||
|
||||
| Artifact | Type | Action |
|
||||
|---|---|---|
|
||||
| `docs/reviewers.yaml` `math_expert_claims` | history (receipt) | keep entry; add quarantine note; do not re-sign |
|
||||
| `docs/decisions/ADR-0120-math-expert-ledger-flip.md` | history | header note: valid-at 2026-05-23, auto-reverted, current = audit-passed |
|
||||
| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | current state | regenerate → `promote_admitted: false` |
|
||||
| `README.md` §"Path to expert" + ledger lines | current state | reconcile narrative to built-attempted-reverted; verify test count |
|
||||
| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | flip 3 red "is-expert" assertions into fail-closed-revert assertions |
|
||||
|
|
@ -8,6 +8,18 @@
|
|||
ADR-0114a.{1,2,3,4,5,6,7,8,9,10} (obligation auditors)
|
||||
**Predecessor PRs:** #173, #176-#180, #182-#194
|
||||
|
||||
> **Reconciliation note (ADR-0200, 2026-06-02).** This flip was **valid when it
|
||||
> was made** (2026-05-23 / #267): code + evidence reproduced the signed digest
|
||||
> `4c46f530...` exactly, and the empirical verdict below was correct *at that
|
||||
> commit*. It has since **auto-reverted to `audit-passed`.** The GSM8K coverage
|
||||
> probe drifted (`3/47 -> 4/46`, PRs #310/#488), changing the evidence-derived
|
||||
> digest to `02f6d3c8...`; the signature no longer matches and the composer
|
||||
> refuses — ADR-0120's documented fail-closed property firing as designed.
|
||||
> **Current ledger status: `audit-passed`.** The empirical block below is
|
||||
> preserved as the historical record. See
|
||||
> [ADR-0200](ADR-0200-expert-claim-reconciliation.md) and the
|
||||
> [claims ledger](../claims_ledger.md).
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
|
|
|||
132
docs/decisions/ADR-0200-expert-claim-reconciliation.md
Normal file
132
docs/decisions/ADR-0200-expert-claim-reconciliation.md
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
# ADR-0200 — Expert-Claim Reconciliation: Record the Fail-Closed Revert as Designed Behavior
|
||||
|
||||
**Status:** Proposed (review-gated — every claim/test change below awaits operator ratification)
|
||||
**Date:** 2026-06-02
|
||||
**Author:** CORE main agent (Opus 4.8) + reviewer (shay-j)
|
||||
**Depends on:** ADR-0120 (expert tier contract + ledger flip), ADR-0131.4 (composite math gate), ADR-0131.5 (GSM8K probe retirement), ADR-0113 (audit-passed naming), ADR-0119.7 (sealed GSM8K)
|
||||
**Companion:** [`docs/claims_ledger.md`](../claims_ledger.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Context
|
||||
|
||||
On 2026-05-23, `mathematics_logic` was signed and promoted to the `expert`
|
||||
ledger tier (ADR-0120 ledger flip) — the first-ever flip. As of 2026-06-02 the
|
||||
live ledger reports it as **`audit-passed`**, and the expert composer refuses:
|
||||
|
||||
> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived
|
||||
> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was
|
||||
> added.`
|
||||
|
||||
This is **not a regression to fix.** It is ADR-0120's load-bearing safety
|
||||
property firing exactly as documented ("Does NOT auto-promote on subsequent
|
||||
evidence-bundle changes … the verdict flips back … and the ledger row drops back
|
||||
to `audit-passed`. This is the load-bearing safety property"). The system
|
||||
**revoked its own expert claim** when its evidence drifted.
|
||||
|
||||
The problem this ADR solves is **documentary drift**, not engine behavior: several
|
||||
committed artifacts still assert the flip *succeeded*, contradicting the live,
|
||||
honest machine state. A skeptical external reader who diffs the README against
|
||||
`reviewers.yaml` against the ledger sees three different stories. This ADR makes
|
||||
the human-authored artifacts tell the **same** true story the engine tells.
|
||||
|
||||
## 2. Proven root cause (Week-1a investigation, 2026-06-02)
|
||||
|
||||
The digest divergence is **genuine single-source evidence-drift, not a
|
||||
non-determinism defect.** Evidence chain (all reproducible on `c058d96`):
|
||||
|
||||
| Test | Result | Conclusion |
|
||||
|---|---|---|
|
||||
| Derive 3× in-process + 2 fresh subprocesses (`PYTHONHASHSEED=0,1`) | all `02f6d3c8…` | **deterministic** — no defect |
|
||||
| `git log` of all 7 digest/obligation modules since signing (#267) | **0 changes** | **no method drift** |
|
||||
| Re-derive inside a worktree at the signing commit (#267) | reproduces `4c46f530`, `VALID-AT-ITS-COMMIT: True` | signature was sound when signed |
|
||||
| Restore signing-era probe bytes **in-place** at the canonical path | reproduces `4c46f530` exactly | the probe is the **sole** drift source |
|
||||
|
||||
The single moved input is `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
|
||||
(GSM8K admission `3/47 → 4/46` via PRs #310, #488 — both *after* signing). The
|
||||
four *gating* B-lanes are unchanged (185/14/40/50, `wrong=0`).
|
||||
|
||||
**Implication for the determinism claim:** the digest is byte-deterministic at any
|
||||
fixed commit. "Byte-identical replay" is safe to assert.
|
||||
|
||||
## 3. The safe-direction wrinkle (documented; fix deferred)
|
||||
|
||||
`composite_math_gate._compute_claim_digest` commits the GSM8K `honest_disclosure`
|
||||
block into the gating digest. But ADR-0131 explicitly designates GSM8K as
|
||||
**non-gating** disclosure. So improving a purely informational coverage metric
|
||||
invalidates the expert signature even though every *gating* criterion still
|
||||
passes (`technical_pass=True`).
|
||||
|
||||
This is a coupling smell, but it **fails toward `audit-passed`, never toward a
|
||||
false `expert`** — so it is **not a `wrong=0` hazard**. Per the project's
|
||||
no-scope-creep discipline, the fix (scope the digest to *gating* inputs only) is
|
||||
**deferred to a future ADR**. This ADR only records the wrinkle so a future
|
||||
reviewer does not rediscover it cold.
|
||||
|
||||
## 4. Decision — reconcile per artifact type
|
||||
|
||||
Reconciliation principle: **artifacts that describe HISTORY keep their content
|
||||
with a dated "valid-at … auto-reverted … current = audit-passed" note (keep the
|
||||
receipt; keep the mismatch-refusal firing); artifacts that assert CURRENT MACHINE
|
||||
STATE reconcile to the truth.**
|
||||
|
||||
| Artifact | Type | Action | Rationale |
|
||||
|---|---|---|---|
|
||||
| `docs/reviewers.yaml` `math_expert_claims` entry | history (receipt) | **Keep the entry**; add a quarantine comment block (valid-at 2026-05-23; auto-reverted; do not re-sign). | The entry's *mismatch* is what makes the ledger refuse — keeping it keeps the safety mechanism visibly firing. |
|
||||
| `ADR-0120-math-expert-ledger-flip.md` | history | Add a dated header note: valid at 2026-05-23/#267; auto-reverted; current ledger = `audit-passed`; see ADR-0200. | A decision record stays true to its moment; the note prevents misreading it as current state. |
|
||||
| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | **current state** | **Regenerate** → `promote_admitted: false`, `reviewer_signature_matches: false`. | A machine-generated current-state file with a committed `true` is the live overclaim. The receipt survives in git history + the quarantined registry entry. |
|
||||
| `README.md` §"Path to expert" + line-81 test count | current state | Reconcile the "next gate" narrative to the built-attempted-reverted story; verify/correct the test count. | The ledger table is already accurate; only the narrative tense and the test count are stale. |
|
||||
| `docs/decisions/README.md` | current state | Repoint any "math = expert" reference to `audit-passed`. | Index must match the ledger. |
|
||||
| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | Flip 3 currently-RED "is-expert" assertions into **fail-closed-revert** assertions (status==audit-passed; expert_reason explains the digest mismatch; `reviewer_signature_matches is False`). | These assert current machine state; reconciling them to the truth converts 3 red overclaim-tests into green mechanism-proving tests. |
|
||||
|
||||
### 4.1 Cascade safety (verified)
|
||||
|
||||
- **Regenerating the signed JSON does not cascade:** it is **not** SHA-pinned
|
||||
(absent from `scripts/verify_lane_shas.py` and `scripts/generate_claims.py`),
|
||||
no test reads its bytes, and only `core capability math-expert-promote` writes
|
||||
it. The digest `4c46f530` appears only in the JSON itself and the (quarantined)
|
||||
`reviewers.yaml` entry.
|
||||
- **The 3 red tests are pre-existing** (red since the #310/#488 drift), not caused
|
||||
by this reconciliation. They are folded in here so the expert-claim surface is
|
||||
reconciled as one unit.
|
||||
|
||||
## 5. What this ADR does NOT do
|
||||
|
||||
- Does **not** re-sign the expert claim. Re-signing `02f6d3c8` would re-assert an
|
||||
expert claim that (a) rests on CORE-authored lanes, not external GSM8K, and
|
||||
(b) would be immediately re-broken by the next parser improvement. The honest
|
||||
posture is `audit-passed` with the receipt preserved.
|
||||
- Does **not** change any eval gate, threshold, or safety boundary.
|
||||
- Does **not** fix the disclosure-in-digest coupling (§3) — deferred.
|
||||
- Does **not** weaken any invariant. `wrong=0`, determinism, exact recall, and
|
||||
versor closure are untouched.
|
||||
|
||||
## 6. Trust boundary
|
||||
|
||||
- **Reads:** the capability ledger, the committed evidence bundle, `reviewers.yaml`.
|
||||
- **Writes:** documentation + the one regenerated current-state artifact + two
|
||||
test files — all under operator ratification, on a branch, via PR. Nothing
|
||||
lands on `main` directly.
|
||||
- No dynamic imports, no network, no normalization, no stochastic path.
|
||||
|
||||
## 7. Consequences
|
||||
|
||||
- The README, the ledger, `reviewers.yaml`, and the tests tell **one** true story:
|
||||
`audit-passed`, expert auto-reverted, fail-closed working.
|
||||
- CI gains 3 green mechanism-proving tests where it had 3 red overclaim tests.
|
||||
- A future ADR may scope the promotion digest to gating inputs only (§3).
|
||||
- The reconciliation is the substrate for the CTO brief, whose honesty narrative
|
||||
leads with this revert: *a non-gating metric drifted and the system revoked its
|
||||
own expert claim — refuse-rather-than-guess applied to CORE's own status.*
|
||||
|
||||
## 8. CLAUDE.md PR-checklist
|
||||
|
||||
- **Capability/boundary protected:** restores claim↔machine-state coherence; keeps
|
||||
the fail-closed refusal visibly firing; preserves every invariant.
|
||||
- **Invariant proving field validity:** the ledger reports `expert` iff a signed
|
||||
digest re-derives — and it currently, correctly, does not.
|
||||
- **CLI/eval proving the lane:** `uv run core capability ledger` (status =
|
||||
audit-passed); `uv run pytest tests/test_mathlogic_expert_ledger_flip.py tests/test_adr_0120_math_expert_promotion.py`.
|
||||
- **Avoided hidden normalization / stochastic / approximate / unreviewed mutation:**
|
||||
Yes — documentation + one deterministic regeneration; no engine change.
|
||||
- **Trust boundary:** read-only inputs; writes are review-gated on a branch.
|
||||
|
|
@ -143,7 +143,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0119.7](ADR-0119.7-sealed-gsm8k-test.md) | Sealed GSM8K Test Set as gsm8k_math Holdout | Accepted |
|
||||
| [ADR-0119.8](ADR-0119.8-lane-gate.md) | gsm8k_math Overall Lane Gate (`gsm8k_capability_shape`) | Accepted |
|
||||
| [ADR-0120](ADR-0120-expert-promotion-contract.md) | First `expert` Promotion Contract | Proposed (contract-only; no domain promoted with this ADR) |
|
||||
| [ADR-0120](ADR-0120-math-expert-ledger-flip.md) | ADR-0120 (math, ledger flip) — Mathematics-Logic Domain Promoted to `expert` | Accepted — first `expert`-tier domain in the capability ledger |
|
||||
| [ADR-0120](ADR-0120-math-expert-ledger-flip.md) | ADR-0120 (math, ledger flip) — Mathematics-Logic Domain Promoted to `expert` | Accepted 2026-05-23; **auto-reverted to `audit-passed`** when evidence drifted — current status audit-passed (see [ADR-0200](ADR-0200-expert-claim-reconciliation.md)) |
|
||||
| [ADR-0120](ADR-0120-math-expert-promotion-wireup.md) | ADR-0120 (math) — Math-Expert Promotion Composer Wire-Up | Accepted (technical pass on first evaluation; awaiting reviewer signature for ledger admission) |
|
||||
| [ADR-0121](ADR-0121-mathematics-logic-expert-deferred.md) | `mathematics_logic` `expert` Promotion — Deferred (first attempt) | Accepted (the deferral is the decision) |
|
||||
| [ADR-0122](ADR-0122-parser-rate-per-unit.md) | Parser Expansion: Rate / Per-Unit Reasoning (substrate-only; lift deferred) | Accepted (substrate landed; sealed-lift gate deferred — the |
|
||||
|
|
@ -252,6 +252,7 @@ ADRs record significant architectural decisions: what was decided, why, what alt
|
|||
| [ADR-0197](ADR-0197-vision-compiler-delta-crdt.md) | CORE-native Vision Compiler over the Delta-CRDT Substrate | Proposed |
|
||||
| [ADR-0198](ADR-0198-motor-efferent-decoder-spike.md) | Motor as Efferent Modality — Protocol Gap & Governance (Design Spike) | Proposed (design spike — no implementation) |
|
||||
| [ADR-0199](ADR-0199-cross-domain-learning-arena-contract.md) | Cross-Domain Learning Arena Contract | Proposed |
|
||||
| [ADR-0200](ADR-0200-expert-claim-reconciliation.md) | Expert-Claim Reconciliation: record the math fail-closed revert as designed behavior | Proposed |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -51,6 +51,20 @@ audit_passed_claims:
|
|||
evidence_revision: "adr-0124:reviewed:2026-05-22"
|
||||
signed_by: shay-j
|
||||
claim_digest: "17e24436b6875b89f6d1a5c2992557413c7ef456250f549d463159f54438c407"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# QUARANTINED (ADR-0200, 2026-06-02). The math_expert_claims entry below was a
|
||||
# VALID signature when made (2026-05-23 / #267): code + evidence reproduced
|
||||
# claim_digest 4c46f530... exactly. It is RETAINED as a historical receipt and
|
||||
# is intentionally NOT re-signed. It no longer validates: the GSM8K coverage
|
||||
# probe drifted (3/47 -> 4/46, PRs #310/#488), so the evidence-derived digest
|
||||
# is now 02f6d3c8... and the expert composer REFUSES promotion. The live ledger
|
||||
# reports mathematics_logic = audit-passed. Keeping this mismatch in place is
|
||||
# ADR-0120's fail-closed property firing by design. DO NOT re-sign to force the
|
||||
# ledger flip; the honest current status is audit-passed (expert rests on
|
||||
# CORE-authored lanes, not external GSM8K).
|
||||
# See docs/decisions/ADR-0200-expert-claim-reconciliation.md.
|
||||
# ---------------------------------------------------------------------------
|
||||
math_expert_claims:
|
||||
- domain_id: mathematics_logic
|
||||
evidence_lanes:
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"adr": "0120-math",
|
||||
"all_obligations_passed": true,
|
||||
"claim_digest": "4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4",
|
||||
"claim_digest": "02f6d3c8803b0a3154b84e1ba6a292556ae6a7ed4765507e617576fe342a9c94",
|
||||
"composite_gate_passed": true,
|
||||
"composite_gate_refusal": "",
|
||||
"domain": "mathematics_logic",
|
||||
|
|
@ -77,8 +77,8 @@
|
|||
"title": "operation provenance via pack"
|
||||
}
|
||||
],
|
||||
"promote_admitted": true,
|
||||
"refusal_reason": "",
|
||||
"promote_admitted": false,
|
||||
"refusal_reason": "reviewer claim_digest mismatch \u2014 registry has '4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4', evidence-derived digest is '02f6d3c8803b0a3154b84e1ba6a292556ae6a7ed4765507e617576fe342a9c94'; the evidence bundle has changed since the signature was added",
|
||||
"reviewer_signature": {
|
||||
"claim_digest": "4c46f530351ce96fbdacf29d242b8d99d40e0c5a92d12b10c4862315147ebfa4",
|
||||
"domain_id": "mathematics_logic",
|
||||
|
|
@ -91,7 +91,7 @@
|
|||
"evidence_revision": "adr-0120-math:reviewed:2026-05-23",
|
||||
"signed_by": "shay-j"
|
||||
},
|
||||
"reviewer_signature_matches": true,
|
||||
"reviewer_signature_matches": false,
|
||||
"schema_version": 1,
|
||||
"technical_pass": true
|
||||
}
|
||||
|
|
|
|||
|
|
@ -159,11 +159,13 @@ def test_obligation_9_refuses_when_a_lane_is_invalid_json(tmp_path: Path) -> Non
|
|||
|
||||
|
||||
def test_composer_runs_on_current_main_with_all_obligations_passing() -> None:
|
||||
"""The load-bearing snapshot. Updated by the ledger-flip PR after
|
||||
the reviewer added the signed entry to ``docs/reviewers.yaml`` —
|
||||
every obligation auditor passes, the composite gate passes,
|
||||
technical_pass is True, the reviewer signature is present and
|
||||
matches the computed digest, so promote_admitted is True.
|
||||
"""Fail-closed revert snapshot (ADR-0200). On current main every
|
||||
obligation auditor passes, the composite gate passes, technical_pass is
|
||||
True, and the reviewer signature is present — but a non-gating GSM8K
|
||||
coverage metric drifted after signing, so the signature no longer matches
|
||||
the evidence-derived digest. The composer therefore refuses:
|
||||
promote_admitted is False with a digest-mismatch reason. This is
|
||||
ADR-0120's load-bearing fail-closed safety property.
|
||||
"""
|
||||
v = evaluate_math_expert_promotion()
|
||||
assert v.all_obligations_passed is True, (
|
||||
|
|
@ -174,9 +176,9 @@ def test_composer_runs_on_current_main_with_all_obligations_passing() -> None:
|
|||
assert v.technical_pass is True
|
||||
assert v.reviewer_signature is not None
|
||||
assert v.reviewer_signature.get("signed_by") == "shay-j"
|
||||
assert v.reviewer_signature_matches is True
|
||||
assert v.promote_admitted is True
|
||||
assert v.refusal_reason == ""
|
||||
assert v.reviewer_signature_matches is False
|
||||
assert v.promote_admitted is False
|
||||
assert "mismatch" in v.refusal_reason.lower()
|
||||
assert v.claim_digest # non-empty
|
||||
assert len(v.claim_digest) == 64 # SHA-256 hex
|
||||
|
||||
|
|
|
|||
|
|
@ -125,19 +125,21 @@ def _math_row(report: dict) -> dict:
|
|||
raise AssertionError("mathematics_logic row missing from ledger")
|
||||
|
||||
|
||||
def test_mathlogic_status_is_expert_with_signed_evidence() -> None:
|
||||
"""The load-bearing snapshot. Given the signed entry in
|
||||
docs/reviewers.yaml AND every obligation auditor passing AND the
|
||||
composite gate passing, the ledger reports
|
||||
``mathematics_logic.status == "expert"``."""
|
||||
def test_mathlogic_status_is_audit_passed_after_evidence_drift() -> None:
|
||||
"""Fail-closed revert snapshot (ADR-0200). The signed expert entry in
|
||||
docs/reviewers.yaml no longer re-derives — a non-gating GSM8K coverage
|
||||
metric drifted after signing — so the composer refuses and the ledger
|
||||
reports ``mathematics_logic.status == "audit-passed"``, not "expert".
|
||||
This is ADR-0120's fail-closed property firing by design."""
|
||||
report = ledger_report()
|
||||
row = _math_row(report)
|
||||
assert row["status"] == "expert", (
|
||||
f"expected status=expert; got {row['status']!r}; "
|
||||
assert row["status"] == "audit-passed", (
|
||||
f"expected status=audit-passed (expert fail-closed-reverted); "
|
||||
f"got {row['status']!r}; "
|
||||
f"audit_passed_reason={row.get('audit_passed_reason')}; "
|
||||
f"expert_reason={row.get('expert_reason')}"
|
||||
)
|
||||
assert row["predicates"]["expert"] is True
|
||||
assert row["predicates"]["expert"] is False
|
||||
assert row["predicates"]["audit_passed"] is True
|
||||
|
||||
|
||||
|
|
@ -145,7 +147,9 @@ def test_mathlogic_row_carries_expert_reason() -> None:
|
|||
report = ledger_report()
|
||||
row = _math_row(report)
|
||||
assert "expert_reason" in row
|
||||
assert "admitted" in row["expert_reason"].lower()
|
||||
# Fail-closed revert (ADR-0200): the reason explains the digest mismatch,
|
||||
# not an admission.
|
||||
assert "mismatch" in row["expert_reason"].lower()
|
||||
|
||||
|
||||
def test_other_domains_have_expert_false_no_composer_wired() -> None:
|
||||
|
|
|
|||
Loading…
Reference in a new issue