From 96b6c5e4b2ddb308efb6913c021cd3dde1dacde8 Mon Sep 17 00:00:00 2001 From: Shay Date: Tue, 2 Jun 2026 09:35:44 -0700 Subject: [PATCH] =?UTF-8?q?docs(claims):=20ADR-0200=20+=20claims=20ledger?= =?UTF-8?q?=20=E2=80=94=20single=20source=20of=20truth=20(additive)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add docs/claims_ledger.md (every public claim -> in-repo evidence, with the four GSM8K numbers separated and labeled) and ADR-0200 recording the mathematics_logic expert fail-closed revert as designed behavior (proven single-source evidence-drift; determinism intact). Additive only: no existing claim, eval gate, or test is touched. The review-gated reconciliation diffs (reviewers.yaml quarantine note, ADR-0120 header note, signed-JSON regen, README narrative, 3 test reconciliations) are listed in ADR-0200 section 4 and await operator ratification. --- docs/claims_ledger.md | 182 ++++++++++++++++++ .../ADR-0200-expert-claim-reconciliation.md | 132 +++++++++++++ 2 files changed, 314 insertions(+) create mode 100644 docs/claims_ledger.md create mode 100644 docs/decisions/ADR-0200-expert-claim-reconciliation.md diff --git a/docs/claims_ledger.md b/docs/claims_ledger.md new file mode 100644 index 00000000..6f2a5a98 --- /dev/null +++ b/docs/claims_ledger.md @@ -0,0 +1,182 @@ +# Claims Ledger — Single Source of Truth + +**Purpose.** Every public or material claim CORE makes — in `README.md`, +`CLAIMS.md`, ADRs, eval summaries, and any outward-facing brief or site copy — +maps here to its exact in-repo evidence and its *honest* framing. When a number +or status appears in any external material, it must reconcile against this file. +If a claim cannot be substantiated from the repo, it is softened or cut, not +defended. + +> **Discipline:** Honesty is the product. CORE refuses rather than guesses; that +> discipline applies to how we describe ourselves, not only to what the engine +> outputs. Overclaiming to hit a deadline is failure, not success. + +- **Base commit:** `c058d96` (origin/main). +- **Verification date:** 2026-06-02. +- **Reproduce the machine-derived rows:** + ```bash + uv run core capability ledger # Tier-1 status (audit-passed ×3; NO expert) + uv run python scripts/verify_lane_shas.py # Tier-2 pinned lane SHAs + uv run python scripts/generate_claims.py --check # CLAIMS.md is current + ``` + +--- + +## 1. The Four GSM8K Numbers — never conflate these + +CORE has **four** distinct GSM8K-related measurements. They measure different +things on different data, and only **one** is the real external benchmark. Any +material that blurs them is a credibility risk. + +| Tag | What it is | Result | Real GSM8K? | Gates expert? | Evidence | +|---|---|---|---|---|---| +| **A** | **Sealed real GSM8K test** (1,319 cases, HuggingFace `openai/gsm8k` test split) | **0 correct / 0 wrong / 1,319 refused** | **Yes** | No | `ADR-0119.7`; ciphertext `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` | +| **B** | **CORE-authored synthetic "public" split** (150 cases, rule-based, written to exercise the grammar) | **150 correct / 0 wrong / 0 refused** (rate 1.0) | **No** | No | `evals/gsm8k_math/baselines/comparison_v1.json`, `frontier.json` | +| **C** | **Real `train_sample`** (50-case dev sample of real GSM8K) | **6 correct / 44 refused / 0 wrong** | Yes (sample) | No | `evals/gsm8k_math/train_sample/v1/report.json` | +| **D** | **Expert-promotion composite gate** (CORE-authored lanes B1/B2/B3) | **185/185 + 14/14 + 40/40 + 50/50, wrong=0** | **No** | Yes (but reverted — §2) | `ADR-0131.4`; `core/capability/composite_math_gate.py` | + +**Per-number honest framing:** + +- **(A) The honest external number.** `0/0/1319` — CORE's grammar covers **zero** + real GSM8K test problems today. This is the truthful gap. The load-bearing + property is **`wrong == 0` against the external corpus**: CORE *refuses* what it + cannot grammar-handle; it does not confabulate. The seal is one-way + (encrypted to an off-repo key); the number is a **recorded measurement**, not + CI-reproducible without the holdout key. Never present A as an accuracy + achievement — present it as zero-confabulation discipline plus an honest + coverage gap. +- **(B) A frontier *comparison*, not a benchmark result.** 150/150 is on a split + **CORE wrote**, designed to exercise its own grammar without data + contamination. `frontier.json` itself flags the "apples-vs-oranges" caveat: + frontier LLMs (Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%) are + scored on *real* GSM8K. **Never cite B as "CORE scores 100% on GSM8K."** The + qualitative differentiator is `wrong=0`, not the rate. +- **(C) The dev-sample reality.** 6/44/0 on 50 real cases — the lane's + `exit_criterion` (`correct_min: 10`) is **NOT met**. A stricter candidate-graph + coverage probe reports **4/46/0** on the same 50 cases + (`train_sample_coverage_report.json`); both preserve `wrong=0`. The two readers + differ on 2 fast-path cases. The probe's movement (`3→4`) is precisely what + reverted the expert claim (§2). +- **(D) CORE-authored, and currently NOT conferring expert.** All four B-lanes are + written and scored by CORE; none is external GSM8K. ADR-0131.4 replaced the + original real-GSM8K gate (`correct_rate ≥ 0.60`) with this composite; ADR-0131.5 + retired the GSM8K probe from per-iteration gating. **Expert status therefore + rests on CORE-authored evals, not external GSM8K — and must always be stated + that way.** It is also *currently reverted* (§2). + +--- + +## 2. Capability ledger status — and the expert fail-closed revert + +**No domain is at `expert`.** The live ledger (`core capability ledger`) reports: + +| Domain | Status | Note | +|---|---|---| +| `mathematics_logic` | **audit-passed** | expert promotion **fail-closed-reverted** — see below | +| `physics` | audit-passed | no expert composer wired | +| `systems_software` | audit-passed | no expert composer wired | +| `hebrew_greek_textual_reasoning` | reasoning-capable | — | +| `philosophy_theology` | reasoning-capable | — | + +**What `audit-passed` means (and does NOT).** Per `ADR-0113`, `audit-passed` +verifies *CORE claim-shape compliance* — signed digest, replay determinism, typed +refusal, exact recall, grounding provenance. These are shapes a transformer LLM +cannot structurally produce regardless of raw accuracy. **It is explicitly NOT a +raw-capability claim.** A frontier LLM might score higher on the same benchmark +and still fail this contract. + +**The expert revert (the honesty story).** On 2026-05-23 `mathematics_logic` was +signed and promoted to `expert` (the first-ever flip). It **auto-reverted to +`audit-passed`** when its evidence bundle later drifted. The live composer refuses: + +> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived +> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was +> added.` + +Root cause (proven 2026-06-02, Week-1a): **genuine single-source evidence-drift, +not a determinism defect.** The digest is byte-stable across processes and +`PYTHONHASHSEED`; all digest/obligation modules are unchanged since signing; the +signature was valid at its commit; the **only** moved input is the GSM8K coverage +probe (`train_sample_coverage_report.json`, `3/47 → 4/46` via PRs #310/#488), and +restoring those bytes in-place reproduces `4c46f530` exactly. This is the +ADR-0120 fail-closed property working as designed — **CORE revoked its own expert +claim when its evidence moved.** Full record: `ADR-0200`. + +> **Known safe-direction wrinkle (documented, not yet fixed):** a *non-gating* +> GSM8K disclosure value is committed into the *gating* digest, so improving +> coverage invalidates the signature. It fails toward `audit-passed`, never toward +> a false `expert` — so it is not a `wrong=0` hazard. Digest-scoping refinement is +> deferred to a future ADR (`ADR-0200` §Consequences). + +--- + +## 3. Verified invariants (safe to assert) + +| Claim | Evidence | Reproduce | +|---|---|---| +| Versor closure `‖F·reverse(F) − 1‖_F < 1e-6` at all times | `algebra/`, `tests/test_versor_closure.py` | `uv run pytest tests/test_versor_closure.py` | +| **Byte-identical replay / determinism** (same inputs → same trace hash) | `core/cognition/trace.py`; lane reports byte-equal across runs | digest determinism proven Week-1a (stable across processes/seeds) | +| **Refuse-rather-than-guess / `wrong = 0`** | typed refusal across all four GSM8K numbers; `verify.py` gate | A/B/C/D all `wrong=0` | +| Exact CGA recall (no ANN / HNSW / cosine) | `vault/store.py`; `docs/Yellowpaper.md` recall section | `uv run core test --suite algebra` | +| Safety pack: add-but-never-remove, fail-closed on load | `packs/safety/core_safety_axes_v1.json`; `ADR-0029` | `uv run core test --suite smoke` | +| On-device, non-LLM, no sampling/gradient/tokenizer | architecture; `README.md` "Third Door" | — | + +--- + +## 4. Multimodal status — text only is *capability*; the rest is substrate or proposal + +The patent application's "multimodal" title is broader than what is demonstrated +in-repo. **External materials must not imply working vision or motor.** + +| Modality | Status | Evidence | +|---|---|---| +| **Text** (English, Hebrew, Koine Greek) | **Active capability** | `sensorium/adapters/text.py`; language packs | +| **Audio** | **Substrate landed, gate CLOSED (no capability claim)** | `sensorium/audio/*`; `make_audio_pack(gate_engaged=False)`; determinism + order-invariant CRDT merge + no-PCM-in-trace all gate-tested | +| **Vision** (`ADR-0197`) | **Proposed only** — no code, no pack, no eval | `Modality.VISION` enum exists; no `sensorium/vision/` | +| **Motor** (`ADR-0198`) | **Proposed only** — "design spike, no implementation" | registry has no `decode()` path; `persona/motor.py` is an *internal CGA screw motion*, not an actuator | + +Honest line: *text is a working modality; audio is a determinism-proven substrate +with its capability gate deliberately closed; vision and motor are design +proposals with no implementation.* + +--- + +## 5. Attestation — single-signer (a known boundary) + +The reviewer registry has exactly **one** signer: `shay-j`, `domains: ["*"]`, +`role: primary` (`docs/reviewers.yaml`; `reviewer_count: 1`). Every +`audit_passed_claims` and the (now-quarantined) `math_expert_claims` entry is +signed by the same identity. This is a real single-point-of-capture a partner may +probe. The migration toward a multi-reviewer / threshold-signing registry is +scoped as a Week-4 deliverable (`ADR-02xx`, design-only — no fake signers). + +--- + +## 6. Claims we explicitly do NOT make + +- ❌ "A domain is at `expert`." → No domain is; `mathematics_logic` is + `audit-passed` (expert reverted). +- ❌ "CORE is externally validated on GSM8K." → The expert gate rests on + **CORE-authored** lanes; the real external number is A = `0/0/1319`. +- ❌ "CORE scores ~100% on GSM8K." → That is B, a CORE-authored synthetic split. +- ❌ "CORE has vision / motor / working multimodal perception." → Audio is + substrate-only (gate closed); vision/motor are proposals. +- ❌ "`audit-passed` means expert-level capability." → It is claim-shape + compliance, not raw capability (`ADR-0113`). + +--- + +## 7. Reconciliation log (proposed — pending operator ratification) + +These reconcile stale artifacts to the truth above. **History-describing** +artifacts keep their content with a dated "valid-at … auto-reverted … current = +audit-passed" note (keep the receipt; keep the mismatch-refusal firing). +**Current-machine-state** artifacts reconcile to the truth. Tracked in `ADR-0200`. + +| Artifact | Type | Action | +|---|---|---| +| `docs/reviewers.yaml` `math_expert_claims` | history (receipt) | keep entry; add quarantine note; do not re-sign | +| `docs/decisions/ADR-0120-math-expert-ledger-flip.md` | history | header note: valid-at 2026-05-23, auto-reverted, current = audit-passed | +| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | current state | regenerate → `promote_admitted: false` | +| `README.md` §"Path to expert" + ledger lines | current state | reconcile narrative to built-attempted-reverted; verify test count | +| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | flip 3 red "is-expert" assertions into fail-closed-revert assertions | diff --git a/docs/decisions/ADR-0200-expert-claim-reconciliation.md b/docs/decisions/ADR-0200-expert-claim-reconciliation.md new file mode 100644 index 00000000..10d77d87 --- /dev/null +++ b/docs/decisions/ADR-0200-expert-claim-reconciliation.md @@ -0,0 +1,132 @@ +# ADR-0200 — Expert-Claim Reconciliation: Record the Fail-Closed Revert as Designed Behavior + +**Status:** Proposed (review-gated — every claim/test change below awaits operator ratification) +**Date:** 2026-06-02 +**Author:** CORE main agent (Opus 4.8) + reviewer (shay-j) +**Depends on:** ADR-0120 (expert tier contract + ledger flip), ADR-0131.4 (composite math gate), ADR-0131.5 (GSM8K probe retirement), ADR-0113 (audit-passed naming), ADR-0119.7 (sealed GSM8K) +**Companion:** [`docs/claims_ledger.md`](../claims_ledger.md) + +--- + +## 1. Context + +On 2026-05-23, `mathematics_logic` was signed and promoted to the `expert` +ledger tier (ADR-0120 ledger flip) — the first-ever flip. As of 2026-06-02 the +live ledger reports it as **`audit-passed`**, and the expert composer refuses: + +> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived +> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was +> added.` + +This is **not a regression to fix.** It is ADR-0120's load-bearing safety +property firing exactly as documented ("Does NOT auto-promote on subsequent +evidence-bundle changes … the verdict flips back … and the ledger row drops back +to `audit-passed`. This is the load-bearing safety property"). The system +**revoked its own expert claim** when its evidence drifted. + +The problem this ADR solves is **documentary drift**, not engine behavior: several +committed artifacts still assert the flip *succeeded*, contradicting the live, +honest machine state. A skeptical external reader who diffs the README against +`reviewers.yaml` against the ledger sees three different stories. This ADR makes +the human-authored artifacts tell the **same** true story the engine tells. + +## 2. Proven root cause (Week-1a investigation, 2026-06-02) + +The digest divergence is **genuine single-source evidence-drift, not a +non-determinism defect.** Evidence chain (all reproducible on `c058d96`): + +| Test | Result | Conclusion | +|---|---|---| +| Derive 3× in-process + 2 fresh subprocesses (`PYTHONHASHSEED=0,1`) | all `02f6d3c8…` | **deterministic** — no defect | +| `git log` of all 7 digest/obligation modules since signing (#267) | **0 changes** | **no method drift** | +| Re-derive inside a worktree at the signing commit (#267) | reproduces `4c46f530`, `VALID-AT-ITS-COMMIT: True` | signature was sound when signed | +| Restore signing-era probe bytes **in-place** at the canonical path | reproduces `4c46f530` exactly | the probe is the **sole** drift source | + +The single moved input is `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json` +(GSM8K admission `3/47 → 4/46` via PRs #310, #488 — both *after* signing). The +four *gating* B-lanes are unchanged (185/14/40/50, `wrong=0`). + +**Implication for the determinism claim:** the digest is byte-deterministic at any +fixed commit. "Byte-identical replay" is safe to assert. + +## 3. The safe-direction wrinkle (documented; fix deferred) + +`composite_math_gate._compute_claim_digest` commits the GSM8K `honest_disclosure` +block into the gating digest. But ADR-0131 explicitly designates GSM8K as +**non-gating** disclosure. So improving a purely informational coverage metric +invalidates the expert signature even though every *gating* criterion still +passes (`technical_pass=True`). + +This is a coupling smell, but it **fails toward `audit-passed`, never toward a +false `expert`** — so it is **not a `wrong=0` hazard**. Per the project's +no-scope-creep discipline, the fix (scope the digest to *gating* inputs only) is +**deferred to a future ADR**. This ADR only records the wrinkle so a future +reviewer does not rediscover it cold. + +## 4. Decision — reconcile per artifact type + +Reconciliation principle: **artifacts that describe HISTORY keep their content +with a dated "valid-at … auto-reverted … current = audit-passed" note (keep the +receipt; keep the mismatch-refusal firing); artifacts that assert CURRENT MACHINE +STATE reconcile to the truth.** + +| Artifact | Type | Action | Rationale | +|---|---|---|---| +| `docs/reviewers.yaml` `math_expert_claims` entry | history (receipt) | **Keep the entry**; add a quarantine comment block (valid-at 2026-05-23; auto-reverted; do not re-sign). | The entry's *mismatch* is what makes the ledger refuse — keeping it keeps the safety mechanism visibly firing. | +| `ADR-0120-math-expert-ledger-flip.md` | history | Add a dated header note: valid at 2026-05-23/#267; auto-reverted; current ledger = `audit-passed`; see ADR-0200. | A decision record stays true to its moment; the note prevents misreading it as current state. | +| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | **current state** | **Regenerate** → `promote_admitted: false`, `reviewer_signature_matches: false`. | A machine-generated current-state file with a committed `true` is the live overclaim. The receipt survives in git history + the quarantined registry entry. | +| `README.md` §"Path to expert" + line-81 test count | current state | Reconcile the "next gate" narrative to the built-attempted-reverted story; verify/correct the test count. | The ledger table is already accurate; only the narrative tense and the test count are stale. | +| `docs/decisions/README.md` | current state | Repoint any "math = expert" reference to `audit-passed`. | Index must match the ledger. | +| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | Flip 3 currently-RED "is-expert" assertions into **fail-closed-revert** assertions (status==audit-passed; expert_reason explains the digest mismatch; `reviewer_signature_matches is False`). | These assert current machine state; reconciling them to the truth converts 3 red overclaim-tests into green mechanism-proving tests. | + +### 4.1 Cascade safety (verified) + +- **Regenerating the signed JSON does not cascade:** it is **not** SHA-pinned + (absent from `scripts/verify_lane_shas.py` and `scripts/generate_claims.py`), + no test reads its bytes, and only `core capability math-expert-promote` writes + it. The digest `4c46f530` appears only in the JSON itself and the (quarantined) + `reviewers.yaml` entry. +- **The 3 red tests are pre-existing** (red since the #310/#488 drift), not caused + by this reconciliation. They are folded in here so the expert-claim surface is + reconciled as one unit. + +## 5. What this ADR does NOT do + +- Does **not** re-sign the expert claim. Re-signing `02f6d3c8` would re-assert an + expert claim that (a) rests on CORE-authored lanes, not external GSM8K, and + (b) would be immediately re-broken by the next parser improvement. The honest + posture is `audit-passed` with the receipt preserved. +- Does **not** change any eval gate, threshold, or safety boundary. +- Does **not** fix the disclosure-in-digest coupling (§3) — deferred. +- Does **not** weaken any invariant. `wrong=0`, determinism, exact recall, and + versor closure are untouched. + +## 6. Trust boundary + +- **Reads:** the capability ledger, the committed evidence bundle, `reviewers.yaml`. +- **Writes:** documentation + the one regenerated current-state artifact + two + test files — all under operator ratification, on a branch, via PR. Nothing + lands on `main` directly. +- No dynamic imports, no network, no normalization, no stochastic path. + +## 7. Consequences + +- The README, the ledger, `reviewers.yaml`, and the tests tell **one** true story: + `audit-passed`, expert auto-reverted, fail-closed working. +- CI gains 3 green mechanism-proving tests where it had 3 red overclaim tests. +- A future ADR may scope the promotion digest to gating inputs only (§3). +- The reconciliation is the substrate for the CTO brief, whose honesty narrative + leads with this revert: *a non-gating metric drifted and the system revoked its + own expert claim — refuse-rather-than-guess applied to CORE's own status.* + +## 8. CLAUDE.md PR-checklist + +- **Capability/boundary protected:** restores claim↔machine-state coherence; keeps + the fail-closed refusal visibly firing; preserves every invariant. +- **Invariant proving field validity:** the ledger reports `expert` iff a signed + digest re-derives — and it currently, correctly, does not. +- **CLI/eval proving the lane:** `uv run core capability ledger` (status = + audit-passed); `uv run pytest tests/test_mathlogic_expert_ledger_flip.py tests/test_adr_0120_math_expert_promotion.py`. +- **Avoided hidden normalization / stochastic / approximate / unreviewed mutation:** + Yes — documentation + one deterministic regeneration; no engine change. +- **Trust boundary:** read-only inputs; writes are review-gated on a branch.