96b6c5e4b2
Add docs/claims_ledger.md (every public claim -> in-repo evidence, with the four GSM8K numbers separated and labeled) and ADR-0200 recording the mathematics_logic expert fail-closed revert as designed behavior (proven single-source evidence-drift; determinism intact). Additive only: no existing claim, eval gate, or test is touched. The review-gated reconciliation diffs (reviewers.yaml quarantine note, ADR-0120 header note, signed-JSON regen, README narrative, 3 test reconciliations) are listed in ADR-0200 section 4 and await operator ratification.
182 lines
11 KiB
Markdown
182 lines
11 KiB
Markdown
# Claims Ledger — Single Source of Truth
|
||
|
||
**Purpose.** Every public or material claim CORE makes — in `README.md`,
|
||
`CLAIMS.md`, ADRs, eval summaries, and any outward-facing brief or site copy —
|
||
maps here to its exact in-repo evidence and its *honest* framing. When a number
|
||
or status appears in any external material, it must reconcile against this file.
|
||
If a claim cannot be substantiated from the repo, it is softened or cut, not
|
||
defended.
|
||
|
||
> **Discipline:** Honesty is the product. CORE refuses rather than guesses; that
|
||
> discipline applies to how we describe ourselves, not only to what the engine
|
||
> outputs. Overclaiming to hit a deadline is failure, not success.
|
||
|
||
- **Base commit:** `c058d96` (origin/main).
|
||
- **Verification date:** 2026-06-02.
|
||
- **Reproduce the machine-derived rows:**
|
||
```bash
|
||
uv run core capability ledger # Tier-1 status (audit-passed ×3; NO expert)
|
||
uv run python scripts/verify_lane_shas.py # Tier-2 pinned lane SHAs
|
||
uv run python scripts/generate_claims.py --check # CLAIMS.md is current
|
||
```
|
||
|
||
---
|
||
|
||
## 1. The Four GSM8K Numbers — never conflate these
|
||
|
||
CORE has **four** distinct GSM8K-related measurements. They measure different
|
||
things on different data, and only **one** is the real external benchmark. Any
|
||
material that blurs them is a credibility risk.
|
||
|
||
| Tag | What it is | Result | Real GSM8K? | Gates expert? | Evidence |
|
||
|---|---|---|---|---|---|
|
||
| **A** | **Sealed real GSM8K test** (1,319 cases, HuggingFace `openai/gsm8k` test split) | **0 correct / 0 wrong / 1,319 refused** | **Yes** | No | `ADR-0119.7`; ciphertext `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` |
|
||
| **B** | **CORE-authored synthetic "public" split** (150 cases, rule-based, written to exercise the grammar) | **150 correct / 0 wrong / 0 refused** (rate 1.0) | **No** | No | `evals/gsm8k_math/baselines/comparison_v1.json`, `frontier.json` |
|
||
| **C** | **Real `train_sample`** (50-case dev sample of real GSM8K) | **6 correct / 44 refused / 0 wrong** | Yes (sample) | No | `evals/gsm8k_math/train_sample/v1/report.json` |
|
||
| **D** | **Expert-promotion composite gate** (CORE-authored lanes B1/B2/B3) | **185/185 + 14/14 + 40/40 + 50/50, wrong=0** | **No** | Yes (but reverted — §2) | `ADR-0131.4`; `core/capability/composite_math_gate.py` |
|
||
|
||
**Per-number honest framing:**
|
||
|
||
- **(A) The honest external number.** `0/0/1319` — CORE's grammar covers **zero**
|
||
real GSM8K test problems today. This is the truthful gap. The load-bearing
|
||
property is **`wrong == 0` against the external corpus**: CORE *refuses* what it
|
||
cannot grammar-handle; it does not confabulate. The seal is one-way
|
||
(encrypted to an off-repo key); the number is a **recorded measurement**, not
|
||
CI-reproducible without the holdout key. Never present A as an accuracy
|
||
achievement — present it as zero-confabulation discipline plus an honest
|
||
coverage gap.
|
||
- **(B) A frontier *comparison*, not a benchmark result.** 150/150 is on a split
|
||
**CORE wrote**, designed to exercise its own grammar without data
|
||
contamination. `frontier.json` itself flags the "apples-vs-oranges" caveat:
|
||
frontier LLMs (Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%) are
|
||
scored on *real* GSM8K. **Never cite B as "CORE scores 100% on GSM8K."** The
|
||
qualitative differentiator is `wrong=0`, not the rate.
|
||
- **(C) The dev-sample reality.** 6/44/0 on 50 real cases — the lane's
|
||
`exit_criterion` (`correct_min: 10`) is **NOT met**. A stricter candidate-graph
|
||
coverage probe reports **4/46/0** on the same 50 cases
|
||
(`train_sample_coverage_report.json`); both preserve `wrong=0`. The two readers
|
||
differ on 2 fast-path cases. The probe's movement (`3→4`) is precisely what
|
||
reverted the expert claim (§2).
|
||
- **(D) CORE-authored, and currently NOT conferring expert.** All four B-lanes are
|
||
written and scored by CORE; none is external GSM8K. ADR-0131.4 replaced the
|
||
original real-GSM8K gate (`correct_rate ≥ 0.60`) with this composite; ADR-0131.5
|
||
retired the GSM8K probe from per-iteration gating. **Expert status therefore
|
||
rests on CORE-authored evals, not external GSM8K — and must always be stated
|
||
that way.** It is also *currently reverted* (§2).
|
||
|
||
---
|
||
|
||
## 2. Capability ledger status — and the expert fail-closed revert
|
||
|
||
**No domain is at `expert`.** The live ledger (`core capability ledger`) reports:
|
||
|
||
| Domain | Status | Note |
|
||
|---|---|---|
|
||
| `mathematics_logic` | **audit-passed** | expert promotion **fail-closed-reverted** — see below |
|
||
| `physics` | audit-passed | no expert composer wired |
|
||
| `systems_software` | audit-passed | no expert composer wired |
|
||
| `hebrew_greek_textual_reasoning` | reasoning-capable | — |
|
||
| `philosophy_theology` | reasoning-capable | — |
|
||
|
||
**What `audit-passed` means (and does NOT).** Per `ADR-0113`, `audit-passed`
|
||
verifies *CORE claim-shape compliance* — signed digest, replay determinism, typed
|
||
refusal, exact recall, grounding provenance. These are shapes a transformer LLM
|
||
cannot structurally produce regardless of raw accuracy. **It is explicitly NOT a
|
||
raw-capability claim.** A frontier LLM might score higher on the same benchmark
|
||
and still fail this contract.
|
||
|
||
**The expert revert (the honesty story).** On 2026-05-23 `mathematics_logic` was
|
||
signed and promoted to `expert` (the first-ever flip). It **auto-reverted to
|
||
`audit-passed`** when its evidence bundle later drifted. The live composer refuses:
|
||
|
||
> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived
|
||
> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was
|
||
> added.`
|
||
|
||
Root cause (proven 2026-06-02, Week-1a): **genuine single-source evidence-drift,
|
||
not a determinism defect.** The digest is byte-stable across processes and
|
||
`PYTHONHASHSEED`; all digest/obligation modules are unchanged since signing; the
|
||
signature was valid at its commit; the **only** moved input is the GSM8K coverage
|
||
probe (`train_sample_coverage_report.json`, `3/47 → 4/46` via PRs #310/#488), and
|
||
restoring those bytes in-place reproduces `4c46f530` exactly. This is the
|
||
ADR-0120 fail-closed property working as designed — **CORE revoked its own expert
|
||
claim when its evidence moved.** Full record: `ADR-0200`.
|
||
|
||
> **Known safe-direction wrinkle (documented, not yet fixed):** a *non-gating*
|
||
> GSM8K disclosure value is committed into the *gating* digest, so improving
|
||
> coverage invalidates the signature. It fails toward `audit-passed`, never toward
|
||
> a false `expert` — so it is not a `wrong=0` hazard. Digest-scoping refinement is
|
||
> deferred to a future ADR (`ADR-0200` §Consequences).
|
||
|
||
---
|
||
|
||
## 3. Verified invariants (safe to assert)
|
||
|
||
| Claim | Evidence | Reproduce |
|
||
|---|---|---|
|
||
| Versor closure `‖F·reverse(F) − 1‖_F < 1e-6` at all times | `algebra/`, `tests/test_versor_closure.py` | `uv run pytest tests/test_versor_closure.py` |
|
||
| **Byte-identical replay / determinism** (same inputs → same trace hash) | `core/cognition/trace.py`; lane reports byte-equal across runs | digest determinism proven Week-1a (stable across processes/seeds) |
|
||
| **Refuse-rather-than-guess / `wrong = 0`** | typed refusal across all four GSM8K numbers; `verify.py` gate | A/B/C/D all `wrong=0` |
|
||
| Exact CGA recall (no ANN / HNSW / cosine) | `vault/store.py`; `docs/Yellowpaper.md` recall section | `uv run core test --suite algebra` |
|
||
| Safety pack: add-but-never-remove, fail-closed on load | `packs/safety/core_safety_axes_v1.json`; `ADR-0029` | `uv run core test --suite smoke` |
|
||
| On-device, non-LLM, no sampling/gradient/tokenizer | architecture; `README.md` "Third Door" | — |
|
||
|
||
---
|
||
|
||
## 4. Multimodal status — text only is *capability*; the rest is substrate or proposal
|
||
|
||
The patent application's "multimodal" title is broader than what is demonstrated
|
||
in-repo. **External materials must not imply working vision or motor.**
|
||
|
||
| Modality | Status | Evidence |
|
||
|---|---|---|
|
||
| **Text** (English, Hebrew, Koine Greek) | **Active capability** | `sensorium/adapters/text.py`; language packs |
|
||
| **Audio** | **Substrate landed, gate CLOSED (no capability claim)** | `sensorium/audio/*`; `make_audio_pack(gate_engaged=False)`; determinism + order-invariant CRDT merge + no-PCM-in-trace all gate-tested |
|
||
| **Vision** (`ADR-0197`) | **Proposed only** — no code, no pack, no eval | `Modality.VISION` enum exists; no `sensorium/vision/` |
|
||
| **Motor** (`ADR-0198`) | **Proposed only** — "design spike, no implementation" | registry has no `decode()` path; `persona/motor.py` is an *internal CGA screw motion*, not an actuator |
|
||
|
||
Honest line: *text is a working modality; audio is a determinism-proven substrate
|
||
with its capability gate deliberately closed; vision and motor are design
|
||
proposals with no implementation.*
|
||
|
||
---
|
||
|
||
## 5. Attestation — single-signer (a known boundary)
|
||
|
||
The reviewer registry has exactly **one** signer: `shay-j`, `domains: ["*"]`,
|
||
`role: primary` (`docs/reviewers.yaml`; `reviewer_count: 1`). Every
|
||
`audit_passed_claims` and the (now-quarantined) `math_expert_claims` entry is
|
||
signed by the same identity. This is a real single-point-of-capture a partner may
|
||
probe. The migration toward a multi-reviewer / threshold-signing registry is
|
||
scoped as a Week-4 deliverable (`ADR-02xx`, design-only — no fake signers).
|
||
|
||
---
|
||
|
||
## 6. Claims we explicitly do NOT make
|
||
|
||
- ❌ "A domain is at `expert`." → No domain is; `mathematics_logic` is
|
||
`audit-passed` (expert reverted).
|
||
- ❌ "CORE is externally validated on GSM8K." → The expert gate rests on
|
||
**CORE-authored** lanes; the real external number is A = `0/0/1319`.
|
||
- ❌ "CORE scores ~100% on GSM8K." → That is B, a CORE-authored synthetic split.
|
||
- ❌ "CORE has vision / motor / working multimodal perception." → Audio is
|
||
substrate-only (gate closed); vision/motor are proposals.
|
||
- ❌ "`audit-passed` means expert-level capability." → It is claim-shape
|
||
compliance, not raw capability (`ADR-0113`).
|
||
|
||
---
|
||
|
||
## 7. Reconciliation log (proposed — pending operator ratification)
|
||
|
||
These reconcile stale artifacts to the truth above. **History-describing**
|
||
artifacts keep their content with a dated "valid-at … auto-reverted … current =
|
||
audit-passed" note (keep the receipt; keep the mismatch-refusal firing).
|
||
**Current-machine-state** artifacts reconcile to the truth. Tracked in `ADR-0200`.
|
||
|
||
| Artifact | Type | Action |
|
||
|---|---|---|
|
||
| `docs/reviewers.yaml` `math_expert_claims` | history (receipt) | keep entry; add quarantine note; do not re-sign |
|
||
| `docs/decisions/ADR-0120-math-expert-ledger-flip.md` | history | header note: valid-at 2026-05-23, auto-reverted, current = audit-passed |
|
||
| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | current state | regenerate → `promote_admitted: false` |
|
||
| `README.md` §"Path to expert" + ledger lines | current state | reconcile narrative to built-attempted-reverted; verify test count |
|
||
| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | flip 3 red "is-expert" assertions into fail-closed-revert assertions |
|