core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

docs(claims): ADR-0200 + claims ledger — single source of truth (additive)

Browse source 
Add docs/claims_ledger.md (every public claim -> in-repo evidence, with the
four GSM8K numbers separated and labeled) and ADR-0200 recording the
mathematics_logic expert fail-closed revert as designed behavior (proven
single-source evidence-drift; determinism intact).

Additive only: no existing claim, eval gate, or test is touched. The
review-gated reconciliation diffs (reviewers.yaml quarantine note, ADR-0120
header note, signed-JSON regen, README narrative, 3 test reconciliations) are
listed in ADR-0200 section 4 and await operator ratification.
This commit is contained in:
Shay 2026-06-02 09:35:44 -07:00
parent c058d966f9
commit 96b6c5e4b2
2 changed files with 314 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

182
docs/claims_ledger.md Normal file
View file

@ -0,0 +1,182 @@
# Claims Ledger — Single Source of Truth
**Purpose.** Every public or material claim CORE makes — in `README.md`,
`CLAIMS.md`, ADRs, eval summaries, and any outward-facing brief or site copy —
maps here to its exact in-repo evidence and its *honest* framing. When a number
or status appears in any external material, it must reconcile against this file.
If a claim cannot be substantiated from the repo, it is softened or cut, not
defended.
> **Discipline:** Honesty is the product. CORE refuses rather than guesses; that
> discipline applies to how we describe ourselves, not only to what the engine
> outputs. Overclaiming to hit a deadline is failure, not success.
- **Base commit:** `c058d96` (origin/main).
- **Verification date:** 2026-06-02.
- **Reproduce the machine-derived rows:**
```bash
uv run core capability ledger # Tier-1 status (audit-passed ×3; NO expert)
uv run python scripts/verify_lane_shas.py # Tier-2 pinned lane SHAs
uv run python scripts/generate_claims.py --check # CLAIMS.md is current
```
---
## 1. The Four GSM8K Numbers — never conflate these
CORE has **four** distinct GSM8K-related measurements. They measure different
things on different data, and only **one** is the real external benchmark. Any
material that blurs them is a credibility risk.
| Tag | What it is | Result | Real GSM8K? | Gates expert? | Evidence |
|---|---|---|---|---|---|
| **A** | **Sealed real GSM8K test** (1,319 cases, HuggingFace `openai/gsm8k` test split) | **0 correct / 0 wrong / 1,319 refused** | **Yes** | No | `ADR-0119.7`; ciphertext `evals/gsm8k_math/holdouts/v1/cases.jsonl.age` |
| **B** | **CORE-authored synthetic "public" split** (150 cases, rule-based, written to exercise the grammar) | **150 correct / 0 wrong / 0 refused** (rate 1.0) | **No** | No | `evals/gsm8k_math/baselines/comparison_v1.json`, `frontier.json` |
| **C** | **Real `train_sample`** (50-case dev sample of real GSM8K) | **6 correct / 44 refused / 0 wrong** | Yes (sample) | No | `evals/gsm8k_math/train_sample/v1/report.json` |
| **D** | **Expert-promotion composite gate** (CORE-authored lanes B1/B2/B3) | **185/185 + 14/14 + 40/40 + 50/50, wrong=0** | **No** | Yes (but reverted — §2) | `ADR-0131.4`; `core/capability/composite_math_gate.py` |
**Per-number honest framing:**
- **(A) The honest external number.** `0/0/1319` — CORE's grammar covers **zero**
real GSM8K test problems today. This is the truthful gap. The load-bearing
property is **`wrong == 0` against the external corpus**: CORE *refuses* what it
cannot grammar-handle; it does not confabulate. The seal is one-way
(encrypted to an off-repo key); the number is a **recorded measurement**, not
CI-reproducible without the holdout key. Never present A as an accuracy
achievement — present it as zero-confabulation discipline plus an honest
coverage gap.
- **(B) A frontier *comparison*, not a benchmark result.** 150/150 is on a split
**CORE wrote**, designed to exercise its own grammar without data
contamination. `frontier.json` itself flags the "apples-vs-oranges" caveat:
frontier LLMs (Claude 3.5 Sonnet 96.4%, GPT-4 92.0%, Gemini 1.5 Pro 90.8%) are
scored on *real* GSM8K. **Never cite B as "CORE scores 100% on GSM8K."** The
qualitative differentiator is `wrong=0`, not the rate.
- **(C) The dev-sample reality.** 6/44/0 on 50 real cases — the lane's
`exit_criterion` (`correct_min: 10`) is **NOT met**. A stricter candidate-graph
coverage probe reports **4/46/0** on the same 50 cases
(`train_sample_coverage_report.json`); both preserve `wrong=0`. The two readers
differ on 2 fast-path cases. The probe's movement (`3→4`) is precisely what
reverted the expert claim (§2).
- **(D) CORE-authored, and currently NOT conferring expert.** All four B-lanes are
written and scored by CORE; none is external GSM8K. ADR-0131.4 replaced the
original real-GSM8K gate (`correct_rate ≥ 0.60`) with this composite; ADR-0131.5
retired the GSM8K probe from per-iteration gating. **Expert status therefore
rests on CORE-authored evals, not external GSM8K — and must always be stated
that way.** It is also *currently reverted* (§2).
---
## 2. Capability ledger status — and the expert fail-closed revert
**No domain is at `expert`.** The live ledger (`core capability ledger`) reports:
| Domain | Status | Note |
|---|---|---|
| `mathematics_logic` | **audit-passed** | expert promotion **fail-closed-reverted** — see below |
| `physics` | audit-passed | no expert composer wired |
| `systems_software` | audit-passed | no expert composer wired |
| `hebrew_greek_textual_reasoning` | reasoning-capable | — |
| `philosophy_theology` | reasoning-capable | — |
**What `audit-passed` means (and does NOT).** Per `ADR-0113`, `audit-passed`
verifies *CORE claim-shape compliance* — signed digest, replay determinism, typed
refusal, exact recall, grounding provenance. These are shapes a transformer LLM
cannot structurally produce regardless of raw accuracy. **It is explicitly NOT a
raw-capability claim.** A frontier LLM might score higher on the same benchmark
and still fail this contract.
**The expert revert (the honesty story).** On 2026-05-23 `mathematics_logic` was
signed and promoted to `expert` (the first-ever flip). It **auto-reverted to
`audit-passed`** when its evidence bundle later drifted. The live composer refuses:
> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived
> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was
> added.`
Root cause (proven 2026-06-02, Week-1a): **genuine single-source evidence-drift,
not a determinism defect.** The digest is byte-stable across processes and
`PYTHONHASHSEED`; all digest/obligation modules are unchanged since signing; the
signature was valid at its commit; the **only** moved input is the GSM8K coverage
probe (`train_sample_coverage_report.json`, `3/47 → 4/46` via PRs #310/#488), and
restoring those bytes in-place reproduces `4c46f530` exactly. This is the
ADR-0120 fail-closed property working as designed — **CORE revoked its own expert
claim when its evidence moved.** Full record: `ADR-0200`.
> **Known safe-direction wrinkle (documented, not yet fixed):** a *non-gating*
> GSM8K disclosure value is committed into the *gating* digest, so improving
> coverage invalidates the signature. It fails toward `audit-passed`, never toward
> a false `expert` — so it is not a `wrong=0` hazard. Digest-scoping refinement is
> deferred to a future ADR (`ADR-0200` §Consequences).
---
## 3. Verified invariants (safe to assert)
| Claim | Evidence | Reproduce |
|---|---|---|
| Versor closure `‖F·reverse(F) − 1‖_F < 1e-6` at all times | `algebra/`, `tests/test_versor_closure.py` | `uv run pytest tests/test_versor_closure.py` |
| **Byte-identical replay / determinism** (same inputs → same trace hash) | `core/cognition/trace.py`; lane reports byte-equal across runs | digest determinism proven Week-1a (stable across processes/seeds) |
| **Refuse-rather-than-guess / `wrong = 0`** | typed refusal across all four GSM8K numbers; `verify.py` gate | A/B/C/D all `wrong=0` |
| Exact CGA recall (no ANN / HNSW / cosine) | `vault/store.py`; `docs/Yellowpaper.md` recall section | `uv run core test --suite algebra` |
| Safety pack: add-but-never-remove, fail-closed on load | `packs/safety/core_safety_axes_v1.json`; `ADR-0029` | `uv run core test --suite smoke` |
| On-device, non-LLM, no sampling/gradient/tokenizer | architecture; `README.md` "Third Door" | — |
---
## 4. Multimodal status — text only is *capability*; the rest is substrate or proposal
The patent application's "multimodal" title is broader than what is demonstrated
in-repo. **External materials must not imply working vision or motor.**
| Modality | Status | Evidence |
|---|---|---|
| **Text** (English, Hebrew, Koine Greek) | **Active capability** | `sensorium/adapters/text.py`; language packs |
| **Audio** | **Substrate landed, gate CLOSED (no capability claim)** | `sensorium/audio/*`; `make_audio_pack(gate_engaged=False)`; determinism + order-invariant CRDT merge + no-PCM-in-trace all gate-tested |
| **Vision** (`ADR-0197`) | **Proposed only** — no code, no pack, no eval | `Modality.VISION` enum exists; no `sensorium/vision/` |
| **Motor** (`ADR-0198`) | **Proposed only** — "design spike, no implementation" | registry has no `decode()` path; `persona/motor.py` is an *internal CGA screw motion*, not an actuator |
Honest line: *text is a working modality; audio is a determinism-proven substrate
with its capability gate deliberately closed; vision and motor are design
proposals with no implementation.*
---
## 5. Attestation — single-signer (a known boundary)
The reviewer registry has exactly **one** signer: `shay-j`, `domains: ["*"]`,
`role: primary` (`docs/reviewers.yaml`; `reviewer_count: 1`). Every
`audit_passed_claims` and the (now-quarantined) `math_expert_claims` entry is
signed by the same identity. This is a real single-point-of-capture a partner may
probe. The migration toward a multi-reviewer / threshold-signing registry is
scoped as a Week-4 deliverable (`ADR-02xx`, design-only — no fake signers).
---
## 6. Claims we explicitly do NOT make
- ❌ "A domain is at `expert`." → No domain is; `mathematics_logic` is
`audit-passed` (expert reverted).
- ❌ "CORE is externally validated on GSM8K." → The expert gate rests on
**CORE-authored** lanes; the real external number is A = `0/0/1319`.
- ❌ "CORE scores ~100% on GSM8K." → That is B, a CORE-authored synthetic split.
- ❌ "CORE has vision / motor / working multimodal perception." → Audio is
substrate-only (gate closed); vision/motor are proposals.
- ❌ "`audit-passed` means expert-level capability." → It is claim-shape
compliance, not raw capability (`ADR-0113`).
---
## 7. Reconciliation log (proposed — pending operator ratification)
These reconcile stale artifacts to the truth above. **History-describing**
artifacts keep their content with a dated "valid-at … auto-reverted … current =
audit-passed" note (keep the receipt; keep the mismatch-refusal firing).
**Current-machine-state** artifacts reconcile to the truth. Tracked in `ADR-0200`.
| Artifact | Type | Action |
|---|---|---|
| `docs/reviewers.yaml` `math_expert_claims` | history (receipt) | keep entry; add quarantine note; do not re-sign |
| `docs/decisions/ADR-0120-math-expert-ledger-flip.md` | history | header note: valid-at 2026-05-23, auto-reverted, current = audit-passed |
| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | current state | regenerate → `promote_admitted: false` |
| `README.md` §"Path to expert" + ledger lines | current state | reconcile narrative to built-attempted-reverted; verify test count |
| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | flip 3 red "is-expert" assertions into fail-closed-revert assertions |

132
docs/decisions/ADR-0200-expert-claim-reconciliation.md Normal file
View file

@ -0,0 +1,132 @@
# ADR-0200 — Expert-Claim Reconciliation: Record the Fail-Closed Revert as Designed Behavior
**Status:** Proposed (review-gated — every claim/test change below awaits operator ratification)
**Date:** 2026-06-02
**Author:** CORE main agent (Opus 4.8) + reviewer (shay-j)
**Depends on:** ADR-0120 (expert tier contract + ledger flip), ADR-0131.4 (composite math gate), ADR-0131.5 (GSM8K probe retirement), ADR-0113 (audit-passed naming), ADR-0119.7 (sealed GSM8K)
**Companion:** [`docs/claims_ledger.md`](../claims_ledger.md)
---
## 1. Context
On 2026-05-23, `mathematics_logic` was signed and promoted to the `expert`
ledger tier (ADR-0120 ledger flip) — the first-ever flip. As of 2026-06-02 the
live ledger reports it as **`audit-passed`**, and the expert composer refuses:
> `reviewer claim_digest mismatch — registry has '4c46f530…', evidence-derived
> digest is '02f6d3c8…'; the evidence bundle has changed since the signature was
> added.`
This is **not a regression to fix.** It is ADR-0120's load-bearing safety
property firing exactly as documented ("Does NOT auto-promote on subsequent
evidence-bundle changes … the verdict flips back … and the ledger row drops back
to `audit-passed`. This is the load-bearing safety property"). The system
**revoked its own expert claim** when its evidence drifted.
The problem this ADR solves is **documentary drift**, not engine behavior: several
committed artifacts still assert the flip *succeeded*, contradicting the live,
honest machine state. A skeptical external reader who diffs the README against
`reviewers.yaml` against the ledger sees three different stories. This ADR makes
the human-authored artifacts tell the **same** true story the engine tells.
## 2. Proven root cause (Week-1a investigation, 2026-06-02)
The digest divergence is **genuine single-source evidence-drift, not a
non-determinism defect.** Evidence chain (all reproducible on `c058d96`):
| Test | Result | Conclusion |
|---|---|---|
| Derive 3× in-process + 2 fresh subprocesses (`PYTHONHASHSEED=0,1`) | all `02f6d3c8…` | **deterministic** — no defect |
| `git log` of all 7 digest/obligation modules since signing (#267) | **0 changes** | **no method drift** |
| Re-derive inside a worktree at the signing commit (#267) | reproduces `4c46f530`, `VALID-AT-ITS-COMMIT: True` | signature was sound when signed |
| Restore signing-era probe bytes **in-place** at the canonical path | reproduces `4c46f530` exactly | the probe is the **sole** drift source |
The single moved input is `evals/gsm8k_math/train_sample/v1/train_sample_coverage_report.json`
(GSM8K admission `3/47 → 4/46` via PRs #310, #488 — both *after* signing). The
four *gating* B-lanes are unchanged (185/14/40/50, `wrong=0`).
**Implication for the determinism claim:** the digest is byte-deterministic at any
fixed commit. "Byte-identical replay" is safe to assert.
## 3. The safe-direction wrinkle (documented; fix deferred)
`composite_math_gate._compute_claim_digest` commits the GSM8K `honest_disclosure`
block into the gating digest. But ADR-0131 explicitly designates GSM8K as
**non-gating** disclosure. So improving a purely informational coverage metric
invalidates the expert signature even though every *gating* criterion still
passes (`technical_pass=True`).
This is a coupling smell, but it **fails toward `audit-passed`, never toward a
false `expert`** — so it is **not a `wrong=0` hazard**. Per the project's
no-scope-creep discipline, the fix (scope the digest to *gating* inputs only) is
**deferred to a future ADR**. This ADR only records the wrinkle so a future
reviewer does not rediscover it cold.
## 4. Decision — reconcile per artifact type
Reconciliation principle: **artifacts that describe HISTORY keep their content
with a dated "valid-at … auto-reverted … current = audit-passed" note (keep the
receipt; keep the mismatch-refusal firing); artifacts that assert CURRENT MACHINE
STATE reconcile to the truth.**
| Artifact | Type | Action | Rationale |
|---|---|---|---|
| `docs/reviewers.yaml` `math_expert_claims` entry | history (receipt) | **Keep the entry**; add a quarantine comment block (valid-at 2026-05-23; auto-reverted; do not re-sign). | The entry's *mismatch* is what makes the ledger refuse — keeping it keeps the safety mechanism visibly firing. |
| `ADR-0120-math-expert-ledger-flip.md` | history | Add a dated header note: valid at 2026-05-23/#267; auto-reverted; current ledger = `audit-passed`; see ADR-0200. | A decision record stays true to its moment; the note prevents misreading it as current state. |
| `evals/math_expert_claims/v1/expert_claims_math_v1_signed.json` | **current state** | **Regenerate** → `promote_admitted: false`, `reviewer_signature_matches: false`. | A machine-generated current-state file with a committed `true` is the live overclaim. The receipt survives in git history + the quarantined registry entry. |
| `README.md` §"Path to expert" + line-81 test count | current state | Reconcile the "next gate" narrative to the built-attempted-reverted story; verify/correct the test count. | The ledger table is already accurate; only the narrative tense and the test count are stale. |
| `docs/decisions/README.md` | current state | Repoint any "math = expert" reference to `audit-passed`. | Index must match the ledger. |
| `tests/test_mathlogic_expert_ledger_flip.py`, `tests/test_adr_0120_math_expert_promotion.py` | current state | Flip 3 currently-RED "is-expert" assertions into **fail-closed-revert** assertions (status==audit-passed; expert_reason explains the digest mismatch; `reviewer_signature_matches is False`). | These assert current machine state; reconciling them to the truth converts 3 red overclaim-tests into green mechanism-proving tests. |
### 4.1 Cascade safety (verified)
- **Regenerating the signed JSON does not cascade:** it is **not** SHA-pinned
(absent from `scripts/verify_lane_shas.py` and `scripts/generate_claims.py`),
no test reads its bytes, and only `core capability math-expert-promote` writes
it. The digest `4c46f530` appears only in the JSON itself and the (quarantined)
`reviewers.yaml` entry.
- **The 3 red tests are pre-existing** (red since the #310/#488 drift), not caused
by this reconciliation. They are folded in here so the expert-claim surface is
reconciled as one unit.
## 5. What this ADR does NOT do
- Does **not** re-sign the expert claim. Re-signing `02f6d3c8` would re-assert an
expert claim that (a) rests on CORE-authored lanes, not external GSM8K, and
(b) would be immediately re-broken by the next parser improvement. The honest
posture is `audit-passed` with the receipt preserved.
- Does **not** change any eval gate, threshold, or safety boundary.
- Does **not** fix the disclosure-in-digest coupling (§3) — deferred.
- Does **not** weaken any invariant. `wrong=0`, determinism, exact recall, and
versor closure are untouched.
## 6. Trust boundary
- **Reads:** the capability ledger, the committed evidence bundle, `reviewers.yaml`.
- **Writes:** documentation + the one regenerated current-state artifact + two
test files — all under operator ratification, on a branch, via PR. Nothing
lands on `main` directly.
- No dynamic imports, no network, no normalization, no stochastic path.
## 7. Consequences
- The README, the ledger, `reviewers.yaml`, and the tests tell **one** true story:
`audit-passed`, expert auto-reverted, fail-closed working.
- CI gains 3 green mechanism-proving tests where it had 3 red overclaim tests.
- A future ADR may scope the promotion digest to gating inputs only (§3).
- The reconciliation is the substrate for the CTO brief, whose honesty narrative
leads with this revert: *a non-gating metric drifted and the system revoked its
own expert claim — refuse-rather-than-guess applied to CORE's own status.*
## 8. CLAUDE.md PR-checklist
- **Capability/boundary protected:** restores claim↔machine-state coherence; keeps
the fail-closed refusal visibly firing; preserves every invariant.
- **Invariant proving field validity:** the ledger reports `expert` iff a signed
digest re-derives — and it currently, correctly, does not.
- **CLI/eval proving the lane:** `uv run core capability ledger` (status =
audit-passed); `uv run pytest tests/test_mathlogic_expert_ledger_flip.py tests/test_adr_0120_math_expert_promotion.py`.
- **Avoided hidden normalization / stochastic / approximate / unreviewed mutation:**
Yes — documentation + one deterministic regeneration; no engine change.
- **Trust boundary:** read-only inputs; writes are review-gated on a branch.