Rank 5 on the docket, and the first item this arc whose expected outcome is LOSING capability. conservative_floor is a one-sided Wilson lower bound, and Wilson assumes independent trials. CORE's pipeline is deterministic, so replaying an identical case is not a second trial — it is the same trial with a guaranteed outcome. The deduction sealer folded the raw practice corpus and counted every replay, so bands recorded 720/720 committed whether that was 720 decisions or 28 decisions seen 26 times each. conservative_floor(720,720)=0.990868 cleared theta_SERVE=0.99; conservative_floor(28,28)=0.808413 is not close. RESULT: 25 licensed bands -> 4. Twenty-one demoted. Survivors: en_conditional_chain, en_disjunctive, en_verb_fact, en_verb_universal — the only bands whose corpus holds enough independent cases. What the demotion did and did not do, stated exactly. NO ANSWER CHANGED. No answer became wrong. `wrong` stayed 0 across all 25 bands. The engine is exactly as correct as it was. What changed is the CLAIM attached to the answer: 21 bands now serve the same sound conclusion prefixed with "(reasoned, but I haven't yet earned a verified track record on arguments of this shape)". The reasoning did not get worse; the boast did. The producer is hardened, which is the half that stops this recurring. seal_ledger now refuses outright — assert_sealed_evidence_distinct compares the ledger about to be written against the corpus's distinct-case count and raises BEFORE anything is written. The curriculum sealer has carried this guarantee since ADR-0264 R9; the deduction sealer had no equivalent, which is exactly how the exposure arose. Catching it at seal time rather than in an audit matters: an audit finds a padded ledger after it is committed, trusted, and gating a live flag, and unwinding that then needs a ruling — it needed one. A HOLLOW GUARD OF MY OWN, caught by sabotage rather than trusted. The first version compared all_gold_problems() with distinct_gold_problems() — two functions that agree with each other by construction, and neither of which is what build_ledger folds. Reverting build_ledger to the raw corpus walked straight past it AND re-sealed the inflated artifact. Same defect class the ledger already suffered: a guard checking something adjacent to the thing it protects. It now takes the BUILT LEDGER as input, because the invariant is about the artifact. Re-running the identical attack: refused, wrote nothing. Pinned as test_sealer_refuses_a_ledger_that_counts_replays. A SECOND MISS, caught by the gate rather than by me. My blast-radius scan grepped for deduction_serve_license / deduction_serve_ledger / 720 and found nine files. It missed tests/test_construction_inventory.py, which asserts deduction SURFACE STRINGS and never mentions the ledger — so a pattern search over the wrong noun could not see it. The full gate found it. Recorded because "I scanned for the blast radius" is worth exactly as much as the scan's key, and mine was the wrong key. That test is the #138 fabrication defect pin, and it is updated WITHOUT being weakened. Both surfaces now carry the disclosure prefix, and the fabrication survives untouched — which is a sharper finding than before: a disclosure prefix looks like it might cover the hazard, and it does not. The user is still told they said "furthermore". Exact-match assertions kept on the full strings so neither the fabrication nor the licence state can drift unnoticed, plus an isolating assertion that the only difference between clean and leaked recitals is the fabricated premise. Four records corrected in the same PR, each of which would otherwise have gone on asserting capability that no longer exists: - CAPABILITY_LEDGERS' manifest note ("25 bands at 720/720 wrong=0") - the published CLAIMS.md claim — caught immediately by G-22's own pin, which is that fix earning its keep three days after landing - test_volume_honesty.py's module docstring, which said the exposure was "NOT silently repaired ... changing it is Shay's ratification, not a test's". R-13 IS that ratification, so the line went stale by being acted on - the exposure inventory's framing: it now pins the repair, not the gap test_volume_honesty's sealed-vs-producer assertion is strictly stronger than it was. It compared sealed committed counts to the producer's committed counts and both were the inflated 720 — the assertion held perfectly while the number it agreed on was the wrong number. Two records agreeing is not evidence that either is right. It now pins committed == distinct, plus a non-vacuity check. Nine tests encoded the unearned expectation and were updated to the honest one. Closes G-19 and H-1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4
284 lines
9.7 KiB
Python
284 lines
9.7 KiB
Python
"""Generate CLAIMS.md from the capability ledger and pinned lane SHAs.
|
|
|
|
CLAIMS.md is a single-page, auditable list of every CI-pinned
|
|
capability claim CORE currently makes. It is **mechanically derived**
|
|
from two ground-truth sources:
|
|
|
|
- ``core.capability.ledger_report()`` — domain ratification rows
|
|
(Tier 1: "this domain is reasoning-capable").
|
|
- ``scripts.verify_lane_shas.PINNED_SHAS`` + ``LANE_SPECS`` —
|
|
per-lane pinned report SHA-256 (Tier 2: "this lane's report bytes
|
|
hash to X under deterministic replay").
|
|
|
|
The generator is **deterministic**: same inputs → byte-identical
|
|
output. ``tests/test_claims_md_is_current.py`` regenerates the file
|
|
into a temp path and asserts byte equality with the on-disk copy.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import hashlib
|
|
import sys
|
|
from dataclasses import dataclass
|
|
from pathlib import Path
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
|
CLAIMS_PATH = REPO_ROOT / "CLAIMS.md"
|
|
|
|
# Import via path so this script remains runnable without installing the
|
|
# package.
|
|
sys.path.insert(0, str(REPO_ROOT))
|
|
|
|
from core.capability import ledger_report # noqa: E402
|
|
from scripts.verify_lane_shas import LANE_SPECS, PINNED_SHAS # noqa: E402 # type: ignore[import-not-found]
|
|
|
|
|
|
# ADR → human-readable lane purpose. Keep this in lockstep with the
|
|
# ADR docs themselves; the generator fails fast if any lane lacks a
|
|
# mapping here, which surfaces drift before CI does.
|
|
_LANE_ADR: dict[str, tuple[str, str]] = {
|
|
"reviewer_registry": (
|
|
"ADR-0092",
|
|
"Reviewer registry schema validates + bootstrap entry self-seals",
|
|
),
|
|
"miner_loop_closure": (
|
|
"ADR-0095",
|
|
"Miner-sourced proposals route through single reviewed teaching path",
|
|
),
|
|
"curriculum_loop_closure": (
|
|
"ADR-0104",
|
|
"Curriculum-sourced proposals route through single reviewed teaching path",
|
|
),
|
|
"domain_contract_validation": (
|
|
"ADR-0093",
|
|
"All ratified packs satisfy the 9 ADR-0091 contract predicates",
|
|
),
|
|
"fabrication_control_summary": (
|
|
"ADR-0096",
|
|
"Phantom endpoints / cross-pack non-bridges / sibling collapses refuse",
|
|
),
|
|
"demo_composition": (
|
|
"ADR-0098",
|
|
"Demos compose from shipped modules; no parallel mechanism",
|
|
),
|
|
"public_demo": (
|
|
"ADR-0099",
|
|
"Public showcase runs deterministically under 30s; all claims supported",
|
|
),
|
|
"math_teaching_corpus_v1": (
|
|
"ADR-0131",
|
|
"Math teaching corpus replays deterministically; all chains pass exit criterion (correct_rate=1.0, wrong=0)",
|
|
),
|
|
"deductive_logic_v1": (
|
|
"ADR-0206",
|
|
"Propositional entailment scored against an independent truth-table oracle; dev+holdout+external 716/716 correct, wrong=0, refused=0",
|
|
),
|
|
"curriculum_serve_v1": (
|
|
"ADR-0262",
|
|
"Flag-gated curriculum serving answers exam questions from a subject's RATIFIED chain corpus only; untaught facts return UNKNOWN (anti-recall probes enforced), wrong=0",
|
|
),
|
|
"deduction_serve_v1": (
|
|
"ADR-0256",
|
|
"Flag-gated deduction serving decides real English/member/fused/verb/existential arguments end-to-end, wrong=0 across all splits; 4 of 25 shape-bands hold an earned SERVE licence on DISTINCT evidence (R-13 re-count, 2026-07-28) and the other 21 decide with disclosure",
|
|
),
|
|
}
|
|
|
|
|
|
# Domain → ratifying ADR. The ledger row already carries provenance
|
|
# inside each contract (e.g. "adr-0101:reviewed:2026-05-21"), but we
|
|
# pin the canonical ADR id here so the table reads cleanly even if a
|
|
# pack later carries a different provenance prefix.
|
|
_DOMAIN_ADR: dict[str, str] = {
|
|
"systems_software": "ADR-0101",
|
|
"mathematics_logic": "ADR-0097",
|
|
"physics": "ADR-0100",
|
|
"hebrew_greek_textual_reasoning": "ADR-0102",
|
|
"philosophy_theology": "ADR-0085",
|
|
}
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class DomainRow:
|
|
domain: str
|
|
status: str
|
|
adr: str
|
|
pack_count: int
|
|
open_gaps_count: int
|
|
|
|
|
|
@dataclass(frozen=True, slots=True)
|
|
class LaneRow:
|
|
lane_id: str
|
|
adr: str
|
|
purpose: str
|
|
report_path: str
|
|
pinned_sha: str
|
|
|
|
|
|
def _collect_domain_rows() -> list[DomainRow]:
|
|
report = ledger_report()
|
|
rows: list[DomainRow] = []
|
|
for entry in report["domains"]:
|
|
domain = entry["domain"]
|
|
if domain not in _DOMAIN_ADR:
|
|
raise RuntimeError(
|
|
f"domain {domain!r} missing from _DOMAIN_ADR mapping in "
|
|
f"scripts/generate_claims.py — add its ratifying ADR id."
|
|
)
|
|
rows.append(
|
|
DomainRow(
|
|
domain=domain,
|
|
status=entry["status"],
|
|
adr=_DOMAIN_ADR[domain],
|
|
pack_count=len(entry["packs"]),
|
|
open_gaps_count=len(entry["open_gaps"]),
|
|
)
|
|
)
|
|
# Deterministic ordering: by ADR id (lexicographic), then domain.
|
|
return sorted(rows, key=lambda r: (r.adr, r.domain))
|
|
|
|
|
|
def _collect_lane_rows() -> list[LaneRow]:
|
|
rows: list[LaneRow] = []
|
|
for spec in LANE_SPECS:
|
|
if spec.lane_id not in _LANE_ADR:
|
|
raise RuntimeError(
|
|
f"lane {spec.lane_id!r} missing from _LANE_ADR mapping in "
|
|
f"scripts/generate_claims.py — add its ADR id and purpose."
|
|
)
|
|
adr, purpose = _LANE_ADR[spec.lane_id]
|
|
pinned = PINNED_SHAS.get(spec.lane_id)
|
|
if not pinned:
|
|
raise RuntimeError(
|
|
f"lane {spec.lane_id!r} has no entry in PINNED_SHAS — "
|
|
f"run scripts/verify_lane_shas.py --update first."
|
|
)
|
|
rows.append(
|
|
LaneRow(
|
|
lane_id=spec.lane_id,
|
|
adr=adr,
|
|
purpose=purpose,
|
|
report_path=spec.report_relative,
|
|
pinned_sha=pinned,
|
|
)
|
|
)
|
|
return sorted(rows, key=lambda r: r.adr)
|
|
|
|
|
|
def _render(
|
|
*, domain_rows: list[DomainRow], lane_rows: list[LaneRow]
|
|
) -> str:
|
|
"""Render CLAIMS.md as a deterministic markdown string.
|
|
|
|
Two tables. No timestamps, no git revs, no wall-clock — anything
|
|
volatile would break the "regenerate and compare bytes" test.
|
|
"""
|
|
lines: list[str] = [
|
|
"# CLAIMS",
|
|
"",
|
|
"<!--",
|
|
"AUTO-GENERATED by scripts/generate_claims.py.",
|
|
"Do not hand-edit. Run `python3 scripts/generate_claims.py` to",
|
|
"regenerate after any change to the capability ledger or pinned",
|
|
"lane SHAs (scripts/verify_lane_shas.py).",
|
|
"-->",
|
|
"",
|
|
"Every row below is mechanically derived from in-tree state and",
|
|
"verified by CI. Tier 1 rows come from `core.capability.ledger_report`;",
|
|
"Tier 2 rows come from `scripts/verify_lane_shas.py`'s pinned SHAs.",
|
|
"",
|
|
"## Tier 1 — Ratified domains",
|
|
"",
|
|
"Each row asserts: the domain's packs pass all nine ADR-0091 contract",
|
|
"predicates, declared operator chains meet the ≥8 / ≥3-intent floor,",
|
|
"the reviewer registry resolves the primary reviewer, and the ledger",
|
|
"status predicate evaluates to `reasoning-capable` with no open gaps.",
|
|
"",
|
|
"| Domain | Status | ADR | Packs | Open gaps |",
|
|
"| --- | --- | --- | --- | --- |",
|
|
]
|
|
for row in domain_rows:
|
|
lines.append(
|
|
f"| `{row.domain}` | {row.status} | {row.adr} | {row.pack_count} | "
|
|
f"{row.open_gaps_count} |"
|
|
)
|
|
lines.extend(
|
|
[
|
|
"",
|
|
"## Tier 2 — Pinned lane reports",
|
|
"",
|
|
"Each row asserts: running the lane's runner produces a JSON",
|
|
"report whose SHA-256 matches the pinned value below. Mismatch",
|
|
"is a CI failure (`.github/workflows/lane-shas.yml`).",
|
|
"",
|
|
"| ADR | Lane | Purpose | Report path | Pinned SHA-256 |",
|
|
"| --- | --- | --- | --- | --- |",
|
|
]
|
|
)
|
|
for row in lane_rows:
|
|
lines.append(
|
|
f"| {row.adr} | `{row.lane_id}` | {row.purpose} | "
|
|
f"`{row.report_path}` | `{row.pinned_sha}` |"
|
|
)
|
|
lines.extend(
|
|
[
|
|
"",
|
|
"## Verification",
|
|
"",
|
|
"```bash",
|
|
"python3 scripts/verify_lane_shas.py # verify all Tier 2 SHAs",
|
|
"core test --suite full -q # exercises Tier 1 invariants",
|
|
"```",
|
|
"",
|
|
]
|
|
)
|
|
return "\n".join(lines)
|
|
|
|
|
|
def render_claims() -> bytes:
|
|
domain_rows = _collect_domain_rows()
|
|
lane_rows = _collect_lane_rows()
|
|
text = _render(domain_rows=domain_rows, lane_rows=lane_rows)
|
|
return text.encode("utf-8")
|
|
|
|
|
|
def main(argv: list[str] | None = None) -> int:
|
|
parser = argparse.ArgumentParser(description="generate CLAIMS.md")
|
|
parser.add_argument(
|
|
"--check",
|
|
action="store_true",
|
|
help="exit non-zero if CLAIMS.md differs from regenerated bytes",
|
|
)
|
|
parser.add_argument(
|
|
"--output",
|
|
type=Path,
|
|
default=CLAIMS_PATH,
|
|
help="write to this path (default: repo-root CLAIMS.md)",
|
|
)
|
|
args = parser.parse_args(argv)
|
|
|
|
payload = render_claims()
|
|
sha = hashlib.sha256(payload).hexdigest()
|
|
|
|
if args.check:
|
|
if not CLAIMS_PATH.exists():
|
|
print(f"FAIL: {CLAIMS_PATH} does not exist; run without --check")
|
|
return 1
|
|
on_disk = CLAIMS_PATH.read_bytes()
|
|
if on_disk != payload:
|
|
print("FAIL: CLAIMS.md is stale — regenerate with:")
|
|
print(" python3 scripts/generate_claims.py")
|
|
print(f" on-disk sha256: {hashlib.sha256(on_disk).hexdigest()}")
|
|
print(f" computed sha256: {sha}")
|
|
return 1
|
|
print(f"OK: CLAIMS.md is current (sha256: {sha})")
|
|
return 0
|
|
|
|
args.output.write_bytes(payload)
|
|
print(f"wrote {args.output} (sha256: {sha})")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|