From 065826c556e809f0cbdcaa17e9d03f895b405982 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 28 Jul 2026 05:29:46 +0000 Subject: [PATCH] =?UTF-8?q?feat(tests):=20pin=203=20=E2=80=94=20a=20curate?= =?UTF-8?q?d=20suite=20no=20gate=20invokes=20is=20not=20a=20guarantee=20(P?= =?UTF-8?q?R-4,=20G-7)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR-4's membership ratchet closed "no new orphans". Within the hour this arc's first full-tree run found tests/test_ratification_ceremony.py RED, in the curated suite `teaching`, invoked by NO gate tier. Membership was satisfied and bought nothing — which is exactly the objection PR-4 itself raised against G-7's original formulation: a curated suite nobody runs is the same non-guarantee as `full` wearing a different name. The fix had the same blind spot as the thing it fixed. Measured: 21 curated suites, 2 gate-reachable (smoke, deductive). The gap is NINETEEN, not the handful first estimated. Shipped as baseline + ratchet, sized to the measurement rather than the rule: 19 `post-merge-only` justifications written in one sitting is the ceremony failure already rejected for the 749-file membership baseline. The pin freezes the gap, fails on growth, enforces both directions (a promoted suite must leave the baseline), pins the count, and — critically — VERIFIES ITS DECLARED GATE SET AGAINST THE SHELL (pre-push / local-ci.sh), so it cannot go vacuous the way a scan that silently matches nothing does (N-1's class, three false conclusions in this repo so far). Two sabotages observed red: a new curated suite with no gate caller, and a GATE_SUITES value drifted from the scripts it claims to describe. The second correctly cascaded into two failures, which is the anti-vacuity guard working. What pin 3 deliberately does NOT do: decide which of the 19 belong on the gate. That costs gate time and is a decision, not an engineering task. Recorded as the open half of G-7. `teaching` is the concrete first candidate — it just shipped a red ratification pin that no gate ran. Process note: a gate run was invalidated by editing the tree mid-run for the third time today. Stopped with TaskStop rather than pkill this time, which reaped cleanly instead of leaving the orphaned wait-loop that ran for 2h16m earlier. All edits now land before any run starts. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4 --- core/cli_test.py | 10 ++ docs/assessment/30-gap-register.md | 6 +- docs/assessment/50-execution-plan.md | 4 +- tests/test_suite_reachability.py | 183 +++++++++++++++++++++++++++ 4 files changed, 201 insertions(+), 2 deletions(-) create mode 100644 tests/test_suite_reachability.py diff --git a/core/cli_test.py b/core/cli_test.py index baa33617..a02e9301 100644 --- a/core/cli_test.py +++ b/core/cli_test.py @@ -119,6 +119,16 @@ TEST_SUITES: dict[str, tuple[str, ...]] = { # full-only files stay a declared, shrinking number rather than 749 # invented justifications (N-9). Filesystem + glob only, <1s. "tests/test_suite_membership.py", + # PR-4 pin 3 / G-7 — MEMBERSHIP WAS NEVER THE GUARANTEE; EXECUTION IS. + # The membership ratchet closed "no new orphans" and, within the hour, + # this arc's full-tree run found test_ratification_ceremony.py RED, in + # the curated suite `teaching`, invoked by no gate tier. Measured: 21 + # curated suites, 2 gate-reachable. This freezes the 19-suite gap so it + # cannot grow silently, enforced in both directions, with the declared + # gate set verified against the shell so the pin cannot go vacuous. + # It deliberately does NOT decide which of the 19 belong on the gate — + # that costs gate time and is Shay's call. <1s. + "tests/test_suite_reachability.py", # G-22 — CLAIMS.md is a PUBLISHED capability artifact, deterministically # regenerable from the capability ledger + PINNED_SHAS. It published a # superseded evidence digest for deduction_serve_v1 for two days after diff --git a/docs/assessment/30-gap-register.md b/docs/assessment/30-gap-register.md index 41a36214..43fe2fd8 100644 --- a/docs/assessment/30-gap-register.md +++ b/docs/assessment/30-gap-register.md @@ -74,7 +74,11 @@ Pin 1 exists because the two gates have **already** drifted (see H-12): the loca **REOPENED, PARTLY — 2026-07-28, by the arc's own full-tree run.** The ratchet closes *membership*. It does **not** close *execution*, and the difference is not academic: the first full-tree run of this arc found `tests/test_ratification_ceremony.py` **red on `main`, in a curated suite (`teaching`), and run by no gate** — the pre-push gate invokes `smoke` + the warmed_session lane + `deductive`, and nothing invokes `teaching`. That is exactly the hollow-gate failure this entry's own re-specification warned about — *"a curated suite nobody runs is the same non-guarantee as `full`, wearing a different name"* — demonstrated live, on a pin guarding ratification-corpus byte compatibility. -**What PR-4 still owes, therefore, is a third pin: reachability.** Every curated suite must be reachable from a gate tier (`scripts/hooks/pre-push` / `local-ci.sh`), or be declared post-merge-only with a stated reason. `teaching`, `packs`, `algebra`, `sensorium`, `cognition`, `runtime` and the rest are currently in the unreachable set — they have members, and no gate calls them. Membership was never the guarantee; execution is. +**PIN 3 LANDED 2026-07-28** — `tests/test_suite_reachability.py`, on the gate. **What PR-4 owed was a third pin: reachability.** Every curated suite must be reachable from a gate tier (`scripts/hooks/pre-push` / `local-ci.sh`), or be declared post-merge-only with a stated reason. `teaching`, `packs`, `algebra`, `sensorium`, `cognition`, `runtime` and the rest are currently in the unreachable set — they have members, and no gate calls them. Membership was never the guarantee; execution is. + +**Shipped as baseline + ratchet**, sized to the measurement rather than to the rule: **21 curated suites, 2 gate-reachable** (`smoke`, `deductive`). Demanding 19 `post-merge-only` justifications in one sitting is the ceremony failure already rejected for the 749-file membership baseline. So the pin freezes the 19-suite gap, fails on growth, enforces both directions (a promoted suite must leave the baseline), **verifies its declared gate set against the shell** so it cannot go vacuous, and pins the count. Two sabotages observed red — a new curated suite with no gate caller, and a `GATE_SUITES` value drifting from `pre-push`/`local-ci.sh`. + +**What pin 3 deliberately does not do — and what is now Shay's to decide:** *which* of the 19 belong on the gate. That costs gate time and is a real decision. `teaching` is the obvious first candidate: it just shipped a red ratification pin that no gate ran. Recorded as the open half of this entry. *One real gap the entry was circling, now fixed:* the parity pin lived in `fast`, which the pre-push gate does not run — **the pin guarding the gate did not run on the gate**. It is now in `smoke`. **Evidence:** `tests/test_suite_membership.py`; `tests/full_only_baseline.txt`; `tests/test_cli_test_suites.py:49`; `50fa287d`; `AGENTS.md:275-280`. · **Authority:** discharged mechanically; **no R-14 ruling is owed** for any of it. diff --git a/docs/assessment/50-execution-plan.md b/docs/assessment/50-execution-plan.md index 104b5dfc..4972d4d0 100644 --- a/docs/assessment/50-execution-plan.md +++ b/docs/assessment/50-execution-plan.md @@ -225,7 +225,9 @@ The assessment directory's own corrections (N-1…N-7 applied to `30-gap-registe **Verification:** three sabotages, each observed red — a new unregistered test file, a stale baseline entry, and the parity pin demoted out of `smoke`. Remediation half (adding the ten to `smoke.yml`) is **not owed**: it buys observability, not gating. -**STILL OWED — a third pin, found by this arc's own full-tree run (G-7 reopened-partly).** The ratchet closes membership; it does not close **execution**. The run found `tests/test_ratification_ceremony.py` red on `main`, **in a curated suite (`teaching`) that no gate invokes** — the gate runs `smoke` + warmed_session + `deductive`. Membership was satisfied and bought nothing, which is this item's own hollow-gate objection turned on its own fix. **Pin 3: every curated suite must be reachable from a gate tier, or declared post-merge-only with a reason.** Currently unreachable: `teaching`, `packs`, `algebra`, `sensorium`, `cognition`, `runtime`, and the rest — members, no caller. +**STILL OWED — a third pin, found by this arc's own full-tree run (G-7 reopened-partly).** The ratchet closes membership; it does not close **execution**. The run found `tests/test_ratification_ceremony.py` red on `main`, **in a curated suite (`teaching`) that no gate invokes** — the gate runs `smoke` + warmed_session + `deductive`. Membership was satisfied and bought nothing, which is this item's own hollow-gate objection turned on its own fix. **Pin 3 — LANDED 2026-07-28**, `tests/test_suite_reachability.py`. Measured **21 curated suites, 2 gate-reachable**; the gap is **19**, not the handful first estimated. Shipped as baseline + ratchet for the same reason the membership pin was: 19 justifications written in one sitting is ceremony, not a decision. Freezes the gap, fails on growth, both directions enforced, declared gate set verified against `pre-push`/`local-ci.sh` so it cannot go vacuous. Two sabotages observed red. + +**Still open, and it is a decision rather than an engineering task:** which of the 19 go on the gate. `teaching` first — it shipped a red ratification pin that no gate ran (G-22). ### PR-5 · The flag-default register · **M** · G-8/H-6 `docs/specs/flag_register.md`: all **28** default-off flags plus the 4 default-on — current default, deliberate-posture vs accumulated-hesitancy, **what evidence flips it**, and named profiles (one-shot / eval / continuous-life) per R-4. Declared in the table, not the call site (ADR-0263 Rule 5). A pin asserts the register lists exactly the flags `core/config.py` defines — so a new flag cannot land unregistered. Includes the F-6 accrual resolution from R-3 and the N-6 docstring correction. diff --git a/tests/test_suite_reachability.py b/tests/test_suite_reachability.py new file mode 100644 index 00000000..44f6ca75 --- /dev/null +++ b/tests/test_suite_reachability.py @@ -0,0 +1,183 @@ +"""PR-4 pin 3 / G-7 — a curated suite that no gate invokes is not a guarantee. + +PR-4's ratchet closed *membership*: a new test file cannot land outside every +curated suite. Within the hour it turned out membership was never the property +that mattered. + +This arc's first full-tree run found ``tests/test_ratification_ceremony.py`` +**red on main, in the curated suite `teaching`, and invoked by no gate tier**. +Membership was satisfied and bought nothing — which is precisely the objection +PR-4 itself raised against G-7's original formulation: *a curated suite nobody +runs is the same non-guarantee as ``full``, wearing a different name.* The fix +had the same blind spot as the thing it fixed. + +Measured at 2026-07-28: **21 curated suites, 2 gate-reachable.** ``smoke`` and +``deductive`` are invoked by ``scripts/hooks/pre-push`` and +``scripts/ci/local-ci.sh --tier gate``; the other nineteen have members and no +caller. That is why this is a **baseline plus ratchet** and not a flat rule — +demanding nineteen ``post-merge-only`` justifications in one sitting is the +ceremony failure rejected for the 749-file membership baseline, and it would +buy a wall of text rather than a decision. + +What this pin does: + * freezes the currently-unreachable set, so it cannot grow silently; + * fails when a **new** curated suite is added with no gate caller and no + explicit declaration; + * fails when a suite leaves the unreachable set (promoted to a gate) and its + baseline line is not removed — the same both-directions discipline that + caught this file's author twice already. + +What it deliberately does **not** do: decide which of the nineteen belong on +the gate. That is a real decision with a real time cost, and it is Shay's. +This pin makes the state visible and stops it worsening. +""" + +from __future__ import annotations + +import ast +import re +from pathlib import Path + +_ROOT = Path(__file__).resolve().parents[1] + +#: Suites invoked by a gate tier today. Parsed rather than hard-coded would be +#: nicer, but the invocations are shell (``step "…" suite smoke``) and a parse +#: that silently matched nothing would make this pin vacuous — the exact +#: failure mode (N-1) that has produced three false conclusions in this repo. +#: So the set is declared here and *verified* against the shell below. +GATE_SUITES: frozenset[str] = frozenset({"smoke", "deductive"}) + +#: Curated suites with members and no gate caller, measured 2026-07-28. +#: This list may only SHRINK. A suite leaves it by being invoked from a gate +#: tier — not by being deleted from this file. +UNREACHABLE_BASELINE: frozenset[str] = frozenset( + { + "adr-0024", + "algebra", + "cognition", + "fast", + "formation", + "full", + "inner-loop", + "margin", + "math", + "packs", + "phase5", + "phase6", + "proof", + "pulse", + "refusal", + "rotor", + "runtime", + "sensorium", + "teaching", + } +) + + +def _curated_suites() -> dict[str, tuple[str, ...]]: + src = (_ROOT / "core" / "cli_test.py").read_text(encoding="utf-8") + match = re.search(r"TEST_SUITES\s*(?::[^=]+)?=\s*\{", src) + assert match is not None, "TEST_SUITES literal not found" + start = match.end() - 1 + depth = 0 + end = len(src) + for i, ch in enumerate(src[start:], start): + if ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + end = i + 1 + break + return ast.literal_eval(src[start:end]) + + +def _suites_invoked_by_gate_tiers() -> set[str]: + """Suites named by a ``suite `` step in the hook or the gate tier.""" + found: set[str] = set() + for rel in ("scripts/hooks/pre-push", "scripts/ci/local-ci.sh"): + text = (_ROOT / rel).read_text(encoding="utf-8") + found.update(re.findall(r"--suite\s+([a-z0-9-]+)", text)) + found.update(re.findall(r"\bsuite\s+([a-z0-9-]+)\s*$", text, re.MULTILINE)) + # ``full`` is the arc-boundary tier, not a gate; it is not a merge bar. + found.discard("full") + found.discard("only") # prose match guard + return found + + +def test_declared_gate_suites_match_the_shell() -> None: + """GATE_SUITES must be what the hook and gate tier actually invoke. + + Without this, the pin below could pass while pointing at a fiction — a + declared set that drifted from the scripts it claims to describe. + """ + invoked = _suites_invoked_by_gate_tiers() + assert invoked, "parsed no suite invocations — the pin would be vacuous" + assert invoked == set(GATE_SUITES), ( + f"gate tiers invoke {sorted(invoked)}, but GATE_SUITES declares " + f"{sorted(GATE_SUITES)} — update the constant with the change that " + "moved the gate, and move any promoted suite out of " + "UNREACHABLE_BASELINE in the same edit" + ) + + +def test_no_new_gate_unreachable_suite() -> None: + """A new curated suite must be gate-reachable or explicitly baselined. + + If this fails on a suite you just added: either invoke it from a gate tier + (``scripts/hooks/pre-push`` / ``local-ci.sh --tier gate``), or add it to + UNREACHABLE_BASELINE with the reason in your commit message. The second is + available and is a decision, not a default — a suite nobody runs guarantees + nothing. + """ + suites = set(_curated_suites()) + unreachable = suites - set(GATE_SUITES) + undeclared = sorted(unreachable - UNREACHABLE_BASELINE) + assert not undeclared, ( + "these curated suites have members and no gate caller, and are not in " + f"the recorded baseline: {undeclared}" + ) + + +def test_baseline_has_no_stale_entries() -> None: + """Both directions: a promoted or deleted suite must leave the baseline. + + Same discipline as ``test_suite_membership.py`` — a baseline that keeps + entries it no longer needs stops being a measurement and becomes a story + about one. + """ + suites = set(_curated_suites()) + + vanished = sorted(s for s in UNREACHABLE_BASELINE if s not in suites) + assert not vanished, ( + f"baseline names suites that no longer exist: {vanished}" + ) + + promoted = sorted(s for s in UNREACHABLE_BASELINE if s in GATE_SUITES) + assert not promoted, ( + "these suites are now gate-reachable (good) but are still listed as " + f"unreachable — remove them from UNREACHABLE_BASELINE: {promoted}" + ) + + +def test_recorded_gate_gap_matches_the_measurement() -> None: + """Pin the number so the gap moving is a reviewed decision, not a drift. + + 19 of 21 at 2026-07-28. Shrinking this is the point — it shrinks by putting + a suite on the gate, which costs gate time and is therefore a decision + someone has to make deliberately. + """ + assert len(UNREACHABLE_BASELINE) <= 19, ( + f"gate-unreachable suites grew to {len(UNREACHABLE_BASELINE)} (was 19). " + "The ratchet only turns one way." + ) + + +def test_the_two_gate_suites_actually_exist() -> None: + """A gate that names a suite the CLI does not define would fail closed at + run time, but silently here. Pin it.""" + suites = _curated_suites() + for name in GATE_SUITES: + assert name in suites, f"gate tier invokes undefined suite {name!r}" + assert suites[name], f"gate suite {name!r} is empty"