core/scripts
Claude 68e68589a2
feat(tests): curated-suite membership ratchet — and PR-4's parity pin withdrawn (PR-4, G-7, N-9)
N-9 is the finding, and it PREVENTS work rather than re-scoping it.

WHAT THE PLAN GOT WRONG. N-3 concluded the 23-vs-13 smoke delta was unintended
drift, on two in-repo comments asserting parity, and called that "the strongest
evidence available that the drift was never intended". H-12 repeated it; R-14
built three options on it recommending "raise CI, +46s measured"; PR-4
specified a bidirectional parity pin as deliverable one.

A third artifact answers all of it, and none of them read it.
tests/test_cli_test_suites.py::test_cli_smoke_suite_covers_ci_smoke_gate has
pinned smoke.yml SUBSET-OF TEST_SUITES["smoke"] since before this assessment —
which is why every measurement finds CI-only == 0; a pin, not luck — beneath a
comment headed "DELIBERATELY ONE-DIRECTIONAL — do not 'complete' this by also
asserting local_paths <= ci_paths". Its history is 50fa287d (2026-07-25):
"revert(tests): drop the local-to-CI smoke parity assertion — CI is not a
gate." An earlier agent read the same comment, drew the same inference, made
the assertion symmetric, and reverted it hours later: "That was wrong, and
AGENTS.md says so in a line I had already read."

AGENTS.md:280 — GitHub is "a mirror only; its Actions are billing-locked and
produce dead signals — never chase them." §277 — the workflows are "secondary
observability only." So the local suite is the SOURCE and is deliberately
broader; the delta is the design.

  - PR-4 pin 1: WITHDRAWN. It exists, in the direction that protects something.
  - R-14: premise corrected in the packet. Option A spends 46s on a workflow
    that does not gate; option B would delete real protection for a fiction;
    option C's live half is a two-line comment fix, done here.
  - N-3's exposure claim was too GENEROUS to CI: for a push that skipped the
    local gate there is no automatic check at all. Larger than stated, and not
    closable by editing smoke.yml.

WHAT LANDED. Measured: 749 of 877 test files are in no curated suite. The
plan's mechanism ("assign every orphan, or 749 exclusion reasons") is hollow at
that scale — glob topic-suites ("adr": tests/test_adr_*.py absorbs 172) satisfy
the assertion while changing nothing about what executes, which is precisely the
Third-Door objection the plan levels at G-7's own first formulation. All four
demonstrated incidents (#113, #136, negation-survives-articulation,
speculative-subject-lifecycle) were caused by a NEWLY LANDED orphan; none by a
legacy one.

So: a ratchet. tests/test_suite_membership.py + tests/full_only_baseline.txt.
New orphans fail. The baseline is enforced in BOTH directions — a promoted or
deleted file must leave it, so the list can only shrink — and its count is
pinned so bulk movement is a reviewed decision, in the discipline
test_volume_honesty.py uses.

Also: the gate-guarding pin now runs on the gate. test_cli_test_suites.py lived
in `fast`; the pre-push gate runs smoke + deductive. The pin guarding the gate
did not run on the gate.

Three sabotages, each OBSERVED RED: a new unregistered test file, a stale
baseline entry, the parity pin demoted out of smoke.

Note on the delta: now 12 (local 27, CI 15) because this session added three
pins locally. Under the corrected reading that is the design, not widening
drift — the invariant is CI-only == 0, and that is pinned. The registers now
say so; "ten files" was a measurement, never a target.

Process note: `git checkout --` during sabotage cleanup destroyed uncommitted
edits to core/cli_test.py, which were redone. Backups, not checkout, on files
with unstaged work.

Registers: G-7 CLOSED, H-12 amended, R-14 dissolved in the packet, PR-4
re-specified, N-9 added to plan §0.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4
2026-07-28 04:03:51 +00:00
..
benchmarks feat(evals): implement safe metadata and fail-closed evaluator check for GSM1K 2026-06-23 07:52:32 -07:00
ci ci(local): one-command local-first runner; fix a lane roster red on main 2026-07-25 12:51:17 +00:00
hooks feat(tests): curated-suite membership ratchet — and PR-4's parity pin withdrawn (PR-4, G-7, N-9) 2026-07-28 04:03:51 +00:00
ops feat(telemetry): discovery-yield-per-served-turn baseline (ADR-0255) 2026-07-23 07:27:00 -07:00
probes chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
__init__.py
agent_startup.sh chore(dev): enforce fresh-base startup guard for CORE agents (#849) 2026-06-20 20:11:09 -07:00
build_refusal_taxonomy_cases.py fix(refusal-taxonomy): parse new refusal-reason format; reconcile to reader gains 2026-06-03 01:49:41 -07:00
calibrate_ratification_threshold.py
dump-api-schemas.py
dump-enums.py feat(workbench): CORE-Logos Studio LG-2 — /logos route shell + Overview/Identity/Safety 2026-06-14 12:12:47 -07:00
dump-schemas.py feat(workbench): design mastery pass — chain rail, typography, doctrine-as-tests (Wave R R1) 2026-06-12 13:18:44 -07:00
generate_claims.py feat(generalization): curriculum-grounded serving — exams answered from ratified curriculum (ADR-0262) 2026-07-24 14:38:08 -07:00
generate_discourse_paragraph.py
generate_en_numerics_v1.py Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
generate_english_fluency_ood.py
generate_grammatical_v2.py
generate_gsm8k_public_corpus.py feat(evals): Author 200-case GSM8K math evaluation corpus and verification script 2026-07-04 16:03:51 -07:00
generate_identity_curriculum.py
generate_identity_test_cases.py
generate_monotonic_cases.py
generate_phase5_domain_lanes.py
generate_phase5_language_lanes.py Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
gsm8k_experience_flywheel.py feat(gsm8k): add bounded experience flywheel for sealed practice 2026-06-17 21:05:02 -07:00
gsm8k_frontier_report.py feat(derivation): Workstream A inc 2 — frontier report + rate_with_currency apply_rate injection (#797) 2026-06-17 06:38:45 -07:00
gsm8k_microscope.py
gsm8k_post_gate_a1_frontier_microscope.py feat(derivation): Gate A2a unit partition injection (#809) 2026-06-17 18:14:24 -07:00
gsm8k_problem_frame_adequacy.py feat(kernel): implement ProblemFrame proportional-change closure (PR slice 1) (#835) 2026-06-20 10:58:43 -07:00
gsm8k_sealed_attempt_scout.py feat(evals): add GSM8K sealed attempt scout (#812) 2026-06-17 19:55:13 -07:00
gsm8k_substrate_morphology.py Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
measure_grammar_seam.py refactor(generate): one owner per linguistic fact (Phase 2A) 2026-07-26 17:45:27 -07:00
measure_trackb_inc2.py fix(trackb): pure-S1 extract refuse multi-clause overmatch (0361 wrong=0) 2026-07-19 20:04:27 -07:00
measure_trackb_s1_slice.py feat(trackb): S1 symbolic structure-mapping vertical slice (off-serving) 2026-07-19 19:01:23 -07:00
package_apple_uma_demo.py feat(bench): add Apple UMA demo package builder (#913) 2026-06-24 15:51:45 -07:00
publish_pack_measurements.py
ratify_anchor_lens_packs.py Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
ratify_ethics_pack.py
ratify_identity_packs.py
ratify_register_packs.py
ratify_rhetorical_style_packs.py
ratify_safety_pack.py chore: Refactor CLI and Governance Anchors (#926) 2026-07-03 12:34:56 -07:00
review_trace.py
run_examples.py
run_pulse.py Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
scaffold_phase5_domain_lanes.py
score_phase5_holdouts.py
seal_gsm8k_test.py
seal_holdouts.py
setup_codex_environment.sh chore(dev): restore tracked agent session launchers (#903) 2026-06-24 12:19:53 -07:00
start-codex-session.zsh chore(dev): restore tracked agent session launchers (#903) 2026-06-24 12:19:53 -07:00
start-grok-session.zsh chore(dev): restore tracked agent session launchers (#903) 2026-06-24 12:19:53 -07:00
test_ffi_layout.py refactor(kernel): implement Lane 1 VersorBinding and Geometric ContractAssessment 2026-07-03 21:05:25 -07:00
verify_definitional_closure.py Lane 4: Registry Consolidation (language_packs to packs) 2026-07-04 15:11:28 -07:00
verify_lane_shas.py fix(evals): the deduction lane hashes the prose it serves 2026-07-26 19:45:48 -07:00
verify_semantic_equivalence.py test(derivation): add ADR-0184 semantic replay equivalence harness 2026-06-10 14:35:22 -07:00
workbench fix(core): lock python and extend rust cga surface (#925) 2026-07-02 15:32:08 -07:00