fix(tests,claims): G-22 — main was red, and CLAIMS.md was publishing evidence that no longer existed
Ruled: fix, don't document. The argument that settled it — a red test on a
serving-path contract left with an explanatory note IS the H-8 failure mode, a
document acknowledging a gap instead of closing it, which becomes the next
assessor's finding.
Two distinct failures, and the second is the serious one.
1. THE CEREMONY PINS WERE STALE AGAINST CORRECT CODE (H-8e's class).
ChainRecord.polarity defaults to AFFIRMATIVE and is DELIBERATELY OMITTED
from affirmative rows (ADR-0264 R1; teaching/ratification.py:119). Both pins
built ChainRecord(**{k: row[k] for k in __slots__}), so they KeyError'd on
every correctly-serialized row. Now built from the row's own keys, letting
absent fields take their dataclass defaults — which RESTORES the guarantee
rather than weakening it. Consequence of the staleness: ratification-corpus
byte compatibility and unreviewed-status refusal were UNVERIFIED since
ADR-0264 R1 landed.
2. CLAIMS.md PUBLISHED A SUPERSEDED EVIDENCE DIGEST FOR A LICENSED CAPABILITY.
f9e9cc0c (2026-07-26, "the deduction lane hashes the prose it serves")
strengthened the lane, rewrote evals/deduction_serve/report.json, and moved
the authoritative pin in scripts/verify_lane_shas.py:60 to c855d55c —
recording the superseded 0b461a5a on line 56. It did NOT regenerate
CLAIMS.md. So for two days the published claim for deduction_serve_v1
pointed at evidence that no longer existed. The claim TEXT never changed;
only the pointer was wrong — which is exactly the failure a digest exists to
prevent. Regenerated; the diff is one line, the digest.
The drift had reached a THIRD artifact: this assessment's own
M3-comprehension-reasoning.md card cited the same stale 0b461a5a as its
would-fail-if-absent evidence. Corrected.
THE MECHANISM IS G-7's, AGAIN. One commit updated the lane, its verifier and
its lane test, and missed one downstream artifact — and the pin that guards
that artifact (tests/test_claims_md_is_current.py) EXISTED AND RAN ON NO GATE.
Nothing surfaced it until this arc ran the full tree once. Both it and
test_ratification_ceremony.py are now registered in TEST_SUITES["smoke"].
The membership ratchet caught the author a second time: promoting
test_claims_md_is_current.py onto the gate without removing its baseline line
failed test_baseline_has_no_stale_entries immediately. Baseline 748 -> 747.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wcw2pnMBwyvmNyQg4uPEt4
This commit is contained in:
parent
913111c0dc
commit
0329cc99fa
6 changed files with 37 additions and 6 deletions
|
|
@ -43,7 +43,7 @@ is a CI failure (`.github/workflows/lane-shas.yml`).
|
|||
| ADR-0104 | `curriculum_loop_closure` | Curriculum-sourced proposals route through single reviewed teaching path | `evals/curriculum_loop_closure/results/v1_dev.json` | `cb94ca0042d78ec2624129ff6493d52e767b69feea32d2997b85d88f1c0883af` |
|
||||
| ADR-0131 | `math_teaching_corpus_v1` | Math teaching corpus replays deterministically; all chains pass exit criterion (correct_rate=1.0, wrong=0) | `evals/math_teaching_corpus/v1/report.json` | `eaf160d145da29f9050ede8d58bf111b0f651dd40aeae9201857d0b97e014dd4` |
|
||||
| ADR-0206 | `deductive_logic_v1` | Propositional entailment scored against an independent truth-table oracle; dev+holdout+external 716/716 correct, wrong=0, refused=0 | `evals/deductive_logic/report.json` | `97a230949016e38d5e3f37a69e4245b320575ee70e5af92ff7607f7b05f74b5f` |
|
||||
| ADR-0256 | `deduction_serve_v1` | Flag-gated deduction serving decides real English/member/fused/verb/existential arguments end-to-end under earned SERVE licenses; wrong=0 across all splits | `evals/deduction_serve/report.json` | `0b461a5a49c8f8260ca87d0c9c9f9a17232bd1fdedd982e34649eedf9cca30b5` |
|
||||
| ADR-0256 | `deduction_serve_v1` | Flag-gated deduction serving decides real English/member/fused/verb/existential arguments end-to-end under earned SERVE licenses; wrong=0 across all splits | `evals/deduction_serve/report.json` | `c855d55cf316471fdfe092aa0d5c954e5ceb6f30c9a8db283e0b9a5d5e8b419a` |
|
||||
| ADR-0262 | `curriculum_serve_v1` | Flag-gated curriculum serving answers exam questions from a subject's RATIFIED chain corpus only; untaught facts return UNKNOWN (anti-recall probes enforced), wrong=0 | `evals/curriculum_serve/report.json` | `d9e7ba500f040b865870413a940ee9a49910ac22e1a89c9feec1a60bdd2513f1` |
|
||||
|
||||
## Verification
|
||||
|
|
|
|||
|
|
@ -119,6 +119,20 @@ TEST_SUITES: dict[str, tuple[str, ...]] = {
|
|||
# full-only files stay a declared, shrinking number rather than 749
|
||||
# invented justifications (N-9). Filesystem + glob only, <1s.
|
||||
"tests/test_suite_membership.py",
|
||||
# G-22 — CLAIMS.md is a PUBLISHED capability artifact, deterministically
|
||||
# regenerable from the capability ledger + PINNED_SHAS. It published a
|
||||
# superseded evidence digest for deduction_serve_v1 for two days after
|
||||
# f9e9cc0c moved the lane, because that commit updated the lane, the
|
||||
# verifier and the lane test but not the published claim — and this pin,
|
||||
# which would have caught it, ran in no curated suite. A claim whose
|
||||
# evidence digest does not match its evidence is the exact failure the
|
||||
# digest exists to prevent. <1s.
|
||||
"tests/test_claims_md_is_current.py",
|
||||
# G-22 — ratification-corpus byte compatibility and unreviewed-status
|
||||
# refusal. Red on main and unverified since ADR-0264 R1 landed, because
|
||||
# the pins iterated ChainRecord.__slots__ while affirmative rows omit
|
||||
# `polarity` by design. Governance-path contract; belongs on the gate.
|
||||
"tests/test_ratification_ceremony.py",
|
||||
# PR-6 / G-9 — three AGENTS.md prohibitions enforced by test instead of
|
||||
# review: exact recall (no cosine/ANN/HNSW on any path that decides
|
||||
# recalled truth — the prior ban covered ONE physics module while
|
||||
|
|
|
|||
|
|
@ -40,7 +40,7 @@ Operationally, the live path enters M3 after ingest: intent classification → `
|
|||
- **Design:** **ADR-0252 (Accepted, governing)** — the problem-solving paradigm and its §4 conformance bar; ADR-0251 (halt bespoke per-case regex work; the prohibition governing every math-reader increment); ADR-0249/0250 (reader→Hamiltonian compiler); ADR-0256–0261 (six deduction bands); ADR-0262/0264 (curriculum); ADR-0265 (negation in the proposition graph); ADR-0243 (wave-field lifecycle); ADR-0142 (epistemic taxonomy).
|
||||
- **Build:** `partial-wiring-debt`.
|
||||
- **Evidence:**
|
||||
- Deduction serving decides real arguments end-to-end, `wrong=0` across all splits — lane — `evals/deduction_serve/report.json`, pinned SHA `0b461a5a…` in `CLAIMS.md` — would-fail-if-absent: **yes**.
|
||||
- Deduction serving decides real arguments end-to-end, `wrong=0` across all splits — lane — `evals/deduction_serve/report.json`, pinned SHA `c855d55c…` in `CLAIMS.md` *(corrected 2026-07-28, G-22: the card cited the digest superseded by `f9e9cc0c`, which strengthened the lane to hash the prose it serves; `CLAIMS.md` had not been regenerated, so three artifacts published a stale evidence pointer for two days)* — would-fail-if-absent: **yes**.
|
||||
- Propositional entailment scored against an independent truth-table oracle, 716/716 correct, `wrong=0`, `refused=0` — lane — `evals/deductive_logic/report.json`, pinned `97a23094…` — would-fail-if-absent: **yes**.
|
||||
- `deductive` suite exists and carries 20 test files — code-read — `core/cli_test.py` — would-fail-if-absent: **yes**.
|
||||
- 25 sealed bands / 18,000 cases / `wrong=0`; capability index breadth 13, `wrong_total=0` — measurement — `chat/data/deduction_serve_ledger.json`, `evals/capability_index/baseline.json` (confirmed mechanically 2026-07-25).
|
||||
|
|
|
|||
|
|
@ -136,7 +136,13 @@ Own `use_salience`, the two underived constants, the self-narrowing budget feedb
|
|||
- **G-17 · Non-text ingest** — 59 sensorium modules, no serving path, no entry criterion; projection heads do not exist. Position paper is honest about this. Needs either an entry criterion or an explicit deferral ruling (the falsification bench is the standard the track should be held to when it moves). **Authority:** ruling.
|
||||
- **G-18 · Identity-divergence curriculum may still bypass formation's gates** — known gap since 2026-05-17 (`teaching_order.md`); unverified at this SHA. **Authority:** Phase-3-style verification pass, then a routing PR.
|
||||
- **G-19 · Wilson/replay evidence shortfall** — 21/25 ratified bands short if replays were counted as independent trials (see H-1 for the mechanism). Recorded here as *evidence debt on existing licenses*; the counting fix is the hindrance entry. **Note (Phase 6):** the exposure is **already pinned in-repo** — `tests/test_volume_honesty.py` (ADR-0264 R9, in the local smoke gate) pins the 21-of-25 shortfall *"in BOTH directions"* and calls its inventory *"an EXPOSURE INVENTORY, not an approved baseline."* So the open work is **applying** the demotions, not discovering them, and that pin moves in the same PR. **Authority:** ADR amendment + re-count, authorized by R-13.
|
||||
- **G-22 · Three tests are red on `main`, and were red before this arc** *(new, 2026-07-28)* — found by the arc's first full-tree run (`3 failed, 13403 passed`), and reproduced identically at `797ebad5`, so **not arc-caused**. `tests/test_ratification_ceremony.py::{test_row_is_byte_compatible_with_the_committed_corpus, test_unreviewed_status_is_refused}` and `tests/test_claims_md_is_current.py::test_claims_md_matches_generator_output`. **Diagnosis:** the code is correct and the pins are stale. `ChainRecord.__slots__` gained `polarity` (ADR-0264 R1), and the serializer **deliberately omits it for affirmative rows** (`teaching/ratification.py:119`) — so a test that builds `ChainRecord(**{k: row[k] for k in __slots__})` `KeyError`s on a correctly-serialized row. Same class as H-8e: ratified design right, pin not updated. **Consequence:** the ratification-corpus byte-compatibility guarantee and the unreviewed-status refusal have been **unverified since ADR-0264 R1 landed**. `test_ratification_ceremony.py` is curated (`teaching`) but ungated; `test_claims_md_is_current.py` is an orphan. **Authority:** the repair is test-side and mechanical, but these guard governance-path behavior and a published artifact (`CLAIMS.md`), so the disposition is Shay's.
|
||||
- **G-22 · Three tests are red on `main`, and were red before this arc** *(new, 2026-07-28)* — found by the arc's first full-tree run (`3 failed, 13403 passed`), and reproduced identically at `797ebad5`, so **not arc-caused**. `tests/test_ratification_ceremony.py::{test_row_is_byte_compatible_with_the_committed_corpus, test_unreviewed_status_is_refused}` and `tests/test_claims_md_is_current.py::test_claims_md_matches_generator_output`. **Diagnosis:** the code is correct and the pins are stale. `ChainRecord.__slots__` gained `polarity` (ADR-0264 R1), and the serializer **deliberately omits it for affirmative rows** (`teaching/ratification.py:119`) — so a test that builds `ChainRecord(**{k: row[k] for k in __slots__})` `KeyError`s on a correctly-serialized row. Same class as H-8e: ratified design right, pin not updated. **Consequence:** the ratification-corpus byte-compatibility guarantee and the unreviewed-status refusal have been **unverified since ADR-0264 R1 landed**. `test_ratification_ceremony.py` is curated (`teaching`) but ungated; `test_claims_md_is_current.py` is an orphan. **RULED 2026-07-28 (Shay): fix, don't document.** The argument that settled it: *a red test on a serving-path contract, left with an explanatory note, IS the H-8 failure mode* — a document acknowledging a gap instead of closing it, which becomes the next assessor's finding. **FIXED**, and the `CLAIMS.md` half turned out to be the more serious of the two:
|
||||
|
||||
- **The two ceremony pins were stale against correct code** (H-8e's class). `polarity` defaults to `AFFIRMATIVE` and is *deliberately omitted* from affirmative rows; the pins iterated `__slots__` and indexed the row, so they broke on every correctly-serialized row. Now built from the row's own keys, letting absent fields take their defaults — which restores the guarantee rather than weakening it. Ratification-corpus byte compatibility and unreviewed-status refusal had been **unverified since ADR-0264 R1 landed**.
|
||||
- **`CLAIMS.md` was publishing a superseded evidence digest for a licensed capability.** Commit `f9e9cc0c` (2026-07-26, *"the deduction lane hashes the prose it serves"*) strengthened the lane, rewrote `evals/deduction_serve/report.json`, and updated the authoritative pin in `scripts/verify_lane_shas.py:60` to `c855d55c…` — recording the old `0b461a5a…` as superseded on line 56. It did **not** regenerate `CLAIMS.md`. So for two days the published capability claim for `deduction_serve_v1` pointed at evidence that no longer existed. The claim *text* never changed; only the pointer was wrong — but a claim whose evidence digest does not match its evidence is exactly the failure the digest exists to prevent.
|
||||
- **The drift had propagated to a third artifact — this directory.** `10-layer-cards/M3-comprehension-reasoning.md` cited the same stale `0b461a5a…` as its would-fail-if-absent evidence. Corrected.
|
||||
|
||||
**The mechanism, and why it is G-7's story again:** one commit updated the lane, its verifier and its test, and missed one downstream artifact. The pin that guards that artifact — `tests/test_claims_md_is_current.py` — **exists and is an orphan in no curated suite**, so nothing surfaced it until this arc ran the full tree once. Registered on the gate as part of the fix.
|
||||
- **G-21 · The math reader decides 1.0% of `holdout_dev/v1`** *(new, 2026-07-28)* — measured while building the §5 corpus: `parse_and_solve` returns a selected graph for **5 of 500** held-out cases, all carrying the same relational skeleton (`compare_multiplicative`); on the public lane, 24/150. This is a sharper measurement of the comprehension frontier than G-3's construction count, on the corpus the project already treats as its held-out standard, and it is why ADR-0252 §5.1's four-structure corpus is not extractable today. Distinct from G-3 (which counts *constructions* the general reader admits); this counts *cases decided* on a standing eval corpus. **Authority:** the widening program (G-3) + a ruling on whether holdout decision-rate becomes a tracked lane metric.
|
||||
- **G-20 · The `refusal_reason` materialisation** — typed refusal evidence exists and is discarded at the public `str` boundary; the plumbing for materialisation already landed. Cross-listed as H-3. **Authority:** small ADR (anticipated by the ADR-0024 chain).
|
||||
|
||||
|
|
|
|||
|
|
@ -271,7 +271,6 @@ tests/test_cga_f64_exactness.py
|
|||
tests/test_cga_incidence.py
|
||||
tests/test_cga_rust_surface_parity.py
|
||||
tests/test_chiral_orientation_gate.py
|
||||
tests/test_claims_md_is_current.py
|
||||
tests/test_claude_hybrid_verification_demo.py
|
||||
tests/test_claude_tool_authority_demo.py
|
||||
tests/test_cli.py
|
||||
|
|
|
|||
|
|
@ -90,7 +90,16 @@ def test_row_is_byte_compatible_with_the_committed_corpus() -> None:
|
|||
changed and make the real change unreviewable."""
|
||||
committed = corpus_path_for("physics")[1].read_text(encoding="utf-8").splitlines()[0]
|
||||
row = existing_rows("physics")[0]
|
||||
rebuilt = ChainRecord(**{k: row[k] for k in ChainRecord.__slots__}).as_jsonl_line()
|
||||
# Build from the keys the committed row ACTUALLY carries, letting absent
|
||||
# fields take their dataclass defaults. ``polarity`` (ADR-0264 R1) is
|
||||
# defaulted to AFFIRMATIVE and deliberately OMITTED from affirmative rows
|
||||
# (``ChainRecord.as_row``), so iterating ``__slots__`` and indexing the row
|
||||
# raises KeyError on every correctly-serialized affirmative row. That is
|
||||
# what made this pin red — the schema moved and the pin did not, leaving
|
||||
# byte-compatibility unverified since R1 landed (G-22).
|
||||
rebuilt = ChainRecord(
|
||||
**{k: row[k] for k in ChainRecord.__slots__ if k in row}
|
||||
).as_jsonl_line()
|
||||
assert rebuilt.rstrip("\n") == committed
|
||||
|
||||
|
||||
|
|
@ -144,7 +153,10 @@ def test_duplicate_edge_is_refused() -> None:
|
|||
|
||||
def test_unreviewed_status_is_refused() -> None:
|
||||
row = existing_rows("physics")[0]
|
||||
record = ChainRecord(**{**{k: row[k] for k in ChainRecord.__slots__},
|
||||
# See the note in test_row_is_byte_compatible_with_the_committed_corpus:
|
||||
# affirmative rows omit ``polarity`` by design, so only the row's own keys
|
||||
# may be indexed (G-22).
|
||||
record = ChainRecord(**{**{k: row[k] for k in ChainRecord.__slots__ if k in row},
|
||||
"chain_id": "physics-causal-900",
|
||||
"subject": "entropy", "object": "temperature",
|
||||
"review_status": "pending"})
|
||||
|
|
|
|||
Loading…
Reference in a new issue