fix(evals): the deduction lane hashes the prose it serves #133

Merged

core-labs merged 1 commit from feat/lane-pins-see-surfaces into main

2026-07-27 02:46:27 +00:00

core-labs commented

2026-07-27 02:46:18 +00:00

Owner

Closes the gap #132 uncovered. The pinned artifact carried verdict counts only, so CORE's user-visible output was unguarded by its own hash pins.

The runner's docstring justified the exclusion: prose is "presentation, not decision", so the pinned bytes "stay stable against wording-only changes; wording is covered by tests/test_deduction_surface.py."

Measured, that justification was false. With render._display_noun sabotaged so every clause reads all SABOTAGE_dogs are SABOTAGE_animals:

guard	result
11 lane SHA pins	11/11 byte-identical — blind
`test_deduction_serve_lane` + `_license` (20)	20 passed — blind
`tests/test_deduction_surface.py` (41)	41 passed — blind, and this is the file the docstring named as the wording guard
`evals/grammar_roundtrip`	RED — the only witness

That is how the ratified v1b band served all dog are mammal for the entire arc with wrong=0 intact.

The fix

build_report now emits surface_sha256 + per-case surfaces from the real deduction_grounded_surface — the same call chat serving makes, so what gets hashed is what a user reads.

One trap worth naming: build_combined_report re-projects five named fields per split, so a field added to build_report alone never reaches the pinned bytes. My first attempt did exactly that and the pin stayed green. The payload is not the report — both had to change.

Surfaces are recorded, not just digested, so a moved pin shows the exact sentence that changed in review rather than an opaque hash someone has to go re-derive. 2,766 → 37,280 bytes.

`deduction_serve_v1` under sabotage
before	✓ byte-identical (blind)
after	✗ `52370b73` vs `c855d55c`

Re-pinned surgically — one line, old hash recorded in a comment beside it, never --update. Verdicts are untouched (166/166 correct, wrong=0); the hash moved because the payload grew.

test_surface_hash_moves_when_the_renderer_is_sabotaged makes it permanent: it corrupts the renderer, requires a digest to move, and asserts the aggregate counts are unchanged — proving the digest tracks prose rather than smuggling in a decision change. A pin that cannot fail guards nothing.

Accepted cost: wording-only changes now move this pin. That is the intent — a wording change is a user-visible change and should require a deliberate re-pin, which is precisely the discipline whose absence let all dog are mammal ship.

The other 10 lanes are untouched. deduction_serve was fixed because it is the one demonstrably serving prose to users.

[Verification]: in-worktree on CPython 3.12.13 with uv sync --locked — uv run core test --suite smoke -q 621 passed (unchanged); uv run core test --suite deductive -q 405 passed (403 + 2 new); scripts/verify_lane_shas.py 11/11 with the new pin, and 10/11 (RED on deduction_serve_v1) under the sabotage it previously could not see.

Closes the gap #132 uncovered. The pinned artifact carried **verdict counts only**, so CORE's user-visible output was unguarded by its own hash pins. The runner's docstring justified the exclusion: prose is "presentation, not decision", so the pinned bytes "stay stable against wording-only changes; wording is covered by `tests/test_deduction_surface.py`." **Measured, that justification was false.** With `render._display_noun` sabotaged so every clause reads `all SABOTAGE_dogs are SABOTAGE_animals`: | guard | result | |---|---| | 11 lane SHA pins | 11/11 byte-identical — **blind** | | `test_deduction_serve_lane` + `_license` (20) | 20 passed — **blind** | | `tests/test_deduction_surface.py` (41) | 41 passed — **blind**, and this is the file the docstring named as the wording guard | | `evals/grammar_roundtrip` | **RED** — the only witness | That is how the ratified v1b band served `all dog are mammal` for the entire arc with `wrong=0` intact. ## The fix `build_report` now emits `surface_sha256` + per-case `surfaces` from the real `deduction_grounded_surface` — the same call chat serving makes, so what gets hashed is what a user reads. One trap worth naming: `build_combined_report` **re-projects five named fields per split**, so a field added to `build_report` alone never reaches the pinned bytes. My first attempt did exactly that and the pin stayed green. *The payload is not the report* — both had to change. Surfaces are **recorded, not just digested**, so a moved pin shows the exact sentence that changed in review rather than an opaque hash someone has to go re-derive. 2,766 → 37,280 bytes. | `deduction_serve_v1` under sabotage | | |---|---| | before | ✓ byte-identical (blind) | | after | **✗ `52370b73` vs `c855d55c`** | Re-pinned **surgically** — one line, old hash recorded in a comment beside it, never `--update`. Verdicts are untouched (**166/166 correct, `wrong=0`**); the hash moved because the payload grew. `test_surface_hash_moves_when_the_renderer_is_sabotaged` makes it permanent: it corrupts the renderer, requires a digest to move, **and** asserts the aggregate counts are unchanged — proving the digest tracks *prose* rather than smuggling in a decision change. A pin that cannot fail guards nothing. **Accepted cost:** wording-only changes now move this pin. That is the intent — a wording change *is* a user-visible change and should require a deliberate re-pin, which is precisely the discipline whose absence let `all dog are mammal` ship. The other 10 lanes are untouched. `deduction_serve` was fixed because it is the one demonstrably serving prose to users. [Verification]: in-worktree on CPython 3.12.13 with `uv sync --locked` — `uv run core test --suite smoke -q` **621 passed** (unchanged); `uv run core test --suite deductive -q` **405 passed** (403 + 2 new); `scripts/verify_lane_shas.py` **11/11** with the new pin, and **10/11 (RED on `deduction_serve_v1`)** under the sabotage it previously could not see.

core-labs added 1 commit 2026-07-27 02:46:21 +00:00

fix(evals): the deduction lane hashes the prose it serves f9e9cc0c6a

The pinned artifact carried verdict counts only. The runner's docstring
justified that: prose is "presentation, not decision", so the pinned bytes
"stay stable against wording-only changes; wording is covered by
tests/test_deduction_surface.py."

Measured, that justification was false. With render._display_noun sabotaged
so every clause reads "all SABOTAGE_dogs are SABOTAGE_animals":

  11 lane SHA pins                     -> 11/11 byte-identical, blind
  test_deduction_serve_lane + _license -> 20 passed, blind
  tests/test_deduction_surface.py      -> 41 passed, blind
                                          (the named wording guard)
  evals/grammar_roundtrip              -> RED, the only witness

So CORE's user-visible output was unguarded by its own hash pins, which is
how the ratified v1b band served "all dog are mammal" for the entire arc
with wrong=0 intact.

build_report now emits surface_sha256 + per-case surfaces from the real
deduction_grounded_surface — the same call chat serving makes, so what is
hashed is what a user reads. Note build_combined_report re-projects five
named fields per split, so a field added to build_report alone never reaches
the pinned bytes; both had to change. The payload is not the report.

Surfaces are recorded, not just digested, so a moved pin shows the exact
sentence that changed in review instead of an opaque hash to go re-derive.
2,766 -> 37,280 bytes.

  deduction_serve_v1 under sabotage: BEFORE byte-identical (blind)
                                     AFTER  52370b73 vs c855d55c (RED)

Re-pinned surgically, one line, old hash recorded beside it, never --update.
Verdicts untouched: 166/166 correct, wrong=0. The hash moved because the
payload grew.

test_surface_hash_moves_when_the_renderer_is_sabotaged makes it permanent:
it corrupts the renderer, requires a digest to move, AND asserts the
aggregate counts are unchanged — proving the digest tracks PROSE rather than
smuggling in a decision change. A pin that cannot fail guards nothing.

Accepted cost: wording-only changes now move this pin. That is the intent —
a wording change IS a user-visible change and should require a deliberate
re-pin. The other 10 lanes are untouched; deduction_serve was fixed because
it is the one demonstrably serving prose to users.

[Verification]: in-worktree on CPython 3.12.13, uv sync --locked —
smoke 621 unchanged; deductive 405 (403 + 2 new);
scripts/verify_lane_shas.py 11/11 with the new pin, and 10/11 (RED on
deduction_serve_v1) under the sabotage it previously could not see.