Commit graph

1 commit

Author SHA1 Message Date
Shay
f63e790ab0 test(comprehend): structure-preservation + perturbation invariance (close coincidental-correctness)
Addresses the central review finding: the generative wrong=0 tests compared oracle
VERDICTS, so a misread graph that coincidentally yields the same verdict passed
silently (coincidental correctness in cleaner clothes). This adds the conjugate
check — the reader must recover the EXACT structure the prose encodes, not merely a
verdict-equivalent one.

tests/test_comprehension_structure_preserving.py:
  - Structure preservation (all 4 domains): over randomly generated structures
    rendered to prose that FULLY determines them, assert projected structure AND
    query == ground truth exactly (order-insensitive canonicalization), or refuse.
    Empirically this is strictly stronger: under a subject<->predicate premise swap,
    361/400 reads are structurally wrong and 307 of those (85%) coincide in verdict
    — the answer test misses all 307; the structure test catches all 361.
  - Perturbation invariance: meaning-preserving surface changes (premise/clause
    reordering, capitalization, extra whitespace) yield the SAME structure.

The existing answer-preservation property tests stay (verdict agreement is still a
valid, separate check — exactly the "assert structure, then separately assert
oracle agreement" the review recommends). Tests-only; no source change; capability
index unchanged. 7 new + 86 comprehension/capability targeted green.
2026-06-05 23:59:20 -07:00