Strengthen the ruler before it judges real GSM8K frames — NOT a capability change, no new parser behavior, no serving path. The setup-oracle now grades UNITS, read from the binding-graph itself, against an independent expected-unit gold. - New per-symbol unit axis (symbol_unit_signature) — read from the GRAPH's symbols (reader_symbol_units), not the answer projection; covers fact + relation-operand units (operands are symbols). A None unit canonicalizes to "unset" so an unmodelled symbol never silently matches a declared gold unit. - The question-target signature gains the target's expected_unit (reader from BoundUnknown.expected_unit; gold from the fixture). - expected_units.json — independent, hand-authored MODELING unit per entity for the 15 cases (a discrete sortal count -> the generic count unit "item"; money -> "dollars"). Authored from the PROBLEM, not copied from the reader. - The reader's per-symbol units match the independent gold on all 15 (setup_wrong=0) — agreement, not circularity (both apply the same correct count->item / money modelling). - Meaningful-fail: a reading whose STRUCTURE matches but whose UNIT diverges now FAILS (test_unit_mismatch_is_caught, test_target_unit_mismatch_is_caught). The ruler is not decoration on the unit axis either. Gate held: setup-oracle 15/15 setup_wrong=0 (now structure + units + target). The relational_metric answer lane is untouched (oracle-only change). No generate/ change. This is the unit-aware ruler PR-5b (independent R1 gold) and PR-5c (the first R1 comparative/derived-symbol frame) must pass before serving. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| __main__.py | ||
| expected_units.json | ||
| runner.py | ||
| signature.py | ||