core/evals/setup_oracle
Shay 4056069152 feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a)
Strengthen the ruler before it judges real GSM8K frames — NOT a capability change, no
new parser behavior, no serving path. The setup-oracle now grades UNITS, read from the
binding-graph itself, against an independent expected-unit gold.

- New per-symbol unit axis (symbol_unit_signature) — read from the GRAPH's symbols
  (reader_symbol_units), not the answer projection; covers fact + relation-operand units
  (operands are symbols). A None unit canonicalizes to "unset" so an unmodelled symbol
  never silently matches a declared gold unit.
- The question-target signature gains the target's expected_unit (reader from
  BoundUnknown.expected_unit; gold from the fixture).
- expected_units.json — independent, hand-authored MODELING unit per entity for the 15
  cases (a discrete sortal count -> the generic count unit "item"; money -> "dollars").
  Authored from the PROBLEM, not copied from the reader.
- The reader's per-symbol units match the independent gold on all 15 (setup_wrong=0) —
  agreement, not circularity (both apply the same correct count->item / money modelling).
- Meaningful-fail: a reading whose STRUCTURE matches but whose UNIT diverges now FAILS
  (test_unit_mismatch_is_caught, test_target_unit_mismatch_is_caught). The ruler is not
  decoration on the unit axis either.

Gate held: setup-oracle 15/15 setup_wrong=0 (now structure + units + target). The
relational_metric answer lane is untouched (oracle-only change). No generate/ change.

This is the unit-aware ruler PR-5b (independent R1 gold) and PR-5c (the first R1
comparative/derived-symbol frame) must pass before serving.
2026-06-06 17:09:45 -07:00
..
__init__.py feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a) 2026-06-06 17:09:45 -07:00
__main__.py feat(comprehension): question target in the graph (PR-1) + setup-oracle lane (grade the reading) 2026-06-06 16:40:15 -07:00
expected_units.json feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a) 2026-06-06 17:09:45 -07:00
runner.py feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a) 2026-06-06 17:09:45 -07:00
signature.py feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a) 2026-06-06 17:09:45 -07:00