core/evals/setup_oracle
Shay f9ef9e56a4 feat(comprehension): aggregate-then-divide partition frame — "split equally into N boxes" (PR-6d)
PR-6d adds the partition frame: combine all parts into a total, then split that
total equally into N containers. r1-06-subtotal-reused moves refused → correct —
the FIRST case where the divisor applies to a DERIVED symbol (the total), not a
directly given fact. That is real progress toward GSM8K setup comprehension,
where intermediate quantities are the norm.

Scope (kept narrow on purpose):
  No new relation kind.
  No new arithmetic operation.
  No rational support.
  No rounding/flooring.
  No serving path touched.

The frame reuses the already-ratified pieces — SumOf(parts) + Div(Symbol(total),
Literal(N)) → divide_by — so this PR is reader-only (no IR / admissibility /
oracle / signature change).

Frame grammar:
  "They combine their <unit> and split them equally into N <containers>."
  + "How many <unit> are in each <container>?"
  -> total = sum(all facts); per_<container> = total / N; ask per_<container>.

wrong=0 boundaries:
- Exact-divisibility still gates the ANSWER, now over a derived total: 5+6=11,
  11/3 is non-exact -> the setup reads correctly but the answer REFUSES (never
  floors). Setup comprehension and answer exactness are cleanly separated.
- Partition/query coherence: a partition is read ONLY together with its
  "in each <container>" query (and vice versa); container mismatch (box vs jar)
  refuses. Prevents over-reading a story detail into an unused derived value.
  Meaningful-fail verified: disabling the guard makes a dangling partition
  wrongly comprehend.

Gates:
  R1 setup:   4 correct / 0 wrong / 6 refused
  R1 answers: 4 correct / 0 wrong / 6 refused / setup_wrong 0 / gold_error 0
  15-case setup: 15 / 0 / 0
  97 PR-6d tests + 99 relational/invariant tests green. Reader is off-serving
  (no generate.derivation / core.reliability_gate import).
2026-06-06 21:07:19 -07:00
..
__init__.py test(comprehension): expose R1 answer lane 2026-06-06 18:28:49 -07:00
__main__.py test(comprehension): expose R1 answer lane CLI 2026-06-06 18:31:41 -07:00
expected_units.json feat(setup-oracle): make the ruler UNIT-AWARE (setup-oracle v2, PR-5a) 2026-06-06 17:09:45 -07:00
r1_gold.jsonl feat(comprehension): aggregate-then-divide partition frame — "split equally into N boxes" (PR-6d) 2026-06-06 21:07:19 -07:00
runner.py feat(comprehension): add R1 answer-oracle lane 2026-06-06 18:26:30 -07:00
signature.py feat(comprehension): the divisive comparative frame — "half as many" as exact integer division (PR-6c) 2026-06-06 20:18:39 -07:00