r1-07 now reads setup-correct and answers 6 — 'Nia has 9 more beads than Omar. Nia has 15. -> Omar = 6'. The reader binds the unknown base's unit FROM the relation when its subject is a known fact and its referent is the otherwise-ungrounded query target, so the equation is admissible; the answer oracle reverse-solves it (PR-7a). Bounded: single base == query target (no chains), known subject value, base not otherwise grounded, <=1 inverse (multiple_inverse_bases else), never over times/divide.
R1 setup 6/0/4 -> 7/0/3; R1 answers -> 7 correct / 0 wrong; 15-case 15/0/0; setup_wrong stays 0. Off-serving. Refreshes the R1 ledger to 7/0/3 (R1 closed; the 3 remaining refusals are wrong=0 boundaries).
Pin the EXACT reverse-solve semantics in the independent answer oracle before the
reader learns the inverse frame (PR-7b). A single more_than/fewer_than whose entity
is grounded and whose ref is the (otherwise ungrounded) query target inverts to the
base: base = entity -/+ delta (always integer-exact for +/-).
Refuses everything wider (each a meaningful-fail oracle test): more than one inverse
constraint, an inverse base that is not the query target (a chain), an already-grounded
base, a negative count result, or an inverse over times/divide. The duplicate-definition
guard moved into the per-kind branches so a grounded entity on a more/fewer_than is read
as an inverse constraint, not a redefinition; forward more/fewer/times/divide/sum paths
are unchanged (regression-guarded).
Contract only — the reader is NOT touched: r1-07 still refuses (admissibility_refused),
so R1 stays 6 / 0 / 4 and 15-case 15 / 0 / 0. r1-07 gold drops the non-consumed ask_base
(aligning with the reader's future projection) and gains gold=6, dormant until PR-7b.
Off-serving (only evals/tests import the oracle). SHA gate 8/9 (public_demo env flake).
Widen the aggregate-query recognizer so a multi-part total may be asked with a
trailing qualifier after 'have': 'How many X do A and B have altogether?' and
'... in total?'. The qualifier is stripped and honored ONLY for the multi-part
(sumquery) form — a single-entity query carrying it refuses, guarded by
'not aggregate'.
Phrasing-only: no new arithmetic, no new relation kind, no inverse solving, no
distractor/pronoun handling. The parts still flow through the existing sum_of;
an ungrounded part (unit unbound) or a unit-incompatible part (unit mismatch) is
refused downstream by the REAL admissibility check, so the recognizer cannot
over-read. Off-serving organ only (no generate.derivation / reliability_gate).
Flips r1-03 (more+altogether -> 25) and r1-04 (fewer+in total -> 34):
R1 setup 6 / 0 / 4 R1 answers 6 / 0 / 4 (setup_wrong 0, gold_error 0)
15-case 15 / 0 / 0 29 quant tests, 102 affected-file tests green
Tests are meaningful-fail: the single-entity-qualifier, ungrounded-part, and
unit-incompatible-part refusals each fail loudly if their guard is removed.
PR-6d adds the partition frame: combine all parts into a total, then split that
total equally into N containers. r1-06-subtotal-reused moves refused → correct —
the FIRST case where the divisor applies to a DERIVED symbol (the total), not a
directly given fact. That is real progress toward GSM8K setup comprehension,
where intermediate quantities are the norm.
Scope (kept narrow on purpose):
No new relation kind.
No new arithmetic operation.
No rational support.
No rounding/flooring.
No serving path touched.
The frame reuses the already-ratified pieces — SumOf(parts) + Div(Symbol(total),
Literal(N)) → divide_by — so this PR is reader-only (no IR / admissibility /
oracle / signature change).
Frame grammar:
"They combine their <unit> and split them equally into N <containers>."
+ "How many <unit> are in each <container>?"
-> total = sum(all facts); per_<container> = total / N; ask per_<container>.
wrong=0 boundaries:
- Exact-divisibility still gates the ANSWER, now over a derived total: 5+6=11,
11/3 is non-exact -> the setup reads correctly but the answer REFUSES (never
floors). Setup comprehension and answer exactness are cleanly separated.
- Partition/query coherence: a partition is read ONLY together with its
"in each <container>" query (and vice versa); container mismatch (box vs jar)
refuses. Prevents over-reading a story detail into an unused derived value.
Meaningful-fail verified: disabling the guard makes a dangling partition
wrongly comprehend.
Gates:
R1 setup: 4 correct / 0 wrong / 6 refused
R1 answers: 4 correct / 0 wrong / 6 refused / setup_wrong 0 / gold_error 0
15-case setup: 15 / 0 / 0
97 PR-6d tests + 99 relational/invariant tests green. Reader is off-serving
(no generate.derivation / core.reliability_gate import).
PR-6c adds the divisive comparative frame: "half as many" read as EXACT INTEGER
DIVISION. It is the divisor twin of PR-5c's multiplicative frame, and moves the
independent R1 gold's r1-02-half from refused → correct.
No serving path touched. No rational/fractional answer support added. Non-exact
division refuses.
Design (ADR-0134 amended — divide made symmetric with multiply):
- `_check_divide` now admits a SINGLE-DEP divide-by-dimensionless-literal
(item / dimensionless = item), the exact twin of single-dep multiply. The
2-dep rate-divide path is untouched. This keeps the IR's "literal operands
are not deps" invariant (proven in PR-6a) uniform across Mul AND Div, so the
reader builds both without a per-op special case and WITHOUT synthesizing a
divisor symbol that would pollute the setup-oracle's unit signature.
- `Div(Symbol, Literal)` IR node: "ref / divisor", operation_kind "divide",
projects to `divide_by`. Divisor-only contract mirrors the scalar-only one.
- Reader: `_DIVISOR_WORDS={half:2}` slots into the same 8-token "<WORD> as many"
template as the factor words; graph carries only the two entities.
- Gold reconciliation: r1-02 placeholder `times_as_many factor 0.5` → exact
`divide_by divisor 2` (gold 4). Makes the INDEPENDENT gold integer-faithful.
The wrong=0 boundary — exact divisibility:
the oracle admits `divide_by` only when `base % divisor == 0`. An odd base
halved REFUSES (gold_error), never floors to a wrong integer. Divisor must be
a nonzero int (0, 0.5, 1.5, bool all refuse); divisor=1 is intentionally the
identity (pinned). admissibility proves DIMENSION; the oracle proves EXACT VALUE.
Meaningful-fail (CLAUDE.md Schema-Defined Proof Obligations), both verified red:
- drop the `% divisor` guard → test_oracle_refuses_non_exact_division fails (returns 3).
- disable the single-dep divide branch → the admissibility test AND the reader's
`half` test fail (admissibility refuses → reader refuses → half stays refused).
Gates:
R1 setup: 3 correct / 0 wrong / 7 refused
R1 answers: 3 correct / 0 wrong / 7 refused / setup_wrong 0 / gold_error 0
15-case setup: 15 / 0 / 0
91 PR-6c tests + 60 relational lanes + 56 architectural invariants + 502
binding-graph/proof-chain/adapter tests green. All 8 SHA-content lanes match
(serving unmoved; admissibility has no generate.derivation/reliability_gate consumer).
The first capability slice on the R1 arc, gated by the setup-oracle: turn the
"twice / N times as many" reading from REFUSED into a correct setup, without a single
misread. Builds on the typed IR (PR-4) and the R1 gold (PR-5b).
- IR: a Mul(symbol, literal-factor) node — to_canonical_string "ref * factor",
operation_kind "multiply", dependencies {ref}, to_relation -> times_as_many. The
product keeps the symbol's unit (count * scalar = count), admitted by the REAL
check_admissibility multiply path (the literal factor is dimensionless, not a dep).
- Reader: a multiplicative template "Y has <factor> as many <unit> as X" (factor word:
twice/double/triple/quadruple) and "Y has <N> times as many <unit> as X", checked
BEFORE the digit gate (the factor may be a word). 'half' (a /2) is deliberately
deferred — divide-by-literal is a separate admissibility path.
- setup-oracle: relation_signature now canonicalizes times_as_many.
Setup-oracle R1 result: 2 setup_correct (r1-01 twice; r1-05 the multi-step chain
ivy/jon=3*ivy/kim=jon+2), 0 setup_WRONG, 8 setup_refused. Every hard negative stays a
safe refusal: missing-base (Rosa ungrounded), ambiguous referent, distractor, inverse,
partition, 'altogether'/'in total' phrasings, and 'half' (divide). wrong=0 held through
the first capability addition.
Gates green: setup-oracle R1 setup_wrong=0; 15-case setup gate 15/15 setup_wrong=0;
relational_metric answer lane 15/15 wrong=0; binding-graph admissibility + realize +
architectural invariants + chat-runtime + pipeline (122+). No serving path touched (this
reader feeds the relational_metric / setup-oracle lanes, not the candidate-graph serving).
Pure-gold. Defines what "correct" means for the R1 (comparative / derived-symbol /
multi-step) shapes the reader cannot read yet, and verifies the current reader REFUSES
them rather than misreading them. NO frame extraction, no parser change, no serving.
- evals/setup_oracle/r1_gold.jsonl — 10 INDEPENDENT, self-contained fixtures (id, text,
relations, expected_units, query, notes) covering: twice/half-as-many (multiplicative),
more/fewer-than -> total, a multi-step derived chain (jon=3*ivy; kim=jon+2), a derived
subtotal reused downstream (total=lee+mae; per_box=total/3), an inverse target, and the
hard negatives (ambiguous referent, missing base quantity, distractor quantity). The
gold is the authority — each fixture is reviewable without joining files.
- run_r1() scores the CURRENT reader against this gold.
Investigative result (the deliverable): **10/10 setup_refused, setup_wrong=0, setup_correct=0.**
The reader refuses every R1 shape with a TYPED reason (non_digit_quantity /
no_quantity_template / unreadable_quantity_clause / admissibility_refused) — it NEVER
misreads an R1 case as a simpler form. So the wrong=0 posture holds all the way down, and
PR-5c (the R1 frame) starts from a clean foundation: every R1 case is currently a safe
refusal, and the frame's job is refusal -> correct, with the oracle ensuring no
setup_wrong is ever introduced.
`python -m evals.setup_oracle r1` prints the auditable R1 report (exit 0 iff
setup_wrong==0). The 15-case setup gate + relational_metric answer lane are untouched.
Strengthen the ruler before it judges real GSM8K frames — NOT a capability change, no
new parser behavior, no serving path. The setup-oracle now grades UNITS, read from the
binding-graph itself, against an independent expected-unit gold.
- New per-symbol unit axis (symbol_unit_signature) — read from the GRAPH's symbols
(reader_symbol_units), not the answer projection; covers fact + relation-operand units
(operands are symbols). A None unit canonicalizes to "unset" so an unmodelled symbol
never silently matches a declared gold unit.
- The question-target signature gains the target's expected_unit (reader from
BoundUnknown.expected_unit; gold from the fixture).
- expected_units.json — independent, hand-authored MODELING unit per entity for the 15
cases (a discrete sortal count -> the generic count unit "item"; money -> "dollars").
Authored from the PROBLEM, not copied from the reader.
- The reader's per-symbol units match the independent gold on all 15 (setup_wrong=0) —
agreement, not circularity (both apply the same correct count->item / money modelling).
- Meaningful-fail: a reading whose STRUCTURE matches but whose UNIT diverges now FAILS
(test_unit_mismatch_is_caught, test_target_unit_mismatch_is_caught). The ruler is not
decoration on the unit axis either.
Gate held: setup-oracle 15/15 setup_wrong=0 (now structure + units + target). The
relational_metric answer lane is untouched (oracle-only change). No generate/ change.
This is the unit-aware ruler PR-5b (independent R1 gold) and PR-5c (the first R1
comparative/derived-symbol frame) must pass before serving.
Two coupled, additive, off-serving changes toward the typed math-comprehension organ.
No serving path touched; the relational_metric answer lane stays 15/15 wrong=0.
PR-1 — QuantQuery → BoundUnknown. comprehend_quantitative now emits the question
target as a BoundUnknown INSIDE the binding-graph (symbol_id, state_index="terminal",
question_form "count"|"total", expected_unit), so the graph is a real question-bearing
mathematical object and its canonical serialization carries the target. The external
QuantQuery is RETAINED, consistent-by-construction, so the two consumers
(to_relational_metric, realize/quantitative) are byte-identical; a follow-up rewires
them onto graph.unknowns and drops the duplicate field.
Setup-oracle lane (evals/setup_oracle) — grade the READING, not the answer. The
relational_metric lane scores answers, which can bless a semantically-wrong derivation
that coincidentally lands on the right number (the exact hazard the held-out
measurements + the 2/87 resolve_pooled probe exposed). The setup-oracle compares the
reader's comprehended STRUCTURE — a span-free signature of facts + typed equations +
the BoundUnknown target — against the INDEPENDENT gold structure (the relational_metric
cases' own relations/query, authored separately from the binding-graph reader). A
structural mismatch is setup_wrong, the wrong=0-critical count, even when the answer
would be right. v1 grades structure (units deferred — covered by admissibility). The
reader reads all 15 cases with the gold structure (setup_wrong=0); a meaningful-fail
test proves the oracle catches a right-answer/wrong-structure reading (it is not
decoration). `python -m evals.setup_oracle` exits nonzero iff setup_wrong > 0.
This is the measurement rig BEFORE investing in frame families: setup_wrong=0 is the
gate; serving must not move while setup_wrong > 0. It is the first milestone of the
math-comprehension organ, not a path to "solve GSM8K".
Verified: setup-oracle 15/15 setup_correct wrong=0; quantitative + setup-oracle unit
tests (17); realize-binding-graph + binding-graph + architectural invariants (183).
