diff --git a/docs/analysis/r1-inventory-ledger-2026-06-07.md b/docs/analysis/r1-inventory-ledger-2026-06-07.md index 11031933..3765bb31 100644 --- a/docs/analysis/r1-inventory-ledger-2026-06-07.md +++ b/docs/analysis/r1-inventory-ledger-2026-06-07.md @@ -1,21 +1,26 @@ # R1 comprehension inventory ledger -**As of:** `main @ 5ada1392` (post-#616, PR-6d landed) -**Lane state:** R1 setup **4 / 0 / 6** · R1 answers **4 / 0 / 6** (`setup_wrong 0`, `gold_error 0`) · 15-case setup **15 / 0 / 0** +**As of:** PR-7b / C0 (inverse reader frame), on `main @ 0e6a7f9a` (post-#619) + this branch +**Lane state:** R1 setup **7 / 0 / 3** · R1 answers **7 / 0 / 3** (`setup_wrong 0`, `gold_error 0`) · 15-case setup **15 / 0 / 0** -This is a baseline decision artifact, not a capability claim. It records exactly -which `evals/setup_oracle/r1_gold.jsonl` fixtures the typed comprehension organ -now *reads and answers*, which it *refuses*, and — for each refusal — the -semantic family that blocks it and whether that refusal is a coverage gap or a -load-bearing wrong=0 boundary that **should stay refused**. It exists so the next -slice moves only the cases whose semantics are already supported, and never -mistakes a correct refusal for a gap. +This is a decision artifact, not a capability claim. It records exactly which +`evals/setup_oracle/r1_gold.jsonl` fixtures the typed comprehension organ now +*reads and answers*, which it *refuses*, and — for each refusal — the semantic +family that blocks it and whether that refusal is a coverage gap or a load-bearing +wrong=0 boundary that **should stay refused**. It exists so the next slice moves +only the cases whose semantics are already supported, and never mistakes a correct +refusal for a gap. + +> **History.** This ledger opened at **4 / 0 / 6** (`main @ 5ada1392`, post-PR-6d). +> The additive aggregate-query slice (#618) flipped `r1-03`/`r1-04` → **6 / 0 / 4**; +> the inverse reader frame (PR-7b / C0) flips `r1-07` → **7 / 0 / 3**. With that, R1 +> is **closed**: the three remaining refusals are correct wrong=0 boundaries, not gaps. ## Reproduce ```bash -.venv/bin/python -m evals.setup_oracle r1 # setup lane -> 4/0/6 -.venv/bin/python -m evals.setup_oracle r1-answers # answer lane -> 4/0/6, setup_wrong 0, gold_error 0 +.venv/bin/python -m evals.setup_oracle r1 # setup lane -> 7/0/3 +.venv/bin/python -m evals.setup_oracle r1-answers # answer lane -> 7/0/3, setup_wrong 0, gold_error 0 .venv/bin/python -m evals.setup_oracle # 15-case setup gold -> 15/0/0 ``` @@ -32,14 +37,14 @@ mis-valued. | `r1-02-half` | `Dora has half as many … as Carl (8)` | ✅ correct | `4` | divisive (`half`) | — | **admitted** | | `r1-05-chain` | `Jon = 3×Ivy(4); Kim = Jon + 2` | ✅ correct | `14` | multi-step chain (mul → add) | — | **admitted** | | `r1-06-subtotal-reused` | `total = Lee(5)+Mae(7); per_box = total/3` | ✅ correct | `4` | aggregate-then-divide partition | — | **admitted** | -| `r1-03-more-total` | `Evan = Finn(10)+5; ask total **altogether**` | ⛔ refused | ⛔ refused | additive aggregate + aggregate-query phrasing | `unreadable_quantity_query` | **gap (phrasing)** | -| `r1-04-fewer-total` | `Hank = Gail(20)−6; ask total **in total**` | ⛔ refused | ⛔ refused | additive aggregate + aggregate-query phrasing | `unreadable_quantity_query` | **gap (phrasing)** | -| `r1-07-inverse` | `Nia(15) = Omar + 9; ask Omar` | ⛔ refused | ⛔ refused | inverse target (ask the *base* of a relation) | `admissibility_refused` | **gap (real capability)** | +| `r1-03-more-total` | `Evan = Finn(10)+5; ask total **altogether**` | ✅ correct | `25` | additive aggregate + aggregate-query phrasing | — | **admitted** | +| `r1-04-fewer-total` | `Hank = Gail(20)−6; ask total **in total**` | ✅ correct | `34` | additive aggregate + aggregate-query phrasing | — | **admitted** | +| `r1-07-inverse` | `Nia(15) = Omar + 9; ask Omar` | ✅ correct | `6` | inverse target (ask the *base* of a relation) | — | **admitted** | | `r1-08-ambiguous-referent` | `He has 3 more than her; ask she` | ⛔ refused | ⛔ refused | unresolved pronoun referent | `unreadable_quantity_clause` | **correct refusal** | | `r1-09-missing-base` | `Quinn = 2×Rosa; Rosa never given` | ⛔ refused | ⛔ refused | ungrounded base (no grounded fact) | `no_single_quantity_query` | **correct refusal** | | `r1-10-distractor` | `Sam has 7 pencils **and 3 erasers**; ask Tom` | ⛔ refused | ⛔ refused | distractor in a compound clause | `unreadable_quantity_clause` | **correct refusal** | -## The 4 admitted families and the invariant protecting each +## The 7 admitted families and the invariant protecting each Each admitted family is protected by a specific projection/admissibility gate AND the two independent oracles. A family is "admitted" only because a violation of @@ -52,70 +57,32 @@ silently. | divisive | `r1-02` | divisor-only projection: `Div(Symbol, Literal)` is the **only** divisive `to_relation`; single-dep divide admissibility, symmetric with multiply (PR-6c); oracle **exact-divisibility** gate `base % divisor == 0` — an odd base refuses, never rounds. | | multi-step chain | `r1-05` | composition of the above two gates over a derived intermediate; the whole reading refuses if any step's projection/admissibility fails (no partial chain). | | aggregate-then-divide partition | `r1-06` | partition/query coherence guard (`partition ⇔ perquery`, else `partition_query_mismatch`; container match else `partition_container_mismatch`); reuses `SumOf` + `Div` with **no new relation kind**; exact-divisibility gates the answer over a *derived* total (PR-6d). | -| all admitted | — | setup oracle: `reader_rel == gold_rel ∧ reader_units == gold_units ∧ reader_unk == gold_unk` (any drift → `setup_wrong`). Answer oracle: forward-substitution + exact integer arithmetic (any mis-value → `wrong`/`gold_error`). | +| additive aggregate (query-phrasing) | `r1-03`, `r1-04` | the trailing-qualifier recognizer (`altogether` / `in total`) is honored **only** for the multi-part `sumquery` form; an ungrounded or unit-incompatible part is refused downstream at admissibility (`unit_unbound` / `unit_mismatch`). No new arithmetic, no new relation kind (#618). | +| inverse target | `r1-07` | inverse frame (PR-7b): a `more`/`fewer`-than whose **subject is a known fact** and whose **referent is the otherwise-ungrounded query target** binds the base's unit **from the relation** so the equation is admissible; the answer oracle reverse-solves it (PR-7a). Bounded — single base == query target (no chains), known subject value, base not otherwise grounded, ≤1 inverse (`multiple_inverse_bases` else), never over times/divide. | +| all admitted | — | setup oracle: `reader_rel == gold_rel ∧ reader_units == gold_units ∧ reader_unk == gold_unk` (any drift → `setup_wrong`). Answer oracle: forward-substitution + one narrow reverse-solve + exact integer arithmetic (any mis-value → `wrong`/`gold_error`). | -## The 6 refusals, classified +## The 3 refusals, classified — all correct wrong=0 boundaries -The headline: **of 6 refusals, only 2 are pure phrasing gaps; 1 is a real missing -capability; 3 are correct wrong=0 boundaries that must stay refused.** - -### Gaps — phrasing only (2): `r1-03`, `r1-04` - -Structurally **fully supported** — fact + `more_than`/`fewer_than` + `sum_of` are -all admitted families. The *only* blocker is the query template at -`generate/quantitative_comprehension.py:254` requiring `toks[-1] == "have"`: - -- `"… do Evan and Finn have altogether?"` → `toks[-1] == "altogether"` → falls through to `unreadable_quantity_query` (`:262`). -- `"… do Gail and Hank have in total?"` → `toks[-1] == "total"` → same. - -A trailing aggregate adverb (`altogether`) or qualifier (`in total`) defeats the -`have`-terminal check. This is the **next safe slice**: widen the aggregate-query -recognizer to accept the trailing qualifier when all named parts are already -grounded and unit-compatible. No new arithmetic, no new relation kind. Expected -post-slice: **R1 6 / 0 / 4**. - -### Gap — real capability (1): `r1-07` - -`Nia(15) = Omar + 9; ask Omar` is an **inverse** problem: the asked entity `Omar` -appears as a *dependency* of a grounded lhs (`Nia`), not as a derivable lhs with -its own equation. The reader can read the structure honestly, but forward -substitution + forward admissibility cannot invert (`Omar = Nia − 9`); `Omar` has -no grounded unit as a bare ref, so `check_admissibility` raises and the build -returns `admissibility_refused` (`:464`). Closing this needs a genuine -reverse-solve contract, not a phrasing tweak — **defer**. Until then the refusal -is the correct wrong=0 behavior (the gold note pins: setup_correct OR refuse, -**never** setup_wrong). - -### Correct refusals — must stay refused (3): `r1-08`, `r1-09`, `r1-10` - -Admitting any of these would breach wrong=0. They are the refusal boundary -working, not coverage gaps: +The headline: **of 3 remaining refusals, 0 are gaps; all 3 are correct wrong=0 +boundaries that must stay refused.** Admitting any would breach wrong=0; they are +the refusal boundary working, not coverage gaps: - **`r1-08` (pronoun):** `"He has 3 more than her"` has no grounded base for `her`/`she`; binding a guessed referent would fabricate. `unreadable_quantity_clause`. -- **`r1-09` (ungrounded base):** `Quinn = 2×Rosa` with Rosa never given is underdetermined; the `not facts` guard (`:342`) refuses with `no_single_quantity_query`. A "twice as many" with no grounded anchor must refuse. +- **`r1-09` (ungrounded base):** `Quinn = 2×Rosa` with Rosa never given is underdetermined; the `not facts` guard refuses with `no_single_quantity_query`. A "twice as many" with no grounded anchor must refuse. (Note: distinct from `r1-07` — there the *subject* `Nia` is a known fact and only the *base* is unknown, so the inverse frame can solve it; here *nothing* is grounded.) - **`r1-10` (distractor):** `"Sam has 7 pencils and 3 erasers"` is a compound clause the `X has N unit` template intentionally cannot parse; mis-binding the distractor (`3 erasers`) would corrupt the reading. `unreadable_quantity_clause`. Only an intentionally-designed compound-clause parser would move this — and only with a wrong=0 hazard audit first. ## Decision and trajectory -Next capability slice (per pinned scope): **additive aggregate query variants** — -`"… have altogether?"` and `"… have in total?"` — flipping `r1-03` and `r1-04` -only. Constraints: no new arithmetic, no new relation kind, no inverse solving, no -distractor handling, no pronoun resolution; widen the aggregate-query recognizer -**only** when all named parts are already grounded and unit-compatible. +**R1 is closed at 7 / 0 / 3.** Every fixture whose semantics the typed organ can +honestly read is admitted; the three remaining refusals are load-bearing wrong=0 +boundaries (pronoun resolution, ungrounded base, distractor parsing) — each would +need an *intentionally designed* capability with its own hazard audit, and none is +the next priority. -Expected after that slice: - -```text -R1 setup: 6 correct / 0 wrong / 4 refused -R1 answers: 6 correct / 0 wrong / 4 refused / setup_wrong 0 / gold_error 0 -``` - -Remaining 4 refusals after the slice: - -- `r1-07-inverse` — real reverse-solve capability, deferred. -- `r1-08`, `r1-09`, `r1-10` — correct wrong=0 refusals; stay refused unless a - compound-clause/pronoun/inverse capability is *intentionally designed* with its - own wrong=0 hazard audit. - -This keeps the trajectory clean: move only cases whose semantics are already -supported, and leave the refusal boundary intact. +The next capability axis is **not** another R1 slice. It is **R2: finite integer +linear-constraint systems** (two-category problems — buses/seats, chickens/legs, +tickets/prices, coins/values) built as a parallel off-serving organ on the same +disciplined ladder (gold → setup oracle → solver → answer verifier → reader). R1 +stays frozen at 7 / 0 / 3 as that work proceeds; the invariant carried forward is +the same one this ledger tracks: **move only cases whose semantics are already +supported, leave the refusal boundary intact, never produce a `setup_wrong`.** diff --git a/evals/setup_oracle/r1_gold.jsonl b/evals/setup_oracle/r1_gold.jsonl index 62c4bc83..5585004f 100644 --- a/evals/setup_oracle/r1_gold.jsonl +++ b/evals/setup_oracle/r1_gold.jsonl @@ -4,7 +4,7 @@ {"id": "r1-04-fewer-total", "text": "Gail has 20 cards. Hank has 6 fewer cards than Gail. How many cards do Gail and Hank have in total?", "relations": [{"kind": "fact", "entity": "gail", "value": 20}, {"kind": "fewer_than", "entity": "hank", "ref": "gail", "delta": 6}, {"kind": "sum_of", "entity": "total", "parts": ["gail", "hank"]}], "expected_units": {"gail": "item", "hank": "item", "total": "item"}, "query": {"entity": "total", "unit": "item"}, "gold": 34, "notes": "Additive(fewer) + aggregate (aggregate-query slice): the query closes with a two-token qualifier 'in total' after 'have'; the recognizer strips it and reads the multi-part sum, honored ONLY for the multi-part form. hank = gail - 6 = 14; total = gail + hank = 34."} {"id": "r1-05-chain", "text": "Ivy has 4 pens. Jon has 3 times as many pens as Ivy. Kim has 2 more pens than Jon. How many pens does Kim have?", "relations": [{"kind": "fact", "entity": "ivy", "value": 4}, {"kind": "times_as_many", "entity": "jon", "ref": "ivy", "factor": 3}, {"kind": "more_than", "entity": "kim", "ref": "jon", "delta": 2}], "expected_units": {"ivy": "item", "jon": "item", "kim": "item"}, "query": {"entity": "kim", "unit": "item"}, "gold": 14, "notes": "Multi-step derived chain: jon=3*ivy (intermediate), kim=jon+2. The multiplicative middle step has no template -> must REFUSE the whole reading, never read a partial/wrong chain."} {"id": "r1-06-subtotal-reused", "text": "Lee has 5 hats. Mae has 7 hats. They combine their hats and split them equally into 3 boxes. How many hats are in each box?", "relations": [{"kind": "fact", "entity": "lee", "value": 5}, {"kind": "fact", "entity": "mae", "value": 7}, {"kind": "sum_of", "entity": "total", "parts": ["lee", "mae"]}, {"kind": "divide_by", "entity": "per_box", "ref": "total", "divisor": 3}], "expected_units": {"lee": "item", "mae": "item", "total": "item", "per_box": "item"}, "query": {"entity": "per_box", "unit": "item"}, "gold": 4, "notes": "Aggregate-then-divide partition (PR-6d): semantic source is equal PARTITION ('split equally into 3 boxes'); the mathematical setup is total = lee + mae (sum_of) then per_box = total / 3 (divide_by), reusing SumOf + Div with NO new relation kind. First derived-subtotal reuse: the divide's ref is itself a derived symbol. Exact-divisibility still gates the answer (total % 3 == 0; 12 / 3 = 4)."} -{"id": "r1-07-inverse", "text": "Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?", "relations": [{"kind": "fact", "entity": "nia", "value": 15}, {"kind": "more_than", "entity": "nia", "ref": "omar", "delta": 9}], "expected_units": {"nia": "item", "omar": "item"}, "query": {"entity": "omar", "unit": "item"}, "gold": 6, "notes": "Inverse target: omar = nia - 9 = 6, asked via the base of a more_than (nia=15, nia=omar+9, ask omar). PR-7a pins the ORACLE reverse-solve: the gold relations feed oracle_answer (a more_than whose entity is grounded inverts to its ref). The reader is unchanged and STILL refuses (admissibility_refused), so this stays setup_refused / answer-refused until PR-7b reads the inverse frame; gold=6 is dormant until then. The non-consumed ask_base annotation was dropped to match the reader's future projection. setup_correct OR refuse, but NEVER setup_wrong."} +{"id": "r1-07-inverse", "text": "Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?", "relations": [{"kind": "fact", "entity": "nia", "value": 15}, {"kind": "more_than", "entity": "nia", "ref": "omar", "delta": 9}], "expected_units": {"nia": "item", "omar": "item"}, "query": {"entity": "omar", "unit": "item"}, "gold": 6, "notes": "Inverse target: omar = nia - 9 = 6, asked via the base of a more_than (nia=15, nia=omar+9, ask omar). PR-7b reads the inverse frame: the reader binds the unknown base's unit FROM the relation (omar -> item) so the equation is admissible, emitting exactly these gold relations + units + target (setup_correct). The answer lane feeds the gold relations to the PR-7a oracle reverse-solve (a more_than whose entity is grounded inverts to its ref) -> 6. Bounded: single base == query target (no chains), known subject value, base not otherwise grounded, <=1 inverse, never over times/divide. The non-consumed ask_base annotation was dropped to match the reader's projection. setup_correct OR refuse, but NEVER setup_wrong."} {"id": "r1-08-ambiguous-referent", "text": "Pat has 5 marbles. He has 3 more than her. How many marbles does she have?", "relations": [], "expected_units": {}, "query": {"entity": "she", "unit": "item"}, "notes": "Ambiguous pronoun referents (he/her/she) with no grounded base for 'her'/'she' -> must REFUSE (no honest reading), never bind a guessed referent."} {"id": "r1-09-missing-base", "text": "Quinn has twice as many toys as Rosa. How many toys does Quinn have?", "relations": [{"kind": "times_as_many", "entity": "quinn", "ref": "rosa", "factor": 2}], "expected_units": {"quinn": "item", "rosa": "item"}, "query": {"entity": "quinn", "unit": "item"}, "notes": "Missing base quantity: Rosa's count is never given, so Quinn is underdetermined -> must REFUSE."} {"id": "r1-10-distractor", "text": "Sam has 7 pencils and 3 erasers. Tom has 4 more pencils than Sam. How many pencils does Tom have?", "relations": [{"kind": "fact", "entity": "sam", "value": 7}, {"kind": "more_than", "entity": "tom", "ref": "sam", "delta": 4}], "expected_units": {"sam": "item", "tom": "item"}, "query": {"entity": "tom", "unit": "item"}, "notes": "Distractor quantity (3 erasers) in a compound clause the 'X has N unit' template can't parse -> must REFUSE rather than mis-bind the distractor."} diff --git a/generate/quantitative_comprehension.py b/generate/quantitative_comprehension.py index db802762..b40c55cc 100644 --- a/generate/quantitative_comprehension.py +++ b/generate/quantitative_comprehension.py @@ -20,6 +20,13 @@ Templates (function-word + order; digits only — a non-digit quantity REFUSES): - query ``How many does have`` -> ask Y - query ``How many do and have [altogether|in total]`` -> total = X + Y; ask total +Inverse frame (PR-7b): when a ``more/fewer than`` clause's SUBJECT is grounded by a fact +and its REFERENT is the otherwise-ungrounded query target, the referent is the unknown +base. Its unit is bound FROM the relation (same unit) so the equation is admissible, and +the answer oracle reverse-solves the value (``Nia has 9 more beads than Omar. Nia has 15. +-> Omar = 6``). Bounded: a single base that IS the query target (no chains), a known +subject value (a fact), a base not otherwise grounded, ≤1 inverse, never over times/divide. + Refusal-first: an unparseable clause, a non-digit quantity, a non-identifier name, a missing/duplicated query, or an admissibility refusal all return a typed ``Refusal`` — never a fabricated quantity (wrong=0 at the comprehension layer). @@ -399,6 +406,28 @@ def comprehend_quantitative(text: str, source_id: str = "input") -> QuantCompreh role_of[ask_entity] = "total" sum_eq = (ask_entity, parts) + # Narrow inverse frame (PR-7b): a more/fewer-than whose SUBJECT is grounded by a fact + # and whose REFERENT is the otherwise-ungrounded query target is an inverse constraint + # pinning the unknown base. Bind the base's unit FROM the relation (same unit) so the + # equation is admissible and the base carries a unit; the answer oracle reverse-solves + # the value (PR-7a). Strictly bounded — the base must BE the single query target (no + # chains), the subject value must be known (a fact), the base must not be otherwise + # grounded, and at most one such constraint may exist. Only an _Eq (add/subtract) is + # ever an inverse here: a _Mul/_Div with an ungrounded ref stays unbound and refuses, + # so the contract never reverse-solves over times/divide. Non-negativity of the solved + # base is the oracle's boundary (PR-7a), not the reader's — the reader admits the SETUP. + fact_entities = {f.entity for f in facts} + inverse_eqs = [ + e for e in eqs + if e.entity in fact_entities and e.ref not in unit_of and e.ref == ask_entity + ] + if len(inverse_eqs) > 1: + return Refusal("multiple_inverse_bases") + if inverse_eqs: + base = inverse_eqs[0] + unit_of[base.ref] = base.unit + role_of[base.ref] = "count" + referenced: set[str] = set() for f in facts: referenced.add(f.entity) diff --git a/tests/test_quantitative_comprehension.py b/tests/test_quantitative_comprehension.py index d0dc6169..d41a8b48 100644 --- a/tests/test_quantitative_comprehension.py +++ b/tests/test_quantitative_comprehension.py @@ -361,3 +361,71 @@ def test_aggregate_query_unit_incompatible_part_refuses() -> None: assert isinstance(comp, Refusal) assert comp.reason == "admissibility_refused" assert "unit_mismatch" in comp.detail + + +# --------------------------------------------------------------------------- # +# Inverse frame (PR-7b): a more/fewer-than whose SUBJECT is a known fact and whose +# REFERENT is the otherwise-ungrounded query target pins the unknown base. The base's +# unit is bound FROM the relation so the equation is admissible; the answer oracle +# reverse-solves the value (PR-7a). Bounded — single base == query target, known subject, +# base not otherwise grounded, <=1 inverse, never over times/divide. +# --------------------------------------------------------------------------- # + + +def test_inverse_more_than_reads_base_with_bound_unit() -> None: + from evals.relational_metric.oracle import oracle_answer + + # Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have? + comp = _comp("Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?") + # The base (omar) carries the relation's unit even though it has no fact of its own. + units = {s.symbol_id: s.unit for s in comp.binding_graph.symbols} + assert units == {"nia": "item", "omar": "item"} + assert single_unknown(comp.binding_graph).symbol_id == "omar" + relations, query = to_relational_metric(comp) + assert query == {"entity": "omar", "unit": "item"} + assert {"kind": "fact", "entity": "nia", "value": 15} in relations + assert {"kind": "more_than", "entity": "nia", "ref": "omar", "delta": 9} in relations + assert oracle_answer(relations, query) == 6 # omar = 15 - 9 + + +def test_inverse_fewer_than_reads_base() -> None: + from evals.relational_metric.oracle import oracle_answer + + # Quinn has 4 fewer beads than Pat. Quinn has 10 beads. How many beads does Pat have? + comp = _comp("Quinn has 4 fewer beads than Pat. Quinn has 10 beads. How many beads does Pat have?") + assert single_unknown(comp.binding_graph).symbol_id == "pat" + relations, query = to_relational_metric(comp) + assert oracle_answer(relations, query) == 14 # pat = 10 + 4 + + +def test_inverse_base_must_be_query_target_refuses() -> None: + # The unknown base (omar) is NOT what's asked — the question asks the grounded subject + # while the base stays unbound. No inverse fires (ref != query target); the equation's + # ungrounded operand makes admissibility REFUSE rather than guess. (no chains) + comp = comprehend_quantitative( + "Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Nia have?" + ) + assert isinstance(comp, Refusal) + assert comp.reason == "admissibility_refused" + + +def test_multiple_inverse_bases_refuses() -> None: + # Two known subjects each pin the SAME unknown base -> an over-determined inverse, not a + # single base. The reader REFUSES rather than bind from one and drop the other. Without + # the len>1 guard this would emit a setup the oracle then chokes on; refusing is honest. + comp = comprehend_quantitative( + "Nia has 9 more beads than Omar. Pam has 5 more beads than Omar. " + "Nia has 15 beads. Pam has 11 beads. How many beads does Omar have?" + ) + assert isinstance(comp, Refusal) + assert comp.reason == "multiple_inverse_bases" + + +def test_inverse_over_times_as_many_refuses() -> None: + # The inverse frame is add/subtract ONLY. A times-as-many with an ungrounded ref is + # never reverse-solved: the ref stays unit-unbound and admissibility REFUSES. + comp = comprehend_quantitative( + "Nia has twice as many beads as Omar. Nia has 14 beads. How many beads does Omar have?" + ) + assert isinstance(comp, Refusal) + assert comp.reason == "admissibility_refused" diff --git a/tests/test_setup_oracle.py b/tests/test_setup_oracle.py index 7b442e53..2a122259 100644 --- a/tests/test_setup_oracle.py +++ b/tests/test_setup_oracle.py @@ -172,8 +172,11 @@ def test_r1_comparative_supported_rest_refused_wrong_zero() -> None: # the existing sum_of; no new arithmetic or relation kind. assert by_id["r1-03-more-total"] == "correct" assert by_id["r1-04-fewer-total"] == "correct" - assert r["setup_correct"] == 6 - assert r["setup_refused"] == 4 + # Inverse frame (PR-7b): the base of a more_than whose subject is a known fact and + # whose referent is the query target — "Nia has 9 more than Omar. Nia has 15. -> Omar". + assert by_id["r1-07-inverse"] == "correct" + assert r["setup_correct"] == 7 + assert r["setup_refused"] == 3 # No detail is ever WRONG, and every non-correct one is a typed refusal. for d in r["details"]: assert d["outcome"] in ("correct", "refused") @@ -218,8 +221,8 @@ def test_r1_answer_lane_scores_only_setup_correct_fixtures() -> None: assert r["setup_wrong"] == 0 assert r["wrong"] == 0 assert r["gold_error"] == 0 - assert r["correct"] == 6 - assert r["refused"] == 4 + assert r["correct"] == 7 + assert r["refused"] == 3 by_id = {d["id"]: d for d in r["details"]} assert by_id["r1-01-twice"] == {"id": "r1-01-twice", "outcome": "correct", "answer": 12} assert by_id["r1-02-half"] == {"id": "r1-02-half", "outcome": "correct", "answer": 4} @@ -229,9 +232,11 @@ def test_r1_answer_lane_scores_only_setup_correct_fixtures() -> None: # Aggregate-query slice: additive totals via "altogether" / "in total". assert by_id["r1-03-more-total"] == {"id": "r1-03-more-total", "outcome": "correct", "answer": 25} assert by_id["r1-04-fewer-total"] == {"id": "r1-04-fewer-total", "outcome": "correct", "answer": 34} + # Inverse frame (PR-7b): the reverse-solved base — omar = nia(15) - 9 = 6. + assert by_id["r1-07-inverse"] == {"id": "r1-07-inverse", "outcome": "correct", "answer": 6} _supported = { "r1-01-twice", "r1-02-half", "r1-05-chain", "r1-06-subtotal-reused", - "r1-03-more-total", "r1-04-fewer-total", + "r1-03-more-total", "r1-04-fewer-total", "r1-07-inverse", } for fixture_id, detail in by_id.items(): if fixture_id not in _supported: @@ -310,9 +315,9 @@ def test_oracle_divide_by_one_is_identity() -> None: # --------------------------------------------------------------------------- # # PR-7a — narrow reverse-solve oracle contract (the base of one more/fewer_than). -# Pins the EXACT semantics before the reader learns the inverse frame (PR-7b). The -# reader is unchanged here: r1-07 still refuses; these exercise the oracle directly. -# Each refusal is meaningful-fail — drop its guardrail and the case computes a value. +# Pins the EXACT semantics the reader's inverse frame (PR-7b) relies on; these exercise +# the oracle directly (independent of the reader). Each refusal is meaningful-fail — drop +# its guardrail and the case computes a value. # --------------------------------------------------------------------------- # @@ -343,7 +348,7 @@ def test_r1_07_gold_relations_reverse_solve_to_six() -> None: from evals.setup_oracle.runner import _load_r1_gold fx = next(f for f in run_r1()["details"] if f["id"] == "r1-07-inverse") - assert fx["outcome"] == "refused" # reader still refuses in PR-7a (contract only) + assert fx["outcome"] == "correct" # reader now reads the inverse frame (PR-7b) gold = next(g for g in _load_r1_gold() if g["id"] == "r1-07-inverse") assert oracle_answer(gold["relations"], gold["query"]) == gold["gold"] == 6