Merge pull request #624 from AssetOverflow/feat/r1-inverse-frame
feat(comprehension): R1 inverse reader frame — close R1 at 7/0/3 (PR-7b)
This commit is contained in:
Shay
GitHub
commit
8d2c270144
5 changed files with 152 additions and 83 deletions
|
|
@ -1,21 +1,26 @@
|
|||
# R1 comprehension inventory ledger
|
||||
|
||||
**As of:** `main @ 5ada1392` (post-#616, PR-6d landed)
|
||||
**Lane state:** R1 setup **4 / 0 / 6** · R1 answers **4 / 0 / 6** (`setup_wrong 0`, `gold_error 0`) · 15-case setup **15 / 0 / 0**
|
||||
**As of:** PR-7b / C0 (inverse reader frame), on `main @ 0e6a7f9a` (post-#619) + this branch
|
||||
**Lane state:** R1 setup **7 / 0 / 3** · R1 answers **7 / 0 / 3** (`setup_wrong 0`, `gold_error 0`) · 15-case setup **15 / 0 / 0**
|
||||
|
||||
This is a baseline decision artifact, not a capability claim. It records exactly
|
||||
which `evals/setup_oracle/r1_gold.jsonl` fixtures the typed comprehension organ
|
||||
now *reads and answers*, which it *refuses*, and — for each refusal — the
|
||||
semantic family that blocks it and whether that refusal is a coverage gap or a
|
||||
load-bearing wrong=0 boundary that **should stay refused**. It exists so the next
|
||||
slice moves only the cases whose semantics are already supported, and never
|
||||
mistakes a correct refusal for a gap.
|
||||
This is a decision artifact, not a capability claim. It records exactly which
|
||||
`evals/setup_oracle/r1_gold.jsonl` fixtures the typed comprehension organ now
|
||||
*reads and answers*, which it *refuses*, and — for each refusal — the semantic
|
||||
family that blocks it and whether that refusal is a coverage gap or a load-bearing
|
||||
wrong=0 boundary that **should stay refused**. It exists so the next slice moves
|
||||
only the cases whose semantics are already supported, and never mistakes a correct
|
||||
refusal for a gap.
|
||||
|
||||
> **History.** This ledger opened at **4 / 0 / 6** (`main @ 5ada1392`, post-PR-6d).
|
||||
> The additive aggregate-query slice (#618) flipped `r1-03`/`r1-04` → **6 / 0 / 4**;
|
||||
> the inverse reader frame (PR-7b / C0) flips `r1-07` → **7 / 0 / 3**. With that, R1
|
||||
> is **closed**: the three remaining refusals are correct wrong=0 boundaries, not gaps.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m evals.setup_oracle r1 # setup lane -> 4/0/6
|
||||
.venv/bin/python -m evals.setup_oracle r1-answers # answer lane -> 4/0/6, setup_wrong 0, gold_error 0
|
||||
.venv/bin/python -m evals.setup_oracle r1 # setup lane -> 7/0/3
|
||||
.venv/bin/python -m evals.setup_oracle r1-answers # answer lane -> 7/0/3, setup_wrong 0, gold_error 0
|
||||
.venv/bin/python -m evals.setup_oracle # 15-case setup gold -> 15/0/0
|
||||
```
|
||||
|
||||
|
|
@ -32,14 +37,14 @@ mis-valued.
|
|||
| `r1-02-half` | `Dora has half as many … as Carl (8)` | ✅ correct | `4` | divisive (`half`) | — | **admitted** |
|
||||
| `r1-05-chain` | `Jon = 3×Ivy(4); Kim = Jon + 2` | ✅ correct | `14` | multi-step chain (mul → add) | — | **admitted** |
|
||||
| `r1-06-subtotal-reused` | `total = Lee(5)+Mae(7); per_box = total/3` | ✅ correct | `4` | aggregate-then-divide partition | — | **admitted** |
|
||||
| `r1-03-more-total` | `Evan = Finn(10)+5; ask total **altogether**` | ⛔ refused | ⛔ refused | additive aggregate + aggregate-query phrasing | `unreadable_quantity_query` | **gap (phrasing)** |
|
||||
| `r1-04-fewer-total` | `Hank = Gail(20)−6; ask total **in total**` | ⛔ refused | ⛔ refused | additive aggregate + aggregate-query phrasing | `unreadable_quantity_query` | **gap (phrasing)** |
|
||||
| `r1-07-inverse` | `Nia(15) = Omar + 9; ask Omar` | ⛔ refused | ⛔ refused | inverse target (ask the *base* of a relation) | `admissibility_refused` | **gap (real capability)** |
|
||||
| `r1-03-more-total` | `Evan = Finn(10)+5; ask total **altogether**` | ✅ correct | `25` | additive aggregate + aggregate-query phrasing | — | **admitted** |
|
||||
| `r1-04-fewer-total` | `Hank = Gail(20)−6; ask total **in total**` | ✅ correct | `34` | additive aggregate + aggregate-query phrasing | — | **admitted** |
|
||||
| `r1-07-inverse` | `Nia(15) = Omar + 9; ask Omar` | ✅ correct | `6` | inverse target (ask the *base* of a relation) | — | **admitted** |
|
||||
| `r1-08-ambiguous-referent` | `He has 3 more than her; ask she` | ⛔ refused | ⛔ refused | unresolved pronoun referent | `unreadable_quantity_clause` | **correct refusal** |
|
||||
| `r1-09-missing-base` | `Quinn = 2×Rosa; Rosa never given` | ⛔ refused | ⛔ refused | ungrounded base (no grounded fact) | `no_single_quantity_query` | **correct refusal** |
|
||||
| `r1-10-distractor` | `Sam has 7 pencils **and 3 erasers**; ask Tom` | ⛔ refused | ⛔ refused | distractor in a compound clause | `unreadable_quantity_clause` | **correct refusal** |
|
||||
|
||||
## The 4 admitted families and the invariant protecting each
|
||||
## The 7 admitted families and the invariant protecting each
|
||||
|
||||
Each admitted family is protected by a specific projection/admissibility gate AND
|
||||
the two independent oracles. A family is "admitted" only because a violation of
|
||||
|
|
@ -52,70 +57,32 @@ silently.
|
|||
| divisive | `r1-02` | divisor-only projection: `Div(Symbol, Literal)` is the **only** divisive `to_relation`; single-dep divide admissibility, symmetric with multiply (PR-6c); oracle **exact-divisibility** gate `base % divisor == 0` — an odd base refuses, never rounds. |
|
||||
| multi-step chain | `r1-05` | composition of the above two gates over a derived intermediate; the whole reading refuses if any step's projection/admissibility fails (no partial chain). |
|
||||
| aggregate-then-divide partition | `r1-06` | partition/query coherence guard (`partition ⇔ perquery`, else `partition_query_mismatch`; container match else `partition_container_mismatch`); reuses `SumOf` + `Div` with **no new relation kind**; exact-divisibility gates the answer over a *derived* total (PR-6d). |
|
||||
| all admitted | — | setup oracle: `reader_rel == gold_rel ∧ reader_units == gold_units ∧ reader_unk == gold_unk` (any drift → `setup_wrong`). Answer oracle: forward-substitution + exact integer arithmetic (any mis-value → `wrong`/`gold_error`). |
|
||||
| additive aggregate (query-phrasing) | `r1-03`, `r1-04` | the trailing-qualifier recognizer (`altogether` / `in total`) is honored **only** for the multi-part `sumquery` form; an ungrounded or unit-incompatible part is refused downstream at admissibility (`unit_unbound` / `unit_mismatch`). No new arithmetic, no new relation kind (#618). |
|
||||
| inverse target | `r1-07` | inverse frame (PR-7b): a `more`/`fewer`-than whose **subject is a known fact** and whose **referent is the otherwise-ungrounded query target** binds the base's unit **from the relation** so the equation is admissible; the answer oracle reverse-solves it (PR-7a). Bounded — single base == query target (no chains), known subject value, base not otherwise grounded, ≤1 inverse (`multiple_inverse_bases` else), never over times/divide. |
|
||||
| all admitted | — | setup oracle: `reader_rel == gold_rel ∧ reader_units == gold_units ∧ reader_unk == gold_unk` (any drift → `setup_wrong`). Answer oracle: forward-substitution + one narrow reverse-solve + exact integer arithmetic (any mis-value → `wrong`/`gold_error`). |
|
||||
|
||||
## The 6 refusals, classified
|
||||
## The 3 refusals, classified — all correct wrong=0 boundaries
|
||||
|
||||
The headline: **of 6 refusals, only 2 are pure phrasing gaps; 1 is a real missing
|
||||
capability; 3 are correct wrong=0 boundaries that must stay refused.**
|
||||
|
||||
### Gaps — phrasing only (2): `r1-03`, `r1-04`
|
||||
|
||||
Structurally **fully supported** — fact + `more_than`/`fewer_than` + `sum_of` are
|
||||
all admitted families. The *only* blocker is the query template at
|
||||
`generate/quantitative_comprehension.py:254` requiring `toks[-1] == "have"`:
|
||||
|
||||
- `"… do Evan and Finn have altogether?"` → `toks[-1] == "altogether"` → falls through to `unreadable_quantity_query` (`:262`).
|
||||
- `"… do Gail and Hank have in total?"` → `toks[-1] == "total"` → same.
|
||||
|
||||
A trailing aggregate adverb (`altogether`) or qualifier (`in total`) defeats the
|
||||
`have`-terminal check. This is the **next safe slice**: widen the aggregate-query
|
||||
recognizer to accept the trailing qualifier when all named parts are already
|
||||
grounded and unit-compatible. No new arithmetic, no new relation kind. Expected
|
||||
post-slice: **R1 6 / 0 / 4**.
|
||||
|
||||
### Gap — real capability (1): `r1-07`
|
||||
|
||||
`Nia(15) = Omar + 9; ask Omar` is an **inverse** problem: the asked entity `Omar`
|
||||
appears as a *dependency* of a grounded lhs (`Nia`), not as a derivable lhs with
|
||||
its own equation. The reader can read the structure honestly, but forward
|
||||
substitution + forward admissibility cannot invert (`Omar = Nia − 9`); `Omar` has
|
||||
no grounded unit as a bare ref, so `check_admissibility` raises and the build
|
||||
returns `admissibility_refused` (`:464`). Closing this needs a genuine
|
||||
reverse-solve contract, not a phrasing tweak — **defer**. Until then the refusal
|
||||
is the correct wrong=0 behavior (the gold note pins: setup_correct OR refuse,
|
||||
**never** setup_wrong).
|
||||
|
||||
### Correct refusals — must stay refused (3): `r1-08`, `r1-09`, `r1-10`
|
||||
|
||||
Admitting any of these would breach wrong=0. They are the refusal boundary
|
||||
working, not coverage gaps:
|
||||
The headline: **of 3 remaining refusals, 0 are gaps; all 3 are correct wrong=0
|
||||
boundaries that must stay refused.** Admitting any would breach wrong=0; they are
|
||||
the refusal boundary working, not coverage gaps:
|
||||
|
||||
- **`r1-08` (pronoun):** `"He has 3 more than her"` has no grounded base for `her`/`she`; binding a guessed referent would fabricate. `unreadable_quantity_clause`.
|
||||
- **`r1-09` (ungrounded base):** `Quinn = 2×Rosa` with Rosa never given is underdetermined; the `not facts` guard (`:342`) refuses with `no_single_quantity_query`. A "twice as many" with no grounded anchor must refuse.
|
||||
- **`r1-09` (ungrounded base):** `Quinn = 2×Rosa` with Rosa never given is underdetermined; the `not facts` guard refuses with `no_single_quantity_query`. A "twice as many" with no grounded anchor must refuse. (Note: distinct from `r1-07` — there the *subject* `Nia` is a known fact and only the *base* is unknown, so the inverse frame can solve it; here *nothing* is grounded.)
|
||||
- **`r1-10` (distractor):** `"Sam has 7 pencils and 3 erasers"` is a compound clause the `X has N unit` template intentionally cannot parse; mis-binding the distractor (`3 erasers`) would corrupt the reading. `unreadable_quantity_clause`. Only an intentionally-designed compound-clause parser would move this — and only with a wrong=0 hazard audit first.
|
||||
|
||||
## Decision and trajectory
|
||||
|
||||
Next capability slice (per pinned scope): **additive aggregate query variants** —
|
||||
`"… have altogether?"` and `"… have in total?"` — flipping `r1-03` and `r1-04`
|
||||
only. Constraints: no new arithmetic, no new relation kind, no inverse solving, no
|
||||
distractor handling, no pronoun resolution; widen the aggregate-query recognizer
|
||||
**only** when all named parts are already grounded and unit-compatible.
|
||||
**R1 is closed at 7 / 0 / 3.** Every fixture whose semantics the typed organ can
|
||||
honestly read is admitted; the three remaining refusals are load-bearing wrong=0
|
||||
boundaries (pronoun resolution, ungrounded base, distractor parsing) — each would
|
||||
need an *intentionally designed* capability with its own hazard audit, and none is
|
||||
the next priority.
|
||||
|
||||
Expected after that slice:
|
||||
|
||||
```text
|
||||
R1 setup: 6 correct / 0 wrong / 4 refused
|
||||
R1 answers: 6 correct / 0 wrong / 4 refused / setup_wrong 0 / gold_error 0
|
||||
```
|
||||
|
||||
Remaining 4 refusals after the slice:
|
||||
|
||||
- `r1-07-inverse` — real reverse-solve capability, deferred.
|
||||
- `r1-08`, `r1-09`, `r1-10` — correct wrong=0 refusals; stay refused unless a
|
||||
compound-clause/pronoun/inverse capability is *intentionally designed* with its
|
||||
own wrong=0 hazard audit.
|
||||
|
||||
This keeps the trajectory clean: move only cases whose semantics are already
|
||||
supported, and leave the refusal boundary intact.
|
||||
The next capability axis is **not** another R1 slice. It is **R2: finite integer
|
||||
linear-constraint systems** (two-category problems — buses/seats, chickens/legs,
|
||||
tickets/prices, coins/values) built as a parallel off-serving organ on the same
|
||||
disciplined ladder (gold → setup oracle → solver → answer verifier → reader). R1
|
||||
stays frozen at 7 / 0 / 3 as that work proceeds; the invariant carried forward is
|
||||
the same one this ledger tracks: **move only cases whose semantics are already
|
||||
supported, leave the refusal boundary intact, never produce a `setup_wrong`.**
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
{"id": "r1-04-fewer-total", "text": "Gail has 20 cards. Hank has 6 fewer cards than Gail. How many cards do Gail and Hank have in total?", "relations": [{"kind": "fact", "entity": "gail", "value": 20}, {"kind": "fewer_than", "entity": "hank", "ref": "gail", "delta": 6}, {"kind": "sum_of", "entity": "total", "parts": ["gail", "hank"]}], "expected_units": {"gail": "item", "hank": "item", "total": "item"}, "query": {"entity": "total", "unit": "item"}, "gold": 34, "notes": "Additive(fewer) + aggregate (aggregate-query slice): the query closes with a two-token qualifier 'in total' after 'have'; the recognizer strips it and reads the multi-part sum, honored ONLY for the multi-part form. hank = gail - 6 = 14; total = gail + hank = 34."}
|
||||
{"id": "r1-05-chain", "text": "Ivy has 4 pens. Jon has 3 times as many pens as Ivy. Kim has 2 more pens than Jon. How many pens does Kim have?", "relations": [{"kind": "fact", "entity": "ivy", "value": 4}, {"kind": "times_as_many", "entity": "jon", "ref": "ivy", "factor": 3}, {"kind": "more_than", "entity": "kim", "ref": "jon", "delta": 2}], "expected_units": {"ivy": "item", "jon": "item", "kim": "item"}, "query": {"entity": "kim", "unit": "item"}, "gold": 14, "notes": "Multi-step derived chain: jon=3*ivy (intermediate), kim=jon+2. The multiplicative middle step has no template -> must REFUSE the whole reading, never read a partial/wrong chain."}
|
||||
{"id": "r1-06-subtotal-reused", "text": "Lee has 5 hats. Mae has 7 hats. They combine their hats and split them equally into 3 boxes. How many hats are in each box?", "relations": [{"kind": "fact", "entity": "lee", "value": 5}, {"kind": "fact", "entity": "mae", "value": 7}, {"kind": "sum_of", "entity": "total", "parts": ["lee", "mae"]}, {"kind": "divide_by", "entity": "per_box", "ref": "total", "divisor": 3}], "expected_units": {"lee": "item", "mae": "item", "total": "item", "per_box": "item"}, "query": {"entity": "per_box", "unit": "item"}, "gold": 4, "notes": "Aggregate-then-divide partition (PR-6d): semantic source is equal PARTITION ('split equally into 3 boxes'); the mathematical setup is total = lee + mae (sum_of) then per_box = total / 3 (divide_by), reusing SumOf + Div with NO new relation kind. First derived-subtotal reuse: the divide's ref is itself a derived symbol. Exact-divisibility still gates the answer (total % 3 == 0; 12 / 3 = 4)."}
|
||||
{"id": "r1-07-inverse", "text": "Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?", "relations": [{"kind": "fact", "entity": "nia", "value": 15}, {"kind": "more_than", "entity": "nia", "ref": "omar", "delta": 9}], "expected_units": {"nia": "item", "omar": "item"}, "query": {"entity": "omar", "unit": "item"}, "gold": 6, "notes": "Inverse target: omar = nia - 9 = 6, asked via the base of a more_than (nia=15, nia=omar+9, ask omar). PR-7a pins the ORACLE reverse-solve: the gold relations feed oracle_answer (a more_than whose entity is grounded inverts to its ref). The reader is unchanged and STILL refuses (admissibility_refused), so this stays setup_refused / answer-refused until PR-7b reads the inverse frame; gold=6 is dormant until then. The non-consumed ask_base annotation was dropped to match the reader's future projection. setup_correct OR refuse, but NEVER setup_wrong."}
|
||||
{"id": "r1-07-inverse", "text": "Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?", "relations": [{"kind": "fact", "entity": "nia", "value": 15}, {"kind": "more_than", "entity": "nia", "ref": "omar", "delta": 9}], "expected_units": {"nia": "item", "omar": "item"}, "query": {"entity": "omar", "unit": "item"}, "gold": 6, "notes": "Inverse target: omar = nia - 9 = 6, asked via the base of a more_than (nia=15, nia=omar+9, ask omar). PR-7b reads the inverse frame: the reader binds the unknown base's unit FROM the relation (omar -> item) so the equation is admissible, emitting exactly these gold relations + units + target (setup_correct). The answer lane feeds the gold relations to the PR-7a oracle reverse-solve (a more_than whose entity is grounded inverts to its ref) -> 6. Bounded: single base == query target (no chains), known subject value, base not otherwise grounded, <=1 inverse, never over times/divide. The non-consumed ask_base annotation was dropped to match the reader's projection. setup_correct OR refuse, but NEVER setup_wrong."}
|
||||
{"id": "r1-08-ambiguous-referent", "text": "Pat has 5 marbles. He has 3 more than her. How many marbles does she have?", "relations": [], "expected_units": {}, "query": {"entity": "she", "unit": "item"}, "notes": "Ambiguous pronoun referents (he/her/she) with no grounded base for 'her'/'she' -> must REFUSE (no honest reading), never bind a guessed referent."}
|
||||
{"id": "r1-09-missing-base", "text": "Quinn has twice as many toys as Rosa. How many toys does Quinn have?", "relations": [{"kind": "times_as_many", "entity": "quinn", "ref": "rosa", "factor": 2}], "expected_units": {"quinn": "item", "rosa": "item"}, "query": {"entity": "quinn", "unit": "item"}, "notes": "Missing base quantity: Rosa's count is never given, so Quinn is underdetermined -> must REFUSE."}
|
||||
{"id": "r1-10-distractor", "text": "Sam has 7 pencils and 3 erasers. Tom has 4 more pencils than Sam. How many pencils does Tom have?", "relations": [{"kind": "fact", "entity": "sam", "value": 7}, {"kind": "more_than", "entity": "tom", "ref": "sam", "delta": 4}], "expected_units": {"sam": "item", "tom": "item"}, "query": {"entity": "tom", "unit": "item"}, "notes": "Distractor quantity (3 erasers) in a compound clause the 'X has N unit' template can't parse -> must REFUSE rather than mis-bind the distractor."}
|
||||
|
|
|
|||
|
|
@ -20,6 +20,13 @@ Templates (function-word + order; digits only — a non-digit quantity REFUSES):
|
|||
- query ``How many <unit> does <Y> have`` -> ask Y
|
||||
- query ``How many <unit> do <X> and <Y> have [altogether|in total]`` -> total = X + Y; ask total
|
||||
|
||||
Inverse frame (PR-7b): when a ``more/fewer than`` clause's SUBJECT is grounded by a fact
|
||||
and its REFERENT is the otherwise-ungrounded query target, the referent is the unknown
|
||||
base. Its unit is bound FROM the relation (same unit) so the equation is admissible, and
|
||||
the answer oracle reverse-solves the value (``Nia has 9 more beads than Omar. Nia has 15.
|
||||
-> Omar = 6``). Bounded: a single base that IS the query target (no chains), a known
|
||||
subject value (a fact), a base not otherwise grounded, ≤1 inverse, never over times/divide.
|
||||
|
||||
Refusal-first: an unparseable clause, a non-digit quantity, a non-identifier name,
|
||||
a missing/duplicated query, or an admissibility refusal all return a typed
|
||||
``Refusal`` — never a fabricated quantity (wrong=0 at the comprehension layer).
|
||||
|
|
@ -399,6 +406,28 @@ def comprehend_quantitative(text: str, source_id: str = "input") -> QuantCompreh
|
|||
role_of[ask_entity] = "total"
|
||||
sum_eq = (ask_entity, parts)
|
||||
|
||||
# Narrow inverse frame (PR-7b): a more/fewer-than whose SUBJECT is grounded by a fact
|
||||
# and whose REFERENT is the otherwise-ungrounded query target is an inverse constraint
|
||||
# pinning the unknown base. Bind the base's unit FROM the relation (same unit) so the
|
||||
# equation is admissible and the base carries a unit; the answer oracle reverse-solves
|
||||
# the value (PR-7a). Strictly bounded — the base must BE the single query target (no
|
||||
# chains), the subject value must be known (a fact), the base must not be otherwise
|
||||
# grounded, and at most one such constraint may exist. Only an _Eq (add/subtract) is
|
||||
# ever an inverse here: a _Mul/_Div with an ungrounded ref stays unbound and refuses,
|
||||
# so the contract never reverse-solves over times/divide. Non-negativity of the solved
|
||||
# base is the oracle's boundary (PR-7a), not the reader's — the reader admits the SETUP.
|
||||
fact_entities = {f.entity for f in facts}
|
||||
inverse_eqs = [
|
||||
e for e in eqs
|
||||
if e.entity in fact_entities and e.ref not in unit_of and e.ref == ask_entity
|
||||
]
|
||||
if len(inverse_eqs) > 1:
|
||||
return Refusal("multiple_inverse_bases")
|
||||
if inverse_eqs:
|
||||
base = inverse_eqs[0]
|
||||
unit_of[base.ref] = base.unit
|
||||
role_of[base.ref] = "count"
|
||||
|
||||
referenced: set[str] = set()
|
||||
for f in facts:
|
||||
referenced.add(f.entity)
|
||||
|
|
|
|||
|
|
@ -361,3 +361,71 @@ def test_aggregate_query_unit_incompatible_part_refuses() -> None:
|
|||
assert isinstance(comp, Refusal)
|
||||
assert comp.reason == "admissibility_refused"
|
||||
assert "unit_mismatch" in comp.detail
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Inverse frame (PR-7b): a more/fewer-than whose SUBJECT is a known fact and whose
|
||||
# REFERENT is the otherwise-ungrounded query target pins the unknown base. The base's
|
||||
# unit is bound FROM the relation so the equation is admissible; the answer oracle
|
||||
# reverse-solves the value (PR-7a). Bounded — single base == query target, known subject,
|
||||
# base not otherwise grounded, <=1 inverse, never over times/divide.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_inverse_more_than_reads_base_with_bound_unit() -> None:
|
||||
from evals.relational_metric.oracle import oracle_answer
|
||||
|
||||
# Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?
|
||||
comp = _comp("Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Omar have?")
|
||||
# The base (omar) carries the relation's unit even though it has no fact of its own.
|
||||
units = {s.symbol_id: s.unit for s in comp.binding_graph.symbols}
|
||||
assert units == {"nia": "item", "omar": "item"}
|
||||
assert single_unknown(comp.binding_graph).symbol_id == "omar"
|
||||
relations, query = to_relational_metric(comp)
|
||||
assert query == {"entity": "omar", "unit": "item"}
|
||||
assert {"kind": "fact", "entity": "nia", "value": 15} in relations
|
||||
assert {"kind": "more_than", "entity": "nia", "ref": "omar", "delta": 9} in relations
|
||||
assert oracle_answer(relations, query) == 6 # omar = 15 - 9
|
||||
|
||||
|
||||
def test_inverse_fewer_than_reads_base() -> None:
|
||||
from evals.relational_metric.oracle import oracle_answer
|
||||
|
||||
# Quinn has 4 fewer beads than Pat. Quinn has 10 beads. How many beads does Pat have?
|
||||
comp = _comp("Quinn has 4 fewer beads than Pat. Quinn has 10 beads. How many beads does Pat have?")
|
||||
assert single_unknown(comp.binding_graph).symbol_id == "pat"
|
||||
relations, query = to_relational_metric(comp)
|
||||
assert oracle_answer(relations, query) == 14 # pat = 10 + 4
|
||||
|
||||
|
||||
def test_inverse_base_must_be_query_target_refuses() -> None:
|
||||
# The unknown base (omar) is NOT what's asked — the question asks the grounded subject
|
||||
# while the base stays unbound. No inverse fires (ref != query target); the equation's
|
||||
# ungrounded operand makes admissibility REFUSE rather than guess. (no chains)
|
||||
comp = comprehend_quantitative(
|
||||
"Nia has 9 more beads than Omar. Nia has 15 beads. How many beads does Nia have?"
|
||||
)
|
||||
assert isinstance(comp, Refusal)
|
||||
assert comp.reason == "admissibility_refused"
|
||||
|
||||
|
||||
def test_multiple_inverse_bases_refuses() -> None:
|
||||
# Two known subjects each pin the SAME unknown base -> an over-determined inverse, not a
|
||||
# single base. The reader REFUSES rather than bind from one and drop the other. Without
|
||||
# the len>1 guard this would emit a setup the oracle then chokes on; refusing is honest.
|
||||
comp = comprehend_quantitative(
|
||||
"Nia has 9 more beads than Omar. Pam has 5 more beads than Omar. "
|
||||
"Nia has 15 beads. Pam has 11 beads. How many beads does Omar have?"
|
||||
)
|
||||
assert isinstance(comp, Refusal)
|
||||
assert comp.reason == "multiple_inverse_bases"
|
||||
|
||||
|
||||
def test_inverse_over_times_as_many_refuses() -> None:
|
||||
# The inverse frame is add/subtract ONLY. A times-as-many with an ungrounded ref is
|
||||
# never reverse-solved: the ref stays unit-unbound and admissibility REFUSES.
|
||||
comp = comprehend_quantitative(
|
||||
"Nia has twice as many beads as Omar. Nia has 14 beads. How many beads does Omar have?"
|
||||
)
|
||||
assert isinstance(comp, Refusal)
|
||||
assert comp.reason == "admissibility_refused"
|
||||
|
|
|
|||
|
|
@ -172,8 +172,11 @@ def test_r1_comparative_supported_rest_refused_wrong_zero() -> None:
|
|||
# the existing sum_of; no new arithmetic or relation kind.
|
||||
assert by_id["r1-03-more-total"] == "correct"
|
||||
assert by_id["r1-04-fewer-total"] == "correct"
|
||||
assert r["setup_correct"] == 6
|
||||
assert r["setup_refused"] == 4
|
||||
# Inverse frame (PR-7b): the base of a more_than whose subject is a known fact and
|
||||
# whose referent is the query target — "Nia has 9 more than Omar. Nia has 15. -> Omar".
|
||||
assert by_id["r1-07-inverse"] == "correct"
|
||||
assert r["setup_correct"] == 7
|
||||
assert r["setup_refused"] == 3
|
||||
# No detail is ever WRONG, and every non-correct one is a typed refusal.
|
||||
for d in r["details"]:
|
||||
assert d["outcome"] in ("correct", "refused")
|
||||
|
|
@ -218,8 +221,8 @@ def test_r1_answer_lane_scores_only_setup_correct_fixtures() -> None:
|
|||
assert r["setup_wrong"] == 0
|
||||
assert r["wrong"] == 0
|
||||
assert r["gold_error"] == 0
|
||||
assert r["correct"] == 6
|
||||
assert r["refused"] == 4
|
||||
assert r["correct"] == 7
|
||||
assert r["refused"] == 3
|
||||
by_id = {d["id"]: d for d in r["details"]}
|
||||
assert by_id["r1-01-twice"] == {"id": "r1-01-twice", "outcome": "correct", "answer": 12}
|
||||
assert by_id["r1-02-half"] == {"id": "r1-02-half", "outcome": "correct", "answer": 4}
|
||||
|
|
@ -229,9 +232,11 @@ def test_r1_answer_lane_scores_only_setup_correct_fixtures() -> None:
|
|||
# Aggregate-query slice: additive totals via "altogether" / "in total".
|
||||
assert by_id["r1-03-more-total"] == {"id": "r1-03-more-total", "outcome": "correct", "answer": 25}
|
||||
assert by_id["r1-04-fewer-total"] == {"id": "r1-04-fewer-total", "outcome": "correct", "answer": 34}
|
||||
# Inverse frame (PR-7b): the reverse-solved base — omar = nia(15) - 9 = 6.
|
||||
assert by_id["r1-07-inverse"] == {"id": "r1-07-inverse", "outcome": "correct", "answer": 6}
|
||||
_supported = {
|
||||
"r1-01-twice", "r1-02-half", "r1-05-chain", "r1-06-subtotal-reused",
|
||||
"r1-03-more-total", "r1-04-fewer-total",
|
||||
"r1-03-more-total", "r1-04-fewer-total", "r1-07-inverse",
|
||||
}
|
||||
for fixture_id, detail in by_id.items():
|
||||
if fixture_id not in _supported:
|
||||
|
|
@ -310,9 +315,9 @@ def test_oracle_divide_by_one_is_identity() -> None:
|
|||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# PR-7a — narrow reverse-solve oracle contract (the base of one more/fewer_than).
|
||||
# Pins the EXACT semantics before the reader learns the inverse frame (PR-7b). The
|
||||
# reader is unchanged here: r1-07 still refuses; these exercise the oracle directly.
|
||||
# Each refusal is meaningful-fail — drop its guardrail and the case computes a value.
|
||||
# Pins the EXACT semantics the reader's inverse frame (PR-7b) relies on; these exercise
|
||||
# the oracle directly (independent of the reader). Each refusal is meaningful-fail — drop
|
||||
# its guardrail and the case computes a value.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
|
|
@ -343,7 +348,7 @@ def test_r1_07_gold_relations_reverse_solve_to_six() -> None:
|
|||
from evals.setup_oracle.runner import _load_r1_gold
|
||||
|
||||
fx = next(f for f in run_r1()["details"] if f["id"] == "r1-07-inverse")
|
||||
assert fx["outcome"] == "refused" # reader still refuses in PR-7a (contract only)
|
||||
assert fx["outcome"] == "correct" # reader now reads the inverse frame (PR-7b)
|
||||
gold = next(g for g in _load_r1_gold() if g["id"] == "r1-07-inverse")
|
||||
assert oracle_answer(gold["relations"], gold["query"]) == gold["gold"] == 6
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue