a733fc5737
Recovers the multi-word-NP cases the reader previously refused, by adopting ONE
principled canonicalization contract (evals/comprehension/CANONICALIZATION.md) that
the reader AND the gold lanes both follow — so a committed answer can only match
gold or refuse, never silently mean something else.
Contract: a noun-phrase slot -> tokens lowercased, joined with "_"; a plural class
slot singularizes its head first ("metal objects"->"metal_object",
"North station"->"north_station", "Level one"->"level_one"). JOIN is chosen over
head-word-only ("metal objects"->"metal") because head-word-only is
information-destroying — it collapses "metal objects" and "metal tools" into one
false identity, itself a wrong=0 hazard.
reader.py: slot-based templates chunk multi-token NPs (_chunk / _chunk_class
replace the single-token _one / _one_class). Reserved-function-word guard fires only
INSIDE a multi-token slot (a lone "A" item is content, not the article). Still
parse-or-refuse: reserved-word leaks ("Compare beta with beta in the same order"),
non-pluralizable class heads (adjectival "trained"), and the ambiguous adjacent
two-NP subset query ("Are all <Xs> <Ys>?") all REFUSE.
gold (the contract update, logic-preserving — only term NAMES change):
- sy-v1-0008: metal/soft -> metal_object/soft_object (was head-word-only)
- to-v1-0005: red -> red_rank (was head-word-only)
- to-v1-0004: prose made internally consistent ("is after", "north station") +
north -> north_station (original prose used "North station" in the fact but
"north" in the query — a latent inconsistency)
- to-v1-0007: already conformed (level_one…), no change
Gold-only integrity runners stay 8/8 both lanes (structure+query+gold consistent).
Scores: set_membership 8/8, syllogism 6/8->7/8, total_ordering 4/8->7/8, all
wrong=0. Capability index re-frozen: score 0.814356 -> 0.917231, breadth 6,
wrong_total 0, digest 13d7db6c…
Tests: reader chunking + refusal tests; multi-word generative round-trip added to
the wrong=0 property suite (verified to BITE under a head-word-only mutation —
collapsed ids produce a wrong verdict the test catches); pinned counts updated.
100 comprehension/capability targeted + 87 smoke green.
60 lines
2.8 KiB
Markdown
60 lines
2.8 KiB
Markdown
# Noun-phrase canonicalization contract (comprehension lanes)
|
|
|
|
The comprehension reader (`generate/meaning_graph/reader.py`) and the staged gold
|
|
lanes (`evals/{set_membership,syllogism,total_ordering}/v1`) share **one** rule for
|
|
turning a surface noun phrase into a term/entity id. This contract is what makes
|
|
multi-word noun phrases `wrong=0`-safe: the reader and the gold canonicalize the
|
|
*same* way, so a committed answer can only match gold or refuse — it can never
|
|
silently mean something different.
|
|
|
|
## The rule
|
|
|
|
A noun-phrase slot (the tokens a template isolates between its function words)
|
|
becomes an id by:
|
|
|
|
1. lowercasing every token;
|
|
2. for a **plural class slot** (the subject/predicate of `all/no/some … are …`),
|
|
singularizing the **head** (final) token — `objects → object`, `people → person`;
|
|
item and individual slots are **not** singularized;
|
|
3. joining the resulting tokens with `_`.
|
|
|
|
| surface | role | canonical id |
|
|
|---|---|---|
|
|
| `metal objects` | class | `metal_object` |
|
|
| `soft objects` | class | `soft_object` |
|
|
| `North station` | item | `north_station` |
|
|
| `Level one` | item | `level_one` |
|
|
| `Red rank` | item | `red_rank` |
|
|
| `red car` | individual | `red_car` |
|
|
| `squares` | class | `square` |
|
|
|
|
## Why join, not head-word-only
|
|
|
|
An earlier (implicit) gold used head-word-only in places (`metal objects → metal`,
|
|
`North station → north`). That is **information-destroying** and an active
|
|
`wrong=0` hazard: `metal objects` and `metal tools` both collapse to `metal`,
|
|
creating a *false identity* the oracle would then reason over. Joining preserves
|
|
the distinction, so distinct phrases stay distinct ids.
|
|
|
|
## Refusal cases (still parse-or-refuse)
|
|
|
|
The reader refuses rather than guess when:
|
|
|
|
- a multi-token slot contains a **reserved function word** (article, comparator,
|
|
quantifier, `is/are/than/with/not/the/from/to/order/in/of/on/at/by/and/or`) —
|
|
e.g. `Compare beta with beta in the same order` → the right slot
|
|
`beta in the same order` leaks `in/the/order`, so refuse (never
|
|
`beta_in_the_same_order`). A **single-token** slot is exempt: a literal item
|
|
named `A` is content even though `a` is also the article.
|
|
- a plural class head is not a recognizable plural (e.g. the adjectival predicate
|
|
`trained` in `All pilots are trained`) — cannot singularize → refuse.
|
|
- the two class NPs in `Are all <Xs> <Ys>?` are adjacent with no separating
|
|
function word and either is multi-word — the boundary is unknown → refuse.
|
|
|
|
## Changing this contract
|
|
|
|
This contract is load-bearing for `wrong=0`. If it changes, the reader **and**
|
|
every affected gold case must change together in the same PR, and the
|
|
`tests/test_comprehension_wrong_zero_property.py` round-trip (which renders prose
|
|
from canonical structures and checks the reader reproduces the oracle's verdict)
|
|
must stay green.
|