Shay a733fc5737 feat(comprehend): multi-word NP chunking under a canonicalization contract

Recovers the multi-word-NP cases the reader previously refused, by adopting ONE
principled canonicalization contract (evals/comprehension/CANONICALIZATION.md) that
the reader AND the gold lanes both follow — so a committed answer can only match
gold or refuse, never silently mean something else.

Contract: a noun-phrase slot -> tokens lowercased, joined with "_"; a plural class
slot singularizes its head first ("metal objects"->"metal_object",
"North station"->"north_station", "Level one"->"level_one"). JOIN is chosen over
head-word-only ("metal objects"->"metal") because head-word-only is
information-destroying — it collapses "metal objects" and "metal tools" into one
false identity, itself a wrong=0 hazard.

reader.py: slot-based templates chunk multi-token NPs (_chunk / _chunk_class
replace the single-token _one / _one_class). Reserved-function-word guard fires only
INSIDE a multi-token slot (a lone "A" item is content, not the article). Still
parse-or-refuse: reserved-word leaks ("Compare beta with beta in the same order"),
non-pluralizable class heads (adjectival "trained"), and the ambiguous adjacent
two-NP subset query ("Are all <Xs> <Ys>?") all REFUSE.

gold (the contract update, logic-preserving — only term NAMES change):
  - sy-v1-0008: metal/soft -> metal_object/soft_object (was head-word-only)
  - to-v1-0005: red -> red_rank (was head-word-only)
  - to-v1-0004: prose made internally consistent ("is after", "north station") +
    north -> north_station (original prose used "North station" in the fact but
    "north" in the query — a latent inconsistency)
  - to-v1-0007: already conformed (level_one…), no change
Gold-only integrity runners stay 8/8 both lanes (structure+query+gold consistent).

Scores: set_membership 8/8, syllogism 6/8->7/8, total_ordering 4/8->7/8, all
wrong=0. Capability index re-frozen: score 0.814356 -> 0.917231, breadth 6,
wrong_total 0, digest 13d7db6c…

Tests: reader chunking + refusal tests; multi-word generative round-trip added to
the wrong=0 property suite (verified to BITE under a head-word-only mutation —
collapsed ids produce a wrong verdict the test catches); pinned counts updated.
100 comprehension/capability targeted + 87 smoke green.

2026-06-05 22:47:34 -07:00

2.8 KiB

Raw Permalink Blame History

Noun-phrase canonicalization contract (comprehension lanes)

The comprehension reader (generate/meaning_graph/reader.py) and the staged gold lanes (evals/{set_membership,syllogism,total_ordering}/v1) share one rule for turning a surface noun phrase into a term/entity id. This contract is what makes multi-word noun phrases wrong=0-safe: the reader and the gold canonicalize the same way, so a committed answer can only match gold or refuse — it can never silently mean something different.

The rule

A noun-phrase slot (the tokens a template isolates between its function words) becomes an id by:

lowercasing every token;
for a plural class slot (the subject/predicate of all/no/some … are …), singularizing the head (final) token — objects → object, people → person; item and individual slots are not singularized;
joining the resulting tokens with _.

surface	role	canonical id
`metal objects`	class	`metal_object`
`soft objects`	class	`soft_object`
`North station`	item	`north_station`
`Level one`	item	`level_one`
`Red rank`	item	`red_rank`
`red car`	individual	`red_car`
`squares`	class	`square`

Why join, not head-word-only

An earlier (implicit) gold used head-word-only in places (metal objects → metal, North station → north). That is information-destroying and an active wrong=0 hazard: metal objects and metal tools both collapse to metal, creating a false identity the oracle would then reason over. Joining preserves the distinction, so distinct phrases stay distinct ids.

Refusal cases (still parse-or-refuse)

The reader refuses rather than guess when:

a multi-token slot contains a reserved function word (article, comparator, quantifier, is/are/than/with/not/the/from/to/order/in/of/on/at/by/and/or) — e.g. Compare beta with beta in the same order → the right slot beta in the same order leaks in/the/order, so refuse (never beta_in_the_same_order). A single-token slot is exempt: a literal item named A is content even though a is also the article.
a plural class head is not a recognizable plural (e.g. the adjectival predicate trained in All pilots are trained) — cannot singularize → refuse.
the two class NPs in Are all <Xs> <Ys>? are adjacent with no separating function word and either is multi-word — the boundary is unknown → refuse.

Changing this contract

This contract is load-bearing for wrong=0. If it changes, the reader and every affected gold case must change together in the same PR, and the tests/test_comprehension_wrong_zero_property.py round-trip (which renders prose from canonical structures and checks the reader reproduces the oracle's verdict) must stay green.

2.8 KiB Raw Permalink Blame History