a733fc5737
Recovers the multi-word-NP cases the reader previously refused, by adopting ONE
principled canonicalization contract (evals/comprehension/CANONICALIZATION.md) that
the reader AND the gold lanes both follow — so a committed answer can only match
gold or refuse, never silently mean something else.
Contract: a noun-phrase slot -> tokens lowercased, joined with "_"; a plural class
slot singularizes its head first ("metal objects"->"metal_object",
"North station"->"north_station", "Level one"->"level_one"). JOIN is chosen over
head-word-only ("metal objects"->"metal") because head-word-only is
information-destroying — it collapses "metal objects" and "metal tools" into one
false identity, itself a wrong=0 hazard.
reader.py: slot-based templates chunk multi-token NPs (_chunk / _chunk_class
replace the single-token _one / _one_class). Reserved-function-word guard fires only
INSIDE a multi-token slot (a lone "A" item is content, not the article). Still
parse-or-refuse: reserved-word leaks ("Compare beta with beta in the same order"),
non-pluralizable class heads (adjectival "trained"), and the ambiguous adjacent
two-NP subset query ("Are all <Xs> <Ys>?") all REFUSE.
gold (the contract update, logic-preserving — only term NAMES change):
- sy-v1-0008: metal/soft -> metal_object/soft_object (was head-word-only)
- to-v1-0005: red -> red_rank (was head-word-only)
- to-v1-0004: prose made internally consistent ("is after", "north station") +
north -> north_station (original prose used "North station" in the fact but
"north" in the query — a latent inconsistency)
- to-v1-0007: already conformed (level_one…), no change
Gold-only integrity runners stay 8/8 both lanes (structure+query+gold consistent).
Scores: set_membership 8/8, syllogism 6/8->7/8, total_ordering 4/8->7/8, all
wrong=0. Capability index re-frozen: score 0.814356 -> 0.917231, breadth 6,
wrong_total 0, digest 13d7db6c…
Tests: reader chunking + refusal tests; multi-word generative round-trip added to
the wrong=0 property suite (verified to BITE under a head-word-only mutation —
collapsed ids produce a wrong verdict the test catches); pinned counts updated.
100 comprehension/capability targeted + 87 smoke green.
8 lines
4 KiB
JSON
8 lines
4 KiB
JSON
{"id":"sy-v1-0001","seed":20260605,"text":"All mammals are animals. All whales are mammals. Therefore all whales are animals.","structure":{"terms":["whale","mammal","animal"],"domain_size":3,"premises":[{"form":"A","subject":"mammal","predicate":"animal"},{"form":"A","subject":"whale","predicate":"mammal"}]},"query":{"kind":"validity","conclusion":{"form":"A","subject":"whale","predicate":"animal"}},"gold":{"valid":true,"conclusion":{"form":"A","subject":"whale","predicate":"animal"}},"class":"barbara_valid"}
|
|
{"id":"sy-v1-0002","seed":20260605,"text":"No reptiles are mammals. All snakes are reptiles. Therefore no snakes are mammals.","structure":{"terms":["snake","reptile","mammal"],"domain_size":3,"premises":[{"form":"E","subject":"reptile","predicate":"mammal"},{"form":"A","subject":"snake","predicate":"reptile"}]},"query":{"kind":"validity","conclusion":{"form":"E","subject":"snake","predicate":"mammal"}},"gold":{"valid":true,"conclusion":{"form":"E","subject":"snake","predicate":"mammal"}},"class":"celarent_valid"}
|
|
{"id":"sy-v1-0003","seed":20260605,"text":"All poets are artists. Some students are poets. Therefore some students are artists.","structure":{"terms":["student","poet","artist"],"domain_size":3,"premises":[{"form":"A","subject":"poet","predicate":"artist"},{"form":"I","subject":"student","predicate":"poet"}]},"query":{"kind":"validity","conclusion":{"form":"I","subject":"student","predicate":"artist"}},"gold":{"valid":true,"conclusion":{"form":"I","subject":"student","predicate":"artist"}},"class":"darii_valid"}
|
|
{"id":"sy-v1-0004","seed":20260605,"text":"No mammals are reptiles. Some pets are mammals. Therefore some pets are not reptiles.","structure":{"terms":["pet","mammal","reptile"],"domain_size":3,"premises":[{"form":"E","subject":"mammal","predicate":"reptile"},{"form":"I","subject":"pet","predicate":"mammal"}]},"query":{"kind":"validity","conclusion":{"form":"O","subject":"pet","predicate":"reptile"}},"gold":{"valid":true,"conclusion":{"form":"O","subject":"pet","predicate":"reptile"}},"class":"ferio_valid"}
|
|
{"id":"sy-v1-0005","seed":20260605,"text":"All cats are mammals. All dogs are mammals. Therefore all dogs are cats.","structure":{"terms":["dog","cat","mammal"],"domain_size":3,"premises":[{"form":"A","subject":"cat","predicate":"mammal"},{"form":"A","subject":"dog","predicate":"mammal"}]},"query":{"kind":"validity","conclusion":{"form":"A","subject":"dog","predicate":"cat"}},"gold":{"valid":false,"conclusion":null},"class":"invalid_undistributed_middle"}
|
|
{"id":"sy-v1-0006","seed":20260605,"text":"All pilots are people. All pilots are trained. Therefore some trained people exist.","structure":{"terms":["pilot","person","trained"],"domain_size":3,"premises":[{"form":"A","subject":"pilot","predicate":"person"},{"form":"A","subject":"pilot","predicate":"trained"}]},"query":{"kind":"validity","conclusion":{"form":"I","subject":"trained","predicate":"person"}},"gold":{"valid":false,"conclusion":null},"class":"invalid_existential_import"}
|
|
{"id":"sy-v1-0007","seed":20260605,"text":"Some musicians are teachers. All teachers are workers. Therefore some musicians are workers.","structure":{"terms":["musician","teacher","worker"],"domain_size":3,"premises":[{"form":"I","subject":"musician","predicate":"teacher"},{"form":"A","subject":"teacher","predicate":"worker"}]},"query":{"kind":"validity","conclusion":{"form":"I","subject":"musician","predicate":"worker"}},"gold":{"valid":true,"conclusion":{"form":"I","subject":"musician","predicate":"worker"}},"class":"datisi_valid"}
|
|
{"id":"sy-v1-0008","seed":20260605,"text":"No metal objects are soft objects. Some tools are metal objects. Therefore some tools are not soft objects.","structure":{"terms":["metal_object","soft_object","tool"],"domain_size":3,"premises":[{"form":"E","subject":"metal_object","predicate":"soft_object"},{"form":"I","subject":"tool","predicate":"metal_object"}]},"query":{"kind":"validity","conclusion":{"form":"O","subject":"tool","predicate":"soft_object"}},"gold":{"valid":true,"conclusion":{"form":"O","subject":"tool","predicate":"soft_object"}},"class":"ferio_valid"}
|