feat(evals): the reader/writer construction overlap is 6 — and the reader fabricates on 22 more #138

Merged

core-labs merged 2 commits from feat/construction-inventory into main

2026-07-28 01:10:41 +00:00

core-labs commented

2026-07-27 21:20:34 +00:00

Owner

Phase 5 item 1 of docs/plans/grammar-unification-2026-07-26.md. §6's RESULT asked for the overlap to be sized "by measuring the reader's construction set against the writer's rather than by growing corpora blindly." This does that.

What it measures

New lane evals/construction_inventory. It sweeps the writer's whole parameter space — every RhetoricalMove × IntentTag × predicate × quantifier × tense × aspect, both public entry points, 32292 cells — quotients it by the reader's own function-word skeleton, and comprehends each construction under three vocabularies.

The quotient is read off reader._RESERVED, so it cannot be gerrymandered to flatter a number and it widens automatically when the reader learns a word.


writer constructions	1739
reader constructions	19 (mint-site AST-guarded)
overlap, faithful	6
accepts but mis-reads	22
refuses	1711
vocabulary-dependent	0

The overlap is the same six from both directions and is entirely set-theoretic: X is a Y, the X is a Y, and the four categoricals. The writer cannot express one ordering, propositional or interrogative construction; the reader has no template for the bare transitive clause that is the writer's whole default output. Two reader constructions need a ?, which no writer template contains — unreachable by shape, so corpus work cannot close that gap.

This corrects §6, using a metric that was already on the page

Faithfulness, not acceptance, is the criterion — and that reverses two claims I wrote into §6's RESULT:

g_read_rate is 1/293 but g_args_rate is 0.0. The one surface that "reads" — all molecules are defined as compounds — is comprehended as subset(molecule, defined_as_compound). The reader chunked the writer's verb phrase into a class name. It accepted; it did not comprehend.
"one construction wide" was wrong both ways: zero on that corpus, six over the writer's actual output space. A corpus cannot report an inventory's size.

The lane's contract had separated read from args for exactly this reason; the RESULT put its weight on the wrong one. Kept in the doc as a correction rather than edited away, per that block's own convention. evals/grammar_roundtrip/contract.md now carries a standing warning never to quote g_read_rate without g_args_rate.

Three findings that re-order Phase 5 item 1

1. Growing the inventory is not the first move — the reader fabricates. On 22 constructions it neither reads nor refuses; it invents.

leak	example	minted
determiner glued to entity	`every dog is a mammal`	`member(every_dog, mammal)`
verb phrase chunked as class	`all dogs are defined as mammals`	`subset(dog, defined_as_mammal)`
connective promoted to proposition	`furthermore, all dogs are mammals`	`asserted(furthermore)`

The first and third are ordinary user English, not writer artefacts. One root cause: _RESERVED lacks the function words the writer emits, and _parse_propositional accepts any single token as a fact. This breaks the reader's own stated contract — "a fabricated reading is the only failure."

2. It reaches served output.

"if p then q. p. therefore q."               -> Given: p implies q; p. ...
"furthermore, if p then q. p. therefore q."  -> Given: furthermore; p implies q; p. ...

CORE recites a premise the user never stated. chat/runtime.py additionally realizes declarative turns into the held self, so a fabricated atom is vault-writable. Widening the inventory before closing this would widen the fabrication surface with it.

3. ADR-0265's defect class survives inside ADR-0265's own designated owner of clause grammar. The four aspect arms of _inflect_predicate bind negated to a wildcard and never read it:

aspect=perfective    affirm: dog has been defined as mammal
                     denied: dog has been defined as mammal   <-- identical

10530 of 16146 parameter points. Not reachable today — no producer sets aspect on the serving path — so a loaded gun, not a casualty; the distinction ADR-0265 got wrong once already, so it is stated rather than assumed. It survived because ADR-0265's invariant is structural (is negated threaded to the call?) and cannot see an arm that receives the flag and ignores it. A behavioural sweep catches what a threading invariant cannot.

Deliberately not applied here

The two fabrication fixes are written already — as the mutations that turn the defect pins red (every/each into _RESERVED; discourse connectives stripped in _split_clauses). They are not in this PR: they change what CORE comprehends from user input, which is a serving change on the truth path. Authorization-gated, ADR first, on the ADR-0261 §5.1 refuse-don't-drop precedent.

This PR changes zero served bytes. It is measurement plus pins.

Guards, so the tables cannot rot

reader constructions pinned to an AST count of the reader's 10 mint sites — a new template goes red until it is entered in the table
declared tense/aspect axes pinned to _inflect_predicate's match arms
committed corpus pinned to the fillers actually in use
three vocabularies (regular, -f/-ves, suppletive) so a morphology gap cannot masquerade as a construction gap — currently 0 filler-dependence

[Verification]

In-worktree, canonical CPython 3.12.13, uv sync --locked:

deductive  517  (was 503, +14 — the count MOVED, registration confirmed)
smoke      641  (unchanged; the file is registered in `deductive`)
lane SHAs  11/11 match, no pin edited
pyright    0 errors on all new files
mutations  13/13 observed RED, including both fabrication fixes

The suite count was checked deliberately: the first registration attempt silently landed in a different tuple than intended, and only the count delta caught it.

🤖 Generated with Claude Code

Phase 5 item 1 of `docs/plans/grammar-unification-2026-07-26.md`. §6's RESULT asked for the overlap to be sized **"by measuring the reader's construction set against the writer's rather than by growing corpora blindly."** This does that. ## What it measures New lane `evals/construction_inventory`. It sweeps the writer's whole parameter space — every `RhetoricalMove` × `IntentTag` × predicate × quantifier × tense × aspect, both public entry points, **32292 cells** — quotients it by the reader's *own* function-word skeleton, and comprehends each construction under three vocabularies. The quotient is read off `reader._RESERVED`, so it cannot be gerrymandered to flatter a number and it widens automatically when the reader learns a word. | | | |---|---| | writer constructions | **1739** | | reader constructions | **19** (mint-site AST-guarded) | | **overlap, faithful** | **6** | | accepts but **mis-reads** | **22** | | refuses | 1711 | | vocabulary-dependent | **0** | The overlap is the same six from both directions and is **entirely set-theoretic**: `X is a Y`, `the X is a Y`, and the four categoricals. The writer cannot express one ordering, propositional or interrogative construction; the reader has no template for the bare transitive clause that is the writer's whole default output. Two reader constructions need a `?`, which no writer template contains — unreachable **by shape**, so corpus work cannot close that gap. ## This corrects §6, using a metric that was already on the page **Faithfulness, not acceptance, is the criterion** — and that reverses two claims I wrote into §6's RESULT: - `g_read_rate` is `1/293` but **`g_args_rate` is `0.0`**. The one surface that "reads" — `all molecules are defined as compounds` — is comprehended as `subset(molecule, defined_as_compound)`. The reader chunked the writer's verb phrase into a class name. It **accepted; it did not comprehend.** - **"one construction wide" was wrong both ways**: zero on that corpus, six over the writer's actual output space. A corpus cannot report an inventory's size. The lane's contract had separated `read` from `args` for exactly this reason; the RESULT put its weight on the wrong one. Kept in the doc as a correction rather than edited away, per that block's own convention. `evals/grammar_roundtrip/contract.md` now carries a standing warning never to quote `g_read_rate` without `g_args_rate`. ## Three findings that re-order Phase 5 item 1 **1. Growing the inventory is not the first move — the reader fabricates.** On 22 constructions it neither reads nor refuses; it invents. | leak | example | minted | |---|---|---| | determiner glued to entity | `every dog is a mammal` | `member(every_dog, mammal)` | | verb phrase chunked as class | `all dogs are defined as mammals` | `subset(dog, defined_as_mammal)` | | connective promoted to proposition | `furthermore, all dogs are mammals` | `asserted(furthermore)` | The first and third are **ordinary user English**, not writer artefacts. One root cause: `_RESERVED` lacks the function words the writer emits, and `_parse_propositional` accepts any single token as a fact. This breaks the reader's own stated contract — *"a fabricated reading is the only failure."* **2. It reaches served output.** ``` "if p then q. p. therefore q." -> Given: p implies q; p. ... "furthermore, if p then q. p. therefore q." -> Given: furthermore; p implies q; p. ... ``` CORE recites a premise the user never stated. `chat/runtime.py` additionally *realizes* declarative turns into the held self, so a fabricated atom is vault-writable. **Widening the inventory before closing this would widen the fabrication surface with it.** **3. ADR-0265's defect class survives inside ADR-0265's own designated owner of clause grammar.** The four aspect arms of `_inflect_predicate` bind `negated` to a wildcard and never read it: ``` aspect=perfective affirm: dog has been defined as mammal denied: dog has been defined as mammal <-- identical ``` 10530 of 16146 parameter points. **Not reachable today** — no producer sets `aspect` on the serving path — so a loaded gun, not a casualty; the distinction ADR-0265 got wrong once already, so it is stated rather than assumed. It survived because ADR-0265's invariant is *structural* (is `negated` threaded to the call?) and cannot see an arm that receives the flag and ignores it. **A behavioural sweep catches what a threading invariant cannot.** ## Deliberately not applied here The two fabrication fixes are written already — as the mutations that turn the defect pins red (`every`/`each` into `_RESERVED`; discourse connectives stripped in `_split_clauses`). They are **not** in this PR: they change what CORE *comprehends from user input*, which is a serving change on the truth path. **Authorization-gated, ADR first**, on the ADR-0261 §5.1 refuse-don't-drop precedent. This PR changes **zero served bytes**. It is measurement plus pins. ## Guards, so the tables cannot rot - reader constructions pinned to an **AST count of the reader's 10 mint sites** — a new template goes red until it is entered in the table - declared tense/aspect axes pinned to `_inflect_predicate`'s **match arms** - committed corpus pinned to the fillers actually in use - three vocabularies (regular, `-f/-ves`, suppletive) so a morphology gap cannot masquerade as a construction gap — currently **0** filler-dependence ## [Verification] In-worktree, canonical CPython 3.12.13, `uv sync --locked`: ``` deductive 517 (was 503, +14 — the count MOVED, registration confirmed) smoke 641 (unchanged; the file is registered in `deductive`) lane SHAs 11/11 match, no pin edited pyright 0 errors on all new files mutations 13/13 observed RED, including both fabrication fixes ``` The suite count was checked deliberately: the first registration attempt silently landed in a different tuple than intended, and only the count delta caught it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

core-labs added 1 commit 2026-07-27 21:20:38 +00:00

feat(evals): measure the reader/writer construction inventories against each other c69f99485d

Phase 5 item 1 asked for the overlap to be sized "by measuring the reader's
construction set against the writer's rather than by growing corpora blindly".
New lane `evals/construction_inventory` does that: it sweeps the writer's whole
parameter space (every RhetoricalMove x IntentTag x predicate x quantifier x
tense x aspect, both public entry points, 32292 cells), quotients it by the
reader's OWN function-word skeleton, and comprehends each construction under
three vocabularies.

  writer constructions            1739
  reader constructions              19   (mint-site AST-guarded)
  overlap, faithful                  6
  accepts but MIS-READS             22
  refuses                         1711
  vocabulary-dependent               0

Faithfulness, not acceptance, is the criterion — and that reverses two claims
in the plan's §6 RESULT, using a metric that was already on the page:

* g_read_rate is 1/293 but **g_args_rate is 0.0**. The one surface that "reads",
  `all molecules are defined as compounds`, is comprehended as
  `subset(molecule, defined_as_compound)` — the reader chunked the writer's verb
  phrase into a class name. It accepted; it did not comprehend.
* "one construction wide" was wrong both ways: zero on that corpus, six over the
  writer's actual output space. A corpus cannot report an inventory's size.

Three findings re-order Phase 5 item 1:

1. The reader FABRICATES on 22 constructions — neither reads nor refuses.
   `every dog is a mammal` -> member(every_dog, mammal);
   `furthermore, all dogs are mammals` -> asserted(furthermore).
   Both are ordinary user English, not writer artefacts. Root cause: _RESERVED
   lacks the function words the writer emits, and _parse_propositional accepts
   any single token as a fact.
2. It reaches SERVED output. deduction_surface recites
   `Given: furthermore; p implies q; p.` — a premise the user never stated — and
   chat/runtime.py realizes declarative turns into the held self, so a
   fabricated atom is vault-writable. Widening the inventory first would widen
   the fabrication surface with it.
3. ADR-0265's defect class survives inside ADR-0265's designated owner of clause
   grammar: the four aspect arms of _inflect_predicate bind `negated` to a
   wildcard and never read it, so `dog has been defined as mammal` serves both
   the assertion and the denial (10530/16146 points). Unreachable today — no
   producer sets aspect — so a loaded gun, not a casualty. It survived because
   ADR-0265's invariant is structural (is `negated` threaded?) and cannot see an
   arm that receives the flag and ignores it. A behavioural sweep can.

The two fixes are written already, as the mutations that turn the defect pins
red. They are NOT applied here: they change what CORE comprehends from user
input, which is a serving change on the truth path — authorization-gated, ADR
first, on the ADR-0261 §5.1 refuse-don't-drop precedent.

Guards, so the tables cannot rot: reader constructions are pinned to an AST
count of the reader's 10 mint sites; the declared tense/aspect axes are pinned
to _inflect_predicate's match arms; the committed corpus is pinned to the
fillers in use. 14 pins, 13 mutations observed RED.

[Verification]: in-worktree, CPython 3.12.13, uv sync --locked.
  deductive 517 (was 503, +14 — count moved, registration confirmed)
  smoke     641 (unchanged; the file is registered in `deductive`)
  lane SHAs 11/11 match, no pin edited
  pyright   0 errors on all new files
  mutations 13/13 red, including both fabrication fixes