core/evals/construction_inventory/skeleton.py
Shay c69f99485d feat(evals): measure the reader/writer construction inventories against each other
Phase 5 item 1 asked for the overlap to be sized "by measuring the reader's
construction set against the writer's rather than by growing corpora blindly".
New lane `evals/construction_inventory` does that: it sweeps the writer's whole
parameter space (every RhetoricalMove x IntentTag x predicate x quantifier x
tense x aspect, both public entry points, 32292 cells), quotients it by the
reader's OWN function-word skeleton, and comprehends each construction under
three vocabularies.

  writer constructions            1739
  reader constructions              19   (mint-site AST-guarded)
  overlap, faithful                  6
  accepts but MIS-READS             22
  refuses                         1711
  vocabulary-dependent               0

Faithfulness, not acceptance, is the criterion — and that reverses two claims
in the plan's §6 RESULT, using a metric that was already on the page:

* g_read_rate is 1/293 but **g_args_rate is 0.0**. The one surface that "reads",
  `all molecules are defined as compounds`, is comprehended as
  `subset(molecule, defined_as_compound)` — the reader chunked the writer's verb
  phrase into a class name. It accepted; it did not comprehend.
* "one construction wide" was wrong both ways: zero on that corpus, six over the
  writer's actual output space. A corpus cannot report an inventory's size.

Three findings re-order Phase 5 item 1:

1. The reader FABRICATES on 22 constructions — neither reads nor refuses.
   `every dog is a mammal` -> member(every_dog, mammal);
   `furthermore, all dogs are mammals` -> asserted(furthermore).
   Both are ordinary user English, not writer artefacts. Root cause: _RESERVED
   lacks the function words the writer emits, and _parse_propositional accepts
   any single token as a fact.
2. It reaches SERVED output. deduction_surface recites
   `Given: furthermore; p implies q; p.` — a premise the user never stated — and
   chat/runtime.py realizes declarative turns into the held self, so a
   fabricated atom is vault-writable. Widening the inventory first would widen
   the fabrication surface with it.
3. ADR-0265's defect class survives inside ADR-0265's designated owner of clause
   grammar: the four aspect arms of _inflect_predicate bind `negated` to a
   wildcard and never read it, so `dog has been defined as mammal` serves both
   the assertion and the denial (10530/16146 points). Unreachable today — no
   producer sets aspect — so a loaded gun, not a casualty. It survived because
   ADR-0265's invariant is structural (is `negated` threaded?) and cannot see an
   arm that receives the flag and ignores it. A behavioural sweep can.

The two fixes are written already, as the mutations that turn the defect pins
red. They are NOT applied here: they change what CORE comprehends from user
input, which is a serving change on the truth path — authorization-gated, ADR
first, on the ADR-0261 §5.1 refuse-don't-drop precedent.

Guards, so the tables cannot rot: reader constructions are pinned to an AST
count of the reader's 10 mint sites; the declared tense/aspect axes are pinned
to _inflect_predicate's match arms; the committed corpus is pinned to the
fillers in use. 14 pins, 13 mutations observed RED.

[Verification]: in-worktree, CPython 3.12.13, uv sync --locked.
  deductive 517 (was 503, +14 — count moved, registration confirmed)
  smoke     641 (unchanged; the file is registered in `deductive`)
  lane SHAs 11/11 match, no pin edited
  pyright   0 errors on all new files
  mutations 13/13 red, including both fabrication fixes
2026-07-27 14:11:07 -07:00

61 lines
2.8 KiB
Python

"""The construction quotient: a surface reduced to what the reader parses by.
The reader is documented as reading "STRUCTURE symbolically from the token
sequence via domain-agnostic templates keyed on FUNCTION WORDS + ORDER". Take
that literally and it defines an equivalence relation on surfaces: replace every
token the reader does *not* key on with a placeholder, and two surfaces with the
same result are — by the reader's own design — indistinguishable to its parser.
That relation is what makes "how many constructions?" a countable question
instead of a judgement call. The quotient is not chosen by this module; it is
read off ``reader._RESERVED``, so it cannot be gerrymandered to flatter a
number, and it widens automatically when the reader learns a new function word.
**The one place the abstraction leaks**, stated here because a silent leak in a
measurement instrument is worse than the defect it measures: ``_chunk_class``
refuses on an unrecognized plural, which depends on the *token*, not on whether
the token is reserved. So the skeleton determines the reader's verdict **up to
morphology**. The lane handles this by sweeping several lexical fillers per
construction and only counting a construction as shared when every filler
agrees — see ``runner.py``. Morphology refusals are then visible as
filler-dependent disagreement rather than being silently folded into the
construction count.
"""
from __future__ import annotations
from generate.meaning_graph.reader import _RESERVED
#: Function words the reader keys on that ``_RESERVED`` does not list.
#:
#: ``_parse_propositional`` dispatches on ``if`` and ``then``, but neither is in
#: ``_RESERVED`` — so ``_chunk`` will happily swallow them into a noun phrase.
#: That asymmetry is a reader defect (see contract.md, "What this lane found"),
#: not an artefact of this module; it is recorded here so the quotient reflects
#: the tokens the parser *actually* branches on rather than the ones it declares.
_UNDECLARED_KEYWORDS: frozenset[str] = frozenset({"if", "then"})
#: The reader's full function-word vocabulary, derived from source.
READER_FUNCTION_WORDS: frozenset[str] = frozenset(_RESERVED) | _UNDECLARED_KEYWORDS
#: Stands in for any token the reader treats as content.
CONTENT = "*"
_PUNCT = ".,;:?!"
def skeleton(surface: str) -> str:
"""Reduce *surface* to its reader-visible construction.
Content tokens collapse to ``*``; function words and attached punctuation
survive, because both change how the reader splits and dispatches.
"""
out: list[str] = []
for token in surface.lower().split():
core = token.strip(_PUNCT)
trailing = token[len(core):] if core and token.startswith(core) else ""
out.append((core if core in READER_FUNCTION_WORDS else CONTENT) + trailing)
return " ".join(out)
__all__ = ("CONTENT", "READER_FUNCTION_WORDS", "skeleton")