Shay c69f99485d feat(evals): measure the reader/writer construction inventories against each other

Phase 5 item 1 asked for the overlap to be sized "by measuring the reader's
construction set against the writer's rather than by growing corpora blindly".
New lane `evals/construction_inventory` does that: it sweeps the writer's whole
parameter space (every RhetoricalMove x IntentTag x predicate x quantifier x
tense x aspect, both public entry points, 32292 cells), quotients it by the
reader's OWN function-word skeleton, and comprehends each construction under
three vocabularies.

  writer constructions            1739
  reader constructions              19   (mint-site AST-guarded)
  overlap, faithful                  6
  accepts but MIS-READS             22
  refuses                         1711
  vocabulary-dependent               0

Faithfulness, not acceptance, is the criterion — and that reverses two claims
in the plan's §6 RESULT, using a metric that was already on the page:

* g_read_rate is 1/293 but **g_args_rate is 0.0**. The one surface that "reads",
  `all molecules are defined as compounds`, is comprehended as
  `subset(molecule, defined_as_compound)` — the reader chunked the writer's verb
  phrase into a class name. It accepted; it did not comprehend.
* "one construction wide" was wrong both ways: zero on that corpus, six over the
  writer's actual output space. A corpus cannot report an inventory's size.

Three findings re-order Phase 5 item 1:

1. The reader FABRICATES on 22 constructions — neither reads nor refuses.
   `every dog is a mammal` -> member(every_dog, mammal);
   `furthermore, all dogs are mammals` -> asserted(furthermore).
   Both are ordinary user English, not writer artefacts. Root cause: _RESERVED
   lacks the function words the writer emits, and _parse_propositional accepts
   any single token as a fact.
2. It reaches SERVED output. deduction_surface recites
   `Given: furthermore; p implies q; p.` — a premise the user never stated — and
   chat/runtime.py realizes declarative turns into the held self, so a
   fabricated atom is vault-writable. Widening the inventory first would widen
   the fabrication surface with it.
3. ADR-0265's defect class survives inside ADR-0265's designated owner of clause
   grammar: the four aspect arms of _inflect_predicate bind `negated` to a
   wildcard and never read it, so `dog has been defined as mammal` serves both
   the assertion and the denial (10530/16146 points). Unreachable today — no
   producer sets aspect — so a loaded gun, not a casualty. It survived because
   ADR-0265's invariant is structural (is `negated` threaded?) and cannot see an
   arm that receives the flag and ignores it. A behavioural sweep can.

The two fixes are written already, as the mutations that turn the defect pins
red. They are NOT applied here: they change what CORE comprehends from user
input, which is a serving change on the truth path — authorization-gated, ADR
first, on the ADR-0261 §5.1 refuse-don't-drop precedent.

Guards, so the tables cannot rot: reader constructions are pinned to an AST
count of the reader's 10 mint sites; the declared tense/aspect axes are pinned
to _inflect_predicate's match arms; the committed corpus is pinned to the
fillers in use. 14 pins, 13 mutations observed RED.

[Verification]: in-worktree, CPython 3.12.13, uv sync --locked.
  deductive 517 (was 503, +14 — count moved, registration confirmed)
  smoke     641 (unchanged; the file is registered in `deductive`)
  lane SHAs 11/11 match, no pin edited
  pyright   0 errors on all new files
  mutations 13/13 red, including both fabrication fixes

2026-07-27 14:11:07 -07:00

7.4 KiB

Raw Blame History

Grammar Round-Trip Eval Lane — Contract

Lane: grammar_roundtrip Version: v1 Created: 2026-07-26 Plan: docs/plans/grammar-unification-2026-07-26.md (Phase 1)

What this lane measures

Whether CORE can read what it writes and write what it reads — and, critically, whether it refuses word salad.

Why it exists

evals/deterministic_fluency reports 1.00 on all six of its predicates and still passes every one of these:

candidate	passes deterministic_fluency?
`"banana does the."`	yes
`"wet ground rains the is."`	yes
`"is is is is."`	yes

It checks terminal punctuation, presence of a verb-shaped token, and two anti-shape regexes. It cannot distinguish English from word salad, so its 100% carries no information about fluency.

Heuristic predicates will always have that failure mode, because grammaticality cannot be measured without a grammar. This lane therefore measures agreement between the two halves of CORE that already encode grammar — the reader and the writer — and requires the measurement to fail on salad. Round-trip is falsifiable with no judge, no embedding, and no gold aesthetic.

The two directions

direction	pipeline	what a failure means
G-round-trip	graph → `realize_target` → surface → `comprehend` → graph	CORE cannot read its own writing

realize_target is eval-only (Phase 4). core/cognition/pipeline.py calls realize_semantic, never realize_target, so g_read_rate is a measurement of the grammar library and not of served English. Run through the serving writer instead, the G-direction reads back nothing (0.0 vs 0.003413) — see tests/test_phase4_realizer_resolution.py. Any claim made from this metric must say which writer it is about.

| S-round-trip | surface → comprehend → graph → categorical renderer → surface | CORE cannot reproduce what it just understood |

Both are reported because they fail for different reasons and have different remedies.

Metrics

metric	definition
`g_write_rate`	fraction of graph cases the writer produced any surface for
`g_read_rate`	fraction of written surfaces the reader comprehended
`g_args_rate`	fraction of expected propositions whose argument pair was recovered
`g_predicates_rate`	fraction whose predicate name was recovered on the same arguments
`g_exact_rate`	fraction recovered exactly (predicate + arguments + polarity)
`s_read_rate`	fraction of positive surfaces comprehended
`s_renderable_rate`	fraction whose projection the categorical renderer can express at all
`s_surface_match_rate`	fraction that render back to the input surface
`reject_rate`	fraction of the negative corpus refused or reduced to zero propositions

g_args_rate and g_predicates_rate are reported separately on purpose. High argument agreement with low predicate agreement means the grammars align and only the vocabulary is split — a materially different remedy from both being low. Collapsing them into one boolean would hide the distinction that decides this arc's direction (plan §6).

Never quote g_read_rate without g_args_rate. They currently disagree: g_read_rate = 0.003413 with g_args_rate = 0.0. The single surface that "reads" is all molecules are defined as compounds, which the reader accepts as subset(molecule, defined_as_compound) — it chunked the writer's verb phrase into a class name. Acceptance is not comprehension, and a reader scored on whether it returned a graph rather than on whether the graph is the one it was given is rewarded for guessing. Plan §6's RESULT drew a conclusion from g_read_rate alone and had to be corrected; see evals/construction_inventory/contract.md, which measures the same question against the writer's whole output space instead of a 293-case corpus.

Corpora

corpus	source	size
positive graphs	committed `english_fluency_ood` + `grammatical_coverage` case files	293
positive surfaces	in-module, each verified comprehensible by probe	8
negative surfaces	hand-authored salad + deterministic token shuffles of the positives	16

The shuffles are load-bearing: they are lexically identical to positive cases — same vocabulary, same length, order destroyed. A lane that rejects hand-authored salad but accepts the shuffles is doing vocabulary checking, not grammar checking. Shuffling uses a fixed rotation rather than a PRNG so reject_rate is byte-reproducible.

Thresholds

metric	v1 requirement	rationale
`reject_rate`	1.00, always	the guarantee; regression is a hard failure
`g_read_rate`	recorded baseline, revised upward only	currently a measured defect
`s_surface_match_rate`	recorded baseline, revised upward only	currently a measured defect

v1 baseline — measured on `main` @ `9696443a`

graph_cases              293
g_write_rate             1.000
g_read_rate              0.000     <-- CORE reads 0% of what it writes
g_propositions_expected  370
g_args_rate              0.000
g_predicates_rate        0.000
g_exact_rate             0.000

surface_cases              8
s_read_rate              1.000
s_renderable_rate        0.625
s_surface_match_rate     0.000     <-- nothing renders back to its input

negative_cases            16
reject_rate              1.000     <-- the guarantee holds

Two of these are pins on known defects, not goals. They are expected to be revised upward by plan Phases 2B and 5, and must never be revised downward to accommodate a regression.

s_renderable_rate = 0.625 is not a defect: 3 of the 8 positive surfaces project to member / less predicates, which have no all X are Y categorical surface at all. Those are reported as unrenderable rather than as match failures, because "cannot write this" and "wrote this wrongly" need different fixes.

What this lane does NOT prove

Round-trip is necessary, not sufficient for fluency. It measures mutual intelligibility — both halves agreeing about a surface. Two halves can agree on an impoverished construction: english_fluency_ood accepts "river flows valley" as a correct surface, and round-trip would be perfectly happy with it.

So a high round-trip rate licenses "CORE means what it says", never "CORE writes well". The negative corpus is what prevents the first failure mode. Nothing in this lane prevents the second, and no metric here should ever be cited as evidence of prose quality.

Mutation guarantees

Every guarantee is paired with a test proving it can break (tests/test_grammar_roundtrip.py):

test_reject_rate_goes_red_when_the_reader_accepts_everything — an accept-everything reader must drive reject_rate to 0.0. Without this, reject_rate == 1.0 would be unfalsifiable, which is precisely the defect that makes the existing fluency lane decoration.
test_shuffled_negatives_reuse_positive_vocabulary — closes the vocabulary-checking escape hatch.
test_positive_surfaces_are_all_inside_the_reader_envelope — a refused positive would silently measure reader coverage instead of round-trip.
test_quantifier_map_matches_reader — the lane keeps a deliberate local copy of the reader's quantifier map so it never becomes a consumer of the thing it measures; this test fails loudly if the reader's map changes.

7.4 KiB Raw Blame History