core/evals/grammar_roundtrip
Shay 7d6760ae5b feat(evals): the fluency lanes report the writer that actually ships (Phase 4)
Phase 4 of the grammar-unification arc, resolved by option (b) of the plan.

The problem: 149 green fluency cases scored `realize_target`, and
`core/cognition/pipeline.py` never calls it -- it calls `realize_semantic`.
`english_fluency_ood` reported 117/117 + 39/39 for a function that does not
speak, for the whole life of the lane.

The plan framed this as "which realizer is better". Reading the source says it
is not a quality question. `render_semantic`'s signature is
`(intent, subject, predicate, obj, secondary, language, root)` -- no `negated`,
no `quantifier`, no `tense`, no `aspect` -- and `realize_semantic` never reads
them off the step. So the serving writer cannot express content the
ArticulationStep is carrying:

    negated=False -> 'Knowledge is defined as opinion.'
    negated=True  -> 'Knowledge is defined as opinion.'

It serves the AFFIRMATIVE of a negated proposition. That is the ADR-0261 §5.1
family, not a fluency defect. Pinned here as a defect; NOT fixed, because
fixing it changes served output and belongs to Shay.

Measured -- identical contract, all seven scored corpora
--------------------------------------------------------
    bucket                                n     realize_target   realize_semantic
    feature-bearing, single node        214          207               49
    no features, multi-node             100          100                3
    no features, single node (CONTROL)   33           33               33
    total                               347          340               85

The control is what makes the rest mean anything. Every corpus hardcodes
IntentTag.UNKNOWN, so "the serving writer scores badly" could have been an
artifact of never giving it a real intent. On the 33 cases carrying nothing it
cannot express, the two are IDENTICAL -- so the gap on the other 314 is the
dropped features and the clause joining, not the intent.

Delivered
---------
- `grammatical_coverage/runner.py` reports `serving_accuracy` beside
  `accuracy`; `english_fluency_ood` delegates to that run_lane and gains it
  for free. The realizer is a parameter now instead of a hardcoded import.
- `tests/test_phase4_realizer_resolution.py` -- the control, the decomposition,
  the negation defect pin, and the §6 evidence.
- The claim is restated at both places it was made: the lane docstring and
  `grammar_roundtrip/contract.md`.

Zero served bytes change. No serving authorization needed.

§6 -- the plan's pre-commitment was wrong, and is corrected rather than
quietly edited
------------------------------------------------------------------------
§6 forks on read_rate: risen => Phase 5 diversity; near-zero => §1.8 graph-model
mismatch => ADR. It pre-committed to the second. Neither is what the
measurement says.

g_read_rate went to 1/293. The unblocked case was blocked by a one-line WRITER
defect (predicate-nominal object agreement, #135), not by §1.8. The other 292:

    no_template_match     289   reader has no SUBJ-VERB-OBJ template at all
    unknown_morphology      2   prepositional objects (reserved_word_in_np)
    unsupported_negation    1   reader has no negated-categorical template

Every one is the reader declining a CONSTRUCTION, not a projection disagreeing
about a graph it parsed. Where a construction is in both inventories the round
trip closes exactly. So the barrier is the OVERLAP of the two construction
inventories, currently one construction wide -- Phase 5's item 1, tractable,
not an ADR-scale model decision.

§1.6 read "uniform no_template_match" as evidence FOR the type mismatch. It is
not: no_template_match is a coverage fact, and the corpus was 289/293 bare
transitives, a construction the reader never claimed to read. The measurement
was mostly reporting the corpus's composition.

And a finding in my own stack
------------------------------
`tests/test_realizer_quantifier_agreement.py` lived ONLY in the `cognition`
suite, which is not on the AGENTS.md pre-push gate. So every pin Phases 3 and 4
added to it -- including the invariant covering all twelve inflection branches
-- ran in NO gate. That is why smoke stayed at 621 across two PRs that added 13
tests between them, and it is the same silent-red shape the smoke list already
calls out for test_adr_index.py.

Registered into `deductive`, which now runs 504 instead of 406.

Mutation
--------
    baseline                                             12 pass
    serving metric computed with realize_target           4 FAIL
    render_semantic GAINS a `negated` parameter           2 FAIL
    realize_semantic delegated to realize_target          6 FAIL

Row 2 matters most: if someone FIXES the negation defect, the pin forces a
deliberate revision instead of passing silently.

Still open, still Shay's: option (a), promoting realize_target to the serving
path. It now has a price tag -- 340/347 over 85/347 and the ability to say
"not", against a move in every surface hash and whatever the Shadow Coherence
Gate ruling in core/cognition/surface_resolution.py was protecting.

[Verification]: in-worktree on CPython 3.12.13 with `uv sync --locked` --
smoke 621, deductive 504 (was 406; +12 new Phase 4 pins, +86 previously
ungated), lane pins 11/11 unchanged, no pin edited.
2026-07-27 11:57:48 -07:00
..
public/v1 feat(evals): grammar round-trip instrument (Phase 1) 2026-07-26 16:30:49 -07:00
__init__.py feat(evals): grammar round-trip instrument (Phase 1) 2026-07-26 16:30:49 -07:00
contract.md feat(evals): the fluency lanes report the writer that actually ships (Phase 4) 2026-07-27 11:57:48 -07:00
corpora.py feat(evals): grammar round-trip instrument (Phase 1) 2026-07-26 16:30:49 -07:00
projection.py feat(evals): grammar round-trip instrument (Phase 1) 2026-07-26 16:30:49 -07:00
runner.py feat(evals): grammar round-trip instrument (Phase 1) 2026-07-26 16:30:49 -07:00