core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

feat(evals): grammar round-trip instrument — CORE cannot read its own writing (0/280) #130

Merged
core-labs merged 1 commit from feat/grammar-roundtrip-instrument into main 2026-07-26 23:51:23 +00:00
Conversation 0 Commits 1 Files changed 9 +1436
core-labs commented 2026-07-26 23:50:53 +00:00
Owner
Copy link

Phase 1 of the plan of record (#128, corrected by #129). New files only — no serving code, no corpus, no pin touched. Rebased onto 041c6202, so the diff is +1436/−0 across 9 files.

What this is

The arc's claim is that CORE has two grammars — one for reading, one for writing — and they were never checked against each other. That claim was unfalsifiable because there was no instrument. This is the instrument.

Three measurements, deliberately reported separately:

  • G-round-trip — read(write(g)): take a graph, write it to English, read it back, compare propositions.
  • S-round-trip — write(read(s)): take a sentence, read it, write it back, compare strings.
  • reject_rate — malformed surfaces the reader must refuse.

evals/grammar_roundtrip/projection.py exists because MeaningGraph and PropositionGraph are not type-compatible (n-ary Relation with source spans vs. SPO GraphNode with discourse edges). Both project into a CanonicalProposition frozenset. The lane keeps its own local copy of _QUANTIFIER_PREDICATE so it never consumes the thing it measures; test_quantifier_map_matches_reader pins the copy to the original.

Baseline, measured on this branch

metric value meaning
graph_cases 280
g_write_rate 1.0 the writer emits English for all 280
g_read_rate 0.0 the reader comprehends 0 of 280
g_propositions_expected 370
g_args_rate / g_predicates_rate / g_exact_rate 0.0 / 0.0 / 0.0
surface_cases 8
s_read_rate 1.0 reader comprehends 8/8
s_renderable_rate 0.625 only 5 of 8 yield a graph the renderer can express at all
s_surface_match_rate 0.0 0 of those 5 return the input string
negative_cases 16 8 committed + 8 runtime shuffles
reject_rate 1.0 16/16 refused

reject_rate is not trivially green. The same reader accepts 8/8 positives and refuses 16/16 negatives — it discriminates. If it simply refused everything, s_read_rate would be 0.0, not 1.0. The negatives reuse the positives' own vocabulary (pinned by test_shuffled_negatives_reuse_positive_vocabulary), so rejection is about structure, not unknown words. Shuffles use a fixed rotation rather than a PRNG, so reject_rate is byte-reproducible.

The defect it caught without being asked to

s_surface_match_rate = 0.0 surfaced a live, user-visible defect on a ratified flag-ON band. generate/proof_chain/render.py::_categorical_clause formats "all {s} are {p}" with the singular subject:

_render_categorical("subset", "dog", "animal")  ==  "all dog are animal"

CORE currently serves "all dog are mammal" — 4 of the 47 ratified deduction_serve corpus cases. wrong=0 is intact: every verdict is still correct. Only the prose is wrong, which is exactly the class of defect a verdict-only gate cannot see. Pinned concretely by test_the_categorical_render_defect_concretely so the fix in Phase 2B has to move a number.

What this does NOT prove

Round-trip agreement is mutual intelligibility, not English quality. A writer and a reader that share the same bug agree perfectly. contract.md says this in the lane's own words. Round-trip is necessary and not sufficient — grammaticality still needs a grammar, which is Phase 2 onward.

Mutation guarantees

A pin that cannot fail is worthless. Each of these goes red under the mutation named:

  • test_reject_rate_goes_red_when_the_reader_accepts_everything — monkeypatches comprehend to always succeed ⇒ reject_rate 1.0 → 0.0.
  • test_the_categorical_render_defect_concretely — fixing the renderer without updating the pin ⇒ red (intentional; Phase 2B must touch it).
  • test_quantifier_map_matches_reader — the lane's local map drifting from reader.py ⇒ red.
  • test_positive_surfaces_are_all_inside_the_reader_envelope — adding a positive the reader can't read ⇒ red, so s_read_rate can't be gamed by corpus choice.

scripts/measure_grammar_seam.py reproduces every §1 number in the plan (tables, agreement, readers, served, roundtrip).

[Verification]: in-worktree on CPython 3.12.13 with uv sync --locked, after rebase onto 041c6202 — uv run core test --suite smoke -q 621 passed; uv run core test --suite deductive -q 364 passed (349 + 15 new, also run by the pre-push gate); scripts/verify_lane_shas.py 11/11 match pinned SHAs. No serving module, corpus, or pin modified — the only non-new-file change is one line adding tests/test_grammar_roundtrip.py to the deductive suite tuple in core/cli_test.py.

Phase 1 of the plan of record (#128, corrected by #129). New files only — no serving code, no corpus, no pin touched. Rebased onto `041c6202`, so the diff is +1436/−0 across 9 files. ## What this is The arc's claim is that CORE has **two grammars** — one for reading, one for writing — and they were never checked against each other. That claim was unfalsifiable because there was no instrument. This is the instrument. Three measurements, deliberately reported separately: - **G-round-trip** — `read(write(g))`: take a graph, write it to English, read it back, compare propositions. - **S-round-trip** — `write(read(s))`: take a sentence, read it, write it back, compare strings. - **reject_rate** — malformed surfaces the reader must refuse. `evals/grammar_roundtrip/projection.py` exists because `MeaningGraph` and `PropositionGraph` are **not type-compatible** (n-ary `Relation` with source spans vs. SPO `GraphNode` with discourse edges). Both project into a `CanonicalProposition` frozenset. The lane keeps its **own local copy** of `_QUANTIFIER_PREDICATE` so it never consumes the thing it measures; `test_quantifier_map_matches_reader` pins the copy to the original. ## Baseline, measured on this branch | metric | value | meaning | |---|---|---| | `graph_cases` | 280 | | | `g_write_rate` | **1.0** | the writer emits English for all 280 | | `g_read_rate` | **0.0** | the reader comprehends **0 of 280** | | `g_propositions_expected` | 370 | | | `g_args_rate` / `g_predicates_rate` / `g_exact_rate` | 0.0 / 0.0 / 0.0 | | | `surface_cases` | 8 | | | `s_read_rate` | **1.0** | reader comprehends 8/8 | | `s_renderable_rate` | 0.625 | only 5 of 8 yield a graph the renderer can express at all | | `s_surface_match_rate` | **0.0** | 0 of those 5 return the input string | | `negative_cases` | 16 | 8 committed + 8 runtime shuffles | | `reject_rate` | **1.0** | 16/16 refused | **`reject_rate` is not trivially green.** The same reader accepts **8/8** positives and refuses **16/16** negatives — it discriminates. If it simply refused everything, `s_read_rate` would be 0.0, not 1.0. The negatives reuse the positives' own vocabulary (pinned by `test_shuffled_negatives_reuse_positive_vocabulary`), so rejection is about structure, not unknown words. Shuffles use a fixed rotation rather than a PRNG, so `reject_rate` is byte-reproducible. ## The defect it caught without being asked to `s_surface_match_rate = 0.0` surfaced a **live, user-visible defect on a ratified flag-ON band**. `generate/proof_chain/render.py::_categorical_clause` formats `"all {s} are {p}"` with the **singular** subject: ``` _render_categorical("subset", "dog", "animal") == "all dog are animal" ``` CORE currently serves **"all dog are mammal"** — 4 of the 47 ratified `deduction_serve` corpus cases. `wrong=0` is intact: every verdict is still correct. Only the prose is wrong, which is exactly the class of defect a verdict-only gate cannot see. Pinned concretely by `test_the_categorical_render_defect_concretely` so the fix in Phase 2B has to move a number. ## What this does NOT prove Round-trip agreement is **mutual intelligibility, not English quality.** A writer and a reader that share the same bug agree perfectly. `contract.md` says this in the lane's own words. Round-trip is necessary and **not sufficient** — grammaticality still needs a grammar, which is Phase 2 onward. ## Mutation guarantees A pin that cannot fail is worthless. Each of these goes red under the mutation named: - `test_reject_rate_goes_red_when_the_reader_accepts_everything` — monkeypatches `comprehend` to always succeed ⇒ `reject_rate` 1.0 → 0.0. - `test_the_categorical_render_defect_concretely` — fixing the renderer without updating the pin ⇒ red (intentional; Phase 2B must touch it). - `test_quantifier_map_matches_reader` — the lane's local map drifting from `reader.py` ⇒ red. - `test_positive_surfaces_are_all_inside_the_reader_envelope` — adding a positive the reader can't read ⇒ red, so `s_read_rate` can't be gamed by corpus choice. `scripts/measure_grammar_seam.py` reproduces every §1 number in the plan (`tables`, `agreement`, `readers`, `served`, `roundtrip`). [Verification]: in-worktree on CPython 3.12.13 with `uv sync --locked`, after rebase onto `041c6202` — `uv run core test --suite smoke -q` **621 passed**; `uv run core test --suite deductive -q` **364 passed** (349 + 15 new, also run by the pre-push gate); `scripts/verify_lane_shas.py` **11/11 match pinned SHAs**. No serving module, corpus, or pin modified — the only non-new-file change is one line adding `tests/test_grammar_roundtrip.py` to the `deductive` suite tuple in `core/cli_test.py`.
core-labs added 1 commit 2026-07-26 23:50:55 +00:00 
The measurement foundation for docs/plans/grammar-unification-2026-07-26.md.

WHY: evals/deterministic_fluency reports 1.00 on all six predicates and
still passes "banana does the.", "wet ground rains the is." and
"is is is is." — it checks terminal punctuation, presence of a verb-shaped
token, and two anti-shape regexes. Heuristic predicates will always have
that failure mode, because grammaticality cannot be measured without a
grammar. So this lane measures agreement between the two halves of CORE
that already encode grammar, and requires the measurement to FAIL on salad.

Two directions, reported separately because they fail for different
reasons and have different remedies:

  G-round-trip  graph -> realize_target -> surface -> comprehend -> graph
  S-round-trip  surface -> comprehend -> graph -> categorical renderer -> surface

v1 baseline on main @ 9696443a:

  graph_cases            280      surface_cases            8
  g_write_rate         1.000      s_read_rate          1.000
  g_read_rate          0.000      s_renderable_rate    0.625
  g_exact_rate         0.000      s_surface_match_rate 0.000
                                  negative_cases          16
                                  reject_rate          1.000

g_read_rate and s_surface_match_rate are pins on measured DEFECTS, not
goals; they may be revised upward only. s_surface_match_rate = 0 is the
§1.7 categorical render defect caught by construction — the lane found it
without being told to look.

g_args_rate and g_predicates_rate are deliberately separate: high argument
agreement with low predicate agreement would mean the grammars align and
only the vocabulary is split, a materially different remedy from both
being low. That distinction decides the arc's direction (plan §6).

Design notes:
- The committed cases.jsonl is the SINGLE source for authored surfaces —
  no in-module duplicate, since a second copy of a corpus is the defect
  this arc exists to remove. Negative shuffles are DERIVED at run time so
  they cannot drift from the positives.
- The shuffles are lexically identical to positives (same vocabulary, same
  length, order destroyed) so the lane cannot pass by vocabulary-checking.
- Fixed rotation, not a PRNG, so reject_rate is byte-reproducible.
- The lane keeps a local copy of the reader's quantifier map ON PURPOSE so
  it never becomes a consumer of what it measures;
  test_quantifier_map_matches_reader fails loudly if the reader changes.
- _render_categorical deliberately reaches a private serving function: a
  lane measuring a private copy would measure what users never see.

Every guarantee is paired with a mutation test. The load-bearing one is
test_reject_rate_goes_red_when_the_reader_accepts_everything: an
accept-everything reader must drive reject_rate to 0.0. Without it,
reject_rate == 1.0 would be unfalsifiable — precisely the defect that
makes the existing fluency lane decoration.

Also documents plainly what round-trip does NOT prove: it measures mutual
intelligibility, not English quality. english_fluency_ood accepts "river
flows valley" and round-trip would be happy with it. No metric here may be
cited as evidence of prose quality.

scripts/measure_grammar_seam.py reproduces every number in the plan's §1
so a reader can check them instead of trusting them.

[Verification]: in-worktree on CPython 3.12.13, uv sync --locked —
smoke 621 (unchanged), deductive 364 (349 + 15 new). Lane SHA pins
verified separately. No serving code touched; new files plus one suite
registration line.
core-labs merged commit 0948f7cdb8 into main 2026-07-26 23:51:23 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: core-labs/core#130
No description provided.
Delete branch "feat/grammar-roundtrip-instrument"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?