4a3e89b730
First Phase 5 lane. Tests whether the deterministic realizer produces grammatical English across all 13 C01-C13 constructions when the (subject, predicate, object) vocabulary is outside the en_core_cognition_v1 seed pack. Four OOD domains: nature, tech, domestic (public), chemistry (holdouts). Public 117/117 (100%) and holdouts 39/39 (100%) — every construction passes on every domain. Realizer fluency is mechanistic and pack-independent; the Phase 5 capability story rests on a sound structural bet. Known v1 gaps (designed around to isolate the structural claim): G1 irregular past tense (realizer applies -ed unconditionally), G2 plural agreement under quantifiers (no pluralisation of subjects under "all"/"some"), G3 rubric-side punctuation strictness in shared _check_word_order. All three are documented in gaps.md with bounded follow-on lanes. Scoring is delegated to evals.grammatical_coverage.runner so the rubric stays consistent. Cases generated by scripts/generate_english_fluency_ood.py for reproducibility.
30 lines
1.1 KiB
Python
30 lines
1.1 KiB
Python
"""English fluency OOD eval lane runner (Phase 5.1).
|
|
|
|
Verifies the deterministic realizer produces grammatical English
|
|
surfaces across all 13 grammatical-coverage constructions when the
|
|
(subject, predicate, object) vocabulary is **out of the
|
|
en_core_cognition_v1 distribution** — drawn from nature, tech,
|
|
domestic, and chemistry domains that the seed pack does not
|
|
contain.
|
|
|
|
The structural claim under test: fluency is mechanistic in the
|
|
realizer (templates over typed graph nodes), not lexical
|
|
(pack-bound). If the claim holds, OOD vocabulary passes the same
|
|
syntactic gates as the seed vocabulary did at v1/v2 of
|
|
`grammatical_coverage`.
|
|
|
|
Scoring is delegated to the grammatical_coverage runner so the
|
|
rubric stays consistent across lanes.
|
|
|
|
Conforms to the framework interface: run_lane(cases, config=None) -> report.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
from evals.grammatical_coverage.runner import LaneReport, run_lane as _run
|
|
|
|
|
|
def run_lane(cases: list[dict[str, Any]], *, config: Any = None) -> LaneReport:
|
|
return _run(cases, config=config)
|