Shay ad7993e861 feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep

Completes the Phase 5 curriculum-era lane checklist alongside 5.1.

English-substrate domain lanes (5.4–5.7) — extend the proven
english_fluency_ood pattern with new vocabulary domains. Same
13-construction realizer, same grammatical_coverage rubric, new
triples. All four lanes land at 100% on both splits:

  5.4 elementary_mathematics_ood    117/117 public + 39/39 holdouts
      domains: arithmetic, set, geometry  |  holdout: probability
  5.5 foundational_physics_ood      117/117 + 39/39
      domains: mechanics, electricity, thermodynamics  |  holdout: optics
  5.6 foundational_biology_ood      117/117 + 39/39
      domains: cell, organism, ecosystem  |  holdout: genetics
  5.7 classical_literature_ood      117/117 + 39/39
      domains: epic, tragedy, lyric  |  holdout: comedy

New-language lanes (5.2 Hebrew, 5.3 Koine Greek) — scoped honestly to
v1 = C01 only, script + length rubric. The realizer's
tense/aspect/quantifier/negation logic in generate/templates.py is
English-only; C02-C13 in HE/GRC requires Hebrew/Greek morphology +
rhetorical templates, named explicitly in each lane's gaps.md as the
v2 unblock path. v1 measures what infrastructure exists:

  5.2 hebrew_fluency       3/3  (predicate-subject-object assembly,
                                  Hebrew script gate)
  5.3 koine_greek_fluency  3/3  (subject-object-predicate assembly,
                                  Greek script gate)

Lane scaffolds follow the established pattern: contract.md, runner.py,
__init__.py, gaps.md, public/v1/cases.jsonl, dev/cases.jsonl,
holdouts/v1/cases.jsonl (5.4–5.7 only; HE/GRC holdouts deferred to v2
when vocabulary expands).

Generators + scorers:
  scripts/generate_phase5_domain_lanes.py      — 5.4–5.7 case emit
  scripts/scaffold_phase5_domain_lanes.py      — 5.4–5.7 contracts/runners
  scripts/generate_phase5_language_lanes.py    — 5.2/5.3 case emit
  scripts/score_phase5_holdouts.py             — parallel holdouts scoring
                                                 via multiprocessing.Pool
                                                 (mirrors the parallel-eval
                                                 pattern from evals/parallel.py)

Lanes are wired into core eval --list automatically through the
framework's lane discovery; parallel sweeps via bash background jobs
(one process per lane).

Regression clean: smoke 54, runtime 19, teaching 17, packs 6,
cognition 57, algebra 132. Cognition eval 100% across all metrics.

2026-05-16 20:59:31 -07:00

1.7 KiB

Raw Blame History

Hebrew fluency eval lane (Phase 5.2)

What it measures (v1, honest scope)

Whether the deterministic articulation layer (generate/articulation.py) produces a Hebrew-script surface in verb-second word order (predicate-subject-object) when given a (subject, predicate, object) triple drawn from the he_core_cognition_v1 / he_logos_micro_v1 seed packs.

This is the C01 (simple declarative) gate only.

Why v1 is C01-only

The realizer's tense/aspect/quantifier/negation logic in generate/templates.py and generate/morphology.py is English-only:

_PREDICATE_DISPLAY maps English predicates to English phrases
_inflect_predicate calls past_tense, present_participle, past_participle, base_form from generate/morphology.py — all English regular-verb morphology
_MOVE_TEMPLATES produce English-only function words ("furthermore", "in contrast", "next", "correction")

Measuring C02–C13 in Hebrew without first building Hebrew morphology

Hebrew rhetorical templates would be measuring code paths that don't exist. The honest v1 scope is what does exist: pack grounding + word-order assembly through _assemble.

Inputs

public/v1/cases.jsonl: 3 cases — one per triple from the seed packs, all C01.
dev/cases.jsonl: 1 case — smoke check for runner integration.

Scoring rubric

A case passes if the realized surface:

Contains Hebrew script (U+0590..U+05FF)
Is in accept_surfaces OR satisfies all constraints (must_contain, max_words)

Provenance

Same realize() entry point as tests/test_articulation.py::test_realize_hebrew_surface_uses_hebrew_script_and_compact, generalized into a lane with multiple cases and the standard scoring shape.

1.7 KiB Raw Blame History Unescape Escape