Shay ad7993e861 feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep

Completes the Phase 5 curriculum-era lane checklist alongside 5.1.

English-substrate domain lanes (5.4–5.7) — extend the proven
english_fluency_ood pattern with new vocabulary domains. Same
13-construction realizer, same grammatical_coverage rubric, new
triples. All four lanes land at 100% on both splits:

  5.4 elementary_mathematics_ood    117/117 public + 39/39 holdouts
      domains: arithmetic, set, geometry  |  holdout: probability
  5.5 foundational_physics_ood      117/117 + 39/39
      domains: mechanics, electricity, thermodynamics  |  holdout: optics
  5.6 foundational_biology_ood      117/117 + 39/39
      domains: cell, organism, ecosystem  |  holdout: genetics
  5.7 classical_literature_ood      117/117 + 39/39
      domains: epic, tragedy, lyric  |  holdout: comedy

New-language lanes (5.2 Hebrew, 5.3 Koine Greek) — scoped honestly to
v1 = C01 only, script + length rubric. The realizer's
tense/aspect/quantifier/negation logic in generate/templates.py is
English-only; C02-C13 in HE/GRC requires Hebrew/Greek morphology +
rhetorical templates, named explicitly in each lane's gaps.md as the
v2 unblock path. v1 measures what infrastructure exists:

  5.2 hebrew_fluency       3/3  (predicate-subject-object assembly,
                                  Hebrew script gate)
  5.3 koine_greek_fluency  3/3  (subject-object-predicate assembly,
                                  Greek script gate)

Lane scaffolds follow the established pattern: contract.md, runner.py,
__init__.py, gaps.md, public/v1/cases.jsonl, dev/cases.jsonl,
holdouts/v1/cases.jsonl (5.4–5.7 only; HE/GRC holdouts deferred to v2
when vocabulary expands).

Generators + scorers:
  scripts/generate_phase5_domain_lanes.py      — 5.4–5.7 case emit
  scripts/scaffold_phase5_domain_lanes.py      — 5.4–5.7 contracts/runners
  scripts/generate_phase5_language_lanes.py    — 5.2/5.3 case emit
  scripts/score_phase5_holdouts.py             — parallel holdouts scoring
                                                 via multiprocessing.Pool
                                                 (mirrors the parallel-eval
                                                 pattern from evals/parallel.py)

Lanes are wired into core eval --list automatically through the
framework's lane discovery; parallel sweeps via bash background jobs
(one process per lane).

Regression clean: smoke 54, runtime 19, teaching 17, packs 6,
cognition 57, algebra 132. Cognition eval 100% across all metrics.

2026-05-16 20:59:31 -07:00

1.5 KiB

Raw Permalink Blame History

Foundational biology fluency OOD eval lane (Phase 5.6)

What it measures

Whether the deterministic realizer produces grammatical English surfaces over foundational-biology vocabulary the seed pack does not contain.

The structural claim under test: fluency is mechanistic in the realizer (templates over typed graph nodes), not lexical (pack-bound). If the claim holds, this OOD vocabulary passes the same syntactic gates the seed vocabulary did at v1/v2 of grammatical_coverage — and the same 13 constructions covered by the Phase 5.1 english_fluency_ood lane.

Inputs

public/v1/cases.jsonl: 117 cases — 13 constructions × 3 public domains (cell, organism, ecosystem) × 3 lexical items per domain.
holdouts/v1/cases.jsonl: 39 cases — 13 constructions × 1 held-out domain (genetics) × 3 items.
dev/cases.jsonl: 13 cases — one of each construction from the first public domain.

Each case is one PropositionGraph JSON with realised constraints, generated by scripts/generate_phase5_domain_lanes.py.

Scoring rubric

Delegated to evals/grammatical_coverage/runner.py so the rubric stays consistent across all fluency-OOD lanes:

surface satisfies accept_surfaces or constraints (must_contain, word_order, max_words)
surface is not in reject_surfaces

Target

100% on public + holdouts. Any miss indicates a real realizer or inflection gap and is filed in gaps.md.

Provenance

Same harness as evals/english_fluency_ood/ (Phase 5.1). Vocabulary is the only material change between lanes.

1.5 KiB Raw Permalink Blame History Unescape Escape