core-labs/core
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
core/evals/elementary_mathematics_ood/gaps.md
Shay ad7993e861 feat(phase5): land 5.2–5.7 — six new fluency lanes, parallel sweep 
Completes the Phase 5 curriculum-era lane checklist alongside 5.1.

English-substrate domain lanes (5.4–5.7) — extend the proven
english_fluency_ood pattern with new vocabulary domains. Same
13-construction realizer, same grammatical_coverage rubric, new
triples. All four lanes land at 100% on both splits:

  5.4 elementary_mathematics_ood    117/117 public + 39/39 holdouts
      domains: arithmetic, set, geometry  |  holdout: probability
  5.5 foundational_physics_ood      117/117 + 39/39
      domains: mechanics, electricity, thermodynamics  |  holdout: optics
  5.6 foundational_biology_ood      117/117 + 39/39
      domains: cell, organism, ecosystem  |  holdout: genetics
  5.7 classical_literature_ood      117/117 + 39/39
      domains: epic, tragedy, lyric  |  holdout: comedy

New-language lanes (5.2 Hebrew, 5.3 Koine Greek) — scoped honestly to
v1 = C01 only, script + length rubric. The realizer's
tense/aspect/quantifier/negation logic in generate/templates.py is
English-only; C02-C13 in HE/GRC requires Hebrew/Greek morphology +
rhetorical templates, named explicitly in each lane's gaps.md as the
v2 unblock path. v1 measures what infrastructure exists:

  5.2 hebrew_fluency       3/3  (predicate-subject-object assembly,
                                  Hebrew script gate)
  5.3 koine_greek_fluency  3/3  (subject-object-predicate assembly,
                                  Greek script gate)

Lane scaffolds follow the established pattern: contract.md, runner.py,
__init__.py, gaps.md, public/v1/cases.jsonl, dev/cases.jsonl,
holdouts/v1/cases.jsonl (5.4–5.7 only; HE/GRC holdouts deferred to v2
when vocabulary expands).

Generators + scorers:
  scripts/generate_phase5_domain_lanes.py      — 5.4–5.7 case emit
  scripts/scaffold_phase5_domain_lanes.py      — 5.4–5.7 contracts/runners
  scripts/generate_phase5_language_lanes.py    — 5.2/5.3 case emit
  scripts/score_phase5_holdouts.py             — parallel holdouts scoring
                                                 via multiprocessing.Pool
                                                 (mirrors the parallel-eval
                                                 pattern from evals/parallel.py)

Lanes are wired into core eval --list automatically through the
framework's lane discovery; parallel sweeps via bash background jobs
(one process per lane).

Regression clean: smoke 54, runtime 19, teaching 17, packs 6,
cognition 57, algebra 132. Cognition eval 100% across all metrics.
2026-05-16 20:59:31 -07:00

1.2 KiB
Raw Permalink Blame History

Elementary mathematics fluency OOD — gaps

v1 (current)

  • Constructions covered: all 13 (C01–C13) via grammatical_coverage rubric.
  • Domains covered: public — arithmetic, set, geometry; holdout — probability.
  • Predicates default to regular verbs to keep morphology gaps from confounding the structural fluency claim.

Known gaps for v2

The Phase 5.1 English fluency OOD lane already names three realizer gaps that apply equally here:

G1. Irregular past tense — past_tense() handles regular -ed but not irregular forms (e.g. "ran", "stood"). G2. Plural agreement — the realizer does not pluralise nouns or conjugate to plural subjects. G3. Punctuation strictness — the rubric pins comma-bounded relative clauses exactly; small punctuation differences still fail.

These are not lane-specific. When 5.1 closes them, every domain lane gains the same coverage for free.

Out of scope for this lane

  • Domain-specific semantic correctness (whether "ribosome assembles protein" is true biology). This lane measures fluency, not factual correctness. Truth checks live in the epistemic_status surface (ADR-0021) and future curriculum lanes.