ad7993e861
Completes the Phase 5 curriculum-era lane checklist alongside 5.1.
English-substrate domain lanes (5.4–5.7) — extend the proven
english_fluency_ood pattern with new vocabulary domains. Same
13-construction realizer, same grammatical_coverage rubric, new
triples. All four lanes land at 100% on both splits:
5.4 elementary_mathematics_ood 117/117 public + 39/39 holdouts
domains: arithmetic, set, geometry | holdout: probability
5.5 foundational_physics_ood 117/117 + 39/39
domains: mechanics, electricity, thermodynamics | holdout: optics
5.6 foundational_biology_ood 117/117 + 39/39
domains: cell, organism, ecosystem | holdout: genetics
5.7 classical_literature_ood 117/117 + 39/39
domains: epic, tragedy, lyric | holdout: comedy
New-language lanes (5.2 Hebrew, 5.3 Koine Greek) — scoped honestly to
v1 = C01 only, script + length rubric. The realizer's
tense/aspect/quantifier/negation logic in generate/templates.py is
English-only; C02-C13 in HE/GRC requires Hebrew/Greek morphology +
rhetorical templates, named explicitly in each lane's gaps.md as the
v2 unblock path. v1 measures what infrastructure exists:
5.2 hebrew_fluency 3/3 (predicate-subject-object assembly,
Hebrew script gate)
5.3 koine_greek_fluency 3/3 (subject-object-predicate assembly,
Greek script gate)
Lane scaffolds follow the established pattern: contract.md, runner.py,
__init__.py, gaps.md, public/v1/cases.jsonl, dev/cases.jsonl,
holdouts/v1/cases.jsonl (5.4–5.7 only; HE/GRC holdouts deferred to v2
when vocabulary expands).
Generators + scorers:
scripts/generate_phase5_domain_lanes.py — 5.4–5.7 case emit
scripts/scaffold_phase5_domain_lanes.py — 5.4–5.7 contracts/runners
scripts/generate_phase5_language_lanes.py — 5.2/5.3 case emit
scripts/score_phase5_holdouts.py — parallel holdouts scoring
via multiprocessing.Pool
(mirrors the parallel-eval
pattern from evals/parallel.py)
Lanes are wired into core eval --list automatically through the
framework's lane discovery; parallel sweeps via bash background jobs
(one process per lane).
Regression clean: smoke 54, runtime 19, teaching 17, packs 6,
cognition 57, algebra 132. Cognition eval 100% across all metrics.
55 lines
2 KiB
Python
55 lines
2 KiB
Python
"""Score holdouts split for each Phase 5.4–5.7 domain lane.
|
||
|
||
The `core eval` CLI only exposes dev/public splits. Holdouts are
|
||
sealed-test infrastructure scored by direct runner invocation, which
|
||
is how `evals/english_fluency_ood/results/v1_holdouts_metrics.json`
|
||
was produced.
|
||
|
||
This script mirrors that pattern across the four new lanes, runs each
|
||
lane's holdouts in parallel via multiprocessing.Pool (one process per
|
||
lane), and writes `results/v1_holdouts_metrics.json` +
|
||
`results/v1_holdouts_details.json` per lane.
|
||
|
||
Run:
|
||
.venv/bin/python scripts/score_phase5_holdouts.py
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import multiprocessing as mp
|
||
from importlib import import_module
|
||
from pathlib import Path
|
||
|
||
LANES = [
|
||
"elementary_mathematics_ood",
|
||
"foundational_physics_ood",
|
||
"foundational_biology_ood",
|
||
"classical_literature_ood",
|
||
]
|
||
|
||
|
||
def _score_lane(lane: str) -> tuple[str, dict, int, int]:
|
||
repo = Path(__file__).resolve().parent.parent
|
||
cases_path = repo / "evals" / lane / "holdouts" / "v1" / "cases.jsonl"
|
||
runner = import_module(f"evals.{lane}.runner")
|
||
cases = [json.loads(line) for line in cases_path.read_text().splitlines() if line.strip()]
|
||
report = runner.run_lane(cases)
|
||
|
||
results_dir = repo / "evals" / lane / "results"
|
||
results_dir.mkdir(parents=True, exist_ok=True)
|
||
(results_dir / "v1_holdouts_metrics.json").write_text(
|
||
json.dumps(report.metrics, indent=2, sort_keys=True) + "\n"
|
||
)
|
||
(results_dir / "v1_holdouts_details.json").write_text(
|
||
json.dumps(report.case_details, indent=2, sort_keys=True) + "\n"
|
||
)
|
||
return lane, report.metrics, report.metrics["passed"], report.metrics["total"]
|
||
|
||
|
||
if __name__ == "__main__":
|
||
ctx = mp.get_context("spawn")
|
||
with ctx.Pool(processes=min(len(LANES), 4)) as pool:
|
||
for lane, metrics, passed, total in pool.map(_score_lane, LANES):
|
||
acc = metrics["accuracy"] * 100
|
||
print(f"{lane:38s} {passed:3d}/{total:3d} {acc:5.1f}%")
|