core-labs/core
core-labs/core
1
0
Fork 0
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity

fix(evals): introspect run_lane signature before passing workers kwarg

Browse source 
PR #46 added the `workers` kwarg to framework dispatch (evals/framework.py:176)
but only the cognition runner was updated to accept it. The three serial
lanes (cold_start_grounding, deterministic_fluency, warmed_session_consistency)
— and ~30 other runners — raised TypeError on every framework invocation,
producing 18 test failures across the full suite.

Fix at the dispatch site rather than per-runner: inspect the target
run_lane signature and pass `workers=` only when it accepts the kwarg
(or has **kwargs). This keeps the framework contract backward-compatible
with the legacy two-arg shape and forward-compatible with future
parallelized runners — no runner needs updating.

Full lane: 2859 passed, 3 skipped, 0 failed (was 2841/18 failed).
Cognition eval byte-identical: 100/100/91.7/100.
This commit is contained in:
Shay 2026-05-20 05:59:51 -07:00
parent ae45e768ec
commit 21b10028b5
1 changed files with 14 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
evals/framework.py
View file

@ -15,6 +15,7 @@ from __future__ import annotations
import importlib import importlib
import importlib.util import importlib.util
import inspect
import json import json
import sys import sys
from dataclasses import dataclass, field from dataclasses import dataclass, field
@ -173,7 +174,19 @@ def run_lane(
cases = load_cases(cases_path) cases = load_cases(cases_path)
runner_module = load_lane_runner(lane) runner_module = load_lane_runner(lane)
report = runner_module.run_lane(cases, config=config, workers=workers) # PR #46 added the `workers` kwarg to enable parallel execution on
# lanes that opt in (currently: cognition). Most lane runners are
# serial and predate this contract. Pass `workers` only when the
# target runner's signature accepts it — otherwise call the legacy
# two-arg form. This keeps the framework dispatch backward-compatible
# without forcing every runner to accept a no-op kwarg.
runner_params = inspect.signature(runner_module.run_lane).parameters
if "workers" in runner_params or any(
p.kind == inspect.Parameter.VAR_KEYWORD for p in runner_params.values()
):
report = runner_module.run_lane(cases, config=config, workers=workers)
else:
report = runner_module.run_lane(cases, config=config)
return LaneResult( return LaneResult(
lane=lane.name, lane=lane.name,