Skip to content

FunctionExecutor.map() got an unexpected keyword argument 'chunk_size' #45

@jmettes

Description

@jmettes

It seems like this airflow plugin is out of date with latest lithops API - expecting chunk_size, which I suspect is now chunksize

Running airflow 2.9.0 via docker-compose, installing lithops v3.6.0 (Python 3.12) via _PIP_ADDITIONAL_REQUIREMENTS

$ grep lithops docker-compose.yml
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-lithops}

Installed this airflow-plugin by manually copying the lithops_airflow_plugin/ directory into the plugins/ directory (and changing __init__.py to from lithops_airflow_plugin.lithops_plugin import *)

Running this

from airflow import DAG
from airflow.operators.python import PythonOperator
from lithops_airflow_plugin import LithopsMapOperator
import random

args = {
    'owner': 'lithops',
}

def add(x, y):
    return x+y

dag = DAG(
    dag_id='LithopsTest',
    default_args=args,
    schedule_interval=None,
)

gen_list = PythonOperator(
    task_id='gen_list',
    python_callable=lambda: [random.randint(1, 100) for _ in range(10)],
    dag=dag
)

mult_num_map = LithopsMapOperator(
    task_id='mult_num_map',
    map_function=add,
    iterdata_from_task={'a': 'gen_list'},
    extra_args={'b': 10},
    dag=dag,
    backend='localhost'
)

gen_list >> mult_num_map

I get

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 400, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/plugins/lithops_airflow_plugin/operators/lithops_operator.py", line 86, in execute
    self._futures = self.execute_callable(context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/airflow/plugins/lithops_airflow_plugin/operators/lithops_operator.py", line 232, in execute_callable
    return self._executor.map(map_function=self.map_function,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: FunctionExecutor.map() got an unexpected keyword argument 'chunk_size'
[2025-07-11, 06:03:29 UTC] {taskinstance.py:1205} INFO - Marking task as FAILED. dag_id=LithopsTest, task_id=mult_num_map, execution_date=20250711T060325, start_date=20250711T060329, end_date=20250711T060329
[2025-07-11, 06:03:29 UTC] {standard_task_runner.py:110} ERROR - Failed to execute job 15 for task mult_num_map (FunctionExecutor.map() got an unexpected keyword argument 'chunk_size'; 98)
[2025-07-11, 06:03:29 UTC] {local_task_job_runner.py:240} INFO - Task exited with return code 1
[2025-07-11, 06:03:29 UTC] {taskinstance.py:3482} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2025-07-11, 06:03:29 UTC] {local_task_job_runner.py:222}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions