Last Updated on September 12, 2022
The multiprocessing pool map() function cannot be used directly with a target function that takes multiple arguments.
Instead, you need to use an alternate function like starmap() or a workaround like a wrapper function.
In this tutorial you will discover how to call the multiprocessing pool map() function with multiple arguments indirectly and how to use alternate approaches to execute target functions that take multiple arguments.
Let’s get started.
Need to Use Pool.map() With Multiple Arguments
The built-in map() function will call a function to each item on an iterable.
For example:
|
1 2 3 4 |
... # call function for each item in an iterable for result in map(task, items): print(result) |
A benefit of the built-in map() function is that it can take multiple iterables. Each iterable will be traversed in lock-step, yielding an argument to the function.
For example:
|
1 2 3 4 |
... # call function for each item in an iterable for result in map(task, items1, items2): print(result) |
The multiprocessing.Pool class provides a parallel version of the map() function. Each call to the target function will be issued to the multiprocessing pool and executed by a child worker process.
For example:
|
1 2 3 4 5 6 |
... # create the process pool with multiprocessing.Pool() as pool: # call function for each item in an iterable in parallel for result in pool.map(task, items): print(result) |
You can learn more about how to use the multiprocessing pool map() function in the tutorial:
A limitation of the multiprocessing pool map() function is that it only takes a single iterable of arguments for the target function.
This means it can only be used to call target functions that take one argument.
How can we use the multiprocessing pool map() function with multiple arguments?
Run loops using all CPUs, download your FREE book to learn how.
How to Use Pool.map() With Multiple Arguments
There are many ways that we can use the multiprocessing pool map() function with a target function that takes multiple arguments.
We will look at 4 common approaches, they are:
- Use Pool.apply_async() instead.
- Use Pool.starmap() instead.
- Change the target function to unpack arguments.
- Use a wrapper function to unpack arguments.
Let’s take a closer look at each approach in turn.
Use Pool.apply_async() Instead
The multiprocessing pool apply() function will execute a single function call in the process pool.
Importantly, it allows a function to be called that takes zero, one, or multiple arguments. As such, it can be used in a loop instead of the map() function.
You can learn more about the multiprocessing apply() function in the tutorial:
A limitation of apply() is that it will block for each call until the target function is complete. As such, it offers no benefit of concurrency.
Instead, we can use the apply_async() function, which is an asynchronous version of the apply() function. We can specify multiple arguments to our target function using the “args” keyword and providing a tuple of arguments.
For example:
|
1 2 3 |
... # issue a task into the process pool with multiple arguments async_result = pool.apply_async(task, args=(arg1, arg2, arg3)) |
It issues the call to the target function into the process pool and returns immediately with an AsyncResult object.
These objects can be collected using a list comprehension.
For example:
|
1 2 3 |
... # issue multiple tasks each with multiple arguments async_results = [pool.apply_async(task, args=(i, i*2, i*3)) for i in range(10)] |
We can then call the get() function on each AsyncResult to retrieve the return value as needed.
This too can be performed in a list comprehension.
For example:
|
1 2 3 |
... # retrieve the return value results results = [ar.get() for ar in async_results] |
You can learn more about the apply_async() function in the tutorial:
Use Pool.starmap() Instead
The multiprocessing pool starmap() function will call the target function with multiple arguments.
As such it can be used instead of the map() function. This is probably the preferred approach for executing a target function in the multiprocessing pool that takes multiple arguments.
The starmap() function takes the name of the target function and an iterable, like the map() function.
The difference is that each item in the iterable must be an iterable that yields results for each call to the target function.
The easiest way to think of this is to have a list where each item in the list is a tuple of arguments for one call to the target function.
For example:
|
1 2 3 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] |
This can then be passed directly to the starmap() function.
For example:
|
1 2 3 |
... # issue multiple tasks each with multiple arguments results = pool.starmap(task, args) |
You can learn more about the starmap() function in the tutorial:
Change The Target Function to Unpack Arguments
The target function can be changed.
Instead of taking multiple arguments, it can be modified to take a single argument, such as a list or tuple of arguments.
The function can then unpack these arguments and operate as per normal.
For example:
|
1 2 3 4 |
# task function executed in a child worker process def task(args): # unpack arguments arg1, arg2, arg3 = args |
The multiprocessing map() function can then be used directly passing a list or tuple of arguments, simulating a call that takes multiple arguments.
For example:
|
1 2 3 4 5 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task, args) |
This requires that you are able to change the target function, which might not be the case in all applications.
Use a Wrapper Function to Unpack Arguments
Unwrapping arguments is an easy fix.
An alternate approach to using this fix is to use a wrapper or proxy function that takes a list or tuple of multiple arguments and is called by map() instead of the true target function.
For example:
|
1 2 3 4 5 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task_wrapper, args) |
The wrapper function will unpack the arguments and then call the target function with multiple arguments.
For example:
|
1 2 3 4 |
# wrapper function for task def task_wrapper(args): # call task() and unpack the arguments return task(*args) |
Recall that the star operator (*) will unpack a list or tuple for us.
This approach is appropriate for cases where you have control over the function called by map() but cannot or do not want to change the target function.
Now that we know how to use the multiprocessing pool map() with multiple arguments, let’s look at some worked examples.
Multiple Arguments with Pool.apply_async()
We can explore how to use apply_async() instead of map() to call a target function with multiple arguments.
In this example, we will define a target function that takes multiple arguments, blocks for a moment to simulate doing work, reports a message then returns a value that combines the arguments. The caller will create the process pool, issue the tasks, then retrieve the return values.
Firstly, we can define the task() function that takes multiple arguments, blocks, reports a message then returns a return value.
|
1 2 3 4 5 6 7 8 |
# task function executed in a child worker process def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 |
Next, in the main process we can create the process pool with default arguments.
|
1 2 3 4 |
... # create the process pool with Pool() as pool: # ... |
We will then call the target function 10 times, each with three arguments.
Each call will return an AsyncResult object that provides a handle on each asynchronous task in the multiprocessing pool.
We can collect these objects in a list using a list comprehension.
For example:
|
1 2 3 |
... # issue multiple tasks each with multiple arguments async_results = [pool.apply_async(task, args=(i, i*2, i*3)) for i in range(10)] |
Next, we can get the return values from each issued task.
This can be achieved by calling the get() function on each AsyncResult object. This function will block until the associated function call finishes and returns a value.
This too can be achieved in a list comprehension.
For example:
|
1 2 3 |
... # retrieve the return value results results = [ar.get() for ar in async_results] |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of multiple arguments with pool.apply_async from time import sleep from random import random from multiprocessing import Pool # task function executed in a child worker process def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # issue multiple tasks each with multiple arguments async_results = [pool.apply_async(task, args=(i, i*2, i*3)) for i in range(10)] # retrieve the return value results results = [ar.get() for ar in async_results] |
Running the example first creates the process pool with a default configuration, using all CPU cores in the system.
Next, 10 tasks with multiple arguments are issued to the process pool and a list of AsyncResult objects are collected immediately.
Tasks begin executing, reporting the three arguments received, confirming that the target function was called with multiple arguments correctly.
Next, the return values from each issued task are collected into a list as the tasks completes.
|
1 2 3 4 5 6 7 8 9 10 |
Task 5, 10, 15. Task 3, 6, 9. Task 8, 16, 24. Task 6, 12, 18. Task 0, 0, 0. Task 4, 8, 12. Task 1, 2, 3. Task 7, 14, 21. Task 2, 4, 6. Task 9, 18, 27. |
Next, we will explore how to use the starmap() function to execute a target function in the multiprocessing pool that takes multiple arguments.
Free Python Multiprocessing Pool Course
Download your FREE Process Pool PDF cheat sheet and get BONUS access to my free 7-day crash course on the Process Pool API.
Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.
Multiple Arguments with Pool.starmap()
We can use the multiprocessing pool starmap() function to execute a target function that takes multiple arguments.
In this example we will define a target function that takes multiple arguments, then in the main process we will prepare a list of tuples, each that provides an argument to the target function, then issues tasks into the process pool to call our target function.
Firstly, we can define a target function that takes multiple arguments, blocks for a moment, reports a message then returns a combination of all three arguments.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# task function executed in a child worker process def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 |
Next, in the main process, we can create the multiprocessing pool with default arguments.
|
1 2 3 4 |
... # create the process pool with Pool() as pool: # ... |
Next, we can create an iterable that contains the arguments for each call to our target function.
|
1 2 3 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] |
Finally, we can call the starmap() function on the pool. This will issue 10 calls to the target task() function with multiple arguments and return an iterable of return values once all tasks have completed.
|
1 2 3 |
... # issue multiple tasks each with multiple arguments results = pool.starmap(task, args) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of multiple arguments with pool.starmap from time import sleep from random import random from multiprocessing import Pool # task function executed in a child worker process def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.starmap(task, args) |
Running the example first creates the process pool.
The arguments for tasks are prepared as a list of tuples. The list contains 10 tuples for the 10 calls to the target function and each tuple contains 3 values for the three arguments for each call to the task() function.
The tasks execute and report their arguments, confirming the multiple arguments were provided as intended and that the tasks executed successfully.
Finally, all tasks complete and an iterable of return values is returned to the main process.
|
1 2 3 4 5 6 7 8 9 10 |
Task 2, 4, 6. Task 3, 6, 9. Task 0, 0, 0. Task 7, 14, 21. Task 1, 2, 3. Task 4, 8, 12. Task 6, 12, 18. Task 5, 10, 15. Task 8, 16, 24. Task 9, 18, 27. |
Next, we will explore how we can call map() and unpack multiple arguments in the target function.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Multiple Arguments with Pool.map() and Unpack Arguments
We can explore how to modify the target function so that it takes a single argument and then unpacks this single argument into multiple arguments.
The map() function can then be called directly with a single argument for each call to the target function.
This approach requires that the target function can be modified.
Firstly, we can update the task() function from previous examples to take a single argument instead of multiple arguments.
The single argument may be a list or a tuple of multiple arguments.
Then, in the first line of the function it unpacks the single argument into 3 arguments.
The updated task() function with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 |
# task function executed in a child worker process def task(args): # unpack arguments arg1, arg2, arg3 = args # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 |
We can then prepare a list of tuples as arguments for the task function, as we did previously for the starmap() function.
Each tuple item in the list represents a call to the target function with one argument.
For example:
|
1 2 3 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] |
We can then call the multiprocessing pool map() function directly.
|
1 2 3 |
... # issue multiple tasks each with multiple arguments results = pool.map(task, args) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of multiple arguments with pool.map and unpack arguments from time import sleep from random import random from multiprocessing import Pool # task function executed in a child worker process def task(args): # unpack arguments arg1, arg2, arg3 = args # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task, args) |
Running the example first creates the process pool.
The arguments for tasks are prepared as a list of tuples. The list contains 10 tuples for the 10 calls to the target function and each tuple contains 3 values for the three values that will be unpacked in the task() function.
Each call to the task() function receives a single tuple argument. This is then unpacked into three arguments, then the function executes normally, blocking, reporting a message, then returning a value that combines the three arguments.
Finally, all tasks complete and an iterable of return values is returned to the main process.
|
1 2 3 4 5 6 7 8 9 10 |
Task 1, 2, 3. Task 0, 0, 0. Task 7, 14, 21. Task 8, 16, 24. Task 5, 10, 15. Task 3, 6, 9. Task 2, 4, 6. Task 9, 18, 27. Task 6, 12, 18. Task 4, 8, 12. |
Next, we will explore how to use a call Pool.map() with a wrapper function that will unpack a tuple of arguments and call the target function with multiple arguments.
Multiple Arguments with Pool.map() and Wrapper Function
We can explore how to call map() to execute our target function indirectly with a wrapper function that will unpack multiple arguments for us.
This approach allows us to use the Pool.map() function and to leave our target task function untouched.
Firstly, we can define the task() function that takes three arguments, blocks for a moment, reports a message then returns a return value.
|
1 2 3 4 5 6 7 8 |
# task function executed in a child worker process def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 |
We can then define a wrapper function that will be called by map().
This function takes a collection of arguments such as a list or tuple.
The function then unpacks these arguments and calls the target function, adding a level of indirection between map() that takes one argument per task and the target function that in this case requires three arguments.
We could unpack the arguments manually and then call the function.
For example:
|
1 2 3 4 5 6 |
# wrapper function for task def task_wrapper(args): # unpack arguments arg1, arg2, arg3 = args # call the target function task(arg1, arg2, arg3) |
However, this can be achieved in a single line using the star operator (*) to unpack the arguments in the call to the target function.
For example:
|
1 2 3 4 |
# wrapper function for task def task_wrapper(args): # call task() and unpack the arguments return task(*args) |
We can then prepare the arguments as a list of tuples, as we did previously, then use the map() function to call the wrapper function.
|
1 2 3 4 5 6 7 |
... # create the process pool with Pool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task_wrapper, args) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of multiple arguments with pool.map and wrapper function from time import sleep from random import random from multiprocessing import Pool # task function executed in a child worker process def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.', flush=True) # return a result return arg1 + arg2 + arg3 # wrapper function for task def task_wrapper(args): # call task() and unpack the arguments return task(*args) # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task_wrapper, args) |
Running the example first creates the process pool.
The arguments for tasks are prepared as a list of tuples. The list contains 10 tuples for the 10 calls to the target function and each tuple contains 3 values for the three values that will be unpacked in the task_wrapper() function.
The task_wrapper() is called each time with one argument, a tuple. It in turn calls the task() function with the unpacked arguments.
Each call the task() function receives all three arguments executing normally, blocking, reporting a message, then returning a value that combines the three arguments.
Finally, all tasks complete and an iterable of return values is returned to the main process.
|
1 2 3 4 5 6 7 8 9 10 |
Task 1, 2, 3. Task 8, 16, 24. Task 4, 8, 12. Task 3, 6, 9. Task 7, 14, 21. Task 6, 12, 18. Task 9, 18, 27. Task 5, 10, 15. Task 2, 4, 6. Task 0, 0, 0. |
Further Reading
This section provides additional resources that you may find helpful.
Books
- Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Pool Class API Cheat Sheet
I would also recommend specific chapters from these books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPool: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to execute target functions that take multiple arguments in the multiprocessing pool.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Lexi Eddington on Unsplash
Do you have any questions?