Last Updated on October 29, 2022
The ThreadPool map() method cannot be used directly with a target function that takes multiple arguments.
Instead, you need to use an alternate method like starmap() or a workaround like a wrapper function.
In this tutorial you will discover how to call the ThreadPool map() function with multiple arguments indirectly and how to use alternate approaches to execute target functions that take multiple arguments.
Let’s get started.
Need to Use ThreadPool map() With Multiple Arguments
The built-in map() function will call a function to each item on an iterable.
For example:
|
1 2 3 4 |
... # call function for each item in an iterable for result in map(task, items): print(result) |
A benefit of the built-in map() function is that it can take multiple iterables. Each iterable will be traversed in lock-step, yielding an argument to the function.
For example:
|
1 2 3 4 |
... # call function for each item in an iterable for result in map(task, items1, items2): print(result) |
The ThreadPool class provides a concurrent version of the map() function. Each call to the target function will be issued to the ThreadPool and executed by a worker thread.
For example:
|
1 2 3 4 5 6 |
... # create the thread pool with ThreadPool() as pool: # call function for each item in an iterable concurrently for result in pool.map(task, items): print(result) |
You can learn more about how to use the ThreadPool map() method in the tutorial:
A limitation of the ThreadPool map() method is that it only takes a single iterable of arguments for the target function.
This means it can only be used to call target functions that take one argument.
How can we use the ThreadPool map() method with multiple arguments?
Run loops using all CPUs, download your FREE book to learn how.
How to Use ThreadPool map() With Multiple Arguments
There are many ways that we can use the ThreadPool map() method with a target function that takes multiple arguments.
We will look at 4 common approaches, they are:
- Use apply_async() instead.
- Use starmap() instead.
- Change the target function to unpack arguments.
- Use a wrapper function to unpack arguments.
Let’s take a closer look at each approach in turn.
Use ThreadPool apply_async() Instead
The apply() method will execute a single function call in the ThreadPool.
Importantly, it allows a function to be called that takes zero, one, or multiple arguments. As such, it can be used in a loop instead of the map() function.
You can learn more about the ThreadPool apply() method in the tutorial:
A limitation of apply() is that it will block for each call until the target function is complete. As such, it offers no benefit of concurrency.
Instead, we can use the apply_async() function, which is an asynchronous version of the apply() function. We can specify multiple arguments to our target function using the “args” keyword and providing a tuple of arguments.
For example:
|
1 2 3 |
... # issue a task into the thread pool with multiple arguments async_result = pool.apply_async(task, args=(arg1, arg2, arg3)) |
It issues the call to the target function into the ThreadPool and returns immediately with an AsyncResult object.
These objects can be collected using a list comprehension.
For example:
|
1 2 3 |
... # issue multiple tasks each with multiple arguments async_results = [pool.apply_async(task, args=(i, i*2, i*3)) for i in range(10)] |
We can then call the get() function on each AsyncResult to retrieve the return value as needed.
This too can be performed in a list comprehension.
For example:
|
1 2 3 |
... # retrieve the return value results results = [ar.get() for ar in async_results] |
You can learn more about the apply_async() function in the tutorial:
Use ThreadPool starmap() Instead
The ThreadPool starmap() method will call the target function with multiple arguments.
As such it can be used instead of the map() method. This is probably the preferred approach for executing a target function in the ThreadPool that takes multiple arguments.
The starmap() method takes the name of the target function and an iterable, like the map() function.
The difference is that each item in the iterable must be an iterable that yields results for each call to the target function.
The easiest way to think of this is to have a list where each item in the list is a tuple of arguments for one call to the target function.
For example:
|
1 2 3 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] |
This can then be passed directly to the starmap() function.
For example:
|
1 2 3 |
... # issue multiple tasks each with multiple arguments results = pool.starmap(task, args) |
You can learn more about the starmap() method in the tutorial:
Change The Target Function to Unpack Arguments
The target function can be changed.
Instead of taking multiple arguments, it can be modified to take a single argument, such as a list or tuple of arguments.
The function can then unpack these arguments and operate as per normal.
For example:
|
1 2 3 4 |
# task function executed in a worker threads def task(args): # unpack arguments arg1, arg2, arg3 = args |
The ThreadPool map() method can then be used directly passing a list or tuple of arguments, simulating a call that takes multiple arguments.
For example:
|
1 2 3 4 5 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task, args) |
This requires that you are able to change the target function, which might not be the case in all applications.
Use a Wrapper Function to Unpack Arguments
Unwrapping arguments is an easy fix.
An alternate approach to using this fix is to use a wrapper or proxy function that takes a list or tuple of multiple arguments and is called by map() instead of the true target function.
For example:
|
1 2 3 4 5 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task_wrapper, args) |
The wrapper function will unpack the arguments and then call the target function with multiple arguments.
For example:
|
1 2 3 4 |
# wrapper function for task def task_wrapper(args): # call task() and unpack the arguments return task(*args) |
Recall that the star operator (*) will unpack a list or tuple for us.
This approach is appropriate for cases where you have control over the function called by map() but cannot or do not want to change the target function.
Now that we know how to use the ThreadPool map() with multiple arguments, let’s look at some worked examples.
Multiple Arguments with ThreadPool apply_async()
We can explore how to use apply_async() instead of map() to call a target function with multiple arguments.
In this example, we will define a target function that takes multiple arguments, blocks for a moment to simulate doing work, reports a message then returns a value that combines the arguments. The caller will create the ThreadPool, issue the tasks, then retrieve the return values.
Firstly, we can define the task() function that takes multiple arguments, blocks, reports a message then returns a return value.
|
1 2 3 4 5 6 7 8 |
# task function executed in a worker thread def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 |
Next, in the main thread we can create the ThreadPool with default arguments.
|
1 2 3 4 |
... # create the thread pool with ThreadPool() as pool: # ... |
We will then call the target function 10 times, each with three arguments.
Each call will return an AsyncResult object that provides a handle on each asynchronous task in the ThreadPool.
We can collect these objects in a list using a list comprehension.
For example:
|
1 2 3 |
... # issue multiple tasks each with multiple arguments async_results = [pool.apply_async(task, args=(i, i*2, i*3)) for i in range(10)] |
Next, we can get the return values from each issued task.
This can be achieved by calling the get() function on each AsyncResult object. This function will block until the associated function call finishes and returns a value.
This too can be achieved in a list comprehension.
For example:
|
1 2 3 |
... # retrieve the return value results results = [ar.get() for ar in async_results] |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of multiple arguments with apply_async from time import sleep from random import random from multiprocessing.pool import ThreadPool # task function executed in a worker thread def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 # protect the entry point if __name__ == '__main__': # create the thread pool with ThreadPool() as pool: # issue multiple tasks each with multiple arguments async_results = [pool.apply_async(task, args=(i, i*2, i*3)) for i in range(10)] # retrieve the return value results results = [ar.get() for ar in async_results] |
Running the example first creates the ThreadPool with a default configuration, using all CPU cores in the system.
Next, 10 tasks with multiple arguments are issued to the ThreadPool and a list of AsyncResult objects are collected immediately.
Tasks begin executing, reporting the three arguments received, confirming that the target function was called with multiple arguments correctly.
Next, the return values from each issued task are collected into a list as the tasks completes.
|
1 2 3 4 5 6 7 8 9 10 |
Task 5, 10, 15. Task 0, 0, 0. Task 9, 18, 27. Task 1, 2, 3. Task 2, 4, 6. Task 7, 14, 21. Task 3, 6, 9. Task 6, 12, 18. Task 4, 8, 12. Task 8, 16, 24. |
Next, we will explore how to use the starmap() function to execute a target function in the ThreadPool that takes multiple arguments.
Free Python ThreadPool Course
Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
Multiple Arguments with ThreadPool starmap()
We can use the ThreadPool starmap() method to execute a target function that takes multiple arguments.
In this example we will define a target function that takes multiple arguments, then in the main thread we will prepare a list of tuples, each that provides an argument to the target function, then issues tasks into the ThreadPool to call our target function.
Firstly, we can define a target function that takes multiple arguments, blocks for a moment, reports a message then returns a combination of all three arguments.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# task function executed in a worker thread def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 |
Next, in the main thread, we can create the ThreadPool with default arguments.
|
1 2 3 4 |
... # create the thread pool with ThreadPool() as pool: # ... |
Next, we can create an iterable that contains the arguments for each call to our target function.
|
1 2 3 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] |
Finally, we can call the starmap() function on the pool. This will issue 10 calls to the target task() function with multiple arguments and return an iterable of return values once all tasks have completed.
|
1 2 3 |
... # issue multiple tasks each with multiple arguments results = pool.starmap(task, args) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of multiple arguments with starmap from time import sleep from random import random from multiprocessing.pool import ThreadPool # task function executed in a worker thread def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 # protect the entry point if __name__ == '__main__': # create the thread pool with ThreadPool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.starmap(task, args) |
Running the example first creates the ThreadPool.
The arguments for tasks are prepared as a list of tuples. The list contains 10 tuples for the 10 calls to the target function and each tuple contains 3 values for the three arguments for each call to the task() function.
The tasks execute and report their arguments, confirming the multiple arguments were provided as intended and that the tasks executed successfully.
Finally, all tasks complete and an iterable of return values is returned to the main thread.
|
1 2 3 4 5 6 7 8 9 10 |
Task 0, 0, 0. Task 3, 6, 9. Task 5, 10, 15. Task 6, 12, 18. Task 4, 8, 12. Task 2, 4, 6. Task 9, 18, 27. Task 8, 16, 24. Task 7, 14, 21. Task 1, 2, 3. |
Next, we will explore how we can call map() and unpack multiple arguments in the target function.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Multiple Arguments with ThreadPool map() and Unpack Arguments
We can explore how to modify the target function so that it takes a single argument and then unpacks this single argument into multiple arguments.
The map() function can then be called directly with a single argument for each call to the target function.
This approach requires that the target function can be modified.
Firstly, we can update the task() function from previous examples to take a single argument instead of multiple arguments.
The single argument may be a list or a tuple of multiple arguments.
Then, in the first line of the function it unpacks the single argument into 3 arguments.
The updated task() function with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 |
# task function executed in a worker thread def task(args): # unpack arguments arg1, arg2, arg3 = args # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 |
We can then prepare a list of tuples as arguments for the task function, as we did previously for the starmap() function.
Each tuple item in the list represents a call to the target function with one argument.
For example:
|
1 2 3 |
... # prepare arguments args = [(i, i*2, i*3) for i in range(10)] |
We can then call the ThreadPool map() function directly.
|
1 2 3 |
... # issue multiple tasks each with multiple arguments results = pool.map(task, args) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of multiple arguments with map and unpack arguments from time import sleep from random import random from multiprocessing.pool import ThreadPool # task function executed in a worker thread def task(args): # unpack arguments arg1, arg2, arg3 = args # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 # protect the entry point if __name__ == '__main__': # create the thread pool with ThreadPool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task, args) |
Running the example first creates the ThreadPool.
The arguments for tasks are prepared as a list of tuples. The list contains 10 tuples for the 10 calls to the target function and each tuple contains 3 values for the three values that will be unpacked in the task() function.
Each call to the task() function receives a single tuple argument. This is then unpacked into three arguments, then the function executes normally, blocking, reporting a message, then returning a value that combines the three arguments.
Finally, all tasks complete and an iterable of return values is returned to the main thread.
|
1 2 3 4 5 6 7 8 9 10 |
Task 6, 12, 18. Task 7, 14, 21. Task 5, 10, 15. Task 0, 0, 0. Task 8, 16, 24. Task 4, 8, 12. Task 9, 18, 27. Task 3, 6, 9. Task 2, 4, 6. Task 1, 2, 3. |
Next, we will explore how to use a call ThreadPool map() with a wrapper function that will unpack a tuple of arguments and call the target function with multiple arguments.
Multiple Arguments with ThreadPool map() and Wrapper Function
We can explore how to call map() to execute our target function indirectly with a wrapper function that will unpack multiple arguments for us.
This approach allows us to use the map() method and to leave our target task function untouched.
Firstly, we can define the task() function that takes three arguments, blocks for a moment, reports a message then returns a return value.
|
1 2 3 4 5 6 7 8 |
# task function executed in a worker thread def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 |
We can then define a wrapper function that will be called by map().
This function takes a collection of arguments such as a list or tuple.
The function then unpacks these arguments and calls the target function, adding a level of indirection between map() that takes one argument per task and the target function that in this case requires three arguments.
We could unpack the arguments manually and then call the function.
For example:
|
1 2 3 4 5 6 |
# wrapper function for task def task_wrapper(args): # unpack arguments arg1, arg2, arg3 = args # call the target function task(arg1, arg2, arg3) |
However, this can be achieved in a single line using the star operator (*) to unpack the arguments in the call to the target function.
For example:
|
1 2 3 4 |
# wrapper function for task def task_wrapper(args): # call task() and unpack the arguments return task(*args) |
We can then prepare the arguments as a list of tuples, as we did previously, then use the map() method to call the wrapper function.
|
1 2 3 4 5 6 7 |
... # create the thread pool with ThreadPool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task_wrapper, args) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of multiple arguments with map and wrapper function from time import sleep from random import random from multiprocessing.pool import ThreadPool # task function executed in a worker thread def task(arg1, arg2, arg3): # block for a moment sleep(random()) # report values print(f'Task {arg1}, {arg2}, {arg3}.') # return a result return arg1 + arg2 + arg3 # wrapper function for task def task_wrapper(args): # call task() and unpack the arguments return task(*args) # protect the entry point if __name__ == '__main__': # create the thread pool with ThreadPool() as pool: # prepare arguments args = [(i, i*2, i*3) for i in range(10)] # issue multiple tasks each with multiple arguments results = pool.map(task_wrapper, args) |
Running the example first creates the ThreadPool.
The arguments for tasks are prepared as a list of tuples. The list contains 10 tuples for the 10 calls to the target function and each tuple contains 3 values for the three values that will be unpacked in the task_wrapper() function.
The task_wrapper() is called each time with one argument, a tuple. It in turn calls the task() function with the unpacked arguments.
Each call the task() function receives all three arguments executing normally, blocking, reporting a message, then returning a value that combines the three arguments.
Finally, all tasks complete and an iterable of return values is returned to the main thread.
|
1 2 3 4 5 6 7 8 9 10 |
Task 7, 14, 21. Task 2, 4, 6. Task 6, 12, 18. Task 8, 16, 24. Task 4, 8, 12. Task 5, 10, 15. Task 1, 2, 3. Task 9, 18, 27. Task 3, 6, 9. Task 0, 0, 0. |
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python ThreadPool Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- ThreadPool PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPool: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
APIs
References
Takeaways
You now know how to execute target functions that take multiple arguments in the ThreadPool.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Ibrahim Rifath on Unsplash
Do you have any questions?