Last Updated on September 12, 2022
You may encounter one among a number of common errors when using the ThreadPoolExecutor in Python.
These errors are typically easy to identify and often involve a quick fix.
In this tutorial, you will discover the common errors when using thread pools in Python and how to fix each in turn.
Let’s get started.
Common Errors When Using ThreadPoolExecutor
The ThreadPoolExecutor is a flexible and powerful thread pool for executing ad hoc tasks in an asynchronous manner.
When you are getting started with the ThreadPoolExecutor in Python, you may encounter one of many common errors.
These errors are typically made because of bugs introduced by copy-and-pasting code, or from a slight misunderstanding in how the ThreadPoolExecutor works.
We will take a closer look at some of the more common errors made when using the ThreadPoolExecutor; they are:
- Error 1: Using a Function Call in submit()
- Error 2: Using a Function Call in map()
- Error 3: Incorrect Function Signature for map()
- Error 4: Incorrect Function Signature for Future Callbacks
Do you have an error using the ThreadPoolExecutor?
Let me know in the comments so I can recommend a fix and add the case to this tutorial.
Run loops using all CPUs, download your FREE book to learn how.
Error 1: Using a Function Call in submit()
A common error is to call your function when using the submit() function.
For example:
|
1 2 3 |
... # submit the task future = executor.submit(task()) |
A complete example with this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# SuperFastPython.com # example of calling submit with a function call from time import sleep from random import random from concurrent.futures import ThreadPoolExecutor # custom task that will sleep for a variable amount of time def task(): # sleep for less than a second sleep(random()) return 'all done' # start the thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(task()) # get the result result = future.result() print(result) |
Running this example will fail with an error.
|
1 2 3 4 |
Traceback (most recent call last): ... result = self.fn(*self.args, **self.kwargs) TypeError: 'str' object is not callable |
You can fix the error by updating the call to submit() to take the name of your function and any arguments, instead of calling the function in the call to submit.
For example:
|
1 2 3 |
... # submit the task future = executor.submit(task) |
Error 2: Using a Function Call in map()
A common error is to call your function when using the map() function.
For example:
|
1 2 3 4 |
... # submit all tasks for result in executor.map(task(), range(5)): print(result) |
A complete example with this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of calling map with a function call from time import sleep from random import random from concurrent.futures import ThreadPoolExecutor # custom task that will sleep for a variable amount of time def task(value): # sleep for less than a second sleep(random()) return value # start the thread pool with ThreadPoolExecutor() as executor: # submit all tasks for result in executor.map(task(), range(5)): print(result) |
Running the example results in a TypeError:
|
1 2 3 |
Traceback (most recent call last): ... TypeError: task() missing 1 required positional argument: 'value' |
This error can be fixed by changing the call to map() to pass the name of the target task function instead of a call to the function.
|
1 2 3 4 |
... # submit all tasks for result in executor.map(task, range(5)): print(result) |
Free Python ThreadPoolExecutor Course
Download your FREE ThreadPoolExecutor PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPoolExecutor API.
Discover how to use the ThreadPoolExecutor class including how to configure the number of workers and how to execute tasks asynchronously.
Error 3: Incorrect Function Signature for map()
Another common error when using map() is to provide no second argument to function, e.g. the iterable.
For example:
|
1 2 3 4 |
... # submit all tasks for result in executor.map(task): print(result) |
A complete example with this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of calling map without an iterable from time import sleep from random import random from concurrent.futures import ThreadPoolExecutor # custom task that will sleep for a variable amount of time def task(value): # sleep for less than a second sleep(random()) return value # start the thread pool with ThreadPoolExecutor() as executor: # submit all tasks for result in executor.map(task): print(result) |
Running the example does not issue any tasks to the thread pool as there was no iterable for the map() function to iterate over.
In this case, no output is displayed.
The fix involves providing an iterable in the call to map() along with your function name.
|
1 2 3 4 |
... # submit all tasks for result in executor.map(task, range(5)): print(result) |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Error 4: Incorrect Function Signature for Future Callbacks
Another common error is to forget to include the Future in the signature for the callback function registered with a Future object.
For example:
|
1 2 3 4 |
... # callback function to call when a task is completed def custom_callback(): print('Custom callback was called') |
A complete example with this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of the wrong signature on the callback function from time import sleep from concurrent.futures import ThreadPoolExecutor # callback function to call when a task is completed def custom_callback(): print('Custom callback was called') # mock task that will sleep for a moment def work(): sleep(1) return 'Task is done' # create a thread pool with ThreadPoolExecutor() as executor: # execute the task future = executor.submit(work) # add the custom callback future.add_done_callback(custom_callback) # get the result result = future.result() print(result) |
Running this example will result in an error when the callback is called by the thread pool.
|
1 2 3 4 5 |
Task is done exception calling callback for <Future at 0x10a05f190 state=finished returned str> Traceback (most recent call last): ... TypeError: custom_callback() takes 0 positional arguments but 1 was given |
Fixing this error involves updating the signature of your callback function to include the future object.
|
1 2 3 4 |
... # callback function to call when a task is completed def custom_callback(future): print('Custom callback was called') |
Further Reading
This section provides additional resources that you may find helpful.
Books
- ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
- Concurrent Futures API Interview Questions
- ThreadPoolExecutor Class API Cheat Sheet
I also recommend specific chapters from the following books:
- Effective Python, Brett Slatkin, 2019.
- See Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPoolExecutor: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Takeaways
You now know about the common errors when using the ThreadPoolExecutor in Python.
Do you have any questions about the common errors using thread pools?
Ask your question in the comments below and I will do my best to answer.
Photo by Andhika Soreng on Unsplash
Do you have any questions?