Last Updated on September 12, 2022
You can expose and handle suppressed exceptions in the ThreadPoolExecutor by getting the results from the task.
In this tutorial, you will discover how to uncover and handle suppressed exceptions in the ThreadPoolExecutor.
Let’s get started.
ThreadPoolExecutor Fails Silently
The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.
You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread.
Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.
You can also submit tasks by calling the map() function and specifying the name of the function to execute and the iterable of items to which your function will be applied.
Tasks executing concurrently in the thread pool can fail by raising an exception.
When tasks in the ThreadPoolExecutor raise an exception, it can look like they fail silently.
Why does the ThreadPoolExecutor suppress exceptions?
Run loops using all CPUs, download your FREE book to learn how.
ThreadPoolExecutor Will Catch Exceptions for You
The ThreadPoolExecutor intentionally suppresses exceptions raised by tasks.
This is to ensure that the failure of a task does not result in the failure of a worker thread in the thread pool. An exception raised by a task will not halt your program.
ThreadPoolExecutor Suppresses Exceptions
If the ThreadPoolExecutor did not suppress exceptions raised by tasks, then a failure in a task will cause the worker thread to unwind and close, meaning that the thread pool would have one fewer worker threads.
For example, if you issue tasks to the thread pool using the submit() function that returns a Future object, and you do not get the result from the task by calling result() on the Future, then you will have no knowledge that the task failed.
|
1 2 3 4 5 6 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit tasks future = executor.submit(task) # no idea of the outcome of the task... |
Alternatively, if you issue tasks to the thread pool by calling the map() function, but do not iterate over the results, again, you will have no knowledge that the task failed.
|
1 2 3 4 5 6 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit tasks results = executor.map(task, items) # no idea of the outcome of the tasks... |
How to Access Suppressed Exceptions
You can discover the outcome of a task issued to the thread pool using submit() in one of two ways.
Firstly, you can call the result() function on the Future object. This will re-raise any exception raised within the task, allowing you to catch and handle it if you choose.
|
1 2 3 4 5 6 7 8 9 10 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit tasks future = executor.submit(task) # handle exceptions raised by the task try: result = future.result() except: # handle task failure... |
Alternatively, you can call the exception() function on the Future object that will return the exception raised during the execution of the task or None if no exception was raised.
The exception() function will block until the task is done (successfully or otherwise), and you can specify a timeout in seconds for how long you are willing to wait for the task to be done.
|
1 2 3 4 5 6 7 8 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit tasks future = executor.submit(task) # handle failure of the task if future.exception() != None: # handle exception... |
Similarly, exceptions raised by a task issued by calling the map() function will be raised when iterating the results returned from calling the map() function, unlike the submit() function; for example:
|
1 2 3 4 5 6 7 8 9 10 11 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit tasks results = executor.map(task, items) # handle an exception in any result try: for result in results: # do something... except: # handle exception |
Now that we are familiar with why the ThreadPoolExecutor will suppress exceptions and how to gain access to those exceptions, let’s look at some worked examples.
Get Suppressed Exceptions From submit()
Let’s take a look at an example of the ThreadPoolExecutor suppressing an exception when calling submit(), then how we can uncover the exception that occurred.
First, let’s define a simple task function that fails by raising an exception.
|
1 2 3 4 5 6 |
# mock work task that sleeps for a moment and raises an exception def task(): sleep(1) raise('Something bad happened') # never gets here return 100 |
We can then start a thread pool and submit the task in the pool for execution.
|
1 2 3 4 5 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit a task future = executor.submit(task) |
Tying this together, the complete example is listed below.
We expect the task to fail, but we also expect to have no knowledge of this failure given that the ThreadPoolExecutor will suppress the exception.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of the thread pool suppressing exceptions in tasks with submit from time import sleep from concurrent.futures import ThreadPoolExecutor # mock work task that sleeps for a moment and raises an exception def task(): sleep(1) raise('Something bad happened') # never gets here return 100 # create the thread pool with ThreadPoolExecutor() as executor: # submit a task future = executor.submit(task) print('All done') |
Running the example, the thread pool is created and the task is submitted.
The program reports all tasks are done.
The problem is that we know the task failed and that it failed silently. The user has no way of knowing that the task has failed.
|
1 |
All done |
We can update the example to get the result from the target task function by calling the result() function on the Future object associated with the task.
This will expose the exception which will be re-raised by the thread pool. We can then choose to handle it.
|
1 2 3 4 5 6 7 8 |
... # handle any exception in the task try: # get the result from the task result = future.result() print(f'Task result: {result}') except: print('The task failed') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of the thread pool suppressing exceptions in tasks with submit from time import sleep from concurrent.futures import ThreadPoolExecutor # mock work task that sleeps for a moment and raises an exception def task(): sleep(1) raise('Something bad happened') # never gets here return 100 # create the thread pool with ThreadPoolExecutor() as executor: # submit a task future = executor.submit(task) # handle any exception in the task try: # get the result from the task result = future.result() print(f'Task result: {result}') except: print('The task failed') print('All done') |
Running the example creates the thread pool and executes the task.
The task fails as we expect. In this case, we attempt to get the result of the task and because the task failed, the thread pool re-raises the exception, which we then handle and report a message.
|
1 2 |
The task failed All done |
Alternatively, we can check if a task has failed by calling the exception() function.
This will return any Exception raised during the execution of the task or None if no exception was raised.
This function may be more helpful to conditionally take action if the task fails and useful in the case where the task does not return a value.
For example:
|
1 2 3 4 |
... # check if the task failed if future.exception(): print('The task failed') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of the thread pool suppressing exceptions in tasks with submit from time import sleep from concurrent.futures import ThreadPoolExecutor # mock work task that sleeps for a moment and raises an exception def task(): sleep(1) raise('Something bad happened') # never gets here return 100 # create the thread pool with ThreadPoolExecutor() as executor: # submit a task future = executor.submit(task) # check if the task failed if future.exception(): print('The task failed') print('All done') |
Running the example, the thread pool is started as per normal and the task is issued for execution.
The task fails and we check for any exceptions raised during the exception of the task. An exception is returned and we report a message that the task failed.
|
1 2 |
The task failed All done |
Free Python ThreadPoolExecutor Course
Download your FREE ThreadPoolExecutor PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPoolExecutor API.
Discover how to use the ThreadPoolExecutor class including how to configure the number of workers and how to execute tasks asynchronously.
Get Suppressed Exceptions From map()
We can also look at an example of the ThreadPoolExecutor suppressing an exception when calling the map() function, then how we can uncover the exception that occurred.
Firstly, we can update the target task function to take an argument, a requirement of using the map() function.
|
1 2 3 4 5 6 |
# mock work task that sleeps for a moment and raises an exception def task(value): sleep(1) raise('Something bad happened') # never gets here return value + 1 |
We can then create the thread pool and call the map() function with the target task() function and a list of items.
|
1 2 3 4 5 |
... # create the thread pool with ThreadPoolExecutor() as executor: # submit the tasks result = executor.map(task, range(5)) |
Tying this together, the complete example is listed below.
We would expect the tasks to be issued to the thread pool and for each task to fail with a raised Exception. We would also expect that running the program will provide no indication that an exception occurred and that all tasks failed.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of the thread pool suppressing exceptions in tasks with map from time import sleep from concurrent.futures import ThreadPoolExecutor # mock work task that sleeps for a moment and raises an exception def task(value): sleep(1) raise('Something bad happened') # never gets here return value + 1 # create the thread pool with ThreadPoolExecutor() as executor: # submit the tasks result = executor.map(task, range(5)) print('All done') |
Running the example, the thread pool is created and five tasks are submitted for execution.
The program ends successfully with no indication to the user that the tasks failed.
|
1 |
All done |
We can update the example to iterate the results of the tasks, which will expose the failure of those tasks.
One approach is to wrap the iteration of results in a try-except block to handle the case that any task in the collection failed.
For example:
|
1 2 3 4 5 6 7 8 |
... # handle a task failure try: # submit tasks an iterate results for result in executor.map(task, range(5)): print(result) except: print('A task failed') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of the thread pool suppressing exceptions in tasks with map from time import sleep from concurrent.futures import ThreadPoolExecutor # mock work task that sleeps for a moment and raises an exception def task(value): sleep(1) raise('Something bad happened') # never gets here return value + 1 # create the thread pool with ThreadPoolExecutor() as executor: # handle a task failure try: # submit tasks an iterate results for result in executor.map(task, range(5)): print(result) except: print('A task failed') print('All done') |
Running the example creates the thread pool and submits all five tasks.
While iterating the results, the first task fails by raising an exception, which is caught and an error message is reported.
|
1 2 |
A task failed All done |
Alternatively, we may want to manually iterate the iterable returned from map() and handle any exceptions along the way.
You might recall from Python basics that we can manually iterate an iterable by calling the next() function on it until a StopIteration is raised.
If an exception is raised by one of the tasks, it will abort the iterable and the StopIteration will also be raised.
Therefore, we can write an infinite while loop and manually iterate over the results of map() until an exception is raised. If the exception is a StopIteration, we can break the while loop, whereas if the exception is anything else, we can handle it, after which the iterable will throw a StopIteration and we can break out of the loop.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
... # submit tasks an iterate results results = executor.map(task, range(5)) # iterate over all results while True: # handle a task failure or end of the iterable try: # manually retrieve the next result result = next(results) print(result) except StopIteration: break except: # a task failed print('A task failed') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of the thread pool suppressing exceptions in tasks with map from time import sleep from concurrent.futures import ThreadPoolExecutor # mock work task that sleeps for a moment and raises an exception def task(value): sleep(1) raise('Something bad happened') # never gets here return value + 1 # create the thread pool with ThreadPoolExecutor() as executor: # submit tasks an iterate results results = executor.map(task, range(5)) # iterate over all results while True: # handle a task failure or end of the iterable try: # manually retrieve the next result result = next(results) print(result) except StopIteration: break except: # a task failed print('A task failed') print('All done') |
Running the example creates the thread pool and submits all tasks for execution.
We then begin iterating the results. The first task fails and raises an exception that we handle and then report. This breaks the iterable and we stop iterating the results.
Note that all five tasks were submitted and all five failed in this case, but only the first failure is reported. If we want to report on the failure of each task separately, we must use the submit() function.
|
1 2 |
A task failed All done |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
- Concurrent Futures API Interview Questions
- ThreadPoolExecutor Class API Cheat Sheet
I also recommend specific chapters from the following books:
- Effective Python, Brett Slatkin, 2019.
- See Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPoolExecutor: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Takeaways
You now know how to uncover and handle suppressed exceptions in the ThreadPoolExecutor.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Greg Trowman on Unsplash
Do you have any questions?