Last Updated on September 12, 2022
You can handle exceptions in the ThreadPoolExecutor raised by tasks by catching them when getting the result.
In this tutorial, you will discover how to handle exceptions in the ThreadPoolExecutor.
Let’s get started.
Need to Handle Exceptions Raised by Tasks
The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.
You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread.
Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.
You can also submit tasks by calling the map() function and specifying the name of the function to execute and the iterable of items to which your function will be applied.
An exception raised in a single threaded program must be caught, otherwise it will unwind the main thread.
Unwinding a thread means that the exception is propagated up to the top of the stack of function calls until it reaches the top level at which it will cause the main thread to exit and close the program.
The ThreadPoolExecutor can be used to execute arbitrary functions.
Each task submitted to the thread pool is picked-up by a worker thread and executed. While executing, the function may raise an exception.
If uncaught, does the exception may unwind the worker thread and it permanently unavailable in the thread pool?
Additionally, we may need to know if a task raised an exception when getting the result so that we can take appropriate action, such as submit a follow-up task or perhaps re-try the task.
How do you handle the exception raised by a task in a ThreadPoolExecutor?
Run loops using all CPUs, download your FREE book to learn how.
How to Handle Exceptions in Tasks
An exception raised by a target function executed by a worker thread will not unwind the worker thread.
The thread pool is robust to target functions that raise exceptions that are not caught. The exception will be caught by the thread pool automatically, and the worker thread will remain available for executing tasks.
There are three ways you may handle exceptions raised by a target function, they are:
- Handle exceptions when getting the task result.
- Check for raised exceptions.
- Handle exceptions within the task function.
Let’s take a closer look at each in turn.
Handle Exceptions When Getting the Result
You can handle exceptions raised within a task when getting the result from the task.
Recall that when you submit a task to the thread pool, you will receive a Future object in return.
When you wish to retrieve the result from the task, you can call the result() function in the Future for the task.
If the task is completed successfully, it will return the result from your function or None if your function does not return a result.
If your target task function raises an exception during execution, the exception will be raised again (re-raised) when calling the result() function, where you may need to handle it.
|
1 2 3 4 5 6 |
... # handle a possible exception raised in a task try: result = future.result() exception: # handle exception... |
The failure of the task will not break the worker thread and will not break the thread pool. The worker thread that executed the task can be reused for additional tasks in the future without any problem.
Check for a Raised Exception
Alternatively, you can call the exception() function on the task.
This will return once the task has completed and will contain the exception raised during the execution of the task, or None if no exception was raised
|
1 2 3 |
... # get the exception raised by the task exception = future.exception() |
Both the result() and exception() functions on the Future object allow you to specify a timeout argument, which is the time in seconds you are willing to wait for a result or an exception respectively if the task is scheduled or running, before giving up.
Handle Exceptions in the Task Function
A third option is to catch the exception within the target task function itself and handle it.
This approach may be desired if you are submitting tasks to the thread pool via the map() function, in which case you will not have a Future object from which to call result() or exception() to handle the exception.
|
1 2 3 4 5 6 |
# target task function def work() try: # do something... exception: # handle exception... |
The downside of this approach is that any recipient of the result from the task, e.g. callers waiting on the result, may not be aware that an exception occurred within the task.
You may need to log the exception within the task and perhaps pass on the failure within the task via a return value from the target task function.
Now that we know how to handle an exception within a task executed by a ThreadPoolExecutor, let’s look at some worked examples.
Example of Not Handling the Exception
Before we explore how to handle an exception raised by a task, let’s explore what happens if we don’t handle it.
First, let’s define a task that sleeps for a moment, then raises an exception.
|
1 2 3 4 5 |
# mock task that will sleep for a moment def work(): sleep(1) raise Exception('Something bad happened') return "Task is done" |
Next, we can start a thread pool and submit the task for execution and receive a Future object in return.
|
1 2 3 4 5 |
... # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) |
We will then not get the result from the task, and instead will wait around for a moment.
We want to see if the exception raised by the task has any effect on the thread pool or the main thread.
|
1 2 3 |
... # wait around for a moment sleep(2) |
Tying this all together, the complete example of executing a task that raises an exception and not handling the exception is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# SuperFastPython.com # example of not handling an exception raised by task from time import sleep from concurrent.futures import ThreadPoolExecutor # task function that blocks for a moment def work(): sleep(1) raise Exception('Something bad happened') return "Task is done" # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) # wait around for a moment sleep(2) print('All done') |
Running the example will start the thread pool and submit the task.
The task will block for a moment and then raises an exception.
The main thread waits around for a moment, then the thread pool is closed and the main thread exits.
This shows that if we don’t attempt to get the result from the task, we have no idea whether the task succeeded or failed. It looks like the task failed silently.
Importantly, it shows that the exception raised by the task function did not unwind the worker thread or the main thread.
|
1 |
All done |
We can update the example to attempt to get the result, again, without handling the exception.
|
1 2 3 4 |
... # get the result from the task result = future.result() print(result) |
Tying this together, the complete example of attempting to get the result from a task that we know has raised an exception is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# SuperFastPython.com # example of attempting to get the result from a task that has raised an exception from time import sleep from concurrent.futures import ThreadPoolExecutor # task function that blocks for a moment def work(): sleep(1) raise Exception('Something bad happened') return "Task is done" # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) # get the result from the task result = future.result() print(result) print('All done') |
Running the example creates the thread pool and submits the task. The task blocks then raises an exception which is caught by the thread pool.
The main thread attempts to get the result from the task and the exception is re-raised by the thread pool and is not caught by the main thread.
This unwinds the main thread and closes the program, just like if we called the task function directly.
|
1 2 3 4 |
Traceback (most recent call last): ... raise Exception('Something bad happened') Exception: Something bad happened |
Free Python ThreadPoolExecutor Course
Download your FREE ThreadPoolExecutor PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPoolExecutor API.
Discover how to use the ThreadPoolExecutor class including how to configure the number of workers and how to execute tasks asynchronously.
Example of Handling the Exception When Getting the Result
Let’s demonstrate how to handle an exception raised within a task executed by the ThreadPoolExecutor.
We can use the same task from the previous section that blocks for a moment, then raises an exception.
|
1 2 3 4 5 |
# mock task that will sleep for a moment def work(): sleep(1) raise Exception('Something bad happened') return "Task is done" |
The task will fail, but it will not break the worker thread that executed or the thread pool.
Next, we can start a thread pool and submit the task for execution and receive a Future object in return.
|
1 2 3 4 5 |
... # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) |
We can then attempt to get the result from the task once it is done.
The result can be retrieved by calling the result() function on the Future object, which we can wrap in a try-except block to handle any exception raised by the task, and re-raised by the thread pool.
|
1 2 3 4 5 6 7 |
.. # get the result from the task try: result = future.result() print(result) except Exception: print('Unable to get the result') |
Tying this together, the complete example of handling an exception in a task executed by the ThreadPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of handling an exception raised by task from time import sleep from concurrent.futures import ThreadPoolExecutor # mock task that will sleep for a moment def work(): sleep(1) raise Exception('Something bad happened') return "Task is done" # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) # get the result from the task try: result = future.result() print(result) except Exception: print('Unable to get the result') |
Running the example will start the thread pool and submit the task as per normal.
We attempt to retrieve the result and the exception is raised, which we handle explicitly and, in this case, report the failure.
|
1 |
Unable to get the result |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Checking for an Exception
An alternative approach to handling the exception is to check for an exception directly.
This can be achieved by calling the exception() function in the future, which returns the exception raised within the task or None if no exception was raised.
|
1 2 3 4 |
... # get the exception from the task when it is finished exception = future.exception() print(exception) |
Tying this together, the updated version of the example that directly checks for an exception raised by the task is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# SuperFastPython.com # example of checking for an exception directly from time import sleep from concurrent.futures import ThreadPoolExecutor # mock task that will sleep for a moment def work(): sleep(1) raise Exception('Something bad happened') return "Task is done" # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) # get the exception from the task when it is finished exception = future.exception() print(exception) |
Running the example starts the thread pool and submits the task as per normal.
We then check for an exception in the task, which does not return until the task is done.
In this case, an exception is returned as expected and reported directly.
|
1 |
Something bad happened |
Example of Handling the Exception within the Task
We can handle the exception directly within the task function.
This requires updating the task function to explicitly handle the exception, for example:
|
1 2 3 4 5 6 7 8 |
# mock task that will sleep for a moment def work(): sleep(1) try: raise Exception('Something bad happened') except: return "Unable to get the result" return "Task is done" |
The downside of this approach is that the caller that retrieves the result has no clear indication that the task failed and instead must rely on the returned value.
Tying this together, the complete example of handling the exception within the task function is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of handling the exception within the task from time import sleep from concurrent.futures import ThreadPoolExecutor # mock task that will sleep for a moment def work(): sleep(1) try: raise Exception('Something bad happened') except: return "Unable to get the result" return "Task is done" # create a thread pool with ThreadPoolExecutor() as executor: # execute our task future = executor.submit(work) # get the result result = future.result() print(result) |
Running the example starts the thread pool and submits the task as per normal.
The task function raises an exception that is immediately caught and handled, returning a message that no result could be calculated.
|
1 |
Unable to get the result |
Further Reading
This section provides additional resources that you may find helpful.
Books
- ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
- Concurrent Futures API Interview Questions
- ThreadPoolExecutor Class API Cheat Sheet
I also recommend specific chapters from the following books:
- Effective Python, Brett Slatkin, 2019.
- See Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPoolExecutor: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Takeaways
You now know how to handle exceptions raised within tasks executed by the ThreadPoolExecutor.
Do you have any questions about how to handle exceptions raised?
Ask your question in the comments below and I will do my best to answer.
Photo by Carter Moorse on Unsplash
Do you have any questions?