How to Handle Exceptions in Tasks With the ThreadPoolExecutor in Python

December 5, 2021 Python ThreadPoolExecutor

You can handle exceptions in the ThreadPoolExecutor raised by tasks by catching them when getting the result.

In this tutorial, you will discover how to handle exceptions in the ThreadPoolExecutor.

Let's get started.

Need to Handle Exceptions Raised by Tasks

The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.

You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread.

Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.

You can also submit tasks by calling the map() function and specifying the name of the function to execute and the iterable of items to which your function will be applied.

An exception raised in a single threaded program must be caught, otherwise it will unwind the main thread.

Unwinding a thread means that the exception is propagated up to the top of the stack of function calls until it reaches the top level at which it will cause the main thread to exit and close the program.

The ThreadPoolExecutor can be used to execute arbitrary functions.

Each task submitted to the thread pool is picked-up by a worker thread and executed. While executing, the function may raise an exception.

If uncaught, does the exception may unwind the worker thread and it permanently unavailable in the thread pool?

Additionally, we may need to know if a task raised an exception when getting the result so that we can take appropriate action, such as submit a follow-up task or perhaps re-try the task.

How do you handle the exception raised by a task in a ThreadPoolExecutor?

How to Handle Exceptions in Tasks

An exception raised by a target function executed by a worker thread will not unwind the worker thread.

The thread pool is robust to target functions that raise exceptions that are not caught. The exception will be caught by the thread pool automatically, and the worker thread will remain available for executing tasks.

There are three ways you may handle exceptions raised by a target function, they are:

Handle exceptions when getting the task result.
Check for raised exceptions.
Handle exceptions within the task function.

Let's take a closer look at each in turn.

Handle Exceptions When Getting the Result

You can handle exceptions raised within a task when getting the result from the task.

Recall that when you submit a task to the thread pool, you will receive a Future object in return.

When you wish to retrieve the result from the task, you can call the result() function in the Future for the task.

If the task is completed successfully, it will return the result from your function or None if your function does not return a result.

If your target task function raises an exception during execution, the exception will be raised again (re-raised) when calling the result() function, where you may need to handle it.

...
# handle a possible exception raised in a task
try:
	result = future.result()
exception:
	# handle exception...

The failure of the task will not break the worker thread and will not break the thread pool. The worker thread that executed the task can be reused for additional tasks in the future without any problem.

Check for a Raised Exception

Alternatively, you can call the exception() function on the task.

This will return once the task has completed and will contain the exception raised during the execution of the task, or None if no exception was raised

...
# get the exception raised by the task
exception = future.exception()

Both the result() and exception() functions on the Future object allow you to specify a timeout argument, which is the time in seconds you are willing to wait for a result or an exception respectively if the task is scheduled or running, before giving up.

Handle Exceptions in the Task Function

A third option is to catch the exception within the target task function itself and handle it.

This approach may be desired if you are submitting tasks to the thread pool via the map() function, in which case you will not have a Future object from which to call result() or exception() to handle the exception.

# target task function
def work()
	try:
		# do something...
	exception:
		# handle exception...

The downside of this approach is that any recipient of the result from the task, e.g. callers waiting on the result, may not be aware that an exception occurred within the task.

You may need to log the exception within the task and perhaps pass on the failure within the task via a return value from the target task function.

Now that we know how to handle an exception within a task executed by a ThreadPoolExecutor, let's look at some worked examples.

Example of Not Handling the Exception

Before we explore how to handle an exception raised by a task, let's explore what happens if we don't handle it.

First, let’s define a task that sleeps for a moment, then raises an exception.

# mock task that will sleep for a moment
def work():
    sleep(1)
    raise Exception('Something bad happened')
    return "Task is done"

Next, we can start a thread pool and submit the task for execution and receive a Future object in return.

...
# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)

We will then not get the result from the task, and instead will wait around for a moment.

We want to see if the exception raised by the task has any effect on the thread pool or the main thread.

...
# wait around for a moment
sleep(2)

Tying this all together, the complete example of executing a task that raises an exception and not handling the exception is listed below.

# SuperFastPython.com
# example of not handling an exception raised by task
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# task function that blocks for a moment
def work():
    sleep(1)
    raise Exception('Something bad happened')
    return "Task is done"

# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)
    # wait around for a moment
    sleep(2)
print('All done')

Running the example will start the thread pool and submit the task.

The task will block for a moment and then raises an exception.

The main thread waits around for a moment, then the thread pool is closed and the main thread exits.

This shows that if we don't attempt to get the result from the task, we have no idea whether the task succeeded or failed. It looks like the task failed silently.

Importantly, it shows that the exception raised by the task function did not unwind the worker thread or the main thread.

All done

We can update the example to attempt to get the result, again, without handling the exception.

...
# get the result from the task
result = future.result()
print(result)

Tying this together, the complete example of attempting to get the result from a task that we know has raised an exception is listed below.

# SuperFastPython.com
# example of attempting to get the result from a task that has raised an exception
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# task function that blocks for a moment
def work():
    sleep(1)
    raise Exception('Something bad happened')
    return "Task is done"

# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)
    # get the result from the task
    result = future.result()
    print(result)
print('All done')

Running the example creates the thread pool and submits the task. The task blocks then raises an exception which is caught by the thread pool.

The main thread attempts to get the result from the task and the exception is re-raised by the thread pool and is not caught by the main thread.

This unwinds the main thread and closes the program, just like if we called the task function directly.

Traceback (most recent call last):
  ...
    raise Exception('Something bad happened')
Exception: Something bad happened

Example of Handling the Exception When Getting the Result

Let's demonstrate how to handle an exception raised within a task executed by the ThreadPoolExecutor.

We can use the same task from the previous section that blocks for a moment, then raises an exception.

# mock task that will sleep for a moment
def work():
    sleep(1)
    raise Exception('Something bad happened')
    return "Task is done"

The task will fail, but it will not break the worker thread that executed or the thread pool.

Next, we can start a thread pool and submit the task for execution and receive a Future object in return.

...
# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)

We can then attempt to get the result from the task once it is done.

The result can be retrieved by calling the result() function on the Future object, which we can wrap in a try-except block to handle any exception raised by the task, and re-raised by the thread pool.

..
# get the result from the task
try:
    result = future.result()
    print(result)
except Exception:
    print('Unable to get the result')

Tying this together, the complete example of handling an exception in a task executed by the ThreadPoolExecutor is listed below.

# SuperFastPython.com
# example of handling an exception raised by task
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# mock task that will sleep for a moment
def work():
    sleep(1)
    raise Exception('Something bad happened')
    return "Task is done"

# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)
    # get the result from the task
    try:
        result = future.result()
        print(result)
    except Exception:
        print('Unable to get the result')

Running the example will start the thread pool and submit the task as per normal.

We attempt to retrieve the result and the exception is raised, which we handle explicitly and, in this case, report the failure.

Unable to get the result

Example of Checking for an Exception

An alternative approach to handling the exception is to check for an exception directly.

This can be achieved by calling the exception() function in the future, which returns the exception raised within the task or None if no exception was raised.

...
# get the exception from the task when it is finished
exception = future.exception()
print(exception)

Tying this together, the updated version of the example that directly checks for an exception raised by the task is listed below.

# SuperFastPython.com
# example of checking for an exception directly
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# mock task that will sleep for a moment
def work():
    sleep(1)
    raise Exception('Something bad happened')
    return "Task is done"

# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)
    # get the exception from the task when it is finished
    exception = future.exception()
    print(exception)

Running the example starts the thread pool and submits the task as per normal.

We then check for an exception in the task, which does not return until the task is done.

In this case, an exception is returned as expected and reported directly.

Something bad happened

Example of Handling the Exception within the Task

We can handle the exception directly within the task function.

This requires updating the task function to explicitly handle the exception, for example:

# mock task that will sleep for a moment
def work():
    sleep(1)
    try:
        raise Exception('Something bad happened')
    except:
            return "Unable to get the result"
    return "Task is done"

The downside of this approach is that the caller that retrieves the result has no clear indication that the task failed and instead must rely on the returned value.

Tying this together, the complete example of handling the exception within the task function is listed below.

# SuperFastPython.com
# example of handling the exception within the task
from time import sleep
from concurrent.futures import ThreadPoolExecutor

# mock task that will sleep for a moment
def work():
    sleep(1)
    try:
        raise Exception('Something bad happened')
    except:
            return "Unable to get the result"
    return "Task is done"

# create a thread pool
with ThreadPoolExecutor() as executor:
    # execute our task
    future = executor.submit(work)
    # get the result
    result = future.result()
    print(result)

Running the example starts the thread pool and submits the task as per normal.

The task function raises an exception that is immediately caught and handled, returning a message that no result could be calculated.

Unable to get the result

Takeaways

You now know how to handle exceptions raised within tasks executed by the ThreadPoolExecutor.