Last Updated on September 12, 2022

You can handle exceptions raised by tasks in the ProcessPoolExecutor by catching them when getting the result.

In this tutorial you will discover how to handle exceptions in the ProcessPoolExecutor.

Let’s get started.

Need to Handle Exceptions Raised By Tasks

The ProcessPoolExecutor in Python provides a pool of reusable processes for executing ad hoc tasks.

You can submit tasks to the process pool by calling the submit() function and passing in the name of the function you wish to execute on another process.

Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.

You can also submit tasks by calling the map() function and specify the name of the function to execute and the iterable of items to which your function will be applied.

It is possible for your target task function to raise an exception during execution.

How do you handle the exception raised by a task in a ProcessPoolExecutor?

How to Handle Exceptions Raised by Tasks

You can handle exceptions raised within a task when getting the result from the task.

Recall that when you submit a task to the process pool you will receive a Future object in return.

When you wish to retrieve the result from the task, you can call the result() function in the Future for the task.

If the task is completed successfully, it will return the result from your function or None if your function does not return a result.

If your target task function raises an exception during execution, the exception will be raised again (re-raised) when calling the result() function, where you may need to handle it.

The failure of the task will not break the worker process and will not break the process pool. The worker process that executed the task can be reused for additional tasks in the future without any problem.

Alternatively, you can call the exception() function on the task.

This will return once the task has completed and will contain the exception raised during the execution of the task, or None if no exception was raised.

Both the result() and exception() functions on the Future object allow you to specify a “timeout” argument which is the time in seconds you are willing to wait for a result or an exception respectively if the task is scheduled or running, before giving up.

A third option is to catch the exception within the target task function itself and handle it.

This approach may be desired if you are submitting tasks to the process pool via the map() function, in which case you will not have a Future object from which to call result() or exception() to handle the exception.

The downside of this approach is that any recipient of the result from the task, e.g. callers waiting on the result, may not be aware that an exception occurred within the task.

You may need to log the exception within the task and perhaps pass on the failure within the task via a return value from the target task function.

Now that we know how to handle an exception within a task executed by a ProcessPoolExecutor, let’s look at a worked example.

Example of Handling a Task Exception

Let’s demonstrate how to handle an exception raised within a task executed by the ProcessPoolExecutor.

First, let’s define a task that blocks for a moment then raises an exception.

The task will fail, but it will not break the worker process that executed it, or the process pool.

Next, we can start a process pool and submit the task for execution and receive a Future object in return.

We can then attempt to get the result from the task once it is done.

The result can be retrieved by calling the result() function on the Future object, which we can wrap in a try-except block to handle any exception raised by the task, and re-raised by the process pool.

Tying this together, the complete example of handling an exception in a task executed by the ProcessPoolExecutor is listed below.

Running the example will start the process pool and submit the task as per normal.

We attempt to retrieve the result and the exception is raised which we handle explicitly and in this case report the failure.

Example of Checking For an Exception

An alternative approach is to check for an exception directly.

This can be achieved by calling the exception() function in the future which returns the exception raised within the task or None if no exception was raised.

Tying this together, the updated version of the example that directly checks for an exception raised by the task is listed below.

Running the example starts the process pool and submits the task as per normal.

We then check for an exception in the task, which does not return until the task is done.

In this case, an exception is returned as expected and reported directly.

Further Reading

This section provides additional resources that you may find helpful.

Books

• ProcessPoolExecutor Jump-Start, Jason Brownlee (my book!)
• Concurrent Futures API Interview Questions
• ProcessPoolExecutor PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Effective Python, Brett Slatkin, 2019.
  • See Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ProcessPoolExecutor: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python Pool: The Complete Guide

APIs

• concurrent.futures - Launching parallel tasks
• Multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.
• Futures and promises, Wikipedia.

Takeaways

You now know how to handle exceptions raised within tasks executed by the ProcessPoolExecutor.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by John McArthur on Unsplash