Last Updated on September 12, 2022

You can wait for a task to finish in a ThreadPoolExecutor by calling the wait() module function.

In this tutorial you will discover how to wait for tasks to finish in a Python thread pool.

Let’s get started.

Need To Wait for All Tasks to Finish

The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.

You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread. Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.

You can also submit tasks to the thread pool by calling the map() function and pass in the name of the function and the iterable of items that your function will be applied to asynchronously.

After submitting tasks to the thread pool, you may want to wait for the tasks to complete before continuing.

There are many reasons for this to be the case, for example:

• Perhaps you don’t have the computational resources to perform the tasks and continue on with the program.
• Perhaps you need the results from all tasks.
• Perhaps you need the action performed by each task to have been performed before continuing.

How can you wait for tasks submitted to the ThreadPoolExecutor to complete?

How to Wait For All Tasks To Finish

There are a number of ways that you can wait for all tasks to complete in the ThreadPoolExecutor.

We will look at three approaches you can use, they are calling the wait() module function, getting the result from each task, and shutting down the thread pool.

Let’s take a look at each approach in turn.

Approach 1: Call the wait() Module Function

The most common approach is to call the wait() module function and pass in a collection of Future objects created when calling submit().

The wait function allows you fine grained control over the specific tasks that you wish to wait to complete and also allows you to specify a timeout for how long you are willing to wait in seconds.

Approach 2: Get The Results From Each Task

Alternatively, you can enumerate the list of Future objects and attempt to get the result from each.

This iteration will complete when all results are available meaning that all tasks were completed.

A similar approach could be used with the as_completed() module function that will iterate over Future objects in the order that tasks are completed, rather than the order they were stored in the list.

For example:

Approach 3: Call shutdown()

Perhaps you don’t have Future objects for your tasks because you submitted them to the thread pool by calling map(), or perhaps you wish to wait for all tasks in the thread pool to complete rather than a subset.

In this case, you can wait for tasks to complete while shutting down the thread pool.

You can shutdown the thread pool by calling the shutdown() function. We can set the wait argument to True so that the call will not return until all running tasks complete and set cancel_futures to True which will cancel all scheduled tasks.

You can also shutdown the pool and not cancel the scheduled tasks, yet still wait for all tasks to complete.

This will ensure all running and scheduled tasks are completed before the function returns. This is the default behavior of the shutdown function, but is a good idea to specify explicitly.

Now that we have seen a few ways to wait for tasks to complete in the ThreadPoolExecutor, let’s look at a worked example.

Example of Waiting For All Tasks To Complete With wait()

Let’s explore how to wait for tasks to complete in the ThreadPoolExecutor with an example.

First, let’s define a simple task that will sleep for a fraction of a second and report when it is completed.

Next, we can create a thread pool with 2 worker threads and issue ten tasks to the pool for execution by calling the submit() function for each task.

Each call to submit will return a Future object which we will collect into a list.

Next, we can call the wait() module function and pass in the list of the ten Future objects we collected when calling submit().

This call will return once all tasks associated with the Future objects in the collection have completed.

We can then report that all tasks are completed.

Tying this together, the complete example of starting tasks and waiting for them all to complete before continuing on is listed below.

Running the example first submits all tasks into the thread pool.

We then start waiting for the tasks to complete, reporting a message.

The tasks complete, reporting a message as they do.

Finally, all tasks are completed and we are free to carry on.

Example of Waiting For All Tasks To Complete With shutdown()

Let’s look at an alternative approach for waiting for all tasks to complete.

Perhaps we have many tasks in the thread pool and we don’t have Future objects for them.

This might happen if we add tasks to the pool using the map() function and choose not to enumerate the results for the tasks.

We can wait for all of the scheduled and running tasks to complete by calling the shutdown() function explicitly and setting wait=True and cancel_futures=False, the default arguments.

Because we are using the context manager, the thread pool will be closed automatically using the default parameters.

Therefore, we don’t need to add any extra code in order to wait for all tasks in the thread pool to complete before continuing on.

We can demonstrate this with a worked example listed below.

Running the example, we see that all tasks are submitted to the pool using the map() function and we do not have Future objects associated with each task.

We then wait for the tasks to complete at the end of the context manager. Tasks report their progress as they complete.

Finally, all tasks are completed, the context manager closes the thread pool, and the program ends.

Further Reading

This section provides additional resources that you may find helpful.

Books

• ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
• Concurrent Futures API Interview Questions
• ThreadPoolExecutor Class API Cheat Sheet

I also recommend specific chapters from the following books:

• Effective Python, Brett Slatkin, 2019.
  • See Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPoolExecutor: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• concurrent.futures - Launching parallel tasks
• threading — Thread-based parallelism

References

• Thread (computing), Wikipedia.
• Futures and promises, Wikipedia.
• Thread pool, Wikipedia.

Takeaways

You now know how to wait for tasks to complete in the ThreadPoolExecutor in Python.

Do you have any questions about how to wait for tasks to complete?
Ask your question in the comments below and I will do my best to answer.

Photo by Alejandro Lopez on Unsplash