Last Updated on September 12, 2022

The multiprocessing pool will be closed automatically by the Python garbage collector, if required.

It will not prevent the main process from exiting and the child worker processes will not keep running if the main process exits without closing the pool.

In this tutorial, we will explore what happens if you forget or are unable to close the multiprocessing pool in Python.

Let’s get started.

Will The Pool Stop The Main Process From Exiting?

The multiprocessing pool provides a pool of reusable workers for executing ad hoc tasks with process-based concurrency.

An instance of the multiprocessing.Pool class can be created, specifying the number of workers to create, otherwise a default number of workers will be created to match the number of logical CPUs in the system.

Once created, ad hoc tasks can be issued to the pool for execution via the Pool.apply() function. Multiple tasks may be executed in the book by calling the same function with different arguments via the Pool.map() function.

Tasks may be executed asynchronously in the pool via Pool.apply_async() and Pool.map_async(), allowing the caller to carry on with other tasks.

Once concern with the multiprocessing pool is whether it will prevent the main process from exiting if it is not explicitly closed.

For example, we may not close the multiprocessing pool correctly for many reasons, such as:

• We forget to call the close() or terminate() function.
• An error is raised that prevents a normal exit.

Sometimes in Python concurrency, a running thread or running child process can prevent the main process, and therefore the program from exiting normally.

Additionally, we may be concerned that if the pool is not closed properly, that the child worker processes will keep running if the parent process exits.

Will a running multiprocessing pool prevent the main process from exiting normally (or will the program hang)?

Pools Do Not Stop The Main Process From Exiting

A running multiprocessing pool does not stop the main process from exiting.

This means that we can forget to close a multiprocessing pool and the main process will exit normally.

For example:

It also means that if an exception or error is raised and we are unable to close the pool normally, that it will not prevent the program from exiting.

For example:

In both cases, the pool will be closed automatically by the garbage collector in the Python interpreter (parent process) and the child worker processes will be closed and released.

Although this occurs, it should not be relied upon.

Note that it is not correct to rely on the garbage collector to destroy the pool as CPython does not assure that the finalizer of the pool will be called

— multiprocessing — Process-based parallelism

Additionally, the child worker processes are created as daemon processes.

This means that they are intended to run in the background and to not prevent the main process from exiting.

You can learn more about daemon processes in the tutorial:

• Daemon Process in Python

Now that we know that the multiprocessing pool will not stop the main process from exiting and that the python garbage collector will close the pool for us if needed, let’s look at a worked example.

Example of Not Closing a Multiprocessing Pool

We can explore the effect of not closing the multiprocessing pool.

In this example we will create a multiprocessing pool with a fixed number of workers. We will then confirm that all child worker processes are running then attempt to exit the main process.

Firstly, we can create a multiprocessing pool with four workers.

Don’t worry if you have more or fewer CPU cores, as we will not be running any tasks.

Next, we can wait a moment for the pool to initialize completely. This is probably not required, but will likely avoid any possible race conditions in checking if the workers are running.

We can then get a list of all child processes for the current process. This will include all workers in the process pool.

This can be achieved via the multiprocessing.active_children() function.

We will then attempt to exit the main process normally, with the multiprocessing pool still running.

Tying this together, the complete example is listed below.

Running the example first creates a multiprocessing pool with four child worker processes.

The main process then blocks a moment to wait for the pool to initialize completely.

Next, a list of all child processes is retrieved and their details are reported.

We can notice that the child worker processes are daemon processes, meaning that they are intended to run in the background.

We can see all four child worker processes in the pool.

The main process then ends normally, and the program exits.

The Python interpreter garbage collector in the parent process terminates the multiprocessing pool automatically.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPool: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know what happens if you forget or are unable to close the multiprocessing pool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Zachary Shea on Unsplash