Last Updated on September 12, 2022
You can shutdown the process pool via the Pool.close() or Pool.terminate() functions.
In this tutorial you will discover how to shutdown a process pool in Python.
Let’s get started.
Need to Close a Process Pool
The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.
A process pool can be configured when it is created, which will prepare the child workers.
A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.
— multiprocessing — Process-based parallelism
We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map(). Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().
The process pool must be closed once we are finished with it in order to release the child worker processes.
How can we safely close the process pool once we are finished with it?
Run loops using all CPUs, download your FREE book to learn how.
Problem With Not Closing Process Pool
We may encounter problems if we do not close the process pool once we are finished with it.
The process pool maintains multiple child worker processes. Each process is a heady weight object maintained by the operating system with its own main thread and stack space.
In addition, the process pool has worker threads responsible for dispatching tasks and gathering results from the child processes.
If the process pool is not explicitly closed, it means that the resources required to operate the process pool, e.g. the child processes, their threads, and stack space, may not be released and made available to the program.
multiprocessing.pool objects have internal resources that need to be properly managed (like any other resource) by using the pool as a context manager or by calling close() and terminate() manually.
— multiprocessing — Process-based parallelism
In addition, it is possible for worker threads to prevent the main process from exiting if they are still running.
The child worker processes are daemon processes, which means that the main process can exit if worker processes are running.
You can learn more about daemon processes in the tutorial:
Nevertheless, there is a known issue where if we let the Python garbage collector shutdown and delete the process pool for us (e.g. finalize), and an exception is raised during the shutdown process, then it is possible for the pool to not shutdown properly.
… Failure to do this can lead to the process hanging on finalization. Note that it is not correct to rely on the garbage collector to destroy the pool as CPython does not assure that the finalizer of the pool will be called (see object.__del__() for more information).
— multiprocessing — Process-based parallelism
Therefore, it is a best practice to always shutdown the process pool.
How to Shutdown the Process Pool
There are two ways to shutdown the process pool.
They are:
- Call Pool.close().
- Call Pool.terminate().
We can also call Pool.terminate() automatically via the context manager interface.
Let’s take a closer look at each of these cases in turn.
How to Close the Process Pool
The process pool can be shutdown by calling the Pool.close() function.
For example:
|
1 2 3 |
... # close the process pool pool.close() |
This will prevent the pool from accepting new tasks.
Once all issued tasks are completed, the resources of the process pool, such as the child worker processes, will be released.
Prevents any more tasks from being submitted to the pool. Once all the tasks have been completed the worker processes will exit.
— multiprocessing — Process-based parallelism
How to Terminate the Process Pool
The process pool can be forcefully shutdown by calling the Pool.terminate() function.
For example:
|
1 2 3 |
... # terminate the process pool pool.terminate() |
This will prevent the pool from accepting new tasks.
It will immediately terminate all child worker processes, even if they are in the middle of executing a task.
Stops the worker processes immediately without completing outstanding work.
— multiprocessing — Process-based parallelism
This approach to shutting down the process pool is not recommended, as it may leave your task data or program state in an inconsistent state.
How to Automatically Terminate the Process Pool
The Pool.terminate() function can be called automatically in two circumstances:
- By the Python garbage collector, once the pool is no longer used by the program.
- By the context manager interface.
As mentioned previously, it is not a good idea to rely on the terminate() function being called by the Python garbage collector in order to shutdown the process pool as it can result in the main process hanging in rare circumstances.
The process pool class provides the context manager interface.
This means that we can create and configure a process pool, use it within the context manager block, and once the block is exited, the process pool will be closed automatically for us.
For example:
|
1 2 3 4 5 6 |
... # create a process pool with Pool() as pool: # use the process pool # ... # process pool is closed automatically |
Exiting the block of the context manager normally or via an error will result in the terminate() function of the process pool being called automatically.
Pool objects now support the context management protocol […]. __enter__() returns the pool object, and __exit__() calls terminate().
— multiprocessing — Process-based parallelism
You can learn more about the process pool context manager interface in the tutorial:
Shutdown With Pool.close() vs Pool.terminate()
The key difference between close() and terminate() is that terminate shuts down the pool immediately, even if tasks are running, whereas close() waits for the task to complete before shutting down.
Use close() for a normal shutdown of the pool once you are finished with it.
Use terminate() for emergencies, such as if an exception was raised and the pool must be shutdown immediately, or if the program is exiting.
You can also use terminate() after close as a safety check in another part of the application (e.g. a consistent shutdown procedure) to confirm that the pool is completely shut down.
Now that we know how to shutdown the process pool, let’s look at some worked examples.
Free Python Multiprocessing Pool Course
Download your FREE Process Pool PDF cheat sheet and get BONUS access to my free 7-day crash course on the Process Pool API.
Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.
Example of Closing the Process Pool
We can explore how to close the process pool safely.
In this example we will create a process pool, issue a task, wait for the task to complete, then close the process pool. We will define a new custom function to execute as a task in the pool which will report a message and sleep for a moment.
Firstly, we can define a function to run in a new task in the process pool.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) |
Next, in the main process we can create a process pool with a default configuration.
|
1 2 3 |
... # create and configure the process pool pool = Pool() |
Next, we can issue our task() function to the process pool asynchronously.
Notice that we don’t have to wait for the task to complete.
|
1 2 3 |
... # issue tasks to the process pool result = pool.apply_async(task) |
Next, we can close the process pool, then call join() to wait a moment for all resources to be released.
|
1 2 3 4 5 6 7 |
... # close the process pool pool.close() # wait a moment pool.join() # report a message print('Main all done.') |
Finally, we can report the number of child processes for the main process, which we expect to be zero, as the process pool has now shut down.
|
1 2 3 4 |
... # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of closing the process pool from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool pool = Pool() # issue tasks to the process pool result = pool.apply_async(task) # close the process pool pool.close() # wait a moment pool.join() # report a message print('Main all done.') # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') |
Running the example first creates the process pool then issues the task to the process pool.
The process pool begins executing the task in a child worker process.
The main process then closes the process pool while the task is running.
This prevents the pool from taking any further tasks, then closes all child worker processes once all tasks are completed. The main process blocks waiting for all child processes to be released.
The task in the process pool finishes and the worker processes in the pool are closed.
The main process carries on and reports the number of active child processes, which is zero as expected.
|
1 2 3 4 |
Task executing Task done Main all done. Active children: 0 |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Terminating the Process Pool
We can explore how to shutdown the process pool by forcefully terminating the child processes.
In this example, we can update the previous example and let the issued task run for a moment then terminate the pool before the task has had a chance to complete.
|
1 2 3 4 5 6 7 |
... # issue tasks to the process pool result = pool.apply_async(task) # wait a moment sleep(0.5) # close the process pool pool.terminate() |
As before, we can wait for the child worker processes in the pool to finish, then report the number of active child processes for the main process, which we expect to be zero.
|
1 2 3 4 5 6 7 8 |
... # wait for the child processes to close pool.join() # report a message print('Main all done.') # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# SuperFastPython.com # example of terminating the process pool from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool pool = Pool() # issue tasks to the process pool result = pool.apply_async(task) # wait a moment sleep(0.5) # close the process pool pool.terminate() # wait for the child processes to close pool.join() # report a message print('Main all done.') # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') |
Running the example first creates the process pool then issues the task to the process pool.
The process pool begins executing the task in a child worker process.
The main process blocks for a moment, then terminates the process pool.
This prevents the pool from taking any further tasks, then immediately closes all child worker processes. The main process blocks waiting for all child processes to be terminated.
The task in the process pool does not get a chance to finish. The worker processes in the pool are closed.
The main process carries on and reports the number of active child processes, which is zero as expected.
|
1 2 3 |
Task executing Main all done. Active children: 0 |
Automatically Terminate the Process Pool with Garbage Collection
We can explore the Python garbage collector terminating the process pool for us.
As mentioned above, this approach is not recommended.
Nevertheless, it is important to understand the behavior of the pool under these circumstances.
We can update the example to issue above to issue the task to the pool, wait a moment, then exit the main process without explicitly closing the process pool.
|
1 2 3 4 5 6 7 |
... # issue tasks to the process pool result = pool.apply_async(task) # wait a moment sleep(0.5) # report a message print('Main all done.') |
This will trigger the Python garbage collector to call finalize() on the process pool object.
At this time the process pool will still be running and executing one task.
The terminate() function will be called to immediately terminate the pool, allowing the main process to exit.
Note, the child worker processes in the process pool are daemon processes, therefore will not prevent the main process from exiting.
We can demonstrate this by reporting the daemon status of the child processes.
|
1 2 3 4 5 6 7 |
... # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') # report the daemon status of the child for child in children: print(f'Worker daemon={child.daemon}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of not closing the process pool from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool pool = Pool() # issue tasks to the process pool result = pool.apply_async(task) # wait a moment sleep(0.5) # report a message print('Main all done.') # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') # report the daemon status of the child for child in children: print(f'Worker daemon={child.daemon}') |
Running the example first creates the process pool then issues the task to the process pool.
The process pool begins executing the task in a child worker process.
The main process blocks for a moment.
It then reports the number of active processes, which is 8 in this case.
Note, the number of active child processes will differ depending on your system and the default number of worker processes that were created.
The daemon status of all child worker processes is then reported.
We can see that all child workers are daemon processes, as expected. This means they will not prevent the main process from exiting.
The main process exits. The Python garbage collector triggers the multiprocessing.pool.Pool class to be deleted and indirectly results in the terminate() function on the pool being called.
This prevents the pool from taking any further tasks, then immediately closes all child worker processes.
The task in the process pool does not get a chance to finish. The worker processes in the pool are closed.
|
1 2 3 4 5 6 7 8 9 10 11 |
Task executing Main all done. Active children: 8 Worker daemon=True Worker daemon=True Worker daemon=True Worker daemon=True Worker daemon=True Worker daemon=True Worker daemon=True Worker daemon=True |
Automatically Terminate the Process Pool with the Context Manager
We can automatically call terminate on the process pool once we are finished with it via the context manager interface.
This is a preferred way to work with the process pool, if the work with the pool can fit within the context managers block.
We can update the example to create and configure the pool using the context manager interface.
|
1 2 3 4 |
... # create and configure the process pool with Pool() as pool: # ... |
We can then issue the task within the pool, and wait for it to complete.
|
1 2 3 4 5 |
... # issue tasks to the process pool result = pool.apply_async(task) # wait a moment result.wait() |
It is important that we wait for the task to complete within the context manager block.
This can be achieved by using blocking calls on the process pool or waiting on the result directly as we are in this case.
If we carried on and the context manager block exited, then the pool would be terminated while the task was still running.
One we are finished with the process pool, we can report a message and check the number of active child processes of the main process.
|
1 2 3 4 5 6 |
... # report a message print('Main all done.') # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of automatically terminating the process pool from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool result = pool.apply_async(task) # wait a moment result.wait() # report a message print('Main all done.') # report the number of child processes that are still active children = active_children() print(f'Active children: {len(children)}') |
Running the example first creates the process pool then issues the task to the process pool.
The process pool begins executing the task in a child worker process.
The main process then blocks, waiting for the task to complete.
Once the task is finished, the main process unblocks and continues on. It exits the context manager block which triggers the terminate() function to be called by the __exit__() function of the context manager.
The worker processes in the process pool are terminated immediately.
The main process then reports the number of active child processes, which is zero as expected.
|
1 2 3 4 |
Task executing Task done Main all done. Active children: 0 |
Error When Issuing Task to Closed Pool
We cannot issue new tasks to a process pool that has been shutdown.
That is, if the pool has been shutdown with a call to close() or terminate() and we try to issue a new task, it results in an error.
We can demonstrate this with a worked example.
The example above can be updated so that we close the pool, then issue a second task.
|
1 2 3 4 5 6 7 |
... # issue tasks to the process pool result = pool.apply_async(task) # close the process pool pool.close() # issue another task pool.apply_async(task) |
This is expected to result in an error.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of issuing a task to a pool that has already been closed. from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool pool = Pool() # issue tasks to the process pool result = pool.apply_async(task) # close the process pool pool.close() # issue another task pool.apply_async(task) |
Running the example first creates the process pool then issues the task to the process pool.
The process pool begins executing the task in a child worker process.
The main process then closes the process pool while the task is running.
This prevents the pool from taking any further tasks.
The main process then attempts to issue a new task to the pool. This results in an error, as expected, reporting “Pool not running”.
|
1 2 3 |
Traceback (most recent call last): ... ValueError: Pool not running |
Further Reading
This section provides additional resources that you may find helpful.
Books
- Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Pool Class API Cheat Sheet
I would also recommend specific chapters from these books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPool: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to shutdown the process pool in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Mark Harpur on Unsplash
Jason, these are amazing tutorials – thank you.
I have a question about how to terminate a task after a timeout period. I would like to create a single thread (or daemon process) to collect some data from a remote system, and return the results in a callback function to capture the output.
The collection may or may not complete in timeout period, so I would like the callback to report None (or similar), kill or exit the child process/thread and have the main thread continue.
I think from your tutorials this is possible using combinations of process pools in daemon mode, join(timeout), apply_async and global variables, but i’m having a hard time putting it together.
Could you advise please?
Great question!
I think there are many ways to achieve this.
If the task is just one function call, then you can issue it using a new Process(), then join it with a timeout. If the result is not available after the timeout, the process can be stopped via a call to terminate(). This can all be wrapped up into a function that either returns the result or None.
You can use a queue to get the result back from the process to the caller and wait on the queue directly. This might be easier/cleaner.
The example below shows how to do this:
If the task is many calls, then the same deal as above, but use a Pool instead to issue the tasks using map_async().
Does that help?