Last Updated on September 12, 2022
You can automatically terminate a process pool once you are finished with it using the context manager interface.
In this tutorial you will discover how to use the context manager interface of the process pool in Python.
Let’s get started.
Need to Automatically Shutdown the Process Pool
The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.
A process pool can be configured when it is created, which will prepare the child workers.
A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.
— multiprocessing — Process-based parallelism
We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map(). Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().
The process pool must be closed once we are finished with it in order to release the child worker processes.
How can we safely and automatically close the process pool once we are finished with it?
Run loops using all CPUs, download your FREE book to learn how.
What is a Context Manager
A context manager is an interface on Python objects for defining a new run context.
Python’s with statement supports the concept of a runtime context defined by a context manager. This is implemented using a pair of methods that allow user-defined classes to define a runtime context that is entered before the statement body is executed and exited when the statement ends
— CONTEXT MANAGER TYPES, BUILT-IN TYPES
A run context is a block of Python code. Examples of other run contexts include the content of a function or the content of a loop.
Context managers allow an object to define the code that runs at the beginning and end of a block of code. This is a lot like the try-except-finally pattern, except the code executed before and after the block is hidden within the object and only the body block of code needs to be specified.
The context manager interface has two methods that must be implemented on an object that supports the interface, they are:
- __enter__(): Executed prior to the code block.
- __exit__(): Executed after the code block.
These two methods are always executed, even if an error or exception occurs within the block. In this way, __exit__() acts like a finally block in a try-except-finally pattern.
The with statement is used to wrap the execution of a block with methods defined by a context manager . This allows common try…except…finally usage patterns to be encapsulated for convenient reuse.
— THE WITH STATEMENT.
Context managers on Python objects are used via the “with” statement.
The with statement takes the object instance that implements the context manager interface as an argument. It will execute the code in the object for the beginning of the block, e.g. the __enter__() method and return an instance of an object that can be assigned.
Context managers are useful for operations that require the consistent initialization and shutdown of an operation.
The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code.
— WITH STATEMENT CONTEXT MANAGERS
Common examples of context managers provided in the Python standard library include:
- Opening and closing files.
- Opening and closing sockets.
Now that we know what context managers are, let’s look at the use of context managers with the process pool.
How to Close a Process Pool
The process pool provides a pool of worker child processes.
As such, the pool must be closed directly once we are finished with it to allow the child processes and their resources to be released. Otherwise, the worker processes and their resources such as main threads will remain until the program exits.
multiprocessing.pool objects have internal resources that need to be properly managed (like any other resource) by using the pool as a context manager or by calling close() and terminate() manually. Failure to do this can lead to the process hanging on finalization.
— multiprocessing — Process-based parallelism
In fact, not closing the process pool and relying on the Python garbage collector to close it once we are finished with it can result in the program hanging.
Note that it is not correct to rely on the garbage collector to destroy the pool as CPython does not assure that the finalizer of the pool will be called
— multiprocessing — Process-based parallelism
There are two ways to safely close the process pool, they are:
- Pool.close() method: close the pool once all issued tasks are complete, will not accept additional tasks.
- Pool.terminate() method: close the worker processes immediately, will not accept additional tasks.
As such, the normal usage pattern for the process pool is to:
- Create and configure the process pool.
- Issue tasks to the pool.
- Close the pool.
We need to ensure that the pool is closed, even if an error is raised while tasks are issued to the pool and their results are processed.
There are three main ways we could handle this, they are:
- Directly.
- With a Try-Except-Finally pattern.
- With a Context Manager.
Let’s take a closer look at each in turn.
Approach 1: Directly
We can create, use and close the pool directly.
This can be achieved by calling the class constructor to create and configure the pool.
Issuing tasks to the pool using methods such as apply() and map().
Then, calling the close() method on the pool and perhaps join() to ensure all remaining tasks are completed.
For example:
|
1 2 3 4 5 6 7 8 9 |
... # create the process pool pool = multiprocessing.pool.Pool() # issue tasks to the pool ... # close the process pool pool.close() # wait for issued tasks to complete pool.join() |
If we don’t call the join() function and there are tasks executing the pool, it is possible that the pool will be forcefully terminated by the Python garbage collector and the tasks will not complete.
The danger of this approach is that an Error or Exception could be raised while issuing tasks to the pool and processing their results.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 |
... # create the process pool pool = multiprocessing.pool.Pool() # issue tasks to the pool ... raise Exception('Something bad happened') # ... # close the process pool pool.close() # wait for issued tasks to complete pool.join() |
This is a problem because the pool will not be closed in a controlled manner and it is possible that it will be forcefully terminated, stopping any tasks that are being processed.
Approach 2: Try-Except-Finally
A better approach is to use a try-except-finally pattern.
The issuing of tasks and processing of results can be performed within the try block. Any expected errors or exceptions can be handled in the except block, if desired. The pool can then be closed safely in the finally block.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
... # create the process pool pool = multiprocessing.pool.Pool() try: # issue tasks to the pool # ... except: # handle expected errors and exceptions finally: # close the process pool pool.close() # wait for issued tasks to complete pool.join() |
This is an improvement over directly closing the pool because it ensures that the safe shutdown procedure for the pool is always executed, even if an exception is raised.
Using a try-finally or a try-except-finally pattern means that if an unexpected exception is raised, any other tasks issued to the pool will have a chance to complete before the pool is gracefully closed.
In other programming languages like Java and C#, the try-except-finally pattern is a best practice when creating, using and releasing a resource such as a process pool.
It is not a best practice in Python, because we have something better.
Approach 3: Context Manager
Python provides a context manager interface on the process pool.
This achieves a similar outcome to using a try-except-finally pattern, with less code.
Specifically, it is more like a try-finally pattern, where any exception handling must be added and occur within the code block itself.
For example:
|
1 2 3 4 5 6 |
... # create and configure the process pool with multiprocessing.pool.Pool() as pool: # issue tasks to the pool # ... # close the pool automatically |
There is an important difference with the try-finally block.
If we look at the source code for the multiprocessing.pool.Pool class, we can see that the __exit__() method calls the terminate() method on the process pool when exiting the context manager block.
This means that the pool is forcefully closed once the context manager block is exited. It ensures that the resources of the process pool are released before continuing on, but does not ensure that tasks that have already been issued are completed first.
Therefore, if we wish to wait for all issued tasks to complete before closing the pool, we must explicitly add a call to close() and join().
For example:
|
1 2 3 4 5 6 7 8 9 10 |
... # create and configure the process pool with multiprocessing.pool.Pool() as pool: # issue tasks to the pool # ... # close the process pool pool.close() # wait for issued tasks to complete pool.join() # close the pool automatically |
It means that if we fail to close the pool safely due to an unexpected error or exception, the pool will always be terminated, allowing child processes to be released and ensuring our main process does not hang waiting for the pool.
Now that we know the benefit of context managers when using process pools, let’s look at some examples.
Free Python Multiprocessing Pool Course
Download your FREE Process Pool PDF cheat sheet and get BONUS access to my free 7-day crash course on the Process Pool API.
Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.
Example of the Process Pool with a Context Manager
We can explore how to use the context manager interface of the process pool.
In this example, we will define a custom function to execute in the process pool. We will then create and configure a process pool using the context manager interface. The task will be issued, then we will safely close the process pool.
Firstly, let’s define a task to execute in the process pool.
The task will report a message to indicate that it is starting, blocks for one second, then reports a message that it has finished.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) |
Next, in the main process will create and configure the process pool using the context manager interface.
|
1 2 3 4 |
... # create and configure the process pool with Pool() as pool: # ... |
Next, we will issue a task to the process pool to execute our custom function asynchronously and not wait for the result.
|
1 2 3 |
... # issue tasks to the process pool pool.apply_async(task) |
We can then close the process pool and wait for the issued tasks to complete.
|
1 2 3 4 5 |
... # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of using the process pool context manager from time import sleep from multiprocessing.pool import Pool # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Running the example first creates the process pool.
A task is issued to the process pool, then the main process closes the pool and waits for issued tasks to complete.
The process pool executes the task, reports the messages, then finishes.
The main process continues on. The process pool is then terminated automatically by the context manager interface, although it has no effect in this case.
|
1 2 |
Task executing Task done |
An alternate pattern when using the context manager interface is to explicitly block while waiting for the task result and allow the process pool to close automatically via the context manager interface.
This can be achieved by getting the multiprocessing.pool.AsyncResult returned from apply_async() and waiting on the result via a call to wait().
For example:
|
1 2 3 4 5 |
... # issue tasks to the process pool result = pool.apply_async(task) # wait for the result result.wait() |
In this way, we do not need to explicitly close the process pool and wait for all issued tasks to complete.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of using the process pool context manager and waiting for the result from time import sleep from multiprocessing.pool import Pool # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool result = pool.apply_async(task) # wait for the result result.wait() |
Running the example first creates the process pool.
A task is issued to the process pool and an AsyncResult is returned. The main process blocks on the result, waiting for the task to finish.
The process pool executes the task, reports the messages, then finishes.
The main process continues on. The process pool is then terminated automatically by the context manager interface, releasing all of the resources.
|
1 2 |
Task executing Task done |
Next, let’s consider what happens if the context manager closes the process pool while tasks are running.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Exiting the Context Manager With Running Tasks
The context manager interface of the process pool will forcefully terminate the child worker processes in the pool.
This means that if there are tasks being executed when the context manager with-block exits, then those tasks will be forcefully stopped.
We can update the example from the previous section to demonstrate this.
The task can be issued to the process pool, then the main process can block for a fraction of a second then exit the context manager block of the process pool. This will terminate the task half-way through.
For example:
|
1 2 3 4 5 6 7 |
... # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # block for a moment sleep(0.5) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of using the process pool context manager and forcefully terminating running tasks from time import sleep from multiprocessing.pool import Pool # task executed in a worker process def task(): # report a message print(f'Task executing', flush=True) # block for a moment sleep(1) # report a message print(f'Task done', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # block for a moment sleep(0.5) |
Running the example first creates the process pool.
A task is issued to the process pool. The main process blocks for a fraction of a second.
The process pool starts executing the task and reports the first message.
The main process continues on and exits the context manager. This results in the terminate() method on the process pool being executed automatically.
The child worker processes in the pool are terminated and the running task is stopped half way through.
|
1 |
Task executing |
This highlights that care must be taken to wait for running tasks when using the context manager interface of the process pool.
Further Reading
This section provides additional resources that you may find helpful.
Books
- Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Pool Class API Cheat Sheet
I would also recommend specific chapters from these books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPool: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to use the context manager interface of the process pool in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Brett Jordan on Unsplash
Do you have any questions?