You can create a hosted centralized version of Python objects using a Manager and share proxy objects that allow child processes to interact with the hosted object in a process-safe manager.
In this tutorial you will discover the multiprocessing Manager in Python.
Let’s get started.
Table of Contents
What is a Multiprocessing Manager
Multiprocessing Manager provides a way of creating centralized Python objects that can be shared safely among processes.
Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines.— multiprocessing – Process-based parallelism
Manager objects create a server process which is used to host Python objects. Managers then return proxy objects used to interact with the hosted objects.
The proxy objects automatically ensure process-safety and serialize data to and from the centralized objects.
A proxy is an object which refers to a shared object which lives (presumably) in a different process. The shared object is said to be the referent of the proxy. Multiple proxy objects may have the same referent. A proxy object has methods which invoke corresponding methods of its referent […]. In this way, a proxy can be used just like its referent can— multiprocessing – Process-based parallelism
This means that the proxy objects can be shared among multiple processes allowing the centralized object to be used in parallel seamlessly in a process-safe manner.
A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.— multiprocessing – Process-based parallelism
Run your loops using all CPUs, download my FREE book to learn how.
What are the Benefits of a Manager
Managers provide three key capabilities for process-based concurrency, they are:
- Centralized: A single instance of a shared object is maintained in a separate server process.
- Process-safety: Proxy objects ensure that access to the centralized object is process-safe in order to avoid race conditions.
- Pickability: Proxy objects can be pickled and shared with child processes such as arguments in process pools and items in queues.
In addition to providing safe access to a shared object across processes on one system, managers allow the same object to be accessed safely across processes on other systems via network access.
Confused by the multiprocessing module API?
Download my FREE PDF cheat sheet
How to Use A Manager?
Using a manager involves four steps, they are:
- Create the manager instance.
- Start the manager.
- Create one or more hosted objects to share.
- Shutdown the manager.
1. Create a Manager
A manager can be created by creating an instance of the multiprocessing.Manager class.
# create a manager
manager = multiprocessing.Manager()
2. Start the Manager
After the manager is created, it must be started.
This can be achieved by calling the start() method.
# start the manager
This will start the server process used to host centralized versions of Python objects.
3. Create Hosted Objects
Once the manager is started, it can be used to create hosted objects and return proxy objects for those hosted objects.
Only objects that have been registered with the Manager, e.g. those objects that the manager knows about, can be created.
# create a hosted object and get a proxy object
proxy_object = manager.Lock()
This creates a centralized version of an object in the Manager‘s server process, in this case a mutex Lock class, and returns a proxy for interacting with the hosted object.
4. Shutdown the Manager
Once we are finished with the shared objects, we can shutdown the manager.
This will release all resources used by the manager, including the server process.
This can be achieved by calling the close() function.
# shutdown the manager
Context Manager Interface
An alternate usage pattern for the Manager is to use the context manager interface.
This ensures that the Manager instance is always closed, in case we forget to call close() or in case the close() method is not reached due to an error.
# create and start the manager
with multiprocessing.Manager() as manager:
# create a hosted object and get a proxy object
proxy_object = manager.Lock()
# manager is closed automatically
The manager is created and started at the beginning of the context manager block, all usage of the manager is restricted to the block, then the manager is closed as soon as the block is exited, normally or otherwise.
This is the preferred way to use the Manager in your program, if possible.
Free Python Multiprocessing Course
Download my multiprocessing API cheat sheet and as a bonus you will get FREE access to my 7-day email course.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
What Objects Does a Manager Provide
Managers are provided via the multiprocessing.Manager class, which creates and returns a multiprocessing.managers.SyncManager instance.
The SyncManager allows a suite of Python objects to be created and managed by default, including:
- Data structures
- Shared ctypes
- Concurrency primitives
- Other Objects
Let’s take a closer look at each group:
Managed Data Structures
This includes Python objects we may want to share, such as:
You can see an example of sharing a managed collection among processes in the tutorial:
Managed Shared Ctypes
It includes shared ctypes for primitive values, such as:
If you are new to shared ctypes, I recommend the tutorial:
Managed Concurrency Primitives
It includes concurrency primitives for synchronizing and coordinating processes, such as:
You can see an example of sharing a managed concurrency primitive in the tutorial:
It includes process-safe queues for sharing data between processes, such as:
You can see an example of sharing a managed queue among processes in the tutorial:
Managed Other Objects
Other objects are provided by the SyncManager class, some of which are not documented, such as:
- Pool (along with Iterator and AsyncResult)
You can see an example of sharing a managed Namespace in the tutorial:
You can see an example of using a managed Pool in the tutorial:
You can see an example of sharing a managed custom class in the tutorial:
In fact, if we look at the source code for the multiprocessing.managers.SyncManager class, we can see that the actual classes that are shared are from the “threading” module, not the “multiprocessing” module.
Likely this is because the hosted objects are never used outside of the Manager‘s server process. They are only interacted with indirectly via proxy objects which manage the inter-process communication and serialization for process-safety.
Care must be taken when nesting objects hosted by a manager.
You can see an example of using nested objects with a manager in the tutorial:
Overwheled by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
When to Use a Manager
A manager should be used when an object needs to be shared among processes.
Sometimes the object can be shared directly without problem, other times it cannot and a manager is required for those cases, which include:
- When the shared object cannot be pickled and needs to be pickled in order to be shared.
- When the shared object is not process-safe and needs to be used in multiple processes simultaneously.
If the object to be shared meets either or both of these requirements, then it is a good candidate for being hosted in a manger server process and shared via proxy objects.
What Are Some Examples Where a Manager is Required
There are a number of common use cases where you may need to use a manager.
- When sharing a concurrency primitive (e.g. lock, semaphore, event, etc.) or queue with a multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor. This is because concurrency primitives cannot be pickled and all arguments to process pools must be pickled.
- When sharing an object that cannot be pickled with processes via a multiprocessing.Queue. This is because the multiprocessing.Queue requires that all objects put on the queue be pickled.
This section provides additional resources that you may find helpful.
- Python Multiprocessing Jump-Start, Jason Brownlee, 2022 (my book!).
- Multiprocessing API Interview Questions
- Multiprocessing Module API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
You now know about the multiprocessing Manager in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.