You can use a Manager to host a centralized Python object that can be shared with multiple processes that is both process-safe and changes to the object are propagated and made available to all processes seamlessly.

In this tutorial you will discover how to use a Manager to share an ad hoc Python object with multiple processes.

Let’s get started.

Need Manager to Share Queue

A manager in the multiprocessing module provides a way to create Python objects that can be shared easily between processes.

Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

— multiprocessing — Process-based parallelism

A manager creates a server process that hosts a centralized version of objects that can be shared among multiple processes.

The objects are not shared directly. Instead, the manager creates a proxy object for each object that it manages and the proxy objects are shared among processes.

The proxy objects are used and operate just like the original objects, except that they serialize data, synchronize and coordinate with the centralized version of the object hosted in the manager server process.

A proxy is an object which refers to a shared object which lives (presumably) in a different process. […] A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy).

— multiprocessing — Process-based parallelism

This makes managers a process-safe and preferred way to share Python objects among processes.

You can learn more about multiprocessing managers in the tutorial:

• What is a Multiprocessing Manager

When using multiprocessing, we may need to share an arbitrary Python object with child processes.

It may be a data structure or similar object where the internal state of the object may need to be updated from multiple child processes concurrently, although in a process-safe manner in order to avoid race conditions.

How can we use a multiprocessing Manager to share a Python object with child processes?

How to Use a Manager to Share an Object with Processes

We can use a multiprocessing manager to share an ad hoc Python object with multiple processes.

This requires a few steps:

1. Defining a custom Manager class that extends BaseManager.
2. Registering the Python class with the custom manager.
3. Creating an instance of the custom manager.
4. Creating an instance of the Python object from the manager instance.

Let’s take a closer look at each step in turn.

Step 1. Define a Custom Manager

We must define a custom manager in order to use the manager to create and manage ad hoc Python objects.

This involves defining a new class that extends the multiprocessing.managers.BaseManager class.

The custom class does not need to override the constructor or any methods, nor define any functionality.

For example:

Step 2. Register the Python Class

The next step involves registering the Python class with the custom manager.

This is so when the program requests a new instance of the class from the manager, it knows how to create an instance of that class.

This involves calling a class function (static function) on the custom manager class called register().

The register() function takes two arguments, the string name of the class used by the program when requesting an instance and the name of the Python class to create.

This should probably be the same name, but do not have to be.

For example:

Step 3. Create the Manager

The next step is to create an instance of the custom manager, and start it.

This is so that the program can request an instance of the custom class.

This can be achieved manually by instantiating the class and calling start, for example:

Later we must close the manager to terminate the server process.

The preferred approach is to use the context manager interface and limit usage of the manager to the context manager block.

This will ensure the manager is closed automatically regardless of what happens in our program.

For example:

Step 4. Create a Custom Class Instance

Finally, we can create an instance of our ad hoc Python class using the manager.

This will create the object in the Manager’s server process and return a proxy object.

The proxy object can then be shared among processes and used to interact with the centralized version of the custom class instance, with all data serialization and process-safety handled automatically under the covers.

For example:

Now that we know how to share an ad hoc class with processes using a manager, let’s look at a worked example.

Example of Using a Manager to Share an Object with Processes

We can explore how to use a Manager to share an ad hoc Python object.

In this example, we will use a Manager to host a Python set and share the set among multiple processes.

Recall, we can create a set via the set() built-in function and add items to the set via calling the add() method.

Importantly, if we just created a set and shared it with multiple processes to modify concurrently, it would not have the desired effect. The changes would be made locally within each process and not shared among all processes.

Firstly, we can define our custom Manager instance that we will use to register and host our set instance.

The CustomManager class is defined below.

Next, we can define a function to execute in each child process.

The function will take a unique integer identifier and the shared set as arguments. It will generate a random number between 0 and 1, block for a fraction of a second to simulate computational effort, then add a tuple of its integer argument and generated number to the shared set.

The task() function below implements this.

Finally, in the main process, we will first register our set class with our custom Manager class.

This can be achieved by calling the register() class method on our CustomManager class and specifying the name of the class we are registering (which is arbitrary, but we will match the actual class name) and the actual Python class it maps to.

Next, we can create and start an instance of our custom manager.

In this case, we will use the context manager interface.

We will then use the manager instance to create a new instance of our Python set class.

We will share this set object among many child processes.

Next, we will create and configure 50 child processes to execute our custom task() function with an integer and the share set as arguments.

This can be achieved in a list comprehension which will create a list of configured Process instances.

The main process can then traverse the list of Process instances and start each. Then iterate the same list and join each process in turn, having the effect of blocking waiting for all tasks to complete.

If you are new to joining processes and waiting for them to terminate, you can learn more in the tutoria:

• How to Join a Process in Python

Once all tasks are completed, the main process will report the total number of items in the shared set.

If the share was not hosted on the Manager, then the set would be empty as each process would have added an item to a local copy of the set object.

We cannot check the size of the shared set object directly, because we only have access to a proxy object representation. We can get access to the underlying set object via the _getvalue() function. We can then calculate the length of the set on the value returned.

We can also report the contents of the set.

Tying this together, the complete example is listed below.

Running the example first registers the set class with the custom manager class via a static class function.

Next, an instance of the custom manager class is created and started, starting a new server process for the manager.

A single centralized instance of a Python set object is then created using the manager and a proxy object for the instance is then returned.

A total of 50 child processes are configured to execute our custom task() function and pass an integer argument and the proxy object for the share set.

The child processes are then started and the main process blocks, waiting for all child processes to terminate.

Each child process generates a random number, blocks for a fraction of a second, then adds a tuple to the shared set containing its integer argument and the generated value.

The proxy objects ensure that all method calls on the centralized set object are process-safe. This ensures two things:

1. No race condition is possible by two processes modifying the set object at the same time.
2. Changes made to the hosted and centralized object are available to all other processes.

All tasks complete and the main process continues on.

It first reports the size of the shared set, which matches the execution of 50 items for the 50 tasks that were started.

Next, the content of the set is reported, showing that we have one item from 0 to 49 for each of the 50 tasks that added a tuple.

Further Reading

This section provides additional resources that you may find helpful.

Python Multiprocessing Books

• Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Multiprocessing API Cheat Sheet

I would also recommend specific chapters in the books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism
• PEP 371 - Addition of the multiprocessing package

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to use a Manager to share an ad hoc Python object with multiple processes.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Arnaud Mesureur on Unsplash