Last Updated on February 21, 2024
Methods of objects hosted in a manager are executed by threads in the manager’s server process.
In this tutorial you will discover the multiprocessing manager server process and threads used to execute code of hosted objects.
Let’s get started.
What is a Multiprocessing Manager
A manager in the multiprocessing module provides a way to create Python objects that can be shared easily between processes.
Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.
— multiprocessing — Process-based parallelism
A manager creates a server process that hosts a centralized version of objects that can be shared among multiple processes.
The objects are not shared directly. Instead, the manager creates a proxy object for each object that it manages and the proxy objects are shared among processes.
The proxy objects are used and operate just like the original objects, except that they serialize data, synchronize and coordinate with the centralized version of the object hosted in the manager server process.
A proxy is an object which refers to a shared object which lives (presumably) in a different process. […] A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy).
— multiprocessing — Process-based parallelism
This makes managers a process-safe and preferred way to share Python objects among processes.
You can learn more about multiprocessing managers in the tutorial:
Next, let’s consider the server process of the manager itself.
Run loops using all CPUs, download your FREE book to learn how.
What Process and Thread Execute the Object in the Manager?
When we start a manager, it starts a server process where centralized objects are created.
We provide access to the hosted objects via proxy objects.
Nevertheless, we may have some basic questions about the manager’s process.
For example:
- Is the manager’s process a child of the main process?
- Which process and thread executes the constructor of the hosted object?
- What process and thread executes a method on a shared object?
We can explore these questions.
For example, we can call the multiprocessing.active_children() function to get a list of all active child processes after a manager has been started, to see if the manager’s server process is a child of the main process.
The multiprocessing.current_process() function can be used to get a multiprocessing.Process instance for the process that is executing the current code.
Similarly, we can use the threading.current_thread() function to get the threading.Thread instance for the thread that is executing the current code.
We can then define a custom object that may be created and hosted in a manager. The object can check the current process and thread when it is constructed and when custom methods are called, answering our questions.
Let’s explore these questions.
Is Manager’s Process a Child of the Main Process?
We can explore whether the process created by the manager is a child of the main process.
In this example we will create a manager in the main process, then get a list and report the details of all active child processes of the main process. If the manager process is a child of the main process, then its details will be reported.
We can get a list of all child processes for the current process via the multiprocessing.active_children() function.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# SuperFastPython.com # example determining if the main process can see the manager's process from multiprocessing import active_children from multiprocessing import Manager # protect the entry point if __name__ == '__main__': # create and start the manager with Manager() as manager: # report all active child processes for child in active_children(): print(child) |
Running the example first creates a manager and starts it using the context manager interface.
Next, the main process gets a list of all active child processes and reports the details of each.
In this case, we can see that the main process has a single child process and its name is “SyncManager-1“.
This confirms that the manager’s server process is a child of the main process (or the process that created it).
It also highlights that the manager names the process after the class used to create it. In this case, the name of the manager’s process is named after the multiprocessing.managers.SyncManager class. This is because when we call multiprocessing.Manager() to create a manager, it creates an instance of the SyncManager class.
Note, the process identifiers will be different each time the program is run because they are assigned by the operating system.
Next, let’s explore what thread and process execute code on an object hosted in a manager.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
What Thread and Process Execute a Centralized Object?
We can explore what thread and process execute code on an object hosted in a multiprocessing manager.
This is a little more elaborate than the previous example. It requires that we first define a custom class with a constructor and a method that reports the current thread and process executing them. It then requires that we define a custom manager class so that we can register a custom class that may be created and managed by a manager process. We can then create the custom manager, use the manager to create the custom class, then call methods on the custom class to see what thread and process execute them.
For more on how to host custom objects in a multiprocessing manager, see the tutorial:
First, we can define a custom class.
The class will define a constructor that will call the threading.current_thread() function to get the current thread instance and multiprocessing.current_process() to get the current process instance and then report the details of each.
The class will also define a method that will report the same details, when called.
The CustomClass below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# custom class class CustomClass(): # constructor def __init__(self): # get the current process and thread process = current_process() thread = current_thread() # report details print(f'Constructor:\n {thread}\n {process}') # do something with the data def task(self): # get the current process and thread process = current_process() thread = current_thread() # report details print(f'Task:\n {thread}\n {process}') |
Next, we can define a custom manager class.
This is required so that we can register a custom class with the manager and have the manager create and host it, allowing processes to interact with it via proxy objects.
The custom manager must extend the BaseManager class.
It then does not need to implement or override any methods.
The CustomManager class below implements this.
|
1 2 3 4 |
# custom manager to support custom classes class CustomManager(BaseManager): # nothing pass |
In the main process, we will first report the name of the current thread and the current process.
We expect this to be the MainThread of the MainProcess.
|
1 2 3 4 5 6 |
... # get the current process and thread process = current_process() thread = current_thread() # report details print(f'Main:\n {thread}\n {process}') |
You can learn more about the main process in the tutorial:
Next, we will register our CustomClass with our CustomManager class.
This is so that the manager knows about the CustomClass and how to create a new instance.
This can be achieved via the register() class function that takes a mapping of string name to register to class name to instantiate.
|
1 2 3 |
... # register the custom class on the custom manager CustomManager.register('CustomClass', CustomClass) |
We will then create an instance of the CustomManager and start it via the context manager interface.
|
1 2 3 4 |
... # create a new manager instance with CustomManager() as manager: # ... |
We can then use the manager to create an instance of our CustomClass.
|
1 2 3 |
... # create a shared custom class instance custom = manager.CustomClass() |
This will create an instance of the class hosted on the server and execute the constructor of our class, reporting the details of the specific thread and process that created the class.
We expect that it will be a thread within the Manager‘s server process.
Finally, we will call a method on our hosted object.
|
1 2 3 |
... # call a method on the custom class custom.task() |
Again, the method will report the thread and process that executed the code on the object. We expect this will be a thread on the Manager‘s server process.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
# SuperFastPython.com # example of reporting details of process executing a hosted object from threading import current_thread from multiprocessing import current_process from multiprocessing import Process from multiprocessing.managers import BaseManager # custom class class CustomClass(): # constructor def __init__(self): # get the current process and thread process = current_process() thread = current_thread() # report details print(f'Constructor:\n {thread}\n {process}') # do something with the data def task(self): # get the current process and thread process = current_process() thread = current_thread() # report details print(f'Task:\n {thread}\n {process}') # custom manager to support custom classes class CustomManager(BaseManager): # nothing pass # protect the entry point if __name__ == '__main__': # get the current process and thread process = current_process() thread = current_thread() # report details print(f'Main:\n {thread}\n {process}') # register the custom class on the custom manager CustomManager.register('CustomClass', CustomClass) # create a new manager instance with CustomManager() as manager: # create a shared custom class instance custom = manager.CustomClass() # call a method on the custom class custom.task() |
Running the example first gets the current thread and process and reports their details.
As expected, we can see that the MainThread of the MainProcess is executing our program.
Next, our CustomClass is registered with our CustomManager.
We then create and start an instance of our CustomManager which starts a server process.
The manager is then used to create an instance of our CustomClass.
This calls the constructor of the class and reports the details of the current thread and process.
In this case, we can see that “Thread-2” of the “CustomManager-1” class executed the constructor of our CustomClass. CustomManager-1 is the server process itself, and Thread-2 must be a helper thread used for maintenance tasks, like creating objects in this case.
We then call the task() method on our hosted object, which reports the thread and process details that perform the call.
In this case, we can see that a thread called “MainProcess” of the “CustomManager-1” executed the method.
This highlights that the proxy object for a hosted object dispatch calls to the manager to execute constructors and methods on the hosted object in the Manager‘s server process. It also highlights that the manager uses helper threads to handle such requests, which is not surprising.
Note, the process identifiers will be different each time the program is run because they are assigned by the operating system.
|
1 2 3 4 5 6 7 8 9 |
Main: <_MainThread(MainThread, started 4584087040)> <_MainProcess name='MainProcess' parent=None started> Constructor: <Thread(Thread-2, started daemon 123145459212288)> <SpawnProcess name='CustomManager-1' parent=34011 started> Task: <Thread(MainProcess, started daemon 123145459212288)> <SpawnProcess name='CustomManager-1' parent=34011 started> |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know that the manager process is a child of the main process and which thread and process execute code on an object hosted in a manager.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by frank mckenna on Unsplash
Do you have any questions?