You can share changes to Python object instance variables among processes by changing instance variables to be shared ctypes.
In this tutorial, you will discover how to share changes to Python object instance variables between processes.
Let’s get started.
Object Instance Variables Not Shared With Processes
Python objects can be shared among processes.
For example, we can start a child process to execute a target function and pass an object as an argument.
We might also send an argument to another process via a shared queue.
A problem with sharing Python objects among processes is that changes to those objects do not propagate.
That is, if the object has instance variables (attributes) and those instance variables are changed in a child process, then the changes to those instances will not be reflected in any other processes that share the object.
Why aren’t changes to instance variables propagated among processes?
Moreover, how can we update custom Python objects so that changes in one process propagate to all other processes?
Run loops using all CPUs, download your FREE book to learn how.
How to Share Instance Variables
Changes to Python objects are not shared among processes because each process gets a copy of the object.
This explains why changes to a Python object in one process are not reflected in another process. Each process has a separate and local copy of the object.
There are a few ways to overcome this problem when using process-based concurrency.
One approach is to host one central copy of the object in a server process and share proxy objects that know how to interact with the hosted object.
This can be achieved using a manager.
You can learn more about how to host a centralized version of a Python object in the tutorial:
Another approach is to change the custom Python object so that the instance variables themselves are sharable.
This can be achieved by using shared ctypes for instance variables in custom objects.
This means that each process will get a copy of the Python object, but there will only be one copy of each instance variable and changes to the variables will be propagated among processes.
For example, a shared ctype can be defined using a common data type, like an integer or a floating point value.
|
1 2 3 |
... # define shared ctype data = Value('i', 0) |
The value of the shared ctype can be accessed, and modified via the “value” attribute.
For example:
|
1 2 3 4 5 |
... # update the value data.value = 100 # report value print(data.value) |
You can learn more about shared ctypes in the tutorial:
Now that we know why changes to Python objects don’t propagate to processes and how to update custom Python objects so that changes propagate, let’s look at some worked examples.
Example of Instance Variables of Shared Object Not Updated
Before we look at an example where changes to instance variables are propagated, let’s look at a failure case.
In this example, we will define a custom class that has a single instance variable. The class has a method to change the variable and another to retrieve it.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# custom class class CustomClass(): # constructor def __init__(self): # define instance variable self._var = 0 # increment the variable def update(self, value): self._var = value # get the value def get(self): return self._var |
We can then define a task() function that takes an instance of our custom class as an argument and updates the instance variable.
|
1 2 3 4 5 6 7 8 |
# function executed in child process def task(custom): # report the value print(f'Before update: {custom.get()}') # update the value custom.update(100) # report the value print(f'After update: {custom.get()}') |
The parent process creates an instance of the custom class, then creates a child process to run the task() function and passes it the instance of the custom class.
|
1 2 3 4 5 6 7 |
... # create the custom object custom = CustomClass() # configure a child process to run the task process = Process(target=task, args=(custom,)) # start the process process.start() |
The parent process waits for the child process to complete and then reports the value of the instance variable in the instance of the custom class.
|
1 2 3 4 5 |
... # wait for the process to terminate process.join() # report the value print(f'Main: {custom.get()}') |
Naively, we would expect the object to be updated in the child process and for the updated value to be reported in the parent process.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
# SuperFastPython.com # example of instance variables not being shared between processes from multiprocessing import Process # custom class class CustomClass(): # constructor def __init__(self): # define instance variable self._var = 0 # increment the variable def update(self, value): self._var = value # get the value def get(self): return self._var # function executed in child process def task(custom): # report the value print(f'Before update: {custom.get()}') # update the value custom.update(100) # report the value print(f'After update: {custom.get()}') # protect the entry point if __name__ == '__main__': # create the custom object custom = CustomClass() # configure a child process to run the task process = Process(target=task, args=(custom,)) # start the process process.start() # wait for the process to terminate process.join() # report the value print(f'Main: {custom.get()}') |
Running the example first creates an instance of the custom class.
A child process instance is created and configured to execute the task() function and is passed the instance of our custom class.
The child process is started and the main process blocks and waits for the child process to terminate.
The task() function is executed, reports a message, updates the instance variable in the custom object, then reports a second message showing that the instance variable was changed.
The child process terminates and the main process resumes and reports the value of the instance variable in the custom object.
In this case, it shows that the instance variable is still zero and that the change made by the child process had no effect or was lost.
|
1 2 3 |
Before update: 0 After update: 100 Main: 0 |
The reason for this is that the child process received and operated upon a copy of the instance of the custom class. The object was not shared as expected.
Next, let’s look at how we can update the shared object so that changes to instance variables are propagated as we expect.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Instance Variables of Shared Object Updated
We can update a custom class so that instance variables are defined using shared ctypes.
Any shared ctypes shared among processes will have their changes propagated to all other processes. This happens automatically behind the scenes using inter-process communication.
Our CustomClass definition has a single integer instance variable that can be changed to a shared ctype with an integer type and initialized to zero. Access and change to the instance variable can occur via the “value” attribute on the shared ctype
The updated CustomClass class with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# custom class class CustomClass(): # constructor def __init__(self): # define instance variable self._var = Value('i', 0) # increment the variable def update(self, value): self._var.value = value # get the value def get(self): return self._var.value |
And that’s it.
All changes are encapsulated within the CustomClass class.
The complete example with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
# SuperFastPython.com # example of shared instance variables between processes using shared ctypes from multiprocessing import Process from multiprocessing import Value # custom class class CustomClass(): # constructor def __init__(self): # define instance variable self._var = Value('i', 0) # increment the variable def update(self, value): self._var.value = value # get the value def get(self): return self._var.value # function executed in child process def task(custom): # report the value print(f'Before update: {custom.get()}') # update the value custom.update(100) # report the value print(f'After update: {custom.get()}') # protect the entry point if __name__ == '__main__': # create the custom object custom = CustomClass() # configure a child process to run the task process = Process(target=task, args=(custom,)) # start the process process.start() # wait for the process to terminate process.join() # report the value print(f'Main: {custom.get()}') |
Running the example first creates an instance of the custom class.
A child process instance is created and configured to execute the task() function and is passed the instance of our custom class.
The child process is started and the main process blocks and waits for the child process to terminate.
The task() function is executed, reports a message, updates the instance variable in the custom object, then reports a second message showing that the instance variable was changed.
The child process terminates and the main process resumes and reports the value of the instance variable in the custom object.
In this case, it shows that the changes made to the instance variable in the “shared” object were propagated as we hoped.
|
1 2 3 |
Before update: 0 After update: 100 Main: 100 |
Although the object was shared with the child process, it was still copied as before. In this case, the instance variable was not copied, and instead access to the same variable was shared among both child and parent processes.
Changes made to the instance variable in the child process were propagated to the instance variable in the parent process and we can see this when the main process reported the value at the end of the program.
This highlights how we can update instance variables in a custom object so that changes are propagated among processes.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to share changes to Python object instance variables between processes.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Jannis Lucas on Unsplash
Do you have any questions?