Last Updated on November 23, 2022
You can share process class attributes via multiprocessing.Value and multiprocessing.Array instances.
In this tutorial you will discover how to share attributes on the process class between processes.
Let’s get started.
Need Process Class Attributes
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In concurrent programming we may need to extend the multiprocessing.Process class and add attributes that need to be accessed from multiple processes.
For example, we may extend the process class and add an attribute that is set by the child process and accessed by the parent process.
This will cause an error if implemented naively.
How can we share attributes on the process class among multiple processes?
Run loops using all CPUs, download your FREE book to learn how.
Process Class Attributes Fail
Extending the multiprocessing.Process and adding attributes that are shared among multiple processes will fail with an error.
For example, if we define a new class that extends the multiprocessing.Process class that sets an attribute on the class instance from the run() method executed in a new child process, then this attribute will not be accessible by other processes, such as the parent process.
This is the case even if both parent and child processes share access to the “same” object.
This is because class instance variables are not shared among processes by default. Instead, instance variables added to the multiprocessing.Process are private to the process that added them.
Each process operates on a serialized copy of the object and any changes made to that object are local to that process only, by default.
We can demonstrate this with worked examples.
Set Process Class Attribute in run()
Instance variable attributes added to a class that extends the multiprocessing.Process class will be local to that process only.
We can demonstrate this with an example.
In this example, we will define a new class that extends the multiprocessing.Process. It overrides the run() function that is executed in a child process and defines a new instance variable on the class. The instance variable exists and is accessible in the child process, but is unknown and inaccessible in the parent process.
First, we can define a new class that extends the multiprocessing.Process class.
We will call this class CustomProcess.
|
1 2 3 |
# custom process class class CustomProcess(Process): # ... |
First, we will define a constructor that calls the constructor on the parent class as the first step. We will need to make use of this constructor in later examples.
|
1 2 3 4 |
# override the constructor def __init__(self): # execute the base constructor Process.__init__(self) |
Next, we can override the run() method which will be executed in a child process.
The task will block for one second with a call to time.sleep(), then set an integer value in an instance variable named “data“, then report that the instance variable has been set.
|
1 2 3 4 5 6 7 8 |
# override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data = 99 # report stored value print(f'Child stored: {self.data}') |
Tying this together, the complete CustomProcess class is defined below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# custom process class class CustomProcess(Process): # override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data = 99 # report stored value print(f'Child stored: {self.data}') |
In the main parent process, we will first create an instance of this new class, then start the child process.
This will execute the run() function in a new child process.
|
1 2 3 4 5 |
... # create the process process = CustomProcess() # start the process process.start() |
The parent process will then block until the child process is finished, then report the value of the “data” instance variable set by the child process in the run() method.
|
1 2 3 4 5 6 7 |
... # wait for the process to finish print('Waiting for the child process to finish') # block until child process is terminated process.join() # report the process attribute print(f'Parent got: {process.data}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of extending the Process class and adding shared attributes from time import sleep from multiprocessing import Process # custom process class class CustomProcess(Process): # override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data = 99 # report stored value print(f'Child stored: {self.data}') # entry point if __name__ == '__main__': # create the process process = CustomProcess() # start the process process.start() # wait for the process to finish print('Waiting for the child process to finish') # block until child process is terminated process.join() # report the process attribute print(f'Parent got: {process.data}') |
Running the example first creates an instance of the custom class, then starts the child process.
The parent process then blocks until the child process terminates.
The child process executes the run() method, blocks, sets an integer value in the instance variable, reports the stored value then exits.
The parent process continues on, then attempts to report the value of the instance variable set by the child process.
This fails with an error that no such instance variable attribute exists.
This was expected.
|
1 2 3 4 5 |
Waiting for the child process to finish Child stored: 99 Traceback (most recent call last): ... AttributeError: 'CustomProcess' object has no attribute 'data' |
This error occurred because the child process operates on a copy of the CustomProcess instance that is different from the copy of the CustomProcess instance used in the parent process.
When the child process adds an attribute to the class instance, it only exists in the child process and not the parent process.
What if we initialize the attribute in the class constructor?
We will explore this case in the next example.
Initialize Process Class Attribute in __init__()
In the previous example, we saw that we cannot set an instance variable attribute on a class instance from a child process and access it from the parent process.
We might consider the case of first initializing the attribute in the class constructor.
This will mean that the attribute will be accessible by both parent and child processes. If the child process sets a value of the attribute, will the parent process see the value?
We can explore this by updating the example in the previous section to first define the attribute in the class constructor.
The attribute can be initialized to the value of None.
|
1 2 3 4 5 6 |
# override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # initialize attribute self.data = None |
Tying this together, the updated CustomProcess class is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# custom process class class CustomProcess(Process): # override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # initialize attribute self.data = None # override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data = 99 # report stored value print(f'Child stored: {self.data}') |
And that’s it.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# SuperFastPython.com # example of extending the Process class and adding shared attributes from time import sleep from multiprocessing import Process # custom process class class CustomProcess(Process): # override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # initialize attribute self.data = None # override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data = 99 # report stored value print(f'Child stored: {self.data}') # entry point if __name__ == '__main__': # create the process process = CustomProcess() # start the process process.start() # wait for the process to finish print('Waiting for the child process to finish') # block until child process is terminated process.join() # report the process attribute print(f'Parent got: {process.data}') |
Running the example first creates an instance of the custom class, then starts a child process that executes the run() method.
When an instance of the class is created, the attribute is initialized.
As before, the parent process blocks until the child process terminates.
The child process sets a value in the instance variable and terminates.
The parent process unblocks then reports the value of the instance variable.
In this case, the child process reports the correct value it stored then the parent value reports a value of None in the instance variable.
This too is expected.
|
1 2 3 |
Waiting for the child process to finish Child stored: 99 Parent got: None |
The parent process can access the attribute. This is because it was defined in the class constructor which was executed in the parent process. Then the child process was started which executes a copy of the class instance and sets a value to the attribute which is only accessible within the child process.
The parent process reports the attribute value which is still None because the change made by the child process was not shared with the parent process.
Next, let’s look at how we can share attributes between child and parent processes.
How to Share Process Class Attributes
Instance variable attributes can be shared between processes via the multiprocessing.Value and multiprocessing.Array classes.
These classes explicitly define data attributes designed to be shared between processes in a thread-safe manner.
A thread-safe manner means that only one process can read or access the variable at a time.
Shared variables mean that changes made in one process are always propagated and made available to other processes.
We will focus on the multiprocessing.Value class in this case. In fact, from reviewing the source code, the multiprocessing.Value is just a wrapper around the multiprocessing.sharedctypes.Value class.
An instance of the multiprocessing.Value can be defined in the constructor of a custom class as a shared instance variable.
The constructor of the multiprocessing.Value class requires that we specify the data type and an initial value.
The data type can be specified using ctype “type” or a typecode.
You can learn more about ctypes here:
Typecodes are familiar and easy to use, for example ‘i’ for a signed integer or ‘f’ for a single floating-point value.
You can see a handy table of type codes here:
For example, we can define a multiprocessing.Value shared memory variable that holds a signed integer and is initialized to the value zero.
|
1 2 3 |
... # initialize an integer shared variable data = multiprocessing.Value('i', 0) |
This can be initialized in the constructor of the class that extends the multiprocessing.Process class.
We can change the value of the shared data variable via the “value” attribute.
For example:
|
1 2 3 |
... # change the value of the shared variable data.value = 100 |
We can access the value of the shared data variable via the same “value” attribute.
For example:
|
1 2 3 |
... # access the shared variable value = data.value |
The propagation of changes to the shared variable and mutual exclusion locking of the shared variable is all performed automatically behind the scenes.
You can learn more about shared ctypes in the tutorial:
Now that we know how to define an instance variable that will be correctly shared between processes, let’s look at a worked example.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Sharing Process Class Attributes
We can define an instance attribute as an instance of the multiprocessing.Value which will automatically and correctly be shared between processes.
We can update the above example to use a multiprocessing.Value directly.
Firstly, we must update the constructor of the CustomProcess class to initialize the multiprocessing.Value instance. We will define it to be an integer with an initial value of zero.
|
1 2 3 |
... # initialize integer attribute self.data = Value('i', 0) |
The updated constructor with this change is listed below.
|
1 2 3 4 5 6 |
# override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # initialize integer attribute self.data = Value('i', 0) |
We can then update the run() method to change the “value” attribute on the “data” instance variable and then to report this value via a print statement.
|
1 2 3 4 5 |
... # store the data variable self.data.value = 99 # report stored value print(f'Child stored: {self.data.value}') |
The updated run() function with this change is listed below.
|
1 2 3 4 5 6 7 8 |
# override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data.value = 99 # report stored value print(f'Child stored: {self.data.value}') |
Tying this together, the updated CustomProcess class is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# custom process class class CustomProcess(Process): # override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # initialize integer attribute self.data = Value('i', 0) # override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data.value = 99 # report stored value print(f'Child stored: {self.data.value}') |
Finally, in the parent process, we can update the print statement to correctly access the value of the shared instance variable attribute.
|
1 2 3 |
... # report the process attribute print(f'Parent got: {process.data.value}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
# SuperFastPython.com # example of extending the Process class and adding shared attributes from time import sleep from multiprocessing import Process from multiprocessing import Value # custom process class class CustomProcess(Process): # override the constructor def __init__(self): # execute the base constructor Process.__init__(self) # initialize integer attribute self.data = Value('i', 0) # override the run function def run(self): # block for a moment sleep(1) # store the data variable self.data.value = 99 # report stored value print(f'Child stored: {self.data.value}') # entry point if __name__ == '__main__': # create the process process = CustomProcess() # start the process process.start() # wait for the process to finish print('Waiting for the child process to finish') # block until child process is terminated process.join() # report the process attribute print(f'Parent got: {process.data.value}') |
Running the example first creates an instance of the custom class then starts the child process.
The constructor initializes the instance variable to be a multiprocessing.Value instance with an initial value of zero.
The parent process blocks until the child process terminates.
The child process blocks, then changes the value of the instance variable and reports the change. The change to the instance variable is propagated back to the parent process.
The child process terminates and the parent process continues on. It reports the value of the instance variable which correctly reflects the change made by the child process.
This demonstrates how to share instance variable attributes among parents via the multiprocessing.Value class.
|
1 2 3 |
Waiting for the child process to finish Child stored: 99 Parent got: 99 |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to share process class attributes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Jill Heyer on Unsplash
I need to start child processes as class instances delivering additional arguments by calling the ‘run’ method, e.g. your example modified to:
class CustomProcess(): # not of class type ‘Process’
def init(self):
self.data = Value(‘i’, 0)
: :
def run(self, val):
self.data.value = val
: :
if name == ‘main‘:
: :
procInst = CustomProcess() # create an instance
: :
process = Process(target=procInst.run, args=(val, )) # create process, start calling the ‘run’ method
: :
In this case i again get the message ‘CustomProcess’ object has no attribute ‘data’.
How can a process be started calling its ‘run’ method but also sharing a ‘Value’ object with the parent?
Hi Eduard,
I think you’re asking how to execute an object method in a new process and have the method assign a variable value that is an instance variable of the object, and that the variable is a process-safe multiprocessing.Value.
This should just work directly.
Below is a complete working example:
This results in:
Perhaps there is a bug in your specific example.
Hi Jason
I am working on a huge NLP project where multiple processes are working on tasks involving NLP inference.
I am having two different NLP models that every process is using.
AFAIK Python will copy every one of these objects for every subprocess when the process gets created.
As these models are only read-only, I’m wondering if we can somehow share access to these objects between different processes?
I was trying with global keyword but it seems processes are blocking each other?
I am getting a speedup from 1 to 2 processes but then it stays plus minus the same.
Cheers,
Fabian
Good question, Fabian.
Generally, you could centralize the model using a multiprocessing.Manager, and have each process access the model via proxy objects. I have a tutorial on how to share ad hoc objects using a manager scheduled, it should appear on the blog soon. Until then, this may help:
https://docs.python.org/3/library/multiprocessing.html#managers
I would not recommend this though.
I suspect the slow part of the program involves using the model, e.g. having the model perform inference. If so, you will want a copy of the model in each process to perform inference in parallel.
Perhaps you can benchmark in order to discover the slow part of the program that would benefit from concurrency.