You can create and share a memory block between processes via the SharedMemory class.
In this tutorial, you will discover how to use shared memory between processes in Python.
Let’s get started.
What is SharedMemory
The multiprocessing.shared_memory.SharedMemory class allows a block of memory to be used by multiple Python processes.
A SharedMemory object can be created and shared directly among multiple processes, or it can assigned a meaningful name attached to a process using that name,
Creates a new shared memory block or attaches to an existing shared memory block. Each shared memory block is assigned a unique name. In this way, one process can create a shared memory block with a particular name and a different process can attach to that same shared memory block using that same name.
— multiprocessing.shared_memory — Shared memory for direct access across processes
A SharedMemory has a fixed size and stores byte data.
Python types can be converted to arrays of bytes and stored in a SharedMemory and read as arrays of bytes and converted back into Python types.
It allows processes to read and write from the same memory, which is faster and more efficient than sharing data via message passing, such as via a multiprocessing.Queue or multiprocessing.Pipe.
Processes are conventionally limited to only have access to their own process memory space but shared memory permits the sharing of data between processes, avoiding the need to instead send messages between processes containing that data. Sharing data directly via memory can provide significant performance benefits compared to sharing data via disk or socket or other communications requiring the serialization/deserialization and copying of data.
— multiprocessing.shared_memory — Shared memory for direct access across processes
Now that we know what a SharedMemory object is, let’s look at how to use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use SharedMemory
A SharedMemory can be created in a process by calling the constructor and specifying a “size” in bytes and the “create” argument to True.
For example:
|
1 2 3 |
... # create a shared memory shared_mem = SharedMemory(size=1024, create=True) |
A shared memory object can be assigned a meaningful name via the “name” attribute to the constructor.
For example:
|
1 2 3 |
... # create a shared memory with a name shared_mem = SharedMemory(name='MyMemory', size=1024, create=True) |
Another process can access a shared memory via its name. This is called attaching to a shared memory.
This can be achieved by specifying the name of the shared memory that has already been created and setting the “create” argument to False (the default).
For example:
|
1 2 3 |
... # attach to a shared memory shared_mem = SharedMemory(name='MyMemory', create=False) |
Once created data can be stored in the shared memory via the “buf” attribute that acts like an array of bytes.
For example:
|
1 2 3 |
... # write data to shared memory shared_mem.buf[0] = 1 |
Data can be read from the “buf” attribute in the same manner.
For example:
|
1 2 3 |
... # read data from shared memory data = shared_mem[0] |
Once a process is finished using the shared memory, it can be closed to signal that access is no longer required.
|
1 2 3 |
... # close access to the shared memory shared_mem.close() |
All processes should close the shared memory once they are finished with it.
Closes access to the shared memory from this instance. In order to ensure proper cleanup of resources, all instances should call close() once the instance is no longer needed. Note that calling close() does not cause the shared memory block itself to be destroyed.
— multiprocessing.shared_memory — Shared memory for direct access across processes
Once all processes are finished with the shared memory, it must be explicitly released.
This can be achieved by calling the unklink() method.
Requests that the underlying shared memory block be destroyed. In order to ensure proper cleanup of resources, unlink() should be called once (and only once) across all processes which have need for the shared memory block.
— multiprocessing.shared_memory — Shared memory for direct access across processes
Ideally, the process that created the shared memory would also release it.
For example:
|
1 2 3 |
... # destroy the shared memory shared_mem.unlink() |
Now that we know how to use the SharedMemory object, let’s look at some worked examples.
Example of Using SharedMemory with Strings
We can explore an example of how to share a SharedMemory object between processes and use it to store and retrieve String data.
In this example, we can define a task that executes in a child process. The task function tasks the shared memory as an argument and writes a string to the shared memory. The main process starts the child process, waits for it to complete, then reports the data in the shared memory.
Firstly, we can define the task function that takes the shared memory as an argument.
String data can be written to the shared memory via the “buf” attribute. Recall that a string can be converted to bytes by prepending the “b” expression to the front of it, automatically encoding the string.
|
1 2 3 4 5 6 |
# task executed in a child process def task(shared_mem): # write some string data to the shared memory shared_mem.buf[:24] = b'Hello from child process' # close as no longer needed shared_mem.close() |
The main process can first create the shared memory.
|
1 2 3 |
... # create a shared memory shared_mem = SharedMemory(create=True, size=100) |
Next, the child process can be created and configured to execute our task() function and pass it the shared memory as an argument. The process can be started and the main process can wait for it to complete.
|
1 2 3 4 5 6 7 |
... # create a child process process = Process(target=task, args=(shared_mem,)) # start the child process process.start() # wait for the child process to finish process.join() |
Next, the main process can read the string data from the shared memory.
Recall that we can convert an array of bytes to a string via the decode() method.
|
1 2 3 4 |
... # report the shared memory data = bytes(shared_mem.buf[:24]).decode() print(data) |
Finally, the main process can close and release the shared memory block.
|
1 2 3 4 5 |
... # close the shared memory shared_mem.close() # release the shared memory shared_mem.unlink() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of using shared memory with strings from multiprocessing.shared_memory import SharedMemory from multiprocessing import Process # task executed in a child process def task(shared_mem): # write some string data to the shared memory shared_mem.buf[:24] = b'Hello from child process' # close as no longer needed shared_mem.close() # protect the entry point if __name__ == '__main__': # create a shared memory shared_mem = SharedMemory(create=True, size=100) # create a child process process = Process(target=task, args=(shared_mem,)) # start the child process process.start() # wait for the child process to finish process.join() # report the shared memory data = bytes(shared_mem.buf[:24]).decode() print(data) # close the shared memory shared_mem.close() # release the shared memory shared_mem.unlink() |
Running the example first creates the shared memory.
The child process is then created and configured to run our custom function and pass the shared memory.
The main process starts the child process and waits for it to complete.
The child process runs and stores a string in the shared memory, then terminates.
The main process resumes and retrieves the bytes data from the shared memory and decodes it into a string, then reports it.
We can see that the reported string matches the string that was stored in the child process.
The shared memory is then closed and released.
This highlights how we can share string data between processes using shared memory.
|
1 |
Hello from child process |
Next, let’s look at sharing integer data between processes using shared memory.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Using SharedMemory with Integers
We can explore an example of how to share a SharedMemory object between processes and use it to store and retrieve integer data.
In this example, we can update the previous example to store and retrieve a list of integers from the shared memory.
Firstly, we can update the task() function to store a list of integers. This can be achieved by converting the list into an array of bytes via the bytearray() built-in function.
|
1 2 3 4 5 6 |
# task executed in a child process def task(shared_mem): # write some integer data to the shared memory shared_mem.buf[:5] = bytearray([1, 2, 3, 4, 5]) # close as no longer needed shared_mem.close() |
Next, in the main process, we can retrieve the integer data from the shared memory buffer.
There are many ways to do this. In this case, we will read each item in turn and cast them to an integer type using a list comprehension.
|
1 2 3 |
... # report the shared memory data = [int(shared_mem.buf[i]) for i in range(5)] |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of using shared memory with integers from multiprocessing.shared_memory import SharedMemory from multiprocessing import Process # task executed in a child process def task(shared_mem): # write some integer data to the shared memory shared_mem.buf[:5] = bytearray([1, 2, 3, 4, 5]) # close as no longer needed shared_mem.close() # protect the entry point if __name__ == '__main__': # create a shared memory shared_mem = SharedMemory(create=True, size=100) # create a child process process = Process(target=task, args=(shared_mem,)) # start the child process process.start() # wait for the child process to finish process.join() # report the shared memory data = [int(shared_mem.buf[i]) for i in range(5)] print(data) # close the shared memory shared_mem.close() # release the shared memory shared_mem.unlink() |
Running the example first creates the shared memory.
The child process is then created and configured to run our custom function and pass the shared memory.
The main process starts the child process and waits for it to complete.
The child process runs and stores a list of integers in the shared memory, then terminates.
The main process resumes and retrieves the bytes data from the shared memory and casts it into integer data, then reports the list.
We can see that the list of retrieved integers matches those values stored by the child process.
The shared memory is then closed and released.
This highlights how we can share a list of integer data between processes using shared memory.
|
1 |
[1, 2, 3, 4, 5] |
Next, let’s look at sharing the shared memory with another process by attaching to it.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Attaching to a SharedMemory Block
We can explore an example of how to share a SharedMemory object between processes by attaching to it by its unique name.
In this example, we can update the first example of sharing a string between processes using shared memory. In this case, we won’t pass the shared memory object to the child process as an argument, instead, we will attach to it directly from the child process.
For example:
|
1 2 3 |
... # attach another shared memory block sm = SharedMemory('MyMemory') |
The updated task function with this change is listed below.
|
1 2 3 4 5 6 7 8 |
# task executed in a child process def task(): # attach another shared memory block sm = SharedMemory('MyMemory') # store data in the first shared memory block sm.buf[:11] = b'Hello world' # close as no longer needed sm.close() |
We must create the shared memory in the main process using the meaningful name so the child process can attach to it.
For example:
|
1 2 3 |
... # create a shared memory shared_mem = SharedMemory(name='MyMemory', create=True, size=100) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of attaching shared memory blocks from multiprocessing.shared_memory import SharedMemory from multiprocessing import Process # task executed in a child process def task(): # attach another shared memory block sm = SharedMemory('MyMemory') # store data in the first shared memory block sm.buf[:11] = b'Hello world' # close as no longer needed sm.close() # protect the entry point if __name__ == '__main__': # create a shared memory shared_mem = SharedMemory(name='MyMemory', create=True, size=100) # create a child process process = Process(target=task) # start the child process process.start() # wait for the child process to finish process.join() # report the data in the shared memory print(bytes(shared_mem.buf[:11]).decode()) # close the shared memory shared_mem.close() # release the shared memory shared_mem.unlink() |
Running the example first creates the shared memory.
The child process is then created and configured to run our custom function. The child process does not pass the shared memory as an argument.
The main process starts the child process and waits for it to complete.
The child process runs and first attaches to the shared memory. It then stores a string in the shared memory and terminates.
The main process resumes and retrieves the bytes data from the shared memory and decodes it into a string, then reports it.
We can see that the reported string matches the string that was stored in the child process.
The shared memory is then closed and released.
This highlights how we can attach to shared memory from another process by name alone.
|
1 |
Hello world |
Next, let’s look at what happens if we fail to shut down the shared memory correctly.
Example of Failing to Close and Release Shared Memory
We can explore the case of failing to shut down the shared memory.
In this example, we will create a shared memory object and not call the close() or unlink() methods.
We expect this to result in a memory leak as the memory is never explicitly destroyed.
|
1 2 3 4 5 6 7 8 9 |
# SuperFastPython.com # example of failing to close and release the shared memory from multiprocessing.shared_memory import SharedMemory # protect the entry point if __name__ == '__main__': # create a shared memory shared_mem = SharedMemory(name='MyMemory', create=True, size=100) # fail to close and release the shared memory... |
Running the example first crests the shared memory, then the main process terminates.
This raises a user warning reporting that the shared memory was not shut down, resulting in a memory leak.
This highlights the importance of shutting down shared memory correctly.
|
1 2 |
UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to use shared memory between processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Nathan Trampe on Unsplash
Do you have any questions?