You can use the SharedMemoryManager to create shared memory and automatically release and destroy all shared memory created by the manager.
This can help to avoid memory leaks when using shared memory with process-based concurrency in Python.
In this tutorial, you will discover how to use the SharedMemoryManager to manage shared memory for processes in Python.
Let’s get started.
What is the SharedMemoryManager
Python provides the ability to create a block of memory that can be shared among processes via the multiprocessing.shared_memory.SharedMemory and multiprocessing.shared_memory.ShareableList classes.
A problem with using these classes is that each object must be explicitly closed and unlinked in order to avoid a memory leak.
The multiprocessing.managers.SharedMemoryManager class provides a multiprocessing manager for easily creating and destroying shared memory in Python.
A subclass of BaseManager which can be used for the management of shared memory blocks across processes.
— multiprocessing.shared_memory — Shared memory for direct access across processes
A SharedMemoryManager can be created and in turn, used to create shared memory blocks like SharedMemory and ShareableList objects.
These objects do not need to be explicitly closed and destroyed. Shutting down the manager will automatically close and release all shared memory that it has created.
This can help to avoid memory leaks when using shared memory among processes in a Python program.
Now that we know about the SharedMemoryManager class, let’s look at how to use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use the SharedMemoryManager
The SharedMemoryManager can be used by creating an example, starting it, using it to create SharedMemory and ShareableList objects, then shutting it down in order to release all memory.
Firstly, an instance of the SharedMemoryManager class must be created.
For example:
|
1 2 3 |
... # create a memory manager manager = SharedMemoryManager() |
Next, the manager must be started.
This can be achieved by calling the start() method.
A call to start() on a SharedMemoryManager instance causes a new process to be started. This new process’s sole purpose is to manage the life cycle of all shared memory blocks created through it.
— multiprocessing.shared_memory — Shared memory for direct access across processes
This may create internal threads or processes to manage the memory blocks we will create and use in our program.
For example:
|
1 2 3 |
... # start the memory manager manager.start() |
Once started, we can use the memory manager to create blocks of shared memory.
For example:
|
1 2 3 4 5 |
... # create a shareable list shared_list = manager.ShareableList([1, 2, 3]) # create a memory block shared_mem = manager.SharedMemory(1024) |
We cannot specify a name when creating ShareableList and SharedMemory objects via the manager.
This means we cannot use the manager to attach to a memory block.
We must also pass around the objects to child processes directly, such as via arguments. We cannot attach to them via name as we could if we managed the objects manually.
Once we are finished, we can release all shared memory managed by the manager.
This can be achieved by calling the shutdown() method.
To trigger the release of all shared memory blocks managed by that process, call shutdown() on the instance. This triggers a SharedMemory.unlink() call on all of the SharedMemory objects managed by that process and then stops the process itself.
— multiprocessing.shared_memory — Shared memory for direct access across processes
This will both close and unlink all shared memory that we have created with the manager.
|
1 2 3 |
... # release all shared memory manager.shutdown() |
We can manage the startup and shutdown process of the manager using the context manager interface.
When using a SharedMemoryManager in a with statement, the shared memory blocks created using that manager are all released when the with statement’s code block finishes execution.
— multiprocessing.shared_memory — Shared memory for direct access across processes
This ensures that the shutdown() method is always called automatically once the block of the context manager is exited.
For example:
|
1 2 3 4 5 6 7 |
... # create and start the manager with SharedMemoryManager() as manager: # create a shareable list shared_list = manager.ShareableList([1, 2, 3]) # ... # shutdown automatically. |
Now that we know how to use the SharedMemoryManager, let’s look at some worked examples.
Example of Using the SharedMemoryManager
We can explore an example of using the SharedMemoryManager to create and release shared memory, in this case, a ShareableList.
In this example, we will use the SharedMemoryManager to create a shared list, then share the list with a child process. The child process will execute a task that changes the data in the shared list. Finally, the main process will report the changed values in the shared memory and then release all the shared memory via the manager.
Firstly, we can define a task to execute in the child process.
The task will take the ShareableList object as an argument and then change the values, in this case by multiplying them by ten.
The task() function is listed below.
|
1 2 3 4 5 |
# task executed in a child process def task(sl): # change the list for i in range(len(sl)): sl[i] = sl[i] * 10 |
Next, in the main process, we can create and start the SharedMemoryManager.
|
1 2 3 4 5 |
... # create the manager manager = SharedMemoryManager() # start the manager manager.start() |
We can then use the memory manager to create a shareable list with five elements and initial values from 1 to 5.
|
1 2 3 4 5 |
... # create a shared list using the manager sl = manager.ShareableList([1, 2, 3, 4, 5]) # report the shared list print(sl) |
A child process can then be started to execute our task() function and pass the ShareableList object. The main process will then block and wait for the child process to terminate.
|
1 2 3 4 5 6 7 |
... # create a child process process = Process(target=task, args=(sl,)) # start the child process process.start() # wait for the child process to finish process.join() |
Finally, the main process can report the contents of the shared memory and shut down the SharedMemoryManager in order to release the shared memory.
|
1 2 3 4 5 |
... # report the shared list print(sl) # shutdown the manager manager.shutdown() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of using a sharedmemorymanager to manage a sharablelist from multiprocessing import Process from multiprocessing.managers import SharedMemoryManager # task executed in a child process def task(sl): # change the list for i in range(len(sl)): sl[i] = sl[i] * 10 # protect the entry point if __name__ == '__main__': # create the manager manager = SharedMemoryManager() # start the manager manager.start() # create a shared list using the manager sl = manager.ShareableList([1, 2, 3, 4, 5]) # report the shared list print(sl) # create a child process process = Process(target=task, args=(sl,)) # start the child process process.start() # wait for the child process to finish process.join() # report the shared list print(sl) # shutdown the manager manager.shutdown() |
Running the example first creates the shared memory manager and starts it.
The shared memory manager is then used to create a shareable list, initialized with specific integer values. The contents of the list with its initial values are reported.
We can see that the ShareableList list is assigned a unique name.
Next, a child process is created and configured to execute our task() function. The process is started, and the main process waits for it to complete.
The child process runs, receiving the shareable list as an argument and multiplying each value in the list by 10.
The child process terminates and the main process resumes. It reports the contents of the list, showing that the child process changed the same shared memory block.
Finally, the memory manager is shut down. This releases the shareable list.
|
1 2 |
ShareableList([1, 2, 3, 4, 5], name='psm_cb89a260') ShareableList([10, 20, 30, 40, 50], name='psm_cb89a260') |
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Using the SharedMemoryManager Using the Context Manager
We can explore the case of using a SharedMemoryManager via the context manager interface to automatically destroy all shared memory once we are finished with it.
In this example, we can update the previous example to use the context manager interface for the SharedMemoryManager rather than explicitly starting and shutting down the memory manager.
This greatly simplifies the use of the memory manager and ensures that regardless of the success or failure of the code in the context manager block, all shared memory created by the manager will always be released.
|
1 2 3 4 5 |
... # create and start the manager with SharedMemoryManager() as manager: # create a shared list using the manager sl = manager.ShareableList([1, 2, 3, 4, 5]) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of sharedmemorymanager to manage a sharablelist using context manager from multiprocessing import Process from multiprocessing.managers import SharedMemoryManager # task executed in a child process def task(sl): # change the list for i in range(len(sl)): sl[i] = sl[i] * 10 # protect the entry point if __name__ == '__main__': # create and start the manager with SharedMemoryManager() as manager: # create a shared list using the manager sl = manager.ShareableList([1, 2, 3, 4, 5]) # report the shared list print(sl) # create a child process process = Process(target=task, args=(sl,)) # start the child process process.start() # wait for the child process to finish process.join() # report the shared list print(sl) |
Running the example first creates the shared memory manager and starts it in one line using the context manager interface.
The shared memory manager is then used to create a shareable list, initialized with specific integer values. The contents of the list with its initial values are reported.
A child process is created and configured to execute our task() function. The process is started, and the main process waits for it to complete.
The child process runs, receiving the shareable list as an argument and multiplying each value in the list by 10.
The child process terminates and the main process resumes. It reports the contents of the list, showing that the child process changed the same shared memory block.
Finally, the memory manager block is exited. This automatically calls the shutdown method on the memory manager and releases all shared memory blocks created by the manager, in this case just our single shareable list.
|
1 2 |
ShareableList([1, 2, 3, 4, 5], name='psm_3d2e3005') ShareableList([10, 20, 30, 40, 50], name='psm_3d2e3005') |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to use the SharedMemoryManager to manage shared memory for processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Håkon Sataøen on Unsplash
Do you have any questions?