You can share memory directly between processes in process-based concurrency using classes in the multiprocessing.shared_memory module.

In this tutorial, you will discover how to use shared memory between processes in Python.

Let’s get started.

Sharing Memory Between Processes

Python processes do not have shared memory.

This is unlike threads that are able to read and write the same memory directly.

Instead, processes must simulate shared memory.

One approach is to have new processes receive a copy of data from another process. This can be achieved by forking one process into another, allowing the child to receive a copy of all data. Changes in one process are not reflected in another process.

Another approach is to share data via message passing. A transport object like Queue or Pipe can be created and shared among processes. Processes can send messages to each other via the transport object allowing each to update their internal state in response.

The downside of simulating shared memory with message passing is the added computational cost of using inter-process communication to share data. All data shared between processes using this method must be pickled and unpickled, limiting sharing to those objects that can be pickled, and adding the cost of serialization and deserialization for each item of data shared.

A final approach is to use a manager to host Python objects in one server process. Proxy objects for the hosted objects can then be distributed among multiple processes and used to interact with the single hosted object.

Shared memory via manages is probably the closest that processes come to true shared memory experienced with threads.

Since Python 3.8 a new alternative to shared memory was added to Python in the multiprocessing.shared_memory module.

Module multiprocessing.shared_memory

The multiprocessing.shared_memory module provides shared memory for use with processes.

This module provides a class, SharedMemory, for the allocation and management of shared memory to be accessed by one or more processes on a multicore or symmetric multiprocessor (SMP) machine

— multiprocessing.shared_memory — Shared memory for direct access across processes

The benefit of shared memory is speed and efficiency.

No inter-process communication is required. Instead, processes are able to read and write the same shared memory block directly, although within constraints.

Processes are conventionally limited to only have access to their own process memory space but shared memory permits the sharing of data between processes, avoiding the need to instead send messages between processes containing that data. Sharing data directly via memory can provide significant performance benefits compared to sharing data via disk or socket or other communications requiring the serialization/deserialization and copying of data.

— multiprocessing.shared_memory — Shared memory for direct access across processes

The module provides three classes to support shared memory, they are:

• multiprocessing.shared_memory.SharedMemory
• multiprocessing.shared_memory.ShareableList
• multiprocessing.managers.SharedMemoryManager

The SharedMemory class is the core class for providing shared memory.

It allows a shared memory block to be created, named, and attached to. It provides a “buf” attribute to read and write data in an array-like structure and can be closed and destroyed.

A ShareableList has an internal SharedMemory, although it allows the memory to be used like a fixed-sized Python list, storing any primitive data type.

The SharedMemoryManager is a multiprocessing.Manager that allows multiple SharedMemory and ShareableList to be created via the manager then released as a group when the manager is shut down. The context manager interface of the SharedMemoryManager helps to ensure the memory is always released, regardless of the success or failure of the program.

Now that we know about the support for shared memory, let’s look at the life cycle of a shared memory block.

Shared Memory Life-Cycle

The life-cycle of shared memory has 4 steps, they are:

• 1. Create shared memory.
  • 1a. Attach shared memory.
• 2. Read/Write shared memory.
• 3. Close shared memory.
• 4. Destroy shared memory.

Let’s take a closer look at each step in the life cycle.

Create Shared Memory

Creating shared memory means creating a SharedMemory or ShareableList.

It allocates the memory and makes it available to Python processes.

Attach Shared Memory

Shared memory objects do not need to be shared directly between processes, instead, processes can attach to shared memory.

Attaching to shared memory allows any Python process to gain access to a shared memory block by name.

This is just like creating the block. It results in a SharedMemory or ShareableList object that can then be used as needed.

Read/Write Shared Memory

Once a shared memory object has been created or acquired, it can be used to store data and access data.

Data can be read and written directly, which is very efficient.

Close Shared Memory

Once a process is finished with a shared memory object, it should close it.

This signals to the Python interpreter that it no longer requires access to the resource.

Destroy Shared Memory

When all processes have closed the shared memory, the memory can be released.

This destroys the shared memory and makes it unavailable in the program.

Only one process should release the memory, typically the process that created it.

Now that we know the life cycle of shared memory, let’s look at how to use each class in the multiprocessing.shared_memory module.

How to Use SharedMemory

A SharedMemory object can be created in a process by calling the constructor and specifying a “size” in bytes and the “create” argument to True.

For example:

A shared memory object can be assigned a meaningful name via the “name” attribute to the constructor.

For example:

Another process can access a shared memory via its name. This is called attaching to shared memory.

This can be achieved by specifying the name of the shared memory that has already been created and setting the “create” argument to False (the default).

For example:

Once created data can be stored in the shared memory via the “buf” attribute that acts like an array of bytes.

For example:

Data can be read from the “buf” attribute in the same manner.

For example:

Once a process is finished using the shared memory, it can be closed to signal that access is no longer required.

All processes should close the shared memory once they are finished with it.

Closes access to the shared memory from this instance. In order to ensure proper cleanup of resources, all instances should call close() once the instance is no longer needed. Note that calling close() does not cause the shared memory block itself to be destroyed.

— multiprocessing.shared_memory — Shared memory for direct access across processes

Once all processes are finished with the shared memory, it must be explicitly released.

This can be achieved by calling the unklink() method.

Requests that the underlying shared memory block be destroyed. In order to ensure proper cleanup of resources, unlink() should be called once (and only once) across all processes which have need for the shared memory block.

— multiprocessing.shared_memory — Shared memory for direct access across processes

Ideally, the process that created the shared memory would also release it.

For example:

You can learn more about how to use SharedMemory in the tutorial:

• How to Use SharedMemory in Python

How to Use ShareableList

A ShareableList can be created by calling the class constructor and specifying a sequence.

This initial sequence, like a list or a range, will define both the size of the ShareableList and the initial values.

For example:

This name can be used to attach directly to the shareable list from another process.

This can be achieved by creating a new ShareableList and only specifying the name of the existing ShareableList, perhaps created in another process.

For example:

Unlike a shared ctype multiprocessing.Array, the ShareableList can hold a variety of types.

For example:

Once created, the list can be shared directly with multiple processes.

Values in the list can be accessed and changed like a regular list using the bracket syntax.

For example:

The ShareableList provides methods like count() for getting the number of occurrences of a given value and index() to return the first index of a value.

Once we are finished with the ShareableList, the shared memory block must be explicitly released.

This can be achieved by accessing the “shm” attribute of the ShareableList and calling the close() method followed by the unlink() method.

The close() method signals that no further access is required to the memory. The unlink() method signals that the shared memory block can be destroyed.

For example:

Failing to shutdown the ShareableList in this way will result in a runtime warning.

You can learn more about how to use a ShareableList in the tutorial:

• How to use a ShareableList in Python

How to Use SharedMemoryManager

The SharedMemoryManager class can be used by creating an example, starting it, using it to create SharedMemory and ShareableList objects, then shutting it down in order to release all memory.

Firstly, an instance of the SharedMemoryManager class must be created.

For example:

Next, the manager must be started.

This can be achieved by calling the start() method.

A call to start() on a SharedMemoryManager instance causes a new process to be started. This new process’s sole purpose is to manage the life cycle of all shared memory blocks created through it.

— multiprocessing.shared_memory — Shared memory for direct access across processes

This may create internal threads or processes to manage the memory blocks we will create and use in our program.

For example:

Once started, we can use the memory manager to create blocks of shared memory.

For example:

We cannot specify a name when creating ShareableList and SharedMemory objects via the manager.

This means we cannot use the manager to attach to a memory block.

We must also pass around the objects to child processes directly, such as via arguments. We cannot attach to them via name as we could if we managed the objects manually.

Once we are finished, we can release all shared memory managed by the manager.

This can be achieved by calling the shutdown() method.

To trigger the release of all shared memory blocks managed by that process, call shutdown() on the instance. This triggers a SharedMemory.unlink() call on all of the SharedMemory objects managed by that process and then stops the process itself.

— multiprocessing.shared_memory — Shared memory for direct access across processes

This will both close and unlink all shared memory that we have created with the manager.

We can manage the startup and shutdown process of the manager using the context manager interface.

When using a SharedMemoryManager in a with statement, the shared memory blocks created using that manager are all released when the with statement’s code block finishes execution.

— multiprocessing.shared_memory — Shared memory for direct access across processes

This ensures that the shutdown() method is always called automatically once the block of the context manager is exited.

For example:

You can learn more about how to use the SharedMemoryManager in the tutorial:

• How to Use the SharedMemoryManager in Python

Further Reading

This section provides additional resources that you may find helpful.

Python Multiprocessing Books

• Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Multiprocessing API Cheat Sheet

I would also recommend specific chapters in the books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism
• PEP 371 - Addition of the multiprocessing package

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to use shared memory between processes in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Aaron Huber on Unsplash