You can efficiently share the same list among multiple processes via the ShareableList class.
In this tutorial, you will discover how to use a ShareableList with processes in Python.
Let’s get started.
What is a ShareableList
A multiprocessing.shared_memory.ShareableList is a list that can be shared efficiently between multiple processes.
Shared between processes means that changes made to the list in one process will be visible and accessible in another process.
It is backed by a shared memory block and can be used to store up to 10 megabytes of data as any of Python’s primitive types, e.g. integers, floats, and strings.
Provides a mutable list-like object where all values stored within are stored in a shared memory block. This constrains storable values to only the int, float, bool, str (less than 10M bytes each), bytes (less than 10M bytes each), and None built-in data types.
— multiprocessing.shared_memory — Shared memory for direct access across processes
Unlike the built-in list type, the size of the ShareableList cannot be changed after it is created. Additionally, many of the operations that a regular list support are not supported on the ShareableList.
It also notably differs from the built-in list type in that these lists can not change their overall length (i.e. no append, insert, etc.) and do not support the dynamic creation of new ShareableList instances via slicing.
— multiprocessing.shared_memory — Shared memory for direct access across processes
The ShareableList can be used as an alternative of a list hosted server process via a multiprocessing.Manager and as an alternative to a shared ctype Array.
Now that we know what a ShareableList is, let’s look at how to use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use a ShareableList
A ShareableList can be created by calling the class constructor and specifying a sequence.
This initial sequence, like a list or a range, will define both the size of the ShareableList and the initial values.
For example:
|
1 2 3 |
... # create a shareable list sl = ShareableList([0, 0, 0, 0, 0]) |
We can assign a ShareableList a unique and meaningful name via the name argument to the constructor.
For example:
|
1 2 3 |
... # create a shareable list with a name sl = ShareableList([0, 0, 0, 0, 0], name=’MyList’) |
This meaningful name can be used to attach directly to the shareable list from another process.
This can be achieved by creating a new ShareableList and only specifying the name of the existing ShareableList, perhaps created in another process.
For example:
|
1 2 3 |
… # attach to a shareable list by name sl = ShareableList(name=’MyList)) |
Unlike a shared ctype multiprocessing.Array, the ShareableList can hold a variety of types.
For example:
|
1 2 3 |
... # create a shareable list with different types sl = ShareableList([0, 1.0, 'test']) |
Once created, the list can be shared directly with multiple processes.
Values in the list can be accessed and changed like a regular list using the bracket syntax.
For example:
|
1 2 3 4 5 |
... # report a value print(sl[0]) # change a value sl[1] = 1000 |
The ShareableList provides methods like count() for getting the number of occurrences of a given value and index() to return the first index of a value.
Once we are finished with the ShareableList, the shared memory block must be explicitly released.
This can be achieved by accessing the “shm” attribute of the ShareableList and calling the close() method followed by the unlink() method.
The close() method signals that no further access is required to the memory. The unlink() method signals that the shared memory block can be destroyed.
For example:
|
1 2 3 4 5 |
... # close the shared memory sl.shm.close() # destroy the shared memory sl.shm.unlink() |
Failing to shutdown the ShareableList in this way will result in a runtime warning.
Now that we know how to use the ShareableList, let’s look at some worked examples.
Example of Using ShareableList From Multiple Processes
We can explore how to create a ShareableList and share it between processes.
In this example, we will define a task that takes a ShareableList object as an argument, reports its values, then changes the values in the list. The main process will create the shared list, share it with the task executed in a child process, then report the status of the list to confirm that the changes made in the child process are available to the main process.
Firstly, we can define a task that takes a ShareableList object and modifies it.
In this case, we will expect the ShareableList contains numbers and the task will multiply each value by ten.
The task() function below implements this.
|
1 2 3 4 5 6 7 |
# task executed in a child process def task(sl): # report the shared list print(sl) # change the list for i in range(len(sl)): sl[i] = sl[i] * 10 |
Next, in the main process, we can create an instance of a ShareableList with the values 1 to 5 and report the list values.
|
1 2 3 4 5 |
... # create a shared list sl = ShareableList([1, 2, 3, 4, 5]) # report the shared list print(sl) |
Next, we can configure and start a child process to execute our task() function and pass it the ShareableList instance. The main process then waits for the task to complete.
|
1 2 3 4 5 6 7 |
... # create a child process process = Process(target=task, args=(sl,)) # start the child process process.start() # wait for the child process to finish process.join() |
The main process then reports any changes to the list, then releases the shared memory block.
|
1 2 3 4 5 6 7 |
... # report the shared list print(sl) # close the shared memory sl.shm.close() # release the shared memory sl.shm.unlink() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of sharing a sharablelist between processes from multiprocessing import Process from multiprocessing.shared_memory import ShareableList # task executed in a child process def task(sl): # report the shared list print(sl) # change the list for i in range(len(sl)): sl[i] = sl[i] * 10 # protect the entry point if __name__ == '__main__': # create a shared list sl = ShareableList([1, 2, 3, 4, 5]) # report the shared list print(sl) # create a child process process = Process(target=task, args=(sl,)) # start the child process process.start() # wait for the child process to finish process.join() # report the shared list print(sl) # close the shared memory sl.shm.close() # release the shared memory sl.shm.unlink() |
Running the example first creates the ShareableList and reports its values.
Next, the child process is created to execute our custom task, then started and awaited.
The task runs in the child process. It first reports the ShareableList confirming that it contains the values that we initialized it with. The values in the list are then changed and multiplied by ten.
The child process completes and the main process resumes, reporting the values of the list.
In this case, we can see that the changes made by the child to the ShareableList are reflected in the main process. Each value in the list has been multiplied by ten.
This highlights how we can create and share a ShareableList among multiple processes that are able to access and modify the same list data.
|
1 2 3 |
ShareableList([1, 2, 3, 4, 5], name='psm_033fda8d') ShareableList([1, 2, 3, 4, 5], name='psm_033fda8d') ShareableList([10, 20, 30, 40, 50], name='psm_033fda8d') |
Next, let’s look at an example of not releasing the shared memory block.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Not Closing and Unlinking a ShareableList
We can explore a case of failing to release the shared memory block.
In this example, we will update the above example of not close and unlike the ShareableList object.
We expect that the program will generate a warning that shared memory has not been released.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of not closing and unlinking a sharablelist from multiprocessing import Process from multiprocessing.shared_memory import ShareableList # task executed in a child process def task(sl): # report the shared list print(sl) # change the list for i in range(len(sl)): sl[i] = sl[i] * 10 # protect the entry point if __name__ == '__main__': # create a shared list sl = ShareableList([1, 2, 3, 4, 5]) # report the shared list print(sl) # create a child process process = Process(target=task, args=(sl,)) # start the child process process.start() # wait for the child process to finish process.join() # report the shared list print(sl) |
Running the example first creates the ShareableList and reports its values.
Next, the child process is created to execute our custom task, then started and awaited.
The task runs in the child process. It first reports the ShareableList confirming that it contains the values that we initialized it with. The values in the list are then changed and multiplied by ten.
The child process completes and the main process resumes, reporting the values of the list.
The changes to the shared list are reflected as we expect.
The program ends without closing and releasing the shared memory.
A UserWarning is reported highlighting the memory leak due to the shared memory block not being correctly shut down.
This highlights the need to correctly release the ShareableList once it is no longer required by the program.
|
1 2 3 4 5 6 |
ShareableList([1, 2, 3, 4, 5], name='psm_47882f94') ShareableList([1, 2, 3, 4, 5], name='psm_47882f94') ShareableList([10, 20, 30, 40, 50], name='psm_47882f94') ... UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' |
Next, let’s explore the process safety of the ShareableList data.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of ShareableList Not Process-Safe
We would expect race conditions, such as data loss and corruption if multiple processes modify the same ShareableList object at the same time. The ShareableList does guarantee process safety.
You can learn more about process safety in the tutorial:
We can explore what happens if multiple processes modify the contents of a ShareableList in parallel.
In this example, we will define a task function that loops 100 times. In each loop the task will iterate through each position in the ShareableList and increment the values.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 |
# task executed in a child process def task(sl): # increment values in the shareable list for i in range(100): for j in range(len(sl)): # get the current value val = sl[j] # update the value val = val + 1 # store the new value sl[j] = val |
The shared list will be initialized with all zero values.
|
1 2 3 |
... # create a shared list sl = ShareableList([0, 0, 0, 0, 0]) |
The main process will then start 10 processes to perform this task on the same ShareableList object in parallel, then report the values in the list.
|
1 2 3 4 5 6 7 8 9 10 11 |
... # create processes processes = [Process(target=task, args=(sl,)) for i in range(10)] # start processes for process in processes: process.start() # wait for processes to finish for process in processes: process.join() # report the shared list print(sl) |
Without race conditions, we would expect the final values in the ShareableList object to be 1000 because each value is incremented 100 times by 10 processes (100 * 10).
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# SuperFastPython.com # example showing that the sharablelist is not process safe from multiprocessing import Process from multiprocessing.shared_memory import ShareableList # task executed in a child process def task(sl): # increment values in the shareable list for i in range(100): for j in range(len(sl)): # get the current value val = sl[j] # update the value val = val + 1 # store the new value sl[j] = val # protect the entry point if __name__ == '__main__': # create a shared list sl = ShareableList([0, 0, 0, 0, 0]) # create processes processes = [Process(target=task, args=(sl,)) for i in range(10)] # start processes for process in processes: process.start() # wait for processes to finish for process in processes: process.join() # report the shared list print(sl) # close the shared memory sl.shm.close() # release the shared memory sl.shm.unlink() |
Running the example first crests the ShareableList object, with values initialized to zero.
The 10 child processes are configured to run our custom task function, started then awaited.
The child process iterates as fast as possible in parallel, incrementing the values of the ShareableList.
The child processes finish and the values in the ShareableList object are reported before the memory block is released.
We can see that race conditions occurred. The values in the list are all different and none are the expected value of 1,000.
This highlights that process safety must be considered when using and modifying the ShareableList from multiple processes in parallel.
|
1 |
ShareableList([907, 903, 910, 778, 884], name='psm_46626831') |
The program does not run cleanly every time it executes.
Occasionally, it may report a ValueError.
For example:
|
1 2 3 4 5 6 7 8 9 |
Process Process-1: Traceback (most recent call last): ... ValueError: not enough values to unpack (expected 1, got 0) Process Process-8: Traceback (most recent call last): ... ValueError: not enough values to unpack (expected 1, got 0) ShareableList([807, 801, 810, 661, 810], name='psm_d8c801a9') |
This is not suprising given the abuse we are performing to the ShareableList object.
We can update the example to be process safe when modifying the ShareableList object in parallel.
This can be achieved by using a shared lock and treating changes to the ShareableList object as a critical section.
The updated task() function with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# task executed in a child process def task(sl, lock): # increment values in the shareable list for i in range(100): for j in range(len(sl)): # acquire the lock before the change with lock: # get the current value val = sl[j] # update the value val = val + 1 # store the new value sl[j] = val |
If you are new to using mutex locks to protect data from race conditions in process-based concurrency, see the tutorial:
The main process can create the shared mutex lock and share it with each child process.
|
1 2 3 4 5 6 7 |
... # create a shared lock lock = Lock() # create a shared list sl = ShareableList([0, 0, 0, 0, 0]) # create processes processes = [Process(target=task, args=(sl,lock)) for i in range(10)] |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
# SuperFastPython.com # example of making sharablelist process safe from multiprocessing.shared_memory import ShareableList from multiprocessing import Process from multiprocessing import Lock # task executed in a child process def task(sl, lock): # increment values in the shareable list for i in range(100): for j in range(len(sl)): # acquire the lock before the change with lock: # get the current value val = sl[j] # update the value val = val + 1 # store the new value sl[j] = val # protect the entry point if __name__ == '__main__': # create a shared lock lock = Lock() # create a shared list sl = ShareableList([0, 0, 0, 0, 0]) # create processes processes = [Process(target=task, args=(sl,lock)) for i in range(10)] # start processes for process in processes: process.start() # wait for processes to finish for process in processes: process.join() # report the shared list print(sl) # close the shared memory sl.shm.close() # release the shared memory sl.shm.unlink() |
Running the example first creates the shared Lock and the ShareableList object.
The 10 child processes are configured to run our custom task function, started then awaited.
The child process iterates as fast as possible in parallel, incrementing the values of the ShareableList. In this case, changes to the ShareableList are made sequentially via the use of the mutex lock. Only one process can acquire the lock and make changes at a time and all other processes must wait.
The child processes finish and the values in the ShareableList object are reported before the memory block is released.
We can see that the use of the mutex lock removed all race conditions with the ShareableList in the child processes and all values in the list have the expected value of 1,000.
This highlights how a ShareableList object can be protected from race conditions using a mutex lock.
|
1 |
ShareableList([1000, 1000, 1000, 1000, 1000], name='psm_fdb8e0b8') |
Takeaways
You now know how to use a ShareableList with processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by serjan midili on Unsplash
Do you have any questions?