Last Updated on September 29, 2023
You can share a numpy array between processes by first creating a shared ctype RawArray and then using the RawArray as a buffer for a new numpy array.
In this tutorial, you will discover how to share a numpy array between processes that is backed by a shared ctype RawArray.
Let’s get started.
Need to Share Numpy Array Between Processes
Python offers process-based concurrency via the multiprocessing module.
Process-based concurrency is appropriate for those tasks that are CPU-bound, as opposed to thread-based concurrency in Python which is generally suited to IO-bound tasks given the presence of the Global Interpreter Lock (GIL).
You can learn more about process-based concurrency and the multiprocessing module in the tutorial:
Consider the situation where we need to share numpy arrays between processes.
This may be for many reasons, such as:
- Data is loaded as an array in one process and analyzed differently in different subprocesses.
- Many child processes load small data as arrays that are sent to a parent process for handling.
- Data arrays are loaded in the parent process and processed in a suite of child processes.
Sharing Python objects and data between processes is slow.
This is because any data, like numpy arrays, shared between processes must be transmitted using inter-process communication (ICP) requiring the data first be pickled by the sender and then unpickled by the receiver.
You can learn more about this in the tutorial:
This means that if we share numpy arrays between processes, it assumes that we receive some benefit, such as a speedup, that overcomes the slow speed of data transmission.
For example, it may be the case that the arrays are relatively small and fast to transmit, whereas the computation performed on each array is slow and can benefit from being performed in separate processes.
Alternatively, preparing the array may be computationally expensive and benefit from being performed in a separate process, and once prepared, the arrays are small and fast to transmit to another process that requires them.
Given these situations, how can we share data between Processes in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Share a Numpy Array Backed By RawArray
One way to share a numpy array between processes is via shared ctypes.
The ctypes module provides tools for working with C data types.
Python provides the capability to share ctypes between processes on one system.
You can learn more about sharing ctypes between processes in the tutorial:
We can use a shared ctype array as a buffer that backs a numpy array.
Numpy arrays can be created in each Python process backed by the same shared ctype array and share data directly.
This can be achieved by first creating a multiprocessing.sharedctypes.RawArray with the required type and large enough to hold the data required by the numpy array.
For example:
|
1 2 3 |
... # create the shared array array = RawArray('d', 10000) |
A new numpy array can then be created with the given type specifying the RawArray as the buffer, e.g. the pre-allocated memory for the array.
This can be achieved by creating a new numpy.ndarray directly and specifying the “buff” as the array buffer.
For example:
|
1 2 3 |
... # create a new numpy array that uses the raw array data = ndarray((len(array),), dtype=numpy.double, buffer=array) |
It is generally not recommended to create ndarrays directly.
Instead, we can use the numpy.frombuffer() method to create the array for us with the given size, type and backed by the RawArray.
For example:
|
1 2 3 |
... # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) |
The RawArray instance can then be made available to other Python processes and used to create numpy arrays backed by the same memory.
This allows each process to read and write the same array data directly without having to copy it between processes.
Now that we know how to share a numpy array between processes, let’s look at some worked examples.
Example of Sharing Numpy Array Backed By RawArray Between Processes
We can explore the case of creating a numpy array backed by a RawArray and sharing it among Python processes.
In this example, we will create a RawArray and then create a numpy array from the RawArray. We will then populate it with data in the main process. The RawArray will then be shared with a child process which will use it to create another numpy array, backed by the same data. It will then change the data and terminate. Finally, the main process will confirm that the data in the array was changed by the child process.
Firstly, we can define a function to execute in the child process.
The function will take a RawArray as an argument and use the RawArray to create a new numpy array. It then confirms the content of the array, increments all values in the array, and confirms that the data in the array was modified.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 9 10 |
# task executed in a child process def task(array): # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # check the contents print(f'Child {data[:10]}') # increment the data data[:] += 1 # confirm change print(f'Child {data[:10]}') |
Next, in the main process will create a new RawArray of double values that has 10,000,000 elements.
|
1 2 3 4 5 |
... # define the size of the numpy array n = 10000000 # create the shared array array = RawArray('d', n) |
Next, the main process will create a new numpy array from the RawArray and populate it with one values and confirm that the data in the array was changed.
|
1 2 3 4 5 6 7 |
... # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # populate the array data.fill(1.0) # confirm contents of the new array print(data[:10], len(data)) |
A child process is then configured to execute our task() function and pass it the RawArray. The process is started and the main process blocks until the child process terminates.
|
1 2 3 4 5 6 7 |
... # create a child process child = Process(target=task, args=(array,)) # start the child process child.start() # wait for the child process to complete child.join() |
Finally, the main process confirms that the child process changed the content of the shared array.
|
1 2 3 |
... # check some data in the shared array print(data[:10]) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# share numpy array between processes via a shared raw array from multiprocessing import Process from multiprocessing.sharedctypes import RawArray from numpy import frombuffer from numpy import double # task executed in a child process def task(array): # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # check the contents print(f'Child {data[:10]}') # increment the data data[:] += 1 # confirm change print(f'Child {data[:10]}') # protect the entry point if __name__ == '__main__': # define the size of the numpy array n = 10000000 # create the shared array array = RawArray('d', n) # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # populate the array data.fill(1.0) # confirm contents of the new array print(data[:10], len(data)) # create a child process child = Process(target=task, args=(array,)) # start the child process child.start() # wait for the child process to complete child.join() # check some data in the shared array print(data[:10]) |
Running the example first creates the RawArray with 10 million double elements.
Next, a numpy array is created that holds double values and is backed by the RawArray. That is, it does not allocate new memory for the array and instead reuses the memory already allocated by the RawArray.
Next, the main process fills the numpy array with one values and confirms the content of the array changed and that the shape of the array is as we expect.
Next, a child process is started to execute our task() function, and the main process blocks until the process terminates.
The child process runs. It first creates a new numpy array using the RawArray passed in as an argument.
Passing the RawArray to the child process does not make a copy of the array. Instead, it passes a reference to the RawArray to the process.
The child process confirms the array contains one values, as set by the parent process. Next, it increments all values in the array and confirms that the values were changed.
This highlights both that the child array can directly access the same array as the parent process and that it is able to read and write to this data.
The child process terminates and the main process resumes. It confirms that the child process changed the content of the array.
This highlights that changes made to the array in the child process are reflected in other processes, such as the parent process.
|
1 2 3 4 |
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] 10000000 Child [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] Child [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.] [2. 2. 2. 2. 2. 2. 2. 2. 2. 2.] |
Free Concurrent NumPy Course
Get FREE access to my 7-day email course on concurrent NumPy.
Discover how to configure the number of BLAS threads, how to execute NumPy tasks faster with thread pools, and how to share arrays super fast.
Example of Sharing a 2D Numpy Array Backed By RawArray
A limitation with the numpy.frombuffer() function for creating a numpy array backed by a RawArray is that it will only create a one-dimensional array.
Often, we require a two-dimensional array for images and dataset, or more dimensions in other applications.
We can store a multi-dimensional array in RawArray by calling the flatten() method on an array to convert a multiple dimensional array to a one-dimensional array, and reshape() to convert a one-dimensional array to a multidimensional array.
In this example, we can update the previous example to work with a two-dimensional numpy array backed by a one-dimensional RawArray, shared between processes.
Firstly, we can update the function executed in the child process.
After creating the array using the numpy.frombuffer() function, we will call the reshape() method and specify a new shape of the array, in this case, a 3×3 matrix.
The contents of the array is reported, incremented, and reported again to confirm that it was changed by the child process.
The task() function below implements these changes.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# task executed in a child process def task(array): # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # reshape array into preferred shape data = data.reshape((3,3)) # check the contents print(f'Child\n{data}') # increment the data data[:] += 1 # confirm change print(f'Child\n{data}') |
The main process will then create a RawArray with 3×3 (9) elements, large enough to hold our two-dimensional numpy array.
|
1 2 3 4 5 |
... # define the size of the numpy array n = 3*3 # create the shared array array = RawArray('d', n) |
A numpy array is then created from this RawArray and reshaped into a two-dimensional shape for use in the application.
|
1 2 3 4 5 |
... # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # reshape array into preferred shape data = data.reshape((3,3)) |
The array is filled, the child process runs and is awaited, and the content of the array is reported, all as before.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
... # populate the array data.fill(1.0) # confirm contents of the new array print(f'Parent\n{data}') # create a child process child = Process(target=task, args=(array,)) # start the child process child.start() # wait for the child process to complete child.join() # check some data in the shared array print(f'Parent\n{data}') |
The complete example with these changes is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
# share 2d numpy array via a shared array from multiprocessing import Process from multiprocessing.sharedctypes import RawArray from numpy import frombuffer from numpy import double # task executed in a child process def task(array): # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # reshape array into preferred shape data = data.reshape((3,3)) # check the contents print(f'Child\n{data}') # increment the data data[:] += 1 # confirm change print(f'Child\n{data}') # protect the entry point if __name__ == '__main__': # define the size of the numpy array n = 3*3 # create the shared array array = RawArray('d', n) # create a new numpy array backed by the raw array data = frombuffer(array, dtype=double, count=len(array)) # reshape array into preferred shape data = data.reshape((3,3)) # populate the array data.fill(1.0) # confirm contents of the new array print(f'Parent\n{data}') # create a child process child = Process(target=task, args=(array,)) # start the child process child.start() # wait for the child process to complete child.join() # check some data in the shared array print(f'Parent\n{data}') |
Running the example first creates the RawArray with 9 elements, large enough to hold the two-dimensional numpy array.
A numpy array is then created from the RawArray and reshaped into a 3×3 matrix. The numpy array is filled with one values and reported to confirm that the change was made.
The child process is then started and the main process blocks and waits for it to terminate.
The child process runs, first creating a numpy array from the passed in RawArray. The array is then reshaped into a 3×3 matrix.
The content of the array is reported, confirming that the changes made in the parent process are reflected. The content of the array is incremented and reported, confirming the change was made.
The child process terminates and the main process resumes, reporting the contents of the array.
This highlights that we can easily share a two-dimension numpy array between processes that is backed by a one-dimensional RawArray.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Parent [[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] Child [[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] Child [[2. 2. 2.] [2. 2. 2.] [2. 2. 2.]] Parent [[2. 2. 2.] [2. 2. 2.] [2. 2. 2.]] |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- Concurrent NumPy in Python, Jason Brownlee (my book!)
Guides
- Concurrent NumPy 7-Day Course
- Which NumPy Functions Are Multithreaded
- Numpy Multithreaded Matrix Multiplication (up to 5x faster)
- NumPy vs the Global Interpreter Lock (GIL)
- ThreadPoolExecutor Fill NumPy Array (3x faster)
- Fastest Way To Share NumPy Array Between Processes
Documentation
- Parallel Programming with numpy and scipy, SciPi Cookbook, 2015
- Parallel Programming with numpy and scipy (older archived version)
- Parallel Random Number Generation, NumPy API
NumPy APIs
Concurrency APIs
- threading — Thread-based parallelism
- multiprocessing — Process-based parallelism
- concurrent.futures — Launching parallel tasks
Takeaways
You now know how to share a numpy array between processes that is backed by a shared ctype RawArray.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Bernd 📷 Dittrich on Unsplash
Do you have any questions?