Last Updated on September 29, 2023

You can share numpy arrays with child processes by inheriting global variables.

In this tutorial, you will discover how to share a numpy array with child processes by variable inheritance.

Let’s get started.

Need to Share Numpy Array Between Processes

Python offers process-based concurrency via the multiprocessing module.

Process-based concurrency is appropriate for those tasks that are CPU-bound, as opposed to thread-based concurrency in Python which is generally suited to IO-bound tasks given the presence of the Global Interpreter Lock (GIL).

You can learn more about process-based concurrency and the multiprocessing module in the tutorial:

• Multiprocessing: The Complete Guide

Consider the situation where we need to share numpy arrays between processes.

This may be for many reasons, such as:

• Data is loaded as an array in one process and analyzed differently in different subprocesses.
• Many child processes load small data as arrays that are sent to a parent process for handling.
• Data arrays are loaded in the parent process and processed in a suite of child processes.

Sharing Python objects and data between processes is slow.

This is because any data, like numpy arrays, shared between processes must be transmitted using inter-process communication (ICP) requiring the data first be pickled by the sender and then unpickled by the receiver.

You can learn more about this in the tutorial:

• Threads are 4x Faster at Sharing Data Than Processes in Python

This means that if we share numpy arrays between processes, it assumes that we receive some benefit, such as a speedup, that overcomes the slow speed of data transmission.

For example, it may be the case that the arrays are relatively small and fast to transmit, whereas the computation performed on each array is slow and can benefit from being performed in separate processes.

Alternatively, preparing the array may be computationally expensive and benefit from being performed in a separate process, and once prepared, the arrays are small and fast to transmit to another process that requires them.

Given these situations, how can we share data between Processes in Python?

How to Share a Numpy Array Between Processes via a Global Variable

One way to share a numpy array with other processes is via a global variable.

That is, a process can load or create a numpy array, store it in a global variable, and other processes can access it directly.

The benefit of this approach is that it is really fast.

It can be up to 34x faster than sending the array between processes using other methods, such as a function argument.

You can learn more about this approach to sharing data between processes in the tutorial:

• Inheriting is 34x Faster Than Sending Data Between Processes in Python

There are some limitations to this approach though, they are:

1. The array must be stored in a global variable by a parent process.
2. The array can only be accessed by child processes.
3. Child processes must be created using the ‘fork’ start method.
4. Changes to the array made by child processes will not be reflected in each other or in the parent process.

Let’s consider these concerns in detail.

A python program can declare an explicit global variable using the ‘global‘ keyword.

For example:

This is not needed in the parent process when preparing the numpy array, but may be needed in the function executed in the child process to ensure that the program is referred to the inherited global variable and not a newly created local variable.

This is called inheriting a global variable.

You can learn more about inheriting global variables in child processes in the tutorial:

• Multiprocessing Inherit Global Variables in Python

Child processes can be created either by spawning a new instance of the Python interpreter or by forking an existing instance of the Python interpreter. Spawning is the current default on Windows and macOS, whereas forking is the default on Linux.

We can explicitly set the method used to start child processes via the multiprocessing.set_start_method() function and specify the method as a string, such as ‘fork’.

For example:

The fork start method is not supported on Windows at the time of writing.

You can learn more about setting the start method used for creating child processes in the tutorial:

• Multiprocessing Start Methods

The child process will inherit a copy of all global variables from the parent process.

Because they are a copy, any changes made to the global variable in the child process will only be available to the child process. Similarly, any changes to the global variable made in the parent process after the child processes are created will not be reflected in the child processes.

As such, this approach to sharing an array between processes is appropriate in certain situations, such as:

1. The array is large, but the machine can afford to store multiple copies in memory at the same time.
2. Child processes compute something using the array, but changes to the array itself are not required in the parent process or other processes.

A common example that may be appropriate for this approach would be where one or more arrays are prepared or loaded and a suite of computationally expensive statistics needs to be calculated on each. Each statistic or set of statistics can be calculated using a copy of the array in a separate process.

Now that we know how to share a numpy array between processes by inheriting it, let’s look at a worked example.

Example of Sharing a Numpy Array as a Global Variable

We can explore how to share a numpy array as an inherited global variable with child processes.

In this example, we will create a modestly sized numpy array in a parent process and store it as a global variable. We will then start a child process using the fork start method and have it access the inherited numpy array.

Firstly, we can define a task to be executed in a child process.

In this case, the task does not take any arguments. Instead, it declares an inherited global variable, then accesses a portion of it directly.

It then changes the contents of the array, by assigning all values to zero and confirming that the change took effect.

The task() function below implements this.

Next, we can set the start method to fork.

We can then create a modestly sized numpy array, stored in a variable.

We can then create a child process, configured to execute our task() function, and then start the process.

This will make a copy of the parent process, including the numpy array stored in the ‘data‘ variable.

You can learn more about running a function in a child process in the tutorial:

• Run a Function in a Child Process

Finally, we can wait for the child process to terminate and then report the contents of the array.

Tying this together, the complete example is listed below.

Note, this example may not run on Windows as it requires support for the ‘fork’ start method.

Running the example first sets the ‘fork’ start method for creating child processes.

Next, the numpy array is created and initialized to the value.

The child process is then created and started, and the main process blocks until it is terminated.

A forked copy of the parent process is created and runs the task() function. The inherited global variable is declared and then accessed, reporting a small subset of the data.

This confirms that the array was inherited correctly and contains the data initialized in the parent process.

The child process then fills the array with zero values and accesses them to confirm the data was changed.

The child process terminates and the main process then resumes. It reports the contents of a small portion of the array, confirming that the data change made in the child process was not propagated to the parent process.

This highlights how to share a numpy array with child processes by inheritance and that changes made in the child process to the inherited array do not propagate.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Concurrent NumPy in Python, Jason Brownlee (my book!)

Guides

• Concurrent NumPy 7-Day Course
• Which NumPy Functions Are Multithreaded
• Numpy Multithreaded Matrix Multiplication (up to 5x faster)
• NumPy vs the Global Interpreter Lock (GIL)
• ThreadPoolExecutor Fill NumPy Array (3x faster)
• Fastest Way To Share NumPy Array Between Processes

Documentation

• Parallel Programming with numpy and scipy, SciPi Cookbook, 2015
• Parallel Programming with numpy and scipy (older archived version)
• Parallel Random Number Generation, NumPy API 

NumPy APIs

• NumPy documentation
• NumPy API Reference
• Linear algebra (numpy.linalg) API

Concurrency APIs

• threading — Thread-based parallelism
• multiprocessing — Process-based parallelism
• concurrent.futures — Launching parallel tasks

Takeaways

You now know how to share a numpy array with child processes by variable inheritance.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by geng zhang on Unsplash