Last Updated on November 19, 2022
You can share ctypes among processes using the multiprocessing.Value and multiprocessing.Array classes.
Shared ctypes provide a mechanism to share data safely between processes in a process-safe manner.
In this tutorial you will discover how to share ctypes between processes in Python.
Let’s get started.
Need Data Shared Between Processes
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
In multiprocessing programming, we typically need to share data and program state between processes.
This can be achieved using shared memory via shared ctypes.
How can we use shared ctypes among processes?
Run loops using all CPUs, download your FREE book to learn how.
What Are ctypes
The ctypes module provides tools for working with C data types.
The C programming language and related languages like C++ underlie many modern programming languages and are the languages used to implement most modern operating systems.
As such, the data types used in C are somewhat standardized across many platforms.
The ctypes module allows Python code to read, write, and generally interoperate with data using standard C data types.
These C data types are more specific and more pedantic than Python data types.
They include signed and unsigned integers and longs, single and double float point values, and char values.
For a list of C data types, see:
For details on how to work with C data types directly in Python, see:
What Are Shared ctypes
Python provides the capability to share ctypes between processes on one system.
This is primarily achieved via the following classes:
- multiprocessing.Value: manage a shared value.
- multiprocessing.Array: manage an array of shared values.
The multiprocessing.Value class is used to share a ctype of a given type among multiple processes.
The multiprocessing.Array class is used to share an array of ctypes of a given type among multiple processes.
Both the Value and Array classes are aliases for classes in the multiprocessing.sharedctypes module, specifically:
- multiprocessing.sharedctypes.Value
- multiprocessing.sharedctypes.Array
This module also provides additional classes for managing shared ctypes, such as:
- multiprocessing.sharedctypes.RawValue
- multiprocessing.sharedctypes.RawArray
The multiprocessing.sharedctypes.RawValue and multiprocessing.sharedctypes.RawArray differ from the above non-raw types in that the values are not protected internally from race conditions. Specifically, they do not use a multiprocessing.Lock or multiprocessing.RLock to protect concurrent updates to the shared ctype data from multiple processes. This means they are not process-safe.
And the functions multiprocessing.sharedctypes.copy() and multiprocessing.sharedctypes.synchronized(). The copy() function duplicates a shared ctype whereas the synchronized() creates a mutex lock to protect the data within a shared ctype from race conditions, making it process-safe.
Reviewing the source code, both multiprocessing.sharedctypes.Value and multiprocessing.sharedctypes.Array are in fact functions that create instance of multiprocessing.sharedctypes.RawValue and multiprocessing.sharedctypes.RawArray that are synchronized so that they can be shared among processes.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Why Use Shared ctypes?
Share ctypes provide a simple and easy to use way of sharing data between processes.
For example, a shared ctype value can be defined in a parent process, then shared with multiple child processes. All child processes and the parent process can then safely read and modify the data within the shared value.
This can be useful in a number of use cases, such as:
- A counter shared among multiple processes.
- Returning data from a child process to a parent process.
- Sharing results of computation among processes.
Shared ctypes can only be shared among processes on the one system. For sharing data across processes on multiple systems, a multiprocessing.Manager should be used.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
How to Use Shared ctypes
Python provides ctypes that can be shared between processes via the multiprocessing.Value and multiprocessing.Array classes.
Let’s take a closer look at each in turn.
How to Use Multiprocessing Value
The multiprocessing.Value class will create a shared ctype with a specified data type and initial value.
For example:
|
1 2 3 |
... # create a value value = multiprocessing.Value(...) |
The first argument defines the data type for the value. It may be a string type code or a Python ctype class. The second argument may be an initial value.
For example, we can define a signed integer type with the ‘i’ type code and an initial value of zero as follows:
|
1 2 3 |
... # create a integer value variable = multiprocessing.Value('i', 0) |
We can define the same signed integer shared ctype using the ctypes.c_int class.
For example:
|
1 2 3 |
... # create a integer value variable = multiprocessing.Value(ctypes.c_int, 0) |
Once defined, the value can then be shared and used within multiple processes, such as between a parent and a child process.
Internally, the multiprocessing.Value makes use of a multiprocessing.RLock that ensures that access and modification of the data inside the class is mutually exclusive, e.g. process-safe.
This means that only one process at a time can access or change the data within the multiprocessing.Value object.
You can learn more about the multiprocessing.RLock in the tutorial:
The data within the multiprocessing.Value object can be accessed via the “value” attribute.
For example:
|
1 2 3 |
... # get the data data = variable.value |
The data within the multiprocessing.Value can be changed by the same “value” attribute.
For example:
|
1 2 3 |
... # change the data variable.value = 100 |
The mutex lock within the multiprocessing.Value can be retrieved via the get_lock() method.
|
1 2 3 |
... # get the lock used by the value lock = variable.get_lock() |
This is helpful as we can use the lock within the multiprocessing.Value to protect a critical section that may first read the value and then assign the value, such as incrementing or adding to it.
For example:
|
1 2 3 4 5 |
... # acquire the lock on the variable with variable.get_lock(): # increment the variable variable.value += 1 |
You can learn more about race conditions between processes when using shared ctypes in the tutorial:
Next, let’s take a closer look at the array.
How to Use Multiprocessing Array
The multiprocessing.Array class will create an array of shared ctypes with a specified data type and initial value.
For example:
|
1 2 3 |
... # create an array value = multiprocessing.Array(...) |
This is an array in the C data type sense, in that it has a fixed and pre-allocated size and all items in the array have the same data type.
The first argument to the constructor defines the data type for the value. It may be a string type code or a Python ctype class. The second argument may be an initial value or the size of the array.
For example, we can define a signed integer type with the ‘i’ type code and an initial values of zero using a fixed size tuple follows:
|
1 2 3 |
... # create a integer array array = multiprocessing.Array('i', (1, 2, 3, 4, 5)) |
We can define the same signed integer shared ctype using the ctypes.c_int class.
For example:
|
1 2 3 |
... # create a integer array array = multiprocessing.Array(ctypes.c_int, (1, 2, 3, 4, 5)) |
Once defined, the array can then be shared and used within multiple processes, such as between a parent and a child process.
The array may then be iterated to access the values.
For example:
|
1 2 3 4 |
... # print values in the array for item in array: print(item) |
Values in the array may be changed directly via an integer array index.
For example:
|
1 2 3 |
... # change the first value in the array array[0] = 100 |
Like the multiprocessing.Value, the multiprocessing.Array also provides a mutex lock that can be accessed to protect a critical section that might both read and update data within the array, or update multiple values in the array atomically.
For example
|
1 2 3 4 5 6 7 |
... # acquire the lock on the array with array.get_lock(): # iterate over the array for i in range(len(array)): # change values in the array array[i] = i |
Help With Data Types
Data types must be specified when using a multiprocessing.Value or multiprocessing.Array.
This can be challenging to beginners that are not familiar with c data types, specifically with the string type codes commonly used in C printf() statements.
Internally, when we use a string type code, the multiprocessing.sharedctypes module will convert them to Python types from the ctypes module.
Reviewing the code, we can see a typecode_to_type dictionary that maps type codes to ctype types.
For example:
|
1 2 3 4 5 6 7 8 9 |
typecode_to_type = { 'c': ctypes.c_char, 'u': ctypes.c_wchar, 'b': ctypes.c_byte, 'B': ctypes.c_ubyte, 'h': ctypes.c_short, 'H': ctypes.c_ushort, 'i': ctypes.c_int, 'I': ctypes.c_uint, 'l': ctypes.c_long, 'L': ctypes.c_ulong, 'q': ctypes.c_longlong, 'Q': ctypes.c_ulonglong, 'f': ctypes.c_float, 'd': ctypes.c_double } |
The most common data types we are likely to use are:
- signed integer:
- type code: ‘i’
- ctype: ctypes.c_int
- single floating point:
- type code: ‘f’
- ctype: ctypes.c_float
- char array (string):
- type code: ‘c’
- ctype: ctypes.c_char
We can also see a list of string type codes and their mapping to c types and Python types defined in the array module API documentation:
Finally, we can see a list of ctypes and their mapping to C types and Python types defined in the ctypes module documentation:
Now that we know how to use shared ctypes, let’s look at some worked examples.
Example of a Shared Double Value
We can explore how to share a floating point value between processes.
In this example we will define a multiprocessing.Value as a floating point variable and an initial value. The value will be shared with a child process and modified with a random value. The parent process will then access and report the variable, confirming that the change took effect.
Firstly, we will define a function to run in a child process.
The function will have the name task() and will take the shared ctype as an argument.
|
1 2 3 4 |
... # function to execute in a child process def task(variable): # ... |
The function will generate a random value between 0 and 1 using the random.random() function.
|
1 2 3 |
... # generate a double floating point value data = random() |
The function will then store the generated value in the shared variable so that the parent variable can access it.
|
1 2 3 |
... # store value variable.value = data |
Finally, the function will report the stored value, so we can compare it to the value accessed by the parent process.
|
1 2 3 |
... # report progress print(f'Wrote: {data}', flush=True) |
Tying this together, the complete task() function is listed below.
|
1 2 3 4 5 6 7 8 |
# function to execute in a child process def task(variable): # generate a double floating point value data = random() # store value variable.value = data # report progress print(f'Wrote: {data}', flush=True) |
Next, in the main process, we will define the shared ctype.
We will set the data type as ‘f’ for a single floating point value with an initial value of zero.
|
1 2 3 |
... # create shared variable variable = Value('f', 0.0) |
We can then create a new multiprocessing.Process instance and configure it to execute our task() function and pass the shared ctype as an argument.
|
1 2 3 |
... # create a child process process process = Process(target=task, args=(variable,)) |
If you are new to executing a function in a new process, see the tutorial:
The process can then be started and the main process will block until the child process terminates using the join() function.
|
1 2 3 4 5 |
... # start the process process.start() # wait for the process to finish process.join() |
If you are new to joining a child process, see the tutorial:
Finally, the parent process will access the data set in the shared ctype from the child process.
|
1 2 3 |
... # report the value print(f'Read: {data}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of value shared memory for a single floating point from random import random from multiprocessing import Value from multiprocessing import Process # function to execute in a child process def task(variable): # generate a single floating point value data = random() # store value variable.value = data # report progress print(f'Wrote: {data}', flush=True) # protect the entry point if __name__ == '__main__': # create shared variable variable = Value('f', 0.0) # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() # read the value data = variable.value # report the value print(f'Read: {data}') |
Running the example first creates the shared multiprocessing.Value instance to hold a float.
The child process is configured and started and the main process blocks until the child process terminates.
The child process generates a random value. It then stores the generated value in the shared Value instance for the parent process to access and reports the value that was stored.
The child process terminates and the main process unblocks.
The parent process then reports the number stored in the shared Value instance.
We can see that the generated value in the child process matches the value accessed by the parent process, showing that indeed that the variable is shared and updated among the two processes.
Note, your specific results will differ given the use of random numbers.
|
1 2 |
Wrote: 0.5508677463070446 Read: 0.5508677363395691 |
Example of a Shared Integer Value
We can explore how to share a signed integer value between processes.
In this example we will define a multiprocessing.Value as a signed integer variable and an initial value. The value will be shared with a child process and modified with a random value. The parent process will then access and report the variable, confirming that the change took effect.
Firstly, we will define a function to run in a child process.
The function will take the shared ctype as an argument. It will then generate a new random integer value between 0 and 99 using the random.randint() function. The value is then stored in the shared ctype and reported so that we can compare it to the value accessed by the parent process later.
The complete task() function is listed below.
|
1 2 3 4 5 6 7 8 |
# function to execute in a child process def task(variable): # generate a signed integer value data = randint(0, 100) # store value variable.value = data # report progress print(f'Wrote: {data}', flush=True) |
Next, in the main process we will define a multiprocessing.Value with a signed integer type code of ‘i’ and an initial value of zero.
|
1 2 3 |
... # create shared variable variable = Value('i', 0) |
We can then define a new process instance to execute our task() function and take the shared ctype as an argument. The process is then started and the main process will block until the child process terminates.
|
1 2 3 4 5 6 7 |
... # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() |
Finally, the main process accesses the value of the shared ctype and reports it.
|
1 2 3 4 5 |
... # read the value data = variable.value # report the value print(f'Read: {data}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of value shared memory for an integer from random import randint from multiprocessing import Value from multiprocessing import Process # function to execute in a child process def task(variable): # generate a signed integer value data = randint(0, 100) # store value variable.value = data # report progress print(f'Wrote: {data}', flush=True) # protect the entry point if __name__ == '__main__': # create shared variable variable = Value('i', 0) # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() # read the value data = variable.value # report the value print(f'Read: {data}') |
Running the example first creates the shared multiprocessing.Value instance to hold an integer.
The child process is configured and started and the main process blocks until the child process terminates.
The child process generates a random integer value. It then stores the generated value in the shared Value instance for the parent process to access and reports the value that was stored.
The child process terminates and the main process unblocks.
The parent process then reports the number stored in the shared Value instance.
We can see that the generated value in the child process matches the value accessed by the parent process, showing that indeed that the variable is shared and updated among the two processes.
Note, your specific results will differ given the use of random numbers.
|
1 2 |
Wrote: 43 Read: 43 |
Example of a Shared Array
We can explore how to use a shared ctype array.
In this example, we will define an array to hold five signed integer values and to initialize the array with the values 1 to 5. We will then report the initialized values. A child process will then run and change the values in the array, reporting the changes. Finally the parent process will access the array and confirm the changes to the array.
First, we can define a function to execute in a child process that takes the shared ctype array as an argument.
|
1 2 3 4 |
... # function to execute in a child process def task(variable): # ... |
The function will iterate over the shared ctype array and assign random integers to each array index.
|
1 2 3 4 |
... # store new values in array for i in range(len(variable)): variable[i] = randint(0, 100) |
We can then read the array values into a Python list using a list comprehension and report the list of values so that the parent process can confirm the change later.
|
1 2 3 4 5 |
... # read the items in the array into a list data = [item for item in variable] # report progress print(f'Wrote: {data}', flush=True) |
Tying this together, the complete function is listed below.
|
1 2 3 4 5 6 7 8 9 |
# function to execute in a child process def task(variable): # store new values in array for i in range(len(variable)): variable[i] = randint(0, 100) # read the items in the array into a list data = [item for item in variable] # report progress print(f'Wrote: {data}', flush=True) |
Next, in the main process, we can define a shared ctype array with an signed integer and a tuple defining the initial values.
The tuple of initial values also defines the fixed size of the array.
|
1 2 3 |
... # create shared array variable = Array('i', (1, 2, 3, 4, 5)) |
We can then read the array values into a Python list using a list comprehension and then report the initial array values.
|
1 2 3 4 |
... # read the items in the array into a list data = [item for item in variable] print(f'Initialized: {data}') |
A new child process can be configured to run our task() function and take the shared ctype array as an argument. The child process is then started and the main process blocks until the new process finishes.
|
1 2 3 4 5 6 7 |
... # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() |
Finally, the parent process will access the values of the array and report them.
|
1 2 3 4 5 |
... # read the items in the array into a list data = [item for item in variable] # report the value print(f'Read: {data}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of shared ctypes with an array from random import randint from multiprocessing import Array from multiprocessing import Process # function to execute in a child process def task(variable): # store new values in array for i in range(len(variable)): variable[i] = randint(0, 100) # read the items in the array into a list data = [item for item in variable] # report progress print(f'Wrote: {data}', flush=True) # protect the entry point if __name__ == '__main__': # create shared array variable = Array('i', (1, 2, 3, 4, 5)) # read the items in the array into a list data = [item for item in variable] print(f'Initialized: {data}') # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() # read the items in the array into a list data = [item for item in variable] # report the value print(f'Read: {data}') |
Running the example first creates the shared multiprocessing.Array instance to hold five signed integers.
The child process is configured and started and the main process blocks until the child process terminates.
The child process generates new random integer values for each position in the shared ctype array. It then reports the values that were stored.
The child process terminates and the main process unblocks.
The parent process then reports the numbers stored in the shared ctype array.
We can see the initial values are different from those reported by the child process, showing that we correctly changed the array in the child process. We can then see that the generated values in the child process match the values accessed by the parent process, showing that indeed that the array is shared and updated among the two processes.
Note, your specific results will differ given the use of random numbers.
|
1 2 3 |
Initialized: [1, 2, 3, 4, 5] Wrote: [19, 88, 52, 31, 56] Read: [19, 88, 52, 31, 56] |
Example of a Shared String
We can share a string between processes using a shared ctype array of characters.
In this example we will define a multiprocessing.Array to hold characters with an initial string value. The child process will then change the string value and the parent process will access the shared array and confirm the change was propagated and shared among processes.
Firstly, we can define a function to execute in the child process. The function will take the shared ctype array as an argument, will define a new string value, store it in the array and then report the value that was stored so that we can compare it to the value accessed by the parent process later.
We do not have to assign each array index of the string manually.
Recall that a Python string is a list of characters. We can confirm a Python string to bytes using the “b” prefix to the constant string value.
For example:
|
1 2 3 |
... # prepare string data data = b'Hello!' |
This can then be assigned directly into the array, specifically the first six positions of the array will be assigned the six characters of the byte string.
|
1 2 3 |
... # store value variable.value = data |
Tying this together, the complete task() function is listed below.
|
1 2 3 4 5 6 7 8 |
# function to execute in a child process def task(variable): # prepare string data data = b'Hello!' # store value variable.value = data # report progress print(f'Wrote: {data}', flush=True) |
Next, in the main process we can define a shared ctype array to hold a string.
We will use the ‘c’ type code to represent characters, and define an initial string value of ‘Hello World‘, converted to byte values using the ‘b’ prefix to the string.
|
1 2 3 |
... # create shared variable variable = Array('c', b'Hello World') |
This will define a fixed-length char array with 11 elements.
A new child process can be configured to run our task() function and take the shared ctype array as an argument. The child process is then started and the main process blocks until the new process finishes.
|
1 2 3 4 5 6 7 |
... # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() |
Finally, the parent process will access and report the string value.
|
1 2 3 4 5 |
... # read the value data = variable.value # report the value print(f'Read: {data}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of shared ctypes for a string from random import randint from time import sleep from multiprocessing import Array from multiprocessing import Process # function to execute in a child process def task(variable): # prepare string data data = b'Hello!' # store value variable.value = data # report progress print(f'Wrote: {data}', flush=True) # protect the entry point if __name__ == '__main__': # create shared variable variable = Array('c', b'Hello World') # create a child process process process = Process(target=task, args=(variable,)) # start the process process.start() # wait for the process to finish process.join() # read the value data = variable.value # report the value print(f'Read: {data}') |
Running the example first creates the shared multiprocessing.Array to hold characters with a fixed length and an initial string value.
The child process is configured and started and the main process blocks until the child process terminates.
The child process assigned a new string to the char array. The new string has fewer characters than the initial string. The child process then reports the values that were stored.
The child process terminates and the main process unblocks.
The parent process then reports the string stored in the shared ctype array.
We can then see that the replacement string set by the child process matches the string accessed by the parent process, showing that indeed that the array is shared and updated among the two processes.
|
1 2 |
Wrote: b'Hello!' Read: b'Hello!' |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to share ctypes between processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Sumit Chinchane on Unsplash
Thanks Jason for your blog. It’s very thorough and detailed. I have a question. I tried the technique to share a string but it didn’t seem to work. In your example, you used b’Hello World!’ (a b character and a string between quote). I have a variable x which has a string value. How to tell Python that this is a list of characters. I dont have a way to precede it with a b character
Thanks
I just found a workaroundx_a = bytes(x, ‘utf-8’)to switch the string to byteThanks for sharing
Happy to hear that you found a workaround!
These blog posts are awesome!
Thank you! I’m happy they’re helpful.