Last Updated on March 17, 2023
You can use reentrant locks for processes via the multiprocessing.RLock class.
In this tutorial you will discover how to use reentrant mutex locks for processes in Python.
Let’s get started.
Need for a Reentrant Lock
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
When writing concurrent programs we may need to share data or resources between processes, which typically must be protected with a lock.
A standard mutex lock does not allow a process to acquire the lock more than once. This means that code in one critical section cannot call or execute code with another critical section protected by the same lock. Instead, a different type of mutex lock is required called a reentrant lock.
What is a reentrant lock and how can we use it with processes in Python?
First, let’s take a quick look at a standard mutex lock.
Run loops using all CPUs, download your FREE book to learn how.
What is a Mutex Lock
A mutual exclusion lock or mutex lock is a synchronization primitive intended to prevent a race condition.
A race condition is a concurrency failure case when two processes (or threads) run the same code and access or update the same resource (e.g. data variables, stream, etc.) leaving the resource in an unknown and inconsistent state.
Race conditions often result in unexpected behavior of a program and/or corrupt data.
These sensitive parts of code that can be executed by multiple processes concurrently and may result in race conditions are called critical sections. A critical section may refer to a single block of code, but it also refers to multiple accesses to the same data variable or resource from multiple functions.
The most commonly used mechanism for ensuring mutual exclusion is a mutual exclusion lock or mutex, or simply lock. A mutex is a special type of object that has support in the underlying hardware. The basic idea is that each critical section is protected by a lock.
— Page 53, An Introduction to Parallel Programming, 2020.
A mutex lock can be used to ensure that only one process at a time executes a critical section of code at a time, while all other processes trying to execute the same code must wait until the currently executing process is finished with the critical section.
While one thread “owns” the lock—that is, has returned from a call to the lock function but hasn’t yet called the unlock function—any other thread attempting to execute the code in the critical section will wait in its call to the lock function.
— Page 53, An Introduction to Parallel Programming, 2020.
Each process must attempt to acquire the lock at the beginning of the critical section. If the lock has not been obtained, then a process will acquire it and other processes must wait until the process that acquired the lock releases it.
Before a thread can execute the code in the critical section, it must “obtain” the mutex by calling a mutex function, and when it’s done executing the code in the critical section, it should “relinquish” the mutex by calling an unlock function.
— Page 53, An Introduction to Parallel Programming, 2020.
You can learn more about mutex locks for Python processes in the tutorial:
Now that we know what a mutex lock is, let’s take a closer look at a reentrant mutex lock.
What is a Reentrant Lock
A reentrant mutual exclusion lock, “reentrant mutex” or “reentrant lock” for short, is like a mutex lock except it allows a process (or thread) to acquire the lock more than once.
A process may need to acquire the same lock more than once for many reasons.
We can imagine critical sections spread across a number of functions, each protected by the same lock. A process may call across these functions in the course of normal execution and may call into one critical section from another critical section.
A limitation of a (non-reentrant) mutex lock is that if a process has acquired the lock that it cannot acquire it again. In fact, this situation will result in a deadlock as it will wait forever for the lock to be released so that it can be acquired, but it holds the lock and will not release it.
A reentrant lock will allow a process to acquire the same lock again if it has already acquired it. This allows the process to execute critical sections from within critical sections, as long as they are protected by the same reentrant lock.
Each time a process acquires the lock it must also release it, meaning that there are recursive levels of acquire and release for the owning process. As such, this type of lock is sometimes called a “recursive mutex lock”.
Now that we know about the reentrant lock, let’s take a look at how we can use it in Python.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
How to Use the RLock
Python provides a reentrant lock for processes via the multiprocessing.RLock class.
An instance of the multiprocessing.RLock can be created and then acquired by processes before accessing a critical section, and released after the critical section.
For example:
|
1 2 3 4 5 6 7 8 |
... # create a reentrant lock lock = multiprocessing.RLock() # acquire the lock lock.acquire() # ... # release the lock lock.release() |
The process attempting to acquire the lock will block until the lock is acquired, such as if another process currently holds the lock (once or more than once) then releases it.
We can attempt to acquire the lock without blocking by setting the “block” argument to False. If the lock cannot be acquired, a value of False is returned.
|
1 2 3 |
... # acquire the lock without blocking lock.acquire(block=false) |
We can also attempt to acquire the lock with a timeout, that will wait the set number of seconds to acquire the lock before giving up. If the lock cannot be acquired, a value of False is returned.
|
1 2 3 |
... # acquire the lock with a timeout lock.acquire(timeout=10) |
We can also use the reentrant lock via the context manager protocol via the with statement, allowing the critical section to be a block within the usage of the lock and for the lock to be released once the block is exited.
For example:
|
1 2 3 4 5 6 |
... # create a reentrant lock lock = multiprocessing.RLock() # acquire the lock with lock: # ... |
RLocks are also available for threads in Python, you can learn more in the tutorial:
Now that we know how to use the multiprocessing.RLock class, let’s look at a worked example.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Using an RLock
We can develop an example to demonstrate how to use the multiprocessing.RLock for processes.
First, we can define a function to report that a process is done that protects the print() statement with a lock.
|
1 2 3 4 5 |
# reporting function def report(lock, identifier): # acquire the lock with lock: print(f'>process {identifier} done') |
Next, we can define a task function that reports a message, blocks for a moment, then calls the reporting function. All of the work is protected with the lock.
|
1 2 3 4 5 6 7 8 |
# work function def task(lock, identifier, value): # acquire the lock with lock: print(f'>process {identifier} sleeping for {value}') sleep(value) # report report(lock, identifier) |
Given that the target task function is protected with a lock and calls the reporting function that is also protected by the same lock, we can use a reentrant lock so that if a process acquires the lock in task(), it will be able to re-enter the lock in the report() function.
Next, we can create the reentrant lock.
|
1 2 3 |
... # create a shared reentrant lock lock = RLock() |
We can then create many processes, each configured to execute our task() function.
Each process will receive the shared multiprocessing.RLock as an argument as well as an integer id between 0 and 9 and a random time to sleep in seconds between 0 and 1.
We can implement this via a list comprehension, creating a list of ten configured multiprocessing.Process instances.
|
1 2 3 |
... # create processes processes = [Process(target=task, args=(lock, i, random())) for i in range(10)] |
Next, we can start all of the processes.
|
1 2 3 4 |
... # start child processes for process in processes: process.start() |
Finally, we can wait for all of the new child processes to terminate.
|
1 2 3 4 |
... # wait for child processes to finish for process in processes: process.join() |
Tying this together, the complete example of using a lock is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# SuperFastPython.com # example of a reentrant lock for processes from time import sleep from random import random from multiprocessing import Process from multiprocessing import RLock # reporting function def report(lock, identifier): # acquire the lock with lock: print(f'>process {identifier} done') # work function def task(lock, identifier, value): # acquire the lock with lock: print(f'>process {identifier} sleeping for {value}') sleep(value) # report report(lock, identifier) # entry point if __name__ == '__main__': # create a shared reentrant lock lock = RLock() # create processes processes = [Process(target=task, args=(lock, i, random())) for i in range(10)] # start child processes for process in processes: process.start() # wait for child processes to finish for process in processes: process.join() |
Running the example starts up ten processes that execute the target task function.
Only one process can acquire the lock at a time, and then once acquired, blocks and then reenters the same lock again to report the done message via the report() function.
If a non-reentrant lock, e.g. a multiprocessing.Lock was used instead, then the process would block forever waiting for the lock to become available, which it can’t because the process already holds the lock.
Note, your specific results will differ given the use of random numbers. Try running the example a few times.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
>process 0 sleeping for 0.9703475136810683 >process 0 done >process 1 sleeping for 0.10372469305828702 >process 1 done >process 2 sleeping for 0.26627777997152036 >process 2 done >process 3 sleeping for 0.9821832886127358 >process 3 done >process 6 sleeping for 0.005591916432016064 >process 6 done >process 5 sleeping for 0.6150762561153148 >process 5 done >process 4 sleeping for 0.3145220383413917 >process 4 done >process 7 sleeping for 0.8961655132345371 >process 7 done >process 8 sleeping for 0.5968254072867757 >process 8 done >process 9 sleeping for 0.8139723778675512 >process 9 done |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to use the multiprocessing.RLock reentrant mutex lock for processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Maxi am Brunnen on Unsplash
Do you have any questions?