Last Updated on September 12, 2022
You can lock a variable to make it thread-safe using a threading.Lock.
In this tutorial you will discover how to lock a variable in Python.
Let’s get started.
Need to Lock a Variable
A thread is a thread of execution in a computer program.
Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional threads in our program in order to execute code concurrently.
Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.
You can learn more about Python threads in the guide:
In concurrent programming, we may need to protect a variable from being updated from multiple threads and avoid a race condition.
How can we lock a variable in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Lock a Variable
A variable can be locked using a mutual exclusion (mutex) lock.
Mutex Lock
A mutual exclusion lock or mutex lock is a synchronization primitive intended to prevent a race condition.
A mutex lock can be used to ensure that only one thread at a time executes a critical section of code at a time, while all other threads trying to execute the same code must wait until the currently executing thread is finished with the critical section and releases the lock.
Python provides a mutual exclusion lock via the threading.Lock class.
First, an instance of the lock must be created.
|
1 2 3 |
... # create a lock lock = Lock() |
The lock can then be used to protect critical sections of code that require serialized or mutually exclusive access.
This can be achieved by first acquiring the lock by calling the acquire() function before accessing a critical section, then releasing the lock by calling release() after the critical section.
For example:
|
1 2 3 4 5 6 7 |
... # acquire the lock lock.acquire() # critical section # ... # release the lock lock.release() |
Only one thread can have the lock at any time.
If the lock is being held, another thread attempting to acquire the lock will block until the lock is available.
We can also use the lock via the context manager interface.
This will allow the critical section to be a block within the usage of the lock and for the lock to be released automatically once the block has completed.
For example:
|
1 2 3 4 |
... # acquire the lock with lock: # ... |
The context manager is the preferred approach to using the lock as it guarantees the lock is always released, even if an unexpected Error or Exception is raised.
You can learn more about mutex locks in the tutorial:
Now that we know how to use a lock, let’s look at how to lock a shared variable.
Lock a Shared Variable
A variable shared among multiple function calls can be locked.
This requires first that an instance of the threading.Lock class be created alongside the shared variable.
|
1 2 3 4 5 |
... # create shared lock lock = threading.Lock() # create shared variable data = ... |
Each time the shared variable is used or modified it must be protected by the lock.
For example:
|
1 2 3 4 5 |
... # acquire the lock with lock: # modify the variable data = ... |
Additionally, each time the shared variable is passed to a function to be modified, the lock must be passed with it.
|
1 2 3 |
# some function def work(data, lock): # ... |
Then, each time the shared variable is used, it must be protected by the lock.
|
1 2 3 4 5 6 7 |
# some function def work(data, lock): # ... # acquire the lock with lock: # modify the variable data = ... |
If one function that uses the lock and shared variable calls another function that makes use of the lock and shared variable while the lock is being held, then a reentrant lock is required.
This is provided by the threading.RLock class and allows one thread to acquire the same lock more than once.
You can learn more about reentrant locks in this tutorial:
Lock a Global Variable
A global variable that is modified by multiple threads may be locked.
This requires first creating an instance of a threading.Lock alongside the global variable.
|
1 2 3 4 5 |
... # create the global variable lock lock = threading.Lock() # create the global var variable data = ... |
Then, each time the global variable is used or modified, it can be protected by the lock.
For example:
|
1 2 3 4 5 |
... # acquire the lock with lock: # modify the global variable # ... |
Now that we know how to lock a variable, let’s look at some worked examples.
Example of Locking a Variable
We can explore an example of how to lock a variable.
In this example, we will define a variable as an object, then pass this variable to a function where it will be modified. The variable will be shared among multiple threads, that will all call the same function concurrently with the same shared variable, potentially resulting in a race condition and corruption of our variable. Therefore, we will protect our variable from concurrent modification by locking it.
We will first develop the thread unsafe version that results in a race condition in data loss, then update it to use the lock which will fix the race condition.
Example Without Lock (Thread-Unsafe)
In this example, we will not use the lock, which will result in a race condition.
We will use a list as the shared variable object and initialize it to a length of 100 with 0 values in all positions. We will then increment the values on the list from multiple threads concurrently. Threads will race to make these changes and some data will be lost, called a race condition.
First we must develop a task function to execute in new threads.
The function will take the shared list, iterate many times and each iteration it will iterate over the list and increment each value in the list.
The task() function below implements this.
|
1 2 3 4 5 6 |
# task function to be executed in new threads def task(data): # iterate many times for i in range(1000): for j in range(len(data)): data[j] += 1 |
Next, we can initialize the list variable with zero values.
|
1 2 3 |
... # create the shared variable data = [0 for i in range(100)] |
We can then create many new threads and configure each to execute our task() function and operate on the same shared list variable.
This can be achieved in a list comprehension.
For example:
|
1 2 3 |
... # create threads to operate on the shared variable threads = [Thread(target=task, args=(data,)) for _ in range(100)] |
In this case, we will have 100 threads calling the task() function. Each thread will iterate 1,000 times and increment the values in the list. Therefore, without a race condition, we would expect each value in the list to be 100,000 at the end of the program, e.g. 100 threads * 1,000 iterations of being incremented.
Next, we can start the threads, then the main thread will wait until all threads have finished and terminated.
|
1 2 3 4 5 6 7 |
... # start threads for thread in threads: thread.start() # wait for threads to terminate for thread in threads: thread.join() |
Finally, we can report the values in the list.
|
1 2 3 |
... # report data print(data) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of updating a variable from many threads resulting in a race condition from threading import Thread # task function to be executed in new threads def task(data): # iterate many times for i in range(1000): for j in range(len(data)): data[j] += 1 # create the shared variable data = [0 for i in range(100)] # create threads to operate on the shared variable threads = [Thread(target=task, args=(data,)) for _ in range(100)] # start threads for thread in threads: thread.start() # wait for threads to terminate for thread in threads: thread.join() # report data print(data) |
Running the example first initializes the shared list with zero values.
Then one hundred threads are created and configured to execute our task() function on the shared list variable.
The threads are started and the main thread blocks until the threads terminate.
All threads execute concurrently. The global interpreter lock ensures that only one thread can update the Python interpreter at a time, because the operating system will context switch between threads frequently, giving each an opportunity to run for a short time.
Each thread will iterate 1,000 times and increment values in the list.
A race condition exists in the way the values in the list are incremented. An increment involves:
- Reading the current value from a position in the list.
- Calculating a new value.
- Writing the new value to the position in the list.
It is possible for a thread to be context switched in the middle of these steps, and when it releases it may write a value that is invalid, e.g. less than the actual value because another thread updated it.
This race condition means some incremented values in the list will be less than 100,000 at the end of the program.
And this is exactly what we see.
Note, your specific results may differ given how the race condition may manifest on different platforms.
|
1 |
[97428, 99862, 100000, 99844, 100000, 99016, 100000, 99793, 100000, 100000, 99088, 99739, 100000, 100000, 99066, 100000, 100000, 100000, 99894, 100000, 99149, 100000, 100000, 100000, 100000, 100000, 99069, 100000, 100000, 97489, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 99086, 100000, 100000, 100000, 99079, 99256, 100000, 99700, 100000, 100000, 99105, 100000, 100000, 100000, 100000, 100000, 97869, 99138, 99273, 100000, 100000, 100000, 99728, 100000, 100000, 100000, 98113, 99125, 97816, 100000, 100000, 100000, 100000, 100000, 100000, 99760, 97788, 100000, 97992, 99814, 100000, 99908, 100000, 98819, 100000, 98319, 98513, 99296, 99286, 99705, 99182, 100000, 100000, 99012, 100000, 97833, 100000, 100000, 100000, 98497, 97108] |
We can fix this race condition by using a lock to protect the variable.
Example With Lock (Thread-Safe)
The race condition can be fixed by making all usage of the shared variable mutually exclusive.
That means, while one thread is operating on the variable, other threads must wait until the thread is finished.
This can be achieved using a mutex lock provided by the threading.Lock class.
The lock can be created and shared along with the shared list variable. Each time the values in the list are modified by threads in the task() function, the lock must be acquired, then released once the updates are complete.
We can update the example in the previous section to demonstrate this.
First, the task() function must be updated to take the shared lock as an argument, along with the shared list.
For example:
|
1 2 3 4 |
... # task function to be executed in new threads def task(data, lock): # ... |
Next, we must update the function so that the lock is acquired before updating the values in the list.
We have a few options of where we may place the lock.
Such as:
- Lock the entire function, e.g. before the outer iteration.
- Lock each iteration, e.g. within the outer loop.
- Lock each update to the list, e.g. within the inner loop.
The choice of where to place the lock depends on the specifics of the application.
A coarse grained lock, e.g. locking the whole function, may mean that threads acquiring the lock will hold the lock for longer and other threads waiting for the lock will have to wait for longer. Overall, execution of all threads may be faster.
For example:
|
1 2 3 4 5 6 7 8 |
# task function to be executed in new threads def task(data, lock): # acquire the lock with lock: # iterate many times for i in range(1000): for j in range(len(data)): data[j] += 1 |
The fine-grain lock, e.g. locking each increment to the list, may mean that the lock is acquired for a very short duration, and waiting threads will not have to wait long. The lock would be acquired and released many times, adding overhead and slowing down the execution of all threads.
For example:
|
1 2 3 4 5 6 7 8 |
# task function to be executed in new threads def task(data, lock): # iterate many times for i in range(1000): for j in range(len(data)): # acquire the lock with lock: data[j] += 1 |
In this case, we will aim for some middle ground between the two above cases and require that each thread acquire the lock at the beginning of each iteration of the outer loop.
The updated task() function with this change is listed below.
|
1 2 3 4 5 6 7 8 |
# task function to be executed in new threads def task(data, lock): # iterate many times for i in range(1000): # acquire the lock with lock: for j in range(len(data)): data[j] += 1 |
Next, in the main thread, we can create the shared lock along with the shared list variable.
|
1 2 3 |
... # create the shared lock lock = Lock() |
This lock can then be passed to the task() function along with the shared list variable when creating and configuring the new threads.
|
1 2 3 |
... # create threads to operate on the shared variable threads = [Thread(target=task, args=(data,lock)) for _ in range(100)] |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of locking a variable to protect it from a race condition from threading import Thread from threading import Lock # task function to be executed in new threads def task(data, lock): # iterate many times for i in range(1000): # acquire the lock with lock: for j in range(len(data)): data[j] += 1 # create the shared lock lock = Lock() # create the shared variable data = [0 for i in range(100)] # create threads to operate on the shared variable threads = [Thread(target=task, args=(data,lock)) for _ in range(100)] # start threads for thread in threads: thread.start() # wait for threads to terminate for thread in threads: thread.join() # report data print(data) |
Running the example first creates the shared lock, then initializes the shared list with zero values.
Then one hundred threads are created and configured to execute our task() function on the shared list variable.
The threads are started and the main thread blocks until the threads terminate.
Each iteration of the task() function by each thread requires that the lock be acquired before modifying values in the list. One thread will acquire the lock at a time and all other threads must block and wait for the lock to be released.
This process is then repeated until all threads complete their task.
This ensures that the modification of the values in the list variable are mutually exclusive, avoiding the race condition of data loss seen in the previous section.
The results confirm that the race condition was fixed by locking the variable.
Each value in the list has the expected value of 100,000.
|
1 |
[100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000, 100000] |
Free Python Threading Course
Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Further Reading
This section provides additional resources that you may find helpful.
Python Threading Books
- Python Threading Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- Threading Module API Cheat Sheet
I also recommend specific chapters in the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Threading: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to lock a variable in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by vikram sundaramoorthy on Unsplash
Do you have any questions?