Last Updated on October 21, 2022
You can fix race conditions with shared variables using a mutual exclusion lock.
In this tutorial you will discover race conditions with shared variables in Python.
Let’s get started.
Race Condition with a Shared Variable
A race condition is a bug in concurrency programming.
It is a failure case where the behavior of the program is dependent upon the order of execution by two or more threads. This means, the behavior of the program will not be predictable, possibly changing each time it is run.
There are many types of race conditions, although a common type of race condition is when two or more threads attempt to change the same data variable.
NOTE: Race conditions are a real problem in Python when using threads, even in the presence of the global interpreter lock (GIL). The refrain that there are no race conditions in Python because of the GIL is dangerously wrong.
For example, one thread may be adding values to a variable, while another thread is subtracting values from the same variable.
Let’s call them an adder thread and a subtractor thread.
At some point, the operating system may context switch from the adding thread to the subtracting thread in the middle of updating the variable. Perhaps right at the point where it was about to write an updated value with an addition, say from the current value of 100 to the new value of 110.
|
1 2 3 |
... # add to the variable variable = variable + 10 |
You may recall that the operating system controls what threads execute and when, and that a context switch refers to pausing the execution of a thread and storing its state, while unpausing another thread and restoring its state.
You may also notice that the adding or subtracting from the variable is composed of at least three steps:
- Read the current value of the variable.
- Calculate a new value for the variable.
- Write a new value for the variable.
A context switch between threads may occur at any point in this task.
Back to our threads. The subtracting thread runs and reads the current value as 100 and reduces the value from 100 to 90.
|
1 2 3 |
... # subtract from the variable variable = variable - 10 |
This subtraction is performed as per normal and the variable value is now 90.
The operating system context switches back to the adding thread and it picks up where it left off writing the value 110.
This means that in this case, one subtraction operation was lost and the shared balance variable has an inconsistent value. A race condition.
Next, let’s look at how we might fix this race condition.
Run loops using all CPUs, download your FREE book to learn how.
Fix a Race Condition with a Shared Variable
There are many ways to fix a race condition between two or more threads with a shared variable.
The approach taken may depend on the specifics of the application.
Nevertheless, a common approach is to protect the critical section of code. This may be achieved with a mutual exclusion lock, sometimes simply called a mutex.
If you are new to the mutex lock, you can learn more here:
You may recall that a critical section of code is code that may be executed by two or more threads concurrently and may be at risk of a race condition.
Updating a variable shared between threads is an example of a critical section of code.
A critical section can be protected by a mutex lock which will ensure that one and only one thread can access the variable at a time.
This can be achieved by first creating a threading.Lock instance.
|
1 2 3 |
... # create a lock lock = threading.Lock() |
When a thread is about to execute the critical section, it can acquire the lock by calling the acquire() function. It can then execute the critical section and release the lock by calling the release() function on the lock.
For example:
|
1 2 3 4 5 6 7 |
... # acquire the lock lock.acquire() # add to the variable variable = variable + 10 # release the lock lock.release() |
We might also use the context manager on the lock, which will acquire and release the lock automatically for us.
|
1 2 3 4 5 6 |
... # acquire the lock with lock: # add to the variable variable = variable + 10 # release the lock automatically |
If one thread has acquired the lock, another thread cannot acquire it, therefore cannot execute a critical section and in turn cannot update the shared variable.
Instead, any threads attempting to acquire the lock while it is acquired must wait until the lock is released. This waiting for the lock to be released is performed automatically within the call to acquire(), no need to do anything special.
Now that we know how to fix a race condition with a shared variable, let’s look at a worked example.
Example of a Race Condition With a Shared Variable
We can explore how to create a race condition with a shared variable and multiple threads.
In this example, we will have a shared variable that maintains the final value and two threads that change the value.
One thread will make additions and other subtractions. The additions and subtractions will be the same amount and will be symmetric so that we expect the final value of the shared variable to be 0.
First, let’s define a function for making additions. It will take the amount to add and the number of times to add the value in a loop.
The shared variable will be a global variable named “value“.
|
1 2 3 4 5 |
# make additions into the global variable def adder(amount, repeats): global value for _ in range(repeats): value += amount |
Next, we can define a function to subtract from the shared variable, also taking the amount and number of times to repeat the operation.
|
1 2 3 4 5 |
# make subtractions from the global variable def subtractor(amount, repeats): global value for _ in range(repeats): value -= amount |
In the main thread we can first define our global variable.
|
1 2 3 |
... # define the global variable value = 0 |
Next, we can create a threading.Thread instance and configure it to execute our adder() function passing in the amount of 100 and 1,000,000 iterations.
|
1 2 3 4 |
... # start a thread making additions adder_thread = Thread(target=adder, args=(100, 1000000)) adder_thread.start() |
We need many iterations of each add and subtract operation to give the operating system a chance to context switch between the threads and force a race condition.
Next, we can configure and start a thread for making subtractions with the same amount and number of iterations, ensuring symmetry in the updates to the shared value. This symmetry is not required, but it keeps things simple. It means that we expect the final value of the “value” variable to be zero.
|
1 2 3 4 |
... # start a thread making subtractions subtractor_thread = Thread(target=subtractor, args=(100, 1000000)) subtractor_thread.start() |
Finally, the main thread will wait for both adder and subtractor threads to finish before reporting the final value of the shared variable.
|
1 2 3 4 5 6 7 |
... # wait for both threads to finish print('Waiting for threads to finish...') adder_thread.join() subtractor_thread.join() # report the value print(f'Value: {value}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of a race condition with a shared variable from threading import Thread # make additions into the global variable def adder(amount, repeats): global value for _ in range(repeats): value += amount # make subtractions from the global variable def subtractor(amount, repeats): global value for _ in range(repeats): value -= amount # define the global variable value = 0 # start a thread making additions adder_thread = Thread(target=adder, args=(100, 1000000)) adder_thread.start() # start a thread making subtractions subtractor_thread = Thread(target=subtractor, args=(100, 1000000)) subtractor_thread.start() # wait for both threads to finish print('Waiting for threads to finish...') adder_thread.join() subtractor_thread.join() # report the value print(f'Value: {value}') |
Running the example first starts the adder and subtractor threads then waits for both threads to finish.
Both threads run as fast as possible. Because there are many iterations within each thread, there are enough operations for the operating system to context switch between the threads many times.
All it takes is for once a case where a thread calculates a new value but is context switched before it can write.
In this case, we can see that many context switches occurred. We expect the final value of the shared variable to be zero given the symmetrical number of additions and subtractions. In this case the final value was 34,751,200.
Note, your specific final value will differ. And this too is a problem, highlighting the non-deterministic behavior of the program. We expect a final value of zero each time and in fact have no idea what the actual value will be.
|
1 2 |
Waiting for threads to finish... Balance: 34751200 |
Run the program again, and you will see a different final value.
|
1 2 |
Waiting for threads to finish... Balance: 9893100 |
Now that we are familiar with a race condition with a shared variable in Python, let’s look at how we might fix it.
Update, the above example does cause a race condition in Python 3.9 and below, but not in Python 3.10 and above. The next section addresses this directly.
Free Python Threading Course
Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Example of Race Condition in Python 3.10+
A change was made to the CPython interpreter in Python 3.10.
Race conditions that used to readily occur in Python 3.9 and below no longer seem to occur as readily.
If you know the detailed reason, please let me know in the comments.
Nevertheless, the example above that generates a race condition in earlier Python versions, no longer causes a race condition in modern Python versions, 3.10+.
Therefore, we can update the example to force a race condition.
This involves two parts.
Firstly, we can unroll the variable increment and decrement operations into separate Python expressions.
For example:
|
1 2 |
... value += amount |
Becomes:
|
1 2 3 4 5 6 7 |
... # copy the value tmp = value # change the copy tmp = tmp + amount # copy the value back value = tmp |
The updated adder() and subtractor() functions with these changes look as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# make additions into the global variable def adder(amount, repeats): global value for _ in range(repeats): # copy the value tmp = value # change the copy tmp = tmp + amount # copy the value back value = tmp # make subtractions from the global variable def subtractor(amount, repeats): global value for _ in range(repeats): # copy the value tmp = value # change the copy tmp = tmp - amount # copy the value back value = tmp |
Secondly, we need to give the underlying operating system a good reason to choose to context switch between the operations.
This can be achieved by adding a sleep between each operation.
A sleep for zero seconds should be sufficient on many platforms.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 |
... # copy the value tmp = value # suggest a context switch sleep(0) # change the copy tmp = tmp + amount # suggest a context switch sleep(0) # copy the value back value = tmp |
This works fine on most Unix-based systems I’ve tested it on.
If this is optimized away or ignored on some platforms, like Windows, then the sleep can be increased to a tiny fraction of a second.
The updated functions with these changes are listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# make additions into the global variable def adder(amount, repeats): global value for _ in range(repeats): # copy the value tmp = value # suggest a context switch sleep(0) # change the copy tmp = tmp + amount # suggest a context switch sleep(0) # copy the value back value = tmp # make subtractions from the global variable def subtractor(amount, repeats): global value for _ in range(repeats): # copy the value tmp = value # suggest a context switch sleep(0) # change the copy tmp = tmp - amount # suggest a context switch sleep(0) # copy the value back value = tmp |
Tying this together, the complete updated example that will force a race condition on more recent versions of Python is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
# SuperFastPython.com # example of a race condition with a shared variable (Python 3.10+) from time import sleep from threading import Thread # make additions into the global variable def adder(amount, repeats): global value for _ in range(repeats): # copy the value tmp = value # suggest a context switch sleep(0) # change the copy tmp = tmp + amount # suggest a context switch sleep(0) # copy the value back value = tmp # make subtractions from the global variable def subtractor(amount, repeats): global value for _ in range(repeats): # copy the value tmp = value # suggest a context switch sleep(0) # change the copy tmp = tmp - amount # suggest a context switch sleep(0) # copy the value back value = tmp # define the global variable value = 0 # start a thread making additions adder_thread = Thread(target=adder, args=(100, 1000000)) adder_thread.start() # start a thread making subtractions subtractor_thread = Thread(target=subtractor, args=(100, 1000000)) subtractor_thread.start() # wait for both threads to finish print('Waiting for threads to finish...') adder_thread.join() subtractor_thread.join() # report the value print(f'Value: {value}') |
Running the example, we can see that the global variable is left in an inconsistent state each time the code is run.
This race occurs on Python 3.9 and below, as well as Python 3.10 and above.
Did it trigger a race condition for you?
Let me know in the comments below. Post your result.
|
1 2 |
Waiting for threads to finish... Value: -99997400 |
Next, let’s look at how we might fix a general race condition using a mutex lock
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Fixing a Race Condition With a Shared Variable
We can fix a race condition with a shared variable by protecting the shared variable with a mutex lock.
The example from the previous section with a race condition can be updated to use a lock which will fix the race condition. We will know that the race condition is fixed if the program reports the expected value of zero each time it is run.
Firstly, we can update the adder() function to take a shared threading.Lock instance as an argument and to then ensure that the lock is acquired for each update to the shared variable.
You can learn more about mutex locks in the tutorial:
The updated version of the adder() function with this change is listed below.
|
1 2 3 4 5 6 |
# make additions into the global variable def adder(amount, repeats, lock): global value for _ in range(repeats): with lock: value += amount |
We can then make the same change to the subtractor() function.
|
1 2 3 4 5 6 |
# make subtractions from the global variable def subtractor(amount, repeats, lock): global value for _ in range(repeats): with lock: value -= amount |
Finally, we can define the shared threading.Lock instance in the main thread.
|
1 2 3 |
... # define a lock to protect the shared variable lock = Lock() |
And pass it to each new thread as an argument.
|
1 2 3 4 5 6 7 |
... # start a thread making additions adder_thread = Thread(target=adder, args=(100, 1000000, lock)) adder_thread.start() # start a thread making subtractions subtractor_thread = Thread(target=subtractor, args=(100, 1000000, lock)) subtractor_thread.start() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# SuperFastPython.com # example of fixing a race condition with a shared variable from threading import Thread from threading import Lock # make additions into the global variable def adder(amount, repeats, lock): global value for _ in range(repeats): with lock: value += amount # make subtractions from the global variable def subtractor(amount, repeats, lock): global value for _ in range(repeats): with lock: value -= amount # define the global variable value = 0 # define a lock to protect the shared variable lock = Lock() # start a thread making additions adder_thread = Thread(target=adder, args=(100, 1000000, lock)) adder_thread.start() # start a thread making subtractions subtractor_thread = Thread(target=subtractor, args=(100, 1000000, lock)) subtractor_thread.start() # wait for both threads to finish print('Waiting for threads to finish...') adder_thread.join() subtractor_thread.join() # report the value print(f'Value: {value}') |
Running the example creates and starts the adder and subtractor threads.
Each thread runs as fast as possible making changes to the shared variable.
Importantly, only one thread is able to make a change to the variable at a time given the protection offered by the mutex lock.
As such once both threads finish, the final value of the shared variable is reported as zero. In fact, every time the program is run, a value of zero will now be reported, making the program entirely deterministic as intended.
|
1 2 |
Waiting for threads to finish... Balance: 0 |
This shows how we might fix a race condition with a shared variable using a mutex lock.
Further Reading
This section provides additional resources that you may find helpful.
Python Threading Books
- Python Threading Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- Threading Module API Cheat Sheet
I also recommend specific chapters in the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Threading: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Takeaways
You now know about race conditions with shared variables in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Vanessa Serpas on Unsplash
Very Informative artical about python race condition.
Thank you, I’m happy to hear that it helped!
Excellent Jason! Can you also write an article about what makes 3.10+ clear from the first example?
Yes.
The main difference is that the 3.10+ introduces more opportunity for the operating system to context switch between threads.
It does this by working on a temporary variable and adding zero duration sleeps.
Jason, excellent guide how to fix race conditions bugs! But what are the advantages with using temp var & sleep over mutex? Mutex looks a lot cleaner, better and what’s most important the code behave as it should at every run, doesn’t it?
Using a tmp var and a sleep is not a fix, it is way to force a race condition in Python.
To avoid a race condition with a shared variable, we can use a mutex.