Last Updated on September 12, 2022
You can use context managers to avoid race conditions by automatically acquiring and releasing thread concurrency primitives like locks and semaphores.
In this tutorial you will discover how to use context managers for threads in Python.
Let’s get started.
What is a Context Manager
A context manager is an interface on Python objects for defining a new run context.
Python’s with statement supports the concept of a runtime context defined by a context manager. This is implemented using a pair of methods that allow user-defined classes to define a runtime context that is entered before the statement body is executed and exited when the statement ends
— Context Manager Types, Built-in Types
A run context is a block of Python code. Examples of other run contexts include the content of a function or the content of a loop.
Context managers allow an object to define the code that runs at the beginning and end of a block of code. This is a lot like the try-except-finally pattern, except the code executed before and after the block is hidden within the object and only the body block of code needs to be specified.
The context manager interface has two methods that must be implemented on an object that supports the interface, they are:
- __enter__(): Executed prior to the code block.
- __exit__(): Executed after the code block.
These two methods are always executed, even if an error or exception occurs within the block. In this way, __exit__() acts like a finally block in a try-except-finally pattern.
The with statement is used to wrap the execution of a block with methods defined by a context manager . This allows common try…except…finally usage patterns to be encapsulated for convenient reuse.
— 8.5. The with statement, 8. Compound statements.
Context managers on Python objects are used via the “with” statement.
The with statement takes the object instance that implements the context manager interface as an argument. It will execute the code in the object for the beginning of the block, e.g. the __enter__() method and return an instance of an object that can be assigned.
For example:
|
1 2 3 4 5 |
... # execute the context manager, calls __enter__() with object as o: # the main block... # end of the context manager, calls __exit__() |
Context managers are useful for operations that require the consistent initialization and shutdown of an operation.
The context manager handles the entry into, and the exit from, the desired runtime context for the execution of the block of code.
— 3.3.9. With Statement Context Managers
Common examples of context managers provided in the Python standard library include:
- Opening and closing files.
- Opening and closing sockets.
Now that we know what context managers are, let’s look at the use of context managers with threads.
Run loops using all CPUs, download your FREE book to learn how.
What is a Context Manager for Threads
Python provides a suite of concurrency primitives for threads that support the context manager interface.
This includes locks, conditions, and semaphores.
The context manager interface on concurrency primitives provides a way to easily avoid common classes of concurrency failure modes called deadlocks by abstracting away and automating the process of acquiring and releasing the concurrency primitive.
Concurrency primitives are typically used to protect a critical section of code subjected to race conditions if executed concurrently by multiple threads.
The concurrency primitive must be acquired prior to a critical section and released at the end of the critical section.
There are three main ways we could handle this, they are:
- Directly.
- With a Try-Except-Finally pattern.
- With a Context Manager.
Let’s take a closer look at each in turn.
Approach 1: Directly
A concurrency primitive can be acquired and released directly.
This can be achieved by calling the acquire() function, followed by the code block, and ending with a call to release().
For example:
|
1 2 3 4 5 6 7 |
... # acquire the primitive primitive.acquire() # execute code block # ... # release the primitive primitive.release() |
The danger of this approach is that if the code block between the call to acquire() and release() raises an Error or Exception, then release() will never be called.
For example:
|
1 2 3 4 5 6 7 8 9 |
... # acquire the primitive primitive.acquire() # execute code block # ... raise Exception('Something bad happened') # ... # release the primitive primitive.release() |
This is a problem because other threads waiting to acquire the primitive may block forever, resulting in a deadlock.
You can learn more about deadlocks in this tutorial:
Approach 2: Try-Except-Finally
A better approach to acquiring and releasing a concurrency primitive directly is to use a try-except-finally pattern.
The primitive is acquired prior to the try statement, the try block defines the main block of code, the exception handles any expected errors or exceptions in the main block of code and the finally block ensures that primitive is always released.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 |
... # acquire the primitive primitive.acquire() try: # execute code block # ... except: # handle expected errors and exceptions finally: # release the primitive primitive.release() |
This is an improvement over directly acquiring and releasing the primitive because regardless of what happens in the main block of code, the primitive is always released.
This prevents a class of deadlocks where primitives are not released due to expected or unexpected errors and exceptions.
In other programming languages like Java and C#, the try-except-finally pattern is a best practice when acquiring and releasing concurrency primitives.
It is not a best practice in Python, because we have something better.
Approach 3: Context Manager
Python provides a context manager interface on concurrency primitives that need to be acquired and released.
This achieves a similar outcome to using a try-except-finally pattern, with less code.
Specifically, it is more like a try-finally pattern, where any exception handling must be added and occur within the code block itself.
For example:
|
1 2 3 4 5 6 |
... # acquire the primitive with primitive: # execute code block # ... # release the primitive automatically |
The benefit is that we can focus on the content of the block, e.g. the critical section of code protected by the concurrency primitive and the acquisition and releasing of the primitive happens for us behind the scenes.
This gives the benefit of avoiding deadlocks like using a try-finally block, although with less code and better readability.
Now that we know the benefit of context managers when using concurrency primitives, let’s look at some examples.
How to Use Context Managers With Threads
We can take a closer look at the context manager interface provided on concurrency primitives in Python.
Specifically, the interface is provided on the following concurrency primitives:
- Mutex locks provided by the threading.Lock class.
- Reentrant locks provided by the threading.RLock class.
- Conditions aka Monitors provided by the threading.Condition class.
- Semaphores provided by the threading.Semaphore class.
- Bounded semaphores provided by the threading.BoundedSemaphore class.
Let’s take a closer look at each in turn.
Lock Context Manager
A mutual exclusion lock or mutex lock is a synchronization primitive intended to prevent race conditions on shared variables.
Python provides a mutual exclusion lock via the threading.Lock class.
An instance of the Lock can be created and then acquired by threads before accessing a critical section, and released after the critical section.
This can be achieved using the context manager interface.
For example:
|
1 2 3 4 5 6 7 |
... # create a lock lock = threading.Lock() ... # acquire the lock with lock: # ... |
You can learn more about mutex locks in this tutorial:
RLock Context Manager
A reentrant mutual exclusion lock or reentrant lock for short, is like a mutex lock except it allows a thread to acquire the lock more than once.
Python provides a reentrant lock via the threading.RLock class.
An instance of the RLock can be created and then acquired by threads before accessing a critical section, and released after the critical section.
This can be achieved using the context manager interface.
For example:
|
1 2 3 4 5 6 7 |
... # create a reentrant lock lock = threading.RLock() ... # acquire the lock with lock: # ... |
You can learn more about reentrant mutex locks in this tutorial:
Condition Context Manager
A condition, also called a monitor, allows multiple threads to be notified about some result.
Python provides a condition via the threading.Condition class.
We can create a condition object and by default it will create a new reentrant mutex lock (threading.RLock class) by default which will be used internally.
In order for a thread to make use of the condition, it must acquire it and release it, like a mutex lock.
This can be achieved using the context manager interface.
Once acquired, we can choose to either notify waiting threads by calling notify(), or wait to be notified by other threads by calling wait().
For example, we can notify waiting threads:
|
1 2 3 4 5 6 7 8 |
... # create the condition condition = threading.Condition() # ... # acquire the condition with condition: # notify a waiting thread condition.notify() |
We may also wait to be notified by another thread:
|
1 2 3 4 5 6 7 8 |
... # create the condition condition = threading.Condition() # ... # acquire the condition with condition: # wait to be notified condition.wait() |
You can learn more about condition objects in this tutorial:
Semaphore Context Manager
A semaphore is a concurrency primitive that allows a limit on the number of threads that can acquire a lock protecting a critical section.
It is used to limit the number of concurrent threads that can perform an action simultaneously, such as execute a critical section, perform a calculation, or operate an external resource.
Python provides a semaphore via the threading.Semaphore class.
The threading.Semaphore instance must be configured when it is created to set the limit on the internal counter. This limit will match the number of concurrent threads that can hold the semaphore.
A position on the semaphore can then be acquired and then released once work with the limited resource is finished.
This can be achieved using the context manager interface.
For example:
|
1 2 3 4 5 6 7 |
... # create the semaphore semaphore = threading.Semaphore(100) # ... # acquire the semaphore with semaphore: # ... |
You can learn more about how to use semaphores in this tutorial:
BoundedSemaphore Context Manager
A bounded semaphore is a type of semaphore that protects the internal counter going above the initial value.
It ensures that the semaphore cannot be released more than it was acquired.
Python provides a semaphore via the threading.BoundedSemaphore class.
Like a semaphore, the threading.BoundedSemaphore instance must be configured when it is created to set the limit on the internal counter. This limit will match the number of concurrent threads that can hold the semaphore.
A position on the bounded semaphore can then be acquired and then released once work with the limited resource is finished.
This can be achieved using the context manager interface.
For example:
|
1 2 3 4 5 6 7 |
... # create the bounded semaphore bounded_semaphore = threading.BoundedSemaphore(100) # ... # acquire the bounded semaphore with bounded_semaphore: # ... |
Free Python Threading Course
Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Further Reading
This section provides additional resources that you may find helpful.
Python Threading Books
- Python Threading Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- Threading Module API Cheat Sheet
I also recommend specific chapters in the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Threading: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to use context managers with threads in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Vladyslav Melnyk on Unsplash
Do you have any questions?