Last Updated on September 12, 2022

You can create and use context variables via the contextvars.ContextVar class.

In this tutorial you will discover how to use context variables in Python.

Let’s get started.

What are Context Variables

A Context Variables or contextvars for short is a variable that can be defined per thread or per coroutine.

They provide local storage for variables that is distinct per context, where the context may be:

• A thread, meaning each thread will have a different version of a contextvar variable.
• A coroutine, meaning each coroutine task will have a different version of a variable.

Recall, threads are available in Python via the threading module and coroutines are available in Python via the asyncio module.

Context variables were created to provide thread-local-like storage that works the same way in threads and in coroutines.

This concept is similar to thread-local storage (TLS), but, unlike TLS, it also allows correctly keeping track of values per asynchronous task, e.g. asyncio.Task.

— PEP 567 – Context Variables

As such, contextvars were primarily developed for use with asyncio. Strictly, asyncio will define the “current context” within the asyncio framework, such as at the per-task level.

Manipulation of the current context is the responsibility of the task framework, e.g. asyncio.

— PEP 567 – Context Variables

Nevertheless, they can be used with Python thread as well for consistency.

Now that we know what context variables are, let’s consider why we might use them.

Why Use Context Variables?

The use case for context variables is similar to that of thread-local storage.

Specifically, we may have code that relies on state that we would like to be specific per concurrent task, such as a thread or a coroutine.

A common situation occurs where the state is managed in a global variable, for example, one function or a call graph of functions that read and write state in global variables.

If this code was made concurrent with threads or coroutines, then each concurrent task would use the same global state, which would quickly become corrupt.

One solution would be to protect the global state using synchronization primitives, like mutual exclusion (mutex) locks. But this is inappropriate if the tasks do not share state with each other, and instead only use the same global state variables.

Instead, the global state may be changed to use contextvars.

In this case, each concurrent task (thread or coroutine) may then read and write its own values to the contextvars which is isolated automatically from all other concurrent tasks.

This is particularly useful for updating legacy programs that use global state to be concurrent.

Now that we know why we might want to use context variables, let’s contrast them with thread-local storage.

Context Variables vs Thread-Local Storage

Both context variables and thread-local storage manage variables per concurrent task.

The main differences between the two are:

1. Thread-Local Storage only works per-thread, whereas context variables work per thread or per coroutine.
2. They have a different API, the thread-local storage API is simpler, context variables API is slightly more complex.

Thread-Local Storage will manage variables with the same name per thread.

This means the same thread local storage context can define and use a variable with the same name in different threads without any interaction. Thread-local variables are completely isolated from all other threads.

Thread-local data is data whose values are thread specific. To manage thread-local data, just create an instance of local (or a subclass) and store attributes on it

— Thread-Local Data

This mechanism is typically provided by the operating system and Python provides access to this capability via the threading.local() function.

For example, a thread-local context can first be created:

Then a variable can be defined and assigned on the context with an arbitrary name, for example:

Each thread that accesses or modifies the thread-local variable will be isolated and distinct from all other threads.

This allows each thread to re-use the same code without side-effects.

You can learn more about how to use thread-local storage in Python here:

• How to Use Thread-Local Data in Python

Context Variables operate much like thread-local variables.

The main difference is the variables are isolated per “context” rather than per thread, where the context is either per thread or per asyncio Task.

As such, contextvars provide thread-local storage capabilities for concurrent tasks other than threads, specifically for coroutine tasks provided by the asyncio module.

The manner in which context variables are defined and accessed is also very different to thread local data, let’s look at that next.

How to Use Context Variables

Python provides context variables via the contextvars module.

Contextvars were added to Python in version 3.7, ensure you are using this version of Python or higher.

A context variable can be defined via the contextvars.ContextVar() class.

This module provides APIs to manage, store, and access context-local state. The ContextVar class is used to declare and work with Context Variables.

— contextvars — Context Variables

The constructor of the class takes a variable “name“, which is just used for debugging, and an optional default value for the variable.

We can define a context variable with the name “color“.

For example:

We can also create a context variable with a default value of ‘red‘ by specifying the “default” argument.

For example:

The value of the contextvar can be retrieved via the get() function.

For example:

We might also want to get the value of the context variable and specify a default to return if no value has been set.

For example:

The value of the context variable can be set via the set() function.

For example:

Setting the value of a context variable returns a token that captures the value of the context variable before it was changed. This token can be saved.

For example:

Finally, the value of the context variable can be restored using a saved token by calling the reset() function.

For example:

Now that we know how to use context variables, let’s look at a worked example.

Example of Context Variables

We can develop an example to demonstrate how to use context variables.

To make this interesting, we can define a program that uses global state, update it to use threads and show the problems, then adapt it to use context variables that address the problems. We can also update the program to use thread local data to address the problems as a point of comparison with context variables.

Let’s dive in.

Program With Global State

We can develop a simple program that uses global state that we can later update to use contextvars.

In this example, we will have a task function that uses a global variable. Initially the value of the global variable is undefined, then it is set to specific values and the task function is executed.

This is a good task to explore context variables, as the task relies on global state and we may want to execute the task concurrently with different global states in a thread-safe manner.

Firstly, we can define the task function.

The function takes no arguments and uses a “color” global variable that it assumes has been set. The function blocks for a random fraction of a second to simulate doing something useful, then reports the current value of the color.

The color_report() function below implements this.

Next, we can define the global variable and initialize it to an unset by human readable value.

We can then define a series of colors that we may wish to work through using our color task.

Finally, we can iterate over our list of colors. Each iteration we can set the global variable to have the value of a color, then execute the color task.

Tying this together, the complete example is listed below.

Running the program iterates over the list of colors.

Each iteration the color is set and the color report task is executed. The task blocks for a moment then uses the global color variable in a print message.

A sample of the program output is listed below.

Next, let’s look at what happens if we naively try to perform the task concurrently with threads.

Naive Update To Use Threads

We can naively update the program in the previous section to perform each task concurrently with threads.

The program involves executing the same task on different colors. As such, it is a natural case for speed-up by executing each task concurrently.

A naive approach would be to execute each task in a new thread.

This can be achieved by defining a new function that takes a color, sets the global “color” variable to the color, then calls the color_report() function.

The task() function below implements this.

Next, we can then create one thread per color and configure each thread to execute the task() function and pass in a color as an argument.

This can be achieved by creating instances of the threading.Thread class and specifying the function to execute in a new thread via the “target” argument and the arguments to this function as a tuple to the “args” argument.

We can achieve this in a list comprehension, iterating over the list of colors and creating a thread for each.

Next, we can iterate the list of threads and start each one in turn.

Finally, the main thread can wait for all threads to finish.

And that’s it.

Easy enough, except there is a massive problem.

Tying this together, the complete example is listed below.

Running the example creates and configures one thread per color.

The threads are started and each thread executes the task() function. The global variable is assigned to a specific color and the color_report() task is executed.

The problem is, all variables attempt to change the same global variable.

This means that when the color_report() function is executed from each thread, it reads the same global variable shared among all threads that has the same value, e.g. the value assigned by the last thread to run, such as the one for ‘purple‘.

This was a naive adaptation of a program with a global state to use concurrency with threads.

We need a way for each thread to have its own version of the color global variable, protected from being modified by other threads.

A sample output of the program is listed below, showing the corrupted output of the program.

Next, let’s look at how we can protect global state from other threads using context variables.

Update To Use Context Variables

We can update the example from the previous section to use context variables in order to prevent the threads from stepping on each other.

This can be achieved by changing the “color” global variable to be a context variable.

This will require a change to the color_report() function when accessing the value, a change to the task() function when setting the value and a change when defining the variable.

Firstly, we can update the color_report() function to call the get() function on the color global variable when printing its value.

The updated version of the color_report() function with this change is listed below.

Next, we can update the task() function to call the set() function on the color global variable when assigning it to a color for a specific thread.

The updated task() function with this change is listed below.

Finally, we can change the definition of the color global variable to be an instance of the contextvars.ContextVar class with a default value.

Tying this together, the complete example is listed below.

Running the example first creates the context variable global variable, then creates and configures one thread per color.

The threads are started and the main thread blocks until the new threads finish.

Each thread executes the task() function. The context variable is assigned to the color argument for the thread and the color_report() function is called.

The change to the context variable only has an effect for the thread, and is not impacted by changes by other threads.

As such, when the color_report() function is executed by each thread, it uses the thread’s value of the context variable and reports the correct value.

A sample output of the program is listed below, showing that using context variables corrects the problem of threads sharing global state and stepping on each other.

Next, let’s explore how we might also fix the problem using thread-local data.

Update To Use Thread-Local Storage

Because we are using threads, we can also update the program to use thread-local storage that will also address the problem of threads stepping on each other.

This can be achieved by changing the color global variable to be a thread-local context shared among all threads.

This requires changes to the program where the color global variable is used. This includes in the color_report() function where the color variable is used and printed, in the task() function where the global variable is assigned, and in the body of the program where the global variable is defined and initially assigned.

First, we can update the color_report() function and access the color thread-local storage context and the value of the color which we will store as “value“.

The updated color_report() function with this change is listed below.

Next we can update the task() function to assign the variable to the value of the color provided to the thread as an argument.

This will change “value” on the thread-local storage to a value only accessible to the thread.

The updated version of the task() function with this change is listed below.

Finally, we can update the definition of the color global variable.

First, we need to create a thread-local storage context shared among all threads.

We then need to define a variable in this context to hold our color, which we will call “value” and assign it to a default value of “Unknown“.

In fact, this has no effect in any thread other than the main thread, but makes it clear in the program that we expect the color value to be get and set via “color.value“.

Tying this together, the complete example is listed below.

Running the example first creates the global thread-local data context, then creates and configures one thread per color.

The threads are started and the main thread blocks until the new threads finish.

Each thread executes the task() function. The variable on the thread-local storage is assigned to the color argument for the thread and the color_report() function is called.

The change to the thread-local variable only has an effect for the thread, and is not impacted by changes by other threads.

As such, when the color_report() function is executed by each thread, it uses the thread’s value of the thread-local variable and reports the correct value.

A sample output of the program is listed below, showing that using a thread-local variable corrects the problem of threads sharing global state and stepping on each other.

Further Reading

This section provides additional resources that you may find helpful.

Python Threading Books

• Python Threading Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• Threading Module API Cheat Sheet

I also recommend specific chapters in the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Threading: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• threading - Thread-based parallelism
• queue — A synchronized queue class

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to use context variables in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Richard Ciraulo on Unsplash