How to Share Variables Between Threads in Python

Last Updated on September 12, 2022

You can protect data variables shared between threads using a threading.Lock mutex lock, and you can share data between threads explicitly using queue.Queue.

In this tutorial you will discover how to share data between threads safely.

Let’s get started.

Table of Contents

Need to Share Data Between Threads

A thread is a thread of execution in a computer program.

Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create additional threads in our program in order to execute code concurrently.

Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.

You can learn more about Python threads in the guide:

Threading in Python: The Complete Guide

In concurrent programming we may need to share data between threads.

Data may refer to many different things, such as:

Global variables.
Local variables.
Instance variables.

It could be something as simple as a counter or a boolean flag, to application specific data and data structures.

We may need to share data for many reasons because multiple threads need to read and/or write to the same data variable.

The problem with multiple threads reading and writing the same variable is that it can result in a concurrency failure mode called a race condition.

You can learn more about race conditions here:

Race Condition With a Shared Variable in Python

How can we share data between threads safely?

Run loops using all CPUs, download your FREE book to learn how.

How to Share Data Between Threads

There are many ways to share data between threads safely.

The specific method to use depends on the type of data to be shared.

Three common approaches include:

Sharing a boolean variable with a threading.Event.
Protecting shared data with a threading.Lock.
Sharing data with a queue.Queue.

Let’s take a closer look at each in turn.

Share a Boolean Variable with an Event

When sharing a boolean flag, an event can be used via the thread.Event class.

The event class will protect a boolean variable ensuring all access and change to the variable is thread safe, avoiding race conditions.

The event can be created in the unset or False state.

...

# create an event

event = threading.Event()

It can then be shared between threads.

The status of the event can be checked safely via the is_set() function.

For example:

...

# check if the event is set

if event.is_set():

# ...

The value event can be changed by multiple different threads. It can be set to True via the set() function and set False via the clear() function.

For example:

...

# set the event true

event.set()

# ...

# set the event false

event.clear()

You can learn more about the event here:

How to Use an Event Object In Python

Access Shared Data With a Lock

When sharing ad hoc variables between threads, a mutual exclusion lock (mutex) can be used via the threading.Lock class.

A lock can be used to protect one or multiple shared variables and they may be variables of any type. The shared variables will be protected from race conditions as long as all access and changes to the variables are protected by the lock.

Each thread interested in the variable must first acquire the lock, and then release it once they are finished with the variable. This will ensure that only one thread can access the variable at a time. A thread trying to acquire the lock that has already been acquired must wait until the lock has been released again.

This can be achieved using the acquire() and release() functions, for example:

...

# create a shared lock

lock = threading.Lock()

...

# acquire the lock

lock.acquire()

# read or write the shared variable

...

# release the lock

lock.release()

We can achieve the same effect using the context manager, will acquire the lock and ensure it is always released when the block is exited.

For example:

...

# create a shared lock

lock = threading.Lock()

...

# acquire the lock

with lock:

# read or write the shared variable

...

You can learn more about locks here:

How to Use a Mutex Lock in Python

Share Data With a Queue

A queue can be used to share data between threads via the queue.Queue class.

A queue is a thread-safe data structure that can be used to share data between threads without a race condition.

The queue module provides a number of queue types, such as:

Queue: A fully-featured first-in-first-out (FIFO) queue.
SimpleQueue: A FIFO queue with less functionality.
LifoQueue: A last-in-first-out (LIFO) queue.
PriorityQueue: A queue where the first items out are those with the highest priority.

A queue can be created then shared between multiple threads.

...

# create a queue

queue = Queue()

Once shared, we may configure one or multiple threads to add data to the queue.

This can be achieved via the put() function. We can put any Python object we like on the queue.

For example:

...

# loop over data

for i in range(100):

# add a data item to the queue

queue.put(i)

We may then configure one or multiple other threads to read data from the queue.

This can be achieved using the get() function. Once an item of data has been retrieved from the queue, it is removed. This means that only one thread can get each item on the queue.

For example:

...

# loop forever

while True:

# get an item of data from the queue

data = queue.get()

# ...

Now that we know how to share variables, let’s look at sharing variables at different scopes.

Download Now: Free Threading PDF Cheat Sheet

How to Share Variables at Different Scopes

We may need to share variables defined at different scopes.

For example:

Local variables within a function.
Global variables defined in a script.
Instance variables defined in a class.

Let’s take a look at each in turn.

Share Local Variables

A variable defined in a function is called a local variable.

We can share a local variable between threads using a queue.

The queue must be shared and accessible to each thread and within the function where the local variable is defined and used.

For example, we can first create a queue.Queue instance.

...

# create a queue

queue = queue.Queue()

We can then pass the queue to our function as an argument.

# custom task function executed by a thread

def task_function(queue):

# ...

Within our function, we can define our local variable and assign it some value.

...

# create a local variable

data = 55

We can then share our local variable with another thread by putting it in a queue.

For example:

...

# share the local variable

queue.put(data)

The function may look as follows:

# custom task function executed by a thread

def task_function(queue):

# create a local variable

data = 55

# share the local variable

queue.put(data)

The queue instance can then be shared with another thread that may execute some other function.

# custom task function executed by another thread

def another_task_function(queue):

# ...

This other thread can then access the shared local variable via the shared queue instance.

For example:

...

# get shared local data from the queue

data = queue.get()

This function may look as follows:

# custom task function executed by another thread

def another_task_function(queue):

# get shared local data from the queue

data = queue.get()

Next, let’s look at how we might share a global variable between threads.

Share Global Variables

A global variable is a variable defined outside of a function.

For example:

...

# define a global variable

data = 66

If the global variable is defined prior to the definition of functions and classes, then those functions and classes can access and modify it directly.

For example:

# custom function

def custom():

# modify the global variable

data = 33

If the global variable is defined after functions and classes, then the scope of the variable can be specified, allowing the global variable to be accessed at the correct scope.

For example:

# custom function

def custom():

# scope the global variable

global data

# modify the global variable

data = 33

Multiple threads may be able to access the global variable directly, as described above.

We can protect the global variable from race conditions by using a mutual exclusion lock via the threading.Lock class.

First, we can create a lock at the same global scope as the global variable.

For example:

...

# define a global variable

data = 66

# define a lock to protect the global variable

lock = threading.Lock()

Each time the global variable is read or modified, it must be done so via the lock.

Specifically, we must acquire the lock before we attempt to interact with the global variable. This will ensure that only one thread is interacting with the global variable at a time.

For example:

# custom function

def custom():

# acquire the lock for the global variable

with lock:

# modify the global variable

data = 33

Share Instance Variables

An instance variable is a variable defined on an object instance of a class.

For example, it is common to define and initialize instance variables in the constructor of a class.

# custom class

class CustomClass():

# constructor

def __init__(self):

# define an instance variable

self.data = 33

The instance variable may then be used within a method on the class.

For example:

# method on the class

def task(self):

self.data = 22

The entire class may look as follows:

# custom class

class CustomClass():

# constructor

def __init__(self):

# define an instance variable

self.data = 33

# method on the class

def task(self):

self.data = 22

We may create an instance of the class and multiple threads may access the class, such as calling the methods on the class that access or modify the instance variable.

...

# create an instance

custom = CustomClass()

custom.task()

Each thread that accesses methods on the shared object may modify the instance variable, leading to race conditions.

One way to prevent a race with instance variables within the object instance is to ensure that all access to the shared object is protected by a mutex lock.

For example:

...

# create an instance

custom = CustomClass()

# create a lock to protect the instance

lock = threading.Lock()

...

# protect the object instance

with lock:

# interact with shared object

custom.task()

Another approach is to protect the instance variable within the class itself.

This can be achieved by defining a lock as a second instance variable within the class, such as in the class constructor.

For example:

# constructor

def __init__(self):

# define an instance variable

self.data = 33

# define a lock to protect the instance variable

self.lock = threading.Lock()

Then in methods that interact with the instance variable, we can protect the variable with the lock.

# method on the class

def task(self):

# acquire the lock

with self.lock:

# modify the instance variable

self.data = 22

This encapsulates the thread safety measure for the instance variable within the class.

Threads can then call functions on the class and know that the internal state of the class is protected. Comments on the class would need to declare that the class is thread-safe and those methods that attempt to acquire a lock would need to declare that they may block.

Free Python Threading Course

Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.

Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores

Learn more