Last Updated on September 12, 2022
You can use thread-local data by calling threading.local() and sharing the instance between threads.
In this tutorial you will discover how to use thread-local data storage in Python.
Let’s get started.
Need Data Unique to Each Thread
A thread is a thread of execution in a computer program.
Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional threads in our program in order to execute code concurrently.
Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.
Sometimes we may need to store data that is unique for each thread.
Python provides a thread-local object that can be used to store unique data for each thread. The thread-local storage provides an alternative to having to extend the threading.Thread class and define instance variables for thread-specific data.
What is thread-local data exactly?
Run loops using all CPUs, download your FREE book to learn how.
What is Thread-Local Data?
Thread-local data storage is a mechanism in multi-threaded programming that allows data to be stored and accessed in a way that is private to each thread.
It may be called “thread-local storage“, “thread-private“, or simply “thread-local“.
Typically this involves creating an instance of a thread-local object that is shared between objects on which data is set and retrieved.
Data stored and retrieved on the thread-local object may have the same variable names across threads, meaning the same code can be used across threads, a common approach when using worker threads.
Importantly, the reads and writes to the thread-local object are unique and private at the thread-level. This means one thread may write a variable with the name “address” and another thread may read or write a variable with an identical name but it will not interact with the variable stored by the first thread.
If executing the same code and using the same thread-local instance, then each thread has its own private version of a named variable and its assigned value in the thread-local storage.
This can be useful when multiple threads need to store local data such as a partial solution or a temporary variable and need to use the same code while executing, such as the same instance of an object.
Now that we know what thread-local data is, how do we use it?
How to Use Thread Local-Data
Threads can store local data via an instance of the threading.local class.
Thread-local data is data whose values are thread specific.
— Thread-Local Data, threading – Thread-based parallelism.
First, an instance of the local class must be created by calling the threading.local() function.
|
1 2 3 |
... # create a local instance local = threading.local() |
Then data can be stored on the local instance with properties of any arbitrary name.
For example:
|
1 2 3 |
... # store some data local.custom = 33 |
Importantly, other threads can use the same property names on the same instance but the values will be limited to each thread.
This is like a namespace limited to each thread and is called “thread-local data“. It means that threads cannot access or read the local data of other threads.
Importantly, each thread must be able to access the same “local” instance in order to access the stored data.
Now that we know how to use thread-local data, let’s look at some worked examples.
Free Python Threading Course
Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Example of Using Thread-Local Data
In this section we can explore how to use thread-local data with a worked example.
We will define a target task function to execute in a new thread that makes use of thread-local data.
First, we can create an instance of the thread-local structure by calling the threading.local() function.
|
1 2 3 |
... # create a local storage local = threading.local() |
Next, we can store data on it directly.
|
1 2 3 |
... # store data local.value = value |
Finally, we can block for a moment and then report the value that we stored.
|
1 2 3 4 5 |
... # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') |
Importantly, the value stored and used for the duration of the blocking call will be passed in as an argument.
Tying this together, the complete target task function is listed below.
|
1 2 3 4 5 6 7 8 9 10 |
# custom target function def task(value): # create a local storage local = threading.local() # store data local.value = value # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') |
Next, we can run the function in new threads.
We will create a new threading.Thread instance and pass in different arguments to each.
This will mean that each thread will create an instance of the thread-local data storage, store a value and report it.
We will force a gap between starting each thread so that the second thread stores its thread-local value before the first thread reports the result.
For example:
|
1 2 3 4 5 6 7 |
... # create and start a thread threading.Thread(target=task, args=(1,)).start() # wait a moment sleep(0.5) # create and start another thread threading.Thread(target=task, args=(2,)).start() |
This is a good test as if the thread-local data is overwritten by the second thread, we will see it when the first reports it’s value.
Tying this together, the complete example of using thread-local data is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of thread local storage from time import sleep import threading # custom target function def task(value): # create a local storage local = threading.local() # store data local.value = value # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') # create and start a thread threading.Thread(target=task, args=(1,)).start() # wait a moment sleep(0.5) # create and start another thread threading.Thread(target=task, args=(2,)).start() |
Running the example first creates one thread that stores the value “1” against the property “value“, then blocks for one second.
The main thread then blocks for a fraction of a second and then creates the second thread.
The second thread runs and stores the value “2” against the property “value“, then blocks for two seconds.
Each thread then reports their “value” from thread-local storage, matching the value that was stored within each thread-local context.
If the threads used the same variable for storage, then both threads would report “2” for the “value” property, which is not the case because the “value” property is unique to each thread.
|
1 2 |
Stored value: 1 Stored value: 2 |
This is not that exciting as each thread is creating an instance of the thread-local structure themselves. It could just as easily be a dict and achieve the same outcome.
What is exciting about the thread-local is that we can share the same instance across threads.
Next, let’s take a look at an example where both threads share the same thread-local instance.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Sharing a Thread-Local Instance
We can create an instance of the thread-local object and share it across multiple threads.
Importantly, each thread can store unique data in the same thread-local with the same name, and not interfere with each other.
Let’s make this clear by updating the example from the previous section to use the same thread-local instance across each thread.
First, we must update the target task function to take the thread-local instance as an argument to the function, instead of creating it.
The updated task() function is listed below.
|
1 2 3 4 5 6 7 8 |
# custom target function def task(value, local): # store data local.value = value # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') |
Next, we must create the thread-local instance in the main thread.
|
1 2 3 |
... # create a shared thread-local instance local = threading.local() |
We can then pass this thread-local instance to each thread.
|
1 2 3 4 5 6 7 |
... # create and start a thread threading.Thread(target=task, args=(1,local)).start() # wait a moment sleep(0.5) # create and start another thread threading.Thread(target=task, args=(2,local)).start() |
This is a good test of the thread local because if the threads interacted with each other then we would expect the second thread to overwrite the “value” attribute of the thread-local instance and the program to report “2” twice.
Tying this together, the complete example of sharing a thread-local instance across threads is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of thread-local storage with a shared instance from time import sleep import threading # custom target function def task(value, local): # store data local.value = value # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') # create a shared thread-local instance local = threading.local() # create and start a thread threading.Thread(target=task, args=(1,local)).start() # wait a moment sleep(0.5) # create and start another thread threading.Thread(target=task, args=(2,local)).start() |
Running the example first creates the thread-local instance and passes it to the first thread that stores it against the thread-local instance with the name “value” and sleeps for one second.
The main thread then blocks for a fraction of a second.
Next, the second thread is created and started and stores its value against the same thread-local instance with the same name “value” and then blocks for two seconds.
The first thread wakes up and reports its value of “1”, then the second thread wakes up and reports its value as “2”.
This shows that when the thread-local instance is shared across threads and each thread stores data against the same attribute that it is private to each thread.
|
1 2 |
Stored value: 1 Stored value: 2 |
Example of Global a Thread-Local Instance
In the previous example we showed how the same thread-local instance can be used across multiple threads, although it ensures that the variables are private for each thread.
We can achieve the same result by making the thread-local instance a global variable and accessing it directly within each function.
This can be achieved by explicitly defining the global variable within our target task function and then making use of it as before.
Using thread-local as per-thread private storage is a common usage pattern for the thread-local mechanism.
The updated target task function from the previous section that makes use of a global thread-local instance is listed below.
|
1 2 3 4 5 6 7 8 9 |
# custom target function def task(value): global local # store data local.value = value # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of thread-local storage with global instance from time import sleep import threading # custom target function def task(value): global local # store data local.value = value # block for a moment sleep(value) # retrieve value print(f'Stored value: {local.value}') # create a shared thread-local instance local = threading.local() # create and start a thread threading.Thread(target=task, args=(1,)).start() # wait a moment sleep(0.5) # create and start another thread threading.Thread(target=task, args=(2,)).start() |
Running the example stores the unique data for each thread using the same variable instance against the thread-local global variable.
This provides a helpful alternative to having to pass the thread-local instance to each function executed by a thread that needs access to thread-local data.
|
1 2 |
Stored value: 1 Stored value: 2 |
Common Questions About Thread-Local
This section lists some common questions about thread-local data and their answers.
Do you have a question about how to use thread-local data storage?
Ask your question in the comments below and I may add it to this section.
When to Use Thread-Local Data?
Thread-local data is useful when each thread must store distinct data that should not be accessible by other threads.
Examples might include:
- A partial solution to a broader problem.
- Temporary variables.
- Handles external resources.
A thread-local storage mechanism allows each thread to:
- Execute the same code.
- Use the same variable names.
- Share the same thread-local instance.
This is achieved while ensuring the data stored by a thread can only be accessed by that thread, making it private at the thread level, hence the name “thread-local”.
When to Not Use Thread-Local Data?
Thread-local data should not be used to share data between threads.
This is because the data variables are private to each thread and cannot be accessed.
What is an Alternative to Thread-Local Data?
An alternative to thread-local data storage is to extend the threading.Thread class and have the thread-specific data stored as instance variables.
This requires more code and may mean that the data can be accessed by other threads as attributes of the thread object.
Why Not Use a Dict?
A container like a dict can be shared between threads and used to store data, but could not achieve the same result as a thread-local data instance.
The problem is that if each thread used the same variable name (e.g. key in the dict) to store data, then the threads would interact, resulting in race-conditions and overwritten data.
Do We Have To Share The Thread-Local Instance?
No.
But if the same instance of the thread-local is not used by each thread, then it may not be an appropriate solution. You could just as easily use a local data structure.
What Are Common Patterns for Thread-Local?
There are many programming patterns that make use of the thread-local data structure.
Two common patterns you may wish to use include:
- Global Thread-Local: Create a thread-local as a global variable and use it across functions, where the functions are executed by multiple threads.
- Static Thread-Local: Create a thread local as a class variable (e.g. static) and use it across multiple methods and multiple instances of the same object executed by multiple threads.
Further Reading
This section provides additional resources that you may find helpful.
Python Threading Books
- Python Threading Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- Threading Module API Cheat Sheet
I also recommend specific chapters in the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Threading: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Takeaways
You now know how to use thread-local storage in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Harley-Davidson on Unsplash
You don’t need
global localintaskAgreed, but I think it makes the code easier to read and understand and avoid possible conflicts with locals with the same name in the future.
Thank you Jason. This article helped me to solve a issue elegantly
You’re welcome.
I’m very happy to hear that it helped!
Hi Jason, and thank you for the detailed explanations. I have a question though: is it possible for thread A to update the local.value of the main thread instead of its own local.value?
The reason I want to do that is that I have a library which initialization needs to set the local.value of the main thread. My current solution is to force client code to do the first import of the library in the main thread, but I’d like to free client code from this constraint?
What are your thoughts about this?
You’re welcome!
No, one thread cannot directly update the thread local storage of another thread. It is private. It can request that the thread update its thread local data, e.g. set a variable that the other thread checks and responds to.
Maybe you can have the main thread pull the data rather than the new thread push the data to the main thread?
Thank you for your help! I was finally able to solve it at the cost of potential other threads checking once that they are not the main thread until the main thread has a chance to set the value. I could free client code from the constraint at first import.
Nice work!
In the global thread local instance example, thread local object is created only once and all other threads are trying to modify same field (value) of the same object(local object) by accessing through global keyword. What happens if 2 threads simultaneously tries to modify the value inside local object, in that case whether one of the threads get wrong value
If threads modify the same variable (name) within the same local object, they will access different data. There will be no race condition.
The reason is because each thread has its own unique and private version of the variable. Thread local storage is a facilitate provided by the operating system that in turn is tapped into via the Python threading API.
Does that help? Or have I misunderstood the question?