Last Updated on September 12, 2022

You can write to file in a thread-safe manner using a mutex lock via the threading.Lock class.

In this tutorial you will discover how to write thread-safe to a file from many threads.

Let’s get started.

Need to Append File From Multiple Threads

A thread is a thread of execution in a computer program.

Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create additional threads in our program in order to execute code concurrently.

Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.

You can learn more about Python threads in the guide:

• Threading in Python: The Complete Guide

In concurrent programming, we may need to write to a file from multiple threads.

Is writing to a file thread-safe? If not, how can we make appending a file from multiple threads thread-safe?

Writing to File is Not Thread-Safe

Writing to a file means opening a file path and writing or appending data to the file.

A file can be opened with the built-in open() function specifying the file path and open mode such as ‘w‘ for writing text or ‘a‘ for appending text. Once open, data can be written via the write() or writelines() functions.

A best practice is to open a file using the context manager interface so that the file is closed automatically once any writing operations are finished.

For example:

Writing to the same file from multiple threads concurrently is not thread safe and may result in a race condition.

Thread-safe means that writing or appending to the same file from more than one thread may result in a race condition.

The result of a race condition while writing or appending to a file may be that the data written to the file is overwritten or corrupted, resulting in data loss.

Now that we know that writing to a file is not thread-safe, let’s look at how we can make it thread-safe.

How to Thread-Safe Write to File

Writing to a file can be made thread-safe by using a mutual exclusion (mutex) lock.

Any code that opens and writes to the file, or appends to the file can be treated as a critical section of code subject to race conditions.

This code can be protected from race conditions by requiring that the accessing thread first acquire a mutex lock before executing the critical section.

A mutex lock can only be acquired by one thread at a time, and once acquired prevents any other thread acquiring it until the lock is released.

This means that only a single thread will be able to write to the file at a time, making writing to file thread safe.

This can be achieved using the threading.Lock class.

First, a lock can be created and shared among all code that needs to access the same file.

Once created, a thread can then acquire the lock before writing to file by calling the acquire() function. Once writing to the file is finished, the lock can be released by calling the release() function.

For example:

Like opening the file, the context manager interface can be used to ensure that the lock is always released after the block enclosed code is exited.

For example:

If a thread attempts to acquire the lock while it has already been acquired, then it must block or wait until the lock is released. This waiting is performed automatically by the thread when attempting to acquire the lock, no additional code is required.

You can learn more about protecting critical sections with mutex locks in this tutorial:

• How to Use a Mutex Lock in Python

Next, let’s look at some worked examples.

Example of Thread-Unsafe Write to File

Before we look at how to make writing to a file thread-safe, let’s look at how it is not thread-safe.

We can develop an example that demonstrates that appending to a file from many threads is not thread safe and results in data loss.

In this example we will create 1,000 threads each of which will generate 1,000 random numbers between 0 and 1 and write them as a line of text to the same file.

There are a few ways to structure this program, for example:

• Each thread opens the file each time a line of text needs to be written.
• Each thread opens the file once and writes each line as generated.
• The application opens the file once and each thread writes using the same file handle.

The specific approach taken really depends on the details of the application.

In this case, we will open the file once in the application and have all threads write to the file using the same file handle. This is really aggressive and it is easy to show the race condition when many threads write to the same file.

First, we can define a task function to be executed by each thread.

The function will take a unique integer to identify the thread and the file handle as arguments. The task will loop for 1,000 iterations and each iteration it will generate a random number using random.random() and then write this number as part of a line of text to the file.

The task() function below implements this.

Next, in the main thread, we can open the file for appending.

Note, we will write to a file named “output.txt” in the current working directory. This is the same directory in which your Python script for the program is located.

We can then create 1,000 new threads configured to call the task() function each with a unique integer identifier and the shared file handle.

This can be done in a list comprehension.

The main thread can then start all threads by calling the start() method and then wait until all threads have terminated by calling the join() method.

Finally, the main thread will close the file.

Tying this together, the complete example is listed below.

Running the example first opens the file for appending.

Next, 1,000 threads are created and started, then the main thread blocks until all tasks are complete.

Each thread loops and writes 1,000 lines to the file.

All tasks complete and the file is closed.

The expectation is that the file contains 1,000,000 lines, with 1,000 lines written by each thread.

We can check this by opening the file in a text editor and inspect the contents.

Type the following command on a POSIX operating system (e.g. Linux or MacOS) to report the total lines in the file.

In this case, we can see that the file contains fewer than the expected number of lines.

Data was lost. This highlights that having multiple threads appending to the same file (using the same file handle) is not thread safe.

Delete the file and run the program again.

You will likely see a different number of lines each time the code is run, due to the unpredictable nature of the race condition.

Next, let’s look at how we can make writing to the file thread-safe.

Example of Thread-Safe Write to File

We can write or append a file in a thread-safe manner using a mutex lock.

The example in the previous section can be updated to be thread-safe by requiring that each thread first acquires a lock prior to writing to the file.

This can be achieved by passing the lock to the task function as an argument and then acquiring the lock prior to each call to write() on the shared file handle.

The updated version of the task() function with these changes is listed below.

In the main thread, we can then create the shared lock instance.

The lock can then be passed to each thread as an argument to the task() function.

Tying this together, the complete example of thread-safe writing to a file is listed below.

Running the example first creates the shared lock and the shared file handle.

The new threads are created and configured to execute the task() function, receiving the shared file handle and lock as arguments.

The main thread starts the threads and then waits for the tasks to complete.

Each thread loops 1,000 times, each iteration generating a random number, acquiring the lock, and writing a line to the file. If the lock is already acquired, then the thread will wait until the lock is available.

This adds some delay to the program, increasing the run time.

Open the output.txt file in a text file and inspect the contents.

Review the total number of entries in the file.

There will be 1,000,000 lines in the file. The race condition has been avoided as each thread now writes to the same file in a thread-safe manner.

Importantly, 1,000,000 lines are written to the file every time the program is run, making it consistent.

Note, the random numbers written to file will differ each time the program is run.

Acquiring the lock is a bottleneck in the code, increasing the overall running time of the program.

An alternate approach would be for each thread to acquire the lock at the beginning of the task, then generate and write all random numbers.

This would acquire the lock be acquired 1,000 times, once for each thread, rather than 1,000,000 times, once for each write to the file.

For example, an updated version of the task() function with this change is listed below.

Next, let’s look at how we might write safely to a file using a dedicated thread.

Example of Safe Dedicated File Writing Thread

Another approach to writing to a file in a thread-safe manner is to use a dedicated file writing thread.

If only a single thread is responsible for writing to the file, then all file writing will be thread-safe.

This can be achieved by creating a new thread that loops for the duration of the program reading lines from other threads and writing them to the file. Worker threads can communicate to the file writer thread using a thread-safe queue data structure. Workers will put messages on the queue and the file writer thread will get messages from the queue and write them to file.

First, we can write a function to be executed by the single file writer thread.

The function will take the path to the file and the queue instance as arguments.

First, the file will be opened using the context manager. Then the task will loop forever. Each iteration it will retrieve a line of text from the queue by calling get(), write it to file, flush the contents of the file buffer (writing text is buffered), then mark the task as done.

Flushing content after each write is computationally expensive, but ensures that if the file is closed, such as by the Python process, that no content will be waiting to be written.

The file_writer() function that implements this is listed below.

Next, we can update the task() function executed by the worker threads.

The function must receive the queue as an argument, and then add lines to the queue each iteration by calling the put() function.

The updated task() function with these changes is listed below.

Next, in the main thread, we can create the shared queue.

This will be an instance of the queue.Queue class. This is a thread-safe implementation of the queue data structure that allows multiple threads to call get() and put() without race conditions.

Importantly, the get() function will block until new work arrives on the queue. This is helpful, as it means that our file writer thread will wait patiently and not consume resources while waiting for lines of text to print.

Next, we can create and start a new thread to execute our file_writer() function.

Importantly, this new thread will be marked as a daemon thread, which is a background task. This means that the Python process can exit while this thread is still running.

We can then create and configure the worker threads, passing the instance of the shared queue.

Finally, after the worker threads have terminated, the main thread can wait on the queue for all submitted tasks to be processed.

This can be achieved by calling the join() function.

Each call to put() by worker threads will increment a counter within the queue and each subsequent call to task_done() by the printer thread will decrement the counter, indicating that the unit of worker was processed completely.

After the worker threads have terminated, there are no more lines to add to the queue. Therefore, once the queue’s internal counter reaches zero, we know that all work has been processed.

The join() function will block until all submitted work on the queue has been processed.

Tying this together, the complete example of writing to file in thread-safe manner using a dedicated thread and a queue is listed below.

Running the example first creates the shared queue.

Next, the file writer thread is created and configured to execute our file_writer() function as a background daemon thread. It is started and runs forever, reading items from the shared queue and writing them to file.

The worker threads are started and passed an instance of the shared queue. Each thread iterates 1,000 times, generating numbers and writing lines to the shared queue.

The main thread waits for all worker threads to complete. It then blocks on the shared queue until all submitted lines have been written to file.

The main thread terminates, which then terminates the background daemon thread, forcefully closing the open file.

Open the output.txt file in a text editor and review the contents

Inspect the total number of lines in the file.

We can see that it has the expected 1,000,000 lines. Importantly, it will have the same number of lines each time the program is run as there is no race condition in writing to file.

The way we handle closing the file is a little rough.

It is better to actually close the file as part of normal operation of the program.

This can be achieved by updating the file_writer() function. We can check for a special message, and if received, mark it as processed and return from the function, which will terminate the file writer thread. The value None is an object that we can use as the special message.

For example:

After the loop is exited, the file is closed and we can mark the final message put on the queue as processed.

The updated file_writer() function with these changes is listed below.

In the main thread, once all worker threads are finished, we can then signal that no more messages are expected by calling put() on the queue and passing the value None.

This will signal to the file writer thread that no further messages are expected and to terminate.

The main thread can then join the queue and wait for the file writer thread to process all messages, including the signal to terminate.

Tying this together, the complete example with these improvements is listed below.

Running the example produces the same thread-safe writing to the file.

Importantly, we are confident that the file is closed correctly as part of the normal execution of the program.

Further Reading

This section provides additional resources that you may find helpful.

Python Threading Books

• Python Threading Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• Threading Module API Cheat Sheet

I also recommend specific chapters in the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Threading: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• threading - Thread-based parallelism
• queue — A synchronized queue class

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to write to a file from multiple threads in a thread-safe manner.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Thanuj Mathew on Unsplash