Last Updated on September 12, 2022

You can log directly from multiple threads because the logging module is thread-safe.

In this tutorial you will discover how to log safely from many threads.

Let’s get started.

Need to Log From Multiple Threads

A thread is a thread of execution in a computer program.

Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create additional threads in our program in order to execute code concurrently.

Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.

You can learn more about Python threads in the guide:

• Threading in Python: The Complete Guide

In concurrent programming, we may need to log from multiple threads in the application.

This may be for many reasons, such as:

• Different threads may perform different tasks and may encounter a range of events.
• Worker threads may encounter exceptions that need to be stored, but should not halt the program.
• Background tasks may need to report progress over the duration of the program.

What is logging, and is it safe to use from multiple threads?

What is Logging

Logging is a way of tracking events within a program.

There are many types of events that may be logged within a program, ranging in different levels of severity, such as debugging and information to warnings, errors and critical events.

The logging module provides infrastructure for logging within Python programs.

This module defines functions and classes which implement a flexible event logging system for applications and libraries.

— logging — Logging facility for Python

Logging is achieved by first configuring the log handler and then adding calls to the logging infrastructure at key points in the program.

Handler objects are responsible for dispatching the appropriate log messages (based on the log messages’ severity) to the handler’s specified destination.

— Handles, Logging HOWTO

The default handler will report log messages on the command prompt (e.g. terminal or system output stream).

Alternate handlers can be configured to write log messages to file, to a database, or to custom target locations. The handler can specify the severity of messages to store.

For example, we can log to file by calling the logging.basicConfig() function and specifying the file name and path to log to (e.g. application.log), the level of logging to capture to the file (e.g. from logging.DEBUG to logging.CRITICAL).

Events are logged via function calls based on the type of event performed, e.g. logging.debug() for debugging messages.

Logging provides a set of convenience functions for simple logging usage. These are debug(), info(), warning(), error() and critical().

— Logging HOWTO

For example, we may add information messages to application code by calling logging.info() and passing in the string details of the event.

We may also log failures, such as exceptions that are important but not critical to the program via the logger.error() or logger.exception() functions.

For example:

These function calls can then be added throughout an application to capture events of different types, and the log handler or runtime configuration can be used to filter the severity of the messages that are actually stored.

This provides a quick crash-course to Python logging, you can learn more here:

• logging — Logging facility for Python
• Logging HOWTO

Next, let’s consider logging from multiple threads.

How to Log From Multiple Threads

The logging module can be used directly from multiple threads.

The reason is because the logging module is thread-safe.

The logging module is intended to be thread-safe without any special work needing to be done by its clients.

— Thread Safety, logging — Logging facility for Python.

This is achieved using locks.

Internally, the logging module uses mutual exclusion (mutex) locks to ensure that logging handlers are protected from race conditions from multiple threads.

For example, locks are used to ensure that only one thread writes to a log file or log stream at a time. This ensures that log messages are serialized and are not corrupted.

It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O.

— Thread Safety, logging — Logging facility for Python.

Therefore, once the logging module is configured for your process, you can log directly from threads.

Next, let’s look at some examples of logging from multiple threads.

Example of Thread-Safe Logging

We can explore how to log safely from multiple threads.

In this example, we will create five worker threads, each of which will execute a task. The task will loop five times and each iteration it will generate a random number between 0 and 1, block for that many seconds, then if the number is below a threshold, the thread will stop.

We will log whether the task was stopped as a warning message and we will log whether the task was completed successfully as an information message.

All threads will log the information message and some subset of threads will report a warning message.

Firstly, we will define a function to execute in each worker thread.

The task will take a unique integer to represent the thread, and a threshold value as arguments. The task will loop five times, generate a random number, block with a call to time.sleep() and check the value of the number to see if the task should stop.

The task() function below implements this, with warning and info level log messages.

Next, in the main thread, will first configure the logger.

First, will acquire an instance of the root logger and then set the level of messages to log to be debug or higher, essentially all messages.

We will then create five threads configured to execute our task function, each passed a unique integer and a threshold value of 0.1. This is performed in a list comprehension.

We then start all worker threads, and the main thread will block until all tasks are complete.

Tying this together, the complete example of logging from multiple threads is listed below.

Running the example first configures the log to report all messages.

The default logger reports messages to the console (e.g. standard out).

The five threads are then created and started and the main thread blocks until the tasks are complete.

Each worker thread then iterates the task loop, generating a random number, blocking and then checking if the generated number was below the threshold. If below the threshold, a warning message is reported and the task loop is broken.

In this case, we can see that three of the tasks generated random numbers below the threshold and broke the task.

Note, your specific results will differ each time the program is run given the use of random numbers.

This demonstrates how we can directly log from multiple threads in a thread-safe manner.

If logging was not thread-safe, then the log messages would not be serialized correctly, possibly overlapping and corrupting as multiple threads attempted to write to the standard output stream concurrently.

Next, let’s look at how we might log to a file from multiple threads.

Example of Thread-Safe Logging to File

We can explore how multiple threads may log to file in a thread-safe manner.

This can be achieved by configuring the logger to store messages at or above the desired level to a filename and path.

For example, we can update the above example to log all messages at debug level or above to a file in the same directory as the Python program named ‘application.log‘.

This can be achieved with a single line that configures the logger with a filename and level.

Because we are logging to a file, perhaps we can capture more information for later review.

In this case, we will update the task() function to add an information message for the thread starting the task, and a debug message reporting the specific value generated each iteration.

The updated version of the task() function with these changes is listed below.

Tying this together, the complete example of logging to file from multiple threads is listed below.

Running the example first configures the root logger for the process to log all messages at debug level or above to a file named ‘application.log‘.

Note, this file will be located in the same directory as your Python script, e.g. your current working directory.

The five threads are then configured and started, then the main thread blocks until the tasks are complete.

Each thread then executes the task function, logging information messages at the beginning and end of the task, as well as debug messages for each number generated and a warning if a value below the threshold is generated.

In this case, only two tasks failed, although now we can see a log of all values generated by all threads.

A sample of the application.log file is provided below.

Note, your specific results will differ each time the program is run given the use of random numbers.

This demonstrates how we can log to file from multiple threads in a thread-safe manner.

Next, let’s look at how we might automatically include thread information in log messages.

Example of Automatically Logging Thread Details

We can automatically include details about threads in the log messages.

This can be achieved by changing the format of log messages to include attributes such as the thread name or the thread identifier.

In the previous examples, we manually assigned each thread a unique integer which was used in log messages to differentiate each thread. We can remove this unique integer and include each thread’s default and uniquely assigned name in the log messages.

You can learn more about thread names in this tutorial:

• How to Change the Thread Name in Python

Firstly, we can create a new logging.FileHandler instance, configured to log all messages at debug level or higher to a file named ‘application2.log‘.

We can then create a new instance of a logging.Formatter instance specifying the attributes to include in each message. We will include the same attributes as in prior messages with the addition of the thread name.

You can see a list of all attributes that you could include in log messages here:

• LogRecord attributes

Finally, we can configure the logging infrastructure to use our log handler and to log all messages at debug level or above to the handler.

We can then update the task() function to no longer take a unique integer for the thread and to remove it from all log messages.

Finally, we can update the configuration of the new threads to no longer pass each thread a unique integer.

Tying this together, the complete example of including thread name in log messages is listed below.

Running the example first configures the logging infrastructure to log all messages at debug level or above to a file named application2.log located in the current working directory.

The threads are then configured and started.

Each thread performs the task, reporting information messages at the beginning and end, debug messages each iteration and a possible warning message if the threshold is crossed.

Below is a sample of the content of the log file.

We can see that each message includes the name of the thread executing the task, taken from the “name” attribute on the threading.Thread instance, assigned automatically when the thread was created.

Note, your specific results will differ each time the program is run given the use of random numbers.

Further Reading

This section provides additional resources that you may find helpful.

Python Threading Books

• Python Threading Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• Threading Module API Cheat Sheet

I also recommend specific chapters in the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Threading: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• threading - Thread-based parallelism
• queue — A synchronized queue class

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to log safely from multiple threads.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Arrul lin on Unsplash