How to Log From Tasks in the ThreadPoolExecutor in Python

You can log from tasks in the ThreadPoolExecutor by calling a function on the logging module.

In this tutorial, you will discover how to log from tasks in the Python thread pool.

Let's get started.

Log From Tasks in the ThreadPoolExecutor

The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.

You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread.

Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.

You can also submit tasks by calling the map() function and specifying the name of the function to execute and the iterable of items to which your function will be applied.

Tasks often involve interacting with resources and can raise exceptions.

As such, we may wish to log events like warnings, errors, debug, or info messages from within tasks executed by the ThreadPoolExecutor.

If we are logging to a file, database or to the console, we may be concerned as to whether the Python logging supports calls from multiple threads within an application.

That is, is logging from tasks thread safe?

How to Log Directly From Tasks

You can log directly from target task functions when using the ThreadPoolExecutor.

This is because the Python logging module is thread safe.

It is a common question as to whether logging from multiple threads in Python is threads-safe. As such, it is addressed directly in the logging module API documentation; for example:

The logging module is intended to be thread-safe without any special work needing to be done by its clients. It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O.

-- Thread Safety, logging - Logging facility for Python.

We can log directly from target task functions by calling functions on the logging module at the desired logging level.

For example, we can log an exception as an error:

...
# log an error message
logging.error('Something bad happened')

We might also want to log debug messages, such as whether tasks completed successfully and the time they completed.

...
# log a debug message
logging.debug(f'Task completed at {datetime.datetime.now()}')

Now that we know how to log from tasks executed by the ThreadPoolExecutor, let's look at a worked example.

Example of Logging From Tasks in the ThreadPoolExecutor

Let's develop an example that logs directly from target task functions in the ThreadPoolExecutor.

We can define a task that has a unique name received as an argument to the target task function.

The main work for the function is to sleep for a fraction of a second. Once the task is completed, we can log a debug message to indicate that the task has completed. Then return the value for the task.

Tying this together, the task() function below implements our task with debug logging.

# custom task that will sleep for a moment
def task(name):
    # sleep for a moment
    sleep(0.5)
    # log the status of the task
    logging.debug(f'Successfully completed task: {name}')
    return f'Task Done: {name}'

By default, Python logging will report log messages to stdout (standard out, i.e. the console), which is fine for our purposes.

In our program, we can set the logging level to debug before we get started.

...
# set the logging level to info
logging.basicConfig(level=logging.DEBUG)

We can then start a thread pool with ten worker threads using the context manager and submit ten tasks to the pool for execution.

We can then wait for all tasks to complete.

...
# start the thread pool
with ThreadPoolExecutor(10) as executor:
    # submit tasks and collect futures
    futures = [executor.submit(task, i) for i in range(10)]
    # wait for all tasks to complete
    print('Waiting for tasks to complete')
    wait(futures)

Tying this together, the complete example of logging directly from tasks in the ThreadPoolExecutor is listed below.

# SuperFastPython.com
# example of logging from tasks in the thread pool
import logging
from time import sleep
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import wait

# custom task that will sleep for a moment
def task(name):
    # sleep for a moment
    sleep(0.5)
    # log the status of the task
    logging.debug(f'Successfully completed task: {name}')
    return f'Task Done: {name}'

# set the logging level to info
logging.basicConfig(level=logging.DEBUG)
# start the thread pool
with ThreadPoolExecutor(10) as executor:
    # submit tasks and collect futures
    futures = [executor.submit(task, i) for i in range(10)]
    # wait for all tasks to complete
    print('Waiting for tasks to complete')
    wait(futures)

Running the example starts the thread pool and submits all ten tasks.

We can then see that as the tasks complete, they log debug messages that are then reported on the command line.

Waiting for tasks to complete
DEBUG:root:Successfully completed task: 0
DEBUG:root:Successfully completed task: 1
DEBUG:root:Successfully completed task: 2
DEBUG:root:Successfully completed task: 3
DEBUG:root:Successfully completed task: 4
DEBUG:root:Successfully completed task: 5
DEBUG:root:Successfully completed task: 6
DEBUG:root:Successfully completed task: 7
DEBUG:root:Successfully completed task: 8
DEBUG:root:Successfully completed task: 9

Takeaways

You now know how to log directly from tasks executed by the ThreadPoolExecutor.