ThreadPool How to Log From Worker Processes
We can log from tasks directly in the ThreadPool by calling a function on the logging module.
In this tutorial, you will discover how to log from tasks in the Python ThreadPool.
Let's get started.
Need to Log From Tasks in the ThreadPool
The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.
A thread pool object which controls a pool of worker threads to which jobs can be submitted.
-- multiprocessing — Process-based parallelism
The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.
Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.
A ThreadPool can be configured when it is created, which will prepare the new threads.
We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().
Tasks often involve interacting with resources and can raise exceptions.
As such, we may wish to log events like warnings, errors, debug, or info messages from within tasks executed by the ThreadPool.
If we are logging to a file, database, or to the console, we may be concerned as to whether the Python logging supports calls from multiple threads within an application.
That is, is logging from tasks thread-safe?
Call Logging Directly From Tasks
You can log directly from target task functions when using the ThreadPool.
This is because the Python logging module is thread-safe.
It is a common question as to whether logging from multiple threads in Python is threads-safe. As such, it is addressed directly in the logging module API documentation; for example:
The logging module is intended to be thread-safe without any special work needing to be done by its clients. It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O.
-- Thread Safety, logging - Logging facility for Python.
We can log directly from target task functions by calling functions on the logger module at the desired logging level.
For example, we can log an exception as an error:
...
# log an error message
logging.error('Something bad happened')We might also want to log debug messages, such as whether tasks were completed successfully and the time they were completed.
...
# log a debug message
logging.debug(f'Task completed at {datetime.datetime.now()}')Now that we know how to log from tasks executed by the ThreadPool, let's look at a worked example.
How to Log From Tasks in the ThreadPool
Let's develop an example that logs directly from target task functions in the ThreadPool.
We can define a task that has a unique name received as an argument to the target task function.
The main work for the function is to sleep for a fraction of a second. Once the task is completed, we can log a debug message to indicate that the task has been completed. Then return the value for the task.
Tying this together, the task() function below implements our task with debug logging.
# custom task that will sleep for a moment
def task(number):
# block for a moment
sleep(0.5)
# log the status of the task
logging.debug(f'Successfully completed task: {number}')
By default, Python logging will report log messages to stdout (standard out, i.e. the console), which is fine for our purposes.
In our program, we can set the logging level to debug before we get started.
...
# set the logging level to info
logging.basicConfig(level=logging.DEBUG)We can then configure and start a ThreadPool with the default number of worker threads using the context manager.
...
# start the thread pool
with ThreadPool() as pool:
# ...
You can learn more about the ThreadPool context manager interface in the tutorial:
We will then issue ten tasks asynchronously via the map_async() function which returns immediately with an AsyncResult object. We then wait on this object for all issued tasks to be completed.
...
# issue tasks to the thread pool
async_result = pool.map_async(task, range(10))
# wait for tasks to complete
async_result.wait()You can learn more about the map_async() method in the tutorial:
Tying this together, the complete example of logging directly from tasks in the ThreadPool is listed below.
# SuperFastPython.com
# example of logging from tasks in the thread pool
import logging
from time import sleep
from multiprocessing.pool import ThreadPool
# custom task that will sleep for a moment
def task(number):
# block for a moment
sleep(0.5)
# log the status of the task
logging.debug(f'Successfully completed task: {number}')
# protect the entry point
if __name__ == '__main__':
# set the logging level to info
logging.basicConfig(level=logging.DEBUG)
# start the thread pool
with ThreadPool() as pool:
# issue tasks to the thread pool
async_result = pool.map_async(task, range(10))
# wait for tasks to complete
async_result.wait()
Running the example starts the ThreadPool and issues all ten tasks.
We can then see that as the tasks are completed, they log debug messages that are then reported on the command line.
DEBUG:root:Successfully completed task: 1
DEBUG:root:Successfully completed task: 0
DEBUG:root:Successfully completed task: 2
DEBUG:root:Successfully completed task: 5
DEBUG:root:Successfully completed task: 6
DEBUG:root:Successfully completed task: 7
DEBUG:root:Successfully completed task: 3
DEBUG:root:Successfully completed task: 4
DEBUG:root:Successfully completed task: 8
DEBUG:root:Successfully completed task: 9
Takeaways
You now know how to log directly from tasks executed by the ThreadPool.
If you enjoyed this tutorial, you will love my book: Python ThreadPool Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.