Last Updated on September 12, 2022
You can log from multiple processes directly using the log module or safely using a custom log handler.
In this tutorial you will discover how to log from multiple processes in Python.
Let’s get started.
Need to Log From Multiple Processes
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In multiprocessing, we may need to log from multiple processes in the application.
This may be for many reasons, such as:
- Different processes may perform different tasks and may encounter a range of events.
- Worker processes may encounter exceptions that need to be stored, but should not halt the program.
- Background tasks may need to report progress over the duration of the program.
What is logging, and is it safe to use from multiple processes?
Run loops using all CPUs, download your FREE book to learn how.
What is Logging
Logging is a way of tracking events within a program.
There are many types of events that may be logged within a program, ranging in different levels of severity, such as debugging and information to warnings, errors and critical events.
The logging module provides infrastructure for logging within Python programs.
This module defines functions and classes which implement a flexible event logging system for applications and libraries.
— logging — Logging facility for Python
Logging is achieved by first configuring the log handler and then adding calls to the logging infrastructure at key points in the program.
Handler objects are responsible for dispatching the appropriate log messages (based on the log messages’ severity) to the handler’s specified destination.
— logging — Logging facility for Python
The default handler will report log messages on the command prompt (e.g. terminal or system output stream).
Alternate handlers can be configured to write log messages to file, to a database, or to custom target locations. The handler can specify the severity of messages to store.
For example, we can log to file by calling the logging.basicConfig() function and specifying the file name and path to log to (e.g. application.log), the level of logging to capture to the file (e.g. from logging.DEBUG to logging.CRITICAL).
|
1 2 3 |
... # log everything to file logging.basicConfig(filename='application.log', level=logging.DEBUG) |
Events are logged via function calls based on the type of event performed, e.g. logging.debug() for debugging messages.
Logging provides a set of convenience functions for simple logging usage. These are debug(), info(), warning(), error() and critical().
— Logging HOWTO
For example, we may add information messages to application code by calling logging.info() and passing in the string details of the event.
|
1 2 3 |
... # log that data was loaded successfully logger.info('All data was loaded successfully') |
We may also log failures, such as exceptions that are important but not critical to the program via the logger.error() or logger.exception() functions.
For example:
|
1 2 3 4 5 6 |
... # try a thing try: # ... except Exception as err: logger.exception(f'Unable to perform task because: {err.message} ') |
These function calls can then be added throughout an application to capture events of different types, and the log handler or runtime configuration can be used to filter the severity of the messages that are actually stored.
Next, let’s consider logging from multiple processes.
How to Log From Multiple Processes
There are perhaps three main approaches to logging from multiple processes, they are:
- Use the logging module separately from each process.
- Use multiprocessing.get_logger().
- Use custom process-safe logging.
Let’s take a closer look at each approach in turn.
1. Use Logging Module Separately in Each Process
We can log directly from each process using the logging module.
This requires that you configure the logger in each process, including the target (e.g. stream or file) and log level.
For example:
|
1 2 3 4 5 |
... # create a logger logger = logging.getLogger() # log all messages, debug and up logger.setLevel(logging.DEBUG) |
Messages can then be logged directly.
For example:
|
1 2 3 |
... # report a message logging.info('Hello world.') |
This is an easy approach in that it does not require anything sophisticated.
The downside is that you have to duplicate code in order to configure a separate logger for each new process.
The major limitation of this approach is that log messages may be lost or corrupted. This is because multiple processes will attempt to write log messages to the same target, e.g. stream or file.
2. Use Multiprocessing Module Logger (not recommended)
The multiprocessing module has its own logger with the name “multiprocessing“.
This logger is used within objects and functions within the multiprocessing module to log messages, such as debug messages that processes are running or have shutdown.
We can get this logger and use it for logging.
The logger used by the multiprocessing module can be acquired via the multiprocessing.get_logger() function. Once acquired, it can be configured, e.g. to specify a handler and a log level.
Returns the logger used by multiprocessing. If necessary, a new one will be created.
— multiprocessing — Process-based parallelism
For example:
|
1 2 3 4 5 6 7 |
... # get the multiprocessing logger logger = get_logger() # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) |
Messages logged by the multiprocessing module itself will appear in this log at the appropriate log level, e.g. there are many debug messages in the module.
We can also get a logger for the multiprocessing module that will log to standard error (stderr) by calling the multiprocessing.log_to_stderr() function. This will log module messages to stderr.
We can then set the log level.
For example:
|
1 2 3 4 5 |
... # get the multiprocessing logger with stderr stream handler added logger = multiprocessing.log_to_stderr() # log all messages, debug and up logger.setLevel(logging.DEBUG) |
We can also set the level as an argument to the multiprocessing.log_to_stderr() function via the “level” argument.
For example:
|
1 2 3 |
... # get the multiprocessing logger with stderr stream handler added logger = multiprocessing.log_to_stderr(level=logging.DEBUG) |
This logger is not shared among processes and is not process-safe.
It is simply using the logger module directly as in the previous example, although for the “multiprocessing” logging namespace.
As such it has the same downside that the logger must be configured again within each child process and that log messages may be lost or corrupted.
3. Use Custom Process-Safe Logging (recommended)
Effective logging from multiple processes requires custom code.
The logging module cookbook provides examples on how this can be achieved, in the section:
A few approaches are described, including:
- Use a socket handler and send messages to a logging process.
- Use a custom handler that internally uses a shared mutex to ensure one process logs at a time.
- Use a queue handler that uses a shared queue to send messages to a logging process.
All approaches are similar in that they require that log messages are serialized before being stored.
This can be achieved by serializing messages on a shared multiprocessing.Queue via a logging.handlers.QueueHandler and letting one process read messages one at a time and store them to stream or file.
Alternatively, it can be achieved by allowing each process to write directly to the same location, but to use a shared lock to ensure that access to the target resource, such as a stream or file, is mutually exclusive.
The approaches are generally equivalent and a matter of taste and application requirements.
Now that we know how to log from multiple processes, let’s look at some worked examples.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example Using the Logging Module
We can explore using the logging module directly from multiple processes.
This is appropriate for simple Python programs that may log non-critical messages to stderr.
In this example we will create a number of child processes to execute a custom function. A logger will be configured in the main process and in each child process to report all messages debug and higher to stderr.
Firstly, we can prepare the function to run in child functions.
|
1 2 3 |
# task to be executed in child processes def task(): # ... |
The function must first configure a log instance with the appropriate level.
|
1 2 3 4 5 |
... # create a logger logger = logging.getLogger() # log all messages, debug and up logger.setLevel(logging.DEBUG) |
We can then get the current process so that we can include the process name in the messages.
|
1 2 3 |
... # get the current process process = current_process() |
The child process can then log a message, loop a few times and report a message each iteration and block for a random fraction of a second, then log a message before exiting.
|
1 2 3 4 5 6 7 8 9 10 11 |
... # report initial message logging.info(f'Child {process.name} starting.') # simulate doing work for i in range(5): # report a message logging.debug(f'Child {process.name} step {i}.') # block # sleep(random()) # report final message logging.info(f'Child {process.name} done.') |
Tying this together, the complete task() function is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# task to be executed in child processes def task(): # create a logger logger = logging.getLogger() # log all messages, debug and up logger.setLevel(logging.DEBUG) # get the current process process = current_process() # report initial message logging.info(f'Child {process.name} starting.') # simulate doing work for i in range(5): # report a message logging.debug(f'Child {process.name} step {i}.') # block sleep(random()) # report final message logging.info(f'Child {process.name} done.') |
Next, in the main process, we can configure a logger for logging from this process.
|
1 2 3 4 5 |
... # create a logger logger = logging.getLogger() # log all messages, debug and up logger.setLevel(logging.DEBUG) |
The main process will then report a message that it is getting started, then configure and start five processes to execute our custom task() function, wait for the child processes to finish, then report a final message.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
... # report initial message logging.info('Main process started.') # configure child processes processes = [Process(target=task) for i in range(5)] # start child processes for process in processes: process.start() # wait for child processes to finish for process in processes: process.join() # report final message logging.info('Main process done.') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
# SuperFastPython.com # example of logging from multiple processes with the logging module from random import random from time import sleep from multiprocessing import current_process from multiprocessing import Process import logging # task to be executed in child processes def task(): # create a logger logger = logging.getLogger() # log all messages, debug and up logger.setLevel(logging.DEBUG) # get the current process process = current_process() # report initial message logging.info(f'Child {process.name} starting.') # simulate doing work for i in range(5): # report a message logging.debug(f'Child {process.name} step {i}.') # block # sleep(random()) # report final message logging.info(f'Child {process.name} done.') # protect the entry point if __name__ == '__main__': # create a logger logger = logging.getLogger() # log all messages, debug and up logger.setLevel(logging.DEBUG) # report initial message logging.info('Main process started.') # configure child processes processes = [Process(target=task) for i in range(5)] # start child processes for process in processes: process.start() # wait for child processes to finish for process in processes: process.join() # report final message logging.info('Main process done.') |
Running the example first configures the logger in the main process, then configures and starts five child processes.
The main process blocks until the child processes finish.
Each child process configures its own logger, using the default stream handler that sends messages to stderr, and sets the log level to debug.
The child processes then perform their simulated task and log messages.
The child processes finish, then the main process continues on and reports a final message.
This highlights that the log module can be used directly from multiple processes, although it requires that each process configure the logging infrastructure separately.
The danger of this approach is that the target where log messages are stored may not reliably handle messages from multiple processes in parallel, in which case messages may overwrite each other and be lost or corrupt. This is especially the case if the loggers are all configured to write to the same file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
INFO:root:Main process started. INFO:root:Child Process-2 starting. DEBUG:root:Child Process-2 step 0. DEBUG:root:Child Process-2 step 1. DEBUG:root:Child Process-2 step 2. DEBUG:root:Child Process-2 step 3. DEBUG:root:Child Process-2 step 4. INFO:root:Child Process-2 done. INFO:root:Child Process-1 starting. DEBUG:root:Child Process-1 step 0. DEBUG:root:Child Process-1 step 1. DEBUG:root:Child Process-1 step 2. DEBUG:root:Child Process-1 step 3. DEBUG:root:Child Process-1 step 4. INFO:root:Child Process-1 done. INFO:root:Child Process-3 starting. DEBUG:root:Child Process-3 step 0. DEBUG:root:Child Process-3 step 1. DEBUG:root:Child Process-3 step 2. DEBUG:root:Child Process-3 step 3. DEBUG:root:Child Process-3 step 4. INFO:root:Child Process-3 done. INFO:root:Child Process-4 starting. DEBUG:root:Child Process-4 step 0. DEBUG:root:Child Process-4 step 1. DEBUG:root:Child Process-4 step 2. DEBUG:root:Child Process-4 step 3. DEBUG:root:Child Process-4 step 4. INFO:root:Child Process-4 done. INFO:root:Child Process-5 starting. DEBUG:root:Child Process-5 step 0. DEBUG:root:Child Process-5 step 1. DEBUG:root:Child Process-5 step 2. DEBUG:root:Child Process-5 step 3. DEBUG:root:Child Process-5 step 4. INFO:root:Child Process-5 done. INFO:root:Main process done. |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example Using the Multiprocessing Logger
We can explore logging from multiple processes using the multiprocessing module’s logger.
This is the same as logging directly using the logging infrastructure in the previous example in that logging is not process-safe and log messages may be lost or corrupted if written to the same target stream or handler.
The main difference is that it is using the same shared logger used by the multiprocessing module itself. It is not clear why this logger is exposed on the multiprocessing module for general use, other than perhaps to be used by classes that extend objects or functionality in the multiprocessing module.
I do not recommend this approach, however, we can explore how we might use this logger directly to log messages from multiple processes as a proof of concept.
In this example, we can update the previous example to log from multiple child processes using the multiprocessing module logger in each process, including the main process and all child processes.
Firstly, we can change the way we get the logger in order to configure it. Instead of calling logger.getLogger(), we can call multiprocessing.get_logger() to get the module logger.
This can be performed in the child process, for example:
|
1 2 3 |
... # get the multiprocessing logger logger = get_logger() |
This logger does not have a handler installed by default. Therefore, can add a StreamHandler so that messages are logged to stderr.
|
1 2 3 |
... # configure a stream handler logger.addHandler(logging.StreamHandler()) |
We can then set the preferred log level on the logger, in this case debug messages and above.
|
1 2 3 |
... # log all messages, debug and up logger.setLevel(logging.DEBUG) |
Logging at different levels can then be performed on the logger object directly instead of via the logger module functions, for example:
|
1 2 3 |
... # report initial message logger.info(f'Child {process.name} starting.') |
The updated task() function with these changes is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# task to be executed in child processes def task(): # get the multiprocessing logger logger = get_logger() # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) # get the current process process = current_process() # report initial message logger.info(f'Child {process.name} starting.') # simulate doing work for i in range(5): # report a message logger.info(f'Child {process.name} step {i}.') # block sleep(random()) # report final message logger.info(f'Child {process.name} done.') |
We can then make the same changes to configure the logger and use the logger in the main process.
For example:
|
1 2 3 4 5 6 7 |
... # get the multiprocessing logger logger = get_logger() # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
# SuperFastPython.com # example of logging from multiple processes with multiprocessing logging from random import random from time import sleep from multiprocessing import current_process from multiprocessing import Process from multiprocessing import get_logger import logging # task to be executed in child processes def task(): # get the multiprocessing logger logger = get_logger() # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) # get the current process process = current_process() # report initial message logger.info(f'Child {process.name} starting.') # simulate doing work for i in range(5): # report a message logger.info(f'Child {process.name} step {i}.') # block sleep(random()) # report final message logger.info(f'Child {process.name} done.') # protect the entry point if __name__ == '__main__': # get the multiprocessing logger logger = get_logger() # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) # report initial message logger.info(f'Main process started.') # configure child processes processes = [Process(target=task) for i in range(5)] # start child processes for process in processes: process.start() # wait for child processes to finish for process in processes: process.join() # report final message logger.info(f'Main process done.') |
Running the example first acquires the logger used in the multiprocessing module in the main process, then configures and starts five child processes.
The main process blocks until the child processes finish.
Each child process configures its own logger from the multiprocessing module, using a stream handler that sends messages to stderr, and sets the log level to debug.
The child processes then perform their simulated task and log messages.
The child processes finish, and we notice debug messages from the multiprocessing module also appear related to each child process finishing. We can also see that the format of the messages is simplified, specified by the multiprocessing module.
The main process continues on and reports a final message.
This highlights that the logger from the multiprocessing module can be used directly from multiple processes, and like using the logger module directly, it requires that each process configure the logging infrastructure separately.
The danger of this approach is the same as the danger of using the logging module directly from multiple processes, namely that the target where log messages are stored may not reliably handle messages from multiple processes in parallel, in which case messages may overwrite each other and be lost or corrupt.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
Main process started. Child Process-4 starting. Child Process-4 step 0. Child Process-2 starting. Child Process-2 step 0. Child Process-1 starting. Child Process-1 step 0. Child Process-3 starting. Child Process-3 step 0. Child Process-5 starting. Child Process-5 step 0. Child Process-5 step 1. Child Process-4 step 1. Child Process-3 step 1. Child Process-2 step 1. Child Process-2 step 2. Child Process-5 step 2. Child Process-1 step 1. Child Process-4 step 2. Child Process-5 step 3. Child Process-3 step 2. Child Process-1 step 2. Child Process-2 step 3. Child Process-3 step 3. Child Process-5 step 4. Child Process-4 step 3. Child Process-1 step 3. Child Process-2 step 4. Child Process-3 step 4. Child Process-4 step 4. Child Process-5 done. process shutting down running all "atexit" finalizers with priority >= 0 running the remaining "atexit" finalizers process exiting with exitcode 0 Child Process-1 step 4. Child Process-1 done. process shutting down running all "atexit" finalizers with priority >= 0 running the remaining "atexit" finalizers process exiting with exitcode 0 Child Process-2 done. process shutting down running all "atexit" finalizers with priority >= 0 running the remaining "atexit" finalizers process exiting with exitcode 0 Child Process-4 done. process shutting down running all "atexit" finalizers with priority >= 0 running the remaining "atexit" finalizers process exiting with exitcode 0 Child Process-3 done. process shutting down running all "atexit" finalizers with priority >= 0 running the remaining "atexit" finalizers process exiting with exitcode 0 Main process done. process shutting down running all "atexit" finalizers with priority >= 0 running the remaining "atexit" finalizers |
Example Using QueueHandler and a Logging Process
We can explore how to log in a process-safe manner.
The Python logging cookbook has good examples to get started and we can develop a new example inspired by them.
In this example, we will update the previous example so that log messages are all sent to a new logging process and logged in a serial (one-at-a-time) manner.
This can be achieved by configuring the logging infrastructure in each process to use a logging.handlers.QueueHandler that will send log messages to a shared multiprocessing.Queue. A new child process will then consume messages from the queue one-at-a-time and store them, in this case on stderr using a logging.StreamHandler, but it could just as easily be a file.
Importantly, in this example, messages cannot be lost or corrupted as only a single process is responsible for “storing” the log messages, and log messages are sent to this one process safely and reliably using a process-aware queue data structure.
Firstly, we need to define a function to run in a child process responsible for logging.
This function will take a shared multiprocessing.Queue as an argument.
|
1 2 3 |
# executed in a process that performs logging def logger_process(queue): # ... |
We need to configure the logging infrastructure in this process to specify what level of messages to report and where to store them. In this case, we will get a named log for our application called ‘app‘, and send all messages debug level and up to stderr using a StreamHandler.
|
1 2 3 4 5 6 7 |
... # create a logger logger = logging.getLogger('app') # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) |
This child process will then run forever, reading messages from the queue and logging them. If there are no messages on the queue, it will block until a message arrives.
|
1 2 3 4 5 |
... # run forever while True: # consume a log message, block until one arrives message = queue.get() |
If you are new to multiprocessing queues, you can learn more in the tutorial:
If the message is a special message None, then the look is broken and the process will terminate normally. This is called a sentinel message and must be sent by the main process before the program exits.
|
1 2 3 4 |
... # check for shutdown if message is None: break |
Otherwise, if it is a normal log message, we can let our logging infrastructure for this process handle it, e.g. log it to stderr.
|
1 2 3 |
... # log the message logger.handle(message) |
Tying this together, the complete function for running a child process responsible for logging is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# executed in a process that performs logging def logger_process(queue): # create a logger logger = logging.getLogger('app') # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) # run forever while True: # consume a log message, block until one arrives message = queue.get() # check for shutdown if message is None: break # log the message logger.handle(message) |
We can then update the child process to configure its local logging infrastructure to send log messages into the shared queue.
Firstly, we can update the function to take the shared queue as an argument.
|
1 2 3 |
# task to be executed in child processes def task(queue): # ... |
We can then configure a new logger instance to use a QueueHandler that makes use of the shared queue.
|
1 2 3 4 5 6 7 |
... # create a logger logger = logging.getLogger('app') # add a handler that uses the shared queue logger.addHandler(QueueHandler(queue)) # log all messages, debug and up logger.setLevel(logging.DEBUG) |
This configured logger can then be used in call logging calls in the child process.
Finally, we can update the main process to first create the queue that will be shared among all processes.
|
1 2 3 |
... # create the shared queue queue = Queue() |
The main process can then configure its logging infrastructure to use the QueueHandler so that messages are sent to the logging child process.
|
1 2 3 4 5 6 7 8 9 |
... # create the shared queue queue = Queue() # create a logger logger = logging.getLogger('app') # add a handler that uses the shared queue logger.addHandler(QueueHandler(queue)) # log all messages, debug and up logger.setLevel(logging.DEBUG) |
Next, we can create and start a new child process to run our custom logger_process() function responsible for reading log messages and logging them to stderr.
|
1 2 3 4 |
... # start the logger process logger_p = Process(target=logger_process, args=(queue,)) logger_p.start() |
The child processes performing our simulated work need to receive the shared queue as an argument when configured.
|
1 2 3 |
... # configure child processes processes = [Process(target=task, args=(queue,)) for i in range(5)] |
Finally, before the program exits, we need to send the special None sentinel value that will instruct the logging child process to exit its loop and terminate.
|
1 2 3 |
... # shutdown the queue correctly queue.put(None) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
# SuperFastPython.com # example of logging from multiple processes in a process-safe manner from random import random from time import sleep from multiprocessing import current_process from multiprocessing import Process from multiprocessing import Queue from logging.handlers import QueueHandler import logging # executed in a process that performs logging def logger_process(queue): # create a logger logger = logging.getLogger('app') # configure a stream handler logger.addHandler(logging.StreamHandler()) # log all messages, debug and up logger.setLevel(logging.DEBUG) # run forever while True: # consume a log message, block until one arrives message = queue.get() # check for shutdown if message is None: break # log the message logger.handle(message) # task to be executed in child processes def task(queue): # create a logger logger = logging.getLogger('app') # add a handler that uses the shared queue logger.addHandler(QueueHandler(queue)) # log all messages, debug and up logger.setLevel(logging.DEBUG) # get the current process process = current_process() # report initial message logger.info(f'Child {process.name} starting.') # simulate doing work for i in range(5): # report a message logger.debug(f'Child {process.name} step {i}.') # block sleep(random()) # report final message logger.info(f'Child {process.name} done.') # protect the entry point if __name__ == '__main__': # create the shared queue queue = Queue() # create a logger logger = logging.getLogger('app') # add a handler that uses the shared queue logger.addHandler(QueueHandler(queue)) # log all messages, debug and up logger.setLevel(logging.DEBUG) # start the logger process logger_p = Process(target=logger_process, args=(queue,)) logger_p.start() # report initial message logger.info('Main process started.') # configure child processes processes = [Process(target=task, args=(queue,)) for i in range(5)] # start child processes for process in processes: process.start() # wait for child processes to finish for process in processes: process.join() # report final message logger.info('Main process done.') # shutdown the queue correctly queue.put(None) |
Running the example first creates the shared queue and configures the logging infrastructure in the main process to use the QueueHandler with the shared queue.
This ensures that any logging in the main process is sent to the logging process.
Next, a child process is configured and started and is responsible for reading messages from the shared queue and logging them to a central location, in this case to standard error.
The child processes are then configured and started as before and the main process blocks, waiting for them to finish.
Each child process runs and configures its logging infrastructure to use a QueueHandler with the shared queue. The child processes then perform their task and log their messages which are all sent into the shared queue to the logging process.
The logging process reads messages one-at-a-time from the queue and lets its local logging infrastructure handle them, logging them to stderr.
The child processes finish and the main process carries on. It sends a None message into the queue which triggers the logging process to exit its loop and terminate.
All child processes are finished, allowing the main process to terminate.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
Main process started. Child Process-2 starting. Child Process-2 step 0. Child Process-3 starting. Child Process-3 step 0. Child Process-4 starting. Child Process-4 step 0. Child Process-5 starting. Child Process-5 step 0. Child Process-6 starting. Child Process-6 step 0. Child Process-4 step 1. Child Process-5 step 1. Child Process-3 step 1. Child Process-2 step 1. Child Process-4 step 2. Child Process-6 step 1. Child Process-4 step 3. Child Process-5 step 2. Child Process-6 step 2. Child Process-3 step 2. Child Process-6 step 3. Child Process-2 step 2. Child Process-5 step 3. Child Process-4 step 4. Child Process-3 step 3. Child Process-5 step 4. Child Process-6 step 4. Child Process-4 done. Child Process-2 step 3. Child Process-5 done. Child Process-2 step 4. Child Process-6 done. Child Process-3 step 4. Child Process-3 done. Child Process-2 done. |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to log from multiple processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Matthew Fournier on Unsplash
Hi, thanks for the good tutorial
What is not mentioned, however, is that the QueueHandler is extremely slow. QueueHandler must pickle log records when it puts them to queue, and then someone needs to unpickle when reading from the queue.
An application that’s doing a lot of stuff can send tens of thousands of debug messages – with this the queue handler virtually becomes unusable.
Best regards,
Serhii
Thank you for sharing.
Hi there! I just wanted to give you and others a heads up that there is a bit of an issue with the last example. I see duplicate log messages when running on Python 3.11.
It looks like the primary issue is that you call logger.addHandler within the sub-processes, but they already inherited their first handler from the parent. So, the child processes end up with two similar QueueHandlers registered, which causes the log messages to appear twice at the console.
Best,
Craig
Thank you Craig, I will investigate and update the tutorial.
Great resource, thanks! It would be nice to update it to show an example using a QueueListener() instead of explictly creating a log listening process yourself.
I also found I needed to create my queue using a Manager(), in this form:
log_manager = Manager()
log_queue = log_manager.Queue()
This way I can pass the log_queue to the subprocesses without running into pickle issues. I’m spawning the subprocesses using:
with get_context(“spawn”).Pool(processes=N) as pool:
Thanks Tom, great suggestion!
If anyone looking for multiprocess custom logging solution. feel free to checkout below opensource repo. Special thanks to @JasonBrownlee for your great work on these articles helped me learn a lot about multiprocessing in python.
https://github.com/solaikannanpandiyan/log_safe
Thanks for sharing.