You can log exceptions in asyncio programs by calling logging.exception() within tasks or when awaiting tasks.

We can also configure the event loop to automatically log never-retrieved exceptions when the program is shut down.

In this tutorial, you will discover how to log exceptions from asyncio.

Let’s get started.

Need to Log Exceptions in Asyncio

Asyncio programs may raise exceptions.

This may be by design as part of the normal operation of the program. It may also raise exceptions for unexpected situations.

In both cases, it is good practice to log exceptions raised in an asyncio program.

This is essential for many reasons, such as:

1. Error Diagnosis: Logging exceptions helps diagnose issues and errors in asyncio code. It provides details about what went wrong, including error messages and stack traces, making it easier to pinpoint the root cause of problems.
2. Debugging: In asynchronous code, tracing the flow of execution can be challenging. Logging exceptions at critical points in your asyncio code allows you to understand how and where exceptions are raised, helping you identify and resolve bugs.
3. Visibility: Exceptions logged to a central location, such as a file or a centralized logging service, provide visibility into the health of your asyncio application. Monitoring these logs allows you to proactively address issues before they affect users.
4. Production Debugging: In production environments, where debugging interactively is limited, logs are often the primary source of information for diagnosing problems. Exception logs provide a historical record of errors and exceptions that occurred during the application’s operation.
5. Error Reporting: Exception logs serve as a reporting mechanism for errors that may require immediate attention. They can trigger notifications or alerts to alert developers or system administrators when critical errors occur.

Logging exceptions in asyncio programs is crucial for maintaining the reliability and stability of asynchronous applications.

How can we log exceptions in asyncio programs?

How to Log Exceptions in Asyncio

There are perhaps two ways to log exceptions in asyncio programs, they are:

1. Manually Logging Exceptions
2. Automatically Logging Exceptions

Let’s take a closer look at each in turn.

How to Log Exceptions Manually

We can manually log an exception in an asyncio task or coroutine.

This can be achieved by wrapping the offending code in a try-except block and calling the logging.exception() method.

Logs a message with level ERROR on this logger.

— logging — Logging facility for Python

For example:

It is common for tasks to handle a CancelledError in order to close resources and then propagate the raised CancelledError exception to the caller.

You can learn more about best practices for canceling tasks in the tutorial:

• Asyncio Task Cancellation Best Practices

We may want to add debug logging to task cancellation to provide insight into task cancellation.

This can be achieved by logging CancelledError exceptions via logging.debug().

Logs a message with level DEBUG on this logger.

— logging — Logging facility for Python

For example:

How to Log Never-Retrieved Exceptions Automatically

In asyncio, if a background task fails with an unhandled exception and the exception is not retrieved, this is called a “never-retrieved” exception.

They are also called “asyncio silent exceptions”, as they are not reported at the time they occur.

The asyncio event loop will log these never-retrieved exceptions by default when the tasks are garbage-collected, such as when the asyncio program is terminated.

If a Future.set_exception() is called but the Future object is never awaited on, the exception would never be propagated to the user code. In this case, asyncio would emit a log message when the Future object is garbage collected.

— Developing with asyncio

The asyncio event loop provides a handler that is called automatically when reporting never-retrieved exceptions.

This is achieved by getting an instance of the event loop and calling the set_exception_handler() method to configure a handler function to call when never-retrieved exceptions are logged.

We can develop a handler function that logs never-retrieved exceptions.

Generally, we should only call the logging.exception() function from an except block, otherwise we might get unexpected behavior (e.g. the infrastructure expects an exception to have been raised).

This function should only be called from an exception handler.

— logging — Logging facility for Python

Therefore, when logging exceptions from an asyncio exception handler, we need to use a different level, e.g. error().

For example:

We can then register our custom exception handler as a first step in our asyncio program.

For example:

You can learn more about asyncio silent exceptions in the tutorial:

• Log All Silent Exceptions in Asyncio

Now that we know how to log exceptions in asyncio programs, let’s look at some worked examples.

Example of Logging Asyncio Task Exceptions From Within

We can explore an example of logging asyncio exceptions from within a task.

In this example, we will define a task that will conditionally raise an exception. We will log normal operations and catch any raised exceptions and log them. The main coroutine will prepare the logging infrastructure and create many tasks, then wait for them to be completed.

Firstly, we can define the task.

The task takes an integer argument value. It reports a starting message and sleeps for a moment. We then check its integer argument. If it is divisible by two with no remainder (e.g. even), an exception is raised and then handled and logged. Otherwise, a done message is logged.

The task() coroutine below implements this.

Next, we can define the main() coroutine that will first configure the logger to log all messages with a DEBUG level and above.

It will then log a starting message, create a TaskGroup, and start 10 versions of our task() coroutine with arguments from 0 to 9. It then waits for all tasks to complete then logs a final done message.

The main() coroutine below implements this.

Finally, we can start the asyncio event loop.

Tying this together, the complete example is listed below.

Running the program first starts the event loop and runs the main() coroutine.

The main() coroutine runs and sets the logging level, then logs an initial info message.

It then creates a TaskGroup and schedules 10 task() coroutines, then blocks until they are done.

The tasks run, first logging a message and then suspending with a sleep.

The tasks resume and check their argument value. If it is even, an exception is raised, caught and logged, otherwise an info message is logged.

Once all tasks are done, the main() coroutine resumes and logs a final info message.

This highlights how we can log exceptions directly from within our asyncio tasks.

Next, let’s look at how we might log exceptions from tasks that fail while being awaited.

Example of Logging Awaited Task Exceptions

We can explore an example of logging tasks that raise an exception while being awaited.

In this case, we can update the above example and remove the exception handling from the task() coroutine.

We then update the main() to no longer use a TaskGroup and instead issue all tasks and keep the returned asyncio.Task objects. This can be done in a list comprehension.

We can then loop over each task and await each in turn.

If an exception is raised, we can handle and log it directly without disrupting the loop.

Tying this together, the complete example is listed below.

Running the program first starts the event loop and runs the main() coroutine.

The main() coroutine runs and sets the logging level, then logs an initial info message.

It then schedules all 10 tasks and stores the Task objects in a list.

The main() coroutine then iterates over the tasks and awaits the first one.

The tasks run, first logging a message and then suspending with a sleep.

The tasks resume and check their argument value. If it is even, an exception is raised, terminating the task, otherwise, an info message is logged.

The main() coroutine resumes. Some tasks finish normally, while others fail with an exception, which is logged directly.

This highlights how we can log exceptions from awaited tasks that fail.

Next, we can look at logging canceled tasks that raise a CancelledError exception.

Example of Logging CancelledError in Asyncio

We can explore an example of logging a CancelledError exception when a task is canceled.

In this case, we can update the above example to catch a CancelledError exception, log it, and then re-raise it to allow the caller to handle the exception, if it desires (a good practice).

You can learn more about best practices when canceling tasks in the tutorial:

• Asyncio Task Cancellation Best Practices

The main() coroutine can then be updated to create 10 tasks using a list comprehension as before, then wait a moment before canceling the first 3 tasks.

This will cause the tasks to raise a CancelledError exception, which will be logged and re-raised, before terminating the task.

You can learn more about canceling tasks in the tutorial:

• How to Cancel an Asyncio Task

Tying this together, the complete example is listed below.

Running the program first starts the event loop and runs the main() coroutine.

The main() coroutine runs and sets the logging level, then logs an initial info message.

It then schedules all 10 tasks and stores the Task objects in a list.

The main() coroutine then sleeps a moment to allow the tasks to run.

The tasks run, first logging a message and then suspending with a sleep.

The main() coroutine resumes and then cancels the first 3 tasks. It then suspends again and allows the tasks to be completed.

The tasks resume. The first three tasks raise a CancelledError exception, which is caught, logged, and re-raised, terminating the tasks.

The remaining tasks log an info message and terminate normally.

The main() coroutine resumes and logs a final info message.

This highlights how we can log CancelledError exceptions within tasks without disrupting the normal propagation of task cancellation.

Next, we will explore how we can automatically log exceptions that are never retrieved in asyncio.

Example of Logging Never-Retrieved Exceptions in Asyncio

We can explore an example of automatically logging never-retrieved exceptions in asyncio.

The asyncio event loop will log silent exceptions when it is shut down. These silent exceptions are those exceptions raised by a task, but never retrieved by another task.

You can learn more about silent or never-retrieved exceptions in the tutorial:

• Log All Silent Exceptions in Asyncio

We can log these exceptions using the logging infrastructure.

This can be achieved by configuring a log handler function that takes the event loop object and a dict “context” as arguments.

This function can then retrieve the raised exception from the “context” dict via the “exception” key and log it.

We should not log exceptions using logging.exception() when outside of a try-except structure as it can cause problems. Instead, we should use logging.error() or similar.

We will use logging.error() in this case.

logging.ERROR: Due to a more serious problem, the software has not been able to perform some function.

— logging — Logging facility for Python

We can then configure the asyncio event loop to call our custom handler for each never-retrieved exception on shutdown.

This can be achieved by first retrieving the event loop object via the asyncio.get_running_loop() function, then calling the set_exception_handler() function to set the handler function.

Tying this together, the complete example is listed below.

Running the program first starts the event loop and runs the main() coroutine.

The main() coroutine runs and sets the logging level, configures the event loop exception handler function, and then logs an initial info message.

It then schedules all 10 tasks and stores the Task objects in a list.

The main() coroutine then suspends for enough time for all tasks to be completed.

The tasks run, first logging a message and then suspending with a sleep.

The tasks resume and check their argument value. If it is even, an exception is raised, terminating the task, otherwise, an info message is logged.

The main() coroutine resumes and reports the final info message.

The event loop then calls our custom exception handler function for each exception that was never retrieved. We can see error log messages for all of the tasks that received an even argument.

This highlights how we can automatically log never-retrieved exceptions when the event loop is shut down.

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know how to log exceptions from asyncio.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Louis Hansel on Unsplash