Your asyncio program may report a warning: Task exception was never retrieved.
Asyncio will report this message when the program is terminated and there were tasks that failed with an exception and the exception was never retrieved.
It is a good practice to retrieve task exceptions if possible, otherwise to log all never retrieved exceptions automatically.
In this tutorial, you will discover how to automatically log never-retrieved exceptions in asyncio programs.
Let’s get started.
What Are Never-Retrieved Exceptions
A never-retrieved exception is an exception raised in a coroutine or task that is not explicitly retrieved.
If a Future.set_exception() is called but the Future object is never awaited on, the exception would never be propagated to the user code. In this case, asyncio would emit a log message when the Future object is garbage collected.
— Developing with asyncio
If an exception is not retrieved, then a warning message is reported automatically by the asyncio event loop with the message:
- Task exception was never retrieved
This message is reported along with the details of the task for each never-retrieved exception.
The message is not reported until the event loop is terminated, e.g. the program has ended.
We can avoid this message in our programs by retrieving the exception.
An exception can be retrieved from an asyncio task in one of three ways:
- Awaiting the coroutine or task object, the exception is re-raised.
- Calling the result() method to get the return value, the exception is re-raised
- Calling the exception() method to get the exception.
You can learn more about never-retrieved exceptions in the tutorial:
How can we automatically log never-retrieved exceptions raised by the asyncio event loop?
Run loops using all CPUs, download your FREE book to learn how.
How to Log Never-Retrieved Exceptions Automatically
We can log never-retrieved exceptions automatically in our asyncio programs.
There are two approaches we can use:
- Configure the logging infrastructure.
- Add a custom handler for never-retrieved exceptions.
Configure Logging to Log Never-Retrieved Exceptions
This can be achieved by simply configuring the logging infrastructure.
Once configured, the asyncio event loop will use it to log never-awaited exceptions and their stack trace as an error message.
For example, we can configure an asyncio program to log all messages with a DEBUG level and above by adding the following line before starting the event loop:
|
1 2 3 |
... # prepare the logger logging.basicConfig(level=logging.DEBUG) |
Add Custom Handler To Log Never-Retrieved Exceptions
We can develop a custom handler function that is called by the asyncio event loop for each never-retrieved exception.
This can be achieved by first defining the handler function.
The function must take an event loop and context as arguments. The context is a dict that has details about the exception that was never retrieved.
We can then retrieve the exception and log it as an error message.
|
1 2 3 4 5 6 7 |
# define an exception handler def exception_handler(loop, context): # get details of the exception exception = context['exception'] message = context['message'] # log exception logging.error(f'Task failed, msg={message}, exception={exception}') |
We can then configure the asyncio event loop to call our custom handler function for each never-retrieved exception when shutting down the event loop.
This can be achieved by first getting the event loop, then calling the set_exception_handler() method and passing the name of our handler function as an argument.
For example:
|
1 2 3 4 5 |
... # get the event loop loop = asyncio.get_running_loop() # set the exception handler loop.set_exception_handler(exception_handler) |
Now that we know how to log never-retrieved exceptions, let’s look at some worked examples.
Example of Never-Retrieved Exception
We can explore an example with a never-retrieved exception.
In this example, we will define a task that does some work and then raise an exception. The task is run in the background and the exception is never-retrieved. The expectation is that the event loop will report the never-retrieved exception when the program is terminated.
Firstly, we can define a task that raises an exception that is never retrieved.
The task reports a message, sleeps for one second, then raises an exception.
The work() coroutine below implements this.
|
1 2 3 4 5 6 7 8 9 10 |
# task that does work and logs async def work(): # log a message print(f'Task is starting') # simulate doing work await asyncio.sleep(1) # fail with an exception raise RuntimeError('Something bad happened') # log a message print(f'Task is done') |
Next, we can define the main() coroutine.
This involves first reporting a message to indicate the program has started then creating and scheduling the work() coroutine as a background task. The main() coroutine then sleeps for two seconds and reports a final message.
The main() coroutine below implements this.
|
1 2 3 4 5 6 7 8 9 10 |
# main coroutine async def main(): # log a message print(f'Main is starting') # schedule a task task = asyncio.create_task(work()) # wait around await asyncio.sleep(2) # log a message print(f'Main is done') |
Finally, we can start the event loop and run the main() coroutine.
|
1 2 3 |
... # start the event loop asyncio.run(main()) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of a never-retrieved exception import asyncio # task that does work and logs async def work(): # log a message print(f'Task is starting') # simulate doing work await asyncio.sleep(1) # fail with an exception raise RuntimeError('Something bad happened') # log a message print(f'Task is done') # main coroutine async def main(): # log a message print(f'Main is starting') # schedule a task task = asyncio.create_task(work()) # wait around await asyncio.sleep(2) # log a message print(f'Main is done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates and schedules the work() coroutine as a background task, then suspends with a two-second sleep.
The work() task runs and reports a message and then suspends with a one-second sleep. It resumes and fails with an exception.
The main() coroutine resumes and reports a final done message.
The event loop terminates and reports the never-retrieved exception, including:
- The message “Task exception was never retrieved“
- The details of the task that failed with a never-retrieved exception.
- A trace of the exception that was not retrieved.
This highlights the default behavior of the asyncio event loop when the program is terminated that had tasks with never retrieved exceptions.
|
1 2 3 4 5 6 7 8 9 |
Main is starting Task is starting Main is done Task exception was never retrieved future: <Task finished name='Task-2' coro=<work() done, defined at ...:6> exception=RuntimeError('Something bad happened')> Traceback (most recent call last): File "...", line 12, in work raise RuntimeError('Something bad happened') RuntimeError: Something bad happened |
Next, let’s explore logging of never-retrieved exceptions.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Never-Retrieved Exception With Logging
We can explore an example that logs a never-retrieved exception.
This involves updating the above example and configuring the logging infrastructure in the standard library.
We will set the log level to DEBUG so that all messages with a DEBUG level or above are reported. Log messages will be reported on standard output.
|
1 2 3 |
... # prepare the logger logging.basicConfig(level=logging.DEBUG) |
We can also update all messages reported via print() in the work() and main() coroutines to be INFO-level log messages.
For example:
|
1 2 3 4 5 6 7 8 9 10 |
# task that does work and logs async def work(): # log a message logging.info(f'Task is starting') # simulate doing work await asyncio.sleep(1) # fail with an exception raise RuntimeError('Something bad happened') # log a message logging.info(f'Task is done') |
And that’s it.
The expectation is that the asyncio event loop will use the configured logging infrastructure to report the details of the never-retrieved exception.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of a never-retrieved exception import logging import asyncio # task that does work and logs async def work(): # log a message logging.info(f'Task is starting') # simulate doing work await asyncio.sleep(1) # fail with an exception raise RuntimeError('Something bad happened') # log a message logging.info(f'Task is done') # main coroutine async def main(): # log a message logging.info(f'Main is starting') # schedule a task task = asyncio.create_task(work()) # wait around await asyncio.sleep(2) # log a message logging.info(f'Main is done') # prepare the logger logging.basicConfig(level=logging.DEBUG) # start the event loop asyncio.run(main()) |
Running the example first configures the logger, then starts the event loop and runs the main() coroutine.
The event loop is started and logs a debug message indicating the selector that is being used.
The main() coroutine runs and logs a start message.
It then creates and schedules the work() coroutine as a background task, then suspends with a two-second sleep.
The work() task runs and logs a message and then suspends with a one-second sleep. It resumes and fails with an exception.
The main() coroutine resumes and logs a final done message.
The event loop terminates and reports the never-retrieved exception, including:
- An ERROR-level log message with the details “Task exception was never retrieved“.
- The details of the task that failed with a never-retrieved exception.
- A trace of the exception that was not retrieved.
The details of the task and trace were not logged, instead, they were reported to standard output directly.
This highlights that the asyncio event loop will log an ERROR-level message if it is terminated with tasks that had never-retrieved exceptions.
|
1 2 3 4 5 6 7 8 9 10 |
DEBUG:asyncio:Using selector: KqueueSelector INFO:root:Main is starting INFO:root:Task is starting INFO:root:Main is done ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-2' coro=<work() done, defined at ...:7> exception=RuntimeError('Something bad happened')> Traceback (most recent call last): File "...", line 13, in work raise RuntimeError('Something bad happened') RuntimeError: Something bad happened |
Next, let’s look at an example of custom logging of never-retrieved exceptions.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Custom Logging a Never-Retrieved Exception
We can explore an example with the custom logging of a never-retrieved exception.
We can update the above example to define a custom exception handler and configure the asyncio event loop to call our handler when shutting down with tasks that have never-retrieved exceptions.
Firstly, we can define a custom exception handler. It will retrieve the details of the exception and message from the context argument and log an error message.
The exception_handler() function below implements this.
|
1 2 3 4 5 6 7 |
# define an exception handler def exception_handler(loop, context): # get details of the exception exception = context['exception'] message = context['message'] # log exception logging.error(f'Task failed, msg={message}, exception={exception}') |
Note that this is a Python function, not a coroutine.
It will be called automatically by the event loop during shutdown for each task that was left with a never-retrieved exception.
Next, in the main() coroutine we can configure the event loop to call our custom handler function during shutdown for each task that has a never-retrieved exception.
This can be achieved via the set_exception_handler() method on the event loop.
|
1 2 3 4 5 |
... # get the event loop loop = asyncio.get_running_loop() # set the exception handler loop.set_exception_handler(exception_handler) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
# SuperFastPython.com # example of a never-retrieved exception import logging import asyncio # define an exception handler def exception_handler(loop, context): # get details of the exception exception = context['exception'] message = context['message'] # log exception logging.error(f'Task failed, msg={message}, exception={exception}') # task that does work and logs async def work(): # log a message logging.info(f'Task is starting') # simulate doing work await asyncio.sleep(1) # fail with an exception raise RuntimeError('Something bad happened') # log a message logging.info(f'Task is done') # main coroutine async def main(): # get the event loop loop = asyncio.get_running_loop() # set the exception handler loop.set_exception_handler(exception_handler) # log a message logging.info(f'Main is starting') # schedule a task task = asyncio.create_task(work()) # wait around await asyncio.sleep(2) # log a message logging.info(f'Main is done') # prepare the logger logging.basicConfig(level=logging.DEBUG) # start the event loop asyncio.run(main()) |
Running the example first configures the logger, then starts the event loop and runs the main() coroutine.
The event loop is started and logs a debug message indicating the selector that is being used.
The main() coroutine runs and configures the event loop to use our custom exception handler. It then logs a start message.
The main() coroutine then creates and schedules the work() coroutine as a background task, and then suspends with a two-second sleep.
The work() task runs and logs a message and then suspends with a one-second sleep. It resumes and fails with an exception.
The main() coroutine resumes and logs a final done message.
The event loop shuts down and notices a task with a never-retrieved exception. It calls our exception_handler() with the details of the exception.
The exception_handler() function runs and gathers details of the exception from the context and logs an error message.
Unlike the previous cases, the trace of the task details and trace of the exception are not logged to standard output.
This highlights how we can use custom logging to log the details of a never-retrieved task exception.
|
1 2 3 4 |
INFO:root:Main is starting INFO:root:Task is starting INFO:root:Main is done ERROR:root:Task failed, msg=Task exception was never retrieved, exception=Something bad happened |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to automatically log never-retrieved exceptions in asyncio programs.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Zoltan Tasi on Unsplash
Do you have any questions?