We can automatically handle exceptions in coroutines executed via asyncio.gather() by setting the “return_exceptions” argument to True.
By default, if a coroutine is executed by asyncio.gather() fails with an unhandled exception, it will be propagated to the caller. Setting the “return_exceptions” argument to True will trap any unhandled exceptions and provide them as return values, allowing all other tasks in the group to complete.
In this tutorial, you will discover how to handle exceptions with asyncio.gather().
Let’s get started.
asyncio.gather() Exceptions
The asyncio.gather() module function allows the caller to group multiple awaitables.
Once grouped, the awaitables can be executed concurrently, awaited, and canceled.
It is a helpful utility function for both grouping and executing multiple coroutines or multiple tasks.
For example:
|
1 2 3 |
... # run a collection of awaitables results = await asyncio.gather(coro1(), coro2(), coro3()) |
We may use the asyncio.gather() function in situations where we may create many tasks or coroutines up-front and then wish to execute them all at once and wait for them all to complete before continuing on.
You can learn more about how to use asyncio.gather() in the tutorial:
Coroutines and tasks executed and awaited via asyncio.gather() can fail with exceptions.
Two examples include:
- A regular Error or Exception raised in one or more tasks.
- A CancelledError is raised in one or more tasks if the task was canceled.
The default behavior of asyncio.gather() is to propagate the raised exception to the caller.
This means that:
- The task or tasks that raise an unhandled exception are unwound and terminated.
- The exception is re-raised at the point in the program where the asyncio.gather() is awaited.
If the part of the program that is awaiting the call to asyncio.gather() does not handle the propagated exception, then the exception will also unwind the current asyncio tasks, which may be the entry point into the program, causing the entire program to terminate.
How can we handle asyncio.gather() exceptions?
Run loops using all CPUs, download your FREE book to learn how.
How to Handle asyncio.gather() Exceptions
There are a few ways we can handle asyncio exceptions, including:
- Handle exceptions in tasks executed via asyncio.gather().
- Handle exceptions raised while awaiting asyncio.gather().
- Convert raised exceptions to return values.
Let’s take a closer look at each approach in turn.
Handle Exceptions Within coroutines
We can update the coroutines and tasks executed via asyncio.gather() to handle their own exceptions so that they do not propagate beyond the task themselves.
This is straightforward if we have complete control over the code for the target coroutine.
For example:
|
1 2 3 4 5 6 |
# a custom coroutine that handles an exception async def custom_coroutine(): try: ... except Exception as e: ... |
It is a good practice to be specific with the exceptions you intend to handle, rather than catching all exceptions.
If we do not have control over the code for the target coroutine, meaning we cannot change it directly, we can instead wrap it.
For example:
|
1 2 3 4 5 6 |
# a custom coroutine that wraps a target coro and handles an exception async def custom_coroutine(): try: await other_coroutine() except Exception as e: ... |
You can learn more about handling exceptions within asyncio tasks in the tutorial:
Handle Exceptions From asyncio.gather()
Another approach to handling an exception in one or more coroutines executed via asyncio.gather() is to wrap the call to asyncio.gather() in a try-except block.
This will handle the first exception raised in the group.
For example:
|
1 2 3 4 5 6 7 |
... try: ... # run a collection of awaitables results = await asyncio.gather(coro1(), coro2(), coro3()) except Exception as e: ... |
The raised exception will not cancel the other tasks.
If this is required, we can store a reference to the task objects and call the cancel method on each.
For an example of this, see the tutorial:
If an exception is raised, it will not await for the other tasks to complete.
If this is required, again we can store a reference to the awaited tasks and await them one by one or via a call to asyncio.wait().
You can learn more about asyncio.wait() in the tutorial:
Return Exceptions With return_exceptions=True
The preferred approach is to to return any exceptions raised by tasks and coroutines in the asyncio.gather() group.
This can be achieved by setting the “return_exceptions” argument to True.
For example:
|
1 2 3 |
... # run a collection of awaitables results = await asyncio.gather(coro1(), coro2(), coro3(), return_exceptions=True) |
The effect is that it allows all coroutines in the group to complete, even if one or more raises an unhandled exception.
Any exceptions raised are provided as a return value in the “results” list returned from asyncio.gather().
Example of asyncio.gather() With Exceptions
We can explore an example of a coroutine in the asyncio.gather() group failing with an exception.
In this case, we can define a coroutine that takes an argument. It reports the value of the argument, suspends for one second, and then if the value of the argument is a special value, an exception is raised.
|
1 2 3 4 5 6 7 8 9 |
# coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # check for failure if value == 0: raise Exception('Something bad happened') |
We can then create a list of coroutine objects with different arguments from 0 to 9 and await them using asyncio.gather().
Once all tasks are done, we can report a final message.
|
1 2 3 4 5 6 7 8 9 10 |
# coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(*coros) # report a message print('main done') |
We know that one of the coroutines will fail with an unhandled exception and that the final message reported by the main() coroutine will not be reported.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of gather where one task fails with an exception import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # check for failure if value == 0: raise Exception('Something bad happened') # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(*coros) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of task_coro() objects with argument values from 0 to 9.
The list of coroutine objects is then unpacked and provided to the asyncio.gather() coroutine and awaited.
Each task runs, reports a message, then suspends for one second.
The tasks resume and check their arguments against the special value.
The first task with an argument of 0 raises an exception. This unwinds and stops the task.
The exception is then propagated to the main() coroutine where it is not handled. This unwinds and stops the main() coroutine and terminates the program.
This example highlights the default behavior of an asyncio program with asyncio.gather() and a coroutine that fails with an unhandled exception.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing Exception: Something bad happened |
Next, let’s look at how we might explicitly handle the exception.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Handling asyncio.gather() Exceptions
We can explore an example of handling the exception propagated to the main() coroutine.
In this case, we can wrap the call to asyncio.gather() in a try-except block. This will handle the first exception raised in the group.
The other tasks will continue to run and others may also fail.
We can handle this by converting the list of coroutines passed to asyncio.gather() into a list of tasks.
|
1 2 3 |
... # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] |
Then, if an exception occurs, we can wait on all those tasks passed to gather() that are not yet done, and ignore any exceptions.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
... # run the tasks try: await asyncio.gather(*tasks) except Exception as e: print(f'Got exception: {e}') # wait for all coroutines to be done for task in tasks: if not task.done(): try: await task except: pass |
This will allow all other tasks in the group to be completed.
It will also prevent any further exceptions from terminating the main() coroutine.
The updated main() coroutine with these changes is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# coroutine used for the entry point async def main(): # report a message print('main starting') # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] # run the tasks try: await asyncio.gather(*tasks) except Exception as e: print(f'Got exception: {e}') # wait for all coroutines to be done for task in tasks: if not task.done(): try: await task except: pass # report a message print('main done') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# SuperFastPython.com # example of handling exception raised by gather import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # check for failure if value == 0: raise Exception('Something bad happened') # coroutine used for the entry point async def main(): # report a message print('main starting') # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] # run the tasks try: await asyncio.gather(*tasks) except Exception as e: print(f'Got exception: {e}') # wait for all coroutines to be done for task in tasks: if not task.done(): try: await task except: pass # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of tasks with argument values from 0 to 9.
The list of task objects is then unpacked and provided to the asyncio.gather() coroutine and awaited.
Each task runs, reports a message, then suspends for one second.
The tasks resume and check their arguments against the special value.
The first task with an argument of 0 raises an exception. This unwinds and stops the task.
The exception is then propagated to the main() coroutine where it is handled and reported.
The main coroutine then awaits all those tasks in the group that are not yet done.
The final message is reported and the main() coroutine terminates normally.
This example highlights how we can handle exceptions raised within the asyncio.gather() group and how we may safely await all tasks in the group to be done.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing Got exception: Something bad happened main done |
Next, let’s explore how we might retrieve exceptions as return values.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Returning asyncio.gather() Exceptions
We can explore an example of configuring asyncio.gather() to return task exceptions as return values.
In this case, we can update the first example so that the “return_exceptions” argument to asyncio.gather() is set to true.
|
1 2 3 |
... # run the tasks await asyncio.gather(*coros, return_exceptions=True) |
And that’s it.
This will automatically trap any exceptions raised by tasks in the group.
It will allow all tasks in the group to be completed, successfully or otherwise.
It will then return any exceptions raised from tasks as a return value.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of returning exceptions from gather import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # check for failure if value == 0: raise Exception('Something bad happened') # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(*coros, return_exceptions=True) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of task_coro() objects with argument values from 0 to 9.
The list of coroutine objects is then unpacked and provided to the asyncio.gather() coroutine and awaited.
Each task runs, reports a message, then suspends for one second.
The tasks resume and check their arguments against the special value.
The first task with an argument of 0 raises an exception. This unwinds and stops the task.
The exception is trapped automatically and asyncio.gather() continues to wait for all other tasks in the group to complete.
Once all tasks are done, asyncio.gather() returns normally with a return value for each task (e.g. None in this case) and an exception object for each task that failed with an exception.
This example highlights how we can automatically handle exceptions in tasks executed via asyncio.gather().
|
1 2 3 4 5 6 7 8 9 10 11 12 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing main done |
Takeaways
You now know how to handle exceptions with asyncio.gather().
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Ricardo Gomez Angel on Unsplash
Do you have any questions?