We can execute coroutines concurrently with asyncio.gather().
By default, an exception in asyncio.gather() will be propagated to the caller immediately. The asyncio.gather() call will not wait for the other coroutines to complete, and it will not cancel the other tasks.
It is critically important to understand the default behavior of asyncio.gather() because it is the most common way to run multiple coroutines concurrently.
In this tutorial, you will discover that asyncio.gather() will not wait for or cancel other coroutines if one fails with an exception.
Let’s get started.
What Happens If There Is An Exception With asyncio.gather()
The asyncio.gather() function allows us to run multiple coroutines or tasks concurrently.
Coroutines can be provided as positional arguments to asyncio.gather() and a list of return values is returned, for example:
|
1 2 3 |
... # execute coroutines concurrently results = await asyncio.gather(coro1(), coro2()) |
Alternately, we can create a list of coroutines or tasks and unpack them as positional arguments using the star operator (*), for example:
|
1 2 3 4 5 |
... # creates a list of coroutine objects coros = [coro(i) for i in range(100)] # execute coroutines concurrently results = await asyncio.gather(*coros) |
You can learn more about how to use asyncio.gather() in the tutorial:
One or more coroutines or tasks provided to asyncio.gather() may fail with an exception.
This raises a question as to what happens to asyncio.gather().
For example:
If a coroutine fails with an exception, does it unwind asyncio.gather()?
Is the exception propagated to the caller?
Are all other coroutines canceled?
How does asyncio.gather() handle the case of a coroutine failing with an exception?
Run loops using all CPUs, download your FREE book to learn how.
Asyncio gather() Does Not Cancel Tasks On Exception
The default behavior of asyncio.gather() when a coroutine fails with an exception is as follows:
- The unhandled exception will unwind the coroutine or task in which it is raised.
- The asyncio.gather() call will not cancel other coroutines and tasks.
- The asyncio.gather() call will not wait for all other coroutines and tasks to be done.
- The exception will be propagated to the caller immediately.
This default behavior assumes that the return_exceptions argument is False.
Some idea of this behavior is described in the API documentation.
For example:
If return_exceptions is False (default), the first raised exception is immediately propagated to the task that awaits on gather(). Other awaitables in the aws sequence won’t be cancelled and will continue to run.
— Coroutines and Tasks
This behavior can be changed by setting the return_exceptions argument to True.
The effect will be:
- The unhandled exception will unwind the coroutine or task in which it is raised.
- The asyncio.gather() call will not cancel other coroutines and tasks.
- The asyncio.gather() call will wait for all other coroutines and tasks to be done.
- The exception will not be propagated to the caller immediately.
- The exception will be provided as a return value for the failed coroutine or task.
You can learn more about returning exceptions from asyncio.gather() in the tutorial:
It is possible to cancel all coroutines and tasks if one task in asyncio.gather() fails with an exception, although this requires additional code.
You can learn more about how to cancel all tasks if one task in asyncio.gather() fails in the tutorial:
Now that we know about the default behavior of asyncio.gather() with a coroutine that fails with an exception, let’s look at a worked example.
Example of asyncio.gather() With Exception in One Task
We can explore an example that shows the default behavior of asyncio.gather() and an exception.
In this case, we can define a coroutine that conditionally fails. We can then execute many examples of the coroutine concurrently and allow one to fail, then observe the behavior of the other coroutines still running and confirm the expected behavior of asyncio.gather(), specifically:
- That asyncio.gather() will not wait for all other coroutines to complete.
- That asyncio.gather() will not cancel all other coroutines.
- That the exception will be propagated to the caller immediately.
Firstly, we can define a custom coroutine that will conditionally fail.
In this case, the coroutine will take an integer argument. It will first suspend for 4 seconds, then resume and check if the provided argument was zero. If so, an exception is raised, otherwise, the coroutine will suspend for a further 2 seconds, report a final message, and then return the provided argument multiplied by 100.
The task_coro() coroutine below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# coroutine used for a task async def task_coro(value): # sleep for a moment await asyncio.sleep(4) # check for failure if value == 0: raise Exception('Something Bad Happened') # wait some more await asyncio.sleep(2) # report details print(f'>{value} done') # return a result return value * 100 |
Next, we can define the main coroutine.
The main coroutine first reports a message and then creates 10 task_coro() objects with arguments from 0 to 9, meaning 1 of the 10 coroutines is expected to fail with an unhandled exception.
Next, asyncio.gather() is used to execute all 10 coroutines concurrently which is wrapped in a try-except block. If/when an exception is raised, a message is reported and the coroutine suspends, allowing the other coroutines to complete.
The main() coroutine below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] try: # run the tasks results = await asyncio.gather(*coros) # report results print(results) except Exception as e: print(f'Gather failed with exception: {e}') # wait around... await asyncio.sleep(2) # report a message print('Main done') |
The expectation is that all coroutines will run normally, and then one will fail with an exception. The exception will be propagated to main() immediately, which will report a message and suspend. The other coroutines will complete and report their message before the main coroutine resumes and terminates the program. This will clearly show the default behavior of asyncio.gather() when a coroutine fails with an exception.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
# SuperFastPython.com # example of gather() with one task that fails with an exception import asyncio # coroutine used for a task async def task_coro(value): # sleep for a moment await asyncio.sleep(4) # check for failure if value == 0: raise Exception('Something Bad Happened') # wait some more await asyncio.sleep(2) # report details print(f'>{value} done') # return a result return value * 100 # coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] try: # run the tasks results = await asyncio.gather(*coros) # report results print(results) except Exception as e: print(f'Gather failed with exception: {e}') # wait around... await asyncio.sleep(2) # report a message print('Main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message. It then creates a list of 10 task_coro() objects with arguments from 0 to 9.
The list of coroutines is then expanded and provided to asyncio.gather() to be expected concurrently. The main coroutine then suspends and waits for gather() to complete.
The coroutines run concurrently, each suspends for 4 seconds.
The coroutines resume the object with the argument of 0 raising an exception. The exception bubbles up and causes the task to terminate. It immediately propagates to the main() coroutine where it is handled, a message is reported and the main() coroutine suspends again.
The remaining coroutines suspend for 2 seconds, report a final message, and then return their result.
The main coroutine results and reports a final message.
This highlights the default behavior of asyncio.gather() when an exception is raised in one or more coroutines.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting Gather failed with exception: Something Bad Happened >1 done >2 done >3 done >4 done >5 done >6 done >7 done >8 done >9 done Main done |
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Takeaways
You now know how that asyncio.gather() will not wait for or cancel other coroutines if one fails with an exception.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Karl Greif on Unsplash
Do you have any questions?