Last Updated on November 23, 2023
We need ways to work with collections of tasks in a way that they can be treated as a group.
The asyncio.gather() module function provides this capability and will return an iterable of return values from the awaited asks.
In this tutorial, you will discover how to await asyncio tasks concurrently with asyncio.gather().
After completing this tutorial, you will know:
- That the asyncio.gather() function will wait for a collection of tasks to complete and retrieve all return values.
- How to use asyncio.gather() with collections of coroutines and collections of tasks.
- How to use asyncio.gather() to create nested groups of tasks that can be awaited and cancelled.
Let’s get started.
What is Asyncio gather()
The asyncio.gather() module function allows the caller to group multiple awaitables together.
Once grouped, the awaitables can be executed concurrently, awaited, and canceled.
Run awaitable objects in the aws sequence concurrently.
— Coroutines and Tasks
It is a helpful utility function for both grouping and executing multiple coroutines or multiple tasks.
For example:
|
1 2 3 |
... # run a collection of awaitables results = await asyncio.gather(coro1(), asyncio.create_task(coro2())) |
We may use the asyncio.gather() function in situations where we may create many tasks or coroutines up-front and then wish to execute them all at once and wait for them all to complete before continuing on.
This is a likely situation where the result is required from many like-tasks, e.g. same task or coroutine with different data.
The awaitables can be executed concurrently, results returned, and the main program can resume by making use of the results on which it is dependent.
The gather() function is more powerful than simply waiting for tasks to complete.
It allows a group of awaitables to be treated as a single awaitable.
This allows:
- Executing and waiting for all awaitables in the group to be done via an await expression.
- Getting results from all grouped awaitables to be retrieved later via the result() method.
- The group of awaitables to be canceled via the cancel() method.
- Checking if all awaitables in the group are done via the done() method.
- Executing callback functions only when all tasks in the group are done.
And more.
The asyncio.gather() function is similar to asyncio.wait(). It will suspend until all provided tasks are done, except it will return an iterable of return values from all tasks, whereas wait(), by default, will not retrieve task results.
You can learn more about how asyncio.gather() is different from asyncio.wait() in the tutorial:
Now that we know what the asyncio.gather() function is, let’s look at how we might use it.
Run loops using all CPUs, download your FREE book to learn how.
How to use Asyncio gather()
In this section, we will take a closer look at how we might use the asyncio.gather() function.
gather() Takes Tasks and Coroutines
The asyncio.gather() function takes one or more awaitables as arguments.
Recall an awaitable may be a coroutine, a Future or a Task.
Therefore, we can call the gather() function with:
- Multiple tasks
- Multiple coroutines
- Mixture of tasks and coroutines
For example:
|
1 2 3 |
... # execute multiple coroutines asyncio.gather(coro1(), coro2()) |
If Task objects are provided to gather(), they will already be running because Tasks are scheduled as part of being created.
The asyncio.gather() function takes awaitables as position arguments.
We cannot create a list or collection of awaitables and provide it to gather, as this will result in an error.
For example:
|
1 2 3 |
... # cannot provide a list of awaitables directly asyncio.gather([coro1(), coro2()]) |
A list of awaitables can be provided if it is first unpacked into separate expressions using the star operator (*), also called the asterisk operator.
This operator specifically unpacks iterables, like lists, into separate expressions. It is often referred to as the iterable unpacking operator.
An asterisk * denotes iterable unpacking. Its operand must be an iterable. The iterable is expanded into a sequence of items, which are included in the new tuple, list, or set, at the site of the unpacking.
Expression lists
For example:
|
1 2 3 |
... # gather with an unpacked list of awaitables asyncio.gather(*[coro1(), coro2()]) |
If coroutines are provided to gather(), they are wrapped in Task objects automatically.
gather() Returns a Future
The gather() function does not block.
Instead, it returns an asyncio.Future object that represents the group of awaitables.
For example:
|
1 2 3 |
... # get a future that represents multiple awaitables group = asyncio.gather(coro1(), coro2()) |
Once the Future object is created it is scheduled automatically within the event loop.
The awaitable represents the group, and all awaitables in the group will execute as soon as they are able.
This means that if the caller did nothing else, the scheduled group of awaitables will run (assuming the caller suspends).
It also means that you do not have to await the Future that is returned from gather().
For example:
|
1 2 3 4 5 |
... # get a future that represents multiple awaitables group = asyncio.gather(coro1(), coro2()) # suspend and wait a while, the group may be executing.. await asyncio.sleep(10) |
Awaiting gather()’s Future
The returned Future object can be awaited which will wait for all awaitables in the group to be done.
For example:
|
1 2 3 |
... # run the group of awaitables await group |
Awaiting the Future returned from gather() will return a list of return values from the awaitables.
If the awaitables do not return a value, then this list will contain the default “None” return value.
For example:
|
1 2 3 |
... # run the group of awaitables and get return values results = await group |
This is more commonly performed in one line.
For example:
|
1 2 3 |
... # run tasks and get results on one line results = await asyncio.gather(coro1(), coro2()) |
gather() Can Nest Groups of Awaitables
The gather() function takes awaitables and itself returns an awaitable.
Therefore, we can create nested groups of awaitables.
For example:
|
1 2 3 4 5 6 7 |
... # create a group of tasks group1 = asyncio.gather(coro1(), coro2()) # create another group of tasks group2 = asyncio.gather(group1, coro3()) # run group2 which will also run group1 await group2 |
gather() and Exceptions
If an awaitable fails with an exception, the exception is re-raised in the caller and may need to be handled.
For example:
|
1 2 3 4 5 6 |
... try: # run tasks and get results results = await asyncio.gather(coro1(), coro2()) except Exception as e: # ... |
Similarly, if a task in the group is canceled, it will re-raise a CancelledError exception in the caller and may need to be handled.
The “return_exceptions” argument to gather() can be set to True which will catch exceptions and provide them as return values instead of re-raising them in the caller.
This applies to both exceptions raised in awaitables, as well as CancelledError exceptions if the awaitables are canceled.
For example:
|
1 2 3 |
... # run tasks and retrieve exceptions as a result results = await asyncio.gather(coro1(), coro2(), return_exceptions=True) |
gather()’s Future Can Be Canceled
The Future that is returned from gather() can be used just like a normal asyncio.Future object.
We can check if it is done by calling the done() method.
For example:
|
1 2 3 4 |
... # check if a gather group of tasks is done if group.done(): # ... |
We can also cancel the gather()‘s Future, which will cancel all tasks within the group.
For example:
|
1 2 3 |
... # cancel all tasks in the group group.cancel() |
Other helpful methods we might use include adding a done callback function, getting a result, and checking if the group was canceled.
You can learn more about using a done callback on a gather’s task in the tutorial:
Now that we know how to use asyncio.gather(), let’s look at some worked examples.
Examples of gather() with Coroutines
In this section, we will look at how we can run coroutines concurrently using the asyncio.gather() function.
Example of gather() For One Coroutine
We can use gather() to manage a single coroutine.
A coroutine can be executed directly by creating it and await it. Nevertheless, we can use the gather() function to wrap it in a Future object and create a group of one awaitable that can be managed.
The example below defines a task coroutine that is gathered and run.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of gather for one coroutine import asyncio # coroutine used for a task async def task_coro(): # report a message print('task executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create a coroutine coro = task_coro() # gather one coroutine await asyncio.gather(coro) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and executes the main() coroutine as the entry point to the asyncio program.
The main() coroutine reports a message and then creates the task coroutine.
The gather() function is then called, passing the single task coroutine as an argument. This returns a Future object that is awaited.
The main() coroutine is suspended and the task coroutine is executed, reporting a message and sleeping before terminating
The main() coroutine then resumes and reports a final message.
This highlights how we might run a single coroutine using gather().
|
1 2 3 |
main starting task executing main done |
Example of gather() For Many Coroutines
We can use gather() to group many coroutines together into a single task.
This is the most common use case for the gather() function.
The example below defines a task coroutine that takes an argument, reports a message with the argument, and sleeps for a moment.
The main coroutine then calls gather() with three separate calls to the task coroutine, each with different arguments. This allows all three coroutines to be executed concurrently and for the caller to wait for all three coroutines to complete before resuming.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of gather for many coroutines import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # run the tasks await asyncio.gather(task_coro(0), task_coro(1), task_coro(2)) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example executes the main() coroutine as the entry point to the program.
A message is reported and then the gather() function is called. It is passed three different coroutine instances that call the same coroutine function with different arguments.
The coroutines are run concurrently and the main coroutine() suspends and waits for the group of awaitables to be done.
Each coroutine runs, reports its message, sleeps for a moment, and terminates.
The main() coroutine resumes only after all three coroutines are completed, and then reports its final message,
This highlights the most common use case of gathering multiple coroutines and waiting for them to complete.
|
1 2 3 4 5 |
main starting >task 0 executing >task 1 executing >task 2 executing main done |
Example of gather() For Many Coroutines in a List
It is common to create multiple coroutines beforehand and then gather them later.
This allows a program to prepare the tasks that are to be executed concurrently and then trigger their concurrent execution all at once and wait for them to complete.
We can collect many coroutines together into a list either manually or using a list comprehension.
For example:
|
1 2 3 |
... # create many coroutines coros = [task_coro(i) for i in range(10)] |
We can then call gather() with all coroutines in the list.
The list of coroutines cannot be provided directly to the gather() function as this will result in an error.
Instead, the gather() function requires each awaitable to be provided as a separate positional argument.
This can be achieved by unwrapping the list into separate expressions and passing them to the gather() function. The star operator (*) will perform this operation for us.
For example:
|
1 2 3 |
... # run the tasks await asyncio.gather(*coros) |
Are you new to the star or asterisk (*) operator? It is used for iterable unpacking. You can learn more in the expression API documentation.
Tying this together, the complete example of running a list of pre-prepared coroutines with gather() is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of gather for many coroutines in a list import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(*coros) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example executes the main() coroutine as the entry point to the program.
The main() coroutine then creates a list of 10 coroutine objects using a list comprehension.
This list is then provided to the gather() function and unpacked into 10 separate expressions using the star operator.
The main() coroutine then awaits the Future object returned from the call to gather(), suspending and waiting for all scheduled coroutines to complete their execution.
The coroutines run as soon as they are able, reporting their unique messages and sleeping before terminating.
Only after all coroutines in the group are complete does the main() coroutine resume and report its final message.
This highlights how we might prepare a collection of coroutines and provide them as separate expressions to the gather() function.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing main done |
Example of gather() With Return Values
We may execute coroutines that return a value.
Using the gather() function to execute multiple coroutines that return values allows the results of all coroutines to be gathered together at one point for use, giving the gather() function its name.
Recall that the gather() function does not block, but returns immediately with a Future object.
When the Future object that is returned from gather() is awaited, it will return a list of return values from the grouped tasks.
The example below demonstrates this by updating the task coroutine to return a value and gathering and reporting the list of return values from all tasks in the main coroutine.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of gather for many coroutines that return values import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # return a value return value * 10 # coroutine used for the entry point async def main(): # report a message print('main starting') # create many tasks tasks = [task_coro(i) for i in range(10)] # run the tasks values = await asyncio.gather(*tasks) # report the values print(values) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example first creates and runs the main() coroutine as the entry point into the program.
A list of coroutines is then created, each with a different argument.
The gather function is called and the list of coroutines is then unpacked into separate expressions using the star operator.
This schedules each coroutine for execution.
The main coroutine suspends and waits for all coroutines in the group to complete.
Each coroutine runs, reporting a message, sleeping, and returning a value that is a multiple of 10 of the input argument.
All coroutines are completed and the awaited Future returned from gather() provides a list of return values.
The main() coroutine then reports the list of return values.
This highlights how we can retrieve return values from multiple coroutines that return values.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing [0, 10, 20, 30, 40, 50, 60, 70, 80, 90] main done |
Example of gather() With Nested Groups
The gather() function can be used to create one awaitable that represents multiple awaitables.
The natural consequence of this is that we can nest groups of awaitables.
We can demonstrate this with a worked example.
In this example, we create two separate groups of three coroutines. We then define a third group that combines the two previously created groups of coroutines.
This creates a hierarchy with three levels of awaitables, for example:
- group 3
- group 1
- coroutine 0
- coroutine 1
- coroutine 2
- group 2
- coroutine 3
- coroutine 4
- coroutine 5
- group 1
Performing an operation on the top-level awaitable will perform an operation on all nested awaitables.
Most importantly, running the top-level will run all nested awaitables concurrently.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of gather nested groups of coroutines import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create group level 1 group1 = asyncio.gather(task_coro(0), task_coro(1), task_coro(2)) # create group level 2 group2 = asyncio.gather(task_coro(3), task_coro(4), task_coro(5)) # create group level 3 group3 = asyncio.gather(group1, group2) # execute 3, which executes 1 and 2 await group3 # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example first creates and runs the main() coroutine.
The main() coroutine first uses the gather() function to create two separate groups of coroutines. It then calls gather() to create a third group that combines the first two groups of coroutines.
It then awaits the third group.
This schedules all awaitables for execution and suspends the main coroutine until the hierarchy of awaitables is completed.
All awaitables are completed and the main coroutine resumes and reports a final message.
This highlights how we can easily create hierarchies of groups of awaitables using the gather() function.
|
1 2 3 4 5 6 7 8 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing main done |
Example of gather() Mix of Tasks and Coroutines
Like a coroutine, an asyncio.Task is an awaitable.
This means that we can provide a mixture of Task objects as well as coroutines to the gather() function.
- The coroutines provided to gather() will be wrapped in Task objects and scheduled for execution.
- The Task objects will already be scheduled for execution.
- The Future object that is returned from the gather() function will be scheduled for execution.
Although we can await the Future that is returned from the gather() function, this does not execute the group, the group is already scheduled.
Mixing tasks and coroutines in the call to gather makes this more apparent.
It also means that some awaitables provided to a call to gather() could already be done. Awaiting the group only waits for those awaitables that are not already done.
The example below explores the ability to mix coroutines and awaitables in a call to gather().
It also does not explicitly await the Future object returned from the gather() function but instead sleeps while the group of awaitables is executed.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of gather with of tasks and coroutines import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create a mix of awaitables awaitables = [task_coro(0), asyncio.create_task(task_coro(1)), task_coro(2), asyncio.create_task(task_coro(3)), task_coro(4),] # schedule the group _ = asyncio.gather(*awaitables) # wait around for a while await asyncio.sleep(2) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example first creates and runs the main() coroutine.
The main() coroutine then creates a list that contains a mixture of coroutine objects and Task objects.
The Task objects are scheduled for execution as soon as they are created. The coroutines are not.
The list is then provided to the gather() function and the returned Future that represents the group is ignored.
This schedules the group of awaitables for execution in the event loop.
The main() coroutine then suspends with a call to sleep.
All tasks and coroutines are given an opportunity to execute, reporting their message and sleeping.
The main() coroutine resumes after its sleep and all coroutines and tasks happen to be complete by this time.
A final message is reported and the asyncio program then terminates.
This highlights that we can create a group that contains a mixture of tasks and coroutines and that we do not have to explicitly await the Future returned from gather() in order for the group to be executed.
|
1 2 3 4 5 6 7 |
main starting >task 1 executing >task 3 executing >task 0 executing >task 2 executing >task 4 executing main done |
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Examples of gather() with Failures
This section explores using gather with awaitables that fail with an exception or are canceled.
Example of gather() Where One Awaitable Fails
An awaitable in group may fail with an unhandled exception.
If this happens, the exception will be re-raised in the caller and may need to be handled.
Recall that all awaitables are scheduled after the call to gather().
This means that even though one awaitable may have failed with an exception, the remaining awaitable may continue to execute and complete normally, if given the opportunity, e.g. the program is not terminated.
The example below explores this. A total of 10 coroutines are passed to a call to gather() and are scheduled. One of the coroutines will fail with an exception that is re-raised in the caller.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of gather where one task fails with an exception import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # check for failure if value == 0: raise Exception('Something bad happened') # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(*coros) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and runs the main() coroutine as the entry point into the program.
The main() coroutine then creates 10 task coroutines in a list comprehension. These are then provided to a call to gather().
All coroutines are scheduled for execution along with the Future returned from gather().
The main() coroutine then suspends and waits for the tasks to complete.
All coroutines execute, reporting a message and blocking.
One coroutine then raises an exception.
The exception does not impact any other coroutines in the group.
The main() coroutine resumes and the exception raised in the coroutine is re-raised and not handled, terminating the application.
This highlights that exceptions that occur within a group of awaitables is re-raised in the caller.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing Traceback (most recent call last): ... Exception: Something bad happened |
Example of gather() With Return Exceptions
We can prevent an exception in the group of awaitables from being re-raised in the caller by setting the “return_exceptions” argument to True when calling gather().
This will trap the exception and return it as a return value for the awaitable.
The example below demonstrates this, updating the above example to set the return_exceptions argument to True.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of gather with returned exceptions import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # check for failure if value == 0: raise Exception('Something bad happened') # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks results = await asyncio.gather(*coros, return_exceptions=True) # report results print(results) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and runs the main() coroutine as the entry point into the program.
The main() coroutine then creates 10 task coroutines in a list comprehension. These are then provided to a call to gather().
All coroutines are scheduled for execution along with the Future returned from gather().
The main() coroutine then suspends and waits for the tasks to complete.
All coroutines execute, reporting a message and blocking.
One coroutine then raises an exception.
The exception does not impact any other coroutines in the group and is not re-raised in the caller.
The caller retrieves a list of return values from all awaitables. It then reports the values.
We can see that all awaitables return None, except the one that failed, where we can see the Exception object that was returned.
This highlights that exceptions can be prevented from being re-raised and can be retrieved as a return value from a failed awaitable.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
main starting >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing [Exception('Something bad happened'), None, None, None, None, None, None, None, None, None] main done |
Example of gather() Where One Task is Canceled
Tasks can be canceled.
It is possible that we call gather() with one or more Tasks or Future objects that have been canceled.
If a task is canceled in a group, it will raise a CancelledError in the wrapped coroutine and this exception will be re-raised in the caller.
The example below demonstrates this.
Two tasks are created, and the second is provided as a reference to the first. I then cancel the other task.
Both tasks are then grouped with a call to gather and the CancelledError is re-raised in the caller.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of gather where one task is canceled import asyncio # coroutine used for a task async def task_coro(value, friend): # report a message print(f'>task {value} executing') # cancel friend task if friend: friend.cancel() # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create many tasks task0 = asyncio.create_task(task_coro(0, None)) task1 = asyncio.create_task(task_coro(1, task0)) # run the tasks await asyncio.gather(task0, task1) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and runs the main() coroutine as the entry point into the program.
The main() coroutine then creates one task with no paired task. This task is scheduled immediately.
A second task is created and provides the first task as a reference. This task too is scheduled immediately.
Both tasks are then provided to the gather() function and the main coroutine awaits both tasks.
The first task is given an opportunity to execute, report a message, and sleep.
The second task is given an opportunity to execute, it reports a message, then cancels the first task, then sleeps.
The first task resumes and raises a CancelledError exception.
The main() task resumes and re-raises the CancelledError that terminates the asyncio program.
This highlights that tasks in a call to gather() can be canceled which causes the CancelledError to be re-raised in the calling coroutine.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
main starting >task 0 executing >task 1 executing Traceback (most recent call last): ... asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): ... asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): ... asyncio.exceptions.CancelledError |
Example of gather() Where One Task is Canceled With Return Exceptions
It is possible to not re-raise a CancelledError for a canceled task in a call to gather() by setting the “return_exceptions” argument to True.
This will cause the CancelledError to be returned as a return value, just as though the task failed with any arbitrary exception.
The example below demonstrates this by updating the previous example to set the “return_exceptions” argument to True, then retrieve and report the return values from all awaitables.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of gather where one task is canceled with returned exceptions import asyncio # coroutine used for a task async def task_coro(value, friend): # report a message print(f'>task {value} executing') # cancel friend task if friend: friend.cancel() # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create many tasks task0 = asyncio.create_task(task_coro(0, None)) task1 = asyncio.create_task(task_coro(1, task0)) # run the tasks results = await asyncio.gather(task0, task1, return_exceptions=True) # report results print(results) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and runs the main() coroutine as the entry point into the program.
The main() coroutine then creates one task with no paired task. This task is scheduled immediately.
A second task is created and provides the first task as a reference. This task too is scheduled immediately.
Both tasks are then provided to the gather() function with return_exceptions set to True and the main coroutine awaits both tasks.
The first task is given an opportunity to execute, report a message, and sleep.
The second task is given an opportunity to execute, it reports a message, then cancels the first task, then sleeps.
The first task resumes and raises a CancelledError exception.
The main() task resumes and the return values from both tasks are retrieved and reported. We can see the first task returned a CancelledError and the second returned the default value of None.
This highlights that a task in the group can be canceled, but does not have to raise a CancelledError in the calling coroutine.
|
1 2 3 4 5 |
main starting >task 0 executing >task 1 executing [CancelledError(''), None] main done |
Example of Canceling All Tasks in gather()
As discussed, a call to gather() returns a Future object.
This Future object can be canceled by calling the cancel() method.
If canceled, a CancelledError exception is raised in the Future itself which in turn cancels all tasks in the group, if they are not done.
We can demonstrate this with a worked example.
In this example, we define a global variable that will reference the Future object returned from the call to gather(). Initially, it is set to None. It is then assigned within the main() coroutine.
Note, this is different from canceling all tasks in gather() if one task fails. For an example of this, see the tutorial:
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of canceling a gather with many coroutines import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # check if this is the special task if value == 0: global group group.cancel() # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # create a group of tasks global group group = asyncio.gather(*coros) # await the task await group # report a message print('main done') # define a global variable group = None # start the asyncio program asyncio.run(main()) |
Running the example first defines a global variable named “group” then creates the main() coroutine and uses it as the entry point to the program.
The main() coroutine runs, first reporting a message, then creating a list of coroutines. They are provided with a call to gather and the Future object that is returned is assigned to the “group” global variable.
All coroutines are scheduled as tasks at this point.
The main() coroutine then awaits the group and suspends, giving the coroutines an opportunity to execute.
The first coroutine executes, reporting a message. It then accesses the “group” global variable which is a Future object and cancels it.
This has the effect of raising a CancelledError exception in the Future object returned from the gather() function, and in turn, canceling all coroutines in the group.
The tasks terminate with the exception.
The main() coroutine returns and the CancelledError exception is raised, terminating the asyncio program.
This highlights that all awaitables in a group can be canceled.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
main starting >task 0 executing asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): ... asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): ... asyncio.exceptions.CancelledError |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Common Errors With asyncio.gather()
This section considers some of the more common errors when using the asyncio.gather() function.
Example of gather() Error With a List
Perhaps the most common error when using the gather() function occurs when providing a list of awaitables to the function.
For example:
|
1 2 3 4 5 |
... # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(coros) |
This is a problem because the gather() function does not take a list of awaitables, it takes multiple awaitables as separate positional arguments.
The example below demonstrates this common error.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of gather with a list that results in an error import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks await asyncio.gather(coros) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and runs the main() coroutine as the entry point into the program.
It reports a message and then creates a list of coroutines.
The coroutines are then provided with a call to the gather() function.
This fails with a TypeError.
The reason is that the gather() function expects separate awaitable arguments (e.g. a variable number of awaitable arguments), not a list of arguments.
|
1 2 3 4 5 |
main starting Traceback (most recent call last): ... TypeError: unhashable type: 'list' sys:1: RuntimeWarning: coroutine 'task_coro' was never awaited |
Example of gather() Error Without Await
Another common error is to call gather() but not give the tasks an opportunity to execute.
Recall that when we call gather(), all awaitables will be scheduled for execution.
We do not have to await the Future that is returned from the gather() function.
Nevertheless, we do need to give the awaitables an opportunity to execute.
This can be achieved by suspending the calling coroutine, such as by awaiting a call to sleep() or some other task.
If this is not done, the coroutines are scheduled and never executed and the program will exit.
The example below demonstrates this problem.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of gather without await results in an error import asyncio # coroutine used for a task async def task_coro(value): # report a message print(f'>task {value} executing') # sleep for a moment await asyncio.sleep(1) # coroutine used for the entry point async def main(): # report a message print('main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks asyncio.gather(*coros) # report a message print('main done') # start the asyncio program asyncio.run(main()) |
Running the example creates and runs the main() coroutine as the entry point into the program.
It reports a message and then creates a list of coroutines.
The coroutines are then provided with a call to the gather() function.
The main() coroutine then reports a message and exits, terminating the program.
The scheduled tasks are never given an opportunity to execute.
The garbage collector attempts to cancel the scheduled tasks and is reported before the program finishes terminating.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
main starting main done >task 0 executing >task 1 executing >task 2 executing >task 3 executing >task 4 executing >task 5 executing >task 6 executing >task 7 executing >task 8 executing >task 9 executing _GatheringFuture exception was never retrieved future: <_GatheringFuture finished exception=CancelledError()> Traceback (most recent call last): ... asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: asyncio.exceptions.CancelledError |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to run asyncio tasks concurrent with gather() in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Do you have any questions?