You can use the gather() function to wait for multiple coroutines to complete and then process their results, whereas you can use the wait() function to wait on a collection of tasks, such as all or the first to complete or the first to fail.
In this tutorial, you will know the similarities and differences between asyncio.gather() and asyncio.wait() and when to use each in Python.
Let’s get started.
What is asyncio.gather()
The asyncio.gather() function allows the caller to group multiple awaitables together.
Once grouped, the awaitables can be executed concurrently, awaited, and canceled.
Run awaitable objects in the aws sequence concurrently.
— Coroutines and Tasks
It is a helpful utility function for both grouping and executing multiple coroutines or multiple tasks.
For example:
|
1 2 3 |
... # run a collection of awaitables results = await asyncio.gather(coro1(), asyncio.create_task(coro2())) |
We may use the asyncio.gather() function in situations where we may create many tasks or coroutines up-front and then wish to execute them all at once and wait for them all to complete before continuing on.
This is a likely situation where the result is required from many like-tasks, e.g. same task or coroutine with different data.
The awaitables can be executed concurrently, results returned, and the main program can resume by making use of the results on which it is dependent.
The gather() function is more powerful than simply waiting for tasks to complete.
It allows a group of awaitables to be treated as a single awaitable.
This allows:
- Executing and waiting for all awaitables in the group to be done via an await expression.
- Getting results from all grouped awaitables to be retrieved later via the result() method.
- The group of awaitables to be canceled via the cancel() method.
- Checking if all awaitables in the group are done via the done() method.
- Executing callback functions only when all tasks in the group are done.
And more.
Now that we know what the asyncio.gather() function is, let’s look at how we might use it.
gather() Takes Tasks and Coroutines
The asyncio.gather() function takes one or more awaitables as arguments.
Recall an awaitable may be a coroutine, a Future or a Task.
Therefore, we can call the gather() function with:
- Multiple tasks
- Multiple coroutines
- Mixture of tasks and coroutines
For example:
|
1 2 3 |
... # execute multiple coroutines asyncio.gather(coro1(), coro2()) |
If Task objects are provided to gather(), they will already be running because Tasks are scheduled as part of being created.
The asyncio.gather() function takes awaitables as position arguments.
We cannot create a list or collection of awaitables and provide it to gather, as this will result in an error.
For example:
|
1 2 3 |
... # cannot provide a list of awaitables directly asyncio.gather([coro1(), coro2()]) |
A list of awaitables can be provided if it is first unpacked into separate expressions using the star operator (*).
For example:
|
1 2 3 |
... # gather with an unpacked list of awaitables asyncio.gather(*[coro1(), coro2()]) |
If coroutines are provided to gather(), they are wrapped in Task objects automatically.
gather() Returns a Future
The gather() function does not block.
Instead, it returns an asyncio.Future object that represents the group of awaitables.
For example:
|
1 2 3 |
... # get a future that represents multiple awaitables group = asyncio.gather(coro1(), coro2()) |
Once the Future object is created it is scheduled automatically within the event loop.
The awaitable represents the group, and all awaitables in the group will execute as soon as they are able.
This means that if the caller did nothing else, the scheduled group of awaitables will run (assuming the caller suspends).
It also means that you do not have to await the Future that is returned from gather().
For example:
|
1 2 3 4 5 |
... # get a future that represents multiple awaitables group = asyncio.gather(coro1(), coro2()) # suspend and wait a while, the group may be executing.. await asyncio.sleep(10) |
Awaiting gather()’s’ Future
The returned Future object can be awaited which will wait for all awaitables in the group to be done.
For example:
|
1 2 3 |
... # run the group of awaitables await group |
Awaiting the Future returned from gather() will return a list of return values from the awaitables.
If the awaitables do not return a value, then this list will contain the default “None” return value.
For example:
|
1 2 3 |
... # run the group of awaitables and get return values results = await group |
This is more commonly performed in one line.
For example:
|
1 2 3 |
... # run tasks and get results on one line results = await asyncio.gather(coro1(), coro2()) |
gather() Can Nest Groups of Awaitables
The gather() function takes awaitables and itself returns an awaitable.
Therefore, we can create nested groups of awaitables.
For example:
|
1 2 3 4 5 6 7 |
... # create a group of tasks group1 = asyncio.gather(coro1(), coro2()) # create another group of tasks group2 = asyncio.gather(group1, coro3()) # run group2 which will also run group1 await group2 |
gather() and Exceptions
If an awaitable fails with an exception, the exception is re-raised in the caller and may need to be handled.
For example:
|
1 2 3 4 5 6 |
... try: # run tasks and get results results = await asyncio.gather(coro1(), coro2()) except Exception as e: # ... |
Similarly, if a task in the group is canceled, it will re-raise a CancelledError exception in the caller and may need to be handled.
The “return_exceptions” argument to gather() can be set to True which will catch exceptions and provide them as return values instead of re-raising them in the caller.
This applies to both exceptions raised in awaitables, as well as CancelledError exceptions if the awaitables are canceled.
For example:
|
1 2 3 |
... # run tasks and retrieve exceptions as a result results = await asyncio.gather(coro1(), coro2(), return_exceptions=True) |
gather()’s Future Can Be Canceled
The Future that is returned from gather() can be used just like a normal asyncio.Future object.
We can check if it is done by calling the done() method.
For example:
|
1 2 3 4 |
... # check if a gather group of tasks is done if group.done(): # ... |
We can also cancel the gather()‘s Future, which will cancel all tasks within the group.
For example:
|
1 2 3 |
... # cancel all tasks in the group group.cancel() |
Other helpful methods we might use include adding a done callback function, getting a result, and checking if the group was canceled.
You can learn more about the asyncio.gather() function in the tutorial:
Next, let’s take a look at the asyncio.wait() function.
Run loops using all CPUs, download your FREE book to learn how.
What is asyncio.wait()
The asyncio.wait() function can be used to wait for a collection of asyncio tasks to complete.
Recall that an asyncio task is an instance of the asyncio.Task class that wraps a coroutine. It allows a coroutine to be scheduled and executed independently, and the Task instance provides a handle on the task for querying status and getting results.
You can learn more about asyncio tasks in the tutorial:
The wait() function allows us to wait for a collection of tasks to be done.
The call to wait can be configured to wait for different conditions, such as all tasks being completed, the first task completed and the first task failing with an error.
Next, let’s look at how we might use the wait() function.
wait() for Awaitables
The asyncio.wait() function takes a collection of awaitables, typically Task objects.
This could be a list, dict, or set of task objects that we have created, such as via calls to the asyncio.create() task function in a list comprehension.
For example:
|
1 2 3 |
... # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] |
The asyncio.wait() will not return until some condition on the collection of tasks is met.
By default, the condition is that all tasks are completed.
The wait() function returns a tuple of two sets. The first set contains all task objects that meet the condition, and the second contains all other task objects that do not yet meet the condition.
These sets are referred to as the “done” set and the “pending” set.
For example:
|
1 2 3 |
... # wait for all tasks to complete done, pending = await asyncio.wait(tasks) |
Technically, the asyncio.wait() is a coroutine function that returns a coroutine.
We can then await this coroutine which will return the tuple of sets.
For example:
|
1 2 3 4 5 |
... # create the wait coroutine wait_coro = asyncio.wait(tasks) # await the wait coroutine tuple = await wait_coro |
The condition waited for can be specified by the “return_when” argument which is set to asyncio.ALL_COMPLETED by default.
For example:
|
1 2 3 |
... # wait for all tasks to complete done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED) |
wait() for First Awaitable
We can wait for the first task to be completed by setting return_when to FIRST_COMPLETED.
For example:
|
1 2 3 |
... # wait for the first task to be completed done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) |
When the first task is complete and returned in the done set, the remaining tasks are not canceled and continue to execute concurrently.
wait() for First Failed Awaitable
We can wait for the first task to fail with an exception by setting return_when to FIRST_EXCEPTION.
For example:
|
1 2 3 |
... # wait for the first task to fail done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) |
In this case, the done set will contain the first task that failed with an exception. If no task fails with an exception, the done set will contain all tasks, and wait() will return only after all tasks are completed.
wait() With Timeout
We can specify how long we are willing to wait for the given condition via a “timeout” argument in seconds.
If the timeout expires before the condition is met, the tuple of tasks is returned with whatever subset of tasks do meet the condition at that time, e.g. the subset of tasks that are completed if waiting for all tasks to complete.
For example:
|
1 2 3 |
... # wait for all tasks to complete with a timeout done, pending = await asyncio.wait(tasks, timeout=3) |
If the timeout is reached before the condition is met, an exception is not raised and the remaining tasks are not canceled.
You can learn more about the asyncio.wait() function in the tutorial:
Next, let’s take a look at how the asyncio.gather() function compares to the asyncio.wait() function.
asyncio.gather() vs asyncio.wait()
The gather() and wait() functions have a lot of similarities and differences.
Let’s take a closer look at how these functions compare to each other.
Similarities
The asyncio.gather() and asyncio.wait() functions have a lot of similarities.
They are:
- Both functions suspend the caller until the provided tasks are done.
- Both functions take many async tasks as arguments.
- Both functions can take tasks or coroutines.
- Both functions are awaitable.
Generally, if we have multiple asyncio tasks and we want to suspend them until the tasks are complete, we can use either the gather() or wait() functions.
We may have asyncio.Task objects, or we may have coroutines to call. In the latter case, we can call either the wait() or gather() functions and they will automatically wrap the coroutines in asyncio.Task objects and schedules them for execution.
This was true, although, since Python 3.10, the wait() function will no longer wrap coroutines in tasks, instead preferring to take asyncio.Task objects (or Futures) as arguments.
Differences
Although the gather() and wait() functions have a lot in common, they are also quite different.
The main differences between the functions are as follows:
- wait() takes an iterable of tasks, whereas gather() takes separate expressions and does not support an iterable as an argument.
- gather() returns an awaitable (e.g. an asyncio.Future) whereas wait() is a coroutine function.
- wait() can be configured in terms of the condition it is waiting for, whereas the gather() function will only return when all tasks are done.
- wait() supports a timeout, whereas gather() does not.
- gather() supports coroutines and tasks, whereas wait() only supports tasks as arguments.
- gather() returns an iterable of results from coroutines and tasks, whereas wait() returns sets of done and not done tasks.
Generally the asyncio.gather() function is intended to wait for a specific series of coroutines to complete and return their values.
The asyncio.wait() function is intended to wait for a specific condition in a collection of tasks, such as all complete, the first to complete, or the first to fail.
The gather() function returns an awaitable that can be used to add callback functions to handle results once all tasks are complete. The wait() function is just a coroutine function that must be awaited.
When to Use asyncio.gather()
Consider using the gather() function when:
- You have multiple coroutines.
- You want to wait for all coroutines or tasks to complete.
- You want to process the return value results from the coroutines.
- You don’t have a time limit.
- You want to use a callback function to handle the results of all coroutines.
When to Use asyncio.wait()
Consider suing the wait() function when:
- You have a collection of tasks.
- You want to wait for all tasks to complete.
- You want to wait for the first task to complete.
- You want to wait for the first task to fail.
- You don’t need to return values from tasks as a first priority.
- You have a time limit on how long you can wait.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know the similarities and differences between asyncio.gather() and asyncio.wait() and when to use each in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Do you have any questions?