Last Updated on November 14, 2023
After issuing many tasks on asyncio, we may need to wait for a specific condition to occur in the group.
For example, we may want to wait until all tasks are complete, or for the first task to complete or fail and know which task it was.
This can be achieved via the asyncio.wait() function.
In this tutorial, you will discover how to wait for asyncio tasks to complete.
After completing this tutorial, you will know:
- How to use the wait() function to wait on a collection of asyncio tasks.
- How to configure the condition waited for by the wait() function.
- How to wait for a target condition with a timeout.
Let’s get started.
What is asyncio.wait()
The asyncio.wait() function can be used to wait for a collection of asyncio tasks to complete.
Recall that an asyncio task is an instance of the asyncio.Task class that wraps a coroutine. It allows a coroutine to be scheduled and executed independently, and the Task instance provides a handle on the task for querying status and getting results.
You can learn more about asyncio tasks in the tutorial:
The wait() function allows us to wait for a collection of tasks to be done.
The call to wait can be configured to wait for different conditions, such as all tasks being completed, the first task completed and the first task failing with an error.
Next, let’s look at how we might use the wait() function.
Run loops using all CPUs, download your FREE book to learn how.
How to Use asyncio.wait()
The asyncio.wait() function takes a collection of awaitables, typically Task objects.
This could be a list, dict or set of task objects that we have created, such as via calls to the asyncio.create() task function in a list comprehension.
For example:
|
1 2 3 |
... # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] |
The asyncio.wait() will not return until some condition on the collection of tasks is met.
By default, the condition is that all tasks are completed.
The wait() function returns a tuple of two sets. The first set contains all task objects that meet the condition, and the second contains all other task objects that do not yet meet the condition.
These sets are referred to as the “done” set and the “pending” set.
For example:
|
1 2 3 |
... # wait for all tasks to complete done, pending = await asyncio.wait(tasks) |
Technically, the asyncio.wait() is a coroutine function that returns a coroutine.
We can then await this coroutine which will return the tuple of sets.
For example:
|
1 2 3 4 5 |
... # create the wait coroutine wait_coro = asyncio.wait(tasks) # await the wait coroutine tuple = await wait_coro |
The condition waited for can be specified by the “return_when” argument which is set to asyncio.ALL_COMPLETED by default.
For example:
|
1 2 3 |
... # wait for all tasks to complete done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED) |
We can wait for the first task to be completed by setting return_when to FIRST_COMPLETED.
For example:
|
1 2 3 |
... # wait for the first task to be completed done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) |
When the first task is complete and returned in the done set, the remaining tasks are not canceled and continue to execute concurrently.
We can wait for the first task to fail with an exception by setting return_when to FIRST_EXCEPTION.
For example:
|
1 2 3 |
... # wait for the first task to fail done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) |
In this case, the done set will contain the first task that failed with an exception. If no task fails with an exception, the done set will contain all tasks and wait() will return only after all tasks are completed.
We can specify how long we are willing to wait for the given condition via a “timeout” argument in seconds.
If the timeout expires before the condition is met, the tuple of tasks is returned with whatever subset of tasks do meet the condition at that time, e.g. the subset of tasks that are completed if waiting for all tasks to complete.
For example:
|
1 2 3 |
... # wait for all tasks to complete with a timeout done, pending = await asyncio.wait(tasks, timeout=3) |
If the timeout is reached before the condition is met, an exception is not raised and the remaining tasks are not canceled.
Now that we know how to use the asyncio.wait() function, let’s look at some worked examples.
Example of Waiting for All Tasks
We can explore how to wait for all tasks using asyncio.wait().
In this example, we will define a simple task coroutine that generates a random value, sleeps for a fraction of a second, then reports a message with the generated value.
The main coroutine will then create many tasks in a list comprehension with the coroutine and then wait for all tasks to complete.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of waiting for all tasks to complete from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # report the value print(f'>task {arg} done with {value}') # main coroutine async def main(): # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] # wait for all tasks to complete done,pending = await asyncio.wait(tasks) # report results print('All done') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine then creates a list of ten tasks in a list comprehension, each providing a unique integer argument from 0 to 9.
The main() coroutine is then suspended and waits for all tasks to complete.
The tasks execute. Each generates a random value, sleeps for a moment, then reports its generated value.
After all tasks have been completed, the main() coroutine resumes and reports a final message.
This example highlights how we can use the wait() function to wait for a collection of tasks to be completed.
This is perhaps the most common usage of the function.
Note, that the results will differ each time the program is run given the use of random numbers.
|
1 2 3 4 5 6 7 8 9 10 11 |
>task 5 done with 0.0591009105682192 >task 8 done with 0.10453715687017351 >task 0 done with 0.15462838864295925 >task 6 done with 0.4103492027393125 >task 9 done with 0.45567100006991623 >task 2 done with 0.6984682905809402 >task 7 done with 0.7785363531316224 >task 3 done with 0.827386088873161 >task 4 done with 0.9481344994700972 >task 1 done with 0.9577302665040541 All done |
Next, let’s look at how we might wait for the first task to complete.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Waiting for First Task
We can explore how to wait for the first task to complete using asyncio.wait().
In this example, we will reuse the same simple task coroutine that generates a random value, sleeps for a fraction of a second, then reports a message with the generated value.
The main coroutine will then create many tasks in a list comprehension with the coroutine as before. It will then wait for the first task to complete.
Once a task is completed, it is retrieved from the set of done tasks and its details are reported.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of waiting for the first task to complete from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # report the value print(f'>task {arg} done with {value}') # main coroutine async def main(): # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] # wait for the first task to complete done,pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) # report result print('Done') # get the first task to complete first = done.pop() print(first) # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine then creates a list of ten tasks in a list comprehension, each providing a unique integer argument from 0 to 9.
The main() coroutine is then suspended and waits for the first task to complete
The tasks execute. Each generates a random value, sleeps for a moment, then reports its generated value.
As soon as the first task completes, the wait() function returns and the main() coroutine resumes.
The “done” set contains a single task that is finished, whereas the “pending” set contains all other tasks that were provided in the collection to the wait() function.
The single finished task is then retrieved from the “done” set and is reported. The printed status of the task confirms that indeed it is done.
The other tasks are not canceled and continue to run concurrently. Their execution is cut short because the asyncio program is terminated.
This example highlights how we can use the wait() function to wait for the first task to complete.
Note, that the results will differ each time the program is run given the use of random numbers.
|
1 2 3 |
>task 9 done with 0.04034360933451242 Done <Task finished name='Task-11' coro=<task_coro() done, defined at ...> result=None> |
Next, let’s look at how we might wait for the first task to fail with an exception.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Waiting for First Failure
We can explore how to wait for the first task to fail using asyncio.wait().
In this example, we will reuse the same simple task coroutine that generates a random value, sleeps for a fraction of a second, then reports a message with the generated value.
The task coroutine is updated so that conditionally it will fail with an exception if the generated value is below a threshold.
The main coroutine will then create many tasks. It will then wait for the first task to fail with an exception.
Once a task has failed, it is retrieved from the set of done tasks and its details are reported.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of waiting for the first task to fail from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # report the value print(f'>task {arg} done with {value}') # conditionally fail if value < 0.5: raise Exception(f'Something bad happened in {arg}') # main coroutine async def main(): # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] # wait for the first task to fail, or all tasks to complete done,pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION) # report result print('Done') # get the first task to fail first = done.pop() print(first) # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine then creates a list of ten tasks in a list comprehension, each providing a unique integer argument from 0 to 9.
The main() coroutine is then suspended and waits for the first task to complete
The tasks execute. Each generates a random value, sleeps for a moment, then reports its generated value.
Conditionally, some tasks raise an exception if their generated value is above a threshold value. It is possible that all tasks generate a value below the threshold and none raise an exception, although this situation is extremely unlikely.
As soon as the first task fails with an exception, the wait() function returns and the main() coroutine resumes.
The “done” set contains a single task that failed first, whereas the “pending” set contains all other tasks that were provided in the collection to the wait() function.
The single failed task is then retrieved from the “done” set and is reported. The printed status of the task confirms that it indeed failed with an exception in this case.
The other tasks are not canceled and continue to run concurrently. Their execution is cut short because the asyncio program is terminated.
This example highlights how we can use the wait() function to wait for the first task to fail.
Note, that the results will differ each time the program is run given the use of random numbers.
|
1 2 3 |
>task 5 done with 0.13168449673381888 Done <Task finished name='Task-7' coro=<task_coro() done, defined at ...> exception=Exception('Something bad happened in 5')> |
Next, let’s look at how we might wait for tasks with a timeout.
Example of Waiting with a Timeout
We can explore how to wait for all tasks to complete with a timeout using asyncio.wait().
In this example, we will reuse the same simple task coroutine that generates a random value, sleeps for a fraction of a second, then reports a message with the generated value.
The main coroutine will then create many tasks and wait for all tasks to complete. In this case, a timeout is specified, setting an upper limit on how long the caller is willing to wait.
Once all tasks are complete, or the timeout expires, the sets of done and pending tasks are returned. The total number of tasks completed within the timeout is then reported.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of waiting for all tasks to be completed with a timeout from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 10 value = random() * 10 # block for a moment await asyncio.sleep(value) # report the value print(f'>task {arg} done with {value}') # main coroutine async def main(): # create many tasks tasks = [asyncio.create_task(task_coro(i)) for i in range(10)] # wait for all tasks to complete done,pending = await asyncio.wait(tasks, timeout=5) # report results print(f'Done, {len(done)} tasks completed in time') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine then creates a list of ten tasks in a list comprehension, each providing a unique integer argument from 0 to 9.
The main() coroutine is then suspended and waits for all tasks to complete.
It specifies a timeout of 5 seconds. This is about half the duration of the longest tasks that could be generated.
The tasks execute. Each generates a random value between 0 and 10, sleeps for the generated number of seconds, then reports its generated value.
The timeout expires and the main() coroutine resumes. It reports a message indicating the number of tasks that were completed within the timeout, which was 6 in this case.
This example highlights how we can use the wait() function to wait for a collection of tasks to be completed with a timeout.
Note, that the results will differ each time the program is run given the use of random numbers.
|
1 2 3 4 5 6 7 |
>task 7 done with 0.16485206249285955 >task 0 done with 0.73241529734688 >task 4 done with 1.1137310743743878 >task 6 done with 2.396915437441108 >task 5 done with 3.375537014759735 >task 2 done with 4.821848023696365 Done, 6 tasks completed in time |
Example of Waiting for Coroutines
We can explore waiting for coroutines instead of tasks with asyncio.wait().
The wait() function is used to support coroutines as well as tasks. It was since updated to only support tasks.
Support for coroutines in the wait() function is deprecated at the time of writing and will result in a DeprecationWarning in Python 3.10. This support will soon be removed, in Python 3.11.
The example below creates a list of coroutines and then waits for them all to complete.
If you are using Python 3.10 or lower, the coroutines are converted into tasks and scheduled for execution.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of waiting for all coroutines to complete from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # report the value print(f'>task {arg} done with {value}') # main coroutine async def main(): # create many coroutines tasks = [task_coro(i) for i in range(10)] # wait for all tasks to complete done,pending = await asyncio.wait(tasks) # report results print('All done') # start the asyncio program asyncio.run(main()) |
Running the program assumes you are using Python 3.10 or lower. If not, please let me know the results you see in the comments below.
In this case, we can see that a DeprecationWarning is reported.
The example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine then creates a list of ten coroutines (not tasks) in a list comprehension, each providing a unique integer argument from 0 to 9.
The main() coroutine is then suspended and waits for all coroutines to complete. The wait() function converts each coroutine into a Task, which is then scheduled for execution and then awaited.
The tasks execute. Each generates a random value, sleeps for a moment, then reports its generated value.
After all tasks have been completed, the main() coroutine resumes and reports a final message.
This example highlights how we can use the wait() function to wait for a collection of coroutines to complete.
Note, that the results will differ each time the program is run given the use of random numbers.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11. done,pending = await asyncio.wait(tasks) >task 3 done with 0.05716637980412176 >task 2 done with 0.21893288741084582 >task 9 done with 0.2702814255302467 >task 4 done with 0.2989980107703637 >task 6 done with 0.40462737710888264 >task 8 done with 0.5675796788798889 >task 5 done with 0.6219456587051952 >task 1 done with 0.8236981712898166 >task 7 done with 0.8408137669173977 >task 0 done with 0.9844164290735692 All done |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to wait for asyncio tasks to complete in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Chris Stein on Unsplash
You’re the best. You have found a balance between providing enough detail to understand what python is doing under the hood, while also providing easy to understand examples.
Thanks again
Thank you kindly Tim!