You can wait for many independent tasks to complete by first getting the set of all running tasks, removing the current task, then waiting on the remaining tasks in the set.
This will allow an asyncio program to schedule many independent tasks throughout its lifetime and for the main coroutine to wait for them to complete without exiting and closing the event loop, terminating the tasks early.
In this tutorial, you will discover how to wait for all asyncio background tasks to complete in Python.
Let’s get started.
Need to Wait for All Background Tasks in Asyncio
It is common in an asyncio to issue many tasks.
The asyncio.create_task() function can be used to wrap a coroutine in an asyncio.Task and schedule it for execution independent of the calling coroutine.
This allows the caller to continue on with other activities and for the task to run as soon as it is able.
For example:
|
1 2 3 |
... # schedule a task task = asyncio.create_task(coro()) |
You can learn more about creating independent tasks in the tutorial:
May may issue many tasks that all run in the background.
This may be in a loop or periodically throughout our code.
It is possible to exit the main() coroutine while independent tasks are still running in the background.
This will terminate the asyncio event loop and all running tasks.
Instead, we would prefer to wait for any remaining tasks to complete before exiting the event loop.
How can we wait for background tasks in asyncio?
Run loops using all CPUs, download your FREE book to learn how.
How to Wait for All Background Tasks
We can wait for all independent tasks in an asyncio program.
This can be achieved by first getting a set of all currently running tasks via the asyncio.all_tasks() function.
For example:
|
1 2 3 |
... # get a set of all running tasks all_tasks = asyncio.all_tasks() |
You can learn more about getting all running tasks in the tutorial:
This will return a set that contains one asyncio.Task object for each task that is currently running, including the main() coroutine.
We cannot wait on this set directly, as it will block forever as it includes the task that is the current task.
Therefore we can get the asyncio.Task object for the currently running task and remove it from the set.
This can be achieved by first calling the asyncio.current_task() method to get the task for the current coroutine and then remove it from the set via the remove() method.
For example:
|
1 2 3 4 5 |
... # get the current tasks current_task = asyncio.current_task() # remove the current task from the list of all tasks all_tasks.remove(current_task) |
You can learn more about getting the current task in the tutorial:
Finally, we can wait on the set of remaining tasks.
This will suspend the caller until all tasks in the set are complete.
For example:
|
1 2 3 |
... # suspend until all tasks are completed await asyncio.wait(all_tasks) |
You can learn more about waiting on a collection of tasks in the tutorial:
Tying this together, the snippet below added to the end of the main() coroutine will wait for all background tasks to complete.
|
1 2 3 4 5 6 7 8 9 |
... # get a set of all running tasks all_tasks = asyncio.all_tasks() # get the current tasks current_task = asyncio.current_task() # remove the current task from the list of all tasks all_tasks.remove(current_task) # suspend until all tasks are completed await asyncio.wait(all_tasks) |
Now that we know how to wait for all background tasks to complete, let’s look at a worked example.
Example of Waiting for All Background Tasks
We can explore an example of waiting for all background tasks to complete before exiting.
In this example, we will define a task that will sleep for a random number of seconds and then report a message.
We will issue 20 of these tasks to run independently from the main() coroutine and then wait for them to complete using the approach developed in the previous section.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# SuperFastPython.com # example of waiting for all background tasks in asyncio import random import asyncio # coroutine run in the background async def task(number): # generate a random value between 0 and 10 value = random.random() * 10.0 # suspend for some time await asyncio.sleep(value) # report done print(f'>Task {number} done') # main coroutine async def main(): # start many background tasks for i in range(20): asyncio.create_task(task(i)) # allow tasks to start await asyncio.sleep(0) # get a set of all running tasks all_tasks = asyncio.all_tasks() # get the current tasks current_task = asyncio.current_task() # remove the current task from the list of all tasks all_tasks.remove(current_task) # report a message print(f'Main waiting for {len(all_tasks)} tasks...') # suspend until all tasks are completed await asyncio.wait(all_tasks) # run the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.
The main() coroutine runs and schedules 20 tasks for execution, each assigned a unique integer argument from 0 to 19.
The main() coroutine then suspends for zero seconds, allowing the scheduling tasks to run.
The tasks run, first generating a random number between 0 and 10 and then sleeping for that many seconds.
The main() coroutine resumes and then gets a set of all running tasks, which includes all 20 tasks. It then gets the task for itself and removes it from the set before finally suspending and waiting for all tasks to be complete.
Each task completes its sleep and reports a message, then terminates.
Once all tasks are complete, the main() coroutine resumes and terminates.
This highlights how an asyncio program may issue many tasks to run independently in the background and how the main() coroutine can wait for them to complete before closing the event loop.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
Main waiting for 20 tasks... >Task 8 done >Task 1 done >Task 19 done >Task 17 done >Task 4 done >Task 12 done >Task 5 done >Task 6 done >Task 3 done >Task 2 done >Task 7 done >Task 13 done >Task 16 done >Task 18 done >Task 9 done >Task 10 done >Task 15 done >Task 14 done >Task 0 done >Task 11 done |
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to wait for all asyncio background tasks to complete in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Kenny Eliason on Unsplash
Hello,
It’s very useful. Thanks a lot!
Thank you, I’m happy to hear that!