You can develop an asynchronous for-loop in asyncio so all tasks run concurrently.
There are many ways to develop an async for-loop, such as using asyncio.gather(), use asyncio.wait(), use asyncio.as_completed(), create and await a list of asyncio.Task objects and use an asyncio.TaskGroup.
In this tutorial, you will discover how to execute an asyncio for loop concurrently.
Let’s get started.
Need to Make Asyncio For Loop Concurrent
You have a for-loop in your asyncio program and you want to execute each iteration concurrently, e.g. at the same time.
This is a common situation.
The loop involves performing the same operation multiple times with different data.
For example:
|
1 2 3 4 |
... # perform the same operation on each item for item in items: # perform operation on item... |
It most commonly involves awaiting the same coroutine in each iteration with different arguments.
For example:
|
1 2 3 4 5 |
... # await the same coroutine each iteration with different data for item in items: # await a coroutine with one data item await custom_coro(item) |
Each iteration of the for-loop is performed sequentially.
This is a problem because the whole idea of using asyncio is to run many tasks simultaneously.
How can we convert a for-loop to be concurrent in asyncio?
Run loops using all CPUs, download your FREE book to learn how.
How to Develop An Async For-Loop
There are many ways to execute an asyncio for loop concurrently.
The most common ways to develop an async for loop are as follows:
- Method 01. Async for-loop with asyncio.gather()
- Method 02. Async for-loop with asyncio.wait()
- Method 03. Async for-loop with asyncio.as_completed()
- Method 04. Async for-loop with list comprehension of Tasks
- Method 05. Async for-loop with an asyncio.TaskGroup
Do you know of another approach?
Let me know in the comments below.
Let’s take a look at each approach in turn.
Method 01. Async for-loop with asyncio.gather()
We can develop an async for loop using asyncio.gather().
The asyncio.gather() function takes one or more coroutines or tasks and will suspend them until they are all completed.
This can be achieved in a few ways.
One way is to first create a list of coroutine objects, then expand this list as an argument to the asyncio.gather() function using the star operator (*).
Note, that the asyncio.gather() does not take a collection of coroutines directly, rather they are provided as positional arguments, therefore any collection provided as an argument must be expanded first, e.g. why we must use the star operator.
For example:
|
1 2 3 4 5 |
... # create all coroutines coros = [work(i) for i in range(10)] # wait for all tasks to complete concurrently await asyncio.gather(*coros) |
This will allow all coroutines to run concurrently.
You can learn more about how to use asyncio.gather() in the tutorial:
Method 02. Async for-loop with asyncio.wait()
We can develop an async for loop using asyncio.wait().
The asyncio.wait() takes a collection of tasks and will suspend until some condition is met. The default condition is that all provided tasks are done.
It then returns a tuple with all tasks that match the condition and all those that do not, which we can ignore in this case.
This means they must first schedule the coroutines as tasks via asyncio.create_task(), then provide them to asyncio.wait().
For example:
|
1 2 3 4 5 |
... # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] # wait for all tasks to complete concurrently _ = await asyncio.wait(tasks) |
This will allow all coroutines to run concurrently.
You can learn more about how to use asyncio.wait() in the tutorial:
Method 03. Async for-loop with asyncio.as_completed()
We can develop an async for loop asyncio.as_completed().
The asyncio.as_completed() takes a collection of coroutines or tasks as an argument and then yields the provided tasks in the order that they are completed.
This approach is preferred if we need to retrieve results for each task in the loop or report progress.
For example:
|
1 2 3 4 5 6 7 8 9 |
... # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] # report results as tasks complete for task in asyncio.as_completed(tasks): # get task result result = await task # report result print(f'>task done with {result}') |
This will allow all coroutines to run concurrently and report results in the order that tasks are completed.
You can learn more about how to use asyncio.as_completed() in the tutorial:
Method 04. Async for-loop with list comprehension of Tasks
We can develop an async for loop using a list comprehension of asyncio.Task objects.
This involves creating and scheduling each coroutine as an asyncio.Task.
The caller can then suspend and allow all tasks to run. It can await each issued task in turn, to confirm that the tasks are all done.
For example:
|
1 2 3 4 5 6 |
... # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] # wait on tasks one by one for task in tasks: await task |
This will allow all coroutines to run concurrently.
Method 05. Async for-loop with an asyncio.TaskGroup
We can develop an async for loop using an asyncio.TaskGroup.
An asyncio.TaskGroup is a way of creating asyncio.Task in a group and waiting for them to be done using a context manager interface.
For example:
|
1 2 3 4 5 6 |
... # create task group async with asyncio.TaskGroup() as group: # create all tasks _ = [group.create_task(work(i)) for i in range(10)] # wait for tasks to complete... |
This will allow all coroutines to run concurrently.
You can learn more about how to use asyncio.TaskGroup in the tutorial:
Next, let’s consider why we cannot use the “async for” expression to develop an asynchronous for loop.
We Cannot Use The “async for” Expression
We cannot develop an asynchronous for loop using the “async for” expression, at least not directly.
The “async for” expression provides a way to traverse an asynchronous generator or an asynchronous iterator.
It is a type of asynchronous for-loop where the target iterator is awaited each iteration which yields a value.
For example:
|
1 2 3 4 |
... # traverse an asynchronous iterator async for item in async_iterator: print(item) |
It can also be used in a list comprehension:
|
1 2 3 |
... # build a list of results results = [item async for item async_iterator] |
It cannot be used to traverse a list of awaitables, such as a list of coroutines or a list of asyncio.Task objects.
For example, the following will result in an error because the list is not an asynchronous generator or an asynchronous iterator:
|
1 2 3 4 5 6 |
... # create a number of coroutines coros = [simple_task(i) for i in range(10)] # traverse the iterable of awaitables async for item in coros: print(result) |
The error would look like:
|
1 |
TypeError: 'async for' requires an object with __aiter__ method, got list |
You can learn more about the async for expression in the tutorial:
The “async for” expression can be used for asynchronous generators.
These are generators that yield a value each time they are awaited.
For example:
|
1 2 3 4 5 6 7 8 |
# define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i |
You can learn more about asynchronous generators in the tutorial:
The “async for” expression can be used for asynchronous iterators.
These are iterators that yield a value each time they are awaited.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# define an asynchronous iterator class CustomIterator(): # constructor, define some state def __init__(self): self.counter = 0 # create an instance of the iterator def __aiter__(self): return self # return the next awaitable async def __anext__(self): # check for no further items if self.counter >= 10: raise StopAsyncIteration # block to simulate work await asyncio.sleep(1) # increment the counter self.counter += 1 # return the counter value return self.counter |
You can learn more about asynchronous generators in the tutorial:
Now that we know how to develop async for loops, let’s look at some worked examples.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Sequential Asyncio For Loop
Before we develop async for loops, let’s look at a sequential for loop in an asyncio program.
In this example, we will define a coroutine that performs a task. We will then call this coroutine many times in a loop with different data.
Each call will be made sequentially, one after the other in a normal for loop.
This will provide a helpful baseline for comparison and contrast to the asynchronous for loops we will develop later.
Firstly, we can define the coroutine that will perform some task with the provided data.
The work() coroutine below implements this taking a data argument, sleeping for a second to simulate work, and reporting a message with the data argument.
|
1 2 3 4 5 6 |
# async task async def work(data): # simulate some work await asyncio.sleep(1) # report a message print(f'>task done with {data}') |
Next, we can define the main() coroutine.
It first reports a message, then loops 10 times and awaits the work() coroutine with a different argument.
This loop is sequential.
Once the loop is complete, a final message is reported.
|
1 2 3 4 5 6 7 8 9 10 |
# main coroutine async def main(): # report a message print('Main starting') # complete many tasks for data in range(10): # complete one task await work(data) # report a message print('Main done') |
Because each task will block for one second, we expect the loop to take at least 10 seconds to complete sequentially.
Finally, we can start the asyncio event loop and run the main() coroutine.
|
1 2 3 |
... # start the event loop asyncio.run(main()) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of sequential for loop in asyncio import asyncio # async task async def work(data): # simulate some work await asyncio.sleep(1) # report a message print(f'>task done with {data}') # main coroutine async def main(): # report a message print('Main starting') # complete many tasks for data in range(10): # complete one task await work(data) # report a message print('Main done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message. It then loops 10 times, each iteration awaiting the work() coroutine with different data.
The work() coroutine runs, sleeps for one second, then reports a message.
Once the loop is complete, a final message is reported and the program is terminated.
This highlights a sequential loop of async tasks. As expected, the program takes about 10 seconds to complete, e.g. 10 x 1 second tasks performed sequentially.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting >task done with 0 >task done with 1 >task done with 2 >task done with 3 >task done with 4 >task done with 5 >task done with 6 >task done with 7 >task done with 8 >task done with 9 Main done |
Next, let’s look at how to develop an async for loop using asyncio.gather().
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Method 01: Async for-loop with asyncio.gather()
We can explore how to develop an async for loop using asyncio.gather().
In this case, we can update the above example to first create a list of work() coroutines, each with different data as an argument.
|
1 2 3 |
... # create all coroutines coros = [work(i) for i in range(10)] |
We can then expand the list and provide it to the asyncio.gather() function as an argument.
This will execute all work() coroutines concurrently.
|
1 2 3 |
... # wait for all tasks to complete concurrently await asyncio.gather(*coros) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of async for loop with asyncio.gather() import asyncio # async task async def work(data): # simulate some work await asyncio.sleep(1) # report a message print(f'>task done with {data}') # main coroutine async def main(): # report a message print('Main starting') # create all coroutines coros = [work(i) for i in range(10)] # wait for all tasks to complete concurrently await asyncio.gather(*coros) # report a message print('Main done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of coroutines in a list comprehension.
The list of coroutine objects is then expanded using the star operator and provided to the asyncio.gather() function, and awaited.
All 10 work() coroutines run concurrently, sleeping, and reporting their message.
Once all tasks are done and the asyncio.gather() returns. The main() coroutine resumes and reports a final message.
This highlights how we can develop an async for loop by running all async tasks concurrently with asyncio.gather().
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting >task done with 0 >task done with 1 >task done with 2 >task done with 3 >task done with 4 >task done with 5 >task done with 6 >task done with 7 >task done with 8 >task done with 9 Main done |
Next, let’s look at how to develop an async for loop using asyncio.wait().
Example of Method 02: Async for-loop with asyncio.wait()
We can explore how to develop an async for loop using asyncio.wait().
In this case, we can update the above example to first create a list of work() coroutines asyncio.Task instances, each with different data as an argument.
The tasks are scheduled, but not yet running.
|
1 2 3 |
... # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] |
We can then await the group of asyncio.Task objects via the asyncio.wait() function.
|
1 2 3 |
... # wait for all tasks to complete concurrently _ = await asyncio.wait(tasks) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of async for loop with asyncio.wait() import asyncio # async task async def work(data): # simulate some work await asyncio.sleep(1) # report a message print(f'>task done with {data}') # main coroutine async def main(): # report a message print('Main starting') # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] # wait for all tasks to complete concurrently _ = await asyncio.wait(tasks) # report a message print('Main done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of asyncio.Task objects in a list comprehension, one for each work() coroutine with different data.
The asyncio.Task objects are scheduled in the event loop, but not yet running.
The list of tasks is then provided to the asyncio.gather() function, and awaited.
All 10 work() coroutines run concurrently, sleeping, and reporting their message.
Once all tasks are done the asyncio.wait() returns. The main() coroutine resumes and reports a final message.
This highlights how we can develop an async for loop by running all async coroutines as concurrent tasks that are awaited with asyncio.wait().
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting >task done with 0 >task done with 1 >task done with 2 >task done with 3 >task done with 4 >task done with 5 >task done with 6 >task done with 7 >task done with 8 >task done with 9 Main done |
Next, let’s look at how to develop an async for loop using asyncio.as_completed().
Example of Method 03: Async for-loop with asyncio.as_completed()
We can explore how to develop an async for loop using asyncio.as_completed().
In this case, we can update the above example so that the work() coroutine returns a value.
|
1 2 3 4 5 6 |
# async task async def work(data): # simulate some work await asyncio.sleep(1) # return a value return data |
We can then create a list of work() tasks in a list comprehension, as we did above.
|
1 2 3 |
... # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] |
These tasks can then be provided to asyncio.as_completed() to be executed concurrently, which will yield each asyncio.Task instance as it is done. We can retrieve the return value from each task and report it.
|
1 2 3 4 5 6 7 |
... # report results as tasks complete for task in asyncio.as_completed(tasks): # get task result result = await task # report result print(f'>task done with {result}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of async for loop with asyncio.as_completed() import asyncio # async task async def work(data): # simulate some work await asyncio.sleep(1) # return a value return data # main coroutine async def main(): # report a message print('Main starting') # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] # report results as tasks complete for task in asyncio.as_completed(tasks): # get task result result = await task # report result print(f'>task done with {result}') # report a message print('Main done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of asyncio.Task objects in a list comprehension, one for each work() coroutine with different data.
The asyncio.Task objects are scheduled in the event loop, but not yet running.
The list of tasks is then provided to the asyncio.as_completed() function.
All work() tasks run concurrently, sleeping and reporting their message.
The asyncio.as_completed() function yields one work() task at a time, in the order they are completed. Each is awaited in order to retrieve the return value result, which is reported.
This highlights how we can develop an async for loop using asyncio.as_completed().
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting >task done with 0 >task done with 1 >task done with 2 >task done with 3 >task done with 4 >task done with 5 >task done with 6 >task done with 7 >task done with 8 >task done with 9 Main done |
Next, let’s look at how to develop an async for loop using a list comprehension of Tasks.
Example of Method 04: Async for-loop with list comprehension of Tasks
We can explore how to develop an async for loop using a list comprehension of Tasks.
In this case, we will create a list of asyncio.Task objects, as we did before.
This schedules the tasks for execution in the event loop but does not yet start their execution.
|
1 2 3 |
... # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] |
We can then enumerate the list of tasks in order, and await each in turn.
|
1 2 3 4 |
... # wait on tasks one by one for task in tasks: await task |
This suspends the main() coroutine, allowing all tasks to run concurrently.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of async for loop with task list comprehension import asyncio # async task async def work(data): # simulate some work await asyncio.sleep(1) # report a message print(f'>task done with {data}') # main coroutine async def main(): # report a message print('Main starting') # create all tasks tasks = [asyncio.create_task(work(i)) for i in range(10)] # wait on tasks one by one for task in tasks: await task # report a message print('Main done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of asyncio.Task objects in a list comprehension, one for each work() coroutine with different data.
The asyncio.Task objects are scheduled in the event loop, but not yet running.
The main() coroutine then enumerates the list of task objects in order and awaits each in turn.
This suspends the main() coroutine and allows each task to run, sleeping and reporting its message.
The main() coroutine resumes as each task is awaited in order, returning immediately if tasks are already done otherwise suspending until the task is done.
This is similar to the asyncio.as_completed() approach, except the order of the tasks matches the order they were created and scheduled, not the order they are completed (if they were completed at different times).
This highlights how we can develop an async for loop using a list comprehension of asyncio.Task objects.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting >task done with 0 >task done with 1 >task done with 2 >task done with 3 >task done with 4 >task done with 5 >task done with 6 >task done with 7 >task done with 8 >task done with 9 Main done |
Next, let’s look at how to develop an async for loop using an asyncio.TaskGroup.
Example of Method 05: Async for-loop with an asyncio.TaskGroup
We can explore how to develop an async for loop using an asyncio.TaskGroup.
In this case, we can update the above example to create an asyncio.TaskGroup using the context manager interface.
This asyncio.TaskGroup can then be used to create an asyncio.Task for each work() coroutine with different data, in a list comprehension.
|
1 2 3 4 5 6 |
... # create task group async with asyncio.TaskGroup() as group: # create all tasks _ = [group.create_task(work(i)) for i in range(10)] # wait for tasks to complete... |
This schedules each work() coroutine as a task. On exiting the context manager, the main() coroutine suspends and waits for all tasks in the group to be done.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of async for loop with task group import asyncio # async task async def work(data): # simulate some work await asyncio.sleep(1) # report a message print(f'>task done with {data}') # main coroutine async def main(): # report a message print('Main starting') # create task group async with asyncio.TaskGroup() as group: # create all tasks _ = [group.create_task(work(i)) for i in range(10)] # wait for tasks to complete... # report a message print('Main done') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
An asyncio.TaskGroup is then created using the context manager interface.
The main() coroutine then creates all 10 tasks using the asyncio.TaskGroup in a list comprehension.
Exiting the asyncio.TaskGroup context manager suspends the main() coroutine and blocks until all tasks created using the group are done.
All tasks run concurrently, sleeping, reporting their message and terminating normally.
Once all tasks are done, the main() coroutine resumes and exits the asyncio.TaskGroup context manager completely. It reports a final message and terminates.
This highlights how we can develop an async for loop using the asyncio.TaskGroup.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Main starting >task done with 0 >task done with 1 >task done with 2 >task done with 3 >task done with 4 >task done with 5 >task done with 6 >task done with 7 >task done with 8 >task done with 9 Main done |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to execute an asyncio for loop concurrently.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Brother Swagler on Unsplash
Do you have any questions?