SuperFastPython

Async For Loop in Python

February 6, 2024 Python Asyncio

You can develop an asynchronous for-loop in asyncio so all tasks run concurrently.

There are many ways to develop an async for-loop, such as using asyncio.gather(), use asyncio.wait(), use asyncio.as_completed(), create and await a list of asyncio.Task objects and use an asyncio.TaskGroup.

In this tutorial, you will discover how to execute an asyncio for loop concurrently.

Let's get started.

Need to Make Asyncio For Loop Concurrent

You have a for-loop in your asyncio program and you want to execute each iteration concurrently, e.g. at the same time.

This is a common situation.

The loop involves performing the same operation multiple times with different data.

For example:

...
# perform the same operation on each item
for item in items:
	# perform operation on item...

It most commonly involves awaiting the same coroutine in each iteration with different arguments.

For example:

...
# await the same coroutine each iteration with different data
for item in items:
	# await a coroutine with one data item
	await custom_coro(item)

Each iteration of the for-loop is performed sequentially.

This is a problem because the whole idea of using asyncio is to run many tasks simultaneously.

How can we convert a for-loop to be concurrent in asyncio?

How to Develop An Async For-Loop

There are many ways to execute an asyncio for loop concurrently.

The most common ways to develop an async for loop are as follows:

Do you know of another approach?
Let me know in the comments below.

Let's take a look at each approach in turn.

Method 01. Async for-loop with asyncio.gather()

We can develop an async for loop using asyncio.gather().

The asyncio.gather() function takes one or more coroutines or tasks and will suspend them until they are all completed.

This can be achieved in a few ways.

One way is to first create a list of coroutine objects, then expand this list as an argument to the asyncio.gather() function using the star operator (*).

Note, that the asyncio.gather() does not take a collection of coroutines directly, rather they are provided as positional arguments, therefore any collection provided as an argument must be expanded first, e.g. why we must use the star operator.

For example:

...
# create all coroutines
coros = [work(i) for i in range(10)]
# wait for all tasks to complete concurrently
await asyncio.gather(*coros)

This will allow all coroutines to run concurrently.

You can learn more about how to use asyncio.gather() in the tutorial:

Method 02. Async for-loop with asyncio.wait()

We can develop an async for loop using asyncio.wait().

The asyncio.wait() takes a collection of tasks and will suspend until some condition is met. The default condition is that all provided tasks are done.

It then returns a tuple with all tasks that match the condition and all those that do not, which we can ignore in this case.

This means they must first schedule the coroutines as tasks via asyncio.create_task(), then provide them to asyncio.wait().

For example:

...
# create all tasks
tasks = [asyncio.create_task(work(i)) for i in range(10)]
# wait for all tasks to complete concurrently
_ = await asyncio.wait(tasks)

This will allow all coroutines to run concurrently.

You can learn more about how to use asyncio.wait() in the tutorial:

Method 03. Async for-loop with asyncio.as_completed()

We can develop an async for loop asyncio.as_completed().

The asyncio.as_completed() takes a collection of coroutines or tasks as an argument and then yields the provided tasks in the order that they are completed.

This approach is preferred if we need to retrieve results for each task in the loop or report progress.

For example:

...
# create all tasks
tasks = [asyncio.create_task(work(i)) for i in range(10)]
# report results as tasks complete
for task in asyncio.as_completed(tasks):
    # get task result
    result = await task
    # report result
    print(f'>task done with {result}')

This will allow all coroutines to run concurrently and report results in the order that tasks are completed.

You can learn more about how to use asyncio.as_completed() in the tutorial:

Method 04. Async for-loop with list comprehension of Tasks

We can develop an async for loop using a list comprehension of asyncio.Task objects.

This involves creating and scheduling each coroutine as an asyncio.Task.

The caller can then suspend and allow all tasks to run. It can await each issued task in turn, to confirm that the tasks are all done.

For example:

...
# create all tasks
tasks = [asyncio.create_task(work(i)) for i in range(10)]
# wait on tasks one by one
for task in tasks:
    await task

This will allow all coroutines to run concurrently.

Method 05. Async for-loop with an asyncio.TaskGroup

We can develop an async for loop using an asyncio.TaskGroup.

An asyncio.TaskGroup is a way of creating asyncio.Task in a group and waiting for them to be done using a context manager interface.

For example:

...
# create task group
async with asyncio.TaskGroup() as group:
    # create all tasks
    _ = [group.create_task(work(i)) for i in range(10)]
# wait for tasks to complete...

This will allow all coroutines to run concurrently.

You can learn more about how to use asyncio.TaskGroup in the tutorial:

Next, let's consider why we cannot use the "async for" expression to develop an asynchronous for loop.

We Cannot Use The "async for" Expression

We cannot develop an asynchronous for loop using the "async for" expression, at least not directly.

The "async for" expression provides a way to traverse an asynchronous generator or an asynchronous iterator.

It is a type of asynchronous for-loop where the target iterator is awaited each iteration which yields a value.

For example:

...
# traverse an asynchronous iterator
async for item in async_iterator:
	print(item)

It can also be used in a list comprehension:

...
# build a list of results
results =

It cannot be used to traverse a list of awaitables, such as a list of coroutines or a list of asyncio.Task objects.

For example, the following will result in an error because the list is not an asynchronous generator or an asynchronous iterator:

...
# create a number of coroutines
coros = [simple_task(i) for i in range(10)]
# traverse the iterable of awaitables
async for item in coros:
    print(result)

The error would look like:

TypeError: 'async for' requires an object with __aiter__ method, got list

You can learn more about the async for expression in the tutorial:

The "async for" expression can be used for asynchronous generators.

These are generators that yield a value each time they are awaited.

For example:

# define an asynchronous generator
async def async_generator():
    # normal loop
    for i in range(10):
        # block to simulate doing work
        await asyncio.sleep(1)
        # yield the result
        yield i

You can learn more about asynchronous generators in the tutorial:

The "async for" expression can be used for asynchronous iterators.

These are iterators that yield a value each time they are awaited.

For example:

# define an asynchronous iterator
class CustomIterator():
    # constructor, define some state
    def __init__(self):
        self.counter = 0

    # create an instance of the iterator
    def __aiter__(self):
        return self

    # return the next awaitable
    async def __anext__(self):
        # check for no further items
        if self.counter >= 10:
            raise StopAsyncIteration
        # block to simulate work
        await asyncio.sleep(1)
        # increment the counter
        self.counter += 1
        # return the counter value
        return self.counter

You can learn more about asynchronous generators in the tutorial:

Now that we know how to develop async for loops, let's look at some worked examples.

Example of Sequential Asyncio For Loop

Before we develop async for loops, let's look at a sequential for loop in an asyncio program.

In this example, we will define a coroutine that performs a task. We will then call this coroutine many times in a loop with different data.

Each call will be made sequentially, one after the other in a normal for loop.

This will provide a helpful baseline for comparison and contrast to the asynchronous for loops we will develop later.

Firstly, we can define the coroutine that will perform some task with the provided data.

The work() coroutine below implements this taking a data argument, sleeping for a second to simulate work, and reporting a message with the data argument.

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # report a message
    print(f'>task done with {data}')

Next, we can define the main() coroutine.

It first reports a message, then loops 10 times and awaits the work() coroutine with a different argument.

This loop is sequential.

Once the loop is complete, a final message is reported.

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # complete many tasks
    for data in range(10):
        # complete one task
        await work(data)
    # report a message
    print('Main done')

Because each task will block for one second, we expect the loop to take at least 10 seconds to complete sequentially.

Finally, we can start the asyncio event loop and run the main() coroutine.

...
# start the event loop
asyncio.run(main())

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of sequential for loop in asyncio
import asyncio

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # report a message
    print(f'>task done with {data}')

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # complete many tasks
    for data in range(10):
        # complete one task
        await work(data)
    # report a message
    print('Main done')

# start the event loop
asyncio.run(main())

Running the example starts the event loop and runs the main() coroutine.

The main() coroutine runs and reports a message. It then loops 10 times, each iteration awaiting the work() coroutine with different data.

The work() coroutine runs, sleeps for one second, then reports a message.

Once the loop is complete, a final message is reported and the program is terminated.

This highlights a sequential loop of async tasks. As expected, the program takes about 10 seconds to complete, e.g. 10 x 1 second tasks performed sequentially.

Main starting
>task done with 0
>task done with 1
>task done with 2
>task done with 3
>task done with 4
>task done with 5
>task done with 6
>task done with 7
>task done with 8
>task done with 9
Main done

Next, let's look at how to develop an async for loop using asyncio.gather().

Example of Method 01: Async for-loop with asyncio.gather()

We can explore how to develop an async for loop using asyncio.gather().

In this case, we can update the above example to first create a list of work() coroutines, each with different data as an argument.

...
# create all coroutines
coros = [work(i) for i in range(10)]

We can then expand the list and provide it to the asyncio.gather() function as an argument.

This will execute all work() coroutines concurrently.

...
# wait for all tasks to complete concurrently
await asyncio.gather(*coros)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of async for loop with asyncio.gather()
import asyncio

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # report a message
    print(f'>task done with {data}')

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # create all coroutines
    coros = [work(i) for i in range(10)]
    # wait for all tasks to complete concurrently
    await asyncio.gather(*coros)
    # report a message
    print('Main done')

# start the event loop
asyncio.run(main())

Running the example starts the event loop and runs the main() coroutine.

The main() coroutine runs and reports a message.

It then creates a list of coroutines in a list comprehension.

The list of coroutine objects is then expanded using the star operator and provided to the asyncio.gather() function, and awaited.

All 10 work() coroutines run concurrently, sleeping, and reporting their message.

Once all tasks are done and the asyncio.gather() returns. The main() coroutine resumes and reports a final message.

This highlights how we can develop an async for loop by running all async tasks concurrently with asyncio.gather().

Main starting
>task done with 0
>task done with 1
>task done with 2
>task done with 3
>task done with 4
>task done with 5
>task done with 6
>task done with 7
>task done with 8
>task done with 9
Main done

Next, let's look at how to develop an async for loop using asyncio.wait().

Example of Method 02: Async for-loop with asyncio.wait()

We can explore how to develop an async for loop using asyncio.wait().

In this case, we can update the above example to first create a list of work() coroutines asyncio.Task instances, each with different data as an argument.

The tasks are scheduled, but not yet running.

...
# create all tasks
tasks = [asyncio.create_task(work(i)) for i in range(10)]

We can then await the group of asyncio.Task objects via the asyncio.wait() function.

...
# wait for all tasks to complete concurrently
_ = await asyncio.wait(tasks)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of async for loop with asyncio.wait()
import asyncio

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # report a message
    print(f'>task done with {data}')

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # create all tasks
    tasks = [asyncio.create_task(work(i)) for i in range(10)]
    # wait for all tasks to complete concurrently
    _ = await asyncio.wait(tasks)
    # report a message
    print('Main done')

# start the event loop
asyncio.run(main())

Running the example starts the event loop and runs the main() coroutine.

The main() coroutine runs and reports a message.

It then creates a list of asyncio.Task objects in a list comprehension, one for each work() coroutine with different data.

The asyncio.Task objects are scheduled in the event loop, but not yet running.

The list of tasks is then provided to the asyncio.gather() function, and awaited.

All 10 work() coroutines run concurrently, sleeping, and reporting their message.

Once all tasks are done the asyncio.wait() returns. The main() coroutine resumes and reports a final message.

This highlights how we can develop an async for loop by running all async coroutines as concurrent tasks that are awaited with asyncio.wait().

Main starting
>task done with 0
>task done with 1
>task done with 2
>task done with 3
>task done with 4
>task done with 5
>task done with 6
>task done with 7
>task done with 8
>task done with 9
Main done

Next, let's look at how to develop an async for loop using asyncio.as_completed().

Example of Method 03: Async for-loop with asyncio.as_completed()

We can explore how to develop an async for loop using asyncio.as_completed().

In this case, we can update the above example so that the work() coroutine returns a value.

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # return a value
    return data

We can then create a list of work() tasks in a list comprehension, as we did above.

...
# create all tasks
tasks = [asyncio.create_task(work(i)) for i in range(10)]

These tasks can then be provided to asyncio.as_completed() to be executed concurrently, which will yield each asyncio.Task instance as it is done. We can retrieve the return value from each task and report it.

...
# report results as tasks complete
for task in asyncio.as_completed(tasks):
    # get task result
    result = await task
    # report result
    print(f'>task done with {result}')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of async for loop with asyncio.as_completed()
import asyncio

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # return a value
    return data

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # create all tasks
    tasks = [asyncio.create_task(work(i)) for i in range(10)]
    # report results as tasks complete
    for task in asyncio.as_completed(tasks):
        # get task result
        result = await task
        # report result
        print(f'>task done with {result}')
    # report a message
    print('Main done')

# start the event loop
asyncio.run(main())

Running the example starts the event loop and runs the main() coroutine.

The main() coroutine runs and reports a message.

It then creates a list of asyncio.Task objects in a list comprehension, one for each work() coroutine with different data.

The asyncio.Task objects are scheduled in the event loop, but not yet running.

The list of tasks is then provided to the asyncio.as_completed() function.

All work() tasks run concurrently, sleeping and reporting their message.

The asyncio.as_completed() function yields one work() task at a time, in the order they are completed. Each is awaited in order to retrieve the return value result, which is reported.

This highlights how we can develop an async for loop using asyncio.as_completed().

Main starting
>task done with 0
>task done with 1
>task done with 2
>task done with 3
>task done with 4
>task done with 5
>task done with 6
>task done with 7
>task done with 8
>task done with 9
Main done

Next, let's look at how to develop an async for loop using a list comprehension of Tasks.

Example of Method 04: Async for-loop with list comprehension of Tasks

We can explore how to develop an async for loop using a list comprehension of Tasks.

In this case, we will create a list of asyncio.Task objects, as we did before.

This schedules the tasks for execution in the event loop but does not yet start their execution.

...
# create all tasks
tasks = [asyncio.create_task(work(i)) for i in range(10)]

We can then enumerate the list of tasks in order, and await each in turn.

...
# wait on tasks one by one
for task in tasks:
    await task

This suspends the main() coroutine, allowing all tasks to run concurrently.

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of async for loop with task list comprehension
import asyncio

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # report a message
    print(f'>task done with {data}')

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # create all tasks
    tasks = [asyncio.create_task(work(i)) for i in range(10)]
    # wait on tasks one by one
    for task in tasks:
        await task
    # report a message
    print('Main done')

# start the event loop
asyncio.run(main())

Running the example starts the event loop and runs the main() coroutine.

The main() coroutine runs and reports a message.

It then creates a list of asyncio.Task objects in a list comprehension, one for each work() coroutine with different data.

The asyncio.Task objects are scheduled in the event loop, but not yet running.

The main() coroutine then enumerates the list of task objects in order and awaits each in turn.

This suspends the main() coroutine and allows each task to run, sleeping and reporting its message.

The main() coroutine resumes as each task is awaited in order, returning immediately if tasks are already done otherwise suspending until the task is done.

This is similar to the asyncio.as_completed() approach, except the order of the tasks matches the order they were created and scheduled, not the order they are completed (if they were completed at different times).

This highlights how we can develop an async for loop using a list comprehension of asyncio.Task objects.

Main starting
>task done with 0
>task done with 1
>task done with 2
>task done with 3
>task done with 4
>task done with 5
>task done with 6
>task done with 7
>task done with 8
>task done with 9
Main done

Next, let's look at how to develop an async for loop using an asyncio.TaskGroup.

Example of Method 05: Async for-loop with an asyncio.TaskGroup

We can explore how to develop an async for loop using an asyncio.TaskGroup.

In this case, we can update the above example to create an asyncio.TaskGroup using the context manager interface.

This asyncio.TaskGroup can then be used to create an asyncio.Task for each work() coroutine with different data, in a list comprehension.

...
# create task group
async with asyncio.TaskGroup() as group:
    # create all tasks
    _ = [group.create_task(work(i)) for i in range(10)]
# wait for tasks to complete...

This schedules each work() coroutine as a task. On exiting the context manager, the main() coroutine suspends and waits for all tasks in the group to be done.

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of async for loop with task group
import asyncio

# async task
async def work(data):
    # simulate some work
    await asyncio.sleep(1)
    # report a message
    print(f'>task done with {data}')

# main coroutine
async def main():
    # report a message
    print('Main starting')
    # create task group
    async with asyncio.TaskGroup() as group:
        # create all tasks
        _ = [group.create_task(work(i)) for i in range(10)]
    # wait for tasks to complete...
    # report a message
    print('Main done')

# start the event loop
asyncio.run(main())

Running the example starts the event loop and runs the main() coroutine.

The main() coroutine runs and reports a message.

An asyncio.TaskGroup is then created using the context manager interface.

The main() coroutine then creates all 10 tasks using the asyncio.TaskGroup in a list comprehension.

Exiting the asyncio.TaskGroup context manager suspends the main() coroutine and blocks until all tasks created using the group are done.

All tasks run concurrently, sleeping, reporting their message and terminating normally.

Once all tasks are done, the main() coroutine resumes and exits the asyncio.TaskGroup context manager completely. It reports a final message and terminates.

This highlights how we can develop an async for loop using the asyncio.TaskGroup.

Main starting
>task done with 0
>task done with 1
>task done with 2
>task done with 3
>task done with 4
>task done with 5
>task done with 6
>task done with 7
>task done with 8
>task done with 9
Main done

Takeaways

You now know how to execute an asyncio for loop concurrently.



If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.