Last Updated on November 6, 2022
You can use the async for expression to loop over asynchronous iterators and generators in asyncio programs.
In this tutorial, you will discover how to use the asyncio async for expression for asynchronous for-loops in Python.
Let’s get started.
What is the async for loop?
The async for expression is used to traverse an asynchronous iterator.
It is an asynchronous for loop statement.
An asynchronous iterator is an iterator that yields awaitables.
asynchronous iterator: An object that implements the __aiter__() and __anext__() methods. __anext__ must return an awaitable object. async for resolves the awaitables returned by an asynchronous iterator’s __anext__() method until it raises a StopAsyncIteration exception.
— Python Glossary
You may recall that an awaitable is an object that can be waited for, such as a coroutine or a task.
awaitable: An object that can be used in an await expression.
— Python Glossary
An asynchronous generator will automatically implement the asynchronous iterator methods, allowing it to be iterated like an asynchronous iterator.
The async for expression allows the caller to traverse an asynchronous iterator of awaitables and retrieve the result from each.
This is not the same as traversing a collection or list of awaitables (e.g. coroutine objects), instead, the awaitables returned must be provided using the expected asynchronous iterator methods.
Internally, the async for loop will automatically resolve or await each awaitable, scheduling coroutines as needed.
Because it is a for-loop, it assumes, although does not require, that each awaitable being traversed yields a return value.
The async for loop must be used within a coroutine because internally it will use the await expression, which can only be used within coroutines.
Now that we know what the async for expression is, let’s look at how we might use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use async for
The async for expression can be used to traverse an asynchronous iterator within a coroutine.
For example:
|
1 2 3 4 |
... # traverse an asynchronous iterator async for item in async_iterator: print(item) |
This does not execute the for-loop in parallel. The asyncio is unable to execute more than one coroutine at a time within a Python thread.
Instead, this is an asynchronous for-loop.
The difference is that the coroutine that executes the for loop will suspend and internally await for each awaitable.
Behind the scenes, this may require coroutines to be scheduled and awaited, or tasks to be awaited.
We may also use the async for expression in a list comprehension.
For example:
|
1 2 3 |
... # build a list of results results = [item async for item async_iterator] |
This would construct a list of return values from the asynchronous iterator.
Next, let’s consider why we might use an async for loop.
Why Use async for
The async-for expression is used when developing asyncio programs.
For example, it may be common for a stream or socket connection to operate like an iterator or generator, allowing objects or lines of data to be retrieved as awaitables. These can be traversed and processed.
Additionally, custom asynchronous iterators and generators can be developed that return awaitables that can be traversed by an asyncio program.
The use cases become apparent if we consider that the for loop and async for loop both operate on iterators and generators, it just so happens that the async for loop works with asynchronous iterators and generators.
This means that use cases where we may need iterators and generators in our asynchronous programs will require the use of an async for loop.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
“async for” loop vs “for” loop
The for and async for expressions are very similar.
They both can be used to traverse something.
The for expression can be used to traverse an iterable or a generator.
The async for expression can be used to traverse an asynchronous iterable or an asynchronous generator.
The important difference is that they are not interchangeable.
A for loop cannot work with asynchronous iterable and generators and an async for loop cannot work with a traditional iterator or generator.
Doing so will result in a TypeError.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of async for with Asynchronous Generator
We can explore an example of using the async for to traverse an asynchronous generator.
An asynchronous generator is like a normal generator, except it is defined with the async def expression. This makes it a coroutine and able to use the await expression. It must also have a yield expression in order to return a value.
We can consume or traverse an asynchronous generator using the async for expression.
The example below defines an asynchronous generator that returns integers. A coroutine then uses the async for expression to traverse the generator and report values that are returned.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of async for with an asynchronous generator import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # define a simple coroutine async def custom_coroutine(): # asynchronous for loop async for item in async_generator(): # report the result print(item) # start asyncio.run(custom_coroutine()) |
Running the example first creates the custom coroutine and uses it as the entry point into the asyncio program.
The custom coroutine then uses an async for expression to traverse the async_generator().
The first call to the generator executes from the top, starts the loop, blocks, then yields one value, suspending execution at this point.
The custom coroutine is received in the async for loop and then reported in the body of the loop.
The next iteration calls into the generator, resuming execution after the yield statement, which starts the next iteration, blocks, and yields the second value.
This process continues until the generator is exhausted and all yielded values are returned.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 2 3 4 5 6 7 8 9 |
If we defined a conventional generator and attempted to traverse it using the async for, an error is raised.
We can demonstrate this by changing the asynchronous generator into a traditional generator and removing the await sleep.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of async for with a traditional generator import asyncio # define a generator def async_generator(): # normal loop for i in range(10): # yield the result yield i # define a simple coroutine async def custom_coroutine(): # asynchronous for loop async for item in async_generator(): # report the result print(item) # start asyncio.run(custom_coroutine()) |
Running the example results in a TypeError.
This is because the async for expression expects an object that implements the __aiter__() and __anext__() methods and an asynchronous generator implements these methods automatically.
Next, let’s look at how we might use the async for expression with a custom asynchronous iterator.
Example of async for with Asynchronous Iterator
We can explore the async for expression with an asynchronous iterator.
Recall that an asynchronous iterator implements the __aiter__() and __anext__() methods.
The __aiter__() method returns an instance of the iterator whereas the __anext__() method returns the next awaitable.
This is just like a traditional iterator that must implement the __iter__() and __next__() method.
The example below defines an asynchronous iterator that returns an integer each time __anext__() is called and stops once the counter reaches 10.
Because the __anext__() is defined as a coroutine via the async def, it is able to use the await expression. In this case, the function sleeps to simulate some IO tasks.
An instance of the iterator can be created and traversed using the async for expression.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# SuperFastPython.com # example of async for with an asynchronous iterator import asyncio # define an asynchronous iterator class CustomIterator(): # constructor, define some state def __init__(self): self.counter = 0 # create an instance of the iterator def __aiter__(self): return self # return the next awaitable async def __anext__(self): # check for no further items if self.counter >= 10: raise StopAsyncIteration # block to simulate work await asyncio.sleep(1) # increment the counter self.counter += 1 # return the counter value return self.counter # define a simple coroutine async def custom_coroutine(): # asynchronous for loop async for item in CustomIterator(): # report the result print(item) # start asyncio.run(custom_coroutine()) |
Running the example first creates the custom coroutine and uses it as the entry point into the asyncio program.
The custom coroutine then uses an async for expression to traverse the CustomIterator().
Each iteration blocks increments the counter and returns a value. Once the counter equals or exceeds 10, the iteration stops.
The calling coroutine internally receives each awaitable, schedules and waits for each to complete, then receives the return value which is reported.
|
1 2 3 4 5 6 7 8 9 10 |
1 2 3 4 5 6 7 8 9 10 |
The async for cannot be used with a traditional iterator.
Further, if an asynchronous iterator is defined and the __anext__() method is not defined using async def, then an error is raised.
This is because the __anext__() method must be an awaitable, e.g. a coroutine in order to be used with the async for expression.
We can update the example to make this one small change so that __anext__() is defined with the def expression instead of the async def expression.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of async for with an incorrectly defined asynchronous iterator import asyncio # define an iterator class CustomIterator(): # constructor, define some state def __init__(self): self.counter = 0 # create an instance of the iterator def __aiter__(self): return self # return the next awaitable def __anext__(self): # check for no further items if self.counter >= 10: raise StopAsyncIteration # increment the counter self.counter += 1 # return the counter value return self.counter # define a simple coroutine async def custom_coroutine(): # asynchronous for loop async for item in CustomIterator(): # report the result print(item) # start asyncio.run(custom_coroutine()) |
Running the example results in a TypeError.
Although our custom asynchronous iterator class implements the correct method name, it is not implemented as a coroutine via the async def expression.
As such, it cannot be awaited which is a requirement for the async for expression.
|
1 2 3 4 5 6 7 |
TypeError: object int can't be used in 'await' expression The above exception was the direct cause of the following exception: Traceback (most recent call last): ... TypeError: 'async for' received an invalid object from __anext__: int |
Next, let’s take a closer look at a common misconception with the async for expression.
Example of async for Cannot Traverse Collection of Awaitables
A common error assumes that you can use the async for to iterate over a collection of awaitables.
This is not the case.
We can demonstrate this with a worked example.
The example below defines a coroutine named simple_task that takes a number, blocks for a moment then returns the argument. The main coroutine creates a list of simple coroutines in a list comprehension, then attempts to traverse the collection of coroutines.
The expectation is that the async for loop will schedule and await each awaitable in order.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of trying to use async for loop over a collection of awaitables import asyncio # define a simple asynchronous async def simple_task(number): # block for a moment await asyncio.sleep(1) # return the argument return number # define a simple coroutine async def custom_coroutine(): # create a number of coroutines coros = [simple_task(i) for i in range(10)] # traverse the iterable of awaitables async for item in coros: print(result) # start the asyncio program asyncio.run(custom_coroutine()) |
Running the example results in a TypeError.
The async for expression expects that the provided iterable implements the __aiter__() method.
The provided list does not implement this method and is not an asynchronous iterable.
As such, the code does not have the desired effect.
|
1 2 3 4 |
Traceback (most recent call last): ... TypeError: 'async for' requires an object with __aiter__ method, got list sys:1: RuntimeWarning: coroutine 'simple_task' was never awaited |
Instead, we can use a traditional for-loop over the awaitables, then explicitly await each in turn, get the return value and report it.
For example:
|
1 2 3 4 5 6 7 |
... # traverse the iterable of awaitables for item in coros: # await and get the result from the awaitable result = await item # report the results print(result) |
The update example with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of awaiting a collection of awaitables import asyncio # define a simple asynchronous async def simple_task(number): # block for a moment await asyncio.sleep(1) # return the argument return number # define a simple coroutine async def custom_coroutine(): # create a number of coroutines coros = [simple_task(i) for i in range(10)] # traverse the iterable of awaitables for item in coros: # await and get the result from the awaitable result = await item # report the results print(result) # start the asyncio program asyncio.run(custom_coroutine()) |
Running the example executes the entry point coroutine.
A list of simple_task() coroutines is created and then traversed. Each is awaited in turn and the result is reported.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 2 3 4 5 6 7 8 9 |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to use the asyncio async for asynchronous for loops in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Joshua Koblin on Unsplash
Do you have any questions?