Last Updated on November 14, 2023
A generator is a function that has at least one yield expression.
Using a generator returns a generator iterator that can be traversed to yield the generated values. The problem is that conventional generators are not well suited to asyncio programs as we cannot await yielded values.
Instead, we can develop and use asynchronous generators defined using coroutines that yield values. The resulting asynchronous generator iterator can then be traversed, awaiting each yielded value automatically with the async for expression.
In this tutorial, you will discover how to develop and use asynchronous generators.
After completing this tutorial, you will know:
- The difference between conventional generators and asynchronous generators.
- How to develop asynchronous generators for use in asyncio.
- How to use asynchronous generators with the “async for” expression in asyncio.
Let’s get started.
What Are Asynchronous Generators
An asynchronous generator is a coroutine that uses the yield expression.
Before we dive into the details of asynchronous generators, let’s first review classical Python generators.
Generators
A generator is a Python function that returns a value via a yield expression.
For example:
|
1 2 3 4 |
# define a generator def generator(): for i in range(10): yield i |
The generator is executed to the yield expression, after which a value is returned. This suspends the generator at that point. The next time the generator is executed it is resumed from the point it was resumed and runs until the next yield expression.
generator: A function which returns a generator iterator. It looks like a normal function except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function.
— Python Glossary
Technically, a generator function creates and returns a generator iterator. The generator iterator executes the content of the generator function, yielding and resuming as needed.
generator iterator: An object created by a generator function. Each yield temporarily suspends processing, remembering the location execution state […] When the generator iterator resumes, it picks up where it left off …
— Python Glossary
A generator can be executed in steps by using the next() built-in function.
For example:
|
1 2 3 4 5 |
... # create the generator gen = generator() # step the generator result = next(gen) |
Although, it is more common to iterate the generator to completion, such as using a for-loop or a list comprehension.
For example:
|
1 2 3 |
... # traverse the generator and collect results results = [item for item in generator()] |
Next, let’s take a closer look at asynchronous generators.
Asynchronous Generators
An asynchronous generator is a coroutine that uses the yield expression.
Unlike a function generator, the coroutine can schedule and await other coroutines and tasks.
asynchronous generator: A function which returns an asynchronous generator iterator. It looks like a coroutine function defined with async def except that it contains yield expressions for producing a series of values usable in an async for loop.
— Python Glossary
Like a classical generator, an asynchronous generator function can be used to create an asynchronous generator iterator that can be traversed using the built-in anext() function, instead of the next() function.
asynchronous generator iterator: An object created by a asynchronous generator function. This is an asynchronous iterator which when called using the __anext__() method returns an awaitable object which will execute the body of the asynchronous generator function until the next yield expression.
— Python Glossary
This means that the asynchronous generator iterator implements the __anext__() method and can be used with the async for expression.
This means that each iteration of the generator is scheduled and executed as awaitable. The “async for” expression will schedule and execute each iteration of the generator, suspending the calling coroutine and awaiting the result.
You can learn more about the “async for” expression in the tutorial:
Next, let’s consider the similarities and differences between classical and asynchronous generators in more detail.
Generators vs Asynchronous Generators
Both classical generators and asynchronous generators have a lot in common.
Both are types of functions that have at least one yield expression.
A Python generator is any function containing one or more yield expressions
— PEP 525 – Asynchronous Generators
A classical generator can be used generally anywhere in a Python program, whereas an asynchronous generator can only be used in an asyncio Python program, such as called and used within a coroutine.
- Classical Generator: Used generally anywhere in Python Program
- Asynchronous Generator: Only used in asyncio programs.
The classical generator can be stepped using the built-in next() function, whereas asynchronous generators are traversed using the anext() built-in function.
- Classical Generator: Stepped using the next() function.
- Asynchronous Generator: Stepped using the anext() function.
The classical generator is traversed with a classical for loop, whereas the asynchronous generator must be traversed using an “async for” loop expression.
- Classical Generator: Traversed using a for loop.
- Asynchronous Generator: Traversed using an async for loop.
Each iteration of the asynchronous generator returns an awaitable that must be awaited in an asyncio program. Therefore mixing the syntax of classical and asynchronous generators will result in errors.
- Classical Generator: Yields a value.
- Asynchronous Generator: Yields an awaitable.
Now that we know about asynchronous generators, let’s look at how we can define and use them.
Run loops using all CPUs, download your FREE book to learn how.
How to Use an Asynchronous Generator
In this section, we will take a close look at how to define, create, step, and traverse an asynchronous generator in asyncio programs.
Let’s start with how to define an asynchronous generator.
Define an Asynchronous Generator
We can define an asynchronous generator by defining a coroutine that has at least one yield expression.
This means that the function is defined using the “async def” expression.
For example:
|
1 2 3 4 |
# define an asynchronous generator async def async_generator(): for i in range(10) yield i |
Because the asynchronous generator is a coroutine and each iterator returns an awaitable that is scheduled and executed in the asyncio event loop, we can execute and await awaitables within the body of the generator.
For example:
|
1 2 3 4 5 6 7 |
# define an asynchronous generator that awaits async def async_generator(): for i in range(10) # suspend and sleep a moment await asyncio.sleep(1) # yield a value to the caller yield i |
Next, let’s look at how we might use an asynchronous generator.
Create Asynchronous Generator
To use an asynchronous generator we must create the generator.
This looks like calling it, but instead creates and returns an iterator object.
For example:
|
1 2 3 |
... # create the iterator it = async_generator() |
This returns a type of asynchronous iterator called an asynchronous generator iterator.
Step an Asynchronous Generator
One step of the generator can be traversed using the anext() built-in function, just like a classical generator using the next() function.
The result is an awaitable that is awaited.
For example:
|
1 2 3 4 5 |
... # get an awaitable for one step of the generator awaitable = anext(gen) # execute the one step of the generator and get the result result = await awaitable |
This can be achieved in one step.
For example:
|
1 2 3 |
... # step the async generator result = await anext(gen) |
Traverse an Asynchronous Generator
The asynchronous generator can also be traversed in a loop using the “async for” expression that will await each iteration of the loop automatically.
For example:
|
1 2 3 4 |
... # traverse an asynchronous generator async for result in async_generator(): print(result) |
You can learn more about the “async for” expression in the tutorial:
We may also use an asynchronous list comprehension with the “async for” expression to collect the results of the generator.
For example:
|
1 2 3 |
... # async list comprehension with async generator results = [item async for item in async_generator()] |
You can learn more about asynchronous list comprehensions in the tutorial:
Now that we know how to create and use asynchronous generators, let’s look at a worked example.
Example of One Step of an Asynchronous Generator with anext()
We can explore how to create an asynchronous generator and step it one time.
This is a good approach to understanding more deeply what exactly is happening with the generator when it is traversed using an “async for” expression.
In this example, we will first define an asynchronous generator that iterates a number of times, suspends each iteration, and yields an integer value.
This highlights the coroutine nature of the asynchronous generator in that it is able to suspend and await other coroutines and tasks.
The main coroutine will create the generator, get an awaitable that steps the generator one iteration, then await the awaitable in order to retrieve the yielded value.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of one step of an asynchronous generator import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # main coroutine async def main(): # create the async generator gen = async_generator() # step the generator one iteration awaitable = anext(gen) # get the result from one iteration result = await awaitable # report the result print(result) # execute the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine runs and creates an instance of the asynchronous generator.
It then calls the anext() function on the generator. This returns an awaitable, that if awaited will execute one iteration of the generator to the first yield statement.
The awaitable is awaited. This suspends the main() coroutine and executes the asynchronous generator one iteration to the first yield statement and a value is returned. The generator is suspended.
The main() coroutine resumes and retrieves the yielded value and reports it.
The program then terminates.
|
1 |
This highlights what is happening when an asynchronous generator is executed, such as each iteration within an “async for” expression.
Next, let’s look at how we might traverse an asynchronous generator using the “async for” expression.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of an Asynchronous Generator with async for
We can explore how to traverse an asynchronous generator using the “async for” expression.
In this example, we will update the previous example to traverse the generator to completion using an “async for” loop.
This loop will automatically await each awaitable returned from the generator, retrieve the yielded value, and make it available within the loop body so that in this case it can be reported.
This is perhaps the most common usage pattern for asynchronous generators.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of asynchronous generator with async for loop import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # main coroutine async def main(): # loop over async generator with async for loop async for item in async_generator(): print(item) # execute the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine runs and starts the for loop.
An instance of the asynchronous generator is created and the loop automatically steps it using the anext() function to return an awaitable. The loop then awaits the awaitable and retrieves a value which is made available to the body of the loop where it is reported.
This process is then repeated, suspending the main() coroutine, executing an iteration of the generator, and suspending, and resuming the main() coroutine until the generator is exhausted.
This highlights how an asynchronous generator can be traversed using an async for expression.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 2 3 4 5 6 7 8 9 |
Next, let’s look at how we might use an asynchronous generator with an asynchronous list comprehension.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of an Asynchronous Generator with Asynchronous Comprehension
We can explore how to use an asynchronous generator with an asynchronous list comprehension.
In this example, we will update the above example to traverse the asynchronous generator to completion.
In this case, we will traverse it within an asynchronous list comprehension, rather than a “async for” loop. This will collect the yielded values into a list that can be reported.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of asynchronous generator with async comprehension import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # main coroutine async def main(): # loop over async generator with async comprehension results = [item async for item in async_generator()] # report results print(results) # execute the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine runs and executes the asynchronous list comprehension.
This comprehension operates just like the async for loop in the previous section. Each iteration retrieves an awaitable, the awaitable is awaited and suspends the main() coroutine, and an iteration of the generator executes and yields a value which is then collected in the resumed main() coroutine.
The process is repeated until all values are collected into a list.
Once the asynchronous list comprehension is completed, the values are then reported.
This highlights how we might traverse an asynchronous generator using an asynchronous list comprehension.
|
1 |
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
Next, let’s look at some common errors when using asynchronous generators.
Common Errors with Asynchronous Generators
It is common to stumble when getting started with asynchronous generators.
In this section, we will explore a number of common errors when using asynchronous generators and how they may be overcome.
Do you have an error not listed below?
Let me know in the comments below and I will do my best to help.
Error Using Asynchronous Generator with next()
We can explore what happens when we try to use an asynchronous generator with the next() built-in function, instead of the anext() function.
In this example, we will create the generator, then attempt to step it one iteration.
We will use the next() function on the asynchronous generator, which will result in an error.
We will get an error because the next() function expects to operate on an iterator, and instead it has received an asynchronous iterator that implements a different interface.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of error using an asynchronous generator with next() import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # main coroutine async def main(): # create the async generator gen = async_generator() # step the generator one iteration (results in an error) awaitable = next(gen) # execute the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine runs and creates the asynchronous generator as per normal.
The next() function is then called on the asynchronous generator which results in a TypeError exception and terminates the program.
The exception occurs because the next() function is expecting an iterator and was instead provided an asynchronous iterator that implements a different interface.
The fix involves using the anext() function on the asynchronous generator instead of the next() function.
|
1 2 3 |
Traceback (most recent call last): ... TypeError: 'async_generator' object is not an iterator |
Next, let’s look at using an asynchronous generator with a “for” expression.
Error Using Asynchronous Generator with for loop
We can explore what happens if we try to use an asynchronous generator with a “for” expression, instead of an “async for” expression.
In this example, we will create the generator and then attempt to traverse it to completion using a classical for-loop.
This will fail because the for-loop expects an iterator and instead receives an asynchronous iterator that implements a different interface.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of error using an asynchronous generator with a for loop import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # main coroutine async def main(): # classical for loop over async generator (relists in error) for item in async_generator(): print(item) # execute the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.
The main() coroutine runs and creates an instance of the asynchronous generator in the loop and begins iterating.
This fails with a TypeError exception because the asynchronous generator is an asynchronous iterator, not a classical iterator as expected by the for-loop expression.
The fix involves using an “async for” expression instead of a “for” expression.
|
1 2 3 |
Traceback (most recent call last): ... TypeError: 'async_generator' object is not iterable |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to create and use asynchronous generators in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Wes Tindel on Unsplash
Great article.
Thanks!!!
You’re welcome!