Asynchronous Comprehensions in Python

You can traverse an asynchronous generator or asynchronous iterator using an asynchronous comprehension via the “async for” expression.

In this tutorial, you will discover how to use asynchronous comprehensions and await comprehensions in Python.

Let’s get started.

Table of Contents

What are Asynchronous Comprehensions

An async comprehension is an asynchronous version of a classical comprehension.

Asyncio supports two types of asynchronous comprehensions, they are the “async for” comprehension and the “await” comprehension.

PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions
— PEP 530: Asynchronous Comprehensions, What’s New In Python 3.6

Before we look at each, let’s first recall classical comprehensions.

Comprehensions

Comprehensions allow data collections like lists, dicts, and sets to be created in a concise way.

List comprehensions provide a concise way to create lists.
— List Comprehensions

A list comprehension allows a list to be created from a for expression within the new list expression.

For example:

...

# create a list using a list comprehension

result = [a*2 for a in range(100)]

Comprehensions are also supported for creating dicts and sets.

For example:

...

# create a dict using a comprehension

result = {a:i for a,i in zip(['a','b','c'],range(3))}

# create a set using a comprehension

result = {a for a in [1, 2, 3, 2, 3, 1, 5, 4]}

Asynchronous Comprehensions

An asynchronous comprehension allows a list, set, or dict to be created using the “async for” expression with an asynchronous iterable.

We propose to allow using async for inside list, set and dict comprehensions.
— PEP 530 – Asynchronous Comprehensions

For example:

...

# async list comprehension with an async iterator

result = [a async for a in aiterable]

This will create and schedule coroutines or tasks as needed and yield their results into a list.

Recall that the “async for” expression may only be used within coroutines and tasks.

You can learn more about the async for expression in the tutorial:

Asyncio async for loop

Also, recall that an asynchronous iterator is an iterator that yields awaitables.

asynchronous iterator: An object that implements the __aiter__() and __anext__() methods. __anext__ must return an awaitable object. async for resolves the awaitables returned by an asynchronous iterator’s __anext__() method until it raises a StopAsyncIteration exception.
— PYTHON GLOSSARY

The “async for” expression allows the caller to traverse an asynchronous iterator of awaitables and retrieve the result from each.

Internally, the async for loop will automatically resolve or await each awaitable, scheduling coroutines as needed.

An async generator automatically implements the methods for the async iterator and may also be used in an asynchronous comprehension.

asynchronous generator: A function which returns an asynchronous generator iterator.
— PYTHON GLOSSARY

For example:

...

# async list comprehension with an async generator

result = [a async for a in agenerator]

Await Comprehensions

The “await” expression may also be used within a list, set, or dict comprehension, referred to as an await comprehension.

We propose to allow the use of await expressions in both asynchronous and synchronous comprehensions
— PEP 530 – Asynchronous Comprehensions

Like an async comprehension, it may only be used within an asyncio coroutine or task.

This allows a data structure, like a list, to be created by suspending and awaiting a series of awaitables.

For example:

...

# await list compression with a collection of awaitables

results = [await a for a in awaitables]

This will create a list of results by awaiting each awaitable in turn.

The current coroutine will be suspended to execute awaitables sequentially, which is different and perhaps slower than executing them concurrently using asyncio.gather().

Now that we know what async and await comprehensions are, let’s look at some examples.

Run loops using all CPUs, download your FREE book to learn how.

Example of Async List Comprehension With Generator

We can explore how to use an asynchronous list comprehension with an asynchronous generator.

In this example, we will define an asynchronous generator that will sleep for one second each iteration to simulate some blocking work and yield an integer.

You may recall that an asynchronous generator is a coroutine that yields a value when iterated.

We will then populate a list using the asynchronous generator with an asynchronous list comprehension that makes use of the “async for” expression.

The complete example is listed below.

# SuperFastPython.com

# example of an asynchronous list comprehension with an generator

import asyncio

# define an asynchronous generator

async def async_generator():

# normal loop

for i in range(10):

# block to simulate doing work

await asyncio.sleep(1)

# yield the result

yield i

# main coroutine

async def main():

# asynchronous list comprehension

results = [item async for item in async_generator()]

# report results

print(results)

# start the asyncio program

asyncio.run(main())

Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.

The main() coroutine runs and iterates the asynchronous generator in an asynchronous list comprehension using the “async for” expression, intended for asynchronous iterators, of which the generator is one example.

A lot is happening on this one compact line.

The generator is created and the “async for” expression schedules the first iteration and then suspends the main() coroutine. The generator executes the first iteration of the loop, sleeps, then yields a single value. The main() coroutine resumes, retrieves one value, and appends it to the list.

The main() coroutine then iterates the generator and is suspended. The generator resumes from where it yielded and performs another iteration of the loop and yields a value. The process continues until no further iterations are possible.

Each iteration of the asynchronous generator schedules a coroutine to resume the iterator where it left off last time.

The result list is populated and is then reported.

This highlights how we can use an asynchronous list comprehension with an asynchronous generator via the “async for” expression.

1	[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Next, we will look at how to use an asynchronous list comprehension with an asynchronous iterator.

Download Now: Free Asyncio PDF Cheat Sheet

Example of Asynchronous List Comprehension With Iterator

We can explore how to use an asynchronous list comprehension with an asynchronous iterator.

In this example, we will define an asynchronous iterator object. The iterator will maintain a counter and will sleep for a second and return the value of the counter each time it is iterated.

You may recall that an asynchronous iterator is an object that implements the __aiter__() and __anext__() methods.

We will then populate a list using the asynchronous iterator with an asynchronous list comprehension that makes use of the “async for” expression.

The complete example is listed below.

# SuperFastPython.com

# example of an asynchronous list comprehension with an iterator

import asyncio

# define an asynchronous iterator

class AsyncIterator():

# constructor, define some state

def __init__(self):

self.counter = 0

# create an instance of the iterator

def __aiter__(self):

return self

# return the next awaitable

async def __anext__(self):

# check for no further items

if self.counter >= 10:

raise StopAsyncIteration

# block to simulate work

await asyncio.sleep(1)

# increment the counter

self.counter += 1

# return the counter value

return self.counter

# main coroutine

async def main():

# asynchronous list comprehension

results = [item async for item in AsyncIterator()]

# report results

print(results)

# start the asyncio program

asyncio.run(main())

Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.

The main() coroutine runs and iterates the asynchronous iterator in an asynchronous list comprehension using the “async for” expression.

As with the previous example, a lot is happening on this one compact line.

The iterator object is first created. It is then used in the “async for” expression which schedules the first iteration and then suspends the main() coroutine. The iterator executes the __anext__() function and the first iteration of the loop, sleeps, increments the counter, then returns a single value. The main() coroutine resumes, retrieves one value, and appends it to the list.

The main() coroutine then steps the iterator again and is suspended. The iterator executes the __anext__() function again and performs another iteration of the loop and returns a value. The process continues until the counter is equal to 10 and a StopAsyncIteration exception is raised, terminating the iteration.

The result list is populated and is then reported.

This highlights how we can use an asynchronous list comprehension with an asynchronous iterator object via the “async for” expression.

1	[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Next, we will look at how to use an await list comprehension.

Free Python Asyncio Course

Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.

Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.

Learn more

Example of Await List Comprehension

We can explore how to use an await list comprehension with coroutines.

In this example, we will define a coroutine that takes a value as an argument, sleeps, and returns a multiple of the argument.

We will then create a list of these coroutines and then populate a list with their return values, awaiting each, in turn, using an await comprehension.

The complete example is listed below.

# SuperFastPython.com

# example of an await list comprehension

import asyncio

# task coroutine

async def task_coro(value):

# block a moment to simulate work

await asyncio.sleep(1)

# calculate and return

return value * 10

# main coroutine

async def main():

# create a list of awaitables

awaitables = [task_coro(i) for i in range(10)]

# await list comprehension to collect results

results = [await a for a in awaitables]

# report results

print(results)

# run the asyncio program

asyncio.run(main())

Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.

The main() coroutine runs and first creates a list of coroutine objects using a classical list comprehension.

It then executes the list of coroutines using an await list comprehension. In each iteration of this comprehension, the main() coroutine is suspended and the next coroutine is scheduled and executed, rerunning a value that is appended to the result list.

This process continues until all coroutines in the list of awaitables have been awaited.

This example highlights how we can use an await list comprehension to collect results from multiple awaitables, like coroutines.

We can see that the coroutines are executed one by one sequentially, rather than concurrently. This suggests that using an await comprehension may only be appropriate when there is a linear dependency between the tasks.

An alternative would be to execute all awaitables concurrently, then collect their results.

1	[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]

Next, we will look at a faster alternative to using an await comprehension.

Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

Example of an Alternative to an Await Comprehension: asyncio.gather()

We can explore how to use an alternative to an await comprehension.

In this example, we will update the above example that populates a list with the return values from multiple coroutines by awaiting each.

Instead of using an await comprehension, we will use the asyncio.gather() function that will schedule and execute all coroutines in the list concurrently, rather than one at a time sequentially.

You can learn more about the asyncio.gather() function in the tutorial:

Asyncio gather() to Run Many Tasks Concurrently

The complete example is listed below.

# SuperFastPython.com

# example of an alternative to an await list comprehension

import asyncio

# task coroutine

async def task_coro(value):

# block a moment to simulate work

await asyncio.sleep(1)

# calculate and return

return value * 10

# main coroutine

async def main():

# create a list of awaitables

awaitables = [task_coro(i) for i in range(10)]

# gather the awaitables concurrently

results = await asyncio.gather(*awaitables)

# report results

print(results)

# run the asyncio program

asyncio.run(main())

Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.

The main() coroutine runs and first creates a list of coroutine objects using a classical list comprehension.

It then executes the list of coroutines concurrently using the asyncio.gather() function.

This function takes awaitables as positional arguments. We can unpack the list of coroutines into separate expressions using the star expression (*).

In this case, all ten coroutines execute concurrently and their results are collected into the result list, which is then reported.

This differs from the previous example that used an await comprehension and executed each awaitable one at a time, sequentially.

This highlights the asyncio.gather() function as an alternative to an await comprehension that can be used when there is no linear dependence between the awaitables being executed.