Asyncio Custom Awaitable With await()

April 7, 2024 Python Asyncio

We can define a custom awaitable for use in asyncio programs.

This can be achieved by defining a Python object that implements the __await__() method that either yields execution or delegates or returns the result of another awaitables __await__() method.

In this tutorial, you will discover how to define Python objects with the __await__() method.

Let's get started.

What is an Awaitable

An awaitable is a Python object that can be used in an await expression.

An awaitable must implement the __await__() special method (dunder method).

An awaitable object generally implements an __await__() method. Coroutine objects returned from async def functions are awaitable.
-- Python Data Model.

Examples of Python objects that are awaitable include:

Coroutines returned from async def expressions.
asyncio.Task objects.
asyncio.Future objects.

You can learn more about awaitable objects in the tutorial:

What is an Asyncio Awaitable in Python

What is await()

The __await__() method is a special method (dunder method) implemented by awaitable Python objects.

It must return an iterator and allow the object to be used as an await expression within coroutines.

object.__await__(self): Must return an iterator. Should be used to implement awaitable objects. For instance, asyncio.Future implements this method to be compatible with the await expression.
-- Python Data Model.

This does not mean that we can return a regular iterator, as an iterator cannot be used directly in an await expression. It would result in an error.

Instead, it must be an iterator that is comfortable with the await expression, such as a yield statement that suspends execution, or delegation to another iterator. This may be a generator as a generator is an iterator of sorts.

Recall that a generator can delegate to a sub-generator or an iterator can delegate to a sub-iterator via the yield from expression.

When yield from is used, the supplied expression must be an iterable. The values produced by iterating that iterable are passed directly to the caller of the current generator's methods.
-- Python Expressions

You can learn more about Python dunder methods in the tutorial:

Asyncio Dunder Methods (Magic Methods)

Examples of await()

We can learn more about the __await__() method by looking at some examples from the Python standard library.

Below is an example of the __await__() method from the asyncio.Future class.

def __await__(self):
    if not self.done():
        self._asyncio_future_blocking = True
        yield self  # This tells Task to wait for completion.
    if not self.done():
        raise RuntimeError("await wasn't used with future")
    return self.result()  # May raise too.

Notice that the main return is either a yield to self or a return of the result.

The Coroutine class extends the Awaitable class and the __await__() method from this parent class is listed below.

@abstractmethod
def __await__(self):
    yield

Notice that it is simply a yield statement.

Generally, any chain of awaitables will at some point end with a yield statement.

Any yield from chain of calls ends with a yield. This is a fundamental mechanism of how Futures are implemented. Since, internally, coroutines are a special kind of generators, every await is suspended by a yield somewhere down the chain of await calls
-- PEP 492 – Coroutines with async and await syntax

Now that we have seen some examples, let's explore how we might implement the __await__() method ourselves.

How to Define a Custom Awaitable

Generally, we don't define objects that implement the __await__() method in our asyncio programs.

Instead, we define coroutines.

For example:

# custom coroutine
asyncio def custom_coroutine():
	await asyncio.sleep()

You can learn more about Python coroutines in the tutorial:

What is a Coroutine in Python

Nevertheless, this is an interesting exercise in order to learn more about how asyncio works internally.

We can define a custom awaitable object in Python by adding the __await__() method.

As we have seen, the __await__() method must have a yield statement, or delegate to other awaitables that eventually end in a yield statement.

The __await__() method is defined as a regular Python function, not a coroutine. This means it is defined using the "def" expression, not the "async def" expression.

For example:

# awaitable dunder method
def __await__(self):
	# ...

The content of the method can be regular Python statements.

As we have seen, the method must return an iterator. This may be a generator, as a generator is a type of iterator.

The example below shows this in the use of a yield expression.

Therefore the simplest implementation of the __await__() method is a yield statement.

For example:

# awaitable dunder method
def __await__(self):
    # suspend execution
    yield

We can also delegate the yield to another awaitable

This can be achieved using the "yield from" expression used by a generator to delegate to another generator.

PEP 380 adds the yield from expression, allowing a generator to delegate part of its operations to another generator. This allows a section of code containing yield to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator.
-- What's New In Python 3.3

If __await__() always returns an iterator, then we yield from another asyncio coroutine or task's __await__() method.

This has the effect of delegating the call.

For example:

# awaitable dunder method
def __await__(self):
    # suspend execution
    yield from asyncio.sleep(2).__await__()

This does not have to be a yield or yield from expression, we might also return the result directly.

For example:

# awaitable dunder method
def __await__(self):
    # return the generator
    return asyncio.sleep(2).__await__()

Now that we know how to define an __await__(), let's look at some worked examples.

Example of A Custom Awaitable That Yields

We can define a custom awaitable with an __await__() method that has nothing more than a yield statement.

This suspends the execution of the caller, and then returns it again.

The complete example is listed below.

# SuperFastPython.com
# example of a custom awaitable with yield
import asyncio

# define a custom awaitable
class CustomAwaitable(object):
    # awaitable dunder method
    def __await__(self):
        # report a message
        print('Await is running')
        # suspend execution
        yield
        # report a message
        print('Await is done')

# main coroutine
async def main():
    # create and await custom awaitable
    await CustomAwaitable()

# start the asyncio event loop
asyncio.run(main())

Running the example first creates the main() coroutine then starts the asyncio event loop.

The main() coroutine runs and creates an instance of our CustomAwaitable class and awaits it.

The __await__() method runs, reports a message, then yields execution.

If other coroutines were scheduled in the event loop, they would have an opportunity to run at this point

Control is then returned as there are no other tasks running in the event loop. A final message is reported and the method returns.

The main() coroutine then exits and the event loop is terminated.

This highlights how we can define a custom awaitable that yields execution.

Await is running
Await is done

We can make this example clearer by scheduling another task before our custom awaitable, but not allowing it to run.

Instead, it must wait until execution is yielded by our custom awaitable before it can run.

For example, our new coroutine looks as follows:

# test coroutine that prints 10 numbers
async def test():
    for i in range(10):
        print(i)

The complete example is listed below.

# SuperFastPython.com
# example of a custom awaitable with yield
import asyncio

# define a custom awaitable
class CustomAwaitable(object):
    # awaitable dunder method
    def __await__(self):
        # report a message
        print('Await is running')
        # suspend execution
        yield
        # report a message
        print('Await is done')

# test coroutine that prints 10 numbers
async def test():
    for i in range(10):
        print(i)

# main coroutine
async def main():
    asyncio.create_task(test())
    # create and await custom awaitable
    await CustomAwaitable()

# start the asyncio event loop
asyncio.run(main())

Running the example first creates the main() coroutine then starts the asyncio event loop.

The main() coroutine runs and creates the test() coroutine and schedules it for execution.

The main() coroutine then creates an instance of our CustomAwaitable class and awaits it.

The __await__() method runs, reports a message, then yields execution.

This allows our background test() task to run, running its loop and reporting all 10 integers.

Control is then returned to the custom awaitable and a final message is reported.

The main() coroutine then exits and the event loop is terminated.

This highlights how a custom awaitable can yield control and allow other tasks running in the event loop an opportunity to run.

Await is running
0
1
2
3
4
5
6
7
8
9
Await is done

Example of A Custom Awaitable That Delegates

We can explore an example of delegating the __await__() method to another __await__() method.

In this case, we can create an asyncio.sleep() coroutine object and delegate the __await__() method of our custom awaitable to the coroutines __await__() method.

For example:

# awaitable dunder method
def __await__(self):
    # report a message
    print('Await is running')
    # suspend execution
    yield from asyncio.sleep(2).__await__()
    # report a message
    print('Await is done')

This will have the effect of running the target coroutine in the event loop.

It provides a workaround to allow the __await__() method of our custom awaitables to execute other awaitables without having to await them directly, which we cannot do because the __await__() method is not a coroutine.

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of a custom awaitable with yield from
import asyncio

# define a custom awaitable
class CustomAwaitable(object):
    # awaitable dunder method
    def __await__(self):
        # report a message
        print('Await is running')
        # suspend execution
        yield from asyncio.sleep(2).__await__()
        # report a message
        print('Await is done')

# main coroutine
async def main():
    # create and await custom awaitable
    await CustomAwaitable()

# start the asyncio event loop
asyncio.run(main())

Running the example first creates the main() coroutine then starts the asyncio event loop.

The main() coroutine runs and creates an instance of our CustomAwaitable class and awaits it.

The __await__() method runs, reports a message, then creates the asyncio.sleep() coroutine object and delegates to it.

The asyncio.sleep() coroutine's __await__() method runs, suspending the caller for two seconds.

Control is then returned and a final message is reported.

The main() coroutine then exits and the event loop is terminated.

This highlights how we can delegate to another awaitable from without a custom awaitable.

Await is running
Await is done

We don't have to explicitly delegate to another awaitable.

Instead, we could return another awaitable.

This is different as when the control of execution is returned, it will be returned to the caller (that has an await expression), rather than to our custom awaitable.

For example:

# awaitable dunder method
def __await__(self):
    # report a message
    print('Await is running')
    # suspend execution
    return asyncio.sleep(2).__await__()

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of a custom awaitable that returns an awaitable
import asyncio

# define a custom awaitable
class CustomAwaitable(object):
    # awaitable dunder method
    def __await__(self):
        # report a message
        print('Await is running')
        # suspend execution
        return asyncio.sleep(2).__await__()

# main coroutine
async def main():
    # create and await custom awaitable
    await CustomAwaitable()

# start the asyncio event loop
asyncio.run(main())

Running the example first creates the main() coroutine then starts the asyncio event loop.

The main() coroutine runs and creates an instance of our CustomAwaitable class and awaits it.

The __await__() method runs, reports a message, then creates the asyncio.sleep() coroutine object and returns the result of its __await__() method.

The result of the asyncio.sleep() coroutine's __await__() method is returned to the main() coroutine and awaited. This suspends the caller for two seconds.

Control is then returned to the main() coroutine and the event loop is terminated.

This highlights how we can return the result of another awaitables __await__() method from a custom awaitable.

Takeaways

You now know how to define objects with the __await__() method.

If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.

Asyncio Custom Awaitable With __await__()

What is an Awaitable

What is __await__()

Examples of __await__()

How to Define a Custom Awaitable

Example of A Custom Awaitable That Yields

Example of A Custom Awaitable That Delegates

Takeaways

Asyncio Custom Awaitable With await()

What is await()

Examples of await()