We can chain coroutines together into linear sequences

In other languages, this is called promise chaining or future chaining.

This allows a pipeline of dependent or independent tasks to be completed in a prescribed order.

There are many ways we can achieve this in asyncio programs, including having coroutines manually chain themselves together, using a utility to define a linear chain of coroutines, and using done callbacks to connect dependent tasks together.

In this tutorial, you will discover how to chain coroutines in asyncio programs.

Let’s get started.

What is a Coroutine

A coroutine represents a special type of function that can pause its execution at specific points without blocking other tasks.

It allows for concurrent and non-blocking operations, enabling asynchronous programming.

Coroutines are declared using the “async def” expression in Python, distinguishing them from regular functions.

For example:

They can be paused or suspended using the await expression within their body, allowing other coroutines or tasks to run while waiting for potentially blocking operations like I/O or delays.

Coroutines are a fundamental building block in asynchronous programming with Python, enabling efficient handling of concurrent tasks without resorting to thread-based parallelism.

They facilitate cooperative multitasking and make it easier to write asynchronous code that remains responsive and scalable.

You can learn more about coroutines in the tutorial:

• What is a Coroutine in Python

What is Coroutine Chaining

Coroutine chaining refers to the process of linking or chaining together multiple coroutines to execute in a specific sequence.

This technique involves invoking one coroutine after another, allowing for the orderly execution of asynchronous tasks.

By chaining coroutines, we can create a sequence of operations where the output of one coroutine becomes the input for the next one.

This pattern helps in organizing and managing complex asynchronous workflows, ensuring that each coroutine runs in the desired order and passes its results or exceptions to the subsequent coroutine in the chain.

The chaining of coroutines often involves using “await” expressions to invoke and await the completion of each coroutine in the sequence, allowing for cooperative multitasking and efficient handling of asynchronous tasks.

It enables us to structure and manage the flow of asynchronous operations, facilitating cleaner and more maintainable code in asyncio-based applications.

Next, let’s look at how we can implement coroutine chaining in asyncio.

How to Chain Coroutines in Asyncio

There are a number of ways to chain coroutines in asyncio.

We will review three approaches to chaining, including:

1. Manually chaining coroutines
2. Automatically chaining independent coroutines.
3. Automatically chaining dependent coroutines with callbacks.

Let’s take a closer look at each approach.

Manually Chain Coroutines

The simplest way to chain coroutines is to have one coroutine await another coroutine.

This process can then be repeated for as many links in the chain as required.

For example:

In this case, func3() awaits the result from func2() which in turn awaits the result from func1().

This looks like a regular call graph where coroutines are created and awaited in sequence just like would be as function calls.

For example:

You can learn more about the await expression that can be used to chain coroutines in the tutorial:

• What is Asyncio Await in Python

Next, let’s look at how we might automatically chain independent coroutines.

Automatically Chain Independent Coroutines

The asyncio module in Python does not provide a way to automatically chain coroutines (at the time of writing).

One approach would be to define a new coroutine that takes an ordered list of coroutines that are then executed in sequence.

This would assume that the coroutines are independent and that the result from one coroutine is not required by another coroutine.

The chain() coroutine below implements this.

We can use this coroutine by providing a number of coroutines.

For example:

Or, a list of coroutines can be prepared and unpacked using the star operator, for example:

Unlike asyncio.gather(), the chain() coroutine ensures that the provided awaitables are executed in order and that the next awaitable is not executed until the one before it is done.

Because the coroutines are executed sequentially, we lose any benefit we might get from executing the coroutines concurrently, given the coroutines are independent of each other.

A benefit of the function is that it allows coroutines to be grouped and nested.

For example:

Next, let’s look at how we might automatically chain dependent coroutines using callback functions.

Automatically Chain Coroutines With Callbacks

Asyncio provides the ability to add and remove done callback functions to tasks.

Callbacks are regular Python functions that are executed when the task is done and require that the coroutine first be scheduled as an asyncio task.

A callback cannot await a coroutine or a task, but it can create and schedule a new task.

This means that we can define a series of callbacks that are added to asyncio.Task instances in order to chain coroutines together.

For example:

You can learn about done callback functions in the tutorial:

• How to Use Asyncio Task Done Callback Functions

The callback function provides the task instance to which it was added as an argument so it is able to retrieve the result from the task and pass it to the next task via an argument to the coroutine.

For example:

This allows a chain of dependent coroutines to be defined.

For example:

A limitation is that the call that initiates the chain has no way of knowing when the chain will be completed. There is also no easy way to get the result from the end of the chain back to the caller.

This is because each coroutine is executed as an independent asyncio.Task.

We can use a mechanism to signal to the caller that the chain is complete, such as an asyncio.Event.

The caller can create the event, assign it to a global variable, and wait for it to be set.

For example:

The final coroutine in the chain can access the same event global variable and set it when it is done, perhaps via a callback function designed for this purpose.

For example:

You can learn more about asyncio events in the tutorial:

• How to Use an Asyncio Event in Python

Now that we know how to define chains of coroutines in asyncio programs, let’s look at some worked examples.

Example of Manual Coroutine Chaining

We can explore how to manually chain coroutines together.

In this example, we will define three coroutines in a chain:

1. The first coroutine will create and await a result from the second coroutine.
2. The second coroutine will create and await a result from the third coroutine.
3. The third coroutine will prepare and return a result.

The main() coroutine will start the chain by creating the first coroutine and await the result of the chain.

The chain looks as follows

Notice the nesting structure of the chain. This is much like function calls, except here we assume that a non-blocking I/O is required to retrieve the data in each step of the chain.

Tying this together, the complete example is listed below.

Running the example first creates the main() coroutine and starts the asyncio event loop.

The main() coroutine runs and creates the func3() coroutine and waits for it to complete.

The func3() coroutine runs and reports a message, it then creates the func2() coroutine and waits for it to complete.

The func2() coroutine runs and reports a message, then creates the func1() coroutine and waits for it to complete.

The func1() coroutine runs, reports a message, and returns a value.

The func2() coroutine receives the value and adds it to its own value before returning the result.

The func3() coroutine receives the value and adds it to its own value before returning the result.

Finally, the main() coroutine receives the result and reports it.

This highlights how we can manually chain coroutines in our asyncio programs.

Next, let’s look at how we might automatically chain independent coroutines.

Example of Automatic Chaining of Independent Coroutines

We can explore how to automatically chain independent coroutines.

In this example, we will use our chain() utility coroutine developed above.

This allows us to execute a linear sequence of independent coroutines. They are independent because they do not interact with each other. There is no passing of results from one to the next.

In this case, we will define a sequence of 3 sequential tasks where the second task is divided into 2 sub-tasks.

For example:

• 1. task1()
• 2a. task2_sub1()
• 2b. task2_sub2()
• 3. task3()

We will use our chain() utility to group or chain the two task2 subtasks together into one task.

For example:

We will then chain the three tasks together in a linear sequence.

For example:

Each task will return a result and we will report all results once the chain is complete.

Tying this together, the complete example is listed below.

Running the example first creates the main() coroutine and starts the asyncio event loop.

The chain is defined as a sequence of three tasks where the second task consists of two subtasks.

The main() coroutine blocks and waits for the chain to complete.

The chain() coroutine runs and executes each task sequentially, starting with task1() which reports a message and returns a value.

The two task2 subtasks run, report a message, and return a value.

Finally, the task3() coroutine runs, reports a message, and returns a value.

The main() coroutine resumes and reports all return values.

This highlights how we can chain independent coroutines together into a linear sequence and retrieve all of their return values.

Next, let’s explore how we might chain coroutines together using done callback functions.

Example of Automatic Chaining of Coroutines With Callbacks

We can explore how to chain coroutines together using done callback functions.

In this example, we will define three coroutines that are intended to be executed in a linear chain.

1. The first task prepares some data and returns it as a result.
2. The second task requires the result from the first task, uses it, and returns a result.
3. The third task requires the result from the second task, uses it, and returns a result.

We will use done callback functions to connect the coroutines in a chain via background tasks.

The first done callback function will be added to the first coroutine. It will retrieve the result from the first task, create and schedule the second task, and add a done callback to the task.

The second done callback is much like the first. It will retrieve the result from the second task, create the third task, pass it the result from the second task, and then add a done callback to the newly created and scheduled task.

Finally, the callback function for the third task is special.

It accesses a global variable, in this case an asyncio.Event and signals that the chain of tasks is done.

The main coroutine declares and defines the event used to signal the chain is done.

It then creates the first task, adds the callback function, and then waits on the event for the chain to be done.

Tying this together, the complete example is listed below.

Running the example creates the main() coroutine and uses it to start the asyncio event loop.

The main() coroutine runs and declares and defines a global variable for the event.

It then creates the first task in the chain and adds a done callback to it. It then waits for the event to be set.

The first task runs, reports a message, sleeps, and returns a result. The done callback for the task runs, gets the result, creates the second task, and passes the result as an argument. It then adds a done callback to the second task.

The second task runs, it receives the data as an argument, reports it, sleeps a moment, and then returns a result. The done callback for the task runs, retrieves the result from the task, creates the third task, and passes the result before finally adding a done callback to the new task.

The third task runs, receives the result from the second task, reports it, sleeps a moment, then returns a result. The done callback for the task runs, and sets the event.

The main() coroutine sees the event is set, resumes, and reports that the chain is done.

This example highlights how we can chain coroutines together using done callback functions.

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know how to chain coroutines in asyncio programs.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Matteo Paganelli on Unsplash