Last Updated on November 16, 2023

Coroutines are concurrent tasks in asyncio programs.

Python provides first-class coroutines and the asyncio module for running and using them in Python applications.

Coroutines are used to develop concurrent applications but are unlike thread-based and process-based concurrency commonly used in Python.

In this tutorial, you will discover coroutines in Python.

After completing this tutorial, you will know:

• What are coroutines, how do they work, and how do they compare to subroutines and generators.
• How to use coroutines including how to define, create, and run coroutines in Python.
• What is the history of coroutines in Python?

Let’s get started.

What is a Coroutine

A coroutine is a function that can be suspended and resumed.

It is often defined as a generalized subroutine.

A subroutine can be executed, starting at one point and finishing at another point. Whereas, a coroutine can be executed then suspended, and resumed many times before finally terminating.

coroutine: Coroutines are a more generalized form of subroutines. Subroutines are entered at one point and exited at another point. Coroutines can be entered, exited, and resumed at many different points.

— Python Glossary

Specifically, coroutines have control over when exactly they suspend their execution.

This may involve the use of a specific expression, such as an “await” expression in Python, like a yield expression in a Python generator.

A coroutine is a method that can be paused when we have a potentially long-running task and then resumed when that task is finished. In Python version 3.5, the language implemented first-class support for coroutines and asynchronous programming when the keywords async and await were explicitly added to the language.

— Page 3, Python Concurrency with asyncio, 2022.

A coroutine may suspend for many reasons, such as executing another coroutine, e.g. awaiting another task, or waiting for some external resources, such as a socket connection or process to return data.

Coroutines are used for concurrency.

Coroutines let you have a very large number of seemingly simultaneous functions in your Python programs.

— Page 267, Effective Python, 2019.

Many coroutines can be created and executed at the same time. They have control over when they will suspend and resume, allowing them to cooperate as to when concurrent tasks are executed.

This is called cooperative multitasking and is different to the multitasking typically used with threads called preemptive multitasking tasking.

… in order to run multiple applications concurrently, processes voluntarily yield control periodically or when idle or logically blocked. This type of multitasking is called cooperative because all programs must cooperate for the scheduling scheme to work.

— Cooperative multitasking, Wikipedia

Preemptive multitasking involves the operating system choosing what threads to suspend and resume and when to do so, as opposed to the tasks themselves deciding in the case of cooperative multitasking.

Now that we have some idea of what a coroutine is, let’s deepen this understanding by comparing them to other familiar programming constructs.

Comparing Coroutines

We can better understand a coroutine by comparing it to other familiar programming concepts.

Let’s dive in.

Coroutine vs Routine and Subroutine

A “routine” and “subroutine” often refer to the same thing in modern programming.

Perhaps more correctly, a routine is a program, whereas a subroutine is a function in the program.

A routine has subroutines.

It is a discrete module of expressions that is assigned a name, may take arguments and may return a value.

• Subroutine: A module of instructions that can be executed on demand, typically named, and may take arguments and return a value. also called a function

A subroutine is executed, runs through the expressions, and returns somehow. Typically, a subroutine is called by another subroutine.

A coroutine is a generalization of a subroutine. This means that a subroutine is a special type of a coroutine.

A coroutine is like a subroutine in many ways, such as:

• They both are discrete named modules of expressions.
• They both can take arguments, or not.
• They both can return a value, or not.

The main difference is that it chooses to suspend and resume its execution many times before returning and exiting.

Both coroutines and subroutines can call other examples of themselves. A subroutine can call other subroutines. A coroutine executes other coroutines. However, a coroutine can also execute other subroutines.

When a coroutine executes another coroutine, it must suspend its execution and allow the other coroutine, to resume once the other coroutine has completed.

This is like a subroutine calling another subroutine. The difference is the suspension of the coroutine may allow any number of other coroutines to run as well.

This makes a coroutine calling another coroutine more powerful than a subroutine calling another subroutine. It is central to the cooperating multitasking facilitated by coroutines.

Coroutine vs Generator

A generator is a special function that can suspend its execution.

generator: A function which returns a generator iterator. It looks like a normal function except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function.

— Python Glossary

A generator function can be defined like a normal function although it uses a yield expression at the point it will suspend its execution and return a value.

A generator function will return a generator iterator object that can be traversed, such as via a for-loop. Each time the generator is executed, it runs from the last point it was suspended to the next yield statement.

generator iterator: An object created by a generator function. Each yield temporarily suspends processing, remembering the location execution state (including local variables and pending try-statements). When the generator iterator resumes, it picks up where it left off (in contrast to functions which start fresh on every invocation).

— Python Glossary

A coroutine can suspend or yield to another coroutine using an “await” expression. It will then resume from this point once the awaited coroutine has been completed.

Using this paradigm, an await statement is similar in function to a yield statement; the execution of the current function gets paused while other code is run. Once the await or yield resolves with data, the function is resumed.

— Page 218, High Performance Python, 2020.

We might think of a generator as a special type of coroutine and cooperative multitasking used in loops.

Generators, also known as semicoroutines, are a subset of coroutines.

— Coroutine, Wikipedia.

Before coroutines were developed, generators were extended so that they might be used like coroutines in Python programs.

This required a lot of technical knowledge of generators and the development of custom task schedulers.

To implement your own concurrency using generators, you first need a fundamental insight concerning generator functions and the yield statement. Specifically, the fundamental behavior of yield is that it causes a generator to suspend its execution. By suspending execution, it is possible to write a scheduler that treats generators as a kind of “task” and alternates their execution using a kind of cooperative task switching.

— Page 524, Python Cookbook, 2013.

This was made possible via changes to the generators and the introduction of the “yield from” expression.

These were later deprecated in favor of the modern async/await expressions.

Coroutine vs Task

A subroutine and a coroutine may represent a “task” in a program.

However, in Python, there is a specific object called an asyncio.Task object.

A Future-like object that runs a Python coroutine. […] Tasks are used to run coroutines in event loops.

— Asyncio Task Object

A coroutine can be wrapped in an asyncio.Task object and executed independently, as opposed to being executed directly within a coroutine. The Task object provides a handle on the asynchronously execute coroutine.

• Task: A wrapped coroutine that can be executed independently.

This allows the wrapped coroutine to execute in the background. The calling coroutine can continue executing instructions rather than awaiting another coroutine.

A Task cannot exist on its own, it must wrap a coroutine.

Therefore a Task is a coroutine, but a coroutine is not a task.

You can learn more about asyncio.Task objects in the tutorial:

• What is an Asyncio Task

Coroutine vs Thread

A coroutine is more lightweight than a thread.

• Thread: heavyweight compared to a coroutine
• Coroutine: lightweight compared to a thread.

A coroutine is defined as a function.

A thread is an object created and managed by the underlying operating system and represented in Python as a threading.Thread object.

• Thread: Managed by the operating system, represented by a Python object.

This means that coroutines are typically faster to create and start executing and take up less memory. Conversely, threads are slower than coroutines to create and start and take up more memory.

The cost of starting a coroutine is a function call. Once a coroutine is active, it uses less than 1 KB of memory until it’s exhausted.

— Page 267, Effective Python, 2019.

Coroutines execute within one thread, therefore a single thread may execute many coroutines.

Many separate async functions advanced in lockstep all seem to run simultaneously, mimicking the concurrent behavior of Python threads. However, coroutines do this without the memory overhead, startup and context switching costs, or complex locking and synchronization code that’s required for threads.

— Page 267, Effective Python, 2019.

You can learn more about threads in the guide:

• Python Threading: The Complete Guide

Coroutine vs Process

A coroutine is more lightweight than a process.

In fact, a thread is more lightweight than a process.

A process is a computer program. It may have one or many threads.

A Python process is in fact a separate instance of the Python interpreter.

Processes, like threads, are created and managed by the underlying operating system and are represented by a multiprocessing.Process object.

• Process: Managed by the operating system, represented by a Python object.

This means that coroutines are significantly faster than a process to create and start and take up much less memory.

A coroutine is just a special function, whereas a Process is an instance of the interpreter that has at least one thread.

You can learn more about Python processes in the guide:

• Python Multiprocessing: The Complete Guide

How to Use a Coroutine in Python

Python provides coroutines for concurrency.

They are provided in two main ways:

• Through specific additions to the language, e.g. async and await expressions.
• Through a specific module in the standard library, e.g. asyncio module.

Let’s take a tour on how we can create and use coroutines in Python.

How to Define a Coroutine

A coroutine can be defined via the “async def” expression.

This is an extension of the “def” expression for defining subroutines.

It defines a coroutine that can be created and returns a coroutine object.

For example:

A coroutine defined with the “async def” expression is referred to as a “coroutine function”

coroutine function: A function which returns a coroutine object. A coroutine function may be defined with the async def statement, and may contain await, async for, and async with keywords.

— Python Glossary

A coroutine can then use coroutine-specific expressions within it, such as await, async for, and async with.

Execution of Python coroutines can be suspended and resumed at many points (see coroutine). await expressions, async for and async with can only be used in the body of a coroutine function.

— Coroutine function definition

For example:

You can learn more about defining coroutines in the tutorial:

• What is Asyncio async def

How to Create a Coroutine

Once a coroutine is defined, it can be created.

This looks like calling a subroutine.

For example:

This does not execute the coroutine.

It returns a “coroutine” object.

You can think of a coroutine function as a factory for coroutine objects; more directly, remember that calling a coroutine function does not cause any user-written code to execute, but rather just builds and returns a coroutine object.

— Page 516, Python in a Nutshell, 2017.

A “coroutine” Python object has methods, such as send() and close(). It is a type.

We can demonstrate this by creating an instance of a coroutine and calling the type() built-in function in order to report its type.

For example:

Running the example reports that the created coroutine is a “coroutine” class.

We also get a RuntimeError because the coroutine was created but never executed, we will explore that in the next section.

A coroutine object is an awaitable.

This means it is a Python type that implements the __await__() method.

An awaitable object generally implements an __await__() method. Coroutine objects returned from async def functions are awaitable.

— Awaitable Objects

You can learn more about awaitables in the tutorial:

• What is an Asyncio Awaitable in Python

How to Run a Coroutine From Python

Coroutines can be defined and created, but they can only be executed within an event loop.

The event loop is the core of every asyncio application. Event loops run asynchronous tasks and callbacks, perform network IO operations, and run subprocesses.

— Asyncio Event Loop

The event loop that executes coroutines, manages the cooperative multitasking between coroutines.

Coroutine objects can only run when the event loop is running.

— Page 517, Python in a Nutshell, 2017.

The typical way to start a coroutine event loop is via the asyncio.run() function.

This function takes one coroutine and returns the value of the coroutine. The provided coroutine can be used as the entry point into the coroutine-based program.

For example:

Running the example

You learn more about executing coroutines as asyncio programs in the tutorial:

• How to Run an Asyncio Program in Python

When Were Coroutines Added to Python

Generators have slowly been migrating towards becoming first-class coroutines for a long time.

We can explore some of the major changes to Python to add coroutines, which we might consider a subset of the probability addition of asyncio.

New methods like send() and close() were added to generator objects to allow them to act more like coroutines.

These were added in Python 2.5 and described in PEP 342.

This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines.

— PEP 342 – Coroutines via Enhanced Generators

Later, allowing generators to emit a suspension exception as well as a stop exception described in PEP 334.

This PEP proposes a limited approach to coroutines based on an extension to the iterator protocol. Currently, an iterator may raise a StopIteration exception to indicate that it is done producing values. This proposal adds another exception to this protocol, SuspendIteration, which indicates that the given iterator may have more values to produce, but is unable to do so at this time.

— PEP 334 – Simple Coroutines via SuspendIteration

The vast majority of the capabilities for working with modern coroutines in Python via the asyncio module were described in PEP 3156, added in Python 3.3.

This is a proposal for asynchronous I/O in Python 3, starting at Python 3.3. Consider this the concrete proposal that is missing from PEP 3153. The proposal includes a pluggable event loop, transport and protocol abstractions similar to those in Twisted, and a higher-level scheduler based on yield from (PEP 380). The proposed package name is asyncio.

— PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module

A second approach to coroutines, based on generators, was added to Python 3.4 as an extension to Python generators.

A coroutine was defined as a function that used the @asyncio.coroutine decorator.

Coroutines were executed using an asyncio event loop, via the asyncio module.

A coroutine could suspend and execute another coroutine via the “yield from” expression

For example:

The “yield from” expression was defined in PEP 380.

A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing ‘yield’ to be factored out and placed in another generator.

— PEP 380 – Syntax for Delegating to a Subgenerator

The “yield from” expression is still available for use in generators, although is a deprecated approach to suspending execution in coroutines, in favor of the “await” expression.

Note: Support for generator-based coroutines is deprecated and is removed in Python 3.11. Generator-based coroutines predate async/await syntax. They are Python generators that use yield from expressions to await on Futures and other coroutines.

— Asyncio Coroutines and Tasks

We might say that coroutines were added as first-class objects to Python in version 3.5.

This included changes to the Python language, such as the “async def“, “await“, “async with“, and “async for” expressions, as well as a coroutine type.

These changes were described in PEP 492.

It is proposed to make coroutines a proper standalone concept in Python, and introduce new supporting syntax. The ultimate goal is to help establish a common, easily approachable, mental model of asynchronous programming in Python and make it as close to synchronous programming as possible.

— PEP 492 – Coroutines with async and await syntax

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know about coroutines in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Callum Hill on Unsplash