Asyncio I/O Libraries

We need third-party libraries for common I/O tasks in asyncio programs.

This includes libraries that support async file I/O and standard I/O, as well as libraries that support common web application protocols such as HTTP, FTP, and SMTP.

In this tutorial, you will discover the best-of-breed third-party libraries for I/O in asyncio programs.

Let’s get started.

Table of Contents

The asyncio module API in the Python standard library is limited.

The asyncio module in Python’s standard library provides a solid foundation for asynchronous programming, offering a low-level API to manage event loops and coroutines efficiently.

However, asyncio lacks direct support for several crucial capabilities, such as:

Non-Blocking File I/O
Non-Blocking Standard I/O
HTTP Clients
FTP and SMTP Clients

File I/O and Standard I/O are blocking tasks in Python and non-blocking file I/O is not supported on all platforms, requiring simulation of asynchronous programming using threads.

To bridge this gap, we have to turn to third-party libraries that build upon asyncio.

Libraries such as aiofiles extend asyncio to enable asynchronous file I/O operations, while aiohttp facilitates building asynchronous web applications, including HTTP and WebSocket support.

Embracing these third-party libraries allows us to harness the full power of asynchronous programming while accommodating diverse I/O operations and communication protocols beyond the asyncio module’s core capabilities.

Let’s take a tour of the best-of-breed third-party I/O libraries in each of these main areas.

Run loops using all CPUs, download your FREE book to learn how.

Asyncio File I/O Libraries

The asyncio module in the Python library does not provide support for non-blocking file I/O.

asyncio does not support asynchronous operations on the filesystem.
— ThirdParty, Asyncio GitHub Project.

This means that operations on files will block the asyncio event loop.

This includes reading and writing files, but also operations on file systems, like moving, renaming, deleting, and so on.

Non-blocking and therefore async file I/O is available on some platforms, although is not universally available or consistent enough to include in the standard library.

There has been some discussion to simulate non-blocking file IO in the Python standard library using threads.

A workaround is to use a threadpool for file io …
— NIO for files

The solution is to use a third-party library that either offers non-blocking file I/O or simulates non-blocking file I/O with threads.

Popular examples include the following:

aiofiles: File support for asyncio
aiofile: Real asynchronous file operations with asyncio support.
aiopath: Asynchronous pathlib for Python

Some less popular libraries (more hobby projects) include the following:

async-files: A Fast, lightweight, and extensible asyncio file library
asyncfile: Asynchronous file handling in Python

The two main options are aiofiles and aiofile. The aiopath offers non-blocking path operations.

The main solution is to use the aiofiles library.

It wraps the majority of the file I/O operations in the standard library and offers awaitable versions of the function that simulate asynchronous operations using a thread pool.

Ordinary local file IO is blocking, and cannot easily and portably be made asynchronous. This means doing file IO may interfere with asyncio applications, which shouldn’t block the executing thread. aiofiles helps with this by introducing asynchronous versions of files that support delegating operations to a separate thread pool.
— aiofiles GitHub Library.

You can learn how to use the aiofiles library in the tutorial:

AIOFiles for AsyncIO in Python

The aiofile project takes a different approach which is to implement true non-blocking file I/O on those platforms that support this functionality and thread pools for those platforms that do not.

For Linux using implementation based on libaio.
For POSIX (MacOS X and optional Linux) using implementation based on threadpool.
Otherwise using pure-python thread-based implementation.
Implementation chooses automatically depending on system compatibility.
— aiofile GitHub Project

This seems like a preferred approach, although it is less widely adopted.

The aiopath provides an async wrapper around the pathlib module in the standard library.

This allows querying and navigating the file system non-blocking, an often overlooked aspect of file I/O.

aiopath is a complete implementation of Python’s pathlib that’s compatible with asyncio, trio, and the async/await syntax. All I/O performed by aiopath is asynchronous and awaitable.
— aiopath GitHub Project.

The plot below shows the GitHub star rating histories of these three libraries.

We can see that aiofiles has a longer history and a Google search will show that it is widely used. The aiofile project is newer and has less adoption.

Non-Blocking File I/O Libraries in Asyncio Star Rating Histories

Download Now: Free Asyncio PDF Cheat Sheet

Asyncio Stdin/Stdout Libraries

The asyncio module in the Python library does not provide support for non-blocking console I/O.

Currently, asyncio doesn’t provide any helper to read asynchronously data from sys.stdin, like input().
— Implement async input()

Specifically:

Standard Input (stdin)
Standard Output (stdout)
Standard Error (stderr)

This means any asyncio programs that read input from users or report output on stdout or stderr will block the event loop, however briefly.

This can make a difference in those programs that report many debug messages to stdout or stderr. It can be deviating for those programs that block waiting for user input.

Some third-party libraries that address this include:

aioconsole: Asynchronous console and interfaces for asyncio
shellous: asyncio library that provides an API for running subprocesses

The focus of the aioconsole library is to provide a console interface for async programs, and focus of this is async versions of standard I/O streams.

asynchronous equivalents to input, print, exec and code.interact
— aioconsole GitHub Project.

The shellous library is more focused on running commands in subprocesses, providing a shell.

shellous provides a concise API for running subprocesses using asyncio. It is similar to and inspired by sh.
— shellous GitHub Project.

Nevertheless, it provides SteamReader and StreamWriter objects for handling standard I/O.

The streams run.stdout and run.stderr are asyncio.StreamReader objects. The stream run.stdin is an asyncio.StreamWriter object. If we didn’t specify that stdin/stdout are sh.CAPTURE, the streams run.stdin and run.stdout would be None.
— shellous GitHub Project.

Reviewing the GitHub star rating history, it seems that the aioconsole is an older and more popular library.

Non-Blocking Standard I/O Libraries in Asyncio Star Rating Histories

Free Python Asyncio Course

Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.

Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.

Learn more

Asyncio HTTP Libraries

The asyncio module provides streams as well as transports and protocols.

Nevertheless, these are low-level constructs. The API does not implement any high-level web protocols like HTTP.

Instead, it is preferable to use a third-party library to perform non-blocking HTTP requests.

The two more popular asyncio-first libraries for this are the following:

aiohttp: Asynchronous HTTP client/server framework for asyncio and Python
httpx: A next generation HTTP client for Python

The aiohttp provides both an HTTP client and server.

This means that although the predominant use of the library is as an HTTP client in asyncio programs, it can also be used as a web microframework to host an API.

Supports both client and server side of HTTP protocol.
Supports both client and server Web-Sockets out-of-the-box and avoids Callback Hell.
Provides Web-server with middleware and pluggable routing.
— aiohttp GitHub Project.

The httpx is newer than aiohttp, having been released only within the last few years.

It seeks to be fully featured, providing asyncio support, but also a classical synchronized API, as well as HTTP/2 support.

HTTPX is a fully featured HTTP client library for Python 3. It includes an integrated command line client, has support for both HTTP/1.1 and HTTP/2, and provides both sync and async APIs.
— httpx GitHub Project.

For more on asyncio HTTP libraries, see the tutorial:

Python Asyncio HTTP Client Libraries

Reviewing the GitHub star rating history, we can see that aiohttp is an older library and may be more popular, however, we can see that httpx is really taking off and may overtake it soon.

Non-Blocking HTTP Libraries in Asyncio Star Rating Histories

Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

Asyncio FTP/SMTP/etc. Client Libraries

There are other web application protocols we may require in addition to HTTP (above) that are also not provided by the asyncio module in the Python standard library.

Some examples include:

File Transfer Protocol (FTP/SFTP)
Simple Mail Transfer Protocol (SMTP)
Amazon Web Services API (AWS)

Below are some of the best-of-breed third-party libraries that offer support for these protocols for asyncio.

aioftp: ftp client/server for asyncio
aiosmtplib: asyncio smtplib implementation
aioaws: Asyncio compatible SDK for aws services.

The aioftp project provides simple FTP client and server support for asyncio programs.

ftp client/server for asyncio
— aioftp GitHub Project.

Similarly, the aiosmtplib provides simple SMTP client support for asyncio programs.

aiosmtplib is an asynchronous SMTP client for use with asyncio. It is an async version of the smtplib module, with similar APIs.
— aiosmtplib Project Documentation

AWS is perhaps one of the mostly used web/cloud services for developers.

As such, many developer tasks are simplified using the AWS APIs.

The aioaws project provides an async-first SDK for AWS APIs for use in asyncio programs.

Asyncio compatible SDK for aws services. This library does not depend on boto, boto3 or any of the other bloated, opaque and mind thumbing AWS SDKs. Instead, it is written from scratch to provide clean, secure and easily debuggable access to AWS services I want to use.
— aioaws GitHub Project.

Interestingly, aioaws is built upon aiofiles and httpx, two libraries that we reviewed above.

Reviewing the GitHub star rating histories, we can see that aioftp is an older and therefore more supported project. The aioaws project is newer and is seeing rapid adoption.