SuperFastPython

Asyncio I/O Libraries

December 25, 2023 Python Asyncio

We need third-party libraries for common I/O tasks in asyncio programs.

This includes libraries that support async file I/O and standard I/O, as well as libraries that support common web application protocols such as HTTP, FTP, and SMTP.

In this tutorial, you will discover the best-of-breed third-party libraries for I/O in asyncio programs.

Let's get started.

Asyncio I/O Libraries

The asyncio module API in the Python standard library is limited.

The asyncio module in Python's standard library provides a solid foundation for asynchronous programming, offering a low-level API to manage event loops and coroutines efficiently.

However, asyncio lacks direct support for several crucial capabilities, such as:

File I/O and Standard I/O are blocking tasks in Python and non-blocking file I/O is not supported on all platforms, requiring simulation of asynchronous programming using threads.

To bridge this gap, we have to turn to third-party libraries that build upon asyncio.

Libraries such as aiofiles extend asyncio to enable asynchronous file I/O operations, while aiohttp facilitates building asynchronous web applications, including HTTP and WebSocket support.

Embracing these third-party libraries allows us to harness the full power of asynchronous programming while accommodating diverse I/O operations and communication protocols beyond the asyncio module's core capabilities.

Let's take a tour of the best-of-breed third-party I/O libraries in each of these main areas.

Asyncio File I/O Libraries

The asyncio module in the Python library does not provide support for non-blocking file I/O.

asyncio does not support asynchronous operations on the filesystem.

-- ThirdParty, Asyncio GitHub Project.

This means that operations on files will block the asyncio event loop.

This includes reading and writing files, but also operations on file systems, like moving, renaming, deleting, and so on.

Non-blocking and therefore async file I/O is available on some platforms, although is not universally available or consistent enough to include in the standard library.

There has been some discussion to simulate non-blocking file IO in the Python standard library using threads.

A workaround is to use a threadpool for file io ...

-- NIO for files

The solution is to use a third-party library that either offers non-blocking file I/O or simulates non-blocking file I/O with threads.

Popular examples include the following:

Some less popular libraries (more hobby projects) include the following:

The two main options are aiofiles and aiofile. The aiopath offers non-blocking path operations.

The main solution is to use the aiofiles library.

It wraps the majority of the file I/O operations in the standard library and offers awaitable versions of the function that simulate asynchronous operations using a thread pool.

Ordinary local file IO is blocking, and cannot easily and portably be made asynchronous. This means doing file IO may interfere with asyncio applications, which shouldn't block the executing thread. aiofiles helps with this by introducing asynchronous versions of files that support delegating operations to a separate thread pool.

-- aiofiles GitHub Library.

You can learn how to use the aiofiles library in the tutorial:

The aiofile project takes a different approach which is to implement true non-blocking file I/O on those platforms that support this functionality and thread pools for those platforms that do not.

For Linux using implementation based on libaio.
For POSIX (MacOS X and optional Linux) using implementation based on threadpool.
Otherwise using pure-python thread-based implementation.
Implementation chooses automatically depending on system compatibility.

-- aiofile GitHub Project

This seems like a preferred approach, although it is less widely adopted.

The aiopath provides an async wrapper around the pathlib module in the standard library.

This allows querying and navigating the file system non-blocking, an often overlooked aspect of file I/O.

aiopath is a complete implementation of Python's pathlib that's compatible with asyncio, trio, and the async/await syntax. All I/O performed by aiopath is asynchronous and awaitable.

-- aiopath GitHub Project.

The plot below shows the GitHub star rating histories of these three libraries.

We can see that aiofiles has a longer history and a Google search will show that it is widely used. The aiofile project is newer and has less adoption.

Non-Blocking File I/O Libraries in Asyncio Star Rating Histories
Non-Blocking File I/O Libraries in Asyncio Star Rating Histories

Asyncio Stdin/Stdout Libraries

The asyncio module in the Python library does not provide support for non-blocking console I/O.

Currently, asyncio doesn't provide any helper to read asynchronously data from sys.stdin, like input().

-- Implement async input()

Specifically:

This means any asyncio programs that read input from users or report output on stdout or stderr will block the event loop, however briefly.

This can make a difference in those programs that report many debug messages to stdout or stderr. It can be deviating for those programs that block waiting for user input.

Some third-party libraries that address this include:

The focus of the aioconsole library is to provide a console interface for async programs, and focus of this is async versions of standard I/O streams.

asynchronous equivalents to input, print, exec and code.interact

-- aioconsole GitHub Project.

The shellous library is more focused on running commands in subprocesses, providing a shell.

shellous provides a concise API for running subprocesses using asyncio. It is similar to and inspired by sh.

-- shellous GitHub Project.

Nevertheless, it provides SteamReader and StreamWriter objects for handling standard I/O.

The streams run.stdout and run.stderr are asyncio.StreamReader objects. The stream run.stdin is an asyncio.StreamWriter object. If we didn't specify that stdin/stdout are sh.CAPTURE, the streams run.stdin and run.stdout would be None.

-- shellous GitHub Project.

Reviewing the GitHub star rating history, it seems that the aioconsole is an older and more popular library.

Non-Blocking Standard I/O Libraries in Asyncio Star Rating Histories

Asyncio HTTP Libraries

The asyncio module provides streams as well as transports and protocols.

Nevertheless, these are low-level constructs. The API does not implement any high-level web protocols like HTTP.

Instead, it is preferable to use a third-party library to perform non-blocking HTTP requests.

The two more popular asyncio-first libraries for this are the following:

The aiohttp provides both an HTTP client and server.

This means that although the predominant use of the library is as an HTTP client in asyncio programs, it can also be used as a web microframework to host an API.

Supports both client and server side of HTTP protocol.
Supports both client and server Web-Sockets out-of-the-box and avoids Callback Hell.
Provides Web-server with middleware and pluggable routing.

-- aiohttp GitHub Project.

The httpx is newer than aiohttp, having been released only within the last few years.

It seeks to be fully featured, providing asyncio support, but also a classical synchronized API, as well as HTTP/2 support.

HTTPX is a fully featured HTTP client library for Python 3. It includes an integrated command line client, has support for both HTTP/1.1 and HTTP/2, and provides both sync and async APIs.

-- httpx GitHub Project.

For more on asyncio HTTP libraries, see the tutorial:

Reviewing the GitHub star rating history, we can see that aiohttp is an older library and may be more popular, however, we can see that httpx is really taking off and may overtake it soon.

Non-Blocking HTTP Libraries in Asyncio Star Rating Histories
Non-Blocking HTTP Libraries in Asyncio Star Rating Histories

Asyncio FTP/SMTP/etc. Client Libraries

There are other web application protocols we may require in addition to HTTP (above) that are also not provided by the asyncio module in the Python standard library.

Some examples include:

Below are some of the best-of-breed third-party libraries that offer support for these protocols for asyncio.

The aioftp project provides simple FTP client and server support for asyncio programs.

ftp client/server for asyncio

-- aioftp GitHub Project.

Similarly, the aiosmtplib provides simple SMTP client support for asyncio programs.

aiosmtplib is an asynchronous SMTP client for use with asyncio. It is an async version of the smtplib module, with similar APIs.

-- aiosmtplib Project Documentation

AWS is perhaps one of the mostly used web/cloud services for developers.

As such, many developer tasks are simplified using the AWS APIs.

The aioaws project provides an async-first SDK for AWS APIs for use in asyncio programs.

Asyncio compatible SDK for aws services. This library does not depend on boto, boto3 or any of the other bloated, opaque and mind thumbing AWS SDKs. Instead, it is written from scratch to provide clean, secure and easily debuggable access to AWS services I want to use.

-- aioaws GitHub Project.

Interestingly, aioaws is built upon aiofiles and httpx, two libraries that we reviewed above.

Reviewing the GitHub star rating histories, we can see that aioftp is an older and therefore more supported project. The aioaws project is newer and is seeing rapid adoption.

Non-Blocking Web Application Protocol Libraries in Asyncio Star Rating Histories
Non-Blocking Web Application Protocol Libraries in Asyncio Star Rating Histories

Takeaways

You now know about popular asyncio I/O libraries.



If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.