We can use HTTP client libraries in asyncio programs.
The popular Requests client HTTP library performs blocking network I/O when making requests. Using this library directly in our asyncio programs will block the event loop and prevent all other coroutines from progressing, however, there are workarounds such as running blocking calls in a separate thread.
A preferred approach is to use asyncio-native HTTP clients in our asyncio programs. This includes the popular and widely used aiohttp library and the new up-and-coming httpx library.
In this tutorial, you will discover how to use HTTP client libraries in asyncio programs.
Let’s get started.
HTTP Clients Block The Asyncio Event Loop
HTTP clients are a large part of Python development.
Most APIs that we use to access remote resources, websites, and SaaS will use HTTP requests under the covers, typically via a RESTful interface.
HTTP requests are a type of network I/O and we may want to develop an asyncio program that makes use of the API so that we can have many hundreds or thousands of concurrent HTTP client connections.
The problem is that classical HTTP client libraries block the asyncio event loop.
They are not compatible with asynchronous programming, do not use or support coroutines, and are unable to perform non-blocking IO
Three popular HTTP client libraries include:
- urllib.request: Extensible library for opening URLs (stdlib) (formally urllib2)
- Requests: A simple, yet elegant, HTTP library.
The urllib.request is provided in the Python standard library and can be used in a pinch.
The Requests library is used almost universally as the preferred HTTP client in Python, and it is not comparable with asyncio.
Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your PUT & POST data — but nowadays, just use the json method!
— Requests GitHub Project.
The requests library will block the event loop meaning that while an HTTP request is being made, the coroutine will not suspend, preventing all other coroutines in the event loop from progressing.
It’s like the whole asyncio program is paused until the HTTP request is completed.
We could implement the HTTP protocol ourselves and use asyncio non-blocking streams.
For example:
However we prefer to use high-level libraries rather than re-implementing web protocols.
How can we perform HTTP requests in asyncio programs using standard libraries?
Run loops using all CPUs, download your FREE book to learn how.
How to Adapt Requests For Asyncio
We can use the Requests library API in a way that is compatible with asyncio programs.
This can be achieved by performing HTTP requests in a separate thread that simulates a coroutine.
For example, we can make an HTTP GET request with the Requests library by calling the requests.get() function.
|
1 2 3 |
... # perform http get result = requests.get('https://python.org/') |
We can run this function call in a separate thread via the asyncio.to_thread() module function.
This will run the blocking I/O in a separate worker thread and allow the asyncio program to await it as though it were a non-blocking I/O function call.
For example:
|
1 2 3 |
... # perform a blocking http get request in a separate thread result = await asyncio.to_thread(requests.get, 'https://python.org') |
The example below demonstrates this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of an http client with requests in asyncio import requests import asyncio # main coroutine async def main(): # define the url we want to get url = 'https://python.org/' # perform a blocking http get request in a separate thread result = await asyncio.to_thread(requests.get, url) # report results print(f'Status Code: {result.status_code}') print(f'Content Length: {len(result.text)}') # start the event loop asyncio.run(main()) |
Running the program performs the request and reports the status code and length of all downloaded text.
Importantly, the call to requests.get() is performed in a separate thread that allows all other coroutines in the asyncio event loop to continue to make progress.
You can learn more about how to Async Requests in the tutorial:
This works, however, there are other ways.
We can use an async-native library to perform our HTTP requests.
This may be preferred if we don’t want to spin up a pool of worker threads as a workaround for working with synchronous network I/O calls.
Python Asyncio HTTP Client Libraries
There are two async-native HTTP clients that we can use in asyncio, they are:
- aiohttp: Asynchronous HTTP client/server framework for asyncio and Python
- httpx: A next generation HTTP client for Python
We should also mention the WebSockets project. Although not an HTTP client, it is relevant as many APIs are using WebSockets under the covers instead of RESTful APIs via HTTP. The aiohttp supports WebSockets, whereas httpx does not.
- WebSockets: Library for building WebSocket servers and clients in Python
Anyway, we can plot star rating history to compare these projects to the classical Requests library.
The plot shows the dominance of requests as an HTML client. We can see the rise of aiohttp as asyncio became important in the Python ecosystem and the rapid rise of httpx more recently.
Let’s take a closer look at each in turn.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
aiohttp HTTP Client Library
The aiohttp provides both an HTTP client and server.
This means that although the predominant use of the library is as an HTTP client in asyncio programs, it can also be used as a web microframework to host an API.
Supports both client and server side of HTTP protocol.
— aiohttp GitHub Project.
Supports both client and server Web-Sockets out-of-the-box and avoids Callback Hell.
Provides Web-server with middleware and pluggable routing.
The first step is to install the library using your preferred Python package manager, such as pip.
For example:
|
1 |
pip install aiohttp |
The project also recommends installing the aiodns project.
- aiodns: Simple DNS resolver for asyncio
This is an asyncio library for resolving domain names
aiodns provides a simple way for doing asynchronous DNS resolutions using pycares
— aiodns GitHub Project.
This too can be installed using your preferred Python package manager, such as pip.
For example:
|
1 |
pip install aiodns |
We can then develop HTTP clients in our asyncio programs.
This involves creating an HTTP session and using the session to perform HTTP requests.
The session can be used via an asynchronous context manager, as can HTTP requests. This ensures the session and request are closed automatically when we are done, or exit the code block unexpectedly.
The example below provides an example of making a GET request in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of an http client with aiohttp import asyncio import aiohttp # main coroutine async def main(): # define the url we want to get url = 'https://python.org/' # start a client session async with aiohttp.ClientSession() as session: # perform http get async with session.get(url) as response: # report results print(f'Status Code: {response.status}') # retrieve text text = await response.text() # report length of text print(f'Content Length: {len(text)}') # start the event loop asyncio.run(main()) |
Running the program first creates a client session using the async context manager interface.
Next, a get request is made using the async context manager interface. This suspends the caller while the connection is made and HTTP header information is downloaded.
The HTTP response code is reported.
Next, the content is downloaded, suspending the caller.
The main() coroutine results and reports the length of the content.
This highlights how we can make a simple HTTP GET request in an asyncio program with aiohttp.
|
1 2 |
Status Code: 200 Content Length: 51225 |
A common complaint by Python developers coming from requests to aiohttp is why so much code is required, e.g. two levels of async context managers.
The first time you use aiohttp, you’ll notice that a simple HTTP request is performed not with one, but with up to three steps … It’s especially unexpected when coming from other libraries such as the very popular requests
— The aiohttp Request Lifecycle, aiohttp Documentation
The reason for the context managers is so that the blocking calls are performed asynchronously, allowing other coroutines to run, specifically: the HTTP GET, the downloading of the text, and the closing of the session.
Because aiohttp is asynchronous, its API is designed to make the most out of non-blocking network operations. In code like this, requests will block three times, and does it transparently, while aiohttp gives the event loop three opportunities to switch context
— The aiohttp Request Lifecycle, aiohttp Documentation
Next, let’s take a look at httpx.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
httpx HTTP Client Library
The httpx is newer than aiohttp, having been released only within the last few years.
It seeks to be fully featured, providing asyncio support, but also a classical synchronized API, as well as HTTP/2 support.
HTTPX is a fully featured HTTP client library for Python 3. It includes an integrated command line client, has support for both HTTP/1.1 and HTTP/2, and provides both sync and async APIs.
— httpx GitHub Project.
It was developed by Encode, the group also behind the popular Starlette ASGI web toolkit and the Uvicorn ASGI web server, among many other popular asyncio projects.
The first step is to install httpx using your preferred Python package manager, such as pip.
For example:
|
1 |
pip install httpx |
We can then develop HTTP clients in our asyncio programs with httpx.
This involves creating an HTTP session and using the session to perform HTTP requests.
Like aiohttp an asyncio context manager interface can be used to manage the HTTP session, and unlike aiohttp, we can perform an HTTP GET directly, although a stream interface is available via an async context manager.
The example below demonstrates a simple HTTP client using httpx.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of an http client with httpx import asyncio import httpx # main coroutine async def main(): # define the url we want to get url = 'https://www.python.org/' # start a client session async with httpx.AsyncClient() as client: # perform http get response = await client.get(url) # report the status code print(f'Status Code: {response.status_code}') # retrieve text text = response.text # report length of text print(f'Content Length: {len(text)}') # start the event loop asyncio.run(main()) |
Running the example creates a client session and performs the HTTP GET.
In this case, the header and content are retrieved in a single call.
This highlights how we can perform a simple HTTP request using httpx in asyncio.ß
|
1 2 |
Status Code: 200 Content Length: 51225 |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to use HTTP clients in asyncio.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Viktor Bystrov on Unsplash
Do you have any questions?