Asynchronous Requests in Python

December 9, 2023 Python Asyncio

We can make Async Requests in Python.

The Requests Python library does not support asyncio directly.

If we make HTTP requests using the Requests library, it will block the asyncio event loop and prevent all other coroutines in the program from progressing.

Instead, we can make async requests using the asyncio.to_thread() method provided in the asyncio module in the Python standard library. This will run the blocking network I/O call in a separate worker thread, simulating an asynchronous or non-blocking I/O function call.

In this tutorial, you will discover how to safely use the Requests library for HTTP requests in asyncio programs.

Let's get started.

Requests Block the Asyncio Event Loop (this is bad)

HTTP clients are a large part of Python development.

Most APIs that we use to access remote resources, websites, and SaaS will use HTTP requests under the covers, typically via a RESTful interface.

HTTP requests are a type of network I/O and we may want to develop an asyncio program that makes use of the API so that we can have many hundreds or thousands of concurrent HTTP client connections.

The most popular Python library for making HTTP requests is the Requests library.

Requests: A simple, yet elegant, HTTP library.

Requests allows you to send HTTP/1.1 requests extremely easily. There's no need to manually add query strings to your URLs, or to form-encode your PUT & POST data — but nowadays, just use the json method!
-- Requests GitHub Project.

The problem is, the Requests library cannot be used directly in asyncio programs.

The reason is because the Requests library will block the asyncio event loop while opening, writing to, and reading from the HTTP socket connection.

This means that the entire asyncio program will be paused until the HTTP request is completed.

This is bad, because the whole idea of adopting asyncio is to use asynchronous programming to run many tasks concurrently and to perform non-blocking network I/O.

How can we use requests safely in asyncio programs?

How to Use Requests in Asyncio

We can use the Requests library to make HTTP requests safely in asyncio programs by running the blocking call in a new thread.

This can be achieved using the asyncio.to_thread() module function.

The asyncio.to_thread() will make use of a ThreadPoolExecutor behind the scenes to execute the call in a new thread concurrently with the asyncio program.

We need to run the blocking network request in a new thread because asyncio runs all coroutines in one thread. If one coroutine in the asyncio program blocks the current thread, it blocks all coroutines in the thread.

By executing the blocking call in a separate thread, it allows the event loop to continue to run and progress all other coroutines that are running in the program.

The asyncio.to_thread() module function can be treated like a non-blocking call, allowing us to simulate asynchronous requests or non-blocking network I/O with the requests library.

A typical HTTP GET request made with the Requests library is made via the requests.get() function.

For example:

...
# perform http get
result = requests.get('https://python.org/')

This will block the event loop.

We can make this call in a new thread by awaiting a call to asyncio.to_thread().

For example:

...
# perform http get
result = await asyncio.to_thread(requests.get, 'https://python.org/')

Notice that the asyncio.to_thread() takes the name of the method or function to call and each argument to provide to the target function.

You can learn more about the asyncio.to_thread() function in the tutorial:

How to Use Asyncio to_thread()

You can learn more about handling blocking tasks in asyncio programs in the tutorial:

How to Run Blocking Tasks in Asyncio

Another solution would be to use an alternative to the Requests library that supports asyncio directly, e.g. an async-first HTTP client library. Popular examples include aiohttp and httpx.

Now that we know how we can use the Requests library in asyncio safely, let's look at some worked examples.

Example of Requests API (regular Python)

Firstly, let's look at how to make a simple HTTP GET request using the Requests library.

The example below makes a request and reports the HTTP status and the length of the content.

# SuperFastPython.com
# example of an http client with requests
import requests
# define the url we want to get
url = 'https://python.org/'
# perform http get
result = requests.get(url)
# report results
print(f'Status Code: {result.status_code}')
print(f'Content Length: {len(result.text)}')

Running the program performs the request and reports the status code and length of all downloaded text.

We can see that it is a very clean and simple API.

The HTTP GET is a blocking call that opens a network connection, sends an HTTP request, and downloads an HTTP response, then parses the response into structures we can access.

Status Code: 200
Content Length: 51225

Example of Requests API in Asyncio (blocking the event loop)

We can call the requests.get() in asyncio programs, but it will block the event loop, as we have discussed.

The example below updates the example to run in the asyncio event loop.

# SuperFastPython.com
# example of an http client with requests in asyncio
import requests
import asyncio

# main coroutine
async def main():
    # define the url we want to get
    url = 'https://python.org/'
    # perform a blocking http get request
    result = requests.get(url)
    # report results
    print(f'Status Code: {result.status_code}')
    print(f'Content Length: {len(result.text)}')

# start the event loop
asyncio.run(main())

The example works, what's the problem?

The problem is that when we call requests.get() it blocks the thread.

This means that any other coroutines we might have running in our asyncio event loop will not progress.

Blocking the event loop is an anti-pattern in asyncio programs. A bad practice. Antithetical. We use asyncio so that our tasks can cooperate and execute concurrently.

Status Code: 200
Content Length: 51225

Example Confirming Requests API Blocks The Asyncio Event Loop

Let's make this obvious by running another coroutine in the background.

We can define a coroutine that prints a message and sleeps every 10 milliseconds.

# some background task
async def background():
    while True:
        print('Running in the background...')
        await asyncio.sleep(0.01)

We can start this task first, and allow it to begin running immediately.

...
# create the background task
task = asyncio.create_task(background())
# allow the background task to start executing
await asyncio.sleep(0)

We can then perform our request and report a message right before we block the thread with our request.

...
# perform a blocking http get request in a separate thread
print('Starting request')
result = requests.get(url)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of an http client with requests in asyncio
import requests
import asyncio

# some background task
async def background():
    while True:
        print('Running in the background...')
        await asyncio.sleep(0.01)

# main coroutine
async def main():
    # create the background task
    task = asyncio.create_task(background())
    # allow the background task to start executing
    await asyncio.sleep(0)
    # define the url we want to get
    url = 'https://python.org/'
    # perform a blocking http get request in a separate thread
    print('Starting request')
    result = requests.get(url)
    # report results
    print(f'Status Code: {result.status_code}')
    print(f'Content Length: {len(result.text)}')

# start the event loop
asyncio.run(main())

Running the example first starts our background task and allows it to run.

The background task gets one opportunity to run before sleeping.

The main() coroutine resumes, prints a message and makes the HTTP GET request.

This takes a moment and blocks the asyncio event loop for the duration. Our background task is unable to progress and print messages.

The GET finishes, the main() coroutine reports status, exits, and cancels the background task as the event loop is shut down.

We must run the blocking call in a way that does not block the event loop.

Running in the background...
Starting request
Status Code: 200
Content Length: 51225

Example Running Requests API Calls in A Separate Thread

We can perform blocking Requests API calls in a separate thread in our asyncio programs.

This can be achieved via the asyncio.to_thread() function that will use a pool of worker threads behind the scenes to execute blocking tasks.

We can then await the call like any other coroutine.

For example:

...
# perform a blocking http get request in a separate thread
result = await asyncio.to_thread(requests.get, url)

The updated version of our program with this change is listed below.

# SuperFastPython.com
# example of an http client with requests in asyncio
import requests
import asyncio

# some background task
async def background():
    while True:
        print('Running in the background...')
        await asyncio.sleep(0.01)

# main coroutine
async def main():
    # create the background task
    task = asyncio.create_task(background())
    # allow the background task to start executing
    await asyncio.sleep(0)
    # define the url we want to get
    url = 'https://python.org/'
    # perform a blocking http get request in a separate thread
    print('Starting request')
    result = await asyncio.to_thread(requests.get, url)
    # report results
    print(f'Status Code: {result.status_code}')
    print(f'Content Length: {len(result.text)}')

# start the event loop
asyncio.run(main())

The program runs as before, with one important difference.

This does not block the current thread.

The background task is created and started, run one time before the main() coroutine resumes.

The main() coroutine then reports the message “Starting request” and then suspends as it makes the HTTP GET request in a separate thread.

This allows all other coroutines in the event loop to progress. We only have one other coroutine running, our background task, which runs a bunch of times in its loop, happily reporting messages.

The GET request eventually finishes, and the main() coroutine resumes and reports the details before shutting down the event loop.

This highlights firstly how we can run blocking HTTP calls in the Requests library in a separate thread within our asyncio programs. Secondly, it highlights how doing so allows the event loop to progress other coroutines running while the blocking call is running.

Running in the background...
Starting request
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Running in the background...
Status Code: 200
Content Length: 51435

This works, however, there are other ways.

We can use an async-native library to perform our HTTP requests.

This may be preferred if we don't want to spin up a pool of worker threads as a workaround for working with synchronous network I/O calls.

Takeaways

You now know how to use the Requests library in asyncio.

If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.