Last Updated on December 12, 2023

We can make Async Requests in Python.

The Requests Python library does not support asyncio directly.

If we make HTTP requests using the Requests library, it will block the asyncio event loop and prevent all other coroutines in the program from progressing.

Instead, we can make async requests using the asyncio.to_thread() method provided in the asyncio module in the Python standard library. This will run the blocking network I/O call in a separate worker thread, simulating an asynchronous or non-blocking I/O function call.

In this tutorial, you will discover how to safely use the Requests library for HTTP requests in asyncio programs.

Let’s get started.

Requests Block the Asyncio Event Loop (this is bad)

HTTP clients are a large part of Python development.

Most APIs that we use to access remote resources, websites, and SaaS will use HTTP requests under the covers, typically via a RESTful interface.

HTTP requests are a type of network I/O and we may want to develop an asyncio program that makes use of the API so that we can have many hundreds or thousands of concurrent HTTP client connections.

The most popular Python library for making HTTP requests is the Requests library.

• Requests: A simple, yet elegant, HTTP library.

Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your PUT & POST data — but nowadays, just use the json method!

— Requests GitHub Project.

The problem is, the Requests library cannot be used directly in asyncio programs.

The reason is because the Requests library will block the asyncio event loop while opening, writing to, and reading from the HTTP socket connection.

This means that the entire asyncio program will be paused until the HTTP request is completed.

This is bad, because the whole idea of adopting asyncio is to use asynchronous programming to run many tasks concurrently and to perform non-blocking network I/O.

How can we use requests safely in asyncio programs?

How to Use Requests in Asyncio

We can use the Requests library to make HTTP requests safely in asyncio programs by running the blocking call in a new thread.

This can be achieved using the asyncio.to_thread() module function.

The asyncio.to_thread() will make use of a ThreadPoolExecutor behind the scenes to execute the call in a new thread concurrently with the asyncio program.

We need to run the blocking network request in a new thread because asyncio runs all coroutines in one thread. If one coroutine in the asyncio program blocks the current thread, it blocks all coroutines in the thread.

By executing the blocking call in a separate thread, it allows the event loop to continue to run and progress all other coroutines that are running in the program.

The asyncio.to_thread() module function can be treated like a non-blocking call, allowing us to simulate asynchronous requests or non-blocking network I/O with the requests library.

A typical HTTP GET request made with the Requests library is made via the requests.get() function.

For example:

This will block the event loop.

We can make this call in a new thread by awaiting a call to asyncio.to_thread().

For example:

Notice that the asyncio.to_thread() takes the name of the method or function to call and each argument to provide to the target function.

You can learn more about the asyncio.to_thread() function in the tutorial:

• How to Use Asyncio to_thread()

You can learn more about handling blocking tasks in asyncio programs in the tutorial:

• How to Run Blocking Tasks in Asyncio

Another solution would be to use an alternative to the Requests library that supports asyncio directly, e.g. an async-first HTTP client library. Popular examples include aiohttp and httpx.

Now that we know how we can use the Requests library in asyncio safely, let’s look at some worked examples.

Example of Requests API (regular Python)

Firstly, let’s look at how to make a simple HTTP GET request using the Requests library.

The example below makes a request and reports the HTTP status and the length of the content.

Running the program performs the request and reports the status code and length of all downloaded text.

We can see that it is a very clean and simple API.

The HTTP GET is a blocking call that opens a network connection, sends an HTTP request, and downloads an HTTP response, then parses the response into structures we can access.

Example of Requests API in Asyncio (blocking the event loop)

We can call the requests.get() in asyncio programs, but it will block the event loop, as we have discussed.

The example below updates the example to run in the asyncio event loop.

The example works, what’s the problem?

The problem is that when we call requests.get() it blocks the thread.

This means that any other coroutines we might have running in our asyncio event loop will not progress.

Blocking the event loop is an anti-pattern in asyncio programs. A bad practice. Antithetical. We use asyncio so that our tasks can cooperate and execute concurrently.

Example Confirming Requests API Blocks The Asyncio Event Loop

Let’s make this obvious by running another coroutine in the background.

We can define a coroutine that prints a message and sleeps every 10 milliseconds.

We can start this task first, and allow it to begin running immediately.

We can then perform our request and report a message right before we block the thread with our request.

Tying this together, the complete example is listed below.

Running the example first starts our background task and allows it to run.

The background task gets one opportunity to run before sleeping.

The main() coroutine resumes, prints a message and makes the HTTP GET request.

This takes a moment and blocks the asyncio event loop for the duration. Our background task is unable to progress and print messages.

The GET finishes, the main() coroutine reports status, exits, and cancels the background task as the event loop is shut down.

We must run the blocking call in a way that does not block the event loop.

Example Running Requests API Calls in A Separate Thread

We can perform blocking Requests API calls in a separate thread in our asyncio programs.

This can be achieved via the asyncio.to_thread() function that will use a pool of worker threads behind the scenes to execute blocking tasks.

We can then await the call like any other coroutine.

For example:

The updated version of our program with this change is listed below.

The program runs as before, with one important difference.

This does not block the current thread.

The background task is created and started, run one time before the main() coroutine resumes.

The main() coroutine then reports the message “Starting request” and then suspends as it makes the HTTP GET request in a separate thread.

This allows all other coroutines in the event loop to progress. We only have one other coroutine running, our background task, which runs a bunch of times in its loop, happily reporting messages.

The GET request eventually finishes, and the main() coroutine resumes and reports the details before shutting down the event loop.

This highlights firstly how we can run blocking HTTP calls in the Requests library in a separate thread within our asyncio programs. Secondly, it highlights how doing so allows the event loop to progress other coroutines running while the blocking call is running.

This works, however, there are other ways.

We can use an async-native library to perform our HTTP requests.

This may be preferred if we don’t want to spin up a pool of worker threads as a workaround for working with synchronous network I/O calls.

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know how to use the Requests library in asyncio.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Paolo Chiabrando on Unsplash