Last Updated on September 12, 2022

The ThreadPoolExecutor class in Python can be used to check the status of multiple websites concurrently.

Using threads to check website status can dramatically speed-up the process compared to checking the status of each website sequentially, one by one.

In this tutorial, you will discover how to concurrently check the status of websites on the internet using thread pools in Python.

After completing this tutorial, you will know:

• How to check the status of websites on the internet sequentially in Python and how slow it can be.
• How to use the ThreadPoolExecutor to manage a pool of worker threads.
• How to update code to check multiple websites at the same time and dramatically accelerate the process.

Let’s dive in.

Check the Status of Websites One-by-One (slowly)

Checking the status of websites on the internet is a common task.

For example:

• You may need to check if a suite of company websites are accessible and take action if they are not.
• You may need to check if websites start generating errors, such as under heavy load.
• You may need to check if your internet connection is working correctly.

There are professional services we may use for some of these tasks. Nevertheless, sometimes we need a quick script to check on the status of a number of websites.

There are many ways to check on the status of a website in Python via HTTP.

Perhaps the most common is to open a connection to the server using the urlopen() function, then call the getcode() function on the open HTTPSConnection.

You can learn more about these standard library functions here:

• urllib.request – Extensible library for opening URLs
• http.client — HTTP protocol client

For example, we can use the context manager to open the HTTP connection and get the status code, which will ensure that the connection will be closed automatically when we exit the block either normally, or if an exception is raised.

Also, given that we may be calling this script many times under possibly intermittent internet connectivity, we may also want to add a timeout to the connection attempt, perhaps a few seconds.

This can be achieved when calling the urlopen() function via the timeout argument.

Opening an HTTP connection can fail for many reasons, such as malformed URLs to servers.

We can choose to handle some of these cases and return an appropriate message.

For example, if we are able to connect but the server reports an error, then an HTTPError will be raised on which we can retrieve the specific HTTP status code.

Alternatively, a URLError could be thrown if we are unable to connect, perhaps because the domain could not be resolved via DNS, in which case we might choose to retrieve the error message as the status.

Finally, if something else happens, and this is always possible, then we can retrieve traps and return more general faults.

You can learn more about these errors here:

• urllib.error — Exception classes raised by urllib.request

Tying this together, the get_website_status() function below takes a URL and returns the HTTP status, or a useful error message if there is a failure.

Once we have an error status, we need to interpret it.

Python offers the http module and the HTTPStatus class with constants for the most common HTTP status codes.

Nevertheless, we can keep our script simple by checking for a status 200 (OK), otherwise reporting an error.

The get_status() function below takes an HTTP status code returned from the get_website_status() function and interprets it with a human readable message as 'OK' or 'ERROR'.

Next, we need to get the status, interpret it, and report the results for each URL in a list of URLs.

This ties the program together.

The list of URLs may be loaded from a database or a text file in practice. In this case, we will define a constant named URLS that lists ten of the most popular websites on the internet in no particular order.

We can then define a function that takes a list of URLs and gets the status for each with a call first to the get_website_status() function to get the HTTP code, then to get_status() to interpret the code, before calling the print function to report the status.

Tying this together, the named check_status_urls() function that reports on the status of a list of URLs is listed below.

We can then call this function without a predefined list of URLs.

And that’s it.

Tying this together, the complete example of checking the status of a list of URLs is listed below.

Running the example gets the status code, which is then interpreted and reported for each URL in the list.

In this case, all URLs report an OK status except amazon.com, which reports a 503 error. It might be because amazon prefers the www prefix, or because it does not like to be pinged by unknown web clients.

Checking the status of websites one-by-one is slow.

On my system, it took a little less than ten seconds to query all ten websites.

How long did it take to run on your system?
Let me know in the comments below.

Next, we will look at the ThreadPoolExecutor class that can be used to create a pool of worker threads that will allow us to speed up the status checking process.

Create a Pool of Worker Threads With ThreadPoolExecutor

We can use the ThreadPoolExecutor to speed up the download of multiple files listed on an HTML webpage.

The ThreadPoolExecutor class is provided as part of the concurrent.futures module for easily running concurrent tasks.

The ThreadPoolExecutor provides a pool of worker threads, which is different from the ProcessPoolExecutor that provides a pool of worker processes.

Generally, ThreadPoolExecutor should be used for concurrent IO-bound tasks, like downloading URLs, and the ProcessPoolExecutor should be used for concurrent CPU-bound tasks, like calculating.

Using the ThreadPoolExecutor was designed to be easy and straightforward. It is like the “automatic mode” for Python threads.

1. Create the thread pool by calling ThreadPoolExecutor().
2. Submit tasks and get futures by calling submit().
3. Wait and get results as tasks complete by calling as_completed().
4. Shutdown the thread pool by calling shutdown()

Step 1: Create the Thread Pool

First, a ThreadPoolExecutor instance must be created.

By default, it will create a pool equal to the number of logical CPUs you have available plus four.

This is good for most purposes.

You can run tens to hundreds of concurrent IO-bound threads per CPU, although perhaps not thousands or tens of thousands.

You can specify the number of threads to create in the pool via the max_workers argument; for example:

Step 2: Submit Tasks to the Thread Pool

Once created, it can send tasks into the pool to be completed using the submit() function.

This function takes the name of the function to call any and all arguments and returns a Future object.

The Future object is a promise to return the results from the task (if any) and provides a way to determine if a specific task has been completed or not.

The return from a function executed by the thread pool can be accessed via the result() function on the Future object. It will wait until the result is available, if needed, or return immediately if the result is available.

For example:

An alternate and perhaps simpler approach to submitting asynchronous tasks to the thread pool is to use the map() function provided on the ThreadPoolExecutor.

It works just the same as the built-in map() function. It takes the name of the target task function and an iterable. The target function is then called for each item in the iterable, and because we are using a thread pool, each call is submitted as an asynchronous task.

The map() function then returns an iterable over the results from each function call. We can call the map() function and iterate over the results in a for-loop directly; for example:

Step 3: Get Results as Tasks Complete

The beauty of performing tasks concurrently is that we can get results as they become available, rather than waiting for tasks to be completed in the order they were submitted.

The concurrent.futures module provides an as_completed() function that we can use to get results for tasks as they are completed, just like its name suggests.

We can call the function and provide it a list of future objects created by calling submit() and it will return future objects as they are completed in whatever order.

For example, we can use a list comprehension to submit the tasks and create the list of Future objects:

Then get results for tasks as they complete in a for loop:

Step 4: Shutdown the Thread Pool

Once all tasks are completed, we can close down the thread pool, which will release each thread and any resources it may hold (e.g. the stack space).

An easier way to use the thread pool is via the context manager (the with keyword), which ensures it is closed automatically once we are finished with it.

Now that we are familiar with ThreadPoolExecutor and how to use it, let’s look at how we can adapt our program for checking the status of websites to make use of it.

Check the Status of Websites Concurrently

We can update our program to make use of the ThreadPoolExecutor with very few changes.

All of the changes are contained to the check_status_urls() function.

First, we can create a thread pool using the context manager and set the number of worker threads to the number of URLs in our list. This way, we can attempt to query all ten websites at once.

Next, we need to perform asynchronous calls to the get_website_status() function.

Perhaps the simplest approach is to use the map() function on the ThreadPoolExecutor.

Each task is a call to the get_website_status() with a URL argument that returns an HTTP response code that we can then interpret and report.

The problem with this approach is that we have no way to associate the HTTP response code with each URL.

We could add a counter and increment it as we iterate through the results, then use the counter to access the appropriate address in the list of URLs.

A better version of this solution is to first retrieve the iterable of results from the call to map().

We can then call the enumerate() built-in function on the iterable to get an index and the item from the iterable for each iteration in a for loop. The index can be used directly to get the URL in the list for the result.

For example:

Tying this together, the complete example of querying a list of websites asynchronously using a ThreadPoolExecutor in Python is listed below.

Running the example reports the websites and their status as before, but this time, the code is much faster.

On my system, all sites are queried in about 1.5 seconds, which is much faster than nearly ten seconds in the sequential version, about a 5x speedup.

What if we wanted to use the submit() function to dispatch the tasks and report the status for results as the tasks are completed, rather than the order that the URLs are listed?

This would require a slightly more complicated solution.

Recall that the submit() function returns a Future object that provides a handle on the asynchronous task. For example, we can use a list comprehension to create a list of Future objects by calling submit() for the get_website_status() function with each URL in the list of URLs.

The tricky part is we don’t have a way to associate the Future or its result with the URL.

We could enumerate the futures in the same order as the list of URLs and use the enumerate() function as we did with the iterable of results returned when calling map(); for example:

This would work, but it is a more complicated version of calling map(); we may as well just call map().

It also does not give us the benefit of reporting website status as it becomes available via the as_completed() function.

An alternative approach is to create a dictionary that maps each Future object to the data that was passed to the Future’s task, in this case, the URL argument to get_website_status(). This idea is also suggested in the ThreadPoolExecutor’s API documentation.

For example:

We can then call the as_completed() function with the future_to_url dictionary and process the Future objects in the order that they are completed.

We can then look-up the URL value in the future_to_url dictionary for the given Future key; for example:

We now have everything we need to report website status in the order that the status is returned, rather than the order that we query the websites. This might be slightly faster and more responsive than the linear version.

The updated version of the check_status_urls() function is listed below.

Tying this together, the complete example of querying websites and reporting results in task completion order is listed below.

Running the example, we can see that website statuses are reported in a different order from the list of URLs.

Specifically, the results are reported in the order that websites responded to the query. We can see that Amazon was the slowest to respond, likely because it does not like the specific address or Python client we are using to make the query.

On my system, this example is also slightly faster, shaving off a fraction of a second.

Further Reading

This section provides additional resources that you may find helpful.

Books

• ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
• Concurrent Futures API Interview Questions
• ThreadPoolExecutor Class API Cheat Sheet

I also recommend specific chapters from the following books:

• Effective Python, Brett Slatkin, 2019.
  • See Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPoolExecutor: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• concurrent.futures - Launching parallel tasks
• threading — Thread-based parallelism

References

• Thread (computing), Wikipedia.
• Futures and promises, Wikipedia.
• Thread pool, Wikipedia.

Takeaways

In this tutorial you discovered how to concurrently check the status of websites on the internet using thread pools in Python.

• How to check the status of websites on the internet sequentially in Python and how slow it can be.
• How to use the ThreadPoolExecutor to manage a pool of worker threads.
• How to update code to check multiple websites at the same time and dramatically accelerate the process.

Do you have any questions?
Leave your question in a comment below and I will reply fast with my best advice.

Photo by Simon Connellan on Unsplash