Last Updated on September 12, 2022

The ThreadPoolExecutor class in Python can be used to query hundreds of game servers concurrently.

This can dramatically speed up the query process compared to querying each game server sequentially, one by one.

In this tutorial, you will discover how to query Quake 3 game servers faster using ThreadPoolExecutor in Python.

After completing this tutorial, you will know:

• How to query the Quake 3 master server and individual servers for their current status.
• How to use the ThreadPoolExecutor to create and manage thread pools in Python.
• How to speed up the querying of Quake 3 game servers nearly 300% using the ThreadPoolExecutor.

Let’s dive in.

Query Quake3 Servers in Python

Quake III Arena, or Quake 3 for short, is a first person shooter (FPS) computer game that was released in 1999 by id Software.

It remains a popular game today for competitive play and among die-hard gamers.

Players join servers online for play, and each server has a specific configuration that defines the rules used for games played on that server. Additionally, each game server has a current status that indicates the number of players and the current game level that is being played.

Quake servers can be queried for their status using a well-defined socket protocol.

Additionally, a Quake 3 master server is a special type of game server that keeps track of all of the currently active Quake 3 game servers. Similarly, Quake 3 master servers can also be queried for a list of game servers using a well-defined socket protocol.

Quake 3 players typically use third party software to keep track of current master servers and to refresh the current status of all active game servers so that they can choose a game to join online. Additionally, the game itself has a similar interface built in.

We can develop our own Quake 3 server querying program in Python.

Thankfully, there are open source libraries available for querying Quake3 servers so that we don’t have to implement the protocol from scratch.

One example is the pyq3serverlist project maintained by the user cetteup.

The project is based on the “Quake 3 Server List” or q3serverlist PHP project maintained by Jack’lul.

The pyq3serverlist library can be installed using your Python package manager, such as pip. For example:

Once installed, it is relatively easy to use.

A quake 3 master server can be queried by first creating an instance of the PrincipalServer object specifying the host name and port number.

Then the get_servers() function can be called to get a list of all currently active game servers known to the master server, specifying the game protocol number (68 for Quake III Arena).

Once we have a list of servers, we can query each game server in turn by calling the get_status() function on the server object; for example:

This status returns a dictionary of status names to status values from which we can choose specific properties of the game server to display, such as the game server name, current map name, and current number of players.

Let’s develop a simple Quake 3 server query program step by step.

We can divide this process into the following elements:

1. Query a Quake 3 Master Server
2. Query a Quake 3 Game Server
3. Query all Quake 3 Game Servers

Query a Quake 3 Master Server

The first step is to develop an example for querying the Quake 3 master server to get a list of game servers.

Configuring the PrincipalServer is straightforward, as we saw in the previous section.

We will use a master server listed on the “q3serverlist” project page that has the following details:

• host: master.quake3arena.com
• port: 27950
• protocol: 68

For example:

Once connected, we can call the get_servers() function with the protocol number. This function can throw a number of exceptions if the connection fails. Therefore, we must use a try-except block to wrap the call.

We can tie this together into a function named get_quake3_server_list() that takes a tuple containing the master server host and port number and returns a list of server objects or an empty list on error.

We can then call this function and print each server object, which will report the IP address and port of each server.

The complete example is listed below.

Running the example first connects to the master server, then reports the list of servers provided.

An truncated example of the output is listed below.

Now that we know how to get a list of game servers from a master server, let’s look at how to query a single game server for its status.

Query a Quake 3 Game Server

Once we have a list of game server objects, we can query each in turn to determine their status.

As with querying the master server, querying each game server requires some exception handling as the host may be down or may speak a different protocol from what we expect.

In fact, many of the servers received from the master server may, in fact, not be live.

We can iterate game servers, querying each in turn until we get a response, then display the response for review.

For example:

Combining this with the query of the master server from the previous section, the complete example is listed below.

Running the example first reports the number of servers received from the master server.

Then the status of the first game server to respond is displayed.

Your results will differ as the number of servers live at the time you run the code will change, and the first server to respond will also differ.

We can see that, as expected, the status of the server is a dictionary of field names and values summarizing a range of interesting things about the status of the game currently in progress.

Three pieces of information that may interest us include:

• The server name (e.g. 'sv_hostname').
• The current map name (e.g. 'mapname').
• The current number of players (e.g. the length of the 'players' list).

We now have enough information to develop a simple Quake 3 game server query program.

Query All Quake 3 Game Servers (one-by-one)

We can now develop a simple Quake 3 game server query program.

The key part of this program will be the way in which we report the status of each game server.

We selected three key pieces of information to display in the previous section: the server name, map name and number of players.

These fields can be retrieved directly from the dictionary response object from the game server; for example:

The game server names can be pretty wild, such as including non ASCII characters and being very long, perhaps as a type of marketing of servers to game players, a way of standing out.

Nevertheless, we will have to clean up the server names to display them nicely on the command prompt when running the code.

First, let’s trim out all characters in the name that are not ASCII. Python provides the string.printable constant that lists all of the standard ASCII characters. We can enumerate all characters in the game server name and only keep those that are in the printable constant.

We can then trim any leading or trailing white space from the name, just in case.

Finally, we can truncate the length of the name to a fixed length, in this case 40 characters (chosen somewhat arbitrarily).

We will design the program to print one summary line per live game server. It would be nice to have the three fields (server name, map name and number of players) line up, which we can achieve using padding when formatting the output string.

For example:

Tying this together, the query_quake3_server() function below takes a server object returned from querying a master server and returns a string summary of the current status of the server, or None if the server cannot be queried successfully.

Next, we can call our query_quake3_server() function for each server returned from the master server and print the results.

The query_servers() function listed below implements this, taking the details of the master server as input (a tuple of host name and port number), queries the master server for a list of servers, then prints a summary of the status of each game server.

We now have all of the elements for a simple Quake 3 game server query tool.

The complete example is listed below.

Running the example prints the server name, map name, and current number of players on each game server that responds successfully.

Using the padding in the string formatting ensures that all of the fields line up vertically for easy scanning and that the server names are truncated to a sensible length.

Although the program works, it is very slow.

The 650+ game servers took approximately 13 minutes to query when I ran the example.

This is far too long to wait, especially for an impatient gamer looking for a game to join.

We can dramatically speed this up using threads in Python with the ThreadPoolExecutor.

Create a Pool of Worker Threads

We can use the ThreadPoolExecutor to speed up the querying of Quake III Arena servers.

The ThreadPoolExecutor class is provided as part of the concurrent.futures module for easily running concurrent tasks.

The ThreadPoolExecutor provides a pool of worker threads, which is different from the ProcessPoolExecutor that provides a pool of worker processes.

Generally, ThreadPoolExecutor should be used for concurrent IO-bound tasks, like downloading URLs, and the ProcessPoolExecutor should be used for concurrent CPU-bound tasks, like calculating.

Using the ThreadPoolExecutor was designed to be easy and straightforward. It is like the “automatic mode” for Python threads.

1. Create the thread pool by calling ThreadPoolExecutor().
2. Submit tasks and get futures by calling submit() or map().
3. Wait and get results as tasks complete by calling as_completed().
4. Shut down the thread pool by calling shutdown()

Create the Thread Pool

First, a ThreadPoolExecutor instance must be created. By default, it will create a pool with the number of logical CPUs in your system plus four. This is good for most purposes.

You can run tens to hundreds of concurrent IO-bound threads per CPU, although perhaps not thousands or tens of thousands. You can specify the number of threads to create in the pool via the max_workers argument; for example:

Submit Tasks to the Thread Pool

Once created, it can send tasks into the pool to be completed using the submit() function.

This function takes the name of the function to call any and all arguments and returns a Future object.

The Future object is a promise to return the results from the task (if any) and provides a way to determine if a specific task has been completed or not.

The return from a function executed by the thread pool can be accessed via the result() function on the Future object. It will wait until the result is available, if needed, or return immediately if the result is available.

For example:

Get Results as Tasks Complete

The beauty of performing tasks concurrently is that we can get results as they become available, rather than waiting for tasks to be completed in the order they were submitted.

The concurrent.futures module provides an as_completed() function that we can use to get results for tasks as they are completed, just like its name suggests.

We can call the function and provide it a list of Future objects created by calling submit() and it will return future objects as they are completed in whatever order.

For example, we can use a list comprehension to submit the tasks and create the list of Future objects:

Then get results for tasks as they complete in a for loop:

Shutdown the Thread Pool

Once all tasks are completed, we can close down the thread pool, which will release each thread and any resources it may hold (e.g. the stack space).

An easier way to use the thread pool is via the context manager (the with keyword), which ensures it is closed automatically once we are finished with it.

Now that we are familiar with ThreadPoolExecutor and how to use it, let’s look at how we can adapt our program for querying Quake 3 servers to make use of it.

Query Quake 3 Servers Concurrently

Our Quake 3 game server query tool can be adapted to use ThreadPoolExecutor directly.

This is because querying each game server is an independent task; it does not have a dependency on querying other game servers. This means we can perform the game server query operations in separate Python threads.

Specifically, we can call query_quake3_server() in a separate thread for each server object returned from the master server.

The slowest part of this function will be the call to server.get_status() and performing this operation in separate threads will give us a large speed up. A further improvement would be to separate the code so that only this call was executed in a separate thread and the code for distilling the status dictionary into a string for printing could be performed in the main thread.

We can create a thread pool and call the submit() function with the query_quake3_server() function and each server object in the master list; for example:

How many threads should we use in the pool?

Each server has a separate IP address and port; therefore, we don’t have to worry about spamming (denial of service) a single server. With modern hardware and a fast internet connection, we can easily query hundreds of game servers concurrently, e.g. hundreds of parallel threads in Python.

After some testing, I found that 500 threads works well for this example so that is what we will use. You may want to experiment and see what works well for your specific workstation.

We now have a list of Future objects that may finish at different times.

It would be nice to report the status of game servers as they become available, rather than wait for all queries to be completed.

This can be achieved using the as_completed() function, passing in the list of Future objects. It will return each Future object as it is completed.

Calling the result() function in the future will either give a string summary of the server or None if the query was unsuccessful. We can then print the line summary as we did in the non-threaded example.

Tying this together, the updated version of the query_servers() function to use the thread pool for querying game servers concurrently is listed below.

We can now test out the concurrent version of our game server query tool.

If each game server takes about one second to respond or fail and we have about 650 servers, then we might have results within about two seconds, compared to 13 minutes in the non-threaded version.

The complete example is listed below.

Running the example queries all game servers returned from the master server concurrently, with up to 500 concurrent connections.

Below is a truncated version of the results.

It is dramatically faster.

In my case, it took about 2.9 seconds to complete as opposed to about 13 minutes, or about 260% faster!

Now that the program is responsive, we could start to think about other concerns for the user, such as filtering out game servers that have no players or are full and cannot accept new players.

We are using one master server to get a list of game servers to query, but what if we wanted to get lists of servers from multiple master servers?

Query Quake 3 Master Servers Concurrently

We can update the example to first query a number of master servers to create a large list of game servers, then query this list of game servers concurrently as we did previously.

First, we need a list of Quake 3 master servers.

A Google search reveals a site dedicated to listing Quake 3 master servers:

• Quake 3 Arena Master Servers

This gives a list of 8 servers we can use.

Below are these servers in a list of tuples.

Next, we need to query each master server in turn.

It might take one second or more to query each master server, and with eight servers doing this sequentially, it would take up to eight seconds.

Instead, we can query the master servers concurrently, as we did with the game servers themselves in the previous section.

In this case, we will use the map() function to apply the get_quake3_server_list() function that we developed previously to each master server tuple (host name and port number).

This will give us an iterable of results once all queries have been completed.

Notice that we set the number of threads to be the number of master servers, in this case eight.

This will give an iterable over lists of game servers that we can flatten to a single list of game servers using a list comprehension.

We would expect some overlap in the game servers reported by each master server.

As such, we would only be interested in a list of unique servers, such as unique server IP address and port number combinations.

We can achieve this by storing all server objects in a dictionary using a string of IP addresses and port numbers as the key. If a server already exists in the dictionary, we can simply replace it.

Given that we’re only interested in the server objects themselves, we can then retrieve the values from the dictionary.

This list of game servers can then be submitted to another thread pool to be queried, as we did before.

We can tie all of this together into a function named get_all_quake3_game_servers() that will take a list of master servers and return a list of unique Quake 3 game servers.

It also reports progress along the way, including the number of game servers total and the size of the distilled list of unique game servers.

An alternative approach would be to use the submit() function and use the as_complete() function on the list of futures returned, then add the game servers to the dictionary one at a time.

This would be functionally equivalent and perhaps slightly faster if some master servers take much longer to respond than others.

We can then update our query_servers() function to call our get_all_quake3_game_servers() function to get the list of game servers, then query each game server using a thread pool as before.

An alternate design might be to create a pipeline of thread pools connected via Queue objects. We could then feed master servers in at one end (the first thread pool), which would pass servers on to the second pool, and finally give us individual game server summary strings as the output.

This might be a sticky design as careful attention would be needed to ensure only unique game servers are queried in the second thread pool. Nevertheless, it would allow for expansion of functionality, such as another step in the pipeline for filtering the results prior to formatting.

Nevertheless, tying our code together, the complete example of querying multiple master servers using a thread pool, then querying the list of game servers using a second thread pool is listed below.

Running the example first queries all master game servers and creates a master list of unique game servers, then queries all game servers and reports their status.

In this case, we can see that a total of about 2,200 game servers were discovered from the eight master servers compared to the 650 discovered when using a single master server. This list was then distilled to 884 unique game servers, still more than the single game server used in previous examples.

The cost of querying multiple game servers did add about two seconds to the execution time.

Extensions

This section lists ideas for extending the tutorial.

• Separate Server Query From Formatting: Update the example so that only the API call to query the game server is performed in a separate thread and that the formatting of the server status is performed in the main thread.
• Optimize Query Threads: Update the example to time different numbers of query threads in order to discover what works best for your specific workstation hardware and internet connection.
• Filter Server Results: Update the example to filter out game servers that have no players or are full and cannot accept new players.
• Use a Pipeline: Update the final example to use a pipeline of thread pools connected by Queue objects and add an additional step in the pipeline to filter servers before formatting their result.
• Create a graphical interface: Update the example to use a graphical user interface to present the status of all queried game servers.

Share your extensions in the comments below; it would be great to see what you come up with.

Further Reading

This section provides additional resources that you may find helpful.

Books

• ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
• Concurrent Futures API Interview Questions
• ThreadPoolExecutor Class API Cheat Sheet

I also recommend specific chapters from the following books:

• Effective Python, Brett Slatkin, 2019.
  • See Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPoolExecutor: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• concurrent.futures - Launching parallel tasks
• threading — Thread-based parallelism

References

• Thread (computing), Wikipedia.
• Futures and promises, Wikipedia.
• Thread pool, Wikipedia.

Takeaways

In this tutorial, you discovered how to query Quake 3 game servers faster using ThreadPoolExecutor in Python.

• How to query the Quake 3 master server and individual servers for their current status.
• How to use the ThreadPoolExecutor to create and manage thread pools in Python.
• How to speed up the querying of Quake 3 game servers nearly 300% using the ThreadPoolExecutor.

Do you have any questions?
Leave your question in a comment below and I will reply fast with my best advice.

Photo by Robert Bye on Unsplash