How to Query Quake 3 Servers Concurrently in Python

November 28, 2021 Python ThreadPoolExecutor

The ThreadPoolExecutor class in Python can be used to query hundreds of game servers concurrently.

This can dramatically speed up the query process compared to querying each game server sequentially, one by one.

In this tutorial, you will discover how to query Quake 3 game servers faster using ThreadPoolExecutor in Python.

After completing this tutorial, you will know:

How to query the Quake 3 master server and individual servers for their current status.
How to use the ThreadPoolExecutor to create and manage thread pools in Python.
How to speed up the querying of Quake 3 game servers nearly 300% using the ThreadPoolExecutor.

Let's dive in.

Query Quake3 Servers in Python

Quake III Arena, or Quake 3 for short, is a first person shooter (FPS) computer game that was released in 1999 by id Software.

It remains a popular game today for competitive play and among die-hard gamers.

Players join servers online for play, and each server has a specific configuration that defines the rules used for games played on that server. Additionally, each game server has a current status that indicates the number of players and the current game level that is being played.

Quake servers can be queried for their status using a well-defined socket protocol.

Additionally, a Quake 3 master server is a special type of game server that keeps track of all of the currently active Quake 3 game servers. Similarly, Quake 3 master servers can also be queried for a list of game servers using a well-defined socket protocol.

Quake 3 players typically use third party software to keep track of current master servers and to refresh the current status of all active game servers so that they can choose a game to join online. Additionally, the game itself has a similar interface built in.

We can develop our own Quake 3 server querying program in Python.

Thankfully, there are open source libraries available for querying Quake3 servers so that we don't have to implement the protocol from scratch.

One example is the pyq3serverlist project maintained by the user cetteup.

The project is based on the "Quake 3 Server List" or q3serverlist PHP project maintained by Jack'lul.

The pyq3serverlist library can be installed using your Python package manager, such as pip. For example:

pip3 install pyq3serverlist

Once installed, it is relatively easy to use.

A quake 3 master server can be queried by first creating an instance of the PrincipalServer object specifying the host name and port number.

Then the get_servers() function can be called to get a list of all currently active game servers known to the master server, specifying the game protocol number (68 for Quake III Arena).

...
# connect to the master server
principal = PrincipalServer(host, port)
# get a list of game servers
servers = principal.get_servers(master_protocol)

Once we have a list of servers, we can query each game server in turn by calling the get_status() function on the server object; for example:

...
for server in servers:
	status = server.get_status()

This status returns a dictionary of status names to status values from which we can choose specific properties of the game server to display, such as the game server name, current map name, and current number of players.

Let's develop a simple Quake 3 server query program step by step.

We can divide this process into the following elements:

Query a Quake 3 Master Server
Query a Quake 3 Game Server
Query all Quake 3 Game Servers

Query a Quake 3 Master Server

The first step is to develop an example for querying the Quake 3 master server to get a list of game servers.

Configuring the PrincipalServer is straightforward, as we saw in the previous section.

We will use a master server listed on the "q3serverlist" project page that has the following details:

host: master.quake3arena.com
port: 27950
protocol: 68

For example:

...
# configure the master server and connect
principal = PrincipalServer('master.quake3arena.com', 27950)

Once connected, we can call the get_servers() function with the protocol number. This function can throw a number of exceptions if the connection fails. Therefore, we must use a try-except block to wrap the call.

...
try:
    # retrieve a list of servers
    servers = principal.get_servers(68)
except:
    # an empty list if we cannot query the server
    servers = []

We can tie this together into a function named get_quake3_server_list() that takes a tuple containing the master server host and port number and returns a list of server objects or an empty list on error.

# query a master server and get a list of game servers
def get_quake3_server_list(master_server, master_protocol=68):
    # configure the master server and connect
    principal = PrincipalServer(master_server[0], master_server[1])
    try:
        # retrieve a list of servers
        return principal.get_servers(master_protocol)
    except:
        # return an empty list if we cannot query the server
        return []

We can then call this function and print each server object, which will report the IP address and port of each server.

The complete example is listed below.

# SuperFastPython.com
# query a quake3 master server for a list of game servers
from pyq3serverlist import PrincipalServer

# query a master server and get a list of game servers
def get_quake3_server_list(master_server, master_protocol=68):
    # configure the master server and connect
    principal = PrincipalServer(master_server[0], master_server[1])
    try:
        # retrieve a list of servers
        return principal.get_servers(master_protocol)
    except:
        # return an empty list if we cannot query the server
        return []

# entry point
MASTER_SERVER = ('master.quake3arena.com', 27950)
# query for a list of game server
servers = get_quake3_server_list(MASTER_SERVER)
# report the number of servers
print(f'Found {len(servers)} Quake 3 Arena servers.')
# report list
for server in servers:
    print(server)

Running the example first connects to the master server, then reports the list of servers provided.

An truncated example of the output is listed below.

Found 652 Quake 3 Arena servers.
149.28.199.8:29588
198.46.223.132:28408
195.201.30.139:27960
185.107.96.124:1041
97.90.63.254:27961
190.220.150.154:27962
104.196.190.1:27960
149.28.199.8:28819
149.28.199.8:29852
...

Now that we know how to get a list of game servers from a master server, let's look at how to query a single game server for its status.

Query a Quake 3 Game Server

Once we have a list of game server objects, we can query each in turn to determine their status.

As with querying the master server, querying each game server requires some exception handling as the host may be down or may speak a different protocol from what we expect.

In fact, many of the servers received from the master server may, in fact, not be live.

We can iterate game servers, querying each in turn until we get a response, then display the response for review.

For example:

...
# get status for one server
for server in servers:
    try:
        status = server.get_status()
        print(status)
        break
    except:
        continue

Combining this with the query of the master server from the previous section, the complete example is listed below.

# SuperFastPython.com
# query the status of one game server
from pyq3serverlist import PrincipalServer

# query a master server and get a list of game servers
def get_quake3_server_list(master_server, master_protocol=68):
    # configure the master server and connect
    principal = PrincipalServer(master_server[0], master_server[1])
    try:
        # retrieve a list of servers
        return principal.get_servers(master_protocol)
    except:
        # return an empty list if we cannot query the server
        return []

# entry point
MASTER_SERVER = ('master.quake3arena.com', 27950)
# query for a list of game server
servers = get_quake3_server_list(MASTER_SERVER)
# report the number of servers
print(f'Found {len(servers)} Quake 3 Arena servers.')
# get status for one server
for server in servers:
    try:
        status = server.get_status()
        print(status)
        break
    except:
        continue

Running the example first reports the number of servers received from the master server.

Then the status of the first game server to respond is displayed.

Your results will differ as the number of servers live at the time you run the code will change, and the first server to respond will also differ.

We can see that, as expected, the status of the server is a dictionary of field names and values summarizing a range of interesting things about the status of the game currently in progress.

Found 653 Quake 3 Arena servers.
{'ip': '185.107.96.124', 'port': 1041, 'capturelimit': '5', 'g_maxGameClients': '0', 'sv_floodProtect': '1', 'sv_maxPing': '0', 'sv_minPing': '0', 'sv_maxRate': '80000', 'sv_minRate': '0', 'sv_maxclients': '64', 'sv_hostname': 'VOGON NETHERWORLD', 'timelimit': '5', 'fraglimit': '0', 'dmflags': '0', 'version': 'ioq3 1.36 linux-i386 May 30 2010', 'g_gametype': '0', 'protocol': '68', 'mapname': 'q3tourney1', 'sv_privateClients': '0', 'sv_allowDownload': '0', 'bot_minplayers': '0', '.www': 'vogon.top - vogonhq.com', 'gamename': 'VOGON', 'g_needpass': '0', 'players': []}

Three pieces of information that may interest us include:

The server name (e.g. 'sv_hostname').
The current map name (e.g. 'mapname').
The current number of players (e.g. the length of the 'players' list).

We now have enough information to develop a simple Quake 3 game server query program.

Query All Quake 3 Game Servers (one-by-one)

We can now develop a simple Quake 3 game server query program.

The key part of this program will be the way in which we report the status of each game server.

We selected three key pieces of information to display in the previous section: the server name, map name and number of players.

These fields can be retrieved directly from the dictionary response object from the game server; for example:

...
# retrieve useful data to report
name = status['sv_hostname']
mapname = status['mapname']
players = len(status['players'])

The game server names can be pretty wild, such as including non ASCII characters and being very long, perhaps as a type of marketing of servers to game players, a way of standing out.

Nevertheless, we will have to clean up the server names to display them nicely on the command prompt when running the code.

First, let's trim out all characters in the name that are not ASCII. Python provides the string.printable constant that lists all of the standard ASCII characters. We can enumerate all characters in the game server name and only keep those that are in the printable constant.

...
# strip out non-printable characters from the game server name
name = ''.join()

We can then trim any leading or trailing white space from the name, just in case.

...
# strip white space from the server name
name = name.strip()

Finally, we can truncate the length of the name to a fixed length, in this case 40 characters (chosen somewhat arbitrarily).

...
# truncate for 40 characters if longer than 40 characters
if len(name) > 40:
    name = name[:40]

We will design the program to print one summary line per live game server. It would be nice to have the three fields (server name, map name and number of players) line up, which we can achieve using padding when formatting the output string.

For example:

...
line = f'{name:40s}\t{mapname:30s}\t{players:2d} players'

Tying this together, the query_quake3_server() function below takes a server object returned from querying a master server and returns a string summary of the current status of the server, or None if the server cannot be queried successfully.

# query a single game server returning a string summary or None if unreachable
def query_quake3_server(server):
    # query the server i
    try:
        status = server.get_status()
    except:
        return None
    # retrieve useful data to report
    name = status['sv_hostname']
    mapname = status['mapname']
    players = len(status['players'])
    # strip out non-printable characters from the game server name
    name = ''.join()
    # strip white space from the server name
    name = name.strip()
    # truncate for 40 characters if longer than 40 characters
    if len(name) > 40:
        name = name[:40]
    # create a one line summary of the server
    return f'{name:40s}\t{mapname:30s}\t{players:2d} players'

Next, we can call our query_quake3_server() function for each server returned from the master server and print the results.

The query_servers() function listed below implements this, taking the details of the master server as input (a tuple of host name and port number), queries the master server for a list of servers, then prints a summary of the status of each game server.

# get a list of game servers from the master server then query each game server
def query_servers(master_server):
    # get master list
    master_list = get_quake3_server_list(master_server)
    # query each game server in turn
    for server in master_list:
        # query game server
        summary = query_quake3_server(server)
        # skip unreachable servers
        if summary is None:
            continue
        # report
        print(summary)

We now have all of the elements for a simple Quake 3 game server query tool.

The complete example is listed below.

# SuperFastPython.com
# get a list of quake servers and then query the status of each in turn
from string import printable
from pyq3serverlist import PrincipalServer

# query a master server and get a list of game servers
def get_quake3_server_list(master_server, master_protocol=68):
    # configure the master server and connect
    principal = PrincipalServer(master_server[0], master_server[1])
    try:
        # retrieve a list of servers
        return principal.get_servers(master_protocol)
    except:
        # return an empty list if we cannot query the server
        return []

# query a single game server returning a string summary or None if unreachable
def query_quake3_server(server):
    # query the server i
    try:
        status = server.get_status()
    except:
        return None
    # retrieve useful data to report
    name = status['sv_hostname']
    mapname = status['mapname']
    players = len(status['players'])
    # strip out non-printable characters from the game server name
    name = ''.join()
    # strip white space from the server name
    name = name.strip()
    # truncate for 40 characters if longer than 40 characters
    if len(name) > 40:
        name = name[:40]
    # create a one line summary of the server
    return f'{name:40s}\t{mapname:30s}\t{players:2d} players'


# get a list of game servers from the master server then query each game server
def query_servers(master_server):
    # get master list
    master_list = get_quake3_server_list(master_server)
    # query each game server in turn
    for server in master_list:
        # query game server
        summary = query_quake3_server(server)
        # skip unreachable servers
        if summary is None:
            continue
        # report
        print(summary)

# entry point
MASTER_SERVER = ('master.quake3arena.com', 27950)
# report game server status
query_servers(MASTER_SERVER)

Running the example prints the server name, map name, and current number of players on each game server that responds successfully.

Using the padding in the string formatting ensures that all of the fields line up vertically for easy scanning and that the server names are truncated to a sensible length.

VOGON NETHERWORLD                       	q3tourney5                    	 0 players
Guild Of Slackers Arena *West Coast*    	rota3dm3                      	12 players
Argentina<RnR> II                       	ut4_kingdom                   	 0 players
UrbanZone [AU] WWW.DISCORD.ME/URT #2    	ut4_paris                     	 0 players
VOGON HQ-1 ANARCHY                      	q3dm4                         	 1 players
VOGON INSTAKILL                         	pro-q3tourney2                	 2 players
! Freon ! :: Duel :: !                  	q3tourney3                    	 2 players
[LCDLC] Unofficial Maps                 	ut4_train_dl1                 	 0 players
Martyr's Quake Server                   	q3dm17                        	 0 players
PUBZAO :: CTF                           	ut4_terrorism3                	 8 players
...

Although the program works, it is very slow.

The 650+ game servers took approximately 13 minutes to query when I ran the example.

This is far too long to wait, especially for an impatient gamer looking for a game to join.

We can dramatically speed this up using threads in Python with the ThreadPoolExecutor.

Create a Pool of Worker Threads

We can use the ThreadPoolExecutor to speed up the querying of Quake III Arena servers.

The ThreadPoolExecutor class is provided as part of the concurrent.futures module for easily running concurrent tasks.

The ThreadPoolExecutor provides a pool of worker threads, which is different from the ProcessPoolExecutor that provides a pool of worker processes.

Generally, ThreadPoolExecutor should be used for concurrent IO-bound tasks, like downloading URLs, and the ProcessPoolExecutor should be used for concurrent CPU-bound tasks, like calculating.

Using the ThreadPoolExecutor was designed to be easy and straightforward. It is like the "automatic mode" for Python threads.

Create the thread pool by calling ThreadPoolExecutor().
Submit tasks and get futures by calling submit() or map().
Wait and get results as tasks complete by calling as_completed().
Shut down the thread pool by calling shutdown()

Create the Thread Pool

First, a ThreadPoolExecutor instance must be created. By default, it will create a pool with the number of logical CPUs in your system plus four. This is good for most purposes.

...
# create a thread pool with the default number of worker threads
pool = ThreadPoolExecutor()

You can run tens to hundreds of concurrent IO-bound threads per CPU, although perhaps not thousands or tens of thousands. You can specify the number of threads to create in the pool via the max_workers argument; for example:

...
# create a thread pool with 10 worker threads
pool = ThreadPoolExecutor(max_workers=10)

Submit Tasks to the Thread Pool

Once created, it can send tasks into the pool to be completed using the submit() function.

This function takes the name of the function to call any and all arguments and returns a Future object.

The Future object is a promise to return the results from the task (if any) and provides a way to determine if a specific task has been completed or not.

...
# submit tasks
future = pool.submit(my_task, arg1, arg2, ...)

The return from a function executed by the thread pool can be accessed via the result() function on the Future object. It will wait until the result is available, if needed, or return immediately if the result is available.

For example:

...
# get the result from a future
result = future.result()

Get Results as Tasks Complete

The beauty of performing tasks concurrently is that we can get results as they become available, rather than waiting for tasks to be completed in the order they were submitted.

The concurrent.futures module provides an as_completed() function that we can use to get results for tasks as they are completed, just like its name suggests.

We can call the function and provide it a list of Future objects created by calling submit() and it will return future objects as they are completed in whatever order.

For example, we can use a list comprehension to submit the tasks and create the list of Future objects:

...
# submit all tasks into the thread pool and create a list of futures
futures = [pool.submit(my_func, task) for task in tasks]

Then get results for tasks as they complete in a for loop:

...
# iterate over all submitted tasks and get results as they are available
for future in as_completed(futures):
	# get the result
	result = future.result()
	# do something with the result...

Shutdown the Thread Pool

Once all tasks are completed, we can close down the thread pool, which will release each thread and any resources it may hold (e.g. the stack space).

...
# shutdown the thread pool
pool.shutdown()

An easier way to use the thread pool is via the context manager (the with keyword), which ensures it is closed automatically once we are finished with it.

...
# create a thread pool
with ThreadPoolExecutor(max_workers=10) as pool:
	# submit tasks
	futures = [pool.submit(my_func, task) for task in tasks]
	# get results as they are available
	for future in as_completed(futures):
		# get the result
		result = future.result()
		# do something with the result...

Now that we are familiar with ThreadPoolExecutor and how to use it, let's look at how we can adapt our program for querying Quake 3 servers to make use of it.

Query Quake 3 Servers Concurrently

Our Quake 3 game server query tool can be adapted to use ThreadPoolExecutor directly.

This is because querying each game server is an independent task; it does not have a dependency on querying other game servers. This means we can perform the game server query operations in separate Python threads.

Specifically, we can call query_quake3_server() in a separate thread for each server object returned from the master server.

The slowest part of this function will be the call to server.get_status() and performing this operation in separate threads will give us a large speed up. A further improvement would be to separate the code so that only this call was executed in a separate thread and the code for distilling the status dictionary into a string for printing could be performed in the main thread.

We can create a thread pool and call the submit() function with the query_quake3_server() function and each server object in the master list; for example:

...
# create thread pool
with ThreadPoolExecutor() as pool:
    # send all tasks into the thread pool
    futures = [pool.submit(query_quake3_server, serv) for serv in master_list]

How many threads should we use in the pool?

Each server has a separate IP address and port; therefore, we don't have to worry about spamming (denial of service) a single server. With modern hardware and a fast internet connection, we can easily query hundreds of game servers concurrently, e.g. hundreds of parallel threads in Python.

After some testing, I found that 500 threads works well for this example so that is what we will use. You may want to experiment and see what works well for your specific workstation.

We now have a list of Future objects that may finish at different times.

It would be nice to report the status of game servers as they become available, rather than wait for all queries to be completed.

This can be achieved using the as_completed() function, passing in the list of Future objects. It will return each Future object as it is completed.

Calling the result() function in the future will either give a string summary of the server or None if the query was unsuccessful. We can then print the line summary as we did in the non-threaded example.

...
# report results as they are made available
for future in as_completed(futures):
    # get the summary line from the task
    summary = future.result()
    # skip servers that cannot be contacted
    if summary is None:
        continue
    # report
    print(summary)

Tying this together, the updated version of the query_servers() function to use the thread pool for querying game servers concurrently is listed below.

# get a list of game servers from the master server then query each game server
def query_servers(master_server):
    # get master list
    master_list = get_quake3_server_list(master_server)
    # create thread pool
    with ThreadPoolExecutor(500) as pool:
        # send all tasks into the thread pool
        futures = [pool.submit(query_quake3_server, serv) for serv in master_list]
        # report results as they are made available
        for future in as_completed(futures):
            # get the summary line from the task
            summary = future.result()
            # skip servers that cannot be contacted
            if summary is None:
                continue
            # report
            print(summary)

We can now test out the concurrent version of our game server query tool.

If each game server takes about one second to respond or fail and we have about 650 servers, then we might have results within about two seconds, compared to 13 minutes in the non-threaded version.

The complete example is listed below.

# SuperFastPython.com
# get a list of quake servers and then query the status of each concurrently
from string import printable
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
from pyq3serverlist import PrincipalServer

# query a master server and get a list of game servers
def get_quake3_server_list(master_server, master_protocol=68):
    # configure the master server and connect
    principal = PrincipalServer(master_server[0], master_server[1])
    try:
        # retrieve a list of servers
        return principal.get_servers(master_protocol)
    except:
        # return an empty list if we cannot query the server
        return []

# query a single game server returning a string summary or None if unreachable
def query_quake3_server(server):
    # query the server i
    try:
        status = server.get_status()
    except:
        return None
    # retrieve useful data to report
    name = status['sv_hostname']
    mapname = status['mapname']
    players = len(status['players'])
    # strip out non-printable characters from the game server name
    name = ''.join()
    # strip white space from the server name
    name = name.strip()
    # truncate for 40 characters if longer than 40 characters
    if len(name) > 40:
        name = name[:40]
    # create a one line summary of the server
    return f'{name:40s}\t{mapname:30s}\t{players:2d} players'

# get a list of game servers from the master server then query each game server
def query_servers(master_server):
    # get master list
    master_list = get_quake3_server_list(master_server)
    # create thread pool
    with ThreadPoolExecutor(500) as pool:
        # send all tasks into the thread pool
        futures = [pool.submit(query_quake3_server, serv) for serv in master_list]
        # report results as they are made available
        for future in as_completed(futures):
            # get the summary line from the task
            summary = future.result()
            # skip servers that cannot be contacted
            if summary is None:
                continue
            # report
            print(summary)

# entry point
MASTER_SERVER = ('master.quake3arena.com', 27950)
# report game server status
query_servers(MASTER_SERVER)

Running the example queries all game servers returned from the master server concurrently, with up to 500 concurrent connections.

Below is a truncated version of the results.

UrbanZone [AU] WWW.DISCORD.ME/URT #2    	ut4_paris                     	 0 players
[AU] oafps.com CPMA #2                  	cpm24                         	 0 players
NG's house of terror                    	ut4_casa                      	 0 players
UrbanZone [AU] WWW.DISCORD.ME/URT       	ut4_casa                      	 0 players
Profanum Perth #2                       	cpm24                         	 0 players
EternalHavoc CTF                        	ut4_docks                     	 0 players
BlazingPhoenix.org Q3A                  	q3dm12                        	 7 players
[AU] oafps.com CPMA #3                  	cpm24                         	 0 players
noname                                  	q3dm6                         	 0 players
AntiLag Beta 2 server                   	q3dm17                        	 4 players
...

It is dramatically faster.

In my case, it took about 2.9 seconds to complete as opposed to about 13 minutes, or about 260% faster!

Now that the program is responsive, we could start to think about other concerns for the user, such as filtering out game servers that have no players or are full and cannot accept new players.

We are using one master server to get a list of game servers to query, but what if we wanted to get lists of servers from multiple master servers?

Query Quake 3 Master Servers Concurrently

We can update the example to first query a number of master servers to create a large list of game servers, then query this list of game servers concurrently as we did previously.

First, we need a list of Quake 3 master servers.

A Google search reveals a site dedicated to listing Quake 3 master servers:

Quake 3 Arena Master Servers

This gives a list of 8 servers we can use.

Below are these servers in a list of tuples.

...
# entry point
MASTER_HOSTS = [
    ('master.quake3arena.com', 27950),
    ('master2.urbanterror.info', 27900),
    ('master0.excessiveplus.net', 27950),
    ('master.ioquake3.org', 27950),
    ('master.huxxer.de', 27950),
    ('master.urbanterror.info', 27900),
    ('master.maverickservers.com', 27950),
    ('dpmaster.deathmask.net', 27950),
    ('master3.idsoftware.com', 27950)]

Next, we need to query each master server in turn.

It might take one second or more to query each master server, and with eight servers doing this sequentially, it would take up to eight seconds.

...
# query each master server for a list of game servers
all_servers_lists = [get_quake3_server_list(master) for master in master_hosts]

Instead, we can query the master servers concurrently, as we did with the game servers themselves in the previous section.

In this case, we will use the map() function to apply the get_quake3_server_list() function that we developed previously to each master server tuple (host name and port number).

This will give us an iterable of results once all queries have been completed.

...
# get lists of servers from each master server
with ThreadPoolExecutor(len(master_hosts)) as pool:
    all_servers_lists = pool.map(get_quake3_server_list, master_hosts)

Notice that we set the number of threads to be the number of master servers, in this case eight.

This will give an iterable over lists of game servers that we can flatten to a single list of game servers using a list comprehension.

...
# flatten list of lists to single list
all_servers =

We would expect some overlap in the game servers reported by each master server.

As such, we would only be interested in a list of unique servers, such as unique server IP address and port number combinations.

We can achieve this by storing all server objects in a dictionary using a string of IP addresses and port numbers as the key. If a server already exists in the dictionary, we can simply replace it.

...
# filter list to a unique set of hosts and ports
master_list = {}
for server in all_servers:
    # create a unique key
    key = f'{server.ip}-{server.port}'
    # store server
    master_list = server

Given that we're only interested in the server objects themselves, we can then retrieve the values from the dictionary.

...
master_list = master_list.values()

This list of game servers can then be submitted to another thread pool to be queried, as we did before.

We can tie all of this together into a function named get_all_quake3_game_servers() that will take a list of master servers and return a list of unique Quake 3 game servers.

It also reports progress along the way, including the number of game servers total and the size of the distilled list of unique game servers.

# query a list of master servers and return a list of unique game servers
def get_all_quake3_game_servers(master_hosts):
    # get lists of servers from each master server
    with ThreadPoolExecutor(len(master_hosts)) as pool:
        all_servers_lists = pool.map(get_quake3_server_list, master_hosts)
    # flatten list of lists to single list
    all_servers = 
    print(f'Found {len(all_servers)} servers from {len(master_hosts)} master servers.')
    # filter list to a unique set of hosts and ports
    master_list = {}
    for server in all_servers:
        # create a unique key
        key = f'{server.ip}-{server.port}'
        # store server
        master_list = server
    master_list = master_list.values()
    print(f'Reduced server list to {len(master_list)} unique IP:PORT.')
    return master_list

An alternative approach would be to use the submit() function and use the as_complete() function on the list of futures returned, then add the game servers to the dictionary one at a time.

This would be functionally equivalent and perhaps slightly faster if some master servers take much longer to respond than others.

We can then update our query_servers() function to call our get_all_quake3_game_servers() function to get the list of game servers, then query each game server using a thread pool as before.

# get a list of game servers from the master server then query each game server
def query_servers(master_hosts):
    # get a list of game servers
    master_list = get_all_quake3_game_servers(master_hosts)
    # create thread pool
    with ThreadPoolExecutor(500) as pool:
        # send all tasks into the thread pool
        futures = [pool.submit(query_quake3_server, serv) for serv in master_list]
        # report results as they are made available
        for future in as_completed(futures):
            # get the summary line from the task
            summary = future.result()
            # skip servers that cannot be contacted
            if summary is None:
                continue
            # report
            print(summary)

An alternate design might be to create a pipeline of thread pools connected via Queue objects. We could then feed master servers in at one end (the first thread pool), which would pass servers on to the second pool, and finally give us individual game server summary strings as the output.

This might be a sticky design as careful attention would be needed to ensure only unique game servers are queried in the second thread pool. Nevertheless, it would allow for expansion of functionality, such as another step in the pipeline for filtering the results prior to formatting.

Nevertheless, tying our code together, the complete example of querying multiple master servers using a thread pool, then querying the list of game servers using a second thread pool is listed below.

# SuperFastPython.com
# get a lists of quake servers from master servers and then query the status of each
from string import printable
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
from pyq3serverlist import PrincipalServer

# query a master server and get a list of game servers
def get_quake3_server_list(master_server, master_protocol=68):
    # configure the master server and connect
    principal = PrincipalServer(master_server[0], master_server[1])
    try:
        # retrieve a list of servers
        return principal.get_servers(master_protocol)
    except:
        # return an empty list if we cannot query the server
        return []

# query a single game server returning a string summary or None if unreachable
def query_quake3_server(server):
    # query the server i
    try:
        status = server.get_status()
    except:
        return None
    # retrieve useful data to report
    name = status['sv_hostname']
    mapname = status['mapname']
    players = len(status['players'])
    # strip out non-printable characters from the game server name
    name = ''.join()
    # strip white space from the server name
    name = name.strip()
    # truncate for 40 characters if longer than 40 characters
    if len(name) > 40:
        name = name[:40]
    # create a one line summary of the server
    return f'{name:40s}\t{mapname:30s}\t{players:2d} players'

# query a list of master servers and return a list of unique game servers
def get_all_quake3_game_servers(master_hosts):
    # get lists of servers from each master server
    with ThreadPoolExecutor(len(master_hosts)) as pool:
        all_servers_lists = pool.map(get_quake3_server_list, master_hosts)
    # flatten list of lists to single list
    all_servers = 
    print(f'Found {len(all_servers)} servers from {len(master_hosts)} master servers.')
    # filter list to a unique set of hosts and ports
    master_list = {}
    for server in all_servers:
        # create a unique key
        key = f'{server.ip}-{server.port}'
        # store server
        master_list = server
    master_list = master_list.values()
    print(f'Reduced server list to {len(master_list)} unique IP:PORT.')
    return master_list

# get a list of game servers from the master server then query each game server
def query_servers(master_hosts):
    # get a list of game servers
    master_list = get_all_quake3_game_servers(master_hosts)
    # create thread pool
    with ThreadPoolExecutor(500) as pool:
        # send all tasks into the thread pool
        futures = [pool.submit(query_quake3_server, serv) for serv in master_list]
        # report results as they are made available
        for future in as_completed(futures):
            # get the summary line from the task
            summary = future.result()
            # skip servers that cannot be contacted
            if summary is None:
                continue
            # report
            print(summary)

# entry point
MASTER_HOSTS = [
    ('master.quake3arena.com', 27950),
    ('master2.urbanterror.info', 27900),
    ('master0.excessiveplus.net', 27950),
    ('master.ioquake3.org', 27950),
    ('master.huxxer.de', 27950),
    ('master.urbanterror.info', 27900),
    ('master.maverickservers.com', 27950),
    ('dpmaster.deathmask.net', 27950),
    ('master3.idsoftware.com', 27950)]

# report game server status
query_servers(MASTER_HOSTS)

Running the example first queries all master game servers and creates a master list of unique game servers, then queries all game servers and reports their status.

In this case, we can see that a total of about 2,200 game servers were discovered from the eight master servers compared to the 650 discovered when using a single master server. This list was then distilled to 884 unique game servers, still more than the single game server used in previous examples.

The cost of querying multiple game servers did add about two seconds to the execution time.

Found 2181 servers from 9 master servers.
Reduced server list to 884 unique IP:PORT.
UrbanZone [AU] WWW.DISCORD.ME/URT #2    	ut4_paris                     	 0 players
[AU] oafps.com CPMA #2                  	cpm24                         	 0 players
NG's house of terror                    	ut4_casa                      	 0 players
UrbanZone [AU] WWW.DISCORD.ME/URT       	ut4_casa                      	 0 players
Profanum Perth #2                       	cpm24                         	 0 players
EternalHavoc CTF                        	ut4_docks                     	 0 players
noname                                  	q3dm6                         	 0 players
[AU] oafps.com CPMA #3                  	cpm24                         	 0 players
...

Extensions

This section lists ideas for extending the tutorial.

Separate Server Query From Formatting: Update the example so that only the API call to query the game server is performed in a separate thread and that the formatting of the server status is performed in the main thread.
Optimize Query Threads: Update the example to time different numbers of query threads in order to discover what works best for your specific workstation hardware and internet connection.
Filter Server Results: Update the example to filter out game servers that have no players or are full and cannot accept new players.
Use a Pipeline: Update the final example to use a pipeline of thread pools connected by Queue objects and add an additional step in the pipeline to filter servers before formatting their result.
Create a graphical interface: Update the example to use a graphical user interface to present the status of all queried game servers.

Share your extensions in the comments below; it would be great to see what you come up with.

Takeaways

In this tutorial, you discovered how to query Quake 3 game servers faster using ThreadPoolExecutor in Python.

How to query the Quake 3 master server and individual servers for their current status.
How to use the ThreadPoolExecutor to create and manage thread pools in Python.
How to speed up the querying of Quake 3 game servers nearly 300% using the ThreadPoolExecutor.

If you enjoyed this tutorial, you will love my book: Python ThreadPoolExecutor Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.