Benchmark Python with time.thread_time()

September 26, 2023 Python Benchmarking

You can benchmark Python code using the time.thread_time() function.

In this tutorial, you will discover how to benchmark Python code using the time.thread_time() function.

Let's get started.

Need to Benchmark Python Code

Benchmarking Python code refers to comparing the performance of one program to variations of the program.

Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.
-- Benchmarking, Wikipedia.

Typically, we make changes to the programs, such as adding concurrency, in order to improve the performance of the program on a given system.

Improving performance typically means reducing the run time of the program.

Therefore, when we benchmark programs in Python after adding concurrency, we typically are interested in recording how long a program takes to run.

It is critical to be systematic when benchmarking code.

The first step is to record how long an unmodified version of the program takes to run. This provides a baseline in performance to which all other versions of the program must be compared. If we are adding concurrency, then the unmodified version of the program will typically perform tasks sequentially, e.g. one-by-one.

We can then make modifications to the program, such as adding thread pools, process pools, or asyncio. The goal is to perform tasks concurrently (out of order), even in parallel (simultaneously). The performance of the program can be benchmarked and compared to the performance of the unmodified version.

The performance of the modified versions of the program must have better performance than the unmodified version of the program. If they do not, they are not improvements and should not be adopted.

How can we benchmark the performance of programs in Python?

How to Benchmark with time.thread_time()

We can benchmark Python code using the time.thread_time() function.

This is a function that is provided in the time module and is part of the Python standard library.

It can be used to record the time before a piece of code that is being benchmarked, then again to record the time again after the benchmarked code. The difference between the two times can be reported as the relative execution time, e.g. compared to another benchmark.

Let's take a closer look.

What is time.thread_time()

The time.thread_time() reports the time that the current thread has been executing.

The time begins or is zero when the current thread is first created.

Return the value (in fractional seconds) of the sum of the system and user CPU time of the current thread.
-- time — Time access and conversions

It is an equivalent value as the time.process_time(), except calculate a the scope of the current thread, not the current process.

This value is calculated as the sum of the system time and the user time.

thread time = user time + system time

The reported time does not include sleep time.

This means if the thread is blocked by a call to time.sleep() or perhaps is suspended by the operating system, then this time is not included in the reported time. This is called a "thread-wide" or "thread-specific" time.

It does not include time elapsed during sleep.
-- time — Time access and conversions

The time.thread_time() function was added in Python version 3.7.

The new time.thread_time() and time.thread_time_ns() functions can be used to get per-thread CPU time measurements.
-- What’s New In Python 3.7

Why Not Benchmark With time.time()?

It is common to benchmark code with time.time() instead of the time.thread_time().

Nevertheless, the time.time() function has some limitations, such as:

Lack of Precision: time.time() provides time measurements in seconds and fractions of a second, which might not be sufficient for accurately profiling very short code segments or fine-grained optimizations.
Clock Drift and System Load: System clock drift or variations in system load can introduce inaccuracies in the measurements, leading to inconsistent results.
Overhead: The overhead introduced by the time.time() calls themselves can be significant for small and fast operations, potentially skewing the results.
System Clock Adjustments: The clock used by time.time() may be updated, such as when adjusted for leap seconds or synchronizing with a time sever.

The biggest limitation of time.time() is that it is possible for a subsequent time to be before a previous time. The time.time() function is not monotonic. This can be because of many factors such as rounding, system load, time synchronization, and lack of precision.

As such, time.time() is not appropriate for fine-grained benchmarking, such as benchmarking code that has a very short duration, e.g. milliseconds or less.

You can learn more about benchmarking with the time.time() function in the tutorial:

Benchmark Python with time.time()

Now that we know about the time.thread_time() function, let's look at how we can use it to benchmark our Python programs.

How to Benchmark with time.thread_time()

We can use the time.thread_time() function to benchmark Python code.

There are perhaps 3 case studies we may want to consider, they are:

Benchmarking a Python statement.
Benchmarking a Python function.
Benchmarking a Python script (program).

Let's look at how we can benchmark time.thread_time() function.

How to Benchmark a Python Statement

We can use the time.thread_time() function to benchmark arbitrary Python statements.

The procedure is as follows:

Record time.thread_time() before the statement.
Execute the statement.
Record time.thread_time() after the statement.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

# record start time
time_start = thread_time()
# execute the statement
...
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

How to Benchmark a Python Function

We can use the time.thread_time() function to benchmark arbitrary Python functions.

The procedure is as follows:

Record time.thread_time() before the function.
Call the function.
Record time.thread_time() after the function.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

# record start time
time_start = thread_time()
# call the function
...
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

How to Benchmark a Python Script

We can use the time.thread_time() function to benchmark arbitrary Python scripts (Python files).

It requires that the entry point into the script is first moved into a new function, that we will call main(). This is to make it easy for all code in the script to be wrapped in the benchmarking code.

The procedure is as follows:

Move the entry of the script into a main() function (if needed).
Record time.thread_time() before the main() function.
Call the main() function.
Record time.thread_time() after the main() function.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

# protect the entry point
if __name__ == '__main__':
    # record start time
    time_start = thread_time()
    # execute the script
    main()
    # record end time
    time_end = thread_time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

Now that we know how to benchmark using the time.thread_time() function, let's look at some worked examples.

Example of Benchmarking a Statement with time.thread_time()

We can explore how to benchmark a Python statement using time.thread_time() with a worked example.

In this example, we will define a statement that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.

...
# execute the statement
data = [i*i for i in range(100000000)]

We will then surround this statement with benchmarking code.

Firstly, we will record the start time using the time.thread_time() function.

...
# record start time
time_start = thread_time()

Afterward, we will record the end time, calculate the overall execution duration, and report the result.

...
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a statement with time.thread_time()
from time import thread_time
# record start time
time_start = thread_time()
# execute the statement
data = [i*i for i in range(100000000)]
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Running the example first records the start time, a number from an internal clock for the thread.

Next, the Python statement is executed, in this case creating a list of 100 million squared integers.

The end time is then recorded, as a number from an internal clock for the thread.

The difference between the two recorded times is calculated, providing the statement execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the statement took about 5.070 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python statement using the time.thread_time() function.

Took 5.118 seconds

Next, let's explore an example of benchmarking a function using the time.thread_time() function.

Example of Benchmarking a Function with time.thread_time()

We can explore how to benchmark a Python function using time.thread_time() with a worked example.

In this example, we will define a function that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

We will then call this function, and surround the function call with benchmarking code.

Firstly, we will record the start time using the time.thread_time() function.

...
# record start time
time_start = thread_time()

Afterward, we will record the end time, calculate the overall execution duration, and report the result.

...
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a function with time.thread_time()
from time import thread_time

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

# record start time
time_start = thread_time()
# execute the function
task()
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Running the example first records the start time, a number from an internal clock for the thread.

Next, the Python function is called, in this case creating a list of 100 million squared integers.

The end time is then recorded, as a number from an internal clock for the thread.

The difference between the two recorded times is calculated, providing the function execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the function took about 6.431 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python function using the time.thread_time() function.

Took 6.194 seconds

Next, let's explore an example of benchmarking a script using the time.thread_time() function.

Example of Benchmarking a Script with time.thread_time()

We can explore how to benchmark a Python script (Python file) using time.thread_time() with a worked example.

In this example, we will update the above example so that it has a main() function and protects the entry point, like a more elaborate Python script.

# main function for script
def main():
    # call a function
    task()

We will then add benchmarking code around the call to the main() function.

# protect the entry point
if __name__ == '__main__':
    # record start time
    time_start = thread_time()
    # execute the script
    main()
    # record end time
    time_end = thread_time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a script with time.thread_time()
from time import thread_time

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

# main function for script
def main():
    # call a function
    task()

# protect the entry point
if __name__ == '__main__':
    # record start time
    time_start = thread_time()
    # execute the script
    main()
    # record end time
    time_end = thread_time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

Running the example first records the start time, a number from an internal clock for the thread.

Next, the main() function is called which executes the core of the Python script. In this case, it calls our task() function and creates a list of 100 million squared integers.

The end time is then recorded, as a number from an internal clock for the thread.

The difference between the two recorded times is calculated, providing the function execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the function took about 6.438 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python script using the time.thread_time() function.

Took 6.199 seconds

Confirm time.thread_time() Does Not Include Sleep

The time.thread_time() function does not include time spent blocked or sleeping.

When the thread is blocked or sleeping, the clock used by time.thread_time() is paused. When the program is resumed, the clock used by time.thread_time() is then also resumed.

This means any benchmarking performed using the time.thread_time() function will exclude time spent sleeping or blocked.

We can demonstrate this with a worked example.

We can update the example of benchmarking a statement and include a sleep for 2 seconds.

For example:

...
# sleep for a moment
sleep(2)

This will have an effect on the benchmark time, e.g. it will not increase the benchmark time from about 5 seconds to about 7 seconds as we might naively expect

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a statement and sleep with time.thread_time()
from time import thread_time
from time import sleep
# record start time
time_start = thread_time()
# execute the statement
data = [i*i for i in range(100000000)]
# sleep for a moment
sleep(2)
# record end time
time_end = thread_time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Running the example, we can see that the addition of the sleep() after the target code does have the intended effect.

The time of the overall benchmark duration does not increase by the added sleep time, e.g. from 5 to 7 seconds.

This highlights that time spent explicitly sleeping is excluded in the benchmark time when using time.thread_time().

Took 5.111 seconds

Confirm time.thread_time() Is Monotonic and Not Adjustable

The clock used by the time.thread_time() function is not adjustable.

We can confirm this by reporting the details of the "thread_time" function in the time module via the time.get_clock_info() function.

This reports the details of the clock used by a function, like thread_time, such as whether it is adjustable, how it is implemented on the platform, whether it is thread_time, and the resolution on the platform.

The program below reports the details of the "thread_time" clock used by the time.thread_time() function.

# SuperFastPython.com
# details of the clock used by time.thread_time()
from time import get_clock_info
# get details
details = get_clock_info('thread_time')
# report details
print(details)

Running the program reports the details of the "thread_time" clock.

We can see that indeed it is monotonic and that the resolution on the platform is 1e-09.

We can also confirm that it is not adjustable (at least on the platform on which this program was run).

namespace(implementation='clock_gettime(CLOCK_THREAD_CPUTIME_ID)', monotonic=True, adjustable=False, resolution=1e-09)

Takeaways

You now know how to benchmark Python code using the time.thread_time() function.