Benchmark Python with time.time()

September 17, 2023 Python Benchmarking

You can benchmark Python code using the time.time() function.

In this tutorial, you will discover how to benchmark Python code using the time.time() function.

Let's get started.

Need to Benchmark Python Code

Benchmarking Python code refers to comparing the performance of one program to variations of the program.

Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.
-- Benchmarking, Wikipedia.

Typically, we make changes to the programs, such as adding concurrency, in order to improve the performance of a program on a given system.

Improving performance typically means reducing the run time of the program.

Therefore, when we benchmark programs in Python after adding concurrency, we typically are interested in recording how long a program takes to run.

It is critical to be systematic when benchmarking code.

The first step is to record how long an unmodified version of the program takes to run. This provides a baseline in performance to which all other versions of the program must be compared. If we are adding concurrency, then the unmodified version of the program will typically perform tasks sequentially, e.g. one-by-one.

We can then make modifications to the program, such as adding thread pools, process pools, or asyncio. The goal is to perform tasks concurrently (out of order), even in parallel (simultaneously). The performance of the program can be benchmarked and compared to the performance of the unmodified version.

The performance of the modified versions of the program must have better performance than the unmodified version of the program. If they do not, they are not improvements and should not be adopted.

How can we benchmark the performance of programs in Python?

Benchmark with time.time()

We can benchmark Python code using the time.time() function.

This is a function that is provided in the time module and is part of the Python standard module.

It can be used to record the time before a piece of code that is being benchmarked, then again to record the time again after the benchmarked code. The difference between the two times can be reported as the wall clock execution time.

Let's take a closer look.

What is time.time()

The time.time() function reports the number of seconds since the epoch.

Return the time in seconds since the epoch as a floating point number.
-- time — Time access and conversions

Recall that epoch is January 1st 1970, which is used on Unix systems and beyond as an arbitrary fixed time in the past.

In computing, an epoch is a fixed date and time used as a reference from which a computer measures system time. Most computer systems determine time as a number representing the seconds removed from a particular arbitrary date and time. For instance, Unix and POSIX measure time as the number of seconds that have passed since Thursday 1 January 1970 00:00:00 UT, a point in time known as the Unix epoch.
-- Epoch (computing), Wikipedia.

The result is a floating point value, potentially offering fractions of a second (e.g. milliseconds), if the platforms support it.

The time.time() function is not perfect.

It is (theoretically) possible for a subsequent call to time.time() to return a value in seconds less than the previous value, due to rounding.

Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.
-- time — Time access and conversions

This may make the time.time() method of benchmarking code appropriate for code that has a generally longer execution time (e.g. seconds) rather than short execution times, e.g. less than a second or less than 500 milliseconds.

An alternative to time.time() that may be more appropriate for fine-grained micro benchmarking is time.perf_counter().

You can learn more about how to benchmark with the time.perf_counter() function in the tutorial:

Benchmark Python with time.perf_counter()

Why Benchmark With time.time()

Using time.time() to benchmark Python code offers several advantages:

Simple and Lightweight: time.time() is a straightforward method to measure execution time, making it easy to implement and understand without introducing significant overhead.
Fine-Grained Timing: It provides precise time measurements down to the second and fractions of a second, allowing you to capture even small performance differences.
Platform Independence: time.time() is platform-independent, making it suitable for benchmarking code across different operating systems and environments.
Customizable Measurements: By placing time.time() calls strategically around specific code sections, you can focus on measuring only the parts of the code that need optimization.
Rapid Profiling: time.time() is a quick and immediate way to perform ad hoc profiling and gain insights into the time taken by specific code sections.

Limitations of Benchmarking with time.time()

The time.time() function has limitations when used for benchmarking, such as:

Lack of Precision: time.time() provides time measurements in seconds and fractions of a second, which might not be sufficient for accurately profiling very short code segments or fine-grained optimizations.
Clock Drift and System Load: System clock drift or variations in system load can introduce inaccuracies in the measurements, leading to inconsistent results.
Overhead: The overhead introduced by the time.time() calls themselves can be significant for small and fast operations, potentially skewing the results.
System Clock Adjustments: The clock used by time.time() may be updated, such as when adjusted for leap seconds or synchronizing with a time sever.

The biggest limitation of time.time() is that it is possible for a subsequent time to be before a previous time. The time.time() function is not monotonic. This can be because of many factors such as rounding, system load, time synchronization, and lack of precision.

As such, time.time() is not appropriate for benchmarking.

You can learn more about this in the tutorial:

Python time.time() vs time.perf_counter()

Now that we know what the time.time() function is, let's look at how we can use it to benchmark Python code.

How to Benchmark with time.time()

We can use the time.time() function to benchmark Python code.

There are perhaps 3 case studies we may want to consider, they are:

Benchmarking a Python statement.
Benchmarking a Python function.
Benchmarking a Python script (program).

Let's look at how we can benchmark time.time() function.

How to Benchmark a Python Statement

We can use the time.time() function to benchmark arbitrary Python statements.

The procedure is as follows:

Record time.time() before the statement.
Execute the statement.
Record time.time() after the statement.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

# record start time
time_start = time()
# execute the statement
...
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

How to Benchmark a Python Function

We can use the time.time() function to benchmark arbitrary Python functions.

The procedure is as follows:

Record time.time() before the function.
Call the function.
Record time.time() after the function.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

# record start time
time_start = time()
# call the function
...
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

How to Benchmark a Python Script

We can use the time.time() function to benchmark arbitrary Python scripts (Python files).

It requires that the entry point into the script is first moved into a new function, that we will call main(). This is to make it easy for all code in the script to be wrapped in the benchmarking code.

The procedure is as follows:

Move the entry of the script into a main() function (if needed).
Record time.time() before the main() function.
Call the main() function.
Record time.time() after the main() function.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

# protect the entry point
if __name__ == '__main__':
    # record start time
    time_start = time()
    # execute the script
    main()
    # record end time
    time_end = time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

Advanced Benchmarking with time.time()

The time.time() approach to benchmarking Python code can be used in some clever ways, allowing the benchmark process to be hidden or automatic.

Some examples include:

Develop a benchmark helper function.
Develop a benchmark context manager.
Develop a benchmark function decorator.

Let's take a closer look at each in turn.

Benchmark Helper Function

We can develop a helper function to automatically benchmark our Python code.

Our function can take the name of our target function that we wish to benchmark and optional *args arguments.

The *args is an optional list of function arguments. It allows us to specify zero, one, or many arguments for our target function to the benchmark function, which we can pass on to the target function directly.

Our function can then record the start time, call the target function, record the end time, and report the overall duration.

For example:

# benchmark function
def benchmark(fun, *args):
    # record start time
    time_start = time()
    # call the custom function
    fun(*args)
    # record end time
    time_end = time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

We can then call the benchmark() function with the name of the target function to be benchmarked, along with any arguments.

For example:

...
# benchmark a function
benchmark(my_function, arg1, arg2)

You can learn more about developing a helper benchmark function in the tutorial:

Benchmark Helper Function in Python

Benchmark Context Manager

We can hide manual benchmarking of Python code in a context manager.

Recall that a context manager is a Python object that has __enter__() and __exit__() methods and is used via the with expression.

We can define a new class that implements a constructor __init__() the __enter__() and __exit__() methods.

The __init__() constructor can take a name argument for the benchmark case and store it in an object attribute.

The __enter__() method can initialize the start time and store it in object attributes. It can then return an instance of the context manager itself, as a good practice.

The __exit__() method must take some standard arguments about any exception that occurred while running the context code. It can then record the end time, calculate and store the duration and report the calculated duration along with the name of the benchmark case.

For example:

# define the benchmark context manager
class Benchmark(object):
    # constructor
    def __init__(self, name):
        # store the name of this benchmark
        self.name = name

    # enter the context manager
    def __enter__(self):
        # record the start time
        self.time_start = time()
        # return this object
        return self

    # exit the context manager
    def __exit__(self, exc_type, exc_value, traceback):
        # record the end time
        self.time_end = time()
        # calculate the duration
        self.duration = self.time_end - self.time_start
        # report the duration
        print(f'{self.name} took {self.duration:.3f} seconds')
        # do not suppress any exception
        return False

We can then use it by creating an instance of the Benchmark class within the "with" expression and then list any code within the context we wish to benchmark.

For example:

...
# create the benchmark context
with Benchmark('Task'):
    # run the task
    my_function(arg1, arg2, arg3)

You can learn more about how to develop a benchmark context manager in the tutorial:

Benchmark Context Manager in Python

Benchmark Function Decorator

A function decorator in Python allows a custom function to be called automatically that will in turn call our target function.

This can be used to insert code before and after calling our target function, such as recording the start time, and end time, and calculating an overall execution duration.

A function decorator can be defined as a custom function that returns a function that in turn calls our target function.

We can develop a function decorator to benchmark a Python function automatically.

The decorator will be called benchmark_decorator() and takes the function to be decorated.

The inner function must take arguments for the target function, just in case. It then must record the start time before calling the function, and the end time after calling the function. It then calculates the duration and reports it before returning any return value from the target function itself.

For example:

# define the benchmark decorator
def benchmark_decorator(func):
    # inner function that wraps the target function
    @wraps(func)
    def wrapper(*args, **kwargs):
        # record start time
        time_start = time()
        # call the custom function
        result = func(*args, **kwargs)
        # record end time
        time_end = time()
        # calculate the duration
        time_duration = time_end - time_start
        # report the duration
        print(f'Took {time_duration:.3f} seconds')
        # pass on the return value
        return result
    # return the inner function
    return wrapper

To use the decorator, we add "@benchmark_decorator" above the target function.

For example:

@benchmark_decorator
def my_function(ar1, arg2, arg3):
	# ...

You can learn more about developing a function benchmark decorator in the tutorial:

Benchmark Function Decorator in Python

Now that we know how to benchmark Python code using time.time(), let's look at some worked examples.

Example of Benchmarking a Statement with time.time()

We can explore how to benchmark a Python statement using time.time() with a worked example.

In this example, we will define a statement that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.

...
# execute the statement
data = [i*i for i in range(100000000)]

We will then surround this statement with benchmarking code.

Firstly, we will record the start time using the time.time() function.

...
# record start time
time_start = time()

Afterward, we will record the end time, calculate the overall execution duration, and report the result.

...
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a statement with time.time()
from time import time
# record start time
time_start = time()
# execute the statement
data = [i*i for i in range(100000000)]
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Running the example first records the start time, the number of seconds since the epoch.

Next, the Python statement is executed, in this case creating a list of 100 million squared integers.

The end time is then recorded, as the number of seconds since the epoch.

The difference between the two recorded times is calculated, providing the statement execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the statement took about 5.228 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python statement using the time.time() function.

Took 5.228 seconds

Next, let's explore an example of benchmarking a function using the time.time() function.

Example of Benchmarking a Function with time.time()

We can explore how to benchmark a Python function using time.time() with a worked example.

In this example, we will define a function that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

We will then call this function, and surround the function call with benchmarking code.

Firstly, we will record the start time using the time.time() function.

...
# record start time
time_start = time()

Afterward, we will record the end time, calculate the overall execution duration, and report the result.

...
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a function with time.time()
from time import time

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

# record start time
time_start = time()
# execute the function
task()
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Running the example first records the start time, the number of seconds since the epoch.

Next, the Python function is called, in this case creating a list of 100 million squared integers.

The end time is then recorded, as the number of seconds since the epoch.

The difference between the two recorded times is calculated, providing the function execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the function took about 6.220 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python function using the time.time() function.

Took 6.220 seconds

Next, let's explore an example of benchmarking a script using the time.time() function.

Example of Benchmarking a Script with time.time()

We can explore how to benchmark a Python script (Python file) using time.time() with a worked example.

In this example, we will update the above example so that it has a main() function and protects the entry point, like a more elaborate Python script.

# main function for script
def main():
    # call a function
    task()

We will then add benchmarking code around the call to the main() function.

# protect the entry point
if __name__ == '__main__':
    # record start time
    time_start = time()
    # execute the script
    main()
    # record end time
    time_end = time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a script with time.time()
from time import time

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

# main function for script
def main():
    # call a function
    task()

# protect the entry point
if __name__ == '__main__':
    # record start time
    time_start = time()
    # execute the script
    main()
    # record end time
    time_end = time()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')

Running the example first records the start time, the number of seconds since the epoch.

Next, the main() function is called which executes the core of the Python script. In this case, it calls our task() function and creates a list of 100 million squared integers.

The end time is then recorded, as the number of seconds since the epoch.

The difference between the two recorded times is calculated, providing the function execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the function took about 6.341 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python script using the time.time() function.

Took 6.341 seconds

Confirm time.time() Does Include Sleep

The time.time() function does include time spent blocked or sleeping.

When the program is blocked or sleeping, the clock used by time.time() is not paused.

We can demonstrate this with a worked example.

We can update the example of benchmarking a statement and include a sleep for 2 seconds.

For example:

...
# sleep for a moment
sleep(2)

This will have an effect on the benchmark time, e.g. it should increase the benchmark time from about 5 seconds to about 7 seconds.

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a statement and sleep with time.time()
from time import time
from time import sleep
# record start time
time_start = time()
# execute the statement
data = [i*i for i in range(100000000)]
# sleep for a moment
sleep(2)
# record end time
time_end = time()
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

Running the example, we can see that the addition of the sleep() after the target code does have the intended effect.

The time of the benchmark increases from about 5 seconds to about 7 seconds.

This highlights that time spent explicitly sleeping is included in the benchmark time when using time.time().

Took 7.168 seconds

Confirm time.time() Is Not Monotonic and Is Adjustable

The clock used by the time.time() function is adjustable.

This means that the system may change the clock while your program is running, possibly making any benchmark results invalid.

Because the clock can be adjusted, it means that it is not monotonic. This means it is possible for future values of the clock to be less than or before past values, due to updates.

We can confirm this by reporting the details of the "time" function in the time module via the time.get_clock_info() function.

This reports the details of the clock used by a function, such as whether it is adjustable, how it is implemented on the platform, whether it is monotonic, and the resolution on the platform.

The program below reports the details of the clock used by the time.time() function.

# SuperFastPython.com
# details of the clock used by time.time()
from time import get_clock_info
# get details
details = get_clock_info('time')
# report details
print(details)

Running the program reports the details of the "time" clock.

We can see that indeed it is not monotonic.

We can also confirm that it is adjustable.

namespace(implementation='clock_gettime(CLOCK_REALTIME)', monotonic=False, adjustable=True, resolution=1.0000000000000002e-06)

Takeaways

You now know how to benchmark Python code using the time.time() function.