Benchmark Decorator in Python

September 12, 2023 Python Benchmarking

You can develop a custom benchmark decorator that will automatically record and report the execution time of target functions.

This requires defining the decorator function and adding the decoration to the function to be benchmarked.

In this tutorial, you will discover how to automatically benchmark target functions using a benchmark decorator in Python.

Let's get started.

Benchmark Python Code With time.perf_counter()

We can benchmark Python code using the time module.

The time.perf_counter() function will return a value from a high-performance counter.

Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration.
-- time — Time access and conversions

The difference between the two calls to the time.perf_counter() function can provide a high-precision estimate of the execution time of a block of code.

Unlike the time.time() function, the time.perf_counter() function is not subject to updates, such as daylight saving and synchronizing the system clock with a time server. This makes the time.perf_counter() function is a reliable approach to benchmarking Python code.

We can call the time.perf_counter() function at the beginning of the code we wish to benchmark, and again at the end of the code we wish to benchmark.

For example:

...
# record start time
time_start = time.perf_counter()
# call benchmark code
task()
# record end time
time_end = time.perf_counter()

The difference between the start and end time is the total duration of the program in seconds.

For example:

...
# calculate the duration
time_duration = time_end - time_start
# report the duration
print(f'Took {time_duration:.3f} seconds')

You can learn more about benchmarking Python code with the time.perf_counter() function in the tutorial:

Benchmark Python with time.perf_counter()

How can we hide all of this code so that we can benchmark with a simple interface?

Can we develop a function decorator that will benchmark our code automatically?

How to Develop a Benchmark Function Decorator

We can develop our custom function decorator to automatically benchmark our target functions.

Firstly, let's review function decorators.

Function Decorators

A function decorator in Python allows a custom function to be called automatically that will in turn call our target function.

This can be used to insert code before and after calling our target function, such as recording the start time, and end time, and calculating an overall execution duration.

A function decorator can be defined as a custom function that returns a function that in turn calls our target function.

Typically, the new separate function is defined as an inner function, that is, within the called function.

For example:

# define the custom decorator
def custom_decorator(func):
	# inner function that wraps the target function
	def inner_wrapper():
		# call the target function
		func()

The decorator can be added to arbitrary functions in our program as follows:

# function that has the decorator
@custom_decorator
def task():
	# ...

The function we are adding the decorator to, e..g task() may take arguments and may have a return value.

Therefore our inner wrapper function needs to handle this accordingly.

For example:

# define the custom decorator
def custom_decorator(func):
	# inner function that wraps the target function
	def inner_wrapper(*args, **kwargs):
		# call the target function
		return func(*args, **kwargs)

Additionally, we may want to leave the decorator on the target function and have the function look and feel unchanged, e.g. if we print the function or call help() on the function. With the decorator in place, this will not be the case.

Instead, we can have the decorator look like the target function passed in, so that any interrogation of the target function looks normal.

This can be achieved by adding the functools.wraps decorator to our inner wrapper function.

This is a convenience function for invoking update_wrapper() as a function decorator when defining a wrapper function. [...] Without the use of this decorator factory, the name of the example function would have been 'wrapper', and the docstring of the original example() would have been lost.
-- functools — Higher-order functions and operations on callable objects

For example:

# define the custom decorator
def custom_decorator(func):
	# inner function that wraps the target function
	@wraps(func)
	def inner_wrapper(*args, **kwargs):
		# call the target function
		return func(*args, **kwargs)

And that's about it for decorators for now.

Next, let's look at how we can define custom decorators to automatically benchmark our target functions.

One-Off Benchmark Function Decorator

We can develop a function decorator to benchmark a Python function automatically.

The decorator will be called benchmark_decorator() and takes the function to be decorated, e.g. called.

# define the benchmark decorator
def benchmark_decorator(func):
	# ...

Next, we can define the inner wrapper function.

It must take arguments for the target function, just in case. It then must record the start time before calling the function, and the end time after calling the function.

It then calculates the duration and reports it before returning any return value from the target function itself.

For example:

# inner function that wraps the target function
@wraps(func)
def wrapper(*args, **kwargs):
    # record start time
    time_start = perf_counter()
    # call the custom function
    result = func(*args, **kwargs)
    # record end time
    time_end = perf_counter()
    # calculate the duration
    time_duration = time_end - time_start
    # report the duration
    print(f'Took {time_duration:.3f} seconds')
    # pass on the return value
    return result

Tying this together, the complete function decorator for benchmarking target functions is listed below.

# define the benchmark decorator
def benchmark_decorator(func):
    # inner function that wraps the target function
    @wraps(func)
    def wrapper(*args, **kwargs):
        # record start time
        time_start = perf_counter()
        # call the custom function
        result = func(*args, **kwargs)
        # record end time
        time_end = perf_counter()
        # calculate the duration
        time_duration = time_end - time_start
        # report the duration
        print(f'Took {time_duration:.3f} seconds')
        # pass on the return value
        return result
    # return the inner function
    return wrapper

It requires two import statements, and these could be moved into the function itself if needed.

Finally, to use the decorator, we add "@benchmark_decorator" above the target function.

For example:

@benchmark_decorator
def custom_function():
	# ...

This is not a new idea. Many people have proposed developing benchmark function annotations.

A notable example is in the Python Cookbook in the recipe: "14.13. Profiling and Timing Your Program".

Next, let's look at how we might update the decorator to report the average execution time.

Repeated Benchmark Function Decorator

It can be a good idea to repeat a benchmark many times and report the average duration.

This is because each benchmark score may be slightly different due to the Python interpreter having to load modules for the first time, the Python garbage collector, and other programs running in the background of the system at the same time.

We can update the benchmark function decorator from the last section to calculate the average execution time instead of a one-off execution time.

This will require executing the target function multiple times, e.g. 3, which could have side effects if the function is changing the program state.

We can update the internal wrapper() function to loop three times, and each iteration benchmarks the target function.

The scores can be collected in a list.

...
results = list()
# repeat the benchmark many times
n_repeats = 3
for i in range(n_repeats):
    # record start time
    time_start = perf_counter()
    # call the custom function
    result = func(*args, **kwargs)
    # record end time
    time_end = perf_counter()
    # calculate the duration
    time_duration = time_end - time_start
    # store the result
    results.append(time_duration)
    # report progress
    print(f'>run {i+1} took {time_duration:.3f} seconds')

After all executions of the target function, we can calculate the average execution time and report the result, before returning the final return value from the target function.

...
# calculate average duration
avg_duration = sum(results) / n_repeats
# report the average duration
print(f'Took {avg_duration:.3f} seconds on average')
# pass on the return value
return result

Tying this together, the complete updated benchmark decorator is listed below.

# define the average benchmark decorator
def average_benchmark_decorator(func):
    # inner function that wraps the target function
    @wraps(func)
    def wrapper(*args, **kwargs):
        results = list()
        # repeat the benchmark many times
        n_repeats = 3
        for i in range(n_repeats):
            # record start time
            time_start = perf_counter()
            # call the custom function
            result = func(*args, **kwargs)
            # record end time
            time_end = perf_counter()
            # calculate the duration
            time_duration = time_end - time_start
            # store the result
            results.append(time_duration)
            # report progress
            print(f'>run {i+1} took {time_duration:.3f} seconds')
        # calculate average duration
        avg_duration = sum(results) / n_repeats
        # report the average duration
        print(f'Took {avg_duration:.3f} seconds on average')
        # pass on the return value
        return result
    # return the inner function
    return wrapper

As above, this would be used by adding the @average_benchmark_decorator above the target function.

For example:

@average_benchmark_decorator
def custom_function():
	# ...

Now that we know how to develop a benchmark function decorator, let's look at some worked examples.

Example of Benchmark Function Decorator

We can explore how to use our one-off benchmark function decorator to benchmark the execution time of a custom function.

In this example, we will define a custom function that takes a moment to complete.

The function creates a list of 100 million squared integers in a list comprehension.

For example:

# function to benchmark
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

We can then add our @benchmark_decorator decoration to our task() function.

# function to benchmark, with benchmark decorator
@benchmark_decorator
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

Then, all we need to do is call the task() function from the entry point of the program and it will be benchmarked automatically.

Tying this together, the complete example of using our function decorator function to estimate the duration of our task() target function is listed below.

# SuperFastPython.com
# example of benchmarking execution time using a decorator
from time import perf_counter
from functools import wraps

# define the benchmark decorator
def benchmark_decorator(func):
    # inner function that wraps the target function
    @wraps(func)
    def wrapper(*args, **kwargs):
        # record start time
        time_start = perf_counter()
        # call the custom function
        result = func(*args, **kwargs)
        # record end time
        time_end = perf_counter()
        # calculate the duration
        time_duration = time_end - time_start
        # report the duration
        print(f'Took {time_duration:.3f} seconds')
        # pass on the return value
        return result
    # return the inner function
    return wrapper

# function to benchmark, with benchmark decorator
@benchmark_decorator
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

# protect the entry point
if __name__ == '__main__':
    # call the custom function
    task()

Running the program calls the task() function.

The benchmark decorator runs and records the start time.

It then calls the target task() function with any arguments provided to the helper function, in this case, no arguments, and retrieves the return value.

The task function runs normally and returns.

The benchmark function records the end time and then calculates the duration.

The duration is then reported to the standard output.

In this case, we can see that the task() function took about 6.441 seconds to complete.

This highlights how we can benchmark arbitrary Python functions using our benchmark function decorator.

Took 6.441 seconds

Next, let's look at how we might report the average duration using our repeated benchmark decorator.

Example of Repeated Benchmark Function Decorator

We can explore how to repeatedly benchmark our target function using a decorator.

In this example, we will use our benchmark_decorator() decorator developed above which has three repeated benchmark runs to estimate the average run time of our task() function.

# define the average benchmark decorator
def average_benchmark_decorator(func):
    # inner function that wraps the target function
    @wraps(func)
    def wrapper(*args, **kwargs):
        results = list()
        # repeat the benchmark many times
        n_repeats = 3
        for i in range(n_repeats):
            # record start time
            time_start = perf_counter()
            # call the custom function
            result = func(*args, **kwargs)
            # record end time
            time_end = perf_counter()
            # calculate the duration
            time_duration = time_end - time_start
            # store the result
            results.append(time_duration)
            # report progress
            print(f'>run {i+1} took {time_duration:.3f} seconds')
        # calculate average duration
        avg_duration = sum(results) / n_repeats
        # report the average duration
        print(f'Took {avg_duration:.3f} seconds on average')
        # pass on the return value
        return result
    # return the inner function
    return wrapper

We can then change the decorator above our target function.

# function to benchmark, with benchmark decorator
@average_benchmark_decorator
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

And that's it.

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking average execution time using a decorator
from time import perf_counter
from functools import wraps

# define the average benchmark decorator
def average_benchmark_decorator(func):
    # inner function that wraps the target function
    @wraps(func)
    def wrapper(*args, **kwargs):
        results = list()
        # repeat the benchmark many times
        n_repeats = 3
        for i in range(n_repeats):
            # record start time
            time_start = perf_counter()
            # call the custom function
            result = func(*args, **kwargs)
            # record end time
            time_end = perf_counter()
            # calculate the duration
            time_duration = time_end - time_start
            # store the result
            results.append(time_duration)
            # report progress
            print(f'>run {i+1} took {time_duration:.3f} seconds')
        # calculate average duration
        avg_duration = sum(results) / n_repeats
        # report the average duration
        print(f'Took {avg_duration:.3f} seconds on average')
        # pass on the return value
        return result
    # return the inner function
    return wrapper

# function to benchmark, with benchmark decorator
@average_benchmark_decorator
def task():
    # create a large list
    data = [i*i for i in range(100000000)]

# protect the entry point
if __name__ == '__main__':
    # call the custom function
    task()

Running the program calls our custom task() function.

The benchmark decorator runs the main benchmark loop.

Each iteration, the loop records the start time. It then calls the target function with any arguments provided to the helper function, in this case, no arguments.

The task function runs normally and returns. The benchmark decorator records the end time and then calculates the duration and reports it as a progress indicator.

This is repeated three times.

Finally, the average of all runs is calculated and then reported.

In this case, we can see that the task() function took about 6.136 seconds to complete on average.

This highlights how we can automatically calculate and report the average execution time for a target function using a custom benchmark decorator.

>run 1 took 6.191 seconds
>run 2 took 6.077 seconds
>run 3 took 6.138 seconds
Took 6.136 seconds on average

Takeaways

You now know how to automatically benchmark target functions using a benchmark decorator in Python.

If you enjoyed this tutorial, you will love my book: Python Benchmarking. It covers everything you need to master the topic with hands-on examples and clear explanations.