How to Benchmark a Python Function

You can benchmark a Python function using the time.perf_counter() function or the timeit module.

Either method can be used to estimate the execution time of a Python code.

Benchmarking is an important step when improving the execution speed of Python programs. Benchmark results provide hard numbers that can be reported and compared directly. This can help in choosing among variations of code for the faster version and see if the code meets performance requirements.

In this tutorial, you will discover how to benchmark a function in Python.

Let’s get started.

Table of Contents

Need to Benchmark a Python Function

A function is a named block of code.

A function can do many things and can be as long or short as we like.

It may take zero, one, or more arguments and it may return a value or nothing at all.

There are many changes we could make to a Python program.

How do we know which version of a Python function is the fastest?

How can we benchmark a function in Python?

Run loops using all CPUs, download your FREE book to learn how.

How to Benchmark a Python Function

There are two main ways to benchmark a Python function:

Use time.perf_counter()
Use timeit

Let’s take a closer look at each in turn.

Benchmark a Function With time.perf_counter()

We can benchmark a function in Python using the time.perf_counter() function.

The time.perf_counter() function was provided for benchmarking.

It has three properties that make it the preferred function for manual benchmarking in Python over other functions, such as time.time(), they are:

Non-Adjustable.
Monotonic.
High-precision.

The time.perf_counter() function reports the time from a clock that is not adjustable. This means that the clock cannot be changed either manually by the system administrator or automatically by clock synchronization or changes for daylight saving.

The time.perf_counter() is monotonic meaning that each value retrieved will be equal to or greater than the previous value. It will never return a value in the past relative to the last value returned.

Finally, the clock used by time.perf_counter() is high-precision, if available. This means we can benchmark Python code that executes very quickly, such as in the order of nanoseconds.

You can learn more about the time.perf_counter() function for benchmarking in the tutorial:

Benchmark Python with time.perf_counter()

The procedure for benchmarking with the time.perf_counter() function is as follows:

Record time.perf_counter() before the function call.
Execute the target function call.
Record time.perf_counter() after the function call.
Subtract start time from after time to give duration.
Report the duration using print().

For example:

...

# record start time

time_start = perf_counter()

# execute the function

myfunc()

# record end time

time_end = perf_counter()

# calculate the duration

time_duration = time_end - time_start

# report the duration

print(f'Took {time_duration} seconds')

Next, let’s look at how we might benchmark using timeit.

Benchmark a Function With timeit

The timeit module is provided in the Python standard library.

It provides an easy way to benchmark snippets of Python code.

It provides two interfaces for benchmarking.

API interface.
Command-line interface.

The first is an API that can be used via the timeit.Timer object or timeit.timeit() and timeit.repeat() module functions.

The timeit.timeit() function takes the Python function to be benchmarked as a string.

For example:

...

# benchmark a python function

result = timeit.timeit('myfunc()')

Any Python code required to execute the benchmark code can be provided as a string to the “setup” argument.

It might include importing the main module so that required functions are imported.

For example:

...

# benchmark a python function with import in setup

result = timeit.timeit('myfunc()', setup='from __main__ import task')

We can do the same thing via the “globals” argument and specify the globals collection.

For example:

...

# benchmark a python function with globals collection

result = timeit.timeit('myfunc()', globals=globals())

Finally, we can specify the number of repetitions of the benchmark code via the “number” argument.

By default, this is set to one million, e.g. 1,000,000, although can be set to a smaller number if the benchmark code takes a long time to execute.

For example:

...

# benchmark a python function with a smaller number

result = timeit.timeit('...', number=100)

The timeit module also provides a command line interface.

Recall that a Python module can be run as a command on the command line directly via the -m flag, followed by the module name.

The timeit module can be run directly in this way, for example:

1	python -m timeit [-n N] [-r N] [-u U] [-s S] [-h] [code ...]

The setup code can be specified via the -s flag, whereas the number of times the function is executed is specified via the -n flag, and the number of repeats via the -r flag.

For example:

1	python -m timeit -n 1 -r 3 "myfunc()"

You can learn more about how to use timeit for benchmarking in the tutorial:

Benchmark Python with timeit

Now that we know how to benchmark a Python function, let’s look at some worked examples.

Start Now: Free Python Benchmarking Crash Course

Example of Benchmarking a Python Function with time.perf_counter()

We can explore how to benchmark a Python function with time.perf_counter() in a worked example.

In this case, we will benchmark a function that creates a list of 100 million squared integers in a list comprehension.

The work() function implements this below.

# define a custom function

def work():

# do some work

data = [i*i for i in range(100000000)]

We will record the time before the function and after the function with the time.perf_counter(), then calculate and report the difference as the benchmark time.

Tying this together, the complete example is listed below.

# SuperFastPython.com

# example of benchmarking a function with time.perf_counter()

from time import perf_counter

# define a custom function

def work():

# do some work

data = [i*i for i in range(100000000)]

# record start time

time_start = perf_counter()

# execute the function

work()

# record end time

time_end = perf_counter()

# calculate the duration

time_duration = time_end - time_start

# report the duration

print(f'Took {time_duration} seconds')

Running the example records the start time, executes the function, and then records the end time.

In this case, we can see that the benchmark time calculated as the difference between the two times is about 6.196887977886945 seconds.

Your results may vary. This is due to the natural variability in running a program on a computer. You can try to repeat the benchmark measurement many times and calculate the average result.

1	Took 6.196887977886945 seconds

Next, let’s look at an example of benchmarking the same function using the timeit.timeit() function.

Free Python Benchmarking Course

Get FREE access to my 7-day email course on Python Benchmarking.

Discover benchmarking with the time.perf_counter() function, how to develop a benchmarking helper function and context manager and how to use the timeit API and command line.

Learn more

Example of Benchmarking a Python Function with timeit.timeit()

We can explore how to benchmark a Python function with timeit.timeit() in a worked example.

In this case, we can update the above example to benchmark our work() function using the timeit.timeit() function.

This involves defining the target function call as a string and providing it as an argument to the timeit() function.

It also requires setting the “globals” argument to be equal to the globals() collection so that the function definition is available to the the timeit() function.

We will set the number of times to run the function to one via the “number” argument.

...

# benchmark the function

time_duration = timeit('work()', globals=globals(), number=1)

Tying this together, the complete example is listed below.

# SuperFastPython.com

# example of benchmarking a function with timeit.timeit()

from timeit import timeit

# define a custom function

def work():

# do some work

data = [i*i for i in range(100000000)]

# benchmark the function

time_duration = timeit('work()', globals=globals(), number=1)

# report the duration

print(f'Took {time_duration} seconds')

Running the example executes the function one time using the timeit.timeit() function.

In this case, we can see that the benchmark result was about 6.281624699011445 seconds.

Your results may vary. This is due to the natural variability in running a program on a computer. You can try to repeat the benchmark measurement many times and calculate the average result.

1	Took 6.281624699011445 seconds

Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

Frequently Asked Questions

This section lists frequently asked questions about benchmarking and their answers.

Do you have a question?
Share it in the comments below and I will do my best to help.

Why Does Each Approach Give Different Results?

Shouldn’t we get the same score regardless of the method used to benchmark?

No.

Each method introduces small differences in what code is executed and when. These differences are reflected in the variation in the benchmark score and differences between benchmark methods.

We must expect measures to differ between methods.

Instead of aiming for an absolute measure of the execution time of code, we must aim for the relative differences when using one method when measuring different variations of the same code.

This allows us to compare relative execution time and choose code that performs “fastest” when measured in a consistent manner.

Why Do We Get Different Results Each Time?

We will get a different benchmark result each time.

The reason is because of the natural variability of running computer programs caused by things such as other programs running at the same time.

We can reduce this variability by repeating a benchmark measurement many times and calculating the average score. This will be a more stable and reliable estimate of the performance of the program

You can learn more about this in the tutorial:

Repeat Benchmarks to Get Stable Results

Which Benchmark Method Should I Use?

Use the method that best suits you.

Perhaps you prefer the manual control at that time.perf_counter() provides.

Perhaps you prefer to use the timeit API.

Perhaps you prefer the command line interface.

Use the method that is the best fit for your style, then use it consistently when comparing variations of the same code.

How Do We Present Benchmark Results?

It is important to choose a numerical precision (resolution ) that best suits your benchmark results.

Too much precision is confusing and too little can hide detail. You must strike the right level of detail.

It is also important to choose a unit of measure that best suits your measures.

This might be hours, minutes, seconds, milliseconds, microseconds, or nanoseconds.

The choice of the unit of measure also interacts with the numerical precision of the results.

You can learn more about these issues in the tutorial:

Tips When Presenting Benchmark Results

What About profile and cProfile modules?

Profiling is different from benchmarking.

Benchmarking is about estimating the performance of code, typically for the purposes of comparison.

Profiling is about discovering why code is slow, and about what is occupying time when code executes.

We can use profiling to understand code and change it in order to improve a benchmark result.

Profiling cannot be used for benchmarking.

You can learn more about profiling and its relationship to benchmarking in the tutorial:

Profile Python Programs

Loving The Tutorials?

Why not take the next step? Get the book.

Learn more

Takeaways

You now know how to benchmark a function in Python.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Homero Thompson on Unsplash

How to Benchmark a Python Function

Need to Benchmark a Python Function

How to Benchmark a Python Function