Benchmark Python with timeit.timeit()

October 5, 2023 Python Benchmarking

You can benchmark snippets of Python code using the timeit.timeit() function.

In this tutorial, you will discover how to benchmark Python code using the timeit.timeit() function.

Let's get started.

What is timeit

The timeit module is provided in the Python standard library.

It provides an easy way to benchmark single statements and snippets of Python code.

This module provides a simple way to time small bits of Python code. It has both a Command-Line Interface as well as a callable one. It avoids a number of common traps for measuring execution times.
-- timeit — Measure execution time of small code snippets

The timeit module provides two interfaces for benchmarking.

API interface.
Command-line interface.

The first is an API that can be used via the timeit.Timer object or timeit.timeit() and timeit.repeat() module functions.

The second is a command line interface.

Both are intended to benchmark single Python statements, although multiple lines and multiple statements can be benchmarked using the module.

How to Benchmark With timeit.timeit()

The timeit.timeit() benchmarks Python code and reports the duration in seconds.

Create a Timer instance with the given statement, setup code and timer function and run its timeit() method with number executions.
-- timeit — Measure execution time of small code snippets

The timeit.timeit() takes the Python statement to be benchmarked as a string.

For example:

...
# benchmark a python statement
result = timeit.timeit('[i*i for i in range(1000)]')

Setup Argument

Any Python code required to execute the benchmark code can be provided as a string to the "setup" argument.

This might include defining a variable.

The setup code is only executed once prior to the benchmark.

For example:

...
# benchmark a python statement with setup code
result = timeit.timeit('[i*i for i in range(total)]', setup='total=10000')

It might include importing the main module so that required functions are imported.

For example:

...
# benchmark a python statement with import in setup
result = timeit.timeit('task()', setup='from __main__ import task')

Globals Argument

Alternatively, if we have defined code in our program that is required to execute the benchmark code, we can specify the "globals" argument to provide the entire namespace.

We can specify locals() or globals() which will include a namespace from our current program.

For example:

...
# benchmark a python statement with a namespace
result = timeit.timeit('task()', globals=globals())

Number Argument

Finally, we can specify the number of repetitions of the benchmark code via the "number" argument.

By default, this is set to one million, e.g. 1,000,000, although can be set to a smaller number if the benchmark code takes a long time to execute.

For example:

...
# benchmark a python statement with a smaller number
result = timeit.timeit('[i*i for i in range(1000)]', number=100)

The "number" argument should be set so that the overall duration is at least 0.2 or 0.5 seconds, perhaps even more than one second.

Now that we know how to use the timeit.timeit() function, let's look at some worked examples.

Example of Standalone Code with timeit.timeit()

In this example, we will benchmark standalone Python code.

This is code that does not depend on any other variables or functions.

We will benchmark two different ways to calculate a list of 1,000 squared numbers.

The first method will multiply each number by itself.

[i*i for i in range(1000)]

The second method will use the exponent operator (**) and raise each number to the power of 2.

[i**2 for i in range(1000)]

These snippets execute very quickly, therefore we will repeat each snippet 100,000 times when calling the timeit.timeit() function.

...
# benchmark the i*i method
time_method1 = timeit('[i*i for i in range(1000)]', number=100000)
...
# benchmark the i**2 method
time_method2 = timeit('[i**2 for i in range(1000)]', number=100000)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a list of numbers with timeit.timeit()
from timeit import timeit
# benchmark the i*i method
time_method1 = timeit('[i*i for i in range(1000)]', number=100000)
# report the duration
print(f'i*i: {time_method1} seconds')
# benchmark the i**2 method
time_method2 = timeit('[i**2 for i in range(1000)]', number=100000)
# report the duration
print(f'i**2: {time_method2} seconds')

Running the example first executes the i*i 100,000 times and reports the overall duration.

It then runs the i**2 100,000 times and reports the overall duration.

In this case, we can see that the i*i took about 4.180 seconds, whereas the i** took about 25.427 seconds.

This suggests that the i*i method is faster.

This highlights how we can benchmark standalone code using timeit.timeit().

i*i: 4.180790501181036 seconds
i**2: 5.427739103091881 seconds

Next, let's look at including a benchmark that requires setup.

Example of Code With Setup using timeit.timeit()

We can explore a timeit.timeit() benchmark that requires setup code.

In this case, we will add one more method to the benchmark suite for squaring a list of 1,000 numbers.

The new method will use the math.pow() function, which takes the number and the exponent as arguments.

[pow(i,2) for i in range(1000)]

This requires that the pow() function be imported from the math module.

We can achieve this by including the important statement via the "setup" argument to the timeit.timeit() function.

...
# benchmark the math.pow() method
time_method3 = timeit('[pow(i,2) for i in range(1000)]', setup='from math import pow', number=100000)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a list of numbers with timeit.timeit()
from timeit import timeit
# benchmark the i*i method
time_method1 = timeit('[i*i for i in range(1000)]', number=100000)
# report the duration
print(f'i*i: {time_method1} seconds')
# benchmark the i**2 method
time_method2 = timeit('[i**2 for i in range(1000)]', number=100000)
# report the duration
print(f'i**2: {time_method2} seconds')
# benchmark the math.pow() method
time_method3 = timeit('[pow(i,2) for i in range(1000)]', setup='from math import pow', number=100000)
# report the duration
print(f'math.pow(): {time_method3} seconds')

In this case, we benchmark the i*i and i**2 methods as before, achieving similar but not identical results, specifically, 4.180 seconds and 5.515 seconds respectively.

Then the new math.pow() method is benchmarked. In this case, it takes about 9.239 seconds to complete overall.

We can see that it is more than twice as slow as the i*i method.

This highlights how we can benchmark a function that requires importing using timeit.timeit().

i*i: 4.180096419993788 seconds
i**2: 5.515898708021268 seconds
math.pow(): 9.239141091005877 seconds

Next, we will explore how we can benchmark a custom function.

Example of a Custom Function with timeit.timeit()

We can explore a timeit.timeit() benchmark of a custom function.

In this case, we will define a new function that squares a given number.

# define a custom function for squaring numbers
def square(value):
    return value * value

We will then create a list of 1,000 squared numbers that use our custom function.

[square(i) for i in range(1000)]

This requires that the square() function be made available to the benchmark snippet.

We can achieve this via the "globals" argument and provide the current namespace via the globals() built-in function, which will include the definition of our custom square() function.

...
# benchmark the square() method
time_method4 = timeit('[square(i) for i in range(1000)]', globals=globals(), number=100000)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of benchmarking a list of numbers with timeit.timeit()
from timeit import timeit

# define a custom function for squaring numbers
def square(value):
    return value * value

# benchmark the i*i method
time_method1 = timeit('[i*i for i in range(1000)]', number=100000)
# report the duration
print(f'i*i: {time_method1} seconds')
# benchmark the i**2 method
time_method2 = timeit('[i**2 for i in range(1000)]', number=100000)
# report the duration
print(f'i**2: {time_method2} seconds')
# benchmark the math.pow() method
time_method3 = timeit('[pow(i,2) for i in range(1000)]', setup='from math import pow', number=100000)
# report the duration
print(f'math.pow(): {time_method3} seconds')
# benchmark the square() method
time_method4 = timeit('[square(i) for i in range(1000)]', globals=globals(), number=100000)
# report the duration
print(f'square(): {time_method4} seconds')

In this case, we benchmark the i*i, i**2, and math.pow() methods as before, achieving similar but not identical results, specifically, 4.159 seconds, 5.522 seconds, and 9.287 seconds respectively.

Then the new custom square() function approach is benchmarked. In this case, it completes in about 7.319 seconds.

This is slower than the simple i*i method, although it is faster than the math.pow() function.

This highlights how we can benchmark a custom function using timeit.timeit().

i*i: 4.1599748940061545 seconds
i**2: 5.5225951920001535 seconds
math.pow(): 9.287094817002071 seconds
square(): 7.319860859002802 seconds

Takeaways

You now know how to benchmark Python code using the timeit.timeit() function.

If you enjoyed this tutorial, you will love my book: Python Benchmarking. It covers everything you need to master the topic with hands-on examples and clear explanations.