You can benchmark Python statements using the time.perf_counter() function or the timeit module.

Either method can be used to estimate the execution time of a Python code.

Benchmarking is an important step when improving the execution speed of Python programs. Benchmark results provide hard numbers that can be reported and compared directly. This can help in choosing among variations of code for the faster version and see if the code meets performance requirements.

In this tutorial, you will discover how to benchmark a statement in Python.

Let’s get started.

Need to Benchmark a Python Statement

A statement is a line of code in Python.

A statement could be many things, we can get a lot done on one line in Python, for example:

• Perform a calculation and assign the result.
• A list comprehension with sorting.
• String manipulation and formatting.

And so much more.

There are typically many different ways to achieve the same result on one line in Python.

How do we know which approach is fastest?

How can we benchmark a one-line statement in Python?

How to Benchmark a Python Statement

There are two main ways to benchmark a Python statement:

1. Use time.perf_counter()
2. Use timeit

Let’s take a closer look at each in turn.

Benchmark a Statement With time.perf_counter()

We can benchmark a statement in Python using the time.perf_counter() function.

The time.perf_counter() function was provided for benchmarking.

It has three properties that make it the preferred function for manual benchmarking in Python over other functions, such as time.time(), they are:

1. Non-Adjustable.
2. Monotonic.
3. High-precision.

The time.perf_counter() function reports the time from a clock that is not adjustable. This means that the clock cannot be changed either manually by the system administrator or automatically by clock synchronization or changes for daylight saving.

The time.perf_counter() is monotonic meaning that each value retrieved will be equal to or greater than the previous value. It will never return a value in the past relative to the last value returned.

Finally, the clock used by time.perf_counter() is high-precision, if available. This means we can benchmark Python code that executes very quickly, such as in the order of nanoseconds.

You can learn more about the time.perf_counter() function for benchmarking in the tutorial:

• Benchmark Python with time.perf_counter()

The procedure for benchmarking with the time.perf_counter() function is as follows:

1. Record time.perf_counter() before the statement.
2. Execute the statement.
3. Record time.perf_counter() after the statement.
4. Subtract start time from after time to give duration.
5. Report the duration using print().

For example:

Next, let’s look at how we might benchmark using timeit.

Benchmark a Statement With timeit

The timeit module is provided in the Python standard library.

It provides an easy way to benchmark single statements and snippets of Python code.

It provides two interfaces for benchmarking.

1. API interface.
2. Command-line interface.

The first is an API that can be used via the timeit.Timer object or timeit.timeit() and timeit.repeat() module functions.

The timeit.timeit() function takes the Python statement to be benchmarked as a string.

For example:

Any Python code required to execute the benchmark code can be provided as a string to the “setup” argument.

It might include importing the main module so that required functions are imported.

For example:

Finally, we can specify the number of repetitions of the benchmark code via the “number” argument.

By default, this is set to one million, e.g. 1,000,000, although can be set to a smaller number if the benchmark code takes a long time to execute.

For example:

The timeit module also provides a command line interface.

Recall that a Python module can be run as a command on the command line directly via the -m flag, followed by the module name.

The timeit module can be run directly in this way, for example:

The setup statement can be specified via the -s flag, whereas the number of times the statement is executed is specified via the -n flag, and the number of repeats via the -r flag.

For example:

You can learn more about how to use timeit for benchmarking in the tutorial:

• Benchmark Python with timeit

Now that we know how to benchmark a Python statement, let’s look at some worked examples.

Example of Benchmarking a Python Statement with time.perf_counter()

We can explore how to benchmark a Python statement with time.perf_counter() in a worked example.

In this case, we will benchmark a statement that creates a list of 100 million squared integers in a list comprehension.

For example:

We will record the time before the statement and after the statement with the time.perf_counter(), then calculate and report the difference as the benchmark time.

Tying this together, the complete example is listed below.

Running the example records the start time, executes the statement, and then records the end time.

In this case, we can see that the benchmark time calculated as the difference between the two times is about 5.016603998839855 seconds.

Your results may vary. This is due to the natural variability in running a program on a computer. You can try to repeat the benchmark measurement many times and calculate the average result.

Next, let’s look at an example of benchmarking the same statement using the timeit.timeit() function.

Example of Benchmarking a Python Statement with timeit.timeit()

We can explore how to benchmark a Python statement with timeit.timeit() in a worked example.

In this case, we can update the above example to benchmark the list comprehension statement using the timeit.timeit() function.

This involves defining the target statement as a string and providing it as an argument to the timeit() function.

We will set the number of times to run the statement to one via the “number” argument.

Tying this together, the complete example is listed below.

Running the example executes the statement one time using the timeit.timeit() function.

In this case, we can see that the benchmark result was about 6.107251399895176 seconds.

Your results may vary. This is due to the natural variability in running a program on a computer. You can try to repeat the benchmark measurement many times and calculate the average result.

Next, let’s look at how we might benchmark the same statement using the timeit command line interface.

Example of Benchmarking a Python Statement with timeit Command Line

We can explore how to benchmark a Python statement with the timeit command line interface with a worked example.

In this case, we can provide the statement as a string to the timeit command.

We will execute the statement one time and report the best time (smallest time) over 3 repetitions of executing the statement.

Running the example executes the statement three times.

It then reports the best score, which in this case was about 5.94 seconds.

Your results may vary. This is due to the natural variability in running a program on a computer. You can try to repeat the benchmark measurement many times and calculate the average result.

Frequently Asked Questions

This section lists frequently asked questions about benchmarking and their answers.

Do you have a question?
Share it in the comments below and I will do my best to help.

Why Does Each Approach Give Different Results?

Shouldn’t we get the same score regardless of the method used to benchmark?

No.

Each method introduces small differences in what code is executed and when. These differences are reflected in the variation in the benchmark score and differences between benchmark methods.

We must expect measures to differ between methods.

Instead of aiming for an absolute measure of the execution time of code, we must aim for the relative differences when using one method when measuring different variations of the same code.

This allows us to compare relative execution time and choose code that performs “fastest” when measured in a consistent manner.

Why Do We Get Different Results Each Time?

We will get a different benchmark result each time.

The reason is because of the natural variability of running computer programs caused by things such as other programs running at the same time.

We can reduce this variability by repeating a benchmark measurement many times and calculating the average score. This will be a more stable and reliable estimate of the performance of the program

You can learn more about this in the tutorial:

• Repeat Benchmarks to Get Stable Results

Which Benchmark Method Should I Use?

Use the method that best suits you.

Perhaps you prefer the manual control at that time.perf_counter() provides.

Perhaps you prefer to use the timeit API.

Perhaps you prefer the command line interface.

Use the method that is the best fit for your style, then use it consistently when comparing variations of the same code.

How Do We Present Benchmark Results?

It is important to choose a numerical precision (resolution ) that best suits your benchmark results.

Too much precision is confusing and too little can hide detail. You must strike the right level of detail.

It is also important to choose a unit of measure that best suits your measures.

This might be hours, minutes, seconds, milliseconds, microseconds, or nanoseconds.

The choice of the unit of measure also interacts wit the numerical precision of the results.

You can learn more about these issues in the tutorial:

• Tips When Presenting Benchmark Results

What About profile and cProfile modules?

Profiling is different from benchmarking.

Benchmarking is about estimating the performance of code, typically for the purposes of comparison.

Profiling is about discovering why code is slow, and about what is occupying time when code executes.

We can use profiling to understand code and change it in order to improve a benchmark result.

Profiling cannot be used for benchmarking.

You can learn more about profiling and its relationship to benchmarking in the tutorial:

• Profile Python Programs

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python Benchmarking, Jason Brownlee (my book!)

Also, the following Python books have chapters on benchmarking that may be helpful:

• Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
• High Performance Python, 2020. (chapter 2)

Guides

• 4 Ways to Benchmark Python Code
• 5 Ways to Measure Execution Time in Python
• Python Benchmark Comparison Metrics

Benchmarking APIs

• time — Time access and conversions
• timeit — Measure execution time of small code snippets
• The Python Profilers

References

• Benchmark (computing), Wikipedia.
• Profiling (computer programming), Wikipedia.

Takeaways

You now know how to benchmark a statement in Python.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Caesar Aldhela on Unsplash