Last Updated on October 3, 2023

You can benchmark Python code using the time.perf_counter() function.

In this tutorial, you will discover how to benchmark Python code using the time.perf_counter() function.

Let’s get started.

Need to Benchmark Python Code

Benchmarking Python code refers to comparing the performance of one program to variations of the program.

Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.

— Benchmarking, Wikipedia.

Typically, we make changes to the programs, such as adding concurrency, in order to improve the performance of the program on a given system.

Improving performance typically means reducing the run time of the program.

Therefore, when we benchmark programs in Python after adding concurrency, we typically are interested in recording how long a program takes to run.

It is critical to be systematic when benchmarking code.

The first step is to record how long an unmodified version of the program takes to run. This provides a baseline in performance to which all other versions of the program must be compared. If we are adding concurrency, then the unmodified version of the program will typically perform tasks sequentially, e.g. one-by-one.

We can then make modifications to the program, such as adding thread pools, process pools, or asyncio. The goal is to perform tasks concurrently (out of order), even in parallel (simultaneously). The performance of the program can be benchmarked and compared to the performance of the unmodified version.

The performance of modified versions of the program must have better performance than the unmodified version of the program. If they do not, they are not improvements and should not be adopted.

How can we benchmark the performance of programs in Python?

Benchmark With time.perf_counter()

We can benchmark Python code using the time.perf_counter() function.

This is a function that is provided in the time module and is part of the Python standard library.

It can be used to record the time before a piece of code that is being benchmarked, then again to record the time again after the benchmarked code. The difference between the two times can be reported as the relative execution time, e.g. compared to another benchmark.

Let’s take a closer look.

What is time.perf_counter()?

The time.perf_counter() function reports the value of a performance counter on the system. It does not report the time since epoch like time.time().

Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. It does include time elapsed during sleep and is system-wide.

— time — Time access and conversions

The returned value is in seconds with fractional components (e.g. milliseconds and nanoseconds), providing a high-resolution timestamp.

Calculating the difference between two timestamps from the time.perf_counter() allows high-resolution execution time benchmarking, e.g. in the millisecond and nanosecond range.

The timestamp from the time.perf_counter() function is consistent, meaning that two durations can be compared relative to each other in a meaningful way.

The time.perf_counter() function was introduced in Python version 3.3 with the intended use for short-duration benchmarking.

perf_counter(): Performance counter with the highest available resolution to measure a short duration.

— What’s New In Python 3.3

The perf_counter() function was specifically designed to overcome the limitations of other time functions to ensure that the result is consistent across platforms and monotonic (always increasing).

To measure the performance of a function, time.clock() can be used but it is very different on Windows and on Unix. […] The new time.perf_counter() function should be used instead to always get the most precise performance counter with a portable behaviour (ex: include time spend during sleep).

— PEP 418 – Add monotonic time, performance counter, and process time functions

For accuracy, the timeit module uses the time.perf_counter() internally.

The default timer, which is always time.perf_counter().

— timeit — Measure execution time of small code snippets

Why Not Benchmark With time.time()?

It is common to benchmark code with time.time() instead of the time.perf_counter().

Nevertheless, the time.time() function has some limitations, such as:

1. Lack of Precision: time.time() provides time measurements in seconds and fractions of a second, which might not be sufficient for accurately profiling very short code segments or fine-grained optimizations.
2. Clock Drift and System Load: System clock drift or variations in system load can introduce inaccuracies in the measurements, leading to inconsistent results.
3. Overhead: The overhead introduced by the time.time() calls themselves can be significant for small and fast operations, potentially skewing the results.

The biggest limitation of time.time() is that it is possible for a subsequent time to be before a previous time. This can be because of many factors such as rounding, system load, updates to the clock (synchronization), and lack of precision.

As such, time.time() is not appropriate for fine-grained benchmarking, such as benchmarking code that has a very short duration, e.g. milliseconds or less.

You can learn more about benchmarking with the time.time() fiction in the tutorial:

• Benchmark Python with time.time()

Why Use time.perf_counter() over time.time()?

When benchmarking Python code, it’s generally preferable to use time.perf_counter() over time.time() for several reasons:

1. Higher Precision: time.perf_counter() provides higher precision time measurements compared to time.time(). It is specifically designed for measuring short intervals and is less susceptible to clock drift or system time adjustments.
2. No Dependencies: Unlike time.time(), which can depend on system clock settings, time.perf_counter() is more reliable and consistent across different platforms and environments.
3. Monotonic Clock: time.perf_counter() uses a monotonic clock, which ensures that the values are always increasing and unaffected by changes to the system time, making it ideal for accurate interval measurements.
4. Minimal Overhead: time.perf_counter() has less overhead compared to time.time(), which makes it more suitable for benchmarking small and fast code snippets without introducing significant interference.
5. Focused Benchmarking: The precision and low overhead of time.perf_counter() makes it well-suited for profiling specific code sections or functions, enabling more focused and accurate benchmarking.
6. Timing Short Intervals: When measuring very short intervals, such as individual function calls, time.perf_counter() provides a more accurate and reliable measurement than time.time().
7. Comprehensive Profiling: In combination with other profiling tools, time.perf_counter() can offer detailed insights into performance bottlenecks, helping developers optimize critical sections of code.
8. Fine-Grained Analysis: For diagnosing performance issues and fine-tuning optimizations, time.perf_counter() allows developers to measure and analyze even the smallest code execution times.
9. Scientific Accuracy: In scientific and performance analysis contexts, time.perf_counter() is recommended for accurate measurements that may be subject to peer review or validation.

While time.perf_counter() is highly useful for benchmarking and profiling, it’s important to remember that it measures time intervals, not absolute time since epoch. Developers should calculate the difference between two perf_counter() values to determine the time taken for a specific operation.

Additionally, for measuring longer time spans or capturing timestamps for other purposes, time.time() or time.monotonic() may still be appropriate.

You can learn more about time.perf_counter() vs time.time() for benchmarking in the tutorial:

• Python time.time() vs time.perf_counter()

Now that we know what the time.perf_counter() function is, let’s look at how we can use it to benchmark Python code.

How to Benchmark with time.perf_counter()

We can use the time.perf_counter() function to benchmark Python code.

There are perhaps 3 case studies we may want to consider, they are:

1. Benchmarking a Python statement.
2. Benchmarking a Python function.
3. Benchmarking a Python script (program).

Let’s look at how we can benchmark time.perf_counter() function.

How to Benchmark a Python Statement

We can use the time.perf_counter() function to benchmark arbitrary Python statements.

The procedure is as follows:

1. Record time.perf_counter() before the statement.
2. Execute the statement.
3. Record time.perf_counter() after the statement.
4. Subtract start time from after time to give duration.
5. Report the duration using print().

For example:

How to Benchmark a Python Function

We can use the time.perf_counter() function to benchmark arbitrary Python functions.

The procedure is as follows:

1. Record time.perf_counter() before the function.
2. Call the function.
3. Record time.perf_counter() after the function.
4. Subtract start time from after time to give duration.
5. Report the duration using print().

For example:

How to Benchmark a Python Script

We can use the time.perf_counter() function to benchmark arbitrary Python scripts (Python files).

It requires that the entry point into the script is first moved into a new function, that we will call main(). This is to make it easy for all code in the script to be wrapped in the benchmarking code.

The procedure is as follows:

1. Move the entry of the script into a main() function (if needed).
2. Record time.perf_counter() before the main() function.
3. Call the main() function.
4. Record time.perf_counter() after the main() function.
5. Subtract start time from after time to give duration.
6. Report the duration using print().

For example:

Advanced Benchmarking with time.perf_counter()

The time.perf_counter() approach to benchmarking Python code can be used in some clever ways, allowing the benchmark process to be hidden or automatic.

Some examples include:

1. Develop a benchmark helper function.
2. Develop a benchmark context manager.
3. Develop a benchmark function decorator.

Let’s take a closer look at each in turn.

Benchmark Helper Function

We can develop a helper function to automatically benchmark our Python code.

Our function can take the name of our target function that we wish to benchmark and optional *args arguments.

The *args is an optional list of function arguments. It allows us to specify zero, one, or many arguments for our target function to the benchmark function, which we can pass on to the target function directly.

Our function can then record the start time, call the target function, record the end time, and report the overall duration.

For example:

We can then call the benchmark() function with the name of the target function to be benchmarked, along with any arguments.

For example:

You can learn more about developing a helper benchmark function in the tutorial:

• Benchmark Helper Function in Python

Benchmark Context Manager

We can hide manual benchmarking of Python code in a context manager.

Recall that a context manager is a Python object that has __enter__() and __exit__() methods and is used via the with expression.

We can define a new class that implements a constructor __init__() the __enter__() and __exit__() methods.

The __init__() constructor can take a name argument for the benchmark case and store it in an object attribute.

The __enter__() method can initialize the start time and store it in object attributes. It can then return an instance of the context manager itself, as a good practice.

The __exit__() method must take some standard arguments about any exception that occurred while running the context code. It can then record the end time, calculate and store the duration and report the calculated duration along with the name of the benchmark case.

For example:

We can then use it by creating an instance of the Benchmark class within the “with” expression and then list any code within the context we wish to benchmark.

For example:

You can learn more about how to develop a benchmark context manager in the tutorial:

• Benchmark Context Manager in Python

Benchmark Function Decorator

A function decorator in Python allows a custom function to be called automatically that will in turn call our target function.

This can be used to insert code before and after calling our target function, such as recording the start time, and end time, and calculating an overall execution duration.

A function decorator can be defined as a custom function that returns a function that in turn calls our target function.

We can develop a function decorator to benchmark a Python function automatically.

The decorator will be called benchmark_decorator() and takes the function to be decorated.

The inner function must take arguments for the target function, just in case. It then must record the start time before calling the function, and the end time after calling the function. It then calculates the duration and reports it before returning any return value from the target function itself.

For example:

To use the decorator, we add “@benchmark_decorator” above the target function.

For example:

You can learn more about developing a function benchmark decorator in the tutorial:

• Benchmark Function Decorator in Python

Now that we know how to benchmark Python code using time.perf_counter(), let’s look at some worked examples.

Example of Benchmarking a Statement with time.perf_counter()

We can explore how to benchmark a Python statement using time.perf_counter() with a worked example.

In this example, we will define a statement that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.

We will then surround this statement with benchmarking code.

Firstly, we will record the start time using the time.perf_counter() function.

Afterward, we will record the end time, calculate the overall execution duration, and report the result.

Tying this together, the complete example is listed below.

Running the example first records the start time, a number from an internal high-performance counter.

Next, the Python statement is executed, in this case creating a list of 100 million squared integers.

The end time is then recorded, as a number from an internal high-performance counter.

The difference between the two recorded times is calculated, providing the statement execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the statement took about 5.070 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python statement using the time.perf_counter() function.

Next, let’s explore an example of benchmarking a function using the time.perf_counter() function.

Example of Benchmarking a Function with time.perf_counter()

We can explore how to benchmark a Python function using time.perf_counter() with a worked example.

In this example, we will define a function that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.

We will then call this function, and surround the function call with benchmarking code.

Firstly, we will record the start time using the time.perf_counter() function.

Afterward, we will record the end time, calculate the overall execution duration, and report the result.

Tying this together, the complete example is listed below.

Running the example first records the start time, a number from an internal high-performance counter.

Next, the Python function is called, in this case creating a list of 100 million squared integers.

The end time is then recorded, as a number from an internal high-performance counter.

The difference between the two recorded times is calculated, providing the function execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the function took about 6.431 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python function using the time.perf_counter() function.

Next, let’s explore an example of benchmarking a script using the time.perf_counter() function.

Example of Benchmarking a Script with time.perf_counter()

We can explore how to benchmark a Python script (Python file) using time.perf_counter() with a worked example.

In this example, we will update the above example so that it has a main() function and protects the entry point, like a more elaborate Python script.

We will then add benchmarking code around the call to the main() function.

Tying this together, the complete example is listed below.

Running the example first records the start time, a number from an internal high-performance counter.

Next, the main() function is called which executes the core of the Python script. In this case, it calls our task() function and creates a list of 100 million squared integers.

The end time is then recorded, as a number from an internal high-performance counter.

The difference between the two recorded times is calculated, providing the function execution duration in seconds.

Finally, the result is reported, truncated to three decimal places (milliseconds).

In this case, we can see that the function took about 6.438 seconds to complete.

Note, the results on your system may vary.

This highlights how we can benchmark a Python script using the time.perf_counter() function.

Confirm time.perf_counter() Does Include Sleep

The time.perf_counter() function does include time spent blocked or sleeping.

When the program is blocked or sleeping, the clock used by time.perf_counter() is not paused.

We can demonstrate this with a worked example.

We can update the example of benchmarking a statement and include a sleep for 2 seconds.

For example:

This will have an effect on the benchmark time, e.g. it should increase the benchmark time from about 5 seconds to about 7 seconds.

Tying this together, the complete example is listed below.

Running the example, we can see that the addition of the sleep() after the target code does have the intended effect.

The time of the benchmark increases from about 5 seconds to about 7 seconds.

This highlights that time spent explicitly sleeping is included in the benchmark time when using time.perf_counter().

Confirm time.perf_counter() Is Monotonic and Is Not Adjustable

The clock used by the time.perf_counter() function is not adjustable.

This means that the system will not change the clock while your program is running.

Because the clock cannot be adjusted, it means that it is monotonic. This means that all future values of the clock will be after past values of the clock.

We can confirm this by reporting the details of the “perf_counter” function in the time module via the time.get_clock_info() function.

This reports the details of the clock used by a time function, such as whether it is adjustable, how it is implemented on the platform, whether it is monotonic, and the resolution on the platform.

The program below reports the details of the “perf_counter” clock used by the time.perf_counter() function.

Running the program reports the details of the “perf_counter” clock.

We can see that indeed it is monotonic and that the resolution on the platform is 1e-09.

We can also confirm that it is not adjustable.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python Benchmarking, Jason Brownlee (my book!)

Also, the following Python books have chapters on benchmarking that may be helpful:

• Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
• High Performance Python, 2020. (chapter 2)

Guides

• 4 Ways to Benchmark Python Code
• 5 Ways to Measure Execution Time in Python
• Python Benchmark Comparison Metrics

Benchmarking APIs

• time — Time access and conversions
• timeit — Measure execution time of small code snippets
• The Python Profilers

References

• Benchmark (computing), Wikipedia.
• Profiling (computer programming), Wikipedia.

Takeaways

You now know how to benchmark Python code using the time.perf_counter() function.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Ather Energy on Unsplash