You can use the pyperf library to benchmark, analyze, and compare the performance of snippets of Python code.

It extends the timeit module’s capabilities and includes the ability to execute benchmarks concurrently and to gather and report summary statistics.

In this tutorial, you will discover how to benchmark Python code using the pyperf open source library.

Let’s get started.

What is pyperf

The pyperf project is a Python library for benchmarking.

It was developed by the Python Software Foundation and is available as open source on GitHub.

The Python pyperf module is a toolkit to write, run and analyze benchmarks.

— pyperf, GitHub Project.

The pyperf wraps the timeit module and offers more features for reliable benchmarking snippets of Python code.

One key addition is the use of worker threads for executing benchmarks concurrently.

pyperf starts by spawning a first worker process (Run 1) only to calibrate the benchmark: compute the number of outer loops: 2^15 loops on the example. Then pyperf spawns 20 worker processes (Run 2 .. Run 21). Each worker starts by running the benchmark once to “warmup” the process, but this result is ignored in the final result. Then each worker runs the benchmark 3 times.

— Run a benchmark, Perf User Guide

It also offers other additional features, such as making available summary statistics on benchmark results, such as the distribution, mean and standard deviation by default.

It also reports warnings if the benchmark duration is too short and in turn likely unstable.

The pyperf project documentation includes a user guide and developer guide:

• Perf User Guide
• Perf Developer Guide

Now that we know about the pyperf project, let’s look at how we can use it.

How to Use pyperf

Using the pyperf library for benchmarking involves three steps, they are:

1. Install pyperf
2. Run benchmarks
3. Analyze results

Let’s take a closer look at each in turn.

1. Install pyperf

The first step is to install the pyperf library.

This can be achieved using your favorite Python package manager, such as pip.

For example:

2. Run benchmarks

The pyperf library provides both a Python API and a command line interface.

Like the timeit module on which it relies, it seems that the pyperf library’s preferred usage is via the command line interface.

We can benchmark snippets of Python code on the command line by executing the pyperf module via the “python -m” flag, then specify the timeit command followed by the code snippet to execute.

For example:

The “pyperf timeit” command takes many of the same arguments as the Python timeit module command.

You can learn more here:

• pyperf timeit

The pyperf timeit command offers a number of benefits over the timeit module in the standard library.

* It displays the average and the standard deviation
* It runs the benchmark in multiple processes
* By default, it skips the first value in each process to warmup the benchmark
* It does not disable the garbage collector

— pyperf commands

We can save benchmark results to a JSON file via the -o flag followed by a filename.

For example:

This is helpful for analysis and comparison in the next section.

3. Analyze and Compare Results

The pyperf library includes a number of tools for analyzing and comparing benchmark results.

One example is the “pyperf stats” command.

It takes the name of a benchmark JSON file and then reports statistical details of the benchmark results.

* Mean and standard deviation.
* Median and median absolute deviation (MAD).
* Percentiles.
* Outliers: number of values out of the range

— pyperf commands

For example:

We can also compare benchmark results using the “pyperf compare_to” command.

This command takes arguments like the way to present the results followed by the filenames that contain the raw benchmark results.

Importantly, it uses statistical significance to report whether the difference between results is more likely meaningful or a statistical fluke.

pyperf determines whether two samples differ significantly using a Student’s two-sample, two-tailed t-test with alpha equals to 0.95.

— pyperf commands

For example:

Now that we know how to use the pyperf library for benchmarking, let’s look at a worked example.

Example of Benchmarking with pyperf

We can use the pyperf command line interface to benchmark and compare different snippets for creating a list of squared integers.

The three methods we will compare are:

1. Use the math.pow() function.
2. Use the ** power operator.
3. Use the * multiplication operator.

We will calculate and review the benchmark results for each method, then finally compare all methods directly.

Benchmark List of Squared Integers With math.pow()

Firstly, we can benchmark the creation of a list of one million squared integers using the math.pow() function.

This requires importing the math module via the -s setup flag.

We will save the results into the “mathpow.json” function.

For example:

Running the example benchmarks this snippet using the default number of loops and repetitions.

In this case, the mean execution time of the snippet is 113 milliseconds with a standard deviation of about 2 milliseconds.

Your results may vary.

We can review the distribution of results using the “pyperf stats” command.

For example:

We can see the details of the distribution of benchmark results, seeing that we gathered a distribution of 60 samples or 3 iterations to run with 20 runs.

Benchmark List of Squared Integers With i**2

We can perform a similar benchmark, in this case using the ** power operator.

Results will be saved to the powerop.json file.

The updated powerop.json command is listed below.

Running the command, we can see that the average execution time for the snippet was about 74.2 milliseconds with a standard deviation of about 1.5 milliseconds.

Next, we can review the statistics of the collected benchmark results using the “pyperf stats” command:

We can see that again 60 samples were collected and we can see the distribution of results.

Benchmark List of Squared Integers With i*i

Finally, we can perform the final benchmark, in this case with the code updated to use the multiplication operator.

The results are saved to the multiplyop.json file.

The updated “pyperf timeit” command is listed below:

Running the command, we can see that the average duration to execute the snippet was about 61.8 milliseconds with a standard deviation of about 1.4 milliseconds.

Next, we can review the statistics of the collected benchmark results using the “pyperf stats” command:

We can see that 60 samples were collected and we can see the distribution and central tendency of the collected measure results.

Comparison of Results

We have now collected benchmark results from 3 approaches that achieve the same result using different techniques.

We can manually gather the results and report them in a table, for example:

manual

We can also compare the results automatically using the “pyperf compare_to” command.

We will configure the command to show the results in a table via the –table flag and then specify the three result files.

For example:

We can see that all 3 results are reported in a table, with one result per column.

We can see that the second two results are compared to the first, showing the speedup factor.

In this case, the multiplication operator was the fastest with a benchmark score that was about 1.83x faster than the math.pow() function.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python Benchmarking, Jason Brownlee (my book!)

Also, the following Python books have chapters on benchmarking that may be helpful:

• Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
• High Performance Python, 2020. (chapter 2)

Guides

• 4 Ways to Benchmark Python Code
• 5 Ways to Measure Execution Time in Python
• Python Benchmark Comparison Metrics

Benchmarking APIs

• time — Time access and conversions
• timeit — Measure execution time of small code snippets
• The Python Profilers

References

• Benchmark (computing), Wikipedia.
• Profiling (computer programming), Wikipedia.

Takeaways

You now know how to benchmark Python code using the pyperf open-source library.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Nicolas Hoizey on Unsplash