You can use the timeit module to perform microbenchmarking in Python.

The timeit module supports both a Python API and a command line interface and encapsulates best practices when benchmarking such as using a high-precision timer and repeating a benchmark many times.

In this tutorial, you will discover how to perform microbenchmarking in Python.

Let’s get started.

What is Microbenchmarking

Microbenchmarking is a specialized form of benchmarking focused on evaluating the performance of very small and specific code snippets or functions within a program.

Unlike traditional benchmarking, which assesses the overall performance of an entire application or system, microbenchmarking zooms in on isolated, fine-grained code segments.

As such, it is sometimes called a component benchmark or a snippet benchmark.

Key characteristics of microbenchmarking include:

1. Small Code Samples: Microbenchmarks typically examine short and self-contained code segments, often limited to a single function or operation.
2. Precise Measurement: The goal is to obtain highly precise performance measurements, often at the microsecond or nanosecond level, to uncover subtle performance differences.
3. Isolation: Microbenchmarks are carefully isolated from external factors that might influence the results, such as I/O operations, network access, or complex dependencies.
4. Focused Analysis: They provide insights into low-level performance details, like a function call overhead, loop iteration times, or arithmetic operations.
5. Repeated Runs: Microbenchmarks are typically run multiple times to reduce the impact of variability in performance measurements.
6. Tooling: Specialized tools and libraries designed for microbenchmarking are used to ensure accurate measurements and reduce sources of error.

Microbenchmarking is particularly useful in scenarios where absolute performance and optimization at the smallest scale are critical, such as when developing low-level libraries, evaluating the impact of compiler optimizations, or comparing the efficiency of different algorithms for specific operations.

It’s a valuable technique for developers aiming to fine-tune code for maximum efficiency in situations where even minor performance gains matter.

History of Microbenchmarking

Microbenchmarking has a relatively recent history in the realm of software development and performance analysis.

It emerged as a response to the growing need for fine-grained performance measurement and optimization in computing applications. Here’s a brief history of microbenchmarking:

In the early days of software development, performance optimization focused on the macro scale. Developers were primarily concerned with optimizing entire applications, and detailed performance analysis of smaller code segments was often overlooked. However, as technology advanced and the demand for high-performance computing increased, the need for more granular performance analysis became evident.

The term “microbenchmarking” gained recognition in the early 21st century, coinciding with the development of specialized tools and libraries for conducting fine-grained performance tests. These tools allowed developers to isolate and measure the performance of small, specific code segments within their applications. Microbenchmarking became especially important in fields where milliseconds or even nanoseconds mattered, such as real-time systems, high-frequency trading, and scientific computing.

One notable development in microbenchmarking was the introduction of benchmarking libraries for various programming languages. These libraries offered standardized and repeatable methods for measuring the performance of code segments, enabling developers to share and compare their results with a broader community. These tools often included features for statistical analysis, allowing developers to assess the stability and reliability of their benchmarking results.

In recent years, the use of microbenchmarking has expanded beyond low-level programming tasks. It has become an essential tool for assessing the efficiency of common programming constructs, algorithms, and data structures. Microbenchmarking is now standard practice for evaluating the performance of arithmetic operations, string manipulation, data structure access, and more.

Microbenchmarking also plays a crucial role in the development and optimization of programming languages themselves. Language designers and compiler developers use microbenchmarking to evaluate the impact of language features and compiler optimizations on code performance. This helps refine language specifications and identify areas for improvement.

Today, microbenchmarking remains a critical component of software development, contributing to the creation of faster, more efficient applications and the continuous advancement of programming languages and compilers. As computing technology continues to evolve, the role of microbenchmarking in fine-tuning code for optimal performance will likely become even more prominent.

Cases For Microbenchmarking

Microbenchmarking involves assessing the performance of very specific, fine-grained code segments.

Below are some examples of areas where microbenchmarking can be applied:

1. Arithmetic Operations: Evaluating the speed of basic arithmetic operations like addition, multiplication, or division to determine which is the most efficient for a given context.
2. String Manipulation: Comparing the performance of different string manipulation operations, such as concatenation, substring extraction, or searching.
3. Data Structure Operations: Assessing the efficiency of common data structure operations, like appending to a list, adding elements to a dictionary, or accessing elements in an array.
4. Function Call Overhead: Measuring the overhead associated with function calls, especially for small functions, to understand the cost of abstraction.
5. Loop Iterations: Analyzing the time it takes to execute loops with various numbers of iterations to identify the impact of loop size on performance.
6. Regular Expressions: Testing the performance of regular expression patterns against different input strings to optimize text processing tasks.
7. Bitwise Operations: Assessing the speed of bitwise operations, such as bitwise AND, OR, XOR, or bit shifting, for operations like data compression or encryption.
8. Memory Allocation: Evaluating the time taken for memory allocation and deallocation, comparing techniques like stack allocation and heap allocation.
9. File I/O: Measuring file read and write performance for small or specific file sizes to optimize data access operations.
10. Mathematical Functions: Benchmarking mathematical functions like square roots, trigonometric operations, and exponentiation to identify the fastest implementations.

Microbenchmarking is valuable in cases where even minor performance differences can have a significant impact, such as in high-frequency trading applications, embedded systems, or real-time data processing.

It helps developers fine-tune code to achieve optimal performance for specific, critical operations.

Now that we know about microbenchmarking, let’s look at how we can use it in Python

Microbenchmarking in Python

We can use microbenchmarking in Python via the timeit module.

In fact, the timeit module was specifically designed to provide the ability to benchmark small snippets of Python code, e.g. for microbenchmarking before the term was widely used.

This module provides a simple way to time small bits of Python code.

— timeit — Measure execution time of small code snippets

The timeit module provides two interfaces for microbenchmarking, they are

1. The timeit API interface.
2. The timeit Command-line interface.

The first is an API that can be used via the timeit.Timer object or timeit.timeit() and timeit.repeat() module functions.

For example, we can use the timeit.timeit() function and pass it a string of Python code to execute and microbenchmark. It will then return a duration in seconds.

For example:

You can learn more about how to use the timeit Python API in the tutorial:

• Benchmark Python with timeit.timeit()

The second is a command line interface.

The timeit module can be run directly from the command line via the -m flag to the Python interpreter.

For example:

This will then perform a microbenchmark of the snippet of code and report the result.

You can learn more about how to use the timeit command line interface in the tutorial:

• Benchmark timeit Command Line Interface

Now that we know how to perform microbenchmarking in Python, let’s look at some worked examples.

Example of Microbenchmark Creating a List

We can explore how to microbenchmark different ways of creating a list of 1,000 integers.

Some approaches include:

1. Create a list from a range
2. Use a list comprehension.
3. Append in a for loop
4. Adding items in a for-loop

The first approach is to create a list from a range.

With results:

We can try a list comprehension.

With results:

We can try appending a list in a for loop.

With results:

We can also try list concatenation in a for loop.

With results:

Let’s compare the results in a table.

Recall that “usec” is microseconds and that there are 1,000 microseconds in one millisecond and 1,000 milliseconds in one second.

Generally, we discovered that we should create a list from a range directly as this is the fastest approach.

We found that appending to a list or concatenating lists is very slow and should be avoided if possible.

Example of Microbenchmark String Concatenation

We can explore how to microbenchmark different ways of concatenating or joining a list into a string.

Note, some of these examples are inspired by an example in the timeit module’s documentation.

Some approaches include:

1. String concatenation
2. Join using a generator.
3. Join using a list comprehension.
4. Join using map().

We can start with string concatenation.

With results:

Next, we can try a generator directly in the join() function.

With results:

Next, we can try a list comprehension in the join() function.

With results:

Finally, we can try the map() build-in function to apply the str() function to each value in the range.

With results:

Let’s compare the results in a table.

Recall that “usec” is microseconds and that there are 1,000 microseconds in one millisecond and 1,000 milliseconds in one second.

Generally, we can see that string concatenation is the slowest, as we might have suspected.

We found that it is slightly faster to use a list comprehension than using the map() built-in function.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python Benchmarking, Jason Brownlee (my book!)

Also, the following Python books have chapters on benchmarking that may be helpful:

• Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
• High Performance Python, 2020. (chapter 2)

Guides

• 4 Ways to Benchmark Python Code
• 5 Ways to Measure Execution Time in Python
• Python Benchmark Comparison Metrics

Benchmarking APIs

• time — Time access and conversions
• timeit — Measure execution time of small code snippets
• The Python Profilers

References

• Benchmark (computing), Wikipedia.
• Profiling (computer programming), Wikipedia.

Takeaways

You now know how to perform microbenchmarking in Python.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any additional tips?
I’d love to hear about them!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Foad Memariaan on Unsplash