You can automatically benchmark target functions as parts of unit testing via the by pytest-benchmark library.
The pytest-benchmark library is a plugin for the pytest Python unit test library, providing a fixture for benchmarking that can be used within arbitrary unit tests.
In this tutorial, you will discover how to benchmark in unit tests using the pytest-benchmark library.
Let’s get started.
What is pytest-benchmark
The pytest-benchmark project is a Python library for benchmarking in unit tests.
It was developed by Ionel Cristian Mărieș and is available as open source on GitHub.
py.test fixture for benchmarking code
— pytest-benchmark, GitHub.
The pytest-benchmark library is a plugin for the pytest library.
Recall that the pytest is an open-source library for unit testing in Python.
The pytest framework makes it easy to write small, readable tests, and can scale to support complex functional testing for applications and libraries.
— pytest: helps you write better programs
The pytest library supports plugins that offer additional capabilities while developing unit tests and the pytest-benchmark is one such plugin. It provides a “benchmark” fixture that can be used to automatically benchmark target functions as part of a unit test.
This plugin provides a benchmark fixture. This fixture is a callable object that will benchmark any function passed to it.
— pytest-benchmark, GitHub.
Recall that in unit testing, a fixture is a context, state, or capability needed during a test.
In testing, a fixture provides a defined, reliable and consistent context for the tests. This could include environment (for example a database configured with known parameters) or content (such as a dataset). Fixtures define the steps and data that constitute the arrange phase of a test (see Anatomy of a test). In pytest, they are functions you define that serve this purpose. They can also be used to define a test’s act phase; this is a powerful technique for designing more complex tests.
— About fixtures, pytest.
Now that we know about the pytest-benchmark project, let’s look at how we can use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use pytest-benchmark
Using the pytest-benchmark library for benchmarking involves three steps, they are:
- Install pytest-benchmark
- Use benchmark in a unit test.
- Run unit tests.
Let’s take a closer look at each in turn.
1. Install pytest-benchmark
The first step is to install the pytest-benchmark library.
This can be achieved using your favorite Python package manager, such as pip.
For example:
|
1 |
pip install pytest-benchmark |
This may install additional dependencies, such as the pytest library itself to which the pytest-benchmark library is a plugin.
2. Use benchmark fixture
Once installed, the benchmark fixture can be used in unit tests.
This is achieved by defining a test that takes the benchmark fixture as an argument. The fixture will then be provided to the test when the test is run by pytest.
|
1 2 3 |
# custom unit test def test_mystuff(benchmark): # ... |
Within the test, the benchmark fixture can be used to execute a target function.
This involves calling the benchmark() fixture and passing it the name of the target function and any arguments to the function.
For example:
|
1 2 3 |
# custom unit test def test_mystuff(benchmark): result = benchmark(mystuff, arg1, arg2) |
3. Run Unit Tests
Once unit tests have been defined to make use of the fixture, the unit tests can be executed normally using the pytest command.
For example:
|
1 |
pytest mytests.py |
The unit tests will run normally.
As a part of the test output, the benchmark results will be provided for the benchmarked target functions.
There is more to the library and ways to configure the benchmark that is performed. You can learn more in the pytest-benchmark library documentation here:
Now that we know how to use the pytest-benchmark library, let’s look at some worked examples.
Example of Unit Testing a Target Function
Before we explore how to benchmark in unit tests, let’s first develop a simple unit test for pytest.
In this case, we will define a target function that creates and returns a list of squared integers of a given size.
|
1 2 3 |
# create a list of squared integers def create_list(size): return [i**2 for i in range(size)] |
We can then define a unit test for this function that calls the function and confirms that the list has the expected size, matching the provided argument.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# unit test for create_list() def test_create_list(): n = 100000000 data = create_list(n) assert len(data) == n Tying this together, the complete unit test is listed below. # SuperFastPython.com # example of unit testing a target function # create a list of squared integers def create_list(size): return [i**2 for i in range(size)] # unit test for create_list() def test_create_list(): n = 100000000 data = create_list(n) assert len(data) == n |
We can then execute the test using the pytest program on the command line.
|
1 |
pytest all_tests.py |
This executes the test_create_list test for our create_list function.
As expected, the test passes normally.
|
1 2 3 4 5 6 7 8 9 |
platform darwin -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: ... plugins: benchmark-4.0.0 collected 1 item all_tests.py . [100%] ======================================= 1 passed in 7.44s ======================================= |
Next, let’s explore how we might update the unit test to perform a benchmark.
Free Python Benchmarking Course
Get FREE access to my 7-day email course on Python Benchmarking.
Discover benchmarking with the time.perf_counter() function, how to develop a benchmarking helper function and context manager and how to use the timeit API and command line.
Example of Benchmark Unit Testing a Target Function
We can explore how to update our unit test to automatically benchmark the target function.
This involves first changing the test_create_list() unit test to take the “benchmark” fixture as an argument.
The test_create_list() can then be updated to call the create_list() function via the “benchmark” fixture.
The updated test_create_list() unit test with these changes is listed below.
|
1 2 3 4 5 |
# unit test for create_list() def test_create_list(benchmark): n = 100000000 data = benchmark(create_list, n) assert len(data) == n |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# SuperFastPython.com # example of a benchmark unit test with pytest-benchmark # create a list of squared integers def create_list(size): return [i**2 for i in range(size)] # unit test for create_list() def test_create_list(benchmark): n = 100000000 data = benchmark(create_list, n) assert len(data) == n |
We can then execute the unit test as before using the pytest program
|
1 |
pytest all_tests.py |
Running the unit tests executes the test as before, which passes as we expect.
The target function is then benchmarked and the results are provided as summary statistics that include the summary of the distribution and the number of rounds and iterations of the benchmark test.
This highlights how we can automatically benchmark target functions as part of unit testing.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
platform darwin -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0 benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: ... plugins: benchmark-4.0.0 collected 1 item all_tests.py . [100%] ------------------------------------------- benchmark: 1 tests ------------------------------------------ Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations --------------------------------------------------------------------------------------------------------- test_create_list 6.9455 7.0223 6.9717 0.0331 6.9562 0.0501 1;0 0.1434 5 1 --------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean ====================================== 1 passed in 50.17s ======================================= |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python Benchmarking, Jason Brownlee (my book!)
Also, the following Python books have chapters on benchmarking that may be helpful:
- Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
- High Performance Python, 2020. (chapter 2)
Guides
- 4 Ways to Benchmark Python Code
- 5 Ways to Measure Execution Time in Python
- Python Benchmark Comparison Metrics
Benchmarking APIs
- time — Time access and conversions
- timeit — Measure execution time of small code snippets
- The Python Profilers
References
Takeaways
You now know how to benchmark in unit tests using the pytest-benchmark library.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Aleks Marinkovic on Unsplash
Do you have any questions?