Last Updated on September 29, 2023

You can benchmark the execution of Python code using the “time” module in the standard library.

In this tutorial, you will discover how to time the execution of Python code using a suite of different techniques.

Let’s get started.

Need to Time Python Code for Benchmarking

Benchmarking Python code refers to comparing the performance of one program to variations of the program.

Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.

— Benchmarking, Wikipedia.

Typically, we make changes to the programs, such as adding concurrency, in order to improve the performance the program on a given system.

Improving performance typically means reducing the run time of the program.

Therefore, when we benchmark programs in Python after adding concurrency, we typically are interested in recording how long a program takes to run.

It is critical to be systematic when benchmarking code.

The first step is to record how long an unmodified version of the program takes to run. This provides a baseline in performance to which all other versions of the program must be compared. If we are adding concurrency, then the unmodified version of the program will typically perform tasks sequentially, e.g. one-by-one.

We can then make modifications to the program, such as adding thread pools, process pools, or asyncio. The goal is to perform tasks concurrently (out of order), even in parallel (simultaneously). The performance of the program can be benchmarked and compared to the performance of the unmodified version.

The performance of the modified versions of the program must have better performance than the unmodified version of the program. If they do not, they are not improvements and should not be adopted.

How can we benchmark the performance of programs in Python?

How to Measure Execution Time in Python

There are 5 ways to measure execution time manually in Python using the time module, they are:

1. Use time.time()
2. Use time.perf_counter()
3. Use time.monotonic()
4. Use time.process_time()
5. Use time.thread_time()

Note, each function returns a time in seconds and has an equivalent function that returns the time in nanoseconds, e.g. time.time_ns(), time.perf_counter_ns(), time.monotonic_ns(), time.process_time_ns() and time.thread_time_ns().

Recall that there are 1,000 nanoseconds in one microsecond, 1,000 microseconds in 1 millisecond, and 1,000 milliseconds in one second. This highlights that the nanosecond versions of the function are for measuring very short time scales indeed.

Note, there are automatic ways to measure execution time, such as via the timeit module.

Next, let’s take a closer look at each technique in turn.

Measure Execution Time With time.time()

The time.time() function reports the number of seconds since the epoch.

Return the time in seconds since the epoch as a floating point number.

— time — Time access and conversions

Recall that epoch is January 1st 1970, which is used on Unix systems and beyond as an arbitrary fixed time in the past.

In computing, an epoch is a fixed date and time used as a reference from which a computer measures system time. Most computer systems determine time as a number representing the seconds removed from a particular arbitrary date and time. For instance, Unix and POSIX measure time as the number of seconds that have passed since Thursday 1 January 1970 00:00:00 UT, a point in time known as the Unix epoch.

— Epoch (computing), Wikipedia.

The result is a floating point value, potentially offering fractions of a seconds (e.g. milliseconds), if the platforms support it.

The time.time() function is not perfect.

It is possible for a subsequent call to time.time() to return a value in seconds less than the previous value, due to rounding.

Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.

— time — Time access and conversions

You can learn more about benchmarking Python with the time.time() function in the tutorial:

• Benchmark Python with time.time()

Measure Execution Time With time.perf_counter()

The time.perf_counter() function reports the value of a performance counter on the system.

It does not report the time since epoch like time.time().

Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. It does include time elapsed during sleep and is system-wide.

— time — Time access and conversions

The returned value in seconds with fractional components (e.g. milliseconds and nanoseconds), provides a high-resolution timestamp.

Calculating the difference between two timestamps from the time.perf_counter() allows high-resolution execution time benchmarking, e.g. in the millisecond and nanosecond range.

The timestamp from the time.perf_counter() function is consistent, meaning that two durations can be compared relative to each other in a meaningful way.

The time.perf_counter() function was introduced in Python version 3.3 with the intended use for short-duration benchmarking.

perf_counter(): Performance counter with the highest available resolution to measure a short duration.

— What’s New In Python 3.3

The perf_counter() function was specifically designed to overcome the limitations of other time functions to ensure that the result is consistent across platforms and monotonic (always increasing).

To measure the performance of a function, time.clock() can be used but it is very different on Windows and on Unix. […] The new time.perf_counter() function should be used instead to always get the most precise performance counter with a portable behaviour (ex: include time spend during sleep).

— PEP 418 – Add monotonic time, performance counter, and process time functions

For accuracy, the timeit module uses the time.perf_counter() internally.

The default timer, which is always time.perf_counter().

— timeit — Measure execution time of small code snippets

You can learn more about benchmarking Python with the time.perf_counter() function in the tutorial:

• Benchmark Python with time.perf_counter()

Measure Execution Time With time.monotonic()

The time.monotonic() function returns time stamps from a clock that cannot go backwards, as its name suggests.

In mathematics, monotonic, e.g. a monotonic function means a function whose output over increases (or decreaes).

This means that the result from the time.monotonic() function will never be before the result from a prior call.

Return the value (in fractional seconds) of a monotonic clock, i.e. a clock that cannot go backwards.

— time — Time access and conversions

It is a high-resolution time stamp, although is not relative to epoch-like time.time(). Instead, like time.perf_counter() uses a separate timer separate from the system clock.

The time.monotonic() has a lower resolution than the time.perf_counter() function.

This means that values from the time.monotonic() function can be compared to each other, relatively, but not to the system clock.

The clock is not affected by system clock updates. The reference point of the returned value is undefined, so that only the difference between the results of two calls is valid.

— time — Time access and conversions

Like the time.perf_counter() function, time.monotonic() function is “system-wide”, meaning that it is not affected by changes to the system clock, such as updates or clock adjustments due to time synchronization.

Like the time.perf_counter() function, the time.monotonic() function was introduced in Python version 3.3 with the intent of addressing the limitations of the time.time() function tied to the system clock, such as use in short-duration benchmarking.

monotonic(): Monotonic clock (cannot go backward), not affected by system clock updates.

— What’s New In Python 3.3

You can learn more about benchmarking Python with the time.monotonic() function in the tutorial:

• Benchmark Python with time.monotonic()

Measure Execution Time With time.process_time()

The time.process_time() reports the time that the current process has been executed.

The time begins or is zero when the current process is first created.

Return the value (in fractional seconds) of the sum of the system and user CPU time of the current process.

— time — Time access and conversions

This value is calculated as the sum of the system time and the user time.

• process time = user time + system time

Recall system time is time that the CPU is spent executing system calls for the kernel (e.g. the operating system), whereas user time is time spent by the CPU executing calls in the program (e.g. your code).

When a program loops through an array, it is accumulating user CPU time. Conversely, when a program executes a system call such as exec or fork, it is accumulating system CPU time.

— time (Unix), Wikipedia.

The reported time does not include sleep time.

This means if the process is blocked by a call to time.sleep() or perhaps is suspended by the operating system, then this time is not included in the reported time. This is called a “process-wide” time.

It does not include time elapsed during sleep.

— time — Time access and conversions

As such, it only reports the time that the current process was executed since it was created by the operating system.

You can learn more about benchmarking Python with the time.process_time() function in the tutorial:

• Benchmark Python with time.process_time()

Measure Execution Time With time.thread_time()

The time.thread_time() reports the time that the current thread has been executing.

The time begins or is zero when the current thread is first created.

Return the value (in fractional seconds) of the sum of the system and user CPU time of the current thread.

— time — Time access and conversions

It is an equivalent value to the time.process_time(), except calculated at the scope of the current thread, not the current process.

This value is calculated as the sum of the system time and the user time.

• thread time = user time + system time

The reported time does not include sleep time.

This means if the thread is blocked by a call to time.sleep() or perhaps is suspended by the operating system, then this time is not included in the reported time. This is called a “thread-wide” or “thread-specific” time.

It does not include time elapsed during sleep.

— time — Time access and conversions

Comparison of Time Functions

There are many ways to calculate time in Python and their differences are confusing.

The time.get_clock_info(name) function can be used to report the technical details of each timer.

The program below reports these details.

Running the program reports the following details:

We can collect these details into a readable table, below.

This table may help tease apart their important details.

Scope refers to the point of reference for the time calculation.

Monotonic refers to whether subsequent calls always result in a larger return value or not. We can see that time.time() is not monotonic, because it could be adjusted.

Adjustable refers to the fact that the clock on which the time is based may be modified, e.g. updated. This can affect the relative difference between two times if the time jumps around by a leap second.

The resolution provides an idea of the floating point precision of the output of each function. We can see that perf_counter() and monotonic() are high-precision floating point values, whereas time.process_time() and time.time() are not. These values may differ across different operating systems and hardware systems.

Generally, time.time() should be avoided for benchmarking because it could be adjusted by the system while being used. Nevertheless, it is wild used because it is widely understood. It reports wall clock time.

The time.perf_counter() and time.monotonic() functions are high resolution and are intended for benchmarking. I believe the time.perf_counter() could have higher resolution on some systems (e.g. use a different clock or sampling rate).

The time.process_time() function can be used if only the current process needs to be measured and the time.thread_time() for only the current threads. And these function should only be used if sleeps are to be ignored, which is probably desirable but could be confusing as it may differ from wall clock time.

Now that we are familiar with the different ways to benchmark execution time in Python, let’s look at some worked examples of each.

Example of Timing Code with time.time()

We can explore an example of benchmarking execution time using the time.time() function.

In this case, we will create a list of 100,000,000 squared integers.

This will involve first recording the start time, then executing the target code, then finally recording the end time. The difference between the times is calculated to give the duration, which is then reported.

The complete example is listed below.

Running the program first records the start time, relative to the epoch.

The target code is then executed, creating the list of squared integers.

Next, the end time is recorded, relative to the epoch.

The duration is then calculated as the difference between the two times and then reported in seconds.

Note, your result will differ given differences in software, hardware, and perhaps whatever else is happening on the system at the same time.

Example of Timing Code with time.perf_counter()

We can explore an example of benchmarking execution time using the time.perf_counter() function.

In this case, we will create a list of 100,000,000 squared integers.

This will involve first recording the start time, then executing the target code, then finally recording the end time. The difference between the times is calculated to give the duration, which is then reported.

The complete example is listed below.

Running the program first records the start time, relative to the system.

The target code is then executed, creating the list of squared integers.

Next, the end time is recorded, relative to the system.

The duration is then calculated as the difference between the two times and then reported in seconds.

Note, your result will differ given differences in software, hardware, and perhaps whatever else is happening on the system at the same time.

Example of Timing Code with time.monotonic()

We can explore an example of benchmarking execution time using the time.monotonic() function.

In this case, we will create a list of 100,000,000 squared integers.

This will involve first recording the start time, then executing the target code, then finally recording the end time. The difference between the times is calculated to give the duration, which is then reported.

The complete example is listed below.

Running the program first records the start time, relative to the system.

The target code is then executed, creating the list of squared integers.

Next, the end time is recorded, relative to the system.

The duration is then calculated as the difference between the two times and then reported in seconds.

Note, your result will differ given differences in software, hardware, and perhaps whatever else is happening on the system at the same time.

Example of Timing Code with time.process_time()

We can explore an example of benchmarking execution time using the time.process_time() function.

In this case, we will create a list of 100,000,000 squared integers.

We do not need to record a start and end time as the timer is started when the process is created. Therefore we only need to report the end time before the program exits. Nevertheless, we will use the same systematic approach of recording start and end times and calculating the duration, as we did in the previous examples.

This will involve first recording the start time, then executing the target code, then finally recording the end time. The difference between the times is calculated to give the duration, which is then reported.

The complete example is listed below.

Running the program first records the start time, relative to the creation of the process.

The target code is then executed, creating the list of squared integers.

Next, the end time is recorded, relative to the creation of the process.

The duration is then calculated as the difference between the two times and then reported in seconds.

Note, your result will differ given differences in software, hardware, and perhaps whatever else is happening on the system at the same time.

Example of Timing Code with time.thread_time()

We can explore an example of benchmarking execution time using the time.thread_time() function.

In this case, we will create a list of 100,000,000 squared integers.

We do not need to record a start and end time as the timer is started when the main thread is created. Therefore we only need to report the end time before the program exits. Nevertheless, we will use the same systematic approach of recording start and end times and calculating the duration, as we did in the previous examples.

This will involve first recording the start time, then executing the target code, then finally recording the end time. The difference between the times is calculated to give the duration, which is then reported.

The complete example is listed below.

Running the program first records the start time, relative to the creation of the thread.

The target code is then executed, creating the list of squared integers.

Next, the end time is recorded, relative to the creation of the thread.

The duration is then calculated as the difference between the two times and then reported in seconds.

Note, your result will differ given differences in software, hardware, and perhaps whatever else is happening on the system at the same time.

Example of Comparing All The Time Measures

It may be interesting to compare each approach to timing the code all within the same program.

We can expect tiny differences between each time because the function calls will be sequential.

Nevertheless, we can expect large differences between some of the times because of the different clocks used for timing.

Tying this together, the comparison of all five timing methods is listed below.

Running the example we can see that the wall clock time report by time() is very similar to the time reported by perf_counter() and monotonic().

We can see that the process_time() and thread_time() are both similar and both about 100 milliseconds shorter than the wall clock times.

This may be because of some idle time during the execution of the program that is excluded from the thread and process timers and not from the other timers.

Alternately, it may be an artifact of the lower resolution (sampling tick rate) of the clocks used by the thread and process times compared to the high-performance clock used by perf_counter() and monotonic() and the higher resolution system clock.

I suspect the latter in this case.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python Benchmarking, Jason Brownlee (my book!)

Also, the following Python books have chapters on benchmarking that may be helpful:

• Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
• High Performance Python, 2020. (chapter 2)

Guides

• 4 Ways to Benchmark Python Code
• 5 Ways to Measure Execution Time in Python
• Python Benchmark Comparison Metrics

Benchmarking APIs

• time — Time access and conversions
• timeit — Measure execution time of small code snippets
• The Python Profilers

References

• Benchmark (computing), Wikipedia.
• Profiling (computer programming), Wikipedia.

Takeaways

You now know how to time the execution of Python code using a suite of different techniques.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Ash Hayes on Unsplash