Last Updated on September 12, 2022

You can show progress of tasks in the multiprocessing pool using a callback function.

In this tutorial you will discover how to show the progress of tasks in the process pool in Python.

Let’s get started.

Need To Show Progress of Tasks in the Process Pool

The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.

A process pool can be configured when it is created, which will prepare the child workers.

A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.

— multiprocessing — Process-based parallelism

We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().

When issuing many tasks to the process pool, we may need to keep track of how much work remains to be completed.

How can we show progress of completed tasks in the process pool?

How to Show Progress of Tasks

We can show progress of tasks in the process pool using the callback function.

This can be achieved by issuing tasks asynchronously to the process pool, such as via the apply_async() function and specifying a callback function via the “callback” argument.

For example:

Our custom callback function will then be called after each task in the process pool is completed.

We can then perform some action to show the progress of completed tasks in the callback function, such as printing a dot to standard out.

For example:

Now that we know how to show the progress of tasks in the process pool in a standard way, let’s look at a worked example.

Example of Showing Progress in the Process Pool

We can explore how to show progress of tasks issued to the process pool.

In this example, we will define a task that will block for a fraction of a second. We will then issue many of these tasks to the process pool. A callback will be called as each task finished and will print a message to show the progress of tasks as they are completed.

Firstly, we can define a callback function to call each time a task is completed in the process pool.

The function takes the return value of a custom task function, if present, and prints a single dot to standard out without a newline.

The progress() function below implements this.

Next, we can define a custom task function.

The task will first generate a random floating point value between 0 and 1. It will then sleep for that many seconds, which will be less than one second. This sleeping will simulate computational work that takes a variable amount of time.

The task() function below implements this.

Next, in the main process we can create the process pool.

We will create the process pool with the default configuration and use the context manager interface.

You can learn more about the context manager interface in the tutorial:

• Process Pool Context Manager

We can then issue 20 tasks to the process pool, each calling our custom task() function and using the custom progress() function as a callback to show progress.

We can issue each task one-by-one using the apply_asyc() function which returns an AsyncResult object.

This can be achieved in a list comprehension, providing a list of AsyncResult objects, one for each task.

You can learn more about the apply_async() function in the tutorial:

• Multiprocessing Pool.apply_async() in Python

Finally, the main process will close the process pool and block, waiting for the issued task to complete.

You can learn more about joining the process pool in the tutorial:

• Join a Process Pool in Python

After the process pool is closed, a final message can be reported.

Tying this together, the complete example is listed below.

Running the example first creates the process pool.

Then, 20 tasks are issued to the pool, one at a time.

The main process then closes the process pool and blocks waiting for all issued tasks to complete.

Each task blocks for a fraction of a second and finishes.

As each task is finished, the callback function is called, printing a dot.

The dots accumulate on standard output, showing the progress of all issued tasks.

Once all tasks are finished, the main process continues on and reports a final message.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPool: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know how to show the progress of tasks in the process pool in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Drew Dau on Unsplash