Multiprocessing Pool Show Progress in Python
You can show progress of tasks in the multiprocessing pool using a callback function.
In this tutorial you will discover how to show the progress of tasks in the process pool in Python.
Let's get started.
Need To Show Progress of Tasks in the Process Pool
The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.
A process pool can be configured when it is created, which will prepare the child workers.
A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.
-- multiprocessing — Process-based parallelism
We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map().
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().
When issuing many tasks to the process pool, we may need to keep track of how much work remains to be completed.
How can we show progress of completed tasks in the process pool?
How to Show Progress of Tasks
We can show progress of tasks in the process pool using the callback function.
This can be achieved by issuing tasks asynchronously to the process pool, such as via the apply_async() function and specifying a callback function via the "callback" argument.
For example:
...
# issue a task to the process pool
pool.apply_async(task, callback=progress)Our custom callback function will then be called after each task in the process pool is completed.
We can then perform some action to show the progress of completed tasks in the callback function, such as printing a dot to standard out.
For example:
# progress indicator for tasks in the process pool
def progress(results):
print('.', end='', flush=True)
Now that we know how to show the progress of tasks in the process pool in a standard way, let's look at a worked example.
Example of Showing Progress in the Process Pool
We can explore how to show progress of tasks issued to the process pool.
In this example, we will define a task that will block for a fraction of a second. We will then issue many of these tasks to the process pool. A callback will be called as each task finished and will print a message to show the progress of tasks as they are completed.
Firstly, we can define a callback function to call each time a task is completed in the process pool.
The function takes the return value of a custom task function, if present, and prints a single dot to standard out without a newline.
The progress() function below implements this.
# progress indicator for tasks in the process pool
def progress(results):
print('.', end='', flush=True)
Next, we can define a custom task function.
The task will first generate a random floating point value between 0 and 1. It will then sleep for that many seconds, which will be less than one second. This sleeping will simulate computational work that takes a variable amount of time.
The task() function below implements this.
# task executed in a worker process
def task():
# generate a random value
value = random()
# block for a moment
sleep(value)
Next, in the main process we can create the process pool.
We will create the process pool with the default configuration and use the context manager interface.
...
# create and configure the process pool
with Pool() as pool:
# ...
You can learn more about the context manager interface in the tutorial:
We can then issue 20 tasks to the process pool, each calling our custom task() function and using the custom progress() function as a callback to show progress.
We can issue each task one-by-one using the apply_asyc() function which returns an AsyncResult object.
This can be achieved in a list comprehension, providing a list of AsyncResult objects, one for each task.
...
# issue many tasks asynchronously to the process pool
results = [pool.apply_async(task, callback=progress) for _ in range(20)]You can learn more about the apply_async() function in the tutorial:
Finally, the main process will close the process pool and block, waiting for the issued task to complete.
...
# close the pool
pool.close()
# wait for all issued tasks to complete
pool.join()You can learn more about joining the process pool in the tutorial:
After the process pool is closed, a final message can be reported.
...
# report all done
print('\nDone!')Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of showing progress in the process pool with separate tasks
from time import sleep
from random import random
from multiprocessing.pool import Pool
# progress indicator for tasks in the process pool
def progress(results):
print('.', end='', flush=True)
# task executed in a worker process
def task():
# generate a random value
value = random()
# block for a moment
sleep(value)
# protect the entry point
if __name__ == '__main__':
# create and configure the process pool
with Pool() as pool:
# issue many tasks asynchronously to the process pool
results = [pool.apply_async(task, callback=progress) for _ in range(20)]
# close the pool
pool.close()
# wait for all issued tasks to complete
pool.join()
# report all done
print('\nDone!')
Running the example first creates the process pool.
Then, 20 tasks are issued to the pool, one at a time.
The main process then closes the process pool and blocks waiting for all issued tasks to complete.
Each task blocks for a fraction of a second and finishes.
As each task is finished, the callback function is called, printing a dot.
The dots accumulate on standard output, showing the progress of all issued tasks.
Once all tasks are finished, the main process continues on and reports a final message.
....................
Done!Takeaways
You now know how to show the progress of tasks in the process pool in Python.
If you enjoyed this tutorial, you will love my book: Python Multiprocessing Pool Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.