Show Progress with the ThreadPool in Python
You can show the progress of tasks in the ThreadPool using a callback function.
In this tutorial, you will discover how to show the progress of tasks in the ThreadPool in Python.
Let's get started.
Need To Show Progress of Tasks in the ThreadPool
The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.
A thread pool object which controls a pool of worker threads to which jobs can be submitted.
-- multiprocessing — Process-based parallelism
The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.
Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.
A ThreadPool can be configured when it is created, which will prepare the new threads.
We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().
When issuing many tasks to the ThreadPool, we may need to keep track of how much work remains to be completed.
How can we show the progress of completed tasks in the ThreadPool?
How to Show Progress in the ThreadPool
We can show the progress of tasks in the ThreadPool using the callback function.
This can be achieved by issuing tasks asynchronously to the ThreadPool, such as via the apply_async() function and specifying a callback function via the "callback" argument.
For example:
...
# issue a task to the thread pool
pool.apply_async(task, callback=progress)Our custom callback function will then be called after each task in the ThreadPool is completed.
We can then perform some action to show the progress of completed tasks in the callback function, such as printing a dot to standard out.
For example:
# progress indicator for tasks in the thread pool
def progress(results):
print('.', end='')
Now that we know how to show the progress of tasks in the ThreadPool in a standard way, let's look at a worked example.
Example of Showing Progress in the ThreadPool
We can explore how to show the progress of tasks issued to the ThreadPool.
In this example, we will define a task that will block for a fraction of a second. We will then issue many of these tasks to the ThreadPool. A callback will be called as each task is finished and will print a message to show the progress of tasks as they are completed.
Firstly, we can define a callback function to call each time a task is completed in the ThreadPool.
The function takes the return value of a custom task function, if present, and prints a single dot to standard out without a newline.
The progress() function below implements this.
# progress indicator for tasks in the thread pool
def progress(results):
print('.', end='')
Next, we can define a custom task function.
The task will first generate a random floating point value between 0 and 1. It will then sleep for that many seconds, which will be less than one second. This sleeping will simulate computational work that takes a variable amount of time.
The task() function below implements this.
# task executed in a worker thread
def task():
# generate a random value
value = random()
# block for a moment
sleep(value)
Next, in the main thread, we can create the ThreadPool.
We will create the ThreadPool with the default configuration and use the context manager interface.
...
# create and configure the thread pool
with ThreadPool() as pool:
# ...
You can learn more about the context manager interface in the tutorial:
We can then issue 20 tasks to the ThreadPool, each calling our custom task() function and using the custom progress() function as a callback to show progress.
We can issue each task one by one using the apply_asyc() function which returns an AsyncResult object.
This can be achieved in a list comprehension, providing a list of AsyncResult objects, one for each task.
...
# issue many tasks asynchronously to the thread pool
results = [pool.apply_async(task, callback=progress) for _ in range(20)]You can learn more about the apply_async() function in the tutorial:
Finally, the main thread will close the ThreadPool and block, waiting for the issued task to complete.
...
# close the pool
pool.close()
# wait for all issued tasks to complete
pool.join()You can learn more about joining the ThreadPool in the tutorial:
After the ThreadPool is closed, a final message can be reported.
...
# report all done
print('\nDone!')Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of showing progress in the thread pool with separate tasks
from time import sleep
from random import random
from multiprocessing.pool import ThreadPool
# progress indicator for tasks in the thread pool
def progress(results):
print('.', end='')
# task executed in a worker thread
def task():
# generate a random value
value = random()
# block for a moment
sleep(value)
# protect the entry point
if __name__ == '__main__':
# create and configure the thread pool
with ThreadPool() as pool:
# issue many tasks asynchronously to the thread pool
results = [pool.apply_async(task, callback=progress) for _ in range(20)]
# close the pool
pool.close()
# wait for all issued tasks to complete
pool.join()
# report all done
print('\nDone!')
Running the example first creates the ThreadPool.
Then, 20 tasks are issued to the pool, one at a time.
The main thread then closes the ThreadPool and blocks waiting for all issued tasks to be completed.
Each task blocks for a fraction of a second and finishes.
As each task is finished, the callback function is called, printing a dot.
The dots accumulate on standard output, showing the progress of all issued tasks.
Once all tasks are finished, the main thread continues and reports a final message.
....................
Done!Takeaways
You now know how to show the progress of tasks in the ThreadPool in Python.
If you enjoyed this tutorial, you will love my book: Python ThreadPool Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.