Last Updated on September 12, 2022

You can adopt one of the common usage patterns to get the most out of the multiprocessing.Pool in Python.

In this tutorial, you will discover the common usage patterns for Python multiprocessing pools.

Let’s get started.

Multiprocessing Pool Usage Patterns

The multiprocessing.Pool provides a lot of flexibility for executing concurrent tasks in Python.

Nevertheless, there are a handful of common usage patterns that will fit most program scenarios.

This section lists the common usage patterns with worked examples that you can copy and paste into your own project and adapt as needed.

The patterns we will look at are as follows:

1. Pattern 1: map() and Iterate Results Pattern
2. Pattern 2: apply_async() and Forget Pattern
3. Pattern 3: map_async() and Forget Pattern
4. Pattern 4: imap_unordered() and Use as Completed Pattern
5. Pattern 5: imap_unordered() and Wait for First Pattern

We will use a contrived task in each example that will sleep for a random amount of time equal to less than one second. You can easily replace this example task with your own task in each pattern.

Let’s start with the first usage pattern.

Pattern 1: map() and Iterate Results Pattern

This pattern involves calling the same function with different arguments then iterating over the results.

It is a concurrent and parallel version of the built-in map() function with the main difference that all function calls are issued to the process pool immediately and we cannot process results until all tasks are completed.

It requires that we call the map() function with our target function and an iterable of arguments and process return values from each function call in a for loop.

You can learn more about how to use the map() function on the process pool in the tutorial:

• Multiprocessing Pool.map() in Python

This pattern can be used for target functions that take multiple arguments by changing the map() function for the starmap() function.

You can learn more about the starmap() function in the tutorial:

• Multiprocessing Pool.starmap() in Python

Tying this together, the complete example is listed below.

Running the example, we can see that the map() function is called the task() function for each argument in the range 0 to 9.

Watching the example run, we can see that all tasks are issued to the process pool, complete, then once all results are available will the main process iterate over the return values.

Pattern 2: apply_async() and Forget Pattern

This pattern involves issuing one task to the process pool and then not waiting for the result. Fire and forget.

This is a helpful approach for issuing ad hoc tasks asynchronously to the process pool, allowing the main process to continue on with other aspects of the program.

This can be achieved by calling the apply_async() function with the name of the target function and any arguments the target function may take.

The apply_async() function will return an AsyncResult object that can be ignored.

For example:

You can learn more about the apply_async() function in the tutorial:

• Multiprocessing Pool.apply_async() in Python

Once all ad hoc tasks have been issued, we may want to wait for the tasks to complete before closing the process pool.

This can be achieved by calling the close() function on the pool to prevent it from receiving any further tasks, then joining the pool to wait for the issued tasks to complete.

You can learn more about joining the process pool in the tutorial:

• Join a Multiprocessing Pool in Python

Tying this together, the complete example is listed below.

Running the example fires a task into the process pool and forgets about it, allowing it to complete in the background.

The task is issued and the main process is free to continue on with other parts of the program.

In this simple example, there is nothing else to go on with, so the main process then closes the pool and waits for all ad hoc fire-and-forget tasks to complete before terminating.

Pattern 3: map_async() and Forget Pattern

This pattern involves issuing many tasks to the process pool and then moving on. Fire-and-forget for multiple tasks.

This is helpful for applying the same function to each item in an iterable and then not being concerned with the result or return values.

The tasks are issued asynchronously, allowing the caller to continue on with other parts of the program.

This can be achieved with the map_async() function that takes the name of the target task and an iterable of arguments for each function call.

The function returns an AsyncResult object that provides a handle on the issued tasks, that can be ignored in this case.

For example:

You can learn more about the map_async() function in the tutorial:

• Multiprocessing Pool.map_async() in Python

Once all asynchronous tasks have been issued and there is nothing else in the program to do, we can close the process pool and wait for all issued tasks to complete.

Tying this together, the complete example is listed below.

Running the example issues ten tasks to the process pool.

The call returns immediately and the tasks are executed asynchronously. This allows the main process to continue on with other parts of the program.

There is nothing else to do in this simple example, so the process pool is then closed and the main process blocks, waiting for the issued tasks to complete.

Pattern 4: imap_unordered() and Use as Completed Pattern

This pattern is about issuing tasks to the pool and using results for tasks as they become available.

This means that results are received out of order, if tasks take a variable amount of time, rather than in the order that the tasks were issued to the process pool.

This can be achieved with the imap_unordered() function. It takes a function and an iterable of arguments, just like the map() function.

It returns an iterable that yields return values from the target function as the tasks are completed.

We can call the imap_unordered() function and iterate the return values directly in a for-loop.

For example:

You can learn more about the imap_unordered() function in the tutorial:

• Multiprocessing Pool.imap_unordered() in Python

Tying this together, the complete example is listed below.

Running the example issues all tasks to the pool, then receives and processes results in the order that tasks are completed, not the order that tasks were issued to the pool, e.g. unordered.

Pattern 5: imap_unordered() and Wait for First Pattern

This pattern involves issuing many tasks to the process pool asynchronously, then waiting for the first result or first task to finish.

It is a helpful pattern when there may be multiple ways of getting a result but only a single or the first result is required, after which, all other tasks become irrelevant.

This can be achieved by the imap_unordered() function that, like the map() function, takes the name of a target function and an iterable of arguments.

It returns an iterable that yields return values in the order that tasks complete.

This iterable can then be traversed once manually via the next() built-in function which will return only once the first task to finish returns.

For example:

The result can then be processed and the process pool can be terminated, forcing any remaining tasks to stop immediately. This happens automatically via the context manager interface.

TYing this together, the complete example is listed below.

Running the example first issues all of the tasks asynchronously.

The result from the first task to complete is then requested, which blocks until a result is available.

One task completes, returns a value, which is then processed, then the process pool and all remaining tasks are terminated automatically.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPool: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know the common usage patterns for the multiprocessing.Pool in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Faiz Prasla on Unsplash