Last Updated on September 12, 2022

Child worker processes are started automatically after creating an instance of the multiprocessing.Pool class.

In this tutorial you will discover when the child worker processes are created in the multiprocessing pool in Python.

Let’s get started.

Multiprocessing Pool Workers

The multiprocessing pool provides a pool of reusable workers for executing ad hoc tasks with process-based concurrency.

An instance of the multiprocessing.Pool class can be created, specifying the number of workers to create, otherwise a default number of workers will be created to match the number of logical CPUs in the system.

Once created, ad hoc tasks can be issued to the pool for execution via the Pool.apply() function. Multiple tasks may be executed in the book by calling the same function with different arguments via the Pool.map() function.

Tasks may be executed asynchronously in the pool via Pool.apply_async() and Pool.map_async(), allowing the caller to carry on with other tasks.

Once concern with the multiprocessing pool is when exactly are the child worker processes used internally within the pool created.

There are two main options for when child workers could be created:

• Immediately after an instance of the class is created.
• On demand as tasks are issued to the pool.

We may need to know this for many reasons, such as:

• Will there be lag when the first task is issued if worker processes are created on demand.
• Will the pool need to be closed correctly if no tasks have been issued to the pool.

When are child worker processes created in the multiprocessing pool?

When Are Pool Workers Created

Workers in the multiprocessing pool are started immediately after the pool is created.

For example:

Moments after the class is created, it will internally create enough child workers to populate the pool. In this case four.

This also applies when using the pool via the context manager interface.

For example:

Again, moments after the Pool instance is defined, it is populated with child worker processes.

Now that we know when the child worker processes are created in the multiprocessing pool, let’s look at a worked example.

Example of When Workers Are Created in the Pool

We can explore when the child worker processes in the multiprocessing pool are created.

In this example, we will create a multiprocessing pool, wait a moment, then check if the caller process has any running child workers.

If it does have child workers and the number of child processes matches the number of workers configured in the pool, it means that the child workers are created after the class is created.

Otherwise, it means that child worker processes are created on demand.

This is an experimental approach to the question, an alternate approach would be to review the source code of the multiprocessing.pool.Pool class.

Firstly, we can create a multiprocessing pool instance with four child worker processes. Don’t worry if you have more or fewer CPU cores, as we will not be running any tasks.

We will then wait a moment to allow any internal initialization by the pool to be completed.

Next, we can get a list of all active child processes for the current process and report their details.

This can be achieved via the multiprocessing.active_children() function.

This will either be an empty list or a list with 4 items.

Finally, we can close the pool now that we are finished with it.

Tying this together, the complete example is listed below.

Running the example first creates an instance of the multiprocessing pool.

The caller then blocks to allow the pool to start up completely. This is probably not necessarily.

Next, a list of all child processes is retrieved and traversed, reporting the details of each.

In this case, we can see that the list of child processes contains four elements, matching the four child workers configured for the pool.

This confirms that child workers are created after the class is created, and not on demand.

Finally, the multiprocessing pool is closed.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPool: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know when child worker processes are created in the multiprocessing pool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Jeremy Bishop on Unsplash