Last Updated on September 12, 2022

You can get the details of the thread used to initialize child worker processes and to execute tasks in the process pool.

Additionally, the process pool uses three helper threads internally to help with the management of the process pool. We can get the details of these threads in the main process that creates the process pool.

In this tutorial you will discover how to get the details of threads used by the process pool in Python.

Let’s get started.

Need Details of Threads in the Process Pool

The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.

A process pool can be configured when it is created, which will prepare the child workers.

A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.

— multiprocessing — Process-based parallelism

We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().

When using the process pool, we may need the names of the thread used to initialize and execute tasks in worker processes.

This may be for many reasons, although it is likely to aid in designing the thread-safe handling of data or program state in worker processes or to help with debugging.

How can we get the child worker process names in Python?

How to Get Threads Used in Process Pool

We can get the current thread via the threading.current_thread() function.

This will return a threading.Thread instance that can be used to get the details of the current thread in situations such as:

1. A worker process in the process pool executing a task.
2. Initializing a worker process in the process pool.

For example, we may have questions such as:

1. Do worker processes execute tasks in the main thread or in a worker thread?
2. Are worker processes initialized using the main thread or in a worker thread?

You can learn more about utility functions for getting threads in the tutorial:

• How to Use Thread Utility Functions in Python

Additionally, the multiprocessing.pool.Pool itself uses worker threads. We can get the details of these threads, to help differentiate them from the threads in the main process.

Let’s take a closer look at each one of these aspects in turn.

Thread Used to Execute Task

We can get the details of the current thread executing a task in the process pool.

This can be achieved by calling the threading.current_thread() function to get the threading.Thread instance for the thread executing the task in the child worker process.

For example, we can call this function in a custom function executed by the process pool:

Thread Used to Initialize Worker

We can get the details of the current thread initializing a task in the process pool.

This can be achieved by calling the threading.current_thread() function to get the threading.Thread instance for the thread executing the function used to initialize in the child worker process.

For example, we can call this function in a custom function executed used to initialize workers in the process pool:

We can then configure the process pool to call the custom function when initializing worker processes.

You can learn more about how to initialize worker processes in the process pool in the tutorial:

• Process Pool Initializer in Python

Threads Used to Manage Process Pool

The multiprocessing.pool.Pool class uses worker threads internally to help manage the process pool.

Looking at the source code for the process pool class, we can see that at least three threads are created with responsibilities such as:

1. Handling the creation of child worker processes.
2. Handling the dispatching of tasks to worker processes.
3. Handing the task results from worker processes.

In the main process that creates the process pool, we can get a list of all running threads and inspect the details of the threads associated with the process pool.

We can call the threading.enumerate() function to get a list of all actively running threads. This will return a list of threading.Thread instances, one for each running thread.

For example:

We can get this list of threads after the process pool has been created, and ideally after tasks have already been issued so that all workers and maintenance threads have been started.

Now that we know how to get the details of threads associated with the process pool, let’s look at some worked examples.

Example of Getting Thread Used to Execute Task

We can explore how to get the details of the thread used to execute tasks in the process pool.

In this example we will define a custom function to be executed in the process pool. The task will get the current thread and report its details. The main process will create the process pool and issue the task into the process pool.

Firstly, we can define a custom function to execute as a task in the process pool.

THe function will take an argument and then get the threading.Thread instance for the current thread and report the thread name. We could report other details of the thread as well.

The task() function below implements this.

You can learn more about the details of a thread in the tutorial:

• How to Query Thread Attributes in Python

Next, in the main process we will create the process pool with the default configuration.

We will use the context manager interface to ensure that the process pool is closed automatically once we are finished with it.

You can learn more about the context manager interface in the tutorial:

• Process Pool Context Manager

We will issue a single call to the custom function as a task in the process pool.

This can be achieved via the apply_async() function that will issue the task asynchronously and return an AsyncResult as a handle on the task.

You can learn more about issuing tasks via the apply_async() function in the tutorial:

• Multiprocessing Pool.apply_async() in Python

The main process will then wait for the task to finish via the returned AsyncResult object.

Tying this together, the complete example is listed below.

Running the example first creates the process pool.

The main process then issues a task to the process pool then blocks until the task completes.

The custom function runs in the process pool. It gets the current thread and reports its name.

In this case, we can see, as we might expect, that the main thread of the worker process is responsible for executing the custom function in the child worker process.

You can learn more about the main thread in the tutorial:

• Main Thread of Process in Python

Next, let’s explore how we might check which thread is used to initialize the child worker processes.

Example of Getting Thread Used to Initialize Worker

We can explore how to get the details of the thread used to initialize the child worker processes.

In this example we will define a custom function to initialize the worker processes. The initialization function will get the current thread and report its details. To ensure that the worker processes are initialized, we will issue one task into the process pool that will just block for a fraction of a second.

Firstly, we can define the custom function that will initialize the child worker processes.

The function does not take any arguments or return any values. It will get the threading.Thread instance for the current thread, then report its name.

The init_worker() function below implements this.

Next, we can define a custom function to execute a task in the process pool.

The task will take an argument and block for a fraction of a second.

The task() function below implements this.

Finally, in the main process, we can create the process pool with the default configuration using the context manager interface.

We can issue the task using the apply_async() function, get an AsyncResult in return immediately, then wait on the object for the task to complete.

Tying this together, the complete example is listed below.

Running the example first creates the process pool.

The main process then issues the task into the process pool and waits for it to complete.

The process pool is created and starts a default number of worker processes, eight in this case, although the number of processes will differ depending on the number of CPU cores in your system.

Each worker process is then initialized using the custom initialization function that gets the current thread and reports its name.

In this case, we can see that each worker process is initialized using the main thread of the child worker process.

This is not surprising.

Next, let’s investigate the worker threads created to help manage the process pool.

Example of Getting Threads Used To Manage Pool

We can explore the threads created automatically by the process pool.

Looking at the source code for the multiprocessing.pool.Pool class, we can see that helper threads are created by the process pool.

A brief review of the code suggests that the roles of these helper functions might be to:

1. Help to create worker child processes.
2. Help with dispatching work to worker processes.
3. Help handling results sent back from worker processes.

In this example we will create a process pool and issue tasks and wait for them to complete. We will then get a list of all active threads in the main process and report their details.

Firstly, we can define a custom function to execute in the process pool.

The task does not need to do anything other than help start-up the process pool completely.

In this case, the task will take an argument and block for a fraction of a second.

The task() function below implements this.

Next in the main process, we will create the process pool using the default configuration. We will use the context manager interface to ensure the pool is closed automatically once we are finished with it.

Next, we will issue 10 tasks to the process pool, calling our custom task() function with different integer values from 0 to 9.

This can be achieved with the map_async() function that will issue the tasks asynchronously and return immediately with an AsyncResult object. We can then wait on the object until the issued tasks are completed.

You can learn more about the map_async() function in the tutorial:

• Multiprocessing Pool.map_async() in Python

Next, we can get a list of all active threads in the main process and report the name of each.

Tying this together, the complete example is listed below.

Running the example first creates the process pool.

The 10 tasks are then issued to the process pool and the main process blocks until they are complete.

Each task runs in the process pool, blocks for a moment, then returns.

The main process continues on and gets a list of all active threads and reports their names.

We can see that in addition to the main threads, there are three additional threads. These are the helper threads used internally by the process pool.

Unfortunately, they have ambiguous names, e.g. “Thread-1” to “Thread-3“. Hopefully they are renamed in a future version of Python to indicate their association with the process pool.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPool: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know how to get the details of threads used by the process pool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by miguel sinisterra on Unsplash