Last Updated on September 12, 2022
You can get the PID of workers in the process pool using os.getpid() from within the task or multiprocessing.active_children() from the parent process.
In this tutorial you will discover how to get the PID of process pool child worker processes in Python.
Let’s get started.
Need Worker Process PID
The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.
A process pool can be configured when it is created, which will prepare the child workers.
A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.
— multiprocessing — Process-based parallelism
We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map(). Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().
Each Python process is created and managed by the underlying operating system.
As such, each process is allocated a unique process identifier, referred to as a PID.
How can we get the PID for worker processes in the process pool?
Run loops using all CPUs, download your FREE book to learn how.
How to Get Worker Process PID
We can get the worker process PIDs in two ways, they are:
- From within the worker process itself.
- From the parent process that created the process pool.
Let’s take a closer look at each approach.
How to Get Worker PID Within Worker
We can get the PID of worker processes from within the workers themselves.
The task that is executed by a worker process can get its own PID.
This can be achieved by calling the os.getpid() function.
For example:
|
1 2 3 |
... # get the pid pid = getpid() |
Alternately, the task executed by the worker process can get the current process via the multiprocessing.current_process() function, then access the pid attribute.
For example:
|
1 2 3 4 5 |
... # get the current process process = current_process() # get the pid pid = process.pid |
Next, let’s look at how we might get the worker PID from the parent process.
How to Get Worker PID From Parent Process
The parent process that created the process pool can get the child worker PIDs.
This can be achieved by getting all active child processes via the multiprocessing.active_children() function and then accessing their pid attributes.
For example
|
1 2 3 4 5 6 |
... # get the child processes children = active_children() # report the pid for child in children: print(f'Worker pid={child.pid}', flush=True) |
Alternatively, we can access the child processes directly on the pool via the _pool attribute on the multiprocessing.pool.Pool instance.
Note, the _pool attribute is a hidden member internal to the Pool class and may change in the future.
For example:
|
1 2 3 4 |
... # report the pids for worker in pool._pool: print(f'Worker pid={worker.pid}', flush=True) |
Now that we know how to access the PIDs of child worker processes in the process pool, let’s look at some worked examples.
Examples of Getting Worker PID From Within Task
We can get the PIDs of child worker processes within the process pool from within the tasks executed in the pool.
There are two main approaches to access the PID of a worker from within the task, they are:
- Using os.getpid()
- Using multiprocessing.Process.pid
Let’s take a closer look at each.
Get Worker PID using os.getpid()
We can access the PID of the worker child process that is executing a task via the os.getpid() function.
This function can be called directly from within the task executed within the process pool, which will return the PID for the child process that is currently executing the task.
The task() below implements this.
|
1 2 3 4 5 6 |
# task executed in a worker process def task(): # get the pid pid = getpid() # report the pid print(f'Worker pid={pid}', flush=True) |
We can then create a process pool with the default configuration using the context manager interface.
|
1 2 3 4 |
... # create and configure the process pool with Pool() as pool: # ... |
We can then issue a task that executes our custom task() function, then close the pool and wait for all issued tasks to complete.
|
1 2 3 4 5 6 7 |
... # issue tasks to the process pool pool.apply_async(task) # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of getting the worker process pid from within the worker from multiprocessing.pool import Pool from os import getpid # task executed in a worker process def task(): # get the pid pid = getpid() # report the pid print(f'Worker pid={pid}', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Running the example first creates and configures the process pool.
The task is then issued to the process pool and the main process closes the pool and waits for all issued tasks to complete.
The custom function executes as a task within the process pool. The task gets the PID of the process that is executing the task, and reports it.
Note, the PID will be different each time the code example is executed.
|
1 |
Worker pid=62808 |
Get Worker PID using multiprocessing.Process.pid
We can get the PID of the worker process by first getting the current process, then reporting the “pid” attribute.
We can update the example from the previous example to do this within the task() function.
First, we can call the multiprocessing.current_process() function to get the multiprocessing.Process instance for the child worker process that is currently executing the task.
|
1 2 3 |
... # get the current process process = current_process() |
We can then access the “pid” attribute from the multiprocessing.Process instance.
|
1 2 3 |
... # get the pid pid = process.pid |
The updated version of the task() function with these changes is listed below.
|
1 2 3 4 5 6 7 8 |
# task executed in a worker process def task(): # get the current process process = current_process() # get the pid pid = process.pid # report the pid print(f'Worker pid={pid}', flush=True) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of getting the worker process pid from within the worker from multiprocessing.pool import Pool from multiprocessing import current_process # task executed in a worker process def task(): # get the current process process = current_process() # get the pid pid = process.pid # report the pid print(f'Worker pid={pid}', flush=True) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Running the example first creates and configures the process pool.
The task is then issued to the process pool and the main process closes the pool and waits for all issued tasks to complete.
The custom function executes as a task within the process pool. The task gets the current process for the child worker process that is executing the task. It then gets the PID of the process that is executing the task, and reports it.
Note, the PID will be different each time the code example is executed.
|
1 |
Worker pid=62822 |
Next, let’s look at how we can get the PID of worker processes from the parent process.
Free Python Multiprocessing Pool Course
Download your FREE Process Pool PDF cheat sheet and get BONUS access to my free 7-day crash course on the Process Pool API.
Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.
Example of Getting Worker Process PID From Parent
We can get the PIDs of child worker processes from the parent process that created and manages the process pool.
There are two main approaches that we can do this, they are:
- Using multiprocessing.active_children()
- Using multiprocessing.pool.Pool._pool
Let’s take a closer look at each approach.
Get Worker PID using multiprocessing.active_children
The parent process can access the PID of child worker processes by first getting a list of all active child processes via the multiprocessing.active_children().
|
1 2 3 |
... # get the child processes children = active_children() |
It can then access the “pid” attribute on each multiprocessing.Process instance.
This can be done after issuing the tasks in the process pool.
|
1 2 3 4 |
... # report the pid for child in children: print(f'Worker pid={child.pid}', flush=True) |
The downside of this approach is that if the parent process has other child processes running at the same time, then we do not have an easy way of determining which processes belong to the process pool and which do not.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of getting the worker process pid from the parent process from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # block for a moment sleep(1) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # get the child processes children = active_children() # report the pid for child in children: print(f'Worker pid={child.pid}', flush=True) # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Running the example first creates and configures the process pool.
The task is then issued to the process pool.
The custom function executes as a task within the process pool and blocks for a second.
The main process then gets a list of all active child processes, then reports the PID of each child process.
The main process then closes the pool and waits for all issued tasks to complete.
In this case, we can see that a total of 8 child worker processes were created.
Note, you have a different number of worker processes, depending on your system.
Note, the PIDs will be different each time the code example is executed.
|
1 2 3 4 5 6 7 8 |
Worker pid=62856 Worker pid=62861 Worker pid=62857 Worker pid=62855 Worker pid=62860 Worker pid=62859 Worker pid=62858 Worker pid=62862 |
Get Worker PID using multiprocessing.pool.Pool._pool
The parent process can access the multiprocessing.Process instances in the process pool directly via the _pool attribute.
This is a private attribute used internally within the pool. As such, the name of this attribute may change in the future and relying on it in an application could be risky.
The parent process can access the list of worker processes directly via this attribute then report their “pid” attributes.
|
1 2 3 4 |
... # report the pids for worker in pool._pool: print(f'Worker pid={worker.pid}', flush=True) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of getting the worker process pid from the parent process from time import sleep from multiprocessing.pool import Pool from multiprocessing import active_children # task executed in a worker process def task(): # block for a moment sleep(1) # protect the entry point if __name__ == '__main__': # create and configure the process pool with Pool() as pool: # issue tasks to the process pool pool.apply_async(task) # report the pids for worker in pool._pool: print(f'Worker pid={worker.pid}', flush=True) # close the process pool pool.close() # wait for all tasks to complete pool.join() |
Running the example first creates and configures the process pool.
The task is then issued to the process pool.
The custom function executes as a task within the process pool and blocks for a second.
The main process then directly accesses the worker processes and reports the PID of each child process.
The main process then closes the pool and waits for all issued tasks to complete.
In this case, we can see that a total of 8 child worker processes were created.
Note, you have a different number of worker processes, depending on your system.
Note, the PIDs will be different each time the code example is executed.
|
1 2 3 4 5 6 7 8 |
Worker pid=62874 Worker pid=62875 Worker pid=62876 Worker pid=62877 Worker pid=62878 Worker pid=62879 Worker pid=62880 Worker pid=62881 |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Pool Class API Cheat Sheet
I would also recommend specific chapters from these books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPool: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to get the PID of process pool child worker processes.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Brett Jordan on Unsplash
Do you have any questions?