Last Updated on September 12, 2022
You can get the name of a worker process in the process pool by calling multiprocessing.current_process() and then accessing the “name” attribute. Worker processes are named for the start method used, and assigned unique numbers.
In this tutorial you will discover how to get the name of worker processes in the Python process pool.
Let’s get started.
Need Names of Worker Processes
The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.
A process pool can be configured when it is created, which will prepare the child workers.
A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.
— multiprocessing — Process-based parallelism
We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map().
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().
When using the process pool, we may need the names of the worker processes.
This may be for many reasons, such as:
- To uniquely identify the worker in the application.
- To include the process name in logging.
- To debug which worker is completing which task.
How can we get the child worker process names in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Get Worker Process Names
When working with a process pool, there are two situations where we may want to get the process name:
- Get the name for each worker process in the process pool.
- Get the name for the worker completing a given task.
Let’s take a closer look at each approach in turn.
Before we get the name of worker processes, let’s take a brief look at how to get a process name in general.
How To Get Process Name
We can get the name of a process via the “name” property on the multiprocessing.Process class.
For example:
|
1 2 3 |
... # report the name of the process print(process.name) |
We may acquire an instance of the multiprocessing.Process for the current process via the multiprocessing.current_process() function.
For example:
|
1 2 3 4 5 |
... # get the current process instance process = current_process() # report the name of the process print(process.name) |
You can learn more about getting process names in the tutorial:
How to Get All Worker Process Names
We can get the name for all child workers in the process pool.
This can be achieved from the parent process.
First, we can get a list of all active child processes via the multiprocessing.active_children() function. This will return a list of multiprocessing.Process instances.
|
1 2 3 |
... # get all active child processes children = multiprocessing.active_children() |
We can then access the “name” attribute of each.
|
1 2 3 4 |
... # get the name of each child process for child in children: print(child.name) |
The downside of this approach is that the list of active children may include child processes that are not worker processes.
Another approach to getting the process name for all worker processes is to configure the process pool to use an initializer function and to get the worker process name. This function will be called by each child worker process once when it is started in the process pool.
The initialization function does not take any arguments. Within the function we can get and use the process via the multiprocessing.current_process() function, then get the worker name.
For example:
|
1 2 3 4 5 6 |
# initialize the worker process def init_worker(): # get the current worker process process = current_process() # get the name for the current worker process name = process.name |
We can then configure the process pool to call the initialization function as each worker is created.
This can be achieved by setting the “initializer” argument in the multiprocessing.pool.Pool constructor to the name of the initialization function.
For example:
|
1 2 3 |
... # create and configure a process pool pool = multiprocessing.pool.Pool(=initializerinit_worker) |
You can learn more about how to initialize worker processes in the process pool in the tutorial:
How to Get Task Worker Process Name
We can get the worker name within a given task executed by the process pool.
This can be achieved by calling the multiprocessing.current_process() function within the target task function executed in the process pool to get the process.
We can then access the “name” attribute directly.
For example:
|
1 2 3 4 5 6 |
# task executed in a worker process def task(identifier): # get the current worker process process = current_process() # get the name for the current worker process name = process.name |
Now that we know how to get the worker process name, let’s look at some worked examples.
Example of Getting All Worker Process Names
We can explore how to get all worker names using a worker initialization function.
In this example we will define a worker initialization function that will get and report the name of each worker process. We will then create and configure a process pool to use the initialization function, then issue many tasks to the process pool in order that all worker processes are started and their names are reported.
First, we must define a worker process initialization function. The function will not take any arguments and will call the multiprocessing.current_process() function to get the current process, then access the “name” attribute for the worker process and report its value.
The init_worker() function below implements this.
|
1 2 3 4 5 6 |
# initialize the worker process def init_worker(): # get the current worker process process = current_process() # report the current worker process name print(f'Worker Name: {process.name}', flush=True) |
Next, we can define a task that we will execute in the process pool.
The task will take an integer augment then block for a fraction of a second.
The task() function below implements this.
|
1 2 3 4 |
# task executed in a worker process def task(identifier): # block for a moment sleep(0.5) |
Next, in the main process, we will configure the program to use the ‘spawn‘ start method. This will influence the names of the worker processes that are created in the process pool.
|
1 2 3 |
... # set the start method to spawn set_start_method('spawn') |
You can learn more about start methods in the tutorial:
Next, we can create and configure a new process pool.
We will use the context manager interface to ensure the process pool is closed automatically once we are finished with it.
|
1 2 3 4 |
... # create and configure the process pool with Pool(initializer=init_worker) as pool: # ... |
You can learn more about the context manager interface in the tutorial:
We will then issue 10 tasks to the process pool, each calling our task() function with an integer between 0 and 9. We will issue the task asynchronously using the map_async() function.
|
1 2 3 |
... # issues tasks to process pool result = pool.map_async(task, range(10)) |
Once issued, the main process will block on the AsyncResult returned from the map_async() function until the issued tasks are complete.
|
1 2 3 |
... # wait for tasks to complete result.wait() |
You can learn more about the map_async() function in the tutorial:
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of getting child worker process names from time import sleep from multiprocessing import set_start_method from multiprocessing import current_process from multiprocessing.pool import Pool # initialize the worker process def init_worker(): # get the current worker process process = current_process() # report the current worker process name print(f'Worker Name: {process.name}', flush=True) # task executed in a worker process def task(identifier): # block for a moment sleep(0.5) # protect the entry point if __name__ == '__main__': # set the start method to spawn set_start_method('spawn') # create and configure the process pool with Pool(initializer=init_worker) as pool: # issues tasks to process pool result = pool.map_async(task, range(10)) # wait for tasks to complete result.wait() # process pool is closed automatically |
Running the example first configures and creates the process pool.
The ten tasks are then issued to the process pool and the main process blocks.
Each worker process is initialized with a call to the init_worker() function. This gets the worker process instance then reports the process name.
In this case, we can see that each worker has a name with the form “SpawnPoolWorker-%d“, where “%d” is an integer from 1 to 8 for the worker process number. You may have more or fewer worker processes depending on the number of logical CPUs in your system.
We can see that the worker process names are named after the start method used to create them, “spawn” in this case.
Each task is then executed, blocking for a fraction of a second and then returning.
All tasks complete and the main process continues on automatically closing the process pool and then the application itself.
|
1 2 3 4 5 6 7 8 |
Worker Name: SpawnPoolWorker-1 Worker Name: SpawnPoolWorker-2 Worker Name: SpawnPoolWorker-4 Worker Name: SpawnPoolWorker-3 Worker Name: SpawnPoolWorker-5 Worker Name: SpawnPoolWorker-6 Worker Name: SpawnPoolWorker-8 Worker Name: SpawnPoolWorker-7 |
We can change the start method to ‘fork‘ and this will influence the names assigned to worker processes created in the process pool.
|
1 2 3 |
... # set the start method to fork set_start_method('fork') |
Note, the ‘fork‘ start method is not available on all platforms. This example may not work on Windows.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of getting child worker process names from time import sleep from multiprocessing import set_start_method from multiprocessing import current_process from multiprocessing.pool import Pool # initialize the worker process def init_worker(): # get the current worker process process = current_process() # report the current worker process name print(f'Worker Name: {process.name}', flush=True) # task executed in a worker process def task(identifier): # block for a moment sleep(0.5) # protect the entry point if __name__ == '__main__': # set the start method to fork set_start_method('fork') # create and configure the process pool with Pool(initializer=init_worker) as pool: # issues tasks to process pool result = pool.map_async(task, range(10)) # wait for tasks to complete result.wait() # process pool is closed automatically |
Running the example creates the process pool, issues tasks and the main process blocks.
Each worker process is initialized with a call to the init_worker() function. This gets the worker process instance then reports the process name.
In this case, we can see that each worker has a name with the form “ForkPoolWorker-%d“, where “%d” is an integer from 1 to 8 for the worker process number. Again, you may have more or fewer worker processes depending on the number of logical CPUs in your system.
We can see that the choice of start method has an important effect on the name of worker processes in the process pool.
|
1 2 3 4 5 6 7 8 |
Worker Name: ForkPoolWorker-1 Worker Name: ForkPoolWorker-2 Worker Name: ForkPoolWorker-3 Worker Name: ForkPoolWorker-4 Worker Name: ForkPoolWorker-5 Worker Name: ForkPoolWorker-6 Worker Name: ForkPoolWorker-7 Worker Name: ForkPoolWorker-8 |
Next, let’s explore getting the worker names from within a task.
Free Python Multiprocessing Pool Course
Download your FREE Process Pool PDF cheat sheet and get BONUS access to my free 7-day crash course on the Process Pool API.
Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.
Example of Getting Work Process Name in Task
We can get the worker name within a task executed in the process pool.
In this example we will get the name of the current process in the custom task function executed by the process pool. We will then issue a single task to the process pool that reports the name.
Firstly, we can define the custom task function that gets the name for the current worker process then reports the value.
The task() function below implements this.
|
1 2 3 4 5 6 |
# task executed in a worker process def task(identifier): # get the current worker process process = current_process() # report the current worker process name print(f'Worker Name: {process.name}', flush=True) |
Next, in the main process we can create the process pool with a default configuration using the context manager interface.
|
1 2 3 4 |
... # create and configure the process pool with Pool() as pool: # ... |
Next, we can issue a single task asynchronously to the process pool using the apply_async() function on the process pool.
|
1 2 3 |
... # issues tasks to process pool result = pool.apply_async(task, (0,)) |
This will return a single AsyncResult object that we can wait on for the issued task to complete.
|
1 2 3 |
... # wait for tasks to complete result.wait() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of getting child worker process name from time import sleep from multiprocessing import set_start_method from multiprocessing import current_process from multiprocessing.pool import Pool # task executed in a worker process def task(identifier): # get the current worker process process = current_process() # report the current worker process name print(f'Worker Name: {process.name}', flush=True) # protect the entry point if __name__ == '__main__': # set the start method to spawn set_start_method('spawn') # create and configure the process pool with Pool() as pool: # issues tasks to process pool result = pool.apply_async(task, (0,)) # wait for tasks to complete result.wait() # process pool is closed automatically |
Running the example first creates the process pool.
A single task is issued to the process pool and the main process blocks until the task is complete.
The task runs, first getting the name of the current worker process that is running the task, then reporting the value.
In this case, we can see that the worker was named after the start method used, e.g. ‘spawn’. We can also see that the third worker (e.g. “SpawnPoolWorker-3“) executed the task.
Note, the worker process chosen to execute the task may be different each time the program is run.
The task completes, then the main process continues on automatically closing the process pool then terminating the application.
|
1 |
Worker Name: SpawnPoolWorker-3 |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Pool Class API Cheat Sheet
I would also recommend specific chapters from these books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPool: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to get the name of worker processes in the Python process pool.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Greg Jeanneau 🗾 on Unsplash
Do you have any questions?