Process Utility Functions in Python

May 12, 2022 Python Multiprocessing

You can access the process in a Python program using multiprocessing module functions.

In this tutorial you will discover how to access and query processes in Python using multiprocessing utility functions.

Let's get started.

Need Thread Utility Functions

A process is a running instance of a computer program.

Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create new child processes in our program in order to execute code concurrently.

Python provides the ability to create and manage new processes via the multiprocessing.Process class.

You can learn more about multiprocessing in the tutorial:

Multiprocessing in Python: The Complete Guide

In some applications we may have many processes and may need to access details about the number of processes that are running or access properties of multiprocessing.Process instances for specific threads.

How can we get access to the processes in a Python program?

How to Use Process Utility Functions

There are a number of utilities we can use when working with Python processes.

These utilities are provided as multiprocessing module functions.

Examples of popular multiprocessing module utility functions are as follows:

active_children(): Returns a list of all active child processes for the current process.
cpu_count(): Return the number of logical CPU cores in the system.
current_process(): Return the Process instance for the current process.
parent_process(): Return the Process instance for the parent of the current process.

Some additional multiprocessing module functions that are less commonly used include the following:

freeze_support(): Add support for multiprocessing for a frozen Python executable on windows.
get_all_start_methods(): Return a list of process start methods supported on the system.
get_context(): Return a process context object.
get_start_method(): Return the currently configured process start method.
set_executable(): Set the path to the Python executable for child processes.
set_start_method(): Set the start method used for new processes.

Now that we are familiar with the multiprocessing module functions, let's look at some worked examples.

How to Get Active Child Processes

We can get a list of all active child processes for a parent process.

This can be achieved via the multiprocessing.active_children() function that returns a list of all child processes that are currently running.

...
# get a list of all active child processes
children = active_children()

We can demonstrate this with a short example.

# SuperFastPython.com
# list all active child processes
from multiprocessing import active_children
# get a list of all active child processes
children = active_children()
# report a count of active children
print(f'Active Children Count: {len(children)}')
# report each in turn
for child in children:
    print(child)

Running the example reports the number of active child processes.

In this case, there are no active children processes

Active Children Count: 0

We can update the example so that we first start a number of children processes, have the children processes block for a moment, then get the list of active children.

This can be achieved by first defining a function to execute in a new process that blocks for a moment.

The task() function below implements this.

# function to execute in a new process
def task():
    # block for a moment
    sleep(1)

Next, we can create a number of processes configured to run our task() function, then start them.

...
# create a number of child processes
processes = [Process(target=task) for _ in range(5)]
# start the child processes
for process in processes:
    process.start()

We can then report the active child processes as before.

...
# get a list of all active child processes
children = active_children()
# report a count of active children
print(f'Active Children Count: {len(children)}')
# report each in turn
for child in children:
    print(child)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# list all active child processes
from time import sleep
from multiprocessing import active_children
from multiprocessing import Process

# function to execute in a new process
def task():
    # block for a moment
    sleep(1)

# entry point
if __name__ == '__main__':
    # create a number of child processes
    processes = [Process(target=task) for _ in range(5)]
    # start the child processes
    for process in processes:
        process.start()
    # get a list of all active child processes
    children = active_children()
    # report a count of active children
    print(f'Active Children Count: {len(children)}')
    # report each in turn
    for child in children:
        print(child)

Running the example first creates five processes to run our task() function, then starts them.

A list of all active child processes is then retrieved.

The count is reported, which is shown as five, as we expect, then the details of each process are then reported.

This highlights how we can access all active child processes from a parent process.

Active Children Count: 5
<Process name='Process-3' pid=16853 parent=16849 started>
<Process name='Process-4' pid=16854 parent=16849 started>
<Process name='Process-2' pid=16852 parent=16849 started>
<Process name='Process-1' pid=16851 parent=16849 started>
<Process name='Process-5' pid=16855 parent=16849 started>

How to Get The CPU Count

We may want to know the number of CPU cores available.

This can be determined via the multiprocessing.cpu_count() function.

The function returns an integer that indicates the number of logical CPU cores available in the system running the Python program.

Recall that a central processing unit or CPU executes program instructions. A CPU may have one or more physical CPU cores for executing code in parallel. Modern CPU cores may make use of hyperthreading, allowing each physical CPU core to operate like two (or more) logical CPU cores.

Therefore, a computer system with a CPU with four physical cores may report eight logical CPU cores via the multiprocessing.cpu_count() function function.

It may be useful to know the number of CPU cores available to configure a thread pool or the number of processes to create to execute tasks in parallel.

The example below demonstrates how to use the number of CPU cores.

# SuperFastPython.com
# example of reporting the number of logical cpu cores
from multiprocessing import cpu_count
# get the number of cpu cores
num_cores = cpu_count()
# report details
print(num_cores)

Running the example reports the number of logical CPU cores available in the system.

In this case, my system provides eight logical CPU cores, your result may differ.

How to Get The Current Process

We can get a multiprocessing.Process instance for the process running the current code.

This can be achieved via the multiprocessing.current_process() function that returns a multiprocessing.Process instance.

...
# get the current process
thread = current_process()

We can use this function to access the multiprocessing.Process for the MainProcess.

This can be demonstrated with a short example, listed below.

# SuperFastPython.com
# example of executing the current process
from multiprocessing import current_process
# get the current process
process = current_process()
# report details
print(process)

Running the example gets the process instance for the currently running process.

The details are then reported, showing that we have accessed the main process that has no parent process.

<_MainProcess name='MainProcess' parent=None started>

We can use this function within a child process, allowing the child process to access a process instance for itself.

First, we can define a custom function that gets the current process and reports its details.

The task() function below implements this.

# function executed in a new process
def task():
    # get the current process
    process = current_process()
    # report details
    print(process)

Next, we can create a new process to execute our custom task() function. Then start the process and wait for it to terminate.

...
# create a new process
process = Process(target=task)
# start the new process
process.start()
# wait for the child process to terminate
process.join()

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of executing the current process within a child process
from multiprocessing import Process
from multiprocessing import current_process

# function executed in a new process
def task():
    # get the current process
    process = current_process()
    # report details
    print(process)

# entry point
if __name__ == '__main__':
    # create a new process
    process = Process(target=task)
    # start the new process
    process.start()
    # wait for the child process to terminate
    process.join()

Running the example first creates a new process instance, then starts the process and waits for it to finish.

The new process then gets an instance of its own multiprocessing.Process instance and then reports the details.

<Process name='Process-1' parent=16873 started>

How to Get The Parent Process

We may need to access the parent process for a current child process.

This can be achieved via the multiprocessing.parent_process() function. This will return a multiprocessing.Process instance for the parent of the current process.

The MainProcess does not have a parent, therefore attempting to get the parent of the MainProcess will return None.

We can demonstrate this with a worked example.

# SuperFastPython.com
# example of getting the parent process of the main process
from multiprocessing import parent_process
# get the the parent process
process = parent_process()
# report details
print(process)

Running the example attempts the multiprocessing.Process instance for the MainProcess.

The function returns None, as expected as the MainProcess does not have a parent process.

None

We can make the example more interesting by creating a new child process, then getting the parent of the child process, which will be the MainProcess.

This can be achieved by first defining a function to get the parent process and then report its details.

The task() function below implements this.

# function to execute in a new process
def task():
    # get the the parent process
    process = parent_process()
    # report details
    print(process)

Next, we can create a new process instance and configure it to execute our task() function. We can then start the new process and wait for it to terminate.

...
# create a new process
process = Process(target=task)
# start the new process
process.start()
# wait for the new process to terminate
process.join()

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of getting the parent process of a child process
from multiprocessing import parent_process
from multiprocessing import Process

# function to execute in a new process
def task():
    # get the the parent process
    process = parent_process()
    # report details
    print(process)

# entry point
if __name__ == '__main__':
    # create a new process
    process = Process(target=task)
    # start the new process
    process.start()
    # wait for the new process to terminate
    process.join()

Running the example first creates a new process configured to execute our task function, then starts the process and waits for it to terminate.

The new child process then gets a multiprocessing.Process instance for the parent process, then reports its details.

We can see that the parent of the new process is the MainProcess, as we expected.

<_ParentProcess name='MainProcess' parent=None unknown>

Takeaways

You now know how to use process utility functions in Python.

If you enjoyed this tutorial, you will love my book: Python Multiprocessing Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.