How to Extend the Process Class in Python

May 5, 2022 Python Multiprocessing

You can extend the multiprocessing.Process class and override the run() function to run code in a new Process.

In this tutorial you will discover how to extend the multiprocessing.Process class.

Let's get started.

Need to Extend the Process Class

A process is a running instance of a computer program.

Every Python program is executed in a Process, which is a new instance of the Python interpreter. Each Python process has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create additional processes in our program in order to execute code concurrently.

Python provides the ability to create and manage new processes via the multiprocessing.Process class.

You can learn more about multiprocessing in the tutorial:

Multiprocessing in Python: The Complete Guide

One way of running code in a new process is to extend the multiprocessing.Process class. This provides flexibility and allows data to be stored as instance variables and accessed by the new process.

It also allows functions associated with the new process to be grouped with the new object.

How can we extend the multiprocessing.Process class?

How to Extend the Process Class

The multiprocessing.Process class can be extended to run code in another process.

This can be achieved by first extending the class, just like any other Python class.

For example:

# custom process class
class CustomProcess(multiprocessing.Process):
    # ...

Then the run() function of the multiprocessing.Process class must be overridden to contain the code that you wish to execute in another process.

For example:

# override the run function
def run(self):
    # ...

And that's it.

Given that it is a custom class, you can define a constructor for the class and use it to pass in data that may be needed in the run() function, stored such as instance variables (attributes).

You can also define additional functions in the class to split up the work you may need to complete in another process.

Finally, attributes can also be used to store the results of any calculation or IO performed in another process that may need to be retrieved afterward.

Next, let's look at a worked example of extending the multiprocessing.Process class.

Example of Extending the Process Class

First, we can define a class that extends the multiprocessing.Process class.

We will name the class something arbitrary such as "CustomProcess".

# custom process class
class CustomProcess(Process):
    # ...

We can then override the run() instance method and define the code that we wish to execute in another process.

In this case, we will block for a moment and then print a message.

# override the run function
def run(self):
    # block for a moment
    sleep(1)
    # display a message
    print('This is coming from another process')

Next, we can create an instance of our CustomProcess class and call the start() function to begin executing our run() function in another process. Internally, the start() function will call the run() function.

The code will then run in a new process as soon as the operating system can schedule it.

...
# create the process
process = CustomProcess()
# start the process
process.start()

Finally, we wait for the new process to finish executing.

...
# wait for the process to finish
print('Waiting for the process to finish')
process.join()

Tying this together, the complete example of executing code in another process by extending the multiprocessing.Process class is listed below.

# SuperFastPython.com
# example of extending the Process class
from time import sleep
from multiprocessing import Process

# custom process class
class CustomProcess(Process):
    # override the run function
    def run(self):
        # block for a moment
        sleep(1)
        # display a message
        print('This is coming from another process')

# entry point
if __name__ == '__main__':
    # create the process
    process = CustomProcess()
    # start the process
    process.start()
    # wait for the process to finish
    print('Waiting for the process to finish')
    process.join()

Running the example first creates an instance of the process, then executes the content of the run() function.

Meanwhile, the main thread waits for the new process to finish its execution, before exiting.

Waiting for the process to finish
This is coming from another process

Example of Passing Arguments When Extending the Process Class

We may need data or variables within the run() function when extending the multiprocessing.Process class.

This data cannot be passed to the run() function via the start() function when starting the process.

Instead, we can pass any data needed by the run() function to the constructor of the extended multiprocessing.Process class and store them as instance variables (properties or attributes) in the class. We can then access these attributes from within the run() function.

Let's make this concrete with a worked example.

We will update the example of extending the multiprocessing.Process class in the previous section to use an argument that specifies how long the process will block. We need to get this value from the main thread into the run() function of the extended multiprocessing.Process class.

Firstly, we can override the constructor of the multiprocessing.Process class and define a new constructor to take any data we require. In this case, the "value" time in seconds that we wish to block.

# override the constructor
def __init__(self, value):
	# ...

Within the constructor, we must first call the parent constructor on the multiprocessing.Process class.

...
# execute the base constructor
Process.__init__(self)

Next, we can store the "value" argument as an attribute on the object instance.

...
# store the value
self.value = value

We can then update the code in the run() function to use the "value" instance attribute.

...
# block for a moment
sleep(self.value)
# display a message
print(f'This is coming from another process: {self.value}')

Tying this together, the updated CustomProcess class that overrides the constructor to get data into the run() function is listed below.

# custom process class
class CustomProcess(Process):
    # override the constructor
    def __init__(self, value):
        # execute the base constructor
        Process.__init__(self)
        # store the value
        self.value = value

    # override the run function
    def run(self):
        # block for a moment
        sleep(self.value)
        # display a message
        print(f'This is coming from another process: {self.value}')

Finally, we can pass a value to the constructor when creating an instance of our CustomProcess class.

...
# create the process
process = CustomProcess(0.5)

The complete example of getting data variables to the run() function when extending the multiprocessing.Process class is listed below.

# SuperFastPython.com
# example of extending the Process class and passing arguments
from time import sleep
from multiprocessing import Process

# custom process class
class CustomProcess(Process):
    # override the constructor
    def __init__(self, value):
        # execute the base constructor
        Process.__init__(self)
        # store the value
        self.value = value

    # override the run function
    def run(self):
        # block for a moment
        sleep(self.value)
        # display a message
        print(f'This is coming from another process: {self.value}')

# entry point
if __name__ == '__main__':
    # create the process
    process = CustomProcess(0.5)
    # start the process
    process.start()
    # wait for the process to finish
    print('Waiting for the process to finish')
    process.join()

Running the example first creates an instance of our extended multiprocessing.Process class and passing data into the constructor. The override constructor first calls the parent constructor, then stores the new data as an instance variable.

Finally the new process is started by calling start() which in turn internally calls the run() function which makes use of the instance variable data.

Waiting for the process to finish
This is coming from another process: 0.5

Takeaways

You now know how to extend the multiprocessing.Process class in Python.

If you enjoyed this tutorial, you will love my book: Python Multiprocessing Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.