Last Updated on September 12, 2022
You can define a new class that extends multiprocessing.Process that automatically starts new processes.
In this tutorial you will discover how to auto-start new processes in Python.
Let’s get started.
Need to Auto-Start a Process
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
A problem with processes is that you must first create an instance of the multiprocessing.Process class, then call the start() function before the process is started.
In many cases, it would be easier if processes could be started immediately once the class was created.
Is there a way to automatically start processes in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Auto-Start a Process
A process can automatically be started by creating a subclass of multiprocessing.Process that calls the start() function in the constructor.
The typical way a process is started involves two steps:
- Create the instance of multiprocessing.Process.
- Call the start() function.
For example:
|
1 2 3 4 5 |
... # create the process instance process = multiprocessing.Process(...) # start the process process.start() |
This adds an additional step if the goal is to create and start the process immediately.
In fact, in most cases that a process is created it is to be started as the next step. Rarely do we need a delay between creating a process instance and starting the process.
We can create and start a process in one step with a compound statement, for example:
|
1 2 3 |
... # create and start a process multiprocessing.Process(...).start() |
The downsides of this approach is that we cannot maintain a reference to the new process instance, and we must combine multiple statements into a single compound statement, which is bad for readability and debugging.
An alternative is to simplify creating and starting a new process in a single step that involves:
- Define a custom class that extends multiprocessing.Process.
- Override the class constructor and call start().
Importantly, when extending the multiprocessing.Process class we must call the parent constructor first to ensure that the process is set up correctly.
We must also pass on all arguments and keyword arguments to the parent constructor so that the new process is set up correctly, e.g. calls the target task function with any arguments.
You can learn more about extending the multiprocessing.Process class in this tutorial:
The example below defines a custom process class that automatically starts the process.
|
1 2 3 4 5 6 7 8 |
# custom process class that auto starts the process class AutoStartProcess(Process): # constructor def __init__(self, *args, **kwargs): # call the the parent constructor super().__init__(*args, **kwargs) # start the process self.start() |
We can then create an instance of this class that will start the process immediately.
For example:
|
1 2 3 |
... # create and start the new process process = AutoStartProcess(...) |
Next, let’s look at using the automatically starting process in a worked example.
Example of Auto-Starting a Process
We can develop an example to demonstrate how to automatically start a new process.
In this example, we will define a new AutoStartProcess class and then use it to execute a custom target function that blocks for a moment and reports a message.
First, we can define the custom class that extends the multiprocessing.Process class, passes along all arguments to the parent constructor, then starts the process.
We will use the AutoStartProcess class defined in the previous section, provided again below.
|
1 2 3 4 5 6 7 8 |
# custom process class that automatically starts processes when they are constructed class AutoStartProcess(Process): # constructor def __init__(self, *args, **kwargs): # call the the parent constructor super().__init__(*args, **kwargs) # start the process self.start() |
Next, we can define a custom task function to execute in a new process.
The function will first print a message to let us know that the process is running, it will then sleep for one second by calling time.sleep() function, then reports a final message before the process terminates.
The task() function below implements this.
|
1 2 3 4 5 6 7 |
# task function def task(): print('Task starting') # block for a moment sleep(1) # report print('Task all done') |
Finally, the parent process will create an instance of the AutoStartProcess class configured to execute our task() function.
Creating a new instance will start the new process immediately.
|
1 2 3 |
... # create and start the new process process = AutoStartProcess(target=task) |
You can learn more about executing functions in a new process in this tutorial:
The main process will then block until the new process terminates. This can be achieved by calling the join() function.
You can learn more about joining processes in this tutorial:
|
1 2 3 |
... # wait for the new process to finish process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of automatically starting a process from time import sleep from multiprocessing import Process # custom process class that automatically starts processes when they are constructed class AutoStartProcess(Process): # constructor def __init__(self, *args, **kwargs): # call the the parent constructor super().__init__(*args, **kwargs) # start the process self.start() # task function def task(): print('Task starting') # block for a moment sleep(1) # report print('Task all done') # entry point if __name__ == '__main__': # create and start the new process process = AutoStartProcess(target=task) # wait for the new process to finish process.join() |
Running the example first creates an instance of the new process class.
The constructor of the new process class calls the parent constructor of the multiprocessing.Process class, which ensures that the new process is set up correctly. The constructor then starts the new process immediately.
The parent process then joins the new child process and waits for it to terminate.
The new process executes our custom task function. A message is printed, the process sleeps for a moment, then a final message is reported before the process terminates.
|
1 2 |
Task starting Task all done |
This demonstrates how to define a new class for auto-starting a new process, and how it can be used to execute task functions.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to auto-start new processes in Python
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by iSAW Company on Unsplash
With Python 3.10, you can do this to solve the problem of how to start a process and have the reference to it.
from multiprocessing import Process as P
(phandle:=P(target=foo)).start()
This will both start the process and assign the instance to phandle. I don’t know if this is a good practice, but I do use it from time to time.
Great tip, thanks!