Last Updated on September 12, 2022
You can run a function in a new child Process via the “target” argument on the multiprocessing.Process class.
In this tutorial you will discover how to run a function in a new process.
Let’s get started.
Need to Run A Function in a New Process
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. Each Python process has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
One way of running a function in a new process is via an argument on the multiprocessing.Process class.
How can we run a function in a new process using the multiprocessing.Process class?
Run loops using all CPUs, download your FREE book to learn how.
How to Run a Function In a Process
To run a function in another process:
- Create an instance of the multiprocessing.Process class.
- Specify the name of the function via the “target” argument.
- Call the start() function.
First, we must create a new instance of the multiprocessing.Process class and specify the function we wish to execute in a new process via the “target” argument.
|
1 2 3 |
... # create a process process = multiprocessing.Process(target=task) |
The function executed in another process may have arguments in which case they can be specified as a tuple and passed to the “args” argument of the multiprocessing.Process class constructor or as a dictionary to the “kwargs” argument.
|
1 2 3 |
... # create a process process = multiprocessing.Process(target=task, args=(arg1, arg2)) |
We can then start executing the process by calling the start() function.
The start() function will return immediately and the operating system will execute the function in a separate process as soon as it is able.
|
1 2 3 |
... # run the new process process.start() |
A new instance of the Python interrupter will be created and a new thread within the new process will be created to execute our target function.
And that’s all there is to it.
We do not have control over when the process will execute precisely or which CPU core will execute it. Both of these are low-level responsibilities that are handled by the underlying operating system.
Next, let’s look at a worked example of executing a function in a new process.
Example of Running a Function in a Process
First, we can define a custom function that will be executed in another process.
We will define a simple function that blocks for a moment then prints a statement.
The function can have any name we like, in this case we’ll name it “task“.
|
1 2 3 4 5 6 |
# a custom function that blocks for a moment def task(): # block for a moment sleep(1) # display a message print('This is from another process') |
Next, we can create an instance of the multiprocessing.Process class and specify our function name as the “target” argument in the constructor.
|
1 2 3 |
... # create a process process = Process(target=task) |
Once created we can run the process which will execute our custom function in a new native process, as soon as the operating system can.
|
1 2 3 |
... # run the process process.start() |
The start() function does not block, meaning it returns immediately.
We can explicitly wait for the new process to finish executing by calling the join() function.
|
1 2 3 4 |
... # wait for the process to finish print('Waiting for the process...') process.join() |
Tying this together, the complete example of executing a function in another process is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of running a function in another process from time import sleep from multiprocessing import Process # a custom function that blocks for a moment def task(): # block for a moment sleep(1) # display a message print('This is from another process') # entry point if __name__ == '__main__': # create a process process = Process(target=task) # run the process process.start() # wait for the process to finish print('Waiting for the process...') process.join() |
Running the example first creates the multiprocessing.Process then calls the start() function. This does not start the process immediately, but instead allows the operating system to schedule the function to execute as soon as possible.
At some point a new instance of the Python interpreter is created that has a new thread which will execute our target function.
The main thread of our initial process then prints a message waiting for the new process to complete, then calls the join() function to explicitly block and wait for the new process to terminate.
Once the custom function returns, the new process is closed. The join() function then returns and the main thread exits.
|
1 2 |
Waiting for the process... This is from another process |
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Running a Function in a Process With Arguments
We can execute functions in another process that takes arguments.
This can be demonstrated by first updating our task() function from the previous section to take two arguments, one for the time in seconds to block and the second for a message to display.
|
1 2 3 4 5 6 |
# a custom function that blocks for a moment def task(sleep_time, message): # block for a moment sleep(sleep_time) # display a message print(message) |
Next, we can update the call to the multiprocessing.Process constructor to specify the two arguments in the order that our task() function expects them as a tuple via the “args” argument.
|
1 2 3 |
... # create a process process = Process(target=task, args=(1.5, 'New message from another process')) |
Tying this together, the complete example of executing a custom function that takes arguments in a separate process is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of running a function with arguments in another process from time import sleep from multiprocessing import Process # a custom function that blocks for a moment def task(sleep_time, message): # block for a moment sleep(sleep_time) # display a message print(message) # entry point if __name__ == '__main__': # create a process process = Process(target=task, args=(1.5, 'New message from another process')) # run the process process.start() # wait for the process to finish print('Waiting for the process...') process.join() |
Running the example creates the process specifying the function name and the arguments to the function.
The new process is started and the function blocks for the parameterized number of seconds and prints the parameterized message.
|
1 2 |
Waiting for the process... New message from another process |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to execute functions in another process via the “target” argument on the multiprocessing.Process class.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Ivan Ragozin on Unsplash
Hi,
I have implemented the same at my end.. However I noticed that when a child process is taking longer than expected to complete, join() is not being invoked. The parent process keeps waiting. Whereas when the child process takes less time then it is being invoked.What is the issue and how can we solve this?
How do you know that the join() is not being made?
Once the child process is started from the main process, the join() method is called immediately. It will return if the child process has already been terminated, otherwise block until it is terminated.
You can learn more about join() here:
https://superfastpython.com/join-a-process-in-python/