Last Updated on September 12, 2022
You can join a process by calling the Process.join() function.
In this tutorial you will discover how to join child processes in Python.
Let’s get started.
Need to Join a Process
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. Each Python process has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In concurrent programming, we may need to wait until another process has finished running. This may be for many reasons, such as:
- The current process needs a result from the target process.
- A resource is shared between the processes.
- The current process has no other work to complete.
The join() method provides a way for one process to block until another process has finished.
How can we use the join() method to join a process in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Join a Process
A process can be joined in Python by calling the join() method on the process instance.
For example:
|
1 2 3 |
... # join a process process.join() |
This has the effect of blocking the current process until the target process that has been joined has terminated.
The target process that is being joined may terminate for a number of reasons, such as:
- Finishes executing its target function.
- Finishes executing its run() method if it extends the Process class.
- Raised an error or exception.
Once the target process has finished, the join() method will return and the current process can continue to execute.
The join() method requires that you have a multiprocessing.Process instance for the process you wish to join.
This means that if you created the process, you may need to keep a reference to the multiprocessing.Process instance.
Alternatively, you may use a multiprocessing module function to get an instance for a process, such as multiprocessing.active_children() or multiprocessing.parent_process().
The join() method also takes a “timeout” argument that specifies how long the current process is willing to wait for the target process to terminate, in seconds.
Once the timeout has expired and the target process has not terminated, the join() process will return.
|
1 2 3 |
... # join the process with a timeout process.join(timeout=10) |
When using a timeout, it will not be clear whether the join() method returned because the target process terminated or because of the timeout. Therefore, we can call the Process.is_alive() function to confirm the target process is no longer running.
For example:
|
1 2 3 4 5 6 7 8 |
... # join the process with a timeout process.join(timeout=10) # check if the target process is still running if process.is_alive(): # timeout expired, process is still running else: # process has terminated |
Now that we know how to join a process, let’s look at some worked examples.
Example of Joining a Process
We can explore how to join a target process in Python.
In this example we will create a new multiprocessing.Process instance and then join it from the parent process.
First, we can define a target task function to execute our new process.
The function will first block for a moment by calling the sleep() function, then report a message. The complete function is listed below.
|
1 2 3 4 5 6 |
# target function def task(): # block for a moment sleep(1) # report a message print('All done in the new process') |
In the parent process, we will create a new multiprocessing.Process instance configured to execute our new task() function, then execute the process by calling the start() function.
|
1 2 3 4 5 |
... # create a new process process = Process(target=task) # start the new process process.start() |
Next, we wish for the parent process to wait until the new process has terminated.
This can be achieved by calling the join() function on the multiprocessing.Process instance for the new process.
|
1 2 3 4 |
... # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join() |
This call will block, meaning that the parent process will not carry on executing until this function call returns, and this function call will only return once the process that has been joined terminates.
Once the new process terminates, the join() thread returns and the parent process is free to carry on executing.
|
1 2 3 |
... # continue on print('Main: Continuing on') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of joining a process from time import sleep from multiprocessing import Process # target function def task(): # block for a moment sleep(1) # report a message print('All done in the new process') # entry point if __name__ == '__main__': # create a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join() # continue on print('Main: Continuing on') |
Running the example first creates a new process instance configured to execute our target task() function.
We then start the new process from the parent process, and then wait for the new process to finish by calling the join() function. The parent process does not progress, but instead blocks, waiting.
The new process runs, blocks for a moment, reports a message, then terminates.
Once the new process terminates, the join() function called in the parent process returns, and the parent process is free to continue on.
|
1 2 3 |
Main: Waiting for process to terminate... All done in the new process Main: Continuing on |
Next, let’s look at an example of joining a process that terminates with an error.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Joining a Process That Has An Error
It just so happens that the target process in the previous example terminated normally.
It is also possible for the target process to terminate by raising an error or exception.
Any error or exception raised in another process will not reach the parent process in the parent process, but will terminate the new child process and allow the join() function to return and the current process to continue on.
Let’s demonstrate this with an example.
We can modify the above example so that the task() function raises an error instead of finishing gracefully.
|
1 2 3 4 5 6 7 8 |
# target function def task(): # block for a moment sleep(1) # report a message print('All done in the new process') # terminate with an error raise RuntimeError('Something bad happened') |
Tying this together, the complete example of the parent process joining a target child process that will terminate with an error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of joining a process that raises an error from time import sleep from multiprocessing import Process # target function def task(): # block for a moment sleep(1) # report a message print('All done in the new process') # terminate with an error raise RuntimeError('Something bad happened') # entry point if __name__ == '__main__': # create a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join() # continue on print('Main: Continuing on') |
Running the example first creates and starts the new child process as before.
The parent process joins the child process and waits for it to terminate.
The new process blocks, reports a message then terminates with an error. The error is reported on standard error (stderr), the default behavior.
The new process is terminated and the join() method in the parent process returns and the parent process continues on as before.
|
1 2 3 4 5 6 7 |
Main: Waiting for process to terminate... Process Process-1: Traceback (most recent call last): ... RuntimeError: Something bad happened All done in the new process Main: Continuing on |
Next, let’s look at an example of joining a process with a timeout.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Joining a Process With a Timeout
We can join a process and use a timeout.
A timeout allows the parent process to stop waiting for the target process to timeout after a fixed number of seconds.
We can update the first example so that the target process takes longer to execute, in this case five seconds. The updated task() function is listed below.
|
1 2 3 4 5 6 |
# target function def task(): # block for a moment sleep(5) # report a message print('All done in the new process') |
Next, we can join the new process and set a timeout of two seconds.
|
1 2 3 4 |
... # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join(timeout=2) |
This will mean that the parent process will only block for two seconds waiting for the child process to complete, it will not complete in time and return before the child process has terminated.
The parent process will not know whether the join() function returned because the child process terminated or because of the timeout.
Therefore, we can check if the child process is still running by calling the is_alive() method on the child process.
|
1 2 3 4 5 6 |
... # check if the process is still alive if process.is_alive(): print('Main: The target process is still running') else: print('Main: The target process has terminated') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of joining a process with a timeout from time import sleep from multiprocessing import Process # target function def task(): # block for a moment sleep(5) # report a message print('All done in the new process') # entry point if __name__ == '__main__': # create a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join(timeout=2) # check if the process is still alive if process.is_alive(): print('Main: The target process is still running') else: print('Main: The target process has terminated') |
Running the example creates and starts the new child process.
The parent process then joins the new child process and waits two seconds. The new process continues running and fails to terminate in two seconds.
The join() function then returns after the timeout and then checks on the status of the new process and finds that it is still running.
The parent process then terminates.
The child process continues to execute and then reports its messages before terminating itself.
Note, the process will not terminate until all child processes terminate, not when the main thread of the parent process exits.
|
1 2 3 |
Main: Waiting for process to terminate... Main: The target process is still running All done in the new process |
Try changing the timeout or the length of time the new process sleeps in this example to see different messages printed (e.g. change the timeout to 10).
Next, let’s look at an example of joining a process that is not running.
Example of Joining a Process That is Not Running
It is possible to join a process that is not running.
The effect is that the join() function will not block and instead will return immediately.
We can demonstrate this with a worked example.
The parent process can block for a moment in a way that we know that the new child process has finished executing. In this case, it can sleep for two seconds, whereas we know the new process will be finished after one second.
|
1 2 3 4 |
... # block for a moment print('Main: blocking for a moment...') sleep(2) |
The parent process can then attempt to join the new process as before and wait for it to finish, even though we know it has already terminated at this point.
|
1 2 3 4 5 6 |
... # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join() # continue on print('Main: Continuing on') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of joining a process that is not running from time import sleep from multiprocessing import Process # target function def task(): # block for a moment sleep(1) # report a message print('All done in the new process') # entry point if __name__ == '__main__': # create a new process process = Process(target=task) # start the new process process.start() # block for a moment print('Main: blocking for a moment...') sleep(2) # wait for the new process to finish print('Main: Waiting for process to terminate...') process.join() # continue on print('Main: Continuing on') |
Running the example first creates and starts the new process.
The parent process then blocks for two seconds with a call to time.sleep().
The new process finishes executing after one second and reports its message.
The parent process wakes up then attempts to join the new process that has already terminated. The join() method returns immediately and the parent process carries on as per normal.
|
1 2 3 4 |
Main: blocking for a moment... All done in the new process Main: Waiting for process to terminate... Main: Continuing on |
This highlights that it is safe to call the join() method, even if the target process may have already terminated.
Next, let’s take a look at what happens if we try to join the current process.
Example of Joining the Current Process
It is possible for a process to attempt to join itself.
That is, the parent process can call the join() function on a multiprocessing.Process instance that represents itself.
The effect is that a AssertionError is raised.
If an error was not raised, then the call would result in a deadlock as the process would be blocking and waiting for itself to terminate, which of course, it would never terminate.
We can demonstrate this with an example.
First, we can get a multiprocessing.Process instance for the current process by calling the multiprocessing.current_process() function.
|
1 2 3 |
... # get the current process process = current_process() |
We can then call the join() method on this process instance.
|
1 2 3 4 |
... # wait for the current thread to finish print('Main: Waiting for process to terminate...') process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# SuperFastPython.com # example of joining the current process from multiprocessing import current_process # entry point if __name__ == '__main__': # get the current process process = current_process() # wait for the current thread to finish print('Main: Waiting for process to terminate...') process.join() # continue on print('Main: Continuing on') |
Running the example first gets a multiprocessing.Process instance for the current parent process.
The process then attempts to join the current process (itself).
This fails and raises an error, terminating the process and the program as there are no other processes running.
|
1 2 3 4 |
Main: Waiting for process to terminate... Traceback (most recent call last): ... AssertionError: can only join a child process |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to join a process in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Paul Byrne on Unsplash
Do you have any questions?