Last Updated on September 12, 2022
You can fix a RuntimeError when starting child processes by adding if name == ‘main’ to checking for the top-level environment in your code.
In this tutorial you will discover the RuntimeError common on Windows and MacOS when creating child processes and how to fix it.
Let’s get started.
RuntimeError When Starting a Child Process
It is common to get a RuntimeError when starting a new Process in Python.
The content of the error often looks as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. |
This will happen on Windows and MacOS where the default start method is ‘spawn‘. It may also happen when you configure your program to use the ‘spawn‘ start method on other platforms.
We can demonstrate this error with a worked example.
In this example we will configure a new child process to execute our custom task() function. We will then start the child process by calling the start() function.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# SuperFastPython.com # example of missing __name__ == '__main__' from multiprocessing import Process # function executed in a new process def task(): print('Hello from a child process', flush=True) # create and configure a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish process.join() |
Running the example raises a RuntimeError on Windows and MacOS.
The RuntimeError is raised when we try to start the child process.
Specifically on the line:
|
1 2 |
... process.start() |
A sample of the message reported by the RuntimeError is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Traceback (most recent call last): ... RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. |
This is a common error and is easy to fix.
Run loops using all CPUs, download your FREE book to learn how.
Fix RuntimeError When Starting New Process
The fix involves checking if the code is running in the top-level environment and only then, attempt to start a new process.
This is a best practice.
The idiom for this fix, as stated in the message of the RuntimeError, is to use an if-statement and check if the name of the module is equal to the string ‘__main__‘.
For example:
|
1 2 3 4 |
... # check for top-level environment if __name__ == '__main__': # ... |
This is called “protecting the entry point” of the program.
Recall, that __name__ is a variable that refers to the name of the module executing the current code.
When a Python module or package is imported, __name__ is set to the module’s name. Usually, this is the name of the Python file itself without the .py extension:
— __main__ — Top-level code environment
Also, recall that ‘__main__‘ is the name of the top-level environment used to execute a Python program.
__main__ is the name of the environment where top-level code is run. “Top-level code” is the first user-specified Python module that starts running. It’s “top-level” because it imports all other modules that the program needs.
— __main__ — Top-level code environment
Using an if-statement to check if the module is the top-level environment and only starting child processes within that block will resolve the RuntimeError.
It means that if the Python file is imported, then the code protected by the if-statement will not run. It will only run when the Python file is run directly, e.g. is the top-level environment.
Some modules contain code that is intended for script use only, like parsing command-line arguments or fetching data from standard input. If a module like this was imported from a different module, for example to unit test it, the script code would unintentionally execute as well. This is where using the if __name__ == ‘__main__’ code block comes in handy. Code within this block won’t run unless the module is executed in the top-level environment.
— __main__ — Top-level code environment
The if-statement idiom is required, even if the entry point of the program calls a function that itself starts a child process.
This is only used for child processes created by the main process. Child processes themselves can create child processes without this check.
Why Do We Need to Protect Entry Point?
We must protect the entry point when starting new processes from the main process using the ‘spawn‘ method.
The reason is because our Python script will be loaded and imported automatically for us by child processes.
This is required to execute our custom code and functions in new child processes.
If the entry point was not protected with an if-statement idiom checking for the top-level environment, then the script would execute again directly, rather than run a new child process as expected.
Protecting the entry point ensures that the program is only started once, that the tasks of the main process are only executed by the main process and not the child processes.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Fix By Protecting Entry Point
We can demonstrate how to check for the top-level environment when starting a new process.
This can be achieved by updating the example above that resulted in a RuntimeError and adding a check if __name__ is equal to ‘__main__’.
For example:
|
1 2 3 4 5 6 7 8 |
# check for top-level environment if __name__ == '__main__': # create and configure a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# SuperFastPython.com # example of starting a child process only in __main__ from multiprocessing import Process # function executed in a new process def task(): print('Hello from a child process', flush=True) # check for top-level environment if __name__ == '__main__': # create and configure a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish process.join() |
Running the example successfully starts a new child process.
The message is reported from the child process as expected.
|
1 |
Hello from a child process |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Fix by Changing Start Method
We can also fix the issue by changing the start method from ‘spawn‘ to ‘fork‘.
This fix only works on those platforms where the ‘fork’ method is supported, e.g. Unix and MacOS.
This can be achieved via the multiprocessing.set_start_method() function.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of missing __name__ == '__main__' from multiprocessing import set_start_method from multiprocessing import Process # function executed in a new process def task(): print('Hello from a child process', flush=True) # set the start method set_start_method('fork') # create and configure a new process process = Process(target=task) # start the new process process.start() # wait for the new process to finish process.join() |
Running the example successfully starts a new child process.
The message is reported from the child process as expected.
|
1 |
Hello from a child process |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to fix the RuntimeError caused by starting child processes with the ‘spawn‘ start method.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Ivan Ragozin on Unsplash
Do you have any questions?