Last Updated on September 12, 2022
You can change the method used for starting child processes via the multiprocessing.set_start_method() function.
In this tutorial you will discover how to retrieve supported and current start methods and how to change the start method in your Python program.
Let’s get started.
Need to Change Start Method
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
In multiprocessing programming, we may need to change the technique used to start child processes.
This is called the start method.
What is a start method and how can we configure it in Python?
Run loops using all CPUs, download your FREE book to learn how.
What is a Start Method
A start method is the technique used to start child processes in Python.
There are three start methods, they are:
- spawn: start a new Python process.
- fork: copy a Python process from an existing process.
- forkserver: new process from which future forked processes will be copied.
Default Start Methods
Each platform has a default start method.
The following lists the major platforms and the default start methods.
- Windows (win32): spawn
- macOS (darwin): spawn
- Linux (unix): fork
Supported Start Methods
Not all platforms support all start methods.
The following lists the major platforms and the start methods that are supported.
- Windows (win32): spawn
- macOS (darwin): spawn, fork, forkserver.
- Linux (unix): spawn, fork, forkserver.
Generally, a fork is considered not safe on macOS.
You can learn more about this here:
How to Change The Start Method
The multiprocessing package provides functions for getting and setting the start method for creating child processes.
The start method API includes the following functions:
- multiprocessing.get_all_start_methods()
- multiprocessing.get_start_method()
- multiprocessing.set_start_method()
- multiprocessing.get_context()
Let’s take a closer look at each in turn.
How to Get Supported Start Methods
The list of supported start methods can be retrieved via the multiprocessing.get_all_start_methods() function.
The function returns a list of string values, each representing a supported start method.
For example:
|
1 2 3 |
... # get supported start methods methods = multiprocessing.get_all_start_methods() |
How to Get The Current Start Method
The current start method can be retrieved via the multiprocessing.get_start_method() function.
The function returns a string value for the currently configured start method.
For example:
|
1 2 3 |
... # get the current start method method = multiprocessing.get_start_method() |
How to Set Start Method
The start method can be set via the multiprocessing.set_start_method() function.
The function takes a string argument indicating the start method to use.
This must be one of the methods returned from the multiprocessing.get_all_start_methods() for your platform.
For example:
|
1 2 3 |
... # set the start method multiprocessing.set_start_method('spawn') |
It is a best practice, and required on most platforms that the start method be set first, prior to any other code, and to be done so within a if __name__ == ‘__main__’ check called a protected entry point or top-level code environment.
For example:
|
1 2 3 4 5 |
... # protect the entry point if __name__ == '__main__': # set the start method multiprocessing.set_start_method('spawn') |
If the start method is not set within a protected entry point, it is possible to get a RuntimeError such as:
|
1 |
RuntimeError: context has already been set |
It is also a good practice and required on some platforms that the start method only be set once.
Best Practices For Setting the Start Method
In summary, the rules for setting the start method are as follows:
- Set the start method first prior to all other code.
- Set the start method only once in a program.
- Set the start method within a protected entry point.
How to Set Start Method Via Context
A multiprocessing context configured with a given start method can be retrieved via the multiprocessing.get_context() function.
This function takes the name of the start method as an argument, then returns a multiprocessing context that can be used to create new child processes.
For example:
|
1 2 3 |
... # get a context configured with a start method context = multiprocessing.get_context('fork') |
The context can then be used to create a child process, for example:
|
1 2 3 |
... # create a child process via a context process = context.Process(...) |
It may also be possible to force the start method.
This can be achieved via the “force” argument provided on the set_start_method() implementation in the DefaultContext, although not documented.
For example:
|
1 2 3 |
... # set the start method context.set_start_method('spawn', force=True) |
Now that we know how to configure the start method, let’s look at some worked examples.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Getting Supported Start Methods
We can get a list of all supported start methods for the current platform.
This can be achieved via the multiprocessing.get_all_start_methods() function.
The example below gets the list of supported start methods for the current platform, then reports the result.
|
1 2 3 4 5 6 7 8 |
# SuperFastPython.com # example of getting the supported start methods from multiprocessing import get_all_start_methods # protect entry point if __name__ == '__main__': # get the supported start methods methods = get_all_start_methods() print(methods) |
Running the example gets the list of supported start methods for the current platform.
In this case, we can see the current platform supports all three start methods.
Your specific results may differ, depending on your platform.
|
1 |
['spawn', 'fork', 'forkserver'] |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Getting the Current Start Method
We can get the name of the currently configured start method.
This can be achieved via the multiprocessing.get_start_method() function
The example below gets the currently configured start method and reports it.
|
1 2 3 4 5 6 7 8 |
# SuperFastPython.com # example of getting the start method from multiprocessing import get_start_method # protect entry point if __name__ == '__main__': # get the start method method = get_start_method() print(method) |
Running the example retrieves the currently configured start method.
In this case, the ‘spawn‘ method is the default start method.
Your specific result may differ, depending on your platform.
|
1 |
spawn |
Example of Setting the Start Method
We can configure the start method using the multiprocessing.set_start_method() function.
In this section we can explore examples of setting each of the different types of start methods and then using it to start a new child process.
Specifically, we will look at starting a child process using the spawn, fork, and forkserver start methods.
Let’s take a closer look at each.
Example of Spawn Start Method
We can start a child process using the ‘spawn‘ start method.
|
1 2 3 |
... # set the spawn method set_start_method('spawn') |
Once configured, we can confirm that the start method was changed.
|
1 2 3 |
... # report the start method print(f'Start Method: {get_start_method()}') |
We can then create a new child process using the start method to execute a custom function.
|
1 2 3 4 5 |
... # start a child process process = Process(target=task) process.start() process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of setting the spawn start method from multiprocessing import set_start_method from multiprocessing import get_start_method from multiprocessing import Process # function executed in a new process def task(): print('Hello from new process', flush=True) # protect entry point if __name__ == '__main__': # set the spawn method set_start_method('spawn') # report the start method print(f'Start Method: {get_start_method()}') # start a child process process = Process(target=task) process.start() process.join() |
Running the example first configures the start method as ‘spawn‘.
The current start method is retrieved and reported, confirming that the change has taken effect.
A new child process is then created using the start method.
|
1 2 |
Start Method: spawn Hello from new process |
Example of Fork Start Method
We can start a child process using the ‘fork‘ start method.
|
1 2 3 |
... # set the start method set_start_method('fork') |
Once configured, we can confirm that the start method was changed.
|
1 2 3 |
... # report the start method print(f'Start Method: {get_start_method()}') |
We can then create a new child process using the start method to execute a custom function.
|
1 2 3 4 5 |
... # start a child process process = Process(target=task) process.start() process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of setting the fork start method from multiprocessing import set_start_method from multiprocessing import get_start_method from multiprocessing import Process # function executed in a new process def task(): print('Hello from new process', flush=True) # protect entry point if __name__ == '__main__': # set the start method set_start_method('fork') # report the start method print(f'Start Method: {get_start_method()}') # start a child process process = Process(target=task) process.start() process.join() |
Running the example first configures the start method as ‘fork‘.
The current start method is retrieved and reported, confirming that the change has taken effect.
A new child process is then created using the start method.
|
1 2 |
Start Method: fork Hello from new process |
Example of ForkServer Start Method
We can start a child process using the ‘forkserver‘ start method.
Recall that the forkserver method will start a new server process and use a copy of this process each time a child process is created.
|
1 2 3 |
... # set the start method set_start_method('forkserver') |
Once configured, we can confirm that the start method was changed.
|
1 2 3 |
... # report the start method print(f'Start Method: {get_start_method()}') |
We can then create a new child process using the start method to execute a custom function.
|
1 2 3 4 5 |
... # start a child process process = Process(target=task) process.start() process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of setting the forkserver start method from multiprocessing import set_start_method from multiprocessing import get_start_method from multiprocessing import Process # function executed in a new process def task(): print('Hello from new process', flush=True) # protect entry point if __name__ == '__main__': # set the start method set_start_method('forkserver') # report the start method print(f'Start Method: {get_start_method()}') # start a child process process = Process(target=task) process.start() process.join() |
Running the example first configures the start method as ‘forkserver‘.
The current start method is retrieved and reported, confirming that the change has taken effect.
A new child process is then created using the start method.
|
1 2 |
Start Method: forkserver Hello from new process |
Example of Multiple Start Methods
We can use multiple start methods within our program via a multiprocessing context.
In this example we will set the baseline start method to ‘fork‘ and use it to start a new child process. We will then create a new multiprocessing context with the ‘spawn’ start method, then use it to start a child process.
First, we can set the baseline start method to ‘fork‘, confirm it has been set, then start a new process using the method.
|
1 2 3 4 5 6 7 8 9 |
... # set the start method set_start_method('fork') # report the start method print(f'Start Method: {get_start_method()}') # start a child process process = Process(target=task) process.start() process.join() |
Next, we can create a new multiprocessing context using the ‘spawn‘ start method.
|
1 2 3 |
... # change start method context = get_context('spawn') |
We can then confirm that the context uses the configured start method.
|
1 2 3 |
... # report the start method print(f'Start Method: {context.get_start_method()}') |
Finally, we can create a new child process using the context with the different start method.
|
1 2 3 4 5 |
... # start a child process process = context.Process(target=task) process.start() process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of setting the fork start method from multiprocessing import get_context from multiprocessing import set_start_method from multiprocessing import get_start_method from multiprocessing import Process # function executed in a new process def task(): print('Hello from new process', flush=True) # protect entry point if __name__ == '__main__': # set the start method set_start_method('fork') # report the start method print(f'Start Method: {get_start_method()}') # start a child process process = Process(target=task) process.start() process.join() # change start method context = get_context('spawn') # report the start method print(f'Start Method: {context.get_start_method()}') # start a child process process = context.Process(target=task) process.start() process.join() |
Running the example first sets the baseline start method to ‘fork‘.
It then confirms that the start method was changed as expected and then creates and starts a new child process using the start method.
Next, a new multiprocessing context is created with a different start method, ‘spawn‘ in this case.
The start method of the new context is then confirmed and a new child process is created using the start method in the created context, different from the baseline start method.
|
1 2 3 4 |
Start Method: fork Hello from new process Start Method: spawn Hello from new process |
Example of RuntimeError Setting Start Method
We may get a RuntimeError when setting the start method.
This can happen when setting a specific start method, e.g. ‘spawn‘ outside of a protected entry point.
We can demonstrate this with a worked example.
The example below attempts to start the start method to ‘spawn‘ first, directly after the import statements and outside of the protected entry point.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of getting a runtime error when setting the start method from multiprocessing import set_start_method from multiprocessing import get_start_method from multiprocessing import Process # set the start method set_start_method('spawn') # raises a RuntimeError # function executed in a new process def task(): print('Hello from new process', flush=True) # protect entry point if __name__ == '__main__': # report the start method print(f'Start Method: {get_start_method()}') # start a child process process = Process(target=task) process.start() process.join() |
Running the example results in a RuntimeError indicating that the start method has already been set.
|
1 2 3 4 |
Start Method: spawn Traceback (most recent call last): ... RuntimeError: context has already been set |
Interestingly, we don’t get the error if we attempt to set the start method to ‘fork‘ outside of the protected entry point.
The fix involves moving the call to set_start_method() to be the first line within the protected entry point.
For example:
|
1 2 3 4 5 |
... # protect entry point if __name__ == '__main__': # set the start method set_start_method('spawn') |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to configure the start method for processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Greg Rosenke on Unsplash
This is all very nice but you leave out a very specific topic: in cases where multiple types of starts are allowed, why would I want one over another?
Check out https://britishgeologicalsurvey.github.io/science/python-forking-vs-spawn/ for a very good description.
Thanks, the table in the link is fantastic.
More generally, fork is unstable on some platforms, e.g. macos, or is is not available on some systems, e.g. win32.
I’d also say that fork is generally faster than spawn.
I meant to add that you can see https://www.tutorialspoint.com/concurrency_in_python/concurrency_in_python_multiprocessing.htm for a great description as well./
Thanks. I found this more general.