Last Updated on September 12, 2022
You may get a FileNotFoundError when sharing concurrency primitives with child processes.
In this tutorial you will discover the preconditions when getting a FileNotFoundError when using multiprocessing in Python and how to fix this error in your programs.
Let’s get started.
FileNotFoundError With New Process
It is common to get a FileNotFoundError when sharing a concurrency primitive object with a child process.
This error is most common on MacOS and Windows.
This error is well understood, repeatable and there are fixes we can use.
Before we dive into understanding and fixing this error, let’s look at a worked example that recreates it.
The example below runs a custom function named task() in a child process. The function takes a mutex lock, then acquires the lock and blocks for one second. The main process starts first, creates the shared lock, then configures the child process to execute the task() function by passing the lock as an argument.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of a common error when using processes from time import sleep from multiprocessing import Process from multiprocessing import Lock # executed in a new thread def task(lock): # acquire the lock with lock: # block for a moment sleep(1) # entry point if __name__ == '__main__': # create a shared lock lock = Lock() # create a process that uses the lock process = Process(target=task, args=(lock,)) # start the process process.start() |
Running the example on MacOS or Windows results in the error.
|
1 2 3 |
Traceback (most recent call last): ... FileNotFoundError: [Errno 2] No such file or directory |
We can create the same error on other platforms, like Linux.
This can be achieved by configuring the program to use the ‘spawn‘ method when creating the child process, the default on MacOS and Windows.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of a common error when using processes from time import sleep from multiprocessing import set_start_method from multiprocessing import Process from multiprocessing import Lock # executed in a new thread def task(lock): # acquire the lock with lock: # block for a moment sleep(1) # entry point if __name__ == '__main__': # set the spawn start method set_start_method('spawn') # create a shared lock lock = Lock() # create a process that uses the lock process = Process(target=task, args=(lock,)) # start the process process.start() |
Running the example on any platform, e.g. Linux, Windows, or MacOS, results in the error.
|
1 2 3 |
Traceback (most recent call last): ... FileNotFoundError: [Errno 2] No such file or directory |
What are the preconditions for this error to occur?
Run loops using all CPUs, download your FREE book to learn how.
Cause of FileNotFoundError With New Process
This error is known and has been reported, for example:
It is sometimes reported as being specific to MacOS or to a specific concurrency primitive.
Nevertheless, there are three elements or pre-conditions that must be present for this error to occur.
They are:
- Use the ‘spawn’ start method to create new processes.
- Share a concurrent primitive with one or more processes.
- Main process exits before the child process starts up fully.
Let’s take a closer look at each element in turn.
Precondition 1: Spawn Start Method
The error requires that you are using the ‘spawn‘ start method.
Recall, the start method is the technique used by your system to start new child processes.
It may be ‘fork‘ or ‘spawn‘, or sometimes ‘forkserver‘.
For the error to occur, you must be using the ‘spawn‘ start method. This is the default start method for creating child processes on MacOS and Windows.
spawn: Available on Unix and Windows. The default on Windows and macOS.
— multiprocessing — Process-based parallelism
You can check the child process start method you are using via the multiprocessing.get_start_method() function.
For example:
|
1 2 3 4 5 6 7 |
# SuperFastPython.com # example of checking the start method from multiprocessing import get_start_method # get the start method method = get_start_method() # report the start method print(method) |
Running the example reports the current start method used for creating child processes.
|
1 |
spawn |
Precondition 2: Share Concurrency Primitive
The error requires that you share a concurrency primitive between the main process and one or more child processes.
Recall that the main process is the default or first process started to run your program.
Also, recall that concurrency primitives are those objects used to synchronization or coordinate behavior between processes, such as:
- Lock via the multiprocessing.Lock class.
- RLock via the multiprocessing.RLock class.
- Semaphore via the multiprocessing.Semaphore class.
- Event via the multiprocessing.Event class.
- Condition via the multiprocessing.Condition class.
- Barrier via the multiprocessing.Barrier class.
Precondition 3: Main Process Ends Early
The error requires that the main process finishes before the child process (or child processes) have had time to start-up completely.
This typically involves the main process having no further instructions to execute after starting the child process or processes.
This causes the error because the main process will terminate too soon and attempt to clean-up or garbage collect objects that are no longer needed, such as the shared concurrency primitive. Then, when the child process starts and attempts to use the concurrency primitive, it gets an error.
We get a FileNotFoundError because under the covers, processes coordinate with each other via files on the hard drive.
Now that we know the preconditions for the error, let’s look at how we might fix the error.
How to Fix FileNotFoundError With New Process
There are two ways we might fix the error.
The they:
- Change start method.
- Block the main process.
Let’s take a closer look at each in turn.
Change Start Method
We can fix the error by changing the start method to ‘fork‘, if on Windows or MacOS.
This can be achieved via the multiprocessing.set_start_method() which must be called before creating a child process.
For example:
|
1 2 3 |
... # change start method set_start_method('fork') |
This method may not be available on all platforms, e.g. it may not be available on windows.
The complete example with this fix is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of fix for common error with processes from time import sleep from multiprocessing import Process from multiprocessing import Lock from multiprocessing import set_start_method # executed in a new thread def task(lock): # acquire the lock with lock: # block for a moment sleep(1) # entry point if __name__ == '__main__': # change start method set_start_method('fork') # create a shared lock lock = Lock() # create a process that uses the lock process = Process(target=task, args=(lock,)) # start the process process.start() |
Block Main Process
We can fix the error by blocking the main process long enough to allow the child processes to start-up and run.
This could be achieved with a call to time.sleep(), although how long is long enough to sleep?
A preferred approach is to block the main process by waiting for the child process to terminate. This can be achieved via the Process.join() function.
For example:
|
1 2 3 |
... # block main process process.join() |
The complete example with this fix is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of fix for common error with processes from time import sleep from multiprocessing import Process from multiprocessing import Lock # executed in a new thread def task(lock): # acquire the lock with lock: # block for a moment sleep(1) # entry point if __name__ == '__main__': # create a shared lock lock = Lock() # create a process that uses the lock process = Process(target=task, args=(lock,)) # start the process process.start() # block main process process.join() |
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know the preconditions for getting FileNotFoundError when using multiprocessing in Python, and how to fix the error.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Michael Niessl on Unsplash
Thank you very much for this blog post! It really helped me with this problem.
Two comments though:
– I encountered this problem without having used any concurrency primitives. So, I am not quite sure if they really are a precondition for this kind of problem?
– fork(ing) as a start method is not available on Windows, as requested in the first solution.
Thanks again! 🙂
Thanks for sharing, I’m happy it helped!