SuperFastPython

Multiprocessing Freeze Support in Python

June 17, 2022 Python Multiprocessing

You can add support for multiprocessing when freezing your code via the multiprocessing.freeze_support() function.

In this tutorial you will discover how to add freeze support for multiprocessing in your Python program.

Let's get started.

What is Freezing Code

You can package your code with a Python runtime.

For example, you can package your program up as a .exe for Windows or .app for macOS that includes both your program and all its code, as well as the Python runtime and relevant standard libraries required to execute it.

It allows your program to run without the end user having to install anything else, such as Python.

It requires that your code be transformed into C-code, called freezing, performed automatically by a freezing tool.

It converts Python byte code to C arrays; with a C compiler you can embed all your modules into a new program, which is then linked with the standard Python modules. [...] It then compiles the generated C code and links it with the rest of the Python interpreter to form a self-contained binary which acts exactly like your script.

-- How can I create a stand-alone binary from a Python script?

The benefit to the user is that your program will work out of the box with the intended version of Python. A downside is that the application will be much larger, e.g. megabytes, and required updates to the program or the included Python runtime will require an update of the entire package.

It is common to freeze end-user applications written in Python intended to be distributed on Windows and macOS, although frozen versions of a program can be distributed to all platforms.

Python includes a tool to freeze Python scripts called freeze.py.

Alternately, you can freeze your program and create a distribution of your application using a third-party tool, such as:

Now that we know what freezing code means, let's look at a possible problem when using multiprocessing.

Multiprocessing and Freezing Code

When a program is frozen in order to be distributed, some features of Python are not included or disabled by default.

This is for performance and/or security reasons.

One feature that is disabled when freezing a Python program is multiprocessing.

That is, we cannot create new python processes via multiprocessing.Process instances when freezing our program for distribution.

Creating a process in a frozen application results in a RuntimeError.

How can we use multiprocessing in frozen code?

Add Multiprocessing Support When Freezing Code

We can add support for multiprocessing in our program when freezing code via the multiprocessing.freeze_support() function.

Add support for when a program which uses multiprocessing has been frozen to produce a Windows executable. (Has been tested with py2exe, PyInstaller and cx_Freeze.)

-- multiprocessing — Process-based parallelism

For example:

...
# enable support for multiprocessing
multiprocessing.freeze_support()

This function will allow a frozen program to create and start new processes via the multiprocessing.Process class when the program is frozen for distribution on Windows.

If the function is called and the program is not frozen for distribution, then it has no effect.

If the function is called on platforms other than Windows, it has no effect.

Calling freeze_support() has no effect when invoked on any operating system other than Windows. In addition, if the module is being run normally by the Python interpreter on Windows (the program has not been frozen), then freeze_support() has no effect.

-- multiprocessing — Process-based parallelism

This function must be called as the first line of the program, e.g. the first line after protecting the entry point.

One needs to call this function straight after the if __name__ == '__main__' line of the main module.

-- multiprocessing — Process-based parallelism

For example:

# protect the entry point
if __name__ == '__main__':
	# enable support for multiprocessing
	multiprocessing.freeze_support()
	# ...

Now that we know how to enable multiprocessing support in frozen code, let's look at a worked example.

Example of Adding Freeze Support for Multiprocessing

We can add freeze support for multiprocessing.

In this example we will run a custom function in a child process. The main process will first add freeze support and then configure and start the child process.

First, we can define a function to run in a child process.

The function will report a message then sleep for two seconds.

The task() function below implements this.

# function executed in a new child process
def task():
    # report message
    print('Hello from child process', flush=True)
    # block for a moment
    sleep(2)

Next, in the main process, we will first add support for multiprocessing while the program is frozen.

...
# add freeze support for multiprocessing
freeze_support()

We can create and configure a new multiprocessing.Process instance to run our custom function.

...
# configure the child process
child = Process(target=task)

Next, we can start the new child process, then wait for the child process to finish.

...
# start the child process
child.start()
# wait for the child process to terminate
child.join()

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of multiprocessing program with freeze support
from time import sleep
from multiprocessing import Process
from multiprocessing import freeze_support

# function executed in a new child process
def task():
    # report message
    print('Hello from child process', flush=True)
    # block for a moment
    sleep(2)

# protect the entry point
if __name__ == '__main__':
    # add freeze support for multiprocessing
    freeze_support()
    # configure the child process
    child = Process(target=task)
    # start the child process
    child.start()
    # wait for the child process to terminate
    child.join()

First, save the above program as a Python script with the filename "app.py".

We can then freeze the program.

You can use your preferred freezing tool, such as a third-party tool or freeze.py.

In this case, we will use the freeze tool provided with Python. This is in the Tools/ directory of your Python installation and has the name freeze.py.

For example:

/path/to/python/examples/Tools/freeze/freeze.py

If you need help with this tool, see documentation in the Python Wiki.

To freeze our program, call the freeze.py tool and specify the name of our app.py script.

For example:

python examples/Tools/freeze/freeze.py app.py

This will convert the program to C code.

...
freezing urllib.parse ...
freezing urllib.request ...
freezing urllib.response ...
freezing uu ...
freezing warnings ...
freezing weakref ...
freezing webbrowser ...
freezing xml ...
freezing xml.etree ...
freezing xml.etree.ElementPath ...
freezing xml.etree.ElementTree ...
freezing xml.parsers ...
freezing xml.parsers.expat ...
freezing xmlrpc ...
freezing xmlrpc.client ...
freezing zipfile ...
freezing zipimport ...
generating table of frozen modules
Now run "make" to build the target: app

Next, compile the C code.

This can be achieved by calling make.

Note, this assumes you have a C compiler and build system installed, which is typically the case by default for many modern development workstations, e.g. macOS and Linux.

For example:

make

This will compile the generated C-code into a binary for your platform.

Then run the program.

For example:

./app.py

This will execute the native program.

Internally, the program will start a new child Python process and report a message.

Hello from child process

You can learn more about multiprocessing in the tutorial:

Takeaways

You now know how to add freeze support for multiprocessing in your Python program.



If you enjoyed this tutorial, you will love my book: Python Multiprocessing Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.