You can initialize workers in the process pool by setting the “initializer” argument in the multiprocessing.pool.Pool class constructor.

In this tutorial you will discover how to initialize worker processes in the process pool in Python.

Let’s get started.

Need to Initialize Worker Processes

The multiprocessing.pool.Pool in Python provides a pool of reusable processes for executing ad hoc tasks.

A process pool can be configured when it is created, which will prepare the child workers.

A process pool object which controls a pool of worker processes to which jobs can be submitted. It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.

— multiprocessing — Process-based parallelism

We can issue one-off tasks to the process pool using functions such as apply() or we can apply the same function to an iterable of items using functions such as map(). Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().

When using a process pool, we may need to initialize a variable, data, or resource to be used by each worker process across all tasks executed by that process.

For example, perhaps each process is required to have its own handle for logging or connection to a remote server to be held open and reused when executing tasks.

We need a way of calling a function to initialize each worker process in the process pool, prior to executing any tasks.

How can we initialize worker processes in the process pool?

How to Initialize Worker Processes

We can configure worker processes in the process pool to execute an initialization function prior to executing tasks.

This can be achieved by setting the “initializer” argument when configuring the process pool via the class constructor.

By default, there is no initializer function and “initializer” is set to None.

The “initializer” argument can be set to the name of a function that will be called to initialize the worker processes.

If initializer is not None then each worker process will call initializer(*initargs) when it starts.

— multiprocessing — Process-based parallelism

For example:

If our worker process initialization function takes arguments, they can be specified to the process pool constructor via the “initargs” argument, which takes an ordered list or tuple of arguments for the custom initialization function.

For example:

Now that we know how to initialize worker processes in the process pool, let’s look at a worked example.

Example of Initializing Worker Processes

We can develop an example of calling a custom function to initialize each process in the process pool.

In this example we will define a task to simulate work that will report a message and block for a moment. We will also define a simple worker process initialization function that will simply report a message. We will then configure a process pool to initialize the workers with our initialization function and execute a number of tasks.

The messages will show that the workers are initialized once right after they are created in the process pool, then process tasks until all work is complete.

Firstly, we can define a function to execute tasks in the process pool. The function will report a message and block for a moment to simulate computational effort.

The task() function below implements this.

Next, we can define a function to initialize the worker processes.

The initialization function won’t do anything interesting, just report a message so we have some idea that the initialization function was called and when it was called, e.g. once, before executing tasks.

The initialize_worker() function below implements this.

Next, we can create a process pool and configure it to use our initialize_worker() function to initialize each worker process.

We will create the process pool using the context manager interface and configure it to create two worker processes.

Next, we will issue four tasks to the process pool to be executed asynchronously. There are only two workers, so only two tasks will be executed at a time.

Finally, we will close down the process pool and wait for all tasks to complete.

Tying this together, the complete example is listed below.

Running the example first creates and configures the process pool.

Tasks are issued to the pool and worker processes are created as needed to execute the tasks.

After the worker processes are created they are initialized, then start executing the issued tasks.

Importantly, each worker process is initialized once and only before it begins consuming and completing tasks in the pool.

Example of Thread and Process Names that Initialize

It may be helpful to know which process and which thread in that process is used to initialize the worker processes.

For example, if the same thread is used to initialize the worker process as is used to execute tasks in the worker process, we can share data across tasks using mechanisms such as thread-local data and/or global variables.

We can get the current process and current thread in the initialization function and report their name. The same can be done in the task function executed by worker process, and we can see if the names match.

This can be achieved using the multiprocessing.current_process() and threading.current_thread() functions.

We can update the example in the previous section to first get the current process and thread, then report their names.

Firstly, we can update the task function executed by worker processes to report the current process and thread names.

Next, we can update the worker process initialization function to report the process and thread names.

Tying this together, the complete example is listed below.

Running the example creates the thread pool and configures it so that worker processes are initialized using our custom function, as before.

Four tasks are issued to the process pool and the main process waits for all tasks to complete.

THe name of the worker process and thread within the process is reported both when initializing the workers and in executing the task.

Interestingly, we can see that the same process that executes the task is used to initialize the worker and the same thread is used in each case.

For example the first worker process has the name “SpawnPoolWorker-1” and executes both the initialization function and the task function in the main thread named “MainThread” of the process.

This highlights that we can use mechanisms like global variables and thread-local storage in worker processes to share data across tasks executed by a worker process.

Example of Accessing an Initialized Variable in a Worker

Perhaps one of the most common uses of the Pool initializer function is initialize a variable that is made available to the worker.

This variable may be passed into the initializer or may be a resource that is prepared in the initializer.

We can prepare a variable in the initializer function and share it with the worker process by making it a global variable.

Recall that each worker process is initialized with the initializer function. As we have seen in previous examples, the initialization occurs within the worker process.

Therefore defining a global variable in the initializer function will make it available to the worker process.

This can be achieved by first defining a variable in the custom initializer function as global, then assigning it a value.

For example:

Now that we know how to share an initialized variable with a worker, let’s look at a worked example.

In this example, we will pass in an object to the initializer function and store it in a global variable in the worker process. This will make it available to all worker processes.

This is the approach to share a multiprocessing.Event object with all workers in order to stop them gracefully as seen in this tutorial:

• Multiprocessing Pool Stop All Tasks If One Task Fails in Python

The complete example is listed below.

Running the example first defines the global state.

Next, the Pool is created and configured with two workers, and the custom worker initializer function and state as an argument.

Each of the two works are initialized, declaring a global variable, storing the data in the global variable, then reporting the value.

Four tasks are then issued to the pool for execution.

Each worker executes the task, reporting the value of its own copy of the shared data.

The example highlights how a variable can be defined in the initializer and used in the worker, and how we can share data from the main process with each worker without passing it as an argument to the task function.

Further Reading

This section provides additional resources that you may find helpful.

APIs

• multiprocessing — Process-based parallelism

Guides

• Multiprocessing Pool: The Complete Guide

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee, 2022 (my book!). 
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Takeaways

You now know how to initialize workers in the multiprocessing process pool in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by GEORGE DESIPRIS on Unsplash