Last Updated on October 8, 2022

You can use a Manager to create a hosted Queue object and share it via proxy objects with multiple child processes.

In this tutorial you will discover how to share a queue using a manager in Python.

Let’s get started.

Need Manager to Share a Queue

A manager in the multiprocessing module provides a way to create Python objects that can be shared easily between processes.

Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines. A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

— multiprocessing — Process-based parallelism

A manager creates a server process that hosts a centralized version of objects that can be shared among multiple processes.

The objects are not shared directly. Instead, the manager creates a proxy object for each object that it manages and the proxy objects are shared among processes.

The proxy objects are used and operate just like the original objects, except that they serialize data, synchronize and coordinate with the centralized version of the object hosted in the manager server process.

A proxy is an object which refers to a shared object which lives (presumably) in a different process. […] A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy).

— multiprocessing — Process-based parallelism

This makes managers a process-safe and preferred way to share Python objects among processes.

You can learn more about multiprocessing managers in the tutorial:

• What is a Multiprocessing Manager

When using multiprocessing, we may need to share a Queue with child processes.

This may be for many reasons, such as:

• Child processes need to put results to the queue.
• Child processes need to get results from the queue.

We may not be able to share the Queue directly with the child processes.

This may be because a queue object cannot be pickled, and we may have to pickle objects when sending them to a child process such as via a Pipe, via another Queue or as an argument to a task to the Pool class.

How can we use a multiprocessing Manager to share a Queue with child processes?

Manager for Sharing a Sharing Queue

Python provides a process-safe queue in the multiprocessing.Queue class.

A queue is a data structure on which items can be added by a call to put() and from which items can be retrieved by a call to get().

The multiprocessing.Queue provides a first-in, first-out FIFO queue, which means that the items are retrieved from the queue in the order they were added. The first items added to the queue will be the first items retrieved. This is opposed to other queue types such as last-in, first-out and priority queues.

You can learn more about how to use the process-safe Queue in the tutorial:

• Multiprocessing Queue in Python

Managers are provided via the multiprocessing.Manager class, which creates and returns a multiprocessing.managers.SyncManager instance.

The SyncManager allows a suite of Python objects to be created and managed by default, including a Queue.

Specifically the SyncManager provides two registered Queue objects, they are:

• Queue
• JoinableQueue

As such, we can create a Manager instance and use it directly to create a Queue object.

This will create a hosted version of the Queue in the Manager‘s server process and return proxy objects for the Queue that can be pickled and shared with child processes.

For example:

From reviewing the source code for the SyncManager class, we can see in fact that a queue.Queue is created and hosted in the Manager’s server process. This means that all process safety is handled via the proxy objects that are returned.

Now that we know how to use a manager to create a shared Queue, let’s look at a worked example.

Example of Using a Manager for Sharing a Queue

We can use a manager to share a Queue safely with multiple processes.

In this example we will define a task that generates some data and puts it on the shared queue. We will then execute the target from many child processes in parallel. The main process will get data from the queue until the expected number of results have been received.

Firstly, we can define a function to execute in child processes.

The function will take an integer number argument and the shared queue. It will then generate a random number between 0 and 1, block for a fraction of a second, then put the integer argument and generated number on the shared queue as a tuple.

The task() function below implements this.

Next, in the main process, we will create and start a Manager instance using the context manager interface.

We can then use the manager to create a shared queue.

The Queue object will be hosted in the manager server process and returns proxy objects that can be shared among child processes, can be pickled if needed, and used to interact with the Queue in a process-safe manner.

Next, we can create many multiprocessing.Process instances configured to run our task() function and pass an integer and the shared Queue object as arguments.

This can be achieved in a list comprehension that will create a list of configured Process instances.

The main process can then iterate through the processes and start each in turn.

The main process will then loop a fixed number of times to retrieve the expected number of task results. Each iteration, a result will be retrieved from the shared queue and reported.

If no result is ready, the main process will block, waiting for a task result to be made available by a child process.

Tying this together, the complete example is listed below.

Running the example first creates the manager using the context manager interface.

Next, the manager is used to create the shared Queue object.

This creates a centralized version of the Queue on the Manager’s server process and returns proxy objects for interacting with the Queue.

Next, 50 child processes are created and configured to call our custom task() function with the shared Queue, then the processes are started. The main process then blocks waiting to retrieve results from the queue.

Each process generates a random number, blocks for a fraction of a second then puts its result on the shared queue.

As results are put on the queue, the main process gets the results one at a time and reports them. This continues until the expected number of results is retrieved.

Note: The output will vary each time the program is run given the use of random numbers.

This example highlights how we might use a Manager to create a hosted Queue object that can be shared with many child processes.

Further Reading

This section provides additional resources that you may find helpful.

Python Multiprocessing Books

• Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Multiprocessing API Cheat Sheet

I would also recommend specific chapters in the books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism
• PEP 371 - Addition of the multiprocessing package

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to share a queue using a manager in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by John Cameron on Unsplash