Last Updated on October 29, 2022
The multiprocessing.dummy module module provides a wrapper for the multiprocessing module, except implemented using thread-based concurrency.
It provides a drop-in replacement for multiprocessing, allowing a program that uses the multiprocessing API to switch to threads with a single change to import statements.
In this tutorial you will discover the multiprocessing.dummy module in Python.
Let’s get started.
What is multiprocessing.dummy?
The multiprocessing.dummy module provides the multiprocessing module API except implemented using threads.
The API documentation reports that dummy is a wrapper around the threading module that implements the multiprocessing module API.
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.
— multiprocessing — Process-based parallelism
Next, let’s consider why the module exists.
Run loops using all CPUs, download your FREE book to learn how.
Why does multiprocessing.dummy exist?
The multiprocessing.dummy module is not explicitly mentioned in the PEP 371 that describes the introduction of the multiprocessing module.
Nevertheless, the multiprocessing.dummy does appear in Python 2.6 which saw the introduction of the multiprocessing module and the dummy module is described in the API documentation in this version.
For example:
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module
— multiprocessing — Process-based parallelism
It is stated in the PEP that the multiprocessing module seeks to provide the same API as the threading module, except implemented using processes instead of threads.
The processing package mimics the standard library threading module functionality to provide a process-based approach to threaded programming allowing end-users to dispatch multiple tasks that effectively side-step the global interpreter lock.
— PEP 371 – Addition of the multiprocessing package to the standard library
The multiprocessing module introduces additional capabilities that do not exist in the threading API, such as:
- Manager
- Shared ctypes (Value and Array)
- ThreadPool
The best guess is that the multiprocessing.dummy module is provided to be a drop-in replacement for the multiprocessing.
This would allow a program to be implemented using the multiprocessing module API with process-based concurrency and switch to thread-based concurrency with a single line change, e.g. switching the import of multiprocessing to multiprocessing.dummy.
Next, let’s explore what classes are provided in the dummy module.
What Classes does multiprocessing.dummy Provide?
The multiprocessing.dummy module provides many classes and module functions that match the multiprocessing module.
For example, we can report the contents of the module directly via the __all__ list.
|
1 2 3 4 5 6 |
# SuperFastPython.com # example reporting all capabilities of the multiprocessing.dummy module import multiprocessing.dummy # report all items in the module for item in multiprocessing.dummy.__all__: print(item) |
Running the example report the contents of the module, as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
Process current_process active_children freeze_support Lock RLock Semaphore BoundedSemaphore Condition Event Barrier Queue Manager Pipe Pool JoinableQueue |
We can see 3 module methods and the rest are multiprocessing class names and mostly concurrency primitives.
Reviewing the source code for the dummy module, we can see that most of the API involves importing classes from the threading module, such as queues and concurrency primitives such as locks and semaphores.
There a few classes defined in the module, they are:
- DummyProcess, which is aliased as Process and used in the ThreadPool
- Namespace, used in managers
- Array and Value,
- Manager
- Pool
Next, let’s take a closer look at the multiprocessing.dummy.Pool class.
Free Python ThreadPool Course
Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
multiprocessing.dummy.Pool vs multiprocessing.pool.ThreadPool
Perhaps the most significant class in multiprocessing.dummy is Pool.
This is because it returns a ThreadPool class instance that is a wrapper for Pool, except that it uses threads instead of processes for concurrency.
In particular, the Pool function provided by multiprocessing.dummy returns an instance of ThreadPool, which is a subclass of Pool that supports all the same method calls but uses a pool of worker threads rather than worker processes.
— multiprocessing — Process-based parallelism
It is significant because the ThreadPool is a useful, although often neglected class.
It is the only class that is part of multiprocessing.dummy (technically multiprocessing.pool) that warrants mention in the multiprocessing API documentation.
Note A ThreadPool shares the same interface as Pool, which is designed around a pool of processes and predates the introduction of the concurrent.futures module. As such, it inherits some operations that don’t make sense for a pool backed by threads, and it has its own type for representing the status of asynchronous jobs, AsyncResult, that is not understood by any other libraries.
— multiprocessing — Process-based parallelism
ThreadPool is defined in the multiprocessing.pool module, although probably should sit in the multiprocessing.dummy module.
Importantly, it imports the Process class from the dummy module for worker threads, which is an alias for the DummyProcess class that extends threading.Thread.
We can see this in the source code for the dummy module where Pool is defined.
For example:
|
1 2 3 |
def Pool(processes=None, initializer=None, initargs=()): from ..pool import ThreadPool return ThreadPool(processes, initializer, initargs) |
Creating an instance of multiprocessing.dummy.Pool will return an instance of the multiprocessing.pool.ThreadPool.
We can demonstrate this with a worked example.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# SuperFastPython.com # example of creating a dummy pool class from multiprocessing.dummy import Pool # protect the entry point if __name__ == '__main__': # create a pool pool = Pool() # report the details of the pool print(pool) # close the pool pool.close() |
Running the example imports the multiprocessing.dummy.Pool class.
An instance of the Pool is created, and the details of the object are reported.
We can see that the Pool is an instance of the multiprocessing.pool.ThreadPool class.
|
1 |
<multiprocessing.pool.ThreadPool state=RUN pool_size=8> |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python ThreadPool Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- ThreadPool PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPool: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
APIs
References
Takeaways
You now know about the multiprocessing.dummy module in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Kamil Pietrzak on Unsplash
Do you have any questions?