The multiprocessing.dummy module module provides a wrapper for the multiprocessing module, except implemented using thread-based concurrency.
It provides a drop-in replacement for multiprocessing, allowing a program that uses the multiprocessing API to switch to threads with a single change to import statements.
In this tutorial you will discover the multiprocessing.dummy module in Python.
Let’s get started.
Table of Contents
What is multiprocessing.dummy?
The multiprocessing.dummy module provides the multiprocessing module API except implemented using threads.
The API documentation reports that dummy is a wrapper around the threading module that implements the multiprocessing module API.
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.— multiprocessing — Process-based parallelism
Next, let’s consider why the module exists.
Run your loops using all CPUs, download my FREE book to learn how.
Why does multiprocessing.dummy exist?
The multiprocessing.dummy module is not explicitly mentioned in the PEP 371 that describes the introduction of the multiprocessing module.
Nevertheless, the multiprocessing.dummy does appear in Python 2.6 which saw the introduction of the multiprocessing module and the dummy module is described in the API documentation in this version.
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module— multiprocessing — Process-based parallelism
It is stated in the PEP that the multiprocessing module seeks to provide the same API as the threading module, except implemented using processes instead of threads.
The processing package mimics the standard library threading module functionality to provide a process-based approach to threaded programming allowing end-users to dispatch multiple tasks that effectively side-step the global interpreter lock.— PEP 371 – Addition of the multiprocessing package to the standard library
The multiprocessing module introduces additional capabilities that do not exist in the threading API, such as:
- Shared ctypes (Value and Array)
The best guess is that the multiprocessing.dummy module is provided to be a drop-in replacement for the multiprocessing.
This would allow a program to be implemented using the multiprocessing module API with process-based concurrency and switch to thread-based concurrency with a single line change, e.g. switching the import of multiprocessing to multiprocessing.dummy.
Next, let’s explore what classes are provided in the dummy module.
Confused by the ThreadPool class API?
Download my FREE PDF cheat sheet
What Classes does multiprocessing.dummy Provide?
The multiprocessing.dummy module provides many classes and module functions that match the multiprocessing module.
For example, we can report the contents of the module directly via the __all__ list.
# example reporting all capabilities of the multiprocessing.dummy module
# report all items in the module
for item in multiprocessing.dummy.__all__:
Running the example report the contents of the module, as follows:
We can see 3 module methods and the rest are multiprocessing class names and mostly concurrency primitives.
Reviewing the source code for the dummy module, we can see that most of the API involves importing classes from the threading module, such as queues and concurrency primitives such as locks and semaphores.
There a few classes defined in the module, they are:
- DummyProcess, which is aliased as Process and used in the ThreadPool
- Namespace, used in managers
- Array and Value,
Next, let’s take a closer look at the multiprocessing.dummy.Pool class.
Free Python ThreadPool Course
Download my ThreadPool API cheat sheet and as a bonus you will get FREE access to my 7-day email course.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
multiprocessing.dummy.Pool vs multiprocessing.pool.ThreadPool
Perhaps the most significant class in multiprocessing.dummy is Pool.
This is because it returns a ThreadPool class instance that is a wrapper for Pool, except that it uses threads instead of processes for concurrency.
In particular, the Pool function provided by multiprocessing.dummy returns an instance of ThreadPool, which is a subclass of Pool that supports all the same method calls but uses a pool of worker threads rather than worker processes.— multiprocessing — Process-based parallelism
It is significant because the ThreadPool is a useful, although often neglected class.
It is the only class that is part of multiprocessing.dummy (technically multiprocessing.pool) that warrants mention in the multiprocessing API documentation.
Note A ThreadPool shares the same interface as Pool, which is designed around a pool of processes and predates the introduction of the concurrent.futures module. As such, it inherits some operations that don’t make sense for a pool backed by threads, and it has its own type for representing the status of asynchronous jobs, AsyncResult, that is not understood by any other libraries.— multiprocessing — Process-based parallelism
ThreadPool is defined in the multiprocessing.pool module, although probably should sit in the multiprocessing.dummy module.
Importantly, it imports the Process class from the dummy module for worker threads, which is an alias for the DummyProcess class that extends threading.Thread.
We can see this in the source code for the dummy module where Pool is defined.
def Pool(processes=None, initializer=None, initargs=()):
from ..pool import ThreadPool
return ThreadPool(processes, initializer, initargs)
Creating an instance of multiprocessing.dummy.Pool will return an instance of the multiprocessing.pool.ThreadPool.
We can demonstrate this with a worked example.
# example of creating a dummy pool class
from multiprocessing.dummy import Pool
# protect the entry point
if __name__ == '__main__':
# create a pool
pool = Pool()
# report the details of the pool
# close the pool
Running the example imports the multiprocessing.dummy.Pool class.
An instance of the Pool is created, and the details of the object are reported.
We can see that the Pool is an instance of the multiprocessing.pool.ThreadPool class.
<multiprocessing.pool.ThreadPool state=RUN pool_size=8>
Overwheled by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
This section provides additional resources that you may find helpful.
- Python ThreadPool Jump-Start, Jason Brownlee, 2022 (my book!).
- Threading API Interview Questions
- ThreadPool Class API Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
You now know about the multiprocessing.dummy module in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.