You can execute functions, methods, and static methods as tasks in the multiprocessing pool.
Some other types of functions, such as lambda functions and nested inner functions cannot be executed in the multiprocessing pool because they cannot be pickled.
In this tutorial, you will discover how to execute many different function types in the multiprocessing Pool.
Let’s get started.
Multiprocessing Pool Function Types
The multiprocessing pool provides a pool of reusable workers for executing ad hoc tasks with process-based concurrency.
An instance of the multiprocessing.Pool class can be created, specifying the number of workers to create, otherwise, a default number of workers will be created to match the number of logical CPUs in the system.
Once created, ad hoc tasks can be issued to the pool for execution via the Pool.apply() method. Multiple tasks may be executed in the book by calling the same function with different arguments via the Pool.map() method.
Tasks may be executed asynchronously in the pool via Pool.apply_async() and Pool.map_async(), allowing the caller to carry on with other tasks.
One concern is what are the types of functions that we can execute using the pool?
For example, we can define a custom function and execute it in the Pool, but what about other functions?
- Can we execute a lambda in the pool?
- Can we execute a nested inner function in the pool?
- Can we execute an object method in the pool?
- Can we execute a static class method in the pool?
The answer to these questions is not obvious.
Let’s explore exactly what types of functions can be used to issue tasks in the multiprocessing Pool.
Run loops using all CPUs, download your FREE book to learn how.
Example of Calling a Function With Pool
Before we explore different types of functions, let’s start with a regular custom function.
We can define a standalone function using the “def” expression that may or may not take arguments and may or may not return a value.
In this example, we will define a task() function that generates a random value between 0 and 1, sleeps for a fraction of a second, then returns the generated value.
We will then create a pool and issue the task, get and report the return value.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of calling a function from a pool from time import sleep from random import random from multiprocessing import Pool # do work def task(): # generate a random value value = random() # block for a moment sleep(value) # return a value return value # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # execute a task result = pool.apply_async(task) # report the result print(result.get()) |
Running the example first creates the multiprocessing pool with the default number of workers.
Next, the task() function is issued to the pool asynchronously and an AsyncResult object is returned.
The main process then calls the get() method on the AsyncResult object which blocks until the task is completed, then provides the return value of the target function, which is reported.
This highlights the typical use case for the multiprocessing pool, namely executing a custom function in the pool.
|
1 |
0.42523968335660933 |
Next, let’s explore if we can execute a lambda in the multiprocessing pool.
Example of Calling a Lambda With Pool
A lambda is an anonymous function in Python.
That is, a lambda is a function that may take arguments and may return a value, but is not named.
A lambda is defined using the “lambda” expression. It creates a function-like object that can be assigned and executed
For example, the following lambda expression will sleep for one second.
|
1 2 3 4 5 |
... # create a lambda func = lambda : sleep(1) # execute the lambda func() |
We can create a lambda function object and pass it to the multiprocessing pool to be executed.
For example:
|
1 2 3 4 5 |
... # define lambda function l = lambda : sleep(1) # execute a task pool.apply(l) |
The complete example of passing a lambda to the multiprocessing pool is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# SuperFastPython.com # example of using a lambda with the multiprocessing pool from time import sleep from multiprocessing import Pool # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # define lambda function l = lambda : sleep(1) # execute a task pool.apply(l) |
Running the example first creates the multiprocessing pool with the default number of workers.
The lambda is then created and assigned a local variable.
The lambda function object is then passed to the multiprocessing pool for execution.
This fails with an exception.
Specifically, a PicklingError exception is raised.
The reason that a PicklingError is raised is that lambda functions cannot be pickled.
This is mentioned in the API documentation for the pickle module:
This is why lambda functions cannot be pickled: all lambda functions share the same name: <lambda>.
Any function passed to the workers in the multiprocessing pool must be pickled in order to be transmitted to the child processes.
This highlights that we cannot use lambda functions with the multiprocessing pool because lambda cannot be pickled.
|
1 2 3 |
Traceback (most recent call last): ... _pickle.PicklingError: Can't pickle <function <lambda> at 0x10fe39120>: attribute lookup <lambda> on __main__ failed |
Free Python Multiprocessing Pool Course
Download your FREE Process Pool PDF cheat sheet and get BONUS access to my free 7-day crash course on the Process Pool API.
Discover how to use the Multiprocessing Pool including how to configure the number of workers and how to execute tasks asynchronously.
Example of Calling a Nested Inner Function With Pool
An inner function in Python is a function that is defined within a function.
Yes, you can do that.
As such, they are sometimes called nested inner functions or sub-functions, because the function is nested or defined within another function.
We can explore whether we can execute a task defined in an inner function using the multiprocessing Pool.
In the example below, we will define a main() function for running our program and define our task() function within our main() function.
We will then attempt to issue a call to our inner task() function using the multiprocessing Pool.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of executing an inner function in the multiprocessing pool from time import sleep from random import random from multiprocessing import Pool # entry point def main(): # define a nested inner function def task(): # generate a random value value = random() # block for a moment sleep(value) # return a value return value # create the process pool with Pool() as pool: # execute a task result = pool.apply(task) # report the result print(result) # protect the entry point if __name__ == '__main__': main() |
Running the example first defines the task() function within the main() function.
The main function then creates the multiprocessing pool with the default number of workers.
The inner task() function is then issued to the multiprocessing Pool.
This fails with an AttributeError exception.
The inner function cannot be pickled and therefore cannot be executed within the multiprocessing Pool.
This is similar to the reason why lambda functions cannot be executed in the pool.
|
1 2 3 |
Traceback (most recent call last): ... AttributeError: Can't pickle local object 'main.<locals>.task' |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Calling a Method With Pool
We can define a class with methods.
An instance of the class can be executed and methods executed, just like functions.
This is typical object-oriented programming supported by many modern programming languages.
We can explore whether we can execute a task method in the multiprocessing Pool.
In this example, we will define a custom class with a task() method.
We will then create an instance of the object and attempt to execute the task() method using the multiprocessing Pool.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of executing a method in a multiprocessing pool from time import sleep from random import random from multiprocessing import Pool # custom class class CustomClass(): # do work def task(self): # generate a random value value = random() # block for a moment sleep(value) # return a value return value # protect the entry point if __name__ == '__main__': # create the custom object obj = CustomClass() # create the process pool with Pool() as pool: # execute a task result = pool.apply(obj.task) # report the result print(result) |
Running the example first creates an instance of the custom class.
A multiprocessing Pool is created with the default number of worker processes.
The task() method is then issued to the multiprocessing Pool and the result is reported.
This transmits a copy of our CustomClass object to the child worker process, executes the method, then returns the result.
In this case, the method executes normally without incident.
This highlights that we can execute object methods in the multiprocessing Pool.
|
1 |
0.05885682554555838 |
A word of caution. Instance variables are copied between processes, but changes to the instance variables are not transmitted.
You can learn more about this issue in the tutorial:
Example of Calling a Static Class Method With Pool
A class can define static methods.
These are methods defined on the class itself and do not require an instance of the class in order to execute.
We can explore whether we can execute static class methods as task functions in the multiprocessing pool.
In the example below, we define a CustomClass class with a task() static method.
We then create a multiprocessing Pool and attempt to issue a task to the pool that executes the static method.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of executing a static class method in a multiprocessing pool from time import sleep from random import random from multiprocessing import Pool # custom class class CustomClass(): # do work @staticmethod def task(): # generate a random value value = random() # block for a moment sleep(value) # return a value return value # protect the entry point if __name__ == '__main__': # create the process pool with Pool() as pool: # execute a task result = pool.apply(CustomClass.task) # report the result print(result) |
Running the example first creates the multiprocessing Pool with the default number of worker processes.
Next, the static task() method is then issued as a task to the multiprocessing Pool.
The task is executed and the return value is reported.
The static method is executed in the multiprocessing pool without incident.
This highlights that we can define static class methods and execute them in the multiprocessing Pool directly.
|
1 |
0.1767446574408864 |
Further Reading
This section provides additional resources that you may find helpful.
Books
- Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Pool Class API Cheat Sheet
I would also recommend specific chapters from these books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPool: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to execute different function types in the multiprocessing pool.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Cali Naughton on Unsplash
Do you have any questions?