Last Updated on September 12, 2022
The ThreadPoolExecutor allows you to submit tasks using any type of Python function.
In this tutorial you will discover how to submit tasks to the ThreadPoolExecutor using all different function types.
Let’s get started.
ThreadPoolExecutor Function Types
The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.
You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread.
Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.
You can also submit tasks by calling the map() function and specifying the name of the function to execute and the iterable of items to which your function will be applied.
There are many different function types in Python, such as:
- Regular user-defined functions.
- Built-in functions.
- Anonymous lambda functions.
- Object instance methods.
- Object static methods.
How do you submit tasks to the ThreadPoolExecutor using each function type?
Let’s explore each case using a small contrived task of squaring an input value.
Run loops using all CPUs, download your FREE book to learn how.
User-Defined Functions as Tasks
We can execute tasks to the ThreadPoolExecutor by calling submit() or map() with a user-defined function.
For example, we can define a function that calculates the square of a number.
|
1 2 3 |
# a task as a regular function def task(value): return value**2 |
We can then create a thread pool and call submit to execute the function with a given argument, in this case, 10.
|
1 2 3 4 5 6 7 |
... # create a thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(task, 10) # get the result print(future.result()) |
Tying this together, the complete example of calling submit() to execute a user-defined function in the ThreadPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# SuperFastPython.com # example of executing a task with submit in the thread pool from concurrent.futures import ThreadPoolExecutor # a task as a regular function def task(value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(task, 10) # get the result print(future.result()) |
Running the example creates the thread pool and executes the single task, reporting the result.
|
1 |
100 |
We can also use map() to execute user-defined functions as tasks in the thread pool.
The function name can be specified in the call to map() along with the iterable of arguments for each function call, in this case, numbers from 0 to 9.
|
1 2 3 4 |
... # submit the tasks and process results for result in executor.map(task, range(10)): print(result) |
Tying this together, the complete example of calling map() to execute multiple tasks with a user-defined function in the ThreadPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# SuperFastPython.com # example of executing a task with map in the thread pool from concurrent.futures import ThreadPoolExecutor # a task as a regular function def task(value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # submit the tasks and process results for result in executor.map(task, range(10)): print(result) |
Running the example creates the thread pool and executes all ten tasks, reporting the results as the squares of the inputs.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 4 9 16 25 36 49 64 81 |
Built-In Functions as Tasks
We can execute tasks to the ThreadPoolExecutor by calling submit() or map() with a built-in function.
You might recall that built-in functions are those functions provided by the Python interpreter directly, such as print() and pow().
The built-in function can be specified as the task directly when calling the submit() function.
For example, we can use the pow() built-in function to square a given number. In this case, the number 3.
|
1 2 3 |
... # submit the task future = executor.submit(pow, 3, 2) |
Tying this together, the complete example of executing a built-in function task with submit() is listed below.
|
1 2 3 4 5 6 7 8 9 10 |
# SuperFastPython.com # example of executing a built-in function with submit from concurrent.futures import ThreadPoolExecutor # create a thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(pow, 3, 2) # get the result print(future.result()) |
Running the example creates the thread pool and executes the built-in function calculating the square of 3, that is 3 raised to the power 2.
|
1 |
9 |
We can also execute built-in functions using the map() function on the ThreadPoolExecutor.
For example, if we wanted to call the pow() built-in function to square the numbers 0 to 9, we can specify the name of the function, the range of inputs, and a list that repeats the value of two for each call to the function.
|
1 2 3 4 |
... # submit the tasks and process results for result in executor.map(pow, range(10), [2]*10): print(result) |
Tying this together, the complete example of executing a built-in function using map() in the ThreadPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 |
# SuperFastPython.com # example of executing a built-in function with map from concurrent.futures import ThreadPoolExecutor # create a thread pool with ThreadPoolExecutor() as executor: # submit the tasks and process results for result in executor.map(pow, range(10), [2]*10): print(result) |
Running the example creates the thread pool and executes all ten calls to the built-in function, then prints the results as the square of numbers from 0 to 9.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 4 9 16 25 36 49 64 81 |
Free Python ThreadPoolExecutor Course
Download your FREE ThreadPoolExecutor PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPoolExecutor API.
Discover how to use the ThreadPoolExecutor class including how to configure the number of workers and how to execute tasks asynchronously.
Anonymous Lambda Functions as Tasks
Python supports anonymous functions, called lambda functions or lambda expressions.
These are functions that are defined in-line using the syntax:
|
1 |
lambda <arguments> : <expression> |
These expressions are also called anonymous functions in some programming languages, as they are functions that do not have a name.
We can execute tasks to the ThreadPoolExecutor by calling submit() or map() with a lambda function.
A lambda function can be defined that squares a given input number as follows:
|
1 |
lambda x : x**2 |
We can pass this expression directly in a call to submit() on the ThreadPoolExecutor with an argument; for example:
|
1 2 3 |
... # submit the task future = executor.submit(lambda x : x**2, 10) |
The complete example of executing an anonymous function as a task in the ThreadPoolExecutor using the submit() function is listed below.
|
1 2 3 4 5 6 7 8 9 10 |
# SuperFastPython.com # example executing a lambda function task with submit from concurrent.futures import ThreadPoolExecutor # create a thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(lambda x : x**2, 10) # get the result print(future.result()) |
Running the example creates the thread pool then calls submit() to execute the lambda expression in the thread pool to calculate the square of 10.
|
1 |
100 |
We can also execute lambda in the thread pool using the map() function with an iterable.
The lambda expression can be passed to the call to map() in place of a named function, and an iterable can be provided as a second argument to map, each value of which will be passed to the lambda as a separate task to execute.
|
1 2 3 4 |
... # submit the tasks and process results for result in executor.map(lambda x : x**2, range(10)): print(result) |
Tying this together, the complete example of executing a lambda expression using map() in the ThreadPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 |
# SuperFastPython.com # example executing a lambda function task with map from concurrent.futures import ThreadPoolExecutor # create a thread pool with ThreadPoolExecutor() as executor: # submit the tasks and process results for result in executor.map(lambda x : x**2, range(10)): print(result) |
Running the example creates the thread pool, then executes ten tasks in the thread pool using the lambda expression, calculating the square of integer values from 0 to 9.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 4 9 16 25 36 49 64 81 |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Object Instance Methods as Tasks
Python supports object-oriented programming with the definition of custom classes.
A function that belongs to an instance of an object is called an instance function or a method.
We can execute tasks to the ThreadPoolExecutor by calling submit() or map() with an instance method.
First, let’s define a custom class that has an instance method that will calculate the square of an input number.
|
1 2 3 4 5 |
# define a custom class class CustomClass(object): # define a method def task(self, value): return value**2 |
We can then create an instance of this custom class and have the function on the instance executed as a task in the ThreadPoolExecutor by calling submit.
For example:
|
1 2 3 |
... # submit the task future = executor.submit(CustomClass().task, 10) |
Note: just like with a user-defined function or a built-in function, we must specify the name of the function to execute, but in this case, the name of the function must be associated with an instance of the class (an instantiated example of the class).
Tying this together, the complete example of executing an instance method as a task in the ThreadPoolExecutor using the submit() function is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# SuperFastPython.com # example executing an instance method task with submit from concurrent.futures import ThreadPoolExecutor # define a custom class class CustomClass(object): # define a method def task(self, value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(CustomClass().task, 10) # get the result print(future.result()) |
Running the example creates the thread pool and executes the method on the class instance, calculating the square value.
|
1 |
100 |
We can also execute an instance method using the map() function on the thread pool.
Like submit, we must specify the name of the function associated with a specific instance of the class when calling map(). We can then specify an iterable as the second argument to map() that contains a list of items, each of which is passed to the instance function to form a task to be executed in the thread pool.
For example, we can call the instance function to square the numbers between 0 and 9:
|
1 2 3 4 |
... # submit the tasks and process results for result in executor.map(CustomClass().task, range(10)): print(result) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# SuperFastPython.com # example executing an instance method task with map from concurrent.futures import ThreadPoolExecutor # define a custom class class CustomClass(object): # define a method def task(self, value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # submit the tasks and process results for result in executor.map(CustomClass().task, range(10)): print(result) |
Running the example creates the thread pool and then submits ten tasks, then reports the results as the squared values.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 4 9 16 25 36 49 64 81 |
In this example, a single instance of the class executed each task.
Perhaps we wanted to use a separate instance of the user-defined class to execute each task in a call to map().
This could be achieved by first defining a list of instances of the custom class.
We can then call the map() function with the list of instances and the list of arguments, one for each instance.
The trick is to then use a lambda expression that will take two arguments, one for the instance and one for the argument to the instance, then call the function on the instance with the argument.
For example:
|
1 2 3 4 5 6 |
... # create a list of objects objects = [CustomClass() for _ in range(10)] # submit the tasks and process results for result in executor.map(lambda x,i : x.task(i), objects, range(10)): print(result) |
Tying this together, the complete example of executing an instance method on separate instances of a user defined class in the thread pool using map() is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example executing a method on a different instance for each task with map from concurrent.futures import ThreadPoolExecutor # define a custom class class CustomClass(object): # define a method def task(self, value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # create a list of objects objects = [CustomClass() for _ in range(10)] # submit the tasks and process results for result in executor.map(lambda x,i : x.task(i), objects, range(10)): print(result) |
Running the example creates the thread pool then submits the ten tasks into the thread pool, each of which is a lambda that executes an instance method on a separate instance of the user defined class.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 4 9 16 25 36 49 64 81 |
Object Static Methods as Tasks
A class on object-oriented programming may also have a function that does not require an object instance, called a static method.
A static method is executed using the name of the custom class but does not require an instance of the class as with an instance method.
As such, we can execute a class static method as a task in the ThreadPoolExecutor using submit() and map() just like a user-defined function or a built-in function.
First, we can define a static method on a user defined class that will square a given numeric input value.
Note the @staticmethod decorator is used to designate that a function on a class is a static method that can also be executed as an instance method, if we so choose. It’s a good practice to add it to static methods.
|
1 2 3 4 5 6 |
# define a custom class class CustomClass(object): # define a static method @staticmethod def task(value): return value**2 |
We can then call this static method directly by dereferencing it via the name of the class when calling the submit() function.
For example:
|
1 2 3 |
... # submit the task future = executor.submit(CustomClass.task, 10) |
Tying this together, the complete example of executing a static method as a task in the ThreadPoolExecutor using the submit() function is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example executing an static method task with submit from concurrent.futures import ThreadPoolExecutor # define a custom class class CustomClass(object): # define a static method @staticmethod def task(value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # submit the task future = executor.submit(CustomClass.task, 10) # get the result print(future.result()) |
Running the example creates the thread pool, then executes the static method as a task in the thread pool, calculating the square of the input value.
|
1 |
100 |
We can also execute a static method using the map() for an iterable of input values.
Again, this is the same as executing a task using a user-defined function, except that the function is dereferenced using the name of the user-defined class.
|
1 2 3 4 |
... # submit the tasks and process results for result in executor.map(CustomClass.task, range(10)): print(result) |
Tying this together, the complete example of executing a static method using map() in the ThreadPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# SuperFastPython.com # example executing an static method task with map from concurrent.futures import ThreadPoolExecutor # define a custom class class CustomClass(object): # define a static method @staticmethod def task(value): return value**2 # create a thread pool with ThreadPoolExecutor() as executor: # submit the tasks and process results for result in executor.map(CustomClass.task, range(10)): print(result) |
Running the example creates the thread pool and executes ten tasks in the thread pool, one for each call to the static method with an argument from the iterable, squaring each input value.
|
1 2 3 4 5 6 7 8 9 10 |
0 1 4 9 16 25 36 49 64 81 |
Further Reading
This section provides additional resources that you may find helpful.
Books
- ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
- Concurrent Futures API Interview Questions
- ThreadPoolExecutor Class API Cheat Sheet
I also recommend specific chapters from the following books:
- Effective Python, Brett Slatkin, 2019.
- See Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPoolExecutor: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Takeaways
You now know how to submit tasks to the ThreadPoolExecutor using all types of Python functions.
Do you have any questions about how to submit tasks to thread pools?
Ask your questions in the comments below and I will do my best to answer.
Photo by Rokas Niparas on Unsplash
Do you have any questions?