Last Updated on September 12, 2022
You can get results from tasks in the ProcessPoolExecutor by calling the result() function.
In this tutorial you will discover how to get results from tasks submitted to the ProcessPoolExecutor in Python.
Let’s get started.
Need to Get Results From Tasks
The ProcessPoolExecutor in Python provides a pool of reusable processes for executing ad hoc tasks.
You can submit tasks to the process pool by calling the submit() function and passing in the name of the function you wish to execute on another process.
Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.
You can also submit tasks by calling the map() function and specify the name of the function to execute and the iterable of items to which your function will be applied.
Tasks submitted to the process pool are defined by a custom function that may return a value. The returned value is likely the result of the completed task.
You need a way to get the results from completed tasks.
How do you get results from tasks submitted to the ProcessPoolExecutor?
Run loops using all CPUs, download your FREE book to learn how.
How To Get Results From Tasks
You can get the result returned by a target task function by calling the result() function on the Future object associated with the task.
Recall that you can submit tasks to the process pool by calling submit() which will return a Future object.
If your target task function returns a value once the task is complete, you can retrieve it directly by calling the result() function.
|
1 2 3 |
... # get the result of a task result = future.result() |
The result() function is a blocking call.
This means that if the task is not yet done because it is scheduled (queued) or is running, then the result() task will not return until the task is complete.
You can set a time limit in seconds for how long you are willing to wait for a result by setting the “timeout” argument. By default there is no timeout when waiting for a result.
|
1 2 3 |
... # get the result of a task with a timeout result = future.result(timeout=5) |
If the timeout is exceeded before a result is available, then a TimeoutError will be raised by the call to result() which you may have to handle.
If the task target function raises an exception while executing, then the exception will be re-raised by the process pool when calling the result() function, which may need to be handled.
|
1 2 3 4 5 6 |
... # get the result and handle a known possible exception try: result = future.result() exception: # handle exception... |
Alternatively, you may have submitted tasks to the process pool via the map() function.
Recall that this is an asynchronous version of the built-in map() function and will apply a specified function to each item in a provided iterable.
The map() function on the ProcessPoolExecutor will return an iterable for results that you can enumerate directly to get the return value from each task. The iterator will progress as results become available in the order that tasks were submitted to the process pool.
|
1 2 3 4 |
... # submit tasks and get results for result in executor.map(work, range(3)): # process result... |
Now that we know how to get results from tasks submitted to the ProcessPoolExecutor, let’s look at a worked example.
Example of Getting a Task Result
Let’s take a look at a worked example of getting results from the ProcessPoolExecutor.
First, let’s define a simple task that will block for a moment and will return a string.
|
1 2 3 4 |
# task that will sleep for a moment def work(): sleep(1) return 'Task is complete' |
Next, we can start a process pool using the context manager, and submit a task by calling submit() that will return a Future object immediately.
|
1 2 3 4 5 |
... # create a process pool with ProcessPoolExecutor() as executor: # execute the task future = executor.submit(work) |
We can then call the result() function on the Future object that will block until the task is completed and return the result from the task.
|
1 2 3 |
... # get the result from the task result = future.result() |
We can then handle the result, in this case simply print it.
|
1 2 3 |
... # report the result print(result) |
Tying this together, the complete example of getting a result from a task in the ProcessPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of getting results from a task in the process pool from time import sleep from concurrent.futures import ProcessPoolExecutor # mock task that will sleep for a moment def work(): sleep(1) return 'Task is complete' # entry point if __name__ == '__main__': # create a process pool with ProcessPoolExecutor() as executor: # execute the task future = executor.submit(work) # get the result from the task result = future.result() # report the result print(result) |
Running the example creates the process pool and submits the task.
We then attempt to get the result and wait for the task to complete before the result is available.
Once the result is available we report it to the user by calling print.
|
1 |
Task is complete |
Free Python ProcessPoolExecutor Course
Download your FREE ProcessPoolExecutor PDF cheat sheet and get BONUS access to my free 7-day crash course on the ProcessPoolExecutor API.
Discover how to use the ProcessPoolExecutor class including how to configure the number of workers and how to execute tasks asynchronously.
Example of Getting a Result with a Timeout
We can also set a time limit on how long we are willing to wait for a result.
This can be achieved by setting the “timeout” argument to a value in seconds when calling the result() function.
If the specified timeout elapses before the task is done and result is available then a TimeoutError is raised which may need to be handled.
|
1 2 3 4 5 6 7 8 |
... try: # get the result from the task result = future.result(timeout=0.5) # report the result print(result) except TimeoutError: print('Waited too long for a result') |
Tying this together, the example below demonstrates how to get a result from a task in the ProcessPoolExecutor but only waiting a fixed time for the result.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of getting results from a task in the process pool from time import sleep from concurrent.futures import ProcessPoolExecutor from concurrent.futures import TimeoutError # mock task that will sleep for a moment def work(): sleep(1) return 'Task is complete' if __name__ == '__main__': # create a process pool with ProcessPoolExecutor() as executor: # execute the task future = executor.submit(work) try: # get the result from the task result = future.result(timeout=0.5) # report the result print(result) except TimeoutError: print('Waited too long for a result') |
Running the example waits a moment for a result before giving up, as we expected.
The TimeoutError is handled and a message is reported that we were unable to get the result in time.
|
1 |
Waited too long for a result |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Getting a Result With map()
You can also get the result for tasks submitted to the process pool by calling the map() function.
The results are provided in an iterable returned by calling map() that will return the results from tasks as they become available in the order that tasks were submitted to the process pool.
We can update our task to take an argument and to include the argument in the result returned by the function.
|
1 2 3 4 |
# task that will sleep for a moment def work(value): sleep(1) return f'Task is done: {value}' |
We can then submit tasks to the process pool by calling map() with an iterable of arguments, one for each task to submit.
The map() function returns an iterable of results immediately, and will yield results as they become available for the tasks in order.
|
1 2 3 4 |
... # submit tasks and process results for result in executor.map(work, range(3)): print(result) |
Tying this together, the complete example of getting results from tasks in the ProcessPoolExecutor submitted by calling map() is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of getting results when calling map from time import sleep from concurrent.futures import ProcessPoolExecutor # mock task that will sleep for a moment def work(value): sleep(1) return f'Task is done: {value}' # entry point if __name__ == '__main__': # create a process pool with ProcessPoolExecutor() as executor: # submit tasks and process results for result in executor.map(work, range(3)): print(result) |
Running the example, the process pool is created as per normal.
Three tasks are then submitted into the process pool and we iterate over their results in order as they become available.
|
1 2 3 |
Task is done: 0 Task is done: 1 Task is done: 2 |
Further Reading
This section provides additional resources that you may find helpful.
Books
- ProcessPoolExecutor Jump-Start, Jason Brownlee (my book!)
- Concurrent Futures API Interview Questions
- ProcessPoolExecutor PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Effective Python, Brett Slatkin, 2019.
- See Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ProcessPoolExecutor: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python Pool: The Complete Guide
APIs
References
- Thread (computing), Wikipedia.
- Process (computing), Wikipedia.
- Thread Pool, Wikipedia.
- Futures and promises, Wikipedia.
Takeaways
You now know how to get results from tasks submitted to the ProcessPoolExecutor.
Do you have any questions?
Ask your question in the comments below and I will do my best to answer.
Photo by Amarnath Tade on Unsplash
Do you have any questions?