How to Get Results From The ProcessPoolExecutor in Python

January 15, 2022 Python ProcessPoolExecutor

You can get results from tasks in the ProcessPoolExecutor by calling the result() function.

In this tutorial you will discover how to get results from tasks submitted to the ProcessPoolExecutor in Python.

Let's get started.

Need to Get Results From Tasks

The ProcessPoolExecutor in Python provides a pool of reusable processes for executing ad hoc tasks.

You can submit tasks to the process pool by calling the submit() function and passing in the name of the function you wish to execute on another process.

Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.

You can also submit tasks by calling the map() function and specify the name of the function to execute and the iterable of items to which your function will be applied.

Tasks submitted to the process pool are defined by a custom function that may return a value. The returned value is likely the result of the completed task.

You need a way to get the results from completed tasks.

How do you get results from tasks submitted to the ProcessPoolExecutor?

How To Get Results From Tasks

You can get the result returned by a target task function by calling the result() function on the Future object associated with the task.

Recall that you can submit tasks to the process pool by calling submit() which will return a Future object.

If your target task function returns a value once the task is complete, you can retrieve it directly by calling the result() function.

...
# get the result of a task
result = future.result()

The result() function is a blocking call.

This means that if the task is not yet done because it is scheduled (queued) or is running, then the result() task will not return until the task is complete.

You can set a time limit in seconds for how long you are willing to wait for a result by setting the "timeout" argument. By default there is no timeout when waiting for a result.

...
# get the result of a task with a timeout
result = future.result(timeout=5)

If the timeout is exceeded before a result is available, then a TimeoutError will be raised by the call to result() which you may have to handle.

If the task target function raises an exception while executing, then the exception will be re-raised by the process pool when calling the result() function, which may need to be handled.

...
# get the result and handle a known possible exception
try:
	result = future.result()
exception:
	# handle exception...

Alternatively, you may have submitted tasks to the process pool via the map() function.

Recall that this is an asynchronous version of the built-in map() function and will apply a specified function to each item in a provided iterable.

The map() function on the ProcessPoolExecutor will return an iterable for results that you can enumerate directly to get the return value from each task. The iterator will progress as results become available in the order that tasks were submitted to the process pool.

...
# submit tasks and get results
for result in executor.map(work, range(3)):
	# process result...

Now that we know how to get results from tasks submitted to the ProcessPoolExecutor, let's look at a worked example.

Example of Getting a Task Result

Let's take a look at a worked example of getting results from the ProcessPoolExecutor.

First, let's define a simple task that will block for a moment and will return a string.

# task that will sleep for a moment
def work():
    sleep(1)
    return 'Task is complete'

Next, we can start a process pool using the context manager, and submit a task by calling submit() that will return a Future object immediately.

...
# create a process pool
with ProcessPoolExecutor() as executor:
    # execute the task
    future = executor.submit(work)

We can then call the result() function on the Future object that will block until the task is completed and return the result from the task.

...
# get the result from the task
result = future.result()

We can then handle the result, in this case simply print it.

...
# report the result
print(result)

Tying this together, the complete example of getting a result from a task in the ProcessPoolExecutor is listed below.

# SuperFastPython.com
# example of getting results from a task in the process pool
from time import sleep
from concurrent.futures import ProcessPoolExecutor

# mock task that will sleep for a moment
def work():
    sleep(1)
    return 'Task is complete'

# entry point
if __name__ == '__main__':
    # create a process pool
    with ProcessPoolExecutor() as executor:
        # execute the task
        future = executor.submit(work)
        # get the result from the task
        result = future.result()
        # report the result
        print(result)

Running the example creates the process pool and submits the task.

We then attempt to get the result and wait for the task to complete before the result is available.

Once the result is available we report it to the user by calling print.

Task is complete

Example of Getting a Result with a Timeout

We can also set a time limit on how long we are willing to wait for a result.

This can be achieved by setting the "timeout" argument to a value in seconds when calling the result() function.

If the specified timeout elapses before the task is done and result is available then a TimeoutError is raised which may need to be handled.

...
try:
    # get the result from the task
    result = future.result(timeout=0.5)
    # report the result
    print(result)
except TimeoutError:
    print('Waited too long for a result')

Tying this together, the example below demonstrates how to get a result from a task in the ProcessPoolExecutor but only waiting a fixed time for the result.

# SuperFastPython.com
# example of getting results from a task in the process pool
from time import sleep
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import TimeoutError

# mock task that will sleep for a moment
def work():
    sleep(1)
    return 'Task is complete'

if __name__ == '__main__':
    # create a process pool
    with ProcessPoolExecutor() as executor:
        # execute the task
        future = executor.submit(work)
        try:
            # get the result from the task
            result = future.result(timeout=0.5)
            # report the result
            print(result)
        except TimeoutError:
            print('Waited too long for a result')

Running the example waits a moment for a result before giving up, as we expected.

The TimeoutError is handled and a message is reported that we were unable to get the result in time.

Waited too long for a result

Example of Getting a Result With map()

You can also get the result for tasks submitted to the process pool by calling the map() function.

The results are provided in an iterable returned by calling map() that will return the results from tasks as they become available in the order that tasks were submitted to the process pool.

We can update our task to take an argument and to include the argument in the result returned by the function.

# task that will sleep for a moment
def work(value):
    sleep(1)
    return f'Task is done: {value}'

We can then submit tasks to the process pool by calling map() with an iterable of arguments, one for each task to submit.

The map() function returns an iterable of results immediately, and will yield results as they become available for the tasks in order.

...
# submit tasks and process results
for result in executor.map(work, range(3)):
    print(result)

Tying this together, the complete example of getting results from tasks in the ProcessPoolExecutor submitted by calling map() is listed below.

# SuperFastPython.com
# example of getting results when calling map
from time import sleep
from concurrent.futures import ProcessPoolExecutor

# mock task that will sleep for a moment
def work(value):
    sleep(1)
    return f'Task is done: {value}'

# entry point
if __name__ == '__main__':
    # create a process pool
    with ProcessPoolExecutor() as executor:
        # submit tasks and process results
        for result in executor.map(work, range(3)):
            print(result)

Running the example, the process pool is created as per normal.

Three tasks are then submitted into the process pool and we iterate over their results in order as they become available.

Task is done: 0
Task is done: 1
Task is done: 2

Takeaways

You now know how to get results from tasks submitted to the ProcessPoolExecutor.

If you enjoyed this tutorial, you will love my book: Python ProcessPoolExecutor Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.