Last Updated on October 18, 2022

You can issue asynchronous tasks to the process pool which will return a multiprocessing.pool.AsyncResult object.

The AsyncResult provides a handle or issued tasks in the process pool and can be used to check on the status of the tasks and to get task results.

In this tutorial you will discover how to use the AsyncResult in Python.

Let’s get started.

What is the AsyncResult

An multiprocessing.pool.AsyncResult object is returned when issuing tasks to multiprocessing.pool.Pool the process pool asynchronously.

This can be achieved via any of the following methods on the process pool:

• Pool.apply_async() to issue one task.
• Pool.map_async() to issue multiple tasks.
• Pool.starmap_async() to issue multiple tasks that take multiple arguments.

A AsyncResult provides a handle on one or more issued tasks.

It allows the caller to check on the status of the issued tasks, to wait for the tasks to complete, and to get the results once tasks are completed.

How to Use the AsyncResult

The AsyncResult class is straightforward to use.

First, you must get an AsyncResult object by issuing one or more tasks to the process pool using any of the apply_async(), map_async(), or starmap_async() functions.

For example:

Once you have an AsyncResult object, you can use it to query the status and get results from the task.

How to Get a Result

We can get the result of an issued task by calling the AsyncResult.get() function.

Return the result when it arrives.

— multiprocessing — Process-based parallelism

This will return the result of the specific function called to issue the task.

• apply_async(): Returns the return value of the target function.
• map_async(): Returns an iterable over the return values of the target function.
• starmap_async(): Returns an iterable over the return values of the target function.

For example:

If the issued tasks have not yet completed, then get() will block until the tasks are finished.

A “timeout” argument can be specified. If the tasks are still running and do not complete within the specified number of seconds, a multiprocessing.TimeoutError is raised.

If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised.

— multiprocessing — Process-based parallelism

For example:

If an issued task raises an exception, the exception will be re-raised once the issued tasks are completed.

We may need to handle this case explicitly if we expect a task to raise an exception on failure.

If the remote call raised an exception then that exception will be re-raised by get().

— multiprocessing — Process-based parallelism

For example:

How to Wait For Task to Finish

We can wait for all tasks to complete via the AsyncResult.wait() function.

This will block until all issued tasks are completed.

For example:

If the tasks have already completed, then the wait() function will return immediately.

A “timeout” argument can be specified to set a limit in seconds for how long the caller is willing to wait.

Wait until the result is available or until timeout seconds pass.

— multiprocessing — Process-based parallelism

If the timeout expires before the tasks are complete, the wait() function will return.

When using a timeout, the wait() function does not give an indication that it returned because tasks completed or because the timeout elapsed. Therefore, we can check if the tasks completed via the ready() function.

For example:

How to Check if Tasks Are Completed

We can check if the issued tasks are completed via the AsyncResult.ready() function.

Return whether the call has completed.

— multiprocessing — Process-based parallelism

It returns True if the tasks have completed, successfully or otherwise, or False if the tasks are still running.

For example:

How to Check if Tasks Were Successful

We can check if the issued tasks completed successfully via the AsyncResult.successful() function.

Issued tasks are successful if no tasks raised an exception.

If at least one issued task raised an exception, then the call was not successful and the successful() function will return False.

This function should be called after it is known that the tasks have completed, e.g. ready() returns True.

For example:

If the issued tasks are still running, a ValueError is raised.

Return whether the call completed without raising an exception. Will raise ValueError if the result is not ready.

— multiprocessing — Process-based parallelism

For example:

Now that we know how to use the AsyncResult, let’s look at some worked examples.

How to Get a AsyncResult

We can get an AsyncResult object via calling any of the apply_async(), map_async(), or starmap_async() functions to issue tasks to the process pool.

Let’s take a look at each example in turn.

Example of Getting a AsyncResult via apply_async()

We can explore how to get an AsyncResult object by issuing tasks using the apply_async() function.

In this example we will define a task function that takes an integer as an argument, generates a random number, reports the values, blocks for a fraction of a second, then returns the generated value.

Firstly, we can define the task function that takes an argument and returns the generated value.

The task() function below implements this.

Next, in the main thread we can issue a task using the apply_async() function. We can specify the task() function and an argument of 0 via the “args” argument.

This will return an AsyncResult object immediately.

In the main thread, we can create the process pool. We will use the context manager interface to ensure the pool is terminated automatically once we are finished with it, and use the default configuration.

You can learn more about creating a process pool using the context manager interface in the tutorial:

• Process Pool Context Manager

You can learn more about how to use the apply_async() function in the tutorial:

• Multiprocessing Pool.apply_async() in Python

Finally, we can close the process pool to prevent any further tasks being issued to the process pool, then join the pool to wait for all issued tasks to complete.

You can learn more about joining the process pool in the tutorial:

• Join a Process Pool in Python

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then closes the process pool and waits for all issued tasks to complete.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

The issued task completes, and the main process continues on and terminates the program.

Example of Getting a AsyncResult via map_async()

We can explore how to get an AsyncResult object by issuing tasks using the map_async() function.

In this example, we can update the previous example to issue multiple calls to the task() function with different arguments from 0 to 9.

This can be achieved by calling the map_async() and specifying the task() function and a range of arguments.

This call will return immediately with an AsyncResult object.

You can learn more about how to use the map_async() function in the tutorial:

• Multiprocessing Pool.map_async() in Python

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A total of 10 tasks are issued to the process pool and a single AsyncResult object is returned immediately.

The main process then closes the process pool and waits for all issued tasks to complete.

The issued tasks are executed by the process pool. Each task generates a random number, reports a message, blocks for a moment, then returns the generated value.

All issued tasks complete and the main process continues on and terminates the program.

Example of Getting a AsyncResult via starmap_async()

We can explore how to get an AsyncResult object by issuing tasks using the starmap_async() function.

In this example, we can update the previous example so that the task() function takes two arguments and that we issue multiple calls to the task() function with two arguments per function call.

This can be achieved by calling the starmap_async().

First, we must update the task() function to take two arguments. In this case it will take the integer number as before, and generate a floating point value between 0 and 1, instead of generating it itself.

The updated task() function is listed below.

Next, we can prepare arguments for each call to the task() function.

In this case, we will use a list comprehension to prepare a list of arguments, where each item in the list is a tuple of arguments for a call to the task() function.

We can then issue tasks to the process pool by specifying the task() function and the list of arguments for each task.

This call will return immediately with an AsyncResult object.

You can learn more about how to use the starmap_async() function in the tutorial:

• Multiprocessing Pool.starmap_async() in Python

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A total of 10 tasks are issued to the process pool and a single AsyncResult object is returned immediately.

The main process then closes the process pool and waits for all issued tasks to complete.

The issued tasks are executed by the process pool. Each task reports a message, blocks for a moment, then returns the floating point value.

All issued tasks complete and the main process continues on and terminates the program.

Now that we know how to get a AsyncResult, let’s look at how we might get results from completed tasks

How to Get a Result From a AsyncResult

In this section we will explore how to get a result for issued tasks via an AsyncResult.

In these examples, we will use the apply_async() function to issue one task as a pretext for exploring how to get a result. Nevertheless, the examples are general and just as applicable to getting results from an AsyncResult created via map_async() and starmap_async().

Example of Getting an AsyncResult Result

We can explore how to get the results of issued tasks from an AsyncResult.

This can be achieved by updating the previous example that gets an AsyncResult by calling apply_async(), then calling the get() function on an AsyncResult to get the result of the issued task.

For example:

The call will block until the task is complete, then returns the return value from the issued task, in this case from the task() function.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then attempts to get the result from the issued task. It blocks until the task is completed.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

The main process receives the result from the completed task and reports the value with a message.

Example of Getting an AsyncResult Result With a Timeout

We can explore getting a result from an asynchronous task with a timeout.

This can be achieved by updating the previous example and calling the get() function and specifying a “timeout” argument in seconds.

We can use a timeout of half a second, meaning that half the time we will get a result on time, and half the time the timeout will elapse before we get a result.

If the timeout elapses before we get a result, then an exception is raised.

We will handle this exception and report a failure message.

If the exception is raised the task may not complete because the process pool will be terminated by the context manager.

Therefore we can close the pool and wait for all issued tasks to complete.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then attempts to get the result from the issued task with a timeout. It blocks until the task is completed or the timeout elapses.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

In this case, the main process receives the result from the completed task and reports the value with a message.

We can run the example a few times, sometimes the main process will get a result and sometimes it will not.

Rerunning the example again, we may get the failure case.

In this case, the timeout elapses before the task is completed.

An exception is raised by the process pool and caught by the main process. A failure message is reported.

The process pool is then closed and the main process blocks until all issued tasks have completed.

Example of Getting an AsyncResult Result With Exception

We can explore getting a result from issued tasks where at least one task fails with an unhandled exception.

This can be achieved by updating the previous example so that the task function raises an exception. We can then attempt to get the result from the issued task in the main process and handle the exception that will be re-raised by the process pool.

Firstly, we can update the task() function so that it raises an exception.

We can then attempt to get the result from the issued task via the get() function on the AsyncResult.

The process pool will re-raise the exception when we attempt to get the result, therefore we will handle the exception when getting the result.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then attempts to get the result from the issued task. It blocks until the task is completed.

The issued task is executed, generating a random number, reporting a message, blocking, then raises an exception.

The task is completed, but unsuccessfully.

The main process handles the re-raised exception from the completed task and reports the failure with a message.

Next, let’s explore how we might wait for issued tasks to complete.

How to Wait For Tasks to Complete With AsyncResult

We can wait for all tasks associated with an AsyncResult object to complete.

This can be achieved using the wait() function, which may also take a timeout in seconds.

Let’s explore each case of waiting for tasks.

Example of Waiting for a AsyncResult Results

We can explore how to wait for all issued tasks to complete.

This can be achieved by updating the previous example to call the wait() function instead of getting the result.

The wait() function will only return after all issued tasks associated with the AsyncResult object have completed.

Recall, a task is completed if it finishes normally or raises an exception.

For example:

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then waits for the issued task to complete, blocking until the wait() function returns.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

The main process unblocks and continues on, terminating the application.

Example of Waiting for AsyncResult Results with a Timeout

We can explore waiting for all issued tasks to complete with a timeout.

This can be achieved by updating the previous example of waiting for all tasks associated with an AsyncResult to complete, then specifying a “timeout” argument.

We will use a timeout of half a second, meaning that half the time the issued task will complete and the wait() function will return normally and the other half of the time the timeout will elapse and the wait() function will return.

Note, the wait() function does not return a value. This means that it does not give an indication of whether the call returned because the associated tasks completed or because the timeout elapses. Therefore the ready() function should be called to check if the tasks are completed.

Finally, because the wait() function may have returned because the timeout elapsed, we can wait for all issued tasks in the pool to complete by joining the pool.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then waits for the issued task to complete with a timeout, blocking until the wait() function returns or the timeout elapses.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

The main process unblocks after the task completes or the timeout.

The main process then closes the process pool and waits for all issued tasks to complete.

Next, let’s explore how to check the status of tasks issued by the process pool.

How to Check the Status of Tasks via AsyncResult

We can check the status of all tasks issued to a process pool associated with an AsyncResult.

This can be achieved via the ready() function and the successful() function.

Let’s take examples of each in turn.

Example of Checking if AsyncResult Results are Ready

We can explore checking whether issued tasks have completed via the ready() function.

Recall that a completed task may or may not have completed successfully (e.g. raised an exception).

We can update the previous example to sleep for a fraction of a second after issuing the task, then checking if the tasks were completed. We can then report a message if the task is completed and a different message if the task is not completed.

Because the task itself will block for a random fraction of a second we will get a different result each time the program is run.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then blocks for a fraction of a second.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

It then checks if the issued task is completed or not.

In this case, the task was completed by the time the check was performed.

Note, you will get different results each time the program is run given the use of random numbers.

We can explore a failure case.

Running the program again, the main process checks if the issued tasks have completed.

In this case the task has not been completed. A different message is then reported.

The process pool is then closed and the main process blocks until all issued tasks have completed.

Example of Checking if Tasks are Completed Successfully

We can explore whether issued tasks were completed successfully via the successful() function.

We can update the previous example so that we first wait for the issued task to complete by calling the wait() function.

We can then call the successful() function and report a message if the issued task was completed successfully, which we expect it was in this case.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then waits for the issued task to complete, blocking until the wait() function returns.

The issued task is executed, generating a random number, reporting a message, blocking, then returning the generated value.

The main process unblocks and continues on. It checks the success of the tasks associated with the AsyncResult object.

The task was completed successfully and an appropriate message was reported.

Example of Tasks Completed Unsuccessfully

We can explore checking the status of a completed task via the successful() function that we know was not completed successfully.

This can be achieved by updating the previous example so that the task() function raises an exception.

We then expect that checking the success of the issued task will then report a different message.

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then waits for the issued task to complete, blocking until the wait() function returns.

The issued task is executed, generating a random number, reporting a message, blocking, then raises an exception.

The main process unblocks and continues on. It checks the success of the tasks associated with the AsyncResult object.

The task was not completed successfully this time and an appropriate message is reported.

Example of Tasks Not Completed

We can explore checking the status of a task that is not yet complete via the successful() function.

The expectation is that the function will raise an exception as the task is not yet completed.

This can be achieved by updating the example in the previous section to check the successful status of the task immediately after it is issued.

This will raise an exception as the task is not completed

Tying this together, the complete example is listed below.

Running the example first creates the process pool with the default configuration.

A task is issued which returns an AsyncResult object immediately.

The main process then checks the success of the tasks associated with the AsyncResult object.

The task was not yet complete and an exception was raised.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Multiprocessing Pool Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Pool Class API Cheat Sheet

I would also recommend specific chapters from these books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPool: The Complete Guide
• Python Multiprocessing: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know how to use an AsyncResult object associated with tasks issued to the process pool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Vidar Nordli-Mathisen on Unsplash