Last Updated on October 29, 2022

You can specify a custom callback function when using the apply_async(), map_async(), and starmap_async() functions in ThreadPool class via the “callback” argument.

In this tutorial you will discover how to use callback functions with the ThreadPool in Python.

Let’s get started.

Need to Use Callbacks with the ThreadPool

The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.

A thread pool object which controls a pool of worker threads to which jobs can be submitted.

— multiprocessing — Process-based parallelism

The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.

Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.

A ThreadPool can be configured when it is created, which will prepare the new threads.

We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().

When issuing tasks to the ThreadPool asynchronously, we may need to configure a callback function. That is, we may need to have a custom function called automatically with the results of each task.

How can we use callback functions with the ThreadPool in Python?

How to Configure a Callback Function

The ThreadPool supports custom callback functions.

Callback functions are called in two situations:

• With the results of a task.
• When an error is raised in a task.

We are only considering callbacks called with the results of a task, so-called result callbacks.

Result callbacks are supported in the ThreadPool when issuing tasks asynchronously with any of the following functions:

• apply_async(): For issuing a single task asynchronously.
• map_async(): For issuing multiple tasks with a single argument asynchronously.
• starmap_async(): For issuing multiple tasks with multiple arguments asynchronously.

A result callback can be specified via the “callback” argument.

The argument specifies the name of a custom function to call with the result of asynchronous task or tasks.

Note, a configured callback function will be called, even if your task function does not have a return value. In that case, a default return value of None will be passed as an argument to the callback function.

The function may have any name you like, as long as it does not conflict with a function name already in use.

If callback is specified then it should be a callable which accepts a single argument. When the result becomes ready callback is applied to it

— multiprocessing — Process-based parallelism

For example, if apply_async() is configured with a callback, then the callback function will be called with the return value of the task function that was executed.

Alternatively, if map_async() or starmap_async() are configured with a callback, then the callback function will be called with an iterable of return values from all tasks issued to the ThreadPool.

Result callbacks should be used to perform a quick action with the result or results of issued tasks from the ThreadPool.

They should not block or execute for an extended period as they will occupy the resources of the ThreadPool while running.

Callbacks should complete immediately since otherwise the thread which handles the results will get blocked.

— multiprocessing — Process-based parallelism

Now that we know how to configure a result callback function, let’s look at some worked examples.

Example Callback Function for apply_async()

We can explore how to use a result callback with the ThreadPool when issuing tasks via the apply_async() function.

In this example we will define a task that generates a random number, reports the number, blocks for a moment, then returns the value that was generated. A callback function will be defined that receives the return value from the task function and reports the value.

Firstly, we can define the result callback function.

The function takes a single return value from a target task function and reports it directly.

The result_callback() function below implements this.

Next, we can define a target task function.

The function takes a unique integer identifier for the task. It then generates a random number between 0 and 1 and sleeps for a fraction of a second to simulate computational effort. Finally, it returns the random value that was generated.

The task() function below implements this.

We then define a ThreadPool with the default number of worker threads. In this case we use the context manager interface to ensure the ThreadPool closes automatically once we are finished with it.

You can learn more about the context manager interface in the tutorial:

• How to Use the ThreadPool Context Manager

We will then issue the task to the ThreadPool using the apply_async() and specify the result callback function to execute with the result of the task.

• How to Use ThreadPool apply_async() in Python

Finally, the main thread will close the ThreadPool and wait for the issued task to complete.

You can learn more about joining the ThreadPool in the tutorial:

• How to Join a ThreadPool in Python

Tying this together, the complete example is listed below.

Running the example first starts the ThreadPool with the default configuration.

Then the task is issued to the ThreadPool. The main thread then closes the pool and then waits for the issued task to complete.

The task function executes, generating a random number, reporting a message, blocking and returning a value.

The result callback function is then called with the generated value, which is then reported.

The task ends and the main thread wakes up and continues on, closing the program.

Next, let’s look at an example of using a callback function with the map_async() function.

Example Callback Function for map_async()

We can explore how to use a result callback with the ThreadPool when issuing tasks via the map_async() function.

In this example, we can update the previous example to issue multiple calls to the task() function. The callback function will then be called with an iterable of return values from all tasks, that will be iterated and each value reported.

Firstly, we must update the callback function to receive an iterable of return values instead of a single return value. It traverses the return values and reports each value.

The updated result_callback() function is listed below.

The task function does not need any change and can be used as-is.

Next, in the main thread, we will call the map_async() function to issue five calls to the task function with integer values from 0 to 9 and to call the callback function once all tasks are completed.

You can learn more about the map_async() function in the tutorial:

• How to Use ThreadPool map_async() in Python

And that’s it.

Tying this together, the complete example is listed below.

Running the example first starts the ThreadPool with the default configuration.

Then the 5 tasks are issued to the ThreadPool. The main thread then closes the pool and then waits for the issued tasks to complete.

The tasks execute in the ThreadPool. Each task runs, generating a random number, reporting a message, blocking and returning a value.

All tasks complete and their return values are gathered.

The result callback function is then called with an iterable of the return values. The iterable is traversed and each value is then reported.

All tasks end, and the main thread wakes up and continues on, closing the program.

Next, let’s look at an example of using a callback function with the starmap_async() function.

Example Callback Function for starmap_async()

We can explore how to use a result callback with the ThreadPool when issuing tasks via the starmap_async() function.

In this example, we will update the previous example so that the target task function takes two arguments instead of one. The second argument will be a generated random value that will be passed in instead of generated in the task. A list of arguments will be prepared in the main thread that contain the integer identifiers and random values, then tasks with multiple arguments will be issued via the starmap_async() function.

Firstly, we must update the task() function to take the random number as a second argument and to not generate a number as part of the task.

The updated task() function with these changes is listed below.

Next, in the main thread we will first create a list of arguments. Each item in the list will be a tuple of arguments for one call to the target task() function, containing an integer and a generated random floating point value.

We can then issue the 5 tasks via the starmap_async(), configured to call a callback function once all of the issued tasks have completed.

You can learn more about the starmap_async() function in the tutorial:

• How to Use ThreadPool starmap_async() in Python

And that’s it.

Tying this together, the complete example is listed below.

Running the example first starts the ThreadPool with the default configuration.

Then the 5 tasks are issued to the ThreadPool. The main thread then closes the pool and then waits for the issued tasks to complete.

The tasks execute in the ThreadPool. Each task runs, reporting a message, blocking and returning a value.

All tasks complete and their return values are gathered.

The result callback function is then called with an iterable of the return values. The iterable is traversed and each value is then reported.

All tasks end, and the main thread wakes up and continues on, closing the program.

Next, let’s take a look at the specific thread used to execute the callback function.

Which Thread Executes the Callback Function

We can explore the specific thread used to execute the result callback function.

In this example we will define a task that gets and reports the current thread. We expect this to be the worker thread. We will also get and report the main thread for reference. Finally, we will get and report the current thread used to execute the result callback.

Firstly, we can define a result callback function that gets and reports the current thread.

Recall that we can access the current thread via the threading.current_thread() function that returns a threading.Thread instance.

This will report the details of the thread executing the callback function, most notably the names and the ids.

The result_callback() function below implements this.

Next, we can define a target task function that does the same thing. It gets and reports the thread used to execute the task.

The task() function below implements this.

Finally, in the main thread, we will create and configure a ThreadPool using the context manager, then issue a single task to the ThreadPool via the apply_async() function, configured with a result callback function.

We will then get and report the details of the current thread for reference.

We will then close the ThreadPool and wait for the issued task to complete.

Tying this together, the complete example is listed below.

Running the example first starts the ThreadPool.

The main thread then issues a task to the ThreadPool. It then reports the current thread.

In this case, we can see that the program is executed by the “MainThread” as we might expect.

You can learn more about the main thread in the tutorial:

• What is the Main Thread in Python

Next, the task is executed.

In this case, we can see that the task is executed by a worker thread which is an instance of the DummyProcess class and in this case has the name “Thread-1“

Finally, the result callback function is executed. In this case, we can see that the callback is executed by a helper thread in the ThreadPool named “Thread-11“.

Note, the specific thread names and ids will differ each time the program is run.

Next, let’s look at how we might share data with the callback function.

How to Share Data With the Callback Function

We can explore how to share data with the callback function.

As we discovered in the previous section, both the code where we create the ThreadPool and issue tasks and the callback function are executed in the main thread of the application, at least in these examples.

Therefore, we can define a global variable in the program and have it shared and available to the callback function.

We will define a global variable with a random floating point value between 0 and 1. We will then access this global variable in the callback function, then change it. Once all tasks are finished, we will then report the global variable again and confirm that change was reflected.

This will demonstrate how the callback function can both access and change global variables from the program.

Firstly, we can define a callback function that declares the global variable named “data“, reports the value, changes it, then reports the changed value.

The result_callback() function below implements this.

Next, we can define a target task function that blocks for a moment to simulate computational effort.

Finally, in the main thread we can define a global variable and assign it a random number.

We can then create the ThreadPool, issue the task configured with the result callback, then wait for the task to complete.

We will then report the global variable in order to confirm the changes made in the callback function are reflected in the main thread.

Tying this together, the complete example is listed below.

Running the example first creates the global variable and assigns it a random number, then reports the value.

Next, the ThreadPool is created and the task is issued. The main thread then waits for the task in the ThreadPool to finish.

The task executes, blocking for a second, then the task finishes.

The callback function is called, even though there is no return value. E.g. the argument to the function will be None.

The callback function declares the global variable, then reports its value. We can see that the reported value matches the value reported from the main thread. It then assigns a new random value to the global variable and reports its value.

The main thread continues on and reports the current value of the global variable. We can see that the main thread sees the changed value, matching the value that was set in the callback function.

This highlights how data can be made available to the callback function and how the callback function may make data available to the main thread.

Note, the results will differ each time the program is run given the use of random numbers.

Next, let’s take a closer look at when a callback function is executed.

When is the Callback Executed

We can explore when exactly the callback function is called.

In this example we will report a message in the main thread, in the task, and in the callback function, then after the task is completed. The order of the reported messages will give an idea of when exactly the callback function is executed.

Firstly, we can define a callback function that just reports a message that it is finished.

Similarly, we can define a task function that reports a message that it is finished.

We can create a ThreadPool and issue five tasks to the pool, configured with the custom callback function.

We can then report a message, wait for the tasks to complete, then report another message.

Finally, the pool can be closed.

Tying this together, the complete example is listed below.

Running the example first creates the ThreadPool, then issues five tasks.

The main thread then reports a message that the tasks are issued, then blocks until the tasks are done.

Next, each task executes and reports a message that they are done.

The callback function is then called and reports that it is done.

Finally, the main task unblocks, reports a message that all tasks are done, and closes the ThreadPool.

This highlights that the callback is called after all issued tasks are completed, but before the caller is made aware that the tasks are done. That is, the callback is a part of the task from the perspective of the caller waiting on the task to complete.

Next, let’s take a look at what happens if there is an error in a callback function.

What Happens if a Callback Raises an Exception

We can explore what happens if an error occurs in the result callback function.

In this example, we will define a result callback function that raises an exception. We will then issue the task as per normal and wait for it to complete.

Firstly, we can define the custom callback function that reports a message then raises an exception.

Next, we will define a target task function that simply reports a message.

In the main thread, we will create the ThreadPool, then issue five tasks to the pool with the configured callback.

The main thread will then wait for the tasks to complete, then close the thread pool.

Tying this together, the complete example is listed below.

Running the example first creates the ThreadPool.

It then issues 5 tasks and waits for them to complete.

Each task executes, reporting a message as per normal.

The callback function is then called after all tasks are finished. A message is reported, then an exception is raised.

The exception then appears to unwind the helper thread in the ThreadPool. This very likely breaks the ThreadPool.

The main thread in the main thread blocks forever waiting for the issued tasks to complete. The tasks never complete because the callback function never completes successfully.

This highlights that care must be taken in the callback function as an error in the callback may bring down the application.

Note, you will have to manually terminate the program, e.g. Control-C.

Common Questions

This section lists common questions about result callbacks used with asynchronous tasks in the ThreadPool.

Do you have any questions?
Let me know below.

What is a Callback?

A callback is a function called automatically once all asynchronous tasks issued to the ThreadPool in a batch have completed.

When Should I Use a Callback?

You can use a callback function when issuing asynchronous tasks to the ThreadPool.

A callback function can be used to handle the results of issued tasks, such as reporting, storing, or gathering the results of issued tasks.

Callback functions should not take a long time to execute as they will occupy resources of the ThreadPool.

What Argument Does the Callback Receive?

The callback function will receive the return value from the task or tasks issued to the ThreadPool.

In the case of apply_async(), the argument will be a single value.

In the case of map_async() and starmap_async(), the argument will be an iterable of return values.

What if the Task Function Does Not Return a Value?

A configured callback function will be called even if your target task function does not return a value.

In that case, the argument to the callback function will be the default return value of None.

Which Thread Executes the Callback Function?

The callback function is executed in a helper thread of the ThreadPool in the main process, or the process in which the ThreadPool was created and tasks were issued.

Will The Result Callback Be Called Before The Error Callback?

No.

If at least one task raises an error, then the result callback will not be called and the error callback function will be called instead.

Can I Issue Follow-Up Tasks From the Result Callback?

Yes.

The ThreadPool can be made accessible via a global variable and follow-up tasks may be issued directly from within the callback function.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python ThreadPool Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• ThreadPool PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPool: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide

APIs

• multiprocessing - Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know how to use callback functions with the ThreadPool in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by ZHANG FENGSHENG on Unsplash