Return Value From Child Process Without Changing The Target Function

March 14, 2023 Python Multiprocessing

You can execute a target function that returns a value in a child process without changing the target function by using a wrapper function to handle the transmission of the return value back to the caller automatically.

An alternate approach is to define a callback function and have the wrapper function call the callback function to handle the return value automatically, instead of returning it.

In this tutorial, you will discover how to execute a target function that returns a result in a child process without changing the target function.

Let's get started.

Need to Return Value From Child Process Without Changing Target Function

We can run a task in a child process.

This can be achieved by creating an instance of the multiprocessing.Process class and specify the target function to execute via the "target" argument and any arguments to the target function via the "args" argument.

For example:

...
# create and configure a child process
process = multiprocessing.Process(target=task)

You can learn more about running a function in a child process in the tutorial:

Run a Function in a Child Process

When executing a task in a child function, we cannot return a result directly.

Instead, we must simulate a return value, such as by using a shared queue.

You can learn more about simulating a return value from a child process in the tutorial:

Multiprocessing Return Value From Process

The problem with this approach is that it requires that the target function be changed to put the result on the shared queue or shared pipe, rather than return it.

Sometimes we cannot change the target function because we don't have direct access to it, for example, it may be part of a third-party library. Alternatively, the target function may have other uses, requiring that the interface does not change.

How can we return a value from a target function executed in a child process without changing the target function?

How to Simulate Return Value From Child Process

We can simulate a return value from a child process without changing the target function by using a wrapper function.

We can define a new function that executes the target function in a child process, collects the return value, and then transmits it back to the calling process via a queue or pipe.

For example:

# helper function for running a target function in a child process
def run_in_process_helper(function, queue):
    # run the function and get the result
    result = function()
    # put the result on the queue
    queue.put(result)

The caller can then retrieve the return value directly from the shared queue.

Alternatively, we can hide the queue completely in a second function responsible for creating, configuring, and starting the child process, calling the wrapper function, and then returning the value retrieved from the shared queue.

For example:

# run a function with a return value in a child process
def run_in_process(function):
    # create the shared queue
    queue = Queue()
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, queue))
    # start the child process
    process.start()
    # wait for the return value on the queue
    result = queue.get()
    # return the return value
    return result

This encapsulates the execution of the task in the child process and the mechanism for retrieving the return value.

Another approach that could be used is to pass a callback function used to handle the result.

The callback function can be passed and executed in the child process once the result from the target function has been retrieved.

For example:

# helper function for running a target function in a child process
def run_in_process_helper(function, callback):
    # run the function and get the result
    result = function()
    # call the callback with the result
    callback(result)

Alternatively, the callback function could be executed in the calling process, once the result has been retrieved from the shared queue.

For example:

# run a function with a return value in a child process
def run_in_process(function, callback):
    # create the shared queue
    queue = Queue()
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, queue))
    # start the child process
    process.start()
    # wait for the return value on the queue
    result = queue.get()
    # execute the callback
    callback(result)

Now that we have explored some approaches for executing a target function that returns a result in a child process without changing the target function, let's look at some worked examples.

Example of Simulated Return Value From Child Process

We can explore how to return a value from a target function executed in a child process using a wrapper function.

In this example, we will first define a wrapper function to execute in a child process.

The wrapper function takes the name of the function to execute and the shared queue. It executes the target function and puts the result on the shared queue.

# helper function for running a target function in a child process
def run_in_process_helper(function, queue):
    # run the function and get the result
    result = function()
    # put the result on the queue
    queue.put(result)

Next, we can define a function to drive the overall process.

This function takes the name of the target function. It creates the shared queue, then creates and configures a child process to execute the wrapper function with the name of the target function and shared queue as arguments. It starts the child process, then waits for the result on the shared queue, which is then returned.

# run a function with a return value in a child process
def run_in_process(function):
    # create the shared queue
    queue = Queue()
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, queue))
    # start the child process
    process.start()
    # wait for the return value on the queue
    result = queue.get()
    # return the return value
    return result

We can then define our actual target function.

In this case, our target task function does not take any arguments, blocks for one second to simulate work, then returns an integer value.

# task that returns a value
def task():
    # block for a moment to simulate work
    sleep(1)
    # return some data
    return 111

Importantly, the target function can return a value normally and is not concerned without the return value being transmitted from the child process back to the calling process.

Finally, we can call the helper function to run the target function in the child process, passing it the name of the target function to execute.

# protect the entry point
if __name__ == '__main__':
    # run a target function with a return value in a child process
    result = run_in_process(task)
    # report the result
    print(f'got: {result}')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of simulating a return value from a child process
from time import sleep
from multiprocessing import Process
from multiprocessing import Queue

# helper function for running a target function in a child process
def run_in_process_helper(function, queue):
    # run the function and get the result
    result = function()
    # put the result on the queue
    queue.put(result)

# run a function with a return value in a child process
def run_in_process(function):
    # create the shared queue
    queue = Queue()
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, queue))
    # start the child process
    process.start()
    # wait for the return value on the queue
    result = queue.get()
    # return the return value
    return result

# task that returns a value
def task():
    # block for a moment to simulate work
    sleep(1)
    # return some data
    return 111

# protect the entry point
if __name__ == '__main__':
    # run a target function with a return value in a child process
    result = run_in_process(task)
    # report the result
    print(f'got: {result}')

Running the example calls the run_in_process() function with the name of the target function.

This creates the shared queue, then creates and configures a new child process instance to execute the run_in_process_helper() wrapper function and passes it the name of the target function and the shared queue.

The run_in_process() then starts the child process and blocks, waiting for the result to be retrieved from the shared queue.

The child process executes the target function and then places the result on the shared queue.

The main process retrieves the result from the queue and returns it back to the caller, where it is reported.

This highlights how we can use wrapper functions to execute a target function that returns a result in a child process, without making any changes to the target function.

got: 111

Next, let's explore an alternative approach that uses a callback function to handle the result of the target function, instead of returning it to the caller.

Example of Simulated Return Value From Child Process With Callback

We can explore the case of handling the return value from the target function with a callback function, instead of returning it to the caller.

In this example, we will define a callback function that takes the return value from the target function, and in this case simply reports it.

# callback function for handling the return value
def custom_callback(result):
    # report the result
    print(f'got: {result}')

We can then update the run_in_process() function to take the name of the callback function as an argument, and then call the callback function in the calling process, once the result from the target function has been retrieved from the shared queue.

# run a function with a return value in a child process
def run_in_process(function, callback):
    # create the shared queue
    queue = Queue()
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, queue))
    # start the child process
    process.start()
    # wait for the return value on the queue
    result = queue.get()
    # execute the callback
    callback(result)

Finally, we can call the run_in_process() with the name of the target function and the name of the callback function.

# protect the entry point
if __name__ == '__main__':
    # run a target function with a return value in a child process
    run_in_process(task, custom_callback)

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of simulating a return value from a child process with a callback
from time import sleep
from multiprocessing import Process
from multiprocessing import Queue

# helper function for running a target function in a child process
def run_in_process_helper(function, queue):
    # run the function and get the result
    result = function()
    # put the result on the queue
    queue.put(result)

# run a function with a return value in a child process
def run_in_process(function, callback):
    # create the shared queue
    queue = Queue()
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, queue))
    # start the child process
    process.start()
    # wait for the return value on the queue
    result = queue.get()
    # execute the callback
    callback(result)

# task that returns a value
def task():
    # block for a moment to simulate work
    sleep(1)
    # return some data
    return 111

# callback function for handling the return value
def custom_callback(result):
    # report the result
    print(f'got: {result}')

# protect the entry point
if __name__ == '__main__':
    # run a target function with a return value in a child process
    run_in_process(task, custom_callback)

Running the example calls the run_in_process() function with the name of the target function and the name of the callback function.

The run_in_process() then starts the child process and blocks, waiting for the result to be retrieved from the shared queue.

The child process executes the target function and then places the result on the shared queue.

The main process retrieves the result from the queue and returns and then calls the callback function with the return value.

The callback function executes and reports the return value.

This highlights how we can use a callback function to handle the return value from a target function executed in a child process, without changing the target function.

got: 111

Next, let's explore how we might execute the target function in a child process asynchronously and handle the result using a callback function.

Example of Simulated Return Value From Child Process Asynchronous Callback

A benefit of executing a target function in a child process is that it does not have to block the caller.

A downside of the two designs presented so far is that they block the caller until the target function executed in the child process is complete.

In this example, we can update the previous example that handles the result from the target function using a callback function, and execute the target function asynchronously. This allows the caller to resume and execute other tasks if needed.

This can be achieved by executing the callback function using the wrapper function that is executed in the child function. The callback function can be passed to the wrapper function.

Because the result is no longer needed in the calling process, we no longer need the shared queue.

# helper function for running a target function in a child process
def run_in_process_helper(function, callback):
    # run the function and get the result
    result = function()
    # call the callback with the result
    callback(result)

We can then update the run_in_process() function to pass the callback function to the wrapper function and no longer create or wait on a shared queue.

Instead, it starts the child process and returns immediately, without blocking.

# run a function with a return value in a child process
def run_in_process(function, callback):
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, callback))
    # start the child process
    process.start()

And that's it.

The complete example of executing the target function in a child process asynchronously and handling the result with a callback function is listed below.

# SuperFastPython.com
# example of asynchronous a return value from a child process with a callback
from time import sleep
from multiprocessing import Process
from multiprocessing import Queue

# helper function for running a target function in a child process
def run_in_process_helper(function, callback):
    # run the function and get the result
    result = function()
    # call the callback with the result
    callback(result)

# run a function with a return value in a child process
def run_in_process(function, callback):
    # configure the child process
    process = Process(target=run_in_process_helper, args=(function, callback))
    # start the child process
    process.start()

# task that returns a value
def task():
    # block for a moment to simulate work
    sleep(1)
    # return some data
    return 111

# callback function for handling the return value
def custom_callback(result):
    # report the result
    print(f'got: {result}')

# protect the entry point
if __name__ == '__main__':
    # run a target function with a return value in a child process
    run_in_process(task, custom_callback)
    # report a message
    print('Main is done')

Running the example calls the run_in_process() function with the name of the target function and the name of the callback function.

This creates and configures a new child process instance to execute the run_in_process_helper() wrapper function and passes it the name of the target function and the name of the callback function. The child process is started and the run_in_process() returns immediately.

The main process is then done, but the program does not exit because the child process is still executing and is not a daemon process.

The run_in_process_helper() wrapper function executes in the child process. It calls the target function, retrieves the result, then calls the callback function with the result.

The callback function executes and reports the return value.

This highlights how we can execute a target function asynchronously in a child process and handle the return value using a callback, all without having to make any changes to the target function.

Main is done
got: 111

Takeaways

You now know how to execute a target function that returns a result in a child process without changing the target function.

If you enjoyed this tutorial, you will love my book: Python Multiprocessing Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.