Last Updated on November 14, 2023

Sometimes we need to start, run and interact with other programs from within our Python program.

Asyncio provides a way to run commands in subprocesses and to read and write from the subprocesses without blocking. This can be helpful when porting Python programs that run local commands as subprocesses to asyncio and in developing new programs that are required to interact with separate programs.

In this tutorial, you will discover how to use subprocesses in asyncio programs.

After completing this tutorial, you will know:

• What are subprocesses and how we can run new commands in subprocesses from asyncio.
• How to perform non-blocking read and write communication with programs in subprocesses.
• How to send signals, retrieve data, and terminate programs run in subprocesses.

Let’s get started.

What are Asyncio Subprocesses

We can execute commands and run separate programs from asyncio as subprocesses.

These subprocesses are represented by the Process class in the asyncio.subprocess module.

A Process provides a handle on a subprocess in asyncio programs, allowing actions to be performed on it, such as waiting and terminating it.

Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.

— Interacting with Subprocesses

The API is very similar to the multiprocessing.Process class and perhaps more so with the subprocess.Popen class.

Specifically, it shares methods such as wait(), communicate(), send_signal(), and attributes such as stdin, stdout, and stderr with the subprocess.Popen.

Now that we know what the asyncio.subprocess.Process class is, let’s look at how we might use it in our asyncio programs.

How to Create a asyncio.subprocess.Process

We do not create a asyncio.subprocess.Process directly.

Instead, an instance of the class is created for us when executing a subprocess in an asyncio program.

An object that wraps OS processes created by the create_subprocess_exec() and create_subprocess_shell() functions.

— Interacting with Subprocesses

There are two ways to execute an external program as a subprocess and acquire a Process instance, they are:

• asyncio.create_subprocess_exec()
• asyncio.create_subprocess_shell()

Let’s look at examples of each in turn.

Create Process with create_subprocess_exec()

The asyncio.create_subprocess_exec() function can be called to execute a command in a subprocess.

It returns a asyncio.subprocess.Process as a handle on the subprocess.

The create_subprocess_exec() function is a coroutine and must be awaited. It will suspend the caller until the subprocess is started (not completed).

For example:

We can configure the subprocess to receive input from the asyncio program or send output to the asyncio program by setting the stdin, stdout, and stderr attributes to the asyncio.subprocess.PIPE constant.

This will set the stdin, stdout, and stderr attributes on the asyncio.subprocess.Process to be a StreamReader or StreamWriter and allow coroutines to read or write from them via the communicate() method in the subprocess (described later).

For example:

You can learn more about how to execute commands in a subprocess in the tutorial:

• Asyncio Subprocess With create_subprocess_exec()

We can explore how to get a Process instance by executing a command in a subprocess with the create_subprocess_exec() function.

The example below executes the “echo” command in a subprocess that prints out the provided string.

The subprocess is started, then the details of the subprocess are then reported.

The complete example is listed below.

Running the example executes the echo command in a subprocess.

A asyncio.subprocess.Process instance is returned and the details are reported, showing the unique process id (PID).

The echo command then reports the provided string on stdout.

Next, let’s look at an example of creating a subprocess via the create_subprocess_shell() function.

Create Process with create_subprocess_shell()

The asyncio.create_subprocess_shell() function can be called to execute a command in a subprocess.

Unlike the create_subprocess_exec() function, the create_subprocess_shell() function will execute the provided command via the shell. This is the command line interpreter used to execute commands on the system, such as bash or zsh on Linux and macOS or cmd.exe on Windows.

Executing a command via the shell allows the capabilities of the shell to be used in addition to executing the command, such as wildcards and shell variables.

The function returns a asyncio.subprocess.Process as a handle on the subprocess.

The create_subprocess_shell() function is a coroutine and must be awaited. It will suspend the caller until the subprocess is started (not completed).

You can learn more about how to execute commands in a subprocess in the tutorial:

• Asyncio Subprocess With create_subprocess_shell()

We can explore how to get a Process instance by executing a command in a subprocess with the create_subprocess_shell() function.

The example below executes the “echo” command in a subprocess that prints out the provided string. Unlike the create_subprocess_exec() function, the entire command with arguments is provided as a single string.

The subprocess is started, then the details of the subprocess are then reported.

The complete example is listed below.

Running the example executes the echo command in a subprocess.

A asyncio.subprocess.Process instance is returned and the details are reported, showing the unique process id (PID).

The echo command then reports the provided string on stdout.

Next, let’s look at how we can wait for a subprocess to complete.

How to Wait for the Subprocess

We can wait for a subprocess to complete via the wait() method.

Wait for the child process to terminate.

— Asyncio Subprocesses

This is a coroutine that must be awaited.

The caller will be suspended until the subprocess is terminated, normally or otherwise.

If the subprocess is expecting input and is configured to receive input via asyncio via a pipe, then calling wait() can cause a deadlock as the caller cannot provide input to the subprocess if it is suspended.

This method can deadlock when using stdout=PIPE or stderr=PIPE and the child process generates so much output that it blocks waiting for the OS pipe buffer to accept more data. Use the communicate() method when using pipes to avoid this condition.

— Asyncio Subprocesses

We can explore how to wait for a subprocess to complete.

In the example below, we will execute the “sleep” command and sleep for three seconds.

The caller will then wait for the subprocess to complete before resuming.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The coroutine then awaits the process to complete, which takes three seconds.

The subprocess closes and the coroutine resumes and terminates the programs.

This highlights how to wait for a subprocess in an asyncio program.

Next, let’s look at how we might write data to the subprocess.

How to Write Data to the Subprocess

We can write data to the subprocess from an asyncio coroutine via the communicate() method.

This is a coroutine that must be awaited. Data is provided via the “input” argument as bytes.

The optional input argument is the data (bytes object) that will be sent to the child process.

— Asyncio Subprocesses

Writing data to the subprocess via the communicate() method requires that the “stdin” argument in the create_subprocess_shell() or create_subprocess_exec() functions were set to asyncio.subprocess.PIPE.

We can explore how to write data to a subprocess in asyncio.

In the example below we will run the “cat” command that outputs the data provided to it. When called without an argument it will echo data from stdin.

We will write a string to the subprocess via the communicate() method.

The complete example is listed below.

Running the example executes the cat command in a subprocess and configured standard input to be redirected from an asyncio pipe.

The command awaits input from stdin.

The coroutine then writes a string with a trailing new line to the subprocess in byte format.

The string is reported and the subprocess terminates.

This highlights how to write data to a subprocess in an asyncio program.

Next, let’s look at how to read data from a subprocess in asyncio.

How to Read Data to the Subprocess

We can read data from a subprocess in asyncio via the communicate() method.

Reading data from the subprocess requires that the stdout or stderr arguments of the create_subprocess_shell() or create_subprocess_exec() functions was set to asyncio.subprocess.PIPE.

No argument is provided and the method returns a tuple with input from stdout and stderr. Data is read until an end of file (EOF) character is received.

read data from stdout and stderr, until EOF is reached;

— Asyncio Subprocesses

The communicate() method is a coroutine and must be awaited.

If no data can be read, the call will block until the subprocess has terminated.

We can explore how to read data from a subprocess.

In the example below, we execute the “echo” command to echo a string and configure stdout to redirect to an asyncio pipe.

We then read the data from the subprocess and report it.

The complete example is listed below.

Running the example runs the command in a subprocess and configures stdout to redirect to a pipe.

The subprocess runs and outputs the string.

The coroutine reads data from the subprocess, keeping the data from stdout and ignoring stderr.

The returned string is then reported, showing that it was read as byte data.

This highlights how to read data from a subprocess from asyncio.

Next, let’s look at how we might send signals to the subprocess from asyncio.

How to Send Signals to the Subprocess

We can send a signal to the subprocess via the send_signal() method.

Sends the signal signal to the child process.

— Asyncio Subprocesses

The method takes a specific signal type, such as SIGTERM to terminate the subprocess or SIGKILL to terminate the subprocess.

You can learn more about signaling processes in the signal module API documentation:

• signal — Set handlers for asynchronous events

We can explore sending a signal to a subprocess in asyncio.

In the example below, we run the “sleep” command in a subprocess that will sleep for 3 seconds.

The calling coroutine will wait for 1 second, then send a SIGKILL signal to the subprocess in order to terminate it immediately.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The subprocess proceeded to sleep for 3 seconds.

The coroutine then blocks for 1 second. It then resumes and sends the SIGKILL to the subprocess.

This terminates the subprocess immediately, and the program closes.

This highlights how to send a signal to a subprocess from asyncio.

Next, let’s look at how to terminate a subprocess from asyncio.

How to Terminate the Subprocess

We can stop a subprocess via the terminate() method.

Stop the child process. On POSIX systems this method sends signal.SIGTERM to the child process. On Windows the Win32 API function TerminateProcess() is called to stop the child process.

— Asyncio Subprocesses

On most platforms, this sends a SIGTERM signal to the subprocess and terminates it immediately.

We can explore how to stop a subprocess from asyncio.

The example below starts a subprocess that executes the sleep command, then terminates it directly.

This achieves the same result as sending the SIGTERM to the subprocess manually, as we did in the previous example.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The calling coroutine then sleeps a moment to allow the subprocess to run.

It then calls the terminate() method. This stops the subprocess immediately, and the program exits.

This highlights how to terminate a subprocess from asyncio.

Next, we will look at how to kill a subprocess from an asyncio program.

How to Kill the Subprocess

We can kill a subprocess via the kill() method.

Kill the child process. On POSIX systems this method sends SIGKILL to the child process. On Windows this method is an alias for terminate().

— Asyncio Subprocesses

On most platforms, this will send the SIGKILL signal to the subprocess in order to stop it immediately.

Unlike the terminate() method that sends the SIGTERM signal, the SIGKILL signal cannot be handled by the subprocess. This means it is assured to stop the subprocess.

We can explore how to kill a subprocess.

In the example below, we will update the previous example that executes the sleep command in a subprocess.

Rather than terminating it after a short interval, the caller will kill it directly.

It has much the same effect in this case.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The calling coroutine then sleeps a moment to allow the subprocess to run.

It then calls the kill() method. This stops the subprocess immediately, and the program exits.

This highlights how to kill a subprocess from asyncio.

Next, we will look at how to get the PID from a command executing in a subprocess.

How to Get the Process PID

We can get the process identifier (PID) for a subprocess in asyncio via the “pid” attribute.

The PID is a unique positive integer assigned to each process in the system by the underlying operating system.

We can explore how to get and report the PID of a subprocess.

In the example below we execute the “sleep” command in a subprocess, then report the PID of the subprocess.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The PID is then retrieved from the subprocess via the Process instance.

Note, the process identifier will be different each time the example is run.

This highlights how to retrieve the PID from a subprocess in asyncio.

Next, let’s look at how to access the return code for a subprocess.

How to Get the Subprocess Return Code

We can retrieve the return code from a subprocess in asyncio via the “returncode” attribute.

Return code of the process when it exits. A None value indicates that the process has not terminated yet. A negative value -N indicates that the child was terminated by signal N (POSIX only).

— Asyncio Subprocesses

The return code is only available after the process has terminated, otherwise, the value is None.

A return code of zero (0) indicates a successful exit. Any other value indicates an unsuccessful exit.

You can learn more about exit codes from processes in Python in the tutorial:

• Process Exit Codes in Python

We can explore how to access the return code of a subprocess in asyncio.

The example below executes the “sleep” command in a subprocess. The calling coroutine then waits for the subprocess to terminate, then reports the exit code.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The coroutine then suspends and awaits the subprocess to terminate.

Once terminated, the coroutine then reports the return code for the subprocess.

A value of zero (0) is reported, showing a successful exit of the subprocess, as we might expect.

This highlights how to retrieve the return code for a subprocess in asyncio.

Next, let’s look at how to access the streams of the subprocess.

How to Use Subprocess Streams

The asyncio.subprocess.Process allows us to read or write to the subprocess, as we have seen above.

This is achieved via three attributes on a asyncio.subprocess.Process object: stdin, stout, and sterr.

These are set to a StreamWriter and StreamReaders respectively if the process is configured to redirect outputs to a stream via the asyncio.subprocess.PIPE when created.

If configured, these readers and writers can be used directly to read or write data.

We can take a closer look at these attributes in the asyncio.subprocess.Process class.

Example of Streams Not Set

If the subprocess is created without redirecting the streams to pipes, then the stdin, stdout, and sterr attributes on the Process object will not be set.

We can demonstrate this with a worked example.

In the example below, we execute the “sleep” command in a subprocess and not pipe the streams.

We then report the status of each of the streams on the Process object.

The complete example is listed below.

Running the example executes the sleep command in a subprocess.

The coroutine then reports the status of the three streams on the Process object.

We can see that they are not set, as expected.

This highlights that in order to read and write from the subprocess, we must configure the streams when the subprocess is created.

We will take a look at this next.

Example of Piped Streams

We can configure the input and output streams of the subprocess to be piped.

This creates a StreamWriter and StreamReader objects for the streams and makes them available via the asyncio.subprocess.Process object.

We can demonstrate this with a worked example.

In the example below, we execute the sleep command in the subprocess, then pipe all streams.

We then report the status of all of the streams via attributes on the Process object.

The complete example is listed below.

Running the example executes the sleep command in a subprocess and redirects the input and output streams from the process.

The coroutine then reports the status of the streams.

In this case, we can see that the stdin stream is configured as a StreamWriter object, and the stdout and stderr streams are configured as StreamReaders.

We may then choose to write or read from these streams directly from the asyncio program in order to interact with the subprocess.

This highlights how to configure the input and output streams for the subprocess.

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know how to use the asyncio.subprocess.Process in asyncio programs.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Tom Strecker on Unsplash