Asyncio Subprocess With create_subprocess_exec()

December 17, 2022 Python Asyncio

You can run a command as a subprocess with asyncio via the create_subprocess_exec() function.

In this tutorial, you will discover how to run commands with asyncio in Python.

Let's get started.

What is Asyncio create_subprocess_exec()

The create_subprocess_exec() function allows commands to be executed from asyncio.

What is a Command

A command is a program executed on the command line (terminal or command prompt). It is another program that is run directly.

Common examples on Linux and macOS might be:

'ls' to list the contents of a directory
'cat' to report the content of a file
'date' to report the date
'echo' to report back a string
'sleep' to sleep for a number of seconds

And so on.

These are just programs that we can execute on the command line as a command.

Asyncio create_subprocess_exec()

The asyncio.create_subprocess_exec() function takes a command and executes it directly.

This is helpful as it allows the command to be executed in a subprocess and for asyncio coroutines to read, write, and wait for it.

Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel.
-- Asyncio Subprocesses

Unlike the asyncio.create_subprocess_shell() function, the asyncio.create_subprocess_exec() will not execute the command using the shell.

This means that the capabilities provided by the shell, such as shell variables, scripting, and wildcards are not available when executing the command.

It also means that executing the command may be more secure as there is no opportunity for a shell injection.

You can learn more about the create_subprocess_shell() function in the tutorial:

Asyncio Subprocess With create_subprocess_shell()

Now that we know what asyncio.create_subprocess_exec() does, let's look at how to use it.

How to Use Asyncio create_subprocess_exec()

The asyncio.create_subprocess_exec() function will execute a given string command in a subprocess.

It returns a asyncio.subprocess.Process object that represents the subprocess.

Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.
-- Interacting with Subprocesses

The create_subprocess_exec() function is a coroutine, which means we must await it. It will return once the subprocess has been started, not when the subprocess is finished.

For example:

...
# execute a command in a subprocess
process = await asyncio.create_subprocess_exec('ls')

Arguments to the command being executed must be provided as subsequent arguments to the create_subprocess_exec() function.

For example:

...
# execute a command with arguments in a subprocess
process = await asyncio.create_subprocess_exec('ls', '-l')

We can wait for the subprocess to finish by awaiting the wait() method.

For example:

...
# wait for the subprocess to terminate
await process.wait()

We can stop the subprocess directly by calling the terminate() or kill() methods, which will raise a signal in the subprocess.

For example:

...
# terminate the subprocess
process.terminate)

The input and output of the command will be handled by stdin, stderr, and stdout.

We can have the asyncio program handle the input or output for the subprocess.

This can be achieved by specifying the input or output stream and specifying a constant to redirect, such as asyncio.subprocess.PIPE.

For example, we can redirect the output of a command to the asyncio program:

...
# start a subprocess and redirect output
process = await asyncio.create_subprocess_exec('ls', stdout=asyncio.subprocess.PIPE)

We can then read the output of the program via the asyncio.subprocess.Process instance via the communicate() method.

This method is a coroutine and must be awaited. It is used to both send and receives data with the subprocess.

For example:

...
# read data from the subprocess
line = process.communicate()

We can also send data to the subprocess via the communicate() method by setting the "input" argument in bytes.

For example:

...
# start a subprocess and redirect input
process = await asyncio.create_subprocess_exec('ls', stdin=asyncio.subprocess.PIPE)
# send data to the subprocess
process.communicate(input=b'Hello\n')

Behind the scenes the asyncio.subprocess.PIPE configures the subprocess to point to a StreamReader or StreamWriter for sending data to or from the subprocess, and the communicate() method will read or write bytes from the configured reader.

If PIPE is passed to stdin argument, the Process.stdin attribute will point to a StreamWriter instance. If PIPE is passed to stdout or stderr arguments, the Process.stdout and Process.stderr attributes will point to StreamReader instances.
-- Asyncio Subprocesses

We can interact with the StreamReader or StreamWriter directly via the subprocess via the stdin, stdout, and stderr attributes.

For example:

...
# read a line from the subprocess output stream
line = await process.stdout.readline()

Now that we know how to use the create_subprocess_exec() function, let's look at some worked examples.

Example of Asyncio create_subprocess_exec()

We can explore how to run a command in a subprocess from asyncio.

In this example, we will execute the "echo" command to report back a string.

The echo command will report the provided string on standard output directly.

The complete example is listed below.

Note, this example assumes you have access to the "echo" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of executing a command as a subprocess with asyncio
import asyncio

# main coroutine
async def main():
    # start executing a command in a subprocess
    process = await asyncio.create_subprocess_exec('echo', 'Hello World')
    # report the details of the subprocess
    print(f'subprocess: {process}')

# entry point
asyncio.run(main())

Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.

The main() coroutine runs and calls the create_subprocess_exec() function to execute a command.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() coroutine resumes and reports the details of the subprocess. The main() process terminates and the asyncio program terminates.

The output of the echo command is reported on the command line.

This highlights how we can execute a command from an asyncio program.

Hello World
subprocess: <Process 50249>

Example of Waiting for Long Running Subprocess

We can explore how to wait for a long-running command to run in asyncio.

In this example, we will execute the "sleep" command that will block the calling for a given number of seconds.

We will then wait for the subprocess to complete from the coroutine.

The complete example is listed below.

Note, this example assumes you have access to the "sleep" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of waiting for a long running a command with asyncio
import asyncio

# main coroutine
async def main():
    # start executing a command in a subprocess
    process = await asyncio.create_subprocess_exec('sleep', '3')
    # report the details of the subprocess
    print(f'subprocess: {process}')
    # wait for the subprocess to terminate
    await process.wait()

# entry point
asyncio.run(main())

Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.

The main() coroutine runs and calls the create_subprocess_exec() function to execute a command.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() process resumes and reports the details of the subprocess.

It then suspends and waits for the subprocess to terminate.

The subprocess terminates, the main() coroutine resumes and closes the program.

This highlights how an asyncio program can execute a command in a subprocess and wait for the command to complete.

subprocess: <Process 50257>

Example of Reading Output From a Subprocess

We can explore how to read output from a command run in a subprocess.

In this example, we will read data from the command executed in the subprocess.

We will execute the "echo" command and redirect output from stdout to the asyncio program. We will then read the output in the coroutine and report the result.

The complete example is listed below.

Note, this example assumes you have access to the "echo" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of reading output of command executed via in asyncio
import asyncio

# main coroutine
async def main():
    # start executing a command in a subprocess
    process = await asyncio.create_subprocess_exec('echo', 'Hello World', stdout=asyncio.subprocess.PIPE)
    # report the details of the subprocess
    print(f'subprocess: {process}')
    # read a line of output from the program
    data, _ = await process.communicate()
    # report the data
    print(data)

# entry point
asyncio.run(main())

The main() coroutine runs and calls the create_subprocess_exec() function to execute a command and redirect output to the asyncio program.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() coroutine resumes and reports the details of the subprocess.

It then reads data from the subprocess, suspending until all data is received (e.g. an end-of-file character is sent).

The subprocess echos the string which is read by the coroutine and reported.

This highlights how we can read data from a command executed as a subprocess in an asyncio program.

subprocess: <Process 50261>
b'Hello World\n'

Example of Sending Input to a Subprocess

We can explore how to write data to a command run in a subprocess with asyncio.

In this example, we will execute the "cat" command. We will then write a string to the subprocess, which will be reported back via standard output.

The complete example is listed below.

Note, this example assumes you have access to the "cat" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of writing input to a subprocess in asyncio
import asyncio

# main coroutine
async def main():
    # start executing a command in a subprocess
    process = await asyncio.create_subprocess_exec('cat', stdin=asyncio.subprocess.PIPE)
    # report the details of the subprocess
    print(f'subprocess: {process}')
    # write data to the process
    _ = await process.communicate(b'Hello World\n')

# entry point
asyncio.run(main())

The main() coroutine runs and calls the create_subprocess_exec() function to execute a command and redirect input from the asyncio program.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() coroutine resumes and reports the details of the subprocess.

It then sends a string to the subprocess, encoded as bytes.

The main() coroutine terminates.

The subprocess reports the string sent to it and terminates.

This highlights how we can send data to a command executed as a subprocess in an asyncio program.

subprocess: <Process 50266>
Hello World

Takeaways

You now know how to run commands with asyncio in Python.

If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.