Asyncio Subprocess With create_subprocess_shell()

December 16, 2022 Python Asyncio

You can run a command using a shell as a subprocess with asyncio via the create_subprocess_shell() function.

In this tutorial, you will discover how to run commands using the shell with asyncio in Python.

Let's get started.

What is Asyncio create_subprocess_shell()

The asyncio.create_subprocess_shell() function allows commands to be executed using the shell from asyncio.

What is a Command

A command is a program executed on the command line (terminal or command prompt). It is another program that is run directly.

Common examples on Linux and macOS might be:

'ls' to list the contents of a directory
'cat' to report the content of a file
'date' to report the date
'echo' to report back a string
'sleep' to sleep for a number of seconds

And so on.

These are just programs that we can execute on the command line as a command.

What is a Shell

We can execute these commands using the shell.

The shell is a user interface for the command line, called a command line interpreter (CLI).

It will interpret and execute commands on behalf of the user.

It also offers features such as a primitive programming language for scripting, wildcards, piping, shell variables (e.g. PATH), and more.

For example, we can redirect the output of one command as input to another command, such as the contents of the "/etc/services" file into the word count "wc" command and count the number of lines:

cat /etc/services | wc -l

Examples of shells in the Unix based operating systems include:

'sh'
'bash'
'zsh'
And so on.

On Windows, the shell is probably cmd.exe.

See this great list of command line shells:

List of command-line interpreters, Wikipedia.

The shell is already running, it was used to start the Python program.

You don't need to do anything special to get or have access to the shell.

Asyncio create_subprocess_shell()

The asyncio.create_subprocess_shell() function takes a command and executes it using the current user shell.

This is helpful as it not only allows the command to be executed, but allows the capabilities of the shell to be used, such as redirection, wildcards and more.

... the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user's home directory.
-- subprocess — Subprocess management

The command will be executed in a subprocess of the process executing the asyncio program.

Importantly, the asyncio program is able to interact with the subprocess asynchronously, e.g. via coroutines.

Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel.
-- Asyncio Subprocesses

There can be security considerations when executing a command via the shell instead of directly.

This is because there is at least one level of indirection and interpretation between the request to execute the command and the command being executed, allowing possible malicious injection.

Important It is the application’s responsibility to ensure that all whitespace and special characters are quoted appropriately to avoid shell injection vulnerabilities.
-- Asyncio Subprocesses

Now that we know what asyncio.create_subprocess_shell() does, let's look at how to use it.

How to Use Asyncio create_subprocess_shell()

The asyncio.create_subprocess_shell() function will execute a given string command via the current shell.

It returns a asyncio.subprocess.Process object that represents the process.

Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.
-- Interacting with Subprocesses

The create_subprocess_shell() function is a coroutine, which means we must await it. It will return once the subprocess has been started, not when the subprocess is finished.

For example:

...
# start a subprocess
process = await asyncio.create_subprocess_shell('ls')

We can wait for the subprocess to finish by awaiting the wait() method.

For example:

...
# wait for the subprocess to terminate
await process.wait()

We can stop the subprocess directly by calling the terminate() or kill() methods, which will raise a signal in the subprocess.

For example:

...
# terminate the subprocess
process.terminate)

The input and output of the command will be handled by the shell, e.g. stdin, stderr, and stdout.

We can have the asyncio program handle the input or output for the subprocess.

This can be achieved by specifying the input or output stream and specifying a constant to redirect, such as asyncio.subprocess.PIPE.

For example, we can redirect the output of a command to the asyncio program:

...
# start a subprocess and redirect output
process = await asyncio.create_subprocess_shell('ls', stdout=asyncio.subprocess.PIPE)

We can then read the output of the program via the asyncio.subprocess.Process instance via the communicate() method.

This method is a coroutine and must be awaited. It is used to both send and receives data with the subprocess.

For example:

...
# read data from the subprocess
line = process.communicate()

We can also send data to the subprocess via the communicate() method by setting the "input" argument in bytes.

For example:

...
# start a subprocess and redirect input
process = await asyncio.create_subprocess_shell('ls', stdin=asyncio.subprocess.PIPE)
# send data to the subprocess
process.communicate(input=b'Hello\n')

Behind the scenes the asyncio.subprocess.PIPE configures the subprocess to point to a StreamReader or StreamWriter for sending data to or from the subprocess, and the communicate() method will read or write bytes from the configured reader.

If PIPE is passed to stdin argument, the Process.stdin attribute will point to a StreamWriter instance. If PIPE is passed to stdout or stderr arguments, the Process.stdout and Process.stderr attributes will point to StreamReader instances.
-- Asyncio Subprocesses

We can interact with the StreamReader or StreamWriter directly via the subprocess via the stdin, stdout, and stderr attributes.

For example:

...
# read a line from the subprocess output stream
line = await process.stdout.readline()

Now that we know how to use the create_subprocess_shell() function, let's look at some worked examples.

Example of Asyncio create_subprocess_shell()

We can explore how to run a command in a subprocess from asyncio using the shell.

In this example, we will execute the "echo" command to report back a string.

The echo command will report the provided string on standard output directly.

The complete example is listed below.

Note, this example assumes you have access to the "echo" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of executing a shell command as a subprocess with asyncio
import asyncio

# main coroutine
async def main():
    # start executing a shell command in a subprocess
    process = await asyncio.create_subprocess_shell('echo Hello World')
    # report the details of the subprocess
    print(f'subprocess: {process}')

# entry point
asyncio.run(main())

Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.

The main() coroutine runs and calls the create_subprocess_shell() function to execute a command.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() coroutine resumes and reports the details of the subprocess. The main() process terminates and the asyncio program terminates.

The output of the echo command is reported on the command line.

This highlights how we can execute a command using the shell from an asyncio program.

subprocess: <Process 43916>
Hello World

Example of Waiting for Long Running Shell Subprocess

We can explore how to wait for a long-running command to run via the shell in asyncio.

In this example, we will execute the "sleep" command that will block the calling for a given number of seconds.

We will then wait for the subprocess to complete from the coroutine.

The complete example is listed below.

Note, this example assumes you have access to the "sleep" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of waiting for a long running a shell command with asyncio
import asyncio

# main coroutine
async def main():
    # start executing a shell command in a subprocess
    process = await asyncio.create_subprocess_shell('sleep 3')
    # report the details of the subprocess
    print(f'subprocess: {process}')
    # wait for the subprocess to terminate
    await process.wait()

# entry point
asyncio.run(main())

Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.

The main() coroutine runs and calls the create_subprocess_shell() function to execute a command.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() process resumes and reports the details of the subprocess.

It then suspends and waits for the subprocess to terminate.

The subprocess terminates, the main() coroutine resumes and closes the program.

This highlights how an asyncio program can execute a command in a subprocess and wait for the command to complete.

subprocess: <Process 43927>

Example of Reading Output From a Shell Subprocess

We can explore how to read output from a command run in a subprocess via the shell.

In this example, we will read data from the command executed in the subprocess.

We will execute the "echo" command and redirect output from stdout to the asyncio program. We will then read the output in the coroutine and report the result.

The complete example is listed below.

Note, this example assumes you have access to the "echo" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of reading output of command executed via shell in asyncio
import asyncio

# main coroutine
async def main():
    # start executing a shell command in a subprocess
    process = await asyncio.create_subprocess_shell('echo Hello World', stdout=asyncio.subprocess.PIPE)
    # report the details of the subprocess
    print(f'subprocess: {process}')
    # read a line of output from the program
    data, _ = await process.communicate()
    # report the data
    print(data)

# entry point
asyncio.run(main())

The main() coroutine runs and calls the create_subprocess_shell() function to execute a command and redirect output to the asyncio program.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() coroutine resumes and reports the details of the subprocess.

It then reads data from the subprocess, suspending until all data is received (e.g. an end-of-file character is sent).

The subprocess echos the string which is read by the coroutine and reported.

This highlights how we can read data from a command executed as a subprocess in an asyncio program.

subprocess: <Process 43999>
b'Hello World\n'

Example of Sending Input to a Shell Subprocess

We can explore how to write data to a command run in a subprocess via the shell.

In this example, we will execute the "cat" command via the shell. We will then write a string to the subprocess, which will be reported back via standard output.

The complete example is listed below.

Note, this example assumes you have access to the "cat" command, I'm not sure it will work on Windows.

# SuperFastPython.com
# example of writing input to a shell subprocess in asyncio
import asyncio

# main coroutine
async def main():
    # start executing a shell command in a subprocess
    process = await asyncio.create_subprocess_shell('cat', stdin=asyncio.subprocess.PIPE)
    # report the details of the subprocess
    print(f'subprocess: {process}')
    # write data to the process
    _ = await process.communicate(b'Hello World\n')

# entry point
asyncio.run(main())

The main() coroutine runs and calls the create_subprocess_shell() function to execute a command and redirect input from the asyncio program.

The main() coroutine suspends while the subprocess is created. A Process instance is returned.

The main() coroutine resumes and reports the details of the subprocess.

It then sends a string to the subprocess, encoded as bytes.

The main() coroutine terminates.

The subprocess reports the string sent to it and terminates.

This highlights how we can send data to a command executed as a subprocess in an asyncio program.

subprocess: <Process 44209>
Hello World

Takeaways

You now know how to run commands using the shell with asyncio in Python.

If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.