You can run a command using a shell as a subprocess with asyncio via the create_subprocess_shell() function.
In this tutorial, you will discover how to run commands using the shell with asyncio in Python.
Let’s get started.
What is Asyncio create_subprocess_shell()
The asyncio.create_subprocess_shell() function allows commands to be executed using the shell from asyncio.
What is a Command
A command is a program executed on the command line (terminal or command prompt). It is another program that is run directly.
Common examples on Linux and macOS might be:
- ‘ls‘ to list the contents of a directory
- ‘cat‘ to report the content of a file
- ‘date‘ to report the date
- ‘echo‘ to report back a string
- ‘sleep‘ to sleep for a number of seconds
And so on.
These are just programs that we can execute on the command line as a command.
What is a Shell
We can execute these commands using the shell.
The shell is a user interface for the command line, called a command line interpreter (CLI).
It will interpret and execute commands on behalf of the user.
It also offers features such as a primitive programming language for scripting, wildcards, piping, shell variables (e.g. PATH), and more.
For example, we can redirect the output of one command as input to another command, such as the contents of the “/etc/services” file into the word count “wc” command and count the number of lines:
|
1 |
cat /etc/services | wc -l |
Examples of shells in the Unix based operating systems include:
- ‘sh‘
- ‘bash‘
- ‘zsh‘
- And so on.
On Windows, the shell is probably cmd.exe.
See this great list of command line shells:
The shell is already running, it was used to start the Python program.
You don’t need to do anything special to get or have access to the shell.
Asyncio create_subprocess_shell()
The asyncio.create_subprocess_shell() function takes a command and executes it using the current user shell.
This is helpful as it not only allows the command to be executed, but allows the capabilities of the shell to be used, such as redirection, wildcards and more.
… the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
— subprocess — Subprocess management
The command will be executed in a subprocess of the process executing the asyncio program.
Importantly, the asyncio program is able to interact with the subprocess asynchronously, e.g. via coroutines.
Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel.
— Asyncio Subprocesses
There can be security considerations when executing a command via the shell instead of directly.
This is because there is at least one level of indirection and interpretation between the request to execute the command and the command being executed, allowing possible malicious injection.
Important It is the application’s responsibility to ensure that all whitespace and special characters are quoted appropriately to avoid shell injection vulnerabilities.
— Asyncio Subprocesses
Now that we know what asyncio.create_subprocess_shell() does, let’s look at how to use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use Asyncio create_subprocess_shell()
The asyncio.create_subprocess_shell() function will execute a given string command via the current shell.
It returns a asyncio.subprocess.Process object that represents the process.
Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.
— Interacting with Subprocesses
The create_subprocess_shell() function is a coroutine, which means we must await it. It will return once the subprocess has been started, not when the subprocess is finished.
For example:
|
1 2 3 |
... # start a subprocess process = await asyncio.create_subprocess_shell('ls') |
We can wait for the subprocess to finish by awaiting the wait() method.
For example:
|
1 2 3 |
... # wait for the subprocess to terminate await process.wait() |
We can stop the subprocess directly by calling the terminate() or kill() methods, which will raise a signal in the subprocess.
For example:
|
1 2 3 |
... # terminate the subprocess process.terminate) |
The input and output of the command will be handled by the shell, e.g. stdin, stderr, and stdout.
We can have the asyncio program handle the input or output for the subprocess.
This can be achieved by specifying the input or output stream and specifying a constant to redirect, such as asyncio.subprocess.PIPE.
For example, we can redirect the output of a command to the asyncio program:
|
1 2 3 |
... # start a subprocess and redirect output process = await asyncio.create_subprocess_shell('ls', stdout=asyncio.subprocess.PIPE) |
We can then read the output of the program via the asyncio.subprocess.Process instance via the communicate() method.
This method is a coroutine and must be awaited. It is used to both send and receives data with the subprocess.
For example:
|
1 2 3 |
... # read data from the subprocess line = process.communicate() |
We can also send data to the subprocess via the communicate() method by setting the “input” argument in bytes.
For example:
|
1 2 3 4 5 |
... # start a subprocess and redirect input process = await asyncio.create_subprocess_shell('ls', stdin=asyncio.subprocess.PIPE) # send data to the subprocess process.communicate(input=b'Hello\n') |
Behind the scenes the asyncio.subprocess.PIPE configures the subprocess to point to a StreamReader or StreamWriter for sending data to or from the subprocess, and the communicate() method will read or write bytes from the configured reader.
If PIPE is passed to stdin argument, the Process.stdin attribute will point to a StreamWriter instance. If PIPE is passed to stdout or stderr arguments, the Process.stdout and Process.stderr attributes will point to StreamReader instances.
— Asyncio Subprocesses
We can interact with the StreamReader or StreamWriter directly via the subprocess via the stdin, stdout, and stderr attributes.
For example:
|
1 2 3 |
... # read a line from the subprocess output stream line = await process.stdout.readline() |
Now that we know how to use the create_subprocess_shell() function, let’s look at some worked examples.
Example of Asyncio create_subprocess_shell()
We can explore how to run a command in a subprocess from asyncio using the shell.
In this example, we will execute the “echo” command to report back a string.
The echo command will report the provided string on standard output directly.
The complete example is listed below.
Note, this example assumes you have access to the “echo” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# SuperFastPython.com # example of executing a shell command as a subprocess with asyncio import asyncio # main coroutine async def main(): # start executing a shell command in a subprocess process = await asyncio.create_subprocess_shell('echo Hello World') # report the details of the subprocess print(f'subprocess: {process}') # entry point asyncio.run(main()) |
Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.
The main() coroutine runs and calls the create_subprocess_shell() function to execute a command.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() coroutine resumes and reports the details of the subprocess. The main() process terminates and the asyncio program terminates.
The output of the echo command is reported on the command line.
This highlights how we can execute a command using the shell from an asyncio program.
|
1 2 |
subprocess: <Process 43916> Hello World |
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Waiting for Long Running Shell Subprocess
We can explore how to wait for a long-running command to run via the shell in asyncio.
In this example, we will execute the “sleep” command that will block the calling for a given number of seconds.
We will then wait for the subprocess to complete from the coroutine.
The complete example is listed below.
Note, this example assumes you have access to the “sleep” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# SuperFastPython.com # example of waiting for a long running a shell command with asyncio import asyncio # main coroutine async def main(): # start executing a shell command in a subprocess process = await asyncio.create_subprocess_shell('sleep 3') # report the details of the subprocess print(f'subprocess: {process}') # wait for the subprocess to terminate await process.wait() # entry point asyncio.run(main()) |
Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.
The main() coroutine runs and calls the create_subprocess_shell() function to execute a command.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() process resumes and reports the details of the subprocess.
It then suspends and waits for the subprocess to terminate.
The subprocess terminates, the main() coroutine resumes and closes the program.
This highlights how an asyncio program can execute a command in a subprocess and wait for the command to complete.
|
1 |
subprocess: <Process 43927> |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Reading Output From a Shell Subprocess
We can explore how to read output from a command run in a subprocess via the shell.
In this example, we will read data from the command executed in the subprocess.
We will execute the “echo” command and redirect output from stdout to the asyncio program. We will then read the output in the coroutine and report the result.
The complete example is listed below.
Note, this example assumes you have access to the “echo” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of reading output of command executed via shell in asyncio import asyncio # main coroutine async def main(): # start executing a shell command in a subprocess process = await asyncio.create_subprocess_shell('echo Hello World', stdout=asyncio.subprocess.PIPE) # report the details of the subprocess print(f'subprocess: {process}') # read a line of output from the program data, _ = await process.communicate() # report the data print(data) # entry point asyncio.run(main()) |
The main() coroutine runs and calls the create_subprocess_shell() function to execute a command and redirect output to the asyncio program.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() coroutine resumes and reports the details of the subprocess.
It then reads data from the subprocess, suspending until all data is received (e.g. an end-of-file character is sent).
The subprocess echos the string which is read by the coroutine and reported.
This highlights how we can read data from a command executed as a subprocess in an asyncio program.
|
1 2 |
subprocess: <Process 43999> b'Hello World\n' |
Example of Sending Input to a Shell Subprocess
We can explore how to write data to a command run in a subprocess via the shell.
In this example, we will execute the “cat” command via the shell. We will then write a string to the subprocess, which will be reported back via standard output.
The complete example is listed below.
Note, this example assumes you have access to the “cat” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# SuperFastPython.com # example of writing input to a shell subprocess in asyncio import asyncio # main coroutine async def main(): # start executing a shell command in a subprocess process = await asyncio.create_subprocess_shell('cat', stdin=asyncio.subprocess.PIPE) # report the details of the subprocess print(f'subprocess: {process}') # write data to the process _ = await process.communicate(b'Hello World\n') # entry point asyncio.run(main()) |
The main() coroutine runs and calls the create_subprocess_shell() function to execute a command and redirect input from the asyncio program.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() coroutine resumes and reports the details of the subprocess.
It then sends a string to the subprocess, encoded as bytes.
The main() coroutine terminates.
The subprocess reports the string sent to it and terminates.
This highlights how we can send data to a command executed as a subprocess in an asyncio program.
|
1 2 |
subprocess: <Process 44209> Hello World |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to run commands using the shell with asyncio in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Lance Asper on Unsplash
I am new newbie to Python and trying to – what I would think is simple – loop through every single aiohttp session that is open on my client and kill it. I have rebooted, pip uninstalled aithttp and all that and still I get a “Unclosed Client Session” error in my code. I would like to work on fixing my broken code myself, but I’m only asking for [separately] how in the world do I kill ALL aiohttp sessions (btw, this is a Windows box and yes, I have the latest aiohttp, etc, etc.) Thanks so much, sir.