You can run a command as a subprocess with asyncio via the create_subprocess_exec() function.
In this tutorial, you will discover how to run commands with asyncio in Python.
Let’s get started.
What is Asyncio create_subprocess_exec()
The create_subprocess_exec() function allows commands to be executed from asyncio.
What is a Command
A command is a program executed on the command line (terminal or command prompt). It is another program that is run directly.
Common examples on Linux and macOS might be:
- ‘ls‘ to list the contents of a directory
- ‘cat‘ to report the content of a file
- ‘date‘ to report the date
- ‘echo‘ to report back a string
- ‘sleep‘ to sleep for a number of seconds
And so on.
These are just programs that we can execute on the command line as a command.
Asyncio create_subprocess_exec()
The asyncio.create_subprocess_exec() function takes a command and executes it directly.
This is helpful as it allows the command to be executed in a subprocess and for asyncio coroutines to read, write, and wait for it.
Because all asyncio subprocess functions are asynchronous and asyncio provides many tools to work with such functions, it is easy to execute and monitor multiple subprocesses in parallel.
— Asyncio Subprocesses
Unlike the asyncio.create_subprocess_shell() function, the asyncio.create_subprocess_exec() will not execute the command using the shell.
This means that the capabilities provided by the shell, such as shell variables, scripting, and wildcards are not available when executing the command.
It also means that executing the command may be more secure as there is no opportunity for a shell injection.
You can learn more about the create_subprocess_shell() function in the tutorial:
Now that we know what asyncio.create_subprocess_exec() does, let’s look at how to use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use Asyncio create_subprocess_exec()
The asyncio.create_subprocess_exec() function will execute a given string command in a subprocess.
It returns a asyncio.subprocess.Process object that represents the subprocess.
Process is a high-level wrapper that allows communicating with subprocesses and watching for their completion.
— Interacting with Subprocesses
The create_subprocess_exec() function is a coroutine, which means we must await it. It will return once the subprocess has been started, not when the subprocess is finished.
For example:
|
1 2 3 |
... # execute a command in a subprocess process = await asyncio.create_subprocess_exec('ls') |
Arguments to the command being executed must be provided as subsequent arguments to the create_subprocess_exec() function.
For example:
|
1 2 3 |
... # execute a command with arguments in a subprocess process = await asyncio.create_subprocess_exec('ls', '-l') |
We can wait for the subprocess to finish by awaiting the wait() method.
For example:
|
1 2 3 |
... # wait for the subprocess to terminate await process.wait() |
We can stop the subprocess directly by calling the terminate() or kill() methods, which will raise a signal in the subprocess.
For example:
|
1 2 3 |
... # terminate the subprocess process.terminate) |
The input and output of the command will be handled by stdin, stderr, and stdout.
We can have the asyncio program handle the input or output for the subprocess.
This can be achieved by specifying the input or output stream and specifying a constant to redirect, such as asyncio.subprocess.PIPE.
For example, we can redirect the output of a command to the asyncio program:
|
1 2 3 |
... # start a subprocess and redirect output process = await asyncio.create_subprocess_exec('ls', stdout=asyncio.subprocess.PIPE) |
We can then read the output of the program via the asyncio.subprocess.Process instance via the communicate() method.
This method is a coroutine and must be awaited. It is used to both send and receives data with the subprocess.
For example:
|
1 2 3 |
... # read data from the subprocess line = process.communicate() |
We can also send data to the subprocess via the communicate() method by setting the “input” argument in bytes.
For example:
|
1 2 3 4 5 |
... # start a subprocess and redirect input process = await asyncio.create_subprocess_exec('ls', stdin=asyncio.subprocess.PIPE) # send data to the subprocess process.communicate(input=b'Hello\n') |
Behind the scenes the asyncio.subprocess.PIPE configures the subprocess to point to a StreamReader or StreamWriter for sending data to or from the subprocess, and the communicate() method will read or write bytes from the configured reader.
If PIPE is passed to stdin argument, the Process.stdin attribute will point to a StreamWriter instance. If PIPE is passed to stdout or stderr arguments, the Process.stdout and Process.stderr attributes will point to StreamReader instances.
— Asyncio Subprocesses
We can interact with the StreamReader or StreamWriter directly via the subprocess via the stdin, stdout, and stderr attributes.
For example:
|
1 2 3 |
... # read a line from the subprocess output stream line = await process.stdout.readline() |
Now that we know how to use the create_subprocess_exec() function, let’s look at some worked examples.
Example of Asyncio create_subprocess_exec()
We can explore how to run a command in a subprocess from asyncio.
In this example, we will execute the “echo” command to report back a string.
The echo command will report the provided string on standard output directly.
The complete example is listed below.
Note, this example assumes you have access to the “echo” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# SuperFastPython.com # example of executing a command as a subprocess with asyncio import asyncio # main coroutine async def main(): # start executing a command in a subprocess process = await asyncio.create_subprocess_exec('echo', 'Hello World') # report the details of the subprocess print(f'subprocess: {process}') # entry point asyncio.run(main()) |
Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.
The main() coroutine runs and calls the create_subprocess_exec() function to execute a command.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() coroutine resumes and reports the details of the subprocess. The main() process terminates and the asyncio program terminates.
The output of the echo command is reported on the command line.
This highlights how we can execute a command from an asyncio program.
|
1 2 |
Hello World subprocess: <Process 50249> |
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Waiting for Long Running Subprocess
We can explore how to wait for a long-running command to run in asyncio.
In this example, we will execute the “sleep” command that will block the calling for a given number of seconds.
We will then wait for the subprocess to complete from the coroutine.
The complete example is listed below.
Note, this example assumes you have access to the “sleep” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# SuperFastPython.com # example of waiting for a long running a command with asyncio import asyncio # main coroutine async def main(): # start executing a command in a subprocess process = await asyncio.create_subprocess_exec('sleep', '3') # report the details of the subprocess print(f'subprocess: {process}') # wait for the subprocess to terminate await process.wait() # entry point asyncio.run(main()) |
Running the example first creates the main() coroutine and executes it as the entry point into the asyncio program.
The main() coroutine runs and calls the create_subprocess_exec() function to execute a command.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() process resumes and reports the details of the subprocess.
It then suspends and waits for the subprocess to terminate.
The subprocess terminates, the main() coroutine resumes and closes the program.
This highlights how an asyncio program can execute a command in a subprocess and wait for the command to complete.
|
1 |
subprocess: <Process 50257> |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Reading Output From a Subprocess
We can explore how to read output from a command run in a subprocess.
In this example, we will read data from the command executed in the subprocess.
We will execute the “echo” command and redirect output from stdout to the asyncio program. We will then read the output in the coroutine and report the result.
The complete example is listed below.
Note, this example assumes you have access to the “echo” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # example of reading output of command executed via in asyncio import asyncio # main coroutine async def main(): # start executing a command in a subprocess process = await asyncio.create_subprocess_exec('echo', 'Hello World', stdout=asyncio.subprocess.PIPE) # report the details of the subprocess print(f'subprocess: {process}') # read a line of output from the program data, _ = await process.communicate() # report the data print(data) # entry point asyncio.run(main()) |
The main() coroutine runs and calls the create_subprocess_exec() function to execute a command and redirect output to the asyncio program.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() coroutine resumes and reports the details of the subprocess.
It then reads data from the subprocess, suspending until all data is received (e.g. an end-of-file character is sent).
The subprocess echos the string which is read by the coroutine and reported.
This highlights how we can read data from a command executed as a subprocess in an asyncio program.
|
1 2 |
subprocess: <Process 50261> b'Hello World\n' |
Example of Sending Input to a Subprocess
We can explore how to write data to a command run in a subprocess with asyncio.
In this example, we will execute the “cat” command. We will then write a string to the subprocess, which will be reported back via standard output.
The complete example is listed below.
Note, this example assumes you have access to the “cat” command, I’m not sure it will work on Windows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# SuperFastPython.com # example of writing input to a subprocess in asyncio import asyncio # main coroutine async def main(): # start executing a command in a subprocess process = await asyncio.create_subprocess_exec('cat', stdin=asyncio.subprocess.PIPE) # report the details of the subprocess print(f'subprocess: {process}') # write data to the process _ = await process.communicate(b'Hello World\n') # entry point asyncio.run(main()) |
The main() coroutine runs and calls the create_subprocess_exec() function to execute a command and redirect input from the asyncio program.
The main() coroutine suspends while the subprocess is created. A Process instance is returned.
The main() coroutine resumes and reports the details of the subprocess.
It then sends a string to the subprocess, encoded as bytes.
The main() coroutine terminates.
The subprocess reports the string sent to it and terminates.
This highlights how we can send data to a command executed as a subprocess in an asyncio program.
|
1 2 |
subprocess: <Process 50266> Hello World |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to run commands with asyncio in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Janne Aspegren on Unsplash
What is the different between
import asyncio then use subprocess and import subprocess?
Which one is better?
Many thanks
Good question.
They both support subprocesses, except asyncio.subprocess allows coroutines to read/write to subprocesses asynchronously, e.g. without blocking the event loop.