Last Updated on August 21, 2023
You can perform asynchronous file IO in AsyncIO programs using the AIOFiles library.
In this tutorial you will discover how to manipulate files asynchronously in asyncio programs using the aiofiles library in Python.
Let’s get started.
Python Asynchronous I/O
Asynchronous I/O or AsyncIO refers to the Python module that supports coroutine-based concurrency via the async/await syntax.
Coroutines are executed concurrently and explicitly yield control to other coroutines when performing IO operations.
Coroutines are defined by functions or blocks starting with the async keyword, for example:
|
1 2 3 4 |
# defines a function as a coroutine async def work() # yield control at this point asyncio.sleep(1) |
Coroutines can then be created and executed by using the await keyword, for example:
|
1 2 3 |
... # create and execute a coroutine await work() |
In order to execute coroutines, you must start the async event loop and specify the entry point into your program,the top-level coroutine, for example:
|
1 2 3 |
... # start the event loop asyncio.run(main()) |
Run loops using all CPUs, download your FREE book to learn how.
AIOFiles for File Support In AsyncIO
The asyncio module provides a suite of tools for asynchronous programming and is focused on asynchronous networking programming, e.g. socket IO.
asyncio is often a perfect fit for IO-bound and high-level structured network code.
— asyncio — Asynchronous I/O
At the time of writing (Python 3.10), the asyncio module does not support asynchronous file IO operations.
This means that if you want to perform file IO operations in your asyncio program, such as reading or writing data to disk, you must either use blocking IO which is antithetical to the asynchronous programming paradigm, or find a workaround.
One workaround is to perform blocking IO in a worker thread within your asyncio program via the asyncio.to_thread() function.
A more complete solution is to use the aiofiles third-party library.
The aiofiles library was created and maintained by Tin Tvrtković from Croatia.
It is described as “File support for asyncio” and provides a range of standard IO operations for use in asyncio programs.
Ordinary local file IO is blocking, and cannot easily and portably made asynchronous. This means doing file IO may interfere with asyncio applications, which shouldn’t block the executing thread. aiofiles helps with this by introducing asynchronous versions of files that support delegating operations to a separate thread pool.
— aiofiles: File support for asyncio
True asynchronous file IO, also called non-blocking file IO, is provided by modern operating systems such as Windows, MacOS and Linux, although is not provided in a consistent manner. This may be the reason why the Python standard library does not currently support async file IO, or perhaps the focus of asyncio is on network programming.
The aiofiles library does not implement True asynchronous file IO, instead, it simulated asynchronous file IO using worker threads under the covers.
This means the solution is similar to manually calling asyncio.to_thread() for file IO operations, although with a more natural syntax.
Next, let’s look at how we might perform a suite of standard file IO operations using the aiofiles library in asyncio programs.
How to Use AIOFiles with AsyncIO
The aiofiles library provides a wide range of file IO operations for use in asyncio programs.
The library can be installed using your favourite package manager, such as pip, for example:
|
1 |
pip3 install aiofiles |
Once installed, you can import the module and begin using file IO operations in your asyncio programs.
This section will review how to perform a range of standard file IO operations in asyncio programs using the aiofiles library.
The hope is that you can copy-paste the examples and bring asynchronous file IO directly into your project.
Are there any async file operations not listed below that you need help with?
Let me know in the comments below.
Create a File
A file can be opened asynchronously via the aiofiles.open() function.
This function takes the same arguments as the built-in open() function, such as the path and name of the file and the open mode and returns a Python file object.
|
1 2 3 4 5 |
... # create a file handle = await aiofiles.open('test_create.txt', mode='x') # ... handle.close() |
Unlike the built-in function, the aiofiles.open() function can be awaited, yielding execution control.
The example below shows how to create a new file in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# SuperFastPython.com # example of creating a file with asyncio and aiofiles import asyncio import aiofiles # create a file async def main(): # create a file with no content handle = await aiofiles.open('test_create.txt', mode='x') # close the file handle.close() # entry point asyncio.run(main()) |
Running the program creates a new empty file named ‘test_create.txt‘ with no content in your current working directory.
|
1 |
test_create.txt |
Note, if the ‘test_create.txt‘ file already exists because you ran the example twice, you will get an error like the following:
|
1 |
FileExistsError: [Errno 17] File exists: 'test_create.txt' |
Write a File
A file can be written asynchronously by first opening it via the aiofiles.open() function and then calling the write() function.
Like the built-in open() function, the aiofiles.open() function can be called directly or used via a context manager interface via the “with” keyword.
The with block is defined as a coroutine with the “async” keyword, and calls to the write() function can be awaited.
For example:
|
1 2 3 4 5 |
... # create a file async with aiofiles.open('test_write.txt', mode='w') as handle: # write to the file await handle.write('Hello world') |
The example below shows how to write to a file in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# SuperFastPython.com # example of writing to a file with asyncio and aiofiles import asyncio import aiofiles # write a file async def main(): # open the file async with aiofiles.open('test_write.txt', mode='w') as handle: # write to the file await handle.write('Hello world') # entry point asyncio.run(main()) |
Running the program creates a new file named “test_write.txt” in the current working directory and writes the string “Hello world” asynchronously to the file.
|
1 |
Hello world |
Read a File
A file can be read asynchronously by first opening it via the aiofiles.open() function then calling the read() function.
Calls to the read() function can be awaited like calls to write(), yielding control of execution while the operation is being performed.
|
1 2 3 |
... # read the contents of the file data = await handle.read() |
The example below reads the file /etc/services, standard on all POSIX operating systems, then reports the contents.
Note, if you don’t have an ‘/etc/services‘ file, you can change the program to read another standard text file on your system.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# SuperFastPython.com # example of reading to a file with asyncio and aiofiles import asyncio import aiofiles # read a file async def main(): # open the file async with aiofiles.open('/etc/services', mode='r') as handle: # read the contents of the file data = await handle.read() print(f'Read {len(data)} bytes') # entry point asyncio.run(main()) |
Running the example opens and reads the contents of the file asynchronously, and reports the number of bytes read.
|
1 |
Read 677972 bytes |
Make Directories
New directories can be created asynchronously via the aiofiles.os.makedirs() function.
This function takes the same arguments as the os.makedirs() function, such as the path containing directories to create and the “exist_ok” argument that if set to True will ignore the case if the directories already exist.
Importantly, the aiofiles.os.makedirs() function is awaitable.
|
1 2 3 |
... # create a directory await aiofiles.os.makedirs('tmp', exist_ok=True) |
The example below shows how to create directories in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# SuperFastPython.com # example of creating a directory asyncio and aiofiles import asyncio import aiofiles import aiofiles.os # create a directory async def main(): # create a directory await aiofiles.os.makedirs('tmp', exist_ok=True) # entry point asyncio.run(main()) |
Running the example creates a new tmp/ subdirectory asynchronously in the current working directory
|
1 |
tmp/ |
Rename Files
Files can be renamed asynchronously via the aiofiles.os.rename() function.
This function takes the same arguments as the os.rename() function such as the source file path and the destination file path.
Importantly, the call to the aiofiles.os.rename() function is awaitable, meaning that execution control will be yielded while the blocking operation is being performed.
|
1 2 3 |
... # rename the file await aiofiles.os.rename('files_rename.txt', 'files_rename2.txt') |
The example below shows how to rename a file in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# SuperFastPython.com # example of renaming a file asyncio and aiofiles import asyncio import aiofiles import aiofiles.os # rename a file async def main(): # create a file with no content handle = await aiofiles.open('files_rename.txt', mode='x') handle.close() # rename the file await aiofiles.os.rename('files_rename.txt', 'files_rename2.txt') # entry point asyncio.run(main()) |
Running the example first asynchronously creates a file named “files_rename.txt“.
It then asynchronously renames the file “files_rename.txt” to the new name “files_rename2.txt“.
|
1 |
files_rename2.txt |
Move a File
Files can be moved asynchronously via the aiofiles.os.replace() function.
This function takes the same argument as the os.replace() function, such as the source file path and the destination file path.
Importantly, the aiofiles.os.replace() function is awaitable.
|
1 2 3 |
... # move the file into the directory await aiofiles.os.replace('files_move.txt', 'tmp/files_move.txt') |
The example below shows how to move a file in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# SuperFastPython.com # example of moving a file asyncio and aiofiles import asyncio import aiofiles import aiofiles.os # move a file async def main(): # create a file with no content handle = await aiofiles.open('files_move.txt', mode='x') handle.close() # create a directory await aiofiles.os.makedirs('tmp', exist_ok=True) # move the file into the directory await aiofiles.os.replace('files_move.txt', 'tmp/files_move.txt') # entry point asyncio.run(main()) |
Running the example first asynchronously creates a file named “files_move.txt“.
A new subdirectory named tmp/ is then created asynchronously.
Finally, the file “files_move.txt” is asynchronously moved under the tmp/ subdirectory.
|
1 |
tmp/files_move.txt |
Delete a File
Files can be deleted asynchronously via the aiofiles.os.remove() function.
This function takes the same argument as the os.remove() function, such as the file path of the file to be deleted.
Unlike the os.remove() function, the aiofiles.os.remove() function is awaitable meaning that it will yield execution control while the blocking IO operation is being performed.
|
1 2 3 |
... # delete the file await aiofiles.os.remove('files_delete.txt') |
The example below shows how to delete a file in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# SuperFastPython.com # example of deleting a file asyncio and aiofiles import asyncio import aiofiles import aiofiles.os # delete a file async def main(): # create a file with no content handle = await aiofiles.open('files_delete.txt', mode='x') handle.close() # delete the file await aiofiles.os.remove('files_delete.txt') # entry point asyncio.run(main()) |
Running the example, first asynchronously creates a new file named “files_delete.txt” in the current working directory.
Then the file “files_delete.txt” is asynchronously deleted.
Copy a File
Files can be copied asynchronously via the aiofiles.os.sendfile() function.
This function takes the same argument as the os.sendfile() function, such as the file descriptors for the destination file and source file, the offset for reading from the source file and the number of bytes to copy to the destination file.
Importantly, the aiofiles.os.sendfile() function is awaitable.
|
1 2 3 |
... # copy the file await aiofiles.os.sendfile(fd_dst, fd_src, 0, n_bytes) |
The example below shows how to copy a file in an asyncio program.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of copying a file asyncio and aiofiles import asyncio import aiofiles import aiofiles.os # copy a file async def main(): # create a file with some content async with aiofiles.open('files_copy.txt', mode='w') as handle: # write some content await handle.write('hello world') # create file objects for the source and destination handle_src = await aiofiles.open('files_copy.txt', mode='r') handle_dst = await aiofiles.open('files_copy2.txt', mode='w') # get the number of bytes for the source stat_src = await aiofiles.os.stat('files_copy.txt') n_bytes = stat_src.st_size # get the file descriptors for the source and destination files fd_src = handle_src.fileno() fd_dst = handle_dst.fileno() # copy the file await aiofiles.os.sendfile(fd_dst, fd_src, 0, n_bytes) # entry point asyncio.run(main()) |
Running the example first asynchronously creates a file named “files_copy.txt” in the current working directory and writes a string as contents to the file.
It then asynchronously opens and creates file objects for both the source file “files_copy.txt” and the destination file “files_copy2.txt“.
Next, the status of the source file “files_copy.txt” is obtained asynchronously and the size of the contents of the file is recorded.
Finally, the file descriptors are retrieved for each file and the source file is copied to the destination file.
Unfortunately, the os.sendfile() function and similarly the aiofiles.os.sendfile() function is not supported on all platforms.
At the time of writing (Python 3.10) and on my system (MacOS), the example results in the error:
|
1 |
OSError: [Errno 38] Socket operation on non-socket |
Does this example work on your system?
Let me know in the comments below.
Free Concurrent File I/O Course
Get FREE access to my 7-day email course on concurrent File I/O.
Discover patterns for concurrent file I/O, how save files with a process pool, how to copy files with a thread pool, and how to append to a file from multiple threads safely.
Further Reading
This section provides additional resources that you may find helpful.
Books
- Concurrent File I/O in Python, Jason Brownlee (my book!)
Guides
Python File I/O APIs
- Built-in Functions
- os - Miscellaneous operating system interfaces
- os.path - Common pathname manipulations
- shutil - High-level file operations
- zipfile — Work with ZIP archives
- Python Tutorial: Chapter 7. Input and Output
Python Concurrency APIs
- threading — Thread-based parallelism
- multiprocessing — Process-based parallelism
- concurrent.futures — Launching parallel tasks
- asyncio — Asynchronous I/O
File I/O in Asyncio
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to manipulate files asynchronously in asyncio programs using the aiofiles library.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Clint Patterson on Unsplash
Very nice/instructive article. Thanks !!
Just one question :
If there is no “consistent” way to achieve “true” I/O files in Python, how does node.js do (for instance)
Thanks.
No, not at this stage.
Thanks so much! This is really helpful!
Its like starting again with python and I like the challenge and its worth it to async things.
Some async ‘directory type things’ would be helpful too. Like async disk crawl with a minor file operation per file (like a stat) for example.
Thanks again!
You’re very welcome!