Last Updated on August 21, 2023

Loading files from disk in Python is typically a slow operation.

It can become painfully slow in situations where you may need to load thousands of files into memory. The hope is that multi-threading the file loading will speed up the operation.

In this tutorial, you will discover how to explore concurrent file loading in Python.

Let’s dive in.

Create 10,000 Files (to load)

Before we can load thousands of files from disk, we must create them.

We can write a script that will create 10,000 CSV files, each file with 10,000 lines and each line containing 10 random numeric values.

First, we can create a function that will take a file path and string data and save it to file.

The save_file() function below implements this, taking the full path and data, opening the file in ASCII format, then saving the data to the file. The context manager is used so that the file is closed automatically.

Next, we can generate the data to be saved.

First, we can write a function to generate a single line of data. In this case, a line is composed of 10 random numeric values in the range between 0 and 1, stored in CSV format (e.g. separated by a comma per value).

The generate_line() function below implements this, returning a string of random data.

Next, we can write a function to generate the contents of one file.

This will be 10,000 lines of random values, where each line is separated by a new line.

The generate_file_data() function below implements this, returning a string representing the data for a single data file.

Finally, we can generate data for all of the files and save each with a separate file name.

First, we can create a directory to store all of the created files (e.g. ‘tmp‘) under the current working directory.

We can then loop 10,000 times and create the data and a unique filename for each iteration, then save the generated contents of the file to disk.

The generate_all_files() function listed below implements this, creating the 10,000 files of 10,000 lines of data.

Tying this together, the complete example of creating a large number of CSV files is listed below.

Running the example will create 10,000 CSV files of random data into a tmp/ directory.

It takes a long time to run, depending on the speed of your hard drive.

On my system, it takes about 11 minutes to complete.

Next, we can explore ways of loading all of these files into memory.

Load Files One-by-One (slowly)

There are many applications where we need to load many files into memory.

A common example is loading a large number of CSV files into a large in-memory dataset in order to perform some analysis or modelling.

First, we can develop a function to load a single file into memory using the open() built-in function.

The load_file() function below implements this, taking the file path of the file to load and returning the contents of the file.

The file is opened as an ASCII file and returns the contents as a string. The context manager is used when opening the file, to ensure that the file handle is closed once we are finished with it.

We can then call this function for each file listed in a directory, such as ‘tmp‘ we created in the previous section.

First, we can create a list of all file paths that need to be loaded.

We can then iterate over this list of paths and load each file in turn and report progress along the way.

The load_all_files() function below implements this, loading all files in the directory and reporting progress along the way.

Tying this together, the complete example of loading all 10,000 files in the directory sequentially is listed below.

Running the example loads all files in the directory into memory.

The files are loaded in the order that files are returned from the listdir() order, probably not ordered by filename.

It takes a moment to complete.

On my system, all files are loaded in about 10.7 seconds.

How long does it take on your system?
Let me know in the comments below.

Next, let’s explore whether we can speed up the program by using multithreading.

How to Load Files Concurrently

Loading files involves copying data from disk into main memory.

We would expect that this is an IO-bound task. This is because the speed of loading a file is bounded by the speed that the hard drive can locate the file, read the bytes of the file and transfer those bytes into main memory. This is compared to CPU-bound tasks that run as fast as the CPU.

This suggests that using threads is appropriate instead of processes that are better suited to CPU-bound tasks. It also means we can have many more threads than we do CPUs, e.g. perhaps hundreds or thousands of threads in this case.

Naively, we might reasonably expect that hard drives are only able to load a single file into main memory at a time.

If true, this suggests that although loading files might be made concurrent with threads or processes, we may not expect any speed benefit because load operations cannot be performed in parallel.

Nevertheless, there may be some internal buffering or some ability to batch load operations by the operating system or hard disk hardware that we could exploit.

Now that we have some expectations of the effect of concurrent file loading, let’s explore a worked example.

Load Files Concurrently with Threads

The ThreadPoolExecutor provides a pool of worker threads that we can use to multithread the loading of thousands of data files.

Each file in the tmp/ directory will represent a task that will require the file to be loaded from disk into main memory.

First, we can define a function that takes the path of a file to load and performs the read operation returning the contents of the file and the file path.

Next, we can call this function for each file in the tmp/ directory.

First, we can create the thread pool with ten worker threads. We will use the context manager to ensure the thread pool is closed automatically once all pending and running tasks have been completed.

We can then call the submit() function for each file to be loaded, calling the load_file() function with the file path as an argument.

This can be achieved in a list comprehension, resulting in a list of Future objects.

We can then iterate over the Future objects and get the loaded data and file path information and report progress. This is to simulate some use of the loaded data, like collating it into one in-memory data structure or using the file contents to compute something.

The as_completed() function lists us iterating over the tasks in the order that they are completed.

Tying this together, the complete example of multithreaded loading of files from one directory into memory is listed below.

Running the example loads all 10,000 files into memory and reports a message as the contents of each file is made available.

In this case, the operation is performed relatively quickly, but not quite as fast as the single-threaded example.

On my system it takes about 11.93 seconds, compared to 10.7 for the sequential case, or about 1.1x slower.

Perhaps 10 threads is too few.

We can run the example again with 100 threads instead to see if it makes any difference.

This requires a single change to the number of worker threads in the constructor to the ThreadPoolExecutor, for example:

Running the example loads all files from the source directory, as before.

Using 100 threads does result in a small increase in performance compared to using 10 threads.

On my system it takes about 11.26 seconds compared to 10 threads that took about 11.93 seconds, but this may be within the margin of measurement error. Compared to the single-threaded version it is about 1.1x slower.

Next, let’s see if we can get any benefit from batching file loads into chunks for the worker threads.

Load Files Concurrently with Threads in Batch

Each task in the thread pool adds overhead that slows down the overall task.

This includes function calls and object creation for each task, occurring internally within the ThreadPoolExecutor class.

We might reduce this overhead by performing file operations in a batch within each worker thread.

This can be achieved by updating the load_file() function to receive a list of files to load, and splitting up the files in the main() function into chunks to be submitted to worker threads for batch processing.

The hope is that this would reduce some of the overhead required for submitting so many small tasks by replacing it with a few larger batched tasks.

First, we can change the target function to receive a list of file paths to load, listed below.

Next, in the main() function we can select a chunksize based on the number of worker threads and the number of files to load.

In this case, we will use 100 worker threads with 10,000 files to load. Dividing the files evenly between workers (10,000 / 100) or 100 files to load for each worker.

This means that we may have up to 10,000 * 2 megabyte files (20 gigabytes) in memory at one time in the worst case. If you don’t have a ton of main memory, you might want to set a different value to chunksize, like 10 or 50.

Next, we can create the thread pool with the parameterized number of worker threads.

Next, we can iterate over the list of files to load and split them into chunks defined by a chunksize.

This can be achieved by iterating over the list of files and using the chunksize as the increment size in a for-loop. We can then split off a chunk of files to send to a worker thread as a task.

We will keep track of the Future objects for each task for iterating later.

Finally, we can iterate the Future objects in the order of task completion, as before, but this time, report the file paths loaded by each task.

And that’s it.

The complete example of loading files in a batch concurrently using worker threads is listed below.

All 10,000 files are loaded from tmp/ into main memory as before.

In this case, we don’t see any benefit from the concurrent loading of files in batches.

On my system, the example took about 11.16 seconds to complete, compared to about 11.26 seconds without batch mode, which is probably within the margin of measurement error.

Next, let’s see if we can get any benefit using processes instead of threads.

Load Files Concurrently with Processes

We can also try to load files concurrently with processes instead of threads.

It is unclear whether processes can offer a speed benefit in this case. Given that we cannot get a benefit using threads.

Nevertheless, using processes requires data for each task to be serialized which introduces additional overhead that might negate any speed-up from performing file operations with true parallelism via processes.

In fact, the need to transmit the contents of each file from child processes back to the main process will likely add a large overhead, given that the contents of each file must be serialized and then deserialized.

We can explore using processes to load files concurrently using the ProcessPoolExecutor.

This can be achieved by switching out the ThreadPoolExecutor directly and specifying the number of worker processes.

We will use 4 processes in this case as I have 4 physical CPU cores.

It may be interesting to try different configurations of the number of worker processes to see if it makes a difference on the overall running time.

Tying this together, the complete example of loading files concurrently using the process pool is listed below.

Running the example loads all files from the directory as we did before.

In this case, we do not see a speed improvement over the single-threaded case. In fact, we see much worse performance.

On my system, it takes about 30.26 seconds to complete, compared to about 10.7 seconds for the sequential case. This is about 2.8x slower.

The large increase in run time is likely the result of the overhead of inter-process communication, where the content of each file must be pickled by the child process and unpickled by the parent process in order to transmit it from one process to the other.

Next, let’s see if we can reduce the overhead of inter-process communication by batching the file load operations.

Load Files Concurrently with Processes in Batch

Any data sent to a worker process or received from a worker process must be serialized (pickled).

This adds overhead for each task executed by worker threads.

This is likely the cause of worse performance seen when using the process pool in the previous section.

We can address this by batching the load tasks into chunks to be executed by each worker process, just as we did when we batched filenames for the thread pools.

Again, this can be achieved by adapting the batched version of ThreadPoolExecutor from a previous section to use the ProcessPoolExecutor class.

We will use 4 worker processes and a chunk size of 50 files loaded per task.

And that’s it.

The complete example of batch loading files using process pools is listed below.

Running the example loads all 10,000 files into memory as we did in all previous examples.

In this case, concurrent file loading with processes does not offer any performance improvement over processes without batch mode.

On my system, the example takes about 40.16 seconds to complete, compared to about 30.26 seconds without, which is nearly 10 seconds worse or about 3.8x slower.

Again, this is likely related to the manner in which the contents of the files must be serialized in order to be shared between processes.

Next, let’s explore how we might use both thread pools and process pools.

Load Files Concurrently With Processes and Threads in Batch

The example in the previous section loaded batches of files in each worker process.

The slow parts of the example are the IO-bound loading of files within each process and the inter-process communication required to get the loaded data back to the parent process.

We may be able to speed-up the overall task by making the file loading within each process concurrent using threads.

This can be achieved by allowing each worker process to create a thread pool and to use worker threads to load files concurrently.

First, we must define a target task function for the thread tasks to execute.

The load_file() function below takes a file path as an argument, then returns the content of the file. This function can be called by worker threads concurrently.

Next, we must update the load_files() function to use a thread pool instead of loading the files sequentially.

This can be achieved by creating an instance of the ThreadPoolExecutor class.

We will create the thread pool with one worker per file to be loaded.

Next, we can issue one call to the load_file() function for each file path provided to the function.

The load_files() function must return a list of the contents of all files and a parallel list of the file paths.

One approach would be to issue the tasks using map() then convert the returned iterator into a list. For example:

Although simple, this is quite slow, discovered after some ad hoc testing.

A faster approach seems to be to issue the tasks using submit(), then constructing the list of file contents from each Future object using a list comprehension. For example:

Tying this together, the updated version of the list_files() function that uses a thread pool is listed below.

Finally, we can update the main() function slightly.

We will increase the number of processes in the process pool to match the number of logical CPU cores in my system, 8 in this case.

It may be interesting to test different values of the number of worker processes in order to discover the effect on over all task completion time.

We will then set the chunk size as an even fraction of all files to be loaded and the number of worker processes.

Given that we have 10,000 files to load and 8 worker processes, this means there are (10,000 / 8) or 1,250 files to be loaded per process.

Also, given that each process has a thread pool equal to the chunk size, it means we will have 10,000 concurrent threads in our program. Cool (if it works)!

Finally, we can update the configuration of the ProcessPoolExecutor to use the parameterized number of worker processes.

And that’s it.

The complete example of using a thread pool to load files within each worker process in the process pool is listed below.

Running the example first loads all 10,000 files from disk into main memory as before.

In this case, we see a large speed-up compared to the single-threaded version and compared to other versions tried so far.

On my system, the example takes about 4.833 seconds, compared to about 10.7 seconds for the single threaded version. That is about 2.21x faster.

It may be interesting to test different chunk sizes to see if further speed improvements are possible.

Next, let’s explore loading files concurrently using asyncio.

Load Files Concurrently with AsyncIO

We can also load files concurrently using asyncio.

Generally, Python does not support non-blocking IO operations when working with files. It is not provided in the asyncio module. This is because it is challenging to implement in a general cross-platform manner.

The third-party library aiofiles provides file operations for use in asyncio operations, but again, the operations are not true non-blocking IO and instead the concurrency is simulated using thread pools.

Nevertheless, we can use aiofiles in an asyncio program to load files concurrently.

You can install the aiofiles library using your Python package manager, such as pip:

A few minor changes are required to our program to use asyncio and aiofiles.

First, we must update the load_file() function to use the aiofiles.open() function to open files and to await on the call to read() from the file which will yield control.

The updated version of the load_file() function with these changes is listed below.

Next, we can update the main() function to first create a list of calls to the load_file() function as coroutines to execute concurrently. This can be achieved in a list comprehension.

Recall, this will create coroutines and will not actually call the load_file() function.

Next, we can execute the created coroutines and process the results (loaded data and filepath) in the order that the files are loaded.

This can be achieved using the asyncio.as_completed() function that will return coroutines in the order they finish.

We can then await the result from each coroutine and report progress.

The updated version of the main() function with these changes is listed below.

Finally, we can start the asyncio event loop and call the main() function to start the program.

Tying this together, the complete example of concurrently loading files using asyncio is listed below.

There’s just one problem, it does not work.

Running the program results in an error, as follows (or similar):

The error is caused because the program attempts to open all 10,000 files at once.

This fails because the operating system imposes a limit on the maximum number of files that can be opened at one time, such as 256 on MacOS.

Therefore, we must update the program to limit the number of files that it attempts to open at once.

This can be achieved using an asyncio.Semaphore instance.

The Semaphore can be configured with the maximum number of resources, e.g. 50, and each coroutine must acquire the semaphore before it can attempt to load a file.

First, we can create the Semaphore instance in the main() function and pass it as an argument to the load_file() coroutines.

Next, we can update the load_file() function to take the Semaphore as an argument and then acquire a resource before loading the file.

This can be achieved using the context manager on the Semaphore which will yield while waiting for a resource and will automatically release the resource once the block is exited.

The updated version of the load_file() function with these changes is listed below.

Tying this together, the complete example of loading files using asyncio and limiting the maximum number of files open concurrently is listed below.

Running the example loads all 10,000 files into memory as before.

In this case, the asyncio example is faster than the single-threaded and multi-threaded versions.

On my system, the example takes about 9.5 seconds, compared to about 10.7 seconds for the single-threaded version. This slight improvement is about 1.13x faster.

Load Files Concurrently with AsyncIO in Batch

We can update the concurrent loading of files with asyncio from the previous section to load files in batch.

That is, each coroutine can be responsbile for loading a list of files into memory to be returned to the calling coroutine.

This may or may not offer a speed benefit. Interestingly, we are able to avoid using the Semaphore that was required in the previous section.

Firstly, let’s update the load_file() function to load_files() that will load and return the contents of multiple files.

Next, we can update the main() function to split the list of files to load into chunks and the pass each chunk to a call to the load_files() function as a coroutine.

This can be achieved by iterating over the list of file paths in chunks, just as we did with the batch loading examples developed previously.

First, we must define a chunk size. In this case, we will use a size of 10, found after a little trial and error. I recommend testing different chunk sizes in order to discover what works best.

We can then use the chunksize to split up the file paths into chunks and define a coroutine for each chunk.

Then, as we did in the previous section, we can process results of each task in the order they are completed, reporting results as we go.

Tying all of this together, the complete example of loading files concurrently in batch using asyncio is listed below.

Running the example loads all 10,000 files as we did in previous examples.

In this case, we don’t see any speed benefit over the singe-threaded example and slightly worse results than the non-batch asyncio version.

On my system, the example took about 11.866 seconds, compared to about 10.7 for the sequential example, which is about 1.1x slower.

Comparison of Results

We have performed the same file load operations using a suite of different parallel methods.

We can compare the results and get some idea of those methods that may offer a performance benefit over the naive sequential method.

The table below provides a summary of the results.

The results suggest using processes with threads in batch offers the best performance.

Using threads or processes alone does not appear to offer any benefit in this case.

Extensions

This section lists ideas for extending the tutorial.

• Vary Number of Thread Workers. Update the multithreaded example to test different numbers of worker threads and evaluate the effect on overall task duration.
• Vary Number of Process Workers. Update the multiprocess example to test different numbers of worker processes and evaluate the effect on overall task duration.
• Vary Number of Process and Thread Workers: Update the combined multiprocess and multithreaded example and vary the number of worker processes and worker threads and evaluate the effect on overall task duration.
• Vary Asyncio Semaphore: Update the asyncio example and test different numbers of resources for the semaphore and discover the limit for your system and the effect on overall task duration.
• Vary Asyncio Chunk Size: Update the batch asyncio example and test different chunk sizes and their effect on the overall task duration.

Share your extensions in the comments below, it would be great to see what you come up with.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Concurrent File I/O in Python, Jason Brownlee (my book!)

Guides

• Concurrent File I/O 7-Day Course
• Faster File I/O With Concurrency
• Python File IO: A Whirlwind Tour

Python File I/O APIs

• Built-in Functions
• os - Miscellaneous operating system interfaces
• os.path - Common pathname manipulations
• shutil - High-level file operations
• zipfile — Work with ZIP archives
• Python Tutorial: Chapter 7. Input and Output

Python Concurrency APIs

• threading — Thread-based parallelism
• multiprocessing — Process-based parallelism
• concurrent.futures — Launching parallel tasks
• asyncio — Asynchronous I/O

File I/O in Asyncio

• aiofiles: File support for asyncio
• aiofiles, GitHub Project.

References

• Input/output, Wikipedia

Takeaways

In this tutorial, you discovered how to explore concurrent file loading in Python.

Do you have any questions?
Leave your question in a comment below and I will reply fast with my best advice.

Photo by Devon Janse van Rensburg on Unsplash