Last Updated on August 21, 2023

Deleting a single file is fast, but deleting many files can be slow.

It can become painfully slow in situations where you may need to delete thousands of files. The hope is that multithreading can be used to speed up the file deletion operation.

In this tutorial, you will explore how to delete thousands of files using multithreading.

Let’s dive in.

Create 10,000 Files To Delete

Before we can delete thousands of files, we must create them.

We can write a script that will create 10,000 CSV files, each file with 10,000 lines and each line containing 10 random numeric values.

Each file will be about 1.9 megabytes in size, and all files together will be about 19 gigabytes in size.

First, we can create a function that will take a file path and string data and save it to file.

The save_file() function below implements this, taking the full path and data, opens the file in ASCII format, then saves the data to the file. The context manager is used so that the file is closed automatically.

Next, we can generate the data to be saved.

First, we can write a function to generate a single line of data. In this case, a line is composed of 10 random numeric values in the range between 0 and 1, stored in CSV format (e.g. separated by a comma per value).

The generate_line() function below implements this, returning a string of random data.

Next, we can write a function to generate the contents of one file.

This will be 10,000 lines of random values, where each line is separated by a new line.

The generate_file_data() function below implements this, returning a string representing the data for a single data file.

Finally, we can generate data for all of the files and save each with a separate file name.

First, we can create a directory to store all of the created files (e.g. ‘tmp‘) under the current working directory.

This can be achieved using the os.makedirs() function, for example:

We can then loop 10,000 times and create the data and a unique filename for each iteration, then save the generated contents of the file to disk.

The generate_all_files() function listed below implements this, creating the 10,000 files of 10,000 lines of data.

Tying this together, the complete example of creating a large number of CSV files is listed below.

Running the example will create 10,000 CSV files of random data into a tmp/ directory.

It takes a long time to run, depending on the speed of your hard drive, e.g. SSD used in modern computer systems are quite fast.

On my system it takes about 11 minutes to complete.

Next, we can explore ways of deleting all of the files from the directory.

Delete Files One-By-One (slowly)

Deleting all files in one directory to a second directory in Python is relatively straightforward.

A file can be deleted in Python by calling the os.remove() function and specifying the path to the file.

For example:

First, we can create a list of all file paths to delete.

This can be achieved by calling the os.listdir() function that will list the contents of a directory and then calling the os.path.join() function for each file name to join it with the directory name to create a path.

This can be achieved in a list comprehension, for example:

We can then iterate over all of the prepared file paths and call the os.remove() function on each.

Tying this together, the complete example of deleting all files is listed below.

Running the example will iterate over all files in the tmp/ sub-directory and delete each file in turn, one-by-one.

Even though there are 10,000 files and about 19 gigabytes of data total, deleting the files is relatively fast.

On my system it finishes in under a second, specifically 671.6 milliseconds (recall there are 1,000 milliseconds in a second).

How long did the example take to run on your system?
Let me know in the comments below.

Even though the deletion process is fast, we can use this as a baseline to see if the file deletion process can be sped up using multiple threads.

How to Delete Files Concurrently

File deletion is performed by the operating system which then calls down to the hardware, probably to do something with the table of contents for files on the disk (file allocation table or some-such).

We might expect that file deletion is an IO-bound task, limited by the speed that files can be “deleted” (perhaps just dropped from the allocation table) by the hard disk drive.

This suggests that threads may be appropriate to perform concurrent deletion operations. This is because the CPython interpreter will release the global interpreter lock when performing IO operations, perhaps such as deleting a file. This is opposed to processes that are better suited to CPU-bound tasks, that execute as fast as the CPU can run them.

Given that deletion does not involve moving data around on the disk, deletion is a fast process generally.

Ultimately we might expect that file deletion on the disk is performed sequentially. That is the disk is only able to delete one file at a time.

As such, our expectation may be that file deletion cannot be performed in parallel on a single disk and using threads will not offer any speed-up.

Let’s explore this in the next section.

Delete Files Concurrently With Threads

The ThreadPoolExecutor provides a pool of worker threads that we can use to multithread the deleting of thousands of data files.

Each file in the tmp/ directory will represent a task that will require the file to be deleted from disk by a call to the os.remove() function.

First, we can define a function that takes the path of a file to delete, removes the file, then reports progress.

The delete_file() function below implements this.

Next, we can call this function for each file in the tmp/ directory.

We can then create the thread pool with ten worker threads. We will use the context manager to ensure the thread pool is closed automatically once all pending and running tasks have been completed.

We can then call the submit() function for each file to be deleted, calling the delete_file() function with the file path as an argument.

This can be achieved in a list comprehension, and the resulting list of Future objects can be ignored as we do not require any result from the task.

Tying this together, the complete example of multithreaded deletion of files from one directory is listed below.

Running the example deletes all files from the directory, as before.

We do not see any speed benefit to multithreading the file deletion process, or any penalty slowing down the process.

On my system it takes about 629.6 milliseconds, which is roughly the same, probably within the margin of measurement error, as the sequential case that took 671.6 milliseconds.

How long did it take to execute on your system?
Let me know in the comments below.

We can try increasing the number of worker threads to see if it offers any further benefit.

For example, we could try ten times more threads, e.g. 100 workers.

This requires simply changing the argument to the ThreadPoolExecutor class from 10 to 100.

Running the example with the increase in worker threads deletes all files as before.

In this case, we do not see an improvement in performance. In fact we see a small increase in the overall task duration.

On my system it took about 723.3 milliseconds with 100, compared to about 629.6 milliseconds with 10 threads. This may be within the margin of measurement error, or perhaps caused by the overhead of having to manage additional threads.

You might like to test additional configurations such as 500 or even 1,000 threads to see if it makes a reasonable difference in speed.

Delete Files Concurrently with Threads in Batch

Each task in the thread pool adds overhead that slows down the overall task.

This includes function calls and object creation for each task, occurring internally within the ThreadPoolExecutor class.

We might reduce this overhead by performing file operations in a batch within each worker thread.

This can be achieved by updating the delete_file() function to receive a list of files to delete, and splitting up the files in the main() function into chunks to be submitted to worker threads for batch processing.

The hope is that this would reduce some of the overhead required for submitting so many small tasks by replacing it with a few larger batch tasks.

First, we can change the target function to receive a list of file names to delete, listed below.

Next, in the main() function we can select a chunksize based on the number of worker threads and the number of files to delete.

First, we must create a list of all files to delete that we can split up later.

In this case, we will use 10 worker threads with 10,000 files to delete. Dividing the files evenly between workers (10,000 / 10) or 1,000 files to delete for each worker.

Next, we can create the thread pool with the parameterized number of worker threads.

We can then iterate over the list of files to delete and split them into chunks defined by a chunksize.

This can be achieved by iterating over the list of files and using the chunksize as the increment size in a for-loop. We can then split off a chunk of files to send to a worker thread as a task.

And that’s it.

The complete example of deleting files in a batch concurrently using worker threads is listed below.

Running the example deletes all 10,000 files from the tmp/ sub-directory.

In this case, we do see a lift in performance.

On my system it takes about 394.6 milliseconds, compared to about 671.6 milliseconds in the case where files were deleted one-by-one. This is about a 1.7x speed improvement.

Not bad.

Delete Files Concurrently with Processes

We can also try to delete files concurrently with processes instead of threads.

It is unclear whether processes can offer a speed benefit in this case. Given that we can get a benefit using threads with batch deletion, we know it is possible. Nevertheless, using processes requires data for each task to be serialised which introduces additional overhead that might negate any speed-up from performing file operations with true parallelism via processes.

We can explore using processes to delete files concurrently using the ProcessPoolExecutor.

This can be achieved by switching out the ThreadPoolExecutor directly and specifying the number of worker processes.

We will use 8 processes in this case as I have 8 logical CPU cores, although you may want to try different configurations based on the number of logical CPU cores you have available.

Tying this together, the complete example of deleting files concurrently using the process pool is listed below.

Running the example deletes all files as before.

We do not see any lift in performance. In fact, we see a dramatic drop in performance compared to both the sequential and multithreaded examples.

On my system it takes about 2.23 seconds, compared to about 629.6 milliseconds for the single-process example above. This is about 3.3x slower.

The dramatic decrease in performance is very likely due to the overhead in serialising the arguments for the 10,000 tasks.

Delete Files Concurrently with Processes in Batch

Any data sent to a worker process or received from a worker process must be serialised (pickled).

This adds overhead for each task executed by worker threads.

This is likely the cause of worse performance seen when using the process pool in the previous section.

We can address this by batching the delete tasks into chunks to be executed by each worker process, just as we did when we batched filenames for the thread pools.

Again, this can be achieved by adapting the batched version of ThreadPoolExecutor from a previous section to use the ProcessPoolExecutor class.

We will use 8 worker processes. With 10,000 files to be deleted, this means that each worker will receive a batch of 1,250 files to delete.

We can then create the process pool using the parameterized number of workers.

And that’s it.

The complete example of batch deleting files using process pools is listed below.

Running the example deletes all 10,000 files as we did previously.

On my system, the program took about 497.3 milliseconds, which may be slightly slower than the batch multithreaded example that took about 394.6 milliseconds although slightly faster than the sequential example that took about 629.6 milliseconds seconds (about 1.35x faster).

Delete Files Concurrently with AsyncIO

We can also delete files concurrently using asyncio.

Generally, Python does not support non-blocking operations when working with files.

It is not provided in the asyncio module. This is because it is challenging to implement in a general cross-platform manner.

The third-party library aiofiles provides file operations for use in asyncio operations, but again, the operations are not true non-blocking IO and instead the concurrency is simulated using threads.

Nevertheless, we can use aiofiles in an asyncio program to delete files concurrently.

First, we can update the delete_file() function to be an async function that can be awaited and change the call to the os.remove() function to use the aiofiles.os.remove() that can be awaited.

The updated version of delete_file() is listed below.

Next, we can update the main() function.

First, we can create a list of coroutines (asynchronous functions) that will execute concurrently. Each coroutine will be a call to the delete_file() function with a separate file to delete.

This can be achieved in a list comprehension, for example:

You might recall that this defines a list of coroutines, and does not call the asynchronous function while constructing the list.

Next, we can execute the list of coroutines and wait for them to complete. This will attempt to perform all 10,000 file delete operations concurrently, allowing each task to yield control while waiting for the delete operation to complete via the await within the delete_file() function.

This can be achieved using the asyncio.gather() function that takes a collection of async function calls. Given that we have a list of these calls, we can unpack the function calls with the star (*) operator.

Finally, we can start the asyncio event loop by calling the asyncio.run() function and pass it a call to our main() function.

The complete example of deleting files concurrently with asyncio is listed below.

Running the example deletes all 10,000 files as before.

We do not see any benefit in using asyncio to delete files concurrently. In fact, the results are worse than the sequential example we developed first.

On my system, the program completes in about 1.33 seconds which is about 2x worse than the sequential example that took about 629.6 milliseconds.

This is not surprising given that asyncio is operating in a single thread, that aiofiles is simulating non-blocking IO using threads internally and that we are attempting to run 10,000 coroutines simultaneously.

Nevertheless, it was an interesting exercise in case Python does support non-blocking file operations in the future.

Delete Files Concurrently with AsyncIO in Batch

Perhaps we can reduce some of the overhead in deleting files concurrently with asyncio by performing tasks in batch.

In the previous example, each coroutine performed a single IO-bound operation. We can update the delete_file() function to process multiple files in a batch as we did in previous batch processing examples.

This will reduce the number of coroutines in the asyncio runtime and may reduce the amount of data moving around. At the very least, it may be an interesting exercise.

The adaptation of the asyncio example to delete files in batch is very similar to the previous examples that we adapted to operate in batch.

The updated version of the task function that will process a batch of files to delete is listed below. The task will yield control for each delete operation, but only to other coroutines. The for-loop within a task will not progress while it is waiting.

Next, we can choose the number of coroutines we wish to use and then divide the total number of files up evenly into chunks.

We will use 100 in this case, chosen arbitrarily.

We can then iterate over the chunks and add each coroutine to a list for execution at the end.

Finally, we can execute all 100 coroutines concurrently.

Tying this together, a complete example of batch file deleting using asyncio is listed below.

Again, we don’t expect a speed benefit over the single-threaded example, but perhaps a small lift in performance over the previous example where we had 10,000 coroutines operating concurrently.

Running the example deletes approximately 20 gigabytes of files as we did in all previous examples.

In this case, we don’t see any performance benefit. The example appears to take about as long as the non-batch asyncio example above.

On my system it takes about 1.23 seconds compared to about 1.33 seconds for the non-batched example. A small difference that might be within the margin of measurement error.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Concurrent File I/O in Python, Jason Brownlee (my book!)

Guides

• Concurrent File I/O 7-Day Course
• Faster File I/O With Concurrency
• Python File IO: A Whirlwind Tour

Python File I/O APIs

• Built-in Functions
• os - Miscellaneous operating system interfaces
• os.path - Common pathname manipulations
• shutil - High-level file operations
• zipfile — Work with ZIP archives
• Python Tutorial: Chapter 7. Input and Output

Python Concurrency APIs

• threading — Thread-based parallelism
• multiprocessing — Process-based parallelism
• concurrent.futures — Launching parallel tasks
• asyncio — Asynchronous I/O

File I/O in Asyncio

• aiofiles: File support for asyncio
• aiofiles, GitHub Project.

References

• Input/output, Wikipedia

Takeaways

In this tutorial you discovered how to explore using multithreading for batch deleting thousands of files.

Do you have any questions?
Leave your question in a comment below and I will reply fast with my best advice.

Photo by Roberto Nickson on Unsplash