Last Updated on August 21, 2023

Renaming a single file is fast, but renaming many files can be slow.

It can become painfully slow in situations where you may need to rename thousands or millions of files. The hope is that multithreading can be used to speed up the file renaming operation.

In this tutorial, you will explore how to rename thousands of files using multithreading.

Let’s dive in.

Create 10,000 Files To Rename

Before we can rename thousands of files, we must create them.

We can write a script that will create 10,000 CSV files, each file with 10,000 lines and each line containing 10 random numeric values.

Each file will be about 1.9 megabytes, and all files together will be about 19 gigabytes in size.

First, we can create a function that will take a file path and string data and save it to file.

The save_file() function below implements this, taking the full path and data, opens the file in ASCII format, then saves the data to the file. The context manager is used so that the file is closed automatically.

Next, we can generate the data to be saved.

First, we can write a function to generate a single line of data. In this case, a line is composed of 10 random numeric values in the range between 0 and 1, stored in CSV format (e.g. separated by a comma per value).

The generate_line() function below implements this, returning a string of random data.

Next, we can write a function to generate the contents of one file.

This will be 10,000 lines of random values, where each line is separated by a new line.

The generate_file_data() function below implements this, returning a string representing the data for a single data file.

Finally, we can generate data for all of the files and save each with a separate file name.

First, we can create a directory to store all of the created files (e.g. ‘tmp‘) under the current working directory.

This can be achieved using the os.makedirs() function, for example:

We can then loop 10,000 times and create the data and a unique filename for each iteration, then save the generated contents of the file to disk.

The generate_all_files() function listed below implements this, creating the 10,000 files of 10,000 lines of data.

Tying this together, the complete example of creating a large number of CSV files is listed below.

Running the example will create 10,000 CSV files of random data into a tmp/ directory.

It takes a long time to run, depending on the speed of your hard drive, e.g. SSD used in modern computer systems are quite fast.

On my system it takes about 11 minutes to complete.

Next, we can explore ways of renaming a large number of files.

Rename Files One-By-One (slowly)

We are interested in renaming a list of files that could be in the same or different directories.

We will simulate the general file renaming task by renaming a list of files in one source directory.

Renaming all files in one directory in Python is relatively straightforward.

Python provides a number of ways for renaming files, although the os.rename() function is perhaps the standard approach.

In this tutorial, we will use the os.rename() function to rename files and provide it with the relative path to the source file and the relative path to the destination (renamed) file in the same directory.

For example:

First, we can iterate all files in the source directory using the os.listdir() function that will return the name of each file.

For each file to be renamed we will first split the filename into a name and extension part, so that we can devise a new filename. This can be achieved using the os.path.splitext() function, for example:

We can then devise a new filename. In this case we will append the string “-new” to the filename.

Next, we can define the relative path to the existing file and the new file that will be used as arguments to the rename function. This can be achieved using the os.path.join() function that takes the directory as the first argument and the filename as the second argument.

Finally, we can rename the file and report progress.

Tying this together, the complete example of renaming all files in a directory sequentially (one-by-one) is listed below.

Running the program will rename all files in the tmp/ subdirectory to have a “-new” string in the filename.

Renaming files is faster than copying files, although it will still take a moment as there are 10,000 files, each about 2 megabytes in size.

The overall execution time will depend on the speed of your hardware, specifically the speed of your hard drive.

On my system with an SSD hard drive, it takes about 1.36 seconds to complete (resetting the tmp/ directory between runs).

How long does it take to run on your system?
Let me know in the comments below.

Now that we know how to rename thousands of files, let’s consider how we might speed it up using multiple threads.

How to Rename Files Concurrently

File renaming is performed by the operating system which then calls down to the hardware, probably to do something with the table of contents for files on the disk (e.g. a file allocation table or the modern equivalent).

As such, we might expect that file renaming is an IO-bound task, limited by the speed that files can be “renamed” by the disk, e.g. perhaps just updates to the hard drives file allocation table.

This suggests that threads may be appropriate to perform concurrent file renaming operations. This is because the CPython interpreter will release the global interpreter lock when performing IO operations, perhaps such as renaming a file. This is opposed to processes that are better suited to CPU-bound tasks, that execute as fast as the CPU can run them.

Given that renaming files does not involve creating new data or moving data around on the disk, file renaming is a generally fast process.

Ultimately we might expect that file renaming on the disk is performed sequentially. That is the disk is only able to rename one file at a time.

As such, our expectation may be that file renaming cannot be performed in parallel on a single disk and using threads will not offer any speed-up.

Let’s explore this in the next section.

Rename Files Concurrently with Threads

The ThreadPoolExecutor provides a pool of worker threads that we can use to multithread the renaming of thousands of files.

Each file in the tmp/ directory will represent a task that will require the file to be renamed from the source name to the destination name using the os.rename() function.

First, we can define a function that represents the task. The function will take the name of the file and the source directory and will perform the rename operation and report progress to stdout.

The rename_file() function below implements this.

Next, we can call this function for each file in the tmp/ directory.

First, we can create the thread pool with ten worker threads, chosen arbitrarily.

We will use the context manager to ensure the thread pool is closed automatically once all pending and running tasks have been completed.

We can then call the submit() function for each file to be renamed, calling the rename_file() function with the filename and directory arguments.

This can be achieved in a list comprehension, and the resulting list of Future objects can be ignored as we do not require any result from the task.

Tying this together, the complete example of multithreaded renaming of files is listed below.

Running the example renames all files in the tmp/ sub-directory , as before.

We may see a small increase in speed in performing the rename operations concurrently with threads.

On my system, the task completed in about 1.06 seconds, compared to about 1.36 seconds with the sequential case. This is about 1.28x faster.

How long does it take to run on your system?
Let me know in the comments below.

We can try increasing the number of worker threads to see if it offers any further benefit.

For example, we could try ten times more threads, e.g. 100 workers.

This requires simply changing the argument to the ThreadPoolExecutor class from 10 to 100.

Running the example with the increase in worker threads renames all files as before.

In this case, we may see a small further speed increase, or it may be within the margin of error.

On my system it took about 967.6 milliseconds (recall there are 1,000 milliseconds in a second), compared to about 1.36 seconds for the single-threaded version. That is about 1.41x faster.

You might like to test additional configurations such as 500 or even 1,000 threads to see if it makes a reasonable difference in speed.

Rename Files Concurrently with Threads in Batch

Each task in the thread pool adds overhead that slows down the overall task.

This includes function calls and object creation for each task, occurring internally within the ThreadPoolExecutor class.

We might reduce this overhead by performing file operations in a batch within each worker thread.

This can be achieved by updating the rename_file() function to receive a list of files to rename, and splitting up the files in the main() function into chunks to be submitted to worker threads for batch processing.

The hope is that this would reduce some of the overhead required for submitting so many small tasks by replacing it with a few larger batch tasks.

First, we can change the target function to receive a list of file names to rename, listed below.

Next, in the main() function we can select a chunksize based on the number of worker threads and the number of files to rename.

First, we must create a list of all files to rename that we can split up later.

In this case, we will use 100 worker threads with 10,000 files to rename. Dividing the files evenly between workers (10,000 / 100) or 100 files to rename for each worker.

Next, we can create the thread pool with the parameterized number of worker threads.

We can then iterate over the list of files to rename and split them into chunks defined by a chunksize.

This can be achieved by iterating over the list of files and using the chunksize as the increment size in a for-loop. We can then split off a chunk of files to send to a worker thread as a task.

And that’s it.

The complete example of renaming files in a batch concurrently using worker threads is listed below.

Running the example renames all 10,000 files in the tmp/ subdirectory as before.

In this case, see a further minor decrease in overall execution time.

On my system, renaming all files takes about 823 milliseconds compared to about 967.6 milliseconds with 100 workers in non-batch mode and 1.36 seconds in the sequential case. This is about 1.65x faster compared to the sequential example.

Rename Files Concurrently with Processes

We can also rename files concurrently with processes instead of threads.

It is unclear whether processes can offer a speed benefit in this case. Given that we can get a benefit using threads, we know it is possible. Nevertheless, using processes requires data for each task to be serialised which introduces additional overhead that might negate any speed-up from performing file operations concurrently.

We can explore using processes to rename files concurrently using the ProcessPoolExecutor.

This can be achieved by switching out the ThreadPoolExecutor directly and specifying the number of worker processes.

We will use 8 processes in this case as I have 8 logical CPU cores, although you may want to try different configurations based on the number of logical CPU cores you have available.

Tying this together, the complete example of renaming files concurrently using the process pool is listed below.

Running the example renames all 10,000 files in the tmp/ subdirectory as before.

In this case, we do not see a speed-up. Instead, the task is slower than even the sequential case.

On my system it takes about 2.7 seconds, compared to the 1.36 seconds for the sequential case. This is about 2.0x slower.

Rename Files Concurrently with Processes in Batch

Any data sent to a worker process or received from a worker process must be serialised (pickled).

This adds overhead for each task executed by worker threads.

This is likely the cause of worse performance seen when using the process pool in the previous section.

We can address this by batching the rename tasks into chunks to be executed by each worker process, just as we did when we batched filenames for the thread pools in a previous section.

Again, this can be achieved by adapting the batched version of ThreadPoolExecutor from a previous section to use the ProcessPoolExecutor.

We will use 8 worker processes. With 10,000 files to be renamed, this means that each worker will receive a batch of 1,250 files to rename.

We can then create the process pool using the parameterized number of workers.

And that’s it.

The complete example of batch renaming files using process pools is listed below.

Running the example renames all of the files as before.

We do see a small improvement in performance as compared to the sequential case, although the results are not as good as the batch multithreaded case.

On my system it takes about 976 milliseconds, compared to about 823 milliseconds in the multithreaded case, and compared to 1.36 seconds for the sequential case. That is about 1.39x faster.

It is possible that further improvements can be achieved by tuning the division of rename tasks to the worker processes, e.g. optimising chunksize.

Rename Files Concurrently with AsyncIO

We can also rename files concurrently using asyncio.

Generally, Python does not support non-blocking IO operations when working with files. It is not provided in the asyncio module. This is because it is challenging to implement in a general cross-platform manner.

The third-party library aiofiles provides file operations for use in asyncio operations, but again, the operations are not true non-blocking IO and instead the concurrency is simulated using threads.

Nevertheless, we can use aiofiles in an asyncio program to rename files concurrently with coroutines.

First, we can update the rename_file() function to be an async function that can be waited upon and change the rename() function to use the aiofiles.os.rename() that is awaited upon.

The updated version of rename_file() is listed below.

Next, we can update the main() function.

First, we can create a list of coroutines (asynchronous functions) that will execute concurrently. Each coroutine will be a call to the rename_file() function with a separate file to rename.

This can be achieved in a list comprehension, for example:

Recall, this defines a list of coroutines, and does not call the asynchronous function while constructing the list.

Next, we can execute the list of coroutines and wait for them to complete. This will attempt to perform all 10,000 file rename operations concurrently, allowing each task to yield control while waiting for the rename operation to complete via the await within the rename_file() function.

This can be achieved using the asyncio.gather() function that takes a collection of async function calls. Given that we have a list of these calls, we can unpack the function calls with the star (*) operator.

Finally, we can start the asyncio event loop by calling the asyncio.run() function.

The complete example of renaming files concurrently with asyncio is listed below.

Running the example renames all 10,000 files as before.

The asyncio version is slightly slower than the sequential version.

On my system, it takes about 1.63 seconds, compared to about 1.36 for the sequential version which is about 1.2x slower.

It is possible that the task is slower because of the overhead of the asyncio event loop and any simulated concurrent achieved using threads provided by the aiofiles library.

This is not a surprising result, although it was an interesting exercise.

Rename Files Concurrently with AsyncIO in Batch

Perhaps we can reduce some of the overhead in renaming files concurrently with asyncio by performing tasks in batch.

In the previous example, each coroutine performed a single IO-bound operation. We can update the rename_file() function to rename_files() to process multiple files in a batch as we did in previous batch processing examples.

This will reduce the number of coroutines in the asyncio runtime and may reduce the amount of data moving around. At the very least, it may be an interesting exercise.

The adaptation of the asyncio example to rename files in batch is very similar to the previous examples that we adapted to operate in batch.

The updated version of the task function that will process a list of files is listed below. The task will yield control for each rename operation, but only to other coroutines. The for-loop within a task will not progress while it is waiting.

Next, we can choose the number of coroutines we wish to use and then divide the total number of files up evenly into chunks.

We will use 100 in this case, chosen arbitrarily.

We can then iterate over the chunks and add each coroutine to a list for execution at the end.

Finally, we can execute all 100 coroutines concurrently.

Tying this together, a complete example of batch file renaming using asyncio is listed below.

Again, we don’t expect a speed benefit over the single-threaded example, but perhaps a small lift in performance over the previous example where we had 10,000 coroutines operating concurrently.

Running the example renames all 10,000 files in the tmp/ subdirectory as before.

In this case, we may see a small improvement in performance over the non-batch asyncio example, although perhaps still slower than the single-threaded example we developed first.

On my system, the example takes about 1.5 seconds, which may be better than the non-batch version that took 1.63 seconds, or may be within the margin of error. That is about 1.1x slower.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Concurrent File I/O in Python, Jason Brownlee (my book!)

Guides

• Concurrent File I/O 7-Day Course
• Faster File I/O With Concurrency
• Python File IO: A Whirlwind Tour

Python File I/O APIs

• Built-in Functions
• os - Miscellaneous operating system interfaces
• os.path - Common pathname manipulations
• shutil - High-level file operations
• zipfile — Work with ZIP archives
• Python Tutorial: Chapter 7. Input and Output

Python Concurrency APIs

• threading — Thread-based parallelism
• multiprocessing — Process-based parallelism
• concurrent.futures — Launching parallel tasks
• asyncio — Asynchronous I/O

File I/O in Asyncio

• aiofiles: File support for asyncio
• aiofiles, GitHub Project.

References

• Input/output, Wikipedia

Takeaways

In this tutorial you discovered how to explore using multithreading for renaming thousands of files.

Do you have any questions?
Leave your question in a comment below and I will reply fast with my best advice.

Photo by Grant Durr on Unsplash