Last Updated on August 21, 2023

Unzipping a large zip file is typically slow.

It can become painfully slow in situations where you may need to unzip thousands of files to disk. The hope is that multithreading will speed up the unzipping operation.

In this tutorial, you will explore how to unzip large zip files containing many data files using concurrency.

Let’s dive in.

Create a Large Zip File

Before we can explore multithreaded file unzipping, we need a large zip file that contains many files.

We can create this file ourselves.

We can create .CSV data files full of random numbers then add them to one large zip file.

First, we can define a function that creates one line of ten random numeric data values separated by commas.

The generate_line() function below implements this.

Next, we can call this function 10,000 times (e.g. 10,000 lines) to create a CSV data file that is about 2 megabytes when unzipped.

The generate_file_data() function below implements this, returning a string that represents the content of a CSV data file full of random numbers.

Finally, we can create a zip file and fill it with 1,000 different CSV data files.

First, we can open the zip file, which we will call ‘testing.zip‘ stored in our current working directory. This can be achieved using the zipfile.ZipFile class with a context manager to ensure the file is closed once we are finished with it.

We can then iterate 1,000 times and in each iteration generate the data, specify a unique filename and store the data against the filename in the zip file.

Tying these elements together, the complete example of creating the large zip file full of smaller CSV files is listed below.

Running the example will create a single zip file named ‘testing.zip‘ in your current working directory.

Progress is reported along the way and a “Done” message is reported at the end of the run.

It may take a while depending on the speed of your machine.

On my system it takes 210.9 seconds to complete and the resulting zip file is about 847 megabytes in size.

This provides an excellent case study to see if we can use multithreading to speed-up the decompression of a large zip file archive that contains thousands of smaller files.

Before we explore multithreading, let’s write a program to unzip the archive sequentially.

Unzip the Files One-by-One (slowly)

Unzipping a file with Python is straightforward,

We can open the archive using the context manager as we did when creating the zip file, then call the ZipFile.extractall() function and specify a path.

For example:

That’s all there is to it.

The complete example of unzipping the 847 megabytes zip file containing 1,000 data files is listed below.

It has been configured to unzip all files into a tmp/ sub-directory under the current working directory, to avoid clutter.

Running the example will open the ‘testing.zip‘ archive and will uncompress all data files into a tmp/ sub-directory.

It is reasonably fast, depending on the speed of your hardware (e.g. hard drive).

On my system, it takes about 8.0 seconds.

Looking in the tmp/ subdirectory shows 1,000 data files, each about 2 megabytes in size.

We can also unzip the files one by one by calling the ZipFile.extract() function for each filename in the archive.

We can get a list of filenames in the archive by calling the ZipFile.namelist() function.

For example:

Tying this together, the complete example of extracting each file from the archive one at a time by name is listed below.

Running the example does exactly the same thing as the previous example, that is extracts all files into a tmp/ sub-directory.

It also takes about the same amount of time to run, e.g. about 8.36 seconds on my machine, perhaps slightly slower because of the print messages reporting progress.

How long does the serial version take to run on your machine?
Let me know in the comments below.

Next, let’s explore how we might use multi-threading to speed-up the unzipping process.

How to Unzip Files Concurrently

We can adapt the program to be multithreaded with very few changes.

Unzipping files has three elements:

1. Load the compressed data into memory.
2. Decompress data.
3. Save the decompressed data to disk.

Decompressing data in memory is purely algorithmic and intuitively we might think it is CPU-bound. That is, it runs as fast as the CPU.

Saving the decompressed data to file is IO-bound as it is limited by the speed that we can move data from main memory onto the hard drive. Intuitively, we would expect that the IO-bound part of the task is slower than the CPU-bound part of the task. That is, we expect to spend more time waiting for the hard drive than waiting for the CPU.

One approach would be to call the ZipFile.extract() function directly to first decompress the data into memory then save the data to disk.

An alternate approach might be to first decompress the data into memory as a string using the ZipFile.read() function, then create a path and save the file to disk manually using Python disk IO functions. This might offer a benefit if we wish to explore possible speed-ups by separating the two elements of unzipping each file.

Next, let’s start to explore the use of threads to speed-up file unzipping and saving.

Unzip Files Concurrently With Threads

The ThreadPoolExecutor provides a pool of worker threads that we can use.

Each file can be submitted as a task to the thread pool and worker threads can perform the task of loading data from disk into memory.

First, we can create the thread pool with 100 worker threads.

We will use the context manager to ensure the thread pool is closed once we are finished with it.

We can then call the submit() function to issue a task to the worker threads for each file in the archive to unzip.

Each call to the ZipFile.extract() function can be a separate concurrent task, passing the name of the member to unzip and the path location on disk where to save the uncompressed data.

For example:

It appears that reading the compressed data from the archive is thread safe in the ZipFile class, although there no mention of this in the API documentation at the time of writing (Python 3.10).

Tying this together, the complete example of using multiple worker threads to unzip the 847 megabyte file full of smaller data files is listed below.

Running the example unzips the archive into a tmp/ local sub-directory, just like the serial case.

Checking the tmp/ subdirectory shows 1,000 data files, each about 2 megabytes in size.

In this case, we see a large speedup by unzipping files concurrently.

On my system, the example takes about 3.16 seconds, compared to about 8 seconds by unzipping the files sequentially. That is about 2.53x faster.

We can adapt the multithreaded unzip example to use a separate target function to unzip a single file.

This may provide more control over the unzip process which we may require later in more advanced examples.

We can define a function that takes the zip file handle, the name of the member to zip and the path to write the file, then unzip the file and report progress.

The unzip_file() function below implements this.

We can then call this function for each member in the zip file using the thread pool.

The complete example is listed below.

Running the example unzips all 1,000 files as before, although now reports progress.

The example is functionally equivalent to the previous version although makes use of a custom target task function. Therefore, it should take about the same amount of time to complete.

On my system, the example takes about 3.4 seconds, compared to about 8.36 seconds for the sequential version. That is about 2.46x faster.

Unzip Files Concurrently with Threads in Batch

Each task in the thread pool adds overhead that slows down the overall task.

This includes function calls and object creation for each task, occurring internally within the ThreadPoolExecutor class.

We might reduce this overhead by performing file operations in a batch within each worker thread.

This can be achieved by updating the unzip_file() function to receive a list of files to unzip, and splitting up the files in the main() function into chunks to be submitted to worker threads for batch processing.

The hope is that this would reduce some of the overhead required for submitting so many small tasks by replacing it with a few larger batch tasks.

First, we can change the target function to receive a list of file names to unzip, listed below.

Next, in the main() function we can select a chunksize based on the number of worker threads and the number of files to copy.

First, we must create a list of all files we wish to unzip by calling the namelist() function on the zip file.

In this case, we will use 100 worker threads with 1,000 files to unzip. Dividing the files evenly between workers (1,000 / 100) or 10 files to unzip for each worker.

Next, we can create the thread pool with the parameterized number of worker threads.

Next, we can iterate over the list of files to unzip and split them into chunks defined by a chunksize.

This can be achieved by iterating over the list of files and using the chunksize as the increment size in a for-loop. We can then split off a chunk of files to send to a worker thread as a task.

And that’s it.

The complete example of unzipping files in a batch concurrently using worker threads is listed below.

Running the example unzips all 1,000 files to the tmp/ subdirectory as before.

In this case, we don’t see any benefit in terms of reduced execution time.

On my system it takes about 3.3 seconds to complete, compared to about 8.36 seconds for the sequential version. That is about 2.53x faster.

It is slightly faster than the non-batch version that took about 3.4 seconds, although may be within the margin of measurement error.

Unzip Files Concurrently with Processes

We can also try to unzip files concurrently with processes instead of threads.

It is unclear whether processes can offer a speed benefit in this case. Given that we can get a benefit using threads, we know it is possible. Nevertheless, using processes requires data for each task to be serialized which introduces additional overhead that might negate any speed-up from performing file operations with true parallelism via processes.

We can explore using processes to unzip files concurrently using the ProcessPoolExecutor.

Unlike the ThreadPoolExecutor, we cannot simply send the handle to the open zip file to each worker process as the ZipFile class cannot be serialized.

Instead, each process must open the zip archive and extract files separately.

First, we can update the unzip_file() function to take the name of the zip file instead of the file handle. Then open the zip file before then unzipping a single file to the destination directory.

The updated version of the unzip_file() function is listed below.

Next, we must open the zip file in the main thread and get a list of all files to unzip.

We can then submit each filename as a task to the process pool.

In this case, we will use 8 workers in the pool because I have 8 logical CPU cores. You may want to adapt this for the number of logical CPU cores in your system or try different configurations in order to discover what works well.

And that’s it.

The complete example of unzipping files concurrently using processes is listed below.

Running the example unzips all files from the archive as before.

In this case, using a process pool offers a benefit over the single-threaded version, although about the same performance as the multi-threaded example.

On my system, the example takes about 3.26 seconds to complete, compared to about 8.36 seconds for the single-threaded version. That is about 2.56x faster.

Unzip Files Concurrently with Processes in Batch

Opening the zip file for each of the 1,000 files to unzip likely results in too much overhead.

Instead, it may be more efficient to open the zip file once per process, then unzip a subset of the files in batch.

Therefore, we can adapt the example from the previous section to unzip multiple files per task, similar to the multithreaded batch example we developed previously.

First, we can update the unzip_files() function to take a list of files to unzip.

We can then split the list of files into chunks.

In this case, we will use 8 processes and split the 1,000 files to unzip evenly giving 125 files to unzip per process. It may be interesting to explore different divisions of work among the processes.

Finally, we can iterate the list of files to unzip in chunks and send each into the process pool as a task, much like we did in the multithreaded batch unzipping example.

Tying this together, the complete example of unzipping files concurrently using the process pool in batch is listed below.

Running the example unzips all files from the archive as before.

In this case, we see a benefit over all previous examples tried so far.

On my system, the example takes about 2.03 seconds, compared to about 8.36 seconds for the single-threaded version. That is about 4.12x faster.

Interestingly, the ZipFile.extractall() allows a subset of files to be extracted in batch.

We could call this function from each process for a chunk of files. This could be slightly more efficient given the fewer lines of code although it means we cannot report progress along the way.

The unzip_files() function can be updated to call the ZipFile.extractall() function directly, specifying the directory in which the files are to be extracted and a list of names of files to extract from the archive.

The unzip_files() function with these changes is listed below.

Tying this together, the complete example is listed below.

Running the example extracts all 1,000 files from the archive as before.

In this case, we don’t see any significant speed improvement.

On my system, it takes about 2.0 seconds, compared to about 2.03 seconds when calling the ZipFile.extract() function manually for each file to decompress which may be within the margin of measurement error.

Nevertheless, compared to the sequential version that took 8.36 seconds, this version is about 4.18x faster.

Unzip Files Concurrently with Processes and Threads in Batch

It is possible to separate the CPU-bound and IO-bound effort of each file unzip task.

We can decompress a file from the archive to main memory by calling the ZipFile.read() function, then save it to disk manually by creating a path and writing the bytes.

The decompression of each file in memory could be performed by processes and the writing of each file to disk could be performed by threads.

This could be achieved by first using a process pool to decompress batches of files from the archive then using a thread pool within each process to save the decompressed file data to disk.

There are a lot of moving parts and it is unclear whether this would offer a speed improvement. Nevertheless, it is an interesting exercise.

First, we can define a task to save data to disk to be executed in a thread pool.

The function takes the decompressed data, the filename and the directory in which to save the file. A relative path is constructed using the os.path.join() function, then the file is opened in write mode and the decompressed bytes are written to disk and progress is reported.

The save_file() function below implements this.

Next, the unzip_files() function needs to be updated to first decompress each file, then issue a save task to a thread pool.

First, we must create a thread pool. We will use 10 workers per pool in this case, chosen arbitrarily. Recall, each of the eight processes will have its own thread pool.

We can then iterate over the list of files that the process is responsible for unzipping and decompressing each and submit the data to the thread pool for saving.

Because the thread pool is within the process, no inter-process communication is required, avoiding additional computational overhead, such as if the main thread/process were to save the decompressed files.

The updated version of the unzip_files() function with these changes is listed below.

And that is it.

No other changes from the previous example for batch unzipping with processes are required.

The complete example is listed below.

It is more complicated than the previous example, so let’s review what is happening. First, we split the list of files into chunks and send each chunk to a separate process for extraction. Each process then starts a thread pool, extracts the data for each file and throws tasks into the thread pool to save the decompressed data for each file.

The hope is that decompressing data is faster than saving data to disk so that the CPU can get through the decompression steps fast and then wait on the threads to save all the files.

Running the example decompresses all 1,000 data files to the tmp/ sub-directory as with all previous examples and reports progress.

In this case, we may see a small speed increase

On my system, it takes about 1.83 seconds to complete, compared to about 8.36 seconds for the sequential version. That is about 4.57x faster.

Unzip Files Concurrently with AsyncIO

We can also unzip files concurrently using asyncio.

Generally, Python does not support non-blocking IO operations when working with files. It is not provided in the asyncio module. This is because it is challenging to implement in a general cross-platform manner.

The third-party library aiofiles provides file operations for use in asyncio operations, but again, the operations are not true non-blocking IO and instead the concurrency is simulated using thread pools.

Nevertheless, we can use aiofiles in an asyncio program to load files concurrently. It does not support an asynchronous version of the ZipFile class.

You can install the aiofiles library using your Python package manager, such as pip:

Given that the aiofiles library does not support asyncio support for ZipFile, we can develop an example that first decompresses each file in a blocking manner, then save the decompressed data to disk using (simulated) non-blocking IO.

First, we can develop a function to extract data from the zip file and save data to disk using aiofiles.

The data can be extracted from the zip file, which requires a blocking read operation.

Next, we can construct the output path for saving the file using os.path.join() function.

We can then open the file using aiofiles.open() and write the data to disk, yielding control until the task is completed.

The extract_and_save() function below implements this.

Finally, we can update the main() function.

We can create the directory in which all files will be saved using the os.makedirs() function.

After opening the ZipFile for unzipping, we can create a list of coroutines, each with a call to our extract_and_save() function for each filename in the zip archive.

This can be achieved using a list comprehension.

Recall that this will create a list of coroutines and will not call the extract_and_save() function.

We can execute all of the coroutines and await their completion using the asyncio.gather() function. We can unpack the coroutines for the gather function using the * operator, for example:

Finally, we can start the asyncio runtime and call the main() to begin the event loop and start processing coroutines.

Tying this together, the complete example of concurrently unzipping files using asyncio is listed below.

Running the example unzips all files to the tmp/ subdirectory as before.

In this case, we don’t see any speed benefit, in fact, the asyncio version has much worse performance than other multithreaded versions.

On my system, the example takes about 7.6 seconds, compared to about 8.36 seconds with the sequential version. This is about 1.10x faster.

Unzip Files Concurrently with AsyncIO in Batch

Perhaps we can reduce some of the overhead in unzipping files concurrently with asyncio by performing tasks in batch.

In the previous example, each coroutine performed a single IO-bound operation. We can update the extract_and_save() function to process multiple files in a batch as we did in previous batch processing examples.

This will reduce the number of coroutines in the asyncio runtime and may reduce the amount of data moving around.

The updated version of the task function that will process a list of files is listed below. The task will yield control for each file write operation, but only to other coroutines. The for-loop within a task will not progress while it is waiting.

Next, we can prepare a list of all files we expect to unzip from the archive.

We can then choose the number of coroutines we wish to use and then divide the total number of files up evenly into chunks.

We will use 100 in this case, chosen arbitrarily.

We can then iterate over the chunks and add each coroutine to a list for execution at the end.

Finally, we can execute all 100 coroutines concurrently.

Tying this together, a complete example of batch file unzipping using asyncio is listed below.

Running the example unzips all files to the tmp/ subdirectory as before.

In this case we see a modest speed improvement over the non-batch asyncio version of unzipping files.

On my system, the example takes about 6.4 seconds to complete, compared to 8.36 seconds for the sequential version. That is about 1.31x faster.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Concurrent File I/O in Python, Jason Brownlee (my book!)

Guides

• Concurrent File I/O 7-Day Course
• Faster File I/O With Concurrency
• Python File IO: A Whirlwind Tour

Python File I/O APIs

• Built-in Functions
• os - Miscellaneous operating system interfaces
• os.path - Common pathname manipulations
• shutil - High-level file operations
• zipfile — Work with ZIP archives
• Python Tutorial: Chapter 7. Input and Output

Python Concurrency APIs

• threading — Thread-based parallelism
• multiprocessing — Process-based parallelism
• concurrent.futures — Launching parallel tasks
• asyncio — Asynchronous I/O

File I/O in Asyncio

• aiofiles: File support for asyncio
• aiofiles, GitHub Project.

References

• Input/output, Wikipedia

Takeaways

In this tutorial, you discovered how to explore concurrent file unzipping in Python.

Do you have any questions?
Leave your question in a comment below and I will reply fast with my best advice.

Photo by Appic on Unsplash