How ZIP compression works: DEFLATE explained

Open a .zip in a hex editor and you will see it starts with the bytes PK\x03\x04. That is a local file header: it names one entry, states its compression method, its packed and unpacked sizes and a CRC-32 checksum, and is immediately followed by that entry's compressed bytes. Header, data, header, data — all the way to the end of the file's payload.

After the last entry comes the central directory: a second, complete list of every entry, each record starting with PK\x01\x02 and carrying the same metadata plus the offset where that entry's local header sits. The very last 22 bytes of the file are the end-of-central-directory record, PK\x05\x06, which says how many entries there are and where the directory begins.

That trailing index is why a ZIP viewer can list a huge archive instantly: it reads a few kilobytes from the end and never touches the compressed data. It is also why a ZIP survives being appended to — self-extracting archives put an executable stub in front, and signed Android packages insert a signing block, yet the directory at the end still describes everything correctly.

Almost every entry you will ever meet uses compression method 8, DEFLATE, standardised in RFC 1951. It works in two stages. First LZ77 slides a 32 KB window over the data and replaces any sequence it has seen before with a back-reference — a distance of up to 32,768 bytes and a length of 3 to 258 bytes — so the second copy of a repeated phrase costs a couple of bytes instead of its full length.

Then Huffman coding takes the resulting stream of literals and back-references and assigns short bit patterns to the symbols that occur often and long ones to the rare symbols. In English text e and space end up a few bits each; an unusual byte may cost a dozen. The two stages compose: LZ77 removes repetition, Huffman removes the waste of spending eight bits on every byte regardless of how common it is.

This is why compression level mostly changes how hard the encoder looks for matches, not what the decoder does. A file deflated at level 1 and the same file at level 9 both decompress with the same algorithm at roughly the same speed; the slow setting simply searched the window harder and found longer matches.

Method 0 is called store: the bytes go in unchanged. Archivers switch to it automatically when deflating made the entry bigger, which happens more often than you would think. JPEG, PNG, WebP, MP3, MP4, and every other media format already ran their own entropy coder over the data, so there is essentially no repetition left for LZ77 to find and no skew left for Huffman to exploit.

A ZIP of a holiday photo folder therefore lands at 99% of the original size, and that is the correct result rather than a broken one. Source code, logs, CSV exports, XML and SQL dumps go the other way: they are full of repeated identifiers, timestamps and tag names, and 70–95% savings are routine. A directory of 500 near-identical log files can compress by three orders of magnitude.

There is one structural limitation worth knowing: in ZIP each entry is compressed independently, so identical files stored twice cost twice the space. Formats that compress the whole stream at once — tar.gz, 7z with solid blocks — exploit cross-file repetition and win badly here. What ZIP buys with that inefficiency is random access: any entry can be extracted without decompressing the ones before it.

Every entry shows two sizes. The uncompressed size is what will land on your disk; the compressed size is what the archive actually spends on it. The ratio between them is the only real measure of how well compression worked, and it varies per entry — a single archive routinely contains entries at 0% and at 99%.

A ratio far above 100× deserves a second look. It is perfectly normal for a log file or a sparse disk image full of zero bytes, and it is also the signature of a zip bomb: a small archive engineered to expand into something that fills your disk. The number itself is not a verdict, but it tells you how much space extraction will need before you start.

The timestamp column deserves scepticism too. The classic ZIP date field is a packed MS-DOS value with two-second resolution, no time zone and a range starting in 1980, so a file modified at 09:14:23 is stored as 09:14:22 local time in whatever zone the packer happened to be in. Some archivers add an extra field with a proper Unix timestamp; many do not.

The original header fields are 32-bit, which caps an archive and each entry at 4 GB and the entry count at 65,535. ZIP64, added to the specification in 2001, keeps those fields at their maximum values as a marker and stores the real 64-bit numbers in an extra field, plus a second end-of-central-directory record. Everything modern reads it; a few very old tools still do not, which is why some archivers ask before using it.

ZIP encryption comes in two incompatible flavours. The original ZipCrypto has been known to be weak since 1994, when Biham and Kocher published a known-plaintext attack that recovers the internal keys from about a dozen known bytes — and in an archive of ordinary files, known plaintext is easy to come by. The modern option is the WinZip AES scheme, which wraps the real compression method inside method 99 and uses AES-128, 192 or 256 with a proper key derivation.

Neither scheme hides the metadata. File names, sizes, timestamps and CRCs live in the central directory in plain text no matter how strong the encryption is, which is why a listing of a password-protected archive still tells you a great deal — and why attackers who genuinely want to hide a file list reach for a format that can encrypt its headers instead.