parallel-zip 1.4.0: 26× faster — and here's exactly what that costs

parallel-zip 1.4.0 is out. The short version: it’s a lot faster than 1.3.0, for a reason worth explaining properly, and that reason has a real cost I want to be upfront about rather than bury in a changelog.

Sampling vs. reading the label

1.3.0 already had a defense against wasting CPU on content that won’t compress: probe the first 64 KiB of a large entry, and if DEFLATE barely shrinks the sample, store the rest as-is instead of paying for a full pass that won’t pay off. That’s a statistical bet — a good one, but still a bet. It only samples the head of an entry, which can be unrepresentative (a jar’s META-INF/MANIFEST.MF sits right at the front and compresses fine, even when the class files behind it don’t), and it only kicks in for entries large enough to be worth the sample in the first place.

1.4.0 adds something more direct: check the entry’s own file signature — the same magic bytes your OS uses to guess a file type — against a list of formats that are already compressed. ZIP and everything built on it (jar/war/ear/apk/whl/docx/xlsx…), gzip, bzip2, xz, 7-zip, zstd, rar, lz4, png, jpeg, gif, webp, mp4/mov, webm/mkv, ogg, woff/woff2. A match STOREs the entry immediately, no DEFLATE attempt at all — cheaper than the statistical probe, and it doesn’t care how big the entry is or what its first sampled chunk happens to look like.

It’s the same idea as verbatim copy in shaded-jar: stop opening the sealed box to check if it’s worth repacking. This time the box doesn’t even carry a central-directory shortcut to lean on — just its own leading bytes, which turn out to be enough.

The numbers

I re-ran the full benchmark suite: the same 9 real open-source projects’ real Zip tasks from the 1.3.0 release, this time comparing the actual published 1.3.0 jar against the new 1.4.0 build, plus a fresh pass over all 11 fixed-corpus projects (full detail in the plugin’s benchmarks doc).

Geometric-mean speedup over stock Gradle Zip across those nine tasks: ~26.5×, up from ~5.4× on 1.3.0. JBake and Micronaut Starter CLI top the chart for the same reason — their distributions are dense with small, already-compressed jars, exactly the shape this feature targets. SonarQube sits at the bottom (5.89×) not because the feature doesn’t help there, but because its installer archive was already close to this floor on 1.3.0 — there’s simply less headroom left to win back.

The part I’m not going to hide

Skipping a DEFLATE pass on a “probably already compressed” entry means skipping it even when DEFLATE could still have squeezed out a percent or two more than the plugin now leaves on the table. That’s not a bug — it’s the actual trade being made — but it’s a real one, and it shows up as archive size:

6–17% larger than stock Zip, across every single project in the suite. 1.3.0, by contrast, stayed within about 0.3% of stock — essentially free. That’s a meaningfully different deal than “faster for nothing,” and I’d rather show you the honest chart than a highlight reel.

The escape hatch

Looking at that second chart, “always on by default” stopped feeling like the right call for every archive. So 1.4.0 ships a task property to turn it off:

tasks.register('dist', com.ljarocki.parallelzip.ParallelZip) {
    skipAlreadyCompressed = false   // default: true
}

false falls back to always attempting DEFLATE — closer to 1.3.0’s size, at the cost of most of the speedup above. The statistical sniff from 1.3.0 stays active either way; this flag only governs the new signature check. Default stays true, because for most real archives the speed is worth it — but “most” isn’t “all,” and now you don’t have to accept the trade you didn’t ask for.

(While wiring the flag through, I also found and fixed a real gap: the code path that compresses several small files in one native call — a batching optimization for archives with lots of tiny entries — wasn’t consulting the signature check at all, only the singleton path was. Small already-compressed files that happened to get batched together were missing the optimization entirely. Fixed now, and the new flag governs that path consistently too.)

Two smaller fixes

Not everything in this release is about the compression trade:

• A Windows-only file-lock bug, fixed. Large entries can get compressed via a memory-mapped view of their source file for speed. On Windows, an unreleased mapping keeps its backing file locked against deletion or rewrite until garbage collection gets around to it — which isn’t bounded, and was occasionally still holding the lock by the time a later task in the same Gradle daemon tried to touch that same file. The mapping is now released explicitly, right after the native call returns, instead of waiting on GC.
• A guard against OutOfMemoryError on small Gradle daemon heaps. The plugin already capped how many bytes of buffered compression work it keeps in memory at once, scaled to a fraction of the daemon’s heap — but that budget didn’t account for how a handful of large in-flight buffers can fragment a small heap before the nominal budget is even reached. The mmap-path buffer size now scales down with heap size too, and there’s a one-time warning if your daemon’s heap looks too small for the archive you’re building.

Getting it

plugins {
    id 'com.ljarocki.parallel-zip' version '1.4.0'
}

Full release notes are on GitHub; the complete benchmark breakdown — all 9 in-build projects, all 11 fixed-corpus projects, four codecs — lives in docs/BENCHMARKS.md.