Did you know that you can navigate the posts by swiping left and right?

xz - Your next default compression tool.

The xz compression program, run with defaults, can be so slow that you might wonder why anybody uses it. But on a modern machine, xz -0 --threads=0 will run with less aggressive compression goals and use all available threads on a multi-core CPU, exceeding gzip performance by a good measure. One nice bonus with the xz format is that individual archives can be concatenated. cat file*.xz > all-files.xz without decompressing first, and then unarchive in one step - nifty! You can even null-pad the space between concatenated files if you need to block-align the archive for random access on slow media. You might consider setting your XZ_DEFAULTS environment variable to use these higher performance options.

Let’s look at compressing this Postscript file:

$ ls -lh certificate5896072-3.ps 
-rw-r--r-- 1 user user 30M Nov 20  2012 certificate5896072-3.ps

$ time gzip -9 -k certificate5896072-3.ps

real    0m1.828s
user    0m1.697s
sys     0m0.029s

$ ls -lh certificate5896072-3.ps.gz
-rw-r--r-- 1 user user 28M Nov 20  2012 certificate5896072-3.ps.gz

$ time xz -0 --threads=0 certificate5896072-3.ps

real    0m1.250s
user    0m8.781s
sys     0m0.077s

$ ls -lh certificate5896072-3.ps.xz
-rw-r--r-- 1 user user 27M Nov 20  2012 certificate5896072-3.ps.xz

$ time gzip -1 -k certificate5896072-3.ps

real    0m1.365s
user    0m1.330s
sys     0m0.031s

$ ls -lh certificate5896072-3.ps.gz
-rw-r--r-- 1 user user 28M Nov 20  2012 certificate5896072-3.ps.gz

$ time xz certificate5896072-3.ps

real    0m1.974s
user    0m9.843s
sys     0m0.122s

$ ls -lh certificate5896072-3.ps.xz
-rw-r--r-- 1 user user 26M Nov 20  2012 certificate5896072-3.ps.xz

$ time xz -9 --threads=0 certificate5896072-3.ps

real    0m13.182s
user    0m12.980s
sys     0m0.190s

$ ls -lh certificate5896072-3.ps.xz
-rw-r--r-- 1 user user 23M Nov 20  2012 certificate5896072-3.ps.xz

$ time xz -6 --threads=1 certificate5896072-3.ps

real    0m13.615s
user    0m13.494s
sys     0m0.081s

$ ls -lh certificate5896072-3.ps.xz
-rw-r--r-- 1 user user 24M Nov 20  2012 certificate5896072-3.ps.xz

As you can see, the fastest xz exceeds the compression ratio of the slowest gzip and the speed of the fastest gzip, by using multiple cores (test machine is an 8-thread/4-core portable i7). Allowing slightly more time than gzip gives slighly better compression, and allowing significantly more time than gzip yields marginally better compression still. But note the last test - with the xz defaults, the time is the worst and the compression isn’t the best. In that case, compression is more than 10% better but the time is 10 times worse. These defaults will only hinder the adoption of xz, which is a shame because faster defaults would get more users and already provide better results than gzip on typical workloads on modern CPU’s.