History

Sean Purcell 0f5c95af44 Disambiguate pledgedSrcSize == 0 - Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size - Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize		2017-02-08 15:12:46 -08:00
..
images	Minor tweaks to pzstd graph	2016-09-01 17:53:23 -07:00
test	Clean imports and shorten tests	2017-01-27 10:43:18 -08:00
utils	Add BUCK files for Nuclide support	2017-01-27 10:43:12 -08:00
.gitignore	zbufftest only depends on standard C time.h	2016-09-01 18:11:12 -07:00
BUCK	Add BUCK files for Nuclide support	2017-01-27 10:43:12 -08:00
ErrorHolder.h	Clean up compiler warnings + Build pzstd on travis	2016-09-02 12:23:49 -07:00
Logging.h	Revert unnecessary change to Logging.h	2017-01-27 13:26:44 -08:00
main.cpp	Clean imports and shorten tests	2017-01-27 10:43:18 -08:00
Makefile	Move -std=c++11 cxxflag to PZSTD_CXXFLAGS	2016-12-26 23:08:23 +08:00
Options.cpp	Clean imports and shorten tests	2017-01-27 10:43:18 -08:00
Options.h	Disambiguate pledgedSrcSize == 0	2017-02-08 15:12:46 -08:00
Pzstd.cpp	[pzstd] Cast unused parameters to void	2016-11-15 17:46:28 -08:00
Pzstd.h	[pzstd] Fix frame size for small files + add logging	2016-11-15 16:39:09 -08:00
README.md	[pzstd] Remove gtest dependency from make all	2016-11-14 11:56:28 -08:00
SkippableFrame.cpp	Clean up compiler warnings + Build pzstd on travis	2016-09-02 12:23:49 -07:00
SkippableFrame.h	Add PZstandard to contrib/	2016-09-01 15:25:31 -07:00

README.md

Parallel Zstandard (PZstandard)

Parallel Zstandard is a Pigz-like tool for Zstandard. It provides Zstandard format compatible compression and decompression that is able to utilize multiple cores. It breaks the input up into equal sized chunks and compresses each chunk independently into a Zstandard frame. It then concatenates the frames together to produce the final compressed output. Pzstandard will write a 12 byte header for each frame that is a skippable frame in the Zstandard format, which tells PZstandard the size of the next compressed frame. PZstandard supports parallel decompression of files compressed with PZstandard. When decompressing files compressed with Zstandard, PZstandard does IO in one thread, and decompression in another.

Usage

PZstandard supports the same command line interface as Zstandard, but also provides the -p option to specify the number of threads. Dictionary mode is not currently supported.

Basic usage

pzstd input-file -o output-file -p num-threads -#          # Compression
pzstd -d input-file -o output-file -p num-threads          # Decompression

PZstandard also supports piping and fifo pipes

cat input-file | pzstd -p num-threads -# -c > /dev/null

For more options

pzstd --help

PZstandard tries to pick a smart default number of threads if not specified (displayed in pzstd --help). If this number is not suitable, during compilation you can define PZSTD_NUM_THREADS to the number of threads you prefer.

Benchmarks

As a reference, PZstandard and Pigz were compared on an Intel Core i7 @ 3.1 GHz, each using 4 threads, with the Silesia compression corpus.

Compression Speed vs Ratio with 4 Threads	Decompression Speed with 4 Threads

The test procedure was to run each of the following commands 2 times for each compression level, and take the minimum time.

time pzstd -# -p 4    -c silesia.tar     > silesia.tar.zst
time pzstd -d -p 4    -c silesia.tar.zst > /dev/null

time pigz  -# -p 4 -k -c silesia.tar     > silesia.tar.gz
time pigz  -d -p 4 -k -c silesia.tar.gz  > /dev/null

PZstandard was tested using compression levels 1-19, and Pigz was tested using compression levels 1-9. Pigz cannot do parallel decompression, it simply does each of reading, decompression, and writing on separate threads.

Tests

Tests require that you have gtest installed. Set GTEST_INC and GTEST_LIB in Makefile to specify the location of the gtest headers and libraries. Alternatively, run make googletest, which will clone googletest and build it. Run make tests && make check to run tests.