01b2331ad1
to v1.4.3 |
||
---|---|---|
.. | ||
windres | ||
.gitignore | ||
benchfn.c | ||
benchfn.h | ||
benchzstd.c | ||
benchzstd.h | ||
BUCK | ||
datagen.c | ||
datagen.h | ||
dibio.c | ||
dibio.h | ||
fileio.c | ||
fileio.h | ||
Makefile | ||
platform.h | ||
README.md | ||
timefn.c | ||
timefn.h | ||
util.c | ||
util.h | ||
zstd.1 | ||
zstd.1.md | ||
zstdcli.c | ||
zstdgrep | ||
zstdgrep.1 | ||
zstdgrep.1.md | ||
zstdless | ||
zstdless.1 | ||
zstdless.1.md |
Command Line Interface for Zstandard library
Command Line Interface (CLI) can be created using the make
command without any additional parameters.
There are however other Makefile targets that create different variations of CLI:
zstd
: default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd formatszstd_nolegacy
: Same aszstd
but without support for legacy zstd formatszstd-small
: CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formatszstd-compress
: version of CLI which can only compress into zstd formatzstd-decompress
: version of CLI which can only decompress zstd format
Compilation variables
zstd
scope can be altered by modifying the following make
variables :
-
HAVE_THREAD : multithreading is automatically enabled when
pthread
is detected. It's possible to disable multithread support, by settingHAVE_THREAD=0
. Example :make zstd HAVE_THREAD=0
It's also possible to force multithread support, usingHAVE_THREAD=1
. In which case, linking stage will fail if neitherpthread
norwindows.h
library can be found. This is useful to ensure this feature is not silently disabled. -
ZSTD_LEGACY_SUPPORT :
zstd
can decompress files compressed by older versions ofzstd
. Starting v0.8.0, all versions ofzstd
produce frames compliant with the specification, and are therefore compatible. But older versions (< v0.8.0) produced different, incompatible, frames. By default,zstd
supports decoding legacy formats >= v0.4.0 (ZSTD_LEGACY_SUPPORT=4
). This can be altered by modifying this compilation variable.ZSTD_LEGACY_SUPPORT=1
means "support all formats >= v0.1.0".ZSTD_LEGACY_SUPPORT=2
means "support all formats >= v0.2.0", and so on.ZSTD_LEGACY_SUPPORT=0
means DO NOT support any legacy format. ifZSTD_LEGACY_SUPPORT >= 8
, it's the same as0
, since there is no legacy format after7
. Note :zstd
only supports decoding older formats, and cannot generate any legacy format. -
HAVE_ZLIB :
zstd
can compress and decompress files in.gz
format. This is ordered through command--format=gzip
. Alternatively, symlinks namedgzip
orgunzip
will mimic intended behavior..gz
support is automatically enabled whenzlib
library is detected at build time. It's possible to disable.gz
support, by settingHAVE_ZLIB=0
. Example :make zstd HAVE_ZLIB=0
It's also possible to force compilation with zlib support, usingHAVE_ZLIB=1
. In which case, linking stage will fail ifzlib
library cannot be found. This is useful to prevent silent feature disabling. -
HAVE_LZMA :
zstd
can compress and decompress files in.xz
and.lzma
formats. This is ordered through commands--format=xz
and--format=lzma
respectively. Alternatively, symlinks namedxz
,unxz
,lzma
, orunlzma
will mimic intended behavior..xz
and.lzma
support is automatically enabled whenlzma
library is detected at build time. It's possible to disable.xz
and.lzma
support, by settingHAVE_LZMA=0
. Example :make zstd HAVE_LZMA=0
It's also possible to force compilation with lzma support, usingHAVE_LZMA=1
. In which case, linking stage will fail iflzma
library cannot be found. This is useful to prevent silent feature disabling. -
HAVE_LZ4 :
zstd
can compress and decompress files in.lz4
formats. This is ordered through commands--format=lz4
. Alternatively, symlinks namedlz4
, orunlz4
will mimic intended behavior..lz4
support is automatically enabled whenlz4
library is detected at build time. It's possible to disable.lz4
support, by settingHAVE_LZ4=0
. Example :make zstd HAVE_LZ4=0
It's also possible to force compilation with lz4 support, usingHAVE_LZ4=1
. In which case, linking stage will fail iflz4
library cannot be found. This is useful to prevent silent feature disabling. -
BACKTRACE :
zstd
can display a stack backtrace when execution generates a runtime exception. By default, this feature may be degraded/disabled on some platforms unless additional compiler directives are applied. When triaging a runtime issue, enabling this feature can provide more context to determine the location of the fault. Example :make zstd BACKTRACE=1
Aggregation of parameters
CLI supports aggregation of parameters i.e. -b1
, -e18
, and -i1
can be joined into -b1e18i1
.
Symlink shortcuts
It's possible to invoke zstd
through a symlink.
When the name of the symlink has a specific value, it triggers an associated behavior.
zstdmt
: compress using all cores available on local system.zcat
: will decompress and output target file using any of the supported formats.gzcat
andzstdcat
are also equivalent.gzip
: if zlib support is enabled, will mimicgzip
by compressing file using.gz
format, removing source file by default (use--keep
to preserve). If zlib is not supported, triggers an error.xz
: if lzma support is enabled, will mimicxz
by compressing file using.xz
format, removing source file by default (use--keep
to preserve). If xz is not supported, triggers an error.lzma
: if lzma support is enabled, will mimiclzma
by compressing file using.lzma
format, removing source file by default (use--keep
to preserve). If lzma is not supported, triggers an error.lz4
: if lz4 support is enabled, will mimiclz4
by compressing file using.lz4
format. If lz4 is not supported, triggers an error.unzstd
andunlz4
will decompress any of the supported format.ungz
,unxz
andunlzma
will do the same, and will also remove source file by default (use--keep
to preserve).
Dictionary builder in Command Line Interface
Zstd offers a training mode, which can be used to tune the algorithm for a selected
type of data, by providing it with a few samples. The result of the training is stored
in a file selected with the -o
option (default name is dictionary
),
which can be loaded before compression and decompression.
Using a dictionary, the compression ratio achievable on small data improves dramatically. These compression gains are achieved while simultaneously providing faster compression and decompression speeds. Dictionary work if there is some correlation in a family of small data (there is no universal dictionary). Hence, deploying one dictionary per type of data will provide the greater benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will rely more and more on previously decoded content to compress the rest of the file.
Usage of the dictionary builder and created dictionaries with CLI:
- Create the dictionary :
zstd --train PathToTrainingSet/* -o dictionaryName
- Compress with the dictionary:
zstd FILE -D dictionaryName
- Decompress with the dictionary:
zstd --decompress FILE.zst -D dictionaryName
Benchmark in Command Line Interface
CLI includes in-memory compression benchmark module for zstd.
The benchmark is conducted using given filenames. The files are read into memory and joined together.
It makes benchmark more precise as it eliminates I/O overhead.
Multiple filenames can be supplied, as multiple parameters, with wildcards,
or names of directories can be used as parameters with -r
option.
The benchmark measures ratio, compressed size, compression and decompression speed.
One can select compression levels starting from -b
and ending with -e
.
The -i
parameter selects minimal time used for each of tested levels.
Usage of Command Line Interface
The full list of options can be obtained with -h
or -H
parameter:
Usage :
zstd [args] [FILE(s)] [-o file]
FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
-# : # compression level (1-19, default: 3)
-d : decompression
-D file: use `file` as Dictionary
-o file: result stored into `file` (only if 1 input file)
-f : overwrite output without prompting and (de)compress links
--rm : remove source file(s) after successful de/compression
-k : preserve source file(s) (default)
-h/-H : display help/long help and exit
Advanced arguments :
-V : display Version number and exit
-v : verbose mode; specify multiple times to increase verbosity
-q : suppress warnings; specify twice to suppress errors too
-c : force write to standard output, even if it is the console
-l : print information about zstd compressed files
--ultra : enable levels beyond 19, up to 22 (requires more memory)
--long : enable long distance matching (requires more memory)
--no-dictID : don't write dictID into header (dictionary compression)
--[no-]check : integrity check (default: enabled)
-r : operate recursively on directories
--format=gzip : compress files to the .gz format
--format=xz : compress files to the .xz format
--format=lzma : compress files to the .lzma format
--test : test compressed file integrity
--[no-]sparse : sparse mode (default: disabled)
-M# : Set a memory usage limit for decompression
-- : All arguments after "--" are treated as files
Dictionary builder :
--train ## : create a dictionary from a training set of files
--train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args
--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,shrink[=#],accel=#] : use the fastcover algorithm with optional args
--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
-o file : `file` is dictionary name (default: dictionary)
--maxdict=# : limit dictionary to specified size (default: 112640)
--dictID=# : force dictionary ID to specified value (default: random)
Benchmark arguments :
-b# : benchmark file(s), using # compression level (default: 3)
-e# : test all compression levels from -bX to # (default: 1)
-i# : minimum evaluation time in seconds (default: 3s)
-B# : cut file into independent blocks of size # (default: no block)
--priority=rt : set process priority to real-time
Restricted usage of Environment Variables
Using environment variables to set compression/decompression parameters has security implications. Therefore,
we intentionally restrict its usage. Currently, only ZSTD_CLEVEL
is supported for setting compression level.
If the value of ZSTD_CLEVEL
is not a valid integer, it will be ignored with a warning message.
Note that command line options will override corresponding environment variable settings.
Long distance matching mode
The long distance matching mode, enabled with --long
, is designed to improve
the compression ratio for files with long matches at a large distance (up to the
maximum window size, 128 MiB
) while still maintaining compression speed.
Enabling this mode sets the window size to 128 MiB
and thus increases the memory
usage for both the compressor and decompressor. Performance in terms of speed is
dependent on long matches being found. Compression speed may degrade if few long
matches are found. Decompression speed usually improves when there are many long
distance matches.
Below are graphs comparing the compression speed, compression ratio, and
decompression speed with and without long distance matching on an ideal use
case: a tar of four versions of clang (versions 3.4.1
, 3.4.2
, 3.5.0
,
3.5.1
) with a total size of 244889600 B
. This is an ideal use case as there
are many long distance matches within the maximum window size of 128 MiB
(each
version is less than 128 MiB
).
Compression Speed vs Ratio | Decompression Speed |
---|---|
Method | Compression ratio | Compression speed | Decompression speed |
---|---|---|---|
zstd -1 |
5.065 |
284.8 MB/s |
759.3 MB/s |
zstd -5 |
5.826 |
124.9 MB/s |
674.0 MB/s |
zstd -10 |
6.504 |
29.5 MB/s |
771.3 MB/s |
zstd -1 --long |
17.426 |
220.6 MB/s |
1638.4 MB/s |
zstd -5 --long |
19.661 |
165.5 MB/s |
1530.6 MB/s |
zstd -10 --long |
21.949 |
75.6 MB/s |
1632.6 MB/s |
On this file, the compression ratio improves significantly with minimal impact on compression speed, and the decompression speed doubles.
On the other extreme, compressing a file with few long distance matches (such as the Silesia compression corpus) will likely lead to a deterioration in compression speed (for lower levels) with minimal change in compression ratio.
The below table illustrates this on the Silesia compression corpus.
Method | Compression ratio | Compression speed | Decompression speed |
---|---|---|---|
zstd -1 |
2.878 |
231.7 MB/s |
594.4 MB/s |
zstd -1 --long |
2.929 |
106.5 MB/s |
517.9 MB/s |
zstd -5 |
3.274 |
77.1 MB/s |
464.2 MB/s |
zstd -5 --long |
3.319 |
51.7 MB/s |
371.9 MB/s |
zstd -10 |
3.523 |
16.4 MB/s |
489.2 MB/s |
zstd -10 --long |
3.566 |
16.2 MB/s |
415.7 MB/s |
zstdgrep
zstdgrep
is a utility which makes it possible to grep
directly a .zst
compressed file.
It's used the same way as normal grep
, for example :
zstdgrep pattern file.zst
zstdgrep
is not compatible with dictionary compression.
To search into a file compressed with a dictionary,
it's necessary to decompress it using zstd
or zstdcat
,
and then pipe the result to grep
. For example :
zstdcat -D dictionary -qc -- file.zst | grep pattern