AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
9d6ed9def3
zstd/lib
History
Jennifer Liu 9d6ed9def3 Merge fastCover into DictBuilder (#1274) 
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
..
common fixed several minor issues detected by scan-build 2018-08-14 16:55:35 -07:00
compress fixed more issues detected by recent version of scan-build 2018-08-16 15:20:25 -07:00
decompress Makefile: added variable SCANBUILD 2018-08-16 16:44:13 -07:00
deprecated fixed zbufftest 2017-10-19 14:06:02 -07:00
dictBuilder Merge fastCover into DictBuilder (#1274) 2018-08-23 12:06:20 -07:00
dll fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
legacy Makefile: added variable SCANBUILD 2018-08-16 16:44:13 -07:00
.gitignore makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819) 2017-09-11 14:09:34 -07:00
BUCK Merge fastCover into DictBuilder (#1274) 2018-08-23 12:06:20 -07:00
libzstd.pc.in updated pkg config file 2016-11-30 11:06:58 -08:00
Makefile fix %zu support under minGW 2018-07-30 16:56:18 +02:00
README.md Documentation 2018-06-04 09:56:37 -07:00
zstd.h Add CCtx Param Controlling Dict Attachment Behavior 2018-06-21 17:29:25 -04:00

README.md

Zstandard library files

The lib directory is split into several sub-directories, in order to make it easier to select or exclude features.

Building

Makefile script is provided, supporting all standard Makefile conventions, including commands variables, staged install, directory variables and standard targets.

  • make : generates both static and dynamic libraries
  • make install : install libraries in default system directories

libzstd default scope includes compression, decompression, dictionary building, and decoding support for legacy formats >= v0.4.0.

API

Zstandard's stable API is exposed within lib/zstd.h.

Advanced API

Optional advanced features are exposed via :

  • lib/common/zstd_errors.h : translates size_t function results into an ZSTD_ErrorCode, for accurate error handling.
  • ZSTD_STATIC_LINKING_ONLY : if this macro is defined before including zstd.h, it unlocks access to advanced experimental API, exposed in second part of zstd.h. These APIs are not "stable", their definition may change in the future. As a consequence, it shall never be used with dynamic library ! Only static linking is allowed.

Modular build

It's possible to compile only a limited set of features.

  • Directory lib/common is always required, for all variants.
  • Compression source code lies in lib/compress
  • Decompression source code lies in lib/decompress
  • It's possible to include only compress or only decompress, they don't depend on each other.
  • lib/dictBuilder : makes it possible to generate dictionaries from a set of samples. The API is exposed in lib/dictBuilder/zdict.h. This module depends on both lib/common and lib/compress .
  • lib/legacy : source code to decompress legacy zstd formats, starting from v0.1.0. This module depends on lib/common and lib/decompress. To enable this feature, it's required to define ZSTD_LEGACY_SUPPORT during compilation. Typically, with gcc, add argument -DZSTD_LEGACY_SUPPORT=1. Using higher number limits versions supported. For example, ZSTD_LEGACY_SUPPORT=2 means : "support legacy formats >= v0.2.0". ZSTD_LEGACY_SUPPORT=3 means : "support legacy formats >= v0.3.0", and so on. Starting v0.8.0, all versions of zstd produce frames compliant with specification. As a consequence, ZSTD_LEGACY_SUPPORT=8 (or more) doesn't trigger legacy support. Also, ZSTD_LEGACY_SUPPORT=0 means "do not support legacy formats". Once enabled, this capability is transparently triggered within decompression functions. It's also possible to invoke directly legacy API, as exposed in lib/legacy/zstd_legacy.h. Each version also provides an additional dedicated set of advanced API. For example, advanced API for version v0.4 is exposed in lib/legacy/zstd_v04.h . Note : lib/legacy only supports decoding legacy formats.
  • Similarly, you can define ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION, ZSTD_LIB_DICTBUILDER, and ZSTD_LIB_DEPRECATED as 0 to forgo compilation of the corresponding features. This will also disable compilation of all dependencies (eg. ZSTD_LIB_COMPRESSION=0 will also disable dictBuilder).

Multithreading support

Multithreading is disabled by default when building with make. Enabling multithreading requires 2 conditions :

  • set macro ZSTD_MULTITHREAD
  • on POSIX systems : compile with pthread (-pthread compilation flag for gcc)

Both conditions are automatically triggered by invoking make lib-mt target. Note that, when linking a POSIX program with a multithreaded version of libzstd, it's necessary to trigger -pthread flag during link stage.

Multithreading capabilities are exposed via advanced API ZSTD_compress_generic() defined in lib/zstd.h. This API is still considered experimental, but is expected to become "stable" at some point in the future.

Windows : using MinGW+MSYS to create DLL

DLL can be created using MinGW+MSYS with the make libzstd command. This command creates dll\libzstd.dll and the import library dll\libzstd.lib. The import library is only required with Visual C++. The header file zstd.h and the dynamic library dll\libzstd.dll are required to compile a project using gcc/MinGW. The dynamic library has to be added to linking options. It means that if a project that uses ZSTD consists of a single test-dll.c file it should be linked with dll\libzstd.dll. For example:

    gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll

The compiled executable will require ZSTD DLL which is available at dll\libzstd.dll.

Deprecated API

Obsolete API on their way out are stored in directory lib/deprecated. At this stage, it contains older streaming prototypes, in lib/deprecated/zbuff.h. These prototypes will be removed in some future version. Consider migrating code towards supported streaming API exposed in zstd.h.

Miscellaneous

The other files are not source code. There are :

  • LICENSE : contains the BSD license text
  • Makefile : make script to build and install zstd library (static and dynamic)
  • BUCK : support for buck build system (https://buckbuild.com/)
  • libzstd.pc.in : for pkg-config (used in make install)
  • README.md : this file