AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,040 Commits 1 Branch 0 Tags 27 MiB
88fac5d514
Commit Graph

639 Commits

Author SHA1 Message Date
Lzu Tao
ef2e761937 Helper script to install symlink in meson 2018-12-01 23:18:59 +07:00
Lzu Tao
3175188407 No need these helpers 2018-12-01 23:18:59 +07:00
Lzu Tao
337f914dc8 Fix lib soversion and no install cover.h header 2018-12-01 23:18:59 +07:00
Lzu Tao
c9f0144302 Fix meson tests build 2018-12-01 23:18:59 +07:00
Lzu Tao
5a36a57cf5 Bump to 1.3.8 and fix run_command function 
The run_command is run from an unspecified directory. Therefore we cannot assume
which directory it is running our command.
 2018-12-01 23:18:59 +07:00
Lzu Tao
8a160680d1 Update legacy support to 5 2018-12-01 23:18:59 +07:00
Lzu Tao
f727808731 Minor fix for meson build 
Use files function instead of constructing path with meson.current_source_dir()
 2018-12-01 23:18:59 +07:00
Lzu Tao
9a721e5216 Update meson build system 
NOTE: This commit only tested on Linux (Ubuntu 18.04). Windows
build may not work as expected.

* Use meson >= 0.47.0 cause we use install_man function
* Add three helper Python script:
  * CopyFile.py: To copy file
  * CreateSymlink.py: To make symlink (both Windows and Unix)
  * GetZstdLibraryVersion.py: Parse lib/zstd.h to get zstd version
  These help emulating equivalent functions in CMake and Makefile.
* Use subdir from meson to split meson.build
  * Add contrib build
  * Fix other build
* Add new build options
  * build_programs: Enable programs build
  * build_contrib: Enable contrib build
  * build_tests: Enable tests build
  * use_static_runtime: Link to static run-time libraries on MSVC
  * zlib_support: Enable zlib support
  * lzma_support: Enable lzma support
 2018-11-28 01:08:34 +07:00
Lzu Tao
9bd8f6a00c Rename and update build instruction in README file to README.md 2018-11-28 01:08:34 +07:00
Lzu Tao
2abd5139a5 Add meson build guide 2018-11-28 01:08:34 +07:00
Yann Collet
5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet
b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet
b830ccca5c changed benchfn api 
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
 2018-11-13 13:12:50 -08:00
Yann Collet
d38063f8ae separated bench module into benchfn and benchzstd 
it shall be possible to use benchfn
without any dependency on zstd.
 2018-11-13 11:01:59 -08:00
Yann Collet
483759a3de Improves decompression speed when using cold dictionary 
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).

Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)

The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.

This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).

Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
 2018-11-08 17:00:23 -08:00
Rohit Jain
705e0b18ab Making changes to make it compile on my laptop 2018-10-11 15:51:57 -07:00
Yann Collet
123fac6b6d fix pzstd compatibility with mingw 
some details changed with introduction of gcc7
 2018-09-21 17:36:00 -07:00
Yann Collet
00ce26725b
Merge pull request #1324 from ko-zu/fixclangcode 
Fix largeNbDicts bench for clangbuild
 2018-09-17 14:10:17 -07:00
Nick Terrell
8f27e8cf3d
Merge pull request #1322 from azat-archive/seekable-fixes-pull 
Fixes read write past end of input buffer.
 2018-09-17 11:04:51 -07:00
ko-zu
b053bec2f4 Fix largeNbDicts bench for clangbuild 
Remove unsigned to size_t promotion to fix implicit down conversion errors in clangbuild target.
 2018-09-17 13:09:08 +09:00
Azat Khuzhin
d707692e05
seekable_decompression: support offset greater then UNIT_MAX 2018-09-16 18:05:32 +03:00
Azat Khuzhin
b52867a97f
zstdseek_decompress: fix decompression with data left in input buffer 2018-09-16 18:05:32 +03:00
Yann Collet
c49ccbc8e7 largeNbDicts : can select a nb of blocks 
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
 2018-09-12 11:31:28 -07:00
Yann Collet
50b216146f
Merge pull request #1304 from facebook/largeNbDicts 
contrib/largeNbDicts
 2018-09-06 09:50:56 -07:00
Yann Collet
c57a856d64 fixed minor static analyzer warning 2018-09-05 14:33:51 -07:00
Yann Collet
1d487d587f updated documentation 2018-09-04 14:57:45 -07:00
Yann Collet
11b8b8c100 silenced false-positive scan-build warning 2018-08-31 10:01:06 -07:00
Yann Collet
0ff67511e6 fixed link order for old compilers 2018-08-30 16:43:28 -07:00
Yann Collet
f76253bb70 minor : createDictionaryBuffer() can create dictionaries of different sizes 2018-08-30 16:24:44 -07:00
Yann Collet
39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Yann Collet
39ef91a599 -std=c99 for largeNbDicts 2018-08-30 14:59:23 -07:00
Yann Collet
4086b2871b largeNbDicts compatible with multiple source files 
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
 2018-08-30 14:38:49 -07:00
Yann Collet
a5a77965d3 make all includes contrib/largeNbDicts 2018-08-29 16:17:22 -07:00
Yann Collet
d89fa814c1 added a README 
for documentation
 2018-08-28 18:19:19 -07:00
Yann Collet
6444c50035 increases randomness of ddict ptrs 2018-08-28 18:13:46 -07:00
Yann Collet
6c398df241 level, block size and nb dicts can be set on command line 2018-08-28 18:05:31 -07:00
Yann Collet
0c66a44d1b first working test program 
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
 2018-08-28 15:47:07 -07:00
Yann Collet
274b60e6e6 largeNbDicts can compress and compare dict vs noDict 2018-08-27 17:08:44 -07:00
Yann Collet
6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Jennifer Liu
9d6ed9def3 Merge fastCover into DictBuilder (#1274) 
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
 2018-08-23 12:06:20 -07:00
Yann Collet
36d6165a2d Makefile: added variable SCANBUILD 
so that a different version of scan-build can be selected
 2018-08-16 16:44:13 -07:00
Yann Collet
42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
Jennifer Liu
0acb0abd1e Add non-optimize FASTCOVER (#1260) 
* Add non-optimize FASTCOVER

* Minor fix

* Pass param as value instead of pointer
 2018-08-01 11:06:16 -07:00
Jennifer Liu
4e29bc2469 Use CDict instead of CCtx in analyzeEntropy 2018-07-31 10:36:45 -07:00
Jennifer Liu
31229e527b Increment frequency for every dmer occurence within same sample instead of at most once per sample 2018-07-30 12:54:22 -07:00
Jennifer Liu
51b109c1b5 Delete old benchmarking result 2018-07-27 17:31:33 -07:00
Jennifer Liu
53ef22a4bc Undo deleting clean in make 2018-07-27 16:56:50 -07:00
Jennifer Liu
96d84ee235 Revert test.sh 2018-07-27 16:54:05 -07:00
Jennifer Liu
61262f6c0d Save segmentFreqs in ctx instead of malloc and memset in SelectSegment 2018-07-27 16:51:38 -07:00
Jennifer Liu
49b398e93f Use same param after optimizing cover and fastCover and record k and d for benchmarking 2018-07-27 13:39:19 -07:00
Jennifer Liu
759c543312 Rerun cover and fastCover with optimized values 2018-07-26 19:03:01 -07:00
Jennifer Liu
3d7941ce41 Benchmark different f values 2018-07-26 16:24:13 -07:00
Jennifer Liu
3b163e0b5b Add array to keep track of frequency within active segment, fix malloc bug, update benchmarking result 2018-07-26 13:53:13 -07:00
Jennifer Liu
2333ecb173 Allow d=6 2018-07-25 18:10:09 -07:00
Jennifer Liu
1e85f314d8 Benchmark fast cover optimize vs k=200 2018-07-25 17:53:38 -07:00
Jennifer Liu
d1fc507ef9 Initial benchmarking result for fastCover 2018-07-25 17:05:54 -07:00
Jennifer Liu
f5407e398a Make hash value const 2018-07-25 16:54:08 -07:00
Jennifer Liu
7f3f70f766 Add Fast Cover Dictionary Builder 2018-07-25 16:34:07 -07:00
Nick Terrell
77068a8447
Merge pull request #1246 from jennifermliu/benchmark 
Benchmark dictionary builders
 2018-07-20 18:09:31 -07:00
Jennifer Liu
b6c5d4982c Minor fix 2018-07-20 17:41:22 -07:00
Jennifer Liu
71e767ac09 Refactoring and benchmark without dictionary 2018-07-20 17:03:47 -07:00
Jennifer Liu
470c8d42f4 Benchmark dictionary builders 2018-07-20 11:32:39 -07:00
Nick Terrell
4d1ad5cdb2
Merge pull request #1238 from jennifermliu/random 
Add random dictionary builder
 2018-07-19 13:52:15 -07:00
Jennifer Liu
0c5eaef248 Update Makefile 2018-07-19 13:44:27 -07:00
Jennifer Liu
5bb46a898e Rename cleanup 2018-07-18 12:15:49 -07:00
Jennifer Liu
52e7cf0e40 Add cleanup to trainfromFiles and move RANDOM_segment_t declaration 2018-07-18 10:40:13 -07:00
Jennifer Liu
ce09fb723d Update freeSampleInfo 2018-07-17 16:13:40 -07:00
Jennifer Liu
896ff0644a Fix deallocation problem and add documentation 2018-07-17 16:01:44 -07:00
Jennifer Liu
e6fe405838 Make test PHONY target 2018-07-17 12:42:53 -07:00
Jennifer Liu
49acfaeaec Move file loading functions to new file for access by benchmarking tool 2018-07-17 12:35:09 -07:00
Jennifer Liu
4d32339b75 Remove CLevel cli option which was accidentally added back in the last commit 2018-07-16 18:59:18 -07:00
Jennifer Liu
1f7fa5cdd6 Fix spacing and Edit Makefile (now run with make instead of make run) 2018-07-16 16:31:59 -07:00
Jennifer Liu
b5806d33db Refactor RANDOM 2018-07-16 16:03:04 -07:00
Codecat
044cd81ce6
Fix wrong conditions 2018-07-14 12:34:03 +02:00
Codecat
1a61bdb9c0
Update zstd.lua 2018-07-14 12:27:42 +02:00
Jennifer Liu
0e5fbc10fa Update README 2018-07-13 17:41:09 -07:00
Jennifer Liu
31731df4da Remove clevel and update documentation 2018-07-13 17:38:53 -07:00
Jennifer Liu
a23a3b95f9 Add random dictionary builder 2018-07-13 16:05:14 -07:00
Codecat
23d77c531e
Added premake4/GENie script to contrib folder 2018-07-11 18:02:18 +02:00
Jon Turney
87579d51ee
meson: fix build 2018-07-02 15:06:46 +01:00
Ryan Schmidt
bf30b9caf4 Add CXXFLAGS to ALL_LDFLAGS (#1178) 
pzstd requires C++11, which older C++ standard libraries like libstdc++
as used on OS X 10.8 and earlier don't support. The user might address
this by setting "CXXFLAGS=-stdlib=libc++". This flag must be used both
at compile time and at link time. Asking the user to also put the flag
in LDFLAGS is undesirable because then the flag would also be used when
linking C code, which would be inappropriate.
 2018-06-14 13:05:33 -07:00
Yann Collet
750ee87a92
Merge pull request #1175 from ryandesign/macos 
Fix name of macOS
 2018-06-13 11:32:06 -04:00
Ryan Schmidt
8ebd0f3a17 Remove duplicate CXXFLAGS 2018-06-10 18:04:21 -05:00
Ryan Schmidt
b567ce9d68 Fix name of macOS 2018-06-09 14:31:17 -05:00
Yann Collet
97c60cdf36 fixed seekable_format type mismatch 
and some minor "unused variable" warnings.
Also : zstd_seekable.h is actually depending on zstd.h for ZSTDLIB_API
 2018-06-06 13:10:29 -07:00
Yann Collet
30ee23e905 ensure seekable_format/examples generated libzstd.a 
when it's not already present in the expected directory
 2018-06-06 12:09:58 -07:00
Chris Lamb
2dbe408a49 Make the build reproducible 
Whilst working on the Reproducible Builds effort [0], we noticed
that zstd could not be built reproducibly.

This is due to the manual page encoding the number of CPUs from the
build machine and thus varies across builds.

This was originally filed in Debian as #897904 [1].

 [0] https://reproducible-builds.org/
 [1] https://bugs.debian.org/897904

Signed-off-by: Chris Lamb <lamby@debian.org>
 2018-05-04 08:39:51 -07:00
Adam Borowski
00d52282d0 Use -pthread rather than -lpthread. 
It can have other effects, such as pulling in extra libraries.  Without it,
riscv build fails with undefined reference to `__atomic_compare_exchange_1'.
 2018-04-09 01:50:49 +02:00
Yann Collet
9e6ba88363 moved snap to /contrib 2018-03-19 16:15:06 -07:00
Yann Collet
a7b532a347 added docker readme 2018-03-19 16:13:12 -07:00
Yann Collet
b06db3b3c5
Merge pull request #1052 from Varunram/dockerfile 
Add Dockerfile
 2018-03-19 16:07:41 -07:00
Varunram
7616200eaf Add Dockerfile 
Dockerfile initially proposed by @gyscos at #880
 2018-03-18 14:53:48 +05:30
Yann Collet
5373e44ba7 fixed contrib/adaptive-compression 2018-03-15 17:10:15 -07:00
Yann Collet
355cb645bf fixed seekable format example 2018-03-15 16:29:28 -07:00
Yann Collet
38cbcb5f1a removed LRM exploratory experiment 2018-03-15 16:26:08 -07:00
Yann Collet
e916b9090e gen_html: changed CFLAGS for CXXFLAGS 
since it's associated with $(CXX)
 2018-03-09 11:52:14 -08:00
Shawn Landden
ea41b580eb meson: allow -Dlegacy_support=true, fix -Dlegacy_support=0 2017-12-29 10:18:33 -08:00
Shawn Landden
6ff43c0051 get soversion right 2017-12-24 10:05:43 -08:00
Shawn Landden
daffe435c0 meson: support differn't legacy levels. 
Default to v0.4.0+
 2017-12-21 15:47:38 -08:00
Shawn Landden
3ddfa42fe8 meson: fix build 
used absolute paths which are deprecated in meson, also missing some sources
that got split

also move source files each to their own line so future diffs are clearer.
 2017-12-19 22:02:03 -08:00
Yann Collet
c173dbd6e7 no longer supported starting C++17 2017-12-04 18:00:53 -08:00
Yann Collet
0d58aaf6f0 /contrib: fixed license header 
removed last reference to PATENTS file
 2017-10-02 02:07:17 -07:00
Yann Collet
f1a3ce34b7 Merge pull request #826 from terrelln/license 
[linux-kernel] Update license
 2017-08-31 12:52:23 -07:00
Nick Terrell
f9252d8347 [linux-kernel] Update license 2017-08-31 12:48:36 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00
Yann Collet
e9dc204f42 fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
Yann Collet
394bdd7db9 changed license for examples 
intentionnally this time
 2017-08-29 09:24:11 -07:00
Yann Collet
dd63533b29 Merge pull request #797 from paulcruz74/documentation 
Updating README.md
 2017-08-16 00:37:55 -07:00
Paul Cruz
733ca51360 Updating README.md 2017-08-15 17:48:23 -07:00
Yann Collet
2daeab12f4 Merge pull request #795 from terrelln/squashfs-tools2 
[kernel] Update squashfs-tools patch
 2017-08-15 11:59:41 -07:00
Nick Terrell
57e2df6651 [kernel] Update squashfs-tools patch 2017-08-14 22:43:36 -07:00
Paul Cruz
fc90469587 updated program name print statement 2017-08-10 16:11:59 -07:00
Paul Cruz
75c3ad947c Merge remote-tracking branch 'upstream/dev' into dev 2017-08-10 14:46:31 -07:00
Yann Collet
8915b9b518 Merge pull request #786 from terrelln/squashfs-tools 
[linux-kernel] Update patches for v4 and v5
 2017-08-09 16:00:11 -07:00
Nick Terrell
8b6702a00d [linux-kernel] Update patches for v5 2017-08-09 13:03:40 -07:00
Yann Collet
a1fe656165 Merge pull request #779 from paulcruz74/adapt-approach-4 
Patching warnings, adding ability to bound compression variation
 2017-08-07 14:50:38 -07:00
Paul Cruz
7069bb9458 Merge branch 'adapt-approach-4' into bug-fixes 2017-08-07 13:18:53 -07:00
Paul Cruz
e100a311eb removed direct assignment of 22, used ZSTD_maxCLevel() instead 2017-08-07 13:11:07 -07:00
Nick Terrell
7393b49fbd [linux-kernel] Update patches for v4 2017-08-04 16:57:03 -07:00
Paul Cruz
01237e3b35 changed multi to zstd-adaptive in the help menu 2017-08-03 15:13:49 -07:00
Paul Cruz
8be7bba08c added mutex for compression level to avoid data race 2017-08-02 10:27:33 -07:00
Paul Cruz
69ef22c0ac added detach statements to prevent resource leak 2017-08-01 17:36:13 -07:00
Paul Cruz
f8c7b191e7 Merge branch 'adapt-approach-4' into dev 2017-07-31 15:42:57 -07:00
Paul Cruz
0295737ad7 change signal to broadcast for jobCompressed condition varaible since multiple threads waiting 2017-07-31 13:43:03 -07:00
Paul Cruz
cf92221671 Merge remote-tracking branch 'upstream/dev' into dev 2017-07-31 11:43:40 -07:00
Paul Cruz
c2f1bda8a1 Merge branch 'adapt-approach-4' into dev 2017-07-31 11:43:01 -07:00
Paul Cruz
9ea7df03de add install target in makefile 2017-07-31 11:04:17 -07:00
Paul Cruz
f60cd3f99b print defaults and range, remove EXT 2017-07-31 09:47:09 -07:00
Paul Cruz
e22b60cb76 removed ternary operation, added assert statement, check to make sure initial compression level is within bounds 2017-07-28 17:46:51 -07:00
Paul Cruz
cb9af53e77 delete empty line 2017-07-28 17:28:25 -07:00
Paul Cruz
51788225db remove exe extension from makefile, reinclude pthread flag 2017-07-28 17:27:36 -07:00
Yann Collet
ee27f6937c Merge pull request #758 from stellamplau/ldm 
Add working prototype of a long distance matcher
 2017-07-28 17:07:21 -07:00
Paul Cruz
4d904ac800 add flags for multithreading 2017-07-28 16:12:58 -07:00
Paul Cruz
0f4cb67b00 add tests for compression bounds, fix another warning 2017-07-28 15:55:02 -07:00
Paul Cruz
ff54fced64 patched style errors, add ability to bound compression level variation 2017-07-28 15:30:46 -07:00
Stella Lau
8fae41c412 Return error code in verify() and minor code cleanup 2017-07-27 17:14:05 -07:00
Stella Lau
1294a4a897 Fix typo 2017-07-27 15:49:46 -07:00
Stella Lau
627621839c Add checks in initialization code 2017-07-27 15:37:37 -07:00
Stella Lau
c105f605e6 Update README 2017-07-27 11:11:35 -07:00
Yann Collet
e1222544be Merge pull request #753 from paulcruz74/adapt-approach-3 
adaptive compression v1
 2017-07-27 10:00:10 -07:00
Paul Cruz
2320e7378a remove unused variable, add documentation for context fields 2017-07-26 17:02:47 -07:00
Stella Lau
9eaf3d22d0 Allow HASH_ONLY_EVERY_LOG to be configured in ldm.h 2017-07-26 16:43:25 -07:00
Paul Cruz
ab5a78547e fix leaky abstraction regarding measuring completion 2017-07-26 16:40:05 -07:00
Paul Cruz
715f36ca81 added definitions for conversion constants, moved forced compression check to top of adaptCompressionLevel, used ZSTD_BLOCKSIZE_MAX 2017-07-26 15:52:15 -07:00
Paul Cruz
6c1c1242fc set the window log value before performing compression 2017-07-26 14:29:59 -07:00
Stella Lau
40759bade9 Add README and clean up code 2017-07-26 13:24:03 -07:00
Paul Cruz
a959cc881a moved reset of completion to right after wait 2017-07-26 10:34:48 -07:00
Paul Cruz
305d5ee70f change to >= convergence counter 2017-07-26 10:20:29 -07:00
Paul Cruz
be92a38d6a decrease completion requirements for change, move create thread wait, merge cases where compression thread should wait 2017-07-26 10:05:10 -07:00