AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,177 Commits 1 Branch 0 Tags 27 MiB
bdb673666f
Commit Graph

5177 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
184e1c8b8c updated benchmark figures 2018-03-22 18:22:10 -07:00
Yann Collet
01082a39bd restored simple status line during zstd compression 
the more advanced one, featuring amount of data buffered,
is triggered on `-v`.
 2018-03-22 17:49:46 -07:00
Yann Collet
29b021f9a0
Merge pull request #1067 from facebook/targetLength 
removed limit ZSTD_TARGETLENGTH_MAX
 2018-03-22 10:38:33 -07:00
Yann Collet
1340cafbeb
Merge pull request #1068 from terrelln/mt-fixes 
Fix broken assertion
 2018-03-22 10:38:17 -07:00
Yann Collet
f15e2fd55e fixed fuzz test 2018-03-21 16:45:15 -07:00
Nick Terrell
ad344033df Fix broken assertion 
The `avgJobSize` must not be lower than 256 KB for single-pass mode.
In `zstd.h` we say the minimum value for `ZSTD_p_jobSize` is 1 MB,
so ensure that we always pick a size >= 1 MB.

Found by libFuzzer fuzzer tests with large input limits.
 2018-03-21 16:20:30 -07:00
Yann Collet
b7b80e1569 fixed paramgrill 2018-03-21 16:09:15 -07:00
Yann Collet
93b8262c02 added large negative level test case 2018-03-21 15:54:44 -07:00
Yann Collet
153bc1c004 removed limit ZSTD_TARGETLENGTH_MAX 
this makes it possible to specify extremely large negative compression levels,
achieving the side effect as "no compression".

It will also be possible to define larger targetlength for ultra compression mode.

There is no adverse side effect due to removing this limit.
 2018-03-21 15:50:05 -07:00
Haydn Trigg
57e6b98eb1 Refined build script 2018-03-22 09:10:39 +10:30
Yann Collet
1be8660eca
Merge pull request #1062 from facebook/advancedDecompress 
Advanced decompression API
 2018-03-21 14:52:13 -07:00
Yann Collet
a99c4a3621 Merge branch 'dev' into advancedDecompress 2018-03-21 06:08:28 -07:00
Yann Collet
87b0cf05bd
Merge pull request #1057 from facebook/lrmSettings 
LRM parameters
 2018-03-21 05:59:39 -07:00
Yann Collet
1292a738d8
Merge pull request #1063 from Varunram/manpage 
Update man page variables in CMakeLists
 2018-03-21 05:36:31 -07:00
Haydn Trigg
321de5682c Generic Visual Studio 2017 build script 2018-03-21 22:57:42 +10:30
Varunram
0034c759b0 Update man page variables in CMakeLists 2018-03-21 14:10:42 +05:30
Yann Collet
453d487807 fixed minor declaration warning 2018-03-20 18:03:56 -07:00
Yann Collet
d1bf609abf
Merge pull request #1059 from terrelln/mt-ldm 
Integrate ldm with zstdmt
 2018-03-20 17:50:20 -07:00
Yann Collet
e0cb8d19c6 fixed legacy test case 2018-03-20 17:48:22 -07:00
Yann Collet
878728dc26 fixed several comments by @terrelln 2018-03-20 16:35:14 -07:00
Yann Collet
e1c52faace
Merge pull request #1060 from facebook/compressImpl 
merge bmi2 implementation of encodeSequence into zstd_compress.c
 2018-03-20 16:19:42 -07:00
Yann Collet
b18cb7e0b7 zstreamtest : slightly decreased time to build dictionary size 2018-03-20 16:18:48 -07:00
Yann Collet
6cda8c932c added test with ZSTD_decompress_generic() + ZSTD_DCtx_refPrefix() 
also :
clarified stage condition to accept new parameters,
fixed initializers correspondingly.
 2018-03-20 16:16:13 -07:00
Yann Collet
2af41592ea added a test with ZSTD_decompress_generic() + ZSTD_DCtx_loadDictionary_byReference() 2018-03-20 15:59:25 -07:00
Yann Collet
0dadb6b70d implemented ZSTD_DCtx_refPrefix*() 2018-03-20 15:45:56 -07:00
Yann Collet
569b8ba4d9 implemented ZSTD_DCtx_refDDict() 2018-03-20 15:43:49 -07:00
Nick Terrell
a3b76a77ef Quiet appveyor warnings 2018-03-20 15:34:40 -07:00
Yann Collet
6873fec658 changed dictMore for dictContentType 
which seems clearer to describe what the variable/argument is about.
 2018-03-20 15:13:14 -07:00
Yann Collet
31b54b6eea updated ZSTD_initStaticDDict() prototype 
can also specify dictContentType.
 2018-03-20 14:52:02 -07:00
Nick Terrell
136b9e2392 Fix external sequence corner cases 
* Clear external sequences when we reset the `ZSTD_CCtx`.
* Skip external sequences when a block is too small to compress.
 2018-03-20 14:50:28 -07:00
Yann Collet
353117c5d7 implemented ZSTD_DCtx_loadDictionary*() 
this required updating ZSTD_createDDict_advanced()
to accept a dictContentType parameter (raw, full, auto).
 2018-03-20 13:40:29 -07:00
Yann Collet
451357f37f
Merge pull request #1058 from facebook/cctxParams 
updated CCtxParams API
 2018-03-20 12:36:12 -07:00
Yann Collet
2ed5af0766 merge bmi2 implementation of encodeSequence into zstd_compress.c 2018-03-19 19:10:31 -07:00
Nick Terrell
d19f803a3b Fix window size for 1 worker + flushing 2018-03-19 18:56:39 -07:00
Nick Terrell
24d9edbdd8 Set ldmParams to 0 when disabled 2018-03-19 18:23:54 -07:00
Nick Terrell
4b92574feb Fix corner cases exposed by zstreamtest 2018-03-19 17:54:04 -07:00
Nick Terrell
94c77710a9 Integrate ldm with zstdmt 
Integrate ldm into zstdmt by running it in serial and in order in the first
step of each job, in the same place as the hash gets updated. The input
buffer is sized to fit the whole LDM window and 2 full buffers of slack.
Input buffers cannot be reused until the LDM step is done with them.
After the LDM step is finished, the jobs don't actually have access to the
full window, only the overlap.

Tested on a few different multi-GB files with and without sanitizers,
and with different numbers of threads.
 2018-03-19 16:29:03 -07:00
Yann Collet
9e6ba88363 moved snap to /contrib 2018-03-19 16:15:06 -07:00
Yann Collet
5ad6cc4780
Merge pull request #1015 from elopio/snapcraft 
Add the packaging metadata to build the zstd snap
 2018-03-19 16:14:42 -07:00
Yann Collet
a7b532a347 added docker readme 2018-03-19 16:13:12 -07:00
Yann Collet
b06db3b3c5
Merge pull request #1052 from Varunram/dockerfile 
Add Dockerfile
 2018-03-19 16:07:41 -07:00
Yann Collet
e5ddfaede8 add a test for unordered parameters 2018-03-19 16:02:51 -07:00
Nick Terrell
aa4dbd09a1
Pull job/overlap log logic into common function (#1055) 
Prepares for LDM integration by separating the job size and overlap logic
into helper functions.
 2018-03-19 15:56:36 -07:00
Yann Collet
c8b3d389fd updated CCtxParams API 
to respect naming convention :
ZSTD_CCtxParams_*()
 2018-03-19 15:07:26 -07:00
Yann Collet
6f4d0778a5 make it possible to express compression parameters in any order 2018-03-19 14:41:23 -07:00
Yann Collet
dc6a5471eb
Merge pull request #1056 from terrelln/xxh-parallel 
Move XXH64_update() into worker threads
 2018-03-19 14:13:25 -07:00
Yann Collet
9f6b7b6ef4
Merge pull request #1049 from facebook/contrib 
`/contrib` projects as part of CI tests
 2018-03-19 12:04:27 -07:00
Nick Terrell
2253d01b27 Move XXH64_update() into worker threads 
* Computes the XXH hash in the worker threads.
* Workers get a sequence number and wait until ther number shows up. On
  error, ensures that its sequence is finished, so future threads don't
  get blocked.
* Sets up for ldm integration, which will go in the same spot.
 2018-03-19 11:08:27 -07:00
Yann Collet
9618c0c804 make it possible to specify LDM parameters in any order 2018-03-19 11:07:04 -07:00
Yann Collet
f2562c02e4
Merge pull request #1053 from Varunram/cmakelists 
Add missing checks to CMakeLists
 2018-03-18 12:00:24 -07:00