AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,243 Commits 1 Branch 0 Tags 27 MiB
f1571dad8f
Commit Graph

1809 Commits

Author SHA1 Message Date
Yann Collet
d6ddb879da Merge pull request #817 from terrelln/pool-custom-alloc 
[pool] Accept custom allocators
 2017-08-29 13:05:39 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Nick Terrell
9822f97721 [error] Don't guard undef X with ifdef X 2017-08-29 11:54:38 -07:00
Stella Lau
b5b9275e67 Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced() 2017-08-29 10:49:29 -07:00
Stella Lau
0e56a84a1e Fix getting cParams from CCtxParams 2017-08-28 19:25:17 -07:00
Nick Terrell
02033be08c [pool] Visual Studios disallows empty structs 2017-08-28 17:19:01 -07:00
Nick Terrell
7c365eb02c [threading] Fix ERROR macro after including windows.h 2017-08-28 16:25:02 -07:00
Bernhard M. Wiedemann
cf689b84f9 Sort input file list 
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
 2017-08-26 17:08:00 +02:00
Stella Lau
024098a47d Fix parameter retrieval from cdict 2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8 Fix typo with ZSTDMT_parameter 2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff Remove ZSTD_setCCtxParameter() 2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38 Add function initializing cctxParams from clevel 2017-08-25 13:36:47 -07:00
Stella Lau
9911153723 Move jobSize and overlapLog in zstdmt to cctxParams 2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d Distinguish between jobParams and cctxParams in zstdmt 2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a Remove ZSTD_p_refDictContent and dictContentByRef 2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df [zstdmt] Use POOL_create_advanced() 2017-08-24 18:12:28 -07:00
Nick Terrell
de6c6bce85 Fix zstd_internal.h for C++ mode 2017-08-24 18:09:50 -07:00
Nick Terrell
26dc040a7b [pool] Accept custom allocators 2017-08-24 17:01:41 -07:00
Nick Terrell
89dc856cae [pool] Fix formatting 2017-08-24 16:48:32 -07:00
Stella Lau
15fdeb9e41 Enforce nbThreads<=1 for estimateCCtxSize 2017-08-24 16:28:49 -07:00
Nick Terrell
376f435914 [dictBuilder] Set default compression level to 3 2017-08-24 16:21:05 -07:00
Stella Lau
2fbf0285b2 Fix interaction with ZSTD_setCCtxParameter() and cleanup 2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516 Fix forceWindow and dictMode setting for zstdmt jobs 2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50 Ensure zstdmt uses 'job version' of cctx parameters 2017-08-23 17:03:31 -07:00
Stella Lau
1c81f725ff Remove duplicated testing code 2017-08-23 15:47:15 -07:00
Stella Lau
64ce49426b Fix cstream compression level 2017-08-23 12:30:47 -07:00
Stella Lau
5bc2c1e982 Add prototype support for customMem with cctxParams 2017-08-23 12:03:30 -07:00
Yann Collet
e9ce1208a1 Merge pull request #812 from facebook/longRangeFix 
fixed extraordinary scenario where all fields use maximum nbBits
 2017-08-23 11:35:28 -07:00
Yann Collet
74cde5a4d8 Merge pull request #813 from stellamplau/highbit32fix 
Fix undefined behavior when srcSize==1
 2017-08-23 11:31:06 -07:00
Stella Lau
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Dmitriy Titarenko
20f715d709 Fix displayLevel overflow 2017-08-23 15:56:15 +05:00
Stella Lau
11303778d0 Add function to make cctxParams from ZSTD_parameters 2017-08-22 14:53:13 -07:00
Yann Collet
bd9c8ca146 Merge pull request #811 from terrelln/segmentSize 
[cover] Fix end condition for small dictionary
 2017-08-22 14:36:30 -07:00
Stella Lau
23fc0e41fa Remove 'opaque' naming from internal functions 2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776 Remove unused functions 2017-08-22 13:33:58 -07:00
Yann Collet
6b2b6a9bd5 fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously 
can only happen if windowLog>=27  (level 22 --ultra)
 2017-08-22 12:09:21 -07:00
Stella Lau
e50ed1fa3a Fix undefined behavior when srcSize==1 2017-08-22 11:55:42 -07:00
Stella Lau
60e1bc617c Explicitly create a job cctxParam for multithreading 2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753 Comment out CCtx_param versions of CDict functions 2017-08-21 14:49:16 -07:00
Nick Terrell
29c2d9a4d0 [cover] Turn down notification for ZDICT subroutines 2017-08-21 14:28:31 -07:00
Nick Terrell
98de3f6847 [cover] Add dictionary size to compressed size 2017-08-21 14:23:17 -07:00
Yann Collet
78c3d16bf4 Merge pull request #809 from terrelln/dev 
[cover] Fix divide by zero
 2017-08-21 13:33:19 -07:00
Nick Terrell
9a54a315aa [cover] Convert score to U32 and check for zero 2017-08-21 13:30:07 -07:00
Stella Lau
fd8a25786e Check parameters are valid in initCCtxParams 2017-08-21 13:23:35 -07:00
Stella Lau
1c0dbe81b1 Add documentation for CCtx_params 2017-08-21 13:18:00 -07:00
Nick Terrell
d49eb40c03 [cover] Stop when segmentSize is less than d 2017-08-21 13:10:03 -07:00
Stella Lau
939f954285 Pass ZSTD_CCtx_params as const ptr when possible 2017-08-21 12:57:18 -07:00
Stella Lau
73c73bf16a Reduce code duplication in zstreamtest 2017-08-21 12:41:19 -07:00
Stella Lau
560b34f6d2 Return error code when initializing NULL cctxParams 2017-08-21 11:52:26 -07:00
Stella Lau
25be09c6b4 Set some parameters to zero before initializing cdict 2017-08-21 11:35:46 -07:00
Yann Collet
232d62b637 fixed a few headers that were too hastily copy/pasted during last license change 2017-08-21 11:24:32 -07:00
Nick Terrell
f306d400c0 [cover] Fix divide by zero 2017-08-21 11:12:11 -07:00
Stella Lau
502031ca10 Use cctxParam version of createCDict internally 2017-08-21 11:00:44 -07:00
Stella Lau
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
Yann Collet
7db552676e reduced pool queue to 0 to save memory 
fixed : pool performance when jobs are fires fast and queueSize==0
 2017-08-19 15:07:54 -07:00
Stella Lau
6cee6e07e5 Add internal createCDict function 2017-08-18 22:48:31 -07:00
Stella Lau
d775519296 Add cctxParam versions of internal functions 2017-08-18 17:37:58 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Stella Lau
63b8c98531 Pass cctx parameters to MTCtx 2017-08-18 16:17:24 -07:00
Stella Lau
399ae013d4 Add function to apply cctx params 2017-08-18 13:01:55 -07:00
Stella Lau
81d89d82a6 Move nbThreads to cctx params 2017-08-18 12:08:57 -07:00
Stella Lau
2300c58a6f Move dictContentByRef to cctx params 2017-08-18 12:03:16 -07:00
Stella Lau
b6cb2ed8cb Move dictMode to cctxParams 2017-08-18 11:43:31 -07:00
Stella Lau
97e27affcb Move compression level to cctx params 2017-08-18 11:20:08 -07:00
Stella Lau
c0221124d5 Add function to set opaque parameters 2017-08-17 19:30:22 -07:00
Stella Lau
4169f49171 Add initialization/allocation functions for opaque params 2017-08-17 18:45:04 -07:00
Stella Lau
ade95b8bed Add opaque interfaces for static initialization 2017-08-17 18:13:08 -07:00
Stella Lau
699f11b4f7 Create opaque parameter structure 2017-08-17 17:33:46 -07:00
Yann Collet
f9e6590715 Merge pull request #796 from terrelln/is-error 
[FSE][HUF] Inline error checks
 2017-08-15 12:37:28 -07:00
Yann Collet
2dbcfc6994 Merge pull request #794 from terrelln/force-inline 
[libzstd] Fix FORCE_INLINE macro
 2017-08-15 12:03:44 -07:00
Nick Terrell
07c6ff588e [FSE][HUF] Inline error checks 
Caught by Clang's optimization remarks.
 2017-08-15 11:23:28 -07:00
Nick Terrell
565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Roman Gershman
b9d4f4fb74 Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged 2017-08-13 13:29:42 +03:00
Nick Terrell
9ba97182d1 [CI] Add gcc7build test 2017-08-08 13:28:56 -07:00
Yann Collet
d9f2893eb9 Merge pull request #782 from terrelln/dstSizeTooSmall 
Fix compression failure on incompressible data
 2017-08-07 14:52:02 -07:00
Yann Collet
8049556928 Merge pull request #778 from terrelln/bad-huff 
[libzstd] Fix bug in Huffman decompresser
 2017-08-07 14:05:58 -07:00
Nick Terrell
abe12b3399 [libzstd] Fix bug in Huffman decompresser 
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
 2017-08-07 12:37:48 -07:00
Nick Terrell
308047eb5d Fix compression failure on incompressible data 
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.

Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
 2017-08-07 11:45:24 -07:00
Stella Lau
73ba58955f Signal after finishing job when queueSize=0 2017-08-01 20:12:06 -07:00
Stella Lau
1d76da1d87 Replace marker with queueEmpty variable and update pool.h comment 2017-08-01 12:30:16 -07:00
Stella Lau
5adceeed01 Allow queueSize=0 in pool.c and update poolTests 2017-07-31 10:10:16 -07:00
Yann Collet
e1222544be Merge pull request #753 from paulcruz74/adapt-approach-3 
adaptive compression v1
 2017-07-27 10:00:10 -07:00
Nick Terrell
ae20d413da [libzstd] Fix CHECK_V_F macros 2017-07-25 12:52:01 -07:00
Yann Collet
a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet
b4d460f32c pool.c : blindfix for Visual warnings 2017-07-20 01:13:14 -07:00
Yann Collet
3974d2b38a blind fix for Windows Multithreading module 
adds a fake 0 return value for mutex/cond init
 2017-07-19 13:33:21 -07:00
Paul Cruz
6945b3c43d removed previous version of completion for compression 2017-07-19 11:51:50 -07:00
Yann Collet
b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Nick Terrell
cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Yann Collet
77d67fb167 Merge pull request #766 from terrelln/real-block-split 
[libzstd] Pull optimal parser state out of seqStore_t
 2017-07-18 08:26:24 -07:00
Yann Collet
14c83b05c7 Merge pull request #765 from terrelln/real-block-split 
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
 2017-07-17 19:25:55 -07:00
Nick Terrell
7a28b9e4a3 [libzstd] Pull optimal parser state out of seqStore_t 2017-07-17 15:29:11 -07:00
Yann Collet
3381bf4b84 Merge pull request #764 from terrelln/real-block-split 
[libzstd] Refactor ZSTD_compressSequences()
 2017-07-17 14:46:01 -07:00
Nick Terrell
e198230645 [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences() 2017-07-17 12:27:24 -07:00
Nick Terrell
634f012420 [libzstd] Refactor ZSTD_compressSequences() 2017-07-17 11:36:11 -07:00
Paul Cruz
50ce4eaeb6 added error detection for pthread initialization, added compression completion measurement, fixed const values 2017-07-17 10:12:44 -07:00
Yann Collet
3b0cff3c33 fixed clang's -Wdocumentation 2017-07-13 18:58:30 -07:00
Yann Collet
2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00