AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
2,544 Commits 1 Branch 0 Tags 27 MiB
bbfcc24309
Commit Graph

1259 Commits

Author SHA1 Message Date
Yann Collet
107bcbbbc2 zstdmt : changed internal naming from frame to chunk 
Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.

minor : ensures that content size is written in header
 2017-01-12 01:25:46 +01:00
Yann Collet
5eb749e734 ZSTDMT_compress() creates a single frame 
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
 2017-01-11 18:21:25 +01:00
Yann Collet
04cbc36499 minor refactor (release CCtx 1st) and comment clarification 2017-01-11 16:08:08 +01:00
Yann Collet
085179bb78 fixed ZSTDMT_createCCtx() : checked inner objects are properly created 2017-01-11 15:58:05 +01:00
Yann Collet
8ce1cc2bec improved ZSTD_createCCtxPool() cancellation 
use ZSTD_freeCCtxPool() to release the partially created pool.
avoids to duplicate logic.

Also : identified a new difficult corner case :
when freeing the Pool, all CCtx should be previously released back to the pool.
Otherwise, it means some CCtx are still in use.
There is currently no clear policy on what to do in such a case.
Note : it's supposed to never happen.
Since pool creation/usage is static, it has no external user,
which limits risks.
 2017-01-11 15:44:26 +01:00
Yann Collet
47557ba2b2 fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails 2017-01-11 15:35:56 +01:00
Nick Terrell
8d984699db Document memory requirements for COVER algorithm 2017-01-09 18:20:10 -08:00
Nick Terrell
555e281637 Handle large input size in 32-bit mode correctly 2017-01-09 18:20:06 -08:00
Nick Terrell
3a1fefcf00 Simplify COVER parameters 2017-01-02 17:51:38 -08:00
Nick Terrell
96b39f65fa Add COVER dictionary builder 2017-01-02 13:22:51 -08:00
Yann Collet
6334b04d61 compile object files, for faster recompilation 2017-01-02 03:22:18 +01:00
Yann Collet
f1cb55192c fixed linux warnings 2017-01-02 01:11:55 +01:00
Yann Collet
0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet
2ec635a162 use pthread_cond to send signals between threads 2017-01-01 17:31:33 +01:00
Nick Terrell
bb13387d7d Fix pool for threading.h 2016-12-31 19:10:47 -05:00
Nick Terrell
4204e03e77 Add threading.h condition variables 2016-12-31 19:10:29 -05:00
Yann Collet
3b9d434356 extended ZSTDMT code support for non-MT systems and WIN32 (preliminary) 2016-12-31 16:32:19 +01:00
Yann Collet
c8efc1c874 simplified Buffer Pool 2016-12-31 14:45:33 +01:00
Yann Collet
3b29dbd9e8 new zstdmt version using generic treadpool 2016-12-31 06:04:25 +01:00
Yann Collet
c6a6417458 bench correctly measures time for multi-threaded compression (posix only) 2016-12-31 03:31:26 +01:00
Yann Collet
f765a375a5 Merge pull request #504 from terrelln/thread-pool 
[zstdmt] Add thread pool
 2016-12-30 15:31:49 +01:00
Nick Terrell
e777a5be6b Add a thread pool for ZSTDMT and COVER 2016-12-29 23:39:44 -08:00
Yann Collet
e70912c72b Changed : input divided into roughly equal parts. 
Debug : can measure time waiting for mutexes to unlock.
 2016-12-29 01:24:01 +01:00
Yann Collet
6c0ed9483a compression threads use ZSTD_compressCCtx() 2016-12-28 17:08:28 +01:00
Yann Collet
8d7432914f Merge pull request #503 from inikep/dev11 
Dev11
 2016-12-28 16:50:39 +01:00
Yann Collet
ce9e1452fd protect buffer pool with a mutex 2016-12-28 15:31:19 +01:00
Przemyslaw Skibinski
75f3a3a335 changed default PREFIX and MANDIR 2016-12-28 12:32:41 +01:00
Yann Collet
3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00
Yann Collet
39c105c605 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-23 22:25:31 +01:00
Yann Collet
aca113f4f5 fixed ZSTD_sizeof_?Dict() 2016-12-23 22:25:03 +01:00
Yann Collet
c07d2e3a31 Merge pull request #499 from inikep/dev11 
improved *BSD and Solaris compatibility
 2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski
63b0014b96 BSD: improved "make install" 2016-12-23 10:05:49 +01:00
Nick Terrell
1b5d4a7d53 ZDICT_finalizeDictionary() flipped comparison 2016-12-22 18:14:57 -08:00
Nick Terrell
bcbe77e994 ZDICT_finalizeDictionary() flipped comparison 
`ZDICT_finalizeDictionary()` had a flipped comparison.
I also allowed `dictBufferCapacity == dictContentSize`.
It might be the case that the user wants to fill the dictionary
completely up, and then let zstd take exactly the space it needs
for the entropy tables.
 2016-12-22 18:01:14 -08:00
Nick Terrell
78a0072d5a Fix failing test due to deprecation warning 2016-12-22 17:36:16 -08:00
Yann Collet
d76d1a9ef0 added ZDICT_finalizeDictionary() 2016-12-22 20:18:43 +01:00
Przemyslaw Skibinski
b999170311 Solaris: working "make -C lib install" 2016-12-22 20:14:37 +01:00
Yann Collet
ba75e9d8c3 fix : zlib wrapper compile in gnu90 mode 2016-12-21 19:57:18 +01:00
Yann Collet
0819abe3c1 added ZSTD_createDDict_byReference() body 2016-12-21 19:25:15 +01:00
Yann Collet
4e5eea61a8 added ZSTD_createDDict_byReference() 2016-12-21 16:44:35 +01:00
Yann Collet
8333106b8a Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:44:24 +01:00
Yann Collet
0d7e84899f Merge pull request #489 from inikep/v112 
improved detection of POSIX
 2016-12-21 16:42:46 +01:00
Yann Collet
1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski
2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski
5736db219e fix basic types redefinition 2016-12-21 09:26:00 +01:00
Nick Terrell
8157a4c3cc Fix dictionary loading bug causing an MSAN failure 
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
 2016-12-20 10:47:52 -08:00
Przemyslaw Skibinski
f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112 
# Conflicts:
#	appveyor.yml
 2016-12-19 08:20:26 +01:00
Yann Collet
d564faa3c6 fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header 2016-12-18 21:39:15 +01:00
Yann Collet
1496c3dc47 Fix : size estimation when some samples are very large 2016-12-18 11:58:23 +01:00
Yann Collet
d46ecb58a5 added dll compilation tests 2016-12-17 16:28:12 +01:00
Nick Terrell
8de46ab51a Export all API functions 2016-12-16 13:27:30 -08:00
Przemyslaw Skibinski
0a1caeef6d VS: fixed 32-bit DLL compilation 2016-12-15 12:12:46 +01:00
Przemyslaw Skibinski
4e10bd339d appveyor.yml: added tests of fullbench-dll fullbench-lib 2016-12-15 12:09:23 +01:00
Przemyslaw Skibinski
60f10aab6c introduced ZSTDLIB_VISIBILITY 2016-12-15 11:32:31 +01:00
Yann Collet
2b36b238d3 changed variable name to estimatedSrcSize, to emphasize it does not need to be exact 2016-12-13 17:59:55 +01:00
Yann Collet
e795c8a5f6 Added ZSTD_initCStream_srcSize(). 
Added relevant test cases in zstreamtest
 2016-12-13 17:00:14 +01:00
Yann Collet
5397a66b19 minor BMI version check 2016-12-13 15:21:06 +01:00
Yann Collet
35168679bd Merge pull request #478 from terrelln/wildcopy-ub 
Fix execSequence wildcopy undefined behavior
 2016-12-13 11:33:00 +01:00
Nick Terrell
064a143520 Fix execSequence wildcopy undefined behavior 
execSequence relied on pointer overflow to handle cases where
`sequence.matchLength < 8`.  Instead of passing an `size_t` to
wildcopy, pass a `ptrdiff_t`.
 2016-12-12 19:01:23 -08:00
Nick Terrell
e474aa55b4 Fix decompression buffer overrun 
Allows an adversary to write up to 3 bytes beyond the end of the buffer.
Occurs if the match overlaps the `extDict` and `currentPrefix`, and the
match length in the `currentPrefix` is less than `MINMATCH`, and
`op-(16-MINMATCH) >= oMatchEnd > op-16`.
 2016-12-12 18:05:30 -08:00
Yann Collet
c3a5c4bef8 introduced cycleLog 2016-12-12 00:47:30 +01:00
Yann Collet
c261f71f6a minor variation of rescale fix 2016-12-12 00:25:07 +01:00
Nick Terrell
3826207a70 Simplify segfault fix 
Take advantage of the fact that `chainLog <= windowLog`.
 2016-12-10 18:46:55 -08:00
Nick Terrell
0012332ce0 Fix compression segfault 
When the overflow protection kicks in, it makes sure that ip - ctx->base
isn't too large.  However, it didn't ensure that saved offsets are
still valid.  This change ensures that any valid offsets (<= windowLog)
are still representable after the update.

The bug would shop up on line 1056, when `offset_1 > current + 1`, which
causes an underflow.  This in turn, would cause a segfault on line 1063.

The input must necessarily be longer than 1 GB for this issue to occur.
Even then, it only occurs if one of the last 3 matches is larger than
the chain size and block size.
 2016-12-09 17:15:33 -08:00
Yann Collet
383b8088a3 minor lib build refactoring 2016-12-08 18:42:27 -08:00
Yann Collet
6e754fe76a fixed lib soname. 
example : simple_compression : size overflow check
 2016-12-08 18:26:56 -08:00
Przemyslaw Skibinski
7687913178 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2016-12-08 10:42:42 +01:00
Yann Collet
426a9d4b71 changed : dll : only approved ZSTD symbols are now exposed. All other symbols remain internal. 2016-12-07 16:39:34 -08:00
Przemyslaw Skibinski
4da53219a0 zstd Manual updated to 1.1.2 2016-12-07 11:18:40 +01:00
Yann Collet
379908be3d fixed zstd.h for manual 2016-12-06 10:36:15 -08:00
Yann Collet
9dfd0af164 ZBUFF_ as a wrapper to ZSTD streaming API. 2016-12-06 17:16:41 +01:00
Yann Collet
825dffbc43 moved zbuff source files into lib/deprecated 2016-12-05 19:28:19 -08:00
Yann Collet
8f8e2b0b4a fixed initialization warning 2016-12-05 18:00:50 -08:00
Yann Collet
e7a41a5955 added : dictID retrieval functions. 
added : unit tests for dictID retrieval functions
 2016-12-05 16:21:06 -08:00
Yann Collet
9ffbeea875 API : changed : streaming decompression : implicit reset on starting new frames 2016-12-02 18:37:38 -08:00
Yann Collet
2238312c2f fix dict loading 2016-12-02 11:36:11 -08:00
Przemyslaw Skibinski
821bf1febc fixed Doxygen trailing comment 2016-12-02 16:13:41 +01:00
Yann Collet
b89af20353 reduced table sizes for HUF_readDTableX4 2016-12-01 18:24:59 -08:00
Yann Collet
a0d742b1e4 introduced HUF_buildCTable_wksp(), to reduce stack memory usage 2016-12-01 17:47:30 -08:00
Yann Collet
643d9a234b replaced usage of FSE_buildCTable by FSE_buildCTable_wksp, using less stack space in the process 2016-12-01 16:24:04 -08:00
Yann Collet
e928f7e16d introduced ext_wksp variants of count to reduce stack memory usage 2016-12-01 16:13:35 -08:00
Yann Collet
979cab412b fixed some minor visual silent cast warnings. 
introduced FSE_count_parallel_wksp().
 2016-11-30 18:10:38 -08:00
Yann Collet
5e00b848a8 FSE_compress_wksp() uses less stack space 2016-11-30 16:46:13 -08:00
Yann Collet
d79a9a00d9 Introduced FSE_compress_wksp() and FSE_buildCTable_wksp() to reduce stack memory usage 2016-11-30 15:52:20 -08:00
Yann Collet
766431909f introduced FSE_decompress_wksp(), to use less stack space 2016-11-30 12:36:45 -08:00
Yann Collet
95eb43be09 updated pkg config file 2016-11-30 11:06:58 -08:00
Yann Collet
42247705a3 Merge pull request #461 from obache/neatsrc/pkgconfig 
libzstd.pc.in: Change to use variables for libdir and includedir
 2016-11-30 20:03:42 +01:00
Yann Collet
ff504de391 minor decompression speed improvement 2016-11-29 17:42:46 -08:00
Yann Collet
25f46dcc0f minor const 2016-11-29 16:59:27 -08:00
Yann Collet
a56ac2815c restored normal decoder speed 2016-11-29 15:30:23 -08:00
Yann Collet
37870d7a66 fixed minor visual warning 2016-11-29 14:31:57 -08:00
Yann Collet
013a2b58ae Merge pull request #464 from terrelln/guards 
Fix ZSTD_STATIC_LINKING_ONLY with double include
 2016-11-29 23:06:10 +01:00
Yann Collet
167c494748 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-11-29 14:05:15 -08:00
Yann Collet
4f5350f610 long matches support overflow 2016-11-29 13:12:24 -08:00
Nick Terrell
05c00f2ff7 Fix ZSTD_STATIC_LINKING_ONLY with double include 2016-11-29 11:54:39 -08:00
OBATA Akio
c53eea7c1a libzstd.pc.in: Change to use variables for libdir and includedir 2016-11-29 16:47:53 +09:00
Yann Collet
52e136ed3d long decoder compatible with round and separate buffers 2016-11-28 19:59:11 -08:00
Yann Collet
ce3527ca0c combined normal and long decoder 2016-11-28 18:38:52 -08:00
Yann Collet
8993bee997 restored normal mode 2016-11-28 16:11:30 -08:00
Yann Collet
764e70a4f3 added decodeSequencesLong 2016-11-28 15:50:16 -08:00