AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
3,806 Commits 1 Branch 0 Tags 27 MiB
560b34f6d2
Commit Graph

1691 Commits

Author SHA1 Message Date
Yann Collet
40156a4967 bumped version nb to v1.3.1 2017-07-08 04:55:09 -07:00
Yann Collet
0f4fc6c20a fixed several conversion warnings 2017-07-07 17:13:12 -07:00
Yann Collet
9bde061a0b fixed minor Visual compilation limitation 2017-07-07 16:14:17 -07:00
Yann Collet
593d517ebf fixed minor cast warning 2017-07-07 16:09:47 -07:00
Yann Collet
ead4dd48f6 new field frameHeader.headerSize 2017-07-07 15:51:24 -07:00
Yann Collet
46396523c0 ZSTD_getFrameHeader : control of windowSize limits is delegated to caller 
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
 2017-07-07 15:32:12 -07:00
Yann Collet
990449b89d new field : ZSTD_frameHeader.frameType 
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
 2017-07-07 15:21:35 -07:00
Yann Collet
e622330a3b extended frameHeader.windowSize to unsigned long long 2017-07-07 14:19:01 -07:00
Yann Collet
f04deff4fc fixed #718, reported by @GregSlazinski, solution suggested by @mcmilk 2017-07-06 01:42:46 -07:00
Yann Collet
d75c0e71c4 minor code refactoring 2017-07-05 18:10:07 -07:00
Yann Collet
49af41820d clarified status of zstdmt_compress.h API 2017-07-05 17:21:13 -07:00
Yann Collet
27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet
2cb9774f5e more precise estimation of amount to flush at end of stream (single thread mode) 
also : can use DEBUGLEVEL variable in /tests
 2017-07-04 12:39:26 -07:00
Yann Collet
6383372dec fixed : 0-copy in NULL is UB 2017-07-04 10:36:41 -07:00
Yann Collet
5051dd39ca Merge pull request #743 from facebook/fullbench 
compress_generic() automatic optimization opportunities
 2017-07-03 21:26:38 -07:00
Yann Collet
2de2396a36 refactor ZSTDMT_compress() 2017-07-03 16:23:36 -07:00
Yann Collet
2084b041f4 fixed comments 2017-07-03 15:52:19 -07:00
Yann Collet
5a77361595 fixed wrong function name in comment 2017-07-03 15:21:24 -07:00
Nick Terrell
c80fc50a8d [libzstd] Fix memcpy() on potential NULL source 
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
  to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
  the source should not be `NULL`.
 2017-07-03 12:31:55 -07:00
Yann Collet
2485f88bf8 fixed legacy version init bug 2017-07-01 09:09:34 -07:00
cyan4973
21fdf97e00 Merge branch 'dev' into fullbench 2017-07-01 07:01:08 -07:00
cyan4973
1bafe393e4 fix : ZSTDMT_compressStream_generic() can accept NULL input 
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
 2017-07-01 06:59:24 -07:00
Yann Collet
58bd0e70fc fixed : dictionary compression with new advanced API in Multi-threading mode 2017-06-30 16:01:02 -07:00
Yann Collet
d8b33a598d Optimized ZSTDMT single-pass mode speed on large sources 
by ensuring job sizes remain "not too large"
 2017-06-30 15:44:57 -07:00
Yann Collet
d5c046c609 implemented shortcut for zstd_compress_generic() in MT mode 
added ZSTDMT_compress_advanced() API
 2017-06-30 14:51:01 -07:00
Yann Collet
7f40bb1c39 Merge pull request #742 from stellamplau/stack-space 
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
 2017-06-30 14:50:23 -07:00
Stella Lau
32df49e9f8 Fix typo 2017-06-30 12:56:24 -07:00
Stella Lau
b0513b519c Add comment to HUF_DECOMPRESS_WORKSPACE_SIZE 2017-06-30 12:53:56 -07:00
Stella Lau
4c71f59c77 Clarify typedef of rankVal_t and rankValCol_t 2017-06-30 09:52:20 -07:00
Stella Lau
28f711ef95 Rename ALIGN and ALIGN_MASK to HUF_ALIGN and HUF_ALIGN_MASK 2017-06-30 09:38:11 -07:00
Stella Lau
70ad6829e7 Delegate HUF_decompress4X_hufOnly to workspace version 2017-06-29 16:22:32 -07:00
Stella Lau
104c4d57c1 Fix bitshift error 2017-06-29 15:40:49 -07:00
Yann Collet
a3d9926c40 compression optimization opportunity 
switch to single-pass mode directly into output buffer
when outputSize >= ZSTD_compressBound(inputSize).
Speed gains observed with fullbench (~+15% on level 1)
 2017-06-29 14:44:49 -07:00
Stella Lau
fedc94de8c Fix pointer casting warning 2017-06-29 13:04:15 -07:00
Stella Lau
c6a5275a28 Fix alignment warnings with pointer casting 2017-06-29 12:39:34 -07:00
Stella Lau
99e315999c Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2 2017-06-29 11:49:59 -07:00
Yann Collet
97f2bf66da minor : fix typo 2017-06-29 11:31:40 -07:00
Yann Collet
acbef3decd ZSTD_getFrameContentSize() is promoted to "stable" status 2017-06-29 05:19:51 -07:00
Yann Collet
590937df20 Merge pull request #739 from facebook/refPrefix 
ZSTD_refPrefix
 2017-06-29 04:36:03 -07:00
Yann Collet
811deaea6f Merge pull request #736 from terrelln/cover-default-api 
[zdict] Make COVER the default algorithm
 2017-06-28 20:25:36 -07:00
Yann Collet
037466245f refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget() 
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.

Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
 2017-06-28 20:24:08 -07:00
Yann Collet
2bf428df45 Merge branch 'advancedAPI2' into refPrefix 2017-06-28 16:35:49 -07:00
Yann Collet
1ca76039af fixed -Wdeclaration-after-statement 2017-06-28 15:40:21 -07:00
Yann Collet
813535105b added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0] 
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
 2017-06-28 15:34:56 -07:00
Yann Collet
adbe74a8ac adjusted compression levels to guarantee a monotonically increasing memory budget 2017-06-28 13:22:37 -07:00
Yann Collet
33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet
2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
Yann Collet
b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
Yann Collet
7d3816183f exposed ZSTD_MAGIC_DICTIONARY in zstd.h 
makes it easier to explain ZSTD_dictMode
 2017-06-27 13:50:34 -07:00
Yann Collet
fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
Nick Terrell
5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Yann Collet
c7fb884eea fixed minor conversion warning 2017-06-26 18:02:23 -07:00
Yann Collet
dde10b23fe refactored ZSTD_estimateDStreamSize() 
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
 2017-06-26 17:44:26 -07:00
Yann Collet
09ae03a570 ZSTD_estimateCDictSize_advanced() 
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
 2017-06-26 16:47:32 -07:00
Yann Collet
0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00
Yann Collet
31af8290d1 ZSTD_estimateCCtx_advanced() 
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.

ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
 2017-06-26 15:52:39 -07:00
Yann Collet
ef269c1b68 Merge pull request #725 from facebook/advancedAPI2 
New Advanced API
 2017-06-23 09:50:47 -07:00
Yann Collet
ecb0f46866 add controls over streaming buffers 2017-06-21 17:25:01 -07:00
Yann Collet
dce789281b fixed : decompression of skippable frames in streaming mode 2017-06-21 15:53:42 -07:00
Yann Collet
204b6b7ef6 fixed streaming buffered allocation with CDict compression 2017-06-21 15:13:00 -07:00
Yann Collet
1e4129b27b fixed dangling pointer risk, detected by @terrelln 2017-06-21 13:26:10 -07:00
Yann Collet
83095970e6 free cdictLocal faster, suggested by @terrelln 2017-06-21 12:26:40 -07:00
Yann Collet
7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Yann Collet
9c56b12938 Merge pull request #723 from paulcruz74/dev 
Adding zstd -l
 2017-06-21 09:41:55 -07:00
Yann Collet
e51d51bdf7 fixed memcpy() overlap 2017-06-20 17:44:55 -07:00
Yann Collet
466f92eaa6 removed one useless streaming compression stage, detected by @terrelln 2017-06-20 16:25:29 -07:00
Yann Collet
c3bce24ef4 fixed potential dangling pointer, detected by @terrelln 2017-06-20 16:09:11 -07:00
Yann Collet
78b8234554 fixed comments, following suggestion by @terrelln 2017-06-20 14:26:48 -07:00
Yann Collet
b44ab82f7a ensure new ZSTD_strategy starts at value 1 2017-06-20 14:11:49 -07:00
Yann Collet
c08e649e95 first implementation of bench.c with new API ZSTD_compress_generic() 
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.

Also : fixed a long standing bug in ZSTDMT streaming API.
 2017-06-19 18:25:35 -07:00
Yann Collet
695a0a3449 fixed IA64 compilation error, by @mcmilk 2017-06-19 15:27:30 -07:00
Yann Collet
fe234bf48b fix attempts : fullbench for VS2008 2017-06-19 15:23:19 -07:00
Nick Terrell
55f9cd4942 [libzstd] Fix UBSAN failure 2017-06-19 15:12:28 -07:00
Yann Collet
bf99150be3 update new api presentation in zstd.h and manual 2017-06-19 12:56:25 -07:00
Yann Collet
d7a3bffba9 new api : setting compression parameters is refused if a dictionary is already loaded 2017-06-19 11:53:01 -07:00
Yuri
92bafda406 INSTALL_DATA instead of INSTALL_LIB for libzstd.a 
INSTALL_LIB can be passed -s flag to strip symbols. Static libraries should not be stripped, only dynamic ones should be stripped.
 2017-06-17 00:23:41 -07:00
Yann Collet
381e66cfbd added ZSTD_clampCParams() 
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
 2017-06-16 17:34:54 -07:00
Yann Collet
aee916e37c fixed +/-1 error for pledgedSrcSizePlusOne 2017-06-16 17:02:35 -07:00
Yann Collet
d3de3d51a3 fix attempt 2 : Visual sign conversion warning 2017-06-16 16:51:33 -07:00
Yann Collet
944be54774 fixed attempt : minor Visual sign conversion warning 2017-06-16 14:05:01 -07:00
Yann Collet
b26728c9c8 added ZSTD_startNewCompression() 2017-06-16 14:00:46 -07:00
Yann Collet
a0ba849fe6 changed frameContentSize field to pledgedSrcSizePlusOne 
pledgedSrcSize is proper : it's a promise, not yet fulfilled.
It will be controlled at the end.

PlusOne is meant to have 0 (default) == unknown
 2017-06-16 13:29:17 -07:00
Yann Collet
2cf7755da7 fix : pledgedSrcSize correctly reset to unknown in "continue" mode 2017-06-16 12:34:41 -07:00
Yann Collet
9e73f2f320 fix : correctly reset pledgedSrcSize to unknown status 
when starting a new compression with an existing context
 2017-06-16 12:24:01 -07:00
Yann Collet
33873f0e74 fixed : new advanced AIP : setting nbThreads to the same value > 1 2017-06-16 12:04:21 -07:00
Yann Collet
559ee82e90 fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps 2017-06-16 11:58:21 -07:00
Yann Collet
bd18c885a3 added ZSTD_CCtx_reset 2017-06-16 10:17:50 -07:00
Yann Collet
cc9f9b7f4c protection : ZSTD_CONTENTSIZE_UNKNOWN automatically disables contentSizeFlag 2017-06-15 18:17:34 -07:00
Yann Collet
05ae4b2190 added protection : MT incompatible with Static allocation 2017-06-15 18:03:34 -07:00
Paul Cruz
a9b77c83e5 cleaning up code for analyzing frames 2017-06-15 14:13:28 -07:00
Yann Collet
f129fd3970 disabled MT code path when ZSTD_MULTITHREAD is not defined 2017-06-11 18:46:09 -07:00
Yann Collet
23aace9778 added control stage to MT mode 2017-06-11 18:32:36 -07:00
Yann Collet
f35e2de61c linked newAPI to ZSTDMT 2017-06-05 18:32:48 -07:00
cyan4973
c59162e053 minor fix for -Wdocumentation 2017-06-05 00:12:13 -07:00
cyan4973
8bcbf42617 fixed g++ prototype mismatch 2017-06-04 23:52:00 -07:00
Yann Collet
8c910d2097 updated ZSTDMT streaming API 
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
 2017-06-03 01:15:02 -07:00
Yann Collet
58e8d793e1 made debug definitions common within zstd_internal.h 2017-06-02 18:20:48 -07:00
Yann Collet
8ddf4c22d5 fixed missing initialization 2017-06-02 17:16:49 -07:00
Yann Collet
33a7e679e5 significant zlib wrapper code refactoring 
code indentation
variable scope and names
constify

Only coding style changes.
The logic should remain the same.
 2017-06-02 17:10:49 -07:00
Yann Collet
4effccbf56 zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing 
more generic and safer than using own routing for magic number comparison
 2017-06-02 14:27:11 -07:00