AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
3,147 Commits 1 Branch 0 Tags 27 MiB
af4f45b682
Commit Graph

1408 Commits

Author SHA1 Message Date
Yann Collet
af4f45b682 Improved code comments for block functions 2017-04-18 03:17:44 -07:00
Yann Collet
4f818182b8 clarified frame parameters for ZSTD_compress*_usingCDict() 
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
 2017-04-17 18:29:06 -07:00
Yann Collet
c47c68f6ca proper evaluation of Huffman CTable size 2017-04-17 16:14:21 -07:00
Yann Collet
88009a8ba2 removed srcSize control from CStream 
since it's already done from lower bufferless API level
 2017-04-12 00:51:24 -07:00
Yann Collet
20d5e03893 content size is controlled at bufferless level 
so it's active for all entry points

Also : added relevant test (wrong content size) in fuzzer
 2017-04-11 18:34:02 -07:00
Yann Collet
4ee6b15dac force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict() 
because by definition srcSize is not known when using this prototype.
added relevant test

Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
 2017-04-11 11:59:44 -07:00
Yann Collet
ab9162ebb4 simplified call graph 
by calling ZSTD_compressBegin_internal() instead of ZSTD_compressBegin_advanced()
 2017-04-11 10:46:20 -07:00
Yann Collet
e88034fe26 simplified ZSTD_initCStream*() flow 
all variants converge towards ZSTD_initCStream_stage2()
 2017-04-10 22:24:02 -07:00
Yann Collet
4b987ad8ce Introduce ZSTD_initCStream_internal() 
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
 2017-04-10 17:50:44 -07:00
Yann Collet
0181fef545 ensure cctx internal buffer is correctly sized in case of memory error 2017-04-06 01:25:26 -07:00
Yann Collet
36c2a03757 updated comments for ZSTD_resetCStream() 2017-04-05 22:06:21 -07:00
Yann Collet
003a244324 DStream : ensure correct size of internal buffers in case of error 2017-04-05 15:28:56 -07:00
Yann Collet
02d37aa1c1 ensure correct size of internal buffers in case of error 2017-04-05 14:53:51 -07:00
Nick Terrell
405d2a1027 Explicitly convert scratchBuffer to unsigned* 2017-04-04 16:35:31 -07:00
Nick Terrell
16a739cab0 Switch call of FSE_count() to FSE_count_wksp() 2017-04-04 16:17:21 -07:00
Yann Collet
7cf78f1be7 Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference 
Will issue an error (GENERIC) is cdict==NULL
 2017-04-04 12:38:14 -07:00
Nick Terrell
26b046a7c4 Remove unnecessary dictID store 2017-04-03 21:46:28 -07:00
Nick Terrell
39a6cc5172 Make ZSTD_compress_usingCDict() respect contentSizeFlag 2017-04-03 21:09:55 -07:00
Nick Terrell
62ecad3819 Fix ZSTD_initCStream_usingCDict() to use dictionary 2017-04-03 21:05:59 -07:00
Yann Collet
30c7698970 optimize ZSTDMT_compress() memory usage 
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
 2017-03-31 18:27:03 -07:00
Yann Collet
3f75d52527 Changed ZSTD_compressBound() 
required so that if Total = A+B
compressBound(Total) <= compressBound(A) + compressBound(B)
under condition of a minimum size for A and B

Will help for ZSTDMT_compress() memory allocation
 2017-03-31 17:11:38 -07:00
Yann Collet
7b70a1969e Merge branch 'dev' into zstdmt 2017-03-31 16:22:33 -07:00
Yann Collet
53203e7c38 Merge pull request #640 from facebook/memAccess 
Changed memory strategy to __packed for gcc
 2017-03-31 15:49:12 -07:00
Yann Collet
eea7858e2b fixed minor warnings in debug code 2017-03-30 16:47:19 -07:00
Yann Collet
34cc487d05 overlap at full windowSize for max compression level 
as it provides max compression ratio
 2017-03-30 16:23:22 -07:00
Yann Collet
458e955c23 improved ZSTDMT_compress() 
Use a bit more threads by default.
Uses overlap segments to boost compression ratio (like the streaming variant)
 2017-03-30 15:51:58 -07:00
Yann Collet
6476c51b86 Merge pull request #637 from facebook/zstdmt 
Zstdmt
 2017-03-30 14:18:37 -07:00
Yann Collet
274f59919d Changed memory strategy to __packed for gcc 
Method 1 __packed is always as good or better than memcpy().
But it's not portable, as it depends on compiler extension.

For gcc, __pakced directive works fine.
Furthermore, gcc has serious performance issues with memcpy() on ARM 32 bits.
See #620
 2017-03-30 12:52:14 -07:00
Nick Terrell
5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet
ca5a8bbe36 re-added patch ... 2017-03-29 17:15:27 -07:00
Yann Collet
2e2e78de47 removed unnecessary restriction on minmatchLength 
it's now transparently translated to nearest value when unsupported
(7->6) (3->4)
 2017-03-29 16:02:47 -07:00
Yann Collet
26769d88bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-29 15:21:30 -07:00
Yann Collet
933ce4a1dd fix : minmatch 7 conversion 
minmatch 7 now converted to minmatch 6 for strategies which do not support 7
Used to folded into "default", which applied minmatch 4
 2017-03-29 14:35:38 -07:00
Sean Purcell
4708394bdd Remove extra 'F' from skippable magic mask 2017-03-29 11:46:57 -07:00
Yann Collet
4cf0093571 restored bonus rule 2017-03-26 14:51:00 -07:00
Yann Collet
69017bf253 Merge branch 'dev' into LegacyDictBuilder 2017-03-26 14:39:13 -07:00
Yann Collet
582760818f minor refactor 
add const
changed if for easier to add new conditions
 2017-03-26 03:04:56 -07:00
Yann Collet
858f72eeb8 fixed dictBuilder issue 
dictionary loading would fail during entropy analysis
 2017-03-26 02:50:00 -07:00
Yann Collet
ecee9f2ef8 fixed conversion warnings 2017-03-26 00:59:14 -07:00
Yann Collet
0246d5c531 Merge pull request #630 from facebook/advancedCliCommands 
changed advanced commands --maxdict= and --dictID=
 2017-03-26 00:13:35 -07:00
Yann Collet
4c41d37fcc changed test for new syntax 
--dictID= and --maxdict=
 2017-03-24 18:36:56 -07:00
Yann Collet
d41f707e88 minor improvement : remove duplicates with 1 char prefix difference 2017-03-24 17:56:45 -07:00
Yann Collet
b364caf455 Merge pull request #628 from facebook/dictBuilder_limits 
Ensure all limits derived from same constants
 2017-03-24 17:54:42 -07:00
Yann Collet
2238870eb6 Merge pull request #625 from facebook/loadCDict 
limited CDict acceptation criteria to be the same as DDict
 2017-03-24 16:06:20 -07:00
Yann Collet
96aa3019b2 changed advanced commands --maxdict= and --dictID= 
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
 2017-03-24 16:04:29 -07:00
Yann Collet
9da3b215ec Ensure all limits derived from same constants 
Now uses ZDICT_DICTSIZE_MIN and ZDICT_CONTENTSIZE_MIN
from zdict.h.

Also : reduced values to 256 and 128 respectively
 2017-03-24 15:02:09 -07:00
Yann Collet
ebe9963cf6 Merge pull request #626 from facebook/stricterDictBuilder 
dictBuilder fails to create dictionary on certain input
 2017-03-24 14:27:28 -07:00
Yann Collet
16a0b10781 fixed ZSTD_loadZstdDictionary() 
forgot to add the dictionary content
(tests were not failing, just compressing less).

Also : added size protections when adding dict content
since hc/bt table filling would fail if size < 8
 2017-03-24 12:46:46 -07:00
Yann Collet
23776ce290 fixed ERROR_GENERIC on dstSize_tooSmall 
required by users which depends on this error code to size dest buffer
 2017-03-23 17:59:50 -07:00
Yann Collet
f332ece468 dictBuilder fails to create dictionary on certain input 
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
 2017-03-23 16:24:02 -07:00