AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,405 Commits 1 Branch 0 Tags 27 MiB
e649fad7aa
Commit Graph

1189 Commits

Author SHA1 Message Date
Yann Collet
95b152ab33 updated clevel table for 16K 
to introduce btultra2
 2018-12-07 20:12:43 -08:00
Yann Collet
d613fd9afe linked btultra2 as strategy9 
and ensure zstdbench detects out-of-bound parameters
 2018-12-06 19:27:37 -08:00
Yann Collet
ae370b0e12 minor bound refinements 2018-12-06 16:51:17 -08:00
Yann Collet
39e28982cf introduced constants ZSTD_STRATEGY_MIN and ZSTD_STRATEGY_MAX 2018-12-06 16:16:16 -08:00
Yann Collet
c3c3488981 fixed c++ assignment to enum 2018-12-06 15:57:55 -08:00
Yann Collet
be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 
also : fixed paramgrill, and limit conditions
 2018-12-06 15:00:52 -08:00
Yann Collet
e9448cdf4c introduced strategy btultra2 
note : not yet applied on any compression level
 2018-12-06 13:38:09 -08:00
Yann Collet
3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_* 
for naming consistency
 2018-12-05 17:26:02 -08:00
Yann Collet
aec945f0dc implemented ZSTD_dParam_getBounds() 
and ZSTD_DCtx_setParameter()
 2018-12-04 15:35:37 -08:00
Yann Collet
6ced8f7c7c joined normal streaming API with advanced one 2018-12-03 14:22:38 -08:00
Yann Collet
d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2() 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
 2018-11-30 11:25:56 -08:00
Yann Collet
d4d4e109e9 getParameter fills an int* 
rather than an unsigned*
for consistency
since type of setParameter() changed to int.
 2018-11-21 15:37:26 -08:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
5d3592398d fixed fall-through 2018-11-20 16:09:33 -08:00
Yann Collet
5c6d4b18ac completed implementation of ZSTD_cParam_getBounds() 
for all parameters
 2018-11-20 16:06:00 -08:00
Yann Collet
2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet
114bd4346e changed enum type name to ZSTD_ResetDirective 
for naming consistency :
types should start with a capital letter (after prefix)
 2018-11-20 12:00:20 -08:00
Yann Collet
3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Yann Collet
5c68639186 updated ZSTD_DCtx_reset() 
signature and behavior is now the same as ZSTD_CCtx_reset()
 2018-11-15 16:12:39 -08:00
Yann Collet
06c8d5a4f4 Merge branch 'dev' into advancedAPI 
fixed rsyncable
 2018-11-15 10:51:24 -08:00
Nick Terrell
b9693d3a49 [lib] Add rsyncable mode 
- Add rsyncable mode to multithreaded mode
- Factor out LDM's hash function for reuse
 2018-11-14 16:59:57 -08:00
Yann Collet
7b0391e37e finalized retrofit of ZSTD_CCtx_reset() 
updated all depending sources
 2018-11-14 13:05:35 -08:00
Yann Collet
ff8d371708 modified ZSTD_CCtx_reset() 
which now accepts an enum,
to distinguish between resetting the session, or the parameters (or both).

removed ZSTD_CCtx_resetParameters(), which is redundant.

start replacing invocation of ZSTD_CCtx_reset*() functions

Updated advanced API documentation

trimmed down amount of API staged in RC,
in particular, all functions related to ZSTD_CCtxParams()
seem too advanced.
 2018-11-14 12:33:57 -08:00
Yann Collet
d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT 
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
 2018-11-13 18:12:34 -08:00
Yann Collet
b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet
f28af025d9
Merge pull request #1413 from felixhandte/attach-dict-fix-unsigned-compare 
Fix #1412: Perform Signed Comparison When Setting Attach Dict Param
 2018-11-12 17:53:11 -08:00
Yann Collet
626040ab53 changed PREFETCH() macro into PREFETCH_L2() 
which is more accurate
 2018-11-12 17:05:32 -08:00
W. Felix Handte
5faef4d378 Const 2018-11-12 14:48:42 -08:00
W. Felix Handte
2d9332eb21 Fix Types 2018-11-12 12:52:31 -08:00
W. Felix Handte
4127de5fa6 Switch Enum to Only Non-Negative Values, Update Comments 2018-11-12 12:47:47 -08:00
W. Felix Handte
596f7d1256 Fix #1412: Perform Signed Comparison When Setting Attach Dict Param 2018-11-12 12:07:57 -08:00
Yann Collet
e0701d3c5d
Merge pull request #1404 from facebook/T36302429 
fixed T36302471
 2018-11-06 11:53:20 -08:00
Yann Collet
3e5cdf1b6a fixed T36302429 2018-11-05 17:50:30 -08:00
Yann Collet
2caa995558 just add an assert() in ZSTD_insertBtAndGetAllMatches() 
to express a condition on ll0 .
May help static analyzer as in #1397
 2018-11-05 17:13:32 -08:00
Yann Collet
3a90229616
Merge pull request #1395 from facebook/decompressblock 
created zstd_decompress_block module
 2018-10-29 16:28:09 -07:00
Yann Collet
8d56f4baee added a few comments for clarifications 2018-10-26 15:21:52 -07:00
Yann Collet
7b74405150 refactor HUF_compress_internal for clarity 
changed workspace parameter convention
to always provide workspaceSize,
so that size can be explicitly checked.

Also, use more enum to make the meaning of some parameters more explicit.
 2018-10-26 13:21:37 -07:00
W. Felix Handte
b8235be865 Avoid Searching Dictionary in ZSTD_btlazy2 When an Optimal Match is Found 
Bailing here is important to avoid reading past the end of the input buffer.
 2018-10-08 15:59:32 -07:00
W. Felix Handte
d121b3451c Clean Up Debug Log Statements 2018-10-08 15:59:32 -07:00
W. Felix Handte
08da9ad316 Remove Unused Variable 2018-10-08 15:59:32 -07:00
Yann Collet
22ddf3523a fixed msan warning 
on btlazy2 strategy with dictAttach
 2018-10-02 18:20:20 -07:00
Yann Collet
228c6e5147
Merge pull request #1317 from felixhandte/split-logs 
Independent Dictionary and Working Context Table Logs
 2018-10-01 17:20:12 -07:00
W. Felix Handte
5b296869df Revert Ability to Set HashLog and ChainLog on Context When Dict is Attached 
This capability is not needed / used in the current unit of work. I'll
re-introduce it later, when we start allowing users to override the deduced
working context logs.
 2018-10-01 13:28:13 -07:00
W. Felix Handte
c2369fedc4 Restore Passing CParams to ZSTD_insertAndFindFirstIndex_internal 2018-09-28 17:12:54 -07:00
W. Felix Handte
bad74c4781 Use Working Ctx Logs when not in DMS Mode 
We pre-hash the ptr for the dict match state sometimes. When that actually
happens, a hashlog of 0 can produce undefined behavior (right shift a long
long by 64). Only applies to unoptimized compilations, since when
optimizations are applied, those hash operations are dropped when we're not
actually in dms mode.
 2018-09-28 17:12:54 -07:00
W. Felix Handte
c38acff94f When Attaching Dictionary, Size Working Tables Based on Input Size Only 2018-09-28 17:12:54 -07:00
W. Felix Handte
9d87d50878 Remove Log Overriding for the Time Being 2018-09-28 17:12:54 -07:00
W. Felix Handte
77fd17d93f Remove Strategy-Dependency in Making Attachment Decision 2018-09-28 17:12:54 -07:00
W. Felix Handte
00c088b32d Support Split Logs in ZSTD_btopt..ZSTD_btultra 2018-09-28 17:12:54 -07:00
W. Felix Handte
0783492178 Bump Split Log Support to ZSTD_btultra 2018-09-28 17:12:54 -07:00
W. Felix Handte
e4ac4a0f16 Support Split Logs in ZSTD_greedy..ZSTD_btlazy2 2018-09-28 17:12:54 -07:00
W. Felix Handte
e710dc3369 Bump Split Log Support to ZSTD_btlazy2 2018-09-28 17:12:54 -07:00
W. Felix Handte
22fcb8d4c7 Support Split Logs in ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
a232b3bb7c Bump Split Log Support to ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
fe96e98f81 Support a Separate Hash Log in ZSTD_fast 2018-09-28 17:12:54 -07:00
W. Felix Handte
bc880ebe8f Stop Passing in hashLog and stepSize to ZSTD_compressBlock_fast_generic 2018-09-28 17:12:54 -07:00
W. Felix Handte
b3107c7799 Temporary Commit to Retain Requested Hash and Chain Logs During Dict Attach 2018-09-28 17:12:54 -07:00
W. Felix Handte
34e0193129 Allow Setting Hash and Chain Logs on Contexts with Attached CDict 2018-09-28 17:12:54 -07:00
W. Felix Handte
eae8232f50 For Supported Strategies, Attach Dict Even When Params Don't Match 2018-09-28 17:12:54 -07:00
W. Felix Handte
01ff945eae Split Attach and Copy Reset Strategies into Separate Implementation Functions 2018-09-28 17:12:54 -07:00
W. Felix Handte
a6d6bbeae1 Pull Attachment Decision into Separate Function 2018-09-28 17:12:54 -07:00
W. Felix Handte
b7fba599ae And Then Avoid the Unused Parameter Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte
1f188ae655 Move Asserts into Function to Avoid Unused Function Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte
7212b5e5c2 Move Match State CParams Setting into resetCCtx and continueCCtx 2018-09-28 17:12:54 -07:00
W. Felix Handte
01e34d365b Strengthen Assertion to Assert Equality 2018-09-28 17:12:53 -07:00
W. Felix Handte
50cc1cf4d5 Remove CParams Arg from ZSTD_ldm_blockCompress 2018-09-28 17:12:53 -07:00
W. Felix Handte
14764de49f Stop Separately Passing CParams in ZSTD_lazy Internal Functions 2018-09-28 17:12:53 -07:00
W. Felix Handte
97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
W. Felix Handte
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
W. Felix Handte
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
03103269de Assert ctx and ms cparams Equivalency 2018-09-28 17:10:42 -07:00
W. Felix Handte
4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
W. Felix Handte
76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
Nick Terrell
6391cd1030 [zstd] Fix newly added test case 2018-09-28 12:09:28 -07:00
Nick Terrell
a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
Nick Terrell
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Yann Collet
2a5cd8535a
Merge pull request #1342 from facebook/fixcatyd 
fix : huge (>4GB) chain of blocks
 2018-09-27 10:20:14 -07:00
Yann Collet
404a7bfed0 moved again overflow correction 
cannot work from within ZSTD_compressBlock()
 2018-09-26 18:06:53 -07:00
Yann Collet
0e2dbac18a changed overflow correction place 
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
 2018-09-26 15:35:38 -07:00
Yann Collet
f98c69d77c fix : huge (>4GB) stream of blocks 
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
 2018-09-26 14:24:28 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet
bfff4f4809 ensure all writes to job->cSize are mutex protected 
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
 2018-09-21 16:00:39 -07:00
Yann Collet
32b7cf1bcf fixed tautological tests 
involving ZSTD_TARGETLENGTH_MIN (== 0)
 2018-09-21 15:04:43 -07:00
Yann Collet
c044345f8f Merge branch 'mingw' into minclevel 2018-09-21 14:56:57 -07:00
Yann Collet
de6c75e4e5
Merge pull request #1318 from felixhandte/shadow-dict-matches 
Don't Search Dictionary Context When Working Context Search Resulted in Mismatch
 2018-09-21 12:15:33 -07:00
Yann Collet
a54c86cfc6 defined a minimum negative level 
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
 2018-09-20 16:52:03 -07:00
Yann Collet
7992942d66 fixed complex tsan issue 
when job->consumed == job->src.size , compression job is presumed completed,
so it must be the very last action done in worker thread.
 2018-09-20 13:47:31 -07:00
Yann Collet
6b07a66aec fixed minor reporting discrepancy in MT mode 2018-09-19 16:30:55 -07:00
Yann Collet
ca02ebee07 removed static variables 
so that --adapt can work on multiple input files too
 2018-09-19 15:25:50 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
ko-zu
18b4a1da61 Fix clang build 
Fix dixygen comment
Fix clang binary path
 2018-09-16 10:27:02 +09:00
W. Felix Handte
b76c888497 ZSTD_dfast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:24:25 -07:00
W. Felix Handte
b048af5999 ZSTD_fast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:23:35 -07:00
Yann Collet
31ebb26945
Merge pull request #1301 from terrelln/lit-size 
[zstd] Fix seqStore growth
 2018-08-28 17:10:25 -07:00
Nick Terrell
5e580de6da [zstd] Fix seqStore growth 
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
 2018-08-28 13:24:44 -07:00
Yann Collet
b37a0a6bde
Merge pull request #1298 from facebook/bench 
Refactored bench.c
 2018-08-28 12:25:02 -07:00
Yann Collet
af23d39eb8
Merge pull request #1297 from felixhandte/check-offset-table 
Fix Missing Offset Table Check
 2018-08-24 17:36:44 -07:00