AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,360 Commits 1 Branch 0 Tags 27 MiB
e32e3e8662
Commit Graph

3144 Commits

Author SHA1 Message Date
Sen Huang
04fb42b4f3 Integrated refactor into getDictHeaderSize, now passes tests 2019-11-08 13:57:26 -05:00
Sen Huang
0bcaf6db08 First working pass at refactor of loadZstdDictionary() 2019-11-08 13:57:26 -05:00
Sen Huang
4b141b63e0 Revert "Move decompress symbols into zstd_internal.h, remove dependency" 
This reverts commit a152b4c67a5266f611db4a2eac4a79003852a795.
 2019-11-08 13:57:26 -05:00
Sen Huang
84404cff6e Move decompress symbols into zstd_internal.h, remove dependency 2019-11-08 13:57:26 -05:00
Sen Huang
341e0641ed Checks malloc() for failure, returns 0 if so 2019-11-08 13:57:26 -05:00
Sen Huang
97b7f712f3 Change to heap allocation, remove implicit type conversion 2019-11-08 13:57:25 -05:00
Sen Huang
3c36a7f13a Add ZDICT_getHeaderSize() 2019-11-08 13:57:08 -05:00
Nick Terrell
8c474f9845 Fix parameter selection and adjustment with srcSize == 0 2019-11-07 08:58:43 -08:00
Felix Handte
5688447758
Merge pull request #1873 from felixhandte/make-overlap-log-multithread-only 
Fix #1861: Restrict overlapLog Parameter When Not Built With Multithreading
 2019-11-06 16:56:37 -05:00
Felix Handte
ba4613602f
Merge pull request #1843 from moozzyk/issue-1637 
Take ZSTD_parameters as a const pointer
 2019-11-06 16:56:14 -05:00
W. Felix Handte
c13f81905a Fix #1861: Restrict overlapLog Parameter When Not Built With Multithreading 
This parameter is unused in single-threaded compression. We should make it
behave like the other multithread-only parameters, for which we only accept
zero when we are not built with multithreading.
 2019-11-06 16:05:02 -05:00
Sen Huang
13bb7500e8 Fix frame argument to compression 2019-11-05 16:15:55 -05:00
Sen Huang
f2932fb5eb Fix more merge conflicts 2019-11-05 15:54:05 -05:00
Sen Huang
7ce891870c Fix merge conflicts 2019-11-05 15:51:25 -05:00
Bimba Shrestha
3fb5b106da Replacing some literals with constants 2019-11-05 10:26:57 -08:00
Nick Terrell
60205fec02 Fix 2 bugs in dictionary loading 
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
  This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
 2019-11-01 16:52:07 -07:00
Sen Huang
b9ede1c8c2 Make sure contentsize is known 2019-10-30 16:03:58 -04:00
Nick Terrell
9c1860861e Fix assert in ZSTD_safecopy 
In the case that `op >= oend_w` it is possible that `diff < 8` because
the two buffers could be adjacent.

Credit to OSS-Fuzz, which found the bug. It isn't reproducible because
it depends on the memory layout.
 2019-10-28 17:51:17 -07:00
Felix Handte
01ec595b85
Merge pull request #1851 from felixhandte/pkg-config-prefix-fix 
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
 2019-10-28 14:24:56 -04:00
Yann Collet
74065da4c5 updated API inline doc and manual 
regarding ZSTD_CDict created without a dictBuffer.
 2019-10-28 11:15:41 -07:00
W. Felix Handte
74bd76c3ff In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time 
Addresses #1794. Instead of deriving the lib dir and include dir at
build-time, let's do it like everyone else does at pkg-config run-time.

This has the disadvantage that we can no longer override LIBDIR and
INCLUDEDIR in the Makefile and have that reflected in the .pc file.
 2019-10-25 15:07:31 -04:00
Yann Collet
c2140e9db0
Merge pull request #1845 from facebook/zbuff 
improve deprecation warning macro
 2019-10-25 09:59:00 -07:00
Yann Collet
a9a216a846
Merge pull request #1824 from senhuang42/new_path_for_cdict 
Avoid using CDict params when input is large.
 2019-10-23 12:04:40 -07:00
Yann Collet
63e435dda1 improve deprecation warning macro 
fix #1488

although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
 2019-10-23 11:59:32 -07:00
moozzyk
eda7946a36 Take ZSTD_parameters as a const pointer 
Fixes: #1637
 2019-10-22 23:21:54 -07:00
Yann Collet
f966cd080a added documentation on DYNAMIC_BMI2 build macro 2019-10-22 17:43:09 -07:00
Yann Collet
5d5c895b18 fix initCStream_advanced() for fast strategies 
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.

Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.

Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After  : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550

Fix #1787.

To follow : refactor the test which was supposed to catch this issue (and failed)
 2019-10-22 15:01:38 -07:00
Yann Collet
111b0c53b0 update documentation on deprecated functions 
mostly : note that these functions will soon generate deprecation warnings
 2019-10-22 13:51:18 -07:00
Nick Terrell
b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data 
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
 2019-10-21 21:16:17 -07:00
Sen Huang
c2e1e54f24 ((x or y) or z) == (x or y or z), remove brackets 2019-10-21 19:16:50 -04:00
Sen Huang
59c81aa31b Line up comments :) 2019-10-21 19:12:15 -04:00
Sen Huang
dbda8c318a Trailing comma 2019-10-21 19:10:13 -04:00
Sen Huang
0c00455ea6 Merge branch 'dev' of github.com:senhuang42/zstd into new_path_for_cdict 2019-10-21 19:06:51 -04:00
Sen Huang
5b2f4ac1a8 merge 2019-10-21 19:02:52 -04:00
Sen Huang
2ab484a5f9 Fix bad merge 2019-10-21 18:55:17 -04:00
Nick Terrell
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset 
[zstdmt] Don't memset the jobDescription
 2019-10-21 15:53:57 -07:00
Sen Huang
b6c3459d50 merge 2019-10-21 18:46:17 -04:00
Yann Collet
6cf04c0344
Merge pull request #1834 from facebook/winFix 
Windows fixes
 2019-10-21 13:45:17 -07:00
Sen Huang
676f89902a Added multiplier, renamed new enum to something more useful 2019-10-21 15:36:12 -04:00
Sen Huang
1f3a51fb52 Updated forceAttachDict param bounds 2019-10-21 15:36:12 -04:00
Sen Huang
8f69c47643 Add enum to decision process 2019-10-21 15:36:12 -04:00
Sen Huang
e4de8b098a Added support for forcing new CDict behavior and updated enum 2019-10-21 15:36:12 -04:00
Sen Huang
9294f4826b Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang
f0fccc8847 Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang
bb2df8c499 Trailing whitespace 2019-10-21 15:36:12 -04:00
Sen Huang
cf51501d2f Fix test 2019-10-21 15:36:12 -04:00
Sen Huang
ea3cb6988f Cast to BYTE to appease appveyor 2019-10-21 15:36:12 -04:00
Sen Huang
a727a85a7e merge conflicts round 2 2019-10-21 15:36:12 -04:00
Sen Huang
053a35fd64 formatting 2019-10-21 15:35:33 -04:00
Sen Huang
3fa4daaa55 Fix error 2019-10-21 15:35:33 -04:00
Sen Huang
3328348c63 Add compressionlevel to cdict 2019-10-21 15:32:39 -04:00
Felix Handte
cf725630a6
Merge pull request #1795 from felixhandte/workspace-asan 
Add Poisoned Redzones to the Workspace When Compiling with ASAN
 2019-10-21 12:15:17 -04:00
Sen Huang
e8aa3e486d Updated forceAttachDict param bounds 2019-10-20 22:01:08 -04:00
Sen Huang
6d297265f9 Add enum to decision process 2019-10-20 19:02:47 -04:00
Sen Huang
1daa898c93 Added support for forcing new CDict behavior and updated enum 2019-10-20 14:03:09 -04:00
Nick Terrell
0bc39bc3a0 [zstdmt] Don't memset the jobDescription 2019-10-18 15:05:51 -07:00
Nick Terrell
243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Yann Collet
1795133c45 refactored FIO_compressMultipleFilenames() prototype 
for consistency
 2019-10-17 15:32:03 -07:00
Yann Collet
6446ffb277
Merge pull request #1827 from facebook/dm_Dct 
updated erroneous comments using ZSTD_dm_*
 2019-10-17 10:30:58 -07:00
Yann Collet
19741c7d99
Merge pull request #1815 from facebook/zlibwrap 
make zlibWrapper strict ISO-C90 compatible
 2019-10-16 16:45:15 -07:00
Yann Collet
6323966e53 updated erroneous comments using ZSTD_dm_* 
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
 2019-10-16 16:14:04 -07:00
Yann Collet
2d5201b0ab removed wildcopy8() 
which is no longer used,
noticed by @davidbolvansky
 2019-10-16 14:51:33 -07:00
Sen Huang
4455f00cb8 Changed to int from BYTE 2019-10-16 15:06:02 -04:00
Sen Huang
4f7d26b0ee Changed to int from BYTE 2019-10-16 15:05:29 -04:00
Sen Huang
cf00ea367a Trailing whitespace 2019-10-16 10:31:27 -04:00
Sen Huang
8cb2174446 Fix test 2019-10-16 10:29:31 -04:00
Sen Huang
5e901b6f32 Cast to BYTE to appease appveyor 2019-10-15 13:58:44 -04:00
Sen Huang
5c010c9d2d merge conflicts round 2 2019-10-15 13:10:05 -04:00
Sen Huang
a06b51879c merge conflict 2019-10-15 12:58:50 -04:00
Sen Huang
23dac23a49 formatting 2019-10-15 12:44:48 -04:00
Sen Huang
0c8df5c928 Fix error 2019-10-15 12:28:23 -04:00
Sen Huang
a65eb39f9d Add compressionlevel to cdict 2019-10-15 10:22:06 -04:00
Yann Collet
fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api 
Adding api for extracting sequences from seqstore
 2019-10-10 13:11:18 -07:00
W. Felix Handte
ede31da2ea Fix CCtx Size Estimation 2019-10-10 15:02:08 -04:00
W. Felix Handte
bd6a20b8a0 Expand Default Redzone Size 2019-10-10 13:45:55 -04:00
W. Felix Handte
2c80a9f8ac Check if CCtx in Workspace after Null Check 2019-10-10 13:40:16 -04:00
W. Felix Handte
b6987acbbf Declare the ASAN Functions We Need, Don't Include the Header 2019-10-10 13:40:16 -04:00
W. Felix Handte
0ffae7e440 Stop Allocating Extra Space for Table Redzones 2019-10-10 13:40:16 -04:00
W. Felix Handte
a07037b784 Don't Try to Redzone the Tables 2019-10-10 13:40:16 -04:00
W. Felix Handte
0cc481ef66 Fix Workspace Size Calculation 2019-10-10 13:40:16 -04:00
W. Felix Handte
b6c0a02a17 Fix ZSTD_sizeof_matchState() Calculation 2019-10-10 13:40:16 -04:00
W. Felix Handte
8cffd6ed08 Avoid ASAN Failure in ZSTD_cwksp_free() 2019-10-10 13:40:16 -04:00
W. Felix Handte
ef0b5707c5 Refactor Freeing CCtxes / CDicts Inside Workspaces 2019-10-10 13:40:16 -04:00
W. Felix Handte
143b296cf6 Surround Workspace Allocs with Dead Zone 2019-10-10 13:40:16 -04:00
W. Felix Handte
19a0955ec9 Add ZSTD_cwksp_alloc_size() to Help Calculate Needed Workspace Size 2019-10-10 13:40:16 -04:00
W. Felix Handte
da88c35d41 Stop Assuming Tables are Adjacent 2019-10-10 13:40:16 -04:00
W. Felix Handte
35c30d6ca7 Poison Unused Workspace Memory 2019-10-10 13:40:16 -04:00
W. Felix Handte
edb6d884a5 Detect Whether We're Being Compiled with ASAN 2019-10-10 13:40:16 -04:00
W. Felix Handte
dc1fb684bf Remove Unused MEM_SKIP_MSAN Macro 2019-10-10 13:40:16 -04:00
Bimba Shrestha
36528b96c4 Manually moving instead of memcpy on decoder and using genBuffer() 2019-10-03 09:26:51 -07:00
Bimba Shrestha
61ec4c2e7f Cleaning sequence parsing logic 2019-10-03 06:42:40 -07:00
Yann Collet
cb18fffe65 enforce C90 compatibility for zlibWrapper 2019-09-24 17:50:58 -07:00
Yann Collet
ad2a2785f7 bump version number to v1.4.4 
so that future reports on `dev` branch use this number instead
 2019-09-24 15:15:33 -07:00
Bimba Shrestha
c04245b257 Replacing assert with memory_allocation error code throw 2019-09-23 15:42:16 -07:00
Bimba Shrestha
be0bebd24e Adding test and null check for malloc 2019-09-23 15:08:18 -07:00
Dávid Bolvanský
1ab1a40c9c Fixed one more place 2019-09-23 21:32:56 +02:00
Dávid Bolvanský
1f7228c040 Use clz ^ 31 instead of 31 - clz; better codegen for GCC 2019-09-23 21:23:09 +02:00
Nick Terrell
7451c6578c
Merge pull request #1804 from terrelln/wild-and-fast 
Optimize (de)compression and fix wildcopy overread
 2019-09-21 17:04:36 -07:00
Nick Terrell
5cb7615f1f Add UNUSED_ATTR to ZSTD_storeSeq() 2019-09-20 21:37:13 -07:00
Nick Terrell
5dc0a1d659 HINT_INLINE ZSTD_storeSeq() 
Clang on Mac wasn't inlining `ZSTD_storeSeq()` in level 1, which was
causing a 5% performance regression. This fixes it.
 2019-09-20 16:39:27 -07:00