AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,468 Commits 1 Branch 0 Tags 27 MiB
0154866749
Commit Graph

3177 Commits

Author SHA1 Message Date
Yann Collet
5120883a9c bumped version number 
so that potential issue report do not confuse `dev` with latest release
 2019-12-03 17:06:42 -08:00
Bimba Shrestha
1fc9352f81 Using bss var instead of creating new bool 2019-12-02 21:39:06 -08:00
Bimba Shrestha
1f681d8592 Merge branch 'oss' of https://github.com/bimbashrestha/zstd into oss 2019-11-27 10:56:54 -08:00
Bimba Shrestha
a3a3c62b81 [fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights (#1898) 
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.

* Only setting loaded dict huf table to valid on non-zero

* Adding hasNoZeroWeights test to fse tables

* Forbiding nbBits != 0 when weight == 0

* Reverting the last commit

* Setting table log to 0 when weight == 0

* Small (invalid) zero weight dict test

* Small (valid) zero weight dict test

* Initializing repeatMode vars to check before zero check

* Removing FSE changes to seperate pr

* Reverting accidentally changed file

* Negating bool, using unsigned, optimization nit
 2019-11-26 12:24:19 -08:00
Bimba Shrestha
d4e17d0776 Negating bool, updating bool on inner branches 2019-11-26 12:17:43 -08:00
Nick Terrell
718f00ff6f
Optimize decompression speed for gcc and clang (#1892) 
* Optimize `ZSTD_decodeSequence()`
* Optimize Huffman decoding
* Optimize `ZSTD_decompressSequences()`
* Delete `ZSTD_decodeSequenceLong()`
 2019-11-25 18:26:19 -08:00
Bimba Shrestha
826b555463
Merge branch 'dev' into oss 2019-11-22 17:29:33 -08:00
Bimba Shrestha
10bce1919e Mixed declration fix 2019-11-21 13:08:27 -08:00
Bimba Shrestha
0451accab1 Checking noCompressBlock explicitly for rep code confirmation 2019-11-21 13:06:26 -08:00
Nick Terrell
659e9f05cf Fix null pointer addition 2019-11-20 18:36:04 -08:00
Yann Collet
2d4dcce55f
Merge pull request #1894 from felixhandte/doc-clarify-dctx-reset 
Easy: Update Comment on `ZSTD_initDStream()`
 2019-11-19 16:18:56 -08:00
Nick Terrell
e0d6daabac Fix Appveyor failure 2019-11-19 11:12:26 -08:00
Bimba Shrestha
8f0c2d04c8 Going back to original flow but removing else return 2019-11-19 10:03:07 -08:00
W. Felix Handte
722149cf2b Easy: Update Comment on ZSTD_initDStream() 2019-11-19 01:57:15 -05:00
Nick Terrell
6a7f65117e
Merge pull request #1866 from legrosbuffle/dev 
Optimized loop bounds to allow the compiler to unroll the loop.
 2019-11-18 16:16:30 -08:00
Nick Terrell
a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed 
RLE test and re-enable RLE in main compression loop
 2019-11-18 16:09:33 -08:00
Bimba Shrestha
80586f5e80 Reversing condition order and forwarding error 2019-11-18 13:53:55 -08:00
Bimba Shrestha
dade64428f Output regular uncompressed block when compressSequences fails 2019-11-18 08:43:14 -08:00
Bimba Shrestha
2d5d961a60 Typo in comment 2019-11-15 19:00:53 -08:00
Bimba Shrestha
dba767c0bb Leaving room for checksum 2019-11-15 18:44:51 -08:00
Vincent Torri
6b5c10b48c shared library: rename import library with .dll.a extension 
mort of open source project are using this extension for the import library.
The Win32 linker is supporting this extension, see
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/win32.html
section "direct linking to a dll"
 2019-11-15 19:46:06 +01:00
Clement Courbet
b3c9fc27b4 Optimized loop bounds to allow the compiler to unroll the loop. 
This has no measurable impact on large files but improves small file
decompression by ~1-2% for 10kB, benchmarked with:

head -c 10000 silesia.tar > /tmp/test
make CC=/usr/local/bin/clang-9 BUILD_STATIC=1 && ./lzbench -ezstd -t1,5 /tmp/test
 2019-11-15 08:27:05 +01:00
Sen Huang
d9646dcbb5 Fixed main compression logic changes 2019-11-14 19:39:09 -05:00
Yann Collet
4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed 
Superblocks rebased for merge
 2019-11-14 13:31:34 -08:00
Sen Huang
c26d32c91c Change superblock #include to be last 2019-11-14 13:12:17 -05:00
Yann Collet
d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size 
Method to get dictionary header size
 2019-11-14 09:44:07 -08:00
Sen Huang
c85d10d0ea Remove mixed declarations 2019-11-08 13:57:26 -05:00
Sen Huang
d9c475f3b3 Fix static analyze error, use proper bounds for dictEnd 2019-11-08 13:57:26 -05:00
Sen Huang
d06b90692b Move asserts to loadZstdDictionary() 2019-11-08 13:57:26 -05:00
Sen Huang
b39149e156 Expose ZSTD_reset_compressedBlockState() to shared API 2019-11-08 13:57:26 -05:00
Sen Huang
6ce335371b Add error forwarding to loadCEntropy(), make check for dictSize >= 8 from bad merge 2019-11-08 13:57:26 -05:00
Sen Huang
4a61aaf368 Remove redundant comment 2019-11-08 13:57:26 -05:00
Sen Huang
c787b351ea Use ZSTD Error codes, improve explanation of ZSTD_loadCEntropy() and ZSTD_loadDEntropy() 2019-11-08 13:57:26 -05:00
Sen Huang
04fb42b4f3 Integrated refactor into getDictHeaderSize, now passes tests 2019-11-08 13:57:26 -05:00
Sen Huang
0bcaf6db08 First working pass at refactor of loadZstdDictionary() 2019-11-08 13:57:26 -05:00
Sen Huang
4b141b63e0 Revert "Move decompress symbols into zstd_internal.h, remove dependency" 
This reverts commit a152b4c67a5266f611db4a2eac4a79003852a795.
 2019-11-08 13:57:26 -05:00
Sen Huang
84404cff6e Move decompress symbols into zstd_internal.h, remove dependency 2019-11-08 13:57:26 -05:00
Sen Huang
341e0641ed Checks malloc() for failure, returns 0 if so 2019-11-08 13:57:26 -05:00
Sen Huang
97b7f712f3 Change to heap allocation, remove implicit type conversion 2019-11-08 13:57:25 -05:00
Sen Huang
3c36a7f13a Add ZDICT_getHeaderSize() 2019-11-08 13:57:08 -05:00
Nick Terrell
8c474f9845 Fix parameter selection and adjustment with srcSize == 0 2019-11-07 08:58:43 -08:00
Felix Handte
5688447758
Merge pull request #1873 from felixhandte/make-overlap-log-multithread-only 
Fix #1861: Restrict overlapLog Parameter When Not Built With Multithreading
 2019-11-06 16:56:37 -05:00
Felix Handte
ba4613602f
Merge pull request #1843 from moozzyk/issue-1637 
Take ZSTD_parameters as a const pointer
 2019-11-06 16:56:14 -05:00
W. Felix Handte
c13f81905a Fix #1861: Restrict overlapLog Parameter When Not Built With Multithreading 
This parameter is unused in single-threaded compression. We should make it
behave like the other multithread-only parameters, for which we only accept
zero when we are not built with multithreading.
 2019-11-06 16:05:02 -05:00
Sen Huang
13bb7500e8 Fix frame argument to compression 2019-11-05 16:15:55 -05:00
Sen Huang
f2932fb5eb Fix more merge conflicts 2019-11-05 15:54:05 -05:00
Sen Huang
7ce891870c Fix merge conflicts 2019-11-05 15:51:25 -05:00
Bimba Shrestha
3fb5b106da Replacing some literals with constants 2019-11-05 10:26:57 -08:00
Nick Terrell
60205fec02 Fix 2 bugs in dictionary loading 
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
  This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
 2019-11-01 16:52:07 -07:00
Sen Huang
b9ede1c8c2 Make sure contentsize is known 2019-10-30 16:03:58 -04:00
Nick Terrell
9c1860861e Fix assert in ZSTD_safecopy 
In the case that `op >= oend_w` it is possible that `diff < 8` because
the two buffers could be adjacent.

Credit to OSS-Fuzz, which found the bug. It isn't reproducible because
it depends on the memory layout.
 2019-10-28 17:51:17 -07:00
Felix Handte
01ec595b85
Merge pull request #1851 from felixhandte/pkg-config-prefix-fix 
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
 2019-10-28 14:24:56 -04:00
Yann Collet
74065da4c5 updated API inline doc and manual 
regarding ZSTD_CDict created without a dictBuffer.
 2019-10-28 11:15:41 -07:00
W. Felix Handte
74bd76c3ff In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time 
Addresses #1794. Instead of deriving the lib dir and include dir at
build-time, let's do it like everyone else does at pkg-config run-time.

This has the disadvantage that we can no longer override LIBDIR and
INCLUDEDIR in the Makefile and have that reflected in the .pc file.
 2019-10-25 15:07:31 -04:00
Yann Collet
c2140e9db0
Merge pull request #1845 from facebook/zbuff 
improve deprecation warning macro
 2019-10-25 09:59:00 -07:00
Yann Collet
a9a216a846
Merge pull request #1824 from senhuang42/new_path_for_cdict 
Avoid using CDict params when input is large.
 2019-10-23 12:04:40 -07:00
Yann Collet
63e435dda1 improve deprecation warning macro 
fix #1488

although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
 2019-10-23 11:59:32 -07:00
moozzyk
eda7946a36 Take ZSTD_parameters as a const pointer 
Fixes: #1637
 2019-10-22 23:21:54 -07:00
Yann Collet
f966cd080a added documentation on DYNAMIC_BMI2 build macro 2019-10-22 17:43:09 -07:00
Yann Collet
5d5c895b18 fix initCStream_advanced() for fast strategies 
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.

Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.

Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After  : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550

Fix #1787.

To follow : refactor the test which was supposed to catch this issue (and failed)
 2019-10-22 15:01:38 -07:00
Yann Collet
111b0c53b0 update documentation on deprecated functions 
mostly : note that these functions will soon generate deprecation warnings
 2019-10-22 13:51:18 -07:00
Nick Terrell
b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data 
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
 2019-10-21 21:16:17 -07:00
Sen Huang
c2e1e54f24 ((x or y) or z) == (x or y or z), remove brackets 2019-10-21 19:16:50 -04:00
Sen Huang
59c81aa31b Line up comments :) 2019-10-21 19:12:15 -04:00
Sen Huang
dbda8c318a Trailing comma 2019-10-21 19:10:13 -04:00
Sen Huang
0c00455ea6 Merge branch 'dev' of github.com:senhuang42/zstd into new_path_for_cdict 2019-10-21 19:06:51 -04:00
Sen Huang
5b2f4ac1a8 merge 2019-10-21 19:02:52 -04:00
Sen Huang
2ab484a5f9 Fix bad merge 2019-10-21 18:55:17 -04:00
Nick Terrell
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset 
[zstdmt] Don't memset the jobDescription
 2019-10-21 15:53:57 -07:00
Sen Huang
b6c3459d50 merge 2019-10-21 18:46:17 -04:00
Yann Collet
6cf04c0344
Merge pull request #1834 from facebook/winFix 
Windows fixes
 2019-10-21 13:45:17 -07:00
Sen Huang
676f89902a Added multiplier, renamed new enum to something more useful 2019-10-21 15:36:12 -04:00
Sen Huang
1f3a51fb52 Updated forceAttachDict param bounds 2019-10-21 15:36:12 -04:00
Sen Huang
8f69c47643 Add enum to decision process 2019-10-21 15:36:12 -04:00
Sen Huang
e4de8b098a Added support for forcing new CDict behavior and updated enum 2019-10-21 15:36:12 -04:00
Sen Huang
9294f4826b Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang
f0fccc8847 Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang
bb2df8c499 Trailing whitespace 2019-10-21 15:36:12 -04:00
Sen Huang
cf51501d2f Fix test 2019-10-21 15:36:12 -04:00
Sen Huang
ea3cb6988f Cast to BYTE to appease appveyor 2019-10-21 15:36:12 -04:00
Sen Huang
a727a85a7e merge conflicts round 2 2019-10-21 15:36:12 -04:00
Sen Huang
053a35fd64 formatting 2019-10-21 15:35:33 -04:00
Sen Huang
3fa4daaa55 Fix error 2019-10-21 15:35:33 -04:00
Sen Huang
3328348c63 Add compressionlevel to cdict 2019-10-21 15:32:39 -04:00
Felix Handte
cf725630a6
Merge pull request #1795 from felixhandte/workspace-asan 
Add Poisoned Redzones to the Workspace When Compiling with ASAN
 2019-10-21 12:15:17 -04:00
Sen Huang
e8aa3e486d Updated forceAttachDict param bounds 2019-10-20 22:01:08 -04:00
Sen Huang
6d297265f9 Add enum to decision process 2019-10-20 19:02:47 -04:00
Sen Huang
1daa898c93 Added support for forcing new CDict behavior and updated enum 2019-10-20 14:03:09 -04:00
Nick Terrell
0bc39bc3a0 [zstdmt] Don't memset the jobDescription 2019-10-18 15:05:51 -07:00
Nick Terrell
243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Yann Collet
1795133c45 refactored FIO_compressMultipleFilenames() prototype 
for consistency
 2019-10-17 15:32:03 -07:00
Yann Collet
6446ffb277
Merge pull request #1827 from facebook/dm_Dct 
updated erroneous comments using ZSTD_dm_*
 2019-10-17 10:30:58 -07:00
Yann Collet
19741c7d99
Merge pull request #1815 from facebook/zlibwrap 
make zlibWrapper strict ISO-C90 compatible
 2019-10-16 16:45:15 -07:00
Yann Collet
6323966e53 updated erroneous comments using ZSTD_dm_* 
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
 2019-10-16 16:14:04 -07:00
Yann Collet
2d5201b0ab removed wildcopy8() 
which is no longer used,
noticed by @davidbolvansky
 2019-10-16 14:51:33 -07:00
Sen Huang
4455f00cb8 Changed to int from BYTE 2019-10-16 15:06:02 -04:00
Sen Huang
4f7d26b0ee Changed to int from BYTE 2019-10-16 15:05:29 -04:00
Sen Huang
cf00ea367a Trailing whitespace 2019-10-16 10:31:27 -04:00
Sen Huang
8cb2174446 Fix test 2019-10-16 10:29:31 -04:00
Sen Huang
5e901b6f32 Cast to BYTE to appease appveyor 2019-10-15 13:58:44 -04:00