AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
2,864 Commits 1 Branch 0 Tags 27 MiB
3d95925a59
Commit Graph

291 Commits

Author SHA1 Message Date
Yann Collet
407a11f63e fixed Visual compatibility 2016-11-03 15:52:01 -07:00
Nick Terrell
d82efd8a70 ZSTD_compress_usingDict() when dict gets loaded 
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
 2016-11-02 18:07:16 -07:00
Yann Collet
ee5b725823 ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used 2016-10-27 14:20:55 -07:00
Yann Collet
335ad5d4d4 added ZSTD_initDStream_usingDDict() . 
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
 2016-10-25 17:47:02 -07:00
Yann Collet
9516234e67 first sketch for ZSTD_initCStream_usingCDict() 2016-10-25 16:19:52 -07:00
Yann Collet
62d9a7ddfd Merge pull request #429 from inikep/btopt2 
Btopt2
 2016-10-25 14:48:43 -07:00
Przemyslaw Skibinski
5c5f01f3da added ZSTD_btopt2 strategy 2016-10-25 12:25:07 +02:00
Nick Terrell
b2c39a22b0 Fix compiler narrowing warning 2016-10-24 14:50:13 -07:00
Nick Terrell
f698ad6deb Merge remote-tracking branch 'upstream/dev' into fixes 
* upstream/dev:
  added doc\zstd_manual.html
  added contrib\gen_html
  zstd_compression_format.md moved to doc/
  Fix small bug in ZSTD_execSequence()
  improved ZSTD_compressBlock_opt_extDict_generic
  protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
  zstd_opt.h: small improvement in compression ratio
  improved dicitonary segment merge
  use implicit rules to compile zstd_decompress.c
  detect early impossible decompression scenario in legacy decoder v0.5
  no repeat mode in legacy v0.5
  fixed invalid invocation of dictionary in legacy decoder v0.5
  fix edge case
  fix command line interpretation
  fixed minor corner case
  zstd.h: added the Introduction section
  fixed clang 3.5 warnings
  zstd.h: updated comments
 2016-10-24 13:10:13 -07:00
Nick Terrell
f9c9af3c2e Reject dictionaries with incomplete entropy tables 
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.

Ensure that dictionaries support all `matchLength`, and `literalLength`
codes.  They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
 2016-10-24 10:42:44 -07:00
Przemyslaw Skibinski
3ee94a7600 zstd_compression_format.md moved to doc/ 2016-10-24 15:58:07 +02:00
Przemyslaw Skibinski
4732074a71 improved ZSTD_compressBlock_opt_extDict_generic 2016-10-21 11:19:00 +02:00
Przemyslaw Skibinski
d365ae3497 zstd_opt.h: small improvement in compression ratio 2016-10-20 11:49:02 +02:00
Yann Collet
197a55ee7b fix edge case 2016-10-18 11:27:52 -07:00
Nick Terrell
fd98087047 Fix stack buffer overflow in HUF_readCTable() 
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
 2016-10-17 18:16:59 -07:00
Nick Terrell
bfd943ace5 Fix buffer overrun in ZSTD_loadDictEntropyStats() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`.  This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.

The benchmarks look good, there is no obvious compression performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 271.6 MB/s , 716.8 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 204.8 MB/s , 671.1 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.8 MB/s , 658.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 136.4 MB/s , 665.3 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.9 MB/s , 647.0 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  65.2 MB/s , 670.4 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.9 MB/s , 688.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  32.4 MB/s , 711.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.1 MB/s , 718.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  22.2 MB/s , 721.8 MB/s

  > ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 273.8 MB/s , 722.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 203.2 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 157.4 MB/s , 666.5 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 132.1 MB/s , 661.9 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.8 MB/s , 641.6 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  63.1 MB/s , 677.0 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.3 MB/s , 678.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  33.1 MB/s , 708.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.5 MB/s , 710.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  21.9 MB/s , 723.9 MB/s
 2016-10-17 16:55:52 -07:00
Yann Collet
2b361cf2f1 minor opt 2016-10-14 16:09:07 -07:00
Nick Terrell
3b9cdf9220 Fix ubsan failures (pass NULL to memcpy) 2016-10-12 20:54:42 -07:00
Yann Collet
cf409a7e2a fixed : init*_advanced() followed by reset() with different pledgedSrcSiz 2016-09-26 16:41:05 +02:00
Yann Collet
97b378a6f8 Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize. 2016-09-21 17:20:19 +02:00
Yann Collet
993060e0f2 cli : better adaptation to small files 2016-09-21 16:46:08 +02:00
Yann Collet
a6bdf55759 fixed memory leak 2016-09-15 17:02:06 +02:00
Yann Collet
644a8da88a fixed minor conversion warning 2016-09-15 16:16:21 +02:00
Yann Collet
4cb212938c introduced ZSTD_resetCStream() 2016-09-15 14:54:07 +02:00
Yann Collet
fa0c09760c variable renaming 2016-09-15 14:11:01 +02:00
Yann Collet
d7c6589df8 support ZSTD_sizeof_*() on NULL 
added ZSTD_sizeof_CDict()
 2016-09-15 02:57:27 +02:00
Yann Collet
64deef3bee Fixed srcSize=1 2016-09-14 00:16:07 +02:00
Yann Collet
ac8bace6b1 support large skippable frames 2016-09-07 14:54:23 +02:00
Yann Collet
95d07d7447 introduced CHECK_E 2016-09-06 16:38:51 +02:00
Yann Collet
3e21ec5b01 introduced CHECK_F 2016-09-06 15:36:19 +02:00
Yann Collet
5c956d593c FORCE_INLINE common definition 2016-09-06 15:05:19 +02:00
Yann Collet
edbcd9f5b2 fixed zbufftest 2016-09-06 14:30:57 +02:00
Yann Collet
b624922b14 fixed checksum 2016-09-06 11:16:57 +02:00
Yann Collet
a7737f6a60 improved compression on small files when using same parameters 2016-09-06 09:44:59 +02:00
Yann Collet
7ae67bb18a small compression speed gains with using_CDict 2016-09-06 06:28:05 +02:00
Yann Collet
855766d73d clarified dictionary in format description 2016-09-02 17:04:49 -07:00
Yann Collet
1563bfeabc fixing FORCE_INLINE for older compilers (#330) 2016-09-02 11:44:21 -07:00
David Lam
e10f7f3dcb merge 2016-08-30 12:03:36 -07:00
Yann Collet
4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
Yann Collet
14200a20f0 Fixed issue #304, reported by @borzunov 2016-08-30 06:51:00 -07:00
David Lam
da9d3b7057 Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c 2016-08-29 17:31:51 -07:00
Yann Collet
23b6e05d8e ZSTD_malloc() and ZSTD_free(), to simplify customMem 2016-08-28 21:05:43 -07:00
Yann Collet
e19a9ef05d update compression level table 2016-08-26 20:02:49 +02:00
Yann Collet
87c18b2ebd fixed multiple minor warnings for XCode 2016-08-26 01:43:47 +02:00
Yann Collet
0d59a6f73a removed debug strings 2016-08-25 22:42:46 +02:00
Yann Collet
5a02b69215 reinforced fix for huge files 2016-08-25 22:24:59 +02:00
Yann Collet
96bdd87de4 fixed : compression bug on very large files 2016-08-25 14:34:42 +02:00
inikep
a3a47ec4d0 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other 2016-08-24 21:25:49 +02:00
Yann Collet
a2cdffe556 fixed wrong parameter issue 2016-08-24 19:42:15 +02:00
inikep
e416e30019 remove unnecessary comments 2016-08-24 17:32:09 +02:00
inikep
4e90f6c1e0 removed ZSTD_LOG_ENCODE and ZSTD_LOG_BLOCK 2016-08-24 17:24:11 +02:00
inikep
83388e109f removed ZSTD_LOG_PARSER 2016-08-24 17:22:20 +02:00
inikep
8a36f8527c removed stats in debug mode 2016-08-24 17:19:12 +02:00
Yann Collet
24b68a5b23 update cLevel table for 256KB 2016-08-24 14:22:26 +02:00
Yann Collet
c54692faeb improved level 3 2016-08-24 01:45:46 +02:00
Yann Collet
70e3b31306 fixed playtests on os-x 2016-08-23 01:18:06 +02:00
Yann Collet
cb3276329a added sizeof CStream and DStream 2016-08-23 00:31:59 +02:00
Yann Collet
d1733f7417 fixed crc bug in rare timing conditions within bench.c 2016-08-21 01:04:46 +02:00
Yann Collet
c411902230 fixed g++ conversion warning 2016-08-17 01:50:54 +02:00
Yann Collet
53e17fbd5e updated streaming API 2016-08-17 01:39:22 +02:00
Yann Collet
104e5b072d added : streaming decompression API 2016-08-16 15:11:28 +02:00
Yann Collet
5a0c8e2439 new streaming API (compression) 2016-08-16 15:11:27 +02:00
inikep
5f49eba512 added usage of rep[0]-1 for the optimal parser 2016-08-10 15:01:53 +02:00
inikep
98e08cbe34 fixed: tree not updated after finding very long rep matches 2016-08-10 15:00:30 +02:00
Yann Collet
280f9a8754 minor comment 2016-08-08 00:44:00 +02:00
Yann Collet
0763905f44 ZSTD_compress_usingCDict() correctly provides original size by default in frame header 
Fixed dictionary examples
 2016-08-03 01:57:57 +02:00
Yann Collet
346efccc35 fixed doc typo 2016-08-02 14:26:00 +02:00
Yann Collet
c55eb18c11 Merge pull request #267 from inikep/dev08 
fixed ZSTD_compressBlock_opt_extDict_generic
 2016-07-31 22:00:16 +02:00
inikep
056df510aa fixed ZSTD_compressBlock_opt_extDict_generic 2016-07-31 20:08:53 +02:00
Yann Collet
917fe188f1 Implemented repOffset "minus 1" on ll==0 2016-07-31 04:01:57 +02:00
Yann Collet
3b2bd1d11c zstd_opt uses same tables as zstd_compress 2016-07-30 13:21:41 +02:00
Yann Collet
3c6b808870 minor decompression speed gains 2016-07-30 03:20:47 +02:00
Yann Collet
c0ce4f1211 slightly improved compression speed 2016-07-30 00:55:13 +02:00
Yann Collet
ed57d8530a new seqStore 2016-07-29 21:22:17 +02:00
Yann Collet
c00d30fbe4 Merge pull request #264 from inikep/dev08 
Dev08
 2016-07-29 17:42:30 +02:00
inikep
6b68ba2079 zstd_opt.h: fixed checking of rep codes (2) 2016-07-29 16:45:39 +02:00
inikep
59b86fc141 zstd_opt.h: fixed checking of rep codes 2016-07-29 11:00:33 +02:00
Yann Collet
60ba31c570 zbuff uses ZSTD_compressEnd() 2016-07-28 19:55:09 +02:00
Yann Collet
16e73033ad introduced stage zbf_end 2016-07-28 16:32:34 +02:00
Yann Collet
62470b4bab Changed ZSTD_compressEnd() 2016-07-28 15:29:08 +02:00
Yann Collet
19c1002e46 applied ZSTD_compressContinueThenEnd() 2016-07-28 01:25:46 +02:00
Yann Collet
5b56739b63 created ZSTD_compressContinueThenEnd() 2016-07-28 01:17:22 +02:00
Yann Collet
c991cc1828 new frame end, 32-bits checksums 2016-07-28 00:55:43 +02:00
Yann Collet
d4180cad9c minor code refactoring 2016-07-27 21:21:36 +02:00
Yann Collet
731ef16fc1 minor code style refactoring 2016-07-27 21:05:12 +02:00
Yann Collet
4110534886 ZSTD_maxCLevel() is promoted to "stable" API (#254, by @FrancescAlted) 2016-07-27 15:09:11 +02:00
Yann Collet
c154d9d6a2 better support for large dictionaries (> 128 KB) 2016-07-27 14:37:00 +02:00
inikep
003c7a8568 optimal parser: removed ZSTD_REP_INIT 2016-07-27 11:07:13 +02:00
Eric Biggers
e4d0265ea9 Replace remaining references to "direct mode" with "single segment mode" 2016-07-26 13:22:27 -07:00
Yann Collet
38b75ddeb2 removed special case all-1 huffman distribution 2016-07-24 15:35:59 +02:00
Yann Collet
7ed5e33b89 minor comment changes 2016-07-24 14:26:11 +02:00
Yann Collet
f8e7b5363f unified encoding types 2016-07-23 16:31:49 +02:00
Yann Collet
c2e1a68d81 changed streamNb order to 1-4-4-4 2016-07-22 17:30:52 +02:00
Yann Collet
32faf6c8e7 fixed conversion warnings 2016-07-22 14:37:09 +02:00
Yann Collet
198e6aac44 Literals header fields use little endian convention 2016-07-22 14:37:09 +02:00
Yann Collet
6fa05a2371 cBlockSize uses little-endian convention 2016-07-22 14:37:09 +02:00
Yann Collet
5894ea8d01 updated cLevels 2016-07-22 14:36:46 +02:00
Yann Collet
d5c5a77990 minor comments clarifications 2016-07-20 13:35:14 +02:00
Yann Collet
cf05b9d477 ZSTD_getBlockSizeMax() 2016-07-18 16:52:10 +02:00
Yann Collet
e557fd5e92 minor compression level corrections 2016-07-17 16:21:37 +02:00
Yann Collet
6cacd34d44 minor formatting changes 2016-07-15 17:58:13 +02:00
Yann Collet
98c8884999 added target zstd in root Makefile 2016-07-15 16:12:38 +02:00
Yann Collet
961b6a0e34 ZSTD_compressBlock() limits block size depending on windowLog parameter 2016-07-15 11:58:49 +02:00
Yann Collet
227cc39e15 improved efficiency for large messages with small dictionaries 2016-07-15 11:27:09 +02:00
Yann Collet
ea2ecdc315 fixed issue with small dictionary 2016-07-14 23:27:31 +02:00
Yann Collet
b23e1ce319 removed debugging traces 2016-07-14 17:46:38 +02:00
Yann Collet
8847238cac simplified ZSTD_estimateCCtxSize() 2016-07-14 17:05:38 +02:00
Yann Collet
69c2cdb45c fixed conversion warning 2016-07-14 16:52:45 +02:00
Yann Collet
5e80dd3261 fixed minor coverity warnings 2016-07-13 19:21:57 +02:00
Yann Collet
2b1a3638e6 changed macro name to ZSTDCLI_CLEVEL_DEFAULT 2016-07-13 15:16:00 +02:00
Yann Collet
3c242e79d3 updated compression levels table 2016-07-13 14:56:24 +02:00
Yann Collet
eed2081e55 fixed conversion warning 2016-07-12 15:11:40 +02:00
Yann Collet
a43a854cdb updated paramgrill 2016-07-12 13:42:10 +02:00
Yann Collet
73d74a05b9 fixed dfast strategy 2016-07-12 13:03:48 +02:00
Yann Collet
45dc35628c first version of doubleFast 2016-07-12 09:47:31 +02:00
Yann Collet
3ae543ce75 added ZSTD_estimateCCtxSize() 2016-07-11 03:12:17 +02:00
Yann Collet
e09d38e921 removed mem.h dependency from zbuff.h (experimental section) 2016-07-07 13:17:37 +02:00
Yann Collet
52c04fe58f removed mem.h dependency from zstd.h (experimental section) 2016-07-07 11:53:18 +02:00
Yann Collet
d57dffbe76 ZSTD_storeSeq takes an U32 as offset type 2016-07-03 01:48:26 +02:00
Yann Collet
302ff036f6 simplified repcodes for lazy_extDict 2016-07-03 01:28:16 +02:00
Yann Collet
9634f67107 fix lazy parser 2016-07-03 01:23:58 +02:00
Yann Collet
92d75667e4 fix for fast mode 2016-07-03 01:10:53 +02:00
Yann Collet
5e734ad09b revert fix 2016-07-02 23:55:34 +02:00
Yann Collet
0d5bf8f06f fixed risk of segfault on very large files (multiple GB) 2016-07-02 21:39:47 +02:00
Yann Collet
2fa9904844 update specification and comments 2016-07-01 20:55:28 +02:00
Yann Collet
c093208ab8 fix : potential leak (#229) 2016-06-30 14:07:30 +02:00
Yann Collet
6c6e1751f6 use ZSTD_getParams() to simplify code 2016-06-27 15:28:45 +02:00
Yann Collet
3d2cd7f816 Introduced ZSTD_getParams() 
bench now uses ZSTD_createCDict_advanced()
 2016-06-27 15:12:26 +02:00
Yann Collet
d4f4e58ee1 fixed ZSTD_decompressBlock() using multiple blocks 2016-06-27 01:31:35 +02:00
Yann Collet
3755eb8fea fixed strict-aliasing warning on gcc6 2016-06-22 13:15:53 +02:00
Yann Collet
391a128794 fix : segfault in command line during automatic overwrite protection mode 2016-06-21 17:06:25 +02:00
Yann Collet
bda68c253b refactored ZBUFF_compressEnd() for better maintainability 2016-06-21 15:18:11 +02:00
Yann Collet
aa29226b7c fix : ZBUFF_compressEnd() gives right amount remaining to flush, including future epilogue 2016-06-21 14:04:57 +02:00
Yann Collet
f15c1cb00c Fixed : ZBUFF_compressEnd() called multiple times with too small dst buffer (#206) 2016-06-21 13:11:48 +02:00
Yann Collet
22d76322ce minor refactor 2016-06-21 08:01:51 +02:00
Yann Collet
a436a529bc minor : fast_extDict does no longer skip first byte 2016-06-20 23:34:04 +02:00
Yann Collet
4623d11571 new correction, less extreme replacement value 2016-06-20 19:15:37 +02:00
Yann Collet
5477cc25f7 fixed corruption error related to inter-blocks rep-offset 2016-06-20 18:31:25 +02:00
Yann Collet
06d9a73b48 minor refactor, using WILDCOPY_OVERLENGTH macro instead of hard-coded 8 2016-06-19 14:27:21 +02:00
Yann Collet
19cab46f2f Joined seqStore initialization at dispatch point 2016-06-17 12:54:52 +02:00
Yann Collet
23ba41533a Fixed zstd_opt encoding error with repeat-offsets 2016-06-16 13:20:46 +02:00
Yann Collet
736d419289 strengthened dict loading on decompresson side 2016-06-16 01:05:04 +02:00
Yann Collet
52a0622beb RepsCodes are saved into Dict 
(uncomplete : need decompression to regenerate them)
 2016-06-16 01:05:04 +02:00
Yann Collet
efd0b4993a fixed fuzzer error (inter-block repeated offsets) 2016-06-16 00:53:56 +02:00
Yann Collet
d059092897 fixed conversion warnings 2016-06-14 15:34:24 +02:00
Yann Collet
45c03c564f fixed corruption with inter-blocks repeated offsets 2016-06-14 13:46:11 +02:00
Yann Collet
4266c0a2fd adding inter-blocks rep-offsets 2016-06-14 01:49:25 +02:00
Yann Collet
43dfe01919 Check repIndex for validity 2016-06-13 21:43:06 +02:00
Yann Collet
9dd12742f3 litBlockType_t is an enum 2016-06-10 00:12:26 +02:00
Yann Collet
302fb53a76 Removed ZSTD_*_usingPrepared?Ctx() declaration from public space 2016-06-07 12:16:49 +02:00