AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
8,095 Commits 1 Branch 0 Tags 27 MiB
ea92fb3a68
Commit Graph

126 Commits

Author SHA1 Message Date
Sean Purcell
ba2ad9f25c ZSTD_decompress now handles multiple frames 2017-02-08 14:50:10 -08:00
Sean Purcell
4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Yann Collet
b5fd15ccb2 fixed : legacy decoders v04 and v05 2017-01-30 10:45:58 -08:00
Yann Collet
cafdd31a38 fixed MSAN warnings in legacy decoders 
In some extraordinary circumstances,
*Length field can be generated from reading a partially uninitialized memory segment.
Data is correctly identified as corrupted later on,
but the read taints some later pointer arithmetic operation.
 2017-01-27 10:44:03 -08:00
Nick Terrell
8157a4c3cc Fix dictionary loading bug causing an MSAN failure 
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
 2016-12-20 10:47:52 -08:00
Yann Collet
35168679bd Merge pull request #478 from terrelln/wildcopy-ub 
Fix execSequence wildcopy undefined behavior
 2016-12-13 11:33:00 +01:00
Nick Terrell
064a143520 Fix execSequence wildcopy undefined behavior 
execSequence relied on pointer overflow to handle cases where
`sequence.matchLength < 8`.  Instead of passing an `size_t` to
wildcopy, pass a `ptrdiff_t`.
 2016-12-12 19:01:23 -08:00
Nick Terrell
e474aa55b4 Fix decompression buffer overrun 
Allows an adversary to write up to 3 bytes beyond the end of the buffer.
Occurs if the match overlaps the `extDict` and `currentPrefix`, and the
match length in the `currentPrefix` is less than `MINMATCH`, and
`op-(16-MINMATCH) >= oMatchEnd > op-16`.
 2016-12-12 18:05:30 -08:00
Nick Terrell
4359d21ad7 Merge two memset() calls into one 2016-11-14 17:52:51 -08:00
Nick Terrell
24701de877 Fix uninitialized memory read 2016-11-14 13:57:05 -08:00
Nick Terrell
dc904ad17b Fix bug in zstd v0.{5, 6} dictionary decompression 
Introduced by bb68062c59.
 2016-11-04 16:18:59 -07:00
Nick Terrell
f698ad6deb Merge remote-tracking branch 'upstream/dev' into fixes 
* upstream/dev:
  added doc\zstd_manual.html
  added contrib\gen_html
  zstd_compression_format.md moved to doc/
  Fix small bug in ZSTD_execSequence()
  improved ZSTD_compressBlock_opt_extDict_generic
  protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
  zstd_opt.h: small improvement in compression ratio
  improved dicitonary segment merge
  use implicit rules to compile zstd_decompress.c
  detect early impossible decompression scenario in legacy decoder v0.5
  no repeat mode in legacy v0.5
  fixed invalid invocation of dictionary in legacy decoder v0.5
  fix edge case
  fix command line interpretation
  fixed minor corner case
  zstd.h: added the Introduction section
  fixed clang 3.5 warnings
  zstd.h: updated comments
 2016-10-24 13:10:13 -07:00
Nick Terrell
ae1cb3b3d0 Fix small bug in ZSTD_execSequence() 
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.

Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.

These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.

Introduced by commit 7158584399
 2016-10-21 12:13:44 -07:00
Nick Terrell
d760529a05 Fix stack buffer overrun when weightTotal == 0 
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
 2016-10-19 11:39:11 -07:00
Nick Terrell
bb68062c59 Unitialized memory read in ZSTD_decodeSeqHeaders() 
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
 2016-10-18 16:41:33 -07:00
Nick Terrell
7b06ad7a05 Backport fix from commit 125d817 
This fixes a read of unitialized memory.
Full commit hash: 125d81774f.
 2016-10-18 14:52:34 -07:00
Nick Terrell
f45b157d95 Backport fix from commit 9e8b09a 
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
 2016-10-18 14:22:49 -07:00
Yann Collet
f7906d5955 detect early impossible decompression scenario in legacy decoder v0.5 2016-10-18 13:48:32 -07:00
Yann Collet
9313c8d953 no repeat mode in legacy v0.5 2016-10-18 13:36:15 -07:00
Yann Collet
83d7bdee4b fixed invalid invocation of dictionary in legacy decoder v0.5 2016-10-18 12:25:43 -07:00
Nick Terrell
4db751668f Fix buffer overrun in ZSTD_loadEntropy() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`.  This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.

The benchmarks look good, there is no obvious performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 268.2 MB/s , 701.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.5 MB/s , 666.9 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 154.9 MB/s , 655.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 128.9 MB/s , 648.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.4 MB/s , 633.4 MB/s

  > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 266.1 MB/s , 703.7 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.0 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.2 MB/s , 656.2 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 133.2 MB/s , 647.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.3 MB/s , 633.3 MB/s
 2016-10-17 15:51:15 -07:00
Nick Terrell
ccfcc643da Check if dict is empty before reading first byte 2016-10-17 11:46:03 -07:00
Nick Terrell
7158584399 Fix ZSTD_execSequence() edge case 2016-10-12 10:05:26 -07:00
Christophe Chevalier
dc245e91cb Changed to use ZSTDLIBv06_API and ZSTDLIBv07_API for DLL exports to fix warning 
- changed name to prevent collision with ZSTDLIB_API used by non-legacy dll exports
 2016-09-23 17:09:36 +02:00
inikep
45db83f98d ZSTD_decodeLiteralsBlock renamed to ZSTDv01_decodeLiteralsBlock 2016-09-05 14:46:24 +02:00
inikep
476964f6a1 ZSTD_decodeSeqHeaders renamed to ZSTDv01_decodeSeqHeaders 2016-09-05 13:34:57 +02:00
inikep
c13faa1b0f legacy decoders: restored #include <intrin.h> for VC++ 2016-09-05 13:25:07 +02:00
inikep
8161e7321a unified error codes for legacy decoders 2016-09-05 12:29:51 +02:00
Yann Collet
1563bfeabc fixing FORCE_INLINE for older compilers (#330) 2016-09-02 11:44:21 -07:00
Yann Collet
4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
Yann Collet
5f53b0335e fixed continuation context 2016-08-28 10:00:49 -07:00
Yann Collet
767d8f66fa legacy contexts can be re-used 2016-08-28 08:19:47 -07:00
Yann Collet
4bf317dd00 first version supporting legacy streams (transparent decoding) 2016-08-28 07:43:34 -07:00
Yann Collet
9a021c1aae fixed some minor clang warnings 2016-08-26 09:05:06 +02:00
Yann Collet
87c18b2ebd fixed multiple minor warnings for XCode 2016-08-26 01:43:47 +02:00
inikep
57ef4b1a0d zstd_v07.c: removed unused macros 2016-08-24 17:16:56 +02:00
inikep
a7bb322a93 removed never referenced functions 2016-08-18 10:30:21 +02:00
inikep
48849f86f0 fixed compilation with Intel Compiler with Windows 2016-08-10 14:26:35 +02:00
Yann Collet
7adc2328a3 fixed --test on zero-length files, reported by @amnilsson 2016-07-26 15:49:24 +02:00
inikep
4178f5c289 fixed gcc warning: always_inline function might not be inlinable 2016-07-25 21:17:45 +02:00
inikep
fca90f8f60 legacy decoder for v0.7 format 2016-07-25 17:49:08 +02:00
Yann Collet
d5c5a77990 minor comments clarifications 2016-07-20 13:35:14 +02:00
Yann Collet
572b817be3 Merge pull request #253 from gymdis/heapmode_off_legacy_fix 
Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0
 2016-07-19 13:52:03 +02:00
Christopher Bergqvist
780a9fa857 Fix compile issue with ZSTD_LEGACY_SUPPORT=1 and ZSTD_HEAPMODE=0 2016-07-19 13:25:38 +02:00
luben karavelov
10f999f856 Add legacy support for the low-level streaming API 2016-07-17 01:03:26 +02:00
Yann Collet
5e80dd3261 fixed minor coverity warnings 2016-07-13 19:21:57 +02:00
Yann Collet
3c174f4da9 fixed minor coverity warning 2016-07-13 17:25:53 +02:00
Yann Collet
19c27d27f1 simplified legacy functions, no longer need magic number 2016-07-07 14:40:13 +02:00
Yann Collet
f323bf7d32 added : ZSTD_getDecompressedSize() 2016-07-07 13:14:21 +02:00
Yann Collet
289bbd52e5 Updated huff0 2016-06-11 01:31:54 +02:00
inikep
4923222412 fixed warnings from Travis 2016-06-09 20:03:30 +02:00
inikep
4000945a1d project updated for legacy decoder zstd_v06.c 2016-06-09 18:12:06 +02:00
inikep
bf853d5510 added legacy decoder for v0.6 format 2016-06-09 17:59:18 +02:00
Yann Collet
ea4daa308d added ZSTDv05_getFrameParams() to "zstd_v05.h" 2016-05-17 12:01:55 +02:00
Yann Collet
3329d98df9 fixed minor asan warning in legacy decoder 2016-05-11 11:07:54 +02:00
Yann Collet
f6ca09b5ff Reduced console display on loading lots of files with zstd --train. Reported by @KrzysFR, see #177 2016-05-09 04:44:45 +02:00
Yann Collet
e1a69a88c0 Expose Advanced Streaming API for v0.5.x legacy 2016-05-08 09:13:40 +02:00
Yann Collet
18dedece91 Support legacy ZSTD_decompress_usingDict() (starting v0.5+) 2016-05-06 16:43:23 +02:00
Yann Collet
8283a2f0aa Fixed decoding corruption error with quad-symbols huffman on legacy decoders 2016-05-06 01:51:31 +02:00
Yann Collet
ddb8ebd5b3 Stricter gnu90 tests 2016-05-05 04:59:53 +02:00
inikep
ea4ee3eee0 added compatibility with Visual C++ 2012 2016-04-25 13:09:06 +02:00
inikep
23a0889301 separation of lib/ into common/, compress/, decompress/, dictBuilder/, legacy/ 2016-04-22 12:43:18 +02:00
Yann Collet
9990986bdc fixed minor warnings when compiling with gcc -g (aka -std=gnu90) 2016-04-09 16:17:18 +02:00
Yann Collet
029267ab3f Integrated Legacy v0.5.x decoder (provided by @inikep) 2016-04-09 09:42:27 +02:00
Yann Collet
4488661678 minor cosmetics 2016-02-11 04:17:50 +01:00
Yann Collet
6bff748e43 Fixed minor warnings in legacy modes 2016-02-09 17:55:01 +01:00
Yann Collet
464fa996e9 added v0.4 legacy support 2016-02-03 01:09:46 +01:00
Yann Collet
ffec740d37 fixed visual and clang errors 2016-01-21 15:50:11 +01:00
Yann Collet
494c786a3b fixed outstanding strict aliasing warnings 2016-01-06 12:54:02 +01:00
Yann Collet
1fdd82312d Fixed : strict aliasing warnings in legacy decoders 2016-01-06 12:35:42 +01:00
Yann Collet
29a2c838c5 support for legacy 0.3 format 2015-11-26 16:02:04 +01:00
Yann Collet
c2ce890ac2 fixed bug in legacy decoder v0.2, reported by Maciej Adamczyk 2015-11-26 14:12:23 +01:00
Yann Collet
5054ee0cc0 rollbuffer refactor 2015-11-23 13:34:21 +01:00
Yann Collet
ec43ba4a95 updated visual studio projects 2015-10-30 11:51:26 +01:00
Yann Collet
aa074051d6 decoding support for legacy v0.2 format 2015-10-30 11:21:50 +01:00
Yann Collet
b1f3f4b565 update huff0 2015-10-18 22:18:32 +01:00