Nick Terrell
24701de877
Fix uninitialized memory read
2016-11-14 13:57:05 -08:00
Yann Collet
8e4901eccd
removed zbuff.h from include installation
2016-11-08 15:45:39 -08:00
Yann Collet
fd3be6bc97
bump version number to 1.1.2
2016-11-07 14:35:41 -08:00
Nick Terrell
dc904ad17b
Fix bug in zstd v0.{5, 6} dictionary decompression
...
Introduced by bb68062c59
.
2016-11-04 16:18:59 -07:00
Przemyslaw Skibinski
38b590ad69
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
...
# Conflicts:
# lib/Makefile
2016-11-04 10:10:54 +01:00
Yann Collet
407a11f63e
fixed Visual compatibility
2016-11-03 15:52:01 -07:00
Yann Collet
11812260d1
Merge pull request #439 from terrelln/dev
...
ZSTD_compress_usingDict() specify dict == NULL
2016-11-03 14:15:36 -07:00
Nick Terrell
c8a9ac312b
Fix dynamic libzstd symlinks
2016-11-03 12:32:48 -07:00
Przemyslaw Skibinski
3a415594b1
fixed MinGW compilation
2016-11-03 12:59:20 +01:00
Yann Collet
7347869fb6
fixed make install
2016-11-02 22:28:37 -07:00
Nick Terrell
d82efd8a70
ZSTD_compress_usingDict() when dict gets loaded
...
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
2016-11-02 18:07:16 -07:00
Yann Collet
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
Yann Collet
f3f13211ae
Fix #419 : no warning when setting custom LDFLAGS
2016-11-02 17:02:45 -07:00
Yann Collet
0a5a5fb7fd
Fix #418 : printing selected segments in zdict debug mode can segfault with certain pathological patterns
2016-11-02 13:57:55 -07:00
Yann Collet
31e660e7aa
more accurate default maximum window size
2016-10-29 03:56:45 -07:00
Yann Collet
2115724c22
Merge pull request #430 from terrelln/exec-sequences
...
ZSTD_execSequence() accepts match in last 7 bytes
2016-10-28 10:45:05 -07:00
Nick Terrell
10bfd0c0d5
Fix ZSTD_execSequence() performance regression
...
Commit ae1cb3b3d0
caused the regression.
It is an instruction alignment issue, because if it is `U64 i` instead
of `U32 i`, the regression returns. This patch fixes the regression
in gcc, but only gets some of the clang performance back.
Benchmarks:
Run on `silesia.tar`. I only show levels 1-5 because the performance
regression was uniform across all levels. I did one run on levels
1-19 and it looked good.
| Build | Level | Before | While | After |
|-------|-------|-------:|------:|------:|
| gcc | 1 | 931.4 | 904.4 | 932.8 |
| gcc | 2 | 849.1 | 822.6 | 851.2 |
| gcc | 3 | 815.6 | 790.6 | 818.9 |
| gcc | 4 | 794.1 | 770.7 | 798.0 |
| gcc | 5 | 785.7 | 760.7 | 788.8 |
| clang | 1 | 705.5 | 683.2 | 693.8 |
| clang | 2 | 670.0 | 649.2 | 660.7 |
| clang | 3 | 659.6 | 639.8 | 651.4 |
| clang | 4 | 652.5 | 634.7 | 645.9 |
| clang | 5 | 646.9 | 625.5 | 637.7 |
2016-10-27 16:19:57 -07:00
Yann Collet
ee5b725823
ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used
2016-10-27 14:20:55 -07:00
Nick Terrell
eb7873a048
ZSTD_execSequence() accepts match in last 7 bytes
...
The zstd reference compressor will not emit a match in the last 7
bytes of a block. The decompressor will also not accept a match
in the last 7 bytes. This patch makes the decompressor accept a
match in the last 7 bytes.
2016-10-25 21:24:15 -07:00
Yann Collet
335ad5d4d4
added ZSTD_initDStream_usingDDict() .
...
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
2016-10-25 17:47:02 -07:00
Yann Collet
9516234e67
first sketch for ZSTD_initCStream_usingCDict()
2016-10-25 16:19:52 -07:00
Yann Collet
62d9a7ddfd
Merge pull request #429 from inikep/btopt2
...
Btopt2
2016-10-25 14:48:43 -07:00
Przemyslaw Skibinski
5c5f01f3da
added ZSTD_btopt2 strategy
2016-10-25 12:25:07 +02:00
Yann Collet
7b5948cca7
Merge pull request #426 from terrelln/fixes
...
Fix various {A, M}SAN bugs
2016-10-24 23:42:26 -07:00
Yann Collet
37d130031d
updated comments on context re-use
2016-10-24 17:22:12 -07:00
Nick Terrell
b2c39a22b0
Fix compiler narrowing warning
2016-10-24 14:50:13 -07:00
Nick Terrell
f698ad6deb
Merge remote-tracking branch 'upstream/dev' into fixes
...
* upstream/dev:
added doc\zstd_manual.html
added contrib\gen_html
zstd_compression_format.md moved to doc/
Fix small bug in ZSTD_execSequence()
improved ZSTD_compressBlock_opt_extDict_generic
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
zstd_opt.h: small improvement in compression ratio
improved dicitonary segment merge
use implicit rules to compile zstd_decompress.c
detect early impossible decompression scenario in legacy decoder v0.5
no repeat mode in legacy v0.5
fixed invalid invocation of dictionary in legacy decoder v0.5
fix edge case
fix command line interpretation
fixed minor corner case
zstd.h: added the Introduction section
fixed clang 3.5 warnings
zstd.h: updated comments
2016-10-24 13:10:13 -07:00
Yann Collet
4239a207dd
Merge pull request #425 from inikep/dev11
...
Doc
2016-10-24 11:11:40 -07:00
Nick Terrell
f9c9af3c2e
Reject dictionaries with incomplete entropy tables
...
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.
Ensure that dictionaries support all `matchLength`, and `literalLength`
codes. They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
2016-10-24 10:42:44 -07:00
Przemyslaw Skibinski
984b66cd72
added contrib\gen_html
2016-10-24 15:59:51 +02:00
Przemyslaw Skibinski
3ee94a7600
zstd_compression_format.md moved to doc/
2016-10-24 15:58:07 +02:00
Yann Collet
97611611a3
Merge pull request #423 from terrelln/exec-seq-patch
...
Fix small bug in ZSTD_execSequence()
2016-10-21 17:02:06 -07:00
Nick Terrell
ae1cb3b3d0
Fix small bug in ZSTD_execSequence()
...
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.
Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.
These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.
Introduced by commit 7158584399
2016-10-21 12:13:44 -07:00
Przemyslaw Skibinski
4732074a71
improved ZSTD_compressBlock_opt_extDict_generic
2016-10-21 11:19:00 +02:00
Yann Collet
da3bd8b6de
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
2016-10-20 20:11:00 -07:00
Przemyslaw Skibinski
d365ae3497
zstd_opt.h: small improvement in compression ratio
2016-10-20 11:49:02 +02:00
Przemyslaw Skibinski
575ab00db7
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-10-20 11:01:52 +02:00
Nick Terrell
d760529a05
Fix stack buffer overrun when weightTotal == 0
...
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
2016-10-19 11:39:11 -07:00
Nick Terrell
bb68062c59
Unitialized memory read in ZSTD_decodeSeqHeaders()
...
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
2016-10-18 16:41:33 -07:00
Yann Collet
52c1bf93fe
improved dicitonary segment merge
2016-10-18 16:34:58 -07:00
Nick Terrell
7b06ad7a05
Backport fix from commit 125d817
...
This fixes a read of unitialized memory.
Full commit hash: 125d81774f
.
2016-10-18 14:52:34 -07:00
Nick Terrell
f45b157d95
Backport fix from commit 9e8b09a
...
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
2016-10-18 14:22:49 -07:00
Yann Collet
f7906d5955
detect early impossible decompression scenario in legacy decoder v0.5
2016-10-18 13:48:32 -07:00
Yann Collet
9313c8d953
no repeat mode in legacy v0.5
2016-10-18 13:36:15 -07:00
Yann Collet
83d7bdee4b
fixed invalid invocation of dictionary in legacy decoder v0.5
2016-10-18 12:25:43 -07:00
Yann Collet
197a55ee7b
fix edge case
2016-10-18 11:27:52 -07:00
Nick Terrell
fd98087047
Fix stack buffer overflow in HUF_readCTable()
...
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
2016-10-17 18:16:59 -07:00
Yann Collet
06573e17be
fixed minor corner case
2016-10-17 17:28:28 -07:00
Nick Terrell
bfd943ace5
Fix buffer overrun in ZSTD_loadDictEntropyStats()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`. This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.
The benchmarks look good, there is no obvious compression performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 271.6 MB/s , 716.8 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 204.8 MB/s , 671.1 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.8 MB/s , 658.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 136.4 MB/s , 665.3 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.9 MB/s , 647.0 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 65.2 MB/s , 670.4 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.9 MB/s , 688.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 32.4 MB/s , 711.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.1 MB/s , 718.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 22.2 MB/s , 721.8 MB/s
> ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 273.8 MB/s , 722.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 203.2 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 157.4 MB/s , 666.5 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 132.1 MB/s , 661.9 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.8 MB/s , 641.6 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 63.1 MB/s , 677.0 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.3 MB/s , 678.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 33.1 MB/s , 708.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.5 MB/s , 710.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 21.9 MB/s , 723.9 MB/s
2016-10-17 16:55:52 -07:00
Nick Terrell
4db751668f
Fix buffer overrun in ZSTD_loadEntropy()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`. This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.
The benchmarks look good, there is no obvious performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 268.2 MB/s , 701.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 199.5 MB/s , 666.9 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 154.9 MB/s , 655.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 128.9 MB/s , 648.4 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.4 MB/s , 633.4 MB/s
> ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 266.1 MB/s , 703.7 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 199.0 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.2 MB/s , 656.2 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 133.2 MB/s , 647.4 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.3 MB/s , 633.3 MB/s
2016-10-17 15:51:15 -07:00