Commit Graph

1830 Commits

Author SHA1 Message Date
Nick Terrell
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet
586df82a78 Merge pull request from terrelln/static
[zstd] Backport kernel patch from @ColinIanKing
2017-09-25 17:02:40 -07:00
Yann Collet
5d8fdd1641 Merge pull request from terrelln/maxoff
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085 [zstd] Backport kernel patch from @ColinIanKing
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
2017-09-25 16:18:23 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Yann Collet
8977224b9b Merge pull request from terrelln/31
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
Yann Collet
da74aabc00 Merge pull request from terrelln/fse-optimal
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804 Merge pull request from terrelln/30
[bitstream] Allow adding 31 bits at a time
2017-09-19 14:25:10 -07:00
Nick Terrell
74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676 [ldm] Fix corner case where minMatch < 8
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff [libzstd] Fix bad window size assert
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
2017-09-19 13:47:59 -07:00
Yann Collet
cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream()
previous result was over-estimated
by counting streaming buffers twice
2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Yann Collet
72a80515ec Merge pull request from terrelln/fparams
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-18 13:48:31 -07:00
Yann Collet
539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Nick Terrell
5f22479517 [block] Don't use fParams in ZSTD_decompressBlock() 2017-09-15 17:37:20 -07:00
Yann Collet
77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet
335780c427 fixed too strong alignment assert in ZSTD_initStaticCCtx()
64-bits fields are only 32-bits aligned on 32-bits CPU
2017-09-13 16:35:29 -07:00
Yann Collet
f1571dad8f Merge pull request from stellamplau/ldm-mergeDev
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
4120a7fd5a Merge pull request from facebook/libzstd-nomt
makes it possible to compile libzstd in single-thread mode without zs…
2017-09-12 17:13:17 -07:00
Yann Collet
3306bcb0e6 fix : GCC v3.x 32-bits doesn't define 64-bits intrinsic
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
2017-09-11 15:17:31 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Stella Lau
f902bf9676 Merge branch 'ldm-integrate' into ldm-mergeDev 2017-09-11 14:55:29 -07:00
Yann Collet
f325ee4e84 fixed pass-through warning 2017-09-11 14:37:03 -07:00
Stella Lau
0d1b54db61 Explicitly cast raw numerals when left-shifting 2017-09-11 14:28:18 -07:00
Yann Collet
0d6ecc72a3 makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c () 2017-09-11 14:09:34 -07:00
Yann Collet
ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min()
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
360428c5d9 Move ldm functions to their own file 2017-09-06 18:09:26 -07:00
Yann Collet
baa37c3362 programs/Makefile : better support for GNU conventions
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
2017-09-06 16:53:59 -07:00
Yann Collet
3a12531a3d lib/Makefile : better support for GNU conventions
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
2017-09-06 16:35:49 -07:00
Yann Collet
1c7b914cdf update README on BUCK file 2017-09-06 16:23:39 -07:00
Yann Collet
36374cc3b4 update and clarify lib/README 2017-09-06 16:15:18 -07:00
Stella Lau
2b99d696de Remove debug code 2017-09-06 15:57:26 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Yann Collet
ad0046244f Merge pull request from terrelln/split-compress
Split parsers out of zstd_compress.c
2017-09-06 10:01:27 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
98b85426f1 Fix setting of nextToUpdate at end of ldm matcher 2017-09-05 20:41:37 -07:00
Nick Terrell
721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00
Stella Lau
08d33fe1c9 Fix parameter handling in copyCCtx with cdict 2017-09-05 15:50:20 -07:00
Stella Lau
fd0071da29 Fix parameter handling with ZSTD_copyCCtx 2017-09-05 15:34:17 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Nick Terrell
423b133568 [POOL] Allow free on NULL when multithreading is disabled 2017-09-05 11:18:13 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00