AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,315 Commits 1 Branch 0 Tags 27 MiB
d6770f80af
Commit Graph

1832 Commits

Author SHA1 Message Date
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
586df82a78 Merge pull request #862 from terrelln/static 
[zstd] Backport kernel patch from @ColinIanKing
 2017-09-25 17:02:40 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff 
[libzstd] Increase MaxOff
 2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085 [zstd] Backport kernel patch from @ColinIanKing 
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
 2017-09-25 16:18:23 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Yann Collet
8977224b9b Merge pull request #859 from terrelln/31 
Prepare for ZSTD_WINDOWLOG_MAX == 31
 2017-09-22 09:01:39 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Yann Collet
da74aabc00 Merge pull request #850 from terrelln/fse-optimal 
[fse] Fix FSE_optimalTableLog() for srcSize==1
 2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804 Merge pull request #849 from terrelln/30 
[bitstream] Allow adding 31 bits at a time
 2017-09-19 14:25:10 -07:00
Nick Terrell
74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676 [ldm] Fix corner case where minMatch < 8 
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
 2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff [libzstd] Fix bad window size assert 
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
 2017-09-19 13:47:59 -07:00
Yann Collet
cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() 
previous result was over-estimated
by counting streaming buffers twice
 2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Yann Collet
72a80515ec Merge pull request #848 from terrelln/fparams 
[block] Don't use fParams in ZSTD_decompressBlock()
 2017-09-18 13:48:31 -07:00
Yann Collet
539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Nick Terrell
5f22479517 [block] Don't use fParams in ZSTD_decompressBlock() 2017-09-15 17:37:20 -07:00
Yann Collet
77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet
335780c427 fixed too strong alignment assert in ZSTD_initStaticCCtx() 
64-bits fields are only 32-bits aligned on 32-bits CPU
 2017-09-13 16:35:29 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Yann Collet
4120a7fd5a Merge pull request #837 from facebook/libzstd-nomt 
makes it possible to compile libzstd in single-thread mode without zs…
 2017-09-12 17:13:17 -07:00
Yann Collet
3306bcb0e6 fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic 
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
 2017-09-11 15:17:31 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Stella Lau
f902bf9676 Merge branch 'ldm-integrate' into ldm-mergeDev 2017-09-11 14:55:29 -07:00
Yann Collet
f325ee4e84 fixed pass-through warning 2017-09-11 14:37:03 -07:00
Stella Lau
0d1b54db61 Explicitly cast raw numerals when left-shifting 2017-09-11 14:28:18 -07:00
Yann Collet
0d6ecc72a3 makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819) 2017-09-11 14:09:34 -07:00
Yann Collet
ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() 
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
 2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min() 
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
 2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Stella Lau
360428c5d9 Move ldm functions to their own file 2017-09-06 18:09:26 -07:00
Yann Collet
baa37c3362 programs/Makefile : better support for GNU conventions 
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
 2017-09-06 16:53:59 -07:00
Yann Collet
3a12531a3d lib/Makefile : better support for GNU conventions 
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
 2017-09-06 16:35:49 -07:00
Yann Collet
1c7b914cdf update README on BUCK file 2017-09-06 16:23:39 -07:00
Yann Collet
36374cc3b4 update and clarify lib/README 2017-09-06 16:15:18 -07:00
Stella Lau
2b99d696de Remove debug code 2017-09-06 15:57:26 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Yann Collet
ad0046244f Merge pull request #831 from terrelln/split-compress 
Split parsers out of zstd_compress.c
 2017-09-06 10:01:27 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau
98b85426f1 Fix setting of nextToUpdate at end of ldm matcher 2017-09-05 20:41:37 -07:00
Nick Terrell
721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00
Stella Lau
08d33fe1c9 Fix parameter handling in copyCCtx with cdict 2017-09-05 15:50:20 -07:00
Stella Lau
fd0071da29 Fix parameter handling with ZSTD_copyCCtx 2017-09-05 15:34:17 -07:00
Stella Lau
643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00