AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,315 Commits 1 Branch 0 Tags 27 MiB
d6770f80af
Commit Graph

4315 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9fe50ed623 fixed maximum windowLog for zstreamtest --newapi 
for compatibility with low memory VM
 2017-09-28 01:42:06 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet
aa800c4793 reduced memory usage of zstreamtest --newapi 
to run on memory-constrained VM
 2017-09-27 18:00:15 -07:00
Yann Collet
bc32b40b98 reduced zstreamtest --mt memory load 
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
 2017-09-27 17:27:38 -07:00
Yann Collet
02502191e5 separated --long tests between short and long tests 
A fast subset of these tests is now played in short test mode
 2017-09-27 15:48:06 -07:00
Yann Collet
f9de54acfb reduced memory requirements for --long tests 
in new --long test section
 2017-09-27 15:38:27 -07:00
Yann Collet
bdd0f6f046 improved make clean in tests/fuzz 2017-09-27 15:20:08 -07:00
Yann Collet
60059df051 shorter make test to avoid time out on travis CI 
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
 2017-09-27 15:16:27 -07:00
Yann Collet
495b5269ea Merge pull request #870 from facebook/cpptest 
Change c++ test to use CXX and CXXFLAGS environment variables
 2017-09-27 13:58:12 -07:00
Yann Collet
60ca44b545 switched name to cxxtest 2017-09-27 12:24:13 -07:00
Yann Collet
763f8b5e45 Change c++ test to use CXX and CXXFLAGS environment variables 
Fix OS-X warning on compiling .c files with clang++
Also changed test name from gpptest to cpptest, since it's no longer g++ specific
 2017-09-27 12:09:52 -07:00
Yann Collet
91ffe71ec8 Merge pull request #864 from terrelln/long 
Increase maximum window size
 2017-09-26 16:26:29 -07:00
Yann Collet
bfbec3c1ae Merge pull request #867 from terrelln/fast-fuzz 
[fuzz] Speed up round trip tests
 2017-09-26 14:57:24 -07:00
Nick Terrell
471aa385b3 [fuzz] Speed up round trip tests 
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
 2017-09-26 14:03:43 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
8a4d0abc1c Merge pull request #861 from terrelln/zip 
[fuzz][CI] Set up regression tests
 2017-09-25 17:11:35 -07:00
Yann Collet
586df82a78 Merge pull request #862 from terrelln/static 
[zstd] Backport kernel patch from @ColinIanKing
 2017-09-25 17:02:40 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff 
[libzstd] Increase MaxOff
 2017-09-25 16:34:29 -07:00
Nick Terrell
76cb38d085 [zstd] Backport kernel patch from @ColinIanKing 
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
 2017-09-25 16:18:23 -07:00
Nick Terrell
917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell
77d5bc2d62 [fuzz][CI] Add regression tests to the CI 2017-09-25 15:32:31 -07:00
Nick Terrell
11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell
6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell
bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell
23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell
1c23b64049 [fuzz] fuzz.py can minimize and zip corpora 
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
 2017-09-25 12:04:12 -07:00
Yann Collet
8977224b9b Merge pull request #859 from terrelln/31 
Prepare for ZSTD_WINDOWLOG_MAX == 31
 2017-09-22 09:01:39 -07:00
Yann Collet
14946af10c Merge pull request #858 from facebook/controlSize 
added control of frame content size at end of decompression
 2017-09-21 17:24:43 -07:00
Nick Terrell
d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Yann Collet
da74aabc00 Merge pull request #850 from terrelln/fse-optimal 
[fse] Fix FSE_optimalTableLog() for srcSize==1
 2017-09-19 14:59:21 -07:00
Yann Collet
c399ab4804 Merge pull request #849 from terrelln/30 
[bitstream] Allow adding 31 bits at a time
 2017-09-19 14:25:10 -07:00
Yann Collet
a361e9af08 Merge pull request #857 from terrelln/bad-assert 
Fix minor fuzz test failures
 2017-09-19 14:22:08 -07:00
Nick Terrell
74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell
6c9ed76676 [ldm] Fix corner case where minMatch < 8 
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
 2017-09-19 13:49:37 -07:00
Nick Terrell
18442a31ff [libzstd] Fix bad window size assert 
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
 2017-09-19 13:47:59 -07:00
Yann Collet
bdc83f523b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 15:50:14 -07:00
Yann Collet
9c1908a3cd added streaming_memory_usage example 2017-09-18 15:49:59 -07:00
Yann Collet
b0c0e3a3fb Merge pull request #853 from terrelln/blog 
[zstdcli] Fix LDM advanced options parsing
 2017-09-18 15:21:23 -07:00
Yann Collet
79e8ae61a7 Merge pull request #852 from facebook/fix851 
fix #851 : sudo zstd -t file.zst changes /dev/null permissions
 2017-09-18 15:10:02 -07:00
Nick Terrell
1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Yann Collet
cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet
7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() 
previous result was over-estimated
by counting streaming buffers twice
 2017-09-18 14:47:34 -07:00
Nick Terrell
cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Yann Collet
72a80515ec Merge pull request #848 from terrelln/fparams 
[block] Don't use fParams in ZSTD_decompressBlock()
 2017-09-18 13:48:31 -07:00
Yann Collet
92889709f9 fix #851 : sudo zstd -t file.zst changes /dev/null permissions 
reported by @mike155
 2017-09-18 13:41:54 -07:00
Yann Collet
539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Yann Collet
4a52a89026 Merge pull request #846 from facebook/splitDict 
Split dict
 2017-09-15 23:26:23 -07:00