Commit Graph

548 Commits

Author SHA1 Message Date
Nick Terrell
471aa385b3 [fuzz] Speed up round trip tests
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
2017-09-26 14:03:43 -07:00
Yann Collet
8a4d0abc1c Merge pull request from terrelln/zip
[fuzz][CI] Set up regression tests
2017-09-25 17:11:35 -07:00
Yann Collet
5d8fdd1641 Merge pull request from terrelln/maxoff
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
Nick Terrell
917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell
11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell
6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell
bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell
23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell
1c23b64049 [fuzz] fuzz.py can minimize and zip corpora
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
2017-09-25 12:04:12 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
Nick Terrell
1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Yann Collet
31829cb057 Merge pull request from terrelln/fuzzer
[fuzzer] Fuzz long range matching & new API
2017-09-15 12:09:00 -07:00
Nick Terrell
39357c41cb [fuzzer] Fuzz long range matching & new API 2017-09-14 14:48:08 -07:00
Yann Collet
218c09e5b3 Merge pull request from terrelln/fuzzer
Fuzzer
2017-09-14 11:40:25 -07:00
Nick Terrell
9712d5ebe6 [fuzzer] Fix bugs in fuzz.py 2017-09-13 19:08:35 -07:00
Nick Terrell
a6f08b4783 [fuzzer] Fix FUZZ_seed() 2017-09-13 18:41:32 -07:00
Nick Terrell
6c6412cef9 [fuzzer] Update README.md 2017-09-13 18:23:52 -07:00
Nick Terrell
6b8236cf7e [fuzz] Add fuzzing helper script 2017-09-13 17:45:21 -07:00
Nick Terrell
b7e1522330 Add block fuzzers 2017-09-13 17:44:41 -07:00
Nick Terrell
def3214d74 [fuzzer] Handle single empty directory 2017-09-13 17:44:30 -07:00
Yann Collet
739b620814 Merge pull request from stellamplau/decodeCorpus-maxSize
Add flag to limit max decompressed size in decodeCorpus
2017-09-13 17:26:55 -07:00
Nick Terrell
8b6c80ada8 Update fuzzer Makefile 2017-09-13 16:16:57 -07:00
Nick Terrell
677c2cbf89 Update fuzzer sources 2017-09-13 16:16:57 -07:00
Stella Lau
963558a072 Fix implicit conversion error 2017-09-13 16:01:16 -07:00
Stella Lau
40bf0ced7d Add flag to limit max decompressed size in decodeCorpus 2017-09-13 15:16:56 -07:00
Yann Collet
f1571dad8f Merge pull request from stellamplau/ldm-mergeDev
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
be1f2dac5b Merge pull request from facebook/utilTimeAPI
modified util::time API (T19505791)
2017-09-13 11:41:01 -07:00
Yann Collet
a1bc08834f Merge pull request from stellamplau/decodeCorpus-blocks
Make decodecorpus generate raw compressed blocks
2017-09-13 09:34:04 -07:00
Yann Collet
c95c0c9725 modified util::time API
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
2017-09-12 18:12:46 -07:00
Stella Lau
e89065506e Make decodecorpus generate raw compressed blocks 2017-09-12 17:18:45 -07:00
Stella Lau
3d8e313f64 Reduce ldm hash table size in test 2017-09-11 17:21:28 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min()
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
af4068a697 Fix function name in tests/fuzz/regression_driver 2017-09-05 22:14:41 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Stella Lau
767a0b3be1 Move ldm hashLog, bucketLog, and mml to cctxParams 2017-09-01 12:24:59 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Eiichi Tsukata
7492e7f1c7 tests/fuzz: change ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
ZSTD_BLOCKSIZE_ABSOLUTEMAX is changed at the commit:
fa3671eac7
2017-09-01 16:37:39 +09:00
Eiichi Tsukata
6639395979 tests/fuzz: fix make all target names 2017-09-01 16:32:40 +09:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Yann Collet
e0cecd8736 fixed poolTests
needs more dependencies from zstd for custom allocators and error codes
2017-08-31 15:13:31 -07:00
Stella Lau
6a546efb8c Add long distance matcher
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
Yann Collet
b0cb081dc8 last batch of header files changed to reflect new license ()
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00