Commit Graph

587 Commits

Author SHA1 Message Date
Yann Collet
241c57adff playtest : minor cosmetic changes 2017-10-16 14:01:42 -07:00
Yann Collet
43914f6a20 add test for presence of decompressed size field in header
previous test was "display" only, it wouldn't trigger an error in case of missing header.
2017-10-13 23:47:01 -07:00
Yann Collet
beb9b4b398 fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default
and a wrong test in zstreamtest --mt
2017-10-13 19:09:30 -07:00
Yann Collet
213ef3b510 fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
Yann Collet
3c1e3f8ec9 contentSizeFlag enabled by default would also fail for streaming and MT operations
fixed
2017-10-13 18:32:06 -07:00
Yann Collet
fb44516641 ensure fParams.contentSizeFlag starts at 1
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
2017-10-13 17:39:13 -07:00
Yann Collet
7f6a783862 fixed a small error in decodeCorpus
a compressed block must be strictly smaller than its decompressed size.
2017-10-07 15:19:52 -07:00
Yann Collet
4252621e26 playtests: do not use cat on large files
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
2017-10-05 20:21:59 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
Yann Collet
e963800e27 zstdmt : fixed : buffer dst0 wasn't properly set to null after usage
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
2017-09-28 23:01:31 -07:00
Yann Collet
e0065cf660 make test : removed zstreamtest unit tests for variants
slightly reduced time to create dictionary
at beginning of unit tests
2017-09-28 18:34:38 -07:00
Yann Collet
47c6a95d07 zstreamtest : run unit tests only during "normal" session
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
2017-09-28 18:27:22 -07:00
W. Felix Handte
d0519d4b0c Add CLI Program Name Detection for LZ4 2017-09-28 19:18:15 -04:00
W. Felix Handte
5705d9f25a Add basic tests for the lz4 integration 2017-09-28 19:16:43 -04:00
Yann Collet
e4ec427720 Merge branch 'dev' into shorterTests
fixed conflicts
2017-09-28 12:19:28 -07:00
Yann Collet
bbef058ae6 zstreamtest --newapi : reduced maximum allocated memory 2017-09-28 11:48:45 -07:00
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9fe50ed623 fixed maximum windowLog for zstreamtest --newapi
for compatibility with low memory VM
2017-09-28 01:42:06 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
Yann Collet
824f75ea7c Merge pull request #863 from facebook/newFormats
magicless frames (#591)
2017-09-28 00:32:16 -07:00
Nick Terrell
d9c1e9125f [fuzz] Small changes for oss-fuzz integration 2017-09-27 18:23:06 -07:00
Yann Collet
aa800c4793 reduced memory usage of zstreamtest --newapi
to run on memory-constrained VM
2017-09-27 18:00:15 -07:00
Yann Collet
bc32b40b98 reduced zstreamtest --mt memory load
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
2017-09-27 17:27:38 -07:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
Yann Collet
02502191e5 separated --long tests between short and long tests
A fast subset of these tests is now played in short test mode
2017-09-27 15:48:06 -07:00
Yann Collet
f9de54acfb reduced memory requirements for --long tests
in new --long test section
2017-09-27 15:38:27 -07:00
Yann Collet
bdd0f6f046 improved make clean in tests/fuzz 2017-09-27 15:20:08 -07:00
Yann Collet
60059df051 shorter make test to avoid time out on travis CI
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
2017-09-27 15:16:27 -07:00
Yann Collet
ea1f50bf73 removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.

Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.

A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
Yann Collet
bfabd1d4dc fixed zstreamtest decoding error
same error (wrong output buffer size) was present
on --mt and --new_api tests.
2017-09-27 01:01:11 -07:00
Yann Collet
ca306c1c84 fixed a bug in zstreamtest
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
Yann Collet
4791561c4a silence minor gcc warning -Wempty-body
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
Yann Collet
91ffe71ec8 Merge pull request #864 from terrelln/long
Increase maximum window size
2017-09-26 16:26:29 -07:00
Yann Collet
df4e9bba25 fixed constant errors for gcc in c99 mode
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
Yann Collet
9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Nick Terrell
471aa385b3 [fuzz] Speed up round trip tests
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
2017-09-26 14:03:43 -07:00
Nick Terrell
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet
8a4d0abc1c Merge pull request #861 from terrelln/zip
[fuzz][CI] Set up regression tests
2017-09-25 17:11:35 -07:00
Yann Collet
52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
Nick Terrell
917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell
11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell
6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames
decoder not implemented yet
2017-09-25 14:26:26 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell
bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell
23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell
1c23b64049 [fuzz] fuzz.py can minimize and zip corpora
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
2017-09-25 12:04:12 -07:00
Yann Collet
96f0cde31a minor function rename
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00