Commit Graph

623 Commits

Author SHA1 Message Date
Yann Collet
311878dec3 Improved tests
- building cli from /tests preserves potential flags in MOREFLAGS (such as asan/usan)
- MT dictionary tests check for MT capability (MT is not enabled by default for zstd32)
2017-12-13 11:48:30 -08:00
Yann Collet
dba8016d2d Merge branch 'dev' into fix944 2017-12-13 11:20:09 -08:00
Yann Collet
04a1557e28 fuzzer tests : added -v command
so that unit tests get listed in CI console
and we can see which one failed.
2017-12-13 10:08:52 -08:00
Yann Collet
98ccc115ed added cli test veryfying bug #944
this test fails on v1.3.2
2017-12-12 19:47:38 -08:00
Yann Collet
e9dc0ab884 zstd binary, when compiled from /tests, enables debug mode
adds ZSTD_DEBUG=1 and -g
2017-12-12 18:20:04 -08:00
Yann Collet
e28305fcca fix #944 : ZSTDMT with large files and dictionary now works correctly
windowLog is now enforced from provided compression parameters,
instead of being copied blindly from `cdict`
where it could be smaller.

also :
- fix a minor bug in zstreamtest --mt : advanced parameters must be set before init
- changed advanced parameter name to ZSTDMT_jobSize
2017-12-12 18:04:58 -08:00
Yann Collet
03832b7aa5 re-added test case
messing with revert ... :(
2017-12-12 14:01:54 -08:00
Yann Collet
8a104fda05 Revert "Created a test case which reliably reproduces bug #944"
This reverts commit 5098d1fbe2.
2017-12-12 12:51:49 -08:00
Yann Collet
5098d1fbe2 Created a test case which reliably reproduces bug #944
in zstreamtest.
2017-12-12 12:48:31 -08:00
Yann Collet
21f7672d43 zstreamtest : add test which detects when ZSTD_initCStream() inflate data
as in #942, when ZSTD_initCStream() believes data is empty, instead of unknown.
2017-12-07 03:06:01 -05:00
Yann Collet
7e05ef851a Merge branch 'dev' into qemu32panic 2017-12-03 11:14:36 -08:00
Yann Collet
a1b24e6262
Merge pull request #938 from terrelln/time
Use util.h for timing
2017-12-01 16:40:38 -08:00
Yann Collet
6fc63f521f Merge branch 'dev' into qemu32panic 2017-12-01 10:51:01 -08:00
Yann Collet
453fb5e830 removed -ftrapv from tests/ debug flags
-ftrapv is apparently buggy for gcc.
versions >= 5 are supposed to work better,
but even then, some complaints say it's still flaky when optimizations are enabled.
I even saw a post saying it only works if one creates its own signal handler,
which would make this flag no longer transparent.

on clang, it seems to work correctly.
But we would need to add a method to selectively add flags depending on compiler.
That's too much troubles for the intended benefit
(just catch integer overflows, which we can also do using ubsan).
2017-12-01 10:30:53 -08:00
Nick Terrell
dab8cfa3c7 Combine definitions of SEC_TO_MICRO 2017-11-30 19:40:53 -08:00
Nick Terrell
9a2f6f477b Use util.h for timing 2017-11-30 14:57:25 -08:00
Yann Collet
d3c59edac9 removed long-range-mode tests from zstreamtest --no-big-tests 2017-11-29 16:42:20 -08:00
Yann Collet
b3d76e0a94 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-11-28 15:03:38 -08:00
Yann Collet
f5747afe0f add integer overflow trapping to debug builds
doc: https://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Code-Gen-Options.html
2017-11-28 15:02:48 -08:00
W. Felix Handte
62c746dcf9 Add Test on LZ4 Format Input Buffer Overrun 2017-11-28 12:06:48 -05:00
Yann Collet
bd88f633ac zstreamtest : in -T#s, s considered a suffix meaning "seconds"
avoid unintentionnally triggering `seedset`,
so that seed gets automatically determined when not set.
2017-11-27 12:15:23 -08:00
Yann Collet
42c1e64270 slightly improved ratio at -22
merging of repcode search into btsearch introduced a small compression ratio regressio at max level :
1.3.2 : 52728769
after repMerge patch : 52760789 (+32020)

A few minor changes have produced this difference.
They can be hard to spot.

This patch buys back about half of the difference,
by no longer inserting position at hc3 when a long match is found there.
It feels strangely counter-intuitive, but works :
after this patch : 52742555 (-18234)
2017-11-19 14:00:55 -08:00
Yann Collet
99435dbbab minor : search early-out on sufficient_len for hc3 and rep
very very small speed and ratio increases
2017-11-19 12:58:04 -08:00
Yann Collet
23767e950a fix one UB pointer arithmetic in encoder
Instead of calculating distance between 2 memory objects, which is UB,
we extract the offset from object 1, and transfer it into object 2.
2017-11-17 13:24:51 -08:00
Yann Collet
15768cabb5 fixed some complex scenarios
Fixed : multithreading to compress some small data with dictionary
Fixed : ZSTD_initCStream_usingCDict()
Improved streaming memory usage when pledgedSrcSize is known.
2017-11-16 15:18:18 -08:00
Yann Collet
aec56a52fb
Merge pull request #908 from facebook/ubsan
Modified one pointer arithmetic expression to a more conformant way.
2017-11-07 11:45:34 -08:00
Yann Collet
3b59a01039 fixed zstreamtest declaration after statement 2017-11-01 17:36:31 -07:00
Yann Collet
150354c5fe minor refactor
added some traces and assert
related to hunting a potential ubsan error in 32-bits more
(it ends up being a compiler-side issue : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82802).

Modified one pointer arithmetic expression for a more conformant way.
2017-11-01 16:57:48 -07:00
Nick Terrell
e19b0822bc Test large skippable frames 2017-11-01 13:10:03 -07:00
Nick Terrell
86b8134cad [libzstd] Fix parameter selection for empty input
ZSTD_compress() and friends would treat an empty input as an unknown size
when selecting parameters. Thus, they would drastically overallocate the
context. Tell ZSTD_getParams() that the source size is 1 when it is empty.
2017-10-25 17:24:15 -07:00
Yann Collet
97dccbbb2b fixed zbufftest
preserve "pledgedSrcSize=0" means "unknown" in init_advanced()
2017-10-19 14:06:02 -07:00
Yann Collet
9d377c6e88 Merge pull request #895 from facebook/fileSize_unknown
Distinguish 0-size from size-unavailable
2017-10-18 16:50:17 -07:00
Yann Collet
1ff8a8c109 Merge pull request #891 from facebook/contentSize
Content size
2017-10-17 17:24:51 -07:00
Yann Collet
5891f1d952 added a check to ensure 0-size is present in frame header 2017-10-17 16:23:20 -07:00
Yann Collet
18b795374a UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.

Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
2017-10-17 16:14:25 -07:00
Nick Terrell
e600b5d0da [test] Exercise all codes in dictionary tables 2017-10-16 18:05:36 -07:00
Yann Collet
241c57adff playtest : minor cosmetic changes 2017-10-16 14:01:42 -07:00
Yann Collet
43914f6a20 add test for presence of decompressed size field in header
previous test was "display" only, it wouldn't trigger an error in case of missing header.
2017-10-13 23:47:01 -07:00
Yann Collet
beb9b4b398 fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default
and a wrong test in zstreamtest --mt
2017-10-13 19:09:30 -07:00
Yann Collet
213ef3b510 fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
Yann Collet
3c1e3f8ec9 contentSizeFlag enabled by default would also fail for streaming and MT operations
fixed
2017-10-13 18:32:06 -07:00
Yann Collet
fb44516641 ensure fParams.contentSizeFlag starts at 1
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
2017-10-13 17:39:13 -07:00
Yann Collet
7f6a783862 fixed a small error in decodeCorpus
a compressed block must be strictly smaller than its decompressed size.
2017-10-07 15:19:52 -07:00
Yann Collet
4252621e26 playtests: do not use cat on large files
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
2017-10-05 20:21:59 -07:00
Yann Collet
e580dc6a4a Merge pull request #860 from felixhandte/zstd-lz4-support-tests
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
Yann Collet
e963800e27 zstdmt : fixed : buffer dst0 wasn't properly set to null after usage
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
2017-09-28 23:01:31 -07:00
Yann Collet
e0065cf660 make test : removed zstreamtest unit tests for variants
slightly reduced time to create dictionary
at beginning of unit tests
2017-09-28 18:34:38 -07:00
Yann Collet
47c6a95d07 zstreamtest : run unit tests only during "normal" session
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
2017-09-28 18:27:22 -07:00
W. Felix Handte
d0519d4b0c Add CLI Program Name Detection for LZ4 2017-09-28 19:18:15 -04:00
W. Felix Handte
5705d9f25a Add basic tests for the lz4 integration 2017-09-28 19:16:43 -04:00