Commit Graph

4626 Commits

Author SHA1 Message Date
Yann Collet
5432ef6921 fixes adaptation on srcSize
This patch restores capability for each file to receive adapted compression parameters depending on its size.

The bug breaking this feature was relatively silly :
setting a parameter with a value "0" is supposed to be a no-op.
Unfortunately, it would pin down compression parameters as if they were manually set,
preventing later automatic adaptation.

Unfortunately, I'm currently short of a test case that could check this situation and trigger an error.
Compression parameters selection between tableID 0,1,2,3 is largely internal,
leaving no trace to outside world, not even in frame header.
2017-12-13 17:45:26 -08:00
Nick Terrell
4680e85bdf Allow -o with multiple files 2017-12-13 17:44:34 -08:00
Yann Collet
4d0dfafa7b
Merge pull request #949 from terrelln/rrm
[fileio] Refuse to remove non-regular file
2017-12-13 17:36:39 -08:00
Yann Collet
d23eb9a098 zstreamtest : added missing CHECK_Z() 2017-12-13 15:35:49 -08:00
Nick Terrell
90d38f6a53
Merge pull request #945 from terrelln/dev
Fix cdict compressor repcodes
2017-12-13 14:24:21 -08:00
Nick Terrell
82bc8fe0cc [fileio] Refuse to remove non-regular file 2017-12-13 13:38:26 -08:00
Yann Collet
aa81aac2dd
Merge pull request #948 from terrelln/mb
[fileio] Fix window size MB calculation
2017-12-13 12:17:05 -08:00
Yann Collet
311878dec3 Improved tests
- building cli from /tests preserves potential flags in MOREFLAGS (such as asan/usan)
- MT dictionary tests check for MT capability (MT is not enabled by default for zstd32)
2017-12-13 11:48:30 -08:00
Nick Terrell
22727a7467 Fix cdict compressor repcodes 2017-12-13 11:31:20 -08:00
Yann Collet
dba8016d2d Merge branch 'dev' into fix944 2017-12-13 11:20:09 -08:00
Nick Terrell
b5e7f6c0f3 [fileio] Fix window size MB calculation
Test command:
```
head -c 10000 /dev/zero | ./zstd -c --zstd=wlog=12 | ./zstd -M2048 -t
```
2017-12-13 10:57:01 -08:00
Yann Collet
04a1557e28 fuzzer tests : added -v command
so that unit tests get listed in CI console
and we can see which one failed.
2017-12-13 10:08:52 -08:00
Yann Collet
98ccc115ed added cli test veryfying bug #944
this test fails on v1.3.2
2017-12-12 19:47:38 -08:00
Yann Collet
e9dc0ab884 zstd binary, when compiled from /tests, enables debug mode
adds ZSTD_DEBUG=1 and -g
2017-12-12 18:20:04 -08:00
Yann Collet
e28305fcca fix #944 : ZSTDMT with large files and dictionary now works correctly
windowLog is now enforced from provided compression parameters,
instead of being copied blindly from `cdict`
where it could be smaller.

also :
- fix a minor bug in zstreamtest --mt : advanced parameters must be set before init
- changed advanced parameter name to ZSTDMT_jobSize
2017-12-12 18:04:58 -08:00
Yann Collet
03832b7aa5 re-added test case
messing with revert ... :(
2017-12-12 14:01:54 -08:00
Yann Collet
8a104fda05 Revert "Created a test case which reliably reproduces bug #944"
This reverts commit 5098d1fbe2.
2017-12-12 12:51:49 -08:00
Yann Collet
5098d1fbe2 Created a test case which reliably reproduces bug #944
in zstreamtest.
2017-12-12 12:48:31 -08:00
Yann Collet
ac8e022806
Merge pull request #943 from facebook/fix942
Fix #942
2017-12-08 13:53:08 -05:00
Yann Collet
dfc697e967 comment clarification 2017-12-08 12:16:49 -05:00
Yann Collet
c029ee1f0b ZSTD_initCStream_srcSize() considers "0" to mean "unknown"
to not break existing programs relying on this behavior.
Might be changed to mean "empty" in the future.
2017-12-07 17:13:10 -05:00
Yann Collet
21f7672d43 zstreamtest : add test which detects when ZSTD_initCStream() inflate data
as in #942, when ZSTD_initCStream() believes data is empty, instead of unknown.
2017-12-07 03:06:01 -05:00
Yann Collet
3aa2b27a89 fix #942 : streaming interface does not compress after ZSTD_initCStream()
While the final result is still, technically, a frame,
the resulting frame expands initial data instead of compressing it.
This is because the streaming API creates a tiny 1-byte buffer for input,
because it believes input is empty (0-bytes),
because in the past, 0 used to mean "unknown" instead.

This patch fixes the issue.
Todo : add a test which traps the issue.
2017-12-07 02:52:50 -05:00
Yann Collet
b9c84e0fd6
Merge pull request #941 from facebook/registerLess
remove `register` keyword
2017-12-05 07:34:03 -08:00
Yann Collet
87ed94c800
Merge pull request #940 from facebook/gettime
fix #911 : changed detection macro for `clock_gettime()`
2017-12-04 18:21:02 -08:00
Yann Collet
c173dbd6e7 no longer supported starting C++17 2017-12-04 18:00:53 -08:00
Yann Collet
31293330d0 It's still necessary to check PLATFORM_POSIX_VERSION for clock_gettime()
glibc/uclibc is not enough
2017-12-04 16:31:59 -08:00
Yann Collet
0097469238 removed a few redundant #include 2017-12-04 16:02:42 -08:00
Yann Collet
e46194bbf9 fix #911 : changed detection macro for clock_gettime()
The new macro might be a bit too restrictive.
Systems which do not support new test will simply default to <time.h>'s `clock_t clock()`,
suffering lesser benchmark accuracy.
Should it matter, the detection macro will have to be upgraded.
2017-12-04 15:57:01 -08:00
Yann Collet
896980ec15
Merge pull request #936 from facebook/qemu32panic
Reduce memory stress for qemu tests
2017-12-04 14:16:55 -08:00
Yann Collet
55faa5492d fileio: fixed LZ4F invocation from assert() 2017-12-04 11:26:59 -08:00
Yann Collet
2efc5e6f84 Merge branch 'dev' into qemu32panic 2017-12-04 11:23:21 -08:00
Yann Collet
af2fbbcb0d
Merge pull request #939 from facebook/shorterCircleCI
Faster CircleCI tests
2017-12-04 11:22:30 -08:00
Yann Collet
7e05ef851a Merge branch 'dev' into qemu32panic 2017-12-03 11:14:36 -08:00
Yann Collet
5e1f34b7e4 setParameter : no side-effect on setting a compression parameter
last such side-effect was modifying cctx->loadedDictEnd on setting forceWindow.
It is no a useless operation, so it's removed.
No side-effect left when setting a compression parameter.
2017-12-01 21:17:09 -08:00
Yann Collet
6f1377301f circleci: removed -m32 tests
installing dependencies costs ~2mn, making it the longest test
(test itself was < 15sec)
2017-12-01 18:34:22 -08:00
Yann Collet
272c142330 circleci : balance some tests
try to group them by "equivalent" duration
since next job will not start as long as previous one is not completed on both VM
2017-12-01 18:11:34 -08:00
Yann Collet
6127592391 circleci : removed cmaketest
strange error on pthread detection
(seems a bug in cmake detection, since
`make zstd` works perfectly with `-pthread` on same conf)
2017-12-01 17:56:16 -08:00
Yann Collet
71f012e5bf zstdcli: fixed minor warning when bench module not enabled
one variable defined but not used
2017-12-01 17:42:46 -08:00
Yann Collet
c6f5bf071c fixed powerpc test
misspelled CFLAGS
2017-12-01 17:35:29 -08:00
Yann Collet
74991c2764 circleci: removed gccX/clang tests
does no longer need to install these large packages
2017-12-01 17:28:24 -08:00
Yann Collet
30eeb248a8 added environment variable through circleci dashboard
check if it works better
2017-12-01 17:22:04 -08:00
Yann Collet
363355448b circleci: removed one test node 2017-12-01 17:02:55 -08:00
Yann Collet
a1b24e6262
Merge pull request #938 from terrelln/time
Use util.h for timing
2017-12-01 16:40:38 -08:00
Yann Collet
6f30e60a96
Merge pull request #934 from facebook/opt3
zstd_opt: changed cost formula
2017-12-01 16:40:16 -08:00
Yann Collet
519cf3b400 circleci: add CFLAGS=-O0 environment variable
for faster compilation
2017-12-01 16:31:40 -08:00
Yann Collet
4413ed27e4 circleci : make libzstd-nomt had its own test 2017-12-01 16:27:24 -08:00
Yann Collet
6fc63f521f Merge branch 'dev' into qemu32panic 2017-12-01 10:51:01 -08:00
Yann Collet
453fb5e830 removed -ftrapv from tests/ debug flags
-ftrapv is apparently buggy for gcc.
versions >= 5 are supposed to work better,
but even then, some complaints say it's still flaky when optimizations are enabled.
I even saw a post saying it only works if one creates its own signal handler,
which would make this flag no longer transparent.

on clang, it seems to work correctly.
But we would need to add a method to selectively add flags depending on compiler.
That's too much troubles for the intended benefit
(just catch integer overflows, which we can also do using ubsan).
2017-12-01 10:30:53 -08:00
Nick Terrell
dab8cfa3c7 Combine definitions of SEC_TO_MICRO 2017-11-30 19:40:53 -08:00