Commit Graph

5847 Commits

Author SHA1 Message Date
Yann Collet
2a5cd8535a
Merge pull request #1342 from facebook/fixcatyd
fix : huge (>4GB) chain of blocks
2018-09-27 10:20:14 -07:00
Yann Collet
404a7bfed0 moved again overflow correction
cannot work from within ZSTD_compressBlock()
2018-09-26 18:06:53 -07:00
Yann Collet
9bce916732
Merge pull request #1340 from terrelln/generate-artifacts
Publish artifacts with CircleCI
2018-09-26 15:52:23 -07:00
Yann Collet
0e2dbac18a changed overflow correction place
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
2018-09-26 15:35:38 -07:00
Yann Collet
e74eade251
Merge pull request #1339 from facebook/grep_colors
fixed usage of grep in Makefile
2018-09-26 14:39:20 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt
Adaptive compression
2018-09-26 14:39:08 -07:00
Yann Collet
f98c69d77c fix : huge (>4GB) stream of blocks
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
2018-09-26 14:24:28 -07:00
Yann Collet
3dae90c4e0
Merge pull request #1338 from facebook/rfc
updated format documentation
2018-09-26 13:40:35 -07:00
Yann Collet
65ed6eeefb
Merge pull request #1337 from facebook/test_failure
fixed `!` tests
2018-09-26 13:40:20 -07:00
Yann Collet
8ff17a6a09
Merge pull request #1329 from facebook/v04isout
Changed default legacy support to v0.5+
2018-09-26 13:39:05 -07:00
Nick Terrell
3ff6040848 Publish artifacts with CircleCI
* Updates CircleCI to use workflows.
  We can now specify any number of test jobs to run in parallel.
* Switch the image to `buildpack-deps:trusty` which is only 500 MB
  instead of 7 GB, so that saves 7 minutes to download it if it isn't
  already cached on the host.
* Publish the source tarball and sha256sum as artifacts.
* If the `GITHUB_TOKEN` environment variable is set, we will also
  add the tarball + sha256sum to the tagged release, after manual
  approval.
2018-09-26 13:23:28 -07:00
Yann Collet
08f68d83c5 fixed usage of grep in Makefile
when terminal uses colors
as suggested by @danielshir (#1294)
2018-09-25 16:56:53 -07:00
Yann Collet
72a3adf826 updated format documentation
to match last edits of RFC8478.
2018-09-25 16:34:26 -07:00
Yann Collet
6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
Yann Collet
63abaf2171 fixed ! tests
Sometimes, it's necessary to test that a certain command fail, as expected.
Such failure is actually a success, and must not stop the flow of tests.

Several tests were prefixed with `!` to invert return code.
This does not work : it effectively makes the tests pass no matter what.

Use instead function die(), which is meant to trap successes, and transform them into errors.
2018-09-25 15:57:28 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
9bb6c15f79
Merge pull request #1332 from facebook/minclevel
defined a minimum negative level
2018-09-24 16:01:13 -07:00
Yann Collet
8d69c303e4
Merge pull request #1336 from facebook/scanbuild
fixed minor scan-build warnings
2018-09-24 14:49:16 -07:00
Yann Collet
3ece0316d8
Merge pull request #1335 from facebook/limits
added some tests based on limits.h
2018-09-24 14:49:03 -07:00
Yann Collet
0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet
292d8e4a83 added some tests based on limits.h
in order to ensure proper type mapping
when not using stdint.h
2018-09-23 23:57:30 -07:00
Yann Collet
0fc07eb1fd fixed zstd-decompress
which cannot support ZSTD_minCLevel()
2018-09-22 17:21:39 -07:00
Yann Collet
7d9f386a0d
Merge pull request #1331 from facebook/mingw
fix mingw compatibility
2018-09-22 16:11:55 -07:00
Yann Collet
364041c6dd enforce minimum compression level limit
using ZSTD_minCLevel()
2018-09-22 16:10:10 -07:00
Yann Collet
0e211bdd18 fixed constant comparison on 32-bits systems 2018-09-21 18:23:32 -07:00
Yann Collet
71a5210617 avoid recompiling dll every time under mingw 2018-09-21 17:40:30 -07:00
Yann Collet
123fac6b6d fix pzstd compatibility with mingw
some details changed with introduction of gcc7
2018-09-21 17:36:00 -07:00
Yann Collet
484f40697b fix constant redeclaration in paramgrill 2018-09-21 17:28:37 -07:00
Yann Collet
00c18c0c88 simplified "slows down when compression blocked" 2018-09-21 16:35:43 -07:00
Yann Collet
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet
bfff4f4809 ensure all writes to job->cSize are mutex protected
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
2018-09-21 16:00:39 -07:00
Yann Collet
32b7cf1bcf fixed tautological tests
involving ZSTD_TARGETLENGTH_MIN (== 0)
2018-09-21 15:04:43 -07:00
Yann Collet
c044345f8f Merge branch 'mingw' into minclevel 2018-09-21 14:56:57 -07:00
Yann Collet
54001f3dbc fix mingw compatibility
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
2018-09-21 14:46:09 -07:00
Yann Collet
de6c75e4e5
Merge pull request #1318 from felixhandte/shadow-dict-matches
Don't Search Dictionary Context When Working Context Search Resulted in Mismatch
2018-09-21 12:15:33 -07:00
Yann Collet
a54c86cfc6 defined a minimum negative level
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
2018-09-20 16:52:03 -07:00
Yann Collet
db97310ace fixed versions-test to only test v0.5+
since zstd_devel is no longer compatible with v0.4+
2018-09-20 14:59:11 -07:00
Yann Collet
b2939163e1 Changed default legacy support to v0.5+
thus dropping read support for v0.4.

It's always possible to re-enable it, by changing build macro ZSTD_LEGACY_SUPPORT to 4.
2018-09-20 14:30:20 -07:00
Yann Collet
7992942d66 fixed complex tsan issue
when job->consumed == job->src.size , compression job is presumed completed,
so it must be the very last action done in worker thread.
2018-09-20 13:47:31 -07:00
Yann Collet
15519479ba fixed minor gcc warning on a unused variable 2018-09-20 13:00:11 -07:00
Yann Collet
45010da074 updated man page
and added `--adapt` test in `playTests.sh`
2018-09-19 17:37:22 -07:00
Yann Collet
6b07a66aec fixed minor reporting discrepancy in MT mode 2018-09-19 16:30:55 -07:00
Yann Collet
ca02ebee07 removed static variables
so that --adapt can work on multiple input files too
2018-09-19 15:25:50 -07:00
Yann Collet
89bc309d90 error out when --adapt is associated with --single-thread
since they are not compatible
2018-09-19 14:49:13 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Yann Collet
c8ff5200f9
Merge pull request #1326 from facebook/refPrefix
updated documentation of *refPrefix()
2018-09-18 15:28:34 -07:00
Yann Collet
005f000aed updated documentation of *refPrefix()
indicating the equivalence with `diff` operation.
2018-09-18 13:07:08 -07:00
Nick Terrell
f884e9dd8e
Merge pull request #1321 from bket/non-portable
'head -c BYTES' is non-portable.
2018-09-17 14:40:27 -07:00
Yann Collet
00ce26725b
Merge pull request #1324 from ko-zu/fixclangcode
Fix largeNbDicts bench for clangbuild
2018-09-17 14:10:17 -07:00
Björn Ketelaars
06fd1e473d 'head -c BYTES' is non-portable.
tests/playTests.sh uses 'head -c' in a couple of tests to truncate the
last byte of a file. The '-c' option is non-portable (not in POSIX).
Instead use a wrapper around dd (truncateLastByte).
2018-09-17 20:39:35 +02:00