Commit Graph

5980 Commits

Author SHA1 Message Date
W. Felix Handte
14764de49f Stop Separately Passing CParams in ZSTD_lazy Internal Functions 2018-09-28 17:12:53 -07:00
W. Felix Handte
97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
W. Felix Handte
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
W. Felix Handte
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
03103269de Assert ctx and ms cparams Equivalency 2018-09-28 17:10:42 -07:00
W. Felix Handte
4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
W. Felix Handte
76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
Yann Collet
ec1cb8e996 changed macro name
from EXIT_IF() to RETURN_IF()

EXIT could be misunderstood as exit(), which terminates program execution.
But the macro only leaves the function, not the program.
2018-09-28 16:04:00 -07:00
Yann Collet
05c0a072b7 minor improvement in the multi-format suffix selection 2018-09-28 15:57:35 -07:00
Nick Terrell
5aa9a1dd2d Fix minigzip in std=c99 mode 2018-09-28 15:52:25 -07:00
Nick Terrell
09231dad4d [util] Fix lstat feature test macro 2018-09-28 15:23:09 -07:00
Nick Terrell
43146d8a16 Add -Werror to *build rules 2018-09-28 14:59:40 -07:00
Nick Terrell
eb4423e7ed Fix another warning 2018-09-28 14:24:38 -07:00
Nick Terrell
0e7a7f1def Fix warnings 2018-09-28 12:14:24 -07:00
Nick Terrell
6391cd1030 [zstd] Fix newly added test case 2018-09-28 12:09:28 -07:00
Nick Terrell
146049a1ea [zstreamtest] Add failing test case 2018-09-28 12:09:14 -07:00
Yann Collet
d987ab5983 fixed unreachable section warning on Visual 2018-09-28 09:34:16 -07:00
Nick Terrell
e06f91a169
Merge pull request #1343 from terrelln/faster-ci
[zstreamtest] Reduce memory of newapi tests
2018-09-27 19:29:02 -07:00
Yann Collet
913a0365b6 Merge branch 'dev' into donotdelete 2018-09-27 19:23:55 -07:00
Yann Collet
8b86004cff
Merge pull request #1345 from facebook/fixLegacy
fixed legacy compilation tests
2018-09-27 19:23:27 -07:00
Yann Collet
ef1272737b fixed minor Visual conversion warnings 2018-09-27 18:29:15 -07:00
Yann Collet
ff36513556 fixed longmatch test too 2018-09-27 18:24:41 -07:00
Yann Collet
73773c6b6a fixed legacy compilation tests
for some reason, these tests started failing recently on CircleCI
2018-09-27 18:15:14 -07:00
Yann Collet
7318753d47
Merge pull request #1344 from terrelln/grub-fixes
Fixes for GRUB
2018-09-27 17:36:54 -07:00
Yann Collet
9b45db7fa6 minor refactoring of --list
trying to reduce recurrent patterns.
2018-09-27 16:49:08 -07:00
Nick Terrell
a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
Nick Terrell
d8c73cd607 Reset number of threads less often 2018-09-27 15:49:31 -07:00
Nick Terrell
aec1a3ec58 Change byte to value to avoid a GRUB typedef 2018-09-27 15:24:48 -07:00
Nick Terrell
109bd37474 Include stddef.h for size_t 2018-09-27 15:24:48 -07:00
Nick Terrell
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Nick Terrell
7ee910e86b More aggressive limitations 2018-09-27 13:55:24 -07:00
Nick Terrell
ca0cfa3dbd [zstreamtest] Reduce memory of newapi tests
We could allocate up to 2^28 bytes of memory when using 2 threads with
window log = 24. Now, we limit it to 2^26 bytes of memory when not running
big tests.

I chose max window log = 22 since that is the maximum source size when
big tests are disabled. Hopefully this will be enough to reduce or
eliminate the test failures.
2018-09-27 12:48:29 -07:00
Yann Collet
2a5cd8535a
Merge pull request #1342 from facebook/fixcatyd
fix : huge (>4GB) chain of blocks
2018-09-27 10:20:14 -07:00
Yann Collet
404a7bfed0 moved again overflow correction
cannot work from within ZSTD_compressBlock()
2018-09-26 18:06:53 -07:00
Yann Collet
9bce916732
Merge pull request #1340 from terrelln/generate-artifacts
Publish artifacts with CircleCI
2018-09-26 15:52:23 -07:00
Yann Collet
0e2dbac18a changed overflow correction place
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
2018-09-26 15:35:38 -07:00
Yann Collet
e74eade251
Merge pull request #1339 from facebook/grep_colors
fixed usage of grep in Makefile
2018-09-26 14:39:20 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt
Adaptive compression
2018-09-26 14:39:08 -07:00
Yann Collet
f98c69d77c fix : huge (>4GB) stream of blocks
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
2018-09-26 14:24:28 -07:00
Yann Collet
3dae90c4e0
Merge pull request #1338 from facebook/rfc
updated format documentation
2018-09-26 13:40:35 -07:00
Yann Collet
65ed6eeefb
Merge pull request #1337 from facebook/test_failure
fixed `!` tests
2018-09-26 13:40:20 -07:00
Yann Collet
8ff17a6a09
Merge pull request #1329 from facebook/v04isout
Changed default legacy support to v0.5+
2018-09-26 13:39:05 -07:00
Nick Terrell
3ff6040848 Publish artifacts with CircleCI
* Updates CircleCI to use workflows.
  We can now specify any number of test jobs to run in parallel.
* Switch the image to `buildpack-deps:trusty` which is only 500 MB
  instead of 7 GB, so that saves 7 minutes to download it if it isn't
  already cached on the host.
* Publish the source tarball and sha256sum as artifacts.
* If the `GITHUB_TOKEN` environment variable is set, we will also
  add the tarball + sha256sum to the tagged release, after manual
  approval.
2018-09-26 13:23:28 -07:00
Yann Collet
08f68d83c5 fixed usage of grep in Makefile
when terminal uses colors
as suggested by @danielshir (#1294)
2018-09-25 16:56:53 -07:00
Yann Collet
72a3adf826 updated format documentation
to match last edits of RFC8478.
2018-09-25 16:34:26 -07:00
Yann Collet
6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
Yann Collet
63abaf2171 fixed ! tests
Sometimes, it's necessary to test that a certain command fail, as expected.
Such failure is actually a success, and must not stop the flow of tests.

Several tests were prefixed with `!` to invert return code.
This does not work : it effectively makes the tests pass no matter what.

Use instead function die(), which is meant to trap successes, and transform them into errors.
2018-09-25 15:57:28 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
9bb6c15f79
Merge pull request #1332 from facebook/minclevel
defined a minimum negative level
2018-09-24 16:01:13 -07:00