Commit Graph

6667 Commits

Author SHA1 Message Date
Nick Terrell
ee130a9889 [libzstd] Check the size in readSkippableFrameSize() 2019-04-17 11:41:55 -07:00
Nick Terrell
5922f4e2ae [legacy] Return the right error code 2019-04-17 11:34:52 -07:00
Nick Terrell
09caa4d800 [fuzzer] Add a fuzzer for frame info functions
Add a fuzzer that fuzzes all helper functions that take compressed
input. This fuzzer caught one out of bounds read in
`ZSTD_decompressBound()`.
2019-04-17 11:29:42 -07:00
Nick Terrell
450feb0f95 [libzstd] Fix ZSTD_decompressBound() on bad skippable frames
The function didn't verify that the skippable frame size is correct.
2019-04-17 11:29:42 -07:00
Nick Terrell
83b51e9f88
Merge pull request #1586 from facebook/dev
v1.4.0
2019-04-16 15:37:24 -07:00
Nick Terrell
b85f7d7790
Merge pull request #1589 from terrelln/appveyorTest
[libzstd] Require ZSTD_MULTITHREAD to create a ZSTDMT_CCtx
2019-04-16 12:32:19 -07:00
Nick Terrell
a17fe4c9e5 [visual] Fix unreachable code warning 2019-04-16 11:32:35 -07:00
Yann Collet
41ad46f1a9
Merge pull request #1588 from terrelln/release-prep
[fullbench] Fix speed measurements
2019-04-16 10:11:28 -07:00
Nick Terrell
82d5a9870e [visual] Compile with ZSTD_MULTITHREAD=1
This branch `appveyorTests` runs the tests that `master` runs, so
passing Appveyor means that the tests are fixed.
2019-04-15 23:04:46 -07:00
Nick Terrell
de0499f7fa [libzstd] Require ZSTD_MULTITHREAD to create a ZSTDMT_CCtx
ZSTDMT was broken when compiled without ZSTD_MULTITHREAD defined,
because `ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, nbWorkerss)`
failed. It was detected by the MSVC test which runs the fuzzer with
multithreading disabled.

This is a very niche use case of a deprecated API, because the API is
inefficient and synchronous, since `threading.h` will be synchronous.
Users almost certainly don't want this, and anyone who tested their code
should realize that it is broken. Therefore, I think it is safe to
require `ZSTD_MULTITHREAD` to be defined to use ZSTDMT.
2019-04-15 23:04:46 -07:00
Nick Terrell
a48e32ca50 [fullbench] Fix measurements 2019-04-15 21:01:48 -07:00
Nick Terrell
a821aee3b1
Merge pull request #1587 from terrelln/release-prep
[playTets] Don't pass status=none to dd
2019-04-15 16:42:26 -07:00
Nick Terrell
6996804d05 [playTets] Don't pass status=none to dd
The OS X test machine on TravisCI doesn't recognize the parameter
https://travis-ci.org/facebook/zstd/jobs/520517933.
2019-04-15 16:21:46 -07:00
Nick Terrell
d86c7ad458
Merge pull request #1585 from terrelln/release-prep
Prepare for the zstd-1.4.0 release
2019-04-15 14:41:50 -07:00
Nick Terrell
bef61221a3 Update the changelog for zstd-1.4.0 2019-04-15 14:33:46 -07:00
Nick Terrell
203aac5bc6 Update benchmarks in README 2019-04-15 13:57:14 -07:00
Josh Soref
a880ca239b Spelling (#1582)
* spelling: accidentally

* spelling: across

* spelling: additionally

* spelling: addresses

* spelling: appropriate

* spelling: assumed

* spelling: available

* spelling: builder

* spelling: capacity

* spelling: compiler

* spelling: compressibility

* spelling: compressor

* spelling: compression

* spelling: contract

* spelling: convenience

* spelling: decompress

* spelling: description

* spelling: deflate

* spelling: deterministically

* spelling: dictionary

* spelling: display

* spelling: eliminate

* spelling: preemptively

* spelling: exclude

* spelling: failure

* spelling: independence

* spelling: independent

* spelling: intentionally

* spelling: matching

* spelling: maximum

* spelling: meaning

* spelling: mishandled

* spelling: memory

* spelling: occasionally

* spelling: occurrence

* spelling: official

* spelling: offsets

* spelling: original

* spelling: output

* spelling: overflow

* spelling: overridden

* spelling: parameter

* spelling: performance

* spelling: probability

* spelling: receives

* spelling: redundant

* spelling: recompression

* spelling: resources

* spelling: sanity

* spelling: segment

* spelling: series

* spelling: specified

* spelling: specify

* spelling: subtracted

* spelling: successful

* spelling: return

* spelling: translation

* spelling: update

* spelling: unrelated

* spelling: useless

* spelling: variables

* spelling: variety

* spelling: verbatim

* spelling: verification

* spelling: visited

* spelling: warming

* spelling: workers

* spelling: with
2019-04-12 11:18:11 -07:00
Nick Terrell
13fddac46d
Merge pull request #1583 from lzutao/meson
Fix tests and meson build
2019-04-12 11:13:20 -07:00
Lzu Tao
e31e7ca9ed travis: Make Mesonbuild non-optional 2019-04-12 23:43:34 +07:00
Lzu Tao
6c9b023f3e meson: Fix build 2019-04-12 21:28:36 +07:00
Lzu Tao
fb6901b2c0 tests: Add missing header timefn.h 2019-04-12 21:28:17 +07:00
Yann Collet
8ac2831f3d
Merge pull request #1581 from facebook/benchfn
benchfn's reduced dependencies
2019-04-11 14:23:04 -07:00
Yann Collet
1e01560b83 fixed timespec_get() initialization bug on some targets
not sure why, but msan fires an "unitialized variable" error
when time gets properly initialized by timespec_get().
Maybe in some cases, not all bytes of the structure are initialized ?
Or maybe msan fails to detect the initialization ?

Anyway, pre-initializing the variable before passing it to timespec_get() works.
2019-04-11 13:46:30 -07:00
Yann Collet
058da605cb fixed minor conversion warning 2019-04-11 12:25:27 -07:00
Yann Collet
f8e9bec73a fixed poolTests on Windows
must use ZSTD_ prefix in front of pthread types
so that they get properly translated for Windows.
2019-04-11 12:03:42 -07:00
Yann Collet
8f56fa2f58 fixed poolTests
added poolTests to all
2019-04-11 09:50:39 -07:00
Nick Terrell
9563fb4bfc
Merge pull request #1580 from terrelln/dict-fuzz
[libzstd] Fix decompression dictionary bugs and clean up initialization
2019-04-10 19:29:09 -07:00
Yann Collet
30c26ab726 fixed minor warning
unused variable when assert() turned off in fileio.c
2019-04-10 17:48:56 -07:00
Yann Collet
fbdd30d68e fixed cmake build script for test programs 2019-04-10 17:47:01 -07:00
Nick Terrell
aafe97b67d [libzstd] Switch dictUses to an enum 2019-04-10 16:50:35 -07:00
Yann Collet
526ec646b7 alternate static assert
to circumvent Visual's C4804 warning
2019-04-10 16:05:02 -07:00
Yann Collet
9703a59121 fixed minor conversion warning 2019-04-10 15:54:55 -07:00
Yann Collet
885476fb5b FreeBSD_11 specific fix
C11 mandates the definition of timespec_get() and TIME_UTC.
However, FreeBSD11 announce C11 compliance, but does not provifr timespec_get(),
breaking link stage for benchfn.
Since it does not provide TIME_UTC either, which is also required by C11,
test this macro: this will automatically rule out FreeBSD 11 for this code path
(it will use the backup C90 path instead, based on clock_t).

The issue seeems fixed in FreeBSD 12.
2019-04-10 15:22:18 -07:00
Yann Collet
2fa4f2e246 updated Visual projects
added timefn
2019-04-10 15:07:36 -07:00
Yann Collet
2c6b14ed22 fixed Windows header
cmake build script: added timefn
2019-04-10 14:54:13 -07:00
Yann Collet
3d346579d8 no more need for CLOCK_MONOTONIC 2019-04-10 14:16:39 -07:00
Yann Collet
36d2dfd846 moved C11 code path to timespec_get 2019-04-10 14:15:11 -07:00
Yann Collet
4765929271 fixed perror include 2019-04-10 14:04:11 -07:00
Yann Collet
70802cde6d fixed error message
using stdlib's perror()
2019-04-10 14:01:18 -07:00
Yann Collet
4b8185c7fc tried a blindfix for unix + c11 2019-04-10 13:26:27 -07:00
Nick Terrell
50b9c41196 [libzstd] Fix decompression dictionary bugs and clean up initialization
Bugs:

* `ZSTD_DCtx_refPrefix()` didn't clear the dictionary after the first
  use. Fix and add a test case.
* `ZSTD_DCtx_reset()` always cleared the dictionary. Fix and add a test
  case.
* After calling `ZSTD_resetDStream()` you could no longer load a
  dictionary, since the stage was set to `zdss_loadHeader`. Fix and add
  a test case.

Cleanup:

* Make `ZSTD_initDStream*()` and `ZSTD_resetDStream()` wrap the new
 advanced API, and add test cases.
* Document the equivalent of these functions in the advanced API and
  document the unstable functions as deprecated.
2019-04-10 12:59:02 -07:00
Yann Collet
59a7116cc2 benchfn dependencies reduced to only timefn
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.

Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.

Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
2019-04-10 12:37:03 -07:00
Yann Collet
094c000904 Merge branch 'dev' into benchfn 2019-04-10 11:57:05 -07:00
Nick Terrell
f86d4bd1d5
Merge pull request #1576 from terrelln/dict-fuzz
Add new fuzzers and fix exposed bugs
2019-04-10 11:29:15 -07:00
Nick Terrell
5f6ca3c6ce
Merge pull request #1578 from orip/r-flag-typo
Fixed `-r` typo
2019-04-10 11:19:02 -07:00
Yann Collet
90c0462d63 minor presentation refactoring
and removed some // comment style
2019-04-10 10:03:06 -07:00
Ori Peleg
bdeb4786b5 Fixed -r typo 2019-04-10 13:37:41 +03:00
Nick Terrell
c45dec12c5 [fuzzer] Use ZSTD_DCtx_loadDictionary_advanced() half the time 2019-04-09 18:02:22 -07:00
Nick Terrell
10a3d4dca9 [fuzzer] Make the regression_driver work while fuzzers are active 2019-04-09 18:01:49 -07:00
Nick Terrell
824aaa695f [libzstd] Fix ZSTD_decompressDCtx() with a dictionary
* `ZSTD_decompressDCtx()` did not use the dictionary loaded by
  `ZSTD_DCtx_loadDictionary()`.
* Add a unit test.
* A stacked diff uses `ZSTD_decompressDCtx()` in the
  `dictionary_round_trip` and `dictionary_decompress` fuzzers.
2019-04-09 17:59:27 -07:00