Commit Graph

6605 Commits

Author SHA1 Message Date
Nick Terrell
a821aee3b1
Merge pull request #1587 from terrelln/release-prep
[playTets] Don't pass status=none to dd
2019-04-15 16:42:26 -07:00
Nick Terrell
6996804d05 [playTets] Don't pass status=none to dd
The OS X test machine on TravisCI doesn't recognize the parameter
https://travis-ci.org/facebook/zstd/jobs/520517933.
2019-04-15 16:21:46 -07:00
Nick Terrell
d86c7ad458
Merge pull request #1585 from terrelln/release-prep
Prepare for the zstd-1.4.0 release
2019-04-15 14:41:50 -07:00
Nick Terrell
bef61221a3 Update the changelog for zstd-1.4.0 2019-04-15 14:33:46 -07:00
Nick Terrell
203aac5bc6 Update benchmarks in README 2019-04-15 13:57:14 -07:00
Josh Soref
a880ca239b Spelling (#1582)
* spelling: accidentally

* spelling: across

* spelling: additionally

* spelling: addresses

* spelling: appropriate

* spelling: assumed

* spelling: available

* spelling: builder

* spelling: capacity

* spelling: compiler

* spelling: compressibility

* spelling: compressor

* spelling: compression

* spelling: contract

* spelling: convenience

* spelling: decompress

* spelling: description

* spelling: deflate

* spelling: deterministically

* spelling: dictionary

* spelling: display

* spelling: eliminate

* spelling: preemptively

* spelling: exclude

* spelling: failure

* spelling: independence

* spelling: independent

* spelling: intentionally

* spelling: matching

* spelling: maximum

* spelling: meaning

* spelling: mishandled

* spelling: memory

* spelling: occasionally

* spelling: occurrence

* spelling: official

* spelling: offsets

* spelling: original

* spelling: output

* spelling: overflow

* spelling: overridden

* spelling: parameter

* spelling: performance

* spelling: probability

* spelling: receives

* spelling: redundant

* spelling: recompression

* spelling: resources

* spelling: sanity

* spelling: segment

* spelling: series

* spelling: specified

* spelling: specify

* spelling: subtracted

* spelling: successful

* spelling: return

* spelling: translation

* spelling: update

* spelling: unrelated

* spelling: useless

* spelling: variables

* spelling: variety

* spelling: verbatim

* spelling: verification

* spelling: visited

* spelling: warming

* spelling: workers

* spelling: with
2019-04-12 11:18:11 -07:00
Nick Terrell
13fddac46d
Merge pull request #1583 from lzutao/meson
Fix tests and meson build
2019-04-12 11:13:20 -07:00
Lzu Tao
e31e7ca9ed travis: Make Mesonbuild non-optional 2019-04-12 23:43:34 +07:00
Lzu Tao
6c9b023f3e meson: Fix build 2019-04-12 21:28:36 +07:00
Lzu Tao
fb6901b2c0 tests: Add missing header timefn.h 2019-04-12 21:28:17 +07:00
Yann Collet
8ac2831f3d
Merge pull request #1581 from facebook/benchfn
benchfn's reduced dependencies
2019-04-11 14:23:04 -07:00
Yann Collet
1e01560b83 fixed timespec_get() initialization bug on some targets
not sure why, but msan fires an "unitialized variable" error
when time gets properly initialized by timespec_get().
Maybe in some cases, not all bytes of the structure are initialized ?
Or maybe msan fails to detect the initialization ?

Anyway, pre-initializing the variable before passing it to timespec_get() works.
2019-04-11 13:46:30 -07:00
Yann Collet
058da605cb fixed minor conversion warning 2019-04-11 12:25:27 -07:00
Yann Collet
f8e9bec73a fixed poolTests on Windows
must use ZSTD_ prefix in front of pthread types
so that they get properly translated for Windows.
2019-04-11 12:03:42 -07:00
Yann Collet
8f56fa2f58 fixed poolTests
added poolTests to all
2019-04-11 09:50:39 -07:00
Nick Terrell
9563fb4bfc
Merge pull request #1580 from terrelln/dict-fuzz
[libzstd] Fix decompression dictionary bugs and clean up initialization
2019-04-10 19:29:09 -07:00
Yann Collet
30c26ab726 fixed minor warning
unused variable when assert() turned off in fileio.c
2019-04-10 17:48:56 -07:00
Yann Collet
fbdd30d68e fixed cmake build script for test programs 2019-04-10 17:47:01 -07:00
Nick Terrell
aafe97b67d [libzstd] Switch dictUses to an enum 2019-04-10 16:50:35 -07:00
Yann Collet
526ec646b7 alternate static assert
to circumvent Visual's C4804 warning
2019-04-10 16:05:02 -07:00
Yann Collet
9703a59121 fixed minor conversion warning 2019-04-10 15:54:55 -07:00
Yann Collet
885476fb5b FreeBSD_11 specific fix
C11 mandates the definition of timespec_get() and TIME_UTC.
However, FreeBSD11 announce C11 compliance, but does not provifr timespec_get(),
breaking link stage for benchfn.
Since it does not provide TIME_UTC either, which is also required by C11,
test this macro: this will automatically rule out FreeBSD 11 for this code path
(it will use the backup C90 path instead, based on clock_t).

The issue seeems fixed in FreeBSD 12.
2019-04-10 15:22:18 -07:00
Yann Collet
2fa4f2e246 updated Visual projects
added timefn
2019-04-10 15:07:36 -07:00
Yann Collet
2c6b14ed22 fixed Windows header
cmake build script: added timefn
2019-04-10 14:54:13 -07:00
Yann Collet
3d346579d8 no more need for CLOCK_MONOTONIC 2019-04-10 14:16:39 -07:00
Yann Collet
36d2dfd846 moved C11 code path to timespec_get 2019-04-10 14:15:11 -07:00
Yann Collet
4765929271 fixed perror include 2019-04-10 14:04:11 -07:00
Yann Collet
70802cde6d fixed error message
using stdlib's perror()
2019-04-10 14:01:18 -07:00
Yann Collet
4b8185c7fc tried a blindfix for unix + c11 2019-04-10 13:26:27 -07:00
Nick Terrell
50b9c41196 [libzstd] Fix decompression dictionary bugs and clean up initialization
Bugs:

* `ZSTD_DCtx_refPrefix()` didn't clear the dictionary after the first
  use. Fix and add a test case.
* `ZSTD_DCtx_reset()` always cleared the dictionary. Fix and add a test
  case.
* After calling `ZSTD_resetDStream()` you could no longer load a
  dictionary, since the stage was set to `zdss_loadHeader`. Fix and add
  a test case.

Cleanup:

* Make `ZSTD_initDStream*()` and `ZSTD_resetDStream()` wrap the new
 advanced API, and add test cases.
* Document the equivalent of these functions in the advanced API and
  document the unstable functions as deprecated.
2019-04-10 12:59:02 -07:00
Yann Collet
59a7116cc2 benchfn dependencies reduced to only timefn
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.

Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.

Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
2019-04-10 12:37:03 -07:00
Yann Collet
094c000904 Merge branch 'dev' into benchfn 2019-04-10 11:57:05 -07:00
Nick Terrell
f86d4bd1d5
Merge pull request #1576 from terrelln/dict-fuzz
Add new fuzzers and fix exposed bugs
2019-04-10 11:29:15 -07:00
Nick Terrell
5f6ca3c6ce
Merge pull request #1578 from orip/r-flag-typo
Fixed `-r` typo
2019-04-10 11:19:02 -07:00
Yann Collet
90c0462d63 minor presentation refactoring
and removed some // comment style
2019-04-10 10:03:06 -07:00
Ori Peleg
bdeb4786b5 Fixed -r typo 2019-04-10 13:37:41 +03:00
Nick Terrell
c45dec12c5 [fuzzer] Use ZSTD_DCtx_loadDictionary_advanced() half the time 2019-04-09 18:02:22 -07:00
Nick Terrell
10a3d4dca9 [fuzzer] Make the regression_driver work while fuzzers are active 2019-04-09 18:01:49 -07:00
Nick Terrell
824aaa695f [libzstd] Fix ZSTD_decompressDCtx() with a dictionary
* `ZSTD_decompressDCtx()` did not use the dictionary loaded by
  `ZSTD_DCtx_loadDictionary()`.
* Add a unit test.
* A stacked diff uses `ZSTD_decompressDCtx()` in the
  `dictionary_round_trip` and `dictionary_decompress` fuzzers.
2019-04-09 17:59:27 -07:00
Nick Terrell
c5d70b7dbb [fuzzer] Sometimes fuzz with one less output byte
Zstd compression sometimes does different stuff when it has at least
`ZSTD_compressBound()` output bytes, or not. Half of the time fuzz with
`ZSTD_compressBound() - 1` output bytes. Ensure that we have at least
one byte of overhead by disabling either the dictionary ID or checksum.
2019-04-09 16:47:59 -07:00
Nick Terrell
48a6427d22 [libzstd] Fix ZSTD_compress2() for multithreaded compression
`ZSTD_compress2()` wouldn't wait for multithreaded compression to
finish. We didn't find this because ZSTDMT will block when it can
compress all in one go, but it can't do that if it doesn't have enough
output space, or if `ZSTD_c_rsyncable` is enabled.

Since we will already sometimes block when using `ZSTD_e_end`, I've
changed `ZSTD_e_end` and `ZSTD_e_flush` to guarantee maximum forward
progress. This simplifies the API, and helps users avoid the easy bug
that was made in `ZSTD_compress2()`

* Found by the libfuzzer fuzzers.
* Added a test case that catches the problem.
* I will make the fuzzers sometimes allocate less than
  `ZSTD_compressBound()` output space.
2019-04-09 16:24:17 -07:00
Nick Terrell
7a1fde2957 [fuzzer] Add dictionary fuzzers 2019-04-08 21:07:28 -07:00
Nick Terrell
462918560c [fuzzer] Fix stream_round_trip for the new options 2019-04-08 21:06:19 -07:00
Nick Terrell
f871b5144e [fuzz] Use the new advanced API 2019-04-08 20:01:38 -07:00
Nick Terrell
e649fad7aa [dictBuilder] Fix displayLevel for corpus warning
Pass the displaylevel into the corpus warning, because it is used in
fast cover and cover, so it needs to respect the local level.
2019-04-08 20:00:18 -07:00
Nick Terrell
bfcd5b81d7 [libzstd] Don't check the dictID in fuzzing mode
When `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined don't check
the dictID. This check makes the fuzzers job harder, and it is at the
very beginning.
2019-04-08 19:57:41 -07:00
Nick Terrell
1a90133b15
Merge pull request #1575 from terrelln/zstdmt
[libzstd] Remove ZSTDMT from the shared library
2019-04-08 16:51:40 -07:00
Nick Terrell
947548c24f Remove double the from README 2019-04-08 16:50:18 -07:00
Nick Terrell
641e594309 [libzstd] Remove ZSTDMT from the shared object
* Remove ZSTDMT from the shared object by default.
* Provide a macro `ZSTD_LEGACY_MULTITHREADED_API` to override it.
* Document it in `lib/README.md`.
2019-04-07 18:47:52 -07:00
Nick Terrell
d5910a5d94
Merge pull request #1574 from terrelln/examples
Stabilize ZSTD_getDictID_*() functions and clean up examples
2019-04-05 23:29:32 -07:00