Commit Graph

2708 Commits

Author SHA1 Message Date
Yann Collet
3ebec8383b Merge pull request #549 from iburinoc/matchoverlenwrite
Fix wildcopy overwriting data still in window
2017-02-16 08:39:52 -08:00
Sean Purcell
887eaa9e21 Fix wildcopy overwriting data still in window 2017-02-15 16:43:45 -08:00
Yann Collet
2252d29a5a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-15 12:00:50 -08:00
Yann Collet
4596037042 updated fse version
feature minor refactoring (removing FSE_abs())
also : fix a few minor issues recently introduced in examples
2017-02-15 12:00:03 -08:00
Yann Collet
44f82d781f Merge pull request #545 from terrelln/force-window
[zstdmt] Fix MSAN failure with ZSTD_p_forceWindow
2017-02-15 10:20:15 -08:00
Yann Collet
f0b9a8dddb Merge pull request #547 from inikep/dev11
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 12:29:00 -08:00
Yann Collet
c09d16ba8c preset behavior for gzip, gunzip and gzcat
when zstd is called through a link named gzip, gunzip or gzcat,
provides the same behavior as the related program.
gzip compresses using --format=gz
both gzip and gunzip enable --rm by default
2017-02-14 10:45:19 -08:00
Yann Collet
aca067f19d Merge pull request #543 from inikep/gz_compress
Gz compress
2017-02-14 10:30:22 -08:00
Nick Terrell
74b81ada25 Don't run test-pool with QEMU
> make test -n
    ...
    ./pool
    > make test -n QEMU_SYS=valgrind
    ...
    ./legacy
    # ./pool not run
2017-02-14 10:08:18 -08:00
Yann Collet
9696bfc2ad Merge pull request #544 from ds77/avoid-empty
Portable way to avoid empty unit warning in threading.c
2017-02-14 00:54:55 -08:00
Przemyslaw Skibinski
ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Yann Collet
1decff24b8 Merge pull request #546 from zefanxu/dev
fix memory leak
2017-02-14 00:49:06 -08:00
Przemyslaw Skibinski
970419535f fixed function name (2) 2017-02-14 09:47:29 +01:00
Przemyslaw Skibinski
bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski
abd6302423 Windows resources updated to v1.1.4 2017-02-14 09:39:09 +01:00
Przemyslaw Skibinski
442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski
b876b96ce1 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-14 09:26:03 +01:00
Przemyslaw Skibinski
98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
zefanxu2
2bb6fc2a94 fix memory leak 2017-02-13 21:12:59 -06:00
Nick Terrell
ecf90ca24b [zstdmt] Fix MSAN failure with ZSTD_p_forceWindow
Reproduction steps:

```
make zstreamtest CC=clang CFLAGS="-O3 -g -fsanitize=memory -fsanitize-memory-track-origins"
./zstreamtest -vv -t4178 -i4178 -s4531
```

How to get to the error in gdb (may be a more efficient way):

* 2 breaks at zstd_compress.c:2418  -- in ZSTD_compressContinue_internal()
* 2 breaks at zstd_compress.c:2276  -- in ZSTD_compressBlock_internal()
* 1 break at zstd_compress.c:1547

Why the error occurred:

When `zc->forceWindow == 1`, after calling `ZSTD_loadDictionaryContent()` we
have `zc->loadedDictEnd == zc->nextToUpdate == 0`. But, we've really loaded up
to `iend` into the dictionary. Then in `ZSTD_compressBlock_internal()` we see
that `current > zc->nextToUpdate + 384`, so we load the last 192 bytes a second
time. In this case the bytes we are loading are a block of all 0s, starting in
the previous block. So when we are loading the last 192 bytes, we find a `match`
in the future, 183 bytes beyond `ip`. Since the block is all 0s, the match
extends to the end of the block. But in `ZSTD_count()` we only check that
`pIn < pInLoopLimit`, but since `pMatch > pIn`, `pMatch` eventually points past
the end of the buffer, causing the MSAN failure.

The fix:

The line changed sets sets `zc->nextToUpdate` to the end of the dictionary.
This is the behavior that existed before `ZSTD_p_forceWindow` was introduced.
This fixes the exposing test case. Since the code doesn't fail without
`zc->forceWindow`, it makes sense that this works. I've run the command
`./zstreamtest -T2mn` 64 times without failures. CI should also verify nothing
obvious broke.
2017-02-13 19:11:22 -08:00
Yann Collet
58af614ef2 push version and NEWS to v1.1.4 2017-02-13 18:32:44 -08:00
ds77
08e6a88a97 avoid empty translation unit warning without #pragma 2017-02-14 00:46:47 +01:00
Przemyslaw Skibinski
1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski
48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev'
# Conflicts:
#	programs/fileio.c
2017-02-13 21:35:39 +01:00
Yann Collet
db2666c10c Merge pull request #536 from iburinoc/multiframe
Simple API multiframe decoding
2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski
64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
Yann Collet
73d7a15707 Merge pull request #542 from ds77/large-sparse-fix
zstdcli: fix writing 2GB+ sparse files under Windows
2017-02-13 11:57:35 -08:00
Przemyslaw Skibinski
5a3bb05bb2 Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress
# Conflicts:
#	programs/Makefile
2017-02-13 20:47:01 +01:00
Przemyslaw Skibinski
35bf23c086 MinGW-w64 requires _FILE_OFFSET_BITS 64 2017-02-13 13:57:29 +01:00
Przemyslaw Skibinski
09c8e5390d __builtin_bswap requires gcc 4.3+ 2017-02-13 12:45:53 +01:00
Przemyslaw Skibinski
5a7a612607 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-13 12:15:10 +01:00
ds77
6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
Przemyslaw Skibinski
2f70fec81b Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-13 09:48:15 +01:00
ds77
168d9b8006 fix seeking 2GB+ files under Windows
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
2017-02-12 16:05:19 +01:00
Yann Collet
acfa151622 Merge pull request #540 from ds77/dev-stat64-fix
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
2017-02-11 21:08:08 -08:00
Yann Collet
145f450a1b Merge pull request #541 from inikep/mingw
fixed "mingw32" AppVeyor target
2017-02-11 21:07:53 -08:00
Przemyslaw Skibinski
eb132530cd revert last commit 2017-02-10 21:15:49 +01:00
Sean Purcell
d7bfcac18a Expose frameSrcSize to experimental API 2017-02-10 11:55:44 -08:00
Przemyslaw Skibinski
645f5b9856 fix for original MinGW 2017-02-10 20:09:28 +01:00
Sean Purcell
5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
ds77
45f0c207ab use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks. 2017-02-10 18:37:57 +01:00
Przemyslaw Skibinski
456330948a Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw
# Conflicts:
#	programs/platform.h
2017-02-10 13:38:54 +01:00
Yann Collet
bbba42acd1 Merge pull request #537 from terrelln/small-bugs
Fix small bugs
2017-02-10 04:35:43 -08:00
-
7ec315df0d fix previous commit
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
2017-02-10 13:27:43 +01:00
Przemyslaw Skibinski
192e20338f appveyor.yml: fixed clang test 2017-02-10 13:10:25 +01:00
Yann Collet
a28c34cb7a Merge pull request #538 from iburinoc/errorstring
Fix ZSTD_getErrorString and add tests
2017-02-10 03:59:56 -08:00
Yann Collet
2333afcebe Merge pull request #533 from inikep/dev11
--priority=rt
2017-02-10 03:50:08 -08:00
Przemyslaw Skibinski
7059784192 appveyor.yml: reordering of tests 2017-02-10 12:41:31 +01:00
Przemyslaw Skibinski
bc2bfa4c9a fix missing " 2017-02-10 12:32:30 +01:00
Przemyslaw Skibinski
cb8d2d9d96 appveyor.yml: add clang target 2017-02-10 12:01:14 +01:00