AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
7,675 Commits 1 Branch 0 Tags 27 MiB
93cec0c1d6
Commit Graph

7675 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
76e726e3be updated documentation for ZSTD_estimate*() 
make it clearer that tighter memory estimation
can be provided using advanced functions
on the condition of a defined input size bound.
 2020-05-11 19:21:50 -07:00
Nick Terrell
80d3585e31 [lib] Fix lazy parser with dictionary + repcodes 2020-05-11 19:04:30 -07:00
Nick Terrell
1185dfb8d1 [fuzz] Add raw dictionary content fuzzer 2020-05-11 19:03:33 -07:00
Nick Terrell
301a62fe08 [fuzz] Fix compress bound for dictionary_round_trip 2020-05-11 19:00:52 -07:00
Bimba Shrestha
e7d2391e9a
[doc] measuring performance docs (#2117) 
* performance measuring docs

* spelling

* combining advanced and simple section

* zstd benchmark title change
 2020-05-11 18:55:01 -07:00
Yann Collet
ed9eb17471
Merge pull request #2124 from facebook/initStatic_tests 
re-inforced tests for initStaticCCtx
 2020-05-11 18:54:22 -07:00
Yann Collet
f13eb2ec01
Merge pull request #2123 from facebook/zstd_help 
updated `zstd -h`
 2020-05-11 18:54:07 -07:00
Yann Collet
91ad01218e updated initStatic tests 
differentiate small CCtx for small inputs
from full CCtx
from CStream contexts.

Ensure allocation & resize tests are more precise.
 2020-05-11 18:50:10 -07:00
Yann Collet
608f1bfc4c fixed context downsize with initStatic 
When context is created using initStatic,
no resize is possible.

fix : only bump oversizeDuration when !initStatic
 2020-05-11 18:16:38 -07:00
Yann Collet
58227db405 Merge branch 'dev' into initStatic_tests 2020-05-11 16:51:13 -07:00
Yann Collet
d8b40fe0be
Merge pull request #2129 from felixhandte/fix-cctx-size-estimate 
Fix ZSTD_estimateCCtxSize() Under ASAN
 2020-05-11 16:36:46 -07:00
W. Felix Handte
c6636afbbb Fix ZSTD_estimateCCtxSize() Under ASAN 
`ZSTD_estimateCCtxSize()` provides estimates for one-shot compression, which
is guaranteed not to buffer inputs or outputs. So it ignores the sizes of the
buffers, assuming they'll be zero. However, the actual workspace allocation
logic always allocates those buffers, and when running under ASAN, the
workspace surrounds every allocation with 256 bytes of redzone. So the 0-sized
buffers end up consuming 512 bytes of space, which is accounted for in the
actual allocation path through the use of `ZSTD_cwksp_alloc_size()` but isn't
in the estimation path, since it ignores the buffers entirely.

This commit fixes this.
 2020-05-11 18:58:19 -04:00
Nick Terrell
738f49178b
Merge pull request #2128 from terrelln/benches 
Update benchmarks and add fast mode
 2020-05-11 13:53:18 -07:00
Nick Terrell
f2f86b50ae Update benchmarks and add fast mode 2020-05-11 12:36:36 -07:00
Bimba Shrestha
0c04e641e9
Merge pull request #2127 from bimbashrestha/docs1 
[doc] documenting non-supported grep options for zstdgrep to README
 2020-05-11 12:51:35 -05:00
Bimba Shrestha
ef467fd428 documenting non-supported grep options for zstdgrep 2020-05-11 12:22:45 -05:00
Yann Collet
93ff2fb329
Merge pull request #2120 from facebook/clevel_doc 
updated zstd CLI documentation
 2020-05-11 09:42:33 -07:00
Mingye Wang
c3398d2dda
programs/Makefile: make it actually work with clang 
Recent versions of clang seem to require a step of conversion from the profraw file to a "compiler-readable" profdata file to let PGO work.
 2020-05-10 16:36:08 +08:00
Yann Collet
dd026ca505 re-inforced tests for initStaticCCtx 
ensure that `estimateCCtxSize()` works as intended.
 2020-05-09 11:30:45 -07:00
Felix Handte
aa9474ee9a
Merge pull request #2122 from felixhandte/unused-variables 
Compile Under `-Wunused-variable`
 2020-05-08 18:17:38 -04:00
Yann Collet
37b2dafbed minor clarification on -o file 2020-05-08 14:20:47 -07:00
Yann Collet
5c67a4e737 Merge branch 'dev' of github.com:facebook/zstd into dev 2020-05-08 14:15:56 -07:00
Yann Collet
e5366bee16 updated zstd -h 
to help readability.
Group arguments into categories,
try to give better visibility to more common arguments.
 2020-05-08 14:14:46 -07:00
Felix Handte
3b39ce6b52
Merge pull request #2116 from felixhandte/pkg-config-again-again 
Fix pkg-config File Generation Again Again
 2020-05-08 17:14:43 -04:00
W. Felix Handte
376c26bf56 Use Unused Variable 2020-05-08 16:43:39 -04:00
Yann Collet
d564d56440 document additional compilation macros 
ZSTD_NOBENCH, ZSTD_NODICT, ZSTD_NOCOMPRESS, ZSTD_NODECOMPRESS
 2020-05-08 13:35:59 -07:00
W. Felix Handte
87c541c5f9 Only Trigger libzstd.pc Build on Unix-Like Platforms 
We don't even define the rule on unsupported platforms.
 2020-05-08 16:11:32 -04:00
W. Felix Handte
85801b99d7 Test libzstd.pc Build on Travis OS X 2020-05-08 16:11:32 -04:00
W. Felix Handte
78aa9373cb Add libzstd.pc Build to More Aggregate Targets in Makefiles 2020-05-08 16:11:32 -04:00
W. Felix Handte
15561bcf74 Fix pkg-config File Generation Again Again 
Resubmission of #2001. This switches the `sed` invocations to use `-E`,
extended regex syntax, which is better standardized across platforms.
I guess.

Same test plan:

```
make -C lib clean libzstd.pc
cat lib/libzstd.pc

echo # should fail
make -C lib clean libzstd.pc     LIBDIR=/foo
make -C lib clean libzstd.pc INCLUDEDIR=/foo
make -C lib clean libzstd.pc     LIBDIR=/usr/localfoo
make -C lib clean libzstd.pc INCLUDEDIR=/usr/localfoo
make -C lib clean libzstd.pc     LIBDIR=/usr/local/lib     prefix=/foo
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local/include prefix=/foo

echo # should succeed
make -C lib clean libzstd.pc     LIBDIR=/usr/local/foo
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local/foo
make -C lib clean libzstd.pc     LIBDIR=/usr/local/
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local/
make -C lib clean libzstd.pc     LIBDIR=/usr/local
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local
make -C lib clean libzstd.pc     LIBDIR=/tmp/foo prefix=/tmp
make -C lib clean libzstd.pc INCLUDEDIR=/tmp/foo prefix=/tmp
make -C lib clean libzstd.pc     LIBDIR=/tmp/foo prefix=/tmp/foo
make -C lib clean libzstd.pc INCLUDEDIR=/tmp/foo prefix=/tmp/foo

echo # should also succeed
make -C lib clean libzstd.pc prefix=/foo LIBDIR=/foo/bar INCLUDEDIR=/foo/
cat lib/libzstd.pc

mkdir out
cd out
cmake ../build/cmake
make
cat lib/libzstd.pc
```
 2020-05-08 16:11:32 -04:00
Felix Handte
a036f60255
Merge pull request #2078 from nocnokneo/fix-destdir-installs 
CMake: Fix DESTDIR installs and the Package Config file
 2020-05-08 16:09:44 -04:00
Yann Collet
4f2cb944a8 Merge branch 'dev' into clevel_doc 2020-05-08 11:59:45 -07:00
Nick Terrell
32b5a8ec41
Merge pull request #2118 from Artoria2e5/patch-1 
programs/README.md: update help text
 2020-05-08 11:13:44 -07:00
Yann Collet
e873697625 updated documentation regarding ZSTD_CLEVEL 
make it clearer that it's useful for `tar --zstd`.
 2020-05-08 10:51:37 -07:00
Bimba Shrestha
602dae3509
Merge pull request #2119 from bimbashrestha/mingw 
[build] Adding $TARGET_SYSTEM for cross-compilation Fedora fix
 2020-05-08 10:54:06 -05:00
Bimba Shrestha
df9e5b6f4c adding 2020-05-07 22:07:40 -05:00
Yann Collet
efc656c9a6
Merge pull request #2114 from facebook/verbose 
support for verbose make
 2020-05-07 13:16:33 -07:00
Yann Collet
1afe57cff7
Merge pull request #2112 from facebook/cfast 
small speed improvement for strategy fast
 2020-05-07 13:13:34 -07:00
caoyzh
969ba4f2b9 Change the modification of ZSTD_wildcopy() 2020-05-07 13:10:46 -07:00
caoyzh
a7e34ff693 revert ZSTD_reduceTable_internal()'s modificatiion 2020-05-07 13:10:46 -07:00
caoyzh
9e802ede9c Modify indent of comments 2020-05-07 13:10:46 -07:00
caoyzh
7f75f05e84 Change "arm_neon.h" to system include <arm_neon.h> 2020-05-07 13:10:46 -07:00
caoyzh
b2e56f7f7f Optimize compression by using neon function. 2020-05-07 13:10:46 -07:00
Mingye Wang
37d10b6c75
programs/README.md: update help text 
I noticed that the help is outdated as there is no mention of --adapt. Y'all should probably put "update help text" into your release checklist.
 2020-05-08 02:52:18 +08:00
Nick Terrell
45c66dd298 [zdict] Stabilize ZDICT_finalizeDictionary() 2020-05-07 10:37:01 -07:00
Bimba Shrestha
625924774e
Merge pull request #2115 from bimbashrestha/docs 
[doc] Adding static analysis section and note
 2020-05-07 12:03:33 -05:00
Bimba Shrestha
1b1cc3d1b7 adding some static-anslysis docs 2020-05-07 10:38:06 -05:00
Yann Collet
cf854f4660 support for verbose make 
A commonly accepted makefile idiom is V=1 or VERBOSE=1
to request the printing of all commands.

This is not "default" though, and must be manually added.

Example :
Before :
```
make libzstd
compiling dynamic library 1.4.5
creating versioned links

make libzstd V=1
compiling dynamic library 1.4.5
creating versioned links
```

After :
```
make libzstd
compiling dynamic library 1.4.5
creating versioned links

make libzstd V=1
compiling dynamic library 1.4.5
cc -DXXH_NAMESPACE=ZSTD_ -DZSTD_LEGACY_SUPPORT=5 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -O3 common/debug.c common/entropy_common.c common/error_private.c common/fse_decompress.c common/pool.c common/threading.c common/xxhash.c common/zstd_common.c compress/fse_compress.c compress/hist.c compress/huf_compress.c compress/zstd_compress.c compress/zstd_compress_literals.c compress/zstd_compress_sequences.c compress/zstd_compress_superblock.c compress/zstd_double_fast.c compress/zstd_fast.c compress/zstd_lazy.c compress/zstd_ldm.c compress/zstd_opt.c compress/zstdmt_compress.c decompress/huf_decompress.c decompress/zstd_ddict.c decompress/zstd_decompress.c decompress/zstd_decompress_block.c deprecated/zbuff_common.c deprecated/zbuff_compress.c deprecated/zbuff_decompress.c dictBuilder/cover.c dictBuilder/divsufsort.c dictBuilder/fastcover.c dictBuilder/zdict.c legacy/zstd_v05.c legacy/zstd_v06.c legacy/zstd_v07.c -shared -fPIC -fvisibility=hidden -Wl,-soname=libzstd.so.1 -o libzstd.so.1.4.5
creating versioned links
ln -sf libzstd.so.1.4.5 libzstd.so.1
ln -sf libzstd.so.1.4.5 libzstd.so
```
 2020-05-07 08:04:10 -07:00
Yann Collet
54144285fd small speed improvement for strategy fast 
gcc 9.3.0 :
kennedy : 459 -> 466
silesia : 360 -> 365
enwik8  : 267 -> 269

clang 10.0.0 :
kennedy : 436 -> 441
silesia : 364 -> 366
enwik8  : 271 -> 272
 2020-05-07 06:15:58 -07:00
Nick Terrell
5717bd39ee [lib] Fix NULL pointer dereference 
When the output buffer is `NULL` with size 0, but the frame content size
is non-zero, we will write to the NULL pointer because our bounds check
underflowed.

This was exposed by a recent PR that allowed an empty frame into the
single-pass shortcut in streaming mode.

* Fix the bug.
* Fix another NULL dereference in zstd-v1.
* Overflow checks in 32-bit mode.
* Add a dedicated test.
* Expose the bug in the dedicated simple_decompress fuzzer.
* Switch all mallocs in fuzzers to return NULL for size=0.
* Fix a new timeout in a fuzzer.

Neither clang nor gcc show a decompression speed regression on x86-64.
On x86-32 clang is slightly positive and gcc loses 2.5% of speed.

Credit to OSS-Fuzz.
 2020-05-06 12:09:02 -07:00