AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,624 Commits 1 Branch 0 Tags 27 MiB
be3d2e2de8
Commit Graph

6624 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
a968099038 minor code cleaning for new index invalidation strategy 2019-05-31 16:52:37 -07:00
Yann Collet
d605f482c7 make double_fast compatible with new index invalidation strategy 2019-05-31 16:50:04 -07:00
Yann Collet
a30febaeeb Made fast strategy compatible with new offset validation strategy 
fast mode does the same thing as before :
it pre-emptively invalidates any index that could lead to offset > maxDistance.
It's supposed to help speed.

But this logic is performed inside zstd_fast,
so that other strategies can select a different behavior.
 2019-05-31 16:34:55 -07:00
Yann Collet
58adb1059f extended exact window size to greedy/lazy modes 2019-05-31 16:08:48 -07:00
Yann Collet
bc601bdc6d first implementation of small window size for btopt 
noticeably improves compression ratio
when window size is small (< 18).

enwik7	level 19

windowLog	`dev`	`smallwlog`	improvement
23	3.577	3.577	0.02%
22	3.536	3.538	0.06%
21	3.462	3.467	0.14%
20	3.364	3.377	0.39%
19	3.244	3.272	0.86%
18	3.110	3.166	1.80%
17	2.843	3.057	7.53%
16	2.724	2.943	8.04%
15	2.594	2.822	8.79%
14	2.456	2.686	9.36%
13	2.312	2.523	9.13%
12	2.162	2.361	9.20%
11	2.003	2.182	8.94%
 2019-05-31 15:55:12 -07:00
Yann Collet
b13a9207f9
Merge pull request #1623 from facebook/fullbench 
fullbench minor improvements
 2019-05-31 14:40:19 -07:00
Yann Collet
78056a3e9c consistency : changed constant name to kSampleSizeDefault 
as suggested by @terrelln.

added <assert.h> include
 2019-05-30 10:13:56 -07:00
Yann Collet
ab3625cffd define CONTROL, to ensure check cannot be disabled 
assert() can be disabled with NDEBUG
 2019-05-29 16:18:22 -07:00
Yann Collet
904d4da239 fullbench : minor refactoring, for readability 2019-05-29 16:08:49 -07:00
Yann Collet
ed38b645db fullbench: pass proper parameters in scenario 43 2019-05-29 15:26:06 -07:00
Yann Collet
b84274da0f
Merge pull request #1622 from facebook/nextToUpdate3 
nextToUpdate3 does not need to be maintained outside of zstd_opt.c
 2019-05-28 18:26:21 -07:00
Yann Collet
c63081623f fullbench 43: ensure context is freed after each usage 2019-05-28 17:27:52 -07:00
Yann Collet
eb6b199038 fullbench: added streaming with fresh CCtx scenario 
worst case situation, where context must be recreated every time
but without knowledge of the input size
thus sizing the context for some large input.
 2019-05-28 16:50:49 -07:00
Yann Collet
4c4149452c fullbench can now select sample size with command -B# 2019-05-28 16:37:03 -07:00
Yann Collet
9719fd616c removed nextToUpdate3 from ZSTD_window 
it's now a local variable of ZSTD_compressBlock_opt()
 2019-05-28 16:18:12 -07:00
Yann Collet
33dabc8c80 get bt matches : made it a bit clearer which parameters are input and output 2019-05-28 16:11:32 -07:00
Yann Collet
327cf6fac1 nextToUpdate3 does not need to be maintained outside of zstd_opt.c 
It's re-synchronized with nextToUpdate at beginning of each block.
It only needs to be tracked from within zstd_opt block parser.

Made the logic clear, so that no code tried to maintain this variable.

An even better solution would be to make nextToUpdate3
an internal variable of ZSTD_compressBlock_opt_generic().
That would make it possible to remove it from ZSTD_matchState_t,
thus restricting its visibility to only where it's actually useful.

This would require deeper changes though,
since the matchState is the natural structure to transport parameters into and inside the parser.
 2019-05-28 15:26:52 -07:00
Yann Collet
6453f8158f complementary code comments 
on variables used / impacted during maxDist check
 2019-05-28 14:12:16 -07:00
Yann Collet
4baecdf72a added comments to better understand enforceMaxDist() 2019-05-28 13:15:48 -07:00
Michael Forney
1bc77ee174 Skip --adapt and --rsyncable tests when built without thread support 
These options require ZSTD_MULTITHREAD.
 2019-05-26 03:20:37 -07:00
Jonathan Scott
3f5737afd7 CMake: Check for existing custom target 'uninstall' 2019-05-25 17:30:04 -07:00
Felix Handte
21bb78e908
Merge pull request #1618 from felixhandte/zstgrep-f-flag 
Handle `-f` Flag in `zstgrep`
 2019-05-24 20:59:11 -04:00
W. Felix Handte
6a0638048a Add Test 2019-05-24 17:21:44 -04:00
W. Felix Handte
61025d5b7d zstdgrep: Handle -f Flag 2019-05-24 16:55:59 -04:00
W. Felix Handte
ff0be17cf7 Build Manual 2019-05-24 16:55:43 -04:00
Tyler-Tran
cb47871a0a [dictBuilder] Be more specific than ERROR(generic) (#1616) 
* Specify errors at a finer granularity than `ERROR(generic)`.
* Add tests for bad parameters in the dictionary builder.
 2019-05-22 18:57:50 -07:00
Nick Terrell
b114477f2a
Merge pull request #1613 from neheb/patch-1 
fileio: Disable BACKTRACE under uClibc
 2019-05-06 17:12:03 -07:00
Rosen Penev
23b5ee1236
fileio: Disable BACKTRACE under uClibc 
uClibc does not support BACKTRACE. It also defines __GLIBC__ and minor as both 2.
This fixes compilation under such a setup.
 2019-05-06 15:02:47 -07:00
Nick Terrell
bfbef51afb
Merge pull request #1610 from terrelln/pool-test 
[test] Fix poolTests deadlock and flakiness
 2019-05-02 15:13:50 -07:00
Nick Terrell
81f1f76479 [tests] Only sleep 10ms in poolTests 2019-05-02 14:35:25 -07:00
Nick Terrell
2953d45a0f [test] Fix poolTests deadlock and flakiness 2019-05-02 13:23:41 -07:00
lzutao
1d70bc3ba9 meson: Enable testing in CI (#1609) 
* tests: Fix shellcheck warnings in playTests.sh

* tests: Do not use ../programs which is relative to tests dirs

This commit fixes error when running playTests.sh in Meson.
Mesonbuild runs out of tree, so ./datagen not in `zstd/tests` dir,
it lies in <mesonbuilddir>/tests. This leads to ../programs invalid.

* tests: Replace relative paths for zstd/tests dir

* playTests: Set shell options explicitly, not in shebang

* playTests: Replace echo -e with printf

* meson: Fix test-zstd

Use std=gnu99 to build and test just like `make test`.

* meson: Fix legacy test

* meson: Enable testing in CI

Run build under release mode for faster test time.

* meson: Increase timeout time for test-zstream
 2019-05-02 12:35:37 -07:00
Nick Terrell
f8178ec74e
Merge pull request #1602 from lzutao/meson 
meson: Update default project version
 2019-04-30 08:45:00 -07:00
Lzu Tao
5d900ff747 meson: Error out if fail to extracted version number 2019-04-30 13:12:00 +07:00
Nick Terrell
69baaee3e4
Merge pull request #1604 from vejnar/dev 
Make zstdcat to follow symbolic links
 2019-04-29 17:02:42 -07:00
cev
3e1e49d775 add test for zstdcat and zcat on symlink 2019-04-29 18:47:47 -04:00
Nick Terrell
9ef732f653
Merge pull request #1603 from sjnam/examples-test-fail 
fails to "make test" in examples
 2019-04-29 13:45:58 -07:00
cev
c4a40dbf65 set followLinks option true to cat, zcat and gzcat programs 2019-04-27 19:06:29 -04:00
sjnam
bee9e5f38c fix test fail 2019-04-28 05:05:45 +09:00
Lzu Tao
4107b734bd meson: Update default project version 
* Update to use ninja v1.9.0 on CI
 2019-04-27 01:42:22 +07:00
Nick Terrell
585b5a1039
Merge pull request #1600 from myzhang1029/long-tell 
Fix #1591 - Not building on MinGW-W64
 2019-04-24 21:05:25 -07:00
myzhang1029
f837326f11
Fix #1591 - Not building on MinGW-W64 
Add a static function LONG_TELL for the forth #if branch
 2019-04-24 20:06:05 +08:00
Nick Terrell
3d673f3275
Merge pull request #1598 from terrelln/decode-seq 
[libzstd] Error if all sequence bits aren't consumed
 2019-04-23 15:44:20 -07:00
Nick Terrell
5f228f8db2 [libzstd] Add a ZSTD_STATIC_ASSERT for BIT_DStream_status 2019-04-23 14:22:16 -07:00
Nick Terrell
a892e25374 [libzstd] Error if all sequence bits aren't consumed 2019-04-23 14:07:36 -07:00
Nick Terrell
b758250eb1
Merge pull request #1595 from terrelln/legacy-fix 
[legacy] Fix ZSTDv0*_decodeSequence()
 2019-04-19 12:23:50 -07:00
Nick Terrell
0fd322f812 [legacy] Fix ZSTDv0*_decodeSequence() 
* Version <= 0.5 could read beyond the end of `dumps`, which points into
  the input buffer.
* Check the validity of `dumps` before using it, if it is out of bounds
  return garbage values. There is no return code for this function.
* Introduce `MEM_readLE24()` for simplicity, since I don't want to trust
  that there is an extra byte after `dumps`.
 2019-04-19 11:34:52 -07:00
Nick Terrell
9ad7ea44ec
Merge pull request #1594 from terrelln/legacy-fix 
[legacy] Fix Huffman jump table reads in v01 and v05
 2019-04-18 17:29:32 -07:00
Nick Terrell
2536771134 [legacy] Fix Huffman jump table reads in v01 and v05 2019-04-18 16:20:42 -07:00
Nick Terrell
a8db4bd3fe
Merge pull request #1593 from terrelln/legacy-fix 
[fuzzer] Run fuzzers in legacy mode and fix legacy code
 2019-04-18 16:07:42 -07:00