AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,681 Commits 1 Branch 0 Tags 27 MiB
cb57c107ff
Commit Graph

4681 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
cb57c107ff zstdmt: minor variable renaming, for clarity 2018-01-17 11:39:07 -08:00
Yann Collet
3e1e57db27 fix fileio progression status update 
The compression % is no longer correct,
since it's no longer possible to make direct correlation
between nb bytes read and nb bytes written
due to large internal buffer inside CCtx
(exacerbated with --long).

The current "fix" is to no longer display the %.

A more complex solution will have to count exactly how much data has been consumed and compressed internally, within CCtx buffers.
 2018-01-16 17:35:00 -08:00
Yann Collet
10c213761a cli: fix for no-MT mode 
when cli is compiled without MT support,
invoking ZSTD_p_nonBlockingMode result in an error code.

This patch only sets ZSTD_p_nonBlockingMode when ZSTD_MULTITHREAD is set, meaning there is MT support.

The error code could also be intentionnally ignored (there is no side effect).
 2018-01-16 17:28:11 -08:00
Yann Collet
1dba98d563 introduced parameter ZSTD_p_nonBlockingMode 
This new parameter makes it possible to call
streaming ZSTDMT with a single thread set
which is non blocking.

It makes it possible for the main thread to do other tasks in parallel
while the worker thread does compression.
Typically, for zstd cli, it means it can do I/O stuff.

Applied within fileio.c, this patch provides non-negligible gains during compression.

Tested on my laptop, with enwik9 (1000000000 bytes) : time zstd -f enwik9

With traditional single-thread blocking mode :
real    0m9.557s
user    0m8.861s
sys     0m0.538s

With new single-worker non blocking mode :
real    0m7.938s
user    0m8.049s
sys     0m0.514s

=> 20% faster
 2018-01-16 16:15:47 -08:00
Yann Collet
6025465e42 ZSTDMT : minor CCtx memory optimization 
can be useful when a compression job only has small amount of data to compress.
 2018-01-16 15:34:41 -08:00
Yann Collet
2e23333094 ZSTDMT can now work in non-blocking mode with 1 thread 
it still fallbacks to single-thread blocking invocation
when input is small (<1job)
or when invoking ZSTDMT_compress(), which is blocking.

Also : fixed a bug in new block-granular compression routine.
 2018-01-16 15:28:43 -08:00
Yann Collet
8e83c5c910 Merge branch 'dev' into progressiveMT 2018-01-16 12:54:33 -08:00
Yann Collet
f0d703243f
Merge pull request #956 from terrelln/mf-struct 
Split ZSTD_CCtx into smaller sub-structures
 2018-01-16 12:37:44 -08:00
Yann Collet
0bc7a75468
Merge pull request #987 from facebook/checkTag 
Check tag
 2018-01-16 11:52:32 -08:00
Yann Collet
f4e58455f6 ensure MOREFLAGS are not lost in root->tests Makefile invocation 2018-01-16 11:50:16 -08:00
Yann Collet
19cb37f837 travis ci : added gcc-7 test 
also added `-Werror` to sanitizer tests
 2018-01-16 11:40:42 -08:00
Nick Terrell
aae267a2e1 Reorganize block state 2018-01-16 11:17:50 -08:00
Nick Terrell
887cd4e35e Split ZSTD_CCtx into smaller sub-structures 2018-01-16 11:17:50 -08:00
Yann Collet
a187486e3c
Merge pull request #990 from krab/dev-cmake-libdir 
CMake: use GNUInstallDirs for library install dir
 2018-01-15 18:55:21 -08:00
Alexey Ivanov
403e2db139 CMake: use GNUInstallDirs for library install dir 
Libraries now will be installed in the correct directory on x86_64 linux systems,
and can be changed with `-DCMAKE_INSTALL_LIBDIR=<dirname>` option.
 2018-01-15 22:48:46 +03:00
Yann Collet
815eddeda4 added tag-triggered test 
ensure tag and libzstd version are compatible
 2018-01-14 17:06:21 -08:00
Yann Collet
f6e17b8c7a added tests/checkTag 
compared provided tag with current libzstd version
 2018-01-14 17:03:45 -08:00
Yann Collet
4792ac6689 make -C tests legacy : minor flag alteration 
ZSTD_LEGACY_SUPPORT is a macro constant,
so it should be part of CPPFLAGS, instead of CFLAGS
 2018-01-14 14:09:17 -08:00
Yann Collet
99a0a8bbd2 tests/Makefile : fixed target allnothread 
ensures MT is disabled
 2018-01-13 22:00:05 -08:00
Yann Collet
f8a05932b9 added make list capability to tests/Makefile 2018-01-13 21:54:21 -08:00
Yann Collet
9477f6529d
Merge pull request #984 from terrelln/dict-load 
Load more dictionary positions into table if empty
 2018-01-13 13:20:42 -08:00
Yann Collet
58ecf13e02 zstdmt : can compress at block granularity 
offering perspective of more accurate progression report.
 2018-01-13 13:18:57 -08:00
Nick Terrell
9a211d1f05 Load more dictionary positions into table if empty 
If the hash table is empty load positions into the hash table
that we would otherwise skip.

| Level | Data Set     | Improvement |
|-------|--------------|-------------|
| 1     | github       | 0.44%       |
| 1     | hg-changelog | 0.13%       |
| 1     | hg-commands  | 1.28%       |
| 1     | hg-manifest  | 0.70%       |
| 3     | github       | 0.74%       |
| 3     | hg-changelog | 0.87%       |
| 3     | hg-commands  | 1.74%       |
| 3     | hg-manifest  | 0.23%       |
 2018-01-12 16:17:22 -08:00
Yann Collet
863b2f8db4
Merge pull request #983 from terrelln/dict-wlog 
Increase windowLog from CDict based on the srcSize when known
 2018-01-12 07:47:43 -08:00
Yann Collet
9d65a5c316
Merge pull request #970 from facebook/dubtlazy 
Delayed update Binary Tree for Lazy strategy
 2018-01-12 07:28:59 -08:00
Nick Terrell
4b7c4e5f41 Add test for cdict window log adjustment 2018-01-11 16:45:16 -08:00
Nick Terrell
b610b777d3 Increase windowLog from CDict based on the srcSize when known 2018-01-11 16:23:21 -08:00
Yann Collet
cacf47cbee Merge branch 'dev' into dubtlazy 
and fixed conflicts
 2018-01-11 13:25:08 -08:00
Yann Collet
04c00f9388
Merge pull request #982 from facebook/fix304 
Fix for #304 and #977 : error during dictionary creation
 2018-01-11 13:20:59 -08:00
Yann Collet
b9a14900ff changed function name to ZSTD_DUBT_findBestMatch() 2018-01-11 12:38:31 -08:00
Yann Collet
1d623e60a1
Merge pull request #981 from facebook/fix976 
fixed bug #976, reported by @indygreg
 2018-01-11 11:40:07 -08:00
Yann Collet
752bae4a48 added warning message 
when pathological dataset is detected
(note : cover_optimize needs -v to display the warning)
 2018-01-11 11:29:28 -08:00
Yann Collet
e8093dde09 fixed #304 
Pathological samples may result in literal section being incompressible.
This case is now detected,
and literal distribution is replaced by one that can be written into the dictionary.
 2018-01-11 11:16:32 -08:00
Yann Collet
218e9fe0fc added a test case for dictBuilder failure 
cyclic data set makes the entropy stage fails
now, onto a fix for #304 ...
 2018-01-11 09:42:38 -08:00
Yann Collet
55cdbdb465
Merge pull request #980 from conor42/dev 
Fix break condition in decompression noise test
 2018-01-11 08:33:35 -08:00
Yann Collet
2103a62b3d fixed minor warning on prototype definition 2018-01-11 04:49:19 -08:00
Yann Collet
ff795580f2 fixed bug #976, reported by @indygreg 
constants in zstd.h should not depend on MIN() macro which existence is not guaranteed.

Added a test to check the specific constants.
The test is a bit too specific.
But I have found no way to control a more generic "are all macro already defined" condition,
especially as this is a valid construction (the missing macro might be defined later, intentionnally).
 2018-01-10 20:33:45 -08:00
conor42
0e88f6e97b
Fix break condition in decompression noise test 
The bug prevents noise being added
 2018-01-11 11:42:58 +10:00
Yann Collet
06995775b0
Merge pull request #979 from facebook/memdoc 
Update API documentation related to memory management capabilities
 2018-01-10 17:39:37 -08:00
Yann Collet
1edf33764e
Merge pull request #974 from terrelln/dstfile 
[fileio] Improve safety of output file modifications
 2018-01-10 19:02:48 +01:00
Yann Collet
292eeb672f api doc : grouped all ZSTD_create*_advanced() functions together 
in a new "custom memory allocator" paragraph
which is itself part of "memory management" category.

This makes it simpler to see the relation between the type and its usages.
 2018-01-10 09:07:47 -08:00
Yann Collet
3ea156368c API doc : grouped ZSTD_initStatic*() together 
within "memory management" category.
 2018-01-10 08:49:50 -08:00
Yann Collet
b17fb488b0 fixed msan test 
a pointer calculation was wrong in a corner case
 2018-01-06 20:50:36 +01:00
Yann Collet
76321d6bbc
Merge pull request #965 from facebook/tests_obj 
%.o objects files in /tests
 2018-01-06 13:15:53 +01:00
Yann Collet
658d6b8588 Merge branch 'dev' into dubtlazy 2018-01-06 12:40:58 +01:00
Yann Collet
a927fae2a1 fixed ZSTD_reduceIndex() 
following suggestions from @terrelln.
Also added some comments to present logic behind ZSTD_preserveUnsortedMark().
 2018-01-06 12:31:26 +01:00
Yann Collet
3d578eef0c minor tests/Makefile change, as suggested by @terrelln 
https://github.com/facebook/zstd/pull/965#discussion_r160008006
 2018-01-06 07:13:09 +01:00
Yann Collet
752880ffed
Merge pull request #963 from facebook/benchfix 
fix: bench can accept hlog custom parameter
 2018-01-06 06:57:02 +01:00
Nick Terrell
ed9611dc62 [fileio] Don't call FIO_remove() on stdout or /dev/null 2018-01-05 11:50:24 -08:00
Nick Terrell
282ad05e0a [fileio] Use FIO_remove() everywhere for safety 2018-01-05 11:44:45 -08:00