Commit Graph

4885 Commits

Author SHA1 Message Date
Yann Collet
d14cc881b0 zstdmt : fixed very large window sizes
would create too large buffers,
since default job size == window size * 4.

This would crash on 32-bit systems.

Also : jobSize being a 32-bit unsigned, it cannot be >= 4 GB,
so the formula was failing for large window sizes >= 1 GB.
Fixed now : max job Size is 2 GB, whatever the window size.
2018-01-17 12:39:58 -08:00
Yann Collet
58dd7de640 zstdmt: fixed an endless loop on allocation failure
this happened on 32-bits build when requiring a too large input buffer,
typically on wlog=29, creating jobs of 2 GB size.

also : zstd32 now compiles with multithread support enabled by default
(can be disabled with HAVE_THREAD=0)
2018-01-17 12:10:15 -08:00
Nick Terrell
16bd0fd4df Reduce size of ZSTD_CDict
Shaves 492,076 B off of the `ZSTD_CDict`.
The size of a `ZSTD_CDict` created from a 112,640 B dictionary is:

| Level | Before (B) | After (B) |
|-------|------------|-----------|
| 1     | 648,448    | 156,412   |
| 3     | 1,140,008  | 647,932   |
2018-01-17 11:50:49 -08:00
Yann Collet
cb57c107ff zstdmt: minor variable renaming, for clarity 2018-01-17 11:39:07 -08:00
Alexey Ivanov
22303da601 CMake: fixed multithreading build on Windows
`ZSTD_MULTITHREAD_SUPPORT` option fixed for Windows.

Signed-off-by: Alexey Ivanov <alexey.ivanes@gmail.com>
2018-01-17 10:27:52 +03:00
Yann Collet
3e1e57db27 fix fileio progression status update
The compression % is no longer correct,
since it's no longer possible to make direct correlation
between nb bytes read and nb bytes written
due to large internal buffer inside CCtx
(exacerbated with --long).

The current "fix" is to no longer display the %.

A more complex solution will have to count exactly how much data has been consumed and compressed internally, within CCtx buffers.
2018-01-16 17:35:00 -08:00
Yann Collet
10c213761a cli: fix for no-MT mode
when cli is compiled without MT support,
invoking ZSTD_p_nonBlockingMode result in an error code.

This patch only sets ZSTD_p_nonBlockingMode when ZSTD_MULTITHREAD is set, meaning there is MT support.

The error code could also be intentionnally ignored (there is no side effect).
2018-01-16 17:28:11 -08:00
Yann Collet
1dba98d563 introduced parameter ZSTD_p_nonBlockingMode
This new parameter makes it possible to call
streaming ZSTDMT with a single thread set
which is non blocking.

It makes it possible for the main thread to do other tasks in parallel
while the worker thread does compression.
Typically, for zstd cli, it means it can do I/O stuff.

Applied within fileio.c, this patch provides non-negligible gains during compression.

Tested on my laptop, with enwik9 (1000000000 bytes) : time zstd -f enwik9

With traditional single-thread blocking mode :
real    0m9.557s
user    0m8.861s
sys     0m0.538s

With new single-worker non blocking mode :
real    0m7.938s
user    0m8.049s
sys     0m0.514s

=> 20% faster
2018-01-16 16:15:47 -08:00
Yann Collet
6025465e42 ZSTDMT : minor CCtx memory optimization
can be useful when a compression job only has small amount of data to compress.
2018-01-16 15:34:41 -08:00
Yann Collet
2e23333094 ZSTDMT can now work in non-blocking mode with 1 thread
it still fallbacks to single-thread blocking invocation
when input is small (<1job)
or when invoking ZSTDMT_compress(), which is blocking.

Also : fixed a bug in new block-granular compression routine.
2018-01-16 15:28:43 -08:00
Yann Collet
8e83c5c910 Merge branch 'dev' into progressiveMT 2018-01-16 12:54:33 -08:00
Yann Collet
f0d703243f
Merge pull request #956 from terrelln/mf-struct
Split ZSTD_CCtx into smaller sub-structures
2018-01-16 12:37:44 -08:00
Yann Collet
0bc7a75468
Merge pull request #987 from facebook/checkTag
Check tag
2018-01-16 11:52:32 -08:00
Yann Collet
f4e58455f6 ensure MOREFLAGS are not lost in root->tests Makefile invocation 2018-01-16 11:50:16 -08:00
Yann Collet
19cb37f837 travis ci : added gcc-7 test
also added `-Werror` to sanitizer tests
2018-01-16 11:40:42 -08:00
Nick Terrell
aae267a2e1 Reorganize block state 2018-01-16 11:17:50 -08:00
Nick Terrell
887cd4e35e Split ZSTD_CCtx into smaller sub-structures 2018-01-16 11:17:50 -08:00
Yann Collet
a187486e3c
Merge pull request #990 from krab/dev-cmake-libdir
CMake: use GNUInstallDirs for library install dir
2018-01-15 18:55:21 -08:00
Alexey Ivanov
403e2db139 CMake: use GNUInstallDirs for library install dir
Libraries now will be installed in the correct directory on x86_64 linux systems,
and can be changed with `-DCMAKE_INSTALL_LIBDIR=<dirname>` option.
2018-01-15 22:48:46 +03:00
Yann Collet
815eddeda4 added tag-triggered test
ensure tag and libzstd version are compatible
2018-01-14 17:06:21 -08:00
Yann Collet
f6e17b8c7a added tests/checkTag
compared provided tag with current libzstd version
2018-01-14 17:03:45 -08:00
Yann Collet
4792ac6689 make -C tests legacy : minor flag alteration
ZSTD_LEGACY_SUPPORT is a macro constant,
so it should be part of CPPFLAGS, instead of CFLAGS
2018-01-14 14:09:17 -08:00
Yann Collet
99a0a8bbd2 tests/Makefile : fixed target allnothread
ensures MT is disabled
2018-01-13 22:00:05 -08:00
Yann Collet
f8a05932b9 added make list capability to tests/Makefile 2018-01-13 21:54:21 -08:00
Yann Collet
9477f6529d
Merge pull request #984 from terrelln/dict-load
Load more dictionary positions into table if empty
2018-01-13 13:20:42 -08:00
Yann Collet
58ecf13e02 zstdmt : can compress at block granularity
offering perspective of more accurate progression report.
2018-01-13 13:18:57 -08:00
Nick Terrell
9a211d1f05 Load more dictionary positions into table if empty
If the hash table is empty load positions into the hash table
that we would otherwise skip.

| Level | Data Set     | Improvement |
|-------|--------------|-------------|
| 1     | github       | 0.44%       |
| 1     | hg-changelog | 0.13%       |
| 1     | hg-commands  | 1.28%       |
| 1     | hg-manifest  | 0.70%       |
| 3     | github       | 0.74%       |
| 3     | hg-changelog | 0.87%       |
| 3     | hg-commands  | 1.74%       |
| 3     | hg-manifest  | 0.23%       |
2018-01-12 16:17:22 -08:00
Yann Collet
863b2f8db4
Merge pull request #983 from terrelln/dict-wlog
Increase windowLog from CDict based on the srcSize when known
2018-01-12 07:47:43 -08:00
Yann Collet
9d65a5c316
Merge pull request #970 from facebook/dubtlazy
Delayed update Binary Tree for Lazy strategy
2018-01-12 07:28:59 -08:00
Nick Terrell
4b7c4e5f41 Add test for cdict window log adjustment 2018-01-11 16:45:16 -08:00
Nick Terrell
b610b777d3 Increase windowLog from CDict based on the srcSize when known 2018-01-11 16:23:21 -08:00
Yann Collet
cacf47cbee Merge branch 'dev' into dubtlazy
and fixed conflicts
2018-01-11 13:25:08 -08:00
Yann Collet
04c00f9388
Merge pull request #982 from facebook/fix304
Fix for #304 and #977 : error during dictionary creation
2018-01-11 13:20:59 -08:00
Yann Collet
b9a14900ff changed function name to ZSTD_DUBT_findBestMatch() 2018-01-11 12:38:31 -08:00
Yann Collet
1d623e60a1
Merge pull request #981 from facebook/fix976
fixed bug #976, reported by @indygreg
2018-01-11 11:40:07 -08:00
Yann Collet
752bae4a48 added warning message
when pathological dataset is detected
(note : cover_optimize needs -v to display the warning)
2018-01-11 11:29:28 -08:00
Yann Collet
e8093dde09 fixed #304
Pathological samples may result in literal section being incompressible.
This case is now detected,
and literal distribution is replaced by one that can be written into the dictionary.
2018-01-11 11:16:32 -08:00
Yann Collet
218e9fe0fc added a test case for dictBuilder failure
cyclic data set makes the entropy stage fails
now, onto a fix for #304 ...
2018-01-11 09:42:38 -08:00
Yann Collet
55cdbdb465
Merge pull request #980 from conor42/dev
Fix break condition in decompression noise test
2018-01-11 08:33:35 -08:00
Yann Collet
2103a62b3d fixed minor warning on prototype definition 2018-01-11 04:49:19 -08:00
Yann Collet
ff795580f2 fixed bug #976, reported by @indygreg
constants in zstd.h should not depend on MIN() macro which existence is not guaranteed.

Added a test to check the specific constants.
The test is a bit too specific.
But I have found no way to control a more generic "are all macro already defined" condition,
especially as this is a valid construction (the missing macro might be defined later, intentionnally).
2018-01-10 20:33:45 -08:00
conor42
0e88f6e97b
Fix break condition in decompression noise test
The bug prevents noise being added
2018-01-11 11:42:58 +10:00
Yann Collet
06995775b0
Merge pull request #979 from facebook/memdoc
Update API documentation related to memory management capabilities
2018-01-10 17:39:37 -08:00
Yann Collet
1edf33764e
Merge pull request #974 from terrelln/dstfile
[fileio] Improve safety of output file modifications
2018-01-10 19:02:48 +01:00
Yann Collet
292eeb672f api doc : grouped all ZSTD_create*_advanced() functions together
in a new "custom memory allocator" paragraph
which is itself part of "memory management" category.

This makes it simpler to see the relation between the type and its usages.
2018-01-10 09:07:47 -08:00
Yann Collet
3ea156368c API doc : grouped ZSTD_initStatic*() together
within "memory management" category.
2018-01-10 08:49:50 -08:00
Yann Collet
b17fb488b0 fixed msan test
a pointer calculation was wrong in a corner case
2018-01-06 20:50:36 +01:00
Yann Collet
76321d6bbc
Merge pull request #965 from facebook/tests_obj
%.o objects files in /tests
2018-01-06 13:15:53 +01:00
Yann Collet
658d6b8588 Merge branch 'dev' into dubtlazy 2018-01-06 12:40:58 +01:00
Yann Collet
a927fae2a1 fixed ZSTD_reduceIndex()
following suggestions from @terrelln.
Also added some comments to present logic behind ZSTD_preserveUnsortedMark().
2018-01-06 12:31:26 +01:00