Commit Graph

2542 Commits

Author SHA1 Message Date
Nick Terrell
6570167f8b Fix VS fuzzer build configs 2017-01-27 13:56:41 -08:00
Nick Terrell
8959910412 Add threading and pool to build configs 2017-01-27 12:29:27 -08:00
Nick Terrell
2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Yann Collet
cafdd31a38 fixed MSAN warnings in legacy decoders
In some extraordinary circumstances,
*Length field can be generated from reading a partially uninitialized memory segment.
Data is correctly identified as corrupted later on,
but the read taints some later pointer arithmetic operation.
2017-01-27 10:44:03 -08:00
Yann Collet
122b6aa657 updated NEWS 2017-01-27 03:22:37 -08:00
Yann Collet
717c65d690 Merge pull request #519 from inikep/dev11
Dev11
2017-01-26 14:23:44 -08:00
Yann Collet
c8a27d9303 Merge pull request #518 from facebook/zstdmt
zstd with multi-threading
2017-01-26 14:22:27 -08:00
Yann Collet
ef33d00532 fixed : ZSTD_setCCtxParameter() properly exposed in DLL 2017-01-26 12:24:21 -08:00
Yann Collet
4a62f79ec9 fixed clang documentation warning 2017-01-26 09:16:56 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
06e7697f96 added test of new parameter ZSTD_p_forceWindow 2017-01-25 16:39:03 -08:00
Yann Collet
bb0027405a fixed zstdmt corruption issue when enabling overlapped sections
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37 fixed zstdmt cli freeze issue with large nb of threads
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Przemyslaw Skibinski
e98788e0dc Merge remote-tracking branch 'refs/remotes/origin/master' into dev11 2017-01-25 12:24:24 +01:00
Przemyslaw Skibinski
c4874aab4c .travis.yml: different tests for "master" branch 2017-01-25 11:57:28 +01:00
Przemyslaw Skibinski
3bb010a667 .travis.yml: optimized order of short tests 2017-01-25 11:19:35 +01:00
Yann Collet
dc8dae596a overlapped section, for improved compression
Sections 2+ read a bit of data from previous section
in order to improve compression ratio.
This also costs some CPU, to reference read data.

Read data is currently fixed to window>>3 size
2017-01-24 22:32:12 -08:00
Yann Collet
f14a669054 refactor job creation
code shared accross ZSTDMT_{compress,flush,end}Stream(),
for easier maintenance
2017-01-24 17:41:49 -08:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
Yann Collet
3488a4a473 ZSTDMT now supports frame checksum 2017-01-24 11:48:40 -08:00
Przemyslaw Skibinski
74d2cfdee2 .travis.yml: test jobs 12-15 2017-01-24 17:42:28 +01:00
Przemyslaw Skibinski
6ad02e7762 JOB_NUMBER -eq 9 2017-01-24 15:01:46 +01:00
Przemyslaw Skibinski
96f152f708 improved ZSTD_compressBlock_opt_extDict_generic 2017-01-24 13:18:50 +01:00
Yann Collet
94364bf87a refactor ZSTDMT streaming flush code
now shared by both ZSTDMT_compressStream() and ZSTDMT_flushStream()
2017-01-23 11:50:44 -08:00
Yann Collet
1cbf251e43 ZSTDMT streaming : fall back to (regular) single thread mode
when nbThreads==1
2017-01-23 01:43:58 -08:00
Yann Collet
84581ff8d7 ZSTDMT_compressCCtx : fallback to single-thread mode when nbChunks==1 2017-01-23 01:20:27 -08:00
Yann Collet
1a2547f654 ZSTDMT_compressStream() becomes blocking when required to ensure forward progresses
In some (rare) cases, job list could be blocked by a first job still being processed,
while all following ones are completed, waiting to be flushed.
In such case, the current job-table implementation is unable to accept new job.
As a consequence, a call to ZSTDMT_compressStream() can be useless (nothing read, nothing flushed),
with the risk to trigger a busy-wait on the caller side
(needlessly loop over ZSTDMT_compressStream() ).

In such a case, ZSTDMT_compressStream() will block until the first job is completed and ready to flush.
It ensures some forward progress by guaranteeing it will flush at least a part of the completed job.
Energy-wasting busy-wait is avoided.
2017-01-22 23:49:52 -08:00
Yann Collet
c593348722 ZSTDMT_initCStream_usingDict() can outlive dict
Like ZSTD_initCStream_usingDict(),
ZSTDMT_initCStream_usingDict() now keep a copy of dict internally.
This way, dict can be released :
it does not longer have to outlive all future compression sessions.
2017-01-22 16:44:15 -08:00
Yann Collet
bd6bc22612 playtest.sh : changed sdiff into $DIFF 2017-01-22 15:54:14 -08:00
Yann Collet
9d6f7637ec protected (mutex) read to jobCompleted, as suggested by @terrelln 2017-01-21 22:14:08 -08:00
Yann Collet
0cf74fa957 optimized pool allocation by 1 slot 2017-01-21 22:06:49 -08:00
Yann Collet
6ed29a8f44 minor : tab to spaces 2017-01-21 21:56:36 -08:00
Yann Collet
f8804d1014 convert tabs to space
joys of using multiple editors from multiple environments ...
2017-01-20 17:23:19 -08:00
Yann Collet
317604e0ad fixed : compilation of zstreamtest in dll mode 2017-01-20 17:18:41 -08:00
Yann Collet
d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
Yann Collet
900f39e709 skip zstdmt at root directory 2017-01-20 16:36:29 -08:00
cyan4973
f0ffa237da fixed VS2008 project 2017-01-20 15:24:06 -08:00
cyan4973
326575c3a3 fixed VS2010 project 2017-01-20 14:49:44 -08:00
cyan4973
2e3b659ae1 fixed minor warnings (Visual, conversion, doxygen) 2017-01-20 14:43:09 -08:00
cyan4973
5fba09fa41 updated util's time for Windows compatibility
Correctly measures time on Posix systems when running with
Multi-threading

Todo : check Windows measurement under multi-threading
2017-01-20 12:57:31 -08:00
Yann Collet
458c8a94b4 minor refactoring : cleaner MT integration within bench 2017-01-19 17:44:15 -08:00
Yann Collet
b459aad5b4 renamed savedRep into repToConfirm 2017-01-19 17:33:37 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
2017-01-19 17:04:28 -08:00
Yann Collet
19d670ba9d Added ZSTDMT_initCStream_advanced() variant
Correctly compress with custom params and dictionary
Added relevant fuzzer test in zstreamtest

Also :
new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job
(note : a flush() command can still generate a partial job anytime)
2017-01-19 15:32:07 -08:00
Yann Collet
0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00
Yann Collet
f22adae984 fixed minor warning (unused variable) in fuzzer 2017-01-19 13:46:30 -08:00
Yann Collet
736788f8e8 added streaming fuzzer tests for MT API
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
2017-01-19 12:15:29 -08:00
Yann Collet
32dfae6f98 fixed Multi-threaded compression
MT compression generates a single frame.
Multi-threading operates by breaking the frames into independent sections.
But from a decoder perspective, there is no difference :
it's just a suite of blocks.

Problem is, decoder preserves repCodes from previous block to start decoding next block.
This is also valid between sections, since they are no different than changing block.

Previous version would incorrectly initialize repcodes to their default value at the beginning of each section.
When using them, there was a mismatch between encoder (default values) and decoder (values from previous block).

This change ensures that repcodes won't be used at the beginning of a new section.
It works by setting them to 0.
This only works with regular (single segment) variants : extDict variants will fail !
Fortunately, sections beyond the 1st one belong to this category.

To be checked : btopt strategy.
This change was only validated from fast to btlazy2 strategies.
2017-01-19 10:32:55 -08:00