Yann Collet
b05c4828ea
zstdmt : correctly check for cctx and buffer allocation
...
Result from getBuffer and getCCtx could be NULL when allocation fails.
Now correctly checks : job creation stop and last job reports an allocation error.
releaseBuffer and releaseCCtx are now also compatible with NULL input.
Identified a new potential issue :
when early job fails, later jobs are not collected for resource retrieval.
2017-01-12 02:01:28 +01:00
Yann Collet
107bcbbbc2
zstdmt : changed internal naming from frame to chunk
...
Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.
minor : ensures that content size is written in header
2017-01-12 01:25:46 +01:00
Yann Collet
5eb749e734
ZSTDMT_compress() creates a single frame
...
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()
As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().
This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
Yann Collet
04cbc36499
minor refactor (release CCtx 1st) and comment clarification
2017-01-11 16:08:08 +01:00
Yann Collet
085179bb78
fixed ZSTDMT_createCCtx() : checked inner objects are properly created
2017-01-11 15:58:05 +01:00
Yann Collet
8ce1cc2bec
improved ZSTD_createCCtxPool() cancellation
...
use ZSTD_freeCCtxPool() to release the partially created pool.
avoids to duplicate logic.
Also : identified a new difficult corner case :
when freeing the Pool, all CCtx should be previously released back to the pool.
Otherwise, it means some CCtx are still in use.
There is currently no clear policy on what to do in such a case.
Note : it's supposed to never happen.
Since pool creation/usage is static, it has no external user,
which limits risks.
2017-01-11 15:44:26 +01:00
Yann Collet
47557ba2b2
fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails
2017-01-11 15:35:56 +01:00
Yann Collet
b8cdc16969
Merge pull request #509 from terrelln/dict-builder-32
...
Handle cover dictionary builder maximum input size for 32-bit mode
2017-01-10 06:30:28 +01:00
Nick Terrell
8d984699db
Document memory requirements for COVER algorithm
2017-01-09 18:20:10 -08:00
Nick Terrell
555e281637
Handle large input size in 32-bit mode correctly
2017-01-09 18:20:06 -08:00
Nick Terrell
c220d4c74d
Use COVER_MEMMULT when training with COVER.
2017-01-09 16:49:04 -08:00
Yann Collet
56958500fc
minor man page update
2017-01-09 19:47:09 +01:00
Yann Collet
4736dffd41
Merge pull request #507 from terrelln/dict-builder-optimize
...
Add new dictionary builder
2017-01-03 20:33:33 +01:00
Nick Terrell
a8b4fe0481
Add COVER dictionary builder to fuzzer unit tests
2017-01-02 18:45:19 -08:00
Nick Terrell
3a1fefcf00
Simplify COVER parameters
2017-01-02 17:51:38 -08:00
Yann Collet
8566799781
separated ppc and ppc64 tests, for more regular timing
2017-01-03 00:25:01 +01:00
Nick Terrell
cbb3ce376b
Add cover cli to playtests
2017-01-02 14:43:11 -08:00
Nick Terrell
df8415c502
Add COVER to the zstd cli
2017-01-02 14:43:08 -08:00
Nick Terrell
9103ed6c8b
Add cover.c to build files
2017-01-02 13:22:53 -08:00
Nick Terrell
96b39f65fa
Add COVER dictionary builder
2017-01-02 13:22:51 -08:00
Yann Collet
6334b04d61
compile object files, for faster recompilation
2017-01-02 03:22:18 +01:00
Yann Collet
747452677d
fixed cmake tests
2017-01-02 02:05:45 +01:00
Yann Collet
cdb2763f4a
new Makefile target zstdmt
2017-01-02 01:43:56 +01:00
Yann Collet
f1cb55192c
fixed linux warnings
2017-01-02 01:11:55 +01:00
Yann Collet
0ec6a95ba1
minor fixes
2017-01-02 00:49:42 +01:00
Yann Collet
2ec635a162
use pthread_cond to send signals between threads
2017-01-01 17:31:33 +01:00
Yann Collet
e4f70cdbbc
Merge pull request #505 from terrelln/pthread_cond_t
...
pthread_cond_t
2017-01-01 15:53:51 +01:00
Nick Terrell
5ca0fd2045
Shorten thread pool tests
2016-12-31 22:39:32 -05:00
Nick Terrell
bb13387d7d
Fix pool for threading.h
2016-12-31 19:10:47 -05:00
Nick Terrell
4204e03e77
Add threading.h condition variables
2016-12-31 19:10:29 -05:00
Nick Terrell
d132433534
Switch thread pool test to threading.h
2016-12-31 19:10:13 -05:00
Yann Collet
3b9d434356
extended ZSTDMT code support for non-MT systems and WIN32 (preliminary)
2016-12-31 16:32:19 +01:00
Yann Collet
c8efc1c874
simplified Buffer Pool
2016-12-31 14:45:33 +01:00
Yann Collet
3b29dbd9e8
new zstdmt version using generic treadpool
2016-12-31 06:04:25 +01:00
Yann Collet
c6a6417458
bench correctly measures time for multi-threaded compression (posix only)
2016-12-31 03:31:26 +01:00
Yann Collet
f765a375a5
Merge pull request #504 from terrelln/thread-pool
...
[zstdmt] Add thread pool
2016-12-30 15:31:49 +01:00
Nick Terrell
9c499648e3
Add thread pool tests
2016-12-29 23:41:03 -08:00
Nick Terrell
e777a5be6b
Add a thread pool for ZSTDMT and COVER
2016-12-29 23:39:44 -08:00
Przemyslaw Skibinski
59292ef6e1
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-12-29 15:24:18 +01:00
Yann Collet
e70912c72b
Changed : input divided into roughly equal parts.
...
Debug : can measure time waiting for mutexes to unlock.
2016-12-29 01:24:01 +01:00
Yann Collet
6c0ed9483a
compression threads use ZSTD_compressCCtx()
2016-12-28 17:08:28 +01:00
Yann Collet
8d7432914f
Merge pull request #503 from inikep/dev11
...
Dev11
2016-12-28 16:50:39 +01:00
Yann Collet
ab7a579180
added -T command , to set nb of threads
2016-12-28 16:11:09 +01:00
Yann Collet
ce9e1452fd
protect buffer pool with a mutex
2016-12-28 15:31:19 +01:00
Yann Collet
9b587e2135
Merge pull request #502 from Chocobo1/cpp11
...
Move -std=c++11 cxxflag to PZSTD_CXXFLAGS
2016-12-28 14:38:48 +01:00
Przemyslaw Skibinski
75f3a3a335
changed default PREFIX and MANDIR
2016-12-28 12:32:41 +01:00
Yann Collet
3d93f2fce7
first zstdmt sketch
2016-12-27 07:19:36 +01:00
Chocobo1
37a2fb4ce1
Move -std=c++11 cxxflag to PZSTD_CXXFLAGS
...
Fixes the problem that the compiler doesn't enable c++11 mode by default
and the package build system has its own CXXFLAGS
2016-12-26 23:08:23 +08:00
Yann Collet
39c105c605
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-12-23 22:25:31 +01:00
Yann Collet
aca113f4f5
fixed ZSTD_sizeof_?Dict()
2016-12-23 22:25:03 +01:00
