Yann Collet
cc3d1bc262
Merge pull request #525 from terrelln/covermt
...
Multithreaded COVER dictionary training
2017-01-30 10:15:33 -08:00
Nick Terrell
2fe9126591
Add multithread support to COVER
2017-01-27 11:56:02 -08:00
Nick Terrell
9c018cc140
Add BUCK files for Nuclide support
2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski
eb2d23a90c
improved #232 fix
2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski
92a4dbf2e4
Fixed https://github.com/facebook/zstd/issues/232
2017-01-27 10:43:03 -08:00
Yann Collet
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
Yann Collet
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
Yann Collet
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
Yann Collet
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski
5022a18d51
improved #232 fix
2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski
64fa2dbc5e
Fixed https://github.com/facebook/zstd/issues/232
2017-01-25 13:02:33 +01:00
Yann Collet
512cbe8c10
zstdmt cli and API allow selection of section sizes
...
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.
It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()
The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
Yann Collet
f8804d1014
convert tabs to space
...
joys of using multiple editors from multiple environments ...
2017-01-20 17:23:19 -08:00
Yann Collet
d7e3cb58c5
Resolved merge conflict dev+zstdmt
2017-01-20 16:44:50 -08:00
cyan4973
5fba09fa41
updated util's time for Windows compatibility
...
Correctly measures time on Posix systems when running with
Multi-threading
Todo : check Windows measurement under multi-threading
2017-01-20 12:57:31 -08:00
Yann Collet
458c8a94b4
minor refactoring : cleaner MT integration within bench
2017-01-19 17:44:15 -08:00
Yann Collet
500014af49
zstd cli can now compress using multi-threading
...
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads() (fileio)
2017-01-19 17:04:28 -08:00
Yann Collet
0f984d94c4
changed MT enabling macro to ZSTD_MULTITHREAD
2017-01-19 14:05:07 -08:00
Yann Collet
f22adae984
fixed minor warning (unused variable) in fuzzer
2017-01-19 13:46:30 -08:00
Yann Collet
736788f8e8
added streaming fuzzer tests for MT API
...
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
2017-01-19 12:15:29 -08:00
Przemyslaw Skibinski
8a0bc30a2d
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski
d72f4b6b7a
added "Makefile is validated"
2017-01-17 12:40:06 +01:00
Yann Collet
5eb749e734
ZSTDMT_compress() creates a single frame
...
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()
As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().
This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
Nick Terrell
c220d4c74d
Use COVER_MEMMULT when training with COVER.
2017-01-09 16:49:04 -08:00
Yann Collet
56958500fc
minor man page update
2017-01-09 19:47:09 +01:00
Nick Terrell
3a1fefcf00
Simplify COVER parameters
2017-01-02 17:51:38 -08:00
Nick Terrell
df8415c502
Add COVER to the zstd cli
2017-01-02 14:43:08 -08:00
Yann Collet
6334b04d61
compile object files, for faster recompilation
2017-01-02 03:22:18 +01:00
Yann Collet
747452677d
fixed cmake tests
2017-01-02 02:05:45 +01:00
Yann Collet
cdb2763f4a
new Makefile target zstdmt
2017-01-02 01:43:56 +01:00
Yann Collet
f1cb55192c
fixed linux warnings
2017-01-02 01:11:55 +01:00
Yann Collet
0ec6a95ba1
minor fixes
2017-01-02 00:49:42 +01:00
Yann Collet
c6a6417458
bench correctly measures time for multi-threaded compression (posix only)
2016-12-31 03:31:26 +01:00
Yann Collet
f765a375a5
Merge pull request #504 from terrelln/thread-pool
...
[zstdmt] Add thread pool
2016-12-30 15:31:49 +01:00
Yann Collet
e70912c72b
Changed : input divided into roughly equal parts.
...
Debug : can measure time waiting for mutexes to unlock.
2016-12-29 01:24:01 +01:00
Yann Collet
8d7432914f
Merge pull request #503 from inikep/dev11
...
Dev11
2016-12-28 16:50:39 +01:00
Yann Collet
ab7a579180
added -T command , to set nb of threads
2016-12-28 16:11:09 +01:00
Przemyslaw Skibinski
75f3a3a335
changed default PREFIX and MANDIR
2016-12-28 12:32:41 +01:00
Yann Collet
3d93f2fce7
first zstdmt sketch
2016-12-27 07:19:36 +01:00
Yann Collet
c07d2e3a31
Merge pull request #499 from inikep/dev11
...
improved *BSD and Solaris compatibility
2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski
63b0014b96
BSD: improved "make install"
2016-12-23 10:05:49 +01:00
Andrew Janke
fce374a100
zstdless: add shebang and quote $@
2016-12-22 17:40:10 -05:00
Przemyslaw Skibinski
b999170311
Solaris: working "make -C lib install"
2016-12-22 20:14:37 +01:00
Przemyslaw Skibinski
aab442133d
Solaris: working "make -C programs install"
2016-12-22 19:26:01 +01:00
Yann Collet
8333106b8a
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-12-21 16:44:24 +01:00
Yann Collet
0d7e84899f
Merge pull request #489 from inikep/v112
...
improved detection of POSIX
2016-12-21 16:42:46 +01:00
Yann Collet
9518744025
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-12-21 16:20:41 +01:00
Yann Collet
1f57c2ed32
added : ZSTD_createCDict_byReference()
2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski
7a8a03c20d
util.h: restore BSD license for Facebook Open-Source
2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski
97a258d71d
updated comments
2016-12-21 14:00:41 +01:00
