Commit Graph

30 Commits

Author SHA1 Message Date
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
63b8c98531 Pass cctx parameters to MTCtx 2017-08-18 16:17:24 -07:00
Yann Collet
132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
Yann Collet
052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
Yann Collet
49af41820d clarified status of zstdmt_compress.h API 2017-07-05 17:21:13 -07:00
Yann Collet
d5c046c609 implemented shortcut for zstd_compress_generic() in MT mode
added ZSTDMT_compress_advanced() API
2017-06-30 14:51:01 -07:00
Yann Collet
b44ab82f7a ensure new ZSTD_strategy starts at value 1 2017-06-20 14:11:49 -07:00
Yann Collet
f35e2de61c linked newAPI to ZSTDMT 2017-06-05 18:32:48 -07:00
Yann Collet
8c910d2097 updated ZSTDMT streaming API
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
Yann Collet
58e8d793e1 made debug definitions common within zstd_internal.h 2017-06-02 18:20:48 -07:00
Yann Collet
b877e834b1 minor indent 2017-06-02 13:47:11 -07:00
Yann Collet
c35e535002 added support for multithreading parameters 2017-06-01 18:44:06 -07:00
Yann Collet
c4a5a21c5c created ZSTDMT_sizeof_CCtx() and POOL_sizeof()
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
2017-06-01 17:56:14 -07:00
Yann Collet
ae728a43b8 removed defaultCustomMem
now ZSTD_customCMem is promoted as new default.

Advantages : ZSTD_customCMem = { NULL, NULL, NULL},
so it's natural default after a memset.

ZSTD_customCMem is public constant
(defaultCustomMem was private only).

Also : makes it possible to introduce ZSTD_calloc(),
which can now default to stdlib's calloc()
when it detects system default.

Fixed zlibwrapper which depended on defaultCustomMem.
2017-05-30 17:11:39 -07:00
Yann Collet
44e45e8423 added ZSTDMT_createCCtx_advanced()
make it possible to use custom allocators
2017-05-30 16:12:06 -07:00
Yann Collet
48bed91606 Merge pull request #527 from facebook/zstdmt
zstdmt refinements
2017-01-31 16:36:46 -08:00
Yann Collet
88df1aed61 changed advanced parameter overlapLog
Follows a positive logic (increasing value => increasing overlap)
which is easier to use
2017-01-30 11:00:00 -08:00
Nick Terrell
b42dd27ef5 Add include guards and extern C 2017-01-27 16:00:19 -08:00
Yann Collet
8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet
06e7697f96 added test of new parameter ZSTD_p_forceWindow 2017-01-25 16:39:03 -08:00
Yann Collet
dc8dae596a overlapped section, for improved compression
Sections 2+ read a bit of data from previous section
in order to improve compression ratio.
This also costs some CPU, to reference read data.

Read data is currently fixed to window>>3 size
2017-01-24 22:32:12 -08:00
Yann Collet
512cbe8c10 zstdmt cli and API allow selection of section sizes
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
Yann Collet
3488a4a473 ZSTDMT now supports frame checksum 2017-01-24 11:48:40 -08:00
Yann Collet
c593348722 ZSTDMT_initCStream_usingDict() can outlive dict
Like ZSTD_initCStream_usingDict(),
ZSTDMT_initCStream_usingDict() now keep a copy of dict internally.
This way, dict can be released :
it does not longer have to outlive all future compression sessions.
2017-01-22 16:44:15 -08:00
Yann Collet
317604e0ad fixed : compilation of zstreamtest in dll mode 2017-01-20 17:18:41 -08:00
cyan4973
2e3b659ae1 fixed minor warnings (Visual, conversion, doxygen) 2017-01-20 14:43:09 -08:00
Yann Collet
500014af49 zstd cli can now compress using multi-threading
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
2017-01-19 17:04:28 -08:00
Yann Collet
19d670ba9d Added ZSTDMT_initCStream_advanced() variant
Correctly compress with custom params and dictionary
Added relevant fuzzer test in zstreamtest

Also :
new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job
(note : a flush() command can still generate a partial job anytime)
2017-01-19 15:32:07 -08:00
Yann Collet
107bcbbbc2 zstdmt : changed internal naming from frame to chunk
Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.

minor : ensures that content size is written in header
2017-01-12 01:25:46 +01:00
Yann Collet
3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00