AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,085 Commits 1 Branch 0 Tags 27 MiB
6b2b6a9bd5
Commit Graph

110 Commits

Author SHA1 Message Date
Yann Collet
7db552676e reduced pool queue to 0 to save memory 
fixed : pool performance when jobs are fires fast and queueSize==0
 2017-08-19 15:07:54 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Yann Collet
2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet
132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog 
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
 2017-07-13 02:22:58 -07:00
Yann Collet
8ef666c325 slightly increased buffer pool, to cover normal "full load" scenarios 
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
 2017-07-12 14:23:34 -07:00
Yann Collet
052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Yann Collet
2a62f48bf4 release input buffers from inside worker thread 
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
 2017-07-11 15:56:40 -07:00
Yann Collet
57236184af buffer pool : all buffers have same size 
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
 2017-07-11 15:17:25 -07:00
Yann Collet
34b2b95631 zstdmt : intermediate outBuffer allocated from within worker 
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
 2017-07-11 14:59:10 -07:00
Yann Collet
16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet
ef0ff7fe7f zstdmt: removed margin for improved memory usage 2017-07-11 08:54:29 -07:00
Yann Collet
4616fad18b improved ZSTDMT_compress() memory usage 
does not need the input buffer for streaming operations

also : reduced a few tests time length
 2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547 optimized memory usage for ZSTDMT_compress() 
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.

In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).

To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
 2017-07-10 16:30:55 -07:00
Yann Collet
3510efb02d fix : custom allocator correctly propagated to child contexts 2017-07-10 14:21:40 -07:00
Yann Collet
e32fb0c1fe added ZSTD_sizeof_CCtx() test 2017-07-10 12:29:57 -07:00
Yann Collet
27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet
6383372dec fixed : 0-copy in NULL is UB 2017-07-04 10:36:41 -07:00
Yann Collet
2de2396a36 refactor ZSTDMT_compress() 2017-07-03 16:23:36 -07:00
Yann Collet
2084b041f4 fixed comments 2017-07-03 15:52:19 -07:00
cyan4973
1bafe393e4 fix : ZSTDMT_compressStream_generic() can accept NULL input 
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
 2017-07-01 06:59:24 -07:00
Yann Collet
58bd0e70fc fixed : dictionary compression with new advanced API in Multi-threading mode 2017-06-30 16:01:02 -07:00
Yann Collet
d8b33a598d Optimized ZSTDMT single-pass mode speed on large sources 
by ensuring job sizes remain "not too large"
 2017-06-30 15:44:57 -07:00
Yann Collet
d5c046c609 implemented shortcut for zstd_compress_generic() in MT mode 
added ZSTDMT_compress_advanced() API
 2017-06-30 14:51:01 -07:00
Yann Collet
33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet
204b6b7ef6 fixed streaming buffered allocation with CDict compression 2017-06-21 15:13:00 -07:00
Yann Collet
83095970e6 free cdictLocal faster, suggested by @terrelln 2017-06-21 12:26:40 -07:00
Yann Collet
7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Yann Collet
c08e649e95 first implementation of bench.c with new API ZSTD_compress_generic() 
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.

Also : fixed a long standing bug in ZSTDMT streaming API.
 2017-06-19 18:25:35 -07:00
Yann Collet
381e66cfbd added ZSTD_clampCParams() 
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
 2017-06-16 17:34:54 -07:00
Yann Collet
559ee82e90 fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps 2017-06-16 11:58:21 -07:00
Yann Collet
23aace9778 added control stage to MT mode 2017-06-11 18:32:36 -07:00
Yann Collet
f35e2de61c linked newAPI to ZSTDMT 2017-06-05 18:32:48 -07:00
Yann Collet
8c910d2097 updated ZSTDMT streaming API 
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
 2017-06-03 01:15:02 -07:00
Yann Collet
58e8d793e1 made debug definitions common within zstd_internal.h 2017-06-02 18:20:48 -07:00
Yann Collet
c35e535002 added support for multithreading parameters 2017-06-01 18:44:06 -07:00
Yann Collet
c4a5a21c5c created ZSTDMT_sizeof_CCtx() and POOL_sizeof() 
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
 2017-06-01 17:56:14 -07:00
Yann Collet
deee6e523f expose ZSTD_compress_generic_simpleArgs() 
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
 2017-05-30 17:42:00 -07:00
Yann Collet
5bcef1ada2 removed mtctx->cstream 
use the first cctx in pool when ZSTDMT is used in single-thread mode
now that cctx and cstream are the same object.
 2017-05-30 16:37:19 -07:00
Yann Collet
44e45e8423 added ZSTDMT_createCCtx_advanced() 
make it possible to use custom allocators
 2017-05-30 16:12:06 -07:00
Yann Collet
b6dec4c3ae fixed minor cast warning 2017-05-27 17:09:06 -07:00
Yann Collet
e071159101 mtctx->jobs allocate its own memory space 
to make ZSTDMT_CCtx_s size predictable
so that it can be included in CCtx
 2017-05-27 00:21:33 -07:00
Yann Collet
31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Yann Collet
2c5514c759 fixed ZSTDMT_initCStream_advanced() 
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
 2017-04-18 22:52:41 -07:00
Yann Collet
20d5e03893 content size is controlled at bufferless level 
so it's active for all entry points

Also : added relevant test (wrong content size) in fuzzer
 2017-04-11 18:34:02 -07:00
Yann Collet
30c7698970 optimize ZSTDMT_compress() memory usage 
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
 2017-03-31 18:27:03 -07:00
Yann Collet
eea7858e2b fixed minor warnings in debug code 2017-03-30 16:47:19 -07:00
Yann Collet
34cc487d05 overlap at full windowSize for max compression level 
as it provides max compression ratio
 2017-03-30 16:23:22 -07:00
Yann Collet
458e955c23 improved ZSTDMT_compress() 
Use a bit more threads by default.
Uses overlap segments to boost compression ratio (like the streaming variant)
 2017-03-30 15:51:58 -07:00