Yann Collet
369c29dd1a
fixed impact of merge conflict for longRange
2017-08-31 18:25:56 -07:00
Yann Collet
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
Stella Lau
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
Stella Lau
623e3cd40b
Use ZSTD_dm_rawContent in zstdmt_compress
2017-08-29 18:04:32 -07:00
Stella Lau
4e835720bf
Delay creation of ZSTDMT_CCtx
2017-08-29 17:58:32 -07:00
Stella Lau
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
Stella Lau
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
Stella Lau
024098a47d
Fix parameter retrieval from cdict
2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8
Fix typo with ZSTDMT_parameter
2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
Stella Lau
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d
Distinguish between jobParams and cctxParams in zstdmt
2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df
[zstdmt] Use POOL_create_advanced()
2017-08-24 18:12:28 -07:00
Stella Lau
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
Stella Lau
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
Stella Lau
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
Stella Lau
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
Stella Lau
502031ca10
Use cctxParam version of createCDict internally
2017-08-21 11:00:44 -07:00
Stella Lau
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
Yann Collet
7db552676e
reduced pool queue to 0 to save memory
...
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
Stella Lau
d775519296
Add cctxParam versions of internal functions
2017-08-18 17:37:58 -07:00
Yann Collet
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
Stella Lau
63b8c98531
Pass cctx parameters to MTCtx
2017-08-18 16:17:24 -07:00
Yann Collet
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
Yann Collet
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
Yann Collet
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
Yann Collet
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
Yann Collet
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
Yann Collet
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
Yann Collet
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
Yann Collet
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
Yann Collet
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
Yann Collet
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
Yann Collet
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
Yann Collet
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Yann Collet
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
Yann Collet
6383372dec
fixed : 0-copy in NULL is UB
2017-07-04 10:36:41 -07:00
Yann Collet
2de2396a36
refactor ZSTDMT_compress()
2017-07-03 16:23:36 -07:00
Yann Collet
2084b041f4
fixed comments
2017-07-03 15:52:19 -07:00
