Yann Collet
b93598d6a4
zstdmt : reduced maximum nb of threads
...
to avoid memory address space issues on 32-bits systems
(see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876416#17 )
2017-09-28 13:49:12 -07:00
Yann Collet
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
Nick Terrell
6c41adfb28
[libzstd] pthread function prefixed with ZSTD_
...
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
2017-09-27 11:48:48 -07:00
Yann Collet
4791561c4a
silence minor gcc warning -Wempty-body
...
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
Stella Lau
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
Yann Collet
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
Stella Lau
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
Stella Lau
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
Yann Collet
369c29dd1a
fixed impact of merge conflict for longRange
2017-08-31 18:25:56 -07:00
Yann Collet
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
Stella Lau
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
Stella Lau
623e3cd40b
Use ZSTD_dm_rawContent in zstdmt_compress
2017-08-29 18:04:32 -07:00
Stella Lau
4e835720bf
Delay creation of ZSTDMT_CCtx
2017-08-29 17:58:32 -07:00
Stella Lau
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
Stella Lau
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
Stella Lau
024098a47d
Fix parameter retrieval from cdict
2017-08-25 17:58:28 -07:00
Stella Lau
2adde898c8
Fix typo with ZSTDMT_parameter
2017-08-25 16:13:40 -07:00
Stella Lau
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
Stella Lau
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
Stella Lau
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
Stella Lau
de5193422d
Distinguish between jobParams and cctxParams in zstdmt
2017-08-25 11:36:17 -07:00
Stella Lau
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
Nick Terrell
db3f5372df
[zstdmt] Use POOL_create_advanced()
2017-08-24 18:12:28 -07:00
Stella Lau
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
Stella Lau
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
Stella Lau
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
Stella Lau
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
Stella Lau
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
Stella Lau
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
Stella Lau
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
Stella Lau
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
Stella Lau
502031ca10
Use cctxParam version of createCDict internally
2017-08-21 11:00:44 -07:00
Stella Lau
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
Yann Collet
7db552676e
reduced pool queue to 0 to save memory
...
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
Stella Lau
d775519296
Add cctxParam versions of internal functions
2017-08-18 17:37:58 -07:00
Yann Collet
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
Stella Lau
63b8c98531
Pass cctx parameters to MTCtx
2017-08-18 16:17:24 -07:00
Yann Collet
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
Yann Collet
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
Yann Collet
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
Yann Collet
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
Yann Collet
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
Yann Collet
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
Yann Collet
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
Yann Collet
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
Yann Collet
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
