AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,315 Commits 1 Branch 0 Tags 27 MiB
d6770f80af
Commit Graph

4315 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
6733c0777c updated NEWS regarding #760 2017-07-13 15:34:44 -07:00
Stella Lau
361c06df75 Add min/max offset to stats 2017-07-13 15:29:41 -07:00
Paul Cruz
65a4ce2635 added tests for forced compression level 2017-07-13 14:57:24 -07:00
Paul Cruz
0d9665cef5 added additional tests for performance, allowed force compression level for testing purposes 2017-07-13 14:46:54 -07:00
Stella Lau
2b3c7e4199 [ldm] Make some functions shared 2017-07-13 14:39:35 -07:00
Paul Cruz
9165e97fc6 added some tests for correctness, time, and compression ratio 2017-07-13 13:50:23 -07:00
Stella Lau
9306feb8fa [ldm] Switch to using lib/common/mem.h and move typedefs to ldm.h 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

Blame Revision:
 2017-07-13 13:44:48 -07:00
Nick Terrell
830ef4152a [libzstd] Increase granularity of FSECTable repeat mode 2017-07-13 12:45:39 -07:00
Stella Lau
50421d9474 [ldm] Remove old main files 2017-07-13 11:45:00 -07:00
Stella Lau
68c4560701 [ldm] Add TODO and comment for segfaulting in compress function 2017-07-13 10:38:19 -07:00
Yann Collet
d985319337 Merge pull request #759 from terrelln/real-block-split 
[libzstd] Pull CTables into sub-structure
 2017-07-13 10:24:19 -07:00
Yann Collet
3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Paul Cruz
766663f1f1 added altering dictionary size depending on compression level 2017-07-13 10:15:27 -07:00
Yann Collet
132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog 
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
 2017-07-13 02:22:58 -07:00
Yann Collet
4e77f7761d clarified comment on ZSTD_p_contentSizeFlag 2017-07-13 02:09:07 -07:00
Nick Terrell
de0414b736 [libzstd] Pull CTables into sub-structure 2017-07-12 19:49:19 -07:00
Stella Lau
92bed4a7e0 [ldm] Add CHAR_OFFSET in hash function and extend header size 2017-07-12 18:47:26 -07:00
Yann Collet
d6799635a5 Merge pull request #756 from facebook/memOpt 
Memory optimisations for ZSTDMT
 2017-07-12 18:32:37 -07:00
Paul Cruz
7c886db0a8 changed to stderr 2017-07-12 17:28:53 -07:00
Paul Cruz
b5b18cf664 changed to malloc, added comment about adaptive compression level, and changed ternary operators 2017-07-12 17:10:58 -07:00
Paul Cruz
954d999abf fixed up freeCCtx() removed BYTE since it wasn't being used 2017-07-12 16:50:43 -07:00
Paul Cruz
3c16edd26a added copyright header, removed clean from makefile 2017-07-12 16:40:24 -07:00
Stella Lau
8de82b6eb0 [ldm] Clean up versions 2017-07-12 16:31:31 -07:00
Paul Cruz
74d3a6f5ae passes tests with adaptive compression level 2017-07-12 16:18:41 -07:00
Paul Cruz
5353d350ae working with fixed compression level and fixed dictionary size 2017-07-12 16:02:20 -07:00
Stella Lau
8ff8cdb15b [ldm] Clean up code 2017-07-12 15:12:07 -07:00
Yann Collet
33cecaaaca Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-12 14:38:36 -07:00
Yann Collet
8ef666c325 slightly increased buffer pool, to cover normal "full load" scenarios 
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
 2017-07-12 14:23:34 -07:00
Paul Cruz
356ddb649f working with flush job->src.size and fixed cLevel 2017-07-12 12:21:21 -07:00
Stella Lau
3a48ffd4fd Fix sumToHash to use hash space more efficiently 2017-07-12 10:53:19 -07:00
Stella Lau
e0d4162464 Minor fix for non-rolling hash 2017-07-12 09:50:24 -07:00
Stella Lau
50502519fb Switch to using rolling hash only 2017-07-12 09:47:00 -07:00
Stella Lau
583dda17a8 Update rolling hash 2017-07-11 18:13:26 -07:00
Yann Collet
052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Paul Cruz
0a401852c4 added debug statement 2017-07-11 16:50:50 -07:00
Yann Collet
2a62f48bf4 release input buffers from inside worker thread 
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
 2017-07-11 15:56:40 -07:00
Paul Cruz
72a183efad changed dictionary size, added debugging statements 2017-07-11 15:49:52 -07:00
Yann Collet
57236184af buffer pool : all buffers have same size 
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
 2017-07-11 15:17:25 -07:00
Paul Cruz
7c54e09347 updated DEBUG statements 2017-07-11 15:15:41 -07:00
Paul Cruz
a3c077b8c6 added error message, updated copying dictionary into the input buffer 2017-07-11 15:00:52 -07:00
Yann Collet
34b2b95631 zstdmt : intermediate outBuffer allocated from within worker 
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
 2017-07-11 14:59:10 -07:00
Yann Collet
16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet
fe80d45807 Merge pull request #755 from jacquesg/solaris 
Solaris fix
 2017-07-11 12:11:44 -07:00
Paul Cruz
34afb9b23e changed to using ZSTD_compressBegin_usingDict() and fixed strange issue with ZSTD_compressContinue() 2017-07-11 11:50:00 -07:00
Jacques Germishuys
c325c8db82 fix missing symbol 'nanosleep' for Solaris 2017-07-11 19:26:10 +02:00
Paul Cruz
7ec5928626 fixed an error where -c argument wasn't working for single files 2017-07-11 10:23:25 -07:00
Stella Lau
f6c5d07fe2 Save v3 2017-07-11 09:23:44 -07:00
Yann Collet
ef0ff7fe7f zstdmt: removed margin for improved memory usage 2017-07-11 08:54:29 -07:00
Stella Lau
6c3673f4c3 Add rolling hash 2017-07-10 22:27:43 -07:00
Paul Cruz
f918545491 made some progress on improving compression ratio, but problems exist with speed limits, and for some reason higher compression levels are really slow 2017-07-10 18:16:42 -07:00