AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,757 Commits 1 Branch 0 Tags 27 MiB
c72091556b
Commit Graph

4757 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
c72091556b fixed minor nit as per @terrelln's comments 2018-02-09 09:46:08 -08:00
Yann Collet
4beaeaace5 Merge branch 'dev' into flexibleLevel 2018-02-09 09:15:05 -08:00
Yann Collet
d6e841d609 fixed streaming_memory_usage example 
also:
ensure zstd.h is read from ../lib (instead of /usr/include)
 2018-02-07 23:42:09 -08:00
Yann Collet
5188749e1c ensure compression parameters are updated when only compression level is changed 2018-02-02 16:31:20 -08:00
Yann Collet
4b525af53a zstdmt: applies new parameters on the fly 
when invoked from ZSTD_compress_generic()
 2018-02-02 15:58:13 -08:00
Yann Collet
90eca318a7 fileio: create dedicated function to generate zstd frames 
like other formats
 2018-02-02 14:24:56 -08:00
Yann Collet
1291d9d7cf
Merge pull request #1006 from systemcrash/patch-2 
Update README.md
 2018-02-02 10:04:55 -08:00
Yann Collet
549d26ae71
Merge pull request #1005 from systemcrash/dev 
Update zstd.1
 2018-02-02 10:04:40 -08:00
Yann Collet
6c492af284 fixed minor conversion warning 2018-02-01 20:16:00 -08:00
Yann Collet
209df52ba2 Changed nbThreads for nbWorkers 
This makes it easier to explain that nbWorkers=0 --> single-threaded mode,
while nbWorkers=1 --> asynchronous mode (one mode thread on top of the "main" caller thread).
No need for an additional asynchronous mode flag.
nbWorkers>=2 works the same as nbThreads>=2 previously.
 2018-02-01 19:29:30 -08:00
Yann Collet
4b6a94f0cc clarified comments on LDM parameters 2018-02-01 17:07:27 -08:00
Yann Collet
60fa90b6c0 zstdmt: added ability to change compression parameters during compression 2018-02-01 16:13:31 -08:00
Yann Collet
2bfc79ab8d removed bitstream.h dependency 2018-02-01 16:13:04 -08:00
Yann Collet
f25351e23e
Merge pull request #1009 from terrelln/fail-early 
Test for incorrect pledgeSrcSize earlier
 2018-02-01 14:43:32 -08:00
Nick Terrell
48acaddff9 Test for incorrect pledgeSrcSize earlier 2018-02-01 12:04:05 -08:00
Yann Collet
727bb7f090
Merge pull request #1008 from terrelln/hlog3 
Fix hashLog3 size when copying cdict tables
 2018-01-31 12:49:07 -08:00
Nick Terrell
9c8c69e41a [fuzzer] Check ZSTD_initStaticCDict() for every level 2018-01-31 11:12:54 -08:00
Nick Terrell
ab3346af07 Fix hashLog3 size when copying cdict tables 2018-01-31 11:12:17 -08:00
Yann Collet
823a28a1f4
Merge pull request #1000 from facebook/progressiveFlush 
Progressive flush
 2018-01-30 22:49:47 -08:00
Yann Collet
b173b8d8da updated zstd api manual 2018-01-30 15:05:12 -08:00
Yann Collet
a2ba629971 fixed function declaration ZSTD_getBlockSize() 2018-01-30 15:03:39 -08:00
Yann Collet
2cb0740b6b zstdmt: changed naming convention 
to avoid confusion with blocks.

also:
- jobs are cut into chunks of 512KB now, to reduce nb of mutex calls.
- fix function declaration ZSTD_getBlockSizeMax()
- fix outdated comment
 2018-01-30 14:43:36 -08:00
systemcrash
6b57387728
Update README.md 
spelling
 2018-01-29 18:42:20 +01:00
systemcrash
d13a75c969
Update zstd.1 2018-01-29 18:38:02 +01:00
Yann Collet
9f8ed23b5b bumped version number to v1.3.4 
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
 2018-01-27 22:23:26 -08:00
Yann Collet
ba0cd8cf78 fixed minor conversion warning for C++ compilation mode 2018-01-26 18:18:42 -08:00
Yann Collet
caf9e96dc3 job mutex creation is checked 2018-01-26 18:09:25 -08:00
Yann Collet
9c40ae7ff1 zstdmt: there is now one mutex/cond per job 2018-01-26 17:55:08 -08:00
Yann Collet
77e36273de zstdmt: minor code refactor for clarity 2018-01-26 17:08:58 -08:00
Yann Collet
27c5853c42 zstdmt: job table correctly cleaned after synchronous ZSTDMT_compress() 2018-01-26 14:35:54 -08:00
Yann Collet
0d426f6b83 zstdmt : refactor a few member names 
for clarity
 2018-01-26 13:00:14 -08:00
Yann Collet
79b6e28b0a zstdmt : flush() only lock to read shared job members 
Other job members are accessed directly.
This avoids a full job copy, which would access everything,
including a few members that are supposed to be used by worker only,
uselessly requiring additional locks to avoid race conditions.
 2018-01-26 12:15:43 -08:00
Yann Collet
d2b62b6fa5 minor : ZSTDMT_writeLastEmptyBlock() is a void function 
because it cannot fail
 2018-01-26 11:06:34 -08:00
Yann Collet
fca13c6855 zstdmt : fixed memory leak 
writeLastEmptyBlock() must release srcBuffer
as mtctx assumes it's done by job worker.

minor : changed 2 job member names (src->srcBuffer, srcStart->prefixStart) for clarity
 2018-01-26 10:44:09 -08:00
Yann Collet
8e128eaf05 zstdmt : refactor job members 
grouped by sharing properties
 2018-01-26 10:20:38 -08:00
Yann Collet
777d3c1559 fixed minor declaration-after-statement warning 2018-01-25 17:45:18 -08:00
Yann Collet
a1d4041e69 zstdmt: removed job->jobCompleted 
replaced by equivalent signal job->consumer == job->srcSize.

created additional functions
ZSTD_writeLastEmptyBlock()
and
ZSTDMT_writeLastEmptyBlock()
required when it's necessary to finish a frame with a last empty job, to create an "end of frame" marker.

It avoids creating a job with srcSize==0.
 2018-01-25 17:35:49 -08:00
Yann Collet
1272d8e760 zstdmt:: renamed mutex and cond to underline they are context-global 2018-01-25 14:52:34 -08:00
Yann Collet
0fa96a6b78
Merge pull request #1001 from ne-sted/fix-defaults-align 
cli: fix align of defaults
 2018-01-24 09:34:36 -08:00
ne-sted
50aea2f293 cli: fix align of defaults 2018-01-24 15:07:22 +02:00
Yann Collet
4f7c896113 zstdmt : fixed complex sequencing bug 
zstdmt would shortcut to single-thread blocking mode
in some rare cases where data is sent to be compressed but is not yet ready.
 2018-01-23 18:00:51 -08:00
Yann Collet
5f349b129c zstdmt : correctly set end of frame 2018-01-23 15:52:40 -08:00
Yann Collet
c1cc57f270 zstdmt : fix end condition (ZSTD_e_end) 
When ZSTD_e_end directive is provided,
the question is not only "are internal buffers completely flushed",
it is also "is current frame completed".

In some rare cases,
it was possible for internal buffers to be completely flushed,
triggering a @return == 0,
but frame was not completed as it needed a last null-size block to mark the end,
resulting in an unfinished frame.
 2018-01-23 15:19:11 -08:00
Yann Collet
de5e38a7a6 zstdmt: fixed minor race condition 
no real consequence, but pollute tsan tests :

job->dstBuff is being modified inside worker,
while main thread might read it accidentally
because it copies whole job.
But since it doesn't used dstBuff, there is no real consequence.

Other potential solution : only copy useful data, instead of whole job
 2018-01-23 14:03:07 -08:00
Yann Collet
ebd955e26a zstdmt : fixed ending frame with 0-size block 2018-01-23 13:12:40 -08:00
Yann Collet
2054eaf840
Merge pull request #998 from facebook/zcat 
add `zcat` symlink support, suggested by @wtarreau
 2018-01-23 12:01:25 -08:00
Yann Collet
6711396d97 zstreamtest : fixed test 32 : multi-thread compression 
using ZSTD_compress_generic(,,ZSTD_e_end)
Since it already provides ZSTD_e_end as directive,
it should not be followed by ZSTDMT_endStream().
 2018-01-19 22:20:53 -08:00
Yann Collet
a7ef3a219c zstdmt : fixed last job size 2018-01-19 18:19:09 -08:00
Yann Collet
3ad7d4951c zstdmt : finally vanquished an elusive and rare race condition 2018-01-19 17:35:08 -08:00
Yann Collet
940634a610 zstdmt : simplify job creation 
job will not be created when not enough room within job Table
 2018-01-19 13:25:06 -08:00