AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,797 Commits 1 Branch 0 Tags 27 MiB
50bc2ce95e
Commit Graph

4797 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
4b6a94f0cc clarified comments on LDM parameters 2018-02-01 17:07:27 -08:00
Yann Collet
60fa90b6c0 zstdmt: added ability to change compression parameters during compression 2018-02-01 16:13:31 -08:00
Yann Collet
2bfc79ab8d removed bitstream.h dependency 2018-02-01 16:13:04 -08:00
Yann Collet
f25351e23e
Merge pull request #1009 from terrelln/fail-early 
Test for incorrect pledgeSrcSize earlier
 2018-02-01 14:43:32 -08:00
Nick Terrell
48acaddff9 Test for incorrect pledgeSrcSize earlier 2018-02-01 12:04:05 -08:00
Yann Collet
727bb7f090
Merge pull request #1008 from terrelln/hlog3 
Fix hashLog3 size when copying cdict tables
 2018-01-31 12:49:07 -08:00
Nick Terrell
9c8c69e41a [fuzzer] Check ZSTD_initStaticCDict() for every level 2018-01-31 11:12:54 -08:00
Nick Terrell
ab3346af07 Fix hashLog3 size when copying cdict tables 2018-01-31 11:12:17 -08:00
Yann Collet
823a28a1f4
Merge pull request #1000 from facebook/progressiveFlush 
Progressive flush
 2018-01-30 22:49:47 -08:00
Yann Collet
b173b8d8da updated zstd api manual 2018-01-30 15:05:12 -08:00
Yann Collet
a2ba629971 fixed function declaration ZSTD_getBlockSize() 2018-01-30 15:03:39 -08:00
Yann Collet
2cb0740b6b zstdmt: changed naming convention 
to avoid confusion with blocks.

also:
- jobs are cut into chunks of 512KB now, to reduce nb of mutex calls.
- fix function declaration ZSTD_getBlockSizeMax()
- fix outdated comment
 2018-01-30 14:43:36 -08:00
systemcrash
6b57387728
Update README.md 
spelling
 2018-01-29 18:42:20 +01:00
systemcrash
d13a75c969
Update zstd.1 2018-01-29 18:38:02 +01:00
Yann Collet
9f8ed23b5b bumped version number to v1.3.4 
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
 2018-01-27 22:23:26 -08:00
Yann Collet
ba0cd8cf78 fixed minor conversion warning for C++ compilation mode 2018-01-26 18:18:42 -08:00
Yann Collet
caf9e96dc3 job mutex creation is checked 2018-01-26 18:09:25 -08:00
Yann Collet
9c40ae7ff1 zstdmt: there is now one mutex/cond per job 2018-01-26 17:55:08 -08:00
Yann Collet
77e36273de zstdmt: minor code refactor for clarity 2018-01-26 17:08:58 -08:00
Yann Collet
27c5853c42 zstdmt: job table correctly cleaned after synchronous ZSTDMT_compress() 2018-01-26 14:35:54 -08:00
Yann Collet
0d426f6b83 zstdmt : refactor a few member names 
for clarity
 2018-01-26 13:00:14 -08:00
Yann Collet
79b6e28b0a zstdmt : flush() only lock to read shared job members 
Other job members are accessed directly.
This avoids a full job copy, which would access everything,
including a few members that are supposed to be used by worker only,
uselessly requiring additional locks to avoid race conditions.
 2018-01-26 12:15:43 -08:00
Yann Collet
d2b62b6fa5 minor : ZSTDMT_writeLastEmptyBlock() is a void function 
because it cannot fail
 2018-01-26 11:06:34 -08:00
Yann Collet
fca13c6855 zstdmt : fixed memory leak 
writeLastEmptyBlock() must release srcBuffer
as mtctx assumes it's done by job worker.

minor : changed 2 job member names (src->srcBuffer, srcStart->prefixStart) for clarity
 2018-01-26 10:44:09 -08:00
Yann Collet
8e128eaf05 zstdmt : refactor job members 
grouped by sharing properties
 2018-01-26 10:20:38 -08:00
Yann Collet
777d3c1559 fixed minor declaration-after-statement warning 2018-01-25 17:45:18 -08:00
Yann Collet
a1d4041e69 zstdmt: removed job->jobCompleted 
replaced by equivalent signal job->consumer == job->srcSize.

created additional functions
ZSTD_writeLastEmptyBlock()
and
ZSTDMT_writeLastEmptyBlock()
required when it's necessary to finish a frame with a last empty job, to create an "end of frame" marker.

It avoids creating a job with srcSize==0.
 2018-01-25 17:35:49 -08:00
Yann Collet
1272d8e760 zstdmt:: renamed mutex and cond to underline they are context-global 2018-01-25 14:52:34 -08:00
Yann Collet
0fa96a6b78
Merge pull request #1001 from ne-sted/fix-defaults-align 
cli: fix align of defaults
 2018-01-24 09:34:36 -08:00
ne-sted
50aea2f293 cli: fix align of defaults 2018-01-24 15:07:22 +02:00
Yann Collet
4f7c896113 zstdmt : fixed complex sequencing bug 
zstdmt would shortcut to single-thread blocking mode
in some rare cases where data is sent to be compressed but is not yet ready.
 2018-01-23 18:00:51 -08:00
Yann Collet
5f349b129c zstdmt : correctly set end of frame 2018-01-23 15:52:40 -08:00
Yann Collet
c1cc57f270 zstdmt : fix end condition (ZSTD_e_end) 
When ZSTD_e_end directive is provided,
the question is not only "are internal buffers completely flushed",
it is also "is current frame completed".

In some rare cases,
it was possible for internal buffers to be completely flushed,
triggering a @return == 0,
but frame was not completed as it needed a last null-size block to mark the end,
resulting in an unfinished frame.
 2018-01-23 15:19:11 -08:00
Yann Collet
de5e38a7a6 zstdmt: fixed minor race condition 
no real consequence, but pollute tsan tests :

job->dstBuff is being modified inside worker,
while main thread might read it accidentally
because it copies whole job.
But since it doesn't used dstBuff, there is no real consequence.

Other potential solution : only copy useful data, instead of whole job
 2018-01-23 14:03:07 -08:00
Yann Collet
ebd955e26a zstdmt : fixed ending frame with 0-size block 2018-01-23 13:12:40 -08:00
Yann Collet
2054eaf840
Merge pull request #998 from facebook/zcat 
add `zcat` symlink support, suggested by @wtarreau
 2018-01-23 12:01:25 -08:00
Yann Collet
6711396d97 zstreamtest : fixed test 32 : multi-thread compression 
using ZSTD_compress_generic(,,ZSTD_e_end)
Since it already provides ZSTD_e_end as directive,
it should not be followed by ZSTDMT_endStream().
 2018-01-19 22:20:53 -08:00
Yann Collet
a7ef3a219c zstdmt : fixed last job size 2018-01-19 18:19:09 -08:00
Yann Collet
3ad7d4951c zstdmt : finally vanquished an elusive and rare race condition 2018-01-19 17:35:08 -08:00
Yann Collet
940634a610 zstdmt : simplify job creation 
job will not be created when not enough room within job Table
 2018-01-19 13:25:06 -08:00
Yann Collet
dc69623453 zstdmt: fixed corruption issue in ZSTDMT_endStream() 
when invoked directly.
 2018-01-19 12:41:56 -08:00
Yann Collet
cb5eba8e20 add zcat symlink support, suggested by @wtarreau 
added some test
also updated relevant doc

+ fixed a mistake in `lz4` symlink support :
  lz4 utility doesn't remove source files by default (like zstd, but unlike gzip).
  The symlink must behave the same.
 2018-01-19 11:26:35 -08:00
Yann Collet
70f81d6030 zstdmt uses POOL_tryAdd() to call a new worker 
so that it's no longer a blocking call.
This makes it possible to stream out data gradually,
while waiting for a worker to become available.
 2018-01-19 10:01:40 -08:00
Yann Collet
d19dc1903c
Merge pull request #995 from facebook/progressiveMT 
Progressive mt
 2018-01-18 17:59:49 -08:00
Yann Collet
31f45c98f8
Merge pull request #994 from facebook/constCDict 
changed initStatic?Dict() return type to const ZSTD_?Dict*
 2018-01-18 17:57:53 -08:00
Yann Collet
6f7280fb33 fixed frame checksum issue 
and race conditions
 2018-01-18 16:20:26 -08:00
Yann Collet
997e4d0ccd added POOL_tryAdd() 2018-01-18 14:39:51 -08:00
Yann Collet
4f43ef731d Merge branch 'dev' into constCDict 2018-01-18 13:36:43 -08:00
Yann Collet
ef97d5a287 Merge branch 'progressiveMT' into progressiveFlush 2018-01-18 13:35:24 -08:00
Yann Collet
b6ab232f2d Merge branch 'dev' into progressiveMT 2018-01-18 13:34:56 -08:00