AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,782 Commits 1 Branch 0 Tags 27 MiB
3538a535bf
Commit Graph

4782 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
3538a535bf use TIMELOOP_NANOSEC 
as suggested by @terrelln
 2018-02-20 15:33:56 -08:00
Yann Collet
d3364aa39e improve benchmark measurement for small inputs 
by invoking time() once per batch, instead of once per compression / decompression.
Batch is dynamically resized so that each round lasts approximately 1 second.

Also : increases time accuracy to nanosecond
 2018-02-20 14:58:40 -08:00
Yann Collet
70163bf0d3 added clarification comments in zstd_errors.h 
answering some points in #1018
 2018-02-20 12:54:49 -08:00
Yann Collet
7117ea8bec
Merge pull request #1011 from terrelln/bmi2 
[decompress] Support BMI2
 2018-02-15 11:40:34 -08:00
Nick Terrell
4319132312 [decompress] Support BMI2 2018-02-13 17:00:15 -08:00
Yann Collet
5cb1144872 fixed --single-thread 
was incorrectly set to -T0 (use as many cores as possible) previously
 2018-02-13 14:56:35 -08:00
Yann Collet
9716250197
Merge pull request #1014 from facebook/fasterDec 
Faster decoding speed
 2018-02-13 12:05:54 -08:00
Yann Collet
9b184359e2 pretify last unit test output 2018-02-13 10:09:01 -08:00
Yann Collet
2524cbd847 added code comment on how to generate default tables 
as suggested by @terrelln
 2018-02-13 10:02:25 -08:00
Yann Collet
71c07966bb added SEQSYMBOL_TABLE_SIZE() 
as suggested by @terrelln's comment
 2018-02-12 16:52:15 -08:00
Yann Collet
821efa466e fixed logo path 2018-02-10 21:05:48 -08:00
Yann Collet
5f7495371e Merge branch 'dev' into fasterDec 2018-02-10 14:24:44 -08:00
Yann Collet
992c2370f6
Merge pull request #1010 from facebook/flexibleLevel 
Updatable compression parameters
 2018-02-10 14:19:54 -08:00
Yann Collet
9945e60ac4 Merge branch 'dev' into flexibleLevel 2018-02-10 11:54:49 -08:00
Yann Collet
4e3db17cab
Merge pull request #1013 from facebook/fasterDec32 
Disable Long Offset mode in 32-bits
 2018-02-09 16:13:55 -08:00
Yann Collet
75689838e4 specify new command --single-thread 2018-02-09 15:55:41 -08:00
Yann Collet
c72091556b fixed minor nit as per @terrelln's comments 2018-02-09 09:46:08 -08:00
Yann Collet
4beaeaace5 Merge branch 'dev' into flexibleLevel 2018-02-09 09:15:05 -08:00
Yann Collet
6bfe50ad48 re-enabled ZSTD_decompressSequencesLong() 2018-02-09 09:14:25 -08:00
Yann Collet
1850597eaa pre-calculated default decoding tables 2018-02-09 06:01:02 -08:00
Yann Collet
ab75df21ed fixed mono-symbol distribution 2018-02-09 05:12:13 -08:00
Yann Collet
421a2716d8 fixed default fse distributions 
but would be better to pre-calculate tables, for speed
 2018-02-09 04:50:58 -08:00
Yann Collet
95424409ea addBits and baseline into FSE decoding table 
note : unfinished
- need new default tables
- need modify long mode
 2018-02-09 04:25:15 -08:00
Yann Collet
cc61a3694a Merge branch 'dev' into fasterDec 2018-02-09 02:41:02 -08:00
Yann Collet
d6e841d609 fixed streaming_memory_usage example 
also:
ensure zstd.h is read from ../lib (instead of /usr/include)
 2018-02-07 23:42:09 -08:00
Yann Collet
de68c2ff10 Merged ZSTD_preserveUnsortedMark() into ZSTD_reduceIndex() 
as it's faster, due to one memory scan instead of two
(confirmed by microbenchmark).

Note : as ZSTD_reduceIndex() is rarely invoked,
it does not translate into a visible gain.
Consider it an exercise in auto-vectorization and micro-benchmarking.
 2018-02-07 14:22:35 -08:00
Yann Collet
0170cf9a7a minor : modified ZSTD_preserveUnsortedMark() to be more vectorization friendly 2018-02-05 11:46:02 -08:00
Yann Collet
94efb1749d faster decoding in 32-bits mode for long offsets (tentative) 
On my laptop:
Before:
./zstd32 -b --zstd=wlog=27 silesia.tar enwik8 -S
 3#silesia.tar       : 211984896 ->  66683478 (3.179),  97.6 MB/s , 400.7 MB/s
 3#enwik8            : 100000000 ->  35643153 (2.806),  76.5 MB/s , 303.2 MB/s

After:
./zstd32 -b --zstd=wlog=27 silesia.tar enwik8 -S
 3#silesia.tar       : 211984896 ->  66683478 (3.179),  97.4 MB/s , 435.0 MB/s
 3#enwik8            : 100000000 ->  35643153 (2.806),  76.2 MB/s , 338.1 MB/s

Mileage vary, depending on file, and cpu type.
But a generic rule is : x86 benefits less from "long-offset mode" than x64,
maybe due to register pressure.
On "entropy", long-mode is _never_ a win for x86.
On my laptop though, it may, depending on file and compression level
(enwik8 benefits more from "long-mode" than silesia).
 2018-02-04 01:49:31 -08:00
Yann Collet
5188749e1c ensure compression parameters are updated when only compression level is changed 2018-02-02 16:31:20 -08:00
Yann Collet
4b525af53a zstdmt: applies new parameters on the fly 
when invoked from ZSTD_compress_generic()
 2018-02-02 15:58:13 -08:00
Yann Collet
90eca318a7 fileio: create dedicated function to generate zstd frames 
like other formats
 2018-02-02 14:24:56 -08:00
Yann Collet
1291d9d7cf
Merge pull request #1006 from systemcrash/patch-2 
Update README.md
 2018-02-02 10:04:55 -08:00
Yann Collet
549d26ae71
Merge pull request #1005 from systemcrash/dev 
Update zstd.1
 2018-02-02 10:04:40 -08:00
Yann Collet
6c492af284 fixed minor conversion warning 2018-02-01 20:16:00 -08:00
Yann Collet
209df52ba2 Changed nbThreads for nbWorkers 
This makes it easier to explain that nbWorkers=0 --> single-threaded mode,
while nbWorkers=1 --> asynchronous mode (one mode thread on top of the "main" caller thread).
No need for an additional asynchronous mode flag.
nbWorkers>=2 works the same as nbThreads>=2 previously.
 2018-02-01 19:29:30 -08:00
Yann Collet
4b6a94f0cc clarified comments on LDM parameters 2018-02-01 17:07:27 -08:00
Yann Collet
60fa90b6c0 zstdmt: added ability to change compression parameters during compression 2018-02-01 16:13:31 -08:00
Yann Collet
2bfc79ab8d removed bitstream.h dependency 2018-02-01 16:13:04 -08:00
Yann Collet
f25351e23e
Merge pull request #1009 from terrelln/fail-early 
Test for incorrect pledgeSrcSize earlier
 2018-02-01 14:43:32 -08:00
Nick Terrell
48acaddff9 Test for incorrect pledgeSrcSize earlier 2018-02-01 12:04:05 -08:00
Yann Collet
727bb7f090
Merge pull request #1008 from terrelln/hlog3 
Fix hashLog3 size when copying cdict tables
 2018-01-31 12:49:07 -08:00
Nick Terrell
9c8c69e41a [fuzzer] Check ZSTD_initStaticCDict() for every level 2018-01-31 11:12:54 -08:00
Nick Terrell
ab3346af07 Fix hashLog3 size when copying cdict tables 2018-01-31 11:12:17 -08:00
Yann Collet
823a28a1f4
Merge pull request #1000 from facebook/progressiveFlush 
Progressive flush
 2018-01-30 22:49:47 -08:00
Yann Collet
b173b8d8da updated zstd api manual 2018-01-30 15:05:12 -08:00
Yann Collet
a2ba629971 fixed function declaration ZSTD_getBlockSize() 2018-01-30 15:03:39 -08:00
Yann Collet
2cb0740b6b zstdmt: changed naming convention 
to avoid confusion with blocks.

also:
- jobs are cut into chunks of 512KB now, to reduce nb of mutex calls.
- fix function declaration ZSTD_getBlockSizeMax()
- fix outdated comment
 2018-01-30 14:43:36 -08:00
systemcrash
6b57387728
Update README.md 
spelling
 2018-01-29 18:42:20 +01:00
systemcrash
d13a75c969
Update zstd.1 2018-01-29 18:38:02 +01:00
Yann Collet
9f8ed23b5b bumped version number to v1.3.4 
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
 2018-01-27 22:23:26 -08:00