AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,351 Commits 1 Branch 0 Tags 27 MiB
0a3fa6f909
Commit Graph

6351 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell
cd2c8defad [zstdcli] Refuse to overwrite input file 
Compare the input and output files by their inode number and
refuse to open the output file if the input file is the same.

This doesn't work when (de)compressing multiple files to a single
file, but that is a very uncommon use case, mostly used for
benchmarking by me.

Fixes #1422.
 2018-12-18 15:29:54 -08:00
Yann Collet
d0e15f8d32
Merge pull request #1458 from terrelln/estimate 
[libzstd] Fix estimate with negative levels
 2018-12-18 15:12:21 -08:00
Yann Collet
04baecaeed
Merge pull request #1457 from facebook/btultra2.1 
btultra2 and very small input
 2018-12-18 14:46:55 -08:00
Nick Terrell
d7def456d8 [libzstd] Fix estimate with negative levels 
* Fix `ZSTD_estimateCCtxSize()` with negative levels.
* Fix `ZSTD_estimateCStreamSize()` with negative levels.
* Add a unit test to test for this error.
 2018-12-18 14:24:49 -08:00
Yann Collet
ef984e7307 fix debug levels 
as reported by @terrelln.
2 is reserved for temporary usage only.
 2018-12-18 13:40:07 -08:00
W. Felix Handte
0d606ee3db Fix Incorrect assert() 2018-12-18 13:36:39 -08:00
W. Felix Handte
bd4afc389f Add Logic to Makefile to Convert Make Vars to Defines 2018-12-18 13:36:39 -08:00
W. Felix Handte
ece2c18372 Document Macros in README 2018-12-18 13:36:39 -08:00
W. Felix Handte
4e2f6c110e Add Contbuild Tests 2018-12-18 13:36:39 -08:00
W. Felix Handte
c2d51637d9 Add Mutual-Exclusion Error 2018-12-18 13:36:39 -08:00
W. Felix Handte
c560e34c86 Add HUF_FORCE_DECOMPRESS_X2 2018-12-18 13:36:39 -08:00
W. Felix Handte
abd1567d3c Move HUF_DGEN Up Out of X1 Definitions 2018-12-18 13:36:39 -08:00
W. Felix Handte
4a0572b215 Refactor Huffman Decompression Away From Ternary Tree in ZSTD_decodeLiteralsBlock 2018-12-18 13:36:39 -08:00
W. Felix Handte
432314b58a Rename HUF_DECOMPRESS_MINIMAL -> HUF_FORCE_DECOMPRESS_X1 2018-12-18 13:36:39 -08:00
W. Felix Handte
4bbb8a48ad Add ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG 
This macro forces behavior in the opposite direction.
 2018-12-18 13:36:39 -08:00
W. Felix Handte
64553a0e35 Rename ZSTD_DECOMPRESS_MINIMAL -> ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT 2018-12-18 13:36:39 -08:00
W. Felix Handte
605dd576ee Remove Error Strings with ZSTD_STRIP_ERROR_STRINGS 2018-12-18 13:36:39 -08:00
W. Felix Handte
9d5f3963ff Add Option to Not Request Inlining with ZSTD_NO_INLINE 2018-12-18 13:36:39 -08:00
W. Felix Handte
df28e5babd Add ZSTD_DECOMPRESS_MINIMAL Macro, Which Reduces Branching of Decompress Variants 2018-12-18 13:36:39 -08:00
W. Felix Handte
f45c9df42e Totally Hide/Disable X2 Variants when HUF_DECOMPRESS_MINIMAL is Defined 2018-12-18 13:36:39 -08:00
W. Felix Handte
36a84b07a8 Load Dictionaries as X1 Tables 2018-12-18 13:36:39 -08:00
W. Felix Handte
f9cb348776 Add HUF_DECOMPRESS_MINIMAL Macro, Which Avoids Using X2 Variants 2018-12-18 13:36:39 -08:00
Yann Collet
635783da12 btultra2 and very small srcSize 
When srcSize is small,
the nb of symbols produced is likely too small to warrant dedicated probability tables.
In which case, predefined distribution tables will be used instead.

There is a cheap algorithm in btultra initialization :
it presumes default distribution will be used if srcSize <= 1024.

btultra2 now uses the same threshold to shut down probability estimation,
since measured frequencies won't be used at entropy stage,
and therefore relying on them to determine sequence cost is misleading,
resulting in worse compression ratios.

This fixes btultra2 performance issue on very small input.

Note that, a proper way should be
to determine which symbol is going to use predefined probaility
and which symbol is going to use dynamic ones.
But the current algorithm is unable to make a "per-symbol" decision.
So this will require significant modifications.
 2018-12-18 12:32:58 -08:00
Yann Collet
517d8c984c
Merge pull request #1449 from facebook/ovlog_def 
overlapLog default values
 2018-12-18 09:45:53 -08:00
Yann Collet
345af8abc7
Merge pull request #1456 from yijinfb/example-ZSTD_compressCCtx 
Add multiple_simple_compression.c (demonstrates the use of ZSTD_compressCCtx(...)) to examples directory.
 2018-12-18 09:24:18 -08:00
Yi Jin
bc4dc606de break loadFile_orDie() into 2: loadFile_orDie() loads file into a pre-allocated memory buffer, mallocAndLoadFile_orDie() allocates memory first, then calls loadFile_orDie() 2018-12-17 16:54:55 -08:00
Yann Collet
373ff8b983 play around with rescale weights 2018-12-17 15:48:34 -08:00
Yi Jin
452689678b bug fix in loadFile_orDie(): exit with error code if filesize is bigger than provided buffer 2018-12-16 22:44:27 -08:00
Yi Jin
a951ee9fe1 comment out printf() to make output terse 2018-12-16 21:27:52 -08:00
Yi Jin
595401e4c7 edit README.md and remove simple_compressionCCtx.c 2018-12-16 21:09:21 -08:00
Yi Jin
ea4859ec71 add multiple_simple_compression.c 2018-12-16 20:45:43 -08:00
Yi Jin
cce0842c60 Factor out allocMemory_orDIe(...) in simple_compressionCCtx.c, and amend Makefile to account for dependency on utils.h 2018-12-16 15:36:28 -08:00
Yi Jin
0d14153cfb formatting 2018-12-14 22:10:31 -08:00
Yi Jin
04d06ad885 refactor utils.h and implement simple_compressionCCtx.c 2018-12-14 18:12:05 -08:00
Yann Collet
5bdbd997ae
Merge pull request #1452 from lzutao/meson_getversion 
meson: Remove unused sys import
 2018-12-14 16:50:41 -08:00
Nick Terrell
7f6d11e1e2
Merge pull request #1450 from terrelln/no-progress 
[zstdcli] Add --no-progress flag
 2018-12-14 11:50:59 -08:00
Nick Terrell
bdfcaecc0a [zstdcli] Add --no-progress flag 
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.

I've added simple tests to `playTests.sh` to make sure the parsing
works.
 2018-12-14 11:50:25 -08:00
Yann Collet
d4698424ce
Merge pull request #1447 from lzutao/meson_symlink_soversion 
meson: More accurate Windows build support
 2018-12-14 09:18:37 -08:00
Yi Jin
193fbd30f2 WIP 2018-12-13 22:51:35 -08:00
Lzu Tao
066cfc069b meson: Remove unused sys import 2018-12-14 11:03:04 +07:00
Yann Collet
96adc846c5 fixed tests 
with correct pointer type
 2018-12-13 16:50:19 -08:00
Yann Collet
8be145a8c1 fixed default job size 2018-12-13 16:38:08 -08:00
Nick Terrell
75fa3f2eb7
Merge pull request #1446 from terrelln/overflow 
[libzstd] Fix infinite loop in decompression
 2018-12-13 16:21:15 -08:00
Yann Collet
62180b27d5 zstdmt parameter getter/setter use int 2018-12-13 15:47:34 -08:00
Yi Jin
7f975c0c28 create simple_compressionCCtx.c out of simple_compression.c, to be worked on 2018-12-13 15:42:55 -08:00
Nick Terrell
aaea4ef924 [libzstd] Fix infinite loop in decompression 
When we switched `ZSTD_SKIPPABLEHEADERSIZE` to a macro, the places where we do:

    MEM_readLE32(ptr) + ZSTD_SKIPPABLEHEADERSIZE

can now overflow `(unsigned)-8` to `0` and we infinite loop. We now check
the frame size and reject sizes that overflow a U32.

Note that this bug never made it into a release, and was only in the dev branch
for a few days.

Credit to OSS-Fuzz
 2018-12-13 15:13:19 -08:00
Yann Collet
34f01e600f fixed multiple conversions 
from 64-bit to 32-bit
 2018-12-13 14:02:22 -08:00
Yann Collet
4acf139e9f
Merge pull request #1448 from lzutao/travis_reformat 
travis: Use script instead of Cmd env
 2018-12-13 12:04:10 -08:00
Lzu Tao
b3be899469 travis: Use script instead of Cmd env 2018-12-14 02:06:14 +07:00
Lzu Tao
3ee5504fb2 Simplify logic by setting default value for MESON_INSTALL_DESTDIR_PREFIX 2018-12-13 18:07:01 +07:00