AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
5,704 Commits 1 Branch 0 Tags 27 MiB
0e2dbac18a
Commit Graph

5704 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
0e2dbac18a changed overflow correction place 
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
 2018-09-26 15:35:38 -07:00
Yann Collet
f98c69d77c fix : huge (>4GB) stream of blocks 
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
 2018-09-26 14:24:28 -07:00
Yann Collet
9bb6c15f79
Merge pull request #1332 from facebook/minclevel 
defined a minimum negative level
 2018-09-24 16:01:13 -07:00
Yann Collet
8d69c303e4
Merge pull request #1336 from facebook/scanbuild 
fixed minor scan-build warnings
 2018-09-24 14:49:16 -07:00
Yann Collet
3ece0316d8
Merge pull request #1335 from facebook/limits 
added some tests based on limits.h
 2018-09-24 14:49:03 -07:00
Yann Collet
0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet
292d8e4a83 added some tests based on limits.h 
in order to ensure proper type mapping
when not using stdint.h
 2018-09-23 23:57:30 -07:00
Yann Collet
0fc07eb1fd fixed zstd-decompress 
which cannot support ZSTD_minCLevel()
 2018-09-22 17:21:39 -07:00
Yann Collet
7d9f386a0d
Merge pull request #1331 from facebook/mingw 
fix mingw compatibility
 2018-09-22 16:11:55 -07:00
Yann Collet
364041c6dd enforce minimum compression level limit 
using ZSTD_minCLevel()
 2018-09-22 16:10:10 -07:00
Yann Collet
0e211bdd18 fixed constant comparison on 32-bits systems 2018-09-21 18:23:32 -07:00
Yann Collet
71a5210617 avoid recompiling dll every time under mingw 2018-09-21 17:40:30 -07:00
Yann Collet
123fac6b6d fix pzstd compatibility with mingw 
some details changed with introduction of gcc7
 2018-09-21 17:36:00 -07:00
Yann Collet
484f40697b fix constant redeclaration in paramgrill 2018-09-21 17:28:37 -07:00
Yann Collet
32b7cf1bcf fixed tautological tests 
involving ZSTD_TARGETLENGTH_MIN (== 0)
 2018-09-21 15:04:43 -07:00
Yann Collet
c044345f8f Merge branch 'mingw' into minclevel 2018-09-21 14:56:57 -07:00
Yann Collet
54001f3dbc fix mingw compatibility 
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
 2018-09-21 14:46:09 -07:00
Yann Collet
de6c75e4e5
Merge pull request #1318 from felixhandte/shadow-dict-matches 
Don't Search Dictionary Context When Working Context Search Resulted in Mismatch
 2018-09-21 12:15:33 -07:00
Yann Collet
a54c86cfc6 defined a minimum negative level 
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
 2018-09-20 16:52:03 -07:00
Yann Collet
c8ff5200f9
Merge pull request #1326 from facebook/refPrefix 
updated documentation of *refPrefix()
 2018-09-18 15:28:34 -07:00
Yann Collet
005f000aed updated documentation of *refPrefix() 
indicating the equivalence with `diff` operation.
 2018-09-18 13:07:08 -07:00
Nick Terrell
f884e9dd8e
Merge pull request #1321 from bket/non-portable 
'head -c BYTES' is non-portable.
 2018-09-17 14:40:27 -07:00
Yann Collet
00ce26725b
Merge pull request #1324 from ko-zu/fixclangcode 
Fix largeNbDicts bench for clangbuild
 2018-09-17 14:10:17 -07:00
Björn Ketelaars
06fd1e473d 'head -c BYTES' is non-portable. 
tests/playTests.sh uses 'head -c' in a couple of tests to truncate the
last byte of a file. The '-c' option is non-portable (not in POSIX).
Instead use a wrapper around dd (truncateLastByte).
 2018-09-17 20:39:35 +02:00
Nick Terrell
8f27e8cf3d
Merge pull request #1322 from azat-archive/seekable-fixes-pull 
Fixes read write past end of input buffer.
 2018-09-17 11:04:51 -07:00
ko-zu
b053bec2f4 Fix largeNbDicts bench for clangbuild 
Remove unsigned to size_t promotion to fix implicit down conversion errors in clangbuild target.
 2018-09-17 13:09:08 +09:00
Azat Khuzhin
d707692e05
seekable_decompression: support offset greater then UNIT_MAX 2018-09-16 18:05:32 +03:00
Azat Khuzhin
b52867a97f
zstdseek_decompress: fix decompression with data left in input buffer 2018-09-16 18:05:32 +03:00
Yann Collet
a2aedf458d
Merge pull request #1319 from ko-zu/fixclangbuild 
Fix clang build
 2018-09-15 19:31:42 -07:00
ko-zu
18b4a1da61 Fix clang build 
Fix dixygen comment
Fix clang binary path
 2018-09-16 10:27:02 +09:00
Yann Collet
7269fe6cd3 minor code comment update 2018-09-14 16:06:35 -07:00
Yann Collet
0403148315
Merge pull request #1295 from felixhandte/hdr-intro-comment-negative-lvls 
Proposed Update to Zstd.h Introduction Comment
 2018-09-14 15:29:19 -07:00
W. Felix Handte
b76c888497 ZSTD_dfast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:24:25 -07:00
W. Felix Handte
b048af5999 ZSTD_fast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:23:35 -07:00
Yann Collet
0e5b447aaa
Merge pull request #1316 from facebook/coldDict 
Cold dictionary mitigation
 2018-09-14 10:37:46 -07:00
Yann Collet
5512400677 updated code comments, based on @terrelln review 2018-09-13 16:44:04 -07:00
Yann Collet
d195eec97e fixed msan error 
cold dictionary is detected through a comparison with dictEnd,
which was not initialized at the beginning of first DCtx usage.
 2018-09-13 12:29:52 -07:00
Yann Collet
674dd21bd0 final parameter tuning 2018-09-12 17:25:34 -07:00
Yann Collet
419dfd4ea3 clean traces 2018-09-12 16:40:28 -07:00
Yann Collet
2618253da2 fixed PREFETCH() macro 
for corner cases and platforms without this instruction
 2018-09-12 16:15:37 -07:00
Yann Collet
b287b8f7c6
Merge pull request #1315 from cmcginty/dev 
Add build define to manually enable -rdynamic LD flag
 2018-09-12 16:12:00 -07:00
Yann Collet
44d3b83bb1 conditional dict content prefetching 
based on nbSeq.
 2018-09-12 15:35:21 -07:00
Casey McGinty
d019d15f2b
Add build define to manually enable -rdynamic LD flag 2018-09-12 14:00:08 -07:00
Yann Collet
7800d0235c
Merge pull request #1314 from terrelln/cpuid 
[libzstd] Fix cpu for MSFT ARM
 2018-09-12 13:30:16 -07:00
Yann Collet
5fb5ed3b31 adjust heuristic decisions 2018-09-12 12:32:09 -07:00
Nick Terrell
f6daddf2db Also allow x86 2018-09-12 12:05:32 -07:00
Yann Collet
c49ccbc8e7 largeNbDicts : can select a nb of blocks 
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
 2018-09-12 11:31:28 -07:00
Nick Terrell
1e0bac6a9c [libzstd] Fix cpu for MSFT ARM 
The `__cpuid()` and `__cpuidex()` intrinsics are only available
on x86 and x86_64.
 2018-09-12 10:35:16 -07:00
Yann Collet
4de344d505 added conditional prefetch 
depending on amount of work to do.
 2018-09-12 10:29:47 -07:00
Yann Collet
cf5db388f3
Merge pull request #1311 from cmcginty/dev 
Print a stack trace on unexpected term signal (e.g. SIGABRT)
 2018-09-12 10:13:23 -07:00