AuroraMiddleware/zstd
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
6,807 Commits 1 Branch 0 Tags 27 MiB
81208fd7c2
Commit Graph

2937 Commits

Author SHA1 Message Date
W. Felix Handte
429987c9a6 Add Comment 2019-01-28 17:35:31 -05:00
W. Felix Handte
2179ce00e1 Remove CHECK_E Macro 2019-01-28 17:33:13 -05:00
W. Felix Handte
03e040a966 Replace Uses of CHECK_E with RETURN_ERROR_IF(*_isError(... 2019-01-28 17:33:01 -05:00
W. Felix Handte
7ebd897157 Remove CHECK_F Macro 2019-01-28 17:16:32 -05:00
W. Felix Handte
64bb6640f2 Replace CHECK_F Uses in zstdmt_compress.c and zstd_ddict.c 2019-01-28 17:15:57 -05:00
W. Felix Handte
cafc3b1bcb Also Convert zstd_compress.c 2019-01-28 17:05:18 -05:00
W. Felix Handte
324e9654d3 Add grep-able String to Error Macros 2019-01-28 12:50:36 -05:00
W. Felix Handte
32fed9c7be Switch CHECK_F Calls to FORWARD_ERROR 2019-01-28 12:45:34 -05:00
W. Felix Handte
800c87fed0 Switch Unconditional RETURN_ERROR_IF Calls to RETURN_ERROR 2019-01-28 12:45:34 -05:00
W. Felix Handte
a3538bbc6f Add RETURN_ERROR and FORWARD_ERROR Macros 2019-01-28 12:45:26 -05:00
W. Felix Handte
c823237d7b Convert Checks in zstd_decompress.c to RETURN_ERROR_IF 2019-01-28 12:23:14 -05:00
W. Felix Handte
ea031f4ea2 Convert Checks in zstd_decompress_block.c to RETURN_ERROR_IF 2019-01-28 11:56:39 -05:00
W. Felix Handte
54fa31f03b Add RETURN_ERROR_IF Macro That Logs Debug Information When Check Fails 2019-01-28 11:43:33 -05:00
Yann Collet
f9e4f89252 improved comments for adjustCParams() and getCParams() 2019-01-02 12:18:40 -08:00
Yann Collet
0fb4b21d1a updated libzstd documentation 2018-12-25 03:10:07 -08:00
Yann Collet
e980ba212f
Merge pull request #1471 from facebook/nofloat 
guard functions using floating point for debug mode only
 2018-12-23 12:35:51 -08:00
Yann Collet
aae5bc538a
Merge pull request #1470 from facebook/U32 
fix confusion between unsigned <-> U32
 2018-12-23 12:35:39 -08:00
Yann Collet
c9dfb7e445 guard functions using floating point for debug mode only 
they are only used to print debug messages.
Requested in #1386,
 2018-12-22 09:09:40 -08:00
Yann Collet
ededcfca57 fix confusion between unsigned <-> U32 
as suggested in #1441.

generally U32 and unsigned are the same thing,
except when they are not ...

case : 32-bit compilation for MIPS (uint32_t == unsigned long)

A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).

Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
  but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
  but the caller user a pointer to U32 instead.

These fixes are useful.

However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.

As a consequence, the amount of code changed is larger than I would expect for such a patch.

Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.

So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.

I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.

Thoughts ?
 2018-12-21 18:09:41 -08:00
Yann Collet
c8d1fda982 update aarch64 test to xenial 
in an attempt to circumvent the `ld` bug
 2018-12-21 15:08:48 -08:00
Yann Collet
8f35c7f94c
Merge pull request #1466 from facebook/noDictPresent 
fixed : better error message
 2018-12-20 19:01:27 -08:00
Yann Collet
41b45b84a1
Merge pull request #1465 from facebook/noFilePresent 
fixed : detection of non-existing file
 2018-12-20 17:21:04 -08:00
Yann Collet
ed2fb6bd57 fixed : better error message when dictionary missing 
during benchmark.
Also : refactored ZSTD_fillHashTable(),
just for readability (it does the same thing)
 2018-12-20 17:20:07 -08:00
Yann Collet
e4ae24c229
Merge pull request #1420 from felixhandte/zstd-decompress-minimal 
Various Macros to Allow Building Extremely Minimal Decoder Library
 2018-12-20 15:17:37 -08:00
Yann Collet
95784c654c fixed shadowing of stat variable 
some standard lib declares a `stat` variable at global scope
shadowing local declarations ....
 2018-12-20 14:56:44 -08:00
Yann Collet
ffba142406 fixed file identity detection in 32-bit mode 
also :
some library decided to use `index` as a global variable declared in standard header
shadowing the ones used in fastcover.c  :(
 2018-12-20 14:30:30 -08:00
W. Felix Handte
91b7309115 Mask Off Unused Functions When ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG 2018-12-20 12:20:34 -08:00
W. Felix Handte
038aabde28 Mask Off Unused Functions When ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT 2018-12-20 12:15:07 -08:00
Yann Collet
2898afab52 fixed OSSfuzz 11849 
The problem was already masked,
due to no longer accepting tiny blocks for statistics.

But in case it could still happen with not-so-tiny blocks,
there is a stricter control which ensures that
nothing was already loaded prior to statistics collection.
 2018-12-19 16:54:15 -08:00
W. Felix Handte
8e61ac8161 Use Unused Variable in ERR_getErrorString() 2018-12-19 12:36:10 -08:00
Yann Collet
8e0e495ce8 fixed: compression ratio discrepancy 
depending on initialization,
the first byte of a new frame was invalidated or not.

As a consequence, one match opportunity was available or not,
resulting in slightly different compressed sizes
(on average, 1 or 2 bytes once every 20 frames).

It impacted ratio comparison between one-shot and streaming modes.

This fix makes the first byte of a new frame always a valid match.
Now compressed size is always the same.
It also improves compressed size by a negligible amount.
 2018-12-19 10:11:06 -08:00
Yann Collet
d0e15f8d32
Merge pull request #1458 from terrelln/estimate 
[libzstd] Fix estimate with negative levels
 2018-12-18 15:12:21 -08:00
Yann Collet
04baecaeed
Merge pull request #1457 from facebook/btultra2.1 
btultra2 and very small input
 2018-12-18 14:46:55 -08:00
Nick Terrell
d7def456d8 [libzstd] Fix estimate with negative levels 
* Fix `ZSTD_estimateCCtxSize()` with negative levels.
* Fix `ZSTD_estimateCStreamSize()` with negative levels.
* Add a unit test to test for this error.
 2018-12-18 14:24:49 -08:00
Yann Collet
ef984e7307 fix debug levels 
as reported by @terrelln.
2 is reserved for temporary usage only.
 2018-12-18 13:40:07 -08:00
W. Felix Handte
0d606ee3db Fix Incorrect assert() 2018-12-18 13:36:39 -08:00
W. Felix Handte
bd4afc389f Add Logic to Makefile to Convert Make Vars to Defines 2018-12-18 13:36:39 -08:00
W. Felix Handte
ece2c18372 Document Macros in README 2018-12-18 13:36:39 -08:00
W. Felix Handte
c2d51637d9 Add Mutual-Exclusion Error 2018-12-18 13:36:39 -08:00
W. Felix Handte
c560e34c86 Add HUF_FORCE_DECOMPRESS_X2 2018-12-18 13:36:39 -08:00
W. Felix Handte
abd1567d3c Move HUF_DGEN Up Out of X1 Definitions 2018-12-18 13:36:39 -08:00
W. Felix Handte
4a0572b215 Refactor Huffman Decompression Away From Ternary Tree in ZSTD_decodeLiteralsBlock 2018-12-18 13:36:39 -08:00
W. Felix Handte
432314b58a Rename HUF_DECOMPRESS_MINIMAL -> HUF_FORCE_DECOMPRESS_X1 2018-12-18 13:36:39 -08:00
W. Felix Handte
4bbb8a48ad Add ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG 
This macro forces behavior in the opposite direction.
 2018-12-18 13:36:39 -08:00
W. Felix Handte
64553a0e35 Rename ZSTD_DECOMPRESS_MINIMAL -> ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT 2018-12-18 13:36:39 -08:00
W. Felix Handte
605dd576ee Remove Error Strings with ZSTD_STRIP_ERROR_STRINGS 2018-12-18 13:36:39 -08:00
W. Felix Handte
9d5f3963ff Add Option to Not Request Inlining with ZSTD_NO_INLINE 2018-12-18 13:36:39 -08:00
W. Felix Handte
df28e5babd Add ZSTD_DECOMPRESS_MINIMAL Macro, Which Reduces Branching of Decompress Variants 2018-12-18 13:36:39 -08:00
W. Felix Handte
f45c9df42e Totally Hide/Disable X2 Variants when HUF_DECOMPRESS_MINIMAL is Defined 2018-12-18 13:36:39 -08:00
W. Felix Handte
36a84b07a8 Load Dictionaries as X1 Tables 2018-12-18 13:36:39 -08:00
W. Felix Handte
f9cb348776 Add HUF_DECOMPRESS_MINIMAL Macro, Which Avoids Using X2 Variants 2018-12-18 13:36:39 -08:00
Yann Collet
635783da12 btultra2 and very small srcSize 
When srcSize is small,
the nb of symbols produced is likely too small to warrant dedicated probability tables.
In which case, predefined distribution tables will be used instead.

There is a cheap algorithm in btultra initialization :
it presumes default distribution will be used if srcSize <= 1024.

btultra2 now uses the same threshold to shut down probability estimation,
since measured frequencies won't be used at entropy stage,
and therefore relying on them to determine sequence cost is misleading,
resulting in worse compression ratios.

This fixes btultra2 performance issue on very small input.

Note that, a proper way should be
to determine which symbol is going to use predefined probaility
and which symbol is going to use dynamic ones.
But the current algorithm is unable to make a "per-symbol" decision.
So this will require significant modifications.
 2018-12-18 12:32:58 -08:00
Yann Collet
517d8c984c
Merge pull request #1449 from facebook/ovlog_def 
overlapLog default values
 2018-12-18 09:45:53 -08:00
Yann Collet
373ff8b983 play around with rescale weights 2018-12-17 15:48:34 -08:00
Yann Collet
8be145a8c1 fixed default job size 2018-12-13 16:38:08 -08:00
Nick Terrell
75fa3f2eb7
Merge pull request #1446 from terrelln/overflow 
[libzstd] Fix infinite loop in decompression
 2018-12-13 16:21:15 -08:00
Yann Collet
62180b27d5 zstdmt parameter getter/setter use int 2018-12-13 15:47:34 -08:00
Nick Terrell
aaea4ef924 [libzstd] Fix infinite loop in decompression 
When we switched `ZSTD_SKIPPABLEHEADERSIZE` to a macro, the places where we do:

    MEM_readLE32(ptr) + ZSTD_SKIPPABLEHEADERSIZE

can now overflow `(unsigned)-8` to `0` and we infinite loop. We now check
the frame size and reject sizes that overflow a U32.

Note that this bug never made it into a release, and was only in the dev branch
for a few days.

Credit to OSS-Fuzz
 2018-12-13 15:13:19 -08:00
Yann Collet
34f01e600f fixed multiple conversions 
from 64-bit to 32-bit
 2018-12-13 14:02:22 -08:00
Yann Collet
1993f5d412 fixed ovlog tests 
and updated man page
 2018-12-12 21:09:14 -08:00
Yann Collet
f2f86d369b Merge branch 'btultra2' into ovlog_def 2018-12-12 20:58:14 -08:00
Yann Collet
9a92ed401d updated compression results.csv 
and fixed nit
 2018-12-12 20:30:09 -08:00
Yann Collet
9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet
7bb8dfc62f new overlapLog default values 
varies between 6 and 9, depending on strategy
 2018-12-11 18:10:29 -08:00
Yann Collet
eee789b7ea continued: changed to overlapLog 
in deeper code layer.
for consistency.
 2018-12-11 17:41:42 -08:00
Yann Collet
9b784dec7f changed parameter name to ZSTD_c_overlapLog 
from overlapSizeLog.

Reasoning :
`overlapLog` is already used everwhere, in the code, command line and documentation.
`ZSTD_c_overlapSizeLog` feels unnecessarily different.
 2018-12-11 16:55:33 -08:00
Yann Collet
52b94f902c add clarification for ZSTD_CCtx_setPledgedSrcSize() 
as requested in #1391
 2018-12-11 12:08:21 -08:00
Yann Collet
9c3265a53f
Merge pull request #1417 from facebook/advancedAPI 
Advanced API
 2018-12-10 18:48:15 -08:00
Yann Collet
5e6aaa3abb fixed btultra2 usage with prefix 
notably while using multi-threading
 2018-12-10 18:45:03 -08:00
Yann Collet
3619c34399 fix assert position within ZSTD_compress2() 2018-12-10 17:42:35 -08:00
Yann Collet
5a1e01e5f1 clarified experimentalParam 2018-12-10 17:36:20 -08:00
Yann Collet
c226a7b9f3 fixed ZSTD_compress2() 
as suggested by @terrelln
 2018-12-10 17:33:49 -08:00
Yann Collet
37e314a68d updated clevel table for large inputs 2018-12-09 22:38:05 -08:00
Yann Collet
c9c4c7ec8c update clevel table for 256K 2018-12-08 21:40:08 -08:00
Yann Collet
8075d75f9c update clevel table for 128K 2018-12-08 10:42:55 -08:00
Yann Collet
95b152ab33 updated clevel table for 16K 
to introduce btultra2
 2018-12-07 20:12:43 -08:00
Yann Collet
d613fd9afe linked btultra2 as strategy9 
and ensure zstdbench detects out-of-bound parameters
 2018-12-06 19:27:37 -08:00
Yann Collet
34aa401afd updated documentation 
introducing ZSTD_btultra2
 2018-12-06 17:22:19 -08:00
Yann Collet
ae370b0e12 minor bound refinements 2018-12-06 16:51:17 -08:00
Yann Collet
39e28982cf introduced constants ZSTD_STRATEGY_MIN and ZSTD_STRATEGY_MAX 2018-12-06 16:16:16 -08:00
Yann Collet
c3c3488981 fixed c++ assignment to enum 2018-12-06 15:57:55 -08:00
Yann Collet
be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 
also : fixed paramgrill, and limit conditions
 2018-12-06 15:00:52 -08:00
Yann Collet
e9448cdf4c introduced strategy btultra2 
note : not yet applied on any compression level
 2018-12-06 13:38:09 -08:00
Yann Collet
0c404a48f0 moved ZSTD_WINDOWLOG_LIMIT_DEFAULT into static-linking-only area 2018-12-06 10:57:19 -08:00
Yann Collet
96d887429b clarified usage of word "job" 
only applies in MT / async context now.
 2018-12-06 10:14:34 -08:00
Yann Collet
3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_* 
for naming consistency
 2018-12-05 17:26:02 -08:00
Yann Collet
c2053310e5 updated API documentation 2018-12-05 16:23:00 -08:00
Yann Collet
3e042d5cc0 ZSTD_decompressDCtx() is compatible with sticky parameters 2018-12-04 17:30:58 -08:00
Yann Collet
d7da3fc90a merge dedicated dParam setters 2018-12-04 17:06:48 -08:00
Yann Collet
4b5a4f02d7 write the switch()case: differently 
so that it please both
compilers which warn for dead code after the switch
and
compilers which do not detect that all branches terminate.
 2018-12-04 16:59:26 -08:00
Yann Collet
85b02bf142 fixed silent conversion warning 2018-12-04 15:57:16 -08:00
Yann Collet
aec945f0dc implemented ZSTD_dParam_getBounds() 
and ZSTD_DCtx_setParameter()
 2018-12-04 15:35:37 -08:00
Yann Collet
34e146f548 advanced decompression function replaces by normal streaming one 
advanced parameters compatible with ZSTD_decompressStream().
 2018-12-04 10:28:36 -08:00
Yann Collet
7ef7dc561a check availability of --color=never command on grep and egrep 
before applying them.
Fixes #1436
 2018-12-03 15:46:55 -08:00
Yann Collet
6ced8f7c7c joined normal streaming API with advanced one 2018-12-03 14:22:38 -08:00
Yann Collet
da1f3066a3 preparative for ZSTD_DCtx_setParameter() 2018-11-30 15:59:50 -08:00
Yann Collet
d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2() 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
 2018-11-30 11:25:56 -08:00
Mitchell Grenier
a424899637 Fix buck for lib 2018-11-30 13:45:16 +00:00
Yann Collet
d3a0c71259 pushed experimental parameters 
into ZSTD_STATIC_LINKING_ONLY section
 2018-11-21 16:18:55 -08:00
Yann Collet
d4d4e109e9 getParameter fills an int* 
rather than an unsigned*
for consistency
since type of setParameter() changed to int.
 2018-11-21 15:37:26 -08:00
Yann Collet
fea920615c promote ZSTD_findFrameCompressedSize() into staging area 2018-11-21 15:25:50 -08:00
Yann Collet
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet
5d3592398d fixed fall-through 2018-11-20 16:09:33 -08:00
Yann Collet
5c6d4b18ac completed implementation of ZSTD_cParam_getBounds() 
for all parameters
 2018-11-20 16:06:00 -08:00
Yann Collet
2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet
e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet
114bd4346e changed enum type name to ZSTD_ResetDirective 
for naming consistency :
types should start with a capital letter (after prefix)
 2018-11-20 12:00:20 -08:00
Yann Collet
3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Yann Collet
19e5f2a35b removed some constants and _simpleArgs() from staging 
constants that *may* change in the future
will be accessed through functions instead
(to be created).

_simpleArgs() variant do not have (yet) a clear enough added value
to deserve "stable" status.
 2018-11-19 17:38:15 -08:00
Ryan Schmidt
ef4df0df4a Fix i386 build failure "Junk character 13" 2018-11-16 02:16:21 -06:00
Yann Collet
5c68639186 updated ZSTD_DCtx_reset() 
signature and behavior is now the same as ZSTD_CCtx_reset()
 2018-11-15 16:12:39 -08:00
Yann Collet
06c8d5a4f4 Merge branch 'dev' into advancedAPI 
fixed rsyncable
 2018-11-15 10:51:24 -08:00
Nick Terrell
b9693d3a49 [lib] Add rsyncable mode 
- Add rsyncable mode to multithreaded mode
- Factor out LDM's hash function for reuse
 2018-11-14 16:59:57 -08:00
Yann Collet
21a42bf5f9 added advanced decompression api 2018-11-14 16:54:54 -08:00
Yann Collet
cf9f4b63b8 fixed fuzz test src code 2018-11-14 14:46:49 -08:00
Yann Collet
7b0391e37e finalized retrofit of ZSTD_CCtx_reset() 
updated all depending sources
 2018-11-14 13:05:35 -08:00
Yann Collet
ff8d371708 modified ZSTD_CCtx_reset() 
which now accepts an enum,
to distinguish between resetting the session, or the parameters (or both).

removed ZSTD_CCtx_resetParameters(), which is redundant.

start replacing invocation of ZSTD_CCtx_reset*() functions

Updated advanced API documentation

trimmed down amount of API staged in RC,
in particular, all functions related to ZSTD_CCtxParams()
seem too advanced.
 2018-11-14 12:33:57 -08:00
Yann Collet
d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT 
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
 2018-11-13 18:12:34 -08:00
Yann Collet
2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK 
and updated several API comments
 2018-11-13 17:36:35 -08:00
Yann Collet
b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet
768a264200 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-11-13 15:56:36 -08:00
Yann Collet
092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet
f28af025d9
Merge pull request #1413 from felixhandte/attach-dict-fix-unsigned-compare 
Fix #1412: Perform Signed Comparison When Setting Attach Dict Param
 2018-11-12 17:53:11 -08:00
Yann Collet
626040ab53 changed PREFETCH() macro into PREFETCH_L2() 
which is more accurate
 2018-11-12 17:05:32 -08:00
W. Felix Handte
5faef4d378 Const 2018-11-12 14:48:42 -08:00
W. Felix Handte
2d9332eb21 Fix Types 2018-11-12 12:52:31 -08:00
W. Felix Handte
4127de5fa6 Switch Enum to Only Non-Negative Values, Update Comments 2018-11-12 12:47:47 -08:00
W. Felix Handte
596f7d1256 Fix #1412: Perform Signed Comparison When Setting Attach Dict Param 2018-11-12 12:07:57 -08:00
Yann Collet
7b0c551bff
Merge pull request #1411 from facebook/prefetch_dict 
Improves decompression speed when using cold dictionary
 2018-11-09 11:31:35 -08:00
Yann Collet
1b4a9c518b
Merge pull request #1410 from facebook/prefetch_dec 
improve long-range decoder speed
 2018-11-08 18:41:58 -08:00
Yann Collet
483759a3de Improves decompression speed when using cold dictionary 
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).

Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)

The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.

This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).

Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
 2018-11-08 17:00:23 -08:00
Yann Collet
20fb9e7f36 reduced assertion strength 
one limit case can apparently be generated during fuzzer tests
 2018-11-08 12:57:34 -08:00
Yann Collet
9126da5b5c improve long-range decoder speed 
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
 2018-11-08 12:47:46 -08:00
Yann Collet
8bed4012bd fixed decompression-only benchmark 2018-11-08 12:36:39 -08:00
Nick Terrell
a8daa2d683 Signal before unlocking in pool.c 2018-11-08 10:45:53 -08:00
Bartosz Szreder
5c5c476338 Prevent deadlock on malloc() failure. 2018-11-08 10:29:31 +01:00
Yann Collet
e0701d3c5d
Merge pull request #1404 from facebook/T36302429 
fixed T36302471
 2018-11-06 11:53:20 -08:00
Yann Collet
3e5cdf1b6a fixed T36302429 2018-11-05 17:50:30 -08:00
Yann Collet
2caa995558 just add an assert() in ZSTD_insertBtAndGetAllMatches() 
to express a condition on ll0 .
May help static analyzer as in #1397
 2018-11-05 17:13:32 -08:00
Yann Collet
3a90229616
Merge pull request #1395 from facebook/decompressblock 
created zstd_decompress_block module
 2018-10-29 16:28:09 -07:00
Yann Collet
acd75a1448 fixed a second memset() on NULL 
not sure why it only triggers now,
this code has been around for a while.

Introduced a new error code : dstBuffer_null,
I couldn't express anything even remotely similar with existing error codes set.
 2018-10-29 15:03:57 -07:00
Yann Collet
9c58098200 fixed memcpy() on NULL warning 
memcpy(NULL, src, 0) is undefined behavior.
 2018-10-29 13:57:37 -07:00
Yann Collet
ea966c8fb1
Merge pull request #1396 from facebook/huf_refactor 
refactor HUF_compress_internal for clarity
 2018-10-29 13:06:45 -07:00
Yann Collet
fc20b3c441 added flag -Wc++-compat 
for library and cli
 2018-10-26 16:38:23 -07:00
Yann Collet
c3c7deb1e1
Merge pull request #1392 from coetry/dev 
provide consistent spacing to enum field
 2018-10-26 15:29:07 -07:00
Yann Collet
1866bd374a Merge branch 'dev' into huf_Refactor 2018-10-26 15:25:01 -07:00
Yann Collet
8d56f4baee added a few comments for clarifications 2018-10-26 15:21:52 -07:00
Yann Collet
450356b5af Merge branch 'dev' into decompressblock 2018-10-26 15:03:43 -07:00
Yann Collet
7d4960a5e8
Merge pull request #1390 from facebook/nullAsOutput 
support decompressing an empty frame into NULL
 2018-10-26 14:43:16 -07:00
Allen Hai
3783720f70 vertically align code comment 2018-10-26 16:16:06 -05:00
Yann Collet
7b74405150 refactor HUF_compress_internal for clarity 
changed workspace parameter convention
to always provide workspaceSize,
so that size can be explicitly checked.

Also, use more enum to make the meaning of some parameters more explicit.
 2018-10-26 13:21:37 -07:00
Allen Hai
26e34d8a73 provide consistent spacing to enum field 2018-10-25 18:45:20 -05:00
Yann Collet
2b4914082e created zstd_decompress_block module 
isolate all logic associated with block decompression
into its own module.

zstd_decompress is still in charge
of context creation/destruction,
frames, headers, streaming, special blocks, etc.

Compressed blocks themselves are now handled within zstd_decompress_block .
 2018-10-25 16:28:41 -07:00
Yann Collet
cb320a9fc0 added comment on public ddict functions 2018-10-24 16:50:03 -07:00
Yann Collet
806a5c84e4 support decompressing an empty frame into NULL 
fix #1385
decompressing into NULL was an automatic error.
It is now allowed, as long as the content of the frame is empty.

Seems to simplify things for `arrow`.
Maybe some other projects rely on this behavior ?
 2018-10-24 16:34:35 -07:00
Yann Collet
debff3929b fixed warnings in testpools 2018-10-24 10:36:06 -07:00
Yann Collet
cc3612e1c5 added simple guard macros 
in case of accidental multi-includes
 2018-10-23 17:55:23 -07:00
Yann Collet
ccd2d426fc separate DDict logic into its own module 
created zstd_ddict.c within lib/decompress
 2018-10-23 17:25:49 -07:00
Yann Collet
f181799082 fix decodecorpus incorrect frame generation 
fix #1379
decodecorpus was generating one extraneous byte when `nbSeq==0`.
This is disallowed by the specification.

The reference decoder was just skipping the extraneous byte.
It is now stricter, and flag such situation as an error.
 2018-10-20 18:56:21 -07:00
Yann Collet
1e6208e75e bumped version number to v1.3.7 
updated documentation
 2018-10-11 14:40:12 -07:00
Ori Livneh
f31715f5e0 Enable use of bswap intrinsics in clang 
Necessary because clang disguises itself as an older (__GNUC_MINOR__ = 2) GCC.
 2018-10-11 15:01:09 -04:00
Yann Collet
6ed3b526e4 restored bitMask for shift values 
since corrupted bitstreams can generate too large values.

This slightly reduces the benefits from clang on my laptop.
gcc results and code generation are not affected.
 2018-10-10 18:29:50 -07:00
Yann Collet
c012e9540a removed one assert() 
that can be triggered by a corrupted bitstream.
 2018-10-10 17:33:04 -07:00
Yann Collet
7791f192ee removed one assert() 
which can be triggered when input is corrupted.
 2018-10-10 16:39:15 -07:00
Yann Collet
d3ec23313d improved decompression speed 
while reviewing #1364,
I found a decompression speed improvement.

On my laptop, the new code decompresses +5-6% faster on clang
and +2-3% faster on gcc.

not bad for an accidental optimization...
 2018-10-10 15:48:43 -07:00
Yann Collet
942df522cc
Merge pull request #1361 from facebook/streamdoc 
Clarify streaming api doc
 2018-10-08 19:19:34 -07:00
W. Felix Handte
b8235be865 Avoid Searching Dictionary in ZSTD_btlazy2 When an Optimal Match is Found 
Bailing here is important to avoid reading past the end of the input buffer.
 2018-10-08 15:59:32 -07:00
W. Felix Handte
d121b3451c Clean Up Debug Log Statements 2018-10-08 15:59:32 -07:00
W. Felix Handte
08da9ad316 Remove Unused Variable 2018-10-08 15:59:32 -07:00
Yann Collet
8fc79fac07 clarify streaming api doc 
as suggested by @indygreg in #1360
 2018-10-08 15:53:29 -07:00
Yann Collet
11cd2ea43d finalized minor warnings on Haiku 2018-10-03 16:37:50 -07:00
Yann Collet
bc93b801f0
Merge pull request #1330 from korli/haiku 
Enable building zstd on Haiku.
 2018-10-03 13:36:00 -07:00
Jerome Duval
87c10e2f58 Enable building zstd on Haiku. 2018-10-03 09:51:56 +02:00
Yann Collet
22ddf3523a fixed msan warning 
on btlazy2 strategy with dictAttach
 2018-10-02 18:20:20 -07:00
Yann Collet
c9843ec232
Merge pull request #1348 from facebook/donotdelete 
Fix #1082
 2018-10-02 16:37:58 -07:00
Yann Collet
3ca6261223 fixed static analyzer warnings 
note : for some reason,
scan-build version on my laptop found problems within fastcover.c
that scan-build on travisCI does not flag.

They are, as usual, false positive :
the analyzer does not understand that a table (`offset`) is correctly filled before usage.
 2018-10-02 15:59:11 -07:00
Yann Collet
228c6e5147
Merge pull request #1317 from felixhandte/split-logs 
Independent Dictionary and Working Context Table Logs
 2018-10-01 17:20:12 -07:00
W. Felix Handte
5b296869df Revert Ability to Set HashLog and ChainLog on Context When Dict is Attached 
This capability is not needed / used in the current unit of work. I'll
re-introduce it later, when we start allowing users to override the deduced
working context logs.
 2018-10-01 13:28:13 -07:00
W. Felix Handte
c2369fedc4 Restore Passing CParams to ZSTD_insertAndFindFirstIndex_internal 2018-09-28 17:12:54 -07:00
W. Felix Handte
bad74c4781 Use Working Ctx Logs when not in DMS Mode 
We pre-hash the ptr for the dict match state sometimes. When that actually
happens, a hashlog of 0 can produce undefined behavior (right shift a long
long by 64). Only applies to unoptimized compilations, since when
optimizations are applied, those hash operations are dropped when we're not
actually in dms mode.
 2018-09-28 17:12:54 -07:00
W. Felix Handte
c38acff94f When Attaching Dictionary, Size Working Tables Based on Input Size Only 2018-09-28 17:12:54 -07:00
W. Felix Handte
9d87d50878 Remove Log Overriding for the Time Being 2018-09-28 17:12:54 -07:00
W. Felix Handte
77fd17d93f Remove Strategy-Dependency in Making Attachment Decision 2018-09-28 17:12:54 -07:00
W. Felix Handte
00c088b32d Support Split Logs in ZSTD_btopt..ZSTD_btultra 2018-09-28 17:12:54 -07:00
W. Felix Handte
0783492178 Bump Split Log Support to ZSTD_btultra 2018-09-28 17:12:54 -07:00
W. Felix Handte
e4ac4a0f16 Support Split Logs in ZSTD_greedy..ZSTD_btlazy2 2018-09-28 17:12:54 -07:00
W. Felix Handte
e710dc3369 Bump Split Log Support to ZSTD_btlazy2 2018-09-28 17:12:54 -07:00
W. Felix Handte
22fcb8d4c7 Support Split Logs in ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
a232b3bb7c Bump Split Log Support to ZSTD_dfast 2018-09-28 17:12:54 -07:00
W. Felix Handte
fe96e98f81 Support a Separate Hash Log in ZSTD_fast 2018-09-28 17:12:54 -07:00
W. Felix Handte
bc880ebe8f Stop Passing in hashLog and stepSize to ZSTD_compressBlock_fast_generic 2018-09-28 17:12:54 -07:00
W. Felix Handte
b3107c7799 Temporary Commit to Retain Requested Hash and Chain Logs During Dict Attach 2018-09-28 17:12:54 -07:00
W. Felix Handte
34e0193129 Allow Setting Hash and Chain Logs on Contexts with Attached CDict 2018-09-28 17:12:54 -07:00
W. Felix Handte
eae8232f50 For Supported Strategies, Attach Dict Even When Params Don't Match 2018-09-28 17:12:54 -07:00
W. Felix Handte
01ff945eae Split Attach and Copy Reset Strategies into Separate Implementation Functions 2018-09-28 17:12:54 -07:00
W. Felix Handte
a6d6bbeae1 Pull Attachment Decision into Separate Function 2018-09-28 17:12:54 -07:00
W. Felix Handte
b7fba599ae And Then Avoid the Unused Parameter Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte
1f188ae655 Move Asserts into Function to Avoid Unused Function Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte
7212b5e5c2 Move Match State CParams Setting into resetCCtx and continueCCtx 2018-09-28 17:12:54 -07:00
W. Felix Handte
01e34d365b Strengthen Assertion to Assert Equality 2018-09-28 17:12:53 -07:00
W. Felix Handte
50cc1cf4d5 Remove CParams Arg from ZSTD_ldm_blockCompress 2018-09-28 17:12:53 -07:00
W. Felix Handte
14764de49f Stop Separately Passing CParams in ZSTD_lazy Internal Functions 2018-09-28 17:12:53 -07:00
W. Felix Handte
97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
W. Felix Handte
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
W. Felix Handte
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
03103269de Assert ctx and ms cparams Equivalency 2018-09-28 17:10:42 -07:00
W. Felix Handte
4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
W. Felix Handte
76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
Nick Terrell
6391cd1030 [zstd] Fix newly added test case 2018-09-28 12:09:28 -07:00
Yann Collet
73773c6b6a fixed legacy compilation tests 
for some reason, these tests started failing recently on CircleCI
 2018-09-27 18:15:14 -07:00
Nick Terrell
a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
Nick Terrell
aec1a3ec58 Change byte to value to avoid a GRUB typedef 2018-09-27 15:24:48 -07:00
Nick Terrell
109bd37474 Include stddef.h for size_t 2018-09-27 15:24:48 -07:00
Nick Terrell
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Yann Collet
2a5cd8535a
Merge pull request #1342 from facebook/fixcatyd 
fix : huge (>4GB) chain of blocks
 2018-09-27 10:20:14 -07:00
Yann Collet
404a7bfed0 moved again overflow correction 
cannot work from within ZSTD_compressBlock()
 2018-09-26 18:06:53 -07:00
Yann Collet
0e2dbac18a changed overflow correction place 
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
 2018-09-26 15:35:38 -07:00
Yann Collet
e74eade251
Merge pull request #1339 from facebook/grep_colors 
fixed usage of grep in Makefile
 2018-09-26 14:39:20 -07:00
Yann Collet
8883af6a1e
Merge pull request #1327 from facebook/adapt 
Adaptive compression
 2018-09-26 14:39:08 -07:00
Yann Collet
f98c69d77c fix : huge (>4GB) stream of blocks 
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
 2018-09-26 14:24:28 -07:00
Yann Collet
8ff17a6a09
Merge pull request #1329 from facebook/v04isout 
Changed default legacy support to v0.5+
 2018-09-26 13:39:05 -07:00
Yann Collet
08f68d83c5 fixed usage of grep in Makefile 
when terminal uses colors
as suggested by @danielshir (#1294)
 2018-09-25 16:56:53 -07:00
Yann Collet
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet
9bb6c15f79
Merge pull request #1332 from facebook/minclevel 
defined a minimum negative level
 2018-09-24 16:01:13 -07:00
Yann Collet
292d8e4a83 added some tests based on limits.h 
in order to ensure proper type mapping
when not using stdint.h
 2018-09-23 23:57:30 -07:00
Yann Collet
71a5210617 avoid recompiling dll every time under mingw 2018-09-21 17:40:30 -07:00
Yann Collet
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet
bfff4f4809 ensure all writes to job->cSize are mutex protected 
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
 2018-09-21 16:00:39 -07:00
Yann Collet
32b7cf1bcf fixed tautological tests 
involving ZSTD_TARGETLENGTH_MIN (== 0)
 2018-09-21 15:04:43 -07:00
Yann Collet
c044345f8f Merge branch 'mingw' into minclevel 2018-09-21 14:56:57 -07:00
Yann Collet
de6c75e4e5
Merge pull request #1318 from felixhandte/shadow-dict-matches 
Don't Search Dictionary Context When Working Context Search Resulted in Mismatch
 2018-09-21 12:15:33 -07:00
Yann Collet
a54c86cfc6 defined a minimum negative level 
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
 2018-09-20 16:52:03 -07:00
Yann Collet
b2939163e1 Changed default legacy support to v0.5+ 
thus dropping read support for v0.4.

It's always possible to re-enable it, by changing build macro ZSTD_LEGACY_SUPPORT to 4.
 2018-09-20 14:30:20 -07:00
Yann Collet
7992942d66 fixed complex tsan issue 
when job->consumed == job->src.size , compression job is presumed completed,
so it must be the very last action done in worker thread.
 2018-09-20 13:47:31 -07:00
Yann Collet
6b07a66aec fixed minor reporting discrepancy in MT mode 2018-09-19 16:30:55 -07:00
Yann Collet
ca02ebee07 removed static variables 
so that --adapt can work on multiple input files too
 2018-09-19 15:25:50 -07:00
Yann Collet
89bc309d90 error out when --adapt is associated with --single-thread 
since they are not compatible
 2018-09-19 14:49:13 -07:00
Yann Collet
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Yann Collet
005f000aed updated documentation of *refPrefix() 
indicating the equivalence with `diff` operation.
 2018-09-18 13:07:08 -07:00
ko-zu
18b4a1da61 Fix clang build 
Fix dixygen comment
Fix clang binary path
 2018-09-16 10:27:02 +09:00
Yann Collet
7269fe6cd3 minor code comment update 2018-09-14 16:06:35 -07:00
Yann Collet
0403148315
Merge pull request #1295 from felixhandte/hdr-intro-comment-negative-lvls 
Proposed Update to Zstd.h Introduction Comment
 2018-09-14 15:29:19 -07:00
W. Felix Handte
b76c888497 ZSTD_dfast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:24:25 -07:00
W. Felix Handte
b048af5999 ZSTD_fast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:23:35 -07:00
Yann Collet
0e5b447aaa
Merge pull request #1316 from facebook/coldDict 
Cold dictionary mitigation
 2018-09-14 10:37:46 -07:00
Yann Collet
5512400677 updated code comments, based on @terrelln review 2018-09-13 16:44:04 -07:00
Yann Collet
d195eec97e fixed msan error 
cold dictionary is detected through a comparison with dictEnd,
which was not initialized at the beginning of first DCtx usage.
 2018-09-13 12:29:52 -07:00
Yann Collet
674dd21bd0 final parameter tuning 2018-09-12 17:25:34 -07:00
Yann Collet
419dfd4ea3 clean traces 2018-09-12 16:40:28 -07:00