Commit Graph

843 Commits

Author SHA1 Message Date
Dave Watson
28b824921d decompress_generic: re-add fastpath
This is the remaineder of the original 'shortcut'.  If true, we can avoid
the loop in LZ4_wildCopy, and directly copy instead.
2019-02-08 13:57:23 -08:00
Dave Watson
232f1e261f decompress_generic: drop partial copy check in fast loop
We've already checked that we are more than FASTLOOP_SAFE_DISTANCE
away from the end, so this branch can never be true, we will have
already jumped to the second decode loop.
2019-02-08 13:57:23 -08:00
Dave Watson
59332a3026 decompress_generic: Optimize literal copies
Use LZ4_wildCopy16 for variable-length literals.  For literal counts that
fit in the flag byte, copy directly.  We can also omit oend checks for
roughly the same reason as the previous shortcut:  We check once that both
match length and literal length fit in FASTLOOP_SAFE_DISTANCE, including
wildcopy distance.
2019-02-08 13:57:23 -08:00
Dave Watson
5dfa7d422b decompress_generic: optimize match copy
Add an LZ4_wildCopy16, that will wildcopy, potentially smashing up
to 16 bytes, and use it for match copy.  On x64, this avoids many
blocked loads due to store forwarding, similar to issue #411.
2019-02-08 13:57:23 -08:00
Dave Watson
28356e02ad decompress_generic: Add a loop fastpath
Copy the main loop, and change checks such that op is always less
than oend-SAFE_DISTANCE.  Currently these are added for the literal
copy length check, and for the match copy length check.

Otherwise the first loop is exactly the same as the second.  Follow on
diffs will optimize the first copy loop based on this new requirement.

I also tried instead making a separate inlineable function for the copy
loop (similar to existing partialDecode flags, etc), but I think the
changes might be significant enough to warrent doubling the code, instead
pulling out common functionality to separate functions.

This is the basic transformation that will allow several following optimisations.
2019-02-08 13:57:19 -08:00
Dave Watson
4da336062e decompress_generic: Refactor variable length fields
Make a helper function to read variable lengths for literals and
match length.
2019-02-08 13:42:42 -08:00
Jeremy Maitin-Shepard
26e7635a0e Eliminate optimize attribute warning with clang on PPC64LE 2019-02-04 12:22:56 -08:00
Tim Zakian
81441e2462 Make fact that certain variables that are passed into LZ4HC_encodeSequence are changed by the function call 2019-01-09 13:42:12 -08:00
Yann Collet
c750cbe5c1
Merge pull request #631 from qiuyangs/dev
lz4hc.c: change (length >> 8) to (length / 255)
2019-01-09 12:21:39 -08:00
Tim Zakian
8193742251 Make LZ4F_getBlockSize public and publis in experimental section 2019-01-09 10:49:49 -08:00
qiuyangs
660d21272e
lz4hc.c: change (length >> 8) to (length / 255)
Every 0xff byte in the compressed block corresponds to a length of 255 (not 256) in the input data. For long repeating sequences, using (length >> 8) may generate bad compressed blocks.
2019-01-06 16:29:30 +08:00
W. Felix Handte
4e3accccb2 Fix Dict Size Test in LZ4_compress_fast_continue()
Dictionaries don't need to be > 4 bytes, they need to be >= 4 bytes. This test
was overly conservative.

Also removes the test in `LZ4_attach_dictionary()`.
2018-12-05 11:24:33 -08:00
W. Felix Handte
535636ff5c Don't Attach Very Small Dictionaries
Fixes a mismatch in behavior between loading into the context (via
`LZ4_loadDict()`) a very small (<= 4 bytes) non-contiguous dictionary, versus
attaching it with `LZ4_attach_dictionary()`.

Before this patch, this divergence could be reproduced by running

```
make -C tests fuzzer MOREFLAGS="-m32"
tests/fuzzer -v -s1239 -t3146
```

Making sure these two paths behave exactly identically is an easy way to test
the correctness of the attach path, so it's desirable that this remain an
unpolluted, high signal test.
2018-12-04 14:05:11 -08:00
Vincent Torri
9021648ba7 Merge remote-tracking branch 'upstream/dev' into dev 2018-12-02 19:42:38 +01:00
Vincent Torri
b03714dc80 Windows: create import library with gcc directly, remove now useless def file 2018-12-02 18:50:18 +01:00
Yann Collet
6689dae33b
Merge pull request #610 from antinucleon/bootcamp
[amalgamation] lz4frame.c
2018-11-26 12:14:27 -08:00
Bing Xu
b192c86ba4 [amalgamation] lz4frame.c 2018-11-26 11:30:15 -08:00
Vincent Torri
bd2b259760 Uninstall DLL and import lib 2018-11-21 09:07:26 +01:00
Vincent Torri
8e8b658bde Add explanation for the installation of the DLL in the bin directory 2018-11-21 08:40:51 +01:00
Vincent Torri
0314027051 Allow installation of lz4 for Windows (MSYS2 or when cross-compiling) 2018-11-20 21:08:23 +01:00
Vincent Torri
e057e94215 update lib/README.md 2018-11-18 21:03:27 +01:00
Vincent Torri
d966844a95 Add documentation about DLLTOOL variable 2018-11-18 13:22:10 +01:00
Vincent Torri
b5e106220b Add DLLTOOL variable so that one can override dlltool binary
This fix cross compilation on linux for Windows
2018-11-17 14:17:15 +01:00
Vincent Torri
31ce8b56e5 Use / instead of \ when accessing files in the dll subdirectory.
This allow cross-compilation for Windows on Linux
2018-11-17 07:44:23 +01:00
Bing Xu
17f5071e72 Enable amalgamation of lz4hc.c and lz4.c 2018-11-15 22:24:25 -08:00
Yann Collet
1b819bfd63
Merge pull request #593 from felixhandte/lz4hc-publish-static
Extend Macro to Allow Publishing Experimental LZ4HC Functions in Dynamic Libraries
2018-10-16 15:33:58 -07:00
W. Felix Handte
6a2da13cb7 Make LZ4HC Experimental Functions Dynamically Publishable 2018-10-15 17:23:06 -07:00
W. Felix Handte
45dc195f84 Change Comment and Make LZ4LIB_STATIC_API Available to LZ4HC 2018-10-15 17:22:37 -07:00
Yann Collet
bf9bf80f8d updated code documentation
to clarify #589
2018-10-15 11:14:30 -07:00
Oleg Khabinov
f27ea0774e Adding information about dirty context for _HC_ family of functions 2018-10-10 10:33:04 -07:00
Yann Collet
df6d00ede5
Merge pull request #592 from lz4/compressEnd
fix LZ4F_compressEnd()
2018-10-09 15:37:55 -07:00
Yann Collet
6902fa4892 fixed #589
following recommendations by @raggi.

The fix is slightly different, but achieves the same goal,
and is backed by a test tool which proves that it works
(generates the error before the patch, no longer after the patch).
2018-10-09 14:37:51 -07:00
Yann Collet
e07a37d712 added a test for LZ4F_compressEnd()
which actively tries to make it write out of bound.
For this scenario to be possible,
it's necessary to set dstCapacity < LZ4F_compressBound()

When a compression operation fails,
the CCtx context is left in an undefined state,
therefore compression cannot resume.
As a consequence :
- round trip tests must be aborted, since there is nothing valid to decompress
- most users avoid this situation, by ensuring that dstCapacity >= LZ4F_compressBound()

For these reasons, this use case was poorly tested up to now.
2018-10-09 14:25:18 -07:00
Oleg Khabinov
28eb88d988 Some followups and renamings 2018-10-01 15:19:45 -07:00
Yann Collet
b18b6e53e1
Merge pull request #587 from lz4/hintbug
fixed improper hint
2018-09-28 16:58:01 -07:00
Yann Collet
21120549a7 fixed improper hint
when LZ4F_decompress() decodes an uncompressed block,
it provides an incorrect hint for next block
when frame checksum is enabled and block checksum is not.

Impact is low : the hint is just an hint,
the decoder works whatever the amount of input provided.

But the assumption that each call to LZ4F_decompress()
would generate just one complete block if input size hint was respected
was broken by this error.
2018-09-28 14:57:50 -07:00
Oleg Khabinov
f2ae385c2f Rename initCheck to dirtyContext and use it in LZ4_resetStream_fast() to check if full reset is needed. 2018-09-28 14:55:05 -07:00
Yann Collet
cb917827f9
Merge pull request #578 from lz4/support128bit
Support for 128bit pointers like AS400
2018-09-26 13:57:09 -07:00
Yann Collet
c4c19c74b8 changed LZ4_streamDecode member order
to reduce memory usage on 128-bits systems
2018-09-25 14:43:19 -07:00
Yann Collet
b2215f2a89 tried to clean another bunch of cppcheck warnings
so "funny" thing with cppcheck
is that no 2 versions give the same list of warnings.

On Mac, I'm using v1.81, which had all warnings fixed.
On Travis CI, it's v1.61, and it complains about a dozen more/different things.
On Linux, it's v1.72, and it finds a completely different list of a half dozen warnings.

Some of these seems to be bugs/limitations in cppcheck itself.
The TravisCI version v1.61 seems unable to understand %zu correctly, and seems to assume it means %u.
2018-09-19 12:12:49 -07:00
Yann Collet
8bea19d57c fixed minor cppcheck warnings in lib 2018-09-18 15:51:26 -07:00
Yann Collet
c3397520a1 updated xxhash to latest version 2018-09-18 12:14:26 -07:00
Yann Collet
6381d828fd increase size of LZ4 contexts for 128-bit systems 2018-09-17 17:31:57 -07:00
Yann Collet
7b9edd60a0 Merge branch 'dev' into support128bit 2018-09-17 17:07:00 -07:00
Yann Collet
dea044a485
Merge pull request #575 from lz4/apiDoc
unpublish static-only functions
2018-09-17 17:06:34 -07:00
Yann Collet
cb2fb479ef increase lz4hc context size as constant for 128-bit systems 2018-09-17 17:05:17 -07:00
Yann Collet
6103b4c9b4 use byU32 mode for any pointer > 32-bit
including 128-bit, like IBM AS-400
2018-09-14 15:27:48 -07:00
Yann Collet
c820480e12
Merge pull request #574 from lz4/enumComma
avoid final trailing comma for enum lists
2018-09-14 13:51:43 -07:00
Yann Collet
d2f5716c5d
Merge pull request #573 from laffer1/laffer1-pkgconfig-mnbsd
FIx pkgconfig file installation on MidnightBSD
2018-09-14 10:11:49 -07:00
Yann Collet
e8b08f9fbd updated build doc 2018-09-13 16:14:00 -07:00
Yann Collet
a803230f67 unpublish static-only function
these functions are now unpublished in dll by default.
One needs to opt-in, using macro LZ4_PUBLISH_STATIC_FUNCTIONS.

used this opportunity to update a bunch of api comments in lz4.h
2018-09-13 16:02:11 -07:00
Yann Collet
86023f01f2 avoid final trailing comma for enum lists
as detected in #485 by @JoachimSchneider.

Refactored the c_standards tests
so that these issues get automatically detected in CI tests.
2018-09-13 14:29:41 -07:00
Lucas Holt
de4667011a
FIx pkgconfig file installation on MidnightBSD 2018-09-11 21:18:43 -04:00
Yann Collet
6d32240b2e clarify constant MFLIMIT
and separate it from MATCH_SAFEGUARD_DISTANCE.

While both constants have same value,
they do not seve same purpose, hence should not be confused.
2018-09-11 10:00:13 -07:00
Yann Collet
b87a8e9e62 fixed minor warning in fuzzer.c
added a few more comments and assert()
2018-09-10 16:48:41 -07:00
Yann Collet
63fc6fbf7e restored nullifying output
to counter possible (offset==0)
2018-09-10 16:22:16 -07:00
Yann Collet
32272f9866 removed temporary debug traces 2018-09-10 15:51:53 -07:00
Yann Collet
d28389b2dc Merge branch 'dev' into partialDecode 2018-09-10 15:44:40 -07:00
Yann Collet
6e54d8df33
Merge pull request #569 from lz4/circle2
Migrate CircleCI tests from 1.0 towards 2.0
2018-09-10 14:44:15 -07:00
Yann Collet
f8519d454e
Merge pull request #565 from lz4/lz4f_init
Introduced constants LZ4F_INIT_*
2018-09-10 13:11:56 -07:00
Yann Collet
674eac3325 Merge branch 'dev' of github.com:Cyan4973/lz4 into dev 2018-09-10 12:02:42 -07:00
Lucas Holt
3318d573ba Add support for MidnightBSD 2018-09-08 14:46:54 -04:00
Yann Collet
e22bb80074 fixed fuzzer test
and removed one blind copy, since there is no more guarantee that at least 4 bytes are still available in output buffer
2018-09-07 18:22:01 -07:00
Yann Collet
eaed9ea4a1 updated function interface documentation 2018-09-07 16:21:31 -07:00
Yann Collet
bf614d3c51 first sketch for a byte-accurate partial decoder 2018-09-07 15:44:19 -07:00
Yann Collet
e32766cc34 updated API documentation 2018-09-07 11:30:15 -07:00
Yann Collet
0f08c22c31
Merge pull request #563 from lz4/docDict
updated documentation for dictionary compression
2018-09-06 12:43:29 -07:00
Yann Collet
26c42d7ad1 added comments on version numbers 2018-09-05 18:08:51 -07:00
Yann Collet
b2e56d82bf Introduced constants LZ4F_INIT_*
to simplify initialization of lz4frame.h structures.
Partially in response to #546.
2018-09-05 16:06:37 -07:00
Yann Collet
0fea528e3a updated documentation regarding dictionary compression
following suggestion from @stbrumme (#558)

Also : bumped version number, regenerated man page and html doc
2018-09-05 14:05:08 -07:00
Yann Collet
30f6f34328 removed one assert() condition
which is not correct when using LZ4_HC with dictionary and starting from a low address (<0x10000).
2018-09-05 11:25:10 -07:00
Yann Collet
2e4847c2d5 fixed #560
it was a fairly complex scenario,
involving source files > 64K
and some extraordinary conditions related to specific layout of ranges of zeroes.
and only on level 9.
2018-09-04 18:21:40 -07:00
Jack Luo
2e52f03a12 fixed spelling mistake in lz4.h 2018-07-28 22:21:57 -04:00
Yann Collet
e95781dc2a
Merge pull request #547 from jennifermliu/dev
Add --fast command to cli
2018-07-17 15:42:48 +02:00
Nick Terrell
4eca78b5c3 Fix LZ4_compress_fast_continue() docs
Fixes #549.
2018-07-10 11:44:03 -07:00
Jennifer Liu
e778db373b Fixed bugs about incorrect acceleration calculation and benchmarking negative compresion level 2018-06-27 13:36:38 -07:00
Yann Collet
ef4f1e3047
Merge pull request #542 from wbx-github/dev
allow to override uname when cross-compiling
2018-05-29 14:20:48 -07:00
Waldemar Brodkorb
6a7af839b8 allow to override uname when cross-compiling
When cross-compiling for example from Darwin to Linux it might be
useful to override uname output to force Linux and create Linux
libraries instead of Darwin libraries.
2018-05-22 20:38:28 +02:00
W. Felix Handte
b8211544ef Also Fix Appveyor Cast Warning 2018-05-22 11:44:15 -04:00
W. Felix Handte
4248a9bfc0 Add extern "C" Guards Around Experimental HC Declarations 2018-05-21 22:30:10 -04:00
W. Felix Handte
91888f472d Remove #define-rename of LZ4_decompress_safe_forceExtDict 2018-05-21 22:29:40 -04:00
W. Felix Handte
c746a27e91 Test Linking C-Compiled Library and C++-Compiled Tests 2018-05-21 22:29:20 -04:00
fbrosson
2149b1a8f6 Add Haiku as a validated target.
lz4 1.8.2 works fine on Haiku and passes all tests.
2018-05-17 16:52:53 +00:00
Yann Collet
bf6fd938e5
Merge pull request #537 from lz4/xpHCmf2
Speed optimization for optimal parser
2018-05-07 13:06:43 -07:00
Yann Collet
ba1c7148a5 renamed variable for clarity 2018-05-07 12:14:26 -07:00
Yann Collet
abb1f70e17
Merge pull request #538 from lz4/frameTestError
Fix frametest error
2018-05-07 11:33:53 -07:00
Yann Collet
200b2960d5 fixed minor conversion warning 2018-05-06 18:26:14 -07:00
Yann Collet
24b9c485db small PA optimization
which measurably improves speed
on levels 9+
2018-05-06 16:53:33 -07:00
Yann Collet
d7b6c726ed small extDict : fixed side-effect
don't fix dictionaries of size 0.
setting dictEnd == source triggers prefix mode,
thus removing possibility to use CDict.
2018-05-05 19:59:00 -07:00
Yann Collet
af12733467 fixed frametest error
The error can be reproduced using following command :
./frametest -v -i100000000 -s1659 -t31096808

It's actually a bug in the stream LZ4 API,
when starting a new stream
and providing a first chunk to complete with size < MINMATCH.
In which case, the chunk becomes a dictionary.
No hash was generated and stored,
but the chunk is accessible as default position 0 points to dictStart,
and position 0 is still within MAX_DISTANCE.
Then, next attempt to read 32-bits from position 0 fails.

The issue would have been mitigated by starting from index 64 KB,
effectively eliminating position 0 as too far away.

The proper fix is to eliminate such "dictionary" as too small.
Which is what this patch does.
2018-05-05 18:24:11 -07:00
Yann Collet
cdb0275b7f lz4hc: fixed PA / SC parameter order
also :
reserved PA for levels 9+ (instead of 8+).
In most cases, speed is lower, and compression benefit is not worth.
2018-05-05 14:32:57 -07:00
Yann Collet
a4e918d7a6 lz4hc: SC only enabled for opt parser
the trade off is not good for regular HC parser :
compression is a little bit better, but speed cost is too large in comparison.
2018-05-05 14:25:37 -07:00
Yann Collet
d097bf93f8 fixed SC.opt integration with regular HC parser
Only enabled when searching forward.

note : it slighly improves compression ratio,
but measurably decreases speed.
Trade-off to analyse.
2018-05-05 13:46:45 -07:00
Yann Collet
fa89a9e18b lz4hc: fixed performance issue
when combining both PA and CS optimizations
2018-05-05 13:31:03 -07:00
Yann Collet
9699ba5ddf integrated chain swapper into HC match finder
slower than expected
Pattern analyzer and Chain Swapper
work slower when both activated.
Reasons unclear.
2018-05-04 19:13:33 -07:00
Nick Terrell
a7cc0b590a Fix make install
* Uninstall didn't remove the pkg-config correctly.
* Fix `mandir`
* Allow overriding either upper- or lower-case location variables, but
  always use the lower case variables.
* Add test case that ensures overriding both upper- and lower-case
  variables is the same, and that the directory is empty after uninstall.
2018-05-04 13:50:23 -07:00
Yann Collet
434ace7244 implemented search accelerator
greatly improves speed compared to non-accelerated,
especially for slower files.

On my laptop, -b12 :
```
calgary.tar :  4.3 MB/s =>  9.0 MB/s
enwik7      : 10.2 MB/s => 13.3 MB/s
silesia.tar :  4.0 MB/s =>  8.7 MB/s
```

Note : this is the simplified version,
without handling dictionaries, external buffer, nor pattern analyzer.
Current `dev` branch on these samples gives :
```
calgary.tar :  4.2 MB/s
enwik7      :  9.7 MB/s
silesia.tar :  3.5 MB/s
```

interestingly, it's slower,
presumably due to handling of dictionaries.
2018-05-03 16:31:41 -07:00
Yann Collet
dc42707107 created LZ4HC_FindLongestMatch()
simplified match finder
only searching forward and within current buffer,
for easier testing of optimizations.
2018-05-03 15:38:32 -07:00
Yann Collet
f3e84ffd41
Merge pull request #529 from felixhandte/lz4f-fast-reset-for-streaming-only
LZ4F: Only Reset the LZ4_stream_t when Init'ing a Streaming Block
2018-05-03 15:37:51 -07:00
Yann Collet
ffbff1f360 Merge branch 'dev' into lz4fRingBuffer 2018-05-03 11:54:57 -07:00
Yann Collet
95607a749b
Merge pull request #528 from lz4/complexShortcut
Faster decoding speed
2018-05-03 11:35:50 -07:00
Cyan4973
2e2c9f6ff3 fix comments / indentation
as requested by @terrelln
2018-05-03 07:56:33 -07:00
W. Felix Handte
5406c2e479 Only Reset the LZ4 Stream when Init'ing a Streaming Block 2018-05-03 00:03:20 -04:00
Yann Collet
c25eb16666 random lz4f clarifications
the initial intention was to update lz4f ring buffer strategy,
but lz4f doesn't use ring buffer.
Instead, it uses the destination buffer as much as possible,
and merely copies just what's required to preserve history
into its own buffer, at the end.
Pretty efficient.

This patch just clarifies a few comments and add some assert().
It's built on top of #528.

It also updates doc.
2018-05-02 16:05:42 -07:00
Yann Collet
858d12e3e1 Merge branch 'dev' into lz4fRingBuffer 2018-05-02 14:24:24 -07:00
Yann Collet
85be6b8f6d increased nbAttempts for lz4 -12
shaves one more kilobyte from silesia.tar
2018-05-02 14:22:35 -07:00
Yann Collet
93cf628a08 introduce LZ4_decoderRingBufferSize()
fuzzer : fix and robustify ring buffer tests
2018-05-02 13:01:04 -07:00
Yann Collet
1a191b3f8d simplify shortcut 2018-05-02 10:33:12 -07:00
Yann Collet
0114b63b40 Merge branch 'dev' into complexShortcut 2018-05-02 10:08:30 -07:00
Yann Collet
bd470ccd38
Merge pull request #521 from lz4/BD_deterministic
fix lz4hc -BD non-determinism
2018-04-30 20:40:34 -07:00
Cyan4973
6a7d501fed renamed variable for clarity
lowLimit -> lowestMatchIndex
2018-04-30 18:56:16 -07:00
Yann Collet
8c574990a9 lz4hc changed variable
to reduce confusion
dictLowLimit => dictStart
2018-04-30 16:08:16 -07:00
Yann Collet
4c696613a0 clarified streaming decompression function
restrictions for ring buffer
2018-04-30 15:55:33 -07:00
Yann Collet
90374271c2
Merge pull request #527 from svpv/fastDec
lz4.c: two-stage shortcut for LZ4_decompress_generic
2018-04-30 15:32:37 -07:00
Yann Collet
c32e0319a5
Merge pull request #523 from svpv/makeV1
lib/Makefile: show commands with V=1
2018-04-29 08:56:44 -07:00
Yann Collet
41ad238bf9
Merge pull request #515 from svpv/refactorDec
lz4.c: refactor the decoding routines
2018-04-29 07:41:35 -07:00
Alexey Tourbin
45f8603aae lz4.c: two-stage shortcut for LZ4_decompress_generic 2018-04-28 11:16:57 +03:00
Alexey Tourbin
69242a8a08 lib/Makefile: show commands with V=1
`make V=1` will now show the commands executed to build the library.
A similar technique is used in e.g. linux/Makefile.

The bulk of this change is produced with the following vim command:

    :g!/^\t@echo\>/s/^\t@/\t\$(Q)/
2018-04-28 07:22:26 +03:00
Yann Collet
1e6ca25af3
Merge pull request #520 from felixhandte/frame-dict-nits
Minor Fixes to Dictionary Preparation in LZ4 Frame
2018-04-27 13:52:30 -07:00
Yann Collet
de7b274d99 Merge branch 'dev' into BD_deterministic 2018-04-27 12:59:20 -07:00
Yann Collet
19b1267d44 fix lz4hc -BD non-determinism
related to chain table update
2018-04-27 12:46:49 -07:00
Yann Collet
72e99c8939 lz4hc : minor editions for clarity 2018-04-27 12:28:58 -07:00
Yann Collet
47d70e755e
Merge pull request #519 from lz4/fdParser
Faster decoding speed
2018-04-27 11:46:29 -07:00
W. Felix Handte
fefc40fc0a Avoid Possibly Redundant Table Clears When Loading HC Dict 2018-04-27 14:10:27 -04:00
W. Felix Handte
5076aa3e35 Remove Redundant LZ4_resetStream() Call 2018-04-27 13:59:02 -04:00
W. Felix Handte
7d11e34413 Rename LZ4F_applyCDict() -> LZ4F_initStream() 2018-04-27 13:57:10 -04:00
Yann Collet
d294dd7fc6 ensure favorDecSpeed is properly initialized
also :
- fix a potential malloc error
- proper use of ALLOC macro inside lz4hc
- update html API doc
2018-04-27 09:04:09 -07:00
Alexey Tourbin
d81a434c3d lz4.c: fixed the LZ4_decompress_fast_continue case
The change is very similar to that of the LZ4_decompress_safe_continue
case.  The only reason a make this a separate change is to ensure that
the fuzzer, after it's been enhanced, can detect the flaw in
LZ4_decompress_fast_continue, and that the change indeed fixes the flaw.
2018-04-27 15:10:12 +03:00
Yann Collet
0fb3a3b199 fixed a number of minor cast warnings 2018-04-26 18:08:28 -07:00
Yann Collet
00909b27b1
Merge pull request #518 from felixhandte/fix-517-dict-size-truncation
Limit Dictionary Size During LZ4F Decompression
2018-04-26 16:47:50 -07:00
Yann Collet
5c7d3812d9 fasterDecSpeed can be triggered from cli with --favor-decSpeed 2018-04-26 15:49:32 -07:00
Yann Collet
3792d00168 favorDecSpeed feature can be triggered from lz4frame
and lz4hc.
2018-04-26 15:18:44 -07:00
W. Felix Handte
0858362f28 Merge _destSize Compress Variant into LZ4_compress_generic() 2018-04-26 18:01:08 -04:00
W. Felix Handte
a2edeac201 Limit Dictionary Size During LZ4F Decompression
Fixes lz4/lz4#517.
2018-04-26 17:18:40 -04:00
Yann Collet
1148173c5d introduced ability to parse for decompression speed
triggered through an enum.

Now, it's still necessary to properly expose this capability
all the way up to the cli.
2018-04-26 13:01:59 -07:00
Alexey Tourbin
5603d30f81 lz4.c: fixed the LZ4_decompress_safe_continue case
The previous change broke decoding with a ring buffer.  That's because
I didn't realize that the "double dictionary mode" was possible, i.e.
that the decoding routine can look both at the first part of the
dictionary passed as prefix and the second part passed via dictStart+dictSize.

So this change introduces the LZ4_decompress_safe_doubleDict helper,
which handles this "double dictionary" situation.  (This is a bit of
a misnomer, there is only one dictionary, but I can't think of a better
name, and perhaps the designation is not all too bad.)  The helper is
used only once, in LZ4_decompress_safe_continue, it should be inlined
with LZ4_FORCE_O2_GCC_PPC64LE attached to LZ4_decompress_safe_continue.

(Also, in the helper functions, I change the dictStart parameter type
to "const void*", to avoid a cast when calling helpers.  In the helpers,
the upcast to "BYTE*" is still required, for compatibility with C++.)

So this fixes the case of LZ4_decompress_safe_continue, and I'm
surprised by the fact that the fuzzer is now happy and does not detect
a similar problem with LZ4_decompress_fast_continue.  So before fixing
LZ4_decompress_fast_continue, the next logical step is to enhance
the fuzzer.
2018-04-26 08:23:54 +03:00
Alexey Tourbin
b4eda8d08f lz4.c: refactor the decoding routines
I noticed that LZ4_decompress_generic is sometimes instantiated with
identical set of parameters, or (what's worse) with a subtly different
sets of parameters.  For example, LZ4_decompress_fast_withPrefix64k is
instantiated as follows:

    return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize,
		full, 0, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 64 KB);

while the equivalent withPrefix64k call in LZ4_decompress_usingDict_generic
passes 0 for the last argument instead of 64 KB.  It turns out that there
is no difference in this case: if you change 64 KB to 0 KB in
LZ4_decompress_fast_withPrefix64k, you get the same binary code.

Moreover, because it's been clarified that LZ4_decompress_fast doesn't
check match offsets, it is now obvious that both of these fast/withPrefix64k
instantiations are simply redundant.  Exactly because LZ4_decompress_fast
doesn't check offsets, it serves well with any prefixed dictionary.

There's a difference, though, with LZ4_decompress_safe_withPrefix64k.
It also passes 64 KB as the last argument, and if you change that to 0,
as in LZ4_decompress_usingDict_generic, you get a completely different
binary code.  It seems that passing 0 enables offset checking:

    const int checkOffset = ((safeDecode) && (dictSize < (int)(64 KB)));

However, the resulting code seems to run a bit faster.  How come
enabling extra checks can make the code run faster?  Curiouser and
curiouser!  This needs extra study.  Currently I take the view that
the dictSize should be set to non-zero when nothing else will do,
i.e. when passing the external dictionary via dictStart.  Otherwise,
lowPrefix betrays just enough information about the dictionary.

    * * *

Anyway, with this change, I instantiate all the necessary cases as
functions with distinctive names, which also take fewer arguments and
are therefore less error-prone.  I also make the functions non-inline.
(The compiler won't inline the functions because they are used more than
once.  Hence I attach LZ4_FORCE_O2_GCC_PPC64LE to the instances while
removing from the callers.)  The number of instances is now is reduced
from 18 (safe+fast+partial+4*continue+4*prefix+4*dict+2*prefix64+forceExtDict)
down to 7 (safe+fast+partial+2*prefix+2*dict).  The size of the code is
not the only issue here.  Separate helper function are much more
amenable to profile-guided optimization: it is enough to profile only
a few basic functions, while the other less-often used functions, such
as LZ4_decompress_*_continue, will benefit automatically.

This is the list of LZ4_decompress* functions in liblz4.so, sorted by size.
Exported functions are marked with a capital T.

$ nm -S lib/liblz4.so |grep -wi T |grep LZ4_decompress |sort -k2
0000000000016260 0000000000000005 T LZ4_decompress_fast_withPrefix64k
0000000000016dc0 0000000000000025 T LZ4_decompress_fast_usingDict
0000000000016d80 0000000000000040 T LZ4_decompress_safe_usingDict
0000000000016d10 000000000000006b T LZ4_decompress_fast_continue
0000000000016c70 000000000000009f T LZ4_decompress_safe_continue
00000000000156c0 000000000000059c T LZ4_decompress_fast
0000000000014a90 00000000000005fa T LZ4_decompress_safe
0000000000015c60 00000000000005fa T LZ4_decompress_safe_withPrefix64k
0000000000002280 00000000000005fa t LZ4_decompress_safe_withSmallPrefix
0000000000015090 000000000000062f T LZ4_decompress_safe_partial
0000000000002880 00000000000008ea t LZ4_decompress_fast_extDict
0000000000016270 0000000000000993 t LZ4_decompress_safe_forceExtDict
2018-04-25 13:18:06 +03:00
W. Felix Handte
2dfc7cbe82 Change Over Includes in the Project 2018-04-24 16:22:28 -04:00
W. Felix Handte
2be3905fa4 Integrate lz4frame_static.h Declarations into lz4frame.h 2018-04-24 16:22:28 -04:00
Yann Collet
b2637ab7b2
Merge pull request #512 from lz4/HC_dict
In-place unmutable dictionaries for LZ4HC
2018-04-24 13:18:40 -07:00
Yann Collet
8c6ca6283d
Merge pull request #511 from lz4/decFast
Fixed performance issue with LZ4_decompress_fast()
2018-04-24 11:25:57 -07:00
W. Felix Handte
5ed1463bf4 Remove Debug Log Statements 2018-04-24 11:58:51 -04:00
W. Felix Handte
13271a88d7 Revert Stream Size Const to Correct Value 2018-04-24 11:55:53 -04:00
Yann Collet
092cb77597
Merge pull request #504 from baruchsiach/static-only-support
lib: allow to disable shared libraries
2018-04-23 23:44:04 -07:00
Cyan4973
44bff3fd3b re-ordered parenthesis
to avoid mixing && and &
as suggested by @terrelln
2018-04-23 19:26:02 -07:00
Yann Collet
0c2ae72ba8
Merge pull request #507 from lz4/clangPerf
fixed lz4_fast clang performance
2018-04-23 15:55:56 -07:00
Cyan4973
cd0663456f disable shortcut for LZ4_decompress_fast()
improving speed
2018-04-23 15:47:08 -07:00
Cyan4973
bd06fde104 fullbench compiled without assert()
to better reflect release speed
2018-04-23 15:42:27 -07:00
Nick Terrell
672799e814 Fix compilation error and assert. 2018-04-23 14:21:02 -07:00
Nick Terrell
bb83cad98f Fix input size validation edge cases
The bug is a read up to 2 bytes past the end of the buffer.
There are three cases for this bug, one for each test case added.

* An empty input causes `token = *ip++` to read one byte too far.
* A one byte input with `(token >> ML_BITS) == RUN_MASK` causes
  one extra byte to be read without validation. This could be
  combined with the first bug to cause 2 extra bytes to be read.
* The case pointed out in issue #508, where `ip == iend` at the
  beginning of the loop after taking the shortcut.

Benchmarks show no regressions on clang or gcc-7 on both my mac
and devserver.

Fixes #508.
2018-04-23 13:34:18 -07:00